A memory you can search
October 2026 / Bulbul experiment notes
Can a small language model keep useful information outside its weights, make that memory searchable, and use less space?
This experiment gets part of the way there: a 494-million-parameter model plus 5,183 searchable passages in 663.5 MB. A separate recall test checks whether retrieved memory actually changes its answers.
01 / The idea
Give the model a library it can look through.
A transformer turns tokens into keys, which help attention find relevant information, and values, which carry information forward. Usually these intermediate vectors live inside a running model’s cache. Here we also save them as a persistent, inspectable memory.
The useful possibility is a small model with a source library you can update without retraining its weights. The experiment asks two separate questions: can we search that library, and can the model use what we retrieve?
The starting point was Memorizing Transformers. The working prototype below uses a pretrained Qwen model, key-based search, and a separate prefix-cache experiment. It is not a reproduction of the paper’s trained memory layer.
02 / The architecture
Store once. Search when needed.
- 01Source passages
Up to 256 tokens each, with source IDs.
- 02Small Qwen model
Capture projected keys and values at layer 11.
- 03Pack + save
4-bit K/V, source text, token positions, keyword index.
- 01Find candidates
BM25 keyword search picks 50 passages.
- 02Compare keys
The same model encodes the query and reranks candidates.
- 03Inspect the source
Return passages and their stored token K/V pairs.
Only candidate keys need unpacking. The query and document encoders must use matching weights; changing the model means rebuilding its index. The local prototype exposes a browser search box and an HTTP API, so other tools can query the same library.
03 / The experiment
Compress the representation. Count the whole package.
We used Qwen2.5-0.5B. Its decoder matrices use 4-bit storage, with an 8-bit shared embedding/output matrix and calibration-based error compensation inspired by GPTQ. Keys and values in the search index use 4 bits each. Source text is stored once; unused token padding is removed.
59.7% less disk storage. Includes source text, actual token keys and values, and search metadata for 5,183 passages. MB = 1,000,000 bytes. Both sides exclude unused token padding.
The model still has 494,032,768 parameters. These savings come from lower precision and tighter storage. They do not establish an equivalent reduction in working RAM, GPU memory, or latency: this implementation unpacks weights and active prefixes into floating point.
| Check | Recorded result |
|---|---|
| Language modeling WikiText regression | Perplexity +4.34%Lower is better. 95% interval: +3.84–4.78%. |
| Question answering 256 fresh ARC-Easy questions | 56.25% → 56.25%Same accuracy; difference interval −2.73 to +2.74 percentage points. |
| Passage search 298 SciFact regression queries | 0.6629 → 0.6695Float → 4-bit keys, same quantized encoder. nDCG@10; higher is better. |
nDCG measures whether relevant passages appear near the top. The 4-bit search audit was done after inspecting the test cohort; its small score increase is not evidence that compression improves retrieval. Model quality is compared with the original FP32 model; FP16 above is a storage equivalent.
Some smaller versions failed.
Naive all-2-bit weights broke language modeling. Two-bit search keys lost more relevance than the preset tolerance allowed. For generation prefixes, packing every key per token badly damaged continuation quality. Grouping keys across tokens and values across channels, inspired by KIVI, worked much better. The layout matters as much as the bit count.
04 / Does it use the memory?
Take the source away.
We created 256 new station names with randomly assigned four-digit codes, froze the configuration, and asked 128 questions. The combined setup used the compressed model, matching search keys, and 4-bit all-layer prefix caches.
128/128 exact answers
Search finds the source. The model reads its compressed, all-layer prefix and completes the four-digit code.
Recorded aggregate results, not live inference. Each condition uses the same 128 questions. The 95% interval for 128/128 is 97.1–100%.
That is evidence that retrieved memory supplies the answer in this controlled task. The prompt follows a fixed station-code grammar; it does not demonstrate general reasoning. This small fact bank is separate from the 5,183-passage, 663.5 MB search package.
What about data leakage?
Calibration used training text. ARC questions and answer labels were kept out of retrieval memory; the 256 questions above were unused in earlier local experiments. The recall task is explicitly open-book: source facts are allowed, evaluation questions and labels are not indexed.
WikiText and SciFact were already inspected, so their scores are regression measurements. The search precision audit is post-hoc. Public-model pretraining contamination remains unknown, and these custom subsets are not canonical leaderboard scores. Equal observed accuracy is not proof of equal capability.
05 / Making it useful
Start with a local reference library.
The clearest next application is a searchable collection of papers, manuals, or project notes. A small model helps rank passages, while source IDs let a person inspect where each result came from. New documents require indexing, not model retraining.
Repeatedly used documents are also candidates for saved generation prefixes: encode stable context once, then load the relevant prefix when needed. Whether this beats retrieving text and encoding it normally still needs a latency and memory comparison on the target device.
- Bring a real corpus. Split it into short passages, preserve source links, and build the index with the exact model used for queries.
- Compare against ordinary retrieval. Use unseen questions, keyword and embedding baselines, and questions the sources cannot answer. Keep wrong-memory and no-memory controls.
- Measure the device cost. Count the model, index, and any full prefix bank. Then measure peak working memory and end-to-end response time.
The promising part is being able to store, inspect, and search useful source memory cheaply. A production version still needs reliable document updates, access controls, and a way to abstain when the library has no answer.