Deep dive · Folio III
In a classic model, every word goes through every "drawer" of the network, even those that have nothing to do with it. What if we added a huge card index in which the model opens only the two or three cards it needs?
A language model stores much of its knowledge in so-called feed-forward layers. They work like a memory, but a dense one: every word passes through all of it, which costs compute every time.
A memory layer replaces (or complements) this with a giant table of key → value pairs. For each word, the model reads only a few values. Billions of parameters can be added without adding compute, like a library that grows without anyone reading more books.
The problem: finding the best key among a million would normally take a million comparisons. The trick of product keys (Lample and co-authors, 2019): cut the query into two halves, compare each with a list of 1,000 half-keys, then cross the best ones. You get the right result among 1,000 × 1,000 = a million slots with only 2,000 comparisons.
The model then reads a few values (around thirty, say), takes a weighted average, and a "gate" decides how much of it to feed into the rest of the computation.
Source: Lample et al., Large Memory Layers with Product Keys (NeurIPS 2019)
Fig. D — Finding the right card without opening them all
Here, 8 + 8 comparisons are enough to find the best slot among 64. With 1,000 + 1,000 comparisons, you choose among a million.
Illustrative diagram: it shows the idea, not measurements.
Promising
Up to 128 billion memory parameters, added to base models of up to 8 billion, trained on 1 trillion tokens. Models with memory beat dense models using more than twice the compute, and mixture-of-experts models at equal compute and parameters. The gain is clearest on factual questions.
Promising
A memory looked up in constant time from short word sequences in the text. At equal size and compute, the 27-billion-parameter model gains +3 points on MMLU, +5 on reasoning (BBH), and goes from 84.2% to 97% on a test of finding information in a long text. The 100-billion-parameter table can live in the computer's ordinary memory, not in the GPU's scarce, expensive memory, for under 3% throughput loss. Switch it off and factual knowledge collapses (TriviaQA falls to 29% of its level): that is where the facts are stored.
Source: DeepSeek, Engram: Conditional Memory via Scalable Lookup (2026)
Promising
Since each fact uses only a few slots, you can update just those. When learning new facts, a classic model retrained in full loses 89% of its performance on other questions, and 71% with the LoRA method. With targeted updates of memory slots: only 11%, for the same learning.
Source: Lin et al., Continual Learning via Sparse Memory Finetuning (Meta, 2025)
Two families of memory are emerging. Fixed tables (Memory Layers, UltraMem, Engram), learned once and for all. And memories written while reading, like Titans.
Titans keeps what surprises it: that is exactly the idea in our hypothesis H4, where exceptions are what is worth remembering. A machine that memorises only the unexpected is a form of "rules + exceptions" compression.
H4 · Exceptions are the anchor (65 %)
Promising
Mixture-of-experts (MoE) models save compute, but in use they have to fetch many parameters from memory, which slows them down. UltraMem (ByteDance, 20 million memory slots) is up to 6 times faster than an MoE of the same size and compute, cuts usage cost by up to 83%, and runs almost as fast as a dense model. Adding slots barely slows it down.
Source: Huang et al., Ultra-Sparse Memory Network (ByteDance, ICLR 2025)
Promising
At Google, long-term memory is a small network that changes its own weights while reading. It writes down what surprises it (measured by the size of its error) and gradually forgets the rest. Classic attention acts as short-term memory. Result: contexts of over 2 million tokens, with better needle-in-a-haystack retrieval than larger models.
Source: Behrouz et al., Titans: Learning to Memorize at Test Time (Google, 2025)
It is the most direct evidence so far that knowledge can be separated from reasoning inside a model itself. With Engram, cutting the memory makes the facts collapse but not everything else; and the memory even frees the network's early layers for reasoning.
It is also a path to savings: the memory can sit in cheap, ordinary RAM, and an update touches only a few slots instead of retraining the whole model.
Reading from a giant table is cheap in compute but demanding in memory bandwidth: a dedicated implementation was needed to make it efficient. Published tests go up to 8 billion base parameters at Meta and 27 billion at DeepSeek. And this memory remains internal and opaque: unlike an external database, you cannot yet read a source in it or cite it.
Strong support for H2 ("learn the top, look up the bottom"), which we raise from 70 to 75% confidence. With Engram, the best split is roughly a quarter of the parameters in memory and three quarters in compute: a first concrete figure for the boundary raised by our open question no. 3.
A mini-model with a memory layer: teach it a list of facts, then correct just one by touching only its slots. Check that it keeps the correction and forgets nothing else, and see whether slots on the same topic cluster together: that would be a hierarchy emerging on its own.