Living notebook · Codex no. 1
What if we organised knowledge before learning it?
This notebook is our whiteboard. We write down leads, hypotheses and provisional conclusions on one question: how to build AI differently, so that it costs less, uses less energy and makes fewer mistakes. First project: a hierarchy of data, for learning and for looking things up.
Today a model is trained on a "flat soup": trillions of words, all at the same level. A passing post and a physics textbook weigh the same. Our starting intuition: part of AI's enormous appetite for data goes into rebuilding a hierarchy that nobody hands it.
Fig. 1 — The pyramid of knowledge
Established
Controlled experiments show that a well-trained language model stores about 2 bits of knowledge per parameter, no more. Every parameter is precious: filling it with details has a cost.
Source: Allen-Zhu & Li, Physics of Language Models 3.3 (2024)
Established
In the same study, when 7 out of 8 training texts are noise, the capacity to retain useful knowledge falls by up to 20 times. Simply adding the source website's name in front of each text clearly helps the model sort. A provenance label is already a small hierarchy.
Source: Allen-Zhu & Li, Physics of Language Models 3.3 (2024)
First way to use a hierarchy: sort and order the training data so the model learns faster, with less. Like a student following a good textbook rather than a pile of loose notes.
Established
phi-1 (1.3 billion parameters), trained on "textbook quality" data, beat at coding models 10 times larger trained on 100 times more data.
Over 200 models trained with data presented from easy to hard reach the baseline level with 18 to 45% fewer steps. An ordered "warm-up" followed by random order keeps up to a 3.5% lead.
Established
Another 2026 study finds these gains depend on size: ordering mostly stabilises small models. Overall, published results on ordered learning remain mixed.
Source: Elgaar & Amiri, Curriculum Learning for LLM Pretraining (2026)
With a good measure of what deserves to be kept, error can fall exponentially with the size of the data kept, instead of following the usual slow law. Shown in theory, and in practice on image tasks.
Source: Sorscher et al., Beyond neural scaling laws (NeurIPS 2022)
Meta's Large Concept Models reason sentence by sentence, in a meaning space shared by 200 languages: an explicit hierarchy from words to ideas. For now they remain less fluent than classic models.
Established
Training a model on what the previous one produced, replacing the real data, impoverishes it: rare cases disappear first. On the other hand, accumulating real and synthetic data avoids this collapse.
Source: Shumailov et al., AI models collapse… (Nature, 2024), Gerstgrasser et al., Is Model Collapse Inevitable? (2024)
Second way: keep knowledge outside the model, arranged in levels like a library (section, shelf, book, page). The model only goes as deep as it needs to answer.
Fig. 2 — Searching by going down
16 documents · 4 steps
Established
RETRO (DeepMind) draws on a 2-trillion-token database while it writes. It matches GPT-3 with 25 times fewer parameters.
Established
RAPTOR groups the passages of a text, summarises them, then summarises the summaries, until it forms a tree. Paired with GPT-4, it gains 20 points of accuracy on a hard question benchmark (QuALITY).
Established
GraphRAG (Microsoft) links a corpus's entities into a graph, splits it into nested communities and summarises each level. For questions about the whole corpus it beats classic retrieval in 70 to 80% of comparisons. The trade-off: indexing is expensive up front.
Promising
Meta's "memory layers" add millions of key-value pairs looked up on demand: more knowledge without more compute. On factual questions, accuracy more than doubles.
Promising
Recent hierarchical memories for agents narrow the search level by level: the cost grows roughly like the logarithm of the memory size. A 2026 theory shows these systems all follow the same pattern: group, compress, expand on demand.
Source: Talebirad et al., Toward a Theory of Hierarchical Memory (2026), Hsu et al., HORMA, Organize then Retrieve (2026)
What follows is our own reasoning, not a published result. The confidence figure says how much we believe it today; it will move.
H1
If we gave the model the structure of the data (level, provenance, dependencies), it would need fewer examples to rebuild it.
Source: Allen-Zhu & Li, Physics of Language Models 3.3 (2024), Zhang et al., Beyond Random Sampling (EACL 2026)
H2
Since parameters are scarce (about 2 bits each), let's reserve them for principles and ways of reasoning. Facts and details go into an external memory arranged in levels.
Source: Borgeaud et al., RETRO (DeepMind, 2021), Berges et al., Memory Layers at Scale (Meta, 2024)
H3
Just as a physicist remembers the law and the cases that contradict it, we would keep the rules and only the data that escapes them. The rest could be derived.
Source: Sorscher et al., Beyond neural scaling laws (NeurIPS 2022)
H4
In a loop where each model generation learns from knowledge compressed by the previous one, keeping the rare raw data would prevent drift.
Source: Shumailov et al., AI models collapse… (Nature, 2024), Gerstgrasser et al., Is Model Collapse Inevitable? (2024)
H5
Real knowledge is a network: the same idea belongs to several families. The useful hierarchy would be several trees that cross each other.
Source: Edge et al., GraphRAG (Microsoft, 2024), Sarthi et al., RAPTOR (ICLR 2024)
What we think at this stage. "Provisional" is the key word.
External databases, trees of summaries, community graphs: the gains are measured and reproduced. This is the most mature side of the question.
Source: Borgeaud et al., RETRO (DeepMind, 2021), Sarthi et al., RAPTOR (ICLR 2024), Edge et al., GraphRAG (Microsoft, 2024)
Ordering data helps a little; choosing it well helps much more. Training a model on an explicit hierarchy remains almost unexplored: that is where there is most to discover.
Source: Zhang et al., Beyond Random Sampling (EACL 2026), Elgaar & Amiri, Curriculum Learning for LLM Pretraining (2026), Gunasekar et al., Textbooks Are All You Need (2023)
Principles in the weights, details in the memory. This is hypothesis H2, the one most evidence points to.
Hand-written hierarchies, like the Cyc project started in 1984, never scaled. Richard Sutton's "bitter lesson": what learns on its own ends up winning. So the hierarchy should be learned, then made visible, not dictated.
Source: Richard Sutton, The Bitter Lesson (2019), Cyc (Wikipedia)
Fig. 3 — The proposed split
Inside the model
In the memory