AI of tomorrow
EN
FrançaisEspañolItalianoDeutschPortuguês日本語한국어

← Back to the notebook

The notebook's library

Sources

Every reference in the notebook, grouped by the part where it first appears, with the cards that cite it.

55 sources updated

I

Where we start

  1. Allen-Zhu & Li, Physics of Language Models 3.3 (2024)

    Cited in: A model's memory is limitedNoise is expensive, labels helpH1 · The appetite for data comes partly from a missing hierarchy

II

A hierarchy for learning

  1. Gunasekar et al., Textbooks Are All You Need (2023)

    Cited in: Quality beats quantityFor learning, it is the frontier

  2. Zhang et al., Beyond Random Sampling (EACL 2026)

    Cited in: Learning in order helps…H1 · The appetite for data comes partly from a missing hierarchyFor learning, it is the frontier↳ Learning in order

  3. Elgaar & Amiri, Curriculum Learning for LLM Pretraining (2026)

    Cited in: … but not everywhereFor learning, it is the frontier↳ Learning in order

  4. Sorscher et al., Beyond neural scaling laws (NeurIPS 2022)

    Cited in: Throwing away wiselyH3 · Rules + exceptions = compression↳ Throwing away wisely

  5. Meta, Large Concept Models (2024)

    Cited in: Thinking in ideas, not words↳ Thinking in ideas, not words

  6. Shumailov et al., AI models collapse… (Nature, 2024)

    Cited in: The photocopy trapH4 · Exceptions are the anchor

  7. Gerstgrasser et al., Is Model Collapse Inevitable? (2024)

    Cited in: The photocopy trapH4 · Exceptions are the anchor

III

A hierarchy for looking up

  1. Borgeaud et al., RETRO (DeepMind, 2021)

    Cited in: Looking up rather than memorisingKeep knowledge outside the weightsH2 · Learn the top, look up the bottomFor looking up, hierarchy has proven itself

  2. Sarthi et al., RAPTOR (ICLR 2024)

    Cited in: A tree of summariesH5 · A graph, not a treeFor looking up, hierarchy has proven itself

  3. Edge et al., GraphRAG (Microsoft, 2024)

    Cited in: Communities of ideasH5 · A graph, not a treeFor looking up, hierarchy has proven itself

  4. Berges et al., Memory Layers at Scale (Meta, 2024)

    Cited in: A memory inside the modelKeep knowledge outside the weightsH2 · Learn the top, look up the bottom↳ A memory inside the model

  5. Talebirad et al., Toward a Theory of Hierarchical Memory (2026)

    Cited in: The cost becomes logarithmic↳ The cost becomes logarithmic

  6. Hsu et al., HORMA, Organize then Retrieve (2026)

    Cited in: The cost becomes logarithmic↳ The cost becomes logarithmic

IV

Seven other leads

  1. DeepSeek-AI, DeepSeek-V3 Technical Report (2024)

    Cited in: Switch on only part of the modelH6 · The savings multiply↳ Switch on only part of the model↳ Compute with fewer bits

  2. Raposo et al., Mixture-of-Depths (2024)

    Cited in: Switch on only part of the model↳ Switch on only part of the model

  3. Chen, Zaharia & Zou, FrugalGPT (2023)

    Cited in: Ask the small model firstH6 · The savings multiply↳ Ask the small model first

  4. Ong et al., RouteLLM (ICLR 2025)

    Cited in: Ask the small model first↳ Ask the small model first

  5. Cai et al., Large Language Models as Tool Makers (ICLR 2024)

    Cited in: Distil into programs, not models↳ Distil into programs, not models

  6. Google DeepMind, AlphaEvolve (2025)

    Cited in: Distil into programs, not models↳ Distil into programs, not models

  7. Ma et al., The Era of 1-bit LLMs: BitNet b1.58 (2024)

    Cited in: Compute with fewer bitsH6 · The savings multiply↳ Compute with fewer bits

  8. NVIDIA, Pretraining Large Language Models with NVFP4 (2025)

    Cited in: Compute with fewer bits↳ Compute with fewer bits

  9. Modha et al., NorthPole, Science (2023)

    Cited in: Change the machineThe real cost is moving data↳ Change the machine

  10. Ahmed et al., Universal photonic AI acceleration, Nature (2025)

    Cited in: Change the machine↳ Change the machine

  11. Leviathan, Kalman & Matias, Speculative Decoding (ICML 2023)

    Cited in: Checking is cheaper than writing↳ Ask the small model first↳ Checking is cheaper than writing

  12. Brown et al., Large Language Monkeys (2024)

    Cited in: Checking is cheaper than writing↳ Checking is cheaper than writing

V

Hypotheses and conclusions

  1. DeepSeek, Engram: Conditional Memory via Scalable Lookup (2026)

    Cited in: H2 · Learn the top, look up the bottom↳ A memory inside the model

  2. Chhikara et al., Mem0: Scalable Long-Term Memory (2025)

    Cited in: H5 · A graph, not a tree↳ The cost becomes logarithmic

  3. Xu et al., A-MEM: Agentic Memory for LLM Agents (NeurIPS 2025)

    Cited in: H5 · A graph, not a tree↳ The cost becomes logarithmic

  4. Horowitz, Computing's Energy Problem (ISSCC 2014)

    Cited in: H6 · The savings multiplyThe real cost is moving data↳ Compute with fewer bits↳ Change the machine

  5. Richard Sutton, The Bitter Lesson (2019)

    Cited in: History calls for caution↳ Thinking in ideas, not words

  6. Cyc (Wikipedia)

    Cited in: History calls for caution

VI

Deep dives

  1. Xie et al., DoReMi: Optimizing Data Mixtures (NeurIPS 2023)

    Cited in: ↳ Learning in order

  2. Lin et al., Rho-1: Not All Tokens Are What You Need (NeurIPS 2024)

    Cited in: ↳ Learning in order↳ Throwing away wisely

  3. OLMo et al., 2 OLMo 2 Furious (Ai2, 2025)

    Cited in: ↳ Learning in order

  4. Predicting Training Re-evaluation Curves (2025)

    Cited in: ↳ Learning in order

  5. Abbas et al., SemDeDup (Meta, 2023)

    Cited in: ↳ Throwing away wisely

  6. Penedo et al., The FineWeb Datasets (Hugging Face, 2024)

    Cited in: ↳ Throwing away wisely

  7. Li et al., DataComp-LM (2024)

    Cited in: ↳ Throwing away wisely

  8. Muennighoff et al., Scaling Data-Constrained Language Models (NeurIPS 2023)

    Cited in: ↳ Throwing away wisely

  9. Lample et al., Large Memory Layers with Product Keys (NeurIPS 2019)

    Cited in: ↳ A memory inside the model

  10. Lin et al., Continual Learning via Sparse Memory Finetuning (Meta, 2025)

    Cited in: ↳ A memory inside the model

  11. Huang et al., Ultra-Sparse Memory Network (ByteDance, ICLR 2025)

    Cited in: ↳ A memory inside the model

  12. Behrouz et al., Titans: Learning to Memorize at Test Time (Google, 2025)

    Cited in: ↳ A memory inside the model

  13. Packer et al., MemGPT: Towards LLMs as Operating Systems (2023)

    Cited in: ↳ The cost becomes logarithmic

  14. Rezazadeh et al., MemTree: Dynamic Tree Memory (2024)

    Cited in: ↳ The cost becomes logarithmic

  15. MemForest: Hierarchical Temporal Indexing for Agent Memory (2026)

    Cited in: ↳ The cost becomes logarithmic

  16. Pagnoni et al., Byte Latent Transformer (Meta, ACL 2025)

    Cited in: ↳ Thinking in ideas, not words

  17. Hwang, Wang & Gu, H-Net: Dynamic Chunking (2025)

    Cited in: ↳ Thinking in ideas, not words

  18. Wang et al., Hierarchical Reasoning Model (Sapient, 2025)

    Cited in: ↳ Thinking in ideas, not words

  19. ARC Prize, The Hidden Drivers of HRM's Performance (2025)

    Cited in: ↳ Thinking in ideas, not words

  20. Jolicoeur-Martineau, Less is More: Tiny Recursive Model (Samsung, 2025)

    Cited in: ↳ Thinking in ideas, not words

  21. Bae et al., Mixture-of-Recursions (NeurIPS 2025)

    Cited in: ↳ Switch on only part of the model

  22. Ambrogio et al., An analog-AI chip for speech recognition, Nature (2023)

    Cited in: ↳ Change the machine

  23. Li et al., EAGLE-3 (2025)

    Cited in: ↳ Checking is cheaper than writing

Back to top