AI of tomorrow Glossary
EN

← Back to the notebook

Deep dive · Hypotheses

Putting the hypotheses to the test

Until now, each hypothesis carried a single number, chosen in one go. A single number hides where the doubt lies. Here we break each one into links that must all hold, score each link, multiply, and check the whole against the studies published since. Four hypotheses drop clearly, three rise, two are reworded. Here is the reasoning, number by number.

Speculative updated

I

Why reopen the hypotheses

The notebook has nine hypotheses. Each states something we believe about tomorrow's AI, with a level of confidence: 75%, 40%, 55%… These numbers were honest, but global: we looked at the hypothesis as a whole and asked "is this likely?".

The problem is that a hypothesis is rarely a single claim. It is a chain: A must be true, and B must follow, and C must hold in practice. A weather forecaster does not say "70% chance of rain tomorrow" by instinct: they combine the chance that a front arrives, that it is wet enough, that it does not veer off. That is what we do here, for each of the nine.

II

The method: break it into links

Step 1. Write down the claims the hypothesis needs in order to be true, and only those. These are links a, b, c.

Step 2. Score each link assuming the earlier ones are true. This is called a conditional probability: "if a is true, what is the chance of b?". The detail matters: links in the same chain often support each other, and scoring them separately would wrongly drag the total down.

Step 3. Multiply. If a is worth 90% and b, given a, 50%, the chain is worth 0.9 × 0.5 = 45%. A product is always smaller than its smallest factor: a chain cannot be stronger than its weakest link.

Two traps lie in wait, pulling in opposite directions. The first: a rich, coherent story seems more likely than a simple one, when it is necessarily less likely, since it requires more things. That was our flaw with global numbers. The second: cutting into too many links, each scored cautiously, melts the total artificially. So we stick to two or three links per hypothesis, each one truly necessary.

One last rule, about consistency: if one hypothesis implies another, the first cannot be more likely than the second. A special case cannot be surer than the general rule it follows from. This rule is what led us to our first mistake.

III

First finding: two hypotheses contradicted each other

H7 said "the judge can teach": a model improves by trying and being graded. We gave it 50%. H9, added three days ago, said a small model improves on an ordinary computer thanks to an outside judge. We gave it 55%.

But H9 is a special case of H7: if a judge makes a small model improve on a PC, then a judge can teach. The special case could not be more likely than the general one. One of the two numbers was wrong.

Digging in, the contradiction came from one word. "Teach" mixed two very different things. Sharpening: making reliable what the model found only sometimes. That is settled; Yue and colleagues measure it, every study confirms it. Extending: making the model find what it never found. That is what is debated.

So we correct H7. Sharpening becomes a fact, no longer a hypothesis. H7 is now about extension, the real open question. H9 stays focused on sharpening, and the two numbers no longer conflict.

Source: Yue et al., Does RL Really Incentivize Reasoning Capacity Beyond the Base Model? (NeurIPS 2025)

IV

H1 · The missing hierarchy: from 50% to 55%

What H1 says: giving the model the structure of the data (provenance, level, dependencies) would reduce the number of examples it needs.

Link a, 85%: structural cues really save data, at least up to a few billion parameters. Link b, 65%: the gain does not fade at large sizes. Gains from the order of learning shrank with size, hence our caution; but the result below holds from 600 million to 8 billion parameters.

Calculation: 0.85 × 0.65 = 55%. H1 rises a little. What remains open: is it the hierarchy that helps, or a simple label of origin? The study uses a flat label, the site address. Our H1 includes provenance, so it counts; but the most ambitious form, with levels and dependencies, is still to be tested.

Established

New: the site address as a cue (2025)

Gao and colleagues, at Princeton, simply add the source address (for example en.wikipedia.org) in front of each text during training, then remove it at the end. A 1.6-billion-parameter model matches standard training with 33% less data, with no extra computation, from 600 million to 8 billion parameters. Topics generated by a model also work as labels.

Source: Gao et al., Metadata Conditioning Accelerates Language Model Pre-training (2025)

V

H2 · Learn the top, look up the bottom: from 75% to 64%

What H2 says: keep the parameters for principles, and store facts in an external memory.

Link a, 92%: facts can be moved out of the model with no loss on knowledge questions. RETRO, memory layers, Engram, and now HippoRAG 2 show it. Link b, 82%: the freed space really goes to better reasoning, at equal budget. Engram gains 3 points, but partly on knowledge questions: a clue, not a proof. Link c, 85%: the cost of looking things up (time, memory traffic) stays acceptable in real use.

Calculation: 0.92 × 0.82 × 0.85 = 64%. This is the biggest lesson of method: each link is solid, but three solid links make an average chain. Our global 75% looked at link a, the most convincing one, and forgot the other two.

Source: DeepSeek, Engram: Conditional Memory via Scalable Lookup (2026), Berges et al., Memory Layers at Scale (Meta, 2024)

VI

H3 · Rules and exceptions: from 40% to 48%, by narrowing it

What H3 said: everywhere, keeping the rules and only the data that escape them would compress knowledge.

Broken down, the general form collapses. Link a, 80%: where a program can check, we know how to draw rules from examples (AlphaEvolve, FunSearch). Link b, 60%: keeping rules + exceptions really saves training data; choosing data well changes the shape of the learning curve, which is encouraging. Link c, 45%: this extends to everyday language, full of exceptions and nuance. Product: 0.8 × 0.6 × 0.45 = 22%.

Rather than keep an over-broad hypothesis, we narrow it to checkable domains, where our reasons to believe are real: 0.8 × 0.6 = 48%. Link c does not disappear: it becomes a question for later. Narrowing is not cheating, as long as we say so: a narrower hypothesis claims less, so it is more likely to be true.

Source: Google DeepMind, AlphaEvolve (2025), Sorscher et al., Beyond neural scaling laws (NeurIPS 2022)

VII

H4 · Exceptions as the anchor: from 70% to 44%

What H4 says: when each generation of model learns from the previous one, keeping the rare raw data would prevent drift.

Link a, 90%: drift hits rare cases first. Settled since Shumailov. Link b, 75%: keeping real data prevents drift. Gerstgrasser shows that accumulating rather than replacing avoids it; but a 2025 study shows that a small share of artificial data can be enough to stall progress. Link c, 65%: keeping only the rare cases, not everything, is enough. This is our frugal version, and nobody has tested it.

Calculation: 0.9 × 0.75 × 0.65 = 44%. The biggest drop in the notebook. Our 70% rested on link a, very solid, and assumed the other two. Let us run the reverse test: is it more likely than not that keeping only the exceptions falls short? Yes, slightly, as long as nobody has tried. That is exactly what 44% says.

Source: Shumailov et al., AI models collapse… (Nature, 2024), Gerstgrasser et al., Is Model Collapse Inevitable? (2024)

Established

New: strong collapse (2025)

Dohmatob and colleagues show, in a simplified mathematical setting and then on language models, that even 1% of artificial data in training can stop the model from benefiting from more data: the learning curve stops going down. And larger models can amplify the effect. So mixing is not always enough: you have to know what to keep.

Source: Dohmatob et al., Strong Model Collapse (ICLR 2025)

VIII

H5 · A graph, not a tree: from 65% to 64%

What H5 says: knowledge is a network, an idea belongs to several families; useful memory would be made of trees that cross.

Link a, 85%: structures where a passage belongs to several groups beat simple trees on questions that connect several facts. GraphRAG, RAPTOR and HippoRAG 2 point that way. Link b, 75%: the cost of building and maintaining them stays reasonable at large scale.

Calculation: 0.85 × 0.75 = 64%. Almost unchanged: our global number was already well calibrated. A good sign for the method, which does not lower everything it touches.

Source: Edge et al., GraphRAG (Microsoft, 2024), Sarthi et al., RAPTOR (ICLR 2024)

IX

H6 · The gains multiply: from 35% to 42%

What H6 says: by stacking experts, cascades, low precision and better hardware, the cost of an answer would drop 100-fold without a new giant model.

Link a, 95%: each lever alone divides the cost by 2 to 4. Link b, 55%: the levers are independent enough for their product to hold, within a factor of 3. This is the weak link: Amdahl's law says that speeding up one part only helps for that part. Link c, 80%: 100 is reached in practice.

Calculation: 0.95 × 0.55 × 0.8 = 42%. Why raise b compared with our synthesis? Because the real world ran the experiment for us: at equal performance, prices fell far faster than any single lever explains. That is indirect evidence that the levers stack. But those drops also come from new, better-trained models, which H6 excludes: hence link b at 55%, no more.

Established

New: the measured fall in prices (2025)

Epoch AI measures that, at fixed performance, the price of an answer falls 9 to 900 times a year depending on the task; reaching GPT-4's level on PhD-level science questions got 40 times cheaper each year. Another, more cautious study finds 5 to 10 times a year for frontier models, and notes that using the frontier models is instead getting more expensive, because they are bigger and reason for longer.

Source: Epoch AI, LLM inference prices have fallen rapidly but unequally across tasks (2025), The Price of Progress: Algorithmic Efficiency and the Falling Cost of AI Inference (2025)

X

H7 · Can the judge make the model discover? 43%, reworded

What H7 now says: with long and varied training, a reliable judge would make the model find solutions it never found, even in thousands of tries.

Link a, 95%: a judge makes the model more reliable. Settled. Link b, 45%: with longer training, it extends what the model can find. Two teams, often the same researchers, answer each other. One shows real extension, especially in code; the other shows that in general the model loses access to more solutions than it gains. A telling detail: the title of the second study acquired a question mark in its latest version.

Calculation: 0.95 × 0.45 = 43%. Lower than the old 50%, but for a stronger and clearer claim. This is the hypothesis that matters most for the rest of the notebook: if it falls, a small model will never go beyond, on its own, what it already found by chance.

Source: Yue et al., Does RL Really Incentivize Reasoning Capacity Beyond the Base Model? (NeurIPS 2025)

Promising

For: ProRL, prolonged training (2025)

At NVIDIA, Liu and colleagues train a 1.5-billion-parameter model for a long time, stabilising the learning and varying the tasks. The model solves problems where the starting model failed however many tries it had. But the authors note it themselves: in maths, where the starting model was already strong, success over 128 tries often goes down.

Source: Liu et al., ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries (NeurIPS 2025)

Promising

Against: the invisible leash (2025)

Wu and colleagues, including several ProRL authors, follow training step by step. Their finding across many tests: reinforcement improves the first try, but loses access to more correct answers than it gains. The model stays leashed to its starting point; breaking free would require seeding ideas in the regions it never explores.

Source: Wu et al., The Invisible Leash? Why RLVR May or May Not Escape Its Origin (2025)

XI

H8 · The well-supported small model: from 55% to 46%

What H8 says: on checkable tasks, a well-supported 8-billion-parameter model would match a model a hundred times larger, at comparable or lower cost.

Link a, 90%: it reaches the same score. Established in maths: a 1-billion-parameter model with 512 sorted tries beats a 405-billion-parameter model. Link b, 60%: at comparable total cost, counting every try and the judge, against mixture-of-experts giants that only switch on a small part of themselves. Link c, 85%: this holds beyond maths and code, on other checkable tasks.

Calculation: 0.9 × 0.6 × 0.85 = 46%. Link b is the heart of the doubt, and our old 55% glossed over it. One new piece of data helps the small model, though: what you can run at home catches up quickly with the giants.

Source: Liu et al., Can 1B LLM Surpass 405B LLM? (2025)

XII

H9 · The world as teacher: from 55% to 54%

What H9 says: on an ordinary computer, an outside judge makes a small model improve clearly, up to the ceiling of what it found over many tries.

Link a, 75%: the gain is clear (at least 5 points) on any modern small model, not just a chosen family. Link b, 90%: it fits on a gaming card in a week; these are our calculations from the previous page. Link c, 80%: the description of the ceiling is right; the invisible leash confirms it, ProRL challenges it, but after training far longer than a week.

Calculation: 0.75 × 0.9 × 0.8 = 54%. Almost unchanged, but the doubt has moved: it now sits in link a, because of the study below. Our planned experiment uses Llama, precisely the family where the easy effects vanish. All the better: it will be all the more convincing.

Established

New: random rewards that work… on one family only (2026)

Shao and colleagues train Qwen2.5-Math-7B by reinforcement with rewards drawn at random: it gains 21.4 points on the MATH-500 test, almost as much as the 29.1 points obtained with the real answers. Reinforcement mostly awakens a habit that was already there (reasoning in code). On Llama 3 or OLMo 2, these fake rewards give almost nothing. Lesson: test on several families before declaring victory.

Source: Shao et al., Spurious Rewards: Rethinking Training Signals in RLVR (ICML 2026)

XIII

The overall picture

What goes down: H2 (75 → 64), H4 (70 → 44), H8 (55 → 46), H7 (50 → 43, reworded), and H9 by one point. Almost always for the same reason: we had scored the most visible link and assumed the others.

What goes up: H1 (50 → 55) thanks to a new study, H6 (35 → 42) thanks to the measured fall in prices, H3 (40 → 48) by narrowing it. H5 barely moves.

The notebook's big thesis rests on three hypotheses: an external memory (H2), organised as a graph (H5), and a small model that improves with a judge (H9). If they were independent: 0.64 × 0.64 × 0.54 = 22%. They are not quite independent (the same technical reality underlies them), so the true value is higher, without exceeding the weakest, 54%. We say about 30%: less than an even chance that the whole programme holds. That is why we need to settle it, link by link.

Fig. O — Where the doubt lies

confidence today (product of the links)previous confidencelinks a, b, c

Illustrative diagram: it shows the idea, not measurements.

Each row is a hypothesis. The circles are its links, each scored assuming the earlier ones are true; the bar is their product, that is, the new confidence; the red tick, the old one. A hypothesis is as fragile as its weakest link, and a little more: the bar always ends to the left of the leftmost circle.
XIV

Three experiments to settle it

Which experiment would move our numbers the most? For each, we write down its possible outcomes, their probability and the confidence that would follow. A consistency rule applies: the average of the confidences after the experiment must equal the confidence before. Otherwise we would already know which way it will fall, and we should change our number right away. We checked this rule for all three.

We then measure the expected shift: the probability-weighted sum of the changes in confidence. The bigger it is, the more the experiment teaches us.

Speculative

E1 · Two families, real judge versus random judge · about €20

The experiment from the previous page, run on Llama and Qwen, with real rewards and random rewards: four weeks of a gaming card. Three outcomes: Llama improves clearly with the real judge only, ceiling intact (55%) → H9 66%, H7 37%; only Qwen improves (25%) → H9 25%, H7 38%; Llama improves and its success over 64 tries jumps (20%) → H9 60%, H7 65%. Averages: H9 54.6%, H7 42.9%, consistent. Expected shift: 24 points.

Speculative

E2 · Hierarchical labels versus a flat label · about 16,000 GPU hours

Train 1- and 8-billion-parameter models with no label, with the source address, and with levels and dependencies. Three outcomes: the hierarchy beats the flat label and holds at 8 billion (50%) → H1 75%; only the flat label helps and its gain melts (30%) → H1 40%; nothing left at 8 billion (20%) → H1 25%. Average 54.5%, consistent. Expected shift: about 20 points. Cost: 6 operations per parameter and per word, over 20 words per parameter, three times, at 400 trillion useful operations per second per chip (an assumption): about 16,000 data-centre GPU hours. A job for a lab.

Speculative

E3 · Five generations, keep everything, the rare cases or nothing · about 50 GPU hours

Small models (125 million parameters) each learn from the previous one, five times in a row, keeping all the real data, only the rarest 5%, or nothing. Three outcomes: the rare 5% do as well as everything (40%) → H4 75%; they help halfway (45%) → H4 25%; even keeping everything is not enough (15%) → H4 20%. Average 44.3%, consistent. Expected shift: about 25 points, for some fifty rented GPU hours (our estimate, including generating the texts).

XV

What we will do with these numbers

The order of the experiments follows from the calculation: E1 and E3 teach at least as much as E2, for hundreds to thousands of times less. E1 fits on an ordinary computer; E3, on a few hours of rented computing. These are the ones we recommend running first.

Our revision rules, so that we do not cheat later. A hypothesis only changes its number if a link changes, and we will say which. A hypothesis with a link that falls below 20% will be reworded or withdrawn, not kept out of habit. And every new hypothesis will arrive already broken into links.

What this page gains is not one more number. It is knowing, for each hypothesis, which precise sentence would have to be proved or refuted. A hypothesis you know how to kill is a useful hypothesis.

Back to top