AI of tomorrow Glossary
EN

← Back to the notebook

Deep dive · Folio V

A small model that improves on its own, on an ordinary computer?

On the previous page, a giant served as the small model's teacher. Here we take it away. What is left is a model with 8 billion parameters, a gaming computer and free nights. Our answer fits in one sentence: the teacher does not need to be smarter than the student, it only needs to tell right from wrong. Here are the reasoning, the calculations, and the ceiling that is not so easy to break through.

Promising updated

I

The idea, simply

Think of a student alone with a workbook and the answer key. Nobody explains anything. They try, check against the key, remember what worked, and start again. After a few weeks, they solve on the first try exercises they used to solve only one time in five.

A language model can do exactly the same. It tries the same problem several times, a judge sorts out the right attempts, and the model trains on its own successes. This is self-training. The STaR method showed it as early as 2022, and ReST-EM, at Google, did better by learning from its own sorted attempts than by learning from solutions written by humans.

The key point: the judge is not a big model. It is the answer key, a test, a rule of the game. Checking is much easier than finding. Checking that a sum is right is instant; finding the proof of a theorem can take years. This asymmetry lets a "dumb" teacher teach a student smarter than itself.

Source: Zelikman et al., STaR: Bootstrapping Reasoning With Reasoning (2022), Singh et al., Beyond Human Data: Scaling Self-Training for Problem-Solving (2023)

II

Why it can work: the hidden margin

A model has two scores. The first is its success on the first try. The second is its success over many tries: the chance that at least one of its attempts is right. Researchers write pass@1 and pass@k. The gap between the two can be huge: in Brown and colleagues' work, the share of problems solved climbs steadily with each doubling of the number of tries.

That gap is hidden skill: the model "knows" how to find the answer, but not reliably. Self-training turns hidden skill into reliable skill. It creates almost nothing new; it makes sure what used to be hit-or-miss.

That is also its limit, and we will come back to it: Yue and colleagues observe that reinforcement raises pass@1, but that with many tries, the starting model solves as many problems, or even more. Keep the picture in mind: training fills the glass, it does not make it bigger. Good news for an old model like Llama 3 8B: less reliable than a recent model, it often has an emptier glass, so more room to fill.

Source: Brown et al., Large Language Monkeys (2024), Yue et al., Does RL Really Incentivize Reasoning Capacity Beyond the Base Model? (NeurIPS 2025)

III

Who says "right" when there is no teacher?

Everything rests on the judge. Here are the judges an ordinary computer can run on its own, from the safest to the most fragile.

Promising

The test: run the code and look

For code, the judge is the computer itself: run the program and see whether it passes the tests. Absolute Zero goes further: the model invents its own coding exercises, a Python interpreter checks them, and it learns with no outside data at all. Its authors report doing better than models trained on tens of thousands of examples chosen by humans.

Source: Zhao et al., Absolute Zero: Reinforced Self-play Reasoning with Zero Data (2025)

Promising

The rule of the game: a problem that checks itself

The game Countdown: reach a target number with a few numbers and the four operations. Checking a solution takes one line of arithmetic. TinyZero, at Berkeley, taught this game to a 3-billion-parameter model through reinforcement, and the model developed on its own the habit of checking its calculations and searching differently when it goes wrong. All for less than 30 dollars of computing.

Source: Pan et al., TinyZero (2025)

Speculative

You: the user who corrects

On your computer, you are a free judge. The suggested code crashed? The answer was corrected by hand? Every correction is a labelled example. This is our own idea, not tested here: a local model that learns at night from the day's corrections. The risk: a human judge gets tired and makes mistakes.

Promising

The vote: trusting the majority of its own answers

With no answer key, one can answer 64 times and take the most frequent answer as the "truth". TTRL does exactly that on problems with no known answer and reports a rise of about 211% on the AIME 2024 competition for a 7-billion-parameter model. Fragile by nature: when the majority is wrong, the model learns its mistake.

Source: Zuo et al., TTRL: Test-Time Reinforcement Learning (2025)

IV

What has already been shown

Two results give a sense of what to expect, one older and solid, the other recent and spectacular.

Established

STaR: learning from its own reasoning (2022)

The model writes a line of reasoning; those that lead to the right answer are kept. When it fails, it is given the answer and rewrites a line of reasoning that leads to it: this is "rationalization". On CommonsenseQA, the result rivals a model 30 times larger trained the usual way.

Source: Zelikman et al., STaR: Bootstrapping Reasoning With Reasoning (2022)

Promising

A single example goes a long way (2025)

With one problem and checked reinforcement, a 1.5-billion-parameter model goes from 36.0% to 73.6% on the MATH500 test. Caution: much of the jump comes from the model learning to present its answer in the right format; the authors measure 8.6 points beyond that effect. It is proof that the skill was already there, hidden.

Source: Wang et al., Reinforcement Learning for Reasoning with One Training Example (2025)

V

The ordinary computer: does it fit?

Let us do the sums for Llama 3.1 8B, that is 8.03 billion parameters. Stored in 16 bits, they weigh 16 GB: too much for most graphics cards. Two tricks change everything.

First trick: round the weights to 4 bits, as in QLoRA. 8.03 billion × 0.5 byte = 4.0 GB. QLoRA showed that this way a 65-billion-parameter model can be fine-tuned on a single 48 GB card without losing quality.

Second trick: leave the weights alone and add an adapter (LoRA). The 8 billion weights are frozen and only small matrices grafted alongside are trained. With a rank of 16 on all layers of Llama 3 8B, the count gives 41.9 million parameters, that is 0.5% of the model and 84 MB. Training keeps four 4-byte numbers for each (the parameter, its correction and two optimiser averages): 41.9 M × 16 bytes = 0.67 GB.

Add the working memory. To write, the model keeps 128 KB in memory for each word already written; 16 tries of 1,000 words in parallel need 2.1 GB. Count a few more gigabytes for the training itself. Total: 9 to 10 GB. That fits on a 12 GB gaming card, and comfortably on 16 or 24 GB.

An unexpected bonus: the adapter forgets less. Biderman and colleagues show that LoRA learns less than full retraining in code and maths, but better preserves what the model could already do elsewhere. For a model training on its own for weeks, forgetting little matters more than learning fast.

Source: Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs (2023), Biderman et al., LoRA Learns Less and Forgets Less (2024), Meta, Llama 3.1 8B Instruct model card (2024)

VI

One night of work, hour by hour

Take an 8-hour night and split it into three parts: 6 h of tries, 1 h of learning, 1 h of checking.

The tries (6 h). A high-end gaming card reads its memory at about 1,000 GB per second; a more modest card, two to three times slower, and the night then gives two to three times fewer tries. To write one word, the model rereads its 4 GB of weights: alone, it would top out around 250 words per second. By running 16 tries side by side, the same read serves 16 words. We take a cautious assumption of 1,000 words per second. In 6 h: 6 × 3,600 × 1,000 = 21.6 million words, or about 21,600 tries of 1,000 words. Enough to attempt 2,500 exercises 8 times each (20,000 tries), with room to spare.

The learning (1 h). We keep at most two right tries per exercise, so at most 5,000 examples of about 500 words: 2.5 million words. Learning costs, counting generously, 6 operations per parameter and per word: 6 × 8 billion × 2.5 million = 1.2 × 10¹⁷ operations. At an effective rate of 40 trillion operations per second (an assumption, well below the maximum of these cards), that takes 50 minutes.

The checking (1 h). The model sits a small exam it has never seen, plus a set of general questions, to check that it has improved without forgetting anything. If the general score drops, the night's adapter is thrown away. This is the seat belt of the whole method.

The electricity. A gaming PC at full load draws about 450 W (our assumption). 450 W × 8 h = 3.6 kWh. At the French regulated price of October 2026, about €0.20 per kWh: €0.72 a night, about €5 a week.

VII

The calculation: night after night, how far?

Let us build a toy model. The exercises are not all equally hard for our model: say a quarter are never solved, 15% are solved one time in ten, 25% six times in ten and 35% nine times in ten. Its first-try score: 0.15 × 10% + 0.25 × 60% + 0.35 × 90% = 48%.

Each night, it tries each exercise 8 times. An exercise solved one time in ten gives at least one right try with a chance of 1 − 0.9⁸ = 57%: roughly every other night, it provides an example to learn from. An exercise never solved provides nothing, ever.

Suppose each example learned moves the exercise 30% of the remaining way towards success. The score goes from 48% to 60% in two nights, 73% in a week, then slows down. With learning three times slower (10%), it only reaches 61% in a week. But in both cases, the curve approaches the same line: 75%, exactly the share of exercises the starting model solved at least once in 64 tries.

Here is the ceiling law, in one sentence: self-training brings the first-try score closer to the many-tries score, not beyond it. It is our translation into numbers of the observation by Yue and colleagues. The toy model is optimistic on one point: Llama 3.1 8B has already been fine-tuned after the fact on exercises of this kind, so part of its margin is already used up.

Source: Yue et al., Does RL Really Incentivize Reasoning Capacity Beyond the Base Model? (NeurIPS 2025)

Fig. N — Night after night, the score climbs towards a ceiling

Fast learning: 73% after 7 nightsSlow learning: 61% after 7 nights

Illustrative diagram: it shows the idea, not measurements.

A toy model, not a measurement. A quarter of the exercises are never solved at the start, 15% are solved one time in ten, 25% six times in ten, 35% nine times in ten: 48% on the first try. Each night, 8 tries per exercise; when at least one is right, learning moves that exercise closer to success (by 30% or by 10% of the remaining way). The speed depends on learning; the ceiling does not move: an exercise never solved never gives anything to learn.
VIII

Raising the ceiling without a big model

If the ceiling is what the model finds in many tries, there are three ways to raise it, and none of them needs a giant.

Whisper the answer. This is STaR's rationalization: for an exercise never solved, give the final answer and ask for a line of reasoning that leads to it. The exercise with p = 0 finally gives a signal. The risk: reasoning that "lands right" by a wrong route. So a judge is needed for the reasoning too, not just for the answer.

Grade the exercises. An exercise impossible today becomes doable if simpler versions come first. The model can invent these steps itself, as in Absolute Zero, where it proposes exercises right at the edge of what it can do. This is the notebook's idea of a curriculum, applied to the student on their own.

Give it tools. A calculator, a Python interpreter, a search engine over a local library. With a tool, out-of-reach exercises become doable sometimes, and that "sometimes" is enough to get the loop started.

Hence our hypothesis H9: the world can serve as the teacher. Answer keys, tests, rules, tools, and the length of a night.

Source: Zelikman et al., STaR: Bootstrapping Reasoning With Reasoning (2022), Zhao et al., Absolute Zero: Reinforced Self-play Reasoning with Zero Data (2025)

IX

A thousand ordinary computers

What if a thousand volunteers lent their PCs at night? The calculation is surprising. A thousand machines × 21,600 tries = 21.6 million tries per night. On our 2,500 exercises, that is more than 8,000 tries per exercise per night. Computing stops being the problem: the bottleneck becomes the stock of checkable exercises.

What should be exchanged? Two options. One can exchange the adapters (84 MB each) and merge them. We know how to train a model together over the internet: DiLoCo matches standard training while communicating 500 times less, and INTELLECT-1, a 10-billion-parameter model, was trained across three continents by some thirty compute providers. But the weights received have to be trusted, and they cannot be read back.

We prefer the second: exchange checked examples. A right solution weighs about 2 KB; a night's 5,000 examples, 10 MB. Above all, anyone can rerun the judge on a received example in a fraction of a second. A cheater who sends fake examples is caught at once. We exchange proofs, not weights. This is our proposal, not a published result.

Source: Douillard et al., DiLoCo: Distributed Low-Communication Training (2023), Jaghouar et al., INTELLECT-1 Technical Report (2024)

X

What resists

The ceiling. We calculated it: with no new exercises, no whispered answers and no tools, the score stops at what the model could already find in many tries.

Collapse. A model that learns only from its own texts can grow poorer, generation after generation: Shumailov and colleagues showed it. The outside judge limits this risk, because it filters on correctness and not on style, but it does not remove it: the solutions kept look more and more alike. Keeping the original data in the mix helps.

Gaming the judge. A model learns what is rewarded, not what was meant. If the tests are weak, it will learn to get around them. A lax judge produces a cheating student.

The runaway vote. When the judge is the majority of its own answers, a frequent mistake becomes a "truth" that gets reinforced. Over one night, it is a risk; over months with no outside anchor, it is our main worry.

Tasks with no judge. Writing a letter, giving advice: no program says which try is best. Here, the ordinary computer does not replace a teacher, human or model.

Source: Shumailov et al., AI models collapse… (Nature, 2024), Yue et al., Does RL Really Incentivize Reasoning Capacity Beyond the Base Model? (NeurIPS 2025), Zuo et al., TTRL: Test-Time Reinforcement Learning (2025)

XI

Our bets, with numbers

What follows is our own reasoning, not a published result. Each bet carries a confidence level, like the notebook's hypotheses.

Speculative

A week of nights is enough for a clear gain · 65%

Llama 3.1 8B, trained for 7 nights on its own tries sorted by an answer key, gains at least 5 points of first-try success on the MATH-500 test, on a single gaming card. This bet would fall if the gain stayed under 2 points.

Speculative

The ceiling holds · 70%

Its success over 64 tries moves by less than 3 points. What the loop adds is reliability, not reach. If this bet falls, it is excellent news: the student on their own would really discover something new.

Speculative

Together, the bottleneck is the exercises, not the computing · 70%

A network of volunteers exchanging checked examples improves faster than each one alone, but it quickly runs out of available exercises. Value shifts from computing power to judges and exercises.

Speculative

With no outside anchor at all, it drifts · 60%

A model that learns only from the vote of its own answers improves at first, then, over months, reinforces its frequent mistakes and loses diversity. A piece of the world is always needed to say right or wrong.

XII

The experiment to run, and our predictions

Take Llama 3.1 8B, or Llama 3 8B to make it harder, a 24 GB card and a rank-16 adapter. Draw 2,500 problems from the MATH training set, with their answers. Each night: 8 tries per problem, keep at most two right tries, learn, then check. Seven nights.

Measure before and after: first-try success and success over 64 tries on MATH-500, and a set of general questions (MMLU) for forgetting.

Our predictions, written down before any experiment so that we can be proved wrong. On the first try: +5 to +12 points. Over 64 tries: less than 3 points better. General questions: less than one point lost. Cost: about €5 of electricity. If the first-try gain stays under 2 points, we will lower H9. If it exceeds 15 points, or if the 64-try score jumps, our ceiling law is wrong, and we will gladly revise the notebook.

Back to top