Llama 3 8B has 8 billionparameters. The largest open models have a trillion or more. Can that gap be closed without building a new giant? Our answer fits in one sentence: a small model does not need to be big, it needs to have access to what the big one keeps in its weights. Here are the reasoning, the calculations and the limits.
Promisingupdated
I
The question, asked honestly
Llama 3 8B came out in April 2024. It was trained on more than 15 trillion tokens, but it has only 8 billion parameters. Against it, Kimi K2 has a trillion, and the leading closed models are probably bigger still, although no official figure exists. The size gap therefore runs from 125 times to several hundred times.
"As good" has to be pinned down, or the question has no answer. As good at what, and at what cost? A small model that matches a giant while spending ten times more has gained nothing. So we are looking for a small model that gets as many right answers, at a comparable or lower cost, and we will say for which kinds of task that seems possible.
Where does the gap come from? Four different things
When a big model does better, it is not one advantage but four. They do not close in the same way, and that is the whole key to the problem.
1. Knowledge, meaning the facts retained. The notebook saw it from the start: a model stores about 2 bits of knowledge per parameter. Let us do the sum: 8 billion × 2 bits = 16 billion bits, or about 2 GB of knowledge for Llama 3 8B. For a trillion-parameter model: about 250 GB. A ratio of 125.
2. Reach: can the model, at least now and then, find the right answer? Call p its probability of being right on the first try. If p is zero, the problem is out of its reach. 3. Reliability: p is not zero, but it is small. The model knows how, it just gets it wrong often.
4. Breadth: knowing a little about everything, linking distant fields, grasping a nuance, judging an open question with no single "right answer".
Our central idea: each part has its own remedy. Knowledge goes into an external memory. Reach is learned from a teacher or borrowed from a tool. Reliability is bought by trying several times, provided there is a judge. Breadth has no simple remedy: that is where, we think, parameters really buy something.
A model never finds an answer it has no way of building. Two remedies: learn from a teacher who already knows (distillation), or borrow the ability from a tool that has it (a calculator, a code interpreter). These two studies show that an "old" skeleton is enough.
Established
An old skeleton, a new teacher: DeepSeek-R1-Distill-Llama-8B
DeepSeek took Llama 3.1 8B, from the same family as Llama 3, and trained it on reasoning written by its large R1 model. The result: 50.4% on the AIME 2024 maths competition, against 9.3% for GPT-4o; 89.1% on MATH-500, against 74.6%; and in science (GPQA Diamond), 49.0% against 49.9%, almost level. For comparison, Meta reports 51.9% on MATH for its own Llama 3.1 8B, a similar but not identical test. The number of parameters did not change: what they were taught did.
Borrowing the ability: ToRA, with a calculation tool
ToRA teaches a model to alternate between reasoning in words and small programs it has executed: calculations, symbolic algebra. Built on Llama 2 (2023), ToRA-7B reaches 44.6% on MATH, 22 points more than the best open model of the time, which was ten times bigger (70 billion). The 34-billion version even beats GPT-4 reasoning without a tool. The tool makes no arithmetic mistakes: the model only has to know what to ask it.
Closing the reliability gap: try several times, and sort
When p is small but not zero, the remedy is simple in principle: try several times and keep the right attempt. Everything then rests on the judge that recognises it. Three studies measure how far this goes.
Established
Large Language Monkeys: 250 attempts (2024)
As attempts multiply, the share of problems solved by at least one attempt climbs steadily across four orders of magnitude. On SWE-bench Lite, a benchmark made of real issues taken from GitHub projects, one model goes from 15.9% with one attempt to 56% with 250, above the best single-attempt score of the time (43%). But where nothing checks automatically, choosing by vote or with a scoring model plateaus beyond a few hundred attempts.
With a well-tuned strategy of attempts and a model that scores each reasoning step, Llama 3.2 1B scores 72.2% on MATH-500 with 512 attempts, against 71.4% for Llama 3.1 405B in one attempt. A narrow win, and the small one loses on AIME 2024, which is harder. Our arithmetic: 512 attempts by a 1-billion-parameter model use about as many parameters per word as one attempt by the 405-billion model (512 against 405), not counting the judge. At almost equal compute, the small one draws level.
rStar-Math: searching like a chess program (Microsoft, 2025)
A small model explores solutions step by step, the way a chess program explores moves, guided by a second small model that judges each step. Both improve together over four rounds, on 747,000 problems. Qwen2.5-Math-7B goes from 58.8% to 90.0% on MATH, and Phi3-mini (3.8 billion) from 41.4% to 86.4%, above o1-preview. The authors distilled no stronger model.
Closing the knowledge gap: a memory, and specialists
For facts, the remedy seems obvious: do not learn them, look them up. The results are more nuanced than you might think. And for specialised skills, another idea works: many small specialists rather than one generalist.
Established
MassiveDS: a trillion tokens to look up (2024)
A University of Washington team built a store of 1.4 trillion tokens that the model consults as it answers. On factual questions (TriviaQA, Natural Questions), Llama 2 7B with a store of under 100 billion tokens beats Llama 2 13B and Llama 3 8B without one. But on MMLU and MedQA, which call for more reasoning, the memory does not let the small model overtake the large one. Looking things up brings facts, not the reasoning that connects them.
Predibase fine-tuned 310 small models (10 base models × 31 tasks) with 4-bit LoRA adapters. On average, they beat their base model by 34 points and GPT-4 by 10 points, each on its own task. And because the base model is shared, 25 Mistral 7B specialists fit on a single 80 GB graphics card. The limit: these are narrow, well-defined tasks, not free conversation.
Let us build a simple model. One attempt by the small model succeeds with probability p. With k independent attempts and a perfect judge, the chance that at least one is right is 1 − (1 − p)ᵏ. For example, with p = 30% and 5 attempts: 1 − 0.7⁵ = 1 − 0.168 = 83%.
How many attempts to match a large model that is right 80% of the time on the first try? We need 1 − (1 − p)ᵏ ≥ 0.8, so k ≥ ln(0.2) / ln(1 − p). For p = 30%: 5 attempts. For 12%: 13 attempts. For 5%: 32 attempts. For 0%: never. That is the figure below.
Then there is cost. One attempt by the small model costs a fraction c of one attempt by the large one, roughly the ratio of active parameters per word. The small model pays off if k × c < 1. Combining the two formulas gives a threshold: the small model wins as soon as p exceeds 1 − 0.2ᶜ. That threshold depends enormously on the giant it faces.
Against a dense 2,000-billion-parameter giant, c = 8 / 2,000 = 0.004 and the threshold falls to 0.6%: almost anything the small model can do, even rarely, becomes worthwhile. Against a mixture-of-experts giant with 100 billion active parameters, c = 0.08 and the threshold rises to 12%. Against Kimi K2 and its 32 billion active parameters, c = 0.25 and the threshold reaches 33%.
Two lessons. First, the more frugal the giant, the harder it is to beat: mixture-of-experts giants, which switch on only a small part of themselves, shrink the small model's advantage. Second, the judge is the keystone: without it, you cannot tell which of the k attempts to keep. Finally, an honest warning: our calculation assumes independent attempts and a free judge. In reality a model often repeats the same mistake, and Brown and his co-authors observe a slower rise than our formula: roughly the same gain with each doubling of the number of attempts. So more attempts are needed than this calculation suggests.
Fig. M — Catching up with the giant by trying again
Right 30% of the time: 5 attempts are enough12%: 13 attempts5%: 32 attempts0%: never
Illustrative diagram: it shows the idea, not measurements.
Each curve follows the formula 1 − (1 − p)ᵏ: the chance that at least one of k attempts is right, if a judge can recognise it. The dots mark where the small model catches up with a large model that is right 80% of the time on the first try. An idealised calculation: in reality, attempts from the same model resemble each other, and more are needed.VII
The plan: the small model's workshop
Here is our proposal. It assembles the notebook's leads around the small model, like the apprentices and tools around the master in Leonardo's workshop. Six pieces.
1. A router recognises the kind of question (calculation, code, fact, writing) and picks the useful pieces. 2. A library of specialists: LoRA adapters, one per domain, learned from a large model. Our arithmetic: a rank-16 adapter across all of Llama 3 8B's layers has about 42 million parameters, 0.5% of the model and about 84 MB. A thousand specialists weigh 84 GB and only one switches on at a time: a mixture of experts, but at the scale of the system.
3. A hierarchical external memory, as in parts II and III: the facts are not in the weights but in a store organised as trees and graphs. 4. A toolbox of tools and programs: calculator, code interpreter, solvers, and programs already found, reused like Voyager's skill library.
5. A judge: tests for code, a recalculation for maths, a model that scores the steps elsewhere. The small model tries several times, the judge sorts. 6. A fallback: if nothing passes the judge, the question climbs to the large model, as in a cascade. Its answer, once checked, is used to train the next specialist: the top of the pyramid teaches the bottom (hypothesis H7).
One last calculation says whether the whole thing is worth it. Each question costs k × c (the small model's attempts) plus the share e of questions that climb to the large one. With 8 attempts at c = 0.08, that makes 0.64 + e. The workshop pays off as long as fewer than 36% of questions climb. At 10%, it costs 0.74, about a quarter less than the giant alone. And every question that climbs enriches the specialists, so e should fall over time.
Good news: the age of the weights matters less than what they are taught afterwards. ToRA was built on Llama 2 (2023); DeepSeek-R1-Distill-Llama-8B, on Llama 3.1 8B, from the same family as Llama 3. In both cases, extra training on good examples made the results jump.
Llama 3 8B's real limits lie elsewhere. Its context window: 8,000 tokens, about 6,000 words, a dozen pages. That is little for reading documents pulled from a memory; Llama 3.1 raised it to 128,000. Its starting point in maths: 30% on MATH, according to Meta. The lower the starting point, the smaller p, and the more attempts are needed, or a teacher.
Our recipe for an old model, in order: first an adapter learned from a teacher, so that it can reason at length and call tools; then the memory and the tools, for facts and calculations; finally multiple attempts with a judge, where one exists. The order matters: multiple attempts only help if p is already large enough, and it is the teacher that raises it.
Questions without a judge. Writing, advising, settling an open question: no program says which attempt is best. Choosing by vote or with a scoring model plateaus after a few hundred attempts.
Zero reach. If p is zero, no number of attempts helps. Reinforcement learning itself does not go beyond what the base model finds over many attempts. A teacher or a tool is then needed, and one does not always exist.
Breadth. Linking a thousand scattered facts, grasping a nuance, keeping a long document in mind. Memory helps for a precise fact, much less for reasoning over many facts at once: MassiveDS did not help the small model overtake the large one on MMLU. That is where, we think, parameters really buy something.
Time. Thirty-two attempts take time, or machines in parallel. For a fluid conversation that is a real brake, even when the total cost is lower.
On checkable tasks, the gap is mostly about reliability · 65%
In maths, tested code and formal proofs, the small model often finds the right answer sometimes: multiplying it by attempts and a judge closes most of the gap. This bet would fall if we found many problems that the giant solves and the small model never solves, even in a thousand attempts.
Speculative
Isolated facts are bought with disk space, not parameters · 60%
For a question whose answer fits in one document, a good external memory is worth hundreds of billions of parameters. For a question that requires combining many facts, much less.
Speculative
A well-equipped old 8B matches a giant on a closed list of tasks · 55%
On a defined set of checkable tasks, a Llama 3 8B with adapters, a memory, tools and a judge would reach the level of a trillion-parameter model, at a comparable cost. This is our hypothesis H8.
Speculative
On open-ended tasks, the gap will remain · 70%
Writing, advising, judging: without a verifier, multiplying attempts is almost useless, and breadth cannot be distilled into one adapter. We think the giants keep a clear advantage here.
XI
The experiment to run, and our predictions
Take Llama 3.1 8B, or Llama 3 8B to make it harder, and build the workshop. Compare it, at equal cost, with a leading model on four families of tasks: maths (MATH-500, AIME), code with tests (LiveCodeBench), facts with a memory (TriviaQA, Natural Questions) and open questions judged by people.
Our predictions, written before any experiment so that we can be proved wrong. In maths and code: within 5 points of the giant, at equal or lower cost. On facts: within 10 points with a good memory. On open questions: a clear gap. If the maths gap exceeds 15 points, our bet on reliability falls, and we will lower H8.