Deep dive · Folio IV
Solving a sudoku takes time; checking a filled grid takes seconds. In many problems, checking is far easier than finding. AI is starting to rely on this asymmetry at two scales: the word and the whole answer.
A language model writes one word at a time, and each word requires reading all its weights. But checking several proposed words can be done in a single pass, about as fast as writing one. Hence the idea: a small, fast model guesses, and the big one only confirms.
At the scale of a whole answer the same asymmetry holds: generate many attempts, then let a verifier (tests, a proof checker, a calculator) keep the right one.
In speculative decoding (Google, 2023), the small model proposes a few words; the big model computes, in parallel, the probability it would have given to each. A clever acceptance rule keeps the right ones and resamples the first wrong one, so the output follows exactly the same distribution as the big model alone.
The more often the small model guesses right, the bigger the gain. Later work therefore focuses on making the drafter guess better, notably by giving it access to the big model's internal states.
Source: Leviathan, Kalman & Matias, Speculative Decoding (ICML 2023)
Fig. K — Draft, then check
thecatsleepsunderaon
4 words written for a single pass of the large model
Illustrative diagram: it shows the idea, not measurements.
Established
On T5-XXL, 2 to 3 times faster with identical outputs, without retraining or modifying the model. It is now a standard part of inference servers.
Source: Leviathan, Kalman & Matias, Speculative Decoding (ICML 2023)
Established
The drafter is a small head plugged into the big model's internal layers rather than a separate model. Reported speed-up: up to 6.5 times, 1.4 times better than EAGLE-2, and 1.38 times more throughput even with 64 requests served together.
Source: Li et al., EAGLE-3 (2025)
Promising
With 250 attempts and the projects' own tests as verifier, an open model fixes 56 % of real GitHub issues (SWE-bench Lite) against 15.9 % in one attempt, beating the best single-attempt result of the time (43 %). Five attempts of the cheap model cost less than one attempt of the expensive models, and solved more.
Everything depends on the verifier. For code with tests or a mathematical proof, checking is reliable. For an essay or advice there is no automatic test: majority votes and scoring models level off after a few hundred attempts, far below what a perfect checker would allow. And speculative decoding speeds up writing, not thinking: it does not make the big model smarter.
Verification strengthens the other levers of H6: a cascade needs a judge, a tool needs tests, a small model can try often if something checks its work. It also suggests a role for our hierarchy: the rules at the top of the pyramid could serve as checkers for the details below.
For questions about OrigoTech articles whose answer is a figure or a date, let a small model answer 10 times and keep only answers confirmed by the article text itself. Compare the accuracy with a single answer from a large model.