Deep dive · Folio IV
Nobody calls a surgeon for a splinter. Yet many AI services send every question, easy or hard, to their largest model. Cascades change the order: the cheapest model tries first.
Questions are not equally hard. “What is the capital of Italy?” does not need the same model as a tricky proof. A cascade orders models from cheapest to most expensive and stops as soon as an answer is good enough.
A router does the same thing in one go: it reads the question and predicts, before any answer, which model will be enough.
Everything depends on the judge that decides whether an answer is acceptable. In FrugalGPT (Stanford, 2023), it is a small scoring model trained to estimate whether an answer is right; the cascade learns the best order of models and a threshold for each step.
The judge must cost much less than the next step, or you lose what you hoped to save. And each failed attempt is paid for: a question that climbs to the top costs more than if it had gone straight there.
Source: Chen, Zaharia & Zou, FrugalGPT (2023)
Fig. G — The ladder of models
Questions:12345678
Average cost per question: 5.4 · everything to the large model: 20 · −73 %
Illustrative diagram: it shows the idea, not measurements.
Promising
On its benchmarks, the learned cascade matches GPT-4 with up to 98 % lower cost, or beats GPT-4 by 4 points of accuracy for the same price. The study starts from one observation: the prices of different models differ by up to two orders of magnitude.
Promising
A router trained on human preferences (Chatbot Arena) chooses, for each question, between a strong model and a weak one. On MT Bench, it keeps 95 % of GPT-4's quality for 3.66 times less, without the waiting of a cascade.
Source: Ong et al., RouteLLM (ICLR 2025)
Established
The same idea at the smallest scale: a small model proposes the next words and the big model only checks them. Here the result is guaranteed identical to that of the big model alone, 2 to 3 times faster. See the deep dive on verification.
Source: Leviathan, Kalman & Matias, Speculative Decoding (ICML 2023)
The judge can be wrong: a wrong answer accepted at the bottom of the ladder is never corrected. Waiting: a question that climbs three rungs waits for three answers, and a cascade can hardly display its answer word by word while unsure whether it will keep it. Drift: a router tuned on one kind of question can be badly calibrated when users start asking different things.
Second factor of H6: the share of questions that reach the big model. And a direct echo of H2, “learn the top, consult the bottom”: a cascade is a hierarchy of models, where the expensive one is only consulted when the cheap one is not enough.
H6 · The savings multiply (40 %) H2 · Learn the top, look up the bottom (75 %)
Sort the questions readers could ask about OrigoTech into three levels (a fact in one article, a link between articles, real reasoning) and measure what share a small model answers correctly on its own. If it is above half, the gain is real for a site like ours.