Deep dive · Folio IV
Your brain does not fire all its neurons to read one word. Classic models do: every parameter works on every word. A new generation wakes up only the part that is needed.
A classic (“dense”) model is like a company where every employee reads every letter. A mixture of experts replaces one big block with many smaller specialists and a dispatcher who sends each word to a few of them.
The bet: the model can know a lot (many parameters) while computing little for each word (few active parameters). Total size and cost per word become two separate dials.
At each layer, a small network, the router, gives each expert a score for the current word and keeps the best few. Only those experts compute. The difficulty is balance: if the router always picks the same experts, the others never learn and part of the hardware sits idle.
DeepSeek-V3 (December 2024) has 256 specialised experts plus one shared expert per layer, and activates 8 of them per word: 37 billion active parameters out of 671 billion. It balances the load without the usual penalty, by nudging up the score of under-used experts and down that of over-used ones.
Source: DeepSeek-AI, DeepSeek-V3 Technical Report (2024)
Fig. F — Two experts out of sixteen
2 / 16 experts switched on · 12.5 % of the layer
Illustrative diagram: it shows the idea, not measurements.
Established
671 billion parameters, 5.5 % active per word. The full training took 2.788 million GPU hours on H800 chips, a modest budget for its level, and the model competes with the best closed models of late 2024 on many benchmarks.
Promising
Instead of choosing between experts, each layer chooses which words to process; the others pass straight through. Models trained this way at Google DeepMind match a standard model while using up to 50 % fewer operations per pass, and generate up to 50 % faster.
Promising
One block of layers is applied several times, like rereading a hard sentence, and a router decides how many passes each word gets. From 135 million to 1.7 billion parameters, it beats standard models at equal compute, with fewer parameters (NeurIPS 2025).
Memory does not shrink: all 671 billion parameters must still be stored, often spread over many chips. You save computation, not storage, and words travel between machines. Grouping requests gets harder: when each word takes a different path, serving many users at once, which is what makes servers efficient, works less well; Mixture-of-Depths shines mostly one request at a time. Fragile training: routers can collapse onto a few experts.
It is the first factor of H6: the share of the model that works for each word. It also echoes our hierarchy. Experts split knowledge horizontally (by speciality), whereas our pyramid splits it vertically (by level of abstraction). The two could be combined: experts for principles, a memory for details.
Look inside an open mixture-of-experts model and check, on OrigoTech articles, whether its experts specialise by topic (economy, AI, health) or by form (punctuation, numbers, grammar). If it is mostly form, experts do not sort knowledge the way a library would, and our vertical hierarchy remains to be built.