Strict order
Sorting from easy to hard by vocabulary diversity gives +1.8% average accuracy with 17.9% fewer steps.
Deep dive · Folio II
Learning in order
Nobody learns to read with Proust. Yet models are trained on everything shuffled at random. What happens if we present texts from the simplest to the hardest?
The idea, simply
Today, billions of training texts are shuffled at random and read once. A children's text can follow an advanced mathematics paper.
Curriculum learning, proposed by Yoshua Bengio in 2009, starts from a school intuition: begin with the easy, climb step by step. It is a hierarchy in time: the data stays the same, only the order changes.
How it is done
It takes three ingredients. A difficulty measure: a machine doesn't know what is "hard", it needs a computable proxy. A pace: jump straight from easy to hard, or climb a gentle slope. A moment: order the whole training run, or only the start.
The study by Zhang and co-authors (EACL 2026) screened 15 measures and kept six. The most useful: the compression ratio (a text that compresses well is repetitive, so easy), vocabulary diversity (MTLD) and text length. Perplexity, the model's surprise, helps at first and then drags results down: "surprising" texts are often just noise.
Source: Zhang et al., Beyond Random Sampling (EACL 2026)
Fig. A — Two conveyor belts
Random
Ordered
Illustrative diagram: it shows the idea, not measurements.
What the experiments show
Over 200 training runs, mostly on 0.5-billion-parameter models and 10 billion tokens, with a check at 1 billion parameters.
A gentle slope
With linear pacing, compression ratio reaches the best baseline level with 39.5% fewer steps. Interleaved, the gain rises to 41.7%.
Warm up, then shuffle
Ordering only the start, then returning to random, leaves a lasting lead: up to +3.5% at 0.5 billion parameters and +3.2% at 1 billion.
It really is the order
Decisive control: shuffling the same "easy" data helps a little, but less than presenting it in order. So the gain does not come from sorting alone.
The counterpoint
Another study (models from 14 to 410 million parameters, 300 billion tokens) finds that ordering mostly stabilises small models and that the advantage shrinks with size.
Source: Elgaar & Amiri, Curriculum Learning for LLM Pretraining (2026)
Industry's reverse order
Labs often do the opposite of a school curriculum: they keep the best data for the end. OLMo 2 (Ai2) spends its last 5 to 10% of compute on a high-quality mix (maths, science, Q&A). This is called mid-training or "annealing".
Even this placement is debated: a 2025 study shows the very end is not always the best moment, and explains why annealing on maths exercises helped the 8-billion-parameter Llama 3 but not the 405-billion one. The right order depends on the moment in learning, not just on the data.
Source: OLMo et al., 2 OLMo 2 Furious (Ai2, 2025), Predicting Training Re-evaluation Curves (2025)
The limits
These results come from mid-sized models, in English only. Difficulty is computed once, before training: it does not adapt to what the model already knows. And the gains, though real, stay at a few percent.
What it changes for our hypotheses
It is modest support for hypothesis H1: structuring the data helps, especially early on. But the "difficulty" measured here has a single dimension (short or long, simple or varied). Our idea is richer: levels of abstraction (observation, fact, rule, principle). As far as we know, nobody has yet ordered a training run by these levels.
H1 · The appetite for data comes partly from a missing hierarchy (55 %)
What we would like to test
Two opposite curricula on a mini-model: examples first, rule second (like a child), versus rule first, examples second (like a textbook). And an adaptive curriculum: measure what the model already knows and give it the next step.