Deep dive · Folio II
A library is not better because it has more books. It is better once the duplicates, the torn pages and the books you already know by heart are gone. Can we do the same with an AI's data?
Large corpora come from the web: lots of repetition, meaningless pages, near-identical texts. Every useless text costs compute, energy and, as we saw in part I, some of the model's memory capacity.
Pruning means choosing what to keep. The question is not "how much data?" but "which data?".
Remove the noise: a small classifier scores each text (for example its educational value) and only the best are kept. Remove duplicates in meaning: each text is placed in a space where texts close in meaning are neighbours, and one representative per group is kept. Remove what the model already knows: keep the examples that still teach it something.
The whole art lies in the measure: a good score lets you throw away a lot without losing anything, a bad one throws away treasures.
Fig. B — The sieve
useful duplicate noise
Illustrative diagram: it shows the idea, not measurements.
Usually, error falls slowly as data is added (a "power law"). Sorscher and co-authors show that with a good pruning measure, it can fall exponentially, so much faster. Tested in vision (CIFAR-10, ImageNet), with a cheap measure that doesn't even need labels.
Source: Sorscher et al., Beyond neural scaling laws (NeurIPS 2022)
A subtle result: when you have little data, keep the easy examples; when you have plenty, keep the hard ones. In other words, once the basics are in place, the rare, difficult cases teach the most.
Source: Sorscher et al., Beyond neural scaling laws (NeurIPS 2022)
SemDeDup (Meta) removes texts or images that are nearly identical in meaning. On a large image corpus, removing 50% of the data costs almost nothing in performance and halves training time. On text (C4), the efficiency gain is about 15%.
FineWeb-Edu (Hugging Face) scores each web page's educational value from 0 to 5, with a small model trained on Llama 3 70B's judgements. It keeps only scores ≥ 3: 1.3 trillion tokens out of 15. Result: 33.6% on MMLU at just 38 billion tokens, where the best competitor needed 300 billion.
Source: Penedo et al., The FineWeb Datasets (Hugging Face, 2024)
A filter has the tastes of whoever trained it. FineWeb-Edu's clearly favours school subjects and formal writing: you gain quality, you lose diversity. Very long training runs also need enough unique text, and filters that are too strict eventually run short. Finally, most of the spectacular results on the exponential curve come from images, not yet from text at scale.
Source: Penedo et al., The FineWeb Datasets (Hugging Face, 2024)
The "easy when scarce, hard when plentiful" result is direct support for H3 and H4: once the rules are learned, the exceptions are worth gold. Good pruning looks like our compression idea: keep the rules and what escapes them, drop the rest. What is still missing: a measure that can tell a precious exception from mere noise, since both are "surprising".
H3 · Rules + exceptions = compression (35 %) H4 · Exceptions are the anchor (65 %)
On a small corpus, compare three sieves: by quality, by duplicates in meaning, and a "rules + exceptions" sieve that keeps one typical example per group plus every case that departs from it. Then see which one best retains rare cases.