Deep dive · Folio II
A classic model reads text in small pieces all of the same kind, cut in advance by a frozen dictionary. We read letters, then words, then ideas. What if the model built these levels itself?
Before training, a tool called a tokenizer cuts text into "tokens" using a fixed dictionary. The model never sees the letters, and never reasons at a level higher than those tokens: everything happens on a single level.
The work on this page adds levels and, above all, lets the model decide for itself where to cut.
Picture a factory with three workshops. A small workshop reads the text letter by letter. It groups letters into chunks and passes these bundles to a large central workshop, which does most of the thinking on a much shorter sequence. A last small workshop unrolls the result letter by letter.
The gain: the central workshop, the most expensive one, works on fewer items. And the cutting follows the content: a predictable passage forms a big chunk, a hard passage is cut finely. Compute goes where the information is.
Fig. C — From letters to ideas
Illustrative diagram: it shows the idea, not measurements.
The unit of thought becomes the whole sentence, placed in SONAR, a meaning space shared by 200 written and 76 spoken languages. The model predicts the next idea, not the next word. It beats a same-size Llama 3.1 at multilingual summarisation, but is less fluent.
No more dictionary: the model reads bytes and groups them into variable-size "patches", cut where what follows becomes unpredictable. For the first time it matches token-based models, up to 8 billion parameters and 4 trillion bytes, and scales better at equal inference cost.
Source: Pagnoni et al., Byte Latent Transformer (Meta, ACL 2025)
The cutting is entirely learned, with no rule supplied. On its own, the model invents chunks of about 4.5 to 5 bytes, comparable to classic tokens. With two levels, it matches a token model twice its size, and gains nearly 4 times in data efficiency on DNA.
This is exactly the path our conclusion IV suggested: a hierarchy that is learned, not dictated. The tokenizer was a hand-written rule; H-Net replaces it with learning. It is Sutton's "bitter lesson" applied to the hierarchy itself.
And it is a path to frugality: by allocating compute according to difficulty, less is spent on what is predictable.
These architectures are harder to train: sequences have variable lengths and each level has its own settings. They have been tested up to a few billion parameters, far from the largest models. LCMs still struggle with fluency, and depend on a fixed meaning space (SONAR) that they do not learn themselves.
Support for H1: giving (or letting the model build) an explicit structure makes learning more efficient, clearly so on DNA, Chinese and code, where classic cutting is poor. And a link with H5: an LCM's "ideas" live in a continuous space where one sentence is close to several families, closer to a graph than a tree.
H1 · The appetite for data comes partly from a missing hierarchy (55 %) H5 · A graph, not a tree (60 %)
Stack one more level: from letters to chunks (H-Net), from chunks to ideas (LCM), from ideas to rules. The top level would become the model's compact memory, the one in our hypothesis H2.