AI of tomorrow Glossary
EN

← Back to the notebook

Deep dive · Method

How our H1 experiments work

A plain-language guide to our small tests: what the model is, what a label is, why we scramble it, and how to read a null result.

Speculative updated

I

What we actually train

A tiny language model: two layers, 64 numbers wide, starting from random values, so it knows nothing. It reads text one byte at a time and learns to guess the next byte. We train it for a few hundred updates on the first 96 bytes of old newsgroup messages, 300 per category, which takes under a minute on a GPU.

Later runs use a larger model: four layers, 128 numbers wide, with the same training recipe. It is still small next to a real language model.

II

The labels

Before each snippet we write two or three extra symbols. With no label they are blank. With a flat label each category has its own symbol. With a hierarchy the first symbol names a group (for instance computers or recreation) and the next ones narrow it down. A scrambled code has the same shape but assigns the codes to the wrong categories. We also test the group alone, and on the three-level taxonomy the topic alone and topic plus subtopic.

III

Why we scramble

A hierarchy carries two things: information (which category a text belongs to) and structure (which categories are siblings). A scrambled code carries the same information, but its groups mean nothing. If the true hierarchy beats scrambled labels, the structure is what helps; if not, any code does as well. Because one scrambling could be lucky, we repeat with five, and we reject any that keeps the real groups.

IV

How we score

We measure the loss on held-out messages, in nats per byte (lower is better). A model guessing every byte at random would score about 5.5; ours end near 2.5. Within a seed, every condition gets the same data, the same batches and the same starting values, so only the label differs. To see whether the model really uses its label, we also score it with the label blanked out: that is an ablation, and the gap between the two scores is how much the model relies on it.

V

Less data, more passes

In the small-data runs we keep only the first 30, 100 or 300 messages of each category; smaller sets are nested inside larger ones, and every size is scored on the same 640 held-out messages, so the losses can be compared. The number of updates stays at 800, so a small set is seen many times: about 80 with 30 messages per category. A model that sees the same few examples that often starts to learn them by heart: that is overfitting, and it shows up as a held-out loss that rises again (from 2.78 to 4.12 in our sweep).

VI

Brakes against overfitting

To limit overfitting we add two standard brakes, together called regularization. Dropout 0.2 switches off random units of the network at each training step, so no single path can memorize an example. Weight decay 0.1 pulls the weights towards small values at each step. We use one setting, chosen once and not tuned, the same for every condition. The loss on held-out messages is always measured with dropout off.

The larger-model runs start from the same 0.2 and 0.1, then use a stronger setting, dropout 0.3 and weight decay 0.3, to check whether the bigger model was learning its examples by heart. We still tune nothing.

VII

Predictions first

Each protocol and its predictions are written before the run. The dataset revision is pinned, the official test split is never used, and the code is fingerprinted in the result file. A null result is published like any other.

Within a protocol we also list, beforehand, the cells we will read: a cell is one combination of data size and number of updates. A pre-registered cell is fixed before the run. In the replication runs the four cells were registered before the run but chosen after seeing an earlier one; we say so, because a reader should trust them a little less.

VIII

Material or not

Scrambled codes do not all give the same loss: their range, from the best to the worst of the five (averaged over seeds), measures the noise that comes from the code alone. We call a gap material when the true hierarchy beats the mean of the scrambled codes by more than that range. Below it, the gap could come from which scrambling we happened to draw, so we call it a non-material trend. Another figure, the label-use gap, is the rise in loss when the label is hidden: it tells whether the model uses its labels, not whether the hierarchy is better than scrambled ones.

IX

Toy worlds for H2

The H2 pilots use a toy world rather than text: random facts and a rule (addition modulo 10) that needs two or three of them, so that what a model has memorised and what it has computed can be told apart. As for H1, the predictions are written before each run and the judged size is fixed in advance. In most pilots the memory is an oracle: the right facts are placed in the context, so the cost and the errors of searching are ignored; only pilots 4 and 6 corrupt retrieval, by random substitution or random distractors. Details and numbers: H2 pilots.

X

How to read a result

Differences of a thousandth or two of a nat per byte are smaller than the variation between seeds or between scrambled codes, so we claim no significance and never turn them into a percentage of data saved. What would change our mind: the true hierarchy ranking first consistently, with a gap larger than that spread, and then the same on a larger pretrained model.

XI

Files

Every figure in these notes comes from a file you can download. Longer training: results and protocol. Five scrambled codes: results and protocol. Three-level taxonomy: results and protocol. The results are in JSON; the protocols are in Markdown.

Later runs: Data-scale sweep: results and protocol. Regularized small data: results and protocol. Replication with new seeds: results and protocol. Other categories: results and protocol. Same formats.

Larger model: results and protocol. Larger model, other categories: results and protocol. Larger model, stronger regularization: results and protocol. Same formats.

Back to top