Deep dive · Hypotheses
H1 under test: our experiments
Hierarchy helps early on synthetic data; eleven real-text runs find no material advantage of the true hierarchy over scrambled labels, only a tiny trend that reproduces. Explore the protocols and limits.
The question
H1 asks whether giving a model data structure can help it learn from fewer examples. Our pilots compare no metadata, flat, hierarchical and scrambled labels at matched budgets.
H1 · The appetite for data comes partly from a missing hierarchy (55 %)
Four synthetic pilots
Three initial seeds and five new seeds showed faster early learning with hierarchy. One control scrambled the hierarchy; another crossed five scramblings with five seeds. At 100 updates mean loss was 2.0061 versus 2.0988; at 500 it was 2.0003 versus 2.0045. Five comparisons per seed share a reference and are not independent.
First real-text pilot
Across eight 20 Newsgroups categories, a small randomly initialized byte model was evaluated on 640 validation messages with three seeds. At 200 updates mean loss was 2.73333 for the true hierarchy versus 2.73326 for scrambled labels: no clear advantage. The official test split remains unused. The no-label model actually scored lowest (2.72897, versus 2.73162 for the flat label): at this stage the model does not yet use the labels, so this test could not separate the two hierarchies. Stronger versions follow in the sections below.
Source: SetFit, 20 Newsgroups dataset
Longer training, labels removed
The first real-text pilot stopped at 200 updates, before the model used its labels. We trained the same tiny model for 800 updates (three seeds, the same eight categories) and, at each checkpoint, also scored it with the labels hidden. That gap is zero for a model trained without labels; at 800 updates it averages 0.009 nats per byte for the flat label, 0.020 for the group alone, 0.025 for the true hierarchy and 0.031 for scrambled labels, so the labels are now being used.
Yet at 800 updates the true hierarchy and the scrambled labels reach the same loss, 2.5210 nats per byte, against 2.5292 with no label and 2.5170 with only the computer/recreation group. Details: results, protocol.
Source: OrigoTech, H1 real-text run: longer training with labels removed (results, JSON; protocol alongside)
Five scrambled codes
One scrambling could be a lucky draw. We repeated the test with five different scrambled codes, none of which keeps the computer/recreation split, and five seeds. At 800 updates the true hierarchy reaches 2.5279 and the mean of the five scrambled codes 2.5295: a difference of 0.0016 nats per byte.
The true hierarchy beats that mean on four seeds out of five, but its rank among the six codes goes from first to last (3rd, 1st, 6th, 1st and 2nd), and the five scrambled codes alone spread by about 0.005. The difference is smaller than that spread, so we read it as a null result.
Source: OrigoTech, H1 real-text run: five scrambled codes (results, JSON; protocol alongside)
A three-level taxonomy
Two groups of four categories is a thin hierarchy. We took 16 newsgroups arranged on three levels (4 topics, 8 subtopics, 16 leaves) and added partial codes: topic only, then topic plus subtopic. At 800 updates every coded variant lands between 2.5187 and 2.5217 nats per byte, against 2.5153 with no label at all, which is the lowest.
The true hierarchy (2.5198) beats the mean of five scrambled codes (2.5206) by only 0.0007, on four seeds out of five, again smaller than the 0.005 spread between scrambled codes. Adding a level does not clearly help: the successive steps (topic, then subtopic, then leaf) change the loss by +0.0046, −0.0013 and +0.0012.
Source: OrigoTech, H1 real-text run: three-level taxonomy (results, JSON; protocol alongside)
Data-scale sweep: 30, 100 and 300 per category
The idea behind H1 is that structure should help most when data is scarce, so we trained the same tiny model for 800 updates on 30, 100 and 300 messages per category (smaller sets nested inside larger ones; five seeds, five scrambled codes, the same eight categories) and wrote three predictions beforehand. At 300 per category we recover the earlier picture: the true hierarchy sits 0.0016 nats per byte below the scrambled codes, less than the 0.0052 spread between them (prediction 1, confirmed).
At 30 per category the model overfits: with no label, its loss on the held-out messages rises from 2.78 at 200 updates to 4.12 at 800, as it goes about 80 times over the same 240 messages. The true hierarchy is 0.0124 below the scrambled codes, on four seeds out of five, but the scrambled codes themselves spread by 0.0891, so the gap is lost in the noise (prediction 3, a clear gap at 30 per category, failed). The model does lean harder on its labels when data is scarce: the loss rises by 0.1243 when the label of the true hierarchy is hidden at 30 per category, against 0.0474 at 300 (prediction 2, confirmed). The no-label model stays the best at 30 and at 100 per category. Details: results, protocol.
Regularization on small data
Overfitting means the model learns its few examples by heart instead of what carries over to new ones. We repeated the test at 30 and 100 per category with two standard brakes: dropout 0.2 (random units are switched off during training) and weight decay 0.1 (a pull towards small weights). Same seeds and scrambled codes as the sweep, and again three predictions written beforehand.
Overfitting drops: at 30 per category and 800 updates the no-label loss is 2.8208, against 4.12 without regularization (prediction 1, confirmed), and the rise in loss when the label of the true hierarchy is hidden falls from 0.1243 to 0.0448 (prediction 2, confirmed). The spread between scrambled codes falls from 0.0891 to 0.0073. The true hierarchy is 0.0023 below them on four seeds out of five at 30 per category, and level with them (0.0000, three seeds out of five, spread 0.0020) at 100. Prediction 3 (at least four seeds out of five and a gap larger than the spread, at 30 or 100) failed. Details: results, protocol.
Replication with new seeds and permutations
The previous run left a small negative sign: the true hierarchy slightly below the scrambled codes at 200 and 400 updates. One sign can be luck, so we replayed the test with five new seeds and five new scrambled codes, with the same data and settings. We watch four cells (30 and 100 per category, at 200 and 400 updates), chosen after seeing the previous run, which weakens the test. Three predictions were written beforehand.
The sign is negative in 4 cells out of 4 (prediction 1, confirmed), the true hierarchy wins on at least four seeds out of five in 3 cells out of 4 (prediction 2, confirmed), and no cell shows a material gap, that is, a gap larger than the spread between scrambled codes (prediction 3, at least two cells out of four, failed: 0 out of 4). The gaps run from 0.0001 to 0.0013 nats per byte, against spreads of 0.0012 to 0.0037. Details: results, protocol.
Other categories: science against politics and society
So far, apart from the three-level taxonomy, every run used the same eight categories, four about computers and four about recreation. We repeated the replication on another split: four science groups (cryptography, electronics, medicine, space) against four groups on politics and society (guns, the Middle East, general politics, Christian religion), with new seeds, new scrambled codes and the same settings. A fourth prediction checks that the model really uses its labels.
Result: the sign is negative in 4 cells out of 4, the true hierarchy wins on at least four seeds out of five in 4 cells out of 4, and 0 cells out of 4 are material. The largest gap is 0.0016 nats per byte (30 per category, 400 updates), against a spread of 0.0044. The control holds (prediction 4, confirmed): at 100 per category and 800 updates, hiding the label raises the loss by 0.0103 with the true hierarchy and by 0.0110 with scrambled codes, so the model uses its label whether it is consistent or not. Details: results, protocol.
A larger model: 4 layers, width 128
Every run so far used the same tiny model (2 layers, width 64). If the benefit of hierarchy grew with the size of the model, a bigger one would show it, so we changed only the architecture: 4 layers, width 128, with the same first eight categories, 100 messages per category, five seeds, five scrambled codes, dropout 0.2 and weight decay 0.1 (not retuned). The two cells read, 200 and 400 updates, were fixed beforehand; 800 updates is shown too, but it was not pre-registered.
The true hierarchy sits 0.0006 below the scrambled codes at 200 updates and 0.0003 below at 400 (gaps of −0.0006 and −0.0003 nats per byte), against spreads of 0.0024 and 0.0062 between scrambled codes. It wins on three seeds out of five in both cells, and 0 cells out of 2 are material. At 800 updates (not pre-registered) the sign flips: +0.0005, one seed out of five, spread 0.0086. So the gap does not grow with the size of the model. The model does use its label (hiding it raises the loss by 0.0191 at 800 updates), yet the no-label model stays the best at 800 updates: 2.3925, against 2.3980 for the true hierarchy and 2.3975 for the scrambled codes. Details: results, protocol.
Larger model, other categories
Is the result tied to the computer and recreation categories? We kept the larger model and swapped the data for the other split: four science groups against four groups on politics and society, with new seeds and new scrambled codes. Same cells, same settings.
The gap is −0.0004 at 200 updates (three seeds out of five, spread 0.0018) and +0.0002 at 400 (two seeds out of five, spread 0.0050): one cell out of 2 is negative and 0 are material. At 800 updates (not pre-registered) it is −0.0015 (three seeds out of five), still inside the spread of 0.0078. Hiding the label raises the loss by 0.0179 at 800 updates, and the no-label model is again the best (2.3672, against 2.3753 for the true hierarchy). Details: results, protocol.
Stronger regularization: dropout 0.3, weight decay 0.3
The larger model of the first run may simply have been under-regularized: with more room, it can learn its training messages by heart. We kept its model, data and categories and raised the two brakes together, to dropout 0.3 and weight decay 0.3 (so their separate effects cannot be told apart), with five new seeds and five new scrambled codes.
The gap is −0.0008 at 200 updates (five seeds out of five, spread 0.0017) and −0.0010 at 400 (four seeds out of five, spread 0.0032): both cells are negative, both win on at least four seeds out of five, and 0 are material. At 800 updates (not pre-registered) the sign turns positive, +0.0011 (one seed out of five, spread 0.0042). The no-label model improves: 2.3717 at 800 updates, against 2.3925 in the first larger-model run (an unpaired comparison, with other seeds), which suggests that run was probably under-regularized. Hiding the label raises the loss by 0.0116. Details: results, protocol.
Overall reading: a non-material trend, reproduced
Over three sets of seeds and two sets of categories, the true hierarchy sits very slightly below the scrambled codes at 200 to 400 updates: gaps of about 0.0002 to 0.0016 nats per byte, around 0.1%, always below the spread between scrambled codes. We call this a non-material trend, reproduced: the sign comes back, but a gap smaller than the noise between permutations is not an effect we can claim.
A second result matters as much: the no-label model is better or equal in almost every cell. Consistent labels hurt a little less than wrong ones; they do not beat having no label. A pooled analysis of the two regularized runs on the first categories (ten seeds, ten scrambled codes) points the same way, for instance 10 seeds out of 10 at 100 per category and 400 updates, with a gap of 0.0006 against a spread of 0.0013. It is exploratory: no protocol planned it, and it is not a significance test.
With the larger model (three runs), the gaps at 200 and 400 updates lie between −0.0010 and +0.0002 nats per byte; 0 of the 6 cells are material, and the gap does not grow with the size of the model. Stronger regularization gives the most regular sign (negative in both cells, on five and four seeds out of five), still below the spread between scrambled codes. At 800 updates the no-label model is better or equal in all three runs. We read this as a weak, non-material trend, and the large model of the first run was probably under-regularized. We claim no significance and no percentage of data saved; the 55% confidence in H1 is unchanged.
What these numbers do not prove
The synthetic data deliberately favor hierarchy. The real-text runs use a tiny randomly initialized byte model trained for at most 800 updates, on short prefixes of old English messages from one corpus, and only three category structures (two splits into two groups of four, and a three-level taxonomy), all chosen by us; the labels are present at evaluation. The four cells replicated in the last two runs were chosen after the regularized run, and they are not independent, since the same seeds feed two checkpoints. The regularization is a single setting, dropout 0.2 and weight decay 0.1, not optimized; the larger-model runs add only one bigger architecture (4 layers, width 128), which is not a scaling law, and one stronger setting (dropout 0.3 and weight decay 0.3, changed together). Comparisons between runs are unpaired, with other seeds. We run no significance test, and these tests neither measure a percentage of data saved nor validate or refute H1 for large models. The 55% confidence remains unchanged.
And H10?
H10 remains paused. The public Space holds 14 experiment exports and 1,034 numeric values; the full series has 15 rounds, detailed on the translator page. These counts are exported values, not independent findings or a quality score. Detailed H1 protocols and measurements are also available there.
Other deep dives
- Learning in order
- Throwing away wisely
- A memory inside the model
- The cost becomes logarithmic
- Thinking in ideas, not words
- Switch on only part of the model
- Ask the small model first
- Distil into programs, not models
- Compute with fewer bits
- Change the machine
- Checking is cheaper than writing
- How the seven leads combine
- A small model on a par with a giant?
- A small model that improves on its own, on an ordinary computer?
- Putting the hypotheses to the test
- Translate a model and keep only what is needed?
- How our H1 experiments work
- H2 under test: our toy pilots
Sources
- SetFit, 20 Newsgroups dataset
- OrigoTech, H1 real-text run: longer training with labels removed (results, JSON; protocol alongside)
- OrigoTech, H1 real-text run: five scrambled codes (results, JSON; protocol alongside)
- OrigoTech, H1 real-text run: three-level taxonomy (results, JSON; protocol alongside)
- OrigoTech, H1 real-text run: data-scale sweep, 30, 100 and 300 per category (results, JSON; protocol alongside)
- OrigoTech, H1 real-text run: regularized small data, dropout 0.2 and weight decay 0.1 (results, JSON; protocol alongside)
- OrigoTech, H1 real-text run: replication with new seeds and permutations (results, JSON; protocol alongside)
- OrigoTech, H1 real-text run: other category groups, science against politics and society (results, JSON; protocol alongside)
- OrigoTech, H1 real-text run: larger model, 4 layers and width 128 (results, JSON; protocol alongside)
- OrigoTech, H1 real-text run: larger model, other categories, science against politics and society (results, JSON; protocol alongside)
- OrigoTech, H1 real-text run: larger model, stronger regularization, dropout 0.3 and weight decay 0.3 (results, JSON; protocol alongside)
- OrigoTech AI Research Lab — H1, H2 and H10 results