# H1 real-text pilot v9 protocol (data-scale sweep)

Written before execution. Code: `h1_realtext_v9.py`, entry point `run_realtext_v9()`.

## Question
In v6 and v7, hierarchy was about equal to scrambled labellings at 300 training examples per category. Does the hierarchy beat mixed (scrambled) codes when training data is scarce? The idea behind H1 is that structure should help most when data is rare, so we sweep the amount of training data.

## Data
Identical to v7: `SetFit/20_newsgroups`, frozen revision `f1b91292074e7cfb69be58b642d583ec262f30ed`, `train.jsonl` only (the official test split is unused). Same 8 categories (4 comp.*, 4 rec.*), whitespace-normalised UTF-8, first 96 bytes. `load_data()` is unchanged: 300 training (idx < 300) and 80 validation (idx 300..379) messages per category, so `selected_data_sha256` equals the v7 value. Only aggregate numbers are saved.

New: training sizes `SIZES = (30, 100, 300)` examples PER CATEGORY. The training subset of size n is the first n examples (idx < n) of each category, so subsets are nested (30 in 100 in 300). It is built by the pure function `subsample_train(train, n)`, which keeps category order. The validation set is IDENTICAL for all sizes (the 80 examples idx 300..379 of each category, 640 messages), so NLL is comparable across sizes.

## Model and training
Identical to v7: randomly initialised 2-layer transformer (width 64, 4 heads, FF 128), AdamW lr 0.001, weight decay 0.01, batch 24, TF32 off. Seeds 2027, 3037, 4049, 5051, 6067 (paired). The number of updates is constant: 800 at every size, checkpoints at 200, 400, 800. Small sizes therefore see many more epochs; this is intended and listed as a limitation. For each (size, seed) the batch sequence is `torch.randint(len(train_n), (800, 24), generator=torch.Generator().manual_seed(seed + 30000))`, identical for all conditions of that (size, seed). Initial weights are identical for all conditions of a seed (`torch.manual_seed(seed)`).

## Conditions (2-token prefix)
Same as v7: `none` [256, 256]; `hierarchy` [265 + category // 4, 267 + category % 4]; `scrambled_1..5` = hierarchy code of a permutation `perm[category]`. Permutation seeds (31415, 27182, 16180, 14142, 17320), fallback (11235, 13579, 24680, 97531), and the rejection rule (reject if categories 0-3 all fall in one `code // 4` group) are the v7 ones, via the same `select_permutations()`.

## Metrics
As in v7, per size and checkpoint, on the same 640 validation messages, in nats per byte (lower is better): `eval_nll_labeled`, `eval_nll_ablated` (dummy prefix [256, 256]), `label_use_gap` = ablated - labeled, and perplexities. Each entry of `runs` carries `train_per_category`.
- `summary[str(size)]`: the v7 `compute_summary` applied to the runs of that size (seed-mean NLL per condition, paired difference hierarchy - mean(scrambled), number of scrambled better than hierarchy, hierarchy rank among the 6 codes, range between the 5 scrambled permutations).
- `summary_label_use[str(size)][str(step)]`: seed-mean `label_use_gap` for each condition, with the scrambled aggregated as `scrambled_mean` (computed by the pure function `compute_label_use`).
No significance test is run and no data-savings percentage is computed.

## Predictions declared in advance
- P1: at 300 per category we recover v7: hierarchy is about equal to the mixed codes (mean paired gap smaller than the range between permutations, about 0.005 nats/byte).
- P2: `label_use_gap` of the hierarchy condition at 30 per category is larger than at 300 per category.
- P3: hierarchy beats the mean of the scrambled at 30 per category on at least 4 of 5 seeds AND with a mean paired gap larger than the range between permutations. If this is not the case, the null result is confirmed over the whole range of sizes. If it is the case, this is a first positive signal for H1, to be replicated before any conclusion.
- Reading rule: no update of confidence in H1 is decided before the analysis; no claim of significance is made.

## Limitations
Five seeds, five permutations, only two groups of four categories, short 2-token prefixes, a tiny model, one historical text dataset (20 Newsgroups). The number of updates is held at 800 whatever the size, so small sizes see many more epochs (at 30 per category, 800 x 24 draws over 240 examples is about 80 epochs) and may overfit; the size effect is therefore mixed with an epoch effect. No significance test and no percentage of data saved. A null result is informative. Not a pretrained-LLM or general data-savings result. Labels are present at labeled evaluation.

## Budget and running
3 sizes x 5 seeds x 7 conditions = 105 models; estimated about 335 s (v7: 35 models in 111 s). The GPU budget is guarded at 600 s with a 15 s margin (abort at 585 s) and the run aborts rather than report a partial result. Call `run_realtext_v9()` on a CUDA machine. Smoke tests: `run_realtext_v9(device='cpu', steps=..., seeds=..., sizes=...)` (marked `smoke_test_not_protocol`, no budget guard).
