# H1 real-text pilot v10 protocol (regularised small data)

Written before execution. Code: `h1_realtext_v10.py`, entry point `run_realtext_v10()`.

## Question
In v9, at 30 training examples per category the model overfits (validation NLL 2.78 at 200 updates, 4.12 at 800; about 80 epochs), and the hierarchy-minus-mixed gap (-0.012) is drowned by the range between permutations (0.089). If we regularise to limit this overfitting, does the hierarchy beat the mixed (scrambled) codes at small data sizes?

## Data
Identical to v9 (and v7): `SetFit/20_newsgroups`, frozen revision `f1b91292074e7cfb69be58b642d583ec262f30ed`, `train.jsonl` only (the official test split is unused). Same 8 categories (4 comp.*, 4 rec.*), whitespace-normalised UTF-8, first 96 bytes. `load_data()` is unchanged: 300 training (idx < 300) and 80 validation (idx 300..379) messages per category; `selected_data_sha256` equals the v7/v9 value. Only aggregate numbers are saved.

Training sizes `SIZES = (30, 100)` examples PER CATEGORY (size 300 is already covered by v7 and v9). The subset of size n is the first n examples of each category (nested, via the pure function `subsample_train`). The validation set is identical for both sizes (640 messages).

## Model and training
As in v9 except the regularisation: randomly initialised 2-layer transformer (width 64, 4 heads, FF 128), batch 24, TF32 off, AdamW lr 0.001, 800 updates.
- `DROPOUT = 0.2` (v9: 0), the `dropout` argument of `TransformerEncoderLayer`, active in training only. The model is in `train()` mode for the whole training (including before the first checkpoint); `evaluate` switches to `eval()` for every evaluation and restores the previous mode afterwards.
- `WEIGHT_DECAY = 0.1` (v9: 0.01), built by `make_optimizer(model)`.
The regularisation is a single setting chosen before the run and not tuned. Seeds 2027, 3037, 4049, 5051, 6067 (paired). Batch sequence per (size, seed): `torch.randint(len(train_n), (800, 24), generator=torch.Generator().manual_seed(seed + 30000))`, identical for all conditions. Initial weights identical for all conditions of a seed.

Checkpoints: 100, 200, 400, 800 updates. All checkpoints are reported. There is NO checkpoint selection on the validation set (that would leak); the primary criterion is fixed in advance at 800 updates.

## Conditions (2-token prefix)
As in v9: `none` [256, 256]; `hierarchy` [265 + category // 4, 267 + category % 4]; `scrambled_1..5` = hierarchy code of a permutation `perm[category]`. Permutation seeds (31415, 27182, 16180, 14142, 17320), fallback (11235, 13579, 24680, 97531), same rejection rule, same `select_permutations()`.

## Metrics
As in v9, per size and checkpoint, on the same 640 validation messages, in nats per byte (lower is better): `eval_nll_labeled`, `eval_nll_ablated` (dummy prefix), `label_use_gap` = ablated - labeled, perplexities. `summary[str(size)]` = `compute_summary` on the runs of that size; `summary_label_use[str(size)][str(step)]` = `compute_label_use`. No significance test, no data-savings percentage. The output `protocol` block records `dropout`, `weight_decay`, `regularisation_chosen_before_run: True`, `v9_reference`, `primary_checkpoint: 800`.

## Predictions declared in advance
- P1 (sanity): regularisation reduces overfitting: at 30 per category and 800 updates, the mean NLL of the `none` condition is below 3.5 (it was 4.12 in v9). If not, the chosen regularisation is insufficient and v10 does not answer its question.
- P2: `label_use_gap` of the `hierarchy` condition at 30 per category and 800 updates is smaller than in v9 (0.124).
- P3 (positive criterion for H1): at 30 OR 100 per category, at 800 updates, the hierarchy beats the mean of the scrambled on at least 4 of 5 seeds AND the mean paired gap (hierarchy - scrambled mean) is below -(mean between-permutation range of the scrambled). Otherwise the null result holds under regularisation. Author's prior: about 75% that the null holds.
- Reading rule: no update of confidence in H1 before the analysis; no claim of significance; a positive result should be replicated with other seeds and permutations.

## Limitations
A single regularisation setting (dropout 0.2, weight decay 0.1), chosen once and not tuned; five seeds, five permutations, only two groups of four categories, short 2-token prefixes, a tiny model, one historical text dataset (20 Newsgroups). The number of updates is held at 800, so at 30 per category there are still about 80 epochs. No significance test and no percentage of data saved. A null result is informative. The comparison to v9 is not matched on regularisation (v9: dropout 0, weight decay 0.01) and only covers sizes 30 and 100. Not a pretrained-LLM or general data-savings result. Labels are present at labeled evaluation.

## Budget and running
2 sizes x 5 seeds x 7 conditions = 70 models; estimated about 230-250 s. The GPU budget is guarded at 600 s with a 15 s margin (abort at 585 s) and the run aborts rather than report a partial result; the guard is disabled in custom (smoke-test) mode. Call `run_realtext_v10()` on a CUDA machine. Smoke tests: `run_realtext_v10(device='cpu', steps=..., seeds=..., sizes=...)` (marked `smoke_test_not_protocol`); smoke checkpoints are {max(1,steps//8), max(1,steps//4), max(1,steps//2), steps}.
