# H1 real-text pilot v7 protocol (several scrambled permutations)

Written before execution. Code: `h1_realtext_v7.py`, entry point `run_realtext_v7()`.

## Question
In v6, hierarchy was about equal to a single scrambled labelling. Does that null result hold when the scrambled condition is not one permutation but five different ones? With a single permutation the comparison could be an accident of that permutation; five give a sense of the spread between permutations.

## Data
Identical to v6: `SetFit/20_newsgroups`, frozen revision `f1b91292074e7cfb69be58b642d583ec262f30ed`, `train.jsonl` only (the official test split is unused). Same 8 categories (4 comp.*, 4 rec.*), whitespace-normalised UTF-8, first 96 bytes, 300 training and 80 validation messages per category (640 validation messages). Selected-data SHA-256 is recorded in the result. Do not persist the messages; only aggregate numbers are saved.

## Model and training
Identical to v6: randomly initialised 2-layer transformer (width 64, 4 heads, FF 128), AdamW lr 0.001, weight decay 0.01, batch 24, TF32 off. Seeds 2027, 3037, 4049, 5051, 6067 (paired). For a given seed, every condition gets the same initial weights and the same sequence of batches. 800 updates; checkpoints at 200, 400, 800. Loss covers text bytes only.

## Conditions (2-token prefix)
- none: [256, 256]
- hierarchy: [265 + category // 4, 267 + category % 4]
- scrambled_1 ... scrambled_5: the hierarchy code of a permutation of the 8 categories, `code = perm[category]`, prefix [265 + code // 4, 267 + code % 4].

Permutations are `random.Random(s).shuffle(list(range(8)))` for s in (31415, 27182, 16180, 14142, 17320), fixed before any result. A permutation is rejected if it does not break the comp/rec grouping, i.e. if categories 0-3 all fall in the same `code // 4` group. A rejected seed is replaced by the next unused seed of the fallback list (11235, 13579, 24680, 97531). Selected and rejected seeds, and the permutations themselves, are written in the JSON (`protocol`). The selection is implemented in the pure function `select_permutations()`.

## Metrics
At each checkpoint, on the same 640 validation messages, in nats per byte (lower is better): `eval_nll_labeled`, `eval_nll_ablated` (dummy prefix [256, 256] for every condition) and `label_use_gap` = ablated - labeled, as in v6. Only labeled evaluation enters the `summary` block, which is computed in the code and contains, per checkpoint:
- seed-mean labeled NLL of `none`, `hierarchy`, each `scrambled_k`, and of the mean of the scrambled;
- paired difference per seed `hierarchy - mean(scrambled_1..5)`: mean, min, max over seeds (negative = hierarchy better);
- number of `scrambled_k` that beat hierarchy (per seed, and mean over the 5 seeds);
- rank of hierarchy among the 6 codes (hierarchy + 5 scrambled), per seed and mean (1 = best);
- range between the 5 scrambled permutations (max - min), as the reference dispersion.
No significance test is run and no data-savings percentage is computed.

## Prediction declared in advance
1. If the hierarchical structure matters, at 800 updates hierarchy should beat the mean of the scrambled on a majority of the 5 seeds (paired difference negative on at least 3 of 5), and few scrambled_k should beat hierarchy.
2. If the paired differences stay comparable in size to the range between the scrambled permutations, the null result of v6 is confirmed (hierarchy is not distinguishable from an arbitrary 4+4 labelling, which is consistent with the model using the label only as an arbitrary category tag).
3. We report the rank of hierarchy among the 6 codes without any claim of significance. With 6 codes, a rank of 1 would occur by chance 1 time in 6 per seed.
4. A null result is informative and will be reported as such.

## Limitations
Five seeds, five permutations, only two groups of four categories (so little room for a hierarchy to differ from a scrambling), short 2-token prefixes, a tiny model trained for 800 updates, one historical text dataset (20 Newsgroups). Labels are present at labeled evaluation. No significance claim and no percentage of data saved. Not a pretrained-LLM or general data-savings result. The default run trains 7 conditions x 5 seeds = 35 models; GPU budget is guarded at 600 s with a 15 s margin (abort at 585 s) and the run aborts rather than report a partial result.

## Running
Call `run_realtext_v7()` on a CUDA machine and save the JSON. Smoke tests: `run_realtext_v7(device='cpu', steps=..., seeds=...)` (marked `smoke_test_not_protocol`).
