# H1 real-text pilot v8 protocol (three-level taxonomy, 16 categories)

Written before execution. Code: `h1_realtext_v8.py`, entry point `run_realtext_v8()`.

## Question
In v6 and v7, a two-token hierarchical label over two groups of four categories (comp.*, rec.*) was not distinguishable from an arbitrary labelling. Does a structure with three levels over 16 categories help where two groups of four showed nothing? And which level brings what: the top group, the middle split, or the leaf?

## Data
Same recipe as v7: `SetFit/20_newsgroups`, frozen revision `f1b91292074e7cfb69be58b642d583ec262f30ed`, `train.jsonl` only (the official test split is unused), whitespace-normalised UTF-8, first 96 bytes, 300 training and 80 validation messages per category (1280 validation messages), in repository order. 16 categories (the exact `label_text` values; spelling checked against the dataset). If a category has fewer than 380 eligible messages the run stops with an error naming the category and the number found; nothing partial is reported. Selected-data SHA-256 is recorded. Messages are not persisted; only aggregate numbers are saved.

## Taxonomy (4 x 2 x 2 = 16 leaves; index = category id)
- 0 comp.sys.ibm.pc.hardware, 1 comp.sys.mac.hardware, 2 comp.graphics, 3 comp.os.ms-windows.misc (top comp; mid hardware / software)
- 4 rec.autos, 5 rec.motorcycles, 6 rec.sport.baseball, 7 rec.sport.hockey (top rec; mid vehicles / sport)
- 8 sci.crypt, 9 sci.electronics, 10 sci.med, 11 sci.space (top sci; mid tech / life-and-space)
- 12 talk.politics.guns, 13 talk.politics.mideast, 14 alt.atheism, 15 soc.religion.christian (top society; mid politics / religion)

`top = c // 4`, `mid = (c // 2) % 2`, `leaf = c % 2`. The mid-level grouping is a judgement call made before any result; several pairs are debatable (e.g. sci.med with sci.space, guns with mideast).

## Model and training
As in v7: randomly initialised 2-layer transformer (width 64, 4 heads, FF 128), AdamW lr 0.001, weight decay 0.01, batch 24, TF32 off, seeds 2027, 3037, 4049, 5051, 6067 (paired), same initial weights and same batch sequence for every condition of a seed. 800 updates; checkpoints at 200, 400, 800. Only the text bytes enter the loss.

## Prefix: always exactly 3 tokens
Vocabulary 281: bytes 0-255, dummy 256, top 257-260, mid 261-262, leaf 263-264, flat 265-280. The sequence is 3 prefix tokens + 96 bytes = 99; the model reads the first 98, and the loss uses the outputs at input positions 2..97 (`logits[:, 2:]`), which predict the 96 text bytes (`chunk[:, 3:]`); the outputs at positions 0 and 1 predict prefix tokens and are not scored. NLL is normalised by 96 bytes per message. The pure function `prefix_for(condition, c)` defines all prefixes.

## Conditions (10; 10 x 5 seeds = 50 models)
- none: [256, 256, 256]
- flat: [265 + c, 256, 256] (16 arbitrary tags, no structure)
- top_only: [257 + top, 256, 256]
- top_mid: [257 + top, 261 + mid, 256]
- hierarchy: [257 + top, 261 + mid, 263 + leaf]
- scrambled_1 ... scrambled_5: a permutation P of the 16 categories, `code = P[c]`, prefix [257 + code // 4, 261 + (code // 2) % 2, 263 + code % 2] (same token shapes as hierarchy, structure destroyed).

Permutations are `random.Random(s).shuffle(list(range(16)))` for s in (31415, 27182, 16180, 14142, 17320), fixed before any result. Rejection rule (chosen as the simplest to test): count the pairs of categories that belong to the same original family of four ({0-3}, {4-7}, {8-11}, {12-15}) and whose codes share `code // 4`; there are 24 such pairs. A permutation is rejected if the count is at least 6. The identity and every permutation that keeps the top-level partition give 24, and a family falling entirely into one code group gives at least 6, so all are rejected; a random permutation averages 4.8. A rejected seed is replaced by the next unused fallback seed (11235, 13579, 24680, 97531). Selected and rejected seeds, and the permutations, are written in the JSON. Selection is the pure function `select_permutations()`.

## Metrics
At each checkpoint, on the same validation messages, in nats per byte (lower is better): `eval_nll_labeled`, `eval_nll_ablated` (dummy prefix [256, 256, 256] for every condition), `label_use_gap` = ablated - labeled. Only labeled evaluation enters the `summary`, computed in the code, per checkpoint:
- seed-mean NLL of each of the 10 conditions and of the mean of the scrambled;
- paired difference per seed `hierarchy - mean(scrambled_1..5)` (mean, min, max; negative = hierarchy better), number of seeds where hierarchy is better, number of scrambled that beat hierarchy, rank of hierarchy among the 6 codes (hierarchy + 5 scrambled; 1 = best), and the range between the 5 scrambled permutations as the reference dispersion (as in v7);
- seed-means of none, flat, top_only, top_mid, hierarchy and the successive differences top_only - none, top_mid - top_only, hierarchy - top_mid.
No significance test is run and no data-savings percentage is computed.

## Prediction declared in advance
1. If the hierarchical structure matters, at 800 updates hierarchy beats the mean of the scrambled on at least 4 of the 5 seeds (paired difference negative) AND the mean paired difference exceeds in absolute value the dispersion between permutations (mean over seeds of the scrambled range).
2. Otherwise the null result of v6 and v7 is confirmed: hierarchy is not distinguishable from an arbitrary three-token labelling.
3. The successive differences top_only/top_mid/hierarchy are reported descriptively, with no claim about which level is "responsible"; flat is reported as the reference for arbitrary 16-way tags.
4. The rank of hierarchy among the 6 codes is reported without a significance claim (1 in 6 by chance per seed).
5. A null result is informative and will be reported as such.

## Limitations
Five seeds, five permutations, one fixed taxonomy whose mid-level split is a judgement call, short 3-token prefixes, a tiny model trained for 800 updates, one historical text dataset (20 Newsgroups, only 16 of its 20 categories). Labels are present at labeled evaluation. Successive differences are confounded by the number of non-dummy tokens (each level adds a token). No significance claim and no percentage of data saved. Not a pretrained-LLM or general data-savings result. The default run trains 50 models (v7: 35 models in about 111 s, so roughly 160 s expected); the GPU budget is guarded at 600 s with a 15 s margin (abort at 585 s) and the run aborts rather than report a partial result.

## Running
Call `run_realtext_v8()` on a CUDA machine and save the JSON. Smoke tests: `run_realtext_v8(device='cpu', steps=..., seeds=...)` (marked `smoke_test_not_protocol`). Logic tests without torch: `python3 -I out8/test_v8_logic.py`.
