# H1 real-text pilot v12 protocol (v11 structure, different category groups)

Written before execution. Code: `h1_realtext_v12.py`, entry point `run_realtext_v12()`.

## Question
v10 and v11 (dropout 0.2, weight decay 0.1) showed the hierarchy code very slightly better than the mean of the scrambled (mixed) codes at 200-400 updates with 30 or 100 examples per category (gaps of about -0.0003 to -0.0013 nats per byte), but below the between-permutation range (a non-material tendency), and always with the same 8 categories (4 comp.*, 4 rec.*). Does the SIGN of the gap reproduce with a DIFFERENT category structure, everything else being identical?

## Data
Same dataset, revision and procedure as v11: `SetFit/20_newsgroups`, frozen revision `f1b91292074e7cfb69be58b642d583ec262f30ed`, `train.jsonl` only (the official test split is unused), whitespace-normalised UTF-8, first 96 bytes, 300 training (idx < 300) and 80 validation (idx 300..379) messages per category. `load_data()` is unchanged apart from the category list. Only aggregate numbers are saved.

New categories (the ONLY data change):
- group 0 (codes 0-3, "sci"): `sci.crypt`, `sci.electronics`, `sci.med`, `sci.space`
- group 1 (codes 4-7, "politics/society"): `talk.politics.guns`, `talk.politics.mideast`, `talk.politics.misc`, `soc.religion.christian`

Eligible messages (>= 96 bytes after whitespace normalisation) counted on the frozen revision before writing the code: sci.crypt 556, sci.electronics 542, sci.med 560, sci.space 547, talk.politics.guns 507, talk.politics.mideast 513, talk.politics.misc 427, soc.religion.christian 570. All are >= 380, so no replacement was needed. `selected_data_sha256` differs from v7-v11 because the categories differ. The hierarchical code stays `category // 4` / `category % 4` (no encoding change).

Training sizes `SIZES = (30, 100)` per category, nested subsets via `subsample_train`; the validation set is identical for both sizes (640 messages).

## Model and training
Identical to v11: randomly initialised 2-layer transformer (width 64, 4 heads, FF 128), batch 24, TF32 off, AdamW lr 0.001, 800 updates, `DROPOUT = 0.2` (training only, eval mode at every evaluation), `WEIGHT_DECAY = 0.1`. Batch sequence per (size, seed): `torch.randint(len(train_n), (800, 24), generator=torch.Generator().manual_seed(seed + 30000))`, identical for all conditions. Initial weights identical for all conditions of a seed.

Changes versus v11:
- Categories (above).
- Seeds `SEEDS = (12101, 13103, 14107, 15109, 16111)`. v10: 2027, 3037, 4049, 5051, 6067; v11: 7079, 8081, 9091, 10093, 11099; no overlap.
- Permutation seeds `PERMUTATION_SEEDS = (10101, 20202, 30303, 40404, 50505)`, fallback `(60606, 70707, 80808, 90909)`. v10: 31415, 27182, 16180, 14142, 17320 (fallback 11235, 13579, 24680, 97531); v11: 42424, 55555, 66666, 77777, 88888 (fallback 99991, 12121, 23232, 34343); no overlap. Same rejection rule (independent of the category names): a permutation is rejected if categories 0-3 all fall in the same code//4 group.
- Output `protocol` adds `replication_of`, `category_groups`, `category_set_differs_from_v10_v11`.

Checkpoints: 100, 200, 400, 800. All are reported; there is no checkpoint selection on the validation set.

## Conditions (2-token prefix)
As in v11: `none` [256, 256]; `hierarchy` [265 + category // 4, 267 + category % 4]; `scrambled_1..5` = hierarchy code of a permutation `perm[category]`.

## Pre-registered cells
Four cells, taken from v10/v11 (chosen AFTER seeing v10): size in {30, 100} x checkpoint in {200, 400}. For each cell `compute_replication_cells` reports `mean_paired_diff` (mean over seeds of hierarchy - mean of the 5 scrambled), `seeds_hierarchy_better` (out of 5), `mean_between_permutation_range`, `sign_negative`, `wins_at_least_4_of_5`, `material` (`mean_paired_diff < -mean_between_permutation_range`), and the aggregate counts `n_cells_sign_negative`, `n_cells_wins_at_least_4_of_5`, `n_cells_material`. Outputs: `summary`, `summary_label_use`, `replication_cells`. No significance test, no data-savings percentage.

## Predictions declared in advance
Baseline: under the null hypothesis, P(mean gap < 0) = 0.5 per cell, so at least 3 negative cells out of 4 happen by chance about 31% of the time (5/16). The cells are not independent (same seeds at the 2 checkpoints), so this figure is only indicative.
- P1 (sign): mean gap hierarchy - scrambled negative in at least 3 of the 4 cells. Prior: about 45%.
- P2 (regularity): hierarchy beats the scrambled mean on at least 4 seeds out of 5 in at least 3 of the 4 cells. Prior: about 25%.
- P3 (materiality): at least 2 of the 4 cells are `material`. Prior: about 10%.
- P4 (sanity, relevance of the grouping): the mean `label_use_gap` of `hierarchy` at 100 per category and 800 updates (`summary_label_use['100']['800']['hierarchy']`) is > 0, i.e. the model uses the label. If not, the grouping is badly conditioned and v12 is uninformative.
- Reading rule: read v12 alone, then together with v10+v11 as a replication on other categories, not as additional seeds. No update of confidence in H1 before that analysis. No claim of significance. A sign that reproduces but is not material is reported as a "non-material tendency".

## Limitations
A single regularisation setting (dropout 0.2, weight decay 0.1), not tuned. The new split (sci vs politics/society) may be more or less semantically clean than comp vs rec. The four cells come from v10/v11 and were chosen AFTER seeing v10. The comparison with v10/v11 is unpaired because the categories change (different data, different validation set). Five seeds, five permutations, two groups of four categories, short 2-token prefixes, a tiny model, one historical text dataset (20 Newsgroups). No significance test and no percentage of data saved. The cells share seeds across checkpoints and are not independent. A null result is informative. Not a pretrained-LLM or general data-savings result. Labels are present at labeled evaluation.

## Budget and running
2 sizes x 5 seeds x 7 conditions = 70 models, about 230-250 s as in v11. GPU budget guarded at 600 s with a 15 s margin (abort at 585 s); the run aborts rather than report a partial result; the guard is disabled in custom (smoke-test) mode. Call `run_realtext_v12()` on a CUDA machine. Smoke tests: `run_realtext_v12(device='cpu', steps=..., seeds=..., sizes=...)` (marked `smoke_test_not_protocol`).
