# H1 real-text pilot v11 protocol (replication of the v10 constant gap)

Written before execution. Code: `h1_realtext_v11.py`, entry point `run_realtext_v11()`.

## Question
In v10 (dropout 0.2, weight decay 0.1), the hierarchy code was almost always very slightly better than the mean of the scrambled (mixed) codes at checkpoints 200 and 400 (paired gaps from -0.0003 to -0.0023 nats per byte, well below the between-permutation range). Does the SIGN of this gap reproduce with new seeds and new permutations, with nothing else changed?

## Data
Identical to v10 (and v7/v9): `SetFit/20_newsgroups`, frozen revision `f1b91292074e7cfb69be58b642d583ec262f30ed`, `train.jsonl` only (the official test split is unused). Same 8 categories (4 comp.*, 4 rec.*), whitespace-normalised UTF-8, first 96 bytes. `load_data()` is unchanged: 300 training (idx < 300) and 80 validation (idx 300..379) messages per category; `selected_data_sha256` equals the v7/v9/v10 value. Only aggregate numbers are saved.

Training sizes `SIZES = (30, 100)` examples PER CATEGORY, nested subsets via `subsample_train`. The validation set is identical for both sizes (640 messages).

## Model and training
Identical to v10: randomly initialised 2-layer transformer (width 64, 4 heads, FF 128), batch 24, TF32 off, AdamW lr 0.001, 800 updates, `DROPOUT = 0.2` (training only, eval mode at every evaluation), `WEIGHT_DECAY = 0.1`. Batch sequence per (size, seed): `torch.randint(len(train_n), (800, 24), generator=torch.Generator().manual_seed(seed + 30000))`, identical for all conditions. Initial weights identical for all conditions of a seed.

The ONLY changes versus v10:
- Seeds `SEEDS = (7079, 8081, 9091, 10093, 11099)` (v10: 2027, 3037, 4049, 5051, 6067; no overlap).
- Permutation seeds `PERMUTATION_SEEDS = (42424, 55555, 66666, 77777, 88888)`, fallback `(99991, 12121, 23232, 34343)` (v10: 31415, 27182, 16180, 14142, 17320 and fallback 11235, 13579, 24680, 97531; no overlap). Same rejection rule and same `select_permutations()`: a permutation is rejected if categories 0-3 all fall in the same code//4 group.
- New pure function `compute_replication_cells(summary)` and the output key `replication_cells`.

Checkpoints: 100, 200, 400, 800. All are reported; there is no checkpoint selection on the validation set.

## Conditions (2-token prefix)
As in v10: `none` [256, 256]; `hierarchy` [265 + category // 4, 267 + category % 4]; `scrambled_1..5` = hierarchy code of a permutation `perm[category]`.

## Pre-registered cells
Four cells, fixed before the run: size in {30, 100} x checkpoint in {200, 400}. For each cell `compute_replication_cells` reports:
- `mean_paired_diff`: mean over seeds of (hierarchy - mean of the 5 scrambled), nats per byte;
- `seeds_hierarchy_better`: number of seeds (out of 5) where hierarchy < scrambled mean;
- `mean_between_permutation_range`: mean over seeds of (max - min over the 5 scrambled);
- `sign_negative`: `mean_paired_diff < 0`;
- `wins_at_least_4_of_5`: `seeds_hierarchy_better >= 4`;
- `material`: `mean_paired_diff < -mean_between_permutation_range`.

Aggregate counts: `n_cells_sign_negative`, `n_cells_wins_at_least_4_of_5`, `n_cells_material`. The result is stored under `replication_cells` (with `cells` and `missing_cells`; a cell absent from a custom/smoke run is omitted and listed in `missing_cells`), in addition to `summary` and `summary_label_use`. No significance test, no data-savings percentage.

## Predictions declared in advance
Baseline: under the null hypothesis (no effect), P(mean gap < 0) = 0.5 per cell, so at least 3 negative cells out of 4 happen by chance about 31% of the time (5/16). The cells are not independent (the same seeds are used at the 2 checkpoints), so even this figure is only indicative.
- P1 (sign): the mean gap hierarchy - scrambled is negative in at least 3 of the 4 cells. Author's prior: about 40%.
- P2 (regularity): the hierarchy beats the scrambled mean on at least 4 seeds out of 5 in at least 3 of the 4 cells. Author's prior: about 25%.
- P3 (materiality): at least 2 of the 4 cells are `material` (mean gap < -mean between-permutation range). Author's prior: about 5%. If P3 fails, any gap remains within the permutation noise even if its sign reproduces.
- Reading rule: the main reading is the combination v10 + v11 (10 seeds, 10 permutations), to be done after the run. No update of confidence in H1 before that analysis. No claim of significance. A sign that reproduces but is not material is reported as a "non-material tendency".

## Limitations
A single regularisation setting (dropout 0.2, weight decay 0.1), unchanged from v10 and not tuned; five more seeds and five more permutations, but still only two groups of four categories, short 2-token prefixes, a tiny model, one historical text dataset (20 Newsgroups). No significance test and no percentage of data saved. The four cells were chosen AFTER seeing v10, so this replication tests a post hoc observation. The cells share seeds across checkpoints and are not independent. A null result is informative. Not a pretrained-LLM or general data-savings result. Labels are present at labeled evaluation.

## Budget and running
2 sizes x 5 seeds x 7 conditions = 70 models; estimated about 230-250 s, same as v10. The GPU budget is guarded at 600 s with a 15 s margin (abort at 585 s) and the run aborts rather than report a partial result; the guard is disabled in custom (smoke-test) mode. Call `run_realtext_v11()` on a CUDA machine. Smoke tests: `run_realtext_v11(device='cpu', steps=..., seeds=..., sizes=...)` (marked `smoke_test_not_protocol`); smoke checkpoints are {max(1,steps//8), max(1,steps//4), max(1,steps//2), steps}.
