# H1 real-text pilot v15 protocol (larger model, stronger regularisation)

Written before execution. Code: `h1_realtext_v15.py`, entry point `run_realtext_v15()`.

## Question
Was the 4-layer model of v13 under-regularised? v15 keeps the v13 model, categories and data, and raises the regularisation (dropout 0.2 -> 0.3, weight decay 0.1 -> 0.3), the only change versus v13.

## Data
Identical to v13 (and v11): `SetFit/20_newsgroups`, frozen revision `f1b91292074e7cfb69be58b642d583ec262f30ed`, `train.jsonl` only (official test unused), whitespace-normalised UTF-8, first 96 bytes, 8 categories (`comp.graphics`, `comp.os.ms-windows.misc`, `comp.sys.ibm.pc.hardware`, `comp.sys.mac.hardware`, `rec.autos`, `rec.motorcycles`, `rec.sport.baseball`, `rec.sport.hockey`), 300 training (idx < 300) and 80 validation (idx 300..379) messages per category. `load_data()` unchanged, so `selected_data_sha256` must equal the v11/v13 value. Hierarchical code `category // 4`, `category % 4`. `SIZES = (100,)` per category (nested subset via `subsample_train`); validation identical (640 messages).

## Model and training
Identical to v13: causal Transformer `N_LAYERS = 4`, `D_MODEL = 128`, 4 heads, `FF = 256`, batch 24, AdamW lr 0.001, 800 updates, TF32 off, 2-token prefix, same batch generator (`seed + 30000`) for all conditions of a seed, identical initial weights per seed. The ONLY changes: `DROPOUT = 0.3` (training only; every evaluation runs in `model.eval()` and restores the train mode) and `WEIGHT_DECAY = 0.3` (used by `make_optimizer`, AdamW). The two change together, so their separate effects cannot be told apart.

Seeds `SEEDS = (23197, 24203, 25219, 26227, 27239)`. Permutation seeds `PERMUTATION_SEEDS = (13013, 24024, 35035, 46046, 57057)`, fallback `(68068, 79079, 81081, 92929)`. No overlap with v10 (seeds 2027, 3037, 4049, 5051, 6067; permutations 31415, 27182, 16180, 14142, 17320 + fallback 11235, 13579, 24680, 97531), v11 (7079, 8081, 9091, 10093, 11099; 42424, 55555, 66666, 77777, 88888 + 99991, 12121, 23232, 34343), v12 (12101, 13103, 14107, 15109, 16111; 10101, 20202, 30303, 40404, 50505 + 60606, 70707, 80808, 90909), v13 (13121, 14129, 15137, 16141, 17149; 11011, 22022, 33033, 44044, 55055 + 66066, 77077, 88088, 99099) or v14 (18163, 19169, 20173, 21179, 22189; 12012, 23023, 34034, 45045, 56056 + 67067, 78078, 89089, 91919); checked by `out15/test_v15_logic.py` for primary and fallback seeds. Same rejection rule as v11-v14 (reject a permutation keeping categories 0-3 in one code//4 group; replace by the next unused fallback seed, in order).

Checkpoints 100, 200, 400, 800, all reported; no checkpoint selection.

## Conditions (2-token prefix)
`none` [256, 256]; `hierarchy` [265 + category // 4, 267 + category % 4]; `scrambled_1..5`. 7 conditions x 5 seeds = 35 models.

## Pre-registered cells
Two cells: size 100 x checkpoint {200, 400} (`100_200`, `100_400`), taken from v10/v11 (chosen AFTER seeing v10). `compute_replication_cells` reports `mean_paired_diff`, `seeds_hierarchy_better` (of 5), `mean_between_permutation_range`, `sign_negative`, `wins_at_least_4_of_5`, `material` (`mean_paired_diff < -mean_between_permutation_range`), and the aggregate counts. Outputs: `summary`, `summary_label_use`, `replication_cells`, `reference_v13_none_nll_100_800`, `p5_vs_v13_indicative`. No significance test, no data-savings percentage.

Frozen reference (constant `REFERENCE_V13_NONE_NLL_100_800 = 2.3925`, from `the earlier run output`, `summary['100']['800']['mean_eval_nll_labeled_over_seeds']['none']` = 2.392498..., v13 seeds 13121..17149, dropout 0.2 / weight decay 0.1): not recomputed.

## Predictions declared in advance
- P1 (sign): mean gap hierarchy - scrambled mean negative in BOTH cells. Prior: about 45%.
- P2 (regularity): hierarchy beats the scrambled mean on at least 4 seeds out of 5 in BOTH cells. Prior: 25%.
- P3 (materiality): at least 1 cell is `material`. Prior: 10%.
- P4 (sanity): the mean `label_use_gap` of `hierarchy` at 100 per category and 800 updates (`summary_label_use['100']['800']['hierarchy']`) is > 0. If not, the model does not use the label and v15 is uninformative.
- P5 (indicative, UNPAIRED): the mean NLL of the `none` model at 800 updates (`summary['100']['800']['mean_eval_nll_labeled_over_seeds']['none']`) is lower than the v13 value 2.3925 (other seeds). Reading: stronger regularisation helps if the NLL falls. Reported in `p5_vs_v13_indicative`.
- Reading rule: no update of confidence in H1. No claim of significance. A sign that reproduces but is not material is reported as a "non-material tendency".

## Limitations
One larger model only (4 layers, width 128): not a scaling law. One stronger setting (dropout 0.3, weight decay 0.3), not tuned, which may under- or over-regularise; dropout and weight decay change together. Five seeds, five permutations, two cells that share seeds across checkpoints and are not independent. The comparison with v13 is unpaired (other seeds) and indicative only. No significance test and no percentage of data saved. Labels are present at labeled evaluation. Cells chosen AFTER seeing v10. One historical text dataset. Not a pretrained-LLM or general data-savings result.

## Budget and running
1 size x 5 seeds x 7 conditions = 35 models of the v13 size. GPU guard 600 s, abort at 585 s without partial result; fail-fast projection after the first finished seed (abort at once if the projection exceeds 585 s). Both guards are disabled in custom (smoke-test) mode (`smoke_test_not_protocol`). Call `run_realtext_v15()` on a CUDA machine; smoke tests: `run_realtext_v15(device='cpu', steps=..., seeds=..., sizes=...)`.
