# H2 pilot v1 protocol (toy: facts in the weights vs facts in an external memory)

Written before execution. Code: `h2_pilot_v1.py`, entry point `run_h2_pilot_v1()`. Hypothesis H2: "learn the top, look up the bottom" - reserve the parameters for principles and reasoning, put the facts in an external memory.

## Question
Toy link tested: does the space freed by an external memory go to reasoning, at equal budget? In a synthetic world where facts must be known to apply a rule, how does the rule accuracy on held-out pairs depend on whether the facts are stored in the weights (`internal_*`) or supplied by an ideal memory (`external_64`), as the number of facts N grows?

## World
N entities (N in {64, 256, 1024}), one fact per entity: `v[e]` in {0..9}, drawn at random with the world seed (`seed + 10000 + N`), fixed during training and evaluation. Vocabulary (1039 tokens): entity tokens 0..1023, digits 1024..1033, `Q_FACT` 1034, `Q_RULE` 1035, `SEP` 1036, `M` (memory marker) 1037, `PAD` 1038 (right padding after the answer, never in the loss). The structural tokens are the "role" tokens.
- FACT: `[Q_FACT, e, SEP] -> v[e]`.
- RULE: `[Q_RULE, a, b, SEP] -> (v[a] + v[b]) mod 10`, a != b.
Loss: cross-entropy on the answer token only (logits at the SEP position, next token as target). Training mix: 50% FACT, 50% RULE (exactly 32 + 32 per batch).

Held-out pairs: a pair is a test pair iff `pair_hash(a, b) % 5 == 0`, a deterministic integer mix independent of any seed. The hash is symmetric in (a, b) because the rule is commutative (otherwise the swapped training pair would leak the test answer). Training samples only non-test pairs (rejection sampling). Evaluation uses 2000 test pairs drawn with a fixed seed, with replacement when fewer exist (at N = 64 there are about 800 ordered test pairs; `heldout_pair_stats` is in the output). FACT evaluation: all N entities (first 1000 at most), i.e. facts seen in training.

## Conditions
All with dropout 0 (no regularisation: capacity is the variable).
- `internal_64`: causal Transformer, 2 layers, width 64, 4 heads, FF 128, no memory: facts must be learned in the weights.
- `internal_128`: 2 layers, width 128, 4 heads, FF 256 (about 2.6x the parameters); same inputs.
- `external_64`: same model as `internal_64`; the retrieved memory (oracle retrieval, search cost ignored) is given in the context. RULE `[M, a, v[a], M, b, v[b], Q_RULE, a, b, SEP]`; FACT `[M, e, v[e], Q_FACT, e, SEP]`. The model only has to learn the rule / copying.
Parameter counts per condition are in `parameter_counts` (identical for N, since the vocabulary is shared). Maximum sequence length 11 (external RULE, answer included); position embedding of 16.

## Training and evaluation
AdamW lr 0.001, weight decay 0.01, batch 64, 2000 updates, TF32 off. Checkpoints 500, 1000, 1500, 2000, all reported; no checkpoint selection; the primary checkpoint for the predictions is fixed at 2000. For a given seed and N, all three conditions see the same batch sequence (`seed + 30000 + N`); `internal_64` and `external_64` also have identical initial weights (same seed, same architecture). Evaluation runs in `model.eval()` and the train mode is restored. Metric: top-1 accuracy of the answer token over the whole vocabulary; `chance_level` = 0.1.

Seeds `SEEDS = (31013, 32027, 33029, 34039, 35051)`, disjoint from the v10-v15 seeds (checked by the logic test file shipped with the code). 3 conditions x 3 N x 5 seeds = 45 models.

## Outputs (aggregates only)
For each (N, condition, checkpoint): `rule_acc_heldout` and `fact_acc` (seed mean and per seed), `mean_rule_acc` (seed mean of `rule_acc_heldout`), `chance_level`. Plus `parameter_counts`, `summary`, `predictions_check` (measured values and pass/fail of P1-P5 at 2000, descriptive), `elapsed_seconds`, `protocol`, `environment`; `smoke_test_not_protocol` only in custom mode. No significance test, no percentage of savings.

## Predictions declared in advance (judged at checkpoint 2000, mean over the 5 seeds)
- P1 (external memory): at N = 1024, `external_64` has `rule_acc_heldout` >= 0.90 AND `internal_64` <= 0.30. Prior 55%.
- P2 (few facts, no cost): at N = 64, the three conditions all have `rule_acc_heldout` >= 0.90. Prior 50%.
- P3 (more parameters do not replace the memory): at N = 1024, `internal_128` < `external_64` by at least 0.20 in `rule_acc_heldout`. Prior 60%.
- P4 (facts consume capacity): `internal_64` `fact_acc` decreases (or stays equal) as N goes 64 -> 256 -> 1024. Prior 85%.
- P5 (control): `external_64` has `rule_acc_heldout` >= 0.90 at N = 64 too (the rule is learnable). Prior 70%. If P5 fails, the experiment is badly conditioned and says nothing about H2.

Reading rule: exploratory and toy; says nothing about real LLMs or about the cost of retrieval (oracle retrieval); no update of the confidence in H2 (64%).

## Limitations
Synthetic world with a single rule (addition mod 10); random facts without structure; perfect retrieval; a single small model family; 2000 updates; 5 seeds; no significance test; the direction of the result is partly expected, so the experiment mostly locates the tipping point; no regularisation; the held-out pairs involve entities seen in training (only the pair is new); `fact_acc` is measured on training facts.

## Budget and running
45 models x 2000 updates. GPU guard 600 s, abort at 585 s without a partial result. Fail-fast: after the first complete seed (all N, all conditions), the total is projected as elapsed x seeds / seeds_done; above 585 s the run aborts at once. Both guards are off in custom (smoke-test) mode (`smoke_test_not_protocol`). Call `run_h2_pilot_v1()` on a CUDA machine; smoke tests: `run_h2_pilot_v1(device='cpu', steps=..., seeds=..., ns=...)`.

## Amendment before any protocol result (2026-11-19)
The first launch with 3000 updates aborted by design after the first seed (projected GPU time 627 s > 585 s); no result was produced. The number of updates is reduced to 2000 (checkpoints 500, 1000, 1500, 2000; primary checkpoint 2000). Nothing else changes. A short CPU timing check of one seed (seed 1, N=1024, 3000 updates) had been run before the first launch to estimate the duration; it is not a protocol result, but its accuracies were seen by the author before the predictions were judged.
