# H2 pilot v7 protocol (toy: three facts in memory, trained longer)

Written before execution. Code: `h2_pilot_v7.py`, entry point `run_h2_pilot_v7()`. Hypothesis H2: "learn the top, look up the bottom" - reserve the parameters for principles and reasoning, put the facts in an external memory.

## Context: what v1-v5 showed (known to the authors when this protocol was written)
- v1: with the facts in the context, `external_64` composes held-out PAIRS (RULE = (v[a] + v[b]) mod 10) at about 1.0 at N = 1024.
- v5: with THREE facts (RULE3 = (v[a] + v[b] + v[c]) mod 10), `external_64` trained 2000 updates reached about 0.27 on average at N = 1024, with a large spread over seeds (best seed about 0.93, worst about chance); `internal_64_steps` (explicit steps without an intermediate sum) stayed near chance (about 0.12) at 2000 updates, while the format with an intermediate partial sum reached about 1.0.
- v7 asks whether the weak v5 results of the external memory and of the plain steps are a matter of too few updates, and whether a warm-up on the two-fact rule helps the external model to learn the three-fact one.

Method note: the predictions below are written AFTER v1-v5 and BEFORE the v6 and v7 results (this protocol was written while v6 was running).

## Question
(a) Does `external_64` learn RULE3 reliably when trained 4000 updates (instead of 2000)? (b) Does a warm-up on the two-fact rule (external format, 1500 updates) followed by RULE3 do at least as well? (c) Do the explicit steps without an intermediate sum learn RULE3 with 4000 updates?

## World (identical to v5 for everything not listed here)
N in {256, 1024} entities, all seen in training, one fact `v[e]` in 0..9 per entity (seed + 230000 + N). N = 1024 is judged, N = 256 is reported only. Vocabulary: the 1040 tokens of v5 (including `Q_RULE3`), position table of 16 entries. Rule `RULE3`: `ans = (v[a] + v[b] + v[c]) mod 10` on triples of DISTINCT entities; held-out triples: a triple is a test triple iff `triple_hash(sorted(a, b, c)) % 5 == 0` (the v5 function, checked equal by the tests; every permutation of a test triple is a test triple); training samples only non-test triples; the 2000 evaluation triples are the first valid candidates of a fixed generator (seed 767676 + N, independent of the run seed). The warm-up uses the two-fact rule `RULE` on non-test pairs of the pair split of v1-v6 (`pair_hash(a, b) % 5 == 0` held out; the test triples of RULE3 are evaluated, not pairs). Training mix: 32 FACT + 32 RULE (RULE3 after the warm-up) per batch, batch 64. Seeds `SEEDS = (61197, 62201, 63211, 64223, 65227)`; generator offsets 230000 (world), 240000 (batches), 250000 (warm-up pairs), evaluation seed 767676: all different from v1-v6 (checked by the logic test file shipped with the code, including the derived generator seeds).
Model: causal Transformer, 2 layers, width 64, 4 heads, FF 128, dropout 0, AdamW lr 0.001, wd 0.01, TF32 off. Same initial weights for the three conditions of a given seed; same FACT entities and same RULE3 triples (for the updates where RULE3 is trained) for the three conditions of a given seed and N. 4000 updates; checkpoints 1000, 2000, 3000, 4000 (primary = 4000, fixed in advance, no checkpoint selection).

## Conditions (3), loss on the output tokens only
FACT: steps `[Q_FACT, e, SEP]` -> `v[e]`; external `[M, e, v[e], Q_FACT, e, SEP]` -> `v[e]`.
- `external_64` (oracle retrieval): RULE3 `[M, a, v[a], M, b, v[b], M, c, v[c], Q_RULE3, a, b, c, SEP]` -> `ans`, for the 4000 updates.
- `external_64_curriculum` (oracle retrieval): updates 1..1500 on the two-fact rule `[M, a, v[a], M, b, v[b], Q_RULE, a, b, SEP]` -> `(v[a] + v[b]) mod 10` (v1-v4 format, 11 tokens; the FACT rows stay the same), then updates 1501..4000 on RULE3 as `external_64`. Optimiser and weights are NOT reset at the change. Evaluated on RULE3 test triples only (the checkpoint 1000 falls in the warm-up: the RULE3 accuracy there is that of a model that has not yet seen RULE3).
- `internal_64_steps_long` (no memory): `[Q_RULE3, a, b, c, SEP]` -> `v[a], v[b], v[c], ans` (the `internal_64_steps` format of v5, no intermediate sum), 4000 updates, greedy autoregressive generation of the 4 tokens at evaluation (no teacher forcing); the answer is the 4th generated token.

## Metrics (2000 held-out triples; top-1 over the whole vocabulary, chance 0.1)
`rule_acc`: the answer (last generated token, or the single output token for the external conditions) is `(v[a] + v[b] + v[c]) mod 10`. `recall_acc` (steps): the generated tokens that must be `v[a]`, `v[b]`, `v[c]` are all correct. `fact_acc`: the FACT question on the first 1000 entities (control).

## Outputs (aggregates only)
`summary[str(N)][condition]["step_<k>"][metric]` = mean over seeds, min, max, and per seed, `chance_level`. Plus `parameter_counts`, `predictions_check` (measured values and pass/fail, descriptive), `protocol` (including the warm-up plan), `environment` (including the code hash), `timing_seconds_mean` (update loops, evaluations excluded), `elapsed_seconds`, `runs` and `smoke_test_not_protocol` (true only in custom mode). No significance test.

## Predictions declared in advance (judged at N = 1024, step 4000, mean over the 5 seeds unless stated)
- P1 (external memory is reliable with more updates): `external_64` `rule_acc` >= 0.90 at 4000 updates on at least 4 of the 5 seeds (per-seed values at N = 1024; the mean and range are also reported).
- P2 (warm-up helps or is neutral): `external_64_curriculum` `rule_acc` >= `external_64` `rule_acc` (means over the 5 seeds, 4000 updates). The gap curriculum minus external with its sign is reported whatever the outcome.
- P3 (steps without a partial sum, longer): `internal_64_steps_long` `rule_acc` >= 0.90 at 4000 updates. If it stays below, it is reported as it is (the prediction is then false).
- P4 (control): `fact_acc` >= 0.98 for all three conditions at N = 1024, 4000 updates (the N = 256 values are reported, not judged). If P4 fails, the other results are not readable.
- P5 (seed luck control): in no (condition, checkpoint) cell of `rule_acc` at N = 1024 does the best seed exceed 0.99 while the mean over the five seeds is at most 0.90 (a "lucky-seed" cell: a high value that only one seed reaches). Operationalisation fixed in advance. The range (min, mean, max) of every cell is reported. P5 holds iff there is no lucky-seed cell.

Reading rule: exploratory and toy; retrieval is ideal (oracle) in the external conditions, search cost not tested; no update of the confidence in H2; no significance test. N = 256 is secondary. No prior probability is attached to P1-P5 in this protocol.

## Limitations
One rule family (sums mod 10 of two or three facts); random independent facts; one small model; 5 seeds; no significance test; one warm-up length (1500 updates) only; the steps format is supervised reasoning (intermediate tokens given in training), not shown for real LLMs; held-out triples involve entities seen in training; `fact_acc` is measured on training facts; the new token `Q_RULE3` makes the initial weights differ from v1-v4 (same as v5); the warm-up changes the number of RULE3 updates of the curriculum condition (2500, against 4000).

## Budget and running
3 conditions x 2 N x 5 seeds = 30 trainings of 4000 updates. From v1-v5 the update loop is about 7.5 s per 2000 updates (launch-bound): about 15 s per training, about 455 s in all, plus evaluations (small): estimate 460-470 s, slightly above the 450 s level at which a reduction would be proposed, below the 585 s abort. The number of updates is kept at 4000 as required (no amendment). GPU guard 600 s, abort at 585 s without a partial result. Fail-fast: after the first complete seed, the total is projected as elapsed x seeds / seeds_done; above 585 s the run aborts at once. Both guards are off in custom (smoke) mode. Call `run_h2_pilot_v7()` on a CUDA machine; smoke tests: `run_h2_pilot_v7(device='cpu', steps=..., seeds=..., ns=...)` (`steps` = number of updates; the checkpoints and the warm-up bound scale by steps / 4000).
