# H2 pilot v5 protocol (toy: composition of three facts)

Written before execution. Code: `h2_pilot_v5.py`, entry point `run_h2_pilot_v5()`. Hypothesis H2: "learn the top, look up the bottom" - reserve the parameters for principles and reasoning, put the facts in an external memory.

## Context: what v1-v4 showed (known to the authors when this protocol was written)
- v1: composing two facts, RULE = (v[a] + v[b]) mod 10 on held-out pairs: an internal model in one pass fails at N = 1024 (about chance) while it retains the facts; an external-memory model (facts in the context, ideal retrieval) succeeds.
- v2: an internal model with explicit steps (`[Q_RULE, a, b, SEP, v[a], v[b], ans]`, no memory) also reaches about 1.0 at N = 1024. In this toy, an external memory is therefore not necessary to compose two facts.
- v3, v4: updating knowledge, and imperfect retrieval.
- v5 asks whether the v2 result ("without a memory, explicit steps solve composition") holds when the rule involves THREE facts.

Method note: the predictions below are written AFTER v1-v4 (not blind to them) and BEFORE the v5 run.

## Question
When the rule needs three facts, (a) does a one-pass internal model still fail, (b) do the explicit steps still solve it without a memory, (c) does an intermediate partial sum in the steps change the result, (d) does the external-memory model still compose?

## World (identical to v1-v4 for everything not listed here)
N in {256, 1024} entities, all seen in training, one fact `v[e]` in 0..9 per entity (seed + 160000 + N). N = 1024 is judged, N = 256 is reported only. Vocabulary: the 1039 tokens of v1-v4 (entities 0..1023, digits 1024..1033, `Q_FACT`, `Q_RULE`, `SEP`, `M`, `PAD`) PLUS one new structural token `Q_RULE3` (id 1039), so 1040 tokens in all; the position table keeps its 16 entries (the longest sequence has 15 tokens).
Rule `RULE3`: `ans = (v[a] + v[b] + v[c]) mod 10` on triples `(a, b, c)` of DISTINCT entities. Held-out triples: a triple is a test triple iff `triple_hash(sorted(a, b, c)) % 5 == 0` (the pair-hash mix of v1-v4 applied to the sorted triple, symmetric and seed independent: every permutation of a test triple is a test triple, so no permutation of a test triple appears in training); training samples only non-test triples; the 2000 evaluation triples are the first valid candidates of a fixed generator (seed 939393 + N, independent of the run seed). Training mix: 32 FACT + 32 RULE3 per batch, batch 64. Seeds `SEEDS = (51151, 52153, 53161, 54167, 55171)`; generator offsets 160000 (world) and 170000 (batches), evaluation seed 939393: all different from v1-v4 (checked by the test file, including the derived generator seeds).
Model: causal Transformer, 2 layers, width 64, 4 heads, FF 128, dropout 0, AdamW lr 0.001, wd 0.01, TF32 off. Same initial weights for the four conditions of a given seed; same batches for the four conditions of a given seed and N. 2000 updates; checkpoints 500, 1000, 2000 (primary = 2000, fixed in advance, no checkpoint selection). A checkpoint at 3000 updates is NOT used: 4 conditions x 2 N x 5 seeds = 40 trainings at about 8 s per 2000 updates on the GPU (from v1-v3) would project to about 480 s of updates alone, above the 450 s level.

## Conditions (4), loss on the output tokens only
FACT: internal `[Q_FACT, e, SEP]` -> `v[e]`; external `[M, e, v[e], Q_FACT, e, SEP]` -> `v[e]`.
- `internal_64` (one pass, no memory): `[Q_RULE3, a, b, c, SEP]` -> `ans`.
- `internal_64_steps` (no memory): `[Q_RULE3, a, b, c, SEP]` -> `v[a], v[b], v[c], ans`. Evaluation by greedy autoregressive generation of the 4 tokens (no teacher forcing); the answer is the 4th generated token.
- `internal_64_steps_partial` (no memory): `[Q_RULE3, a, b, c, SEP]` -> `v[a], v[b], s1, v[c], ans` with `s1 = (v[a] + v[b]) mod 10`: one intermediate partial sum. Greedy autoregressive generation of the 5 tokens; the answer is the 5th. It measures the effect of an intermediate step.
- `external_64` (oracle retrieval): `[M, a, v[a], M, b, v[b], M, c, v[c], Q_RULE3, a, b, c, SEP]` -> `ans`.

## Metrics (2000 held-out triples; top-1 over the whole vocabulary, chance 0.1)
`rule_acc`: the answer (last generated token) is `(v[a] + v[b] + v[c]) mod 10`. `recall_acc` (steps conditions): the generated tokens that must be `v[a]`, `v[b]`, `v[c]` are all correct. `s1_acc` (partial only): the generated `s1` is `(v[a] + v[b]) mod 10`. `fact_acc`: the FACT question on the first 1000 entities (control).

## Outputs (aggregates only)
`summary[str(N)][condition]["step_<k>"][metric]` = mean over seeds, min, max, and per seed, `chance_level`. Plus `parameter_counts`, `predictions_check` (measured values and pass/fail, descriptive), `protocol` (including `heldout_triple_stats`), `environment` (including the code hash), `timing_seconds_mean` (update loops, evaluations excluded), `elapsed_seconds`, `runs` and `smoke_test_not_protocol` (true only in custom mode). No significance test.

## Predictions declared in advance (judged at N = 1024, step 2000, mean over the 5 seeds)
- P1 (one pass fails): `internal_64` `rule_acc` <= 0.15 (chance is 0.10).
- P2 (steps solve it): `internal_64_steps` `rule_acc` >= 0.90.
- P3 (partial step): `internal_64_steps_partial` `rule_acc` minus `internal_64_steps` `rule_acc` >= 0.0. The sign of the gap is reported whatever the outcome.
- P4 (external): `external_64` `rule_acc` >= 0.90.
- P5 (control): `fact_acc` >= 0.98 for all four conditions at N = 1024 (the N = 256 values are reported, not judged). If P5 fails, the other results are not readable.

Reading rule: exploratory and toy; retrieval is ideal in `external_64` (search cost not tested); no update of the confidence in H2; no significance test. N = 256 is secondary. No prior probability is attached to P1-P5 in this protocol.

## Limitations
One rule (sum mod 10 of three facts); random independent facts; one small model; 5 seeds; no significance test; the steps formats are supervised reasoning (intermediate tokens given in training), not shown for real LLMs; training stops at 2000 updates (a longer training of `internal_64` could change its result); held-out triples involve entities seen in training; `fact_acc` is measured on training facts; the steps loss weighs RULE3 rows 4:1 (5:1 for the partial format) against FACT rows per token; a new token (`Q_RULE3`) is added to the vocabulary of v1-v4, so the initial weights differ from v1-v4.

## Budget and running
4 conditions x 2 N x 5 seeds = 40 trainings of 2000 updates. From v1-v3 the update loop is about 8 s per training on the GPU whatever the sequence length (launch-bound): about 320 s, plus evaluations (3 checkpoints, generation of 1-5 tokens on 2000 triples): estimate 340-380 s, under the 450 s level at which a reduction would be proposed. GPU guard 600 s, abort at 585 s without a partial result. Fail-fast: after the first complete seed, the total is projected as elapsed x seeds / seeds_done; above 585 s the run aborts at once. Both guards are off in custom (smoke) mode. Call `run_h2_pilot_v5()` on a CUDA machine; smoke tests: `run_h2_pilot_v5(device='cpu', steps=..., seeds=..., ns=...)` (`steps` = number of updates; the checkpoints scale by steps / 2000).
