# H2 pilot v2 protocol (toy: explicit reasoning steps vs longer training, facts stored in the weights)

Written before execution. Code: `h2_pilot_v2.py`, entry point `run_h2_pilot_v2()`. Hypothesis H2: "learn the top, look up the bottom" - reserve the parameters for principles and reasoning, put the facts in an external memory.

## Context: what v1 showed (known to the authors when this protocol was written)
v1 (`pilot-1-facts-weights-vs-memory-protocol.md`, executed): at 2000 updates and N = 1024, `internal_64` had `rule_acc_heldout` 0.097, `internal_128` 0.123, `external_64` 1.00; but `fact_acc` was 1.0 everywhere. The internal models retain the facts but cannot COMPOSE them (addition mod 10) in a single pass as N grows.

Method note: the v1 figures are known to the authors; the predictions below are written AFTER v1, they are not blind to it.

## Question
Does the internal failure come from (a) too little training time, or (b) from composing in a single step, which explicit reasoning steps (the model first recalls its two facts, then adds) would fix? This tells whether "facts in the weights" block reasoning because of storage or because of format.

## World (identical to v1: same functions, same hash)
N entities, one fact per entity `v[e]` in {0..9}, drawn with the world seed (`seed + 10000 + N`), fixed during training and evaluation. Vocabulary of 1039 tokens as in v1: entities 0..1023, digits 1024..1033, `Q_FACT` 1034, `Q_RULE` 1035, `SEP` 1036, `M` 1037, `PAD` 1038 (right padding, never in the loss). N in {256, 1024}.
Held-out pairs: a pair is a test pair iff `pair_hash(a, b) % 5 == 0` (symmetric, seed independent); training samples only non-test pairs; evaluation uses 2000 test pairs from a fixed seed (`eval_pairs`). Same functions as v1 (checked equal by the logic test file shipped with the code).

## Conditions (4)
All: causal Transformer, 2 layers, width 64, 4 heads, FF 128, dropout 0 (the same parameter count; position embedding of 16, enough for the longest sequence of 11 tokens). Initial weights identical for the four conditions of a given seed (same seed, same architecture; also identical to v1's `internal_64`).
- `internal_64`: as v1. FACT `[Q_FACT, e, SEP, v[e]]`; RULE `[Q_RULE, a, b, SEP, ans]`. 2000 updates.
- `internal_64_long`: same model and format, 4000 updates, no other difference. Its first 2000 updates use the same batches as `internal_64`; updates 2001..4000 are new batches (independent generator, seed + 50000 + N, same train-pair rejection).
- `internal_64_steps`: FACT unchanged; RULE `[Q_RULE, a, b, SEP, v[a], v[b], ans]` (the model emits the two recalled facts then the answer). Loss on the 3 RULE output tokens (v[a], v[b], ans) and on the FACT answer; nothing else. 2000 updates. Evaluation: GREEDY AUTOREGRESSIVE generation of 3 tokens from `[Q_RULE, a, b, SEP]` (no teacher forcing): `rule_acc_heldout` = 3rd generated token correct; `recall_acc` = the first two generated tokens equal v[a], v[b]. `fact_acc` as v1.
- `external_64`: control, as v1 (oracle retrieval in the context, search cost ignored). 2000 updates.

## Training and evaluation
Mix 50% FACT / 50% RULE in every batch (32 + 32), batch 64. AdamW lr 0.001, weight decay 0.01, dropout 0, TF32 off. Checkpoints 500, 1000, 2000 for all conditions, plus 3000 and 4000 for `internal_64_long`; all reported, no checkpoint selection. Primary checkpoint = end of the training of each condition (2000; 4000 for `internal_64_long`), fixed in advance. Evaluation in `model.eval()`, train mode restored. Metric: top-1 accuracy over the whole vocabulary, chance 0.1. Loss: mean cross-entropy over the output tokens of the batch (for `steps`, a RULE row therefore weighs 3 tokens and a FACT row 1; for the other conditions 1 token per row).
Seeds `SEEDS = (36061, 37063, 38069, 39073, 40087)`, disjoint from the v10-v15 seeds and from v1's (31013, 32027, 33029, 34039, 35051); checked by the test file. 4 conditions x 2 N x 5 seeds = 40 models (`internal_64_long` counts double: 50 model-equivalents of 2000 updates).

## Outputs (aggregates only)
Per (N, condition, checkpoint): `rule_acc_heldout` and `fact_acc` (seed mean and per seed), `recall_acc` (for `internal_64_steps` only), `mean_rule_acc`, `chance_level`. Plus `parameter_counts`, `summary`, `predictions_check` (measured values and pass/fail of P1-P5 at the end of training, descriptive), `elapsed_seconds`, `protocol`, `environment`; `smoke_test_not_protocol` in custom mode only. No significance test.

## Predictions declared in advance (judged at the end of training of each condition, mean over the 5 seeds; written after v1)
- P1 (explicit steps): at N = 1024, `internal_64_steps` `rule_acc_heldout` >= 0.90. Prior 60%.
- P2 (time alone is not enough): at N = 1024, `internal_64_long` `rule_acc_heldout` < 0.50 (at 4000 updates). Prior 60%.
- P3 (same at N = 256): at N = 256, `internal_64_steps` exceeds `internal_64` by at least 0.10 in `rule_acc_heldout`. Prior 75%.
- P4 (recall): at N = 1024, `internal_64_steps` `recall_acc` >= 0.99. Prior 65%. If P4 fails, P1 cannot be read as a test of composition.
- P5 (control): `external_64` `rule_acc_heldout` >= 0.99 at both N. Prior 85%.

Reading rule: exploratory and toy; recovery (retrieval cost) is not tested; no update of the confidence in H2 (64%); no significance.

## Limitations
One rule (addition mod 10); random facts without structure; one small model; 5 seeds; no significance test; `internal_64_long` is limited to 4000 updates (grokking beyond that is not excluded); the `steps` format is supervised reasoning (the intermediate tokens are given in training), not shown for real LLMs; held-out pairs involve entities seen in training; `fact_acc` is measured on training facts; the `steps` loss weighs RULE rows 3:1 against FACT rows per token.

## Budget and running
50 model-equivalents x 2000 updates. Estimate from v1 (322 s for 45 models x 2000 updates, about 7 s per model): about 50 x 7.2 = 360 s plus evaluation, well under the 450 s level at which a reduction would be proposed. A CPU smoke of 500 updates (N = 256, accuracies not displayed) gave about 3.7 ms per update, consistent with the GPU figure of about 3.5 ms. GPU guard 600 s, abort at 585 s without a partial result. Fail-fast: after the first complete seed, the total is projected as elapsed x seeds / seeds_done; above 585 s the run aborts at once. Both guards are off in custom (smoke) mode. Call `run_h2_pilot_v2()` on a CUDA machine; smoke tests: `run_h2_pilot_v2(device='cpu', steps=..., seeds=..., ns=...)` (`steps` = base updates; the long condition trains twice as many).
