# H2 pilot v9 protocol (toy: how many RULE examples must be learned? - "learn the top": the abstract rule)

Written before execution. Code: `h2_pilot_v9.py`, entry point `run_h2_pilot_v9()`. Hypothesis H2: "learn the top, look up the bottom" - reserve the parameters for principles and reasoning, put the facts in an external memory. Pilots v1-v8 always trained the rule (RULE = (v[a] + v[b]) mod 10) on every non-test pair of entities, about 840 000 ordered pairs at N = 1024. This pilot asks how many rule examples are really needed: the rule only depends on the 100 couples of digits, so a few pairs may be enough.

## Context: what v1-v8 showed (known to the authors when this protocol was written)
- v1: composing two facts on held-out pairs: an internal one-pass model fails at N = 1024 (about chance); an external-memory model (facts in the context, ideal retrieval) succeeds.
- v2: an internal model with explicit steps (`[Q_RULE, a, b, SEP, v[a], v[b], ans]`) also reaches about 1.0 at N = 1024 with 2000 updates, trained on all non-test pairs.
- v3-v8: updating facts, imperfect retrieval, three facts, distractors, Zipf frequencies. None varied the number of rule examples.
Method note: the predictions below are written AFTER v1-v8 and BEFORE the v9 result. They are not blind to v1-v2 (which are the f = 1.0 case).

## Question
How does the held-out rule accuracy of (a) an external-memory model (oracle retrieval) and (b) an internal model with explicit steps depend on the fraction f of the allowed pairs that appear in the RULE training examples, the facts being always all learned?

## World (identical to v1-v8 for everything not listed here)
Vocabulary of 1039 tokens (entities 0..1023, digits 1024..1033, `Q_FACT`, `Q_RULE`, `SEP`, `M`, `PAD`); position table of 16 entries. N in {256, 1024} entities, all appear in training, one fact `v[e]` in 0..9 per entity (seed + 300000 + N). N = 1024 is judged, N = 256 is reported only.
Test pairs: a pair is a test pair iff `a != b` and `pair_hash(a, b) % 5 == 0` (symmetric, seed independent, the v1-v8 function, checked equal by the tests). The test pairs are the SAME for every f and are never trained.
Training pairs of fraction f: a FIXED subset of the ordered non-test pairs with `a != b` (seed independent, nested: the pairs of a smaller f are among those of a larger f). A pair belongs to it iff `subset_hash(a, b) % 1000000 < round(f * 1000000)`, `subset_hash` being a second symmetric mix of the pair, different from `pair_hash` (checked by the tests: symmetric, different from `pair_hash`, same share among test pairs, nested, realised share close to f, no test pair, f = 1.0 = every non-test pair). The realised numbers are reported per seed in `protocol.train_pair_stats` (ordered / unordered pairs, share of the non-test pairs, entities appearing in a pair, distinct digit couples covered out of 100). At N = 1024: about 16 400 / 41 200 / 167 600 / 838 800 ordered pairs for f = 0.02 / 0.05 / 0.2 / 1.0.
Batches: 32 FACT + 32 RULE per batch (batch 64). The FACT entities are drawn uniformly among ALL the entities whatever f (every fact is learned); they are the same for every f. The RULE pairs of a batch are drawn uniformly among the ORDERED training pairs of fraction f. For f = 1.0 this is the same distribution as in the earlier pilots (uniform over the allowed ordered pairs; the exact random numbers differ since the draw is now a direct index draw instead of rejection sampling). A guard (`assert_pairs_allowed`) raises if any RULE pair of the schedule is a test pair or outside the subset.
Seeds `SEEDS = (71243, 72251, 73257, 74263, 75269)`; generator offsets 300000 (world), 310000 (batches), 320000 (evaluation): all different from v1-v8 (checked by the logic test file shipped with the code, including the derived generator seeds).
Model: causal Transformer, 2 layers, width 64, 4 heads, FF 128, dropout 0, AdamW lr 0.001, wd 0.01, TF32 off. Same initial weights for the two conditions and every f of a given seed; same batches for the two conditions of a given seed, N and f. 2000 updates; checkpoints 500, 1000, 2000; primary = 2000 (fixed in advance).

## Conditions (2), loss on the output tokens only
- `external_64` (facts in the context, oracle retrieval, v1 format): RULE `[M, a, v[a], M, b, v[b], Q_RULE, a, b, SEP]` -> `ans`; FACT `[M, e, v[e], Q_FACT, e, SEP]` -> `v[e]`.
- `internal_64_steps` (weights only, v2 format): RULE `[Q_RULE, a, b, SEP]` -> `v[a], v[b], ans`; FACT `[Q_FACT, e, SEP]` -> `v[e]`; greedy autoregressive generation at evaluation (no teacher forcing), the answer is the 3rd generated token.
Fractions: f in {0.02, 0.05, 0.2, 1.0} at N = 1024; f in {0.05, 1.0} at N = 256 (see Budget). Held-out evaluation: `rule_acc` = top-1 accuracy over the whole vocabulary (chance 0.1) on 2000 held-out test pairs per (seed, N) from a fixed generator (seed + 320000 + N), the same set for every f and condition; `fact_acc` on all the entities (control); `recall_acc` (steps): `v[a]` and `v[b]` both generated correctly (diagnostic).

## Outputs (aggregates only)
`summary[str(N)][condition]["f_<f>"]["step_<k>"][metric]` = mean over seeds, min, max, per seed, `chance_level`. Plus `parameter_counts`, `predictions_check` (measured values and pass/fail, descriptive; every `rule_acc` / `fact_acc` cell under `reported_not_judged`), `protocol` (including `train_pair_stats`), `environment` (including the code hash), `timing_seconds_mean` (update loops, evaluations excluded), `elapsed_seconds`, `runs`, `smoke_test_not_protocol` (true only in custom mode). No significance test.

## Predictions declared in advance (judged at N = 1024, step 2000, mean over the 5 seeds; min / max also reported)
- P1: `external_64` at f = 1.0: `rule_acc` >= 0.98.
- P2: `external_64` at f = 0.05: `rule_acc` >= 0.90 (the rule on digits is learned from few pairs because it depends only on the 100 digit couples).
- P3: `external_64` at f = 0.02: `rule_acc` >= 0.80.
- P4: `internal_64_steps` at f = 0.05: `rule_acc` >= 0.90.
- P5 (monotonicity): for each condition, `rule_acc` at f = 0.02 <= `rule_acc` at f = 1.0 (the gap f = 1.0 minus f = 0.02 is >= 0 for both conditions). Every gap is reported with its sign; any opposite sign is reported under `opposite_sign_conditions`.
No prior probability is attached to P1-P5. No 4000-update primary for `internal_64_steps` (see Budget): the primary is 2000 for both conditions.
Reading rule: exploratory and toy; retrieval is ideal (oracle); no update of the confidence in H2; no significance test. N = 256 is secondary and never enters the judgement. `fact_acc` is a control: if the facts are not retained at some f, the corresponding `rule_acc` is not readable for the internal condition.

## Limitations
One rule (addition mod 10): it depends only on 100 digit couples, so the number of pairs needed is specific to this toy rule; random independent facts; one small model; 5 seeds; no significance test; one fixed training subset per f (not redrawn per seed, so subset-to-subset variation is not averaged); oracle retrieval; N = 256 reduced to two fractions; 2000 updates, so at low f the same pairs are seen many times (the batches draw 64 000 pairs from a subset of about 16 000 ordered pairs at f = 0.02, N = 1024); the steps loss weighs the RULE rows 3:1 against FACT rows per token; held-out pairs involve entities seen in training; no longer training for the internal condition (4000 updates not run).

## Budget and running
N = 1024: 4 fractions x 2 conditions x 5 seeds = 40 trainings; N = 256: 2 fractions x 2 conditions x 5 seeds = 20 trainings; 60 trainings of 2000 updates. From v1-v8 an update loop takes about 7.5 s per 2000 updates (launch-bound): about 450 s in all, plus evaluations (3 checkpoints; generation of 3 tokens for the steps condition) and data building: estimate 450-480 s, i.e. at the 450 s level, which is why the reductions below were taken before the run (with N = 256 at f in {0.05, 0.2, 1.0}: about 540 s; with the 4000-update internal primary: more than 600 s). Reductions decided in advance, before any result: N = 256 limited to f in {0.05, 1.0}; no 4000-update checkpoint for the internal condition. GPU guard 600 s, abort at 585 s without a partial result. Fail-fast: after the first complete seed, the total is projected as elapsed x seeds / seeds_done; above 585 s the run aborts at once. Both guards are off in custom (smoke) mode. Call `run_h2_pilot_v9()` on a CUDA machine; smoke tests: `run_h2_pilot_v9(device='cpu', steps=..., seeds=..., ns=..., fractions=...)` (`steps` = number of updates; the checkpoints scale by steps / 2000; `fractions` replaces the per-N fractions for every N).
