# H2 pilot v6 protocol (toy: distractor items with a curriculum)

Written before execution. Code: `h2_pilot_v6.py`, entry point `run_h2_pilot_v6()`. Hypothesis H2: "learn the top, look up the bottom" - reserve the parameters for principles and reasoning, put the facts in an external memory. This pilot is again about link "c" of H2: the errors and the noise of the search.

## Context: what v1-v5 showed (known to the authors when this protocol was written)
- v1: with the facts given in the context (perfect retrieval), `external_64` composes held-out pairs (RULE = (v[a] + v[b]) mod 10) at about 1.0 at N = 1024.
- v4: `external_64` trained with perfect retrieval and tested with wrong items falls roughly as r^2; `external_64_distr` (4 distractor items mixed with the 2 relevant ones, trained 2000 updates from the start) did NOT learn at all (rule_acc about 0.12 at k = 4, 2000 updates): the question left open is whether it is a matter of optimisation (the task is hard from the first update) rather than of capacity.
- v5: a three-fact rule; `external_64` (2000 updates) learned it on some seeds only (best seed about 0.93, mean about 0.27 at N = 1024): a large spread over seeds.
- v6 repeats the distractor test of v4 with a longer training and a curriculum: distractors are introduced gradually.

Method note: the predictions below are written AFTER v1-v5 (not blind to them) and BEFORE the v6 run.

## Question
Does a model trained with distractor items learn to ignore them (a) when trained 4000 updates with 4 distractors from the first update, (b) when the number of distractors is raised progressively (0, then 2, then 4), and does the curriculum model extend to more distractors (k = 8) than in training?

## World (identical to v1-v5 for everything not listed here)
Vocabulary of 1039 tokens (entities 0..1023, digits 1024..1033, `Q_FACT`, `Q_RULE`, `SEP`, `M`, `PAD`). N in {256, 1024} entities, all seen in training, one fact `v[e]` in 0..9 per entity (seed + 180000 + N). N = 1024 is judged, N = 256 is reported only. Held-out pairs: test pair iff `pair_hash(a, b) % 5 == 0` (symmetric, seed independent; the v1-v5 function, checked equal by the tests); training samples only non-test pairs. Seeds `SEEDS = (56173, 57179, 58181, 59191, 60193)`; generator offsets 180000 (world), 190000 (batches), 200000 (distractors of `external_64_distr`), 210000 (distractors of the curriculum), 220000 (evaluation distractors), evaluation-pair seed 656565: all different from v1-v5 (checked by the logic test file shipped with the code, including the derived generator seeds).
Model: causal Transformer, 2 layers, width 64, 4 heads, FF 128, dropout 0, AdamW lr 0.001, wd 0.01, batch 64 = 32 FACT + 32 RULE, TF32 off, position table of 40 entries (as v4). Same initial weights for the three conditions of a given seed. Same batches `(entity, a, b)` (the relevant items) for the three conditions of a given seed and N; only the distractors differ.

## Conditions (3), loss on the answer token only
- `external_64` (reference, no distractors, perfect retrieval): RULE `[M, a, v[a], M, b, v[b], Q_RULE, a, b, SEP]` -> answer; FACT `[M, e, v[e], Q_FACT, e, SEP]` -> `v[e]`. Trained **2000 updates** (checkpoints 1000, 2000): the budget does not allow 4000 for the reference too (see below); it reached about 1.0 at 2000 updates in v4.
- `external_64_distr` (A, fixed): the 2 relevant items + k = 4 distractor items `[M, x, v[x]]` (distinct entities x different from a and b, drawn at random) at random positions (random permutation of the items, independently per sequence), then `Q_RULE, a, b, SEP`; FACT rows: the relevant item + 4 distractors. Trained **4000 updates** (checkpoints 1000, 2000, 3000, 4000).
- `external_64_distr_curriculum` (B): same format, but the number of distractors k depends on the update: k = 0 for updates 1..1000 (the two relevant items, in random order), k = 2 for 1001..2000, k = 4 for 2001..3000, k = 4 for 3001..4000. 4000 updates (checkpoints 1000, 2000, 3000, 4000). Sequence lengths follow k (right padding inside a batch: the loss is on the answer position of each row, the position index of the answer depends on the phase). The model, optimiser and its state are NOT reset at a phase change.

## Test settings (evaluation, 2000 held-out pairs, fixed generator per N, independent of the run seed)
- `external_64`: setting `clean` (the items (a, b) in this order, no distractor, as in its training).
- Distractor conditions: k in {0, 2, 4, 8} distractors (distinct, different from a and b; nested: the k = 2 distractors are the first 2 of the k = 4 ones, which are the first 4 of the k = 8 ones) mixed with the relevant items at random positions (a fixed permutation per k; for k = 0 the two relevant items are in random order). Settings `k_0`, `k_2`, `k_4`, `k_8`. k = 8 is an extrapolation in length for both distractor conditions (35 tokens against at most 23 in training).
- `rule_acc`: top-1 of the answer over the whole vocabulary (chance 0.1), truth = the true `(v[a] + v[b]) mod 10`. `fact_acc`: the FACT question on the first 1000 entities with the same distractor setting applied to its single memory item.
- Checkpoints 1000, 2000, 3000, 4000 for the distractor conditions and 1000, 2000 for the reference; primary = the last one of each condition, fixed in advance; no checkpoint selection. All checkpoints reported.

## Outputs (aggregates only)
`summary[str(N)][condition][test_setting]["step_<k>"][metric]` = mean over seeds, min, max, and per seed (metrics `rule_acc`, `fact_acc`), `chance_level`. Plus `parameter_counts`, `predictions_check` (measured values and pass/fail, descriptive), `protocol` (including the curriculum plan and the number of updates per condition), `environment` (including the code hash), `timing_seconds_mean` (update loops, evaluations excluded), `elapsed_seconds`, `runs` (one record per seed, N, condition, setting, step) and `smoke_test_not_protocol` (true only in custom mode). No significance test.

## Predictions declared in advance (judged at N = 1024, primary checkpoint, mean over the 5 seeds; written after v1-v5)
- P1 (no distractors): `external_64` `clean` `rule_acc` >= 0.98 (checkpoint 2000). This is the "k = 0" reference of the external format; the k = 0 cells of the two distractor conditions are reported, not judged.
- P2 (curriculum works): `external_64_distr_curriculum` `k_4` `rule_acc` >= 0.90 at 4000 updates.
- P3 (fixed works): `external_64_distr` `k_4` `rule_acc` >= 0.90 at 4000 updates. The gap fixed minus curriculum at `k_4` (4000 updates) is reported with its sign whatever the outcome.
- P4 (extrapolation): `external_64_distr_curriculum` `k_8` `rule_acc` >= 0.70 at 4000 updates.
- P5 (control): `fact_acc` >= 0.95 in every cell of every condition and every setting (all four k for the distractor conditions, `clean` for the reference) at the last checkpoint, N = 1024. If P5 fails, the other results are not readable.

Reading rule: exploratory and toy; the distractors are random entities; the cost of the search is not measured; no update of the confidence in H2; no significance test. N = 256 is secondary (reported, not used to judge). No prior probability is attached to P1-P5 in this protocol.

## Limitations
One rule (addition mod 10); random independent facts; one small model; 5 seeds; no significance test; one curriculum only (k = 0, 2, 4); the reference is trained half as long as the other conditions (2000 against 4000 updates), so it is not a matched-length control; the distractors are random entities (not hard negatives); `fact_acc` under distractors is a control only; the position entries beyond 22 are not used in training (they only decay through the weight decay); the curriculum conditions see fewer long sequences than the fixed one.

## Budget and running
10 (seed, N) cells x (2000 + 4000 + 4000 updates). From v1-v5 the update loop is about 7.6 s per 2000 updates on the GPU whatever the sequence length (launch-bound): about 38 s per cell, 380 s in all, plus evaluations (small in v4 and v5: a few seconds): estimate 390-420 s, under the 450 s level at which a reduction would be proposed. This is why the reference is trained 2000 updates (4000 for it would project to about 460 s). GPU guard 600 s, abort at 585 s without a partial result. Fail-fast: after the first complete seed, the total is projected as elapsed x seeds / seeds_done; above 585 s the run aborts at once. Both guards are off in custom (smoke) mode. Call `run_h2_pilot_v6()` on a CUDA machine; smoke tests: `run_h2_pilot_v6(device='cpu', steps=..., seeds=..., ns=...)` (`steps` = number of updates of the distractor conditions; the reference gets steps // 2; the checkpoints and the curriculum bounds scale by steps / 4000).
