# H2 pilot v4 protocol (toy: imperfect retrieval, wrong items and distractors)

Written before execution. Code: `h2_pilot_v4.py`, entry point `run_h2_pilot_v4()`. Hypothesis H2: "learn the top, look up the bottom" - reserve the parameters for principles and reasoning, put the facts in an external memory. This pilot is about link "c" of H2: the cost and the errors of the search.

## Context: what v1-v3 showed (known to the authors when this protocol was written)
- v1: with the facts given in the context (perfect retrieval), `external_64` composes held-out pairs (RULE = (v[a] + v[b]) mod 10) at about 1.0 at N = 1024; an internal model without steps fails (about chance).
- v2: an internal model with explicit steps (`internal_64_steps`) also reaches about 1.0 at N = 1024. In this toy, an external memory is therefore not necessary to compose.
- v3: what an external memory adds is updating (edited facts, unseen entities) without retraining.
- All three assumed perfect retrieval. v4 asks whether the external model's advantage survives when the retrieval is wrong or noisy.

Method note: the predictions below are written AFTER v1-v3 (not blind to them) and BEFORE the v4 run.

## Question
(A) How does the accuracy of a model trained with perfect retrieval fall when the memory items are wrong with probability 1 - r at test time, compared with a model that has no memory? (B) Can a model trained with distractor items among the relevant ones ignore them, and does that extend to more distractors than in training?

## World (identical to v1-v3 for everything not listed here)
Vocabulary of 1039 tokens (entities 0..1023, digits 1024..1033, `Q_FACT`, `Q_RULE`, `SEP`, `M`, `PAD`). N in {256, 1024} entities, all seen in training, one fact `v[e]` in 0..9 per entity (seed + 110000 + N). N = 1024 is judged, N = 256 is reported only. Held-out pairs: test pair iff `pair_hash(a, b) % 5 == 0` (symmetric, seed independent; the v1-v3 function, checked equal by the tests); training samples only non-test pairs. Seeds `SEEDS = (46127, 47131, 48137, 49139, 50147)`; generator offsets 110000 (world), 120000 (batches), 130000 (distractors of the training), 140000 (evaluation noise), 150000 (evaluation distractors), evaluation-pair seed 838383: all different from v1-v3 (checked by the test file, including the derived generator seeds).
Model: causal Transformer, 2 layers, width 64, 4 heads, FF 128, dropout 0, AdamW lr 0.001, wd 0.01, batch 64 = 32 FACT + 32 RULE, TF32 off. The position table has 40 entries (v1-v3: 16), the same for all three conditions, because `external_64_distr` evaluated at k = 8 needs 35. Same initial weights for the three conditions of a given seed. Same batches `(entity, a, b)` for the three conditions of a given seed and N.

## Trained conditions (3), 2000 updates each, loss on the answer token(s) only
- `external_64`: v1 format, perfect retrieval in training: RULE `[M, a, v[a], M, b, v[b], Q_RULE, a, b, SEP]` -> answer; FACT `[M, e, v[e], Q_FACT, e, SEP]` -> `v[e]`.
- `internal_64_steps`: v2 format, no memory: FACT `[Q_FACT, e, SEP]` -> `v[e]`; RULE `[Q_RULE, a, b, SEP]` -> `v[a], v[b], ans`; evaluation by greedy autoregressive generation (the answer is the 3rd generated token). Reference (it reached about 1.0 at N = 1024 in v2).
- `external_64_distr`: k = 4 distractor items `[M, x, v[x]]` (distinct entities x, different from a and b, drawn at random) mixed with the 2 relevant items at random positions (random permutation of the 6 items, independently per sequence), then `Q_RULE, a, b, SEP`. FACT rows: the relevant item and 4 distractors (different from e) at random positions, then `Q_FACT, e, SEP`. Sequences of 23 tokens (RULE) and 19 (FACT).

## Test settings (evaluation, 2000 held-out pairs, fixed generator per N, independent of the run seed)
- Part A, `external_64` only: each of the two memory items is, independently with probability 1 - r, replaced by the complete item `(M, x, v[x])` of another entity x drawn at random (x different from a and b; the item stays a valid (entity, value) pair of the true world; the model cannot know it is wrong). r in {1.0, 0.95, 0.9, 0.7, 0.5}, settings `r_1.0` ... `r_0.5`. The same draws are used for every r (the replaced items for a larger r are a subset of those for a smaller r). Truth = the true `(v[a] + v[b]) mod 10`. `r_1.0` is the clean case.
- `internal_64_steps`: setting `clean` (no memory; nothing to corrupt). It is the reference for part A.
- Part B, `external_64_distr` only: k in {0, 4, 8} distractors (distinct, different from a and b; the k = 4 case uses the first 4 of the 8 drawn, k = 0 none) mixed with the 2 relevant items at random positions (a fixed permutation per k; for k = 0 the two relevant items are therefore in random order). Settings `k_0`, `k_4`, `k_8`. k = 8 is an extrapolation in length (35 tokens against at most 23 in training; the position entries 22..34 are never used in training, they only decay through the weight decay; the model input has at most 22 positions in training and 34 at k = 8).
- `rule_acc`: top-1 of the answer over the whole vocabulary (chance 0.1). `fact_acc`: the FACT question for the first 1000 entities with the same setting applied to its single memory item (control, not a prediction).
- Checkpoints 500, 1000, 2000 for every condition; primary = 2000, fixed in advance; no checkpoint selection.

## Outputs (aggregates only)
`summary[str(N)][condition][test_setting]["step_<k>"][metric]` = mean over seeds, min, max, and per seed (metrics `rule_acc`, `fact_acc`), `chance_level`. Plus `parameter_counts`, `predictions_check` (measured values and pass/fail, descriptive), `protocol`, `environment` (including the code hash), `timing_seconds_mean` (update loops, evaluations excluded), `elapsed_seconds`, `runs` (one record per seed, N, condition, setting, step) and `smoke_test_not_protocol` (true only in custom mode). No significance test.

## Predictions declared in advance (judged at N = 1024, step 2000, mean over the 5 seeds; written after v1-v3)
- P1 (perfect retrieval): `external_64` `r_1.0` `rule_acc` >= 0.98.
- P2 (accuracy falls with r): for r in {0.9, 0.7, 0.5}, `external_64` `rule_acc` is within 0.08 of r^2 + (1 - r^2) * 0.1, i.e. expected 0.829 (r = 0.9), 0.541 (r = 0.7), 0.325 (r = 0.5); all three must hold. Reading: both items must be right (probability r^2), otherwise chance.
- P3 (the advantage is lost): for r in {0.9, 0.7, 0.5}, `external_64` is below `internal_64_steps` `clean` by at least 0.10 (all three gaps); gaps are reported for each r. (`internal_64_steps` is expected at about 1.0.)
- P4 (distractors in training): `external_64_distr` `k_4` `rule_acc` >= 0.95.
- P5 (length extrapolation is fragile): `external_64_distr` `k_8` is below `k_4` by at least 0.05. If it does not fall, it is reported as it is (the prediction is then false).

Reading rule: exploratory and toy; the noise is a random item substitution, not a model of a real search engine; the cost of the search is not measured; no update of the confidence in H2; no significance test. N = 256 is secondary (reported, not used to judge). No prior probability is attached to P1-P5 in this protocol.

## Limitations
One rule (addition mod 10); random independent facts; one small model; 5 seeds; no significance test; the wrong item is a valid item of another entity (a stricter model would also corrupt values or drop items); `external_64` is evaluated with the 2 items in the order (a, b) as in training; no condition trained with noisy items (part A is test-time noise only); the distractors are random entities (not hard negatives); `fact_acc` under noise is a control only; the position table is larger than in v1-v3 (40 against 16), so the initial weights differ from v1-v3.

## Budget and running
3 conditions x 2 N x 5 seeds = 30 trainings of 2000 updates. From v1-v3 the update loop is about 8 s per training on the GPU whatever the sequence length (launch-bound): about 240 s, plus evaluations (3 checkpoints x 9 settings, about 10-20 s per seed): estimate 300-350 s, at or under the 450 s level at which a reduction would be proposed. The number of updates is kept at 2000 (no amendment). GPU guard 600 s, abort at 585 s without a partial result. Fail-fast: after the first complete seed, the total is projected as elapsed x seeds / seeds_done; above 585 s the run aborts at once. Both guards are off in custom (smoke) mode. Call `run_h2_pilot_v4()` on a CUDA machine; smoke tests: `run_h2_pilot_v4(device='cpu', steps=..., seeds=..., ns=...)` (`steps` = number of updates; the checkpoints scale by steps / 2000).
