# H2 pilot v10 protocol (toy: longer or stronger weight updates of edited facts - follow-up of v3 P2)

Written before execution. Code: `h2_pilot_v10.py`, entry point `run_h2_pilot_v10()`. Hypothesis H2: "learn the top, look up the bottom" - reserve the parameters for principles and reasoning, put the facts in an external memory. v3 asked what updating knowledge costs in the weights: its prediction P2 (after 200 `ft_edit_only` updates, the internal model reaches `edited_acc` >= 0.90) FAILED: 200 updates absorbed only 0.24 of the edited facts (0.34 with replay) while `ft_edit_only` erased the old knowledge (`unedited_acc` fell from 1.0 to 0.13) and `ft_replay` kept it (0.998). v10 asks whether more updates, a larger learning rate, a smaller learning rate without replay, or more edited rows in the replay batch change that picture.

## Context: what v1-v9 showed (known to the authors when this protocol was written)
- v1-v2: in this toy an external memory is not necessary to COMPOSE (an internal model with explicit steps reaches about 1.0); what it adds is UPDATING knowledge.
- v3 (executed): the external model uses edited facts and unseen entities at once (`edited_acc` and `new_acc` 1.0 with no extra training); the internal model keeps the stale values (`edited_acc` 0.107, `edited_stale_acc` 1.0) and answers at chance on new entities (0.10). Fine-tuning the internal model for 200 updates: `ft_edit_only` 0.238 edited / 0.134 unedited, `ft_replay` 0.340 edited / 0.998 unedited, `edited_stale_acc` 0.100 / 0.140.
- v4-v9: imperfect retrieval, three facts, distractors, Zipf frequencies, fraction of rule pairs. None varied the fine-tuning recipe.
Method note: the predictions below are written AFTER v3 and BEFORE the v10 result; they are not blind to v3.

## Question
Can the weights absorb the edited facts (a) with 1000 instead of 200 updates, (b) with a larger learning rate (3e-3), (c) without replay but with a smaller learning rate (3e-4), (d) with a replay batch of 80% instead of 50% edited rows, and at what cost for the non-edited knowledge? The external memory needs no update: its `edited_acc` and `new_acc` are the reference.

## World (same as v3: seen / new / edited, world B v' != v; new seeds and offsets)
Vocabulary of 1039 tokens (entities 0..1023, digits 1024..1033, `Q_FACT`, `Q_RULE`, `SEP`, `M`, `PAD`). N total entities in {256, 1024}; N = 1024 is judged, N = 256 is reported (it is run, the time budget allows it) and never enters the judgement.
- Seen entities: the first 75% (192 / 768). New entities: the remaining 25% (64 / 256); they appear in NO training sequence and NO fine-tuning sequence (checked at run time by `assert_entities_in`, and by the tests).
- World A: `v[e]` in 0..9 for all entities (seed + 330000 + N). Training uses world A only.
- Edited set E: 25% of the seen entities (48 / 192), drawn without replacement (seed + 340000 + N). World B: `v'[e] = (v[e] + delta) mod 10` with `delta` uniform in 1..9 on E (so `v'` != `v`); `v' = v` on the other seen entities; the new entities get an independent uniform value in world B.
- Held-out pairs: test pair iff `pair_hash(a, b) % 5 == 0` (symmetric, seed independent, the v1-v9 function, checked equal by the tests). Training and the replay rule pairs sample only non-test pairs.
- Seeds `SEEDS = (76271, 77277, 78283, 79291, 80299)`; generator offsets 330000 (world A), 340000 (E and B), 350000 (training batches), 360000 (fine-tuning batches), evaluation-pair seed 858585: all different from v1-v9 (checked by the logic test file shipped with the code, including the derived generator seeds).

## Trained conditions (2), as in v3
Causal Transformer, 2 layers, width 64, 4 heads, FF 128, dropout 0, same initial weights for both conditions of a given seed; 2000 updates, batch 64 = 32 FACT + 32 RULE, AdamW lr 0.001, wd 0.01, TF32 off. Training only on seen entities, world A.
- `external_64` (v1 format, facts in the context, oracle retrieval): RULE `[M, a, v[a], M, b, v[b], Q_RULE, a, b, SEP]` -> answer; no training on world B (at evaluation the context carries the facts of the world asked). It is the v3 `external_64`, recomputed with the new seeds.
- `internal_64_steps` (v2 format): RULE `[Q_RULE, a, b, SEP]` -> `v[a], v[b], ans`; FACT `[Q_FACT, e, SEP]` -> `v[e]`; greedy autoregressive generation at evaluation (no teacher forcing), the answer is the 3rd generated token.

## Fine-tuning variants (internal only; 4)
Each starts from a COPY of the trained `internal_64_steps` model (so the four variants start from the same weights) with a FRESH AdamW (wd 0.01, no state carried over); checkpoints at 0 (= the trained model), 200, 500 and 1000 updates in one continuous run (the evaluation at 200 and 500 is on the way to 1000); primary = 1000. Every batch has 64 rows; the draws are shared by the variants (seed + 360000 + N; each variant takes a prefix of the same draws). Loss on the output tokens only. The new entities are never in fine-tuning.
- `ft_replay_lr1e3` (the v3 `ft_replay`, reference): lr 1e-3; 32 edited FACT rows (world B) + 16 FACT + 16 RULE rows of seen non-edited entities (world A, non-test pairs).
- `ft_replay_lr3e3`: the same batches, lr 3e-3.
- `ft_edit_only_lr3e4`: 64 edited FACT rows (world B), no replay, lr 3e-4 (the v3 `ft_edit_only` with a smaller learning rate, to see whether forgetting diminishes).
- `ft_replay_hi_edit`: lr 1e-3; 51 edited FACT rows (79.7% of 64; 80% is not a whole number of rows) + 7 FACT + 6 RULE replay rows (world A); the same edited draws as the other variants (the first 51 of 64).
Choice made before the run: with 6 RULE rows replay keeps a FACT : RULE ratio of about 1 : 1 as in v3 (7 : 6).

## Evaluation (2000 pairs each, with replacement if fewer test pairs exist; fixed generator per pool), as in v3
- `edited_acc`: test pairs with BOTH entities in E, answer from world B. External: the context carries `v'`. Internal: weights only.
- `unedited_acc`: test pairs of seen non-edited entities, answer from world A (= B there); measured at every checkpoint (forgetting). (`seen_heldout_acc` of v3 was the same measurement and is not repeated.)
- `new_acc`: test pairs of two new entities, answer from world B (same convention).
- `edited_stale_acc` (diagnostic): the edited pairs scored against the OLD world-A answer.
With N = 256 the edited and new pools have fewer than 2000 test pairs, so the pairs are resampled with replacement (`heldout_pair_stats`).

## Outputs (aggregates only)
`summary[str(N)][name][phase]` with name in `external_64`, `internal_64_steps` (phase `trained`) and the four variants (phases `ft_0`, `ft_200`, `ft_500`, `ft_1000`): each metric as seed mean, min, max and per seed, `chance_level`. Plus `parameter_counts`, `predictions_check` (measured values and pass/fail, descriptive; every cell under `reported_not_judged`), `protocol`, `environment` (code hash), `timing_seconds_mean` (update loops only), `elapsed_seconds`, `runs`, `smoke_test_not_protocol` (custom mode only). No significance test.

## Predictions declared in advance (judged at N = 1024, mean over the 5 seeds; min / max also reported; written after v3)
- P1: `ft_replay_lr1e3` at 1000 updates does not reach `edited_acc` >= 0.90 on average (holds iff the mean `edited_acc` < 0.90; the value is reported, and its range over seeds).
- P2: `ft_replay_lr3e3` at 1000 updates: `edited_acc` >= 0.70 AND `unedited_acc` >= 0.90 (both needed).
- P3: `ft_edit_only_lr3e4` at 1000 updates: `unedited_acc` >= 0.50 (forgetting diminishes with a small learning rate).
- P4: `ft_replay_hi_edit` at 1000 updates has a higher `edited_acc` than `ft_replay_lr1e3` at 1000 updates (strictly greater; the gap is reported with its sign).
- P5: `external_64` stays at 1.0 on edited and new pairs (mean `edited_acc` >= 0.99 AND mean `new_acc` >= 0.99; the threshold 0.99 is fixed here: "stays at 1.0" read as 1.0 up to 0.01).
No prior probability is attached to P1-P5.
Reading rule: exploratory and toy; the external model has ideal retrieval (search cost, the c link of H2, not measured); no update of the confidence in H2; no significance test; N = 256 is secondary and never enters the judgement.

## Limitations
One rule (addition mod 10); random independent facts; one small model; 5 seeds; no significance test; four fine-tuning recipes only, with one value each of learning rate and replay composition (no search); a continuous 1000-update run evaluated at 200 and 500 on the way (the checkpoints are not independent runs); edited values uniform; no specialised knowledge editor (ROME / MEMIT); the fine-tuning draws are shared by the variants (comparisons are paired); ideal retrieval for the external condition; unseen-entity embeddings are never trained by a gradient but shrink slightly through AdamW weight decay; the replay rows are drawn among 144 / 576 non-edited seen entities; `hi_edit` has 79.7% edited rows (not exactly 80%); the steps loss weighs the RULE rows 3:1 against FACT rows per token (so replay rule rows weigh 48 tokens against 16 FACT tokens + 32 edited tokens in `ft_replay`).

## Budget and running
2 N x 2 conditions x 5 seeds = 20 trainings of 2000 updates (about 7.5 s each on GPU, from v3: 7.9-8.1 s) = about 160 s; fine-tuning 4 variants x 2 N x 5 seeds = 40 runs of 1000 updates (v3: 0.8 s per 200 updates, so about 4 s per 1000, evaluations included) = about 170 s; data building and evaluation about 10-20 s. Estimate: 330-360 s, under the 450 s level at which a reduction would be proposed (N = 256 is therefore kept). GPU guard 600 s, abort at 585 s without a partial result. Fail-fast: after the first complete seed, the total is projected as elapsed x seeds / seeds_done; above 585 s the run aborts at once. Both guards are off in custom (smoke) mode. Call `run_h2_pilot_v10()` on a CUDA machine; smoke tests: `run_h2_pilot_v10(device='cpu', steps=..., seeds=..., ns=...)` (`steps` = training updates; fine-tuning checkpoints scale by steps / 2000, at least 1).
