# H2 pilot v3 protocol (toy: updating facts, external memory vs weights)

Written before execution. Code: `h2_pilot_v3.py`, entry point `run_h2_pilot_v3()`. Hypothesis H2: "learn the top, look up the bottom" - reserve the parameters for principles and reasoning, put the facts in an external memory.

## Context: what v1 and v2 showed (known to the authors when this protocol was written)
- v1 (`pilot-1-facts-weights-vs-memory-protocol.md`, executed): toy world, one fact `v[e]` in 0..9 per entity, rule RULE = (v[a] + v[b]) mod 10 on test pairs never seen in training. At 1024 facts the internal model retains the facts but fails to COMPOSE them (about chance); the external-memory model (facts given in the context, ideal retrieval) succeeds.
- v2 (`pilot-2-steps-vs-longer-training-protocol.md`, executed): an internal model that first writes the recalled facts (steps `[Q_RULE, a, b, SEP, v[a], v[b], ans]`) also reaches 100 %.
- Consequence: in this toy, an external memory is NOT necessary to COMPOSE. What it could still add is UPDATING knowledge: (T1) facts modified after training, (T2) entities never seen in training, both without retraining. v3 also measures what the same update costs in the weights (fine-tuning time, forgetting).

Method note: the v1 and v2 results are known to the authors; the predictions below are written AFTER v1 and v2, they are not blind to them.

## Question
(a) Can an external-memory model use edited facts and unseen entities with no extra training while an internal model cannot? (b) How many updates and how much forgetting does it take to put the edited facts into the weights, with and without replay of old knowledge?

## World
Vocabulary as v1 / v2 (1039 tokens: entities 0..1023, digits 1024..1033, `Q_FACT`, `Q_RULE`, `SEP`, `M`, `PAD`). N total entities in {256, 1024}.
- Seen entities: the first 75 % (192 / 768). New entities: the remaining 25 % (64 / 256); they appear in NO training sequence and NO fine-tuning sequence (checked at run time by `assert_entities_in`, and by the tests).
- World A: `v[e]` in 0..9 for all entities (seed + 60000 + N). Training uses world A only.
- Edited set E: 25 % of the seen entities (48 / 192), drawn without replacement (seed + 70000 + N). World B: `v'[e]` = `(v[e] + delta) mod 10` with `delta` uniform in 1..9 on E (so `v'` != `v`, uniform among the 9 other digits); `v'[e] = v[e]` on the other seen entities; the new entities get an independent uniform value in world B.
- Held-out pairs: a pair is a test pair iff `pair_hash(a, b) % 5 == 0` (symmetric, seed independent; the v1 / v2 function, checked equal by the tests). Training and the replay rule pairs sample only non-test pairs.
- Seeds `SEEDS = (41093, 42101, 43103, 44111, 45121)`; generator offsets 60000 (world A), 70000 (E and B), 80000 (training batches), 90000 (fine-tuning batches), evaluation-pair seed 737373: all different from v1 / v2 (10000, 30000, 50000, 424242); seeds disjoint from v10-v15, v1 (31013, 32027, 33029, 34039, 35051) and v2 (36061, 37063, 38069, 39073, 40087) (checked by the test file, including the derived generator seeds).

## Trained conditions (2)
Causal Transformer, 2 layers, width 64, 4 heads, FF 128, dropout 0, same initial weights for both conditions of a given seed; 2000 updates, batch 64 = 32 FACT + 32 RULE, AdamW lr 0.001, wd 0.01, TF32 off, same hyper-parameters as v2. Training only on seen entities, world A.
- `external_64`: v1 / v2 format; the facts are given in the context: RULE `[M, a, v[a], M, b, v[b], Q_RULE, a, b, SEP]` -> answer, FACT `[M, e, v[e], Q_FACT, e, SEP]`. Loss on the answer.
- `internal_64_steps`: v2 format: FACT `[Q_FACT, e, SEP]` -> `v[e]`; RULE `[Q_RULE, a, b, SEP]` -> `v[a], v[b], ans`. Loss on the output tokens (1 for FACT, 3 for RULE). Evaluation by GREEDY AUTOREGRESSIVE generation (no teacher forcing); the answer is the 3rd generated token.

## Evaluation (2000 pairs each, with replacement if fewer test pairs exist; fixed generator per pool)
- `seen_heldout_acc`: test pairs of seen NON-edited entities, answer from world A (= B). Control.
- `edited_acc`: test pairs with BOTH entities in E, answer from world B. External: the context carries `v'` (no additional training). Internal: the model only has its weights, i.e. it recalls its world-A values.
- `new_acc`: test pairs with both entities new, answer from world B. External: context with `v'`. Internal: no knowledge.
- `unedited_acc`: the same set as `seen_heldout_acc` (same value, reported under both names); measured after training and at every fine-tuning checkpoint (forgetting).
- `edited_stale_acc` (diagnostic, not a prediction): the edited pairs scored against the OLD world-A answer.
With N = 256 the edited (about 420 test pairs) and new (about 820) pools are smaller than 2000, so the pairs are resampled with replacement (`heldout_pair_stats` lists it per seed).

## Fine-tuning (internal only)
From COPIES of the trained `internal_64_steps` model, with a fresh AdamW (lr 0.001, wd 0.01, optimizer state reset). Checkpoints at 0, 50, 100, 200 updates (0 = trained model); at each one all four accuracies are measured (the internal model should not know the new entities; measured anyway). The new entities are never in fine-tuning.
- `ft_edit_only`: batches of 64 FACT questions on entities of E, with world-B values.
- `ft_replay`: 32 FACT on E (world B) + 16 FACT + 16 RULE on seen NON-edited entities (world A; rule pairs not test pairs).
Fine-tuning draws are shared by the two variants (seed + 90000 + N). Loss: output tokens only.

## Outputs (aggregates only)
`summary[str(N)][name][phase]` with name in `external_64`, `internal_64_steps` (phase `trained`), `ft_edit_only`, `ft_replay` (phases `ft_0`, `ft_50`, `ft_100`, `ft_200`): each metric as seed mean and per seed, `chance_level`. Plus `parameter_counts`, `predictions_check` (measured values and pass/fail, descriptive), `protocol`, `environment`, `elapsed_seconds`, `timing_seconds_mean` (mean seconds of the update loops: training and the 200 fine-tuning updates, evaluations excluded), `runs` (one record per seed, N, name, phase), and `smoke_test_not_protocol` in custom mode only. No significance test.

## Predictions declared in advance (judged at N = 1024, mean over the 5 seeds; written after v1 and v2)
- P1 (update without cost): `external_64` `edited_acc` >= 0.95 with no extra training AND `internal_64_steps` `edited_acc` before fine-tuning (`trained`, = `ft_0`) <= 0.20. Prior 60 %. Reading: both clauses must hold.
- P2 (cost of updating in the weights): after 200 updates of `ft_edit_only`, internal `edited_acc` >= 0.90. Prior 50 %.
- P3 (forgetting), reported as two separate predictions:
  - P3a: `ft_edit_only` lowers `unedited_acc` by at least 0.10 at 200 updates relative to 0 updates. Prior 40 %.
  - P3b: `ft_replay` stays within less than 0.05 of decrease of `unedited_acc` at 200 updates relative to 0. Prior 55 %.
- P4 (new entities): `external_64` `new_acc` >= 0.90 AND `internal_64_steps` `new_acc` <= 0.20. Prior 45 % (the embeddings of the unseen entities are never trained, the external model may be disturbed by them).
- P5 (control): `seen_heldout_acc` >= 0.99 for both conditions after training. Prior 80 %. If P5 fails, the experiment is not readable.

Reading rule: exploratory and toy; retrieval is ideal (the search itself is not tested: the retrieval cost, the link "c" of H2, is not measured); no update of the confidence in H2 (64 %); no significance. N = 256 is secondary (reported, not used to judge).

## Limitations
One rule (addition mod 10); random independent facts; one small model; 5 seeds; no significance test; fine-tuning of a single type (same lr); edited values drawn uniformly; no specialised knowledge editor (ROME / MEMIT); ideal retrieval; unseen-entity embeddings are never trained by a gradient but shrink slightly through AdamW weight decay (factor about 0.98 over 2000 updates), so they are not exactly the initial random vectors; `seen_heldout_acc` and `unedited_acc` are the same measurement; the `steps` loss weighs RULE rows 3:1 against FACT rows per token (and in `ft_replay`, edited facts weigh 32 tokens against 16 + 48).

## Budget and running
2 N x 2 conditions x 5 seeds = 20 trainings of 2000 updates (about 7 s each on GPU, from v1) = about 140 s; fine-tuning 2 variants x 2 N x 5 seeds = 20 runs of 200 updates (about 1 s each with evaluation) = about 20 s; data building and evaluations about 10-20 s. Estimate: 170-200 s, under the 450 s level at which a reduction would be proposed. CPU measure (no accuracy displayed, N = 1024): about 4.0 ms per update `external_64`, 2.6 ms `internal_64_steps`, 2.6 ms per fine-tuning update. GPU guard 600 s, abort at 585 s without a partial result. Fail-fast: after the first complete seed, the total is projected as elapsed x seeds / seeds_done; above 585 s the run aborts at once. Both guards are off in custom (smoke) mode. Call `run_h2_pilot_v3()` on a CUDA machine; smoke tests: `run_h2_pilot_v3(device='cpu', steps=..., seeds=..., ns=...)` (`steps` = training updates; fine-tuning checkpoints scale by steps / 2000, at least 1).
