# H2 pilot v8 protocol (toy: Zipf entity frequencies - facts in the weights for the frequent entities, in a memory for the rare ones)

Written before execution. Code: `h2_pilot_v8.py`, entry point `run_h2_pilot_v8()`. Hypothesis H2: "learn the top, look up the bottom" - reserve the parameters for principles and reasoning, put the facts in an external memory. Earlier pilots (v1-v7) made every entity equally frequent: either every fact is in the weights or every fact is in the memory. This pilot is the first with frequencies: a fact that is seen often can be stored in the weights, a fact that is seen rarely should be looked up.

## Context: what v1-v5 showed (known to the authors when this protocol was written)
- v1: composing two facts, RULE = (v[a] + v[b]) mod 10 on held-out pairs: an internal model in one pass fails at N = 1024 (about chance) while it retains the facts; an external-memory model (facts in the context, ideal retrieval) succeeds.
- v2: an internal model with explicit steps (`[Q_RULE, a, b, SEP, v[a], v[b], ans]`, no memory) also reaches about 1.0 at N = 1024, with uniform entity frequencies and 2000 updates.
- v3-v5: updating facts, imperfect retrieval, three facts.
- v8 asks what happens when the entity frequencies are very unequal (Zipf), a few entities being seen very often and most of them rarely.

Method note: the predictions below are written AFTER v1-v5 and BEFORE the v6, v7 and v8 results (this protocol was written while v6 was running).

## Question
With Zipf frequencies, (a) does a model that keeps everything in its weights (steps format) learn the frequent ("head") entities but not the rare ("tail") ones; (b) does a model that gets every fact in a memory (oracle) work for both; (c) does a hybrid, which has memory items only for the tail entities and relies on its weights for the head, work for both, and how does its frequency-weighted accuracy compare with the all-memory model?

## World (identical to v1-v4 for everything not listed here)
Vocabulary of 1039 tokens (entities 0..1023, digits 1024..1033, `Q_FACT`, `Q_RULE`, `SEP`, `M`, `PAD`); position table of 16 entries. N in {256, 1024} entities, all appear in training, one fact `v[e]` in 0..9 per entity (seed + 260000 + N). N = 1024 is judged, N = 256 is reported only. Held-out pairs: test pair iff `pair_hash(a, b) % 5 == 0` (symmetric, seed independent; the v1-v7 function, checked equal by the tests); training samples only non-test pairs.
Frequencies: the entity of rank r (r = 0 the most frequent) has weight `1 / (r + 1)`; the probability of an entity is its weight divided by the sum over the N entities. The ranks are assigned to the entities by a random permutation, different for every seed and every N (generator seed + 280000 + N). Head = the 200 most frequent entities for N = 1024 (the 50 most frequent for N = 256); tail = all the others (824 and 206 entities). For N = 1024 the head carries about 78% of the probability mass (reported per seed in `protocol.frequency_stats`). In training the FACT entities, and the two entities a and b of each RULE pair (independently, then rejection of a = b and of test pairs), are drawn with these probabilities. Training mix: 32 FACT + 32 RULE per batch, batch 64.
Seeds `SEEDS = (66229, 67231, 68233, 69239, 70241)`; generator offsets 260000 (world), 270000 (batches), 280000 (frequency permutation), 290000 (evaluation sets): all different from v1-v7 (checked by the logic test file shipped with the code, including the derived generator seeds).
Model: causal Transformer, 2 layers, width 64, 4 heads, FF 128, dropout 0, AdamW lr 0.001, wd 0.01, TF32 off. Same initial weights for the three conditions of a given seed; same batches for the three conditions of a given seed and N. 3000 updates; checkpoints 1000, 2000, 3000; primary = 3000 (fixed in advance, no checkpoint selection).

## Conditions (3), loss on the output tokens only
FACT: `[Q_FACT, e, SEP]` -> `v[e]` (with memory slots as below for the external and hybrid conditions).
- `internal_64_steps` (everything in the weights, no memory): RULE `[Q_RULE, a, b, SEP]` -> `v[a], v[b], ans` (v2 format); greedy autoregressive generation at evaluation (no teacher forcing), the answer is the 3rd generated token.
- `external_64` (everything in a memory, oracle retrieval): RULE `[M, a, v[a], M, b, v[b], Q_RULE, a, b, SEP]` -> `ans`; FACT `[M, e, v[e], Q_FACT, e, SEP]` -> `v[e]` (v1 format).
- `hybrid_64`: the v1 format with two fixed memory slots of three tokens (the first for `a`, the second for `b`; one slot for `e` in a FACT row). The slot of an entity x is `[M, x, v[x]]` if x is a TAIL entity and `[PAD, PAD, PAD]` if x is a HEAD entity: the model sees a memory token only for tail entities, and nothing else tells it which entities are head (it can only know from its weights, and from the empty slot). RULE `[slot a, slot b, Q_RULE, a, b, SEP]` -> `ans` in ONE pass, no steps (the format of pilot 1 with items only for the tail); FACT `[slot e, Q_FACT, e, SEP]` -> `v[e]`. A pair of two head entities therefore has to be composed from two facts stored in the weights in one pass (which v1 found internal one-pass models unable to do); this is the format required for the pilot and it is kept as it is.

## Evaluation groups (per seed and N; 2000 items per group from a generator seeded by seed + 290000 + N)
- RULE, `rule_acc` (top-1 over the whole vocabulary, chance 0.1, truth the true `(v[a] + v[b]) mod 10`), held-out pairs only: **head** = test pairs with both entities in the head; **tail** = test pairs with both entities in the tail (drawn uniformly among the test pairs of the group, with replacement iff fewer than 2000 exist: this is the case for the head at N = 256); **weighted** = test pairs drawn among ALL test pairs with probability proportional to `p[a] * p[b]` (mixed head-tail pairs included): the plain mean over this sample is the frequency-weighted accuracy (the sample is drawn at random, so it carries its own sampling noise).
- FACT, `fact_acc`: head = all head entities, tail = all tail entities (at most 1000), weighted = 2000 entities drawn with probability p. For `hybrid_64` the memory slot is present for a tail entity only.
- `recall_acc` (steps): the two generated tokens before the answer are `v[a]` and `v[b]` (diagnostic).
Mixed head-tail pairs are only in the weighted group.

## Outputs (aggregates only)
`summary[str(N)][condition][group]["step_<k>"][metric]` = mean over seeds, min, max, and per seed, `chance_level`. Plus `parameter_counts`, `predictions_check` (measured values and pass/fail, descriptive; the `fact_acc` of every cell under `reported_not_judged`), `protocol` (including the head sizes and `frequency_stats`: head / tail sizes, head mass, number of test pairs of the head and of the tail, per seed), `environment` (including the code hash), `timing_seconds_mean` (update loops, evaluations excluded), `elapsed_seconds`, `runs` and `smoke_test_not_protocol` (true only in custom mode). No significance test.

## Predictions declared in advance (judged at N = 1024, step 3000, mean over the 5 seeds)
- P1: `internal_64_steps` head `rule_acc` >= 0.90.
- P2 (rare facts are not learned in the weights): `internal_64_steps` tail `rule_acc` <= 0.50.
- P3: `external_64` tail `rule_acc` >= 0.90.
- P4: `hybrid_64` tail `rule_acc` >= 0.90 AND `hybrid_64` head `rule_acc` >= 0.90 (both are needed).
- P5: `hybrid_64` weighted `rule_acc` >= `external_64` weighted `rule_acc`. The gap hybrid minus external, with its sign, is reported whatever the outcome.
`fact_acc` of every group and condition is reported as a control, not judged; if it shows that the head facts are not retained, the head results are not readable.

Reading rule: exploratory and toy; retrieval is ideal (oracle) and the head / tail label is given by the experimenter (the model does not decide what to look up); no update of the confidence in H2; no significance test. N = 256 is secondary. No prior probability is attached to P1-P5 in this protocol.

## Limitations
One rule (addition mod 10); random independent facts; Zipf with exponent 1 and one head / tail split (about 20% of the entities) only; one small model; 5 seeds; no significance test; oracle retrieval (a tail item in the memory is always the right one, no search cost); the head / tail label determines the slots (the model is not asked to decide what to look up); `hybrid_64` composes in one pass without steps, so its head-head pairs depend on one-pass composition from the weights; the steps conditions have a different loss weighting per token from the one-pass ones; the head group at N = 256 has few test pairs (sampled with replacement); 3000 updates; the weighted group is a random sample of 2000 pairs; held-out pairs involve entities seen in training; mixed pairs appear only in the weighted group.

## Budget and running
3 conditions x 2 N x 5 seeds = 30 trainings of 3000 updates. From v1-v7 the update loop is about 7.5 s per 2000 updates (launch-bound): about 11.4 s per training, about 340 s in all, plus evaluations (3 checkpoints x 3 groups, generation of 3 tokens for the steps condition): estimate 350-380 s, under the 450 s level at which a reduction would be proposed. GPU guard 600 s, abort at 585 s without a partial result. Fail-fast: after the first complete seed, the total is projected as elapsed x seeds / seeds_done; above 585 s the run aborts at once. Both guards are off in custom (smoke) mode. Call `run_h2_pilot_v8()` on a CUDA machine; smoke tests: `run_h2_pilot_v8(device='cpu', steps=..., seeds=..., ns=...)` (`steps` = number of updates; the checkpoints scale by steps / 3000; for N other than 256 and 1024 the head is N // 5 entities).
