llm_moe: the union operator (route/max-merge) vs fusion — and the regime flips at scale

Adds the union-preserving recombination operator that llm_merge lacked (E8's max,
not mean): keep each specialist LoRA intact and SELECT the right one per prompt
(MoE router: oracle, or training-free nearest-centroid over base embeddings) or
per module (max_merge = winner-take-all by delta norm). src/llm/moe.py, kind
llm_moe, reuses the cached specialists.

Result — a clean regime boundary for "merge, don't average":
- 0.5B: union wins. Routing 0.74 / worst-family 0.43 > soup 0.64 / 0.26, with no
  dilution (recovers each specialist's own-family peak). E8's max > mean in real
  weights, because at a weak base averaging dilutes.
- 7B (Imperial CX3, L40S, 9 min): the ordering INVERTS. Fusion wins — soup 0.87 >
  routing 0.84 > max_merge 0.78. Routing is capped at the best parent per family;
  fusion blends and, given a capable base, COMPOSES beyond any parent (soup lists
  0.62 > spec 0.57). Selection can't synthesise better than its best component;
  averaging-that-composes can.

So "merge, don't average" (E4/E8) is a weak-parent / small-model law, not
universal: union wins under dilution, fusion wins under composition. Refines E8
(its additive-landscape max>mean assumed no compositional headroom). The operator
to want is fusion-that-composes + offspring selection = the directed-sex ideal
(E10) — the natural next experiment.

Honest riders: the learned router is trivially perfect (lexically-distinct
families), and router-free max_merge is the weakest union (not input-adaptive).
+2 router unit tests (127 green). Results in results/llm_moe{,_hpc}/ (parquet
gitignored per the reproducibility contract).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

CLAUDE.md

@@ -91,6 +91,10 @@ E4's whole purpose is to isolate the effect of teacher **decorrelation ρ**, so
 
 **Finding (2026-07-05, `llm_merge_hpc` — the 7B firm-up on Imperial CX3; the marginal sign becomes decisive).** Re-ran `llm_merge` at a capable base — **Qwen2.5-7B-Instruct**, 200 test tasks/family, one **L40S (46 GB)** GPU, 8 min walltime — via the `/imperial-hpc` runbook (see `memory/hpc-setup.md`). **Both merges reach 0.87 overall, decisively above the best single specialist (0.77) and above every specialist on every family; worst-family 0.62 vs ≤0.57 for any specialist.** The two 0.5 B caveats are resolved: (1) the strict Fisher–Muller "exceeds every parent overall" claim is now clean (+10 points, not marginal); (2) the **dilution vanishes** — at 7 B the merge *beats* the lists-specialist on lists (0.62 > 0.57), where at 0.5 B averaging diluted it (0.43 → 0.26). **Interpretation: dilution is a small-model artefact; a capable base has enough headroom that weight-space averaging composes rather than dilutes** — the "merge, don't average" concern (E4) softens once parents are strong (soup ≈ ties at K=3). Results synced to `results/llm_merge_hpc/` (README legend + data-driven figure title). The natural refinement is *module-level* union-preserving recombination (MoE-expert / adapter-union merge, the real-weight image of E8's max-merge) rather than delta-averaging.
 
+**Finding (2026-07-05, `llm_moe` — the union operator in real weights; E8's `max` vs `mean`, 0.5B).** Added the *union-preserving* recombination operator that `llm_merge` lacked (`src/llm/moe.py`, `kind: llm_moe`): never average the parents — keep each specialist LoRA intact and **select** the right one per prompt (MoE **router**: `oracle`, or `learned` = training-free nearest-centroid over the *base* model's own prompt embeddings) or per module (`max_merge` = winner-take-all by delta-norm). Reuses the cached `llm_merge` specialists (no retraining). **Result (0.5B, seed 1):** **routing wins decisively over fusion — overall 0.74 / worst-family 0.43 vs soup 0.64/0.26** — and recovers *each* specialist's own-family peak exactly (no dilution: fusion diluted the lists-specialist 0.43→0.26, routing keeps 0.43). This is E8's `max`(union) > `mean`(average) in real LLM weights. **Two honest riders:** (1) the learned router is *trivially perfect* (1.00) because the three families are lexically distinct — routing's win here rests partly on the routing problem being easy (ambiguous/overlapping skills would make the router the bottleneck — the interesting next failure mode); (2) **router-free `max_merge` is a poor union (0.46)** — static per-module winner-take-all isn't input-adaptive, so it collapses toward the strongest-norm modules; the union benefit needs *routing*, not weight surgery. `configs/llm/moe.yaml`, `plot_llm_moe.py`, `results/llm_moe/README.md`, +2 router tests (127 green). The regime question — does routing still beat fusion once a capable base lets fusion *compose* rather than dilute (7B soup already beats its specialists)? — is the `llm_moe_hpc` 7B run below.
+
+**Finding (2026-07-05, `llm_moe_hpc` — the regime *flips* at 7B; "merge, don't average" is a weak-base law).** Re-ran `llm_moe` at **Qwen2.5-7B-Instruct** (L40S, 9 min, reusing the cached 7B specialists). **The union-vs-fusion ordering inverts:** at 0.5B union won (routing 0.74 > soup 0.64); at 7B **fusion wins — soup 0.87 > routing 0.84 > max_merge 0.78.** Mechanism, and it's the deep point: **routing *selects* one intact specialist so it is capped at the best parent per family** (lists 0.57 = spec_lists, strings 0.97 = spec_strings), whereas **fusion *blends* deltas and, at a capable base, composes *beyond* any parent** (soup lists 0.62 > spec 0.57, strings 1.00 > spec 0.97). Selection can't synthesise something better than its best component; averaging-that-composes can. So the E4/E8 "merge, don't average" law is **regime-dependent — a weak-parent / small-model law, not universal**: union wins exactly when averaging *dilutes* (0.5B), fusion wins once the base has headroom to *compose* (7B). This refines rather than contradicts E8 (whose additive-landscape `max>mean` assumed no compositional headroom). The operator to actually want is **fusion-that-composes + selection over recombinant offspring** = the "directed sex" ideal (E10), the natural next experiment. `results/llm_moe_hpc/` (README + regime-aware figure title). Riders unchanged: learned router trivially perfect (lexical families), `max_merge` the weakest union (not input-adaptive).
+
 ## Build order (blueprint §7) — respect the gate
 
 1. Scaffold: repo layout (§5), container, pytest skeleton, config system, seeding utils. `make test` green.