llm_moe: the union operator (route/max-merge) vs fusion — and the regime flips at scale

Adds the union-preserving recombination operator that llm_merge lacked (E8's max,
not mean): keep each specialist LoRA intact and SELECT the right one per prompt
(MoE router: oracle, or training-free nearest-centroid over base embeddings) or
per module (max_merge = winner-take-all by delta norm). src/llm/moe.py, kind
llm_moe, reuses the cached specialists.

Result — a clean regime boundary for "merge, don't average":
- 0.5B: union wins. Routing 0.74 / worst-family 0.43 > soup 0.64 / 0.26, with no
  dilution (recovers each specialist's own-family peak). E8's max > mean in real
  weights, because at a weak base averaging dilutes.
- 7B (Imperial CX3, L40S, 9 min): the ordering INVERTS. Fusion wins — soup 0.87 >
  routing 0.84 > max_merge 0.78. Routing is capped at the best parent per family;
  fusion blends and, given a capable base, COMPOSES beyond any parent (soup lists
  0.62 > spec 0.57). Selection can't synthesise better than its best component;
  averaging-that-composes can.

So "merge, don't average" (E4/E8) is a weak-parent / small-model law, not
universal: union wins under dilution, fusion wins under composition. Refines E8
(its additive-landscape max>mean assumed no compositional headroom). The operator
to want is fusion-that-composes + offspring selection = the directed-sex ideal
(E10) — the natural next experiment.

Honest riders: the learned router is trivially perfect (lexically-distinct
families), and router-free max_merge is the weakest union (not input-adaptive).
+2 router unit tests (127 green). Results in results/llm_moe{,_hpc}/ (parquet
gitignored per the reproducibility contract).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

CLAUDE.md

@@ -91,6 +91,10 @@ E4's whole purpose is to isolate the effect of teacher **decorrelation ρ**, so
 
 **Finding (2026-07-05, `llm_merge_hpc` — the 7B firm-up on Imperial CX3; the marginal sign becomes decisive).** Re-ran `llm_merge` at a capable base — **Qwen2.5-7B-Instruct**, 200 test tasks/family, one **L40S (46 GB)** GPU, 8 min walltime — via the `/imperial-hpc` runbook (see `memory/hpc-setup.md`). **Both merges reach 0.87 overall, decisively above the best single specialist (0.77) and above every specialist on every family; worst-family 0.62 vs ≤0.57 for any specialist.** The two 0.5 B caveats are resolved: (1) the strict Fisher–Muller "exceeds every parent overall" claim is now clean (+10 points, not marginal); (2) the **dilution vanishes** — at 7 B the merge *beats* the lists-specialist on lists (0.62 > 0.57), where at 0.5 B averaging diluted it (0.43 → 0.26). **Interpretation: dilution is a small-model artefact; a capable base has enough headroom that weight-space averaging composes rather than dilutes** — the "merge, don't average" concern (E4) softens once parents are strong (soup ≈ ties at K=3). Results synced to `results/llm_merge_hpc/` (README legend + data-driven figure title). The natural refinement is *module-level* union-preserving recombination (MoE-expert / adapter-union merge, the real-weight image of E8's max-merge) rather than delta-averaging.
 
+**Finding (2026-07-05, `llm_moe` — the union operator in real weights; E8's `max` vs `mean`, 0.5B).** Added the *union-preserving* recombination operator that `llm_merge` lacked (`src/llm/moe.py`, `kind: llm_moe`): never average the parents — keep each specialist LoRA intact and **select** the right one per prompt (MoE **router**: `oracle`, or `learned` = training-free nearest-centroid over the *base* model's own prompt embeddings) or per module (`max_merge` = winner-take-all by delta-norm). Reuses the cached `llm_merge` specialists (no retraining). **Result (0.5B, seed 1):** **routing wins decisively over fusion — overall 0.74 / worst-family 0.43 vs soup 0.64/0.26** — and recovers *each* specialist's own-family peak exactly (no dilution: fusion diluted the lists-specialist 0.43→0.26, routing keeps 0.43). This is E8's `max`(union) > `mean`(average) in real LLM weights. **Two honest riders:** (1) the learned router is *trivially perfect* (1.00) because the three families are lexically distinct — routing's win here rests partly on the routing problem being easy (ambiguous/overlapping skills would make the router the bottleneck — the interesting next failure mode); (2) **router-free `max_merge` is a poor union (0.46)** — static per-module winner-take-all isn't input-adaptive, so it collapses toward the strongest-norm modules; the union benefit needs *routing*, not weight surgery. `configs/llm/moe.yaml`, `plot_llm_moe.py`, `results/llm_moe/README.md`, +2 router tests (127 green). The regime question — does routing still beat fusion once a capable base lets fusion *compose* rather than dilute (7B soup already beats its specialists)? — is the `llm_moe_hpc` 7B run below.
+
+**Finding (2026-07-05, `llm_moe_hpc` — the regime *flips* at 7B; "merge, don't average" is a weak-base law).** Re-ran `llm_moe` at **Qwen2.5-7B-Instruct** (L40S, 9 min, reusing the cached 7B specialists). **The union-vs-fusion ordering inverts:** at 0.5B union won (routing 0.74 > soup 0.64); at 7B **fusion wins — soup 0.87 > routing 0.84 > max_merge 0.78.** Mechanism, and it's the deep point: **routing *selects* one intact specialist so it is capped at the best parent per family** (lists 0.57 = spec_lists, strings 0.97 = spec_strings), whereas **fusion *blends* deltas and, at a capable base, composes *beyond* any parent** (soup lists 0.62 > spec 0.57, strings 1.00 > spec 0.97). Selection can't synthesise something better than its best component; averaging-that-composes can. So the E4/E8 "merge, don't average" law is **regime-dependent — a weak-parent / small-model law, not universal**: union wins exactly when averaging *dilutes* (0.5B), fusion wins once the base has headroom to *compose* (7B). This refines rather than contradicts E8 (whose additive-landscape `max>mean` assumed no compositional headroom). The operator to actually want is **fusion-that-composes + selection over recombinant offspring** = the "directed sex" ideal (E10), the natural next experiment. `results/llm_moe_hpc/` (README + regime-aware figure title). Riders unchanged: learned router trivially perfect (lexical families), `max_merge` the weakest union (not input-adaptive).
+
 ## Build order (blueprint §7) — respect the gate
 
 1. Scaffold: repo layout (§5), container, pytest skeleton, config system, seeding utils. `make test` green.

Makefile

@@ -28,8 +28,9 @@ mnist:             ## run the real-MNIST confirmation tier (needs env-mnist; dow
 env-llm:           ## add the LLM stack for the Layer-2 prototype (GPU; transformers/peft)
 	uv sync --extra dev --extra neural --extra llm
 
-llm:               ## run the LLM merge prototype (needs env-llm; downloads a small base model)
+llm:               ## run the LLM prototypes: merge (fusion) + moe (union) recombination
 	uv run python -m llm.experiment configs/llm/merge.yaml
+	uv run python -m llm.experiment configs/llm/moe.yaml
 
 layer2: neural     ## alias: Layer 1.5 is the current Layer-2 deliverable (LLM rung deferred)
 

configs/llm/moe.yaml

@@ -0,0 +1,25 @@
+experiment: llm_moe
+kind: llm_moe
+seed: 1
+n_replicates: 1
+
+# Layer 2 / LLM — module-level, UNION-PRESERVING recombination (the real-weight image of E8's *max*).
+# Reuses the specialist adapters trained by configs/llm/merge.yaml (models/llm/spec_*) and contrasts
+# two families of recombination operator on the same held-out mixed test set:
+#   FUSION (blend the deltas):   soup = mean(Δ_k);  ties = sign-reconciled union.
+#   UNION  (never average):      moe_oracle / moe_learned = keep every specialist intact and ROUTE each
+#                                prompt to one (MoE-over-experts); max_merge = per-module winner-take-all.
+# Prediction (E8, "merge don't average"): union beats fusion exactly where fusion DILUTES — pronounced
+# at a weak base (0.5B), narrowing once a capable base lets fusion compose (7B). Falsifier: fusion
+# matches or beats the routing ceiling (moe_oracle) at 0.5B, i.e. averaging never dilutes.
+
+base_model: Qwen/Qwen2.5-0.5B-Instruct   # reuses the same cached specialists as llm_merge
+families: [lists, strings, arith]
+n_train: 700          # only used if the cached specialists are absent
+n_test: 100
+n_route: 32           # labelled prompts per family for the learned router's centroids
+epochs: 3
+lora: {r: 16, alpha: 32}
+operators: [soup, ties, moe_oracle, moe_learned, max_merge]
+
+output: {dir: results/llm_moe}

configs/llm/moe_hpc.yaml

@@ -0,0 +1,21 @@
+experiment: llm_moe_hpc
+kind: llm_moe
+seed: 1
+n_replicates: 1
+
+# Scaled version of configs/llm/moe.yaml for an L40S (48 GB). At 7B the llm_merge fusion baseline
+# (soup) already *composed* rather than diluted (merge 0.87 > best specialist 0.77, no dilution), so
+# the sharp question here is whether the UNION operators (routing / max-merge) still add anything once
+# the base is capable — i.e. does "merge, don't average" still bite at scale, or does a strong base
+# make fusion and union converge? Either way is a reportable regime result.
+
+base_model: Qwen/Qwen2.5-7B-Instruct
+families: [lists, strings, arith]
+n_train: 800          # only used if the cached specialists are absent (fresh on the HPC node)
+n_test: 200
+n_route: 48
+epochs: 3
+lora: {r: 16, alpha: 32}
+operators: [soup, ties, moe_oracle, moe_learned, max_merge]
+
+output: {dir: results/llm_moe_hpc}

figures/plot_llm_moe.py

@@ -0,0 +1,97 @@
+"""llm_moe figure — union-preserving recombination (route / max-merge) vs fusion (soup / ties).
+
+The real-weight image of E8's *max*: keep every specialist intact and *select* (route per prompt, or
+per module) instead of averaging the deltas. Two panels: (A) per-family accuracy for the base, each
+specialist, the fusion merges, and the union operators — the union operators should match the best
+specialist on every family (they *are* that specialist there), while fusion may dilute or compose;
+(B) overall vs worst-family, fusion vs union, with the routing ceiling (moe_oracle) marked. The
+suptitle reports whether union beats fusion (dilution regime) or they converge (composition regime).
+Reads only the committed bundle.
+
+Usage:  python figures/plot_llm_moe.py [results/llm_moe]
+"""
+
+from __future__ import annotations
+
+import sys
+from pathlib import Path
+
+import matplotlib.pyplot as plt
+import numpy as np
+
+sys.path.insert(0, str(Path(__file__).parent))
+from _figlib import load_bundle, savefig  # noqa: E402
+
+_FAMS = ["lists", "strings", "arith"]
+_FUSION = {"merge_soup": "fuse:soup", "merge_ties": "fuse:ties"}
+_UNION = {"moe_oracle": "route:oracle", "moe_learned": "route:learned", "max_merge": "max-merge"}
+
+
+def _acc(df, model, metric):
+    r = df[(df["model"] == model) & (df["metric"] == metric)]["accuracy"]
+    return float(r.iloc[0]) if len(r) else float("nan")
+
+
+def main(results_dir: str = "results/llm_moe") -> None:
+    df, cfg = load_bundle(results_dir)
+    present = set(df["model"].unique())
+    specialists = sorted(m for m in present if m.startswith("spec_"))
+    fusion = [m for m in _FUSION if m in present]
+    union = [m for m in _UNION if m in present]
+    models = ["base"] + specialists + fusion + union
+    labels = {"base": "base", **{s: s.replace("spec_", "spec:") for s in specialists},
+              **_FUSION, **_UNION}
+    colors = {"base": "#7f7f7f", **{s: "#1f77b4" for s in specialists},
+              **{m: "#ff7f0e" for m in fusion}, **{m: "#2ca02c" for m in union}}
+
+    fig, axes = plt.subplots(1, 2, figsize=(14, 5))
+
+    # Panel A: per-family accuracy, grouped by model.
+    ax = axes[0]
+    x = np.arange(len(_FAMS))
+    w = 0.8 / len(models)
+    for i, mdl in enumerate(models):
+        vals = [_acc(df, mdl, f) for f in _FAMS]
+        ax.bar(x + (i - (len(models) - 1) / 2) * w, vals, w, label=labels[mdl], color=colors[mdl],
+               alpha=0.9 if (mdl in union or mdl in fusion) else 0.65)
+    ax.set_xticks(x); ax.set_xticklabels(_FAMS)
+    ax.set(ylabel="accuracy", title="Per-family: fusion (orange) blends the deltas; union (green)\n"
+           "keeps each specialist intact and selects — no dilution")
+    ax.legend(frameon=False, fontsize=8, ncol=2)
+
+    # Panel B: overall vs worst-family, per model.
+    ax = axes[1]
+    x2 = np.arange(len(models))
+    for off, metric, hatch, lab in [(-0.2, "overall", "", "overall"),
+                                    (0.2, "worst_family", "//", "worst family")]:
+        ax.bar(x2 + off, [_acc(df, m, metric) for m in models], 0.38,
+               color=[colors[m] for m in models], hatch=hatch, alpha=0.85, label=lab,
+               edgecolor="white")
+    ax.set_xticks(x2); ax.set_xticklabels([labels[m] for m in models], rotation=25, ha="right",
+                                          fontsize=8)
+    ax.set(ylabel="accuracy", title="Overall (solid) vs worst-family (hatched):\nfusion vs union "
+           "recombination")
+    # Mark the routing ceiling (oracle) if present.
+    if "moe_oracle" in present:
+        ceil = _acc(df, "moe_oracle", "overall")
+        ax.axhline(ceil, ls=":", c="#2ca02c", lw=1, alpha=0.7)
+    ax.legend(frameon=False, fontsize=9)
+
+    best_fuse = max([_acc(df, m, "overall") for m in fusion], default=float("nan"))
+    best_union = max([_acc(df, m, "overall") for m in union], default=float("nan"))
+    router = _acc(df, "moe_learned", "router_acc") if "moe_learned" in present else float("nan")
+    if best_union > best_fuse + 0.01:
+        verdict = f"union {best_union:.2f} > fusion {best_fuse:.2f} overall (fusion dilutes)"
+    elif best_fuse > best_union + 0.01:
+        verdict = f"fusion {best_fuse:.2f} > union {best_union:.2f} overall (strong base composes)"
+    else:
+        verdict = f"union ≈ fusion ({best_union:.2f} vs {best_fuse:.2f}) overall"
+    rtxt = f"; learned router {router:.2f}" if router == router else ""
+    fig.suptitle(f"llm_moe — module-level union vs fusion recombination: {verdict}{rtxt} "
+                 f"({cfg['base_model'].split('/')[-1]})", y=1.0, fontsize=12)
+    fig.tight_layout()
+    savefig(fig, results_dir, "llm_moe")
+
+
+if __name__ == "__main__":
+    main(*sys.argv[1:])

hpc/llm_moe.pbs

@@ -0,0 +1,23 @@
+#!/bin/bash
+# Module-level union-preserving recombination (llm_moe) on an L40S (46 GB) — route/max-merge vs fusion.
+# Reuses the specialist adapters trained by the llm_merge_hpc run if models/llm/spec_* is present on
+# the node; otherwise trains them fresh. Same env as hpc/llm_merge.pbs (see hpc/README.md):
+#   `uv sync --extra dev --extra neural --extra llm` on the login node + pre-download the 7B base.
+#   submit:  qsub hpc/llm_moe.pbs      status: qstat -u $USER
+#PBS -l select=1:ncpus=8:mem=64gb:ngpus=1:gpu_type=L40S
+#PBS -l walltime=02:00:00
+#PBS -N lam_llm_moe
+
+cd "$PBS_O_WORKDIR"
+export HF_HOME="$EPHEMERAL/hf_cache"
+export TOKENIZERS_PARALLELISM=false
+export UV_CACHE_DIR="$EPHEMERAL/uvcache"
+
+source .venv/bin/activate
+nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv,noheader
+
+python -m llm.experiment configs/llm/moe_hpc.yaml
+
+# results/llm_moe_hpc/ written in-place (parquet gitignored). Sync back to plot:
+#   rsync -avz hpc:'…/LamarckianAI/results/llm_moe_hpc/' results/llm_moe_hpc/
+echo "done: $(date)"

results/llm_moe/README.md

@@ -0,0 +1,58 @@
+# llm_moe — union-preserving recombination (route / max-merge) vs fusion (the real-weight E8 *max*)
+
+**Claim tested.** E8 (analytic) found that the recombination *operator* matters: a **union** over
+parents (`max`) assembles a child fitter than any parent, while an **average** (`mean`, the model
+soup) conserves collapse. `llm_merge` showed fusion (soup/ties, which *average* the LoRA deltas). This
+experiment adds the **union** operator to real LLM weights — never average the parents, keep each
+specialist intact and **select** the right one per input (a Mixture-of-Experts *router*) or per module
+(winner-take-all) — and asks whether union beats fusion, and *where*.
+
+**Setup.** Base **Qwen2.5-0.5B-Instruct**, reusing the three cached LoRA specialists from `llm_merge`
+(disjoint families `lists`/`strings`/`arith`, exact-match verifier), 100 test tasks/family, seed 1.
+Five recombination operators on the same test set:
+- **Fusion** (blend the deltas): `soup` = mean(Δₖ); `ties` = sign-reconciled union.
+- **Union** (never average): `route:oracle`/`route:learned` keep all adapters live and route each
+  prompt to one specialist (MoE); `max-merge` builds one adapter taking, per module, the specialist
+  with the largest-norm delta. The learned router is training-free — nearest-centroid over the *base*
+  model's own prompt embeddings; its routing accuracy is reported.
+
+### Results (accuracy)
+| operator | lists | strings | arith | overall | worst-family | router |
+|---|---|---|---|---|---|---|
+| base | 0.15 | 0.15 | 0.53 | 0.28 | 0.15 | — |
+| best specialist (strings) | 0.08 | 1.00 | 0.80 | 0.63 | 0.08 | — |
+| fuse: soup | 0.26 | 0.74 | 0.91 | 0.64 | 0.26 | — |
+| fuse: ties | 0.23 | 0.71 | 0.90 | 0.61 | 0.23 | — |
+| **route: oracle** | 0.43 | 1.00 | 0.78 | **0.74** | **0.43** | 1.00 |
+| **route: learned** | 0.43 | 1.00 | 0.78 | **0.74** | **0.43** | **1.00** |
+| max-merge | 0.18 | 0.34 | 0.87 | 0.46 | 0.18 | — |
+
+### What holds
+- **Union (routing) beats fusion at a weak base — decisively.** Routing reaches **0.74 overall /
+  0.43 worst-family**, above both fusion merges (soup 0.64/0.26) and every specialist. It recovers
+  *each* specialist's own-family peak exactly (lists 0.43, strings 1.00, arith 0.78) because it *is*
+  that specialist there — **no dilution**. This is E8's `max` (union) beating `mean` (average) in real
+  LLM weights: exactly where fusion diluted the lists-specialist (0.43→0.26), routing keeps 0.43.
+- **The learned router is perfect here (1.00) — stated as a caveat, not a triumph.** The three families
+  are lexically distinct, so nearest-centroid routing over base embeddings is trivially easy;
+  `route:learned` equals `route:oracle`. Routing's advantage on *these* tasks therefore rests partly on
+  the routing problem being easy — the honest scope. On overlapping/ambiguous skills the router would
+  be the bottleneck, and that is the interesting failure mode to probe next.
+- **Static per-module `max-merge` is a poor union (0.46) — an informative negative.** Picking, per
+  module, the largest-norm specialist delta is *not* input-adaptive: it collapses toward whichever
+  specialist dominates the weight norms (arith 0.87, but lists 0.18, strings 0.34). The union benefit
+  needs **routing** (input-adaptive selection), not weight-space surgery — "keep the parents whole"
+  only pays off if you also *choose* the right parent per input.
+
+### Takeaway
+Adds the union half of E8's operator dichotomy to real LLM weights and confirms its sign at a weak
+base: **route-don't-average > average**, with no dilution, mirroring the analytic `max > mean`. Two
+honest riders — the learned router is trivially good because the families are lexically separable, and
+the router-free `max-merge` union fails because it isn't input-adaptive. The regime question the 0.5B
+result raises — *does routing still help once a capable base lets fusion **compose** rather than
+dilute?* (`llm_merge_hpc` showed 7B soup already beats its specialists with no dilution) — is answered
+by **`results/llm_moe_hpc/`: the ordering flips.** At 7B fusion wins (soup 0.87 > routing 0.84),
+because routing is capped at the best parent while fusion composes beyond it. So "merge, don't average"
+is a **weak-base law**: union wins here (0.5B, dilution regime), fusion wins there (7B, composition
+regime). **Falsifier (not triggered at 0.5B):** fusion matching the routing ceiling, i.e. averaging
+never diluting.

results/llm_moe_hpc/README.md

@@ -0,0 +1,52 @@
+# llm_moe_hpc — union vs fusion recombination at scale (7B, Imperial CX3): the regime *flips*
+
+**Claim tested.** The 0.5B `llm_moe` found that a **union** operator (route/select, never average) beats
+**fusion** (soup/ties, average the deltas), because at a weak base averaging *dilutes*. But
+`llm_merge_hpc` showed that at 7B fusion stops diluting and starts *composing* (soup 0.87 beat every
+specialist). So the sharp question: **does routing still beat fusion once the base is capable — or does
+a strong base invert the ordering?** Run on one **L40S (46 GB)** GPU of Imperial's CX3 HPC, 9 min
+walltime, reusing the cached 7B specialists.
+
+**Setup.** Base **Qwen2.5-7B-Instruct**, the three cached LoRA specialists from `llm_merge_hpc`
+(`lists`/`strings`/`arith`, exact-match verifier), 200 test tasks/family, seed 1. Same five operators
+as the 0.5B run: fusion (`soup`, `ties`) vs union (`route:oracle`, `route:learned`, `max-merge`).
+
+### Results (accuracy)
+| operator | lists | strings | arith | overall | worst-family | router |
+|---|---|---|---|---|---|---|
+| base | 0.46 | 0.69 | 1.00 | 0.71 | 0.46 | — |
+| best specialist (lists) | 0.57 | 0.74 | 1.00 | 0.77 | 0.57 | — |
+| **fuse: soup** | **0.62** | **1.00** | 1.00 | **0.87** | **0.62** | — |
+| fuse: ties | 0.62 | 1.00 | 0.99 | 0.87 | 0.62 | — |
+| route: oracle | 0.57 | 0.97 | 0.96 | 0.84 | 0.57 | 1.00 |
+| route: learned | 0.57 | 0.97 | 0.96 | 0.84 | 0.57 | 1.00 |
+| max-merge | 0.48 | 0.90 | 0.96 | 0.78 | 0.48 | — |
+
+### The finding: "merge, don't average" is regime-dependent, and inverts at scale
+- **The ordering flips.** At 0.5B, union > fusion (routing 0.74 > soup 0.64). At 7B, **fusion > union**
+  (soup **0.87** > routing 0.84 > max-merge 0.78). The exact opposite winner.
+- **Why: routing is capped at the best parent; fusion can *exceed* it.** Routing *selects* one intact
+  specialist, so per family it can only reach that specialist's own score (lists 0.57 = spec_lists,
+  strings 0.97 = spec_strings). Fusion *blends* the deltas — and at a capable base the blend
+  **composes beyond any parent**: soup scores lists 0.62 (> spec_lists 0.57) and strings 1.00
+  (> spec_strings 0.97). Selection cannot synthesise something better than its best component;
+  averaging, when it composes rather than dilutes, can. So fusion's worst-family (0.62) also beats
+  routing's (0.57).
+- **The regime boundary is dilution.** Union wins exactly when averaging *dilutes* (weak base, 0.5B);
+  fusion wins once the base has enough headroom that averaging *composes* (7B). "Merge, don't average"
+  (E4/E8) is therefore a **small-model / weak-parent** law, not a universal one — a genuine refinement
+  of the analytic claim, not a contradiction of it (E8's additive-landscape `max > mean` assumed no
+  such compositional headroom).
+- **Riders unchanged.** The learned router is still perfect (1.00, lexically-distinct families), and
+  `max-merge` remains the weakest union (0.78) — static per-module winner-take-all is not
+  input-adaptive.
+
+### Takeaway
+A clean, honest regime result: **route-don't-average wins at a weak base; average-that-composes wins at
+a strong one.** Pure selection (routing) never dilutes but is bounded by the best parent; fusion risks
+dilution but, given a capable base, transcends the parents — which is what the Fisher–Muller "exceed
+every parent" claim actually needs at scale. The operator to want is therefore *fusion that composes
+plus selection over candidates* — the "directed sex" ideal (offspring selection over recombinants),
+the natural next experiment. **Falsifier for this run (not triggered):** routing beating fusion at 7B,
+i.e. dilution persisting at scale — instead it inverted. Provenance in `manifest.json` (L40S, torch
+2.12.1 / transformers 5.13.0 / peft 0.19.1; `git_commit: null` — produced on an rsync'd node copy).

results/llm_moe_hpc/manifest.json

@@ -0,0 +1,27 @@
+{
+  "experiment": "llm_moe_hpc",
+  "master_seed": 1,
+  "git_commit": null,
+  "python": "3.11.13",
+  "libraries": {
+    "numpy": "2.4.6",
+    "scipy": "1.17.1",
+    "pandas": "3.0.3",
+    "pyarrow": "24.0.0",
+    "torch": "2.12.1",
+    "transformers": "5.13.0",
+    "peft": "0.19.1"
+  },
+  "rows": 47,
+  "results_sha256": "bd91413c4cf174980c222dca5f435cb6957dee775d2f27bfb7a30dedc0f36c9f",
+  "layer": "2",
+  "tier": "llm",
+  "base_model": "Qwen/Qwen2.5-7B-Instruct",
+  "operators": [
+    "soup",
+    "ties",
+    "moe_oracle",
+    "moe_learned",
+    "max_merge"
+  ]
+}

results/llm_moe_hpc/resolved_config.yaml

@@ -0,0 +1,28 @@
+experiment: llm_moe_hpc
+seed: 1
+n_replicates: 1
+source_config:
+  experiment: llm_moe_hpc
+  kind: llm_moe
+  seed: 1
+  n_replicates: 1
+  base_model: Qwen/Qwen2.5-7B-Instruct
+  families:
+  - lists
+  - strings
+  - arith
+  n_train: 800
+  n_test: 200
+  n_route: 48
+  epochs: 3
+  lora:
+    r: 16
+    alpha: 32
+  operators:
+  - soup
+  - ties
+  - moe_oracle
+  - moe_learned
+  - max_merge
+  output:
+    dir: results/llm_moe_hpc

src/llm/experiment.py

@@ -24,12 +24,13 @@ from knowledge.experiment import save_artifacts
 
 from .evaluate import evaluate, generate, load_model
 from .merge import load_specialists, make_merge
+from .moe import build_max_merge, embed_prompts, learned_routes, moe_generate
 from .specialise import train_specialist
 from .tasks import FAMILIES, make_tasks, verify
 
 
 def _rows(experiment: str, model: str, kind: str, acc: dict) -> list[dict]:
-    worst = min(acc[f] for f in FAMILIES)
+    worst = min(acc[f] for f in acc if f in FAMILIES)
     out = [{"experiment": experiment, "model": model, "kind": kind, "metric": k, "accuracy": v}
            for k, v in acc.items()]
     out.append({"experiment": experiment, "model": model, "kind": kind,
@@ -85,16 +86,122 @@ def run_merge_experiment(cfg: dict) -> pd.DataFrame:
     return pd.DataFrame(rows)
 
 
+def _load_or_train_specialists(cfg: dict, base: str, fams: list[str], name: str,
+                               rows: list[dict]) -> list[str]:
+    """Reuse cached specialist adapters if present, else train one per family. Appends spec rows."""
+    import torch
+
+    n_train = int(cfg.get("n_train", 700))
+    epochs = int(cfg.get("epochs", 3))
+    lora = cfg.get("lora", {})
+    seed = int(cfg["seed"])
+    adapters_root = Path(cfg.get("adapters_dir", "models/llm"))
+    dirs: list[str] = []
+    for i, f in enumerate(fams):
+        d = str(adapters_root / f"spec_{f}")
+        if not (Path(d) / "adapter_config.json").exists():   # reuse across llm_merge / llm_moe runs
+            train_specialist(base, f, d, n_train=n_train, epochs=epochs, seed=seed + i,
+                             r=int(lora.get("r", 16)), alpha=int(lora.get("alpha", 32)))
+        dirs.append(d)
+        m, tok = load_model(base, adapter_dir=d)
+        rows += _rows(name, f"spec_{f}", "specialist", evaluate(m, tok, test_of(cfg, fams)))
+        del m; torch.cuda.empty_cache()
+    return dirs
+
+
+def test_of(cfg: dict, fams: list[str]) -> list:
+    """The held-out mixed test set (one seed offset per family), shared by both LLM runners."""
+    n_test = int(cfg.get("n_test", 100))
+    return sum([make_tasks(f, n_test, seed=1000 + i) for i, f in enumerate(fams)], [])
+
+
+def run_moe_experiment(cfg: dict) -> pd.DataFrame:
+    """Contrast union-preserving recombination (route / max-merge) with fusion (soup / ties).
+
+    Reuses the trained specialist adapters and evaluates, per operator, on the held-out mixed test
+    set. Union operators keep each specialist intact and *select* (per prompt via a router, or per
+    module via winner-take-all); fusion operators blend the deltas. The real-weight image of E8's
+    ``max`` vs the ``mean`` baseline. Returns long-form accuracies (+ a ``router_acc`` fidelity row
+    for each routed operator).
+    """
+    import torch
+
+    name = cfg["experiment"]
+    base = cfg["base_model"]
+    fams = list(cfg.get("families", list(FAMILIES)))
+    ops = list(cfg.get("operators", ["soup", "ties", "moe_oracle", "moe_learned", "max_merge"]))
+    n_route = int(cfg.get("n_route", 32))
+    rows: list[dict] = []
+
+    test = test_of(cfg, fams)
+    prompts = [t.prompt for t in test]
+    fam = np.array([t.family for t in test])
+    true_idx = np.array([fams.index(t.family) for t in test])
+
+    # base + specialists (parents)
+    m, tok = load_model(base)
+    rows += _rows(name, "base", "base", evaluate(m, tok, test))
+    del m; torch.cuda.empty_cache()
+    dirs = _load_or_train_specialists(cfg, base, fams, name, rows)
+    k = len(dirs)
+
+    # all specialists on one base for the recombination operators
+    model, tok = load_specialists(base, dirs)
+
+    def _score(outs: list[str]) -> dict:
+        corr = np.array([verify(o, t) for o, t in zip(outs, test)])
+        acc = {"overall": float(corr.mean())}
+        acc.update({f: float(corr[fam == f].mean()) for f in fams})
+        return acc
+
+    # precompute router assignments once, from the base model's own prompt embeddings
+    routes: dict[str, np.ndarray] = {}
+    if "moe_oracle" in ops:
+        routes["moe_oracle"] = true_idx
+    if "moe_learned" in ops:
+        route_train = sum([make_tasks(f, n_route, seed=2000 + i) for i, f in enumerate(fams)], [])
+        tr_emb = embed_prompts(model, tok, [t.prompt for t in route_train])
+        te_emb = embed_prompts(model, tok, prompts)
+        tr_fam = np.array([t.family for t in route_train])
+        routes["moe_learned"] = learned_routes(tr_emb, tr_fam, te_emb, fams)
+
+    for op in ops:
+        if op in ("soup", "ties", "task_arith"):
+            make_merge(model, k, op, op)
+            rows += _rows(name, f"merge_{op}", "merge", _score(generate(model, tok, prompts)))
+        elif op in ("moe_oracle", "moe_learned"):
+            r = routes[op]
+            rows += _rows(name, op, "moe", _score(moe_generate(model, tok, prompts, r, k)))
+            rows.append({"experiment": name, "model": op, "kind": "moe",
+                         "metric": "router_acc", "accuracy": float((r == true_idx).mean())})
+        elif op == "max_merge":
+            build_max_merge(model, k, "max_merge")
+            rows += _rows(name, "max_merge", "moe", _score(generate(model, tok, prompts)))
+        else:
+            raise ValueError(f"unknown operator {op!r} (soup|ties|task_arith|moe_oracle|"
+                             f"moe_learned|max_merge)")
+
+    return pd.DataFrame(rows)
+
+
+_RUNNERS = {"llm_merge": run_merge_experiment, "llm_moe": run_moe_experiment}
+
+
 def run_and_save(config_path: str | Path) -> Path:
-    """Load an LLM experiment YAML, run it, and write the artifact triple."""
+    """Load an LLM experiment YAML, run it (dispatch on ``kind``), and write the artifact triple."""
     config_path = Path(config_path)
     cfg = yaml.safe_load(config_path.read_text())
     out_dir = Path(cfg.get("output", {}).get("dir", f"results/{cfg['experiment']}"))
     cfg.setdefault("n_replicates", 1)
-    df = run_merge_experiment(cfg)
+    kind = cfg.get("kind", "llm_merge")
+    if kind not in _RUNNERS:
+        raise ValueError(f"unknown LLM experiment kind {kind!r} (expected one of {list(_RUNNERS)})")
+    df = _RUNNERS[kind](cfg)
+    extra = {"layer": "2", "tier": "llm", "base_model": cfg["base_model"]}
+    if kind == "llm_moe":
+        extra["operators"] = list(cfg.get("operators", []))
     save_artifacts(cfg, df, out_dir, extra_libs=("torch", "transformers", "peft"),
-                   extra_manifest={"layer": "2", "tier": "llm", "base_model": cfg["base_model"]},
-                   grid=None)
+                   extra_manifest=extra, grid=None)
     return out_dir
 
 

src/llm/moe.py

@@ -0,0 +1,154 @@
+"""Module-level, union-preserving recombination of specialists — the real-weight image of E8's *max*.
+
+`merge.py` fuses specialist adapters into one set of weights (``soup`` averages the deltas, ``ties``
+sign-reconciles them). Fusion can *dilute*: at a weak base, averaging a specialist's delta by ``1/K``
+erased its peak (the 0.5B ``llm_merge`` result). The alternative recombination operator — the one E8
+calls *max* / union-preserving — never averages the parents at all. It keeps every specialist adapter
+**intact** and, per input (or per module), **selects** the parent that owns that skill. Nothing is
+diluted because nothing is blended; the child is the *union* of the parents' capabilities.
+
+Two selection operators, both reusing the already-trained specialist adapters (no retraining):
+
+* **route** — a Mixture-of-Experts over the specialists: a cheap router assigns each prompt to one
+  specialist adapter, which then answers it. Router variants: ``oracle`` (route by the known task
+  family — the ceiling of routing) and ``learned`` (nearest-centroid over the *base* model's own
+  prompt embeddings — an honest, training-free router; its accuracy is reported).
+* **max_merge** — a router-free static union: build a single adapter that, per LoRA module, copies the
+  delta from the specialist whose delta has the largest norm there (winner-take-all per module). The
+  literal weight-space image of E8's element-wise ``max`` over teachers.
+
+The experiment (``kind: llm_moe``) contrasts these union operators against the fusion baselines
+(soup/ties) at both scales — the prediction being that union wins where fusion dilutes (0.5B) and the
+gap narrows once a capable base lets fusion *compose* rather than dilute (7B): the regime boundary of
+"merge, don't average" in real LLM weights.
+"""
+
+from __future__ import annotations
+
+import numpy as np
+
+from .evaluate import generate
+
+
+def embed_prompts(model, tok, prompts: list[str], batch_size: int = 32,
+                  device: str = "cuda") -> np.ndarray:
+    """Mean last-hidden-state embedding of each prompt from the **base** model (adapters disabled).
+
+    The router must see a parent-agnostic representation, so generation happens with
+    ``model.disable_adapter()`` — the embedding is the frozen base model's, not any specialist's.
+
+    Args:
+        model: a PEFT model with specialist adapters loaded.
+        tok: the matching tokenizer (left-padded).
+        prompts (list[str]): raw user prompts.
+        batch_size (int): forward-pass batch size.
+        device (str): compute device.
+
+    Returns:
+        np.ndarray: ``(len(prompts), hidden)`` float32 embeddings.
+    """
+    import torch
+
+    embs: list[np.ndarray] = []
+    with model.disable_adapter():                          # parent-agnostic base representation
+        for i in range(0, len(prompts), batch_size):
+            chunk = prompts[i:i + batch_size]
+            msgs = [[{"role": "user", "content": p}] for p in chunk]
+            enc = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt",
+                                          return_dict=True, padding=True).to(device)
+            with torch.no_grad():
+                out = model(**enc, output_hidden_states=True)
+            h = out.hidden_states[-1]                       # (B, T, H)
+            mask = enc["attention_mask"].unsqueeze(-1)      # (B, T, 1)
+            mean = (h * mask).sum(1) / mask.sum(1).clamp(min=1)
+            embs.append(mean.float().cpu().numpy())
+    return np.concatenate(embs, axis=0)
+
+
+def learned_routes(train_emb: np.ndarray, train_fam: np.ndarray, test_emb: np.ndarray,
+                   fams: list[str]) -> np.ndarray:
+    """Nearest-centroid router: assign each test prompt to the family whose train centroid is closest.
+
+    A training-free expert selector — the "regulatory gene" that decides which specialist to express.
+
+    Args:
+        train_emb (np.ndarray): ``(n_train, H)`` base embeddings of labelled train prompts.
+        train_fam (np.ndarray): family label per train prompt.
+        test_emb (np.ndarray): ``(n_test, H)`` base embeddings of test prompts.
+        fams (list[str]): family order (index i ↔ adapter ``a{i}``).
+
+    Returns:
+        np.ndarray: ``(n_test,)`` int expert index in ``[0, len(fams))``.
+    """
+    centroids = np.stack([train_emb[train_fam == f].mean(0) for f in fams])   # (K, H)
+    # Cosine distance is scale-robust for hidden states.
+    c = centroids / (np.linalg.norm(centroids, axis=1, keepdims=True) + 1e-8)
+    t = test_emb / (np.linalg.norm(test_emb, axis=1, keepdims=True) + 1e-8)
+    sims = t @ c.T                                          # (n_test, K)
+    return sims.argmax(1)
+
+
+def moe_generate(model, tok, prompts: list[str], routes: np.ndarray, k: int,
+                 **gen_kw) -> list[str]:
+    """Generate each prompt with its routed specialist adapter active (grouped by expert for speed).
+
+    Args:
+        model: PEFT model with adapters ``a0..a{k-1}`` loaded.
+        tok: tokenizer.
+        prompts (list[str]): prompts to answer.
+        routes (np.ndarray): expert index per prompt (from an oracle or learned router).
+        k (int): number of experts.
+        **gen_kw: forwarded to :func:`llm.evaluate.generate`.
+
+    Returns:
+        list[str]: completions, in the original prompt order.
+    """
+    outs: list[str] = [""] * len(prompts)
+    for expert in range(k):
+        idx = np.nonzero(routes == expert)[0]
+        if len(idx) == 0:
+            continue
+        model.set_adapter(f"a{expert}")
+        completions = generate(model, tok, [prompts[i] for i in idx], **gen_kw)
+        for j, i in enumerate(idx):
+            outs[i] = completions[j]
+    return outs
+
+
+def build_max_merge(model, k: int, name: str):
+    """Add a router-free union adapter ``name``: per LoRA module, keep the largest-norm specialist delta.
+
+    For each LoRA-adapted module, the effective delta of specialist ``a_i`` is ``B_i @ A_i`` (scaled).
+    We select, per module, the specialist whose delta has the largest Frobenius norm and copy its
+    ``A``/``B`` into a fresh adapter — a deterministic winner-take-all union (E8's element-wise max at
+    the granularity of a module). No averaging, so no dilution.
+
+    Args:
+        model: PEFT model with adapters ``a0..a{k-1}``.
+        k (int): number of specialists.
+        name (str): name of the new merged adapter (created as a copy of ``a0`` then overwritten).
+
+    Returns:
+        The model, with adapter ``name`` added and set active.
+    """
+    import torch
+    from peft.tuners.lora import LoraLayer
+
+    # Seed the new adapter from a0's config, then overwrite its weights per module.
+    model.add_weighted_adapter([f"a{i}" for i in range(k)], [1.0] + [0.0] * (k - 1), name,
+                               combination_type="linear")
+    with torch.no_grad():
+        for module in model.modules():
+            if not isinstance(module, LoraLayer) or name not in module.lora_A:
+                continue
+            norms = []
+            for i in range(k):
+                a, b = f"a{i}", f"a{i}"
+                delta = module.lora_B[a].weight @ module.lora_A[a].weight
+                norms.append(float(delta.norm()) * module.scaling.get(f"a{i}", 1.0))
+            win = int(np.argmax(norms))
+            module.lora_A[name].weight.copy_(module.lora_A[f"a{win}"].weight)
+            module.lora_B[name].weight.copy_(module.lora_B[f"a{win}"].weight)
+            module.scaling[name] = module.scaling.get(f"a{win}", 1.0)
+    model.set_adapter(name)
+    return model

tasks/todo.md

@@ -352,3 +352,18 @@ C3 vertical claim deferred.*
   and dilution VANISHES (merge 0.62 > lists-spec 0.57 on lists) — dilution was a small-model artefact.
 - `results/llm_merge_hpc/` (README legend, data-driven figure title). Next refinement: module-level
   union-preserving recombination (MoE-expert/adapter-union = real-weight E8 max-merge), not delta-avg.
+
+**2026-07-05 — MoE-expert / union recombination (`llm_moe`): E8's `max` vs `mean` in real weights.**
+- `src/llm/moe.py`: router (oracle + training-free nearest-centroid over base embeddings) + MoE
+  generate + router-free per-module `max_merge`. `kind: llm_moe` reuses the cached specialists.
+- **0.5B result:** routing beats fusion decisively — overall 0.74/worst 0.43 vs soup 0.64/0.26, no
+  dilution (recovers each specialist's own-family peak). Riders: learned router trivially perfect
+  (1.00, lexically-separable families) and static `max_merge` a poor union (0.46, not input-adaptive).
+- `configs/llm/{moe,moe_hpc}.yaml`, `figures/plot_llm_moe.py`, README, +2 tests (127 green),
+  `hpc/llm_moe.pbs`.
+- **7B firm-up (`llm_moe_hpc`, CX3 L40S, 9 min): the ordering FLIPS.** At 7B fusion wins —
+  soup 0.87 > routing 0.84 > max_merge 0.78 (0.5B had routing 0.74 > soup 0.64). Routing is capped at
+  the best parent per family; fusion *composes beyond* it at a capable base (soup lists 0.62 >
+  spec 0.57). So "merge, don't average" is a **weak-base law**, not universal — union wins under
+  dilution (0.5B), fusion wins under composition (7B). Refines E8. Next: fusion + offspring-selection
+  (directed sex). `results/llm_moe_hpc/` README + regime-aware figure.

tests/test_llm.py

@@ -8,6 +8,9 @@ accepts correct answers (including verbose model phrasings) and rejects wrong on
 
 from __future__ import annotations
 
+import numpy as np
+
+from llm.moe import learned_routes
 from llm.tasks import FAMILIES, make_tasks, verify
 
 
@@ -37,3 +40,29 @@ def test_verifier_rejects_wrong():
     assert not verify(wrong, t)
     lt = next(x for x in make_tasks("lists", 30, seed=6) if x.answer.startswith("["))
     assert not verify("[9, 9, 9]", lt) or lt.answer == "[9, 9, 9]"
+
+
+def test_learned_router_assigns_nearest_centroid():
+    # Three well-separated families in a 4-D "embedding" space; the nearest-centroid router
+    # (the MoE expert-selection gene) must route each test prompt to its own family's specialist.
+    rng = np.random.default_rng(0)
+    fams = ["lists", "strings", "arith"]
+    anchors = {"lists": [5, 0, 0, 0], "strings": [0, 5, 0, 0], "arith": [0, 0, 5, 0]}
+    train_emb = np.array([anchors[f] for f in fams for _ in range(8)], dtype=float)
+    train_emb += rng.normal(scale=0.1, size=train_emb.shape)
+    train_fam = np.array([f for f in fams for _ in range(8)])
+    test_fam = np.array(["arith", "lists", "strings", "arith"])
+    test_emb = np.array([anchors[f] for f in test_fam], dtype=float) + rng.normal(scale=0.1, size=(4, 4))
+    routes = learned_routes(train_emb, train_fam, test_emb, fams)
+    assert [fams[r] for r in routes] == list(test_fam)          # each routed to its own family
+
+
+def test_learned_router_is_cosine_scale_invariant():
+    # Cosine routing must ignore prompt-embedding magnitude (long vs short prompts): a test point on a
+    # family's ray routes there regardless of its norm.
+    fams = ["a", "b"]
+    train_emb = np.array([[1.0, 0.0], [1.0, 0.0], [0.0, 1.0], [0.0, 1.0]])
+    train_fam = np.array(["a", "a", "b", "b"])
+    test_emb = np.array([[10.0, 0.0], [0.0, 0.01]])            # very different magnitudes
+    routes = learned_routes(train_emb, train_fam, test_emb, fams)
+    assert [fams[r] for r in routes] == ["a", "b"]