llm_moe: the union operator (route/max-merge) vs fusion — and the regime flips at scale

Adds the union-preserving recombination operator that llm_merge lacked (E8's max,
not mean): keep each specialist LoRA intact and SELECT the right one per prompt
(MoE router: oracle, or training-free nearest-centroid over base embeddings) or
per module (max_merge = winner-take-all by delta norm). src/llm/moe.py, kind
llm_moe, reuses the cached specialists.

Result — a clean regime boundary for "merge, don't average":
- 0.5B: union wins. Routing 0.74 / worst-family 0.43 > soup 0.64 / 0.26, with no
  dilution (recovers each specialist's own-family peak). E8's max > mean in real
  weights, because at a weak base averaging dilutes.
- 7B (Imperial CX3, L40S, 9 min): the ordering INVERTS. Fusion wins — soup 0.87 >
  routing 0.84 > max_merge 0.78. Routing is capped at the best parent per family;
  fusion blends and, given a capable base, COMPOSES beyond any parent (soup lists
  0.62 > spec 0.57). Selection can't synthesise better than its best component;
  averaging-that-composes can.

So "merge, don't average" (E4/E8) is a weak-parent / small-model law, not
universal: union wins under dilution, fusion wins under composition. Refines E8
(its additive-landscape max>mean assumed no compositional headroom). The operator
to want is fusion-that-composes + offspring selection = the directed-sex ideal
(E10) — the natural next experiment.

Honest riders: the learned router is trivially perfect (lexically-distinct
families), and router-free max_merge is the weakest union (not input-adaptive).
+2 router unit tests (127 green). Results in results/llm_moe{,_hpc}/ (parquet
gitignored per the reproducibility contract).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

figures/plot_llm_moe.py

@@ -0,0 +1,97 @@
+"""llm_moe figure — union-preserving recombination (route / max-merge) vs fusion (soup / ties).
+
+The real-weight image of E8's *max*: keep every specialist intact and *select* (route per prompt, or
+per module) instead of averaging the deltas. Two panels: (A) per-family accuracy for the base, each
+specialist, the fusion merges, and the union operators — the union operators should match the best
+specialist on every family (they *are* that specialist there), while fusion may dilute or compose;
+(B) overall vs worst-family, fusion vs union, with the routing ceiling (moe_oracle) marked. The
+suptitle reports whether union beats fusion (dilution regime) or they converge (composition regime).
+Reads only the committed bundle.
+
+Usage:  python figures/plot_llm_moe.py [results/llm_moe]
+"""
+
+from __future__ import annotations
+
+import sys
+from pathlib import Path
+
+import matplotlib.pyplot as plt
+import numpy as np
+
+sys.path.insert(0, str(Path(__file__).parent))
+from _figlib import load_bundle, savefig  # noqa: E402
+
+_FAMS = ["lists", "strings", "arith"]
+_FUSION = {"merge_soup": "fuse:soup", "merge_ties": "fuse:ties"}
+_UNION = {"moe_oracle": "route:oracle", "moe_learned": "route:learned", "max_merge": "max-merge"}
+
+
+def _acc(df, model, metric):
+    r = df[(df["model"] == model) & (df["metric"] == metric)]["accuracy"]
+    return float(r.iloc[0]) if len(r) else float("nan")
+
+
+def main(results_dir: str = "results/llm_moe") -> None:
+    df, cfg = load_bundle(results_dir)
+    present = set(df["model"].unique())
+    specialists = sorted(m for m in present if m.startswith("spec_"))
+    fusion = [m for m in _FUSION if m in present]
+    union = [m for m in _UNION if m in present]
+    models = ["base"] + specialists + fusion + union
+    labels = {"base": "base", **{s: s.replace("spec_", "spec:") for s in specialists},
+              **_FUSION, **_UNION}
+    colors = {"base": "#7f7f7f", **{s: "#1f77b4" for s in specialists},
+              **{m: "#ff7f0e" for m in fusion}, **{m: "#2ca02c" for m in union}}
+
+    fig, axes = plt.subplots(1, 2, figsize=(14, 5))
+
+    # Panel A: per-family accuracy, grouped by model.
+    ax = axes[0]
+    x = np.arange(len(_FAMS))
+    w = 0.8 / len(models)
+    for i, mdl in enumerate(models):
+        vals = [_acc(df, mdl, f) for f in _FAMS]
+        ax.bar(x + (i - (len(models) - 1) / 2) * w, vals, w, label=labels[mdl], color=colors[mdl],
+               alpha=0.9 if (mdl in union or mdl in fusion) else 0.65)
+    ax.set_xticks(x); ax.set_xticklabels(_FAMS)
+    ax.set(ylabel="accuracy", title="Per-family: fusion (orange) blends the deltas; union (green)\n"
+           "keeps each specialist intact and selects — no dilution")
+    ax.legend(frameon=False, fontsize=8, ncol=2)
+
+    # Panel B: overall vs worst-family, per model.
+    ax = axes[1]
+    x2 = np.arange(len(models))
+    for off, metric, hatch, lab in [(-0.2, "overall", "", "overall"),
+                                    (0.2, "worst_family", "//", "worst family")]:
+        ax.bar(x2 + off, [_acc(df, m, metric) for m in models], 0.38,
+               color=[colors[m] for m in models], hatch=hatch, alpha=0.85, label=lab,
+               edgecolor="white")
+    ax.set_xticks(x2); ax.set_xticklabels([labels[m] for m in models], rotation=25, ha="right",
+                                          fontsize=8)
+    ax.set(ylabel="accuracy", title="Overall (solid) vs worst-family (hatched):\nfusion vs union "
+           "recombination")
+    # Mark the routing ceiling (oracle) if present.
+    if "moe_oracle" in present:
+        ceil = _acc(df, "moe_oracle", "overall")
+        ax.axhline(ceil, ls=":", c="#2ca02c", lw=1, alpha=0.7)
+    ax.legend(frameon=False, fontsize=9)
+
+    best_fuse = max([_acc(df, m, "overall") for m in fusion], default=float("nan"))
+    best_union = max([_acc(df, m, "overall") for m in union], default=float("nan"))
+    router = _acc(df, "moe_learned", "router_acc") if "moe_learned" in present else float("nan")
+    if best_union > best_fuse + 0.01:
+        verdict = f"union {best_union:.2f} > fusion {best_fuse:.2f} overall (fusion dilutes)"
+    elif best_fuse > best_union + 0.01:
+        verdict = f"fusion {best_fuse:.2f} > union {best_union:.2f} overall (strong base composes)"
+    else:
+        verdict = f"union ≈ fusion ({best_union:.2f} vs {best_fuse:.2f}) overall"
+    rtxt = f"; learned router {router:.2f}" if router == router else ""
+    fig.suptitle(f"llm_moe — module-level union vs fusion recombination: {verdict}{rtxt} "
+                 f"({cfg['base_model'].split('/')[-1]})", y=1.0, fontsize=12)
+    fig.tight_layout()
+    savefig(fig, results_dir, "llm_moe")
+
+
+if __name__ == "__main__":
+    main(*sys.argv[1:])