# llm_moe_hpc — union vs fusion recombination at scale (7B, Imperial CX3): the regime *flips*

**Claim tested.** The 0.5B `llm_moe` found that a **union** operator (route/select, never average) beats
**fusion** (soup/ties, average the deltas), because at a weak base averaging *dilutes*. But
`llm_merge_hpc` showed that at 7B fusion stops diluting and starts *composing* (soup 0.87 beat every
specialist). So the sharp question: **does routing still beat fusion once the base is capable — or does
a strong base invert the ordering?** Run on one **L40S (46 GB)** GPU of Imperial's CX3 HPC, 9 min
walltime, reusing the cached 7B specialists.

**Setup.** Base **Qwen2.5-7B-Instruct**, the three cached LoRA specialists from `llm_merge_hpc`
(`lists`/`strings`/`arith`, exact-match verifier), 200 test tasks/family, seed 1. Same five operators
as the 0.5B run: fusion (`soup`, `ties`) vs union (`route:oracle`, `route:learned`, `max-merge`).

### Results (accuracy)
| operator | lists | strings | arith | overall | worst-family | router |
|---|---|---|---|---|---|---|
| base | 0.46 | 0.69 | 1.00 | 0.71 | 0.46 | — |
| best specialist (lists) | 0.57 | 0.74 | 1.00 | 0.77 | 0.57 | — |
| **fuse: soup** | **0.62** | **1.00** | 1.00 | **0.87** | **0.62** | — |
| fuse: ties | 0.62 | 1.00 | 0.99 | 0.87 | 0.62 | — |
| route: oracle | 0.57 | 0.97 | 0.96 | 0.84 | 0.57 | 1.00 |
| route: learned | 0.57 | 0.97 | 0.96 | 0.84 | 0.57 | 1.00 |
| max-merge | 0.48 | 0.90 | 0.96 | 0.78 | 0.48 | — |

### The finding: "merge, don't average" is regime-dependent, and inverts at scale
- **The ordering flips.** At 0.5B, union > fusion (routing 0.74 > soup 0.64). At 7B, **fusion > union**
  (soup **0.87** > routing 0.84 > max-merge 0.78). The exact opposite winner.
- **Why: routing is capped at the best parent; fusion can *exceed* it.** Routing *selects* one intact
  specialist, so per family it can only reach that specialist's own score (lists 0.57 = spec_lists,
  strings 0.97 = spec_strings). Fusion *blends* the deltas — and at a capable base the blend
  **composes beyond any parent**: soup scores lists 0.62 (> spec_lists 0.57) and strings 1.00
  (> spec_strings 0.97). Selection cannot synthesise something better than its best component;
  averaging, when it composes rather than dilutes, can. So fusion's worst-family (0.62) also beats
  routing's (0.57).
- **The regime boundary is dilution.** Union wins exactly when averaging *dilutes* (weak base, 0.5B);
  fusion wins once the base has enough headroom that averaging *composes* (7B). "Merge, don't average"
  (E4/E8) is therefore a **small-model / weak-parent** law, not a universal one — a genuine refinement
  of the analytic claim, not a contradiction of it (E8's additive-landscape `max > mean` assumed no
  such compositional headroom).
- **Riders unchanged.** The learned router is still perfect (1.00, lexically-distinct families), and
  `max-merge` remains the weakest union (0.78) — static per-module winner-take-all is not
  input-adaptive.

### Takeaway
A clean, honest regime result: **route-don't-average wins at a weak base; average-that-composes wins at
a strong one.** Pure selection (routing) never dilutes but is bounded by the best parent; fusion risks
dilution but, given a capable base, transcends the parents — which is what the Fisher–Muller "exceed
every parent" claim actually needs at scale. The operator to want is therefore *fusion that composes
plus selection over candidates* — the "directed sex" ideal (offspring selection over recombinants),
the natural next experiment. **Falsifier for this run (not triggered):** routing beating fusion at 7B,
i.e. dilution persisting at scale — instead it inverted. Provenance in `manifest.json` (L40S, torch
2.12.1 / transformers 5.13.0 / peft 0.19.1; `git_commit: null` — produced on an rsync'd node copy).