Giorgio Gilestro
8da0dac007
Adds the union-preserving recombination operator that llm_merge lacked (E8's max,
not mean): keep each specialist LoRA intact and SELECT the right one per prompt
(MoE router: oracle, or training-free nearest-centroid over base embeddings) or
per module (max_merge = winner-take-all by delta norm). src/llm/moe.py, kind
llm_moe, reuses the cached specialists.
Result — a clean regime boundary for "merge, don't average":
- 0.5B: union wins. Routing 0.74 / worst-family 0.43 > soup 0.64 / 0.26, with no
dilution (recovers each specialist's own-family peak). E8's max > mean in real
weights, because at a weak base averaging dilutes.
- 7B (Imperial CX3, L40S, 9 min): the ordering INVERTS. Fusion wins — soup 0.87 >
routing 0.84 > max_merge 0.78. Routing is capped at the best parent per family;
fusion blends and, given a capable base, COMPOSES beyond any parent (soup lists
0.62 > spec 0.57). Selection can't synthesise better than its best component;
averaging-that-composes can.
So "merge, don't average" (E4/E8) is a weak-parent / small-model law, not
universal: union wins under dilution, fusion wins under composition. Refines E8
(its additive-landscape max>mean assumed no compositional headroom). The operator
to want is fusion-that-composes + offspring selection = the directed-sex ideal
(E10) — the natural next experiment.
Honest riders: the learned router is trivially perfect (lexically-distinct
families), and router-free max_merge is the weakest union (not input-adaptive).
+2 router unit tests (127 green). Results in results/llm_moe{,_hpc}/ (parquet
gitignored per the reproducibility contract).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
52 lines
3.8 KiB
Markdown
52 lines
3.8 KiB
Markdown
# llm_moe_hpc — union vs fusion recombination at scale (7B, Imperial CX3): the regime *flips*
|
||
|
||
**Claim tested.** The 0.5B `llm_moe` found that a **union** operator (route/select, never average) beats
|
||
**fusion** (soup/ties, average the deltas), because at a weak base averaging *dilutes*. But
|
||
`llm_merge_hpc` showed that at 7B fusion stops diluting and starts *composing* (soup 0.87 beat every
|
||
specialist). So the sharp question: **does routing still beat fusion once the base is capable — or does
|
||
a strong base invert the ordering?** Run on one **L40S (46 GB)** GPU of Imperial's CX3 HPC, 9 min
|
||
walltime, reusing the cached 7B specialists.
|
||
|
||
**Setup.** Base **Qwen2.5-7B-Instruct**, the three cached LoRA specialists from `llm_merge_hpc`
|
||
(`lists`/`strings`/`arith`, exact-match verifier), 200 test tasks/family, seed 1. Same five operators
|
||
as the 0.5B run: fusion (`soup`, `ties`) vs union (`route:oracle`, `route:learned`, `max-merge`).
|
||
|
||
### Results (accuracy)
|
||
| operator | lists | strings | arith | overall | worst-family | router |
|
||
|---|---|---|---|---|---|---|
|
||
| base | 0.46 | 0.69 | 1.00 | 0.71 | 0.46 | — |
|
||
| best specialist (lists) | 0.57 | 0.74 | 1.00 | 0.77 | 0.57 | — |
|
||
| **fuse: soup** | **0.62** | **1.00** | 1.00 | **0.87** | **0.62** | — |
|
||
| fuse: ties | 0.62 | 1.00 | 0.99 | 0.87 | 0.62 | — |
|
||
| route: oracle | 0.57 | 0.97 | 0.96 | 0.84 | 0.57 | 1.00 |
|
||
| route: learned | 0.57 | 0.97 | 0.96 | 0.84 | 0.57 | 1.00 |
|
||
| max-merge | 0.48 | 0.90 | 0.96 | 0.78 | 0.48 | — |
|
||
|
||
### The finding: "merge, don't average" is regime-dependent, and inverts at scale
|
||
- **The ordering flips.** At 0.5B, union > fusion (routing 0.74 > soup 0.64). At 7B, **fusion > union**
|
||
(soup **0.87** > routing 0.84 > max-merge 0.78). The exact opposite winner.
|
||
- **Why: routing is capped at the best parent; fusion can *exceed* it.** Routing *selects* one intact
|
||
specialist, so per family it can only reach that specialist's own score (lists 0.57 = spec_lists,
|
||
strings 0.97 = spec_strings). Fusion *blends* the deltas — and at a capable base the blend
|
||
**composes beyond any parent**: soup scores lists 0.62 (> spec_lists 0.57) and strings 1.00
|
||
(> spec_strings 0.97). Selection cannot synthesise something better than its best component;
|
||
averaging, when it composes rather than dilutes, can. So fusion's worst-family (0.62) also beats
|
||
routing's (0.57).
|
||
- **The regime boundary is dilution.** Union wins exactly when averaging *dilutes* (weak base, 0.5B);
|
||
fusion wins once the base has enough headroom that averaging *composes* (7B). "Merge, don't average"
|
||
(E4/E8) is therefore a **small-model / weak-parent** law, not a universal one — a genuine refinement
|
||
of the analytic claim, not a contradiction of it (E8's additive-landscape `max > mean` assumed no
|
||
such compositional headroom).
|
||
- **Riders unchanged.** The learned router is still perfect (1.00, lexically-distinct families), and
|
||
`max-merge` remains the weakest union (0.78) — static per-module winner-take-all is not
|
||
input-adaptive.
|
||
|
||
### Takeaway
|
||
A clean, honest regime result: **route-don't-average wins at a weak base; average-that-composes wins at
|
||
a strong one.** Pure selection (routing) never dilutes but is bounded by the best parent; fusion risks
|
||
dilution but, given a capable base, transcends the parents — which is what the Fisher–Muller "exceed
|
||
every parent" claim actually needs at scale. The operator to want is therefore *fusion that composes
|
||
plus selection over candidates* — the "directed sex" ideal (offspring selection over recombinants),
|
||
the natural next experiment. **Falsifier for this run (not triggered):** routing beating fusion at 7B,
|
||
i.e. dilution persisting at scale — instead it inverted. Provenance in `manifest.json` (L40S, torch
|
||
2.12.1 / transformers 5.13.0 / peft 0.19.1; `git_commit: null` — produced on an rsync'd node copy).
|