MachineSex/results/llm_moe

llm_moe — union-preserving recombination (route / max-merge) vs fusion (the real-weight E8 max)

Claim tested. E8 (analytic) found that the recombination operator matters: a union over parents (max) assembles a child fitter than any parent, while an average (mean, the model soup) conserves collapse. llm_merge showed fusion (soup/ties, which average the LoRA deltas). This experiment adds the union operator to real LLM weights — never average the parents, keep each specialist intact and select the right one per input (a Mixture-of-Experts router) or per module (winner-take-all) — and asks whether union beats fusion, and where.

Setup. Base Qwen2.5-0.5B-Instruct, reusing the three cached LoRA specialists from llm_merge (disjoint families lists/strings/arith, exact-match verifier), 100 test tasks/family, seed 1. Five recombination operators on the same test set:

• Fusion (blend the deltas): soup = mean(Δₖ); ties = sign-reconciled union.
• Union (never average): route:oracle/route:learned keep all adapters live and route each prompt to one specialist (MoE); max-merge builds one adapter taking, per module, the specialist with the largest-norm delta. The learned router is training-free — nearest-centroid over the base model's own prompt embeddings; its routing accuracy is reported.

Results (accuracy)

operator	lists	strings	arith	overall	worst-family	router
base	0.15	0.15	0.53	0.28	0.15	—
best specialist (strings)	0.08	1.00	0.80	0.63	0.08	—
fuse: soup	0.26	0.74	0.91	0.64	0.26	—
fuse: ties	0.23	0.71	0.90	0.61	0.23	—
route: oracle	0.43	1.00	0.78	0.74	0.43	1.00
route: learned	0.43	1.00	0.78	0.74	0.43	1.00
max-merge	0.18	0.34	0.87	0.46	0.18	—

What holds

• Union (routing) beats fusion at a weak base — decisively. Routing reaches 0.74 overall / 0.43 worst-family, above both fusion merges (soup 0.64/0.26) and every specialist. It recovers each specialist's own-family peak exactly (lists 0.43, strings 1.00, arith 0.78) because it is that specialist there — no dilution. This is E8's max (union) beating mean (average) in real LLM weights: exactly where fusion diluted the lists-specialist (0.43→0.26), routing keeps 0.43.
• The learned router is perfect here (1.00) — stated as a caveat, not a triumph. The three families are lexically distinct, so nearest-centroid routing over base embeddings is trivially easy; route:learned equals route:oracle. Routing's advantage on these tasks therefore rests partly on the routing problem being easy — the honest scope. On overlapping/ambiguous skills the router would be the bottleneck, and that is the interesting failure mode to probe next.
• Static per-module max-merge is a poor union (0.46) — an informative negative. Picking, per module, the largest-norm specialist delta is not input-adaptive: it collapses toward whichever specialist dominates the weight norms (arith 0.87, but lists 0.18, strings 0.34). The union benefit needs routing (input-adaptive selection), not weight-space surgery — "keep the parents whole" only pays off if you also choose the right parent per input.

Takeaway

Adds the union half of E8's operator dichotomy to real LLM weights and confirms its sign at a weak base: route-don't-average > average, with no dilution, mirroring the analytic max > mean. Two honest riders — the learned router is trivially good because the families are lexically separable, and the router-free max-merge union fails because it isn't input-adaptive. The regime question the 0.5B result raises — does routing still help once a capable base lets fusion compose rather than dilute? (llm_merge_hpc showed 7B soup already beats its specialists with no dilution) — is answered by results/llm_moe_hpc/: the ordering flips. At 7B fusion wins (soup 0.87 > routing 0.84), because routing is capped at the best parent while fusion composes beyond it. So "merge, don't average" is a weak-base law: union wins here (0.5B, dilution regime), fusion wins there (7B, composition regime). Falsifier (not triggered at 0.5B): fusion matching the routing ceiling, i.e. averaging never diluting.