Giorgio Gilestro
84124de143
Clarity pass over the main text (36-item audit), Discussion rewrite and cut, acknowledgements, Souly et al. as ref 62, lettered SI panels, model section moved under Results; plus the untracked curriculum/society/compose/smol configs, runners, figures, stats and tests that the SI already cites. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y64o8FKP7rCuXzC48pxpMm
1.5 KiB
llm_moe_hard_seeds_smol — the headroom rule on a second base lineage (SmolLM2-1.7B-Instruct)
Claim tested (manuscript revision 2026-09-12). Union (routing among intact specialists) beats
fusion (weight averaging) wherever the average falls short of attainable performance. Replicates
llm_moe_hard_seeds (hard task variant, 400 training tasks, 80 test tasks per family, soup / TIES /
oracle router / learned router, 3 training seeds) on HuggingFaceTB/SmolLM2-1.7B-Instruct. Ran in
7 min on one L40S (hpc/llm_smol.pbs). Numbers from figures/stats_llm_smol.py.
Result — replicates, 3/3 seeds, larger margin than on Qwen
| contrast (SmolLM2, hard) | overall | worst family |
|---|---|---|
| routing (learned = oracle) − soup | +0.162 ± 0.036 (3/3) | +0.221 ± 0.029 (3/3) |
| soup − best specialist | −0.029 ± 0.017 (3/3 below) | +0.012 ± 0.014 |
| (Qwen2.5-0.5B) routing − soup | +0.100 ± 0.090 (3/3) | +0.150 ± 0.075 (3/3) |
| (Qwen) soup − best specialist | −0.025 ± 0.122 (1/3 above) | 0.000 ± 0.102 |
Absolute levels (SmolLM2): best specialist 0.194, soup 0.165, routing 0.328 overall. Two riders carry over unchanged: the learned router equals the oracle (the families are lexically separable, so routing is easy here), and on this base the soup falls below the best single specialist in every seed, the dilution the Qwen 0.5B hard run showed in one seed of three. Falsifier (not triggered): routing ≤ soup in a majority of seeds.