The easy task families saturated 7B (strings & arith at 1.00), so the earlier
7B nulls — moe: fusion 0.87 > union 0.84; directed ~= soup — could not separate
"refinements don't help at scale" from "tasks too easy at 7B". Adds a hard task
variant (hard: true in tasks.py: multi-step lists, Caesar ciphers / letter
transforms, multi-step & larger arithmetic; same family labels and answer
formats, threaded through make_tasks/train_specialist/runners; hard specialists
cache separately as spec_*_hard) and re-runs both experiments at 7B on Imperial
CX3 (one L40S, 24 min, unsaturated: arith ~0.48, strings 0.67, lists 0.34).
Both nulls flip back to the 0.5B ordering:
- Union beats fusion again: routing 0.500 > fusion 0.40 (soup 0.392 / ties
0.400), the same 10-pt margin as 0.5B. Fusion dilutes the fragile strings
specialist so hard (0.665 -> soup 0.300) that soup even trails the best single
specialist (0.425); routing keeps it intact (0.670).
- Directed selection beats soup again: 0.492 > 0.392 (+10 pts), recovering most
of routing's benefit from one deployable merged model (lifts strings to 0.630).
Correction to the earlier interpretation: the llm_moe_hpc "regime flip" and the
llm_directed_hpc "no headroom" null were driven by TASK SATURATION, not base
capability. The operative variable is headroom — "merge, don't average" (union >
fusion) and "directed sex" (selection > single blend) hold whenever there is room
to lose to dilution: a weak base (0.5B) OR hard tasks at a strong base (7B-hard).
Fusion only wins in the degenerate corner where easy tasks let a strong base
compose to the 1.00 ceiling. Vindicates E8's max > mean in real 7B weights once
saturation is controlled.
Default (easy) task behaviour is unchanged (hard defaults False). +1 hard-task
test (131 green). Excludes the 0.5B smoke bundle (a pipeline gate, not a
deliverable). Results in results/llm_{moe,directed}_hard_hpc/ (parquet gitignored).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds the "directed sex" operator (E10) the moe regime-flip pointed to: don't
commit to one a-priori blend — breed a population of recombinant offspring
(specialists merged at Dirichlet-sampled weights), score each on a held-out
validation split with the verifier, and keep the fittest, reported on a fresh
test split. Two breeding objectives: best-overall and best-worst-family.
src/llm/directed.py + kind llm_directed, reusing the cached specialists.
Result — refinements pay off in proportion to how far the uniform soup is from
optimal:
- 0.5B (soup dilutes): directed selection beats soup on the bred objective —
directed_overall 0.69 > soup 0.64; directed_balanced worst-family 0.37 > 0.26.
Riders: single-objective selection trades off the other axis (overall-breed
tanks lists to 0.17); a global blend still trails per-input routing (0.74).
- 7B (Imperial CX3, soup already composes to ceiling on near-saturated families,
strings/arith 1.00): directed ~= soup (0.868 ~ 0.873, marginally below via a
val/test overfit gap) — no fitter offspring to breed.
Through-line across all four LLM runs: "merge, don't average" and its refinements
(routing, directed selection) are weak-base / suboptimal-default phenomena — they
help at 0.5B and are inert at 7B. Honest limitation kept in the writeup: the 7B
families are near-saturated, which caps the headroom; a harder unsaturated
benchmark is the fair next test.
Also folds in the two llm_moe local manifest/config files missed in 8da0dac.
+3 directed unit tests (130 green). Results in results/llm_directed{,_hpc}/
(parquet gitignored).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds the union-preserving recombination operator that llm_merge lacked (E8's max,
not mean): keep each specialist LoRA intact and SELECT the right one per prompt
(MoE router: oracle, or training-free nearest-centroid over base embeddings) or
per module (max_merge = winner-take-all by delta norm). src/llm/moe.py, kind
llm_moe, reuses the cached specialists.
Result — a clean regime boundary for "merge, don't average":
- 0.5B: union wins. Routing 0.74 / worst-family 0.43 > soup 0.64 / 0.26, with no
dilution (recovers each specialist's own-family peak). E8's max > mean in real
weights, because at a weak base averaging dilutes.
- 7B (Imperial CX3, L40S, 9 min): the ordering INVERTS. Fusion wins — soup 0.87 >
routing 0.84 > max_merge 0.78. Routing is capped at the best parent per family;
fusion blends and, given a capable base, COMPOSES beyond any parent (soup lists
0.62 > spec 0.57). Selection can't synthesise better than its best component;
averaging-that-composes can.
So "merge, don't average" (E4/E8) is a weak-parent / small-model law, not
universal: union wins under dilution, fusion wins under composition. Refines E8
(its additive-landscape max>mean assumed no compositional headroom). The operator
to want is fusion-that-composes + offspring selection = the directed-sex ideal
(E10) — the natural next experiment.
Honest riders: the learned router is trivially perfect (lexically-distinct
families), and router-free max_merge is the weakest union (not input-adaptive).
+2 router unit tests (127 green). Results in results/llm_moe{,_hpc}/ (parquet
gitignored per the reproducibility contract).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Re-ran the specialist-merge experiment at a capable base (Qwen2.5-7B-Instruct,
200 tests/family) on one L40S GPU of Imperial's CX3 HPC (8 min walltime). The
two caveats the 0.5B prototype left marginal are now resolved:
- "exceeds every parent overall" is clean: both merges 0.87 vs best specialist
0.77 (+10 pts), and above every specialist on every family.
- dilution vanishes: at 0.5B averaging diluted the lists-specialist
(0.43->0.26); at 7B the merge beats it (0.62>0.57). Dilution was a
small-model artefact -- a capable base composes rather than dilutes, which
softens E4's "merge, don't average" once the parents are strong.
The figure title is now data-driven (reports ">" for 7B, "~" for 0.5B).
Adds the hpc/ smoke job script and the llm_merge walltime trim. Results synced
to results/llm_merge_hpc/ (parquet gitignored per the reproducibility contract).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Enable running the LLM tier on Imperial's HPC (scheduler: PBS Pro / qsub).
- hpc/probe.pbs: 10-min 1-GPU reconnaissance job resolving the two unknowns
the RCS docs omit -- compute-node internet access and the L40S driver's
CUDA version -- plus TMPDIR/disk and available python/cuda modules.
- hpc/llm_merge.pbs: scaled run on an L40S (48 GB), offline HF-cache wired,
runs configs/llm/merge_hpc.yaml.
- configs/llm/merge_hpc.yaml: Qwen2.5-7B-Instruct (fits the L40S) to reduce
the noise that left the 0.5B prototype's overall-exceeds sign marginal.
- hpc/README.md: the git-based workflow (login-node uv env + model
pre-download -> qsub -> rsync results back), confirmed PBS/GPU directives,
and the code TODOs for the definitive run (multi-seed, more families,
directed/dilution-resistant merge).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
