Giorgio Gilestro
e433e48860
Adds the "directed sex" operator (E10) the moe regime-flip pointed to: don't
commit to one a-priori blend — breed a population of recombinant offspring
(specialists merged at Dirichlet-sampled weights), score each on a held-out
validation split with the verifier, and keep the fittest, reported on a fresh
test split. Two breeding objectives: best-overall and best-worst-family.
src/llm/directed.py + kind llm_directed, reusing the cached specialists.
Result — refinements pay off in proportion to how far the uniform soup is from
optimal:
- 0.5B (soup dilutes): directed selection beats soup on the bred objective —
directed_overall 0.69 > soup 0.64; directed_balanced worst-family 0.37 > 0.26.
Riders: single-objective selection trades off the other axis (overall-breed
tanks lists to 0.17); a global blend still trails per-input routing (0.74).
- 7B (Imperial CX3, soup already composes to ceiling on near-saturated families,
strings/arith 1.00): directed ~= soup (0.868 ~ 0.873, marginally below via a
val/test overfit gap) — no fitter offspring to breed.
Through-line across all four LLM runs: "merge, don't average" and its refinements
(routing, directed selection) are weak-base / suboptimal-default phenomena — they
help at 0.5B and are inert at 7B. Honest limitation kept in the writeup: the 7B
families are near-saturated, which caps the headroom; a harder unsaturated
benchmark is the fair next test.
Also folds in the two llm_moe local manifest/config files missed in
|
||
|---|---|---|
| .. | ||
| llm_directed.pdf | ||
| llm_directed.png | ||
| manifest.json | ||
| README.md | ||
| resolved_config.yaml | ||
llm_directed_hpc — directed sex at scale (7B): no headroom once the soup already composes
Claim tested. At 0.5B, breeding offspring and selecting on the verifier beat the uniform soup
(llm_directed: directed 0.69 > soup 0.64). But llm_merge_hpc showed the 7B soup already composes
to 0.87, beating every specialist. So the honest question: does searching the recombination-weight
space + selecting on the verifier find anything better than that strong default — or is there simply no
headroom left? Run on one L40S (46 GB) GPU of Imperial's CX3 HPC, 9 min walltime, 24 offspring,
reusing the cached 7B specialists.
Setup. Base Qwen2.5-7B-Instruct, cached 7B specialists, 24 offspring (Dirichlet-weighted merges, concentration 0.5, candidate 0 = uniform soup), scored on a 100-task/family validation split, winners reported on a fresh 200-task/family test split. Seed 1.
Results (test accuracy)
| model | lists | strings | arith | overall | worst-family |
|---|---|---|---|---|---|
| best specialist (lists) | 0.57 | 0.74 | 1.00 | 0.77 | 0.57 |
| merge_soup (uniform, candidate 0) | 0.62 | 1.00 | 1.00 | 0.873 | 0.625 |
| directed_overall (bred for overall) | 0.62 | 1.00 | 1.00 | 0.868 | 0.615 |
| directed_balanced (bred for worst-family) | 0.62 | 1.00 | 1.00 | 0.868 | 0.615 |
The finding: directed selection is inert once the default is already near-optimal
- Directed ≈ soup, and marginally below it (0.868 vs 0.873). Both breeding objectives converged to a near-soup blend, and the validation-selected winner is a hair worse than the plain uniform soup on the held-out test set — a small val/test generalisation gap (selection overfits the 100-task/ family validation split). Searching 24 offspring found nothing that beats candidate 0.
- Why: no headroom. The 7B soup already composes to the ceiling on these families — strings and arith are saturated at 1.00, and lists (0.62) is the only slack, itself already above every specialist. When the default blend is already optimal, there is no fitter offspring to breed, so selection can only match it (or lose slightly to val noise).
- Honest limitation. These task families are near-saturated at 7B (2 of 3 at 1.00), which structurally caps the headroom any recombination refinement could exploit. A fair test of directed sex at scale needs a harder, unsaturated benchmark where the optimal blend is non-trivial — this run cannot distinguish "directed sex doesn't help at scale" from "these tasks are too easy at 7B."
Takeaway — the through-line across all four LLM runs
The value of every recombination refinement (routing, directed selection) scales with how far the default uniform soup is from optimal:
- 0.5B — soup dilutes (far from optimal): routing beats soup (0.74 > 0.64), directed selection beats soup (0.69 > 0.64). Refinements pay off.
- 7B — soup composes to near-ceiling on saturated tasks: routing < soup (0.84 < 0.87), directed ≈ soup (0.868 ≈ 0.873). No headroom; refinements are inert.
So "merge, don't average" and its refinements are weak-base / suboptimal-default phenomena. The
open question a capable base leaves is whether directed sex helps on hard, unsaturated tasks at scale
— the natural next benchmark. Falsifier for this run (triggered as a null, reported honestly):
directed offspring failed to exceed the uniform soup at 7B; here they tied/marginally trailed it
because the soup was already optimal on near-saturated families. Provenance in manifest.json
(L40S, torch 2.12.1 / transformers 5.13.0 / peft 0.19.1; git_commit: null — rsync'd node copy).