Giorgio Gilestro
39f6c9f4df
The easy task families saturated 7B (strings & arith at 1.00), so the earlier
7B nulls — moe: fusion 0.87 > union 0.84; directed ~= soup — could not separate
"refinements don't help at scale" from "tasks too easy at 7B". Adds a hard task
variant (hard: true in tasks.py: multi-step lists, Caesar ciphers / letter
transforms, multi-step & larger arithmetic; same family labels and answer
formats, threaded through make_tasks/train_specialist/runners; hard specialists
cache separately as spec_*_hard) and re-runs both experiments at 7B on Imperial
CX3 (one L40S, 24 min, unsaturated: arith ~0.48, strings 0.67, lists 0.34).
Both nulls flip back to the 0.5B ordering:
- Union beats fusion again: routing 0.500 > fusion 0.40 (soup 0.392 / ties
0.400), the same 10-pt margin as 0.5B. Fusion dilutes the fragile strings
specialist so hard (0.665 -> soup 0.300) that soup even trails the best single
specialist (0.425); routing keeps it intact (0.670).
- Directed selection beats soup again: 0.492 > 0.392 (+10 pts), recovering most
of routing's benefit from one deployable merged model (lifts strings to 0.630).
Correction to the earlier interpretation: the llm_moe_hpc "regime flip" and the
llm_directed_hpc "no headroom" null were driven by TASK SATURATION, not base
capability. The operative variable is headroom — "merge, don't average" (union >
fusion) and "directed sex" (selection > single blend) hold whenever there is room
to lose to dilution: a weak base (0.5B) OR hard tasks at a strong base (7B-hard).
Fusion only wins in the degenerate corner where easy tasks let a strong base
compose to the 1.00 ceiling. Vindicates E8's max > mean in real 7B weights once
saturation is controlled.
Default (easy) task behaviour is unchanged (hard defaults False). +1 hard-task
test (131 green). Excludes the 0.5B smoke bundle (a pipeline gate, not a
deliverable). Results in results/llm_{moe,directed}_hard_hpc/ (parquet gitignored).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|---|---|---|
| .. | ||
| llm_directed.pdf | ||
| llm_directed.png | ||
| manifest.json | ||
| README.md | ||
| resolved_config.yaml | ||
llm_directed_hard_hpc — directed sex on HARD (unsaturated) tasks at 7B: offspring selection helps again
Claim tested. llm_directed_hpc found directed selection ≈ soup at 7B (0.868 ≈ 0.873) and flagged
the honest caveat that the easy families were saturated (no headroom for selection to exploit). This
run re-runs directed sex on the hard task variant, where the uniform soup is far from the ceiling —
the regime in which breeding + selection can actually improve on the default blend. One L40S (46 GB)
GPU, Imperial CX3; reuses the spec_*_hard specialists trained by llm_moe_hard_hpc.
Setup. Base Qwen2.5-7B-Instruct, hard: true, 24 offspring (Dirichlet-weighted merges),
100-task/family validation split (selection), 200-task/family test split (report). Seed 1.
Results (test accuracy — unsaturated)
| model | lists | strings | arith | overall | worst-family |
|---|---|---|---|---|---|
| best specialist (strings) | 0.155 | 0.665 | 0.455 | 0.425 | 0.155 |
| merge_soup (uniform, candidate 0) | 0.390 | 0.300 | 0.485 | 0.392 | 0.300 |
| directed_overall | 0.380 | 0.630 | 0.465 | 0.492 | 0.380 |
| directed_balanced | 0.380 | 0.630 | 0.465 | 0.492 | 0.380 |
The finding: the 7B "no headroom" null was also saturation
- Directed selection beats the uniform soup by +10 points (0.492 > 0.392). On hard tasks there is
a better blend than uniform averaging, and breeding 24 offspring + selecting on the verifier finds it
— recovering most of the routing-level performance (0.492 vs routing 0.500) from a single deployable
merged model. The
llm_directed_hpcnull (directed ≈ soup at 7B) was a saturation artefact, exactly as that run's honest caveat predicted. - Selection repairs fusion's dilution. The winning offspring lifts strings from soup's diluted 0.300 back to 0.630 (near the 0.665 specialist) while keeping lists' composition gain (0.380) — i.e. it finds a blend that composes where composition helps and avoids diluting the fragile skill. Both breeding objectives converged to the same winner (overall = balanced), which also improves worst-family (0.380 > soup 0.300).
- Directed ≈ routing here. A single searched-and-selected merged model (0.492) matches the per-input router (0.500) on hard tasks — offspring selection buys most of routing's benefit without needing a router at inference.
Takeaway
On unsaturated tasks, directed sex (breed offspring + select on the verifier) beats the single a-priori
soup at 7B, resolving the earlier null: it was task saturation, not scale, that made selection inert.
Together with llm_moe_hard_hpc this completes the correction — both "merge, don't average" (union
fusion) and "directed sex" (selection > single blend) are headroom phenomena that hold at 7B once the tasks are hard enough to leave room, not weak-base-only effects. Falsifier (not triggered): directed offspring ≤ uniform soup — instead they beat it by 10 points. Provenance in
manifest.json(hard: true, L40S, torch 2.12.1 / transformers 5.13.0 / peft 0.19.1).