MachineSex/results/llm_directed_hard_hpc/README.md

llm_directed_hard_hpc — directed sex on HARD (unsaturated) tasks at 7B: offspring selection helps again

Claim tested. llm_directed_hpc found directed selection ≈ soup at 7B (0.868 ≈ 0.873) and flagged the honest caveat that the easy families were saturated (no headroom for selection to exploit). This run re-runs directed sex on the hard task variant, where the uniform soup is far from the ceiling — the regime in which breeding + selection can actually improve on the default blend. One L40S (46 GB) GPU, Imperial CX3; reuses the spec_*_hard specialists trained by llm_moe_hard_hpc.

Setup. Base Qwen2.5-7B-Instruct, hard: true, 24 offspring (Dirichlet-weighted merges), 100-task/family validation split (selection), 200-task/family test split (report). Seed 1.

Results (test accuracy — unsaturated)

model	lists	strings	arith	overall	worst-family
best specialist (strings)	0.155	0.665	0.455	0.425	0.155
merge_soup (uniform, candidate 0)	0.390	0.300	0.485	0.392	0.300
directed_overall	0.380	0.630	0.465	0.492	0.380
directed_balanced	0.380	0.630	0.465	0.492	0.380

The finding: the 7B "no headroom" null was also saturation

• Directed selection beats the uniform soup by +10 points (0.492 > 0.392). On hard tasks there is a better blend than uniform averaging, and breeding 24 offspring + selecting on the verifier finds it — recovering most of the routing-level performance (0.492 vs routing 0.500) from a single deployable merged model. The llm_directed_hpc null (directed ≈ soup at 7B) was a saturation artefact, exactly as that run's honest caveat predicted.
• Selection repairs fusion's dilution. The winning offspring lifts strings from soup's diluted 0.300 back to 0.630 (near the 0.665 specialist) while keeping lists' composition gain (0.380) — i.e. it finds a blend that composes where composition helps and avoids diluting the fragile skill. Both breeding objectives converged to the same winner (overall = balanced), which also improves worst-family (0.380 > soup 0.300).
• Directed ≈ routing here. A single searched-and-selected merged model (0.492) matches the per-input router (0.500) on hard tasks — offspring selection buys most of routing's benefit without needing a router at inference.

Takeaway

On unsaturated tasks, directed sex (breed offspring + select on the verifier) beats the single a-priori soup at 7B, resolving the earlier null: it was task saturation, not scale, that made selection inert. Together with llm_moe_hard_hpc this completes the correction — both "merge, don't average" (union

fusion) and "directed sex" (selection > single blend) are headroom phenomena that hold at 7B once the tasks are hard enough to leave room, not weak-base-only effects. Falsifier (not triggered): directed offspring ≤ uniform soup — instead they beat it by 10 points. Provenance in manifest.json (hard: true, L40S, torch 2.12.1 / transformers 5.13.0 / peft 0.19.1).

Seeds 1–3 (2026-09-11)

Seeds 2–3 were run on CX3 via hpc/llm_7b_seeds.pbs (seed 1 above was moved to s1/; the bundle layout is now s{seed}/). Fixed test sets, training seed varied. Per-seed values and mean ± 95% CI from figures/stats_llm_7b_seeds.py:

model       metric  n_seeds    s1    s2    s3  mean  ci95
       merge_soup      overall        3 0.392 0.405 0.428 0.408 0.021
       merge_soup worst_family        3 0.300 0.345 0.340 0.328 0.028
 directed_overall      overall        3 0.492 0.480 0.473 0.482 0.011
 directed_overall worst_family        3 0.380 0.405 0.430 0.405 0.028
directed_balanced      overall        3 0.492 0.480 0.473 0.482 0.011
directed_balanced worst_family        3 0.380 0.405 0.430 0.405 0.028

                      contrast       metric  n_seeds   s1    s2    s3  mean  ci95 sign_agrees
 directed_overall − merge_soup      overall        3 0.10 0.075 0.045 0.073 0.031         3/3
 directed_overall − merge_soup worst_family        3 0.08 0.060 0.090 0.077 0.017         3/3
directed_balanced − merge_soup      overall        3 0.10 0.075 0.045 0.073 0.031         3/3
directed_balanced − merge_soup worst_family        3 0.08 0.060 0.090 0.077 0.017         3/3

Reading: directed selection beats the a-priori soup in every seed (+0.073 ± 0.031 overall, +0.077 ± 0.017 worst-family). The two objectives selected the same offspring in all three seeds.