Giorgio Gilestro
39f6c9f4df
The easy task families saturated 7B (strings & arith at 1.00), so the earlier
7B nulls — moe: fusion 0.87 > union 0.84; directed ~= soup — could not separate
"refinements don't help at scale" from "tasks too easy at 7B". Adds a hard task
variant (hard: true in tasks.py: multi-step lists, Caesar ciphers / letter
transforms, multi-step & larger arithmetic; same family labels and answer
formats, threaded through make_tasks/train_specialist/runners; hard specialists
cache separately as spec_*_hard) and re-runs both experiments at 7B on Imperial
CX3 (one L40S, 24 min, unsaturated: arith ~0.48, strings 0.67, lists 0.34).
Both nulls flip back to the 0.5B ordering:
- Union beats fusion again: routing 0.500 > fusion 0.40 (soup 0.392 / ties
0.400), the same 10-pt margin as 0.5B. Fusion dilutes the fragile strings
specialist so hard (0.665 -> soup 0.300) that soup even trails the best single
specialist (0.425); routing keeps it intact (0.670).
- Directed selection beats soup again: 0.492 > 0.392 (+10 pts), recovering most
of routing's benefit from one deployable merged model (lifts strings to 0.630).
Correction to the earlier interpretation: the llm_moe_hpc "regime flip" and the
llm_directed_hpc "no headroom" null were driven by TASK SATURATION, not base
capability. The operative variable is headroom — "merge, don't average" (union >
fusion) and "directed sex" (selection > single blend) hold whenever there is room
to lose to dilution: a weak base (0.5B) OR hard tasks at a strong base (7B-hard).
Fusion only wins in the degenerate corner where easy tasks let a strong base
compose to the 1.00 ceiling. Vindicates E8's max > mean in real 7B weights once
saturation is controlled.
Default (easy) task behaviour is unchanged (hard defaults False). +1 hard-task
test (131 green). Excludes the 0.5B smoke bundle (a pipeline gate, not a
deliverable). Results in results/llm_{moe,directed}_hard_hpc/ (parquet gitignored).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
23 lines
809 B
YAML
23 lines
809 B
YAML
experiment: llm_directed_hard_hpc
|
|
kind: llm_directed
|
|
seed: 1
|
|
n_replicates: 1
|
|
|
|
# Directed sex on the HARD benchmark (L40S). On the easy families the 7B soup already composed to the
|
|
# ceiling, so offspring selection had no headroom (directed ≈ soup). This run uses the harder task
|
|
# variant (hard: true), where the uniform soup should be well below saturation — the regime in which
|
|
# breeding + selection can actually improve on the default blend. Reuses the spec_*_hard specialists
|
|
# trained by moe_hard_hpc if present. The fair test the easy 7B null could not provide.
|
|
|
|
base_model: Qwen/Qwen2.5-7B-Instruct
|
|
hard: true
|
|
families: [lists, strings, arith]
|
|
n_train: 800
|
|
n_val: 100
|
|
n_test: 200
|
|
n_candidates: 24
|
|
concentration: 0.5
|
|
epochs: 3
|
|
lora: {r: 16, alpha: 32}
|
|
|
|
output: {dir: results/llm_directed_hard_hpc}
|