Giorgio Gilestro
39f6c9f4df
The easy task families saturated 7B (strings & arith at 1.00), so the earlier
7B nulls — moe: fusion 0.87 > union 0.84; directed ~= soup — could not separate
"refinements don't help at scale" from "tasks too easy at 7B". Adds a hard task
variant (hard: true in tasks.py: multi-step lists, Caesar ciphers / letter
transforms, multi-step & larger arithmetic; same family labels and answer
formats, threaded through make_tasks/train_specialist/runners; hard specialists
cache separately as spec_*_hard) and re-runs both experiments at 7B on Imperial
CX3 (one L40S, 24 min, unsaturated: arith ~0.48, strings 0.67, lists 0.34).
Both nulls flip back to the 0.5B ordering:
- Union beats fusion again: routing 0.500 > fusion 0.40 (soup 0.392 / ties
0.400), the same 10-pt margin as 0.5B. Fusion dilutes the fragile strings
specialist so hard (0.665 -> soup 0.300) that soup even trails the best single
specialist (0.425); routing keeps it intact (0.670).
- Directed selection beats soup again: 0.492 > 0.392 (+10 pts), recovering most
of routing's benefit from one deployable merged model (lifts strings to 0.630).
Correction to the earlier interpretation: the llm_moe_hpc "regime flip" and the
llm_directed_hpc "no headroom" null were driven by TASK SATURATION, not base
capability. The operative variable is headroom — "merge, don't average" (union >
fusion) and "directed sex" (selection > single blend) hold whenever there is room
to lose to dilution: a weak base (0.5B) OR hard tasks at a strong base (7B-hard).
Fusion only wins in the degenerate corner where easy tasks let a strong base
compose to the 1.00 ceiling. Vindicates E8's max > mean in real 7B weights once
saturation is controlled.
Default (easy) task behaviour is unchanged (hard defaults False). +1 hard-task
test (131 green). Excludes the 0.5B smoke bundle (a pipeline gate, not a
deliverable). Results in results/llm_{moe,directed}_hard_hpc/ (parquet gitignored).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
21 lines
691 B
YAML
21 lines
691 B
YAML
experiment: llm_moe_hard
|
|
kind: llm_moe
|
|
seed: 1
|
|
n_replicates: 1
|
|
|
|
# Local 0.5B SMOKE for the hard benchmark: confirms the harder task variant (hard: true) runs
|
|
# end-to-end — trains fresh hard specialists (cached as spec_*_hard), the verifier still scores them,
|
|
# routing/fusion operators execute. 0.5B will score low on these harder tasks (that is expected; the
|
|
# calibrated comparison is the 7B moe_hard_hpc run). Kept small for speed.
|
|
|
|
base_model: Qwen/Qwen2.5-0.5B-Instruct
|
|
hard: true
|
|
families: [lists, strings, arith]
|
|
n_train: 400
|
|
n_test: 80
|
|
n_route: 32
|
|
epochs: 3
|
|
lora: {r: 16, alpha: 32}
|
|
operators: [soup, ties, moe_oracle, moe_learned, max_merge]
|
|
|
|
output: {dir: results/llm_moe_hard}
|