hard benchmark: the 7B "fusion wins / no headroom" results were saturation artefacts
The easy task families saturated 7B (strings & arith at 1.00), so the earlier
7B nulls — moe: fusion 0.87 > union 0.84; directed ~= soup — could not separate
"refinements don't help at scale" from "tasks too easy at 7B". Adds a hard task
variant (hard: true in tasks.py: multi-step lists, Caesar ciphers / letter
transforms, multi-step & larger arithmetic; same family labels and answer
formats, threaded through make_tasks/train_specialist/runners; hard specialists
cache separately as spec_*_hard) and re-runs both experiments at 7B on Imperial
CX3 (one L40S, 24 min, unsaturated: arith ~0.48, strings 0.67, lists 0.34).
Both nulls flip back to the 0.5B ordering:
- Union beats fusion again: routing 0.500 > fusion 0.40 (soup 0.392 / ties
0.400), the same 10-pt margin as 0.5B. Fusion dilutes the fragile strings
specialist so hard (0.665 -> soup 0.300) that soup even trails the best single
specialist (0.425); routing keeps it intact (0.670).
- Directed selection beats soup again: 0.492 > 0.392 (+10 pts), recovering most
of routing's benefit from one deployable merged model (lifts strings to 0.630).
Correction to the earlier interpretation: the llm_moe_hpc "regime flip" and the
llm_directed_hpc "no headroom" null were driven by TASK SATURATION, not base
capability. The operative variable is headroom — "merge, don't average" (union >
fusion) and "directed sex" (selection > single blend) hold whenever there is room
to lose to dilution: a weak base (0.5B) OR hard tasks at a strong base (7B-hard).
Fusion only wins in the degenerate corner where easy tasks let a strong base
compose to the 1.00 ceiling. Vindicates E8's max > mean in real 7B weights once
saturation is controlled.
Default (easy) task behaviour is unchanged (hard defaults False). +1 hard-task
test (131 green). Excludes the 0.5B smoke bundle (a pipeline gate, not a
deliverable). Results in results/llm_{moe,directed}_hard_hpc/ (parquet gitignored).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
parent
e433e48860
commit
39f6c9f4df
20 changed files with 495 additions and 69 deletions
26
results/llm_directed_hard_hpc/manifest.json
Normal file
26
results/llm_directed_hard_hpc/manifest.json
Normal file
|
|
@ -0,0 +1,26 @@
|
|||
{
|
||||
"experiment": "llm_directed_hard_hpc",
|
||||
"master_seed": 1,
|
||||
"git_commit": null,
|
||||
"python": "3.11.13",
|
||||
"libraries": {
|
||||
"numpy": "2.4.6",
|
||||
"scipy": "1.17.1",
|
||||
"pandas": "3.0.3",
|
||||
"pyarrow": "24.0.0",
|
||||
"torch": "2.12.1",
|
||||
"transformers": "5.13.0",
|
||||
"peft": "0.19.1"
|
||||
},
|
||||
"rows": 35,
|
||||
"results_sha256": "a5e45785a8701c5e1cdcfc64bc1482dffe4e68f8c6978e9d07dc8f04d40fbab6",
|
||||
"layer": "2",
|
||||
"tier": "llm",
|
||||
"base_model": "Qwen/Qwen2.5-7B-Instruct",
|
||||
"hard": true,
|
||||
"directed": {
|
||||
"n_candidates": 24,
|
||||
"concentration": 0.5,
|
||||
"n_val": 100
|
||||
}
|
||||
}
|
||||
Loading…
Add table
Add a link
Reference in a new issue