Giorgio Gilestro
a40ace1821
Analyses (figures/stats_llm_epistasis.py, committed + reproducible): condition-clustered bootstrap CIs (functional measures exclude zero: dis_raw [+0.04,+0.69], conf-weighted [+0.02,+0.68]; gradient alignment [-0.59,-0.06]; geometry straddles zero), PAIRED predictor contrasts (not individually significant — stated), leave-one-condition-out held-out prediction (functional replicates, geometry ~0, performance baseline unstable), three outcome references (ordering sensitive to reference — reported, with the mechanism), between/within-axis decomposition (within-conflict identification impossible by design; the compat axis identifies), and seed-level paired reliability (routing/directed beat soup 3/3 seeds incl. one catastrophic soup failure; CI-width fragility claim withdrawn). Renames and corrections: "decisive experiment" -> "controlled predictive test"; "operational epistasis" -> "confidence-weighted functional conflict (proposed proxy)"; "functional by construction" -> "controls a major source of coordinate mismatch / conflict-associated" (module, configs, READMEs, figures); SI proposition's "chord" defined precisely (endpoint-loss interpolation, invariant) vs the path (not invariant) + no-global-optimality caveat (removable = lower bound, residual = upper); snowball count != performance cliff distinction added; claims table gains four rows (grid finding / weighting NOT supported / functional-vs- all-geometry not established / operator choice open); §1 ladder states the prediction rung as a bounded small-model result. paper/response-to-review-2.md: point-by-point, opening with the bookkeeping correction (E13b/c were in the reviewed draft — revised interpretation, not new results). READMEs rewritten around the four analyses with the chronology (prospective/adaptive/post-hoc) disclosed. 151 tests green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkRLcc18rwT2Lysu6PbG7v
2.6 KiB
Multi-seed LLM recombination (0.5B) — the claims with error bars
PNAS work-order Phase 3: removes the "one seed" objection on the three LLM recombination claims.
Protocol: test sets fixed (seed 1000+i per family), training seed varied (specialists cache
per-seed as spec_<family>[_hard]_s<seed>), so across-seed variance is training variance only.
Figure: llm_seeds.png (this dir) aggregates all three experiments, 95% CI over seeds.
(A) Fisher–Muller, easy benchmark, 5 seeds (llm_merge_seeds)
| model | overall | worst-family |
|---|---|---|
| merge_ties | 0.647 ± 0.027 | 0.282 ± 0.020 |
| merge_soup | 0.632 ± 0.042 | 0.278 ± 0.028 |
| best specialist (strings) | 0.592 ± 0.009 | 0.078 ± 0.011 |
| base | 0.277 | 0.150 |
Both merges beat every specialist overall (ties: non-overlapping CIs; soup: marginal at 0.5B, as in the single-seed run — decisive at 7B) and the worst-family signature is unambiguous: merges ≈0.28 vs ≤0.16 for any parent — only recombined models are competent everywhere.
(B) Union vs fusion, hard benchmark, 3 seeds (llm_moe_hard_seeds/)
Routing (union) beats fusion in every seed (3/3 paired, both metrics; e.g. overall per seed: routing 0.258/0.300/0.262 vs soup 0.212/0.238/0.071). The paired per-seed values also show why: seed 3's soup failed catastrophically (0.071 overall, 0.000 worst-family) while routing was unaffected (0.262/0.225). Seed-level sd: soup 0.090 vs routing 0.023 — suggestive of a genuine variance difference, but with 3 seeds this is an observation, not an estimate; the robust statement is the 3/3 paired ordering plus the observed catastrophic-failure mode that routing is structurally immune to. (Learned router still = oracle: lexically distinct families, known rider.)
(C) Directed offspring selection, hard, 3 seeds (llm_directed_hard_seeds/)
directed_overall beats the a-priori soup in every seed (3/3 paired; 0.225/0.242/0.196 vs 0.212/0.238/0.071 — including rescuing soup's catastrophic seed); directed_balanced worst-family 0.158 vs soup 0.088. Directed selection both beats and stabilises the blend (same 3-seed caveat as panel B); per-input routing remains above any single global blend, as before.
Read together: all three recombination claims hold under seed replication with consistent paired ordering (route > directed-select > soup in every seed on headroom tasks), and the per-seed values surface a failure mode — occasional catastrophic soup merges — that the union/selection operators avoided in every observed case. Variance estimates await more seeds. Base: Qwen2.5-0.5B-Instruct; statistical (per-seed) reproducibility per blueprint §4.