Phase 3: LLM-tier speciation + multi-seed firm-up of the recombination claims
llm_speciation (new kind; src/llm/speciation.py): E13 in LLM weights. LoRA children share the frozen base's coordinates, so merge failure is functional by construction. CONFLICT (ambiguous sort prompts learned under opposite conventions — the BDM structure): function-specific hybrid breakdown — merged coherence 0.02-0.08 falls below BOTH parents (~0.2) on the conflicted function; and in the de-confounded `add` design (private budget fixed, conflict added on top; 3 seeds after a single-seed pilot showed one anomalous point) the merge's private-family accuracy shows NO trend with conflict — the damage is surgical, not global. DURATION (over-trained disjoint specialists, 1->12 epochs): the merge improves (0.84->0.94) and stays above the best parent — the MLP "no emergent isolation" null generalises; relevant to the expert-training-duration report (2607.11997), with the epistasis prediction left to the decisive experiment. Multi-seed firm-up (seeds threaded into specialist caches; `seeds:` list support in the runner; fixed test sets): all three recombination claims hold with CIs — merges beat every specialist (5 seeds, ties 0.647±0.027 > best spec 0.592±0.009; worst-family 0.28 vs <=0.16); union 0.274±0.026 > fusion 0.174±0.102 on hard (3 seeds); directed 0.221±0.026 > soup. NEW finding: fusion is seed-FRAGILE where headroom exists (CI ±0.10) while routing/directed selection are stable (±0.026) — the union/selection operators win on reliability, not just mean. Figures (llm_speciation 3-panel; llm_seeds 3-panel with 95% CI), READMEs, +1 convention test (150 green), make llm-speciation / llm-seeds targets. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkRLcc18rwT2Lysu6PbG7v
This commit is contained in:
parent
58e6c74609
commit
5a23ddaf2a
31 changed files with 956 additions and 11 deletions
|
|
@ -62,7 +62,9 @@ barriers). Patch before the preprint goes up.
|
|||
(no growth ⇒ "models are safer to merge than biology predicts" — an honest bound).
|
||||
- Tests + README + fold into figure. Pre-register the falsifier language before running.
|
||||
|
||||
## Phase 2 — arXiv preprint (priority stake) — end of Week 1
|
||||
## Phase 2 — arXiv preprint package (GG decision 2026-08-11: DO NOT POST until all experiments
|
||||
and analysis are complete — the preprint goes up after Phase 3, with the final analysis folded in;
|
||||
re-run md2tex + tectonic at that point)
|
||||
|
||||
- [x] Citation refresh in `paper/the-evolution-of-sex-for-ai.md` *(done 2026-08-11; author names verified against arXiv API)*: **new concessions** —
|
||||
First-Extinction Law (2509.20101), quantitative-trait collapse (2407.17493), verifier-injection
|
||||
|
|
@ -74,12 +76,18 @@ barriers). Patch before the preprint goes up.
|
|||
- [x] Fold E13b/c results into the speciation section (whatever they show — honestly). *(Done: full-symmetry residual + hybrid-fitness cliff + the emergent converse, in abstract, §5, §13 and the accessible version.)*
|
||||
- [x] LaTeX conversion: `paper/arxiv/` (md2tex.py block-based converter from the Markdown source of
|
||||
truth; main.tex; 3 figures; builds clean under tectonic, 20 pp; arXiv pdflatex hint guarded).
|
||||
- [ ] **Post to arXiv (GG action — package ready):** upload `paper/arxiv/{main.tex,body.tex,figs/}`;
|
||||
categories, license note, and a ≤1,920-char abstract are prepared in `paper/arxiv/ARXIV-SUBMISSION.md`.
|
||||
- [ ] **Post to arXiv — DEFERRED until all experiments/analysis are done (GG).** Package is ready
|
||||
(`paper/arxiv/`, instructions in `ARXIV-SUBMISSION.md`); rebuild after the Phase-3 results are
|
||||
folded into the manuscript, then upload.
|
||||
|
||||
## Phase 3 — New experiments for impact & robustness — Weeks 2–3
|
||||
|
||||
- [ ] **`llm_speciation` — the cliff at the LLM tier (new, highest-impact addition).**
|
||||
- [x] **`llm_speciation` — the cliff at the LLM tier.** *(Run 2026-08-11, 0.5B: DURATION NULL —
|
||||
over-trained disjoint specialists merge better not worse (0.84->0.94, above best parent throughout);
|
||||
the MLP "no emergent isolation" null generalises. CONFLICT — function-specific hybrid breakdown:
|
||||
merged coherence 0.02-0.08 falls below BOTH parents (~0.2) on the conflicted function. Caught a
|
||||
design confound (replace mode ties conflict_frac to private-data budget) -> built the de-confounded
|
||||
`add` variant (conflict_mode: add; configs/llm/speciation_add.yaml). 7B confirm optional later.)*
|
||||
Speciation demonstrated at all three tiers (analytic → MLP → LLM) makes the headline
|
||||
unassailable. Structural bonus to state in the paper: LoRA deltas live in the frozen base's
|
||||
coordinate system, so there is **no permutation ambiguity by construction** — any LoRA-merge
|
||||
|
|
@ -93,12 +101,46 @@ barriers). Patch before the preprint goes up.
|
|||
(2607.11997: optimal expert duration) as *our theory explaining their data* — a strong PNAS
|
||||
move.
|
||||
- 0.5B locally first; one 7B CX3 confirm if the sign is clean (`hpc/` PBS, minutes).
|
||||
- [ ] **Multi-seed the LLM arc (robustness — removes the "one seed" objection).**
|
||||
- [x] **Multi-seed the LLM arc (0.5B tier done 2026-08-11).** All three claims hold with CIs
|
||||
(merge>every specialist; union 0.274±0.026 > fusion 0.174±0.102 hard; directed 0.221±0.026 > soup)
|
||||
+ NEW finding: fusion is seed-FRAGILE on hard tasks (±0.10) while routing/directed are stable
|
||||
(±0.026) — the union/selection operators are the reliable ones. results/llm_*_seeds/ + llm_seeds
|
||||
figure. Remaining: 7B CX3 seeds (1-3) when HPC convenient.
|
||||
- Thread the seed into specialist cache keys (`spec_<family>[_hard]_s<seed>`); verify nothing
|
||||
else assumes the old names.
|
||||
- 0.5B: seeds 1–5 × {merge, moe, directed} × {easy, hard}. 7B on CX3: seeds 1–3 × hard
|
||||
{merge, moe, directed} (8–25 min walltimes → trivial). Aggregate figures with 95% CI; update
|
||||
READMEs; the headroom law now carries error bars.
|
||||
- [ ] **`epistasis_predicts` — the DECISIVE experiment (from the external review, 2026-08-11; highest
|
||||
priority after llm_speciation lands).** The review's exact bar: population-genetic quantities must
|
||||
*predict* (not re-describe) — forecast merge success **before merging**, and beat existing
|
||||
predictors. Design, reusing the llm_speciation machinery:
|
||||
1. Parents with independently controlled interaction structure: sweep `conflict_frac` (ground-truth
|
||||
epistasis) *and* compatible/disjoint + duration variants (spread in divergence WITHOUT conflict),
|
||||
so functional conflict and divergence are decorrelated by construction.
|
||||
2. Pre-merge predictors, none of which touches a merged model: (a) **operational epistasis** =
|
||||
functional-disagreement mass between the parents on a shared probe set (the μ(S) estimate — ours);
|
||||
(b) **gradient alignment** (the Zhou et al. 2601.22285 predictor); (c) **weight-space geometry**
|
||||
(LoRA-delta cosine / norm distance).
|
||||
3. Outcome: merged (soup) performance on private families + convention coherence, held-out test,
|
||||
multi-seed.
|
||||
4. The claim to test: at matched geometric divergence, the epistasis measure predicts merge outcome
|
||||
and the geometry measures do not (R² comparison + an operator-choice decision test — merge vs
|
||||
route — under matched budgets).
|
||||
Pre-registered falsifier: if gradient/geometry predictors match the epistasis measure, the paper's
|
||||
"epistasis, not divergence, sets the cliff" claim stays analytic-only and is labelled as such.
|
||||
- [x] **Manuscript claim-narrowing (external review, 2026-08-11) — done.** Softened identity claims
|
||||
(WF exact only in the minimal model + learning-kernel cited against ourselves; ratchet scoped to the
|
||||
irreversible arm), removed "nobody has / none imports / theory outrun" (priority-dispute bait),
|
||||
added the interpretation/explanation/prediction ladder to §1, stated the merge-don't-average
|
||||
operator boundaries (output-mean vs weight-avg vs routing vs max-with-oracle, budgets, oracle,
|
||||
capacity), added a "what these experiments do and do not establish" scope block to the speciation
|
||||
section (impossibility floor is information-theoretic, not genetic; snowball/epistasis-cliff =
|
||||
hypotheses at the neural tier), replaced "control theory" with "framework" (subtitle included —
|
||||
GG can veto), fixed the §3/§11 overstatements (frozen core ≠ frozen behaviour; Baldwin = echo not
|
||||
identity; archive vs operational irreversibility), added the **claims-at-a-glance table**
|
||||
(status/assumptions/evidence/limits) to §13 + table support in md2tex, and matched the calibration
|
||||
in the accessible version. Adopted the review's framing sentence as the stated core contribution.
|
||||
- [ ] **(Optional, SI-grade) ambiguous-families router stress test** — overlapping-skill families
|
||||
where the router is no longer trivially perfect; documents the honest limit of union-by-routing.
|
||||
Do only if time permits; otherwise keep the existing rider sentence.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue