Giorgio Gilestro
5a23ddaf2a
llm_speciation (new kind; src/llm/speciation.py): E13 in LLM weights. LoRA children share the frozen base's coordinates, so merge failure is functional by construction. CONFLICT (ambiguous sort prompts learned under opposite conventions — the BDM structure): function-specific hybrid breakdown — merged coherence 0.02-0.08 falls below BOTH parents (~0.2) on the conflicted function; and in the de-confounded `add` design (private budget fixed, conflict added on top; 3 seeds after a single-seed pilot showed one anomalous point) the merge's private-family accuracy shows NO trend with conflict — the damage is surgical, not global. DURATION (over-trained disjoint specialists, 1->12 epochs): the merge improves (0.84->0.94) and stays above the best parent — the MLP "no emergent isolation" null generalises; relevant to the expert-training-duration report (2607.11997), with the epistasis prediction left to the decisive experiment. Multi-seed firm-up (seeds threaded into specialist caches; `seeds:` list support in the runner; fixed test sets): all three recombination claims hold with CIs — merges beat every specialist (5 seeds, ties 0.647±0.027 > best spec 0.592±0.009; worst-family 0.28 vs <=0.16); union 0.274±0.026 > fusion 0.174±0.102 on hard (3 seeds); directed 0.221±0.026 > soup. NEW finding: fusion is seed-FRAGILE where headroom exists (CI ±0.10) while routing/directed selection are stable (±0.026) — the union/selection operators win on reliability, not just mean. Figures (llm_speciation 3-panel; llm_seeds 3-panel with 95% CI), READMEs, +1 convention test (150 green), make llm-speciation / llm-seeds targets. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkRLcc18rwT2Lysu6PbG7v
206 lines
16 KiB
Markdown
206 lines
16 KiB
Markdown
# Work order — PNAS submission: "The Evolution of Sex for Artificial Intelligence"
|
||
|
||
*Created 2026-08-11. Target: PNAS Direct Submission research article (~6 pp main + SI Appendix),
|
||
preceded by an arXiv preprint (q-bio.PE × cs.LG). Decision basis: 2026-08-11 literature + venue scan
|
||
(see memory/competitive-landscape.md, August addendum). GG approved PNAS + the analysis.*
|
||
|
||
**Governing constraints.** (1) Dual audience: every claim stated so a computer scientist and a
|
||
biologist can each follow it — keep the "two statements of the same fact" device and the
|
||
dictionary table; define every term at first use. (2) Honesty ledger: concede Riis +
|
||
First-Extinction + quantitative-trait collapse up front; claims are the *cure* and its theory, not
|
||
the diagnosis. (3) Reproducibility is the differentiator: every figure from committed artifacts,
|
||
code+data DOI at submission.
|
||
|
||
---
|
||
|
||
## Phase 1 — Referee-proof the headline (E13 hardening) — Week 1
|
||
|
||
The speciation result (E12+E13) is both our most novel and most exposed claim
|
||
(arXiv:2606.23607, June 2026: a symmetry group larger than permutations removes most transformer
|
||
barriers). Patch before the preprint goes up.
|
||
|
||
- [x] **E13c — scale-aware alignment (the symmetry defense).** *(Done 2026-08-11: canonicalise_scale
|
||
+ gated tests; decomposition & cliff regenerated — conflict residual 0.502→0.497 under the full
|
||
group, independent 0.001; floor proposition drafted in paper/si-notes.md S1. Hybrid-fitness
|
||
readout added: cliff accuracy 0.97→0.03.)*
|
||
- Extend `neural/rebasin.py` with per-unit **positive rescaling** canonicalisation: for the
|
||
no-BatchNorm ReLU MLP, the full function-preserving unit symmetry group is permutation ∘
|
||
positive diagonal scaling. Canonicalise both nets first (rescale each hidden unit to
|
||
‖in-weights‖=1, pushing the norm into the out-weights), *then* permutation-match. Sanity gate:
|
||
a randomly permuted **and rescaled** copy must realign to functional identity (extend
|
||
`tests/test_rebasin.py`).
|
||
- Re-run `speciation_real` + `speciation_real_cliff` reporting `residual_perm` vs
|
||
`residual_perm_scale`. Expected: independent-init residual stays ≈0; conflict residual
|
||
unchanged. Update figure/README.
|
||
- Write the **theoretical floor proposition** (SI): two models with low loss on *contradictory*
|
||
label assignments over the same inputs cannot both be matched by any function-preserving
|
||
transformation — any single merged function errs on at least the disagreement mass, so the
|
||
conflict-condition barrier has an information-theoretic floor independent of the symmetry
|
||
group. Cite 2606.23607 and position: their result strengthens ours (the *removable* part may
|
||
grow with richer groups; the *functional* residual cannot vanish). Also cite the two June-2026
|
||
LMC papers found in the direct arXiv sweep (2026-08-11): "Beyond Structural Symmetries: Linear
|
||
Mode Connectivity via Neuron Identifiability" (2606.03…) and "Functional Equivalence in
|
||
Attention … Applications to Linear Mode Connectivity" (2606.16…) — same objection family, same
|
||
defense. Direct arXiv API sweep also re-confirmed: speciation/DMI/Fisher–Muller/ratchet/
|
||
evolution-of-sex/mating-for-models have zero ML hits on arXiv itself as of the newest listings.
|
||
- [x] **E13b — emergent speciation (the decisive missing experiment).** *(Done 2026-08-11: the
|
||
pre-registered SECOND reading — residual 0.000 at every t_div ≤ 3200 in both `disjoint` and
|
||
`augment`; the merge RESCUES the forgetting specialists (parents 0.535/0.474 → merged 0.955,
|
||
sustained Fisher–Muller). Speciation requires functional conflict in this regime; LLM-scale
|
||
over-specialisation deferred to Phase-3 llm_speciation. Figure panel C + READMEs updated.)* Current E13 *imposes*
|
||
contradiction; a true Dobzhansky–Muller incompatibility is *emergent* (each change harmless
|
||
alone, bad only in combination). Design (`speciation_real` new conditions, CPU/torch, cheap):
|
||
- `disjoint`: fork from the shared MNIST base; child A continues on classes 0–4 only, child B on
|
||
5–9 only (no contradiction anywhere). Sweep divergence time `t_div`. Measure naive barrier and
|
||
scale+perm-aligned residual on the full task, **and merged-model accuracy vs best parent**.
|
||
- `conventions` (optional, purer DMI): same task, different augmentation conventions (A:
|
||
rotations, B: inversions) — representational drift with zero output conflict.
|
||
- **The money curve:** merged accuracy vs `t_div` should trace E12's *compatible →
|
||
outbreeding-depression → inviability* trajectory emergently: at low divergence the merge
|
||
*rescues* the two forgetting specialists (Fisher–Muller), at high divergence it fails
|
||
(speciation). If it comes out, this is a headline panel. Either outcome is publishable
|
||
(no growth ⇒ "models are safer to merge than biology predicts" — an honest bound).
|
||
- Tests + README + fold into figure. Pre-register the falsifier language before running.
|
||
|
||
## Phase 2 — arXiv preprint package (GG decision 2026-08-11: DO NOT POST until all experiments
|
||
and analysis are complete — the preprint goes up after Phase 3, with the final analysis folded in;
|
||
re-run md2tex + tectonic at that point)
|
||
|
||
- [x] Citation refresh in `paper/the-evolution-of-sex-for-ai.md` *(done 2026-08-11; author names verified against arXiv API)*: **new concessions** —
|
||
First-Extinction Law (2509.20101), quantitative-trait collapse (2407.17493), verifier-injection
|
||
(2510.16657); **precursor** — Livnat & Papadimitriou, *Sex as an algorithm* (CACM 2016);
|
||
**adjacent-to-differentiate** — merge-collapse theory (2603.09463), mergeability prediction
|
||
(2601.22285), non-local merging (2410.12766), model kinship (2410.12613), expert-duration
|
||
(2607.11997), symmetry-scaling (2606.23607), Harris (2604.05142), in-context diversity collapse
|
||
(2604.18005, 2603.24676).
|
||
- [x] Fold E13b/c results into the speciation section (whatever they show — honestly). *(Done: full-symmetry residual + hybrid-fitness cliff + the emergent converse, in abstract, §5, §13 and the accessible version.)*
|
||
- [x] LaTeX conversion: `paper/arxiv/` (md2tex.py block-based converter from the Markdown source of
|
||
truth; main.tex; 3 figures; builds clean under tectonic, 20 pp; arXiv pdflatex hint guarded).
|
||
- [ ] **Post to arXiv — DEFERRED until all experiments/analysis are done (GG).** Package is ready
|
||
(`paper/arxiv/`, instructions in `ARXIV-SUBMISSION.md`); rebuild after the Phase-3 results are
|
||
folded into the manuscript, then upload.
|
||
|
||
## Phase 3 — New experiments for impact & robustness — Weeks 2–3
|
||
|
||
- [x] **`llm_speciation` — the cliff at the LLM tier.** *(Run 2026-08-11, 0.5B: DURATION NULL —
|
||
over-trained disjoint specialists merge better not worse (0.84->0.94, above best parent throughout);
|
||
the MLP "no emergent isolation" null generalises. CONFLICT — function-specific hybrid breakdown:
|
||
merged coherence 0.02-0.08 falls below BOTH parents (~0.2) on the conflicted function. Caught a
|
||
design confound (replace mode ties conflict_frac to private-data budget) -> built the de-confounded
|
||
`add` variant (conflict_mode: add; configs/llm/speciation_add.yaml). 7B confirm optional later.)*
|
||
Speciation demonstrated at all three tiers (analytic → MLP → LLM) makes the headline
|
||
unassailable. Structural bonus to state in the paper: LoRA deltas live in the frozen base's
|
||
coordinate system, so there is **no permutation ambiguity by construction** — any LoRA-merge
|
||
failure is *functional* incompatibility, the residual isolated architecturally.
|
||
- Conflict knob: two LoRA children from the same base learn contradictory conventions on a
|
||
shared family (e.g. "sort ascending" vs "sort descending"; answer-format conflicts) on a
|
||
fraction `conflict_frac` of prompts, plus their own disjoint families. Sweep `conflict_frac`,
|
||
soup-merge, evaluate on neutral held-out + both convention sets. Predict a monotone cliff.
|
||
- Emergent knob: take existing disjoint specialists, sweep **training duration**
|
||
(over-specialisation) → merged performance vs steps. Frames the Amazon observation
|
||
(2607.11997: optimal expert duration) as *our theory explaining their data* — a strong PNAS
|
||
move.
|
||
- 0.5B locally first; one 7B CX3 confirm if the sign is clean (`hpc/` PBS, minutes).
|
||
- [x] **Multi-seed the LLM arc (0.5B tier done 2026-08-11).** All three claims hold with CIs
|
||
(merge>every specialist; union 0.274±0.026 > fusion 0.174±0.102 hard; directed 0.221±0.026 > soup)
|
||
+ NEW finding: fusion is seed-FRAGILE on hard tasks (±0.10) while routing/directed are stable
|
||
(±0.026) — the union/selection operators are the reliable ones. results/llm_*_seeds/ + llm_seeds
|
||
figure. Remaining: 7B CX3 seeds (1-3) when HPC convenient.
|
||
- Thread the seed into specialist cache keys (`spec_<family>[_hard]_s<seed>`); verify nothing
|
||
else assumes the old names.
|
||
- 0.5B: seeds 1–5 × {merge, moe, directed} × {easy, hard}. 7B on CX3: seeds 1–3 × hard
|
||
{merge, moe, directed} (8–25 min walltimes → trivial). Aggregate figures with 95% CI; update
|
||
READMEs; the headroom law now carries error bars.
|
||
- [ ] **`epistasis_predicts` — the DECISIVE experiment (from the external review, 2026-08-11; highest
|
||
priority after llm_speciation lands).** The review's exact bar: population-genetic quantities must
|
||
*predict* (not re-describe) — forecast merge success **before merging**, and beat existing
|
||
predictors. Design, reusing the llm_speciation machinery:
|
||
1. Parents with independently controlled interaction structure: sweep `conflict_frac` (ground-truth
|
||
epistasis) *and* compatible/disjoint + duration variants (spread in divergence WITHOUT conflict),
|
||
so functional conflict and divergence are decorrelated by construction.
|
||
2. Pre-merge predictors, none of which touches a merged model: (a) **operational epistasis** =
|
||
functional-disagreement mass between the parents on a shared probe set (the μ(S) estimate — ours);
|
||
(b) **gradient alignment** (the Zhou et al. 2601.22285 predictor); (c) **weight-space geometry**
|
||
(LoRA-delta cosine / norm distance).
|
||
3. Outcome: merged (soup) performance on private families + convention coherence, held-out test,
|
||
multi-seed.
|
||
4. The claim to test: at matched geometric divergence, the epistasis measure predicts merge outcome
|
||
and the geometry measures do not (R² comparison + an operator-choice decision test — merge vs
|
||
route — under matched budgets).
|
||
Pre-registered falsifier: if gradient/geometry predictors match the epistasis measure, the paper's
|
||
"epistasis, not divergence, sets the cliff" claim stays analytic-only and is labelled as such.
|
||
- [x] **Manuscript claim-narrowing (external review, 2026-08-11) — done.** Softened identity claims
|
||
(WF exact only in the minimal model + learning-kernel cited against ourselves; ratchet scoped to the
|
||
irreversible arm), removed "nobody has / none imports / theory outrun" (priority-dispute bait),
|
||
added the interpretation/explanation/prediction ladder to §1, stated the merge-don't-average
|
||
operator boundaries (output-mean vs weight-avg vs routing vs max-with-oracle, budgets, oracle,
|
||
capacity), added a "what these experiments do and do not establish" scope block to the speciation
|
||
section (impossibility floor is information-theoretic, not genetic; snowball/epistasis-cliff =
|
||
hypotheses at the neural tier), replaced "control theory" with "framework" (subtitle included —
|
||
GG can veto), fixed the §3/§11 overstatements (frozen core ≠ frozen behaviour; Baldwin = echo not
|
||
identity; archive vs operational irreversibility), added the **claims-at-a-glance table**
|
||
(status/assumptions/evidence/limits) to §13 + table support in md2tex, and matched the calibration
|
||
in the accessible version. Adopted the review's framing sentence as the stated core contribution.
|
||
- [ ] **(Optional, SI-grade) ambiguous-families router stress test** — overlapping-skill families
|
||
where the router is no longer trivially perfect; documents the honest limit of union-by-routing.
|
||
Do only if time permits; otherwise keep the existing rider sentence.
|
||
|
||
## Phase 4 — The PNAS manuscript — Weeks 3–4
|
||
|
||
- [ ] **Restructure** `paper/the-evolution-of-sex-for-ai.md` → PNAS research article
|
||
(`paper/pnas/`): Significance (~120 w, dual-audience), Abstract (~250 w), Intro (concede the
|
||
diagnosis; thesis: the cure is sex; where this sits), **Results ladder**:
|
||
1. Collapse is drift, grounding is immigration (E1–E3: exact `H_eq`, `g*≈0.048`, tail threshold
|
||
`m·p*≳1`) — concede Riis/First-Extinction, keep the immigration delta.
|
||
2. Merge, don't average — the conservation law (E4 + neural recombination).
|
||
3. Sex: Fisher–Muller offspring exceed every parent; outbreeding depression on rugged
|
||
landscapes; directed sex as the AI advantage (E8–E10).
|
||
4. **Model speciation** (E12 + hardened E13 + `llm_speciation`) — the headline.
|
||
5. The jointly-necessary society (E11) + mating structure (E14, one panel).
|
||
6. Real-weight confirmation tier: neural/MNIST (brief) + multi-seed LLM headroom law.
|
||
Discussion: the borrowed/ours ledger, the design-rules table (average/route/select/don't-merge ×
|
||
landscape), limits (what LLM-scale still lacks: the full grounded society), what biology gets
|
||
back (in-silico tests of sex theory in regimes biology can't reach). Methods: brief + SI.
|
||
- [ ] **Manifesto sections → Discussion or drop** (institutions, four timescales, re-minting
|
||
philosophy, fitness-is-not-truth): compress each to ≤1 paragraph or move to SI "extended
|
||
discussion". The accessible companion doc stays as-is for outreach, not submission.
|
||
- [ ] **Dual-audience devices**: Table 1 = the population-genetics ↔ machine-learning dictionary;
|
||
every term defined in one clause at first use; keep "the ML statement / the genetics statement"
|
||
paired-paragraph device, compressed.
|
||
- [ ] **Figures: consolidate ~25 per-experiment figures into 5 main multi-panel figures**
|
||
(1 concept/dictionary+operator schematic; 2 collapse&grounding incl. the MNIST digit-decay
|
||
montage — the one image both audiences get instantly; 3 sex: Fisher–Muller/outbreeding/directed +
|
||
merge-don't-average; 4 speciation across three tiers; 5 society ablation + headroom law).
|
||
Everything else → SI figures. All regenerated from committed parquet.
|
||
- [ ] **SI Appendix**: results-summary.md as the skeleton; full methods, all closed forms +
|
||
tolerances, per-experiment configs/seeds, the E13 floor proposition, confusion matrices,
|
||
reproduce.sh instructions.
|
||
- [ ] Word budget: main text ≤ ~6,000 words; check PNAS current LaTeX template + submission
|
||
checklist at writing time.
|
||
|
||
## Phase 5 — Submission mechanics — Week 5
|
||
|
||
- [ ] **Code & data availability**: public GitHub snapshot + Zenodo DOI (code + committed results
|
||
artifacts); PNAS data-availability statement; verify `reproduce.sh` end-to-end on a clean clone.
|
||
- [ ] **Cover letter**: the fit argument (Evolvable-AI 2026 precedent; geneticist's-lens
|
||
contribution; what biology gains); position vs Riis/Shumailov explicitly.
|
||
- [ ] **Editor & reviewer suggestions**: identify the editor who handled "Evolvable AI" (PNAS
|
||
2527700123; N.C. Stenseth per scan — verify); suggest 3–5 reviewers mixing (i) an
|
||
evolution-of-sex theorist, (ii) a model-merging ML researcher, (iii) a model-collapse author.
|
||
- [ ] **Policy checks at submission time** (verify, don't assume): PNAS AI-assistance disclosure
|
||
wording; preprint policy; OA/page charges + whether Imperial has a read-publish agreement with
|
||
PNAS; ORCID; competing interests.
|
||
- [ ] Sync arXiv v2 with the submitted text.
|
||
|
||
## Decision items for GG (not blocking Phases 1–3)
|
||
|
||
- [ ] Title: keep *"The Evolution of Sex for Artificial Intelligence"* vs a more declarative PNAS
|
||
title (e.g. *"A population genetics of model merging: why AI societies should reproduce
|
||
sexually"*). Decide at Phase 4.
|
||
- [ ] Authorship & AI-assistance acknowledgement wording (per PNAS policy).
|
||
- [ ] Repo public at submission vs at acceptance (Zenodo DOI needed at submission either way).
|
||
- [ ] APC/OA budget approval.
|
||
|
||
## Discovered During Work
|
||
|
||
*(append here)*
|