paper (Phase 2): fold hardened E13 into both versions, citation refresh, arXiv package
Speciation section rewritten around the hardened results: alignment modulo the full function-preserving symmetry group (answers 2606.23607 preemptively), the hybrid-fitness cliff (0.97 -> 0.03), the mu(S)/2 floor, and the pre-registered emergent converse (no isolation without functional conflict; the merge rescues forgetting specialists) — in the abstract, §5, §13 ledger, and the accessible version. Citation refresh (author names verified via arXiv API): concede First-Extinction Law (Benati 2509.20101) and quantitative-trait collapse (Yoon 2407.17493) alongside Riis; add verifier-injection (Yi 2510.16657), Livnat & Papadimitriou (CACM 2016) as the sex-as-computation precursor, and the adjacent 2024-26 merge/LMC/multi-agent literature (Ainsworth, Pari, Zhou, Cao, Sharma, Hu, Kozodoi, Li & Shen, Harris, Chen, Tanaka). arXiv package (paper/arxiv/): md2tex.py — a small block-based Markdown->LaTeX converter keeping the Markdown as source of truth — main.tex, generated body.tex, 3 vector figures; builds clean under tectonic (20 pp; pdflatex hint guarded for arXiv); ARXIV-SUBMISSION.md carries categories, license note, and a <=1,920-char abstract. 149 tests green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkRLcc18rwT2Lysu6PbG7v
This commit is contained in:
parent
ea051a5f92
commit
d6a5c5cacd
11 changed files with 671 additions and 44 deletions
|
|
@ -83,9 +83,13 @@ prediction — sex has a **limit**: as two models diverge they undergo **speciat
|
|||
merge-compatibility cliff (compatible → outbreeding depression → hybrid inviability) whose onset is set
|
||||
by divergence *and* epistasis via **Bateson–Dobzhansky–Muller incompatibilities**, and whose damage
|
||||
grows *super-linearly* (the Orr–Turelli snowball). We introduce and model this "model speciation"
|
||||
directly, and confirm it in real trained weights: after permutation alignment (Git Re-Basin), a
|
||||
residual, epistasis-driven merge barrier survives that alignment provably cannot remove — reproductive
|
||||
isolation, not a coordinate artefact. AI also has an advantage biology lacks: **directed sex** — unbounded parents, chosen mates,
|
||||
directly, and confirm it in real trained weights: a merge barrier that survives alignment under the
|
||||
*full* function-preserving symmetry group of the network (not just Git Re-Basin permutations), rising
|
||||
with functional conflict while hybrid fitness falls to inviability — with an honest converse we
|
||||
pre-registered and found: absent conflicting training signals, divergently-specialised lineages of
|
||||
shared ancestry developed *no* isolation at any divergence tested, the merge instead *rescuing* the
|
||||
forgetting specialists. Isolation must be provoked by conflict; specialisation alone did not speciate.
|
||||
AI also has an advantage biology lacks: **directed sex** — unbounded parents, chosen mates,
|
||||
and offspring screened before they are kept — which converts recombination from a gamble into a
|
||||
reliable engine and has no biological analogue.
|
||||
|
||||
|
|
@ -129,14 +133,22 @@ with an overtly evolutionary vocabulary: crossover-mutation-selection over LLM p
|
|||
2025), niching and "mate choice" (Sakana's M2N2 — 2025), and evolutionary search over merge recipes
|
||||
(Akiba et al., *Nature Mach. Intell.* 2024/25).
|
||||
|
||||
We are candid about the consequence. Two things we do **not** claim. First, that collapse is
|
||||
Wright–Fisher drift: formalised independently (Riis, 2026; Shumailov et al., 2024) and conceded here.
|
||||
We are candid about the consequence. Three things we do **not** claim. First, that collapse is
|
||||
Wright–Fisher drift: formalised independently (Riis, 2026; Shumailov et al., 2024), sharpened to a
|
||||
closed-form first-extinction law whose onset coincides with collapse (Benati et al., 2025) and to a
|
||||
quantitative-trait-genetics account for diffusion models (Yoon et al., ICLR 2025), and conceded here.
|
||||
Second, the bare empirical facts that a merged model can beat its parents, that decorrelated parents
|
||||
merge better, and that naive averaging is inferior to sign- or routing-based merges (TIES, DARE,
|
||||
mixture-of-experts routing): all established. What is genuinely unoccupied — and what a geneticist is
|
||||
placed to supply — is a **theory** rather than a search heuristic. Every one of the works above uses
|
||||
evolution as *metaphor over an optimiser*; none imports the predictive apparatus of the evolution of
|
||||
sex. Nobody has stated the **"merge, don't average" conservation law**, derived **offspring-exceed-parents
|
||||
mixture-of-experts routing): all established. Third, that merge success can be *predicted at all*:
|
||||
machine-learning-native predictors exist, from interpretable pairwise metrics (gradient alignment —
|
||||
Zhou et al., 2026) to capacity/rate-distortion accounts of "merging collapse" (2026); what they lack,
|
||||
and we supply, is the *mechanism* — when and why the failure is a coordinate artefact versus genuine
|
||||
functional incompatibility, and what moves the cliff. What is genuinely unoccupied — and what a
|
||||
geneticist is placed to supply — is a **theory** rather than a search heuristic. The nearest precursor
|
||||
is a theory-of-computation tradition reading sex as an algorithm for *mixability* (Livnat &
|
||||
Papadimitriou, 2016), pre-dating model merging and never applied to it. Every one of the works above
|
||||
uses evolution as *metaphor over an optimiser*; none imports the predictive apparatus of the evolution
|
||||
of sex. Nobody has stated the **"merge, don't average" conservation law**, derived **offspring-exceed-parents
|
||||
as Fisher–Muller**, predicted **outbreeding depression on rugged task landscapes**, framed **grounding
|
||||
as migration–drift balance** with a critical fraction, or connected **reproductive isolation** to when
|
||||
two models can be merged at all. An evolutionary algorithm that *finds* a super-parent is evidence for
|
||||
|
|
@ -357,22 +369,49 @@ event. (Figure: `results/E12/E12.png`.)
|
|||
**The real-weight confirmation.** The obvious objection to the analytic model is that its
|
||||
"incompatibility" is a re-labelled loss barrier, and loss barriers between independently trained
|
||||
networks are famously a *coordinate* artefact — two nets that learned the same function in a permuted
|
||||
basis look incompatible until their neurons are aligned (Git Re-Basin). We therefore ran the experiment
|
||||
that the objection demands, in real trained weights. Two small MLPs are forked from a shared MNIST base,
|
||||
basis look incompatible until their neurons are aligned (Git Re-Basin), and recent work shows that
|
||||
symmetry groups *richer* than permutations remove still more of the barrier (functionality-preserving
|
||||
rescalings and rotations — Scaling LMC, 2026; neuron-identifiability approaches). We therefore ran the
|
||||
experiment the objection demands, in real trained weights, aligning modulo the **full**
|
||||
function-preserving unit symmetry group of the architecture (per-unit positive rescaling composed with
|
||||
permutation — for a plain ReLU network, all of it). Two small MLPs are forked from a shared MNIST base,
|
||||
trained, weight-averaged, and their linear-mode-connectivity error barrier is measured *before and
|
||||
after* in-house Git Re-Basin permutation alignment; the after-alignment **residual** is the part of the
|
||||
incompatibility that alignment provably cannot explain away. The decomposition is clean (Figure:
|
||||
after* alignment; the after-alignment **residual** is the part of the incompatibility that no
|
||||
re-coordination can explain away. The decomposition is clean (Figure:
|
||||
`results/speciation_real/speciation_real.png`): two nets trained *from different random initialisations
|
||||
on the same task* have a real naive barrier that alignment **removes ~98 % of** (residual ≈ 0.001) —
|
||||
same species, different basis, exactly the canonical Re-Basin result, which also proves our aligner
|
||||
works. Two nets that learned *conflicting* label maps have a large barrier that alignment **removes none
|
||||
of** (residual ≈ 0.50) — genuine reproductive isolation, not a coordinate artefact, and it cannot be
|
||||
dismissed as a failure to align because the very same aligner erased the same-task barrier. Sweeping the
|
||||
fraction of conflicting classes traces the **isolation cliff in real weights**: the residual (after
|
||||
alignment) barrier climbs monotonically from 0 to ~0.49 with task conflict — the real-weight image of
|
||||
E12's analytic cliff, and the direct answer to "isn't this just a permutation artefact?" It is not: the
|
||||
part that survives alignment is real speciation, and it rises with the functional conflict between the
|
||||
lineages exactly as the Dobzhansky–Muller frame predicts.
|
||||
on the same task* have a real naive barrier that alignment removes almost entirely (residual ≈ 0.001,
|
||||
and the aligned merge performs at parent level) — same species, different basis, the canonical Re-Basin
|
||||
result, which also proves the aligner works. Two nets that learned *conflicting* label maps have a
|
||||
large barrier of which the full symmetry group removes **essentially nothing** (0.502 → 0.497) —
|
||||
genuine reproductive isolation, not a missed symmetry, and it cannot be dismissed as a failure to align
|
||||
because the very same aligner erased the same-task barrier. It also carries a floor no future alignment
|
||||
method can breach: models loyal to label maps that conflict on a fraction *μ* of inputs cannot both be
|
||||
served by *any* single merged model, which must err at rate ≥ *μ*/2 against at least one parent
|
||||
(SI proposition). Sweeping the fraction of conflicting classes traces the **isolation cliff in real
|
||||
weights**, now readable directly as *hybrid fitness*: the residual barrier climbs monotonically while
|
||||
the merged model's accuracy falls from 0.97 to 0.03 — E12's compatible → depression → inviability
|
||||
trajectory, measured.
|
||||
|
||||
**And its honest converse: speciation must be provoked; it did not emerge.** A true
|
||||
Dobzhansky–Muller incompatibility is *emergent* — each lineage's changes harmless alone, incompatible
|
||||
only in combination — whereas the conflict condition above *imposes* contradiction. So we pre-registered
|
||||
the emergent test: fork two children from a shared base and let them diverge with **no conflicting
|
||||
training signal anywhere** — one pair as complementary class specialists (one child trains only on
|
||||
digits 0–4, the other only on 5–9), one pair with divergent input conventions (views shifted in
|
||||
opposite directions) — out to divergences 6.4× the base training. The result is the second
|
||||
pre-registered reading, and it sharpens the theory's scope rather than confirming its most dramatic
|
||||
form: the residual barrier is **0.000 at every divergence in both conditions**, and far from failing,
|
||||
the merge *rescues* the two specialists — each parent decays toward ~0.50 on the full task
|
||||
(catastrophically forgetting the classes it no longer sees) while the merged model holds ~0.95
|
||||
throughout, a sustained Fisher–Muller rescue at zero barrier. In real weights, at least in this regime
|
||||
of shared ancestry and compatible tasks, **reproductive isolation requires functional conflict; it does
|
||||
not arise spontaneously from divergent specialisation.** The design rule sharpens accordingly: *merge
|
||||
freely across divergently-specialised lineages of shared ancestry — what speciates model populations is
|
||||
conflicting conventions, not specialisation per se.* Whether long-horizon over-specialisation erodes
|
||||
mergeability at language-model scale — as the empirical merging literature hints (experts trained
|
||||
longer merge worse under averaging) — is exactly the next tier's question, and the theory now makes the
|
||||
prediction crisp: it should depend on whether extended training induces *conflicting conventions on
|
||||
shared circuitry*, not on divergence time itself.
|
||||
|
||||
One question remains, and the rest of the paper is largely about it: recombination combines what the
|
||||
parents kept — but *who decides what each parent keeps, and which offspring are worth keeping?*
|
||||
|
|
@ -641,14 +680,17 @@ shape.)
|
|||
|
||||
**What is borrowed, and what is ours.** We are deliberate about the ledger, because the surrounding
|
||||
literature is crowded and a reader deserves to know exactly where the line falls. **Conceded as prior
|
||||
art:** (a) *model collapse is genetic drift* — derived independently and cleanly (Riis, 2026; and the
|
||||
Wright–Fisher collapse literature following Shumailov et al., 2024); (b) the empirical facts that a
|
||||
merged model can *beat its parents*, that *decorrelated* parents merge better, and that *naive averaging
|
||||
is inferior* to sign-reconciled or routed merges (model soups, TIES, DARE, mixture-of-experts routing);
|
||||
(c) that a *population* of merging or self-improving models can climb (GENOME, M2N2, Multiagent
|
||||
Finetuning, the Darwin–Gödel Machine); and (d) that even the *magnitude* of multi-task merge degradation
|
||||
has a machine-learning-native predictive account (recent stability/scaling analyses). We claim none of
|
||||
these.
|
||||
art:** (a) *model collapse is genetic drift* — derived independently and cleanly (Riis, 2026; the
|
||||
Wright–Fisher collapse literature following Shumailov et al., 2024; the closed-form first-extinction
|
||||
law of Benati et al., 2025; the quantitative-trait account of Yoon et al., 2025); (b) the empirical
|
||||
facts that a merged model can *beat its parents*, that *decorrelated* parents merge better, and that
|
||||
*naive averaging is inferior* to sign-reconciled or routed merges (model soups, TIES, DARE,
|
||||
mixture-of-experts routing); (c) that a *population* of merging or self-improving models can climb
|
||||
(GENOME, M2N2, Multiagent Finetuning, the Darwin–Gödel Machine); (d) that merge success has
|
||||
machine-learning-native *predictors* — interpretable pairwise metrics (Zhou et al., 2026),
|
||||
capacity/rate-distortion accounts of merging collapse (Cao et al., 2026), and stability/scaling
|
||||
analyses of multi-task degradation; and (e) that verifier-screened synthetic data can avert collapse
|
||||
(Yi et al., 2025) — the statistical cousin of our grounding operator. We claim none of these.
|
||||
|
||||
**Ours** is the theory those results have outrun: a **population-genetics of sex** applied to model
|
||||
societies, which is *generative* where the incumbents are empirical. Concretely — the **"merge, don't
|
||||
|
|
@ -668,8 +710,12 @@ models are too diverged to be merged at all*. We model it explicitly (§5), pred
|
|||
compatible → outbreeding-depression → inviability curve, its super-linear (snowball) onset, and its
|
||||
control by epistasis rather than divergence alone — the one place the merge literature has phenomena
|
||||
(Pari et al., 2024; Zhou et al., 2026) but no theory — and we confirm it in real trained weights, where
|
||||
a merge barrier survives permutation alignment (Git Re-Basin) as a residual, epistasis-driven
|
||||
reproductive isolation that the coordinate-artefact account cannot explain away. In one sentence: the field agrees on the disease
|
||||
a merge barrier survives alignment under the *full* function-preserving symmetry group (not only
|
||||
Re-Basin permutations) as a residual, functional reproductive isolation with an information-theoretic
|
||||
floor — together with the pre-registered emergent converse: absent conflicting training signals,
|
||||
divergently-specialised lineages of shared ancestry showed *no* isolation at any divergence tested, the
|
||||
merge instead rescuing the forgetting specialists (isolation must be provoked; specialisation alone did
|
||||
not speciate). In one sentence: the field agrees on the disease
|
||||
and tinkers at the cure with evolutionary metaphors; we bring the evolutionary *theory*, and it makes
|
||||
falsifiable predictions — a merge-compatibility cliff among them — that the metaphors do not.
|
||||
|
||||
|
|
@ -724,6 +770,7 @@ The operators, checked; the living society, next.
|
|||
- Otto, S. P., & Lenormand, T. (2002). Resolving the paradox of sex and recombination. *Nature Reviews Genetics.*
|
||||
- Kondrashov, A. S. (1993). Classification of hypotheses on the advantage of amphimixis. *Journal of Heredity.*
|
||||
- Dobzhansky, T. (1936); Muller, H. J. (1942). Bateson–Dobzhansky–Muller incompatibilities (reproductive isolation).
|
||||
- Livnat, A., & Papadimitriou, C. (2016). Sex as an algorithm: the theory of evolution under the lens of computation. *Communications of the ACM 59(11).* (The theory-of-computation precursor: recombination selects for mixability.)
|
||||
|
||||
*The 2025–2026 landscape this paper positions against:*
|
||||
|
||||
|
|
@ -735,5 +782,19 @@ The operators, checked; the living society, next.
|
|||
- Yu, L., Yu, B., Yu, H., Huang, F., & Li, Y. (2023). Language models are super Mario: absorbing abilities from homologous models (DARE). *arXiv:2311.03099.*
|
||||
- Gerstgrasser, M., et al. (2024). Is model collapse inevitable? Breaking the curse of recursion by accumulating real and synthetic data. *arXiv:2404.01413.*
|
||||
- Guo, D., Wu, J., & Yiu, S. M. (2026). Model collapse as cultural evolution. *arXiv:2605.23054.*
|
||||
- Benati, M., Londei, A., Lanzieri, D., & Loreto, V. (2025). First-extinction law for resampling processes. *arXiv:2509.20101.* (Collapse onset = the Wright–Fisher first-extinction time.)
|
||||
- Yoon, Y., Hu, D., Weissburg, I., Qin, Y., & Jeong, H. (2025). Model collapse in the self-consuming chain of diffusion finetuning: a novel perspective from quantitative trait modeling. *ICLR 2025 / arXiv:2407.17493.*
|
||||
- Yi, B., Liu, Q., Cheng, Y., & Xu, H. (2025). Escaping model collapse via synthetic data verification. *arXiv:2510.16657.*
|
||||
- Ainsworth, S., Hayase, J., & Srinivasa, S. (2022). Git Re-Basin: merging models modulo permutation symmetries. *arXiv:2209.04836.*
|
||||
- Li, T., & Shen, Z. (2026). Scaling linear mode connectivity and merging to billion-parameter pretrained transformers. *arXiv:2606.23607.* (Symmetry groups richer than permutations remove more of the barrier.)
|
||||
- Sharma, E., Roy, D. M., & Dziugaite, G. K. (2024). The non-local model merging problem: permutation symmetries and variance collapse. *arXiv:2410.12766.*
|
||||
- Pari, J., Jelassi, S., & Agrawal, P. (2024). Collective model intelligence requires compatible specialization. *arXiv:2411.02207.*
|
||||
- Zhou, L., Zhao, B., Yu, R., & Rodolà, E. (2026). Demystifying mergeability: interpretable properties to predict model merging success. *arXiv:2601.22285.*
|
||||
- Cao, Y., Ran, D., Guo, Y., Wu, M., Chen, S., et al. (2026). An empirical study and theoretical explanation on task-level model-merging collapse. *arXiv:2603.09463.*
|
||||
- Hu, Y., Yao, Y., Zhang, N., Chen, H., & Deng, S. (2024). Exploring model kinship for merging large language models. *arXiv:2410.12613.*
|
||||
- Kozodoi, N., Afolabi, Z., & Butler, J. (2026). Are we merging the right models? Impact of expert training duration on model merging for LLMs. *arXiv:2607.11997.*
|
||||
- Harris, K. D. (2026). A mathematical theory of evolution for self-designing AIs. *arXiv:2604.05142.*
|
||||
- Chen, N., Tong, Y., Yang, Y., He, Y., Zhang, X., et al. (2026). Diversity collapse in multi-agent LLM systems: structural coupling and collective failure in open-ended idea generation. *arXiv:2604.18005.*
|
||||
- Tanaka, H. (2026). When is collective intelligence a lottery? Multi-agent scaling laws for memetic drift in LLMs. *arXiv:2603.24676.*
|
||||
|
||||
*Still to engage in a full version: the machine-learning-native theory of merge degradation with task count; tacit knowledge (Polanyi) and human capital (Becker).*
|
||||
*Still to engage in a full version: tacit knowledge (Polanyi) and human capital (Becker).*
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue