diff --git a/paper/pnas/body.tex b/paper/pnas/body.tex index 70f7f84..5ee1443 100644 --- a/paper/pnas/body.tex +++ b/paper/pnas/body.tex @@ -4,7 +4,7 @@ Artificial intelligence increasingly consists of populations of models rather th \section*{Abstract} -AI development increasingly resembles a population process. Models are specialised, retrained on model output, and recombined by weight merging, and the practice is described in evolutionary vocabulary with little use of evolutionary theory. We treat multigenerational model populations as systems whose inheritance, diversity, and compatibility must be managed, and we transfer the quantitative framework of the evolution of sex. Its starting point, that training on model output is genetic drift and model collapse its signature, we reached independently; parallel work has formalised the same diagnosis, a convergence we take as support for the frame. In a minimal inheritance model that is exactly Wright--Fisher, and measurably Wright--Fisher plus estimator bias in trained networks, we derive and test remedies. Grounding acts as immigration: a real-data fraction far below one retained most equilibrium diversity, with a per-capability observation floor that makes the rarest knowledge expensive under unstratified sampling. Refitting a child to the mean of its parents' output distributions cancels the multi-parent gain to first order in the rare-item regime; union-preserving operators realise it. Merged language-model specialists exceeded every parent in replicated experiments. Blind recombination fails on rugged task landscapes; screening candidate offspring restores the gain. The optimal mating breadth narrows as skills entangle. Finally, we introduce model speciation: a merge barrier remaining after permutation-and-rescaling alignment tracks functional conflict, isolation did not emerge from compatible specialisation, and in a controlled test pre-merge functional disagreement predicted merge damage while weight-geometry baselines showed no detectable association. +AI development increasingly resembles a population process. Models are specialised, retrained on model output, and recombined by weight merging, and the practice is described in evolutionary vocabulary with little use of evolutionary theory. We treat multigenerational model populations as systems whose inheritance, diversity, and compatibility must be managed, and we transfer the quantitative framework of the evolution of sex. Its starting point, that training on model output is genetic drift and model collapse its signature, is by now established from several independent directions; we develop the structure that follows from it. In a minimal inheritance model that is exactly Wright--Fisher, and measurably Wright--Fisher plus estimator bias in trained networks, we derive and test remedies. Grounding acts as immigration: a real-data fraction far below one retained most equilibrium diversity, with a per-capability observation floor that makes the rarest knowledge expensive under unstratified sampling. Refitting a child to the mean of its parents' output distributions cancels the multi-parent gain to first order in the rare-item regime; union-preserving operators realise it. Merged language-model specialists exceeded every parent in replicated experiments. Blind recombination fails on rugged task landscapes; screening candidate offspring restores the gain. The optimal mating breadth narrows as skills entangle. Finally, we introduce model speciation: a merge barrier remaining after permutation-and-rescaling alignment tracks functional conflict, isolation did not emerge from compatible specialisation, and in a controlled test pre-merge functional disagreement predicted merge damage while weight-geometry baselines showed no detectable association. \medskip\hrule\medskip @@ -14,7 +14,7 @@ Machine learning has become a population-scale phenomenon. Public repositories h The generations are coupled through data as well as through weights. Successive models increasingly learn from model output rather than from fresh human experience: frontier alignment pipelines are now predominantly synthetic (over 98\% in documented cases; 11, 12), self-generated instruction data seeds whole lineages of descendants (13), a large and growing share of the public web is machine-generated or machine-translated text (14, 15), and the stock of human text is projected to be exhausted by frontier training within this decade (16). Meanwhile persistent multi-agent systems and emerging agent economies put many interacting models into sustained contact (17--20). A population whose members inherit from one another, recombine, and retransmit under these conditions is an evolving population in the technical sense. The claim of this paper is that the vocabulary should be given its mathematics: multigenerational model populations are systems whose inheritance, diversity, and compatibility must be managed, not merely collections of models to optimise, and the branch of biology that studies exactly this problem, the population genetics of the evolution of sex, transfers as a quantitative framework. -The frame's entry point is the diagnosis. Training each generation of a model on the previous generation's output degrades it (\emph{model collapse}): rare capabilities vanish first and the lineage drifts toward its own most common behaviour (21). That this is the mathematics of \emph{genetic drift} in a finite population is a conclusion we reached independently in building the present framework, and one that has been derived in parallel from several other directions (22--24), including a closed-form first-extinction law placing collapse onset at the Wright--Fisher first-extinction time (23), and that was anticipated, before deep learning, in an analysis of sequential inference chains as generalised genetic drift (25). We cite these works for priority of publication on the diagnosis and read the convergence, independent arrivals at the same population-genetic account by different routes and in different decades, as corroboration that the frame is the natural one. What none of that parallel work develops, and what this paper is about, is the structure the diagnosis opens: the full arc from drift through its remedies (immigration, recombination, selection, population structure) to its limit (reproductive isolation), carried as one framework from closed forms to trained networks to language models. +The diagnosis comes first. Training each generation of a model on the previous generation's output degrades it (\emph{model collapse}): rare capabilities vanish first, and the lineage drifts toward its own most common behaviour (21). That degradation is, mathematically, \emph{genetic drift}, the loss of rare variants that any finite population suffers when each generation is a finite sample of the last. The identification has been made repeatedly and independently: for sequential inference chains before deep learning (22), for language-model text ecosystems (23), as a closed-form first-extinction law placing collapse onset at the Wright--Fisher first-extinction time (24), and in quantitative-genetic form for self-consuming diffusion models (25). A diagnosis reached so often, from such different starting points, marks population genetics as the natural mathematics of the setting. It is also only the entry point. Population genetics is not, at heart, a theory of decay; it is a theory of the mechanisms that maintain and build populations despite decay (immigration, recombination, selection, population structure) and of where those mechanisms reach their limits. This paper develops that fuller structure for model populations: the arc from drift through its remedies to its limit, reproductive isolation, carried as one framework from closed forms to trained networks to language models. In machine learning's own terms, the problem this frame addresses is the field's oldest, \emph{continual learning}, reappearing one level up. Within a single network, sequential learning overwrites prior knowledge (catastrophic forgetting; 26, 27), and the discipline's remedies are, one by one, the population operators of this paper in single-model form: \emph{rehearsal and replay} of past data is grounding's within-lineage counterpart, and the field's empirically settled replay fractions, on the order of 1\% for instruction tuning (28) and 5\% to 25\% by distribution-shift strength in continual pretraining (29), sit where the minimal model's operational grounding threshold lies, a correspondence for which the framework supplies the missing theory (equilibrium diversity, and a per-capability survival law). \emph{Pseudo-rehearsal}, the replay of the network's own generated samples, proposed as a cure in 1995 (30) and revived as generative replay (31), is this paper's ungrounded null: immigration from a drifting source, benign for one hop and compounding into collapse over generations; verifier-filtering (32, 33) converts it back into grounding. \emph{Parameter isolation} (34, and frozen-base adapters, which forget far less; 35) is the engineered decorrelation our specialists use; \emph{complementary-learning-systems consolidation} (36--38) is our periodic adapter-into-base merge; the recent turn to \emph{merging as a continual-learning mechanism} (39--42) applies recombination within one lineage over time, where we apply it across lineages; and the observation that rare examples and long-tail knowledge are forgotten first (43--45) is tail-allele extinction observed one model at a time. One distinction is kept explicit throughout: catastrophic forgetting is largely deterministic interference from shifted training, whereas collapse is stochastic sampling drift; the two phenomena share their victims, the rare, and their remedies, but not their mechanism. To our knowledge, no prior work carries population-genetic formalism into continual learning itself; that bridge (replay as immigration with a survival law, merging as recombination with a compatibility criterion, consolidation as the slow store of a two-speed memory) is where this framework may matter most. @@ -133,7 +133,7 @@ Predictive test & 13 conditions \(\times\) 3 seeds (0.5B) & Merge penalty vs ora \textbf{What this offers continual learning.} Read into the field where these results most directly land: (i) a first-principles account of the \emph{replay ratio}: the field's constants (\(\approx\)1\%, 5\%, 25\%; 28, 29) acquire an equilibrium theory and a sharper prediction, that the required fraction is set by the rarest capability one refuses to lose (the \texttt{1 − e\textasciicircum{}{−m\(\cdot\)p}} law) rather than by average loss, which is testable against published replay sweeps; (ii) a \emph{failure theory for generative replay}: self-generated rehearsal is safe for short horizons and compounds into collapse across generations unless verifier-filtered back into grounding (30--33); (iii) \emph{pre-merge interference prediction with a mechanism}: where the current state of the art fits regressions over candidate metrics (56), the functional-conflict measure arrives at a convergent signal from principle and comes with an operator prescription --- when conflict is high, do not average; route or breed-and-screen; (iv) a candidate \emph{decision rule for the consolidate-versus-stay-modular question} that currently splits the field's practice (keep adapters separate vs merge them; 54--58): union-preserving operators where headroom exists, fusion where the base composes, consolidation as the slow-store step; and (v) \emph{tail monitoring as the leading indicator}: continual-learning evaluation that averages over capabilities hides exactly the losses that drift theory says come first and, past a threshold, become irreversible. On that last point we note the standing objection that apparent forgetting can be skewed task-inference over latent capability rather than erasure (66); our irreversibility results concern oracle-measured behavioural distributions, and distinguishing latent from extinct capability at language-model scale is an open experiment whose outcome would be decisive for both readings. -\textbf{What is borrowed and what is ours.} The diagnosis --- collapse as drift --- was published first by others and we cite it so (21--24), while noting the derivations are independent and convergent; prior art in the strict sense are the empirical facts that merges can beat parents, that decorrelated parents merge better, and that naive averaging loses to interference-aware or routed merges (4, 54, 55), that model populations can climb (5, 8--10), and that merge success admits ML-native predictors (56, 57). Ours is the framework-level synthesis --- inheritance, diversity, and compatibility as managed quantities --- together with: the conservation law for blending inheritance and its operator boundaries; the per-item grounding floor; the society ablation with its complementary failure modes; model speciation as a named, tested question, with the coordinate-versus-functional decomposition under permutation-and-rescaling alignment and the emergent null that bounds it; and the controlled predictive test with its controls. We claim the framework generated these measurements and experiments; we do not claim their outcomes validate a uniquely population-genetic mechanism, and one refinement it proposed was not supported. +\textbf{What is borrowed and what is ours.} The collapse-as-drift diagnosis is established prior work (21--25); so are the empirical facts that merges can beat parents, that decorrelated parents merge better, and that naive averaging loses to interference-aware or routed merges (4, 54, 55), that model populations can climb (5, 8--10), and that merge success admits ML-native predictors (56, 57). Ours is the framework-level synthesis --- inheritance, diversity, and compatibility as managed quantities --- together with: the conservation law for blending inheritance and its operator boundaries; the per-item grounding floor; the society ablation with its complementary failure modes; model speciation as a named, tested question, with the coordinate-versus-functional decomposition under permutation-and-rescaling alignment and the emergent null that bounds it; and the controlled predictive test with its controls. We claim the framework generated these measurements and experiments; we do not claim their outcomes validate a uniquely population-genetic mechanism, and one refinement it proposed was not supported. \textbf{Limits and open problems.} The demonstrations are deliberately small: exact where small is a virtue, sign-level and seed-replicated at the language-model tier, on constructed task families with a trivially separable router and one model lineage (Qwen, 0.5B--7B). The composed society has not been built at language-model scale. The predictive test's next bars, in order of value: generalisation to \emph{unfamiliar} conflict structures and real task pairs; a demonstrably better \emph{budget-matched} merging decision; then scale replication. Beyond engineering, the framework's hardest open problem is the fitness function itself: selection optimises what is measured, and for knowledge systems the persuasive and the true compete --- grounding against a reality that can refuse is the only anchor we trust, and institutionalising that anchor (verification, replication, and challenge among models) is the society-level problem we pose but do not solve. What biology receives in return is a new model system: populations of learners where every genotype, environment, and mating decision is observable and manipulable --- where the evolution of sex can be studied with interventions (unbounded parents, offspring preview, directed mating) that no living system permits. @@ -173,10 +173,10 @@ Predictive test & 13 conditions \(\times\) 3 seeds (0.5B) & Merge penalty vs ora \item T. Guo, et al., Large language model based multi-agents: A survey of progress and challenges. \emph{Proc. Int. Joint Conf. Artif. Intell.} (2024). https://doi.org/10.48550/arXiv.2402.01680. \item N. Tomasev, et al., Virtual agent economies. arXiv [Preprint] (2025). https://doi.org/10.48550/arXiv.2509.10147. \item I. Shumailov, et al., AI models collapse when trained on recursively generated data. \emph{Nature} \textbf{631}, 755--759 (2024). +\item J. P. Crutchfield, S. Whalen, Structural drift: The population dynamics of sequential learning. \emph{PLOS Comput. Biol.} \textbf{8}, e1002510 (2012). \item S. Riis, Drift and selection in LLM text ecosystems. arXiv [Preprint] (2026). https://doi.org/10.48550/arXiv.2604.08554. \item M. Benati, A. Londei, D. Lanzieri, V. Loreto, First-extinction law for resampling processes. arXiv [Preprint] (2025). https://doi.org/10.48550/arXiv.2509.20101. \item Y. Yoon, D. Hu, I. Weissburg, Y. Qin, H. Jeong, Model collapse in the self-consuming chain of diffusion finetuning: A novel perspective from quantitative trait modeling. \emph{Int. Conf. Learn. Represent.} (2025). https://doi.org/10.48550/arXiv.2407.17493. -\item J. P. Crutchfield, S. Whalen, Structural drift: The population dynamics of sequential learning. \emph{PLOS Comput. Biol.} \textbf{8}, e1002510 (2012). \item M. McCloskey, N. J. Cohen, Catastrophic interference in connectionist networks: The sequential learning problem. \emph{Psychol. Learn. Motiv.} \textbf{24}, 109--165 (1989). \item R. M. French, Catastrophic forgetting in connectionist networks. \emph{Trends Cogn. Sci.} \textbf{3}, 128--135 (1999). \item T. Scialom, T. Chakrabarty, S. Muresan, Fine-tuned language models are continual learners. \emph{Proc. Conf. Empir. Methods Nat. Lang. Process.} (2022). https://doi.org/10.48550/arXiv.2205.12393. diff --git a/paper/pnas/main.md b/paper/pnas/main.md index e24a431..ab13559 100644 --- a/paper/pnas/main.md +++ b/paper/pnas/main.md @@ -23,8 +23,8 @@ model output, and recombined by weight merging, and the practice is described in vocabulary with little use of evolutionary theory. We treat multigenerational model populations as systems whose inheritance, diversity, and compatibility must be managed, and we transfer the quantitative framework of the evolution of sex. Its starting point, that training on model output is -genetic drift and model collapse its signature, we reached independently; parallel work has -formalised the same diagnosis, a convergence we take as support for the frame. In a minimal +genetic drift and model collapse its signature, is by now established from several independent +directions; we develop the structure that follows from it. In a minimal inheritance model that is exactly Wright–Fisher, and measurably Wright–Fisher plus estimator bias in trained networks, we derive and test remedies. Grounding acts as immigration: a real-data fraction far below one retained most equilibrium diversity, with a per-capability observation floor that @@ -67,19 +67,20 @@ diversity, and compatibility must be managed, not merely collections of models t branch of biology that studies exactly this problem, the population genetics of the evolution of sex, transfers as a quantitative framework. -The frame's entry point is the diagnosis. Training each generation of a model on the previous -generation's output degrades it (*model collapse*): rare capabilities vanish first and the lineage -drifts toward its own most common behaviour (21). That this is the mathematics of *genetic drift* in -a finite population is a conclusion we reached independently in building the present framework, and -one that has been derived in parallel from several other directions (22–24), including a closed-form -first-extinction law placing collapse onset at the Wright–Fisher first-extinction time (23), and that -was anticipated, before deep learning, in an analysis of sequential inference chains as generalised -genetic drift (25). We cite these works for priority of publication on the diagnosis and read the -convergence, independent arrivals at the same population-genetic account by different routes and in -different decades, as corroboration that the frame is the natural one. What none of that parallel work develops, and what this paper is about, is the -structure the diagnosis opens: the full arc from drift through its remedies (immigration, -recombination, selection, population structure) to its limit (reproductive isolation), carried as one -framework from closed forms to trained networks to language models. +The diagnosis comes first. Training each generation of a model on the previous generation's output +degrades it (*model collapse*): rare capabilities vanish first, and the lineage drifts toward its own +most common behaviour (21). That degradation is, mathematically, *genetic drift*, the loss of rare +variants that any finite population suffers when each generation is a finite sample of the last. The +identification has been made repeatedly and independently: for sequential inference chains before deep +learning (22), for language-model text ecosystems (23), as a closed-form first-extinction law placing +collapse onset at the Wright–Fisher first-extinction time (24), and in quantitative-genetic form for +self-consuming diffusion models (25). A diagnosis reached so often, from such different starting +points, marks population genetics as the natural mathematics of the setting. It is also only the entry +point. Population genetics is not, at heart, a theory of decay; it is a theory of the mechanisms that +maintain and build populations despite decay (immigration, recombination, selection, population +structure) and of where those mechanisms reach their limits. This paper develops that fuller structure +for model populations: the arc from drift through its remedies to its limit, reproductive isolation, +carried as one framework from closed forms to trained networks to language models. In machine learning's own terms, the problem this frame addresses is the field's oldest, *continual learning*, reappearing one level up. Within a single network, sequential learning @@ -426,9 +427,8 @@ skewed task-inference over latent capability rather than erasure (66); our irrev concern oracle-measured behavioural distributions, and distinguishing latent from extinct capability at language-model scale is an open experiment whose outcome would be decisive for both readings. -**What is borrowed and what is ours.** The diagnosis — collapse as drift — was published first by -others and we cite it so (21–24), while noting the derivations are independent and convergent; prior art -in the strict sense are the empirical facts that merges can beat parents, that decorrelated parents merge better, and that +**What is borrowed and what is ours.** The collapse-as-drift diagnosis is established prior work +(21–25); so are the empirical facts that merges can beat parents, that decorrelated parents merge better, and that naive averaging loses to interference-aware or routed merges (4, 54, 55), that model populations can climb (5, 8–10), and that merge success admits ML-native predictors (56, 57). Ours is the framework-level synthesis — inheritance, diversity, and compatibility as managed quantities — together with: the @@ -525,10 +525,10 @@ publication; every figure in this paper regenerates from committed artifacts wit 19. T. Guo, et al., Large language model based multi-agents: A survey of progress and challenges. *Proc. Int. Joint Conf. Artif. Intell.* (2024). https://doi.org/10.48550/arXiv.2402.01680. 20. N. Tomasev, et al., Virtual agent economies. arXiv [Preprint] (2025). https://doi.org/10.48550/arXiv.2509.10147. 21. I. Shumailov, et al., AI models collapse when trained on recursively generated data. *Nature* **631**, 755–759 (2024). -22. S. Riis, Drift and selection in LLM text ecosystems. arXiv [Preprint] (2026). https://doi.org/10.48550/arXiv.2604.08554. -23. M. Benati, A. Londei, D. Lanzieri, V. Loreto, First-extinction law for resampling processes. arXiv [Preprint] (2025). https://doi.org/10.48550/arXiv.2509.20101. -24. Y. Yoon, D. Hu, I. Weissburg, Y. Qin, H. Jeong, Model collapse in the self-consuming chain of diffusion finetuning: A novel perspective from quantitative trait modeling. *Int. Conf. Learn. Represent.* (2025). https://doi.org/10.48550/arXiv.2407.17493. -25. J. P. Crutchfield, S. Whalen, Structural drift: The population dynamics of sequential learning. *PLOS Comput. Biol.* **8**, e1002510 (2012). +22. J. P. Crutchfield, S. Whalen, Structural drift: The population dynamics of sequential learning. *PLOS Comput. Biol.* **8**, e1002510 (2012). +23. S. Riis, Drift and selection in LLM text ecosystems. arXiv [Preprint] (2026). https://doi.org/10.48550/arXiv.2604.08554. +24. M. Benati, A. Londei, D. Lanzieri, V. Loreto, First-extinction law for resampling processes. arXiv [Preprint] (2025). https://doi.org/10.48550/arXiv.2509.20101. +25. Y. Yoon, D. Hu, I. Weissburg, Y. Qin, H. Jeong, Model collapse in the self-consuming chain of diffusion finetuning: A novel perspective from quantitative trait modeling. *Int. Conf. Learn. Represent.* (2025). https://doi.org/10.48550/arXiv.2407.17493. 26. M. McCloskey, N. J. Cohen, Catastrophic interference in connectionist networks: The sequential learning problem. *Psychol. Learn. Motiv.* **24**, 109–165 (1989). 27. R. M. French, Catastrophic forgetting in connectionist networks. *Trends Cogn. Sci.* **3**, 128–135 (1999). 28. T. Scialom, T. Chakrabarty, S. Muresan, Fine-tuned language models are continual learners. *Proc. Conf. Empir. Methods Nat. Lang. Process.* (2022). https://doi.org/10.48550/arXiv.2205.12393. diff --git a/paper/pnas/main.pdf b/paper/pnas/main.pdf index 96d106d..69eafc7 100644 Binary files a/paper/pnas/main.pdf and b/paper/pnas/main.pdf differ diff --git a/tasks/lessons.md b/tasks/lessons.md index c9eb709..11ad834 100644 --- a/tasks/lessons.md +++ b/tasks/lessons.md @@ -19,3 +19,12 @@ the SSH session's systemd scope and gets reaped on logout (lost ~20 min of the e "SSH disconnected"). Reliable pattern here: `systemd-run --user --collect --unit= --working-directory="$PWD" bash -c ''` — lands in user@.service (kept alive by the desktop session), survives disconnects; check with `systemctl --user is-active `, logs via redirect. + +**2026-08-11 — Internal deliberations must not leak into reader-facing prose.** GG (on the +convergence paragraph): it "carries all the ghosts of our internal brainstorming that should be +completely alien to the readers (this is, alas, a recurring problem with your writing)." The pattern: +strategic concerns from the working process (priority anxieties -> "we cite for priority of +publication"; calibration debates -> "the honest statement"; positioning battles -> "nobody has") +surface as defensive or self-referential prose. Rule: before finalising any reader-facing passage, +ask "does understanding this sentence require knowing how we worked?" If yes, rewrite it as a plain +statement about the literature or the result. Confident papers situate; they do not litigate.