History

Giorgio Gilestro 6f8cef1ac5 main: keep only what reproduces the manuscript; everything else lives on dev Removed from main (all preserved on the dev branch): the arXiv build and its sources, design documents (blueprint, results summary, review responses, essay drafts), tasks/ and CLAUDE.md, the cover letter and reference tooling, two unused manuscript figures, and every experiment that feeds no figure or number in the paper: the collapse null, the sexual-vs-asexual lineage, the NK speciation variant, the 0.5B single-seed LLM prototypes, the compose and society experiments with their calibration and pilot runs, and their configs, runners, tests, figure scripts and PBS jobs. Their result bundles are moved to results/_archive/ (ignored) so the parquets stay on disk. Also: plot_llm_speciation reads the s{seed}/ layout; the mating-breadth plot writes under its bundle name; Makefile targets reduced to the kept experiments; REPRODUCING.md and README point to dev for the rest. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y64o8FKP7rCuXzC48pxpMm		2026-09-13 17:07:23 +01:00
..
fig2_grounding_sweep.pdf	main: keep only what reproduces the manuscript; everything else lives on dev	2026-09-13 17:07:23 +01:00
fig2_grounding_sweep.png	Restructure: descriptive tier and experiment names, paper/manuscript	2026-09-13 17:00:40 +01:00
manifest.json	Restructure: descriptive tier and experiment names, paper/manuscript	2026-09-13 17:00:40 +01:00
README.md	Restructure: descriptive tier and experiment names, paper/manuscript	2026-09-13 17:00:40 +01:00
resolved_config.yaml	Restructure: descriptive tier and experiment names, paper/manuscript	2026-09-13 17:00:40 +01:00

README.md

E2 — A tiny dose of real data rescues diversity (the headline)

Claim tested: how much real data must you mix back in each generation to stop collapse — a lot, or a little?

Setup (Layer 1, pure math). K = 1000 items, Zipf truth p*, n = 200 inherited samples per generation, 500 generations, 100 repeats. Each generation we also mix in m fresh real samples drawn from p*. The knob swept is the grounding fraction g = m/(n+m) — the share of the training pool that is real — across g ∈ {0, 0.005, 0.01, 0.02, 0.05, 0.1, 0.2, 0.4}.

Symbols

g grounding fraction (share of real data); g* the critical value that restores diversity.
p* truth, p_t model, H diversity, H* truth's diversity.
grounding = "immigration" in the genetics analogy: real samples are migrants that re-introduce alleles drift keeps killing.

The four panels

Trajectories. Diversity H over generations, one line per g (dark = dry, bright = more grounding). g = 0 slides toward 0; any g > 0 levels off on a plateau — the collapse is arrested.
The phase boundary (the headline). Dots = stationary diversity vs g; the black dashed curve is the exact closed-form equilibrium H_eq; the red line marks the critical g* ≈ 0.048 (95% CI [0.047, 0.050]) where H reaches 95% of the truth's diversity. Only ~5% real data buys back essentially all the diversity. The g = 0 point is drawn hollow (it is still sliding — its true equilibrium is 0).
Tail coverage. Fraction of the rare tail retained, by item-count (red) and truth-mass-weighted (purple). Both rise with g but stay modest at feasible grounding: a little grounding restores diversity long before it restores the deep tail — which motivates E4 (recombination) and E6.
Per-rarity band. The tail split into rarity bands (bright = shallowest, dark = deepest). Deep bands lag: an item survives only once m·p*ᵢ ≳ 1 (enough real samples per generation to land it at least once). The sharp threshold lives here, in discrete item survival — not in the smooth H.

Takeaway

There is a critical grounding fraction g* ≪ 1: a small, constant trickle of reality indefinitely holds off collapse. Falsifier (not triggered): if diversity had only recovered as g → 1, the practical thesis would die. It recovers at g ≈ 0.05.