Giorgio Gilestro
c435cfba6e
An audit of the figure pipeline found real sync gaps, now closed:
- `paper/pnas/make_figs.py` (which draws every manuscript figure) was invoked
by NO Makefile target or script - a manual step. Added `make paper-figures`.
- `configs/llm/epistasis{,_compat}.yaml` were reachable from nothing at all,
despite producing Fig. 3C-D. Added `make llm-epistasis` (+ its statistics).
- `make figures` never regenerated the MNIST montage that Fig. 2B embeds;
it now runs with the `mnist` target (it needs torch - it re-simulates).
- Added `make llm-society`, `env-notebooks`, `notebooks`.
New REPRODUCING.md is the authoritative map: every manuscript panel -> the
artifact it plots -> the config that produced it -> that config's seed, plus
the determinism policy (biological tier bitwise; GPU tiers statistical), the
seed-provenance statement, and an artifact-hash verification snippet. All 44
committed bundles currently hash-match their manifests, and figure
regeneration is pixel-identical (verified by comparison).
reproduce.sh delivers the one-command reproduction the paper's Methods
promises, writing REPRODUCED.md with recomputed hashes per bundle.
Two executed notebooks: 01 builds the Wright-Fisher model from scratch and
checks both closed forms interactively (runs in ~1 min on a laptop); 02
verifies artifact hashes then regenerates and displays all seven manuscript
figures. Both execute end-to-end (`make notebooks`).
Also pins `.python-version` to 3.14: the interpreter was previously
unpinned, and a `uv sync` silently switched it to 3.11 mid-session (see
tasks/lessons.md). README rewritten - it still described a Layer-1-only repo
of E1-E6 and pointed at a figure_manifest.md that does not exist.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkRLcc18rwT2Lysu6PbG7v
88 lines
4.6 KiB
Markdown
88 lines
4.6 KiB
Markdown
# The evolution of sex for artificial intelligence
|
||
|
||
A population-genetic framework for multigenerational model populations. Knowledge transmission
|
||
between generations of learning agents is modelled *literally* as a Wright–Fisher process, not by
|
||
analogy: a model's knowledge is a distribution `p_t` over `K` discrete items, a fixed true
|
||
distribution `p*` has a rare tail, and each generational step is "sample from the parent (drift) +
|
||
mix in fresh real samples (grounding/immigration) + refit". Model collapse is the loss of rare
|
||
alleles under drift — and the remedies population genetics knows for drift (immigration,
|
||
recombination, selection, population structure) become engineering levers for model populations.
|
||
|
||
The framework is developed at three tiers of increasing realism:
|
||
|
||
| Tier | What it is | Hardware |
|
||
|---|---|---|
|
||
| **Biological model** | Wright–Fisher simulator over knowledge distributions; closed forms, bitwise reproducible | laptop |
|
||
| **Trained networks** | RNN / MLP / VAE on a synthetic mode universe with an exact oracle; convolutional VAE on MNIST | one GPU |
|
||
| **Language models** | LoRA specialists on Qwen2.5-Instruct (0.5B / 7B) with an exact-match verifier | one GPU / L40S |
|
||
|
||
## Reproduce
|
||
|
||
**Start here: [`REPRODUCING.md`](REPRODUCING.md)** — the authoritative map from every manuscript
|
||
figure panel back to the artifact, config, and seed that produced it, plus the determinism policy
|
||
and artifact-hash verification.
|
||
|
||
```bash
|
||
curl -LsSf https://astral.sh/uv/install.sh | sh # one-time, if needed
|
||
|
||
./reproduce.sh # env -> tests -> biological-model tier at committed seeds -> figures
|
||
./reproduce.sh --with-gpu # ... and the trained-network + language-model tiers
|
||
```
|
||
|
||
Or tier by tier:
|
||
|
||
```bash
|
||
make env # build .venv from the committed, hash-pinned uv.lock
|
||
make test # correctness + closed-form scientific validation (the spine of trust)
|
||
make layer1 # the biological model: E1-E12, E14, learning kernel
|
||
make figures # per-experiment figures, from committed parquets (no re-simulation)
|
||
make paper-figures # the manuscript's Fig. 1-7 + rebuild the PDF body
|
||
```
|
||
|
||
`make help` is not defined, but every target carries a `##` description — `grep '##' Makefile`.
|
||
|
||
## Notebooks
|
||
|
||
```bash
|
||
make env-notebooks && jupyter lab notebooks/
|
||
```
|
||
|
||
- `01_biological_model.ipynb` — builds the Wright–Fisher model from scratch, checks it against the
|
||
closed forms (geometric diversity decay, the immigration–drift equilibrium), and derives the
|
||
grounding threshold and its per-item observation floor. Runs on a laptop in under a minute.
|
||
- `02_paper_figures.ipynb` — verifies artifact hashes, then regenerates and displays every
|
||
manuscript figure from the committed artifacts.
|
||
|
||
## Layout
|
||
|
||
```
|
||
src/knowledge/ biological-model tier (imported as `knowledge`)
|
||
src/neural/ trained-network tier
|
||
src/llm/ language-model tier
|
||
configs/ one YAML per experiment: layer1/ neural/ llm/ (each declares its master seed)
|
||
figures/ plot_*.py — per-experiment diagnostics, read results.parquet only
|
||
paper/pnas/ the manuscript: main.md, make_figs.py (Fig. 1-7), build.py, si.md
|
||
notebooks/ executable walkthroughs
|
||
hpc/ PBS job scripts for the 7B tier (Imperial CX3)
|
||
tests/ correctness + test_scientific_validation.py (the closed forms as assertions)
|
||
results/ run artifacts: results.parquet (gitignored) + resolved_config.yaml + manifest.json
|
||
```
|
||
|
||
Design documents: `paper/blueprint.md` (the normative build spec) and `paper/results-summary.md`
|
||
(plain-language + technical summary of every result).
|
||
|
||
## The engineering contract
|
||
|
||
- **Reproducibility is a requirement, not a preference.** The environment is a `uv` venv built from
|
||
a committed, hash-pinned `uv.lock`; the biological-model tier is bitwise reproducible from a
|
||
single master seed, and the GPU tiers are statistically reproducible with per-seed points
|
||
reported.
|
||
- **One master seed per config**, with all sub-randomness derived via `SeedSequence.spawn`. No code
|
||
touches global RNG state; a run is a pure function of its resolved config.
|
||
- **No magic numbers in code.** Every parameter lives in a YAML resolved at run time, and the
|
||
resolved config is written next to the results.
|
||
- **Every run writes the same triple:** `results.parquet` + `resolved_config.yaml` +
|
||
`manifest.json` (seed, git commit, library versions, row count, SHA-256 of the results).
|
||
- **Every figure is a pure function of a committed artifact** — figure scripts never re-simulate.
|
||
- **The scientific-validation tests are the spine of trust.** They assert that the simulator
|
||
reproduces the closed forms to within 0.5%. If they fail, the science is wrong, not just the code.
|