MachineSex/REPRODUCING.md

# Reproducing every number and figure in the paper

This document is the authoritative map from the manuscript back to the code, configs, and seeds
that produced it. Every figure panel, every headline number, and the environment they were computed
in are listed below. If something in the paper is not traceable through this document, that is a
bug — please open an issue.

Manuscript: `paper/pnas/main.md` (built to `paper/pnas/main.pdf`).

---

## 1. The three tiers, and what each costs to reproduce

| Tier | What it is | Hardware | Determinism |
|---|---|---|---|
| Inheritance model | Wright–Fisher simulator over knowledge distributions (pure NumPy/SciPy) | Any laptop, no GPU | **Bitwise** from the master seed |
| Trained networks | RNN / MLP / VAE on a synthetic mode universe; convolutional VAE on MNIST | One consumer GPU (16 GB) | Statistical (GPU non-determinism documented in §5) |
| Language models | LoRA specialists on Qwen2.5-Instruct 0.5B / 7B | 0.5B: one 16 GB GPU · 7B: one L40S (46 GB) | Statistical; per-seed points reported |

The inheritance-model tier carries every quantitative claim in the paper and reproduces exactly on a
laptop in minutes. The two AI tiers are confirmatory (signs, not magnitudes) and need a GPU.

## 2. Environment

The environment is a `uv` venv built from the committed, hash-pinned `uv.lock`. That lockfile — not
a container, not a requirements file — is the single source of truth for "it runs".

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh   # one-time, if you don't have uv

make env             # biological-model tier (pure NumPy/SciPy)
make env-neural      # + torch, for the trained-network tier
make env-mnist       # + torchvision, for the real-MNIST tier
make env-llm         # + transformers/peft, for the language-model tier
make env-notebooks   # + Jupyter, for the walkthrough notebooks
```

## 3. One command

```bash
./reproduce.sh
```

runs the environment build, the test suite (including the closed-form scientific-validation tests),
the entire biological-model tier at its committed seeds, every figure, and writes `REPRODUCED.md`
with the resulting artifact hashes for comparison against the committed manifests. It deliberately
stops at the GPU tiers; pass `--with-gpu` to include them if you have the hardware.

Tier by tier, by hand:

```bash
make test            # correctness + closed-form scientific validation
make layer1          # the inheritance model: E1-E12, E14, learning kernel
make neural          # trained networks (needs a GPU)
make mnist           # real-MNIST tier + the Fig. 2A montage asset (needs torchvision)
make llm             # language-model prototypes (needs a GPU)
make llm-seeds       # the multi-seed firm-ups behind Fig. 3B
make llm-epistasis   # the controlled predictive test behind Fig. 3D-E
make llm-speciation  # the LLM speciation tier behind Fig. 5C-D
make figures         # per-experiment figures, from committed parquets (no re-simulation)
make paper-figures   # the manuscript figures Fig. 1-5 + rebuild the PDF body
```

## 4. The figure map

Every manuscript panel, the artifact it is plotted from, the config that produced that artifact, and
that config's declared seed. All panels are drawn by `paper/pnas/make_figs.py` (function per figure);
`make paper-figures` regenerates all of them. Figures are a **pure function of committed artifacts** —
no panel re-simulates anything, with the single documented exception of the Fig. 2A montage asset.

| Panel | Drawn by | From artifact | Produced by config | Seed(s) |
|---|---|---|---|---|
| Fig. 1A, 1B | `fig1a()`, `fig1b()` | — (schematics; icons in `paper/pnas/figs/icons/`) | — | — |
| Fig. 2A | `fig2()` | `results/mnist_collapse/mnist_montage.png` | `configs/neural/mnist_collapse.yaml` → asset from `figures/mnist_montage.py` | 20260705 |
| Fig. 2B *(reference)* | `fig2()` | `results/E2/` | `configs/layer1/E2.yaml` | 20260704 |
| Fig. 3A | `fig3()` | — (schematic of the compared models; icons in `paper/pnas/figs/icons/`) | — | — |
| Fig. 3B | `fig3()` | `results/llm_merge_seeds/` | `configs/llm/merge_seeds.yaml` | 1, 2, 3, 4, 5 |
| Fig. 3C | `fig3()` | `results/llm_moe_hard_hpc/s{1,2,3}/` | `configs/llm/moe_hard_hpc.yaml` (7B, HPC; seeds 2–3 via `hpc/llm_7b_seeds.pbs`); statistics `figures/stats_llm_7b_seeds.py` | 1, 2, 3 |
| Fig. 3D, 3D | `fig3()` | `results/llm_epistasis/` + `results/llm_epistasis_compat/` | `configs/llm/epistasis.yaml`, `configs/llm/epistasis_compat.yaml` | 1, 2, 3 |
| Fig. 4A | `fig4()` | — (schematic; the syllabus grid is read from `configs/llm/curriculum_v5_s1.yaml` family order, complementarity from the veto bundle) | — | — |
| Fig. 4B | `fig4()` | `results/llm_curriculum_v5/`, `results/llm_curriculum_v5_veto/`, `results/llm_curriculum_v5_stop3/` | `configs/llm/curriculum_v5_s1.yaml` (seeds 2–3 via `hpc/llm_curriculum.pbs`), `configs/llm/curriculum_v5_veto.yaml` (seeds 2–3 via `hpc/llm_veto.pbs`), `configs/llm/curriculum_v5_stop3.yaml` (seeds 2–3 via `hpc/llm_curriculum_controls.pbs`) | 1–3 (training seeds; fixed test splits) |
| Fig. 4C | `fig4()` | `results/llm_curriculum_v5_veto/`, `results/llm_curriculum_v5_decor/` | `configs/llm/curriculum_v5_veto.yaml`, `configs/llm/curriculum_v5_decor.yaml` (seeds 2–3 via `hpc/llm_curriculum_controls.pbs`); statistics `figures/stats_llm_curriculum.py` | 1–3 |
| Fig. S14 | `figures/plot_curriculum_timing.py` | `results/llm_curriculum_v5_{early,late,early_obl,late_obl}/s{1,2,3}/` | `configs/llm/curriculum_v5_{early,late}[_obl].yaml` via `hpc/llm_curriculum_timing.pbs`; statistics `figures/stats_llm_curriculum.py` | 1, 2, 3 |
| Fig. S15 | `figures/plot_curriculum_cull.py` | `results/llm_curriculum_v5_cull/s{1,2,3}/` | `configs/llm/curriculum_v5_cull.yaml` via `hpc/llm_cull.pbs`; statistics `figures/stats_llm_curriculum.py` | 1, 2, 3 |
| Fig. S16 | `figures/plot_llm_smol.py` | `results/llm_{merge_seeds,moe_hard_seeds}_smol/` (+ the Qwen bundles) | `configs/llm/{merge_seeds,moe_hard_seeds}_smol.yaml` via `hpc/llm_smol.pbs`; statistics `figures/stats_llm_smol.py` | 1–5; 1–3 |
| Fig. 4D, 4D, 4E *(reference)* | `fig4()` | `results/E11/` | `configs/layer1/E11.yaml` | 20260705 |
| Fig. 5A | `fig5()` | `results/speciation_real/` | `configs/neural/speciation_real.yaml` | 13 |
| Fig. 5B | `fig5()` | `results/speciation_real_cliff/` | `configs/neural/speciation_real_cliff.yaml` | 13 |
| Fig. 5C, 5D | `fig5()` | `results/llm_speciation/s{1,2,3}/` | `configs/llm/speciation.yaml` (seeds 2–3 via `hpc/llm_speciation_seeds.pbs`); statistics `figures/stats_llm_speciation_seeds.py` | 1, 2, 3 |
| Fig. 5E, 5F *(reference)* | `fig5()` | `results/E12/` | `configs/layer1/E12.yaml` | 12 |

Panels marked *(reference)* are inheritance-model results included to set the expectation the
real-model panels are read against, not as findings in their own right.

**Inheritance-model results reported in SI only.** These have no real-model counterpart in the paper
and reproduce known results, so they are cited as reference values and plotted only in SI, from their
per-experiment figures: blending vs union (E4), Fisher–Muller super-parent (E8), outbreeding
depression (E9), directed recombination (E10), and mate-pool breadth (E14), drawn by
`figures/plot_E{4,8,9,10,14}.py` from `results/E{4,8,9,10,14}/`.

**Replication.** Every panel is replicated: inheritance-model panels over 12–100 internal
replicates; Fig. 3B over five training seeds; Fig. 3C, 3D–E, 4A–B and 5C–D over three (Fig. 5C–D
was single-seed until 2026-09-12; seeds 2–3 ran via `hpc/llm_speciation_seeds.pbs`).

### Results reported in the text but not plotted in the manuscript

| Result | Artifact | Config | Seed |
|---|---|---|---|
| Collapse null (E1) | `results/E1/` | `configs/layer1/E1.yaml` | 20260704 |
| Region-matched grounding (E3) | `results/E3/` | `configs/layer1/E3.yaml` | 20260704 |
| Quality-diversity vs greedy (E5) | `results/E5/` | `configs/layer1/E5.yaml` | 20260704 |
| Re-minting / irreversibility (E6) | `results/E6/` | `configs/layer1/E6.yaml` | 20260704 |
| Advantage of sex, lineage (E7) | `results/E7/` | `configs/layer1/E7.yaml` | 20260705 |
| Incompatibilities on NK (E12_nk) | `results/E12_nk/` | `configs/layer1/E12_nk.yaml` | 12 |
| Learning kernel (estimator bias) | `results/kernel_sharpen/`, `results/kernel_smooth/` | `configs/layer1/kernel_{sharpen,smooth}.yaml` | 20260705 |
| Histogram bridge gate | `results/bridge/` | `configs/neural/bridge.yaml` | 20260704 |
| Neural collapse / grounding / architectures / recombination | `results/{collapse,grounding,architectures,recombination}/` | `configs/neural/*.yaml` | 20260704 |
| Emergent-isolation null | `results/speciation_real_emergent/` | `configs/neural/speciation_real_emergent.yaml` | 813 |
| Budget-controlled speciation (add design) | `results/llm_speciation_add/` | `configs/llm/speciation_add.yaml` | 1, 2, 3 |
| LLM prototypes (0.5B) | `results/llm_{merge,moe,directed}/` | `configs/llm/{merge,moe,directed}.yaml` | 1 |
| 7B firm-ups | `results/llm_*_hpc/` | `configs/llm/*_hpc.yaml` (run via `hpc/*.pbs`) | 1 |

### Per-experiment (exploratory) figures

`figures/plot_*.py` regenerate a diagnostic figure **inside each results bundle**
(`results/<name>/<name>.pdf`), named after the experiment, not after a manuscript figure. They are
the working views, not the manuscript's; the table above is the authority on what appears in the
paper. `figures/stats_llm_epistasis.py` prints the robust statistics quoted in the predictive-test
section (clustered bootstrap, paired contrasts, leave-one-condition-out, outcome-reference
sensitivity).

## 5. Seeds and determinism

**Policy.** One master seed per config. All sub-randomness is derived from it via
`numpy.random.SeedSequence.spawn` (`src/knowledge/seeding.py`); no code touches global RNG state, and
every `rng` is passed explicitly. A run is a pure function of its resolved config.

**Biological-model tier: bitwise reproducible.** Re-running a config on the same lockfile
reproduces its `results.parquet` byte-for-byte; the `results_sha256` in each `manifest.json` is the
check.

**GPU tiers: statistically reproducible.** cuDNN kernel selection and reduction order make bitwise
equality unattainable across machines. Per-seed points are reported rather than seed-averaged
summaries alone, and the multi-seed protocols fix the evaluation sets and vary only the training
seed. Expect sign agreement and magnitudes within noise, not identical digits.

**Seed provenance.** `20260704`/`20260705`/`20260709` are date-stamped master seeds chosen at the
time each experiment was written and never re-drawn. Small integer seeds (`1`, `12`, `13`, `813`)
are likewise fixed at authoring time. No seed in this repository was selected after seeing results.

## 6. Verifying artifact integrity

Every run writes three files next to its results:

- `results.parquet` — the long-form data (the only thing figures read)
- `resolved_config.yaml` — the config **after** sweep expansion, i.e. exactly what ran
- `manifest.json` — master seed, git commit, Python and library versions, row count, and
  `results_sha256` (the content hash of the parquet)

To verify a bundle you have regenerated matches the one behind the paper:

```bash
python - <<'EOF'
import hashlib, json, pathlib
for m in sorted(pathlib.Path("results").glob("*/manifest.json")):
    man = json.loads(m.read_text())
    pq = m.parent / "results.parquet"
    if not pq.exists():
        print(f"{m.parent.name:28s} (no parquet — run its config first)"); continue
    got = hashlib.sha256(pq.read_bytes()).hexdigest()
    ok = "OK  " if got == man.get("results_sha256") else "DIFF"
    print(f"{ok} {m.parent.name:28s} seed={man.get('master_seed')}")
EOF
```

`DIFF` on a biological-model bundle means a genuine discrepancy worth investigating. `DIFF` on a GPU
tier is expected (see §5) — compare the figures and the reported statistics instead.

## 7. HPC (the 7B tier)

The 7B runs were executed on Imperial College's CX3 cluster (PBS Pro, one L40S 46 GB per job). Job
scripts are in `hpc/`; each is self-contained and documents its own submission line. They stage the
same `uv.lock` environment, so the only difference from a local run is the GPU.

```bash
qsub hpc/llm_merge.pbs      # 7B merge firm-up
qsub hpc/llm_hard.pbs       # hard-benchmark moe + directed at 7B
qsub hpc/llm_society.pbs    # the society campaign (array over seeds)
```

## 8. Notebooks

`notebooks/` contains executable walkthroughs (`make env-notebooks`, then `jupyter lab`):

| Notebook | What it does | Needs |
|---|---|---|
| `01_biological_model.ipynb` | Builds the Wright–Fisher model from scratch, checks it against the three closed forms, and derives the grounding threshold interactively | Laptop |
| `02_paper_figures.ipynb` | Regenerates every manuscript figure from the committed artifacts and displays them inline, panel by panel | Laptop (artifacts must be present) |

`make notebooks` executes both end-to-end, which is itself a reproduction check.

## 9. Known gaps

- `results/**/results.parquet` is currently **gitignored** (only manifests, hashes, and resolved
  configs are tracked). A fresh clone therefore has to re-run the experiments before figures can be
  regenerated. The archived deposit (Zenodo DOI, on publication) includes the parquets so that the
  paper's "regenerates from committed artifacts without re-simulation" holds from the archive.
- `figures/mnist_montage.py` re-runs a short dry lineage to draw its montage rather than reading a
  parquet; it is an eyeball diagnostic whose quantitative counterpart is `results/mnist_collapse/`.
- The composed society at language-model scale is an open experiment at the time of writing; see
  `tasks/workorder-llm-society.md`.