Reproducibility pass: figure map, one-command reproduce.sh, notebooks, Makefile gaps
An audit of the figure pipeline found real sync gaps, now closed:
- `paper/pnas/make_figs.py` (which draws every manuscript figure) was invoked
by NO Makefile target or script - a manual step. Added `make paper-figures`.
- `configs/llm/epistasis{,_compat}.yaml` were reachable from nothing at all,
despite producing Fig. 3C-D. Added `make llm-epistasis` (+ its statistics).
- `make figures` never regenerated the MNIST montage that Fig. 2B embeds;
it now runs with the `mnist` target (it needs torch - it re-simulates).
- Added `make llm-society`, `env-notebooks`, `notebooks`.
New REPRODUCING.md is the authoritative map: every manuscript panel -> the
artifact it plots -> the config that produced it -> that config's seed, plus
the determinism policy (biological tier bitwise; GPU tiers statistical), the
seed-provenance statement, and an artifact-hash verification snippet. All 44
committed bundles currently hash-match their manifests, and figure
regeneration is pixel-identical (verified by comparison).
reproduce.sh delivers the one-command reproduction the paper's Methods
promises, writing REPRODUCED.md with recomputed hashes per bundle.
Two executed notebooks: 01 builds the Wright-Fisher model from scratch and
checks both closed forms interactively (runs in ~1 min on a laptop); 02
verifies artifact hashes then regenerates and displays all seven manuscript
figures. Both execute end-to-end (`make notebooks`).
Also pins `.python-version` to 3.14: the interpreter was previously
unpinned, and a `uv sync` silently switched it to 3.11 mid-session (see
tasks/lessons.md). README rewritten - it still described a Layer-1-only repo
of E1-E6 and pointed at a figure_manifest.md that does not exist.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkRLcc18rwT2Lysu6PbG7v
This commit is contained in:
parent
8086504f44
commit
c435cfba6e
18 changed files with 4204 additions and 28 deletions
201
REPRODUCING.md
Normal file
201
REPRODUCING.md
Normal file
|
|
@ -0,0 +1,201 @@
|
|||
# Reproducing every number and figure in the paper
|
||||
|
||||
This document is the authoritative map from the manuscript back to the code, configs, and seeds
|
||||
that produced it. Every figure panel, every headline number, and the environment they were computed
|
||||
in are listed below. If something in the paper is not traceable through this document, that is a
|
||||
bug — please open an issue.
|
||||
|
||||
Manuscript: `paper/pnas/main.md` (built to `paper/pnas/main.pdf`).
|
||||
|
||||
---
|
||||
|
||||
## 1. The three tiers, and what each costs to reproduce
|
||||
|
||||
| Tier | What it is | Hardware | Determinism |
|
||||
|---|---|---|---|
|
||||
| Biological model | Wright–Fisher simulator over knowledge distributions (pure NumPy/SciPy) | Any laptop, no GPU | **Bitwise** from the master seed |
|
||||
| Trained networks | RNN / MLP / VAE on a synthetic mode universe; convolutional VAE on MNIST | One consumer GPU (16 GB) | Statistical (GPU non-determinism documented in §5) |
|
||||
| Language models | LoRA specialists on Qwen2.5-Instruct 0.5B / 7B | 0.5B: one 16 GB GPU · 7B: one L40S (46 GB) | Statistical; per-seed points reported |
|
||||
|
||||
The biological-model tier carries every quantitative claim in the paper and reproduces exactly on a
|
||||
laptop in minutes. The two AI tiers are confirmatory (signs, not magnitudes) and need a GPU.
|
||||
|
||||
## 2. Environment
|
||||
|
||||
The environment is a `uv` venv built from the committed, hash-pinned `uv.lock`. That lockfile — not
|
||||
a container, not a requirements file — is the single source of truth for "it runs".
|
||||
|
||||
```bash
|
||||
curl -LsSf https://astral.sh/uv/install.sh | sh # one-time, if you don't have uv
|
||||
|
||||
make env # biological-model tier (pure NumPy/SciPy)
|
||||
make env-neural # + torch, for the trained-network tier
|
||||
make env-mnist # + torchvision, for the real-MNIST tier
|
||||
make env-llm # + transformers/peft, for the language-model tier
|
||||
make env-notebooks # + Jupyter, for the walkthrough notebooks
|
||||
```
|
||||
|
||||
## 3. One command
|
||||
|
||||
```bash
|
||||
./reproduce.sh
|
||||
```
|
||||
|
||||
runs the environment build, the test suite (including the closed-form scientific-validation tests),
|
||||
the entire biological-model tier at its committed seeds, every figure, and writes `REPRODUCED.md`
|
||||
with the resulting artifact hashes for comparison against the committed manifests. It deliberately
|
||||
stops at the GPU tiers; pass `--with-gpu` to include them if you have the hardware.
|
||||
|
||||
Tier by tier, by hand:
|
||||
|
||||
```bash
|
||||
make test # correctness + closed-form scientific validation
|
||||
make layer1 # the biological model: E1-E12, E14, learning kernel
|
||||
make neural # trained networks (needs a GPU)
|
||||
make mnist # real-MNIST tier + the Fig. 2B montage asset (needs torchvision)
|
||||
make llm # language-model prototypes (needs a GPU)
|
||||
make llm-seeds # the multi-seed firm-ups behind Fig. 3A
|
||||
make llm-epistasis # the controlled predictive test behind Fig. 3C-D
|
||||
make llm-speciation # the LLM speciation tier behind Fig. 7E-F
|
||||
make figures # per-experiment figures, from committed parquets (no re-simulation)
|
||||
make paper-figures # the manuscript figures Fig. 1-7 + rebuild the PDF body
|
||||
```
|
||||
|
||||
## 4. The figure map
|
||||
|
||||
Every manuscript panel, the artifact it is plotted from, the config that produced that artifact, and
|
||||
that config's declared seed. All panels are drawn by `paper/pnas/make_figs.py` (function per figure);
|
||||
`make paper-figures` regenerates all of them. Figures are a **pure function of committed artifacts** —
|
||||
no panel re-simulates anything, with the single documented exception of the Fig. 2B montage asset.
|
||||
|
||||
| Panel | Drawn by | From artifact | Produced by config | Seed(s) |
|
||||
|---|---|---|---|---|
|
||||
| Fig. 1A, 1B | `fig1a()`, `fig1b()` | — (schematics; icons in `paper/pnas/figs/icons/`) | — | — |
|
||||
| Fig. 2A | `fig2()` | `results/E2/` | `configs/layer1/E2.yaml` | 20260704 |
|
||||
| Fig. 2B | `fig2()` | `results/mnist_collapse/mnist_montage.png` | `configs/neural/mnist_collapse.yaml` → asset from `figures/mnist_montage.py` | 20260705 |
|
||||
| Fig. 3A | `fig3()` | `results/llm_merge_seeds/` | `configs/llm/merge_seeds.yaml` | 1, 2, 3, 4, 5 |
|
||||
| Fig. 3B | `fig3()` | `results/llm_moe_hard_hpc/` | `configs/llm/moe_hard_hpc.yaml` (7B, HPC) | 1 (single run) |
|
||||
| Fig. 3C, 3D | `fig3()` | `results/llm_epistasis/` + `results/llm_epistasis_compat/` | `configs/llm/epistasis.yaml`, `configs/llm/epistasis_compat.yaml` | 1, 2, 3 |
|
||||
| Fig. 4A | `fig4()` | `results/E4/` | `configs/layer1/E4.yaml` | 20260704 |
|
||||
| Fig. 4B | `fig4()` | `results/E8/` | `configs/layer1/E8.yaml` | 20260705 |
|
||||
| Fig. 5A | `fig5()` | `results/E9/` | `configs/layer1/E9.yaml` | 20260705 |
|
||||
| Fig. 5B | `fig5()` | `results/E10/` | `configs/layer1/E10.yaml` | 20260705 |
|
||||
| Fig. 5C, 5D | `fig5()` | `results/E14/` | `configs/layer1/E14.yaml` | 20260709 |
|
||||
| Fig. 6A, 6B, 6C | `fig6()` | `results/E11/` | `configs/layer1/E11.yaml` | 20260705 |
|
||||
| Fig. 7A, 7B | `fig7()` | `results/E12/` | `configs/layer1/E12.yaml` | 12 |
|
||||
| Fig. 7C | `fig7()` | `results/speciation_real/` | `configs/neural/speciation_real.yaml` | 13 |
|
||||
| Fig. 7D | `fig7()` | `results/speciation_real_cliff/` | `configs/neural/speciation_real_cliff.yaml` | 13 |
|
||||
| Fig. 7E, 7F | `fig7()` | `results/llm_speciation/` | `configs/llm/speciation.yaml` | 1 (single seed) |
|
||||
|
||||
**Single-run panels.** Fig. 3B and Fig. 7E–F come from single-seed runs and are reported as
|
||||
sign-level confirmations, not estimates; the manuscript labels them as such. Every other panel is
|
||||
replicated (biological-model panels over 12–100 internal replicates; Fig. 3A over five training
|
||||
seeds; Fig. 3C–D over three).
|
||||
|
||||
### Results reported in the text but not plotted in the manuscript
|
||||
|
||||
| Result | Artifact | Config | Seed |
|
||||
|---|---|---|---|
|
||||
| Collapse null (E1) | `results/E1/` | `configs/layer1/E1.yaml` | 20260704 |
|
||||
| Region-matched grounding (E3) | `results/E3/` | `configs/layer1/E3.yaml` | 20260704 |
|
||||
| Quality-diversity vs greedy (E5) | `results/E5/` | `configs/layer1/E5.yaml` | 20260704 |
|
||||
| Re-minting / irreversibility (E6) | `results/E6/` | `configs/layer1/E6.yaml` | 20260704 |
|
||||
| Advantage of sex, lineage (E7) | `results/E7/` | `configs/layer1/E7.yaml` | 20260705 |
|
||||
| Incompatibilities on NK (E12_nk) | `results/E12_nk/` | `configs/layer1/E12_nk.yaml` | 12 |
|
||||
| Learning kernel (estimator bias) | `results/kernel_sharpen/`, `results/kernel_smooth/` | `configs/layer1/kernel_{sharpen,smooth}.yaml` | 20260705 |
|
||||
| Histogram bridge gate | `results/bridge/` | `configs/neural/bridge.yaml` | 20260704 |
|
||||
| Neural collapse / grounding / architectures / recombination | `results/{collapse,grounding,architectures,recombination}/` | `configs/neural/*.yaml` | 20260704 |
|
||||
| Emergent-isolation null | `results/speciation_real_emergent/` | `configs/neural/speciation_real_emergent.yaml` | 813 |
|
||||
| Budget-controlled speciation (add design) | `results/llm_speciation_add/` | `configs/llm/speciation_add.yaml` | 1, 2, 3 |
|
||||
| LLM prototypes (0.5B) | `results/llm_{merge,moe,directed}/` | `configs/llm/{merge,moe,directed}.yaml` | 1 |
|
||||
| 7B firm-ups | `results/llm_*_hpc/` | `configs/llm/*_hpc.yaml` (run via `hpc/*.pbs`) | 1 |
|
||||
|
||||
### Per-experiment (exploratory) figures
|
||||
|
||||
`figures/plot_*.py` regenerate a diagnostic figure **inside each results bundle**
|
||||
(`results/<name>/<name>.pdf`), named after the experiment, not after a manuscript figure. They are
|
||||
the working views, not the manuscript's; the table above is the authority on what appears in the
|
||||
paper. `figures/stats_llm_epistasis.py` prints the robust statistics quoted in the predictive-test
|
||||
section (clustered bootstrap, paired contrasts, leave-one-condition-out, outcome-reference
|
||||
sensitivity).
|
||||
|
||||
## 5. Seeds and determinism
|
||||
|
||||
**Policy.** One master seed per config. All sub-randomness is derived from it via
|
||||
`numpy.random.SeedSequence.spawn` (`src/knowledge/seeding.py`); no code touches global RNG state, and
|
||||
every `rng` is passed explicitly. A run is a pure function of its resolved config.
|
||||
|
||||
**Biological-model tier: bitwise reproducible.** Re-running a config on the same lockfile
|
||||
reproduces its `results.parquet` byte-for-byte; the `results_sha256` in each `manifest.json` is the
|
||||
check.
|
||||
|
||||
**GPU tiers: statistically reproducible.** cuDNN kernel selection and reduction order make bitwise
|
||||
equality unattainable across machines. Per-seed points are reported rather than seed-averaged
|
||||
summaries alone, and the multi-seed protocols fix the evaluation sets and vary only the training
|
||||
seed. Expect sign agreement and magnitudes within noise, not identical digits.
|
||||
|
||||
**Seed provenance.** `20260704`/`20260705`/`20260709` are date-stamped master seeds chosen at the
|
||||
time each experiment was written and never re-drawn. Small integer seeds (`1`, `12`, `13`, `813`)
|
||||
are likewise fixed at authoring time. No seed in this repository was selected after seeing results.
|
||||
|
||||
## 6. Verifying artifact integrity
|
||||
|
||||
Every run writes three files next to its results:
|
||||
|
||||
- `results.parquet` — the long-form data (the only thing figures read)
|
||||
- `resolved_config.yaml` — the config **after** sweep expansion, i.e. exactly what ran
|
||||
- `manifest.json` — master seed, git commit, Python and library versions, row count, and
|
||||
`results_sha256` (the content hash of the parquet)
|
||||
|
||||
To verify a bundle you have regenerated matches the one behind the paper:
|
||||
|
||||
```bash
|
||||
python - <<'EOF'
|
||||
import hashlib, json, pathlib
|
||||
for m in sorted(pathlib.Path("results").glob("*/manifest.json")):
|
||||
man = json.loads(m.read_text())
|
||||
pq = m.parent / "results.parquet"
|
||||
if not pq.exists():
|
||||
print(f"{m.parent.name:28s} (no parquet — run its config first)"); continue
|
||||
got = hashlib.sha256(pq.read_bytes()).hexdigest()
|
||||
ok = "OK " if got == man.get("results_sha256") else "DIFF"
|
||||
print(f"{ok} {m.parent.name:28s} seed={man.get('master_seed')}")
|
||||
EOF
|
||||
```
|
||||
|
||||
`DIFF` on a biological-model bundle means a genuine discrepancy worth investigating. `DIFF` on a GPU
|
||||
tier is expected (see §5) — compare the figures and the reported statistics instead.
|
||||
|
||||
## 7. HPC (the 7B tier)
|
||||
|
||||
The 7B runs were executed on Imperial College's CX3 cluster (PBS Pro, one L40S 46 GB per job). Job
|
||||
scripts are in `hpc/`; each is self-contained and documents its own submission line. They stage the
|
||||
same `uv.lock` environment, so the only difference from a local run is the GPU.
|
||||
|
||||
```bash
|
||||
qsub hpc/llm_merge.pbs # 7B merge firm-up
|
||||
qsub hpc/llm_hard.pbs # hard-benchmark moe + directed at 7B
|
||||
qsub hpc/llm_society.pbs # the society campaign (array over seeds)
|
||||
```
|
||||
|
||||
## 8. Notebooks
|
||||
|
||||
`notebooks/` contains executable walkthroughs (`make env-notebooks`, then `jupyter lab`):
|
||||
|
||||
| Notebook | What it does | Needs |
|
||||
|---|---|---|
|
||||
| `01_biological_model.ipynb` | Builds the Wright–Fisher model from scratch, checks it against the three closed forms, and derives the grounding threshold interactively | Laptop |
|
||||
| `02_paper_figures.ipynb` | Regenerates every manuscript figure from the committed artifacts and displays them inline, panel by panel | Laptop (artifacts must be present) |
|
||||
|
||||
`make notebooks` executes both end-to-end, which is itself a reproduction check.
|
||||
|
||||
## 9. Known gaps
|
||||
|
||||
- `results/**/results.parquet` is currently **gitignored** (only manifests, hashes, and resolved
|
||||
configs are tracked). A fresh clone therefore has to re-run the experiments before figures can be
|
||||
regenerated. The archived deposit (Zenodo DOI, on publication) includes the parquets so that the
|
||||
paper's "regenerates from committed artifacts without re-simulation" holds from the archive.
|
||||
- `figures/mnist_montage.py` re-runs a short dry lineage to draw its montage rather than reading a
|
||||
parquet; it is an eyeball diagnostic whose quantitative counterpart is `results/mnist_collapse/`.
|
||||
- The composed society at language-model scale is an open experiment at the time of writing; see
|
||||
`tasks/workorder-llm-society.md`.
|
||||
Loading…
Add table
Add a link
Reference in a new issue