giorgio/MachineSex
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Reproducibility pass: figure map, one-command reproduce.sh, notebooks, Makefile gaps

Browse source 
An audit of the figure pipeline found real sync gaps, now closed:

- `paper/pnas/make_figs.py` (which draws every manuscript figure) was invoked
  by NO Makefile target or script - a manual step. Added `make paper-figures`.
- `configs/llm/epistasis{,_compat}.yaml` were reachable from nothing at all,
  despite producing Fig. 3C-D. Added `make llm-epistasis` (+ its statistics).
- `make figures` never regenerated the MNIST montage that Fig. 2B embeds;
  it now runs with the `mnist` target (it needs torch - it re-simulates).
- Added `make llm-society`, `env-notebooks`, `notebooks`.

New REPRODUCING.md is the authoritative map: every manuscript panel -> the
artifact it plots -> the config that produced it -> that config's seed, plus
the determinism policy (biological tier bitwise; GPU tiers statistical), the
seed-provenance statement, and an artifact-hash verification snippet. All 44
committed bundles currently hash-match their manifests, and figure
regeneration is pixel-identical (verified by comparison).

reproduce.sh delivers the one-command reproduction the paper's Methods
promises, writing REPRODUCED.md with recomputed hashes per bundle.

Two executed notebooks: 01 builds the Wright-Fisher model from scratch and
checks both closed forms interactively (runs in ~1 min on a laptop); 02
verifies artifact hashes then regenerates and displays all seven manuscript
figures. Both execute end-to-end (`make notebooks`).

Also pins `.python-version` to 3.14: the interpreter was previously
unpinned, and a `uv sync` silently switched it to 3.11 mid-session (see
tasks/lessons.md). README rewritten - it still described a Layer-1-only repo
of E1-E6 and pointed at a figure_manifest.md that does not exist.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkRLcc18rwT2Lysu6PbG7v
This commit is contained in:
Giorgio Gilestro 2026-09-07 15:50:50 +01:00
parent 8086504f44
commit c435cfba6e
18 changed files with 4204 additions and 28 deletions
Download patch file Download diff file Expand all files Collapse all files

1
.python-version Normal file
View file

@ -0,0 +1 @@
3.14

25
Makefile
View file

@ -1,7 +1,8 @@
# Layer 1 + Layer 1.5 automation. The uv venv (built from the committed uv.lock) is the # Layer 1 + Layer 1.5 automation. The uv venv (built from the committed uv.lock) is the
# reproducibility source of truth; every target runs inside it via `uv run`. # reproducibility source of truth; every target runs inside it via `uv run`.
.PHONY: env env-neural env-mnist env-llm test layer1 layer2 neural mnist llm figures clean .PHONY: env env-neural env-mnist env-llm env-notebooks test layer1 layer2 neural mnist llm \
llm-epistasis llm-society figures paper-figures notebooks clean
env: ## build .venv from the committed lockfile env: ## build .venv from the committed lockfile
uv sync --extra dev uv sync --extra dev
@ -27,6 +28,7 @@ mnist: ## run the torchvision tiers: MNIST collapse + E13 real-weigh
uv run python -m neural.experiment configs/neural/speciation_real.yaml uv run python -m neural.experiment configs/neural/speciation_real.yaml
uv run python -m neural.experiment configs/neural/speciation_real_cliff.yaml uv run python -m neural.experiment configs/neural/speciation_real_cliff.yaml
uv run python -m neural.experiment configs/neural/speciation_real_emergent.yaml uv run python -m neural.experiment configs/neural/speciation_real_emergent.yaml
MPLBACKEND=Agg uv run python figures/mnist_montage.py # the asset paper Fig. 2B embeds
env-llm: ## add the LLM stack for the Layer-2 prototype (GPU; transformers/peft) env-llm: ## add the LLM stack for the Layer-2 prototype (GPU; transformers/peft)
uv sync --extra dev --extra neural --extra llm uv sync --extra dev --extra neural --extra llm
@ -40,6 +42,14 @@ llm-speciation: ## LLM-tier speciation: conflict cliff (replace + de-confound
uv run python -m llm.experiment configs/llm/speciation.yaml uv run python -m llm.experiment configs/llm/speciation.yaml
uv run python -m llm.experiment configs/llm/speciation_add.yaml uv run python -m llm.experiment configs/llm/speciation_add.yaml
llm-epistasis: ## the controlled predictive test (feeds paper Fig. 3C-D) + its robust statistics
uv run python -m llm.experiment configs/llm/epistasis.yaml
uv run python -m llm.experiment configs/llm/epistasis_compat.yaml
uv run python figures/stats_llm_epistasis.py
llm-society: ## the composed society at LLM scale (C3): pilot; the campaign runs on HPC
uv run python -m llm.experiment configs/llm/society.yaml
llm-seeds: ## multi-seed firm-up (heavy): merge x5, moe-hard x3, directed-hard x3 llm-seeds: ## multi-seed firm-up (heavy): merge x5, moe-hard x3, directed-hard x3
uv run python -m llm.experiment configs/llm/merge_seeds.yaml uv run python -m llm.experiment configs/llm/merge_seeds.yaml
uv run python -m llm.experiment configs/llm/moe_hard_seeds.yaml uv run python -m llm.experiment configs/llm/moe_hard_seeds.yaml
@ -47,11 +57,22 @@ llm-seeds: ## multi-seed firm-up (heavy): merge x5, moe-hard x3, directe
layer2: neural ## alias: Layer 1.5 is the current Layer-2 deliverable (LLM rung deferred) layer2: neural ## alias: Layer 1.5 is the current Layer-2 deliverable (LLM rung deferred)
figures: ## regenerate figures from committed results figures: ## regenerate per-experiment figures from committed results (pure; no re-simulation)
for e in E1 E2 E3 E4 E5 E6; do MPLBACKEND=Agg uv run python figures/plot_$$e.py; done for e in E1 E2 E3 E4 E5 E6; do MPLBACKEND=Agg uv run python figures/plot_$$e.py; done
for p in figures/plot_*.py; do case "$$p" in */plot_E[1-6].py|*/_*) ;; \ for p in figures/plot_*.py; do case "$$p" in */plot_E[1-6].py|*/_*) ;; \
*) [ -e "$$p" ] && MPLBACKEND=Agg uv run python "$$p" ;; esac; done *) [ -e "$$p" ] && MPLBACKEND=Agg uv run python "$$p" ;; esac; done
paper-figures: ## regenerate the manuscript figures (Fig. 1-7) and rebuild the PDF body
MPLBACKEND=Agg uv run python paper/pnas/make_figs.py
uv run python paper/pnas/build.py
env-notebooks: ## add Jupyter for the walkthrough notebooks
uv sync --extra dev --extra notebooks
notebooks: ## execute every notebook end-to-end (a reproduction check in itself)
for nb in notebooks/*.ipynb; do uv run jupyter nbconvert --to notebook --execute \
--inplace --ExecutePreprocessor.timeout=1800 "$$nb"; done
clean: ## remove caches and generated results (keeps committed manifests) clean: ## remove caches and generated results (keeps committed manifests)
rm -rf .pytest_cache **/__pycache__ rm -rf .pytest_cache **/__pycache__
find results -type f ! -name '.gitkeep' -delete 2>/dev/null || true find results -type f ! -name '.gitkeep' -delete 2>/dev/null || true

99
README.md
View file

@ -1,39 +1,88 @@
# The Lamarckian Society — Layer 1 (analytical core) # The evolution of sex for artificial intelligence
A parametric population-genetics model of knowledge transmission across generations of A population-genetic framework for multigenerational model populations. Knowledge transmission
learning agents. Knowledge transmission is modelled *literally* as a Wright–Fisher between generations of learning agents is modelled *literally* as a Wright–Fisher process, not by
process (not by analogy): a model's knowledge is a distribution `p_t` over `K` discrete analogy: a model's knowledge is a distribution `p_t` over `K` discrete items, a fixed true
items; a fixed true distribution `p*` has a rare tail; each generational step is distribution `p*` has a rare tail, and each generational step is "sample from the parent (drift) +
"sample from the parent (drift) + mix in fresh real samples (grounding/immigration) + mix in fresh real samples (grounding/immigration) + refit". Model collapse is the loss of rare
refit." Model collapse is the loss of rare alleles under drift. alleles under drift — and the remedies population genetics knows for drift (immigration,
recombination, selection, population structure) become engineering levers for model populations.
See `paper/blueprint.md` (the normative build spec), The framework is developed at three tiers of increasing realism:
`paper/the-lamarckian-society-v5.md` (the perspective paper), and
`paper/results-summary.md` (a summary of all results). | Tier | What it is | Hardware |
|---|---|---|
| **Biological model** | Wright–Fisher simulator over knowledge distributions; closed forms, bitwise reproducible | laptop |
| **Trained networks** | RNN / MLP / VAE on a synthetic mode universe with an exact oracle; convolutional VAE on MNIST | one GPU |
| **Language models** | LoRA specialists on Qwen2.5-Instruct (0.5B / 7B) with an exact-match verifier | one GPU / L40S |
## Reproduce ## Reproduce
Environment is a `uv` venv built from the committed, hash-pinned `uv.lock` — that **Start here: [`REPRODUCING.md`](REPRODUCING.md)** — the authoritative map from every manuscript
lockfile is the single source of truth for "it runs" (Layer 1 is pure NumPy/SciPy and figure panel back to the artifact, config, and seed that produced it, plus the determinism policy
bitwise-reproducible from a seed; no container needed). and artifact-hash verification.
```bash ```bash
# one-time: install uv (https://astral.sh/uv) curl -LsSf https://astral.sh/uv/install.sh | sh # one-time, if needed
curl -LsSf https://astral.sh/uv/install.sh | sh
uv sync # build .venv from uv.lock ./reproduce.sh # env -> tests -> biological-model tier at committed seeds -> figures
make test # correctness + scientific-validation tests (the spine of trust) ./reproduce.sh --with-gpu # ... and the trained-network + language-model tiers
make layer1 # run experiments E1–E6
make figures # regenerate figures from committed results
``` ```
Or tier by tier:
```bash
make env # build .venv from the committed, hash-pinned uv.lock
make test # correctness + closed-form scientific validation (the spine of trust)
make layer1 # the biological model: E1-E12, E14, learning kernel
make figures # per-experiment figures, from committed parquets (no re-simulation)
make paper-figures # the manuscript's Fig. 1-7 + rebuild the PDF body
```
`make help` is not defined, but every target carries a `##` description — `grep '##' Makefile`.
## Notebooks
```bash
make env-notebooks && jupyter lab notebooks/
```
- `01_biological_model.ipynb` — builds the Wright–Fisher model from scratch, checks it against the
closed forms (geometric diversity decay, the immigration–drift equilibrium), and derives the
grounding threshold and its per-item observation floor. Runs on a laptop in under a minute.
- `02_paper_figures.ipynb` — verifies artifact hashes, then regenerates and displays every
manuscript figure from the committed artifacts.
## Layout ## Layout
``` ```
src/knowledge/ Layer 1 package (imported as `knowledge`) src/knowledge/ biological-model tier (imported as `knowledge`)
configs/layer1/ one YAML per experiment (E1..E6) src/neural/ trained-network tier
figures/ plot_EX.py — read results.parquet only src/llm/ language-model tier
tests/ test_correctness.py + test_scientific_validation.py (analytic checks) configs/ one YAML per experiment: layer1/ neural/ llm/ (each declares its master seed)
paper/ blueprint.md, perspective paper, figure_manifest.md figures/ plot_*.py — per-experiment diagnostics, read results.parquet only
results/ written artifacts (gitignored; hashes tracked in manifest.json) paper/pnas/ the manuscript: main.md, make_figs.py (Fig. 1-7), build.py, si.md
notebooks/ executable walkthroughs
hpc/ PBS job scripts for the 7B tier (Imperial CX3)
tests/ correctness + test_scientific_validation.py (the closed forms as assertions)
results/ run artifacts: results.parquet (gitignored) + resolved_config.yaml + manifest.json
``` ```
Design documents: `paper/blueprint.md` (the normative build spec) and `paper/results-summary.md`
(plain-language + technical summary of every result).
## The engineering contract
- **Reproducibility is a requirement, not a preference.** The environment is a `uv` venv built from
a committed, hash-pinned `uv.lock`; the biological-model tier is bitwise reproducible from a
single master seed, and the GPU tiers are statistically reproducible with per-seed points
reported.
- **One master seed per config**, with all sub-randomness derived via `SeedSequence.spawn`. No code
touches global RNG state; a run is a pure function of its resolved config.
- **No magic numbers in code.** Every parameter lives in a YAML resolved at run time, and the
resolved config is written next to the results.
- **Every run writes the same triple:** `results.parquet` + `resolved_config.yaml` +
`manifest.json` (seed, git commit, library versions, row count, SHA-256 of the results).
- **Every figure is a pure function of a committed artifact** — figure scripts never re-simulate.
- **The scientific-validation tests are the spine of trust.** They assert that the simulator
reproduces the closed forms to within 0.5%. If they fail, the science is wrong, not just the code.

201
REPRODUCING.md Normal file
View file

@ -0,0 +1,201 @@
# Reproducing every number and figure in the paper
This document is the authoritative map from the manuscript back to the code, configs, and seeds
that produced it. Every figure panel, every headline number, and the environment they were computed
in are listed below. If something in the paper is not traceable through this document, that is a
bug — please open an issue.
Manuscript: `paper/pnas/main.md` (built to `paper/pnas/main.pdf`).
---
## 1. The three tiers, and what each costs to reproduce
| Tier | What it is | Hardware | Determinism |
|---|---|---|---|
| Biological model | Wright–Fisher simulator over knowledge distributions (pure NumPy/SciPy) | Any laptop, no GPU | **Bitwise** from the master seed |
| Trained networks | RNN / MLP / VAE on a synthetic mode universe; convolutional VAE on MNIST | One consumer GPU (16 GB) | Statistical (GPU non-determinism documented in §5) |
| Language models | LoRA specialists on Qwen2.5-Instruct 0.5B / 7B | 0.5B: one 16 GB GPU · 7B: one L40S (46 GB) | Statistical; per-seed points reported |
The biological-model tier carries every quantitative claim in the paper and reproduces exactly on a
laptop in minutes. The two AI tiers are confirmatory (signs, not magnitudes) and need a GPU.
## 2. Environment
The environment is a `uv` venv built from the committed, hash-pinned `uv.lock`. That lockfile — not
a container, not a requirements file — is the single source of truth for "it runs".
```bash
curl -LsSf https://astral.sh/uv/install.sh | sh # one-time, if you don't have uv
make env # biological-model tier (pure NumPy/SciPy)
make env-neural # + torch, for the trained-network tier
make env-mnist # + torchvision, for the real-MNIST tier
make env-llm # + transformers/peft, for the language-model tier
make env-notebooks # + Jupyter, for the walkthrough notebooks
```
## 3. One command
```bash
./reproduce.sh
```
runs the environment build, the test suite (including the closed-form scientific-validation tests),
the entire biological-model tier at its committed seeds, every figure, and writes `REPRODUCED.md`
with the resulting artifact hashes for comparison against the committed manifests. It deliberately
stops at the GPU tiers; pass `--with-gpu` to include them if you have the hardware.
Tier by tier, by hand:
```bash
make test # correctness + closed-form scientific validation
make layer1 # the biological model: E1-E12, E14, learning kernel
make neural # trained networks (needs a GPU)
make mnist # real-MNIST tier + the Fig. 2B montage asset (needs torchvision)
make llm # language-model prototypes (needs a GPU)
make llm-seeds # the multi-seed firm-ups behind Fig. 3A
make llm-epistasis # the controlled predictive test behind Fig. 3C-D
make llm-speciation # the LLM speciation tier behind Fig. 7E-F
make figures # per-experiment figures, from committed parquets (no re-simulation)
make paper-figures # the manuscript figures Fig. 1-7 + rebuild the PDF body
```
## 4. The figure map
Every manuscript panel, the artifact it is plotted from, the config that produced that artifact, and
that config's declared seed. All panels are drawn by `paper/pnas/make_figs.py` (function per figure);
`make paper-figures` regenerates all of them. Figures are a **pure function of committed artifacts** —
no panel re-simulates anything, with the single documented exception of the Fig. 2B montage asset.
| Panel | Drawn by | From artifact | Produced by config | Seed(s) |
|---|---|---|---|---|
| Fig. 1A, 1B | `fig1a()`, `fig1b()` | — (schematics; icons in `paper/pnas/figs/icons/`) | — | — |
| Fig. 2A | `fig2()` | `results/E2/` | `configs/layer1/E2.yaml` | 20260704 |
| Fig. 2B | `fig2()` | `results/mnist_collapse/mnist_montage.png` | `configs/neural/mnist_collapse.yaml` → asset from `figures/mnist_montage.py` | 20260705 |
| Fig. 3A | `fig3()` | `results/llm_merge_seeds/` | `configs/llm/merge_seeds.yaml` | 1, 2, 3, 4, 5 |
| Fig. 3B | `fig3()` | `results/llm_moe_hard_hpc/` | `configs/llm/moe_hard_hpc.yaml` (7B, HPC) | 1 (single run) |
| Fig. 3C, 3D | `fig3()` | `results/llm_epistasis/` + `results/llm_epistasis_compat/` | `configs/llm/epistasis.yaml`, `configs/llm/epistasis_compat.yaml` | 1, 2, 3 |
| Fig. 4A | `fig4()` | `results/E4/` | `configs/layer1/E4.yaml` | 20260704 |
| Fig. 4B | `fig4()` | `results/E8/` | `configs/layer1/E8.yaml` | 20260705 |
| Fig. 5A | `fig5()` | `results/E9/` | `configs/layer1/E9.yaml` | 20260705 |
| Fig. 5B | `fig5()` | `results/E10/` | `configs/layer1/E10.yaml` | 20260705 |
| Fig. 5C, 5D | `fig5()` | `results/E14/` | `configs/layer1/E14.yaml` | 20260709 |
| Fig. 6A, 6B, 6C | `fig6()` | `results/E11/` | `configs/layer1/E11.yaml` | 20260705 |
| Fig. 7A, 7B | `fig7()` | `results/E12/` | `configs/layer1/E12.yaml` | 12 |
| Fig. 7C | `fig7()` | `results/speciation_real/` | `configs/neural/speciation_real.yaml` | 13 |
| Fig. 7D | `fig7()` | `results/speciation_real_cliff/` | `configs/neural/speciation_real_cliff.yaml` | 13 |
| Fig. 7E, 7F | `fig7()` | `results/llm_speciation/` | `configs/llm/speciation.yaml` | 1 (single seed) |
**Single-run panels.** Fig. 3B and Fig. 7E–F come from single-seed runs and are reported as
sign-level confirmations, not estimates; the manuscript labels them as such. Every other panel is
replicated (biological-model panels over 12–100 internal replicates; Fig. 3A over five training
seeds; Fig. 3C–D over three).
### Results reported in the text but not plotted in the manuscript
| Result | Artifact | Config | Seed |
|---|---|---|---|
| Collapse null (E1) | `results/E1/` | `configs/layer1/E1.yaml` | 20260704 |
| Region-matched grounding (E3) | `results/E3/` | `configs/layer1/E3.yaml` | 20260704 |
| Quality-diversity vs greedy (E5) | `results/E5/` | `configs/layer1/E5.yaml` | 20260704 |
| Re-minting / irreversibility (E6) | `results/E6/` | `configs/layer1/E6.yaml` | 20260704 |
| Advantage of sex, lineage (E7) | `results/E7/` | `configs/layer1/E7.yaml` | 20260705 |
| Incompatibilities on NK (E12_nk) | `results/E12_nk/` | `configs/layer1/E12_nk.yaml` | 12 |
| Learning kernel (estimator bias) | `results/kernel_sharpen/`, `results/kernel_smooth/` | `configs/layer1/kernel_{sharpen,smooth}.yaml` | 20260705 |
| Histogram bridge gate | `results/bridge/` | `configs/neural/bridge.yaml` | 20260704 |
| Neural collapse / grounding / architectures / recombination | `results/{collapse,grounding,architectures,recombination}/` | `configs/neural/*.yaml` | 20260704 |
| Emergent-isolation null | `results/speciation_real_emergent/` | `configs/neural/speciation_real_emergent.yaml` | 813 |
| Budget-controlled speciation (add design) | `results/llm_speciation_add/` | `configs/llm/speciation_add.yaml` | 1, 2, 3 |
| LLM prototypes (0.5B) | `results/llm_{merge,moe,directed}/` | `configs/llm/{merge,moe,directed}.yaml` | 1 |
| 7B firm-ups | `results/llm_*_hpc/` | `configs/llm/*_hpc.yaml` (run via `hpc/*.pbs`) | 1 |
### Per-experiment (exploratory) figures
`figures/plot_*.py` regenerate a diagnostic figure **inside each results bundle**
(`results/<name>/<name>.pdf`), named after the experiment, not after a manuscript figure. They are
the working views, not the manuscript's; the table above is the authority on what appears in the
paper. `figures/stats_llm_epistasis.py` prints the robust statistics quoted in the predictive-test
section (clustered bootstrap, paired contrasts, leave-one-condition-out, outcome-reference
sensitivity).
## 5. Seeds and determinism
**Policy.** One master seed per config. All sub-randomness is derived from it via
`numpy.random.SeedSequence.spawn` (`src/knowledge/seeding.py`); no code touches global RNG state, and
every `rng` is passed explicitly. A run is a pure function of its resolved config.
**Biological-model tier: bitwise reproducible.** Re-running a config on the same lockfile
reproduces its `results.parquet` byte-for-byte; the `results_sha256` in each `manifest.json` is the
check.
**GPU tiers: statistically reproducible.** cuDNN kernel selection and reduction order make bitwise
equality unattainable across machines. Per-seed points are reported rather than seed-averaged
summaries alone, and the multi-seed protocols fix the evaluation sets and vary only the training
seed. Expect sign agreement and magnitudes within noise, not identical digits.
**Seed provenance.** `20260704`/`20260705`/`20260709` are date-stamped master seeds chosen at the
time each experiment was written and never re-drawn. Small integer seeds (`1`, `12`, `13`, `813`)
are likewise fixed at authoring time. No seed in this repository was selected after seeing results.
## 6. Verifying artifact integrity
Every run writes three files next to its results:
- `results.parquet` — the long-form data (the only thing figures read)
- `resolved_config.yaml` — the config **after** sweep expansion, i.e. exactly what ran
- `manifest.json` — master seed, git commit, Python and library versions, row count, and
`results_sha256` (the content hash of the parquet)
To verify a bundle you have regenerated matches the one behind the paper:
```bash
python - <<'EOF'
import hashlib, json, pathlib
for m in sorted(pathlib.Path("results").glob("*/manifest.json")):
man = json.loads(m.read_text())
pq = m.parent / "results.parquet"
if not pq.exists():
print(f"{m.parent.name:28s} (no parquet — run its config first)"); continue
got = hashlib.sha256(pq.read_bytes()).hexdigest()
ok = "OK " if got == man.get("results_sha256") else "DIFF"
print(f"{ok} {m.parent.name:28s} seed={man.get('master_seed')}")
EOF
```
`DIFF` on a biological-model bundle means a genuine discrepancy worth investigating. `DIFF` on a GPU
tier is expected (see §5) — compare the figures and the reported statistics instead.
## 7. HPC (the 7B tier)
The 7B runs were executed on Imperial College's CX3 cluster (PBS Pro, one L40S 46 GB per job). Job
scripts are in `hpc/`; each is self-contained and documents its own submission line. They stage the
same `uv.lock` environment, so the only difference from a local run is the GPU.
```bash
qsub hpc/llm_merge.pbs # 7B merge firm-up
qsub hpc/llm_hard.pbs # hard-benchmark moe + directed at 7B
qsub hpc/llm_society.pbs # the society campaign (array over seeds)
```
## 8. Notebooks
`notebooks/` contains executable walkthroughs (`make env-notebooks`, then `jupyter lab`):
| Notebook | What it does | Needs |
|---|---|---|
| `01_biological_model.ipynb` | Builds the Wright–Fisher model from scratch, checks it against the three closed forms, and derives the grounding threshold interactively | Laptop |
| `02_paper_figures.ipynb` | Regenerates every manuscript figure from the committed artifacts and displays them inline, panel by panel | Laptop (artifacts must be present) |
`make notebooks` executes both end-to-end, which is itself a reproduction check.
## 9. Known gaps
- `results/**/results.parquet` is currently **gitignored** (only manifests, hashes, and resolved
configs are tracked). A fresh clone therefore has to re-run the experiments before figures can be
regenerated. The archived deposit (Zenodo DOI, on publication) includes the parquets so that the
paper's "regenerates from committed artifacts without re-simulation" holds from the archive.
- `figures/mnist_montage.py` re-runs a short dry lineage to draw its montage rather than reading a
parquet; it is an eyeball diagnostic whose quantitative counterpart is `results/mnist_collapse/`.
- The composed society at language-model scale is an open experiment at the time of writing; see
`tasks/workorder-llm-society.md`.

489
notebooks/01_biological_model.ipynb Normal file
View file

File diff suppressed because one or more lines are too long

565
notebooks/02_paper_figures.ipynb Normal file
View file

File diff suppressed because one or more lines are too long

BIN
paper/pnas/figs/fig1a.pdf
View file

Binary file not shown.

BIN
paper/pnas/figs/fig1b.pdf
View file

Binary file not shown.

BIN
paper/pnas/figs/fig2.pdf
View file

Binary file not shown.

BIN
paper/pnas/figs/fig3.pdf
View file

Binary file not shown.

BIN
paper/pnas/figs/fig4.pdf
View file

Binary file not shown.

BIN
paper/pnas/figs/fig5.pdf
View file

Binary file not shown.

BIN
paper/pnas/figs/fig6.pdf
View file

Binary file not shown.

BIN
paper/pnas/figs/fig7.pdf
View file

Binary file not shown.

3
pyproject.toml
View file

@ -26,6 +26,9 @@ mnist = ["torchvision>=0.17"]
# Layer 2 / LLM prototype (blueprint C2/C4): LoRA specialists + weight-space merging on a small # Layer 2 / LLM prototype (blueprint C2/C4): LoRA specialists + weight-space merging on a small
# open-weight base. GPU; models download to the HF cache (outside the repo). `uv sync --extra llm`. # open-weight base. GPU; models download to the HF cache (outside the repo). `uv sync --extra llm`.
llm = ["transformers>=4.44", "peft>=0.11", "datasets", "accelerate"] llm = ["transformers>=4.44", "peft>=0.11", "datasets", "accelerate"]
# Jupyter notebooks that walk through the biological model and regenerate every paper figure
# from the committed artifacts. `uv sync --extra notebooks`.
notebooks = ["jupyter>=1.0"]
[build-system] [build-system]
requires = ["hatchling"] requires = ["hatchling"]

82
reproduce.sh Executable file
View file

@ -0,0 +1,82 @@
#!/usr/bin/env bash
# One-command reproduction of the paper's biological-model tier (see REPRODUCING.md).
#
# ./reproduce.sh env -> tests -> biological model at committed seeds -> figures
# ./reproduce.sh --with-gpu also runs the trained-network, MNIST, and language-model tiers
#
# Writes REPRODUCED.md: every artifact's recomputed content hash next to the committed one, so a
# reader can see at a glance which bundles reproduced bitwise. The biological-model tier must match
# exactly; GPU tiers are statistically reproducible only (REPRODUCING.md section 5).
set -euo pipefail
WITH_GPU=0
[[ "${1:-}" == "--with-gpu" ]] && WITH_GPU=1
cd "$(dirname "$0")"
START=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
say() { printf '\n\033[1m== %s\033[0m\n' "$1"; }
say "Environment (uv sync from the committed uv.lock)"
if ! command -v uv >/dev/null; then
echo "uv not found. Install it: curl -LsSf https://astral.sh/uv/install.sh | sh" >&2
exit 1
fi
if [[ $WITH_GPU -eq 1 ]]; then
uv sync --extra dev --extra neural --extra mnist --extra llm
else
uv sync --extra dev
fi
say "Tests (correctness + closed-form scientific validation)"
uv run pytest -q
say "Biological-model tier at the committed seeds"
make layer1
if [[ $WITH_GPU -eq 1 ]]; then
say "Trained-network tier"; make neural
say "Real-MNIST tier"; make mnist
say "Language-model tier"; make llm
say "LLM predictive test"; make llm-epistasis
fi
say "Figures (pure functions of the artifacts)"
make figures
make paper-figures
say "Hash report -> REPRODUCED.md"
uv run python - "$START" "$WITH_GPU" <<'EOF'
import hashlib, json, pathlib, subprocess, sys, datetime
start, with_gpu = sys.argv[1], sys.argv[2] == "1"
commit = subprocess.run(["git", "rev-parse", "HEAD"], capture_output=True, text=True).stdout.strip()
rows, exact, differ, missing = [], 0, 0, 0
for man_path in sorted(pathlib.Path("results").glob("*/manifest.json")):
man = json.loads(man_path.read_text())
pq = man_path.parent / "results.parquet"
want = man.get("results_sha256", "")
if not pq.exists():
status, got, missing = "not run", "-", missing + 1
else:
got = hashlib.sha256(pq.read_bytes()).hexdigest()
if got == want:
status, exact = "bitwise match", exact + 1
else:
status, differ = "differs", differ + 1
rows.append((man_path.parent.name, man.get("master_seed", "-"), status, want[:12], got[:12]))
out = [
"# Reproduction report", "",
f"- Started: {start}", f"- Finished: {datetime.datetime.now(datetime.UTC):%Y-%m-%dT%H:%M:%SZ}",
f"- Commit: `{commit}`", f"- GPU tiers included: {'yes' if with_gpu else 'no'}",
f"- Bundles: {exact} bitwise match, {differ} differ, {missing} not run", "",
"The biological-model tier is bitwise reproducible and must show `bitwise match`. GPU tiers are",
"statistically reproducible only, so `differs` is expected there (see REPRODUCING.md section 5).",
"", "| Bundle | Seed | Status | Committed sha256 | Recomputed |", "|---|---|---|---|---|",
]
out += [f"| `{n}` | {s} | {st} | `{w}…` | `{g}…` |" for n, s, st, w, g in rows]
pathlib.Path("REPRODUCED.md").write_text("\n".join(out) + "\n")
print(f"{exact} bitwise match, {differ} differ, {missing} not run -> REPRODUCED.md")
EOF
say "Done. See REPRODUCED.md, and REPRODUCING.md for the figure-by-figure map."

11
tasks/lessons.md
View file

@ -39,3 +39,14 @@ pop-gen construct is **"the biological model"** (GG decision, 2026-09-07) — us
line "closed forms · bitwise-reproducible" carries that content). Support level formerly line "closed forms · bitwise-reproducible" carries that content). Support level formerly
"Exact" is now "Closed form". Keep "exact" only in technical noun phrases (exact-match "Exact" is now "Closed form". Keep "exact" only in technical noun phrases (exact-match
verifier, exact oracle, exact equilibrium, exact recovery). verifier, exact oracle, exact equilibrium, exact recovery).
## Never `uv sync` while a job is using the venv (2026-09-07)
Adding the `notebooks` extra mid-session ran `uv sync` **without** `--extra llm`, which rebuilt the
shared `.venv` — dropping the LLM stack *and* silently switching the interpreter 3.14 → 3.11 (uv
recreates with the system default when no `.python-version` is pinned). That killed the running
local society seed with a `FileNotFoundError` deep in `huggingface_hub` templates — a failure that
looked scientific but was pure environment churn.
Rules: (1) never mutate `.venv` while a background job is running against it — wait, or build a
throwaway venv elsewhere; (2) `uv sync` is *declarative* — always pass **every** extra the project
needs, or it removes the ones you omit; (3) the repo now pins `.python-version` (3.14) so the
interpreter can never drift silently.

2756
uv.lock generated
View file

File diff suppressed because it is too large Load diff