Reproducibility pass: figure map, one-command reproduce.sh, notebooks, Makefile gaps
An audit of the figure pipeline found real sync gaps, now closed:
- `paper/pnas/make_figs.py` (which draws every manuscript figure) was invoked
by NO Makefile target or script - a manual step. Added `make paper-figures`.
- `configs/llm/epistasis{,_compat}.yaml` were reachable from nothing at all,
despite producing Fig. 3C-D. Added `make llm-epistasis` (+ its statistics).
- `make figures` never regenerated the MNIST montage that Fig. 2B embeds;
it now runs with the `mnist` target (it needs torch - it re-simulates).
- Added `make llm-society`, `env-notebooks`, `notebooks`.
New REPRODUCING.md is the authoritative map: every manuscript panel -> the
artifact it plots -> the config that produced it -> that config's seed, plus
the determinism policy (biological tier bitwise; GPU tiers statistical), the
seed-provenance statement, and an artifact-hash verification snippet. All 44
committed bundles currently hash-match their manifests, and figure
regeneration is pixel-identical (verified by comparison).
reproduce.sh delivers the one-command reproduction the paper's Methods
promises, writing REPRODUCED.md with recomputed hashes per bundle.
Two executed notebooks: 01 builds the Wright-Fisher model from scratch and
checks both closed forms interactively (runs in ~1 min on a laptop); 02
verifies artifact hashes then regenerates and displays all seven manuscript
figures. Both execute end-to-end (`make notebooks`).
Also pins `.python-version` to 3.14: the interpreter was previously
unpinned, and a `uv sync` silently switched it to 3.11 mid-session (see
tasks/lessons.md). README rewritten - it still described a Layer-1-only repo
of E1-E6 and pointed at a figure_manifest.md that does not exist.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkRLcc18rwT2Lysu6PbG7v
This commit is contained in:
parent
8086504f44
commit
c435cfba6e
18 changed files with 4204 additions and 28 deletions
99
README.md
99
README.md
|
|
@ -1,39 +1,88 @@
|
|||
# The Lamarckian Society — Layer 1 (analytical core)
|
||||
# The evolution of sex for artificial intelligence
|
||||
|
||||
A parametric population-genetics model of knowledge transmission across generations of
|
||||
learning agents. Knowledge transmission is modelled *literally* as a Wright–Fisher
|
||||
process (not by analogy): a model's knowledge is a distribution `p_t` over `K` discrete
|
||||
items; a fixed true distribution `p*` has a rare tail; each generational step is
|
||||
"sample from the parent (drift) + mix in fresh real samples (grounding/immigration) +
|
||||
refit." Model collapse is the loss of rare alleles under drift.
|
||||
A population-genetic framework for multigenerational model populations. Knowledge transmission
|
||||
between generations of learning agents is modelled *literally* as a Wright–Fisher process, not by
|
||||
analogy: a model's knowledge is a distribution `p_t` over `K` discrete items, a fixed true
|
||||
distribution `p*` has a rare tail, and each generational step is "sample from the parent (drift) +
|
||||
mix in fresh real samples (grounding/immigration) + refit". Model collapse is the loss of rare
|
||||
alleles under drift — and the remedies population genetics knows for drift (immigration,
|
||||
recombination, selection, population structure) become engineering levers for model populations.
|
||||
|
||||
See `paper/blueprint.md` (the normative build spec),
|
||||
`paper/the-lamarckian-society-v5.md` (the perspective paper), and
|
||||
`paper/results-summary.md` (a summary of all results).
|
||||
The framework is developed at three tiers of increasing realism:
|
||||
|
||||
| Tier | What it is | Hardware |
|
||||
|---|---|---|
|
||||
| **Biological model** | Wright–Fisher simulator over knowledge distributions; closed forms, bitwise reproducible | laptop |
|
||||
| **Trained networks** | RNN / MLP / VAE on a synthetic mode universe with an exact oracle; convolutional VAE on MNIST | one GPU |
|
||||
| **Language models** | LoRA specialists on Qwen2.5-Instruct (0.5B / 7B) with an exact-match verifier | one GPU / L40S |
|
||||
|
||||
## Reproduce
|
||||
|
||||
Environment is a `uv` venv built from the committed, hash-pinned `uv.lock` — that
|
||||
lockfile is the single source of truth for "it runs" (Layer 1 is pure NumPy/SciPy and
|
||||
bitwise-reproducible from a seed; no container needed).
|
||||
**Start here: [`REPRODUCING.md`](REPRODUCING.md)** — the authoritative map from every manuscript
|
||||
figure panel back to the artifact, config, and seed that produced it, plus the determinism policy
|
||||
and artifact-hash verification.
|
||||
|
||||
```bash
|
||||
# one-time: install uv (https://astral.sh/uv)
|
||||
curl -LsSf https://astral.sh/uv/install.sh | sh
|
||||
curl -LsSf https://astral.sh/uv/install.sh | sh # one-time, if needed
|
||||
|
||||
uv sync # build .venv from uv.lock
|
||||
make test # correctness + scientific-validation tests (the spine of trust)
|
||||
make layer1 # run experiments E1–E6
|
||||
make figures # regenerate figures from committed results
|
||||
./reproduce.sh # env -> tests -> biological-model tier at committed seeds -> figures
|
||||
./reproduce.sh --with-gpu # ... and the trained-network + language-model tiers
|
||||
```
|
||||
|
||||
Or tier by tier:
|
||||
|
||||
```bash
|
||||
make env # build .venv from the committed, hash-pinned uv.lock
|
||||
make test # correctness + closed-form scientific validation (the spine of trust)
|
||||
make layer1 # the biological model: E1-E12, E14, learning kernel
|
||||
make figures # per-experiment figures, from committed parquets (no re-simulation)
|
||||
make paper-figures # the manuscript's Fig. 1-7 + rebuild the PDF body
|
||||
```
|
||||
|
||||
`make help` is not defined, but every target carries a `##` description — `grep '##' Makefile`.
|
||||
|
||||
## Notebooks
|
||||
|
||||
```bash
|
||||
make env-notebooks && jupyter lab notebooks/
|
||||
```
|
||||
|
||||
- `01_biological_model.ipynb` — builds the Wright–Fisher model from scratch, checks it against the
|
||||
closed forms (geometric diversity decay, the immigration–drift equilibrium), and derives the
|
||||
grounding threshold and its per-item observation floor. Runs on a laptop in under a minute.
|
||||
- `02_paper_figures.ipynb` — verifies artifact hashes, then regenerates and displays every
|
||||
manuscript figure from the committed artifacts.
|
||||
|
||||
## Layout
|
||||
|
||||
```
|
||||
src/knowledge/ Layer 1 package (imported as `knowledge`)
|
||||
configs/layer1/ one YAML per experiment (E1..E6)
|
||||
figures/ plot_EX.py — read results.parquet only
|
||||
tests/ test_correctness.py + test_scientific_validation.py (analytic checks)
|
||||
paper/ blueprint.md, perspective paper, figure_manifest.md
|
||||
results/ written artifacts (gitignored; hashes tracked in manifest.json)
|
||||
src/knowledge/ biological-model tier (imported as `knowledge`)
|
||||
src/neural/ trained-network tier
|
||||
src/llm/ language-model tier
|
||||
configs/ one YAML per experiment: layer1/ neural/ llm/ (each declares its master seed)
|
||||
figures/ plot_*.py — per-experiment diagnostics, read results.parquet only
|
||||
paper/pnas/ the manuscript: main.md, make_figs.py (Fig. 1-7), build.py, si.md
|
||||
notebooks/ executable walkthroughs
|
||||
hpc/ PBS job scripts for the 7B tier (Imperial CX3)
|
||||
tests/ correctness + test_scientific_validation.py (the closed forms as assertions)
|
||||
results/ run artifacts: results.parquet (gitignored) + resolved_config.yaml + manifest.json
|
||||
```
|
||||
|
||||
Design documents: `paper/blueprint.md` (the normative build spec) and `paper/results-summary.md`
|
||||
(plain-language + technical summary of every result).
|
||||
|
||||
## The engineering contract
|
||||
|
||||
- **Reproducibility is a requirement, not a preference.** The environment is a `uv` venv built from
|
||||
a committed, hash-pinned `uv.lock`; the biological-model tier is bitwise reproducible from a
|
||||
single master seed, and the GPU tiers are statistically reproducible with per-seed points
|
||||
reported.
|
||||
- **One master seed per config**, with all sub-randomness derived via `SeedSequence.spawn`. No code
|
||||
touches global RNG state; a run is a pure function of its resolved config.
|
||||
- **No magic numbers in code.** Every parameter lives in a YAML resolved at run time, and the
|
||||
resolved config is written next to the results.
|
||||
- **Every run writes the same triple:** `results.parquet` + `resolved_config.yaml` +
|
||||
`manifest.json` (seed, git commit, library versions, row count, SHA-256 of the results).
|
||||
- **Every figure is a pure function of a committed artifact** — figure scripts never re-simulate.
|
||||
- **The scientific-validation tests are the spine of trust.** They assert that the simulator
|
||||
reproduces the closed forms to within 0.5%. If they fail, the science is wrong, not just the code.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue