Giorgio Gilestro
c435cfba6e
An audit of the figure pipeline found real sync gaps, now closed:
- `paper/pnas/make_figs.py` (which draws every manuscript figure) was invoked
by NO Makefile target or script - a manual step. Added `make paper-figures`.
- `configs/llm/epistasis{,_compat}.yaml` were reachable from nothing at all,
despite producing Fig. 3C-D. Added `make llm-epistasis` (+ its statistics).
- `make figures` never regenerated the MNIST montage that Fig. 2B embeds;
it now runs with the `mnist` target (it needs torch - it re-simulates).
- Added `make llm-society`, `env-notebooks`, `notebooks`.
New REPRODUCING.md is the authoritative map: every manuscript panel -> the
artifact it plots -> the config that produced it -> that config's seed, plus
the determinism policy (biological tier bitwise; GPU tiers statistical), the
seed-provenance statement, and an artifact-hash verification snippet. All 44
committed bundles currently hash-match their manifests, and figure
regeneration is pixel-identical (verified by comparison).
reproduce.sh delivers the one-command reproduction the paper's Methods
promises, writing REPRODUCED.md with recomputed hashes per bundle.
Two executed notebooks: 01 builds the Wright-Fisher model from scratch and
checks both closed forms interactively (runs in ~1 min on a laptop); 02
verifies artifact hashes then regenerates and displays all seven manuscript
figures. Both execute end-to-end (`make notebooks`).
Also pins `.python-version` to 3.14: the interpreter was previously
unpinned, and a `uv sync` silently switched it to 3.11 mid-session (see
tasks/lessons.md). README rewritten - it still described a Layer-1-only repo
of E1-E6 and pointed at a figure_manifest.md that does not exist.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkRLcc18rwT2Lysu6PbG7v
82 lines
3.3 KiB
Bash
Executable file
82 lines
3.3 KiB
Bash
Executable file
#!/usr/bin/env bash
|
|
# One-command reproduction of the paper's biological-model tier (see REPRODUCING.md).
|
|
#
|
|
# ./reproduce.sh env -> tests -> biological model at committed seeds -> figures
|
|
# ./reproduce.sh --with-gpu also runs the trained-network, MNIST, and language-model tiers
|
|
#
|
|
# Writes REPRODUCED.md: every artifact's recomputed content hash next to the committed one, so a
|
|
# reader can see at a glance which bundles reproduced bitwise. The biological-model tier must match
|
|
# exactly; GPU tiers are statistically reproducible only (REPRODUCING.md section 5).
|
|
set -euo pipefail
|
|
|
|
WITH_GPU=0
|
|
[[ "${1:-}" == "--with-gpu" ]] && WITH_GPU=1
|
|
|
|
cd "$(dirname "$0")"
|
|
START=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
|
|
say() { printf '\n\033[1m== %s\033[0m\n' "$1"; }
|
|
|
|
say "Environment (uv sync from the committed uv.lock)"
|
|
if ! command -v uv >/dev/null; then
|
|
echo "uv not found. Install it: curl -LsSf https://astral.sh/uv/install.sh | sh" >&2
|
|
exit 1
|
|
fi
|
|
if [[ $WITH_GPU -eq 1 ]]; then
|
|
uv sync --extra dev --extra neural --extra mnist --extra llm
|
|
else
|
|
uv sync --extra dev
|
|
fi
|
|
|
|
say "Tests (correctness + closed-form scientific validation)"
|
|
uv run pytest -q
|
|
|
|
say "Biological-model tier at the committed seeds"
|
|
make layer1
|
|
|
|
if [[ $WITH_GPU -eq 1 ]]; then
|
|
say "Trained-network tier"; make neural
|
|
say "Real-MNIST tier"; make mnist
|
|
say "Language-model tier"; make llm
|
|
say "LLM predictive test"; make llm-epistasis
|
|
fi
|
|
|
|
say "Figures (pure functions of the artifacts)"
|
|
make figures
|
|
make paper-figures
|
|
|
|
say "Hash report -> REPRODUCED.md"
|
|
uv run python - "$START" "$WITH_GPU" <<'EOF'
|
|
import hashlib, json, pathlib, subprocess, sys, datetime
|
|
|
|
start, with_gpu = sys.argv[1], sys.argv[2] == "1"
|
|
commit = subprocess.run(["git", "rev-parse", "HEAD"], capture_output=True, text=True).stdout.strip()
|
|
rows, exact, differ, missing = [], 0, 0, 0
|
|
for man_path in sorted(pathlib.Path("results").glob("*/manifest.json")):
|
|
man = json.loads(man_path.read_text())
|
|
pq = man_path.parent / "results.parquet"
|
|
want = man.get("results_sha256", "")
|
|
if not pq.exists():
|
|
status, got, missing = "not run", "-", missing + 1
|
|
else:
|
|
got = hashlib.sha256(pq.read_bytes()).hexdigest()
|
|
if got == want:
|
|
status, exact = "bitwise match", exact + 1
|
|
else:
|
|
status, differ = "differs", differ + 1
|
|
rows.append((man_path.parent.name, man.get("master_seed", "-"), status, want[:12], got[:12]))
|
|
|
|
out = [
|
|
"# Reproduction report", "",
|
|
f"- Started: {start}", f"- Finished: {datetime.datetime.now(datetime.UTC):%Y-%m-%dT%H:%M:%SZ}",
|
|
f"- Commit: `{commit}`", f"- GPU tiers included: {'yes' if with_gpu else 'no'}",
|
|
f"- Bundles: {exact} bitwise match, {differ} differ, {missing} not run", "",
|
|
"The biological-model tier is bitwise reproducible and must show `bitwise match`. GPU tiers are",
|
|
"statistically reproducible only, so `differs` is expected there (see REPRODUCING.md section 5).",
|
|
"", "| Bundle | Seed | Status | Committed sha256 | Recomputed |", "|---|---|---|---|---|",
|
|
]
|
|
out += [f"| `{n}` | {s} | {st} | `{w}…` | `{g}…` |" for n, s, st, w, g in rows]
|
|
pathlib.Path("REPRODUCED.md").write_text("\n".join(out) + "\n")
|
|
print(f"{exact} bitwise match, {differ} differ, {missing} not run -> REPRODUCED.md")
|
|
EOF
|
|
|
|
say "Done. See REPRODUCED.md, and REPRODUCING.md for the figure-by-figure map."
|