SI Methods: a full experimental-procedures appendix

Replaces the three-paragraph methods sketch with a scientific account of how
the study was run (M1-M7):

- M1 design principles: cheapest falsifying tier; match claim precision to
  instrument precision; every tier gets an oracle independent of the model
  being measured; falsifiers declared before running.
- M2 replication: what a replicate *is* differs by tier (independent lineage /
  lineage incl. fresh init and data order / training seed with test sets held
  fixed), and a table giving every experiment's replicate count with the
  reasoning - why 200 for E4 (per-item binary outcomes), 60 for the bridge
  gate (must detect any departure), 3-5 where the contrast is categorical,
  and 1 for the 7B runs, labelled as single runs.
- M3-M5 per-tier procedures: parameter choices and their justification, the
  correlated-parent construction, why the neural sandbox is synthetic (a
  lossless identity code plus style entropy gives an exact oracle while still
  forcing the model to learn a distribution), MNIST modes and the frozen-CNN
  oracle with its confusion matrix as measurement floor, why no-BatchNorm MLPs
  for the alignment analysis, and for the LLM tier: why Qwen 0.5B/7B (one
  family so scale is the only variable), why procedural tasks rather than a
  benchmark (exact verifier, contamination-free, controlled disjointness, a
  difficulty knob), why LoRA (confines each parent to an additive low-rank
  delta over an identical base, which is what makes weight-space
  recombination well defined), the training algorithm, and the split scheme.
- M6 negative controls, including the one that removed a result: the
  compatible-overlap axis collapsed the delta-cosine predictor from rho=+0.60
  to +0.03.
- M7 statistical procedures.

Also: SI voice converted to first person and terminology synced to the
"biological model" rename; removed a process ghost from the preamble
("Skeleton assembled at Phase 4"); build.py now takes a document argument and
no longer eats documents that lack a title block, so the SI compiles via a new
si.tex wrapper (10 pp). `make paper` builds both PDFs.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkRLcc18rwT2Lysu6PbG7v
This commit is contained in:
Giorgio Gilestro 2026-09-07 16:04:52 +01:00
parent c435cfba6e
commit a88289964a
6 changed files with 493 additions and 40 deletions

View file

@ -2,7 +2,7 @@
# reproducibility source of truth; every target runs inside it via `uv run`.
.PHONY: env env-neural env-mnist env-llm env-notebooks test layer1 layer2 neural mnist llm \
llm-epistasis llm-society figures paper-figures notebooks clean
llm-epistasis llm-society figures paper-figures paper notebooks clean
env: ## build .venv from the committed lockfile
uv sync --extra dev
@ -62,9 +62,13 @@ figures: ## regenerate per-experiment figures from committed results (
for p in figures/plot_*.py; do case "$$p" in */plot_E[1-6].py|*/_*) ;; \
*) [ -e "$$p" ] && MPLBACKEND=Agg uv run python "$$p" ;; esac; done
paper-figures: ## regenerate the manuscript figures (Fig. 1-7) and rebuild the PDF body
paper-figures: ## regenerate the manuscript figures (Fig. 1-7) and rebuild the PDF bodies
MPLBACKEND=Agg uv run python paper/pnas/make_figs.py
uv run python paper/pnas/build.py
uv run python paper/pnas/build.py si
paper: paper-figures ## figures + compile both PDFs (needs tectonic)
cd paper/pnas && tectonic main.tex && tectonic si.tex
env-notebooks: ## add Jupyter for the walkthrough notebooks
uv sync --extra dev --extra notebooks