giorgio/MachineSex
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Restructure: descriptive tier and experiment names, paper/manuscript

Browse source 
- paper/pnas -> paper/manuscript (venue-neutral)
- configs/layer1 -> configs/inheritance, src/knowledge -> src/inheritance
  (imported as `inheritance`), make layer1 -> make inheritance; layer2 alias dropped
- inheritance and trained-network bundles named after the manuscript figure
  they feed (fig2_grounding_sweep, figS3_rebaselining, ...), or descriptively
  where they feed none; configs keep their `experiment:` value so parquet
  hashes are unchanged, only output.dir moves
- figure scripts, SI figure sources, notebooks, REPRODUCING.md, README and the
  SI Methods/tables updated; make clean no longer deletes tracked manifests;
  reproduce.sh hashes the s{seed}/ layouts too

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y64o8FKP7rCuXzC48pxpMm
This commit is contained in:
Giorgio Gilestro 2026-09-13 17:00:40 +01:00
parent 84124de143
commit ab3dc10587
240 changed files with 477 additions and 476 deletions
Download patch file Download diff file Expand all files Collapse all files

271
paper/pnas/build.py
View file

@ -1,271 +0,0 @@
r"""Build the manuscript-draft PDF from main.md (Markdown stays the source of truth).
Adapted from paper/arxiv/md2tex.py (same Markdown subset + pipe tables), with one addition: standalone
`*(FIG:name)*` markers place the publication figures produced by make_figs.py (unified, lettered,
codename-free panels re-plotted from the committed artifacts). Run make_figs.py before building when
results change.
Usage: python paper/pnas/build.py && (cd paper/pnas && tectonic main.tex)
"""
from __future__ import annotations
import re
import shutil
from pathlib import Path
ROOT = Path(__file__).resolve().parents[2]
HERE = Path(__file__).resolve().parent
SRC = HERE / "main.md"
OUT = HERE / "body.tex"
SI_FIG_DIR = HERE / "figs" / "si"
import sys # noqa: E402
sys.path.insert(0, str(HERE))
from si_figures import SI_FIGURES # noqa: E402 (supplementary figures, numbered by order in si.md)
# figure name -> (single publication PDF from make_figs.py, caption)
FIGURES: dict[str, tuple[list[str], str]] = {
"fig1": (["paper/pnas/figs/fig1a.pdf", "paper/pnas/figs/fig1b.pdf"],
"A map of the study. (A) Each row is a biological mechanism the paper borrows, each column a level "
"of realism at which it is tested: an inheritance model (an exact simulation of knowledge "
"transmission, green), trained neural networks measured against exact oracles (blue), and language "
"models (blue). Filled cells name the experiments run at each level and, in the corner, the figure "
"or table reporting them; grey cells were not run, either because the result is "
"established in prior work (23, 34) or because that level adds no new test for that question. The "
"inheritance model is the reference column: it sets the expectation the real-model experiments are "
"read against. (B) The change of viewpoint the transfer rests on. A group of models is usually "
"pictured as a society in space, contemporaries exchanging messages. The couplings studied here run "
"between generations: training on model output (inheritance), weight-space merging (recombination), "
"and verified real data entering each generation (immigration from reality). That is a society in "
"time, which is what population genetics describes. Dots are capabilities: the rare one (gold) is "
"lost under single-parent inheritance, reassembled by merging complementary parents, and re-supplied "
"by grounding."),
"fig2": (["paper/pnas/figs/fig2.pdf"],
"How much real data stops model collapse. (A) An image-generating network (a variational "
"autoencoder) is trained on handwritten digits, then a fresh copy is trained only on the digits the "
"previous one drew, for fifteen generations, with no real data added. Each row is a later generation "
"(0, 4, 8, 12, 15) and each column a randomly chosen drawing. The thirty kinds of digit (ten digits "
"$\\times$ three stroke thicknesses, some kinds rare) collapse to one blurred shape; an independent "
"classifier confirms that the number of kinds still drawn falls from 30 to 1, while adding 10\\% real "
"digits each generation keeps all 30 (Fig.~S7; 4 replicates). (B) The same question in the "
"inheritance model, the exact simulation: 1,000 knowledge items, 200 samples drawn per generation, "
"and a fraction $g$ of fresh real samples mixed in. Points are the diversity the population settles "
"at after 500 generations (mean and 95\\% CI over 100 lineages), the dashed line the exact prediction "
"(the immigration--drift equilibrium), the dotted line the diversity of the real data itself. The "
"curve is smooth, so any threshold is a choice: the red line marks the $g$ at which 95\\% of the real "
"data's diversity is kept, about 0.05 (bootstrap CI shaded). The hollow point at $g = 0$ has not yet "
"reached its equilibrium of zero. The trained image model needed about twice this fraction, because "
"a trained network is not the exact copier the simulation assumes (Fig.~S2)."),
"fig4": (["paper/pnas/figs/fig4.pdf"],
"A population of language models over six generations. (A) The set-up. Three lineages start "
"from one frozen 1.5-billion-parameter base (Qwen2.5-1.5B). Each generation, every lineage learns "
"one new skill from a public dataset by continuing to train its parent's adapter (300 new examples "
"plus 150 replayed from earlier skills), may merge with a partner according to its arm's rule "
"(weights averaged at a ratio chosen on validation data), and is tested on all six skills by a "
"verifier; the child becomes the next parent. The six skills are taken in rotated order, so a "
"partner knows things a lineage lacks early on (complementarity 1.0) and nothing it lacks by the "
"end (0.0). Three training seeds. (B) Accuracy over all six skills of the best lineage (mean and "
"95\\% CI). Never merging and merging only when it beats keeping the parent finish level (0.80 and "
"0.79); merging with a contemporary every generation collapses to 0.27, beginning when partners "
"stop being complementary; a control that merges through generation 2 and then stops (dashed) "
"matches the declinable arm in every seed, and a single model taught the whole syllabus alone "
"(diamond) matches the population. (C) How often the declinable lineages refused a merge (bars) "
"against partner complementarity (lines), under the rotated syllabus and under a second syllabus in "
"which complementarity is zero at the start, peaks mid-way and returns to zero. Refusals rise with "
"generation under both; with generation held fixed they do not track complementarity (partial "
"Spearman $\\rho = -0.07$, 95\\% CI $-0.21$ to $0.09$, $n = 36$). (D--F) The simulation that "
"motivated the design: 60 agents evolving on a rugged fitness landscape with all four mechanisms "
"(grounded evaluation, recombination, diversity preservation, mutation) and one removed per arm (12 "
"replicates; mean and 95\\% CI). Removing grounded evaluation, so that agents are scored on "
"agreement with the crowd instead of on the truth, collapses the population onto a confident but "
"wrong consensus (D, F); removing recombination or diversity preservation strands it below the "
"optimum (D) and drains diversity fastest (E). Each removal fails in its own way."),
"fig5": (["paper/pnas/figs/fig5.pdf"],
"Model speciation: when two lineages can no longer merge. (A, B) Small image classifiers "
"(multilayer perceptrons) forked from one trained base. Two networks that compute the same function "
"can still differ in their weights, because hidden units can be renumbered and rescaled without "
"changing the output; alignment undoes this before averaging. The merge error barrier is how much "
"worse the average of two networks is than the networks themselves. (A) Two copies trained from "
"different random starts on the same task have a barrier that alignment removes almost entirely "
"(0.04 to 0.001); two trained on conflicting labels (the same images, some classes relabelled) keep "
"theirs (0.50), and their average is useless (3 replicates). (B) Sweeping the share of classes in "
"conflict moves the merged model's accuracy from 0.97 to 0.03. (C) Language models: two specialists "
"share a set of ambiguous questions (``sort this list'', direction unstated) and are taught opposite "
"conventions. As the share of conflicting training grows, each parent stays good under its own "
"convention while the merged model falls below both, in all three seeds (95\\% CI shaded). (D) The "
"control: specialists trained longer and longer on different tasks, with no conflict, merge better, "
"not worse, in every seed. (E, F) The simulation: 20-position genotypes carrying incompatibilities "
"of the Bateson--Dobzhansky--Muller kind. Hybrid fitness tracks the parents while lineages are "
"compatible, then crashes, sooner the denser the incompatibilities (E), and the probability of a "
"non-viable hybrid rises with divergence (F). What breaks merging is conflicting conventions on "
"shared machinery, not distance or specialisation as such."),
"fig3": (["paper/pnas/figs/fig3.pdf"],
"Merging language-model specialists: when it helps, and predicting when it will hurt. All models "
"are built from one frozen base (Qwen2.5) plus a LoRA adapter, a small set of extra weights trained "
"on one family of tasks (list puzzles, string puzzles or arithmetic); a verifier marks every answer "
"right or wrong, and accuracy is the share marked right on held-out questions. (A) The models "
"compared: the base alone; three specialists (one adapter each); their merge by averaging the "
"adapters; their merge after dropping the changes on which the parents conflict (TIES); and routing, "
"which keeps the specialists separate and sends each question to the one that owns it. (B) Easy "
"tasks, 0.5-billion-parameter base, five training seeds (fixed test sets; mean and 95\\% CI). Both "
"merges beat the best single specialist on the weakest task family (paired t-test over seeds, "
"$p < 10^{-4}$), and the interference-aware merge beats it overall ($p = 0.006$; the plain average "
"$p = 0.09$, ahead in 4 of 5 seeds); the two merges do not differ from each other. Only merged models "
"are competent on every family. (C) Deliberately hard tasks, 7-billion-parameter base, three seeds. "
"Averaging only matches the best specialist overall ($p = 0.96$) although it lifts the weakest "
"family ($p = 0.009$); routing beats averaging overall ($p = 0.007$, ahead in every seed) and beats "
"the best specialist on both measures ($p = 0.018$ and $0.014$). With three seeds, some comparisons "
"that hold in every seed are not significant (ns). (D) Predicting merge damage before merging: 39 "
"pairs of specialists built along three axes, parents taught contradictory conventions (red), "
"parents merely trained longer on different tasks (blue), and parents sharing training data without "
"conflict (green). The horizontal axis is how often the two parents confidently disagree when asked "
"the same questions before merging; the vertical axis is the merge penalty, the accuracy the merged "
"model loses relative to answering each task with the parent that owns it. Damage concentrates in "
"the conflicting pairs. (E) Six pre-merge measures ranked by how strongly they track the penalty "
"(absolute Spearman correlation): measures taken by asking the parents questions carry the signal, "
"measures taken on the parents' weights do not; differences between individual predictors are not "
"significant at this sample size (Table~S2)."),
}
UNICODE = {
# Multi-character sequences first: esc() applies these in insertion order.
"⁽ᵏ⁾": r"\(^{(k)}\)", "⁻³": r"\(^{-3}\)", "⁻⁴": r"\(^{-4}\)",
"—": "---", "–": "--", "→": r"\(\rightarrow\)", "≈": r"\(\approx\)", "≥": r"\(\geq\)",
"≳": r"\(\gtrsim\)", "×": r"\(\times\)", "·": r"\(\cdot\)", "μ": r"\(\mu\)",
"ρ": r"\(\rho\)", "≤": r"\(\leq\)", "≪": r"\(\ll\)", "∝": r"\(\propto\)",
# Greek and math symbols: the typewriter font has no Greek, so these must become math.
"α": r"\(\alpha\)", "β": r"\(\beta\)", "ε": r"\(\varepsilon\)", "Δ": r"\(\Delta\)",
"Σ": r"\(\Sigma\)", "∈": r"\(\in\)", "≠": r"\(\neq\)", "±": r"\(\pm\)",
"∼": r"\(\sim\)", "−": r"\(-\)", "²": r"\(^{2}\)", "³": r"\(^{3}\)", "⁴": r"\(^{4}\)",
}
SPECIALS = {"&": r"\&", "%": r"\%", "#": r"\#", "_": r"\_", "$": r"\$",
"~": r"\textasciitilde{}", "^": r"\textasciicircum{}"}
def esc(s: str) -> str:
s = s.replace("\\", r"\textbackslash{}")
for k, v in SPECIALS.items():
s = s.replace(k, v)
for k, v in UNICODE.items():
s = s.replace(k, v)
return s
def inline(s: str) -> str:
# Code spans are stashed behind sentinels *before* emphasis is applied, so that an italic or
# bold span containing `code` is still matched as one span (splitting first would break it).
codes: list[str] = []
def stash(m: re.Match) -> str:
codes.append(r"\texttt{" + esc(m.group(0)[1:-1]) + "}")
return f"\x00{len(codes) - 1}\x00"
s = re.sub(r"`[^`]*`", stash, s)
s = esc(s)
s = re.sub(r"\[([^\]]+)\]\((https?://[^)]+)\)", r"\\href{\2}{\1}", s)
s = re.sub(r"\*\*([^*]+)\*\*", r"\\textbf{\1}", s)
s = re.sub(r"\*([^*]+)\*", r"\\emph{\1}", s)
s = re.sub(r'"([^"]+)"', r"``\1''", s)
return re.sub(r"\x00(\d+)\x00", lambda m: codes[int(m.group(1))], s)
def figure_env(name: str) -> str:
if name in SI_FIGURES:
# Supplementary figure: copy the committed per-experiment PDF beside the SI build, so the SI
# stays a pure function of the results artifacts; numbered by the LaTeX counter (S-prefixed).
src, caption = SI_FIGURES[name]
SI_FIG_DIR.mkdir(parents=True, exist_ok=True)
dst = SI_FIG_DIR / f"{name}_{(ROOT / src).name}"
shutil.copyfile(ROOT / src, dst)
return "\n".join([f"\\begin{{figure}}[p]\\centering % {name}",
f"\\includegraphics[width=\\textwidth]{{figs/si/{dst.name}}}",
f"\\caption{{{caption}}}\\label{{{name}}}",
"\\end{figure}"])
pdfs, caption = FIGURES[name]
incl = "\\\\[6pt]\n".join(f"\\includegraphics[width=\\textwidth]{{figs/{(ROOT / p).name}}}"
for p in pdfs)
lines = [f"\\begin{{figure*}}[p]\\centering % {name}",
incl,
f"\\caption{{{caption}}}\\label{{{name}}}",
"\\end{figure*}"]
return "\n".join(lines)
def convert(text: str) -> str:
lines = text.split("\n")
# Skip the title block only when the document opens with one (main.md separates it with a rule
# in the first few lines); si.md has no such block, so nothing is dropped there.
i = 0
head = [n for n, ln in enumerate(lines[:10]) if ln.strip() == "---"]
if head:
i = head[0] + 1
blocks: list[list[str]] = []
cur: list[str] = []
for line in lines[i:]:
if line.strip() == "":
if cur:
blocks.append(cur); cur = []
else:
cur.append(line)
if cur:
blocks.append(cur)
def emit_table(block, out):
rows = [[c.strip() for c in line.strip().strip("|").split("|")] for line in block]
header, body = rows[0], rows[2:]
n = len(header)
widths = " ".join([f"p{{{0.92 / n:.3f}\\textwidth}}"] * n)
out += ["\\medskip\\noindent\\begin{center}\\footnotesize",
f"\\begin{{tabular}}{{{widths}}}", "\\hline",
" & ".join(inline(c) for c in header) + " \\\\ \\hline"]
for r in body:
r = (r + [""] * n)[:n]
out.append(" & ".join(inline(c) for c in r) + " \\\\[3pt]")
out += ["\\hline\\end{tabular}\\end{center}\\medskip", ""]
out: list[str] = []
for block in blocks:
first = block[0].strip()
m = re.match(r"^\*?\(FIG:(\w+)\)\*?$", first)
if m:
out.append(figure_env(m.group(1))); out.append("")
elif first.startswith("|") and len(block) >= 2 and set(block[1].strip()) <= set("|-: "):
emit_table(block, out)
elif first == "---" and len(block) == 1:
out.append("\\medskip\\hrule\\medskip"); out.append("")
elif first.startswith("# ") and not first.startswith("## "):
continue # document title: the wrapper supplies it
elif first.startswith("## "):
out.append(f"\\section*{{{inline(first[3:])}}}"); out.append("")
elif first.startswith("### "):
out.append(f"\\subsection*{{{inline(first[4:])}}}"); out.append("")
elif re.match(r"^(- |\d+\. )", first):
env = "itemize" if first.startswith("- ") else "enumerate"
out.append(f"\\begin{{{env}}}")
items: list[str] = []
for l in block:
s = l.strip()
if re.match(r"^(- |\d+\. )", s):
items.append(re.sub(r"^(- |\d+\. )", "", s))
else:
items[-1] += " " + s
for it in items:
out.append("\\item " + inline(it.strip()))
out.append(f"\\end{{{env}}}"); out.append("")
else:
joined = re.sub(r"\s{2,}", " ", " ".join(l.strip() for l in block)).strip()
out.append(inline(joined)); out.append("")
return "\n".join(out) + "\n"
if __name__ == "__main__":
import sys
doc = sys.argv[1] if len(sys.argv) > 1 else "main"
src = HERE / f"{doc}.md"
out = HERE / ("body.tex" if doc == "main" else f"{doc}_body.tex")
out.write_text(convert(src.read_text()))
print(f"wrote {out}")