Giorgio Gilestro
ab3dc10587
- paper/pnas -> paper/manuscript (venue-neutral)
- configs/layer1 -> configs/inheritance, src/knowledge -> src/inheritance
(imported as `inheritance`), make layer1 -> make inheritance; layer2 alias dropped
- inheritance and trained-network bundles named after the manuscript figure
they feed (fig2_grounding_sweep, figS3_rebaselining, ...), or descriptively
where they feed none; configs keep their `experiment:` value so parquet
hashes are unchanged, only output.dir moves
- figure scripts, SI figure sources, notebooks, REPRODUCING.md, README and the
SI Methods/tables updated; make clean no longer deletes tracked manifests;
reproduce.sh hashes the s{seed}/ layouts too
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y64o8FKP7rCuXzC48pxpMm
271 lines
17 KiB
Python
271 lines
17 KiB
Python
r"""Build the manuscript-draft PDF from main.md (Markdown stays the source of truth).
|
||
|
||
Adapted from paper/arxiv/md2tex.py (same Markdown subset + pipe tables), with one addition: standalone
|
||
`*(FIG:name)*` markers place the publication figures produced by make_figs.py (unified, lettered,
|
||
codename-free panels re-plotted from the committed artifacts). Run make_figs.py before building when
|
||
results change.
|
||
|
||
Usage: python paper/manuscript/build.py && (cd paper/manuscript && tectonic main.tex)
|
||
"""
|
||
|
||
from __future__ import annotations
|
||
|
||
import re
|
||
import shutil
|
||
from pathlib import Path
|
||
|
||
ROOT = Path(__file__).resolve().parents[2]
|
||
HERE = Path(__file__).resolve().parent
|
||
SRC = HERE / "main.md"
|
||
OUT = HERE / "body.tex"
|
||
SI_FIG_DIR = HERE / "figs" / "si"
|
||
|
||
import sys # noqa: E402
|
||
|
||
sys.path.insert(0, str(HERE))
|
||
from si_figures import SI_FIGURES # noqa: E402 (supplementary figures, numbered by order in si.md)
|
||
|
||
# figure name -> (single publication PDF from make_figs.py, caption)
|
||
FIGURES: dict[str, tuple[list[str], str]] = {
|
||
"fig1": (["paper/manuscript/figs/fig1a.pdf", "paper/manuscript/figs/fig1b.pdf"],
|
||
"A map of the study. (A) Each row is a biological mechanism the paper borrows, each column a level "
|
||
"of realism at which it is tested: an inheritance model (an exact simulation of knowledge "
|
||
"transmission, green), trained neural networks measured against exact oracles (blue), and language "
|
||
"models (blue). Filled cells name the experiments run at each level and, in the corner, the figure "
|
||
"or table reporting them; grey cells were not run, either because the result is "
|
||
"established in prior work (23, 34) or because that level adds no new test for that question. The "
|
||
"inheritance model is the reference column: it sets the expectation the real-model experiments are "
|
||
"read against. (B) The change of viewpoint the transfer rests on. A group of models is usually "
|
||
"pictured as a society in space, contemporaries exchanging messages. The couplings studied here run "
|
||
"between generations: training on model output (inheritance), weight-space merging (recombination), "
|
||
"and verified real data entering each generation (immigration from reality). That is a society in "
|
||
"time, which is what population genetics describes. Dots are capabilities: the rare one (gold) is "
|
||
"lost under single-parent inheritance, reassembled by merging complementary parents, and re-supplied "
|
||
"by grounding."),
|
||
"fig2": (["paper/manuscript/figs/fig2.pdf"],
|
||
"How much real data stops model collapse. (A) An image-generating network (a variational "
|
||
"autoencoder) is trained on handwritten digits, then a fresh copy is trained only on the digits the "
|
||
"previous one drew, for fifteen generations, with no real data added. Each row is a later generation "
|
||
"(0, 4, 8, 12, 15) and each column a randomly chosen drawing. The thirty kinds of digit (ten digits "
|
||
"$\\times$ three stroke thicknesses, some kinds rare) collapse to one blurred shape; an independent "
|
||
"classifier confirms that the number of kinds still drawn falls from 30 to 1, while adding 10\\% real "
|
||
"digits each generation keeps all 30 (Fig.~S7; 4 replicates). (B) The same question in the "
|
||
"inheritance model, the exact simulation: 1,000 knowledge items, 200 samples drawn per generation, "
|
||
"and a fraction $g$ of fresh real samples mixed in. Points are the diversity the population settles "
|
||
"at after 500 generations (mean and 95\\% CI over 100 lineages), the dashed line the exact prediction "
|
||
"(the immigration--drift equilibrium), the dotted line the diversity of the real data itself. The "
|
||
"curve is smooth, so any threshold is a choice: the red line marks the $g$ at which 95\\% of the real "
|
||
"data's diversity is kept, about 0.05 (bootstrap CI shaded). The hollow point at $g = 0$ has not yet "
|
||
"reached its equilibrium of zero. The trained image model needed about twice this fraction, because "
|
||
"a trained network is not the exact copier the simulation assumes (Fig.~S2)."),
|
||
"fig4": (["paper/manuscript/figs/fig4.pdf"],
|
||
"A population of language models over six generations. (A) The set-up. Three lineages start "
|
||
"from one frozen 1.5-billion-parameter base (Qwen2.5-1.5B). Each generation, every lineage learns "
|
||
"one new skill from a public dataset by continuing to train its parent's adapter (300 new examples "
|
||
"plus 150 replayed from earlier skills), may merge with a partner according to its arm's rule "
|
||
"(weights averaged at a ratio chosen on validation data), and is tested on all six skills by a "
|
||
"verifier; the child becomes the next parent. The six skills are taken in rotated order, so a "
|
||
"partner knows things a lineage lacks early on (complementarity 1.0) and nothing it lacks by the "
|
||
"end (0.0). Three training seeds. (B) Accuracy over all six skills of the best lineage (mean and "
|
||
"95\\% CI). Never merging and merging only when it beats keeping the parent finish level (0.80 and "
|
||
"0.79); merging with a contemporary every generation collapses to 0.27, beginning when partners "
|
||
"stop being complementary; a control that merges through generation 2 and then stops (dashed) "
|
||
"matches the declinable arm in every seed, and a single model taught the whole syllabus alone "
|
||
"(diamond) matches the population. (C) How often the declinable lineages refused a merge (bars) "
|
||
"against partner complementarity (lines), under the rotated syllabus and under a second syllabus in "
|
||
"which complementarity is zero at the start, peaks mid-way and returns to zero. Refusals rise with "
|
||
"generation under both; with generation held fixed they do not track complementarity (partial "
|
||
"Spearman $\\rho = -0.07$, 95\\% CI $-0.21$ to $0.09$, $n = 36$). (D--F) The simulation that "
|
||
"motivated the design: 60 agents evolving on a rugged fitness landscape with all four mechanisms "
|
||
"(grounded evaluation, recombination, diversity preservation, mutation) and one removed per arm (12 "
|
||
"replicates; mean and 95\\% CI). Removing grounded evaluation, so that agents are scored on "
|
||
"agreement with the crowd instead of on the truth, collapses the population onto a confident but "
|
||
"wrong consensus (D, F); removing recombination or diversity preservation strands it below the "
|
||
"optimum (D) and drains diversity fastest (E). Each removal fails in its own way."),
|
||
"fig5": (["paper/manuscript/figs/fig5.pdf"],
|
||
"Model speciation: when two lineages can no longer merge. (A, B) Small image classifiers "
|
||
"(multilayer perceptrons) forked from one trained base. Two networks that compute the same function "
|
||
"can still differ in their weights, because hidden units can be renumbered and rescaled without "
|
||
"changing the output; alignment undoes this before averaging. The merge error barrier is how much "
|
||
"worse the average of two networks is than the networks themselves. (A) Two copies trained from "
|
||
"different random starts on the same task have a barrier that alignment removes almost entirely "
|
||
"(0.04 to 0.001); two trained on conflicting labels (the same images, some classes relabelled) keep "
|
||
"theirs (0.50), and their average is useless (3 replicates). (B) Sweeping the share of classes in "
|
||
"conflict moves the merged model's accuracy from 0.97 to 0.03. (C) Language models: two specialists "
|
||
"share a set of ambiguous questions (``sort this list'', direction unstated) and are taught opposite "
|
||
"conventions. As the share of conflicting training grows, each parent stays good under its own "
|
||
"convention while the merged model falls below both, in all three seeds (95\\% CI shaded). (D) The "
|
||
"control: specialists trained longer and longer on different tasks, with no conflict, merge better, "
|
||
"not worse, in every seed. (E, F) The simulation: 20-position genotypes carrying incompatibilities "
|
||
"of the Bateson--Dobzhansky--Muller kind. Hybrid fitness tracks the parents while lineages are "
|
||
"compatible, then crashes, sooner the denser the incompatibilities (E), and the probability of a "
|
||
"non-viable hybrid rises with divergence (F). What breaks merging is conflicting conventions on "
|
||
"shared machinery, not distance or specialisation as such."),
|
||
"fig3": (["paper/manuscript/figs/fig3.pdf"],
|
||
"Merging language-model specialists: when it helps, and predicting when it will hurt. All models "
|
||
"are built from one frozen base (Qwen2.5) plus a LoRA adapter, a small set of extra weights trained "
|
||
"on one family of tasks (list puzzles, string puzzles or arithmetic); a verifier marks every answer "
|
||
"right or wrong, and accuracy is the share marked right on held-out questions. (A) The models "
|
||
"compared: the base alone; three specialists (one adapter each); their merge by averaging the "
|
||
"adapters; their merge after dropping the changes on which the parents conflict (TIES); and routing, "
|
||
"which keeps the specialists separate and sends each question to the one that owns it. (B) Easy "
|
||
"tasks, 0.5-billion-parameter base, five training seeds (fixed test sets; mean and 95\\% CI). Both "
|
||
"merges beat the best single specialist on the weakest task family (paired t-test over seeds, "
|
||
"$p < 10^{-4}$), and the interference-aware merge beats it overall ($p = 0.006$; the plain average "
|
||
"$p = 0.09$, ahead in 4 of 5 seeds); the two merges do not differ from each other. Only merged models "
|
||
"are competent on every family. (C) Deliberately hard tasks, 7-billion-parameter base, three seeds. "
|
||
"Averaging only matches the best specialist overall ($p = 0.96$) although it lifts the weakest "
|
||
"family ($p = 0.009$); routing beats averaging overall ($p = 0.007$, ahead in every seed) and beats "
|
||
"the best specialist on both measures ($p = 0.018$ and $0.014$). With three seeds, some comparisons "
|
||
"that hold in every seed are not significant (ns). (D) Predicting merge damage before merging: 39 "
|
||
"pairs of specialists built along three axes, parents taught contradictory conventions (red), "
|
||
"parents merely trained longer on different tasks (blue), and parents sharing training data without "
|
||
"conflict (green). The horizontal axis is how often the two parents confidently disagree when asked "
|
||
"the same questions before merging; the vertical axis is the merge penalty, the accuracy the merged "
|
||
"model loses relative to answering each task with the parent that owns it. Damage concentrates in "
|
||
"the conflicting pairs. (E) Six pre-merge measures ranked by how strongly they track the penalty "
|
||
"(absolute Spearman correlation): measures taken by asking the parents questions carry the signal, "
|
||
"measures taken on the parents' weights do not; differences between individual predictors are not "
|
||
"significant at this sample size (Table~S2)."),
|
||
}
|
||
|
||
UNICODE = {
|
||
# Multi-character sequences first: esc() applies these in insertion order.
|
||
"⁽ᵏ⁾": r"\(^{(k)}\)", "⁻³": r"\(^{-3}\)", "⁻⁴": r"\(^{-4}\)",
|
||
"—": "---", "–": "--", "→": r"\(\rightarrow\)", "≈": r"\(\approx\)", "≥": r"\(\geq\)",
|
||
"≳": r"\(\gtrsim\)", "×": r"\(\times\)", "·": r"\(\cdot\)", "μ": r"\(\mu\)",
|
||
"ρ": r"\(\rho\)", "≤": r"\(\leq\)", "≪": r"\(\ll\)", "∝": r"\(\propto\)",
|
||
# Greek and math symbols: the typewriter font has no Greek, so these must become math.
|
||
"α": r"\(\alpha\)", "β": r"\(\beta\)", "ε": r"\(\varepsilon\)", "Δ": r"\(\Delta\)",
|
||
"Σ": r"\(\Sigma\)", "∈": r"\(\in\)", "≠": r"\(\neq\)", "±": r"\(\pm\)",
|
||
"∼": r"\(\sim\)", "−": r"\(-\)", "²": r"\(^{2}\)", "³": r"\(^{3}\)", "⁴": r"\(^{4}\)",
|
||
}
|
||
SPECIALS = {"&": r"\&", "%": r"\%", "#": r"\#", "_": r"\_", "$": r"\$",
|
||
"~": r"\textasciitilde{}", "^": r"\textasciicircum{}"}
|
||
|
||
|
||
def esc(s: str) -> str:
|
||
s = s.replace("\\", r"\textbackslash{}")
|
||
for k, v in SPECIALS.items():
|
||
s = s.replace(k, v)
|
||
for k, v in UNICODE.items():
|
||
s = s.replace(k, v)
|
||
return s
|
||
|
||
|
||
def inline(s: str) -> str:
|
||
# Code spans are stashed behind sentinels *before* emphasis is applied, so that an italic or
|
||
# bold span containing `code` is still matched as one span (splitting first would break it).
|
||
codes: list[str] = []
|
||
|
||
def stash(m: re.Match) -> str:
|
||
codes.append(r"\texttt{" + esc(m.group(0)[1:-1]) + "}")
|
||
return f"\x00{len(codes) - 1}\x00"
|
||
|
||
s = re.sub(r"`[^`]*`", stash, s)
|
||
s = esc(s)
|
||
s = re.sub(r"\[([^\]]+)\]\((https?://[^)]+)\)", r"\\href{\2}{\1}", s)
|
||
s = re.sub(r"\*\*([^*]+)\*\*", r"\\textbf{\1}", s)
|
||
s = re.sub(r"\*([^*]+)\*", r"\\emph{\1}", s)
|
||
s = re.sub(r'"([^"]+)"', r"``\1''", s)
|
||
return re.sub(r"\x00(\d+)\x00", lambda m: codes[int(m.group(1))], s)
|
||
|
||
|
||
def figure_env(name: str) -> str:
|
||
if name in SI_FIGURES:
|
||
# Supplementary figure: copy the committed per-experiment PDF beside the SI build, so the SI
|
||
# stays a pure function of the results artifacts; numbered by the LaTeX counter (S-prefixed).
|
||
src, caption = SI_FIGURES[name]
|
||
SI_FIG_DIR.mkdir(parents=True, exist_ok=True)
|
||
dst = SI_FIG_DIR / f"{name}_{(ROOT / src).name}"
|
||
shutil.copyfile(ROOT / src, dst)
|
||
return "\n".join([f"\\begin{{figure}}[p]\\centering % {name}",
|
||
f"\\includegraphics[width=\\textwidth]{{figs/si/{dst.name}}}",
|
||
f"\\caption{{{caption}}}\\label{{{name}}}",
|
||
"\\end{figure}"])
|
||
pdfs, caption = FIGURES[name]
|
||
incl = "\\\\[6pt]\n".join(f"\\includegraphics[width=\\textwidth]{{figs/{(ROOT / p).name}}}"
|
||
for p in pdfs)
|
||
lines = [f"\\begin{{figure*}}[p]\\centering % {name}",
|
||
incl,
|
||
f"\\caption{{{caption}}}\\label{{{name}}}",
|
||
"\\end{figure*}"]
|
||
return "\n".join(lines)
|
||
|
||
|
||
def convert(text: str) -> str:
|
||
lines = text.split("\n")
|
||
# Skip the title block only when the document opens with one (main.md separates it with a rule
|
||
# in the first few lines); si.md has no such block, so nothing is dropped there.
|
||
i = 0
|
||
head = [n for n, ln in enumerate(lines[:10]) if ln.strip() == "---"]
|
||
if head:
|
||
i = head[0] + 1
|
||
|
||
blocks: list[list[str]] = []
|
||
cur: list[str] = []
|
||
for line in lines[i:]:
|
||
if line.strip() == "":
|
||
if cur:
|
||
blocks.append(cur); cur = []
|
||
else:
|
||
cur.append(line)
|
||
if cur:
|
||
blocks.append(cur)
|
||
|
||
def emit_table(block, out):
|
||
rows = [[c.strip() for c in line.strip().strip("|").split("|")] for line in block]
|
||
header, body = rows[0], rows[2:]
|
||
n = len(header)
|
||
widths = " ".join([f"p{{{0.92 / n:.3f}\\textwidth}}"] * n)
|
||
out += ["\\medskip\\noindent\\begin{center}\\footnotesize",
|
||
f"\\begin{{tabular}}{{{widths}}}", "\\hline",
|
||
" & ".join(inline(c) for c in header) + " \\\\ \\hline"]
|
||
for r in body:
|
||
r = (r + [""] * n)[:n]
|
||
out.append(" & ".join(inline(c) for c in r) + " \\\\[3pt]")
|
||
out += ["\\hline\\end{tabular}\\end{center}\\medskip", ""]
|
||
|
||
out: list[str] = []
|
||
for block in blocks:
|
||
first = block[0].strip()
|
||
m = re.match(r"^\*?\(FIG:(\w+)\)\*?$", first)
|
||
if m:
|
||
out.append(figure_env(m.group(1))); out.append("")
|
||
elif first.startswith("|") and len(block) >= 2 and set(block[1].strip()) <= set("|-: "):
|
||
emit_table(block, out)
|
||
elif first == "---" and len(block) == 1:
|
||
out.append("\\medskip\\hrule\\medskip"); out.append("")
|
||
elif first.startswith("# ") and not first.startswith("## "):
|
||
continue # document title: the wrapper supplies it
|
||
elif first.startswith("## "):
|
||
out.append(f"\\section*{{{inline(first[3:])}}}"); out.append("")
|
||
elif first.startswith("### "):
|
||
out.append(f"\\subsection*{{{inline(first[4:])}}}"); out.append("")
|
||
elif re.match(r"^(- |\d+\. )", first):
|
||
env = "itemize" if first.startswith("- ") else "enumerate"
|
||
out.append(f"\\begin{{{env}}}")
|
||
items: list[str] = []
|
||
for l in block:
|
||
s = l.strip()
|
||
if re.match(r"^(- |\d+\. )", s):
|
||
items.append(re.sub(r"^(- |\d+\. )", "", s))
|
||
else:
|
||
items[-1] += " " + s
|
||
for it in items:
|
||
out.append("\\item " + inline(it.strip()))
|
||
out.append(f"\\end{{{env}}}"); out.append("")
|
||
else:
|
||
joined = re.sub(r"\s{2,}", " ", " ".join(l.strip() for l in block)).strip()
|
||
out.append(inline(joined)); out.append("")
|
||
return "\n".join(out) + "\n"
|
||
|
||
|
||
if __name__ == "__main__":
|
||
import sys
|
||
|
||
doc = sys.argv[1] if len(sys.argv) > 1 else "main"
|
||
src = HERE / f"{doc}.md"
|
||
out = HERE / ("body.tex" if doc == "main" else f"{doc}_body.tex")
|
||
out.write_text(convert(src.read_text()))
|
||
print(f"wrote {out}")
|