- paper/pnas -> paper/manuscript (venue-neutral)
- configs/layer1 -> configs/inheritance, src/knowledge -> src/inheritance
(imported as `inheritance`), make layer1 -> make inheritance; layer2 alias dropped
- inheritance and trained-network bundles named after the manuscript figure
they feed (fig2_grounding_sweep, figS3_rebaselining, ...), or descriptively
where they feed none; configs keep their `experiment:` value so parquet
hashes are unchanged, only output.dir moves
- figure scripts, SI figure sources, notebooks, REPRODUCING.md, README and the
SI Methods/tables updated; make clean no longer deletes tracked manifests;
reproduce.sh hashes the s{seed}/ layouts too
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y64o8FKP7rCuXzC48pxpMm
169 lines
14 KiB
Markdown
169 lines
14 KiB
Markdown
|
||
**2026-08-11 — Stop padding time estimates.** GG: "You generally massively overestimate the time it
|
||
takes to do some work... You're fast." Phase-1 items I scoped as "Week 1" took ~an hour. Rule: state
|
||
what will be done and in what order; give a duration only when compute-bound (training walltime), and
|
||
base it on measured runtimes, not human-project heuristics.
|
||
|
||
**2026-08-11 — Correspondence is not identity; interpretation is not prediction (manuscript reviews).**
|
||
The external review's core corrections, to internalise for all paper claims: (1) distinguish
|
||
interpretation / explanation / prediction and claim only the level the evidence supports; (2) a
|
||
minimal model being exactly Wright-Fisher does not make real training "literally" WF — our own
|
||
learning-kernel result says otherwise (cite it against ourselves); (3) name the operator every claim
|
||
is about (output-mean vs weight-average vs max-with-oracle vs routing are different objects with
|
||
different budgets); (4) don't write "nobody has / none imports" — invite no priority disputes; say
|
||
"to our knowledge" and state the positive contribution; (5) negative results (E13b) are strengths —
|
||
lead with them; (6) "control theory" needs states/controls/dynamics/rule or it's a "framework".
|
||
|
||
**2026-08-11 — tmux does NOT survive SSH disconnects on this machine.** The tmux server starts inside
|
||
the SSH session's systemd scope and gets reaped on logout (lost ~20 min of the epistasis grid; GG:
|
||
"SSH disconnected"). Reliable pattern here: `systemd-run --user --collect --unit=<name>
|
||
--working-directory="$PWD" bash -c '<cmd>'` — lands in user@.service (kept alive by the desktop
|
||
session), survives disconnects; check with `systemctl --user is-active <name>`, logs via redirect.
|
||
|
||
**2026-08-11 — Internal deliberations must not leak into reader-facing prose.** GG (on the
|
||
convergence paragraph): it "carries all the ghosts of our internal brainstorming that should be
|
||
completely alien to the readers (this is, alas, a recurring problem with your writing)." The pattern:
|
||
strategic concerns from the working process (priority anxieties -> "we cite for priority of
|
||
publication"; calibration debates -> "the honest statement"; positioning battles -> "nobody has")
|
||
surface as defensive or self-referential prose. Rule: before finalising any reader-facing passage,
|
||
ask "does understanding this sentence require knowing how we worked?" If yes, rewrite it as a plain
|
||
statement about the literature or the result. Confident papers situate; they do not litigate.
|
||
(Promoted to a general rule in the global ~/.claude/CLAUDE.md, 2026-08-11 — it applies to all
|
||
reader-facing prose in every project.)
|
||
|
||
## Terminology: "model" is overloaded in this project (2026-09-07)
|
||
In the PNAS manuscript and figures, "model" must mean an AI model. The settled term for the
|
||
pop-gen construct is **"the biological model"** (GG decision, 2026-09-07) — used everywhere
|
||
("the multi-locus biological model", "Biological-model tier"), never "minimal model",
|
||
"analytic model", or "exact model"; GG also vetoed "(exact)" as a tier label (the guarantee
|
||
line "closed forms · bitwise-reproducible" carries that content). Support level formerly
|
||
"Exact" is now "Closed form". Keep "exact" only in technical noun phrases (exact-match
|
||
verifier, exact oracle, exact equilibrium, exact recovery).
|
||
|
||
## Never `uv sync` while a job is using the venv (2026-09-07)
|
||
Adding the `notebooks` extra mid-session ran `uv sync` **without** `--extra llm`, which rebuilt the
|
||
shared `.venv` — dropping the LLM stack *and* silently switching the interpreter 3.14 → 3.11 (uv
|
||
recreates with the system default when no `.python-version` is pinned). That killed the running
|
||
local society seed with a `FileNotFoundError` deep in `huggingface_hub` templates — a failure that
|
||
looked scientific but was pure environment churn.
|
||
Rules: (1) never mutate `.venv` while a background job is running against it — wait, or build a
|
||
throwaway venv elsewhere; (2) `uv sync` is *declarative* — always pass **every** extra the project
|
||
needs, or it removes the ones you omit; (3) the repo now pins `.python-version` (3.14) so the
|
||
interpreter can never drift silently.
|
||
|
||
## Before a compute campaign, derive the design from the theory and read the analytic code beside the neural code (2026-09-07)
|
||
The v1 `llm_society` campaign (4 seeds, ~13 L40S-h) returned a null that was *structurally guaranteed*:
|
||
3 families over 8 agents made founders near-clones (E8's ρ=1 control: recombining clones buys nothing);
|
||
2^3 competence states left no room for a child to be "new" (E8 needs L=12); sex was a linear blend at
|
||
0.5B (E4/`llm_moe`: the dilution regime); parents were truncated before breeding (E11 selects on
|
||
survival over parents+offspring — v1 threw away half the families at gen 1 with no operator to
|
||
restore them); `n_test`=40 put every contrast inside one SE. GG caught the first fault by asking what
|
||
the founders knew; the rest fell out of comparing `dynamic_society.py` with `society.py` line by line.
|
||
Rules: (1) a Layer-2 instantiation of an analytic experiment must be checked *operator by operator*
|
||
against the analytic code, not against its description; (2) every free parameter that the theory
|
||
constrains (ρ, L, the operator regime, the observation floor, selection intensity) is set by a
|
||
prediction or a calibration measurement, never by feel; (3) write the falsifiers with numbers and the
|
||
power analysis *before* submission — the pre-registration is `tasks/prereg-llm-society-v2.md`.
|
||
|
||
## Calibrate the inheritance channel before the population (2026-09-07, evening)
|
||
Nine local GPU-hours of calibration found three ceilings a 96-GPU-hour campaign would have hidden:
|
||
(1) with 3 skills over 8 agents, founders were near-clones (E8 ρ=1 → recombination buys nothing);
|
||
(2) a fresh LoRA distilled from a one-skill parent's answers on nine families retains only 0.6–0.8 of
|
||
the skill — interference from confident off-expertise answers, not the E2 observation floor, and
|
||
removable by gating on the source's own confidence; (3) a two-skill child holds each skill at
|
||
~0.85× of its parents at ANY training budget (rank 64 overfits) — the learning budget, not the
|
||
sample budget, is the conserved quantity, and it caps how many skills one adapter can carry. GG's
|
||
call was no-go at 0.5B rather than a campaign that could only test the ablations. Rules: (a) measure
|
||
transmission fidelity of the inheritance channel for one skill, then two, before breeding populations;
|
||
(b) when a gate fails, re-derive it from the data you already have (the conflict gate moved 0.35→0.41
|
||
from the epistasis grid's own outcomes) rather than by feel, and record it as an amendment; (c) a
|
||
pre-registration that ends in a no-go has done its job — write the ceiling up, don't route around it.
|
||
|
||
## A plan for reader-facing prose must carry the dual-audience standard explicitly (2026-09-09)
|
||
GG rejected the approved-in-substance manuscript-revision plan until it stated, as a first-class
|
||
section, that every term from either field is defined at first use with an example from each world.
|
||
The plan had the right content and structure but treated accessibility as a verification
|
||
afterthought; GG's rule is that it is "paramount" and must be designed in, not checked for. Rule:
|
||
before drafting any passage for a mixed readership, build the term table (term / one-clause
|
||
definition / biology example / model example) *in the plan*, and put a two-reader pass in
|
||
verification. The same applies to my status reports — GG twice said "I am lost with all these C3,
|
||
E9, H1"; spell codenames out.
|
||
|
||
## 2026-09-11 — check the figure panel inventory before flagging cross-references
|
||
In the manuscript review I flagged Table 1's "Fig. 4C–E" and "Fig. 3C" as inconsistent with the text. They were
|
||
correct: `make_figs.py` puts the E11 ablation in Fig. 4's bottom row and the predictive grid in Fig. 3C–D; the text
|
||
simply failed to cite them. Rule: before calling a cross-reference wrong, read `paper/manuscript/make_figs.py` and the
|
||
captions in `build.py` for the panel inventory; the fix is usually a missing citation in the text, not a wrong table.
|
||
|
||
## 2026-09-11 — never type a result number that a script has not printed
|
||
Writing the stop3 README I filled the per-seed cells from memory of the mean and had three rows wrong
|
||
until the loader's pivot table exposed it. Rule: every number in a README, SI table or manuscript is
|
||
pasted from a stats-script printout produced in the same step; if the script has not printed it,
|
||
print it first. This is the same rule the plan stated ("copied from the stats-script output, not
|
||
typed") and I broke it within the hour.
|
||
|
||
## Reference numbers hardcoded outside the renumber path (2026-09-11)
|
||
`renumber_refs.py` rewrites main.md, si.md and build.py captions, but `make_figs.py` carries a literal "(refs. N, M)" in the fig1a grounding cell, which went stale after references were added. Rule: after any renumber, grep `refs\.` and `ref\.` across `paper/manuscript/*.py` and fix by hand (or extend renumber_refs.py to cover make_figs.py). Also: fig text is rendered, so verify by `pdftotext figs/fig1a.pdf`, not by grepping the source alone.
|
||
|
||
## 2026-09-12 — GG's manuscript comments: heralds, undefined terms, and phantom SI references
|
||
Forty-five comments on the ODT. Three patterns. (1) **Herald sentences** ("Two boundaries follow.",
|
||
"Modifier theory predicts its fate:", "X is the measurement no other arm produces") — GG: "Breaking
|
||
down sentences like this is also a claudism." Now a HERALD block in the declaudify detector; run
|
||
`--list herald` before handing over any draft. (2) **Every technical term defined at first use, with
|
||
one word per concept**: item/capability/allele, mass, refit, verified real samples, Zipf source,
|
||
practitioner, "which trained networks" — the dual-audience rule from 2026-09-09 applied to *my own*
|
||
vocabulary, not only the biology. Rule: after drafting, grep each noun of art for its first
|
||
occurrence and check a definition precedes it. (3) **Never cite the SI for something the SI does not
|
||
contain.** The text cited "the SI separates three cases" and "the proof is Poisson thinning" and
|
||
neither existed; and every "(SI)" pointer must name a figure or text number. Rule: before writing
|
||
"(SI)" grep si.md for the claim; if absent, write it (SI Text S4) or drop the sentence.
|
||
|
||
## 2026-09-12 (round 2) — the Discussion must discuss; novelty is flagged where the result is shown
|
||
GG on the "What is borrowed and what is new" inventory: "The discussion should discuss, not list."
|
||
And on the Limits paragraph: stating small-scale limits "is usually done by undergraduate students";
|
||
either run the experiment or discuss only problems too big for the paper. Rules: (a) never write a
|
||
Discussion paragraph that is a list of prior-art citations or of caveats; each Discussion paragraph
|
||
argues one point; (b) attribute novelty at the point of the result, with the figure panel, and name
|
||
the prior finding it explains or extends in the same sentence ("an observation reported by others
|
||
and left unexplained (60)"); (c) "much/some/most of X was known" is a hedge that gives novelty away
|
||
without saying what is new; replace with the specific thing prior work lacks. Also: after
|
||
`renumber_refs.py --apply`, the fig1a literal (refs. 22, 33) went stale (Shumailov became 23); the lesson from 2026-09-11 held.
|
||
|
||
## 2026-09-12 — a prediction written into the Discussion must be run before it is printed
|
||
The revised Discussion predicted that differential reproduction would turn recombination's speed
|
||
advantage into a level advantage. Three GPU-hours later it did not (parity, 3/3 seeds). Rules:
|
||
(a) when a Discussion sentence forecasts the outcome of an experiment we can run in under a day,
|
||
run it in the same revision; (b) check adapter/cache directories for seed- and base-specificity
|
||
before any HPC array (speciation shared one dir across seeds; the specialist cache would have loaded
|
||
Qwen adapters into SmolLM2); (c) the local smoke gate for a new base (termination, base accuracy in
|
||
(0.05, 0.95), sample generations) cost 4 minutes and is worth running every time.
|
||
|
||
## 2026-09-13 — a figure must be readable without its caption
|
||
GG on the manuscript figures after reading the student guide: "too unclear, cryptic"; figures should
|
||
"give some clear information without the need to read the legend". The house rule in make_figs.py
|
||
("no per-panel headline titles; interpretation lives in the captions") was the wrong rule for this
|
||
audience and is reversed. Rules: (a) every data panel carries a one-line headline stating its
|
||
finding plus a grey line naming the system and its size; (b) legend entries say in words what is
|
||
plotted ("accuracy on the model's weakest task family", not "worst_family"); (c) where the set-up is
|
||
not obvious, a schematic panel explains it inside the figure; (d) bar comparisons carry the test
|
||
(paired over seeds, stars, key printed under the legend). Layout lesson: headlines longer than the
|
||
panel run into the neighbour; wrap at ~40 characters per line for a half-width panel, ~70 for full
|
||
width, and render before trusting.
|
||
|
||
## 2026-09-13 — figure layout rules I should apply without being told
|
||
GG had to ask three times for things a careful eye catches: headlines running past their panel,
|
||
a schematic strip narrower than the data panels beneath it and not flush with their left edge,
|
||
and a large blank band between a strip and the next row. Rules, now encoded in make_figs.py:
|
||
(a) any panel placed by hand (schematics) is positioned from the neighbouring data axes' geometry:
|
||
left edge = the data panels' frame, right edge = the last panel's frame, bottom = a fixed 0.75 in
|
||
above the headline below, height from the content's designed aspect (never let equal-aspect centre
|
||
a too-wide axes); (b) text wraps to its own panel width (`headline()` measures the axes); (c) after
|
||
every regeneration, render at ≥ 90 dpi and check four things before reporting: nothing crosses a
|
||
panel boundary, nothing overlaps, blank bands are no larger than the row gaps, and left edges of
|
||
stacked panels line up. Report only after that check passes.
|
||
|
||
## 2026-09-13 — prose: no staccato fragments
|
||
- GG flagged "These results say X. They do not say where. The inheritance model does, in closed form."
|
||
as a claudism. Breaking a thought into short declaratives is rarely necessary; join them (colon,
|
||
"because", "and", "but"). Clarity comes from stating the concrete object, not from short sentences.
|
||
- After any rewrite pass, scan for sentences of ≤7 words introduced by the edit and rejoin them.
|