MachineSex/tasks/lessons.md
Giorgio Gilestro ab3dc10587 Restructure: descriptive tier and experiment names, paper/manuscript
- paper/pnas -> paper/manuscript (venue-neutral)
- configs/layer1 -> configs/inheritance, src/knowledge -> src/inheritance
  (imported as `inheritance`), make layer1 -> make inheritance; layer2 alias dropped
- inheritance and trained-network bundles named after the manuscript figure
  they feed (fig2_grounding_sweep, figS3_rebaselining, ...), or descriptively
  where they feed none; configs keep their `experiment:` value so parquet
  hashes are unchanged, only output.dir moves
- figure scripts, SI figure sources, notebooks, REPRODUCING.md, README and the
  SI Methods/tables updated; make clean no longer deletes tracked manifests;
  reproduce.sh hashes the s{seed}/ layouts too

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y64o8FKP7rCuXzC48pxpMm
2026-09-13 17:00:40 +01:00

169 lines
14 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

**2026-08-11 — Stop padding time estimates.** GG: "You generally massively overestimate the time it
takes to do some work... You're fast." Phase-1 items I scoped as "Week 1" took ~an hour. Rule: state
what will be done and in what order; give a duration only when compute-bound (training walltime), and
base it on measured runtimes, not human-project heuristics.
**2026-08-11 — Correspondence is not identity; interpretation is not prediction (manuscript reviews).**
The external review's core corrections, to internalise for all paper claims: (1) distinguish
interpretation / explanation / prediction and claim only the level the evidence supports; (2) a
minimal model being exactly Wright-Fisher does not make real training "literally" WF — our own
learning-kernel result says otherwise (cite it against ourselves); (3) name the operator every claim
is about (output-mean vs weight-average vs max-with-oracle vs routing are different objects with
different budgets); (4) don't write "nobody has / none imports" — invite no priority disputes; say
"to our knowledge" and state the positive contribution; (5) negative results (E13b) are strengths —
lead with them; (6) "control theory" needs states/controls/dynamics/rule or it's a "framework".
**2026-08-11 — tmux does NOT survive SSH disconnects on this machine.** The tmux server starts inside
the SSH session's systemd scope and gets reaped on logout (lost ~20 min of the epistasis grid; GG:
"SSH disconnected"). Reliable pattern here: `systemd-run --user --collect --unit=<name>
--working-directory="$PWD" bash -c '<cmd>'` — lands in user@.service (kept alive by the desktop
session), survives disconnects; check with `systemctl --user is-active <name>`, logs via redirect.
**2026-08-11 — Internal deliberations must not leak into reader-facing prose.** GG (on the
convergence paragraph): it "carries all the ghosts of our internal brainstorming that should be
completely alien to the readers (this is, alas, a recurring problem with your writing)." The pattern:
strategic concerns from the working process (priority anxieties -> "we cite for priority of
publication"; calibration debates -> "the honest statement"; positioning battles -> "nobody has")
surface as defensive or self-referential prose. Rule: before finalising any reader-facing passage,
ask "does understanding this sentence require knowing how we worked?" If yes, rewrite it as a plain
statement about the literature or the result. Confident papers situate; they do not litigate.
(Promoted to a general rule in the global ~/.claude/CLAUDE.md, 2026-08-11 — it applies to all
reader-facing prose in every project.)
## Terminology: "model" is overloaded in this project (2026-09-07)
In the PNAS manuscript and figures, "model" must mean an AI model. The settled term for the
pop-gen construct is **"the biological model"** (GG decision, 2026-09-07) — used everywhere
("the multi-locus biological model", "Biological-model tier"), never "minimal model",
"analytic model", or "exact model"; GG also vetoed "(exact)" as a tier label (the guarantee
line "closed forms · bitwise-reproducible" carries that content). Support level formerly
"Exact" is now "Closed form". Keep "exact" only in technical noun phrases (exact-match
verifier, exact oracle, exact equilibrium, exact recovery).
## Never `uv sync` while a job is using the venv (2026-09-07)
Adding the `notebooks` extra mid-session ran `uv sync` **without** `--extra llm`, which rebuilt the
shared `.venv` — dropping the LLM stack *and* silently switching the interpreter 3.14 → 3.11 (uv
recreates with the system default when no `.python-version` is pinned). That killed the running
local society seed with a `FileNotFoundError` deep in `huggingface_hub` templates — a failure that
looked scientific but was pure environment churn.
Rules: (1) never mutate `.venv` while a background job is running against it — wait, or build a
throwaway venv elsewhere; (2) `uv sync` is *declarative* — always pass **every** extra the project
needs, or it removes the ones you omit; (3) the repo now pins `.python-version` (3.14) so the
interpreter can never drift silently.
## Before a compute campaign, derive the design from the theory and read the analytic code beside the neural code (2026-09-07)
The v1 `llm_society` campaign (4 seeds, ~13 L40S-h) returned a null that was *structurally guaranteed*:
3 families over 8 agents made founders near-clones (E8's ρ=1 control: recombining clones buys nothing);
2^3 competence states left no room for a child to be "new" (E8 needs L=12); sex was a linear blend at
0.5B (E4/`llm_moe`: the dilution regime); parents were truncated before breeding (E11 selects on
survival over parents+offspring — v1 threw away half the families at gen 1 with no operator to
restore them); `n_test`=40 put every contrast inside one SE. GG caught the first fault by asking what
the founders knew; the rest fell out of comparing `dynamic_society.py` with `society.py` line by line.
Rules: (1) a Layer-2 instantiation of an analytic experiment must be checked *operator by operator*
against the analytic code, not against its description; (2) every free parameter that the theory
constrains (ρ, L, the operator regime, the observation floor, selection intensity) is set by a
prediction or a calibration measurement, never by feel; (3) write the falsifiers with numbers and the
power analysis *before* submission — the pre-registration is `tasks/prereg-llm-society-v2.md`.
## Calibrate the inheritance channel before the population (2026-09-07, evening)
Nine local GPU-hours of calibration found three ceilings a 96-GPU-hour campaign would have hidden:
(1) with 3 skills over 8 agents, founders were near-clones (E8 ρ=1 → recombination buys nothing);
(2) a fresh LoRA distilled from a one-skill parent's answers on nine families retains only 0.60.8 of
the skill — interference from confident off-expertise answers, not the E2 observation floor, and
removable by gating on the source's own confidence; (3) a two-skill child holds each skill at
~0.85× of its parents at ANY training budget (rank 64 overfits) — the learning budget, not the
sample budget, is the conserved quantity, and it caps how many skills one adapter can carry. GG's
call was no-go at 0.5B rather than a campaign that could only test the ablations. Rules: (a) measure
transmission fidelity of the inheritance channel for one skill, then two, before breeding populations;
(b) when a gate fails, re-derive it from the data you already have (the conflict gate moved 0.35→0.41
from the epistasis grid's own outcomes) rather than by feel, and record it as an amendment; (c) a
pre-registration that ends in a no-go has done its job — write the ceiling up, don't route around it.
## A plan for reader-facing prose must carry the dual-audience standard explicitly (2026-09-09)
GG rejected the approved-in-substance manuscript-revision plan until it stated, as a first-class
section, that every term from either field is defined at first use with an example from each world.
The plan had the right content and structure but treated accessibility as a verification
afterthought; GG's rule is that it is "paramount" and must be designed in, not checked for. Rule:
before drafting any passage for a mixed readership, build the term table (term / one-clause
definition / biology example / model example) *in the plan*, and put a two-reader pass in
verification. The same applies to my status reports — GG twice said "I am lost with all these C3,
E9, H1"; spell codenames out.
## 2026-09-11 — check the figure panel inventory before flagging cross-references
In the manuscript review I flagged Table 1's "Fig. 4CE" and "Fig. 3C" as inconsistent with the text. They were
correct: `make_figs.py` puts the E11 ablation in Fig. 4's bottom row and the predictive grid in Fig. 3CD; the text
simply failed to cite them. Rule: before calling a cross-reference wrong, read `paper/manuscript/make_figs.py` and the
captions in `build.py` for the panel inventory; the fix is usually a missing citation in the text, not a wrong table.
## 2026-09-11 — never type a result number that a script has not printed
Writing the stop3 README I filled the per-seed cells from memory of the mean and had three rows wrong
until the loader's pivot table exposed it. Rule: every number in a README, SI table or manuscript is
pasted from a stats-script printout produced in the same step; if the script has not printed it,
print it first. This is the same rule the plan stated ("copied from the stats-script output, not
typed") and I broke it within the hour.
## Reference numbers hardcoded outside the renumber path (2026-09-11)
`renumber_refs.py` rewrites main.md, si.md and build.py captions, but `make_figs.py` carries a literal "(refs. N, M)" in the fig1a grounding cell, which went stale after references were added. Rule: after any renumber, grep `refs\.` and `ref\.` across `paper/manuscript/*.py` and fix by hand (or extend renumber_refs.py to cover make_figs.py). Also: fig text is rendered, so verify by `pdftotext figs/fig1a.pdf`, not by grepping the source alone.
## 2026-09-12 — GG's manuscript comments: heralds, undefined terms, and phantom SI references
Forty-five comments on the ODT. Three patterns. (1) **Herald sentences** ("Two boundaries follow.",
"Modifier theory predicts its fate:", "X is the measurement no other arm produces") — GG: "Breaking
down sentences like this is also a claudism." Now a HERALD block in the declaudify detector; run
`--list herald` before handing over any draft. (2) **Every technical term defined at first use, with
one word per concept**: item/capability/allele, mass, refit, verified real samples, Zipf source,
practitioner, "which trained networks" — the dual-audience rule from 2026-09-09 applied to *my own*
vocabulary, not only the biology. Rule: after drafting, grep each noun of art for its first
occurrence and check a definition precedes it. (3) **Never cite the SI for something the SI does not
contain.** The text cited "the SI separates three cases" and "the proof is Poisson thinning" and
neither existed; and every "(SI)" pointer must name a figure or text number. Rule: before writing
"(SI)" grep si.md for the claim; if absent, write it (SI Text S4) or drop the sentence.
## 2026-09-12 (round 2) — the Discussion must discuss; novelty is flagged where the result is shown
GG on the "What is borrowed and what is new" inventory: "The discussion should discuss, not list."
And on the Limits paragraph: stating small-scale limits "is usually done by undergraduate students";
either run the experiment or discuss only problems too big for the paper. Rules: (a) never write a
Discussion paragraph that is a list of prior-art citations or of caveats; each Discussion paragraph
argues one point; (b) attribute novelty at the point of the result, with the figure panel, and name
the prior finding it explains or extends in the same sentence ("an observation reported by others
and left unexplained (60)"); (c) "much/some/most of X was known" is a hedge that gives novelty away
without saying what is new; replace with the specific thing prior work lacks. Also: after
`renumber_refs.py --apply`, the fig1a literal (refs. 22, 33) went stale (Shumailov became 23); the lesson from 2026-09-11 held.
## 2026-09-12 — a prediction written into the Discussion must be run before it is printed
The revised Discussion predicted that differential reproduction would turn recombination's speed
advantage into a level advantage. Three GPU-hours later it did not (parity, 3/3 seeds). Rules:
(a) when a Discussion sentence forecasts the outcome of an experiment we can run in under a day,
run it in the same revision; (b) check adapter/cache directories for seed- and base-specificity
before any HPC array (speciation shared one dir across seeds; the specialist cache would have loaded
Qwen adapters into SmolLM2); (c) the local smoke gate for a new base (termination, base accuracy in
(0.05, 0.95), sample generations) cost 4 minutes and is worth running every time.
## 2026-09-13 — a figure must be readable without its caption
GG on the manuscript figures after reading the student guide: "too unclear, cryptic"; figures should
"give some clear information without the need to read the legend". The house rule in make_figs.py
("no per-panel headline titles; interpretation lives in the captions") was the wrong rule for this
audience and is reversed. Rules: (a) every data panel carries a one-line headline stating its
finding plus a grey line naming the system and its size; (b) legend entries say in words what is
plotted ("accuracy on the model's weakest task family", not "worst_family"); (c) where the set-up is
not obvious, a schematic panel explains it inside the figure; (d) bar comparisons carry the test
(paired over seeds, stars, key printed under the legend). Layout lesson: headlines longer than the
panel run into the neighbour; wrap at ~40 characters per line for a half-width panel, ~70 for full
width, and render before trusting.
## 2026-09-13 — figure layout rules I should apply without being told
GG had to ask three times for things a careful eye catches: headlines running past their panel,
a schematic strip narrower than the data panels beneath it and not flush with their left edge,
and a large blank band between a strip and the next row. Rules, now encoded in make_figs.py:
(a) any panel placed by hand (schematics) is positioned from the neighbouring data axes' geometry:
left edge = the data panels' frame, right edge = the last panel's frame, bottom = a fixed 0.75 in
above the headline below, height from the content's designed aspect (never let equal-aspect centre
a too-wide axes); (b) text wraps to its own panel width (`headline()` measures the axes); (c) after
every regeneration, render at ≥ 90 dpi and check four things before reporting: nothing crosses a
panel boundary, nothing overlaps, blank bands are no larger than the row gaps, and left edges of
stacked panels line up. Report only after that check passes.
## 2026-09-13 — prose: no staccato fragments
- GG flagged "These results say X. They do not say where. The inheritance model does, in closed form."
as a claudism. Breaking a thought into short declaratives is rarely necessary; join them (colon,
"because", "and", "but"). Clarity comes from stating the concrete object, not from short sentences.
- After any rewrite pass, scan for sentences of ≤7 words introduced by the edit and rejoin them.