MachineSex/CLAUDE.md

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Current state: Layer 1 complete; Layer 1.5 (neural) in progress

- **Layer 1** (`src/knowledge/`) — **complete and validated.** All six experiments E1–E6, the
  closed-form scientific-validation tests, figures, and reproducibility harness exist. Headline:
  critical grounding `g* = 0.048 ≪ 1`; the E4 finding that mean-mixture distillation conserves
  collapse while only a union-preserving max-merge realises the recombination benefit.
- **Layer 1.5** (`src/neural/`) — **in progress.** An architecture-general neural existence proof
  (re-scoped Layer 2): the same Wright–Fisher abstractions realised in *real trained generative
  models* (histogram bridge + RNN + MLP; VAE implemented but not fidelity-passing) on a
  fully-synthetic sandbox with an exact oracle, plus real MNIST as a later secondary tier. See
  `tasks/todo.md` for status and `~/.claude/plans/we-are-going-to-cheerful-fog.md` for the plan.
  **Done:** scaffold, the histogram bridge gate (reproduces Layer 1 exactly), `bridge` (neural
  g*=0.047 ≈ Layer 1), `collapse` (in RNN weights), `grounding` (refined; sign confirmed, threshold
  softened by neural smoothing — see finding below), `architectures` (architecture-generality),
  `recombination` (the E4 "merge, don't average" finding reproduced in real weights), all six
  neural figures, and the **real-MNIST external-validity tier** (`mnist_collapse`: a conv-VAE
  collapses to a single mode under dry self-training, ~10% grounding holds all 30 modes; frozen-CNN
  oracle, confusion matrix recorded). **Remaining:** `region_matched`, `remint`, the synthetic-VAE
  fidelity fix — all optional. The LLM/LoRA rung and C3 vertical claim stay deferred. The LLM/LoRA rung and the C3 vertical claim are deferred. Experiments are
  named descriptively (`configs/neural/<name>.yaml`), not by code.

The two design documents are the source of truth for intent:

- `paper/the-lamarckian-society-v4.md` — the *perspective paper* (the "why").
- `paper/blueprint.md` — the *technical blueprint* (the "what"/"how"). **It is normative** for
  Layer 1 and the LLM Layer 2; Layer 1.5 is a cost-staged intermediate the blueprint does not
  cover, designed to preserve the same §1 abstractions.

Everything below summarizes the blueprint so you can orient fast, but the blueprint is the source of truth. When they conflict, the blueprint wins; when the blueprint is silent, minimize decisions and match its established patterns.

## The one idea you must hold in your head

**Knowledge transmission across agent generations is modelled *literally* as a Wright–Fisher population-genetics process** — not by analogy. A model's knowledge is a distribution `p_t` over `K` discrete items on a simplex; a fixed true distribution `p*` has a rare **tail**; each generational step is "sample from parent (drift) + mix in fresh real samples (immigration/grounding) + refit." Model collapse = loss of rare alleles under drift. Every experiment is a manipulation of this single process.

The population-genetics dictionary in blueprint §1 is the spine. Keep its abstractions **identical across both layers** — this is a hard requirement, because it is the only thing that lets a Layer-2 neural result count as confirming a Layer-1 analytic prediction:

| Abstraction | Layer 1 (analytic) | Layer 2 (neural) |
|---|---|---|
| region | disjoint block of the `K` items | task family (e.g. string ops, recursion) |
| rarity / tail | low `p*` items | low-frequency task types |
| grounding fraction `g` | `m/(n+m)` real-vs-inherited samples | proportion of verifier-passed items in pupil's training mix |
| decorrelation `ρ` | shared retained-tail correlation between teachers | LoRA specialists on disjoint task families |
| diversity `H` | heterozygosity `1 − Σ pᵢ²` | solution diversity of generated code |
| reality's "no" | grounding against `p*` | execution-based unit-test verifier |

## Two layers, staged by cost

- **Layer 1 — analytical core** (`src/knowledge/`). Pure NumPy/SciPy Wright–Fisher simulator. Laptop, minutes, no GPU. Carries the paper's quantitative claims. **Three of the five §2.4 predictions are closed-form**, so validation is an **exact** test, not a vibe check — these become `<0.1%`-tolerance assertions in `test_scientific_validation.py`:
  - **Pred. 1** — neutral heterozygosity decay: `E[Hₜ] = H₀(1 − 1/n)ᵗ`.
  - **Pred. 3** — *exact* mutation–drift equilibrium for the implemented immigration model: `H_eq = H* · m(2n+m−1)/(n+2nm+m²)`, with `H* = 1 − Σ(p*ᵢ)²`. The textbook `θ/(1+θ)` (θ=2m) is only the rare-immigrant limit. **Critical nuance: H is *smooth* in m — the sharp phase threshold lives in discrete tail-item survival (Pred. 4: an item survives iff `m·p*ᵢ ≳ 1`), not in H.** Do not describe E2 as a discontinuity in H.
  - **Pred. 5** — closed-form recombination benefit: `U(K_T, ρ, q) = T[ρq + (1−ρ)(1−(1−q)^K_T)]` (expected tail items retained by ≥1 of K_T teachers).
- **Layer 2 — neural existence proof** (`src/neural/`). Small open-weight models (default OLMo-2-1B / SmolLM2-1.7B, fallback Qwen2.5-1.5B-Instruct; pin the HF revision hash, never track `main`), LoRA specialisation, distillation/merging across 2–3 generations, program-synthesis-with-unit-tests as the verifier. One consumer GPU. Only needs to show the **sign** of three effects, not precise magnitudes.

## Experiments and their falsifiers

Each experiment is one config file → one runner invocation → one `results.parquet` → one figure. Every experiment has a **falsifier** — an outcome that would refute the corresponding claim. The design is built to be able to kill the thesis; preserve that.

- Layer 1: **E1** reproduce collapse (null), **E2** grounding phase boundary (headline: is there a critical `g* ≪ 1`?), **E3** region-matched grounding, **E4** multi-teacher decorrelation, **E5** quality-diversity vs. greedy selection, **E6** re-minting gate / irreversibility.
- Layer 2: **C1** dry vs. grounded, **C2** one vs. N complementary teachers at matched budget, **C3** the vertical claim (general knowledge climbs while each specialty is re-earned and *exceeded* — this is load-bearing, prioritize it), **C4** distillation vs. merging (optional).

Blueprint §6 is the claim→experiment→figure→falsifier traceability matrix and is the definition of done.

## The one non-obvious implementation piece: the correlated-teacher construction (§2.7.1)

E4's whole purpose is to isolate the effect of teacher **decorrelation ρ**, so ρ must be a *directly constructed, independently-swept* knob — **never** an emergent quantity you get by tuning drift (that ρ would be confounded with `n`, `m`, tail size, and generation count, i.e. with the very drift E4 holds fixed). The construction is a **shared-switch exchangeable Bernoulli**: for each of the `T` tail items, draw a shared switch `z~Bern(ρ)`, a shared retention `s~Bern(q)`, and per-teacher independent `u⁽ᵏ⁾~Bern(q)`; set teacher k's retention `r⁽ᵏ⁾ = s if z else u⁽ᵏ⁾`. This yields exact marginal retention `q` and exact pairwise correlation `ρ` (provable: Cov = ρq(1−q), Var = q(1−q)), and is exchangeable so ρ is a single scalar. `make_retention_matrix(T, K_T, rho, q, rng)` returns the `(K_T, T)` binary matrix; `make_correlated_teachers` maps it to distributions (head items always kept at `p*`; tail item kept at `p*ᵢ` if retained, else `tail_floor`; renormalise so dropped-tail mass flows to survivors). The exact-construction path is **preferred** for E4; the drift-based path exists only as a realism cross-check. `region_specialisation=True` forces full retention of a teacher's home-region tails and applies the ρ construction only off-home.

**E4 reports two coverages, and their gap is a result, not noise:** the construction-level union `U(K_T,ρ,q)` (must match the closed form exactly) and the post-distillation *surviving* coverage after the pupil's size-`n` resampling. A tail item present in the mixture only survives if its mixture mass clears `~1/n` (Pred. 4) — so the gap is precisely "the tail recombination *supplied* but drift *re-erased* because grounding was too thin," which ties E4 back to E2/E3.

**Finding (2026-07-04, E4) — the recombination operator matters, and mean-mixture distillation does not realise the benefit.** Under the blueprint's mean-mixture pupil (`p̄ = mean(teachers)`), surviving tail coverage is **flat in K_T** — a conservation law: averaging preserves expected pupil tail mass at `q·(tail mass of p*)` regardless of K_T, and in the rare-tail (linear-survival) regime the 1/K_T dilution exactly cancels the union gain. The recombination benefit is realised only under a **union-preserving merge** (`max` over teachers, à la M2N2), where surviving rises with K_T and decorrelation. So E4 reports surviving under **both** operators (`surviving_mean`, `surviving_max`): union = supply (validated vs closed form), max-merge = realised benefit, mean-distill = the null that motivates why merging/grounding is needed. GG decision: report both. This sharpens rather than refutes the thesis, but the paper's recombination claim rests on the *merge* operator, not naive mean distillation — worth carrying into Layer 2 (C4) and the write-up.

**Finding (2026-07-05, neural `grounding`) — grounding's SIGN transfers to trained RNN weights, but the sharp `g*` does not; and tail-survival is the *wrong* neural collapse metric.** Re-ran the phase-boundary sweep at 18 replicates. Two results: (1) **forward-KL is the operative neural collapse metric, not H or tail-survival.** The RNN's smoothing inductive bias keeps *spurious* tail modes alive (it generalises to unseen codewords), so `tail_truth_mass_alive` is flat/**non-monotone** in g (dry 0.54 > most grounded points) and H stays ~0.77–0.85 of H\* throughout — neither shows a threshold. Stationary **forward-KL** falls monotonically (dry 2.08 → g=0.2: 0.75), significant at g≥0.05 (paired t up to 3.3; 89% of lineages improve at g=0.2). This *refines* the earlier "forward-KL AND tail survival" note: for a smoothing model, support-counting decouples from closeness-to-truth. (2) **The sharp `g*≪1` is an exact-operator feature, softened by neural inductive bias.** Half the achievable KL reduction closes by a *median-recovery* grounding g≈0.04 (bootstrap CI [0.004, 0.116]) — a striking echo of Layer-1's 0.048 — but full (95%) recovery needs g≈0.19, far more than the histogram bridge, because smoothing both caps dry collapse (KL~2, not ∞) and slows full recovery. So the quantitative `g*≪1` claim rests on the **histogram bridge** (g\*=0.047, exact reduction to Layer 1), which the trained RNN confirms in *sign* and softens in sharpness. Honest note: the pre-registered 95%-of-H\*/tail-survival falsifier is not met, but that is because those are the wrong metrics for a smoothing model, not because grounding fails — the blueprint §3.5 directional claim (grounding arrests collapse) holds robustly. Robustness fix landed alongside: a fully-degenerate RNN can emit only invalid codewords, so `measure_distribution` returns a terminal-collapse sentinel (fixation on the dominant mode) instead of crashing a long sweep.

**Finding (2026-07-05, real-MNIST `mnist_collapse`) — collapse and grounding-rescue reproduce on real images.** External-validity tier: a small **convolutional VAE** (the canonical generative-collapse model) is retrained each generation on its own generated digits. Modes = digit class × stroke-thickness bin (K=30, Zipf, ~18 tail modes); the oracle is a **frozen CNN + deterministic thickness** at **98.5% mode accuracy** (its 30×30 confusion matrix is recorded in the manifest as the measurement floor). Result (4 reps): the **dry (g=0) lineage collapses to a single mode** — forward-KL 0.5→18, support 30→1, tail truth-mass 1.0→0.06, H→0 — while **10% grounding holds all 30 modes** (KL≈0.6, full tail, H≈0.9). The VAE needs ~10% grounding here vs the synthetic histogram's ~5%, consistent with the `grounding` finding that trained neural models need more grounding than the exact operator. **Confirmation-only (signs, not magnitudes; blueprint §3.5)** — the exact synthetic oracle stays the quantitative anchor. `figures/mnist_montage.py` is an eyeball diagnostic (re-runs a short dry lineage; NOT a parquet figure). Build gates passed: CNN mode accuracy 98.5%; VAE gen-0 recovers full 30/30 support (over-smooths frequencies, KL≈0.5, no prior hole — unlike the *synthetic*-codeword VAE, which is why the MNIST VAE works where that one didn't). The MNIST tier is heavy (torchvision `--extra mnist`, downloads MNIST, ~5 min): `make mnist`, kept out of the `make neural` loop.

**Finding (2026-07-05, learning kernel) — neutral drift is a null both real models fail, oppositely; the estimator bias is a signed operator.** Layer-1 extension (`knowledge/kernel.py`, `LearningKernelCfg`): the refit becomes `p_{t+1} = T_θ(counts/n)` with two pop-gen knobs — **reset `u`** (mutation toward a prior = smoothing) and **temperature `τ`** (sharpening = mode-competition) — both identity at their defaults, so the histogram bridge and every scientific-validation test are unchanged (68 core tests still green). Result: **neutral Wright–Fisher fails both neural architectures, in opposite directions.** VAE regime (`n=6000, K=30`): neutral drift is *inert* (no collapse), yet the real VAE collapsed to one mode — **sharpening `τ=0.8` reproduces it** (the estimator ADDS collapse). RNN regime (`n=200, K=256`): neutral drives `H→0`, but the real RNN only partially collapses — **mutation `u=0.006` reproduces the `H`-floor** (the estimator REMOVES collapse). Honest caveat: uniform-mutation matches the RNN `H`-floor but overshoots its forward-KL (~5 vs ~2), evidence the RNN's smoothing prior is *truth-like, not uniform* (future refinement). Configs `configs/layer1/kernel_{sharpen,smooth}.yaml`, figure `plot_kernel.py`. This mechanistically explains the architecture-generality result and the softened neural `g*`.

**Strategic positioning vs Riis 2026 (arXiv:2604.08554, "Drift and selection in LLM text ecosystems").** Riis independently formalizes **collapse = Wright–Fisher drift** (his Thm 1) with n-gram agents: minority-mass martingale, rare-first extinction, single-token dropout ≈ αe^{−α}, de Bruijn-polytope fixed points, plus descriptive-vs-normative *selection* (Thm 2). **Concede as prior art:** "collapse is literally Wright–Fisher", the martingale, rare-first loss, the WF/effective-population formalism — cite him; do **not** frame these as our contribution. **Crucial distinction that protects us:** his "mixed environment" *retains the lineage's own old synthetic tokens* — there is **no injection of fresh real data from a fixed `p*`**, so his headline is *pessimistic* (Thm 1c: extinction is independent of α — retention only changes speed). Our **grounding is immigration from a non-drifting external truth**, giving a stationary `H_eq>0` and a critical `g*≪1` that *prevents* collapse — the mechanism his closed loop lacks. **Our defensible novelty, ranked:** (1) **recombination + "merge, don't average" conservation law** (E4) — he has no model-merging operator; flagship; (2) **the learning-kernel / estimator-bias axis** — he *explicitly names it as future work*; we now build+measure it; (3) grounding threshold (solid anchor, but immigration–drift balance is classic — not a flagship); (4) architecture-generality in real weights + MNIST; (5) **the Lamarckian society + the vertical/cumulative C3 claim — wholly ours, not yet run.** Reposition the paper from *"collapse is drift"* (now contested) to **a population-genetic *control theory* for sustaining open-ended knowledge**: drift is the diagnosed disease (cite Riis), our contribution is the engineered remedies and their integration.

## Build order (blueprint §7) — respect the gate

1. Scaffold: repo layout (§5), container, pytest skeleton, config system, seeding utils. `make test` green.
2. Layer 1 core + null model + `test_scientific_validation.py` against §2.4 predictions 1–2.
   **HARD GATE: do not proceed until simulated drift matches the analytic heterozygosity decay `E[Hₜ] = H₀(1 − 1/n)ᵗ`.**
3. Layer 1 grounding + E1–E2 (the headline result).
4. Layer 1 E3–E6. Layer 1 is now a complete laptop-reproducible paper on its own.
5. Layer 2 scaffold + verifier (test determinism & sandbox isolation *before* any training).
6. Layer 2 C1 + C3.
7. Layer 2 C2 (+ C4 if compute allows).
8. Reproduction pass.

**Do not start Layer 2 until Layer 1's scientific-validation tests pass.**

## Prescribed structure and commands (do not yet exist — create per blueprint §4–5)

Target module interfaces are given with **normative names** in blueprint §2.7 (Layer 1) and §3.6 (Layer 2); downstream scripts depend on these signatures, so implement to them exactly. Target repo layout is §5. Planned automation:

```
make env        # uv sync -> .venv from committed uv.lock
make test       # correctness tests + scientific-validation tests
make layer1     # run E1–E6
make layer2     # run C1–C3 (C4 optional)
make figures    # regenerate every figure from committed results.parquet
make all
./reproduce.sh  # uv sync → test → run all at committed seeds → regen figures → REPRODUCED.md
```

Single-experiment run pattern: one YAML config per experiment under `configs/layer1/EX.yaml` or `configs/layer2/CX.yaml`, fed to the experiment runner. Figures are regenerated separately by `figures/plot_EX.py` reading **only** `results.parquet` (no re-simulation).

## Non-negotiable engineering standard (blueprint §4)

- **Reproducibility is a hard requirement, not a preference (this is a paper).** The environment is a **`uv` venv built from a committed, hash-pinned `uv.lock`** — that lockfile is the source of truth for "it runs" (Apptainer is dropped; a Dockerfile may later wrap the same lockfile for Layer 2's GPU work). Layer 1 is bitwise-reproducible from a single master seed; Layer 2 is *statistically* reproducible (document residual GPU non-determinism, set determinism flags, report per-seed points).
- **Seeding:** one master seed in config → derive all sub-seeds via `np.random.SeedSequence.spawn`. Never touch global RNG state; pass `rng` explicitly everywhere. Results are a pure function of the resolved config.
- **No magic numbers in code.** Every parameter lives in a YAML resolved at run time; the *resolved* config (after sweep expansion) is written next to results. Sweeps are declared in config, not hard-coded.
- **Output contract for every run:** `results.parquet` (long form) + `resolved_config.yaml` + `manifest.json` (library/CUDA versions, seed, git commit, model revision hashes, content hash of results). Every figure must be a pure function of a committed results artifact.
- **Scientific-validation tests are the spine of trust.** They assert the simulator reproduces the §2.4 closed forms within tolerance; if they fail, the *science* is wrong, not just the code. Keep them.
- **Open science end-to-end:** open-weight models only, permissive/open tooling (uv, MLflow or plain versioned Parquet — avoid closed SaaS trackers), `results/` gitignored but hashes tracked.

## Stack

Python ≥ 3.11. Layer 1: NumPy, SciPy, pandas, matplotlib — no GPU, no heavy deps. Layer 2: PyTorch, HF `transformers` + `peft` (LoRA), `datasets`, optional `vllm`; sandboxed `subprocess` verifier. Config via a thin pydantic + PyYAML loader (not Hydra — its global state/`chdir` fights the pure-function-of-resolved-config contract). Env via a **`uv` venv from a committed `uv.lock`** — the lockfile is the reproducibility source of truth; Layer 1 needs no container.