MachineSex/tasks/todo.md

Layer 1 Execution Plan — The Lamarckian Society (analytical core)

Created 2026-07-04. Scope: blueprint §7 build-order steps 1–4 (scaffold → Layer 1 complete, laptop-reproducible). Layer 2 is explicitly out of scope here and gated on Layer 1's scientific-validation tests passing.

Source of truth: lamarckian-society-technical-blueprint-v1.md. Where it is silent I record a decision below rather than improvise silently.

---

Design decisions to pin before coding (blueprint leaves these open)

These are the only places the spec is genuinely underdetermined. Recommendation given for each; flagged for sign-off.

1. Config framework — recommend thin pydantic + PyYAML, not Hydra. Blueprint says "Hydra or a thin equivalent." Hydra is heavyweight and its global-state/os.chdir behaviour fights the "pass rng explicitly, results are a pure function of resolved config" requirement. A thin loader (pydantic models for validation + a small sweep-expander) is dependency-light, aligns with the stdlib-first preference, and makes the "write resolved config beside results" contract trivial. Decision: thin pydantic loader.

2. Selection fitness f_i (the reality-anchored score). Blueprint: "fitness is predictive accuracy against p*" but gives no formula for the discrete model. Decision: f_i = p*_eff_i (truth frequency = fitness; reality-anchored by construction). Post-selection distribution p'_i ∝ p_i^(1−α) · f_i, matching the blueprint's stated w_i ∝ f_i·(p_i)^(−α) with α=0 recovering fitness-proportional greedy. Document as a modelling choice; select-then-sample vs sample-then-select is the documented robustness switch (§2.2).

3. tail_mask definition. Two knobs exist (tail_frac, tail_threshold). Decision: the metric-bearing tail set is the §2.3 definition {i : p*_i < tail_threshold}. tail_frac only drives the twocomponent construction (fraction of items placed in the low-mass component). Document that for zipf, tail_frac is unused.

4. TrueDist type. Decision: a frozen dataclass TrueDist(p_star: np.ndarray, regions: np.ndarray, tail_mask: np.ndarray) — immutable, so p* cannot be mutated in place (except the deliberate re-mint path, which produces a new object).

5. Region partition. Decision: contiguous blocks; require K % R == 0 (assert with a clear error) to keep per-region math clean for v1.

6. Re-mint semantics (E6). Decision: re-mint replaces p_star_eff with a new TrueDist built from current p_t (fresh tail_mask recomputed on p_t), and the original TrueDist is retained only for the KL-to-original metric, never for grounding. This is the irreversibility the experiment tests.

---

Phase 0 — Scaffold & environment

• Install uv (curl -LsSf https://astral.sh/uv/install.sh | sh; lands in ~/.local/bin, no sudo).
• Reproducibility path = uv venv from a committed, hash-pinned uv.lock (GG decision, 2026-07-04; no Apptainer/Docker for Layer 1). pyproject.toml (Python ≥3.11; deps: numpy, scipy, pandas, pyarrow, matplotlib, pydantic, pyyaml; dev: pytest). Commit uv.lock. Create .venv via uv sync.
• Repo layout per §5: src/knowledge/, configs/layer1/, figures/, results/ (gitignored), tests/, paper/. Add src/lamarckian/ package root or make knowledge importable (decide package name — recommend src/lamarckian/knowledge/... with src-layout).
• .gitignore (.venv/, results/, __pycache__/, *.parquet under results but keep hashes).
• Seeding util lamarckian/utils/seeding.py: master seed → np.random.SeedSequence(seed).spawn(n) → per-replicate np.random.default_rng(child). No global RNG anywhere.
• Config loader lamarckian/config.py: pydantic schema mirroring the §2.7 YAML, a load_config(path), a expand_sweeps(cfg) -> list[ResolvedConfig], and write_resolved(cfg, dir).
• Manifest util: write_manifest(dir, results_df) recording lib versions, master seed, git rev-parse HEAD, content hash of results.parquet.
• Makefile targets (env, test, layer1, figures, clean) + pytest skeleton. Gate: make test green on a trivial test.
• Move lamarckian-society-technical-blueprint-v1.md → paper/blueprint.md per §5 (confirm with GG first — it's referenced by name elsewhere).

Phase 1 — Core primitives + null model + VALIDATION GATE

Implement to the normative signatures in §2.7. Order chosen so each piece is unit-testable before the next depends on it.

• knowledge/truth.py::make_true_distribution → TrueDist. Support tail ∈ {zipf, twocomponent}. Unit tests: normalisation, region block sizes, tail_mask matches threshold, determinism from seed.
• knowledge/metrics.py: forward_kl (with eps floor, logged), heterozygosity, tail_mass, support_size, plus per-region variants. Unit tests on hand-computed small vectors.
• knowledge/step.py::generation_step — null path first (single teacher, m=0, selection none): c ~ Multinomial(n, p_t), p_{t+1}=c/n. Exactly neutral Wright–Fisher.
• knowledge/lineage.py::run_lineage(cfg, seed) → tidy per-generation DataFrame (all §2.3 metrics, global + per-region).
• tests/test_scientific_validation.py — the spine:
  • Pred. 1 heterozygosity decay: mean H_t over replicates matches H₀(1−1/n)^t. Prefer testing the full deterministic mean-recursion trajectory (subsumes the fixed point), within Monte-Carlo CI.
  • Pred. 2 fixation probability = initial frequency (long runs, statistical tolerance).
• HARD GATE: do not proceed until Pred. 1–2 pass. If drift ≠ analytic decay, the harness is wrong — fix here.

Phase 2 — Grounding + E1 + E2 (the headline)

• knowledge/step.py::structured_multinomial(m_vector, p_star, regions, policy, rng) — per-region immigration draws from p* restricted+renormalised to each region; uniform spreads m evenly, matched concentrates on exercised regions. Returns length-K counts.
• Extend generation_step with grounding (pooled draw, g = m/(n+m)).
• Pred. 3 validation — exact equilibrium H_eq = H*·m(2n+m−1)/(n+2nm+m²): run to stationarity (burn-in + late-generation + replicate averaging), assert <0.1% rel. error vs closed form across an m grid. Also assert the m→0 and m→∞ limits.
• Pred. 4 validation — tail-persistence: item of freq p*_i maintained iff m·p*_i ≳ 1; verify the survival transition location statistically.
• knowledge/experiment.py::run_experiment(cfg) — sweep grid × n_replicates; long-form results + CIs; write results.parquet + resolved_config.yaml + manifest.json.
• E1 config + run: m=0, single teacher, no selection. Expect H geometric decay, support→1, KL diverges, tail-first loss.
• E2 config + run: sweep g, single teacher, uniform grounding, no selection. Locate critical g* (transition in tail mass / support, since H is smooth in m — the sharp threshold is in discrete tail survival). Report g* with CI. This is the load-bearing result.
• figures/plot_E1.py, plot_E2.py — read results.parquet only.

Phase 3 — E3–E6

• E3 region-matched grounding. Fixed total m; uniform vs matched; one designated inherited-but-unwatered region with a rare tail. Expect uniform lets that region's tail collapse; matched holds it. Per-region metrics essential. plot_E3.py.
• §2.7.1 correlated-teacher construction — knowledge/teachers.py:
  • make_retention_matrix(T, K_T, rho, q, rng) — shared-switch exchangeable Bernoulli.
  • make_correlated_teachers(...) — retention→distributions (head kept at p*; tail at p*_i if retained else tail_floor; renormalise). region_specialisation option.
  • Pred. 5 validation: make_retention_matrix reproduces marginal q, pairwise ρ, and union coverage U(K_T,ρ,q)=T[ρq+(1−ρ)(1−(1−q)^K_T)] to 3 decimals over a (ρ,q) grid.
• E4 multi-teacher decorrelation. Sweep K_T∈{1,2,3,5}, ρ∈[0,1] at fixed q, matched budget (n/K_T each). Report both union U and post-distillation surviving coverage; show their gap shrinks as g rises. plot_E4.py (coverage surface over (K_T,ρ)).
• E5 QD vs greedy. apply_selection (none/greedy/qd, pinned fitness form). Sweep novelty α. Expect greedy→fixation (H→0), qd holds H plateau + re-introduces tails. plot_E5.py.
• E6 re-mint gate. Re-mint at high vs low H; track KL to original truth. Expect collapsed re-mint locks KL high forever; gated (high-H) does not. plot_E6.py.

Phase 4 — Reproducibility polish (Layer 1 slice)

• configs/layer1/E1..E6.yaml all committed with explicit params (no magic numbers in code).
• paper/figure_manifest.md — the §6 claim→experiment→figure rows for Layer 1.
• make layer1 runs E1–E6; make figures regenerates all figures from committed parquet.
• Full test_correctness.py (shapes, normalisation, determinism) + test_scientific_validation.py (Pred. 1–5) green in CI.
• reproduce.sh (uv sync → make test → make layer1 → make figures → write REPRODUCED.md diffing committed result hashes) + README.md reproduce section. No container — the committed uv.lock is the reproducibility source of truth (per GG, 2026-07-04); a Dockerfile may later wrap the same lockfile for Layer 2's GPU work.

---

Definition of done (Layer 1)

Every Layer-1 row of blueprint §6 has a committed figure produced by make figures from committed results; all §2.4 analytic checks (Pred. 1–5) pass; make layer1 && make figures reproduces from a clean .venv. Then — and only then — Layer 2 may begin.

Falsifier watch (report honestly if hit)

• E2 tail mass flat in g, or only stabilises as g→1 → multigenerational thesis refuted.
• E3 uniform protects as well as matched → region-matching claim dies.
• E4 no surviving-coverage benefit at matched budget → recombination claim dies.
• E5 qd ≤ greedy stationary H → QD does no work.
• E6 collapsed lineage recovers original-truth tails after re-mint → irreversibility overstated.

Review — progress log

2026-07-04 — Phases 0 & 1 complete; hard gate PASSED.

• Reorg: docs → paper/ (blueprint.md, the-lamarckian-society-v4.md). src-layout under src/knowledge/.
• A pre-existing tests/test_scientific_validation.py (author-supplied, 22 KB) turned out to hard-specify the package contract — implemented to it rather than inventing interfaces. Key contracts it locked (now honoured): package imports as knowledge.*; run_lineage(cfg_dict, seed) returns a tidy per-gen frame with a heterozygosity column, rows 0..T; p_0 initialises uniform (H_0=1−1/K); metrics.heterozygosity and teachers.make_retention_matrix match the reference to 1e-12 / closed form.
• Env: uv 0.11.26 installed; pyproject.toml + uv.lock committed; numpy 2.5, pandas 3.0, scipy 1.18, pydantic 2.13, pytest 9.1.
• Modules written: metrics, seeding, config (dataclasses + from_dict), truth, teachers, step, lineage. Config is dataclass-based (not pydantic) — the conformance test passes a raw dict; dataclasses validate cleanly and stay stdlib-simple. Pydantic still a dep for the Phase-2 YAML/experiment layer.
• Results: make test green — 68 passed (48 scientific-validation, 20 correctness). Conformance tests RAN (not skipped): Pred. 1 (neutral decay), Pred. 3 (exact H_eq), Pred. 5 (union coverage) all pass against the real package. The Pred. 1–2 hard gate is passed, and grounding already conforms to the exact equilibrium.

Design decisions #1 (dataclasses now / pydantic at YAML layer), #2 (fitness f_i=p*_i), #3 (threshold tail_mask), #6 (re-mint discard) all implemented as planned. Region design: each region an identical 1/R-mass block (symmetric; reduces to global Zipf at R=1, matching the reference).

2026-07-04 — Phase 2 complete (E1 + E2).

• experiment.py: sweep expansion (Cartesian grid; special-cases g→m), paired replicate seeds (shared across grid points), output contract (results.parquet + resolved_config.yaml + manifest.json with lib versions + git commit + sha256). CLI python -m knowledge.experiment <cfg>.
• E1 (null collapse) — reproduces tail-first collapse: H geometric decay matches H₀(1−1/n)ᵗ within CI; tail items die ~10× faster than head items; support 500→1; forward-KL diverges. Figure results/E1/E1.png.
• E2 (headline) — H_sim tracks the exact H_eq closed form across the sweep; phase boundary at g* ≪ 1: g=0.005 (m=1 real sample vs n=200) → 68% of truth H; g=0.05 → 96%. g=0 slides to ~0.10 over 500 gens. Figure results/E2/E2.png. Headline result achieved.
• Metric subtlety found & fixed: aggregate tail_mass is a drift martingale (mean-conserved), so it's a poor collapse indicator. Added tail_support/head_support/tail_frac_alive/head_frac_alive; E1 & E2 figures now use tail-item survival, which is honest and monotone.
• E2 extended 300→500 generations (GG-approved) so the g=0 arm visibly approaches 0 while g>0 arms sit on plateaus.
• Makefile layer1/figures wired to E1–E2. make test still green (68).

2026-07-04 — Phase 3 complete (E3–E6) + E2 analysis add-ons.

• E3 region-matched grounding: added grounding.exercised knob + per-region tailalive_region_r. Target region tail survival 0.49 (matched) vs 0.07 (uniform). Note: per-region H is mass-confounded — used tail-item survival instead.
• E4 multi-teacher recombination: bespoke run_coverage runner (kind: coverage). Union coverage matches U(K_T,ρ,q) exactly. Key finding (GG-approved): mean-mixture distillation gives NO surviving benefit (conservation law — dilution cancels the union gain); max-merge (M2N2-style) does. E4 reports both. In CLAUDE.md.
• E5 QD vs greedy: greedy → H≈0.01 (fixation); qd holds H 0.48–0.88 rising with α. qd ≫ greedy.
• E6 re-mint gate: added arm multi-override sweep type. Re-mint while collapsed → KL-to-original diverges (lock-in) + accelerates H collapse; diversity gate (H≥0.75) blocks it → bounded; healthy re-mint harmless.
• E2 analysis add-ons (companion work order tasks/workorder-E2-analysis-addons.md, verified): new analysis.py (reduce_to_stationary, critical_grounding bootstrap CI) — real E2 g=0.048, CI [0.047,0.050]*; metrics.tail_band_metrics + per-band lineage logging; tests/test_analysis.py reproduces the work order's verified numbers exactly. E2 figure rebuilt 2×2. Deviation: used truth-mass-weighted tail coverage instead of raw tail_mass (a drift martingale).
• All six figures regenerate via make figures; 71 tests green.

---

Layer 1.5 — Architecture-general neural existence proof (RNN/VAE/MLP + synthetic/MNIST)

Created 2026-07-04. Plan: ~/.claude/plans/we-are-going-to-cheerful-fog.md. Re-scopes Layer 2: build a cheap, architecture-general neural collapse proof in real trained weights on a fully-synthetic sandbox (exact known p*) before the LLM rung. Locked decisions: exact-oracle categorical token sequences; Histogram+RNN+VAE+MLP; real MNIST as secondary confirmation; LLM + C3 vertical claim deferred.

Progress log

2026-07-04 — Stages A, B, plumbing complete.

• Env: installed uv 0.11.26 (~/.local/bin); /home was 100% full — GG approved clearing pip/yay/browser caches (~10 GB freed). Base venv synced; 71 Layer-1 tests green.
• Stage A (scaffold, pure NumPy): src/neural/ — config.py (frozen dataclasses reusing knowledge.config GroundingCfg/RemintCfg/MetricsCfg/_sub), synthetic.py (mode-truth via make_true_distribution; lossless identity + stochastic style token grammar), oracle.py (ExactOracle zero-error + measure_distribution), models.py (GenerativeModel protocol + HistogramModel bridge), evaluate.py (reuses knowledge.metrics, Layer-1 row schema), generation_loop.py (run_generative_lineage, reuses allocate_m/structured_multinomial). 15 correctness tests green.
• Stage B — HARD GATE PASSED: tests/test_neural_validation.py — histogram lineage reproduces Pred. 1 (neutral decay, <3% rel err), Pred. 3 (exact H_eq, <5%), and tracks Layer-1 run_lineage directly (<3%). The neural plumbing reproduces the analytic core.
• Plumbing: neural/experiment.py (run_and_save dispatch on kind, reuses _apply_param g→m, paired seeds); extended knowledge.experiment.save_artifacts (optional extra_libs, extra_manifest, injectable grid; skips missing libs — backward compatible). configs/neural/bridge.yaml, Makefile neural/env-neural/layer2 targets, .gitignore. (Experiments are named descriptively — bridge, collapse, grounding, architectures — not by code.)
• bridge result (17s): neural g* = 0.0474, CI [0.045, 0.052] — reproduces Layer-1 E2's g*=0.048 essentially exactly (g=0.005→67% of H*, g=0.05→96%). 89 tests green.

2026-07-04 — Stage C: torch models + collapse/grounding/architectures.

• Env: torch 2.12.1+cu130 (default PyPI wheel ships CUDA 13, matches RTX A4000 driver; no custom index needed, cp314 wheels exist). --extra neural = torch only; --extra mnist = torchvision (later). UV_CACHE_DIR=/tmp during install (RAM-backed) to spare /home.
• Models: torch_models.py (RNNGenerator, autoregressive GRU), torch_mlp.py (autoregressive MLP, causal-masked), torch_vae.py (sequence VAE), train.py (determinism flags + device/seed helpers derived from the SeedSequence stream). tests/test_neural_torch.py (torch-gated): gen-0 fidelity (rnn+mlp) + dry-collapse/grounded-holds. 92 tests green.
• Validated regime: K=256, n=200, zipf_s=1.3, RNN hidden=128/epochs=25. RNN gen-0 fidelity KL(p*‖p̂)=0.008, 64/64 (or 256/256) modes recovered. MLP fidelity KL=0.011. VAE does NOT clear the gen-0 gate on the Zipf-codeword task (KL≈0.8; prior-hole mismatch — sampling z~N(0,I) misses the aggregate posterior) → excluded from architectures to avoid confounding collapse with underfitting.
• collapse (in weights): dry RNN lineage collapses — forward-KL rises to ~2.2 vs grounded ~1.4; grounding lifts tail survival (tailalive 0.31 dry → 0.50 at g=0.02). Sign confirmed.
• grounding (neural phase boundary): stationary H hovers 80–91% of H* and is noisy / non-monotonic at 5 reps — no crisp g*. KEY FINDING: the neural models' smoothing inductive bias partially resists H-collapse (dry H stays ~83% of H*), so forward-KL and tail survival are the sharp neural collapse metrics, not H (mirrors Layer-1's "H is smooth; the threshold lives in tail survival"). grounding needs (a) forward-KL as the phase metric, (b) more reps (≥10), and/or (c) a stronger-collapse regime for a clean neural g*.
• architectures (architecture-generality) — clean result: collapse + grounding-rescue appear in ALL three model classes (dry→grounded forward-KL: histogram 6.2→4.6, MLP 4.8→1.3, RNN 3.8→1.1; tailalive RNN 0.41→0.64, MLP 0.07→0.20). The WF operator is architecture-general. Bonus: neural smoothing lets RNN/MLP retain more tail than the exact histogram under grounding (they generalise to unseen codewords) — an inductive-bias finding worth the write-up.

2026-07-04 — recombination (load-bearing E4 replication).

• recombine.py mirrors run_coverage but trains K_T specialist RNNs on assignments from the exact shared-switch retention construction (K_T/rho/q clean; union matches the closed form), then recombines the measured teacher distributions two ways: mean (naive pooling) vs max (oracle-guided union / M2N2-style), each followed by size-n resampling. The neural merge is the per-mode max over teacher distributions (oracle-guided), NOT weight-averaging of RNNs.
• Result (8 reps): at rho=0, union rises 0.49→0.96 (supply matches closed form); analytic surviving_max rises 0.043→0.087 while surviving_mean stays flat ~0.045 — the conservation law (averaging cancels the union gain; max-merge realises it). At rho=1 (identical teachers) union AND max are flat — more identical teachers buy nothing. The "merge, don't average" lesson holds in the neural setting. Trained-weight columns show the same signs but noisier: neural smoothing inflates baseline survival and the deep tail barely clears n=200 resampling (compresses magnitude) — the expected inductive-bias caveat. torch-gated test added. 93 tests green.

2026-07-05 — grounding refinement + figure (honest reframing).

• Re-ran at 18 reps (n_eval 15000, 30 gens, g grid refined to 9 points). Falsifier pinned in the config before running.
• forward-KL is the operative neural collapse metric — NOT H or tail-survival. The RNN's smoothing keeps spurious tail modes alive, so tail_truth_mass_alive is flat/non-monotone in g (dry 0.54 > most grounded) and H stays 0.77–0.85 of H*. Stationary forward-KL falls monotonically (dry 2.08 → g=0.2: 0.75), significant at g≥0.05 (paired t→3.3; 89% of lineages improve at g=0.2). Refines the earlier "forward-KL AND tail survival" note.
• The sharp g*≪1 is an exact-operator feature, softened by neural smoothing. Median-recovery grounding (half the KL gap closed) g≈0.04 (bootstrap CI [0.004, 0.116]) — echoes Layer-1's 0.048 — but full (95%) recovery needs g≈0.19. Quantitative g*≪1 is carried by the histogram bridge (0.047); the RNN confirms the SIGN and softens the sharpness (blueprint §3.5 met).
• Honest note: the pre-registered 95%-of-H*/tail-survival falsifier is not met — because those are the wrong metrics for a smoothing model, not because grounding fails. Reported as such.
• Robustness fix: a fully-degenerate RNN can emit only invalid codewords → measure_distribution now returns a terminal-collapse sentinel (fixation on the dominant mode) instead of crashing a long sweep. Edge-case test added. figures/plot_grounding.py written (4-panel, states its own verdict), wired into make figures (glob all plot_*.py except plot_E[1-6]).

2026-07-05 — neural figures (all five).

• figures/plot_{bridge,collapse,grounding,architectures,recombination}.py, each a pure function of its committed bundle (reuse figures/_figlib.py), wired into make figures (glob all plot_*.py except plot_E[1-6] / _*). bridge: neural histogram runner sits exactly on the exact H_eq curve, g*=0.047 (HARD-GATE visual). collapse: dry GRU forward-KL climbs, grounded held; H barely moves. architectures: grouped bars — forward-KL falls / tail survival rises with grounding across histogram/GRU/MLP. recombination: union matches closed form; max-merge rises while mean-distill stays flat (analytic + trained + rho=1 control). grounding: the reframed 4-panel (forward-KL phase boundary, recovery, metric-choice).

2026-07-05 — real-MNIST external-validity tier (mnist_collapse).

• New image tier plugged into the existing contract (metrics/grounding/output are data-agnostic and reused verbatim): mnist_data.py (load, per-class thickness bins, mode = class×thickness bijection, MnistSampler), mnist_oracle.py (frozen CNN + deterministic thickness = ClassifierOracle, confusion matrix), mnist_vae.py (ConvVAEGenerator), mnist_loop.py (run_mnist_lineage), plus kind=mnist_lineage dispatch in experiment.py, configs/neural/mnist_collapse.yaml, figures/plot_mnist.py, figures/mnist_montage.py (eyeball diagnostic), MnistCfg/OracleCfg.
• Gates: CNN mode accuracy 98.5% (30×30 confusion matrix in the manifest = noise floor); VAE gen-0 recovers full 30/30 support (over-smooths freq, KL≈0.5, no prior hole).
• Result (4 reps): dry (g=0) VAE collapses to a single mode (KL 0.5→18, support 30→1, tail 1.0→0.06, H→0); g=0.1 holds all 30 modes (KL≈0.6, full tail, H≈0.9). Collapse + grounding-rescue confirmed on real images. VAE needs ~10% grounding vs synthetic ~5% (cf. the grounding finding). 99 tests green (+5 torchvision-gated). make mnist / make env-mnist (kept out of make neural).

2026-07-05 — learning kernel (Layer-1 extension) + Riis positioning.

• Prompted by revisiting Layer 1 vs 1.5 and the Riis 2026 paper (arXiv:2604.08554). Added knowledge/kernel.py (LearningKernelCfg: reset u = smoothing, temperature τ = sharpening, floor ε), wired into step.generation_step (post-refit) / StepCtx / DynamicsCfg — identity by default, so the 68 Layer-1 scientific-validation + correctness tests are unchanged.
• Result: neutral Wright–Fisher fails BOTH neural models, oppositely. VAE regime (n=6000,K=30): neutral is inert, sharpening τ=0.8 reproduces the collapse-to-one-mode. RNN regime (n=200,K=256): neutral → H=0, mutation u=0.006 reproduces the H-floor (~0.68). Uniform-mutation overshoots the RNN's KL → its prior is truth-like, not uniform (honest caveat, future refinement).
• configs/layer1/kernel_{sharpen,smooth}.yaml, figures/plot_kernel.py (overlays analytic arms vs the committed neural endpoints), READMEs, tests/test_kernel.py (+6). Wired into make layer1.
• Strategic (see CLAUDE.md finding): concede "collapse=drift" to Riis (prior art; cite); his mixed environment retains OLD SYNTHETIC (no real-data injection) → pessimistic, no g* that prevents collapse. Our defensible novelty: recombination "merge-don't-average" (flagship), the learning-kernel axis (he flags as future work), grounding-as-immigration, real-weights+MNIST breadth, and the Lamarckian society + vertical/cumulative C3 claim (not yet run). Reposition: from "collapse is drift" to a population-genetic CONTROL THEORY for sustaining open-ended knowledge.

2026-07-05 — multi-locus society frame (E7/E8): raised the ceiling to enter the society.

• Prompted by "enter the society with a robust theoretical frame." The single-locus fixed-p* model can't express "exceeding" a ceiling. Generalized knowledge to a distribution over genotypes (knowledge/genotype.py: L biallelic loci, K=2^L, additive fitness, recombination = product of per-locus marginals). Reuses all K-mode machinery + make_retention_matrix (locus mastery).
• E8 (star, kind: society, knowledge/society.py) — the vertical claim: decorrelated parents recombined; sexual merge reaches the optimum (12/12, a genotype no parent had) as parent count grows / ρ→0, while best-parent (~8.7) and mean-mixture soup (~11.6) plateau. configs/layer1/E8.yaml, plot_E8.py, README. The Fisher–Muller effect for AI.
• E7 (kind: genotype_lineage, knowledge/genotype_lineage.py) — advantage of sex: sexual lineage adapts faster than asexual (LD→0 vs LD spike). Honest: a speed advantage, not a permanent ratchet gap.
• Metaphor shift (GG): sexual reproduction with unbounded parents, not teacher→pupil (which caps at the ceiling). Collapse = asexual degradation; cure = sex, no parent limit. Unifies E4+E6 under evolution-of-sex theory; beyond Riis's single-locus n-grams. tests/test_genotype.py (+7). Experiment dispatch (kind in {genotype_lineage, society}) + make layer1 wired.

2026-07-05 — sexual-transmission model made rigorous (E9/E10): landscape robustness + directed sex.

• GG excited by the sexual metaphor; wanted it robust before the full society. Added NK landscape (genotype.nk_fitness), finite n-parent crossover, hill_climb (parents = local optima).
• E9 (recomb_landscape) — "why sex?": on rugged/epistatic landscapes, blind recombination → outbreeding depression (offspring below parents, worse with ruggedness + recombination rate); the optimal recombination rate shrinks with ruggedness. Design rule: merge freely when complementary, sparingly + selectively when entangled.
• E10 (directed_sex) — AI beats biology: random ("biological") sex craters with ruggedness (0.66→0.51); directed sex (choose mates + select offspring + unbounded parents, iterated) tracks/ exceeds the best parent at every ruggedness. The distinctly-AI superpower, no biological analog.
• Complete picture: dramatic super-parent offspring when complementary (E8); outbreeding-depression risk when entangled (E9); directed sex resolves it (E10). configs/layer1/{E9,E10}.yaml, plot_{E9,E10}.py, READMEs, +5 tests (117 green).

2026-07-05 — the dynamic Lamarckian society (E11): the vertical claim / C3 realized.

• knowledge/dynamic_society.py: finite population of N agents (genotypes) on a rugged NK landscape (reality); composes grounding + directed sex + quality-diversity selection + mutation. Grounding made load-bearing via consensus-conformity (self-consumption): selection on g·true_fitness + (1-g)·conformity (GG decision). kind: dynamic_society dispatch.
• 4-arm ablation (12 reps), each breaks distinctly (global_opt≈0.79): full 0.78 (climbs to optimum, diversity maintained longest); no_sex 0.77; no_diversity/greedy 0.74; no_grounding 0.48 (self-consumption collapse to unfit consensus). Only the full society climbs. Integrates E1-E6 + kernel + E7-E10 into one system: needs ALL of grounding + directed sex + diversity.
• configs/layer1/E11.yaml, plot_E11.py, README, tests/test_dynamic_society.py (+5, 122 green). Closes C3 analytically; the LLM rung remains the eventual empirical instantiation.

Remaining (all optional / next)

• NK/epistasis landscape (sign epistasis can make recombination harmful — the honest limit of "sex always helps"); multi-allelic loci. Deepens the frame.
• Learning-kernel refinement: truth-like smoothing prior (prior="truth") + measurement floor for a quantitative RNN match; multi-locus / linkage modes (class×style) as the rigorous home for recombination. Both enrich predictive power and separate us further from Riis's single-locus n-grams.
• The Lamarckian society experiments (multi-agent grounding + decorrelated specialists + recombination + QD-selection + re-mint) and the vertical/cumulative C3 claim — the highest-ceiling, wholly-novel frame; not yet entered.
• region_matched grounding (R>1), remint re-mint gate (optional).
• VAE fidelity: fix the prior-hole mismatch (KL-annealing / free-bits / larger latent) so it clears the gen-0 gate, then add to architectures. Or document as a known limitation.
• Real-MNIST secondary tier (ClassifierOracle + confusion matrix; --extra mnist).
• figures/plot_<name>.py (reuse figures/_figlib.py); wire into make figures.

Discovered during work

• E2 grounding policy vs. the analytic H_eq: Pred. 3's closed form is derived for plain immigration Multinomial(m, p*). Implemented as policy="proportional", and every policy reduces to it at R=1. E2 should therefore run at R=1 (or proportional) so the phase-boundary sweep tracks the exact H_eq; region structure is E3's concern. Decide E2's init (uniform vs truth) when building its config.
• init: {uniform|truth} added to TruthCfg (uniform default, mandated by the decay conformance test). E1/E2 may want truth start for a clean "tail collapses from the truth" story — revisit in Phase 2.

Potential agents

(none proposed yet)

2026-07-05 — LLM prototype (llm_merge): first real-LLM step, honest/partial.

• src/llm/ package: tasks+exact-match verifier, batched eval, LoRA specialise (manual SFT), peft weight-merge (soup/ties), runner (kind: llm_merge). Base Qwen2.5-0.5B-Instruct on one 16GB GPU.
• Result (seed 1): merges are the ONLY models competent across all 3 disjoint families (worst-family ~0.25 vs <0.16 for any single specialist) — the Fisher-Muller signature, robust. Overall-exceeds is marginal (soup 0.64 vs best spec 0.63; ties below), and averaging dilutes peaks (lists 0.43->0.26 = "merge don't average" in real weights). Pipeline works end-to-end; strict overall-exceeds needs scale (bigger base/more families/seeds/dilution-resistant merge) = HPC step.
• Python 3.14 + transformers 5.13 OK; note transformers-5.x apply_chat_template returns a dict. make env-llm/make llm; figures/plot_llm_merge.py, README, tests/test_llm.py (+3, 125 green).

2026-07-05 — LLM merge 7B firm-up on Imperial CX3 (llm_merge_hpc): marginal sign → decisive. ✅

• Ran on one L40S (46 GB) via /imperial-hpc runbook; 8 min walltime; Qwen2.5-7B-Instruct, 200 tests/family.
• Both merges 0.87 overall > best specialist 0.77 (decisive +10 pts) and beat every specialist on every family; worst-family 0.62 vs ≤0.57. Both 0.5B caveats resolved: overall-exceeds is now clean, and dilution VANISHES (merge 0.62 > lists-spec 0.57 on lists) — dilution was a small-model artefact.
• results/llm_merge_hpc/ (README legend, data-driven figure title). Next refinement: module-level union-preserving recombination (MoE-expert/adapter-union = real-weight E8 max-merge), not delta-avg.

2026-07-05 — MoE-expert / union recombination (llm_moe): E8's max vs mean in real weights.

• src/llm/moe.py: router (oracle + training-free nearest-centroid over base embeddings) + MoE generate + router-free per-module max_merge. kind: llm_moe reuses the cached specialists.
• 0.5B result: routing beats fusion decisively — overall 0.74/worst 0.43 vs soup 0.64/0.26, no dilution (recovers each specialist's own-family peak). Riders: learned router trivially perfect (1.00, lexically-separable families) and static max_merge a poor union (0.46, not input-adaptive).
• configs/llm/{moe,moe_hpc}.yaml, figures/plot_llm_moe.py, README, +2 tests (127 green), hpc/llm_moe.pbs.
• 7B firm-up (llm_moe_hpc, CX3 L40S, 9 min): the ordering FLIPS. At 7B fusion wins — soup 0.87 > routing 0.84 > max_merge 0.78 (0.5B had routing 0.74 > soup 0.64). Routing is capped at the best parent per family; fusion composes beyond it at a capable base (soup lists 0.62 > spec 0.57). So "merge, don't average" is a weak-base law, not universal — union wins under dilution (0.5B), fusion wins under composition (7B). Refines E8. Next: fusion + offspring-selection (directed sex). results/llm_moe_hpc/ README + regime-aware figure.

2026-07-05 — Directed sex (llm_directed): E10 in weights = breed offspring + select on verifier.

• src/llm/directed.py: sample a population of Dirichlet-weighted merges, score on a held-out VAL split, keep the best-overall + best-worst-family, report on a fresh TEST split. kind: llm_directed.
• 0.5B: directed selection beats the single a-priori soup on the bred objective — directed_overall 0.69 > soup 0.64; directed_balanced worst-family 0.37 > soup 0.26. Riders: single-objective selection trades off the other axis (overall-breed tanks lists 0.17); a global blend still trails per-input routing (0.74). 7B (CX3 L40S, 9 min): directed ≈ soup (0.868 ≈ 0.873) — soup already composes to ceiling on near-saturated families (strings/arith 1.00), no fitter offspring to breed.
• Through-line: recombination refinements pay off ∝ how suboptimal the default soup is — big at 0.5B, nil at 7B. Honest limit: 7B families near-saturated; a harder benchmark is the fair next test.
• configs/llm/{directed,directed_hpc}.yaml, figures/plot_llm_directed.py, READMEs, hpc/llm_directed.pbs, Makefile llm target, +3 tests (130 green). results/llm_directed{,_hpc}/.

2026-07-05 — HARD benchmark: the 7B "fusion wins / no headroom" nulls were SATURATION artefacts. ✅

• Easy families saturated 7B (strings/arith 1.00), confounding the moe/directed 7B nulls. Built a hard task variant (hard: true: multi-step lists, Caesar ciphers, multi-step/larger arith) threaded through make_tasks/train_specialist/runners; hard specialists cache as spec_*_hard. Ran both at 7B on CX3 (one L40S, 24 min, unsaturated).
• Both nulls flip back to the 0.5B ordering: union/routing 0.500 > fusion 0.40 (soup dilutes the strings specialist 0.665→0.300, below even the best single specialist 0.425); directed selection 0.492

  soup 0.392 (+10 pts). The operative variable is HEADROOM, not base-size — "merge, don't average" and "directed sex" hold whenever there's room to lose to dilution (weak base OR hard tasks); fusion only wins where easy tasks let a strong base compose to ceiling. Vindicates E8 max>mean at 7B.

• configs/llm/{moe_hard,moe_hard_hpc,directed_hard_hpc}.yaml, hpc/llm_hard.pbs, hard READMEs+figures, +1 test (131 green). results/llm_{moe,directed}_hard_hpc/.

2026-08-11 — PNAS submission campaign opened. GG approved PNAS as target after the post-hold re-assessment (fresh lit scan: speciation/sex-framing/mating-systems/headroom all still unclaimed; new concessions First-Extinction 2509.20101 + qt-trait 2407.17493 + verifier-injection 2510.16657; E13 exposed to richer-symmetry objection 2606.23607). Full plan: tasks/workorder-pnas-submission.md. Phases: (1) E13 hardening (scale-aware alignment + emergent-divergence condition), (2) arXiv preprint, (3) llm_speciation + multi-seed LLM arc, (4) PNAS-format manuscript (5 consolidated figures, dual audience), (5) submission mechanics (Zenodo DOI, cover letter, editor/reviewer suggestions).