MachineSex/tasks/todo.md

# Layer 1 Execution Plan — The Lamarckian Society (analytical core)

*Created 2026-07-04. Scope: blueprint §7 build-order steps 1–4 (scaffold → Layer 1 complete, laptop-reproducible). Layer 2 is explicitly out of scope here and gated on Layer 1's scientific-validation tests passing.*

Source of truth: `lamarckian-society-technical-blueprint-v1.md`. Where it is silent I record a decision below rather than improvise silently.

---

## Design decisions to pin before coding (blueprint leaves these open)

These are the only places the spec is genuinely underdetermined. Recommendation given for each; flagged for sign-off.

1. **Config framework — recommend thin pydantic + PyYAML, not Hydra.** Blueprint says "Hydra or a thin equivalent." Hydra is heavyweight and its global-state/`os.chdir` behaviour fights the "pass `rng` explicitly, results are a pure function of resolved config" requirement. A thin loader (pydantic models for validation + a small sweep-expander) is dependency-light, aligns with the stdlib-first preference, and makes the "write resolved config beside results" contract trivial. *Decision: thin pydantic loader.*

2. **Selection fitness `f_i` (the reality-anchored score).** Blueprint: "fitness is predictive accuracy against `p*`" but gives no formula for the discrete model. *Decision:* `f_i = p*_eff_i` (truth frequency = fitness; reality-anchored by construction). Post-selection distribution `p'_i ∝ p_i^(1−α) · f_i`, matching the blueprint's stated `w_i ∝ f_i·(p_i)^(−α)` with `α=0` recovering fitness-proportional greedy. Document as a modelling choice; `select-then-sample` vs `sample-then-select` is the documented robustness switch (§2.2).

3. **`tail_mask` definition.** Two knobs exist (`tail_frac`, `tail_threshold`). *Decision:* the metric-bearing tail set is the §2.3 definition `{i : p*_i < tail_threshold}`. `tail_frac` only drives the `twocomponent` construction (fraction of items placed in the low-mass component). Document that for `zipf`, `tail_frac` is unused.

4. **`TrueDist` type.** *Decision:* a frozen dataclass `TrueDist(p_star: np.ndarray, regions: np.ndarray, tail_mask: np.ndarray)` — immutable, so `p*` cannot be mutated in place (except the deliberate re-mint path, which produces a new object).

5. **Region partition.** *Decision:* contiguous blocks; require `K % R == 0` (assert with a clear error) to keep per-region math clean for v1.

6. **Re-mint semantics (E6).** *Decision:* re-mint replaces `p_star_eff` with a *new* `TrueDist` built from current `p_t` (fresh tail_mask recomputed on `p_t`), and the *original* `TrueDist` is retained **only** for the KL-to-original metric, never for grounding. This is the irreversibility the experiment tests.

---

## Phase 0 — Scaffold & environment

- [x] Install `uv` (`curl -LsSf https://astral.sh/uv/install.sh | sh`; lands in `~/.local/bin`, no sudo).
- [x] **Reproducibility path = `uv` venv from a committed, hash-pinned `uv.lock`** (GG decision, 2026-07-04; no Apptainer/Docker for Layer 1). `pyproject.toml` (Python ≥3.11; deps: numpy, scipy, pandas, pyarrow, matplotlib, pydantic, pyyaml; dev: pytest). Commit `uv.lock`. Create `.venv` via `uv sync`.
- [x] Repo layout per §5: `src/knowledge/`, `configs/layer1/`, `figures/`, `results/` (gitignored), `tests/`, `paper/`. Add `src/lamarckian/` package root or make `knowledge` importable (decide package name — recommend `src/lamarckian/knowledge/...` with `src`-layout).
- [x] `.gitignore` (`.venv/`, `results/`, `__pycache__/`, `*.parquet` under results but keep hashes).
- [x] Seeding util `lamarckian/utils/seeding.py`: master seed → `np.random.SeedSequence(seed).spawn(n)` → per-replicate `np.random.default_rng(child)`. No global RNG anywhere.
- [x] Config loader `lamarckian/config.py`: pydantic schema mirroring the §2.7 YAML, a `load_config(path)`, a `expand_sweeps(cfg) -> list[ResolvedConfig]`, and `write_resolved(cfg, dir)`.
- [x] Manifest util: `write_manifest(dir, results_df)` recording lib versions, master seed, `git rev-parse HEAD`, content hash of `results.parquet`.
- [x] `Makefile` targets (`env`, `test`, `layer1`, `figures`, `clean`) + `pytest` skeleton. **Gate: `make test` green on a trivial test.**
- [x] Move `lamarckian-society-technical-blueprint-v1.md` → `paper/blueprint.md` per §5 (confirm with GG first — it's referenced by name elsewhere).

## Phase 1 — Core primitives + null model + VALIDATION GATE

Implement to the normative signatures in §2.7. Order chosen so each piece is unit-testable before the next depends on it.

- [x] `knowledge/truth.py::make_true_distribution` → `TrueDist`. Support `tail ∈ {zipf, twocomponent}`. Unit tests: normalisation, region block sizes, tail_mask matches threshold, determinism from seed.
- [x] `knowledge/metrics.py`: `forward_kl` (with `eps` floor, logged), `heterozygosity`, `tail_mass`, `support_size`, plus per-region variants. Unit tests on hand-computed small vectors.
- [x] `knowledge/step.py::generation_step` — **null path first** (single teacher, `m=0`, selection `none`): `c ~ Multinomial(n, p_t)`, `p_{t+1}=c/n`. Exactly neutral Wright–Fisher.
- [x] `knowledge/lineage.py::run_lineage(cfg, seed)` → tidy per-generation DataFrame (all §2.3 metrics, global + per-region).
- [x] `tests/test_scientific_validation.py` — **the spine:**
  - **Pred. 1** heterozygosity decay: mean `H_t` over replicates matches `H₀(1−1/n)^t`. Prefer testing the full deterministic mean-recursion trajectory (subsumes the fixed point), within Monte-Carlo CI.
  - **Pred. 2** fixation probability = initial frequency (long runs, statistical tolerance).
- [x] **HARD GATE: do not proceed until Pred. 1–2 pass.** If drift ≠ analytic decay, the harness is wrong — fix here.

## Phase 2 — Grounding + E1 + E2 (the headline)

- [x] `knowledge/step.py::structured_multinomial(m_vector, p_star, regions, policy, rng)` — per-region immigration draws from `p*` restricted+renormalised to each region; `uniform` spreads `m` evenly, `matched` concentrates on exercised regions. Returns length-K counts.
- [x] Extend `generation_step` with grounding (pooled draw, `g = m/(n+m)`).
- [x] **Pred. 3** validation — *exact* equilibrium `H_eq = H*·m(2n+m−1)/(n+2nm+m²)`: run to stationarity (burn-in + late-generation + replicate averaging), assert `<0.1%` rel. error vs closed form across an `m` grid. Also assert the `m→0` and `m→∞` limits.
- [x] **Pred. 4** validation — tail-persistence: item of freq `p*_i` maintained iff `m·p*_i ≳ 1`; verify the survival transition location statistically.
- [x] `knowledge/experiment.py::run_experiment(cfg)` — sweep grid × `n_replicates`; long-form results + CIs; write `results.parquet` + `resolved_config.yaml` + `manifest.json`.
- [x] **E1** config + run: `m=0`, single teacher, no selection. Expect `H` geometric decay, support→1, KL diverges, tail-first loss.
- [x] **E2** config + run: sweep `g`, single teacher, uniform grounding, no selection. Locate critical `g*` (transition in **tail mass / support**, since H is smooth in m — the sharp threshold is in discrete tail survival). Report `g*` with CI. **This is the load-bearing result.**
- [x] `figures/plot_E1.py`, `plot_E2.py` — read `results.parquet` only.

## Phase 3 — E3–E6

- [x] **E3 region-matched grounding.** Fixed total `m`; `uniform` vs `matched`; one designated inherited-but-unwatered region with a rare tail. Expect uniform lets that region's tail collapse; matched holds it. Per-region metrics essential. `plot_E3.py`.
- [x] **§2.7.1 correlated-teacher construction** — `knowledge/teachers.py`:
  - `make_retention_matrix(T, K_T, rho, q, rng)` — shared-switch exchangeable Bernoulli.
  - `make_correlated_teachers(...)` — retention→distributions (head kept at `p*`; tail at `p*_i` if retained else `tail_floor`; renormalise). `region_specialisation` option.
  - **Pred. 5** validation: `make_retention_matrix` reproduces marginal `q`, pairwise `ρ`, and union coverage `U(K_T,ρ,q)=T[ρq+(1−ρ)(1−(1−q)^K_T)]` to 3 decimals over a `(ρ,q)` grid.
- [x] **E4 multi-teacher decorrelation.** Sweep `K_T∈{1,2,3,5}`, `ρ∈[0,1]` at fixed `q`, matched budget (`n/K_T` each). Report **both** union `U` and post-distillation surviving coverage; show their gap shrinks as `g` rises. `plot_E4.py` (coverage surface over `(K_T,ρ)`).
- [x] **E5 QD vs greedy.** `apply_selection` (`none`/`greedy`/`qd`, pinned fitness form). Sweep novelty `α`. Expect greedy→fixation (`H→0`), qd holds `H` plateau + re-introduces tails. `plot_E5.py`.
- [x] **E6 re-mint gate.** Re-mint at high vs low `H`; track KL to *original* truth. Expect collapsed re-mint locks KL high forever; gated (high-H) does not. `plot_E6.py`.

## Phase 4 — Reproducibility polish (Layer 1 slice)

- [x] `configs/layer1/E1..E6.yaml` all committed with explicit params (no magic numbers in code).
- [x] `paper/figure_manifest.md` — the §6 claim→experiment→figure rows for Layer 1.
- [x] `make layer1` runs E1–E6; `make figures` regenerates all figures from committed parquet.
- [x] Full `test_correctness.py` (shapes, normalisation, determinism) + `test_scientific_validation.py` (Pred. 1–5) green in CI.
- [x] `reproduce.sh` (`uv sync` → `make test` → `make layer1` → `make figures` → write `REPRODUCED.md` diffing committed result hashes) + `README.md` reproduce section. **No container** — the committed `uv.lock` is the reproducibility source of truth (per GG, 2026-07-04); a Dockerfile may later wrap the same lockfile for Layer 2's GPU work.

---

## Definition of done (Layer 1)

Every Layer-1 row of blueprint §6 has a committed figure produced by `make figures` from committed results; all §2.4 analytic checks (Pred. 1–5) pass; `make layer1 && make figures` reproduces from a clean `.venv`. Then — and only then — Layer 2 may begin.

## Falsifier watch (report honestly if hit)

- E2 tail mass flat in `g`, or only stabilises as `g→1` → multigenerational thesis refuted.
- E3 uniform protects as well as matched → region-matching claim dies.
- E4 no surviving-coverage benefit at matched budget → recombination claim dies.
- E5 qd ≤ greedy stationary `H` → QD does no work.
- E6 collapsed lineage recovers original-truth tails after re-mint → irreversibility overstated.

## Review — progress log

**2026-07-04 — Phases 0 & 1 complete; hard gate PASSED.**

- Reorg: docs → `paper/` (`blueprint.md`, `the-lamarckian-society-v4.md`). src-layout under `src/knowledge/`.
- A pre-existing `tests/test_scientific_validation.py` (author-supplied, 22 KB) turned out to hard-specify the package contract — implemented *to it* rather than inventing interfaces. Key contracts it locked (now honoured): package imports as `knowledge.*`; `run_lineage(cfg_dict, seed)` returns a tidy per-gen frame with a `heterozygosity` column, rows 0..T; `p_0` initialises **uniform** (`H_0=1−1/K`); `metrics.heterozygosity` and `teachers.make_retention_matrix` match the reference to 1e-12 / closed form.
- Env: `uv` 0.11.26 installed; `pyproject.toml` + `uv.lock` committed; numpy 2.5, pandas 3.0, scipy 1.18, pydantic 2.13, pytest 9.1.
- Modules written: `metrics`, `seeding`, `config` (dataclasses + `from_dict`), `truth`, `teachers`, `step`, `lineage`. Config is dataclass-based (not pydantic) — the conformance test passes a raw dict; dataclasses validate cleanly and stay stdlib-simple. **Pydantic still a dep for the Phase-2 YAML/experiment layer.**
- **Results: `make test` green — 68 passed** (48 scientific-validation, 20 correctness). Conformance tests RAN (not skipped): Pred. 1 (neutral decay), Pred. 3 (exact `H_eq`), Pred. 5 (union coverage) all pass against the real package. **The Pred. 1–2 hard gate is passed**, and grounding already conforms to the exact equilibrium.

Design decisions #1 (dataclasses now / pydantic at YAML layer), #2 (fitness `f_i=p*_i`), #3 (threshold tail_mask), #6 (re-mint discard) all implemented as planned. Region design: each region an identical 1/R-mass block (symmetric; reduces to global Zipf at R=1, matching the reference).

**2026-07-04 — Phase 2 complete (E1 + E2).**

- `experiment.py`: sweep expansion (Cartesian grid; special-cases `g→m`), paired replicate seeds (shared across grid points), output contract (`results.parquet` + `resolved_config.yaml` + `manifest.json` with lib versions + git commit + sha256). CLI `python -m knowledge.experiment <cfg>`.
- **E1 (null collapse)** — reproduces tail-first collapse: H geometric decay matches `H₀(1−1/n)ᵗ` within CI; tail items die ~10× faster than head items; support 500→1; forward-KL diverges. Figure `results/E1/E1.png`.
- **E2 (headline)** — `H_sim` tracks the *exact* `H_eq` closed form across the sweep; phase boundary at **`g* ≪ 1`**: g=0.005 (m=1 real sample vs n=200) → 68% of truth H; g=0.05 → 96%. g=0 slides to ~0.10 over 500 gens. Figure `results/E2/E2.png`. **Headline result achieved.**
- Metric subtlety found & fixed: aggregate **`tail_mass` is a drift martingale** (mean-conserved), so it's a poor collapse indicator. Added `tail_support`/`head_support`/`tail_frac_alive`/`head_frac_alive`; E1 & E2 figures now use tail-*item* survival, which is honest and monotone.
- E2 extended 300→500 generations (GG-approved) so the g=0 arm visibly approaches 0 while g>0 arms sit on plateaus.
- Makefile `layer1`/`figures` wired to E1–E2. `make test` still green (68).

**2026-07-04 — Phase 3 complete (E3–E6) + E2 analysis add-ons.**

- **E3** region-matched grounding: added `grounding.exercised` knob + per-region `tailalive_region_r`. Target region tail survival 0.49 (matched) vs 0.07 (uniform). Note: per-region *H* is mass-confounded — used tail-item survival instead.
- **E4** multi-teacher recombination: bespoke `run_coverage` runner (`kind: coverage`). Union coverage matches `U(K_T,ρ,q)` exactly. **Key finding (GG-approved): mean-mixture distillation gives NO surviving benefit (conservation law — dilution cancels the union gain); max-merge (M2N2-style) does.** E4 reports both. In CLAUDE.md.
- **E5** QD vs greedy: greedy → H≈0.01 (fixation); qd holds H 0.48–0.88 rising with α. qd ≫ greedy.
- **E6** re-mint gate: added `arm` multi-override sweep type. Re-mint while collapsed → KL-to-original diverges (lock-in) + accelerates H collapse; diversity gate (H≥0.75) blocks it → bounded; healthy re-mint harmless.
- **E2 analysis add-ons** (companion work order `tasks/workorder-E2-analysis-addons.md`, verified): new `analysis.py` (`reduce_to_stationary`, `critical_grounding` bootstrap CI) — real E2 **g*=0.048, CI [0.047,0.050]**; `metrics.tail_band_metrics` + per-band lineage logging; `tests/test_analysis.py` reproduces the work order's verified numbers exactly. E2 figure rebuilt 2×2. **Deviation:** used truth-mass-weighted tail coverage instead of raw `tail_mass` (a drift martingale).
- All six figures regenerate via `make figures`; **71 tests green**.

---

# Layer 1.5 — Architecture-general neural existence proof (RNN/VAE/MLP + synthetic/MNIST)

*Created 2026-07-04. Plan: `~/.claude/plans/we-are-going-to-cheerful-fog.md`. Re-scopes Layer 2:
build a cheap, architecture-general neural collapse proof in real trained weights on a
fully-synthetic sandbox (exact known `p*`) before the LLM rung. Locked decisions: exact-oracle
categorical token sequences; Histogram+RNN+VAE+MLP; real MNIST as secondary confirmation; LLM +
C3 vertical claim deferred.*

## Progress log

**2026-07-04 — Stages A, B, plumbing complete.**

- **Env:** installed `uv` 0.11.26 (`~/.local/bin`); `/home` was 100% full — GG approved clearing
  pip/yay/browser caches (~10 GB freed). Base venv synced; 71 Layer-1 tests green.
- **Stage A (scaffold, pure NumPy):** `src/neural/` — `config.py` (frozen dataclasses reusing
  `knowledge.config` GroundingCfg/RemintCfg/MetricsCfg/_sub), `synthetic.py` (mode-truth via
  `make_true_distribution`; lossless identity + stochastic style token grammar), `oracle.py`
  (`ExactOracle` zero-error + `measure_distribution`), `models.py` (`GenerativeModel` protocol +
  `HistogramModel` bridge), `evaluate.py` (reuses `knowledge.metrics`, Layer-1 row schema),
  `generation_loop.py` (`run_generative_lineage`, reuses `allocate_m`/`structured_multinomial`).
  15 correctness tests green.
- **Stage B — HARD GATE PASSED:** `tests/test_neural_validation.py` — histogram lineage reproduces
  Pred. 1 (neutral decay, <3% rel err), Pred. 3 (exact `H_eq`, <5%), and tracks Layer-1
  `run_lineage` directly (<3%). The neural plumbing reproduces the analytic core.
- **Plumbing:** `neural/experiment.py` (`run_and_save` dispatch on `kind`, reuses `_apply_param`
  g→m, paired seeds); extended `knowledge.experiment.save_artifacts` (optional `extra_libs`,
  `extra_manifest`, injectable `grid`; skips missing libs — backward compatible). `configs/neural/bridge.yaml`,
  Makefile `neural`/`env-neural`/`layer2` targets, `.gitignore`. (Experiments are named
  descriptively — `bridge`, `collapse`, `grounding`, `architectures` — not by code.)
- **`bridge` result (17s):** neural **g\* = 0.0474, CI [0.045, 0.052]** — reproduces Layer-1 E2's
  g\*=0.048 essentially exactly (g=0.005→67% of H*, g=0.05→96%). **89 tests green.**

**2026-07-04 — Stage C: torch models + collapse/grounding/architectures.**

- **Env:** torch **2.12.1+cu130** (default PyPI wheel ships CUDA 13, matches RTX A4000 driver;
  no custom index needed, cp314 wheels exist). `--extra neural` = torch only; `--extra mnist` =
  torchvision (later). `UV_CACHE_DIR=/tmp` during install (RAM-backed) to spare `/home`.
- **Models:** `torch_models.py` (RNNGenerator, autoregressive GRU), `torch_mlp.py` (autoregressive
  MLP, causal-masked), `torch_vae.py` (sequence VAE), `train.py` (determinism flags + device/seed
  helpers derived from the SeedSequence stream). `tests/test_neural_torch.py` (torch-gated): gen-0
  fidelity (rnn+mlp) + dry-collapse/grounded-holds. **92 tests green.**
- **Validated regime:** K=256, n=200, zipf_s=1.3, RNN hidden=128/epochs=25. RNN gen-0 fidelity
  KL(p*‖p̂)=0.008, 64/64 (or 256/256) modes recovered. MLP fidelity KL=0.011. **VAE does NOT clear
  the gen-0 gate** on the Zipf-codeword task (KL≈0.8; prior-hole mismatch — sampling z~N(0,I) misses
  the aggregate posterior) → excluded from `architectures` to avoid confounding collapse with underfitting.
- **`collapse` (in weights):** dry RNN lineage collapses — forward-KL rises to ~2.2 vs grounded
  ~1.4; grounding lifts tail survival (tailalive 0.31 dry → 0.50 at g=0.02). Sign confirmed.
- **`grounding` (neural phase boundary):** stationary H hovers 80–91% of H* and is **noisy / non-monotonic**
  at 5 reps — no crisp g*. **KEY FINDING:** the neural models' smoothing inductive bias *partially
  resists* H-collapse (dry H stays ~83% of H*), so **forward-KL and tail survival are the sharp
  neural collapse metrics, not H** (mirrors Layer-1's "H is smooth; the threshold lives in tail
  survival"). `grounding` needs (a) forward-KL as the phase metric, (b) more reps (≥10), and/or (c) a
  stronger-collapse regime for a clean neural g*.
- **`architectures` (architecture-generality) — clean result:** collapse + grounding-rescue appear in ALL three
  model classes (dry→grounded forward-KL: histogram 6.2→4.6, MLP 4.8→1.3, RNN 3.8→1.1; tailalive
  RNN 0.41→0.64, MLP 0.07→0.20). The WF operator is architecture-general. Bonus: neural smoothing
  lets RNN/MLP retain *more* tail than the exact histogram under grounding (they generalise to
  unseen codewords) — an inductive-bias finding worth the write-up.

**2026-07-04 — `recombination` (load-bearing E4 replication).**

- `recombine.py` mirrors `run_coverage` but trains K_T specialist RNNs on assignments from the
  exact shared-switch retention construction (K_T/rho/q clean; union matches the closed form), then
  recombines the *measured* teacher distributions two ways: `mean` (naive pooling) vs `max`
  (oracle-guided union / M2N2-style), each followed by size-n resampling. The neural merge is the
  per-mode max over teacher distributions (oracle-guided), NOT weight-averaging of RNNs.
- **Result (8 reps):** at rho=0, **union rises 0.49→0.96** (supply matches closed form); analytic
  **surviving_max rises 0.043→0.087 while surviving_mean stays flat ~0.045** — the conservation law
  (averaging cancels the union gain; max-merge realises it). At rho=1 (identical teachers) union AND
  max are flat — more identical teachers buy nothing. **The "merge, don't average" lesson holds in
  the neural setting.** Trained-weight columns show the same signs but noisier: neural smoothing
  inflates baseline survival and the deep tail barely clears n=200 resampling (compresses magnitude)
  — the expected inductive-bias caveat. torch-gated test added. **93 tests green.**

**2026-07-05 — `grounding` refinement + figure (honest reframing).**

- Re-ran at **18 reps** (n_eval 15000, 30 gens, g grid refined to 9 points). Falsifier pinned in
  the config *before* running.
- **forward-KL is the operative neural collapse metric — NOT H or tail-survival.** The RNN's
  smoothing keeps spurious tail modes alive, so `tail_truth_mass_alive` is flat/**non-monotone**
  in g (dry 0.54 > most grounded) and H stays 0.77–0.85 of H\*. Stationary **forward-KL** falls
  monotonically (dry 2.08 → g=0.2: 0.75), significant at g≥0.05 (paired t→3.3; 89% of lineages
  improve at g=0.2). Refines the earlier "forward-KL AND tail survival" note.
- **The sharp `g*≪1` is an exact-operator feature, softened by neural smoothing.** Median-recovery
  grounding (half the KL gap closed) g≈**0.04** (bootstrap CI [0.004, 0.116]) — echoes Layer-1's
  0.048 — but full (95%) recovery needs g≈0.19. Quantitative `g*≪1` is carried by the histogram
  **bridge** (0.047); the RNN confirms the SIGN and softens the sharpness (blueprint §3.5 met).
- **Honest note:** the pre-registered 95%-of-H\*/tail-survival falsifier is *not* met — because
  those are the wrong metrics for a smoothing model, not because grounding fails. Reported as such.
- **Robustness fix:** a fully-degenerate RNN can emit only invalid codewords → `measure_distribution`
  now returns a terminal-collapse sentinel (fixation on the dominant mode) instead of crashing a long
  sweep. Edge-case test added. `figures/plot_grounding.py` written (4-panel, states its own verdict),
  wired into `make figures` (glob all `plot_*.py` except `plot_E[1-6]`).

**2026-07-05 — neural figures (all five).**

- `figures/plot_{bridge,collapse,grounding,architectures,recombination}.py`, each a pure function
  of its committed bundle (reuse `figures/_figlib.py`), wired into `make figures` (glob all
  `plot_*.py` except `plot_E[1-6]` / `_*`). **bridge**: neural histogram runner sits exactly on the
  exact `H_eq` curve, g*=0.047 (HARD-GATE visual). **collapse**: dry GRU forward-KL climbs, grounded
  held; H barely moves. **architectures**: grouped bars — forward-KL falls / tail survival rises with
  grounding across histogram/GRU/MLP. **recombination**: union matches closed form; max-merge rises
  while mean-distill stays flat (analytic + trained + rho=1 control). **grounding**: the reframed
  4-panel (forward-KL phase boundary, recovery, metric-choice).

**2026-07-05 — real-MNIST external-validity tier (`mnist_collapse`).**

- New image tier plugged into the existing contract (metrics/grounding/output are data-agnostic and
  reused verbatim): `mnist_data.py` (load, per-class thickness bins, mode = class×thickness bijection,
  `MnistSampler`), `mnist_oracle.py` (frozen CNN + deterministic thickness = `ClassifierOracle`,
  confusion matrix), `mnist_vae.py` (`ConvVAEGenerator`), `mnist_loop.py` (`run_mnist_lineage`), plus
  `kind=mnist_lineage` dispatch in `experiment.py`, `configs/neural/mnist_collapse.yaml`,
  `figures/plot_mnist.py`, `figures/mnist_montage.py` (eyeball diagnostic), `MnistCfg`/`OracleCfg`.
- **Gates:** CNN mode accuracy **98.5%** (30×30 confusion matrix in the manifest = noise floor);
  VAE gen-0 recovers full 30/30 support (over-smooths freq, KL≈0.5, no prior hole).
- **Result (4 reps):** dry (g=0) VAE **collapses to a single mode** (KL 0.5→18, support 30→1, tail
  1.0→0.06, H→0); **g=0.1 holds all 30 modes** (KL≈0.6, full tail, H≈0.9). Collapse + grounding-rescue
  confirmed on real images. VAE needs ~10% grounding vs synthetic ~5% (cf. the `grounding` finding).
  **99 tests green** (+5 torchvision-gated). `make mnist` / `make env-mnist` (kept out of `make neural`).

**2026-07-05 — learning kernel (Layer-1 extension) + Riis positioning.**

- Prompted by revisiting Layer 1 vs 1.5 and the Riis 2026 paper (arXiv:2604.08554). Added
  `knowledge/kernel.py` (`LearningKernelCfg`: reset `u` = smoothing, temperature `τ` = sharpening,
  floor `ε`), wired into `step.generation_step` (post-refit) / `StepCtx` / `DynamicsCfg` — **identity
  by default, so the 68 Layer-1 scientific-validation + correctness tests are unchanged.**
- **Result:** neutral Wright–Fisher fails BOTH neural models, oppositely. VAE regime (n=6000,K=30):
  neutral is inert, sharpening `τ=0.8` reproduces the collapse-to-one-mode. RNN regime (n=200,K=256):
  neutral → H=0, mutation `u=0.006` reproduces the H-floor (~0.68). Uniform-mutation overshoots the
  RNN's KL → its prior is truth-like, not uniform (honest caveat, future refinement).
- `configs/layer1/kernel_{sharpen,smooth}.yaml`, `figures/plot_kernel.py` (overlays analytic arms vs
  the committed neural endpoints), READMEs, `tests/test_kernel.py` (+6). Wired into `make layer1`.
- **Strategic (see CLAUDE.md finding):** concede "collapse=drift" to Riis (prior art; cite); his
  mixed environment retains OLD SYNTHETIC (no real-data injection) → pessimistic, no g* that prevents
  collapse. Our defensible novelty: recombination "merge-don't-average" (flagship), the learning-kernel
  axis (he flags as future work), grounding-as-immigration, real-weights+MNIST breadth, and the
  Lamarckian society + vertical/cumulative C3 claim (not yet run). Reposition: from "collapse is drift"
  to a population-genetic CONTROL THEORY for sustaining open-ended knowledge.

**2026-07-05 — multi-locus society frame (E7/E8): raised the ceiling to enter the society.**

- Prompted by "enter the society with a robust theoretical frame." The single-locus fixed-`p*` model
  can't express "exceeding" a ceiling. Generalized knowledge to a distribution over **genotypes**
  (`knowledge/genotype.py`: `L` biallelic loci, `K=2^L`, additive fitness, recombination = product of
  per-locus marginals). Reuses all K-mode machinery + `make_retention_matrix` (locus mastery).
- **E8 (star, `kind: society`, `knowledge/society.py`) — the vertical claim:** decorrelated parents
  recombined; **sexual merge reaches the optimum (12/12, a genotype no parent had)** as parent count
  grows / `ρ→0`, while best-parent (~8.7) and mean-mixture soup (~11.6) plateau. `configs/layer1/E8.yaml`,
  `plot_E8.py`, README. The Fisher–Muller effect for AI.
- **E7 (`kind: genotype_lineage`, `knowledge/genotype_lineage.py`) — advantage of sex:** sexual lineage
  adapts faster than asexual (LD→0 vs LD spike). Honest: a speed advantage, not a permanent ratchet gap.
- **Metaphor shift (GG):** sexual reproduction with **unbounded parents**, not teacher→pupil (which caps
  at the ceiling). Collapse = asexual degradation; cure = sex, no parent limit. Unifies E4+E6 under
  evolution-of-sex theory; beyond Riis's single-locus n-grams. `tests/test_genotype.py` (+7).
  Experiment dispatch (`kind` in {genotype_lineage, society}) + `make layer1` wired.

**2026-07-05 — sexual-transmission model made rigorous (E9/E10): landscape robustness + directed sex.**

- GG excited by the sexual metaphor; wanted it robust before the full society. Added NK landscape
  (`genotype.nk_fitness`), finite n-parent `crossover`, `hill_climb` (parents = local optima).
- **E9 (`recomb_landscape`) — "why sex?":** on rugged/epistatic landscapes, blind recombination →
  **outbreeding depression** (offspring below parents, worse with ruggedness + recombination rate);
  the optimal recombination rate shrinks with ruggedness. Design rule: merge freely when
  complementary, sparingly + selectively when entangled.
- **E10 (`directed_sex`) — AI beats biology:** random ("biological") sex craters with ruggedness
  (0.66→0.51); **directed sex** (choose mates + select offspring + unbounded parents, iterated) tracks/
  exceeds the best parent at every ruggedness. The distinctly-AI superpower, no biological analog.
- Complete picture: dramatic super-parent offspring when complementary (E8); outbreeding-depression
  risk when entangled (E9); directed sex resolves it (E10). `configs/layer1/{E9,E10}.yaml`,
  `plot_{E9,E10}.py`, READMEs, +5 tests (117 green).

**2026-07-05 — the dynamic Lamarckian society (E11): the vertical claim / C3 realized.**

- `knowledge/dynamic_society.py`: finite population of N agents (genotypes) on a rugged NK landscape
  (reality); composes grounding + directed sex + quality-diversity selection + mutation. Grounding
  made load-bearing via consensus-conformity (self-consumption): selection on
  `g·true_fitness + (1-g)·conformity` (GG decision). `kind: dynamic_society` dispatch.
- **4-arm ablation (12 reps), each breaks distinctly (global_opt≈0.79):** full 0.78 (climbs to optimum,
  diversity maintained longest); no_sex 0.77; no_diversity/greedy 0.74; **no_grounding 0.48
  (self-consumption collapse to unfit consensus).** Only the full society climbs. Integrates E1-E6 +
  kernel + E7-E10 into one system: needs ALL of grounding + directed sex + diversity.
- `configs/layer1/E11.yaml`, `plot_E11.py`, README, `tests/test_dynamic_society.py` (+5, 122 green).
  Closes C3 analytically; the LLM rung remains the eventual empirical instantiation.

## Remaining (all optional / next)
- [ ] **NK/epistasis landscape** (sign epistasis can make recombination harmful — the honest limit of
  "sex always helps"); **multi-allelic loci**. Deepens the frame.
- [ ] **Learning-kernel refinement:** truth-like smoothing prior (`prior="truth"`) + measurement floor
  for a quantitative RNN match; **multi-locus / linkage** modes (class×style) as the rigorous home for
  recombination. Both enrich predictive power and separate us further from Riis's single-locus n-grams.
- [ ] **The Lamarckian society experiments** (multi-agent grounding + decorrelated specialists +
  recombination + QD-selection + re-mint) and the **vertical/cumulative C3 claim** — the highest-ceiling,
  wholly-novel frame; not yet entered.
- [ ] **`region_matched`** grounding (R>1), **`remint`** re-mint gate (optional).
- [ ] **VAE fidelity:** fix the prior-hole mismatch (KL-annealing / free-bits / larger latent) so it
  clears the gen-0 gate, then add to `architectures`. Or document as a known limitation.
- [ ] Real-MNIST secondary tier (`ClassifierOracle` + confusion matrix; `--extra mnist`).
- [ ] `figures/plot_<name>.py` (reuse `figures/_figlib.py`); wire into `make figures`.

## Discovered during work

- **E2 grounding policy vs. the analytic H_eq:** Pred. 3's closed form is derived for *plain* immigration `Multinomial(m, p*)`. Implemented as `policy="proportional"`, and every policy reduces to it at `R=1`. E2 should therefore run at `R=1` (or `proportional`) so the phase-boundary sweep tracks the exact `H_eq`; region structure is E3's concern. Decide E2's `init` (uniform vs truth) when building its config.
- `init: {uniform|truth}` added to `TruthCfg` (uniform default, mandated by the decay conformance test). E1/E2 may want `truth` start for a clean "tail collapses from the truth" story — revisit in Phase 2.

## Potential agents

*(none proposed yet)*

**2026-07-05 — LLM prototype (`llm_merge`): first real-LLM step, honest/partial.**

- `src/llm/` package: tasks+exact-match verifier, batched eval, LoRA specialise (manual SFT), peft
  weight-merge (soup/ties), runner (`kind: llm_merge`). Base Qwen2.5-0.5B-Instruct on one 16GB GPU.
- **Result (seed 1):** merges are the ONLY models competent across all 3 disjoint families
  (worst-family ~0.25 vs <0.16 for any single specialist) — the Fisher-Muller signature, robust.
  Overall-exceeds is marginal (soup 0.64 vs best spec 0.63; ties below), and averaging dilutes peaks
  (lists 0.43->0.26 = "merge don't average" in real weights). Pipeline works end-to-end; strict
  overall-exceeds needs scale (bigger base/more families/seeds/dilution-resistant merge) = HPC step.
- Python 3.14 + transformers 5.13 OK; note transformers-5.x apply_chat_template returns a dict.
  `make env-llm`/`make llm`; `figures/plot_llm_merge.py`, README, `tests/test_llm.py` (+3, 125 green).

**2026-07-05 — LLM merge 7B firm-up on Imperial CX3 (`llm_merge_hpc`): marginal sign → decisive.** ✅
- Ran on one L40S (46 GB) via `/imperial-hpc` runbook; 8 min walltime; Qwen2.5-7B-Instruct, 200 tests/family.
- **Both merges 0.87 overall > best specialist 0.77** (decisive +10 pts) and beat every specialist on
  every family; worst-family 0.62 vs ≤0.57. Both 0.5B caveats resolved: overall-exceeds is now clean,
  and dilution VANISHES (merge 0.62 > lists-spec 0.57 on lists) — dilution was a small-model artefact.
- `results/llm_merge_hpc/` (README legend, data-driven figure title). Next refinement: module-level
  union-preserving recombination (MoE-expert/adapter-union = real-weight E8 max-merge), not delta-avg.

**2026-07-05 — MoE-expert / union recombination (`llm_moe`): E8's `max` vs `mean` in real weights.**
- `src/llm/moe.py`: router (oracle + training-free nearest-centroid over base embeddings) + MoE
  generate + router-free per-module `max_merge`. `kind: llm_moe` reuses the cached specialists.
- **0.5B result:** routing beats fusion decisively — overall 0.74/worst 0.43 vs soup 0.64/0.26, no
  dilution (recovers each specialist's own-family peak). Riders: learned router trivially perfect
  (1.00, lexically-separable families) and static `max_merge` a poor union (0.46, not input-adaptive).
- `configs/llm/{moe,moe_hpc}.yaml`, `figures/plot_llm_moe.py`, README, +2 tests (127 green),
  `hpc/llm_moe.pbs`.
- **7B firm-up (`llm_moe_hpc`, CX3 L40S, 9 min): the ordering FLIPS.** At 7B fusion wins —
  soup 0.87 > routing 0.84 > max_merge 0.78 (0.5B had routing 0.74 > soup 0.64). Routing is capped at
  the best parent per family; fusion *composes beyond* it at a capable base (soup lists 0.62 >
  spec 0.57). So "merge, don't average" is a **weak-base law**, not universal — union wins under
  dilution (0.5B), fusion wins under composition (7B). Refines E8. Next: fusion + offspring-selection
  (directed sex). `results/llm_moe_hpc/` README + regime-aware figure.