MachineSex/tasks/todo.md

# Layer 1 Execution Plan — The Lamarckian Society (analytical core)

*Created 2026-07-04. Scope: blueprint §7 build-order steps 1–4 (scaffold → Layer 1 complete, laptop-reproducible). Layer 2 is explicitly out of scope here and gated on Layer 1's scientific-validation tests passing.*

Source of truth: `lamarckian-society-technical-blueprint-v1.md`. Where it is silent I record a decision below rather than improvise silently.

---

## Design decisions to pin before coding (blueprint leaves these open)

These are the only places the spec is genuinely underdetermined. Recommendation given for each; flagged for sign-off.

1. **Config framework — recommend thin pydantic + PyYAML, not Hydra.** Blueprint says "Hydra or a thin equivalent." Hydra is heavyweight and its global-state/`os.chdir` behaviour fights the "pass `rng` explicitly, results are a pure function of resolved config" requirement. A thin loader (pydantic models for validation + a small sweep-expander) is dependency-light, aligns with the stdlib-first preference, and makes the "write resolved config beside results" contract trivial. *Decision: thin pydantic loader.*

2. **Selection fitness `f_i` (the reality-anchored score).** Blueprint: "fitness is predictive accuracy against `p*`" but gives no formula for the discrete model. *Decision:* `f_i = p*_eff_i` (truth frequency = fitness; reality-anchored by construction). Post-selection distribution `p'_i ∝ p_i^(1−α) · f_i`, matching the blueprint's stated `w_i ∝ f_i·(p_i)^(−α)` with `α=0` recovering fitness-proportional greedy. Document as a modelling choice; `select-then-sample` vs `sample-then-select` is the documented robustness switch (§2.2).

3. **`tail_mask` definition.** Two knobs exist (`tail_frac`, `tail_threshold`). *Decision:* the metric-bearing tail set is the §2.3 definition `{i : p*_i < tail_threshold}`. `tail_frac` only drives the `twocomponent` construction (fraction of items placed in the low-mass component). Document that for `zipf`, `tail_frac` is unused.

4. **`TrueDist` type.** *Decision:* a frozen dataclass `TrueDist(p_star: np.ndarray, regions: np.ndarray, tail_mask: np.ndarray)` — immutable, so `p*` cannot be mutated in place (except the deliberate re-mint path, which produces a new object).

5. **Region partition.** *Decision:* contiguous blocks; require `K % R == 0` (assert with a clear error) to keep per-region math clean for v1.

6. **Re-mint semantics (E6).** *Decision:* re-mint replaces `p_star_eff` with a *new* `TrueDist` built from current `p_t` (fresh tail_mask recomputed on `p_t`), and the *original* `TrueDist` is retained **only** for the KL-to-original metric, never for grounding. This is the irreversibility the experiment tests.

---

## Phase 0 — Scaffold & environment

- [x] Install `uv` (`curl -LsSf https://astral.sh/uv/install.sh | sh`; lands in `~/.local/bin`, no sudo).
- [x] **Reproducibility path = `uv` venv from a committed, hash-pinned `uv.lock`** (GG decision, 2026-07-04; no Apptainer/Docker for Layer 1). `pyproject.toml` (Python ≥3.11; deps: numpy, scipy, pandas, pyarrow, matplotlib, pydantic, pyyaml; dev: pytest). Commit `uv.lock`. Create `.venv` via `uv sync`.
- [x] Repo layout per §5: `src/knowledge/`, `configs/layer1/`, `figures/`, `results/` (gitignored), `tests/`, `paper/`. Add `src/lamarckian/` package root or make `knowledge` importable (decide package name — recommend `src/lamarckian/knowledge/...` with `src`-layout).
- [x] `.gitignore` (`.venv/`, `results/`, `__pycache__/`, `*.parquet` under results but keep hashes).
- [x] Seeding util `lamarckian/utils/seeding.py`: master seed → `np.random.SeedSequence(seed).spawn(n)` → per-replicate `np.random.default_rng(child)`. No global RNG anywhere.
- [x] Config loader `lamarckian/config.py`: pydantic schema mirroring the §2.7 YAML, a `load_config(path)`, a `expand_sweeps(cfg) -> list[ResolvedConfig]`, and `write_resolved(cfg, dir)`.
- [x] Manifest util: `write_manifest(dir, results_df)` recording lib versions, master seed, `git rev-parse HEAD`, content hash of `results.parquet`.
- [x] `Makefile` targets (`env`, `test`, `layer1`, `figures`, `clean`) + `pytest` skeleton. **Gate: `make test` green on a trivial test.**
- [x] Move `lamarckian-society-technical-blueprint-v1.md` → `paper/blueprint.md` per §5 (confirm with GG first — it's referenced by name elsewhere).

## Phase 1 — Core primitives + null model + VALIDATION GATE

Implement to the normative signatures in §2.7. Order chosen so each piece is unit-testable before the next depends on it.

- [x] `knowledge/truth.py::make_true_distribution` → `TrueDist`. Support `tail ∈ {zipf, twocomponent}`. Unit tests: normalisation, region block sizes, tail_mask matches threshold, determinism from seed.
- [x] `knowledge/metrics.py`: `forward_kl` (with `eps` floor, logged), `heterozygosity`, `tail_mass`, `support_size`, plus per-region variants. Unit tests on hand-computed small vectors.
- [x] `knowledge/step.py::generation_step` — **null path first** (single teacher, `m=0`, selection `none`): `c ~ Multinomial(n, p_t)`, `p_{t+1}=c/n`. Exactly neutral Wright–Fisher.
- [x] `knowledge/lineage.py::run_lineage(cfg, seed)` → tidy per-generation DataFrame (all §2.3 metrics, global + per-region).
- [x] `tests/test_scientific_validation.py` — **the spine:**
  - **Pred. 1** heterozygosity decay: mean `H_t` over replicates matches `H₀(1−1/n)^t`. Prefer testing the full deterministic mean-recursion trajectory (subsumes the fixed point), within Monte-Carlo CI.
  - **Pred. 2** fixation probability = initial frequency (long runs, statistical tolerance).
- [x] **HARD GATE: do not proceed until Pred. 1–2 pass.** If drift ≠ analytic decay, the harness is wrong — fix here.

## Phase 2 — Grounding + E1 + E2 (the headline)

- [x] `knowledge/step.py::structured_multinomial(m_vector, p_star, regions, policy, rng)` — per-region immigration draws from `p*` restricted+renormalised to each region; `uniform` spreads `m` evenly, `matched` concentrates on exercised regions. Returns length-K counts.
- [x] Extend `generation_step` with grounding (pooled draw, `g = m/(n+m)`).
- [x] **Pred. 3** validation — *exact* equilibrium `H_eq = H*·m(2n+m−1)/(n+2nm+m²)`: run to stationarity (burn-in + late-generation + replicate averaging), assert `<0.1%` rel. error vs closed form across an `m` grid. Also assert the `m→0` and `m→∞` limits.
- [x] **Pred. 4** validation — tail-persistence: item of freq `p*_i` maintained iff `m·p*_i ≳ 1`; verify the survival transition location statistically.
- [x] `knowledge/experiment.py::run_experiment(cfg)` — sweep grid × `n_replicates`; long-form results + CIs; write `results.parquet` + `resolved_config.yaml` + `manifest.json`.
- [x] **E1** config + run: `m=0`, single teacher, no selection. Expect `H` geometric decay, support→1, KL diverges, tail-first loss.
- [x] **E2** config + run: sweep `g`, single teacher, uniform grounding, no selection. Locate critical `g*` (transition in **tail mass / support**, since H is smooth in m — the sharp threshold is in discrete tail survival). Report `g*` with CI. **This is the load-bearing result.**
- [x] `figures/plot_E1.py`, `plot_E2.py` — read `results.parquet` only.

## Phase 3 — E3–E6

- [x] **E3 region-matched grounding.** Fixed total `m`; `uniform` vs `matched`; one designated inherited-but-unwatered region with a rare tail. Expect uniform lets that region's tail collapse; matched holds it. Per-region metrics essential. `plot_E3.py`.
- [x] **§2.7.1 correlated-teacher construction** — `knowledge/teachers.py`:
  - `make_retention_matrix(T, K_T, rho, q, rng)` — shared-switch exchangeable Bernoulli.
  - `make_correlated_teachers(...)` — retention→distributions (head kept at `p*`; tail at `p*_i` if retained else `tail_floor`; renormalise). `region_specialisation` option.
  - **Pred. 5** validation: `make_retention_matrix` reproduces marginal `q`, pairwise `ρ`, and union coverage `U(K_T,ρ,q)=T[ρq+(1−ρ)(1−(1−q)^K_T)]` to 3 decimals over a `(ρ,q)` grid.
- [x] **E4 multi-teacher decorrelation.** Sweep `K_T∈{1,2,3,5}`, `ρ∈[0,1]` at fixed `q`, matched budget (`n/K_T` each). Report **both** union `U` and post-distillation surviving coverage; show their gap shrinks as `g` rises. `plot_E4.py` (coverage surface over `(K_T,ρ)`).
- [x] **E5 QD vs greedy.** `apply_selection` (`none`/`greedy`/`qd`, pinned fitness form). Sweep novelty `α`. Expect greedy→fixation (`H→0`), qd holds `H` plateau + re-introduces tails. `plot_E5.py`.
- [x] **E6 re-mint gate.** Re-mint at high vs low `H`; track KL to *original* truth. Expect collapsed re-mint locks KL high forever; gated (high-H) does not. `plot_E6.py`.

## Phase 4 — Reproducibility polish (Layer 1 slice)

- [x] `configs/layer1/E1..E6.yaml` all committed with explicit params (no magic numbers in code).
- [x] `paper/figure_manifest.md` — the §6 claim→experiment→figure rows for Layer 1.
- [x] `make layer1` runs E1–E6; `make figures` regenerates all figures from committed parquet.
- [x] Full `test_correctness.py` (shapes, normalisation, determinism) + `test_scientific_validation.py` (Pred. 1–5) green in CI.
- [x] `reproduce.sh` (`uv sync` → `make test` → `make layer1` → `make figures` → write `REPRODUCED.md` diffing committed result hashes) + `README.md` reproduce section. **No container** — the committed `uv.lock` is the reproducibility source of truth (per GG, 2026-07-04); a Dockerfile may later wrap the same lockfile for Layer 2's GPU work.

---

## Definition of done (Layer 1)

Every Layer-1 row of blueprint §6 has a committed figure produced by `make figures` from committed results; all §2.4 analytic checks (Pred. 1–5) pass; `make layer1 && make figures` reproduces from a clean `.venv`. Then — and only then — Layer 2 may begin.

## Falsifier watch (report honestly if hit)

- E2 tail mass flat in `g`, or only stabilises as `g→1` → multigenerational thesis refuted.
- E3 uniform protects as well as matched → region-matching claim dies.
- E4 no surviving-coverage benefit at matched budget → recombination claim dies.
- E5 qd ≤ greedy stationary `H` → QD does no work.
- E6 collapsed lineage recovers original-truth tails after re-mint → irreversibility overstated.

## Review — progress log

**2026-07-04 — Phases 0 & 1 complete; hard gate PASSED.**

- Reorg: docs → `paper/` (`blueprint.md`, `the-lamarckian-society-v4.md`). src-layout under `src/knowledge/`.
- A pre-existing `tests/test_scientific_validation.py` (author-supplied, 22 KB) turned out to hard-specify the package contract — implemented *to it* rather than inventing interfaces. Key contracts it locked (now honoured): package imports as `knowledge.*`; `run_lineage(cfg_dict, seed)` returns a tidy per-gen frame with a `heterozygosity` column, rows 0..T; `p_0` initialises **uniform** (`H_0=1−1/K`); `metrics.heterozygosity` and `teachers.make_retention_matrix` match the reference to 1e-12 / closed form.
- Env: `uv` 0.11.26 installed; `pyproject.toml` + `uv.lock` committed; numpy 2.5, pandas 3.0, scipy 1.18, pydantic 2.13, pytest 9.1.
- Modules written: `metrics`, `seeding`, `config` (dataclasses + `from_dict`), `truth`, `teachers`, `step`, `lineage`. Config is dataclass-based (not pydantic) — the conformance test passes a raw dict; dataclasses validate cleanly and stay stdlib-simple. **Pydantic still a dep for the Phase-2 YAML/experiment layer.**
- **Results: `make test` green — 68 passed** (48 scientific-validation, 20 correctness). Conformance tests RAN (not skipped): Pred. 1 (neutral decay), Pred. 3 (exact `H_eq`), Pred. 5 (union coverage) all pass against the real package. **The Pred. 1–2 hard gate is passed**, and grounding already conforms to the exact equilibrium.

Design decisions #1 (dataclasses now / pydantic at YAML layer), #2 (fitness `f_i=p*_i`), #3 (threshold tail_mask), #6 (re-mint discard) all implemented as planned. Region design: each region an identical 1/R-mass block (symmetric; reduces to global Zipf at R=1, matching the reference).

**2026-07-04 — Phase 2 complete (E1 + E2).**

- `experiment.py`: sweep expansion (Cartesian grid; special-cases `g→m`), paired replicate seeds (shared across grid points), output contract (`results.parquet` + `resolved_config.yaml` + `manifest.json` with lib versions + git commit + sha256). CLI `python -m knowledge.experiment <cfg>`.
- **E1 (null collapse)** — reproduces tail-first collapse: H geometric decay matches `H₀(1−1/n)ᵗ` within CI; tail items die ~10× faster than head items; support 500→1; forward-KL diverges. Figure `results/E1/E1.png`.
- **E2 (headline)** — `H_sim` tracks the *exact* `H_eq` closed form across the sweep; phase boundary at **`g* ≪ 1`**: g=0.005 (m=1 real sample vs n=200) → 68% of truth H; g=0.05 → 96%. g=0 slides to ~0.10 over 500 gens. Figure `results/E2/E2.png`. **Headline result achieved.**
- Metric subtlety found & fixed: aggregate **`tail_mass` is a drift martingale** (mean-conserved), so it's a poor collapse indicator. Added `tail_support`/`head_support`/`tail_frac_alive`/`head_frac_alive`; E1 & E2 figures now use tail-*item* survival, which is honest and monotone.
- E2 extended 300→500 generations (GG-approved) so the g=0 arm visibly approaches 0 while g>0 arms sit on plateaus.
- Makefile `layer1`/`figures` wired to E1–E2. `make test` still green (68).

**2026-07-04 — Phase 3 complete (E3–E6) + E2 analysis add-ons.**

- **E3** region-matched grounding: added `grounding.exercised` knob + per-region `tailalive_region_r`. Target region tail survival 0.49 (matched) vs 0.07 (uniform). Note: per-region *H* is mass-confounded — used tail-item survival instead.
- **E4** multi-teacher recombination: bespoke `run_coverage` runner (`kind: coverage`). Union coverage matches `U(K_T,ρ,q)` exactly. **Key finding (GG-approved): mean-mixture distillation gives NO surviving benefit (conservation law — dilution cancels the union gain); max-merge (M2N2-style) does.** E4 reports both. In CLAUDE.md.
- **E5** QD vs greedy: greedy → H≈0.01 (fixation); qd holds H 0.48–0.88 rising with α. qd ≫ greedy.
- **E6** re-mint gate: added `arm` multi-override sweep type. Re-mint while collapsed → KL-to-original diverges (lock-in) + accelerates H collapse; diversity gate (H≥0.75) blocks it → bounded; healthy re-mint harmless.
- **E2 analysis add-ons** (companion work order `tasks/workorder-E2-analysis-addons.md`, verified): new `analysis.py` (`reduce_to_stationary`, `critical_grounding` bootstrap CI) — real E2 **g*=0.048, CI [0.047,0.050]**; `metrics.tail_band_metrics` + per-band lineage logging; `tests/test_analysis.py` reproduces the work order's verified numbers exactly. E2 figure rebuilt 2×2. **Deviation:** used truth-mass-weighted tail coverage instead of raw `tail_mass` (a drift martingale).
- All six figures regenerate via `make figures`; **71 tests green**.

---

# Layer 1.5 — Architecture-general neural existence proof (RNN/VAE/MLP + synthetic/MNIST)

*Created 2026-07-04. Plan: `~/.claude/plans/we-are-going-to-cheerful-fog.md`. Re-scopes Layer 2:
build a cheap, architecture-general neural collapse proof in real trained weights on a
fully-synthetic sandbox (exact known `p*`) before the LLM rung. Locked decisions: exact-oracle
categorical token sequences; Histogram+RNN+VAE+MLP; real MNIST as secondary confirmation; LLM +
C3 vertical claim deferred.*

## Progress log

**2026-07-04 — Stages A, B, plumbing complete.**

- **Env:** installed `uv` 0.11.26 (`~/.local/bin`); `/home` was 100% full — GG approved clearing
  pip/yay/browser caches (~10 GB freed). Base venv synced; 71 Layer-1 tests green.
- **Stage A (scaffold, pure NumPy):** `src/neural/` — `config.py` (frozen dataclasses reusing
  `knowledge.config` GroundingCfg/RemintCfg/MetricsCfg/_sub), `synthetic.py` (mode-truth via
  `make_true_distribution`; lossless identity + stochastic style token grammar), `oracle.py`
  (`ExactOracle` zero-error + `measure_distribution`), `models.py` (`GenerativeModel` protocol +
  `HistogramModel` bridge), `evaluate.py` (reuses `knowledge.metrics`, Layer-1 row schema),
  `generation_loop.py` (`run_generative_lineage`, reuses `allocate_m`/`structured_multinomial`).
  15 correctness tests green.
- **Stage B — HARD GATE PASSED:** `tests/test_neural_validation.py` — histogram lineage reproduces
  Pred. 1 (neutral decay, <3% rel err), Pred. 3 (exact `H_eq`, <5%), and tracks Layer-1
  `run_lineage` directly (<3%). The neural plumbing reproduces the analytic core.
- **Plumbing:** `neural/experiment.py` (`run_and_save` dispatch on `kind`, reuses `_apply_param`
  g→m, paired seeds); extended `knowledge.experiment.save_artifacts` (optional `extra_libs`,
  `extra_manifest`, injectable `grid`; skips missing libs — backward compatible). `configs/neural/N0.yaml`,
  Makefile `neural`/`env-neural`/`layer2` targets, `.gitignore`.
- **N0 result (bridge, 17s):** neural **g\* = 0.0474, CI [0.045, 0.052]** — reproduces Layer-1 E2's
  g\*=0.048 essentially exactly (g=0.005→67% of H*, g=0.05→96%). **89 tests green.**

**2026-07-04 — Stage C: torch models + N1/N2/N5.**

- **Env:** torch **2.12.1+cu130** (default PyPI wheel ships CUDA 13, matches RTX A4000 driver;
  no custom index needed, cp314 wheels exist). `--extra neural` = torch only; `--extra mnist` =
  torchvision (later). `UV_CACHE_DIR=/tmp` during install (RAM-backed) to spare `/home`.
- **Models:** `torch_models.py` (RNNGenerator, autoregressive GRU), `torch_mlp.py` (autoregressive
  MLP, causal-masked), `torch_vae.py` (sequence VAE), `train.py` (determinism flags + device/seed
  helpers derived from the SeedSequence stream). `tests/test_neural_torch.py` (torch-gated): gen-0
  fidelity (rnn+mlp) + dry-collapse/grounded-holds. **92 tests green.**
- **Validated regime:** K=256, n=200, zipf_s=1.3, RNN hidden=128/epochs=25. RNN gen-0 fidelity
  KL(p*‖p̂)=0.008, 64/64 (or 256/256) modes recovered. MLP fidelity KL=0.011. **VAE does NOT clear
  the gen-0 gate** on the Zipf-codeword task (KL≈0.8; prior-hole mismatch — sampling z~N(0,I) misses
  the aggregate posterior) → excluded from N5 to avoid confounding collapse with underfitting.
- **N1 (collapse in weights):** dry RNN lineage collapses — forward-KL rises to ~2.2 vs grounded
  ~1.4; grounding lifts tail survival (tailalive 0.31 dry → 0.50 at g=0.02). Sign confirmed.
- **N2 (neural phase boundary):** stationary H hovers 80–91% of H* and is **noisy / non-monotonic**
  at 5 reps — no crisp g*. **KEY FINDING:** the neural models' smoothing inductive bias *partially
  resists* H-collapse (dry H stays ~83% of H*), so **forward-KL and tail survival are the sharp
  neural collapse metrics, not H** (mirrors Layer-1's "H is smooth; the threshold lives in tail
  survival"). N2 needs (a) forward-KL as the phase metric, (b) more reps (≥10), and/or (c) a
  stronger-collapse regime for a clean neural g*.
- **N5 (architecture-generality) — clean result:** collapse + grounding-rescue appear in ALL three
  model classes (dry→grounded forward-KL: histogram 6.2→4.6, MLP 4.8→1.3, RNN 3.8→1.1; tailalive
  RNN 0.41→0.64, MLP 0.07→0.20). The WF operator is architecture-general. Bonus: neural smoothing
  lets RNN/MLP retain *more* tail than the exact histogram under grounding (they generalise to
  unseen codewords) — an inductive-bias finding worth the write-up.

## Remaining

- [ ] **N2 refinement:** re-run with forward-KL as the phase metric + ≥10 reps (and/or smaller n)
  for a clean neural g*. Pin the falsifier ("g* ≪ 1 exists") before re-running.
- [ ] **N4 (load-bearing):** `recombine.py` — mean-mixture vs union-preserving merge. The neural
  merge MUST be **oracle-guided mixture sampling** (sample from the teacher strongest on each mode),
  NOT weight-averaging of recurrent nets (flag #7). Reproduce the E4 "mean flat, max rises" finding.
- [ ] **N3** region-matched grounding (R>1), **N6** re-mint gate (optional).
- [ ] **VAE fidelity:** fix the prior-hole mismatch (KL-annealing / free-bits / larger latent) so it
  clears the gen-0 gate, then add to N5. Or document as a known limitation.
- [ ] Real-MNIST secondary tier (`ClassifierOracle` + confusion matrix; `--extra mnist`).
- [ ] `figures/plot_N*.py` (reuse `figures/_figlib.py`); wire into `make figures`.

## Discovered during work

- **E2 grounding policy vs. the analytic H_eq:** Pred. 3's closed form is derived for *plain* immigration `Multinomial(m, p*)`. Implemented as `policy="proportional"`, and every policy reduces to it at `R=1`. E2 should therefore run at `R=1` (or `proportional`) so the phase-boundary sweep tracks the exact `H_eq`; region structure is E3's concern. Decide E2's `init` (uniform vs truth) when building its config.
- `init: {uniform|truth}` added to `TruthCfg` (uniform default, mandated by the decay conformance test). E1/E2 may want `truth` start for a clean "tail collapses from the truth" story — revisit in Phase 2.

## Potential agents

*(none proposed yet)*