MachineSex/tasks/todo.md

Layer 1 Execution Plan — The Lamarckian Society (analytical core)

Created 2026-07-04. Scope: blueprint §7 build-order steps 1–4 (scaffold → Layer 1 complete, laptop-reproducible). Layer 2 is explicitly out of scope here and gated on Layer 1's scientific-validation tests passing.

Source of truth: lamarckian-society-technical-blueprint-v1.md. Where it is silent I record a decision below rather than improvise silently.

---

Design decisions to pin before coding (blueprint leaves these open)

These are the only places the spec is genuinely underdetermined. Recommendation given for each; flagged for sign-off.

1. Config framework — recommend thin pydantic + PyYAML, not Hydra. Blueprint says "Hydra or a thin equivalent." Hydra is heavyweight and its global-state/os.chdir behaviour fights the "pass rng explicitly, results are a pure function of resolved config" requirement. A thin loader (pydantic models for validation + a small sweep-expander) is dependency-light, aligns with the stdlib-first preference, and makes the "write resolved config beside results" contract trivial. Decision: thin pydantic loader.

2. Selection fitness f_i (the reality-anchored score). Blueprint: "fitness is predictive accuracy against p*" but gives no formula for the discrete model. Decision: f_i = p*_eff_i (truth frequency = fitness; reality-anchored by construction). Post-selection distribution p'_i ∝ p_i^(1−α) · f_i, matching the blueprint's stated w_i ∝ f_i·(p_i)^(−α) with α=0 recovering fitness-proportional greedy. Document as a modelling choice; select-then-sample vs sample-then-select is the documented robustness switch (§2.2).

3. tail_mask definition. Two knobs exist (tail_frac, tail_threshold). Decision: the metric-bearing tail set is the §2.3 definition {i : p*_i < tail_threshold}. tail_frac only drives the twocomponent construction (fraction of items placed in the low-mass component). Document that for zipf, tail_frac is unused.

4. TrueDist type. Decision: a frozen dataclass TrueDist(p_star: np.ndarray, regions: np.ndarray, tail_mask: np.ndarray) — immutable, so p* cannot be mutated in place (except the deliberate re-mint path, which produces a new object).

5. Region partition. Decision: contiguous blocks; require K % R == 0 (assert with a clear error) to keep per-region math clean for v1.

6. Re-mint semantics (E6). Decision: re-mint replaces p_star_eff with a new TrueDist built from current p_t (fresh tail_mask recomputed on p_t), and the original TrueDist is retained only for the KL-to-original metric, never for grounding. This is the irreversibility the experiment tests.

---

Phase 0 — Scaffold & environment

• Install uv (curl -LsSf https://astral.sh/uv/install.sh | sh; lands in ~/.local/bin, no sudo).
• Reproducibility path = uv venv from a committed, hash-pinned uv.lock (GG decision, 2026-07-04; no Apptainer/Docker for Layer 1). pyproject.toml (Python ≥3.11; deps: numpy, scipy, pandas, pyarrow, matplotlib, pydantic, pyyaml; dev: pytest). Commit uv.lock. Create .venv via uv sync.
• Repo layout per §5: src/knowledge/, configs/layer1/, figures/, results/ (gitignored), tests/, paper/. Add src/lamarckian/ package root or make knowledge importable (decide package name — recommend src/lamarckian/knowledge/... with src-layout).
• .gitignore (.venv/, results/, __pycache__/, *.parquet under results but keep hashes).
• Seeding util lamarckian/utils/seeding.py: master seed → np.random.SeedSequence(seed).spawn(n) → per-replicate np.random.default_rng(child). No global RNG anywhere.
• Config loader lamarckian/config.py: pydantic schema mirroring the §2.7 YAML, a load_config(path), a expand_sweeps(cfg) -> list[ResolvedConfig], and write_resolved(cfg, dir).
• Manifest util: write_manifest(dir, results_df) recording lib versions, master seed, git rev-parse HEAD, content hash of results.parquet.
• Makefile targets (env, test, layer1, figures, clean) + pytest skeleton. Gate: make test green on a trivial test.
• Move lamarckian-society-technical-blueprint-v1.md → paper/blueprint.md per §5 (confirm with GG first — it's referenced by name elsewhere).

Phase 1 — Core primitives + null model + VALIDATION GATE

Implement to the normative signatures in §2.7. Order chosen so each piece is unit-testable before the next depends on it.

• knowledge/truth.py::make_true_distribution → TrueDist. Support tail ∈ {zipf, twocomponent}. Unit tests: normalisation, region block sizes, tail_mask matches threshold, determinism from seed.
• knowledge/metrics.py: forward_kl (with eps floor, logged), heterozygosity, tail_mass, support_size, plus per-region variants. Unit tests on hand-computed small vectors.
• knowledge/step.py::generation_step — null path first (single teacher, m=0, selection none): c ~ Multinomial(n, p_t), p_{t+1}=c/n. Exactly neutral Wright–Fisher.
• knowledge/lineage.py::run_lineage(cfg, seed) → tidy per-generation DataFrame (all §2.3 metrics, global + per-region).
• tests/test_scientific_validation.py — the spine:
  • Pred. 1 heterozygosity decay: mean H_t over replicates matches H₀(1−1/n)^t. Prefer testing the full deterministic mean-recursion trajectory (subsumes the fixed point), within Monte-Carlo CI.
  • Pred. 2 fixation probability = initial frequency (long runs, statistical tolerance).
• HARD GATE: do not proceed until Pred. 1–2 pass. If drift ≠ analytic decay, the harness is wrong — fix here.

Phase 2 — Grounding + E1 + E2 (the headline)

• knowledge/step.py::structured_multinomial(m_vector, p_star, regions, policy, rng) — per-region immigration draws from p* restricted+renormalised to each region; uniform spreads m evenly, matched concentrates on exercised regions. Returns length-K counts.
• Extend generation_step with grounding (pooled draw, g = m/(n+m)).
• Pred. 3 validation — exact equilibrium H_eq = H*·m(2n+m−1)/(n+2nm+m²): run to stationarity (burn-in + late-generation + replicate averaging), assert <0.1% rel. error vs closed form across an m grid. Also assert the m→0 and m→∞ limits.
• Pred. 4 validation — tail-persistence: item of freq p*_i maintained iff m·p*_i ≳ 1; verify the survival transition location statistically.
• knowledge/experiment.py::run_experiment(cfg) — sweep grid × n_replicates; long-form results + CIs; write results.parquet + resolved_config.yaml + manifest.json.
• E1 config + run: m=0, single teacher, no selection. Expect H geometric decay, support→1, KL diverges, tail-first loss.
• E2 config + run: sweep g, single teacher, uniform grounding, no selection. Locate critical g* (transition in tail mass / support, since H is smooth in m — the sharp threshold is in discrete tail survival). Report g* with CI. This is the load-bearing result.
• figures/plot_E1.py, plot_E2.py — read results.parquet only.

Phase 3 — E3–E6

• E3 region-matched grounding. Fixed total m; uniform vs matched; one designated inherited-but-unwatered region with a rare tail. Expect uniform lets that region's tail collapse; matched holds it. Per-region metrics essential. plot_E3.py.
• §2.7.1 correlated-teacher construction — knowledge/teachers.py:
  • make_retention_matrix(T, K_T, rho, q, rng) — shared-switch exchangeable Bernoulli.
  • make_correlated_teachers(...) — retention→distributions (head kept at p*; tail at p*_i if retained else tail_floor; renormalise). region_specialisation option.
  • Pred. 5 validation: make_retention_matrix reproduces marginal q, pairwise ρ, and union coverage U(K_T,ρ,q)=T[ρq+(1−ρ)(1−(1−q)^K_T)] to 3 decimals over a (ρ,q) grid.
• E4 multi-teacher decorrelation. Sweep K_T∈{1,2,3,5}, ρ∈[0,1] at fixed q, matched budget (n/K_T each). Report both union U and post-distillation surviving coverage; show their gap shrinks as g rises. plot_E4.py (coverage surface over (K_T,ρ)).
• E5 QD vs greedy. apply_selection (none/greedy/qd, pinned fitness form). Sweep novelty α. Expect greedy→fixation (H→0), qd holds H plateau + re-introduces tails. plot_E5.py.
• E6 re-mint gate. Re-mint at high vs low H; track KL to original truth. Expect collapsed re-mint locks KL high forever; gated (high-H) does not. plot_E6.py.

Phase 4 — Reproducibility polish (Layer 1 slice)

• configs/layer1/E1..E6.yaml all committed with explicit params (no magic numbers in code).
• paper/figure_manifest.md — the §6 claim→experiment→figure rows for Layer 1.
• make layer1 runs E1–E6; make figures regenerates all figures from committed parquet.
• Full test_correctness.py (shapes, normalisation, determinism) + test_scientific_validation.py (Pred. 1–5) green in CI.
• reproduce.sh (uv sync → make test → make layer1 → make figures → write REPRODUCED.md diffing committed result hashes) + README.md reproduce section. No container — the committed uv.lock is the reproducibility source of truth (per GG, 2026-07-04); a Dockerfile may later wrap the same lockfile for Layer 2's GPU work.

---

Definition of done (Layer 1)

Every Layer-1 row of blueprint §6 has a committed figure produced by make figures from committed results; all §2.4 analytic checks (Pred. 1–5) pass; make layer1 && make figures reproduces from a clean .venv. Then — and only then — Layer 2 may begin.

Falsifier watch (report honestly if hit)

• E2 tail mass flat in g, or only stabilises as g→1 → multigenerational thesis refuted.
• E3 uniform protects as well as matched → region-matching claim dies.
• E4 no surviving-coverage benefit at matched budget → recombination claim dies.
• E5 qd ≤ greedy stationary H → QD does no work.
• E6 collapsed lineage recovers original-truth tails after re-mint → irreversibility overstated.

Review — progress log

2026-07-04 — Phases 0 & 1 complete; hard gate PASSED.

• Reorg: docs → paper/ (blueprint.md, the-lamarckian-society-v4.md). src-layout under src/knowledge/.
• A pre-existing tests/test_scientific_validation.py (author-supplied, 22 KB) turned out to hard-specify the package contract — implemented to it rather than inventing interfaces. Key contracts it locked (now honoured): package imports as knowledge.*; run_lineage(cfg_dict, seed) returns a tidy per-gen frame with a heterozygosity column, rows 0..T; p_0 initialises uniform (H_0=1−1/K); metrics.heterozygosity and teachers.make_retention_matrix match the reference to 1e-12 / closed form.
• Env: uv 0.11.26 installed; pyproject.toml + uv.lock committed; numpy 2.5, pandas 3.0, scipy 1.18, pydantic 2.13, pytest 9.1.
• Modules written: metrics, seeding, config (dataclasses + from_dict), truth, teachers, step, lineage. Config is dataclass-based (not pydantic) — the conformance test passes a raw dict; dataclasses validate cleanly and stay stdlib-simple. Pydantic still a dep for the Phase-2 YAML/experiment layer.
• Results: make test green — 68 passed (48 scientific-validation, 20 correctness). Conformance tests RAN (not skipped): Pred. 1 (neutral decay), Pred. 3 (exact H_eq), Pred. 5 (union coverage) all pass against the real package. The Pred. 1–2 hard gate is passed, and grounding already conforms to the exact equilibrium.

Design decisions #1 (dataclasses now / pydantic at YAML layer), #2 (fitness f_i=p*_i), #3 (threshold tail_mask), #6 (re-mint discard) all implemented as planned. Region design: each region an identical 1/R-mass block (symmetric; reduces to global Zipf at R=1, matching the reference).

2026-07-04 — Phase 2 complete (E1 + E2).

• experiment.py: sweep expansion (Cartesian grid; special-cases g→m), paired replicate seeds (shared across grid points), output contract (results.parquet + resolved_config.yaml + manifest.json with lib versions + git commit + sha256). CLI python -m knowledge.experiment <cfg>.
• E1 (null collapse) — reproduces tail-first collapse: H geometric decay matches H₀(1−1/n)ᵗ within CI; tail items die ~10× faster than head items; support 500→1; forward-KL diverges. Figure results/E1/E1.png.
• E2 (headline) — H_sim tracks the exact H_eq closed form across the sweep; phase boundary at g* ≪ 1: g=0.005 (m=1 real sample vs n=200) → 68% of truth H; g=0.05 → 96%. g=0 slides to ~0.10 over 500 gens. Figure results/E2/E2.png. Headline result achieved.
• Metric subtlety found & fixed: aggregate tail_mass is a drift martingale (mean-conserved), so it's a poor collapse indicator. Added tail_support/head_support/tail_frac_alive/head_frac_alive; E1 & E2 figures now use tail-item survival, which is honest and monotone.
• E2 extended 300→500 generations (GG-approved) so the g=0 arm visibly approaches 0 while g>0 arms sit on plateaus.
• Makefile layer1/figures wired to E1–E2. make test still green (68).

2026-07-04 — Phase 3 complete (E3–E6) + E2 analysis add-ons.

• E3 region-matched grounding: added grounding.exercised knob + per-region tailalive_region_r. Target region tail survival 0.49 (matched) vs 0.07 (uniform). Note: per-region H is mass-confounded — used tail-item survival instead.
• E4 multi-teacher recombination: bespoke run_coverage runner (kind: coverage). Union coverage matches U(K_T,ρ,q) exactly. Key finding (GG-approved): mean-mixture distillation gives NO surviving benefit (conservation law — dilution cancels the union gain); max-merge (M2N2-style) does. E4 reports both. In CLAUDE.md.
• E5 QD vs greedy: greedy → H≈0.01 (fixation); qd holds H 0.48–0.88 rising with α. qd ≫ greedy.
• E6 re-mint gate: added arm multi-override sweep type. Re-mint while collapsed → KL-to-original diverges (lock-in) + accelerates H collapse; diversity gate (H≥0.75) blocks it → bounded; healthy re-mint harmless.
• E2 analysis add-ons (companion work order tasks/workorder-E2-analysis-addons.md, verified): new analysis.py (reduce_to_stationary, critical_grounding bootstrap CI) — real E2 g=0.048, CI [0.047,0.050]*; metrics.tail_band_metrics + per-band lineage logging; tests/test_analysis.py reproduces the work order's verified numbers exactly. E2 figure rebuilt 2×2. Deviation: used truth-mass-weighted tail coverage instead of raw tail_mass (a drift martingale).
• All six figures regenerate via make figures; 71 tests green.

---

Layer 1.5 — Architecture-general neural existence proof (RNN/VAE/MLP + synthetic/MNIST)

Created 2026-07-04. Plan: ~/.claude/plans/we-are-going-to-cheerful-fog.md. Re-scopes Layer 2: build a cheap, architecture-general neural collapse proof in real trained weights on a fully-synthetic sandbox (exact known p*) before the LLM rung. Locked decisions: exact-oracle categorical token sequences; Histogram+RNN+VAE+MLP; real MNIST as secondary confirmation; LLM + C3 vertical claim deferred.

Progress log

2026-07-04 — Stages A, B, plumbing complete.

• Env: installed uv 0.11.26 (~/.local/bin); /home was 100% full — GG approved clearing pip/yay/browser caches (~10 GB freed). Base venv synced; 71 Layer-1 tests green.
• Stage A (scaffold, pure NumPy): src/neural/ — config.py (frozen dataclasses reusing knowledge.config GroundingCfg/RemintCfg/MetricsCfg/_sub), synthetic.py (mode-truth via make_true_distribution; lossless identity + stochastic style token grammar), oracle.py (ExactOracle zero-error + measure_distribution), models.py (GenerativeModel protocol + HistogramModel bridge), evaluate.py (reuses knowledge.metrics, Layer-1 row schema), generation_loop.py (run_generative_lineage, reuses allocate_m/structured_multinomial). 15 correctness tests green.
• Stage B — HARD GATE PASSED: tests/test_neural_validation.py — histogram lineage reproduces Pred. 1 (neutral decay, <3% rel err), Pred. 3 (exact H_eq, <5%), and tracks Layer-1 run_lineage directly (<3%). The neural plumbing reproduces the analytic core.
• Plumbing: neural/experiment.py (run_and_save dispatch on kind, reuses _apply_param g→m, paired seeds); extended knowledge.experiment.save_artifacts (optional extra_libs, extra_manifest, injectable grid; skips missing libs — backward compatible). configs/neural/bridge.yaml, Makefile neural/env-neural/layer2 targets, .gitignore. (Experiments are named descriptively — bridge, collapse, grounding, architectures — not by code.)
• bridge result (17s): neural g* = 0.0474, CI [0.045, 0.052] — reproduces Layer-1 E2's g*=0.048 essentially exactly (g=0.005→67% of H*, g=0.05→96%). 89 tests green.

2026-07-04 — Stage C: torch models + collapse/grounding/architectures.

• Env: torch 2.12.1+cu130 (default PyPI wheel ships CUDA 13, matches RTX A4000 driver; no custom index needed, cp314 wheels exist). --extra neural = torch only; --extra mnist = torchvision (later). UV_CACHE_DIR=/tmp during install (RAM-backed) to spare /home.
• Models: torch_models.py (RNNGenerator, autoregressive GRU), torch_mlp.py (autoregressive MLP, causal-masked), torch_vae.py (sequence VAE), train.py (determinism flags + device/seed helpers derived from the SeedSequence stream). tests/test_neural_torch.py (torch-gated): gen-0 fidelity (rnn+mlp) + dry-collapse/grounded-holds. 92 tests green.
• Validated regime: K=256, n=200, zipf_s=1.3, RNN hidden=128/epochs=25. RNN gen-0 fidelity KL(p*‖p̂)=0.008, 64/64 (or 256/256) modes recovered. MLP fidelity KL=0.011. VAE does NOT clear the gen-0 gate on the Zipf-codeword task (KL≈0.8; prior-hole mismatch — sampling z~N(0,I) misses the aggregate posterior) → excluded from architectures to avoid confounding collapse with underfitting.
• collapse (in weights): dry RNN lineage collapses — forward-KL rises to ~2.2 vs grounded ~1.4; grounding lifts tail survival (tailalive 0.31 dry → 0.50 at g=0.02). Sign confirmed.
• grounding (neural phase boundary): stationary H hovers 80–91% of H* and is noisy / non-monotonic at 5 reps — no crisp g*. KEY FINDING: the neural models' smoothing inductive bias partially resists H-collapse (dry H stays ~83% of H*), so forward-KL and tail survival are the sharp neural collapse metrics, not H (mirrors Layer-1's "H is smooth; the threshold lives in tail survival"). grounding needs (a) forward-KL as the phase metric, (b) more reps (≥10), and/or (c) a stronger-collapse regime for a clean neural g*.
• architectures (architecture-generality) — clean result: collapse + grounding-rescue appear in ALL three model classes (dry→grounded forward-KL: histogram 6.2→4.6, MLP 4.8→1.3, RNN 3.8→1.1; tailalive RNN 0.41→0.64, MLP 0.07→0.20). The WF operator is architecture-general. Bonus: neural smoothing lets RNN/MLP retain more tail than the exact histogram under grounding (they generalise to unseen codewords) — an inductive-bias finding worth the write-up.

2026-07-04 — recombination (load-bearing E4 replication).

• recombine.py mirrors run_coverage but trains K_T specialist RNNs on assignments from the exact shared-switch retention construction (K_T/rho/q clean; union matches the closed form), then recombines the measured teacher distributions two ways: mean (naive pooling) vs max (oracle-guided union / M2N2-style), each followed by size-n resampling. The neural merge is the per-mode max over teacher distributions (oracle-guided), NOT weight-averaging of RNNs.
• Result (8 reps): at rho=0, union rises 0.49→0.96 (supply matches closed form); analytic surviving_max rises 0.043→0.087 while surviving_mean stays flat ~0.045 — the conservation law (averaging cancels the union gain; max-merge realises it). At rho=1 (identical teachers) union AND max are flat — more identical teachers buy nothing. The "merge, don't average" lesson holds in the neural setting. Trained-weight columns show the same signs but noisier: neural smoothing inflates baseline survival and the deep tail barely clears n=200 resampling (compresses magnitude) — the expected inductive-bias caveat. torch-gated test added. 93 tests green.

Remaining

• grounding refinement: re-run with forward-KL as the phase metric + ≥10 reps (and/or smaller n) for a clean neural g*. Pin the falsifier ("g* ≪ 1 exists") before re-running.
• region_matched grounding (R>1), remint re-mint gate (optional).
• VAE fidelity: fix the prior-hole mismatch (KL-annealing / free-bits / larger latent) so it clears the gen-0 gate, then add to architectures. Or document as a known limitation.
• Real-MNIST secondary tier (ClassifierOracle + confusion matrix; --extra mnist).
• figures/plot_<name>.py (reuse figures/_figlib.py); wire into make figures.

Discovered during work

• E2 grounding policy vs. the analytic H_eq: Pred. 3's closed form is derived for plain immigration Multinomial(m, p*). Implemented as policy="proportional", and every policy reduces to it at R=1. E2 should therefore run at R=1 (or proportional) so the phase-boundary sweep tracks the exact H_eq; region structure is E3's concern. Decide E2's init (uniform vs truth) when building its config.
• init: {uniform|truth} added to TruthCfg (uniform default, mandated by the decay conformance test). E1/E2 may want truth start for a clean "tail collapses from the truth" story — revisit in Phase 2.

Potential agents

(none proposed yet)