MachineSex/tasks/todo.md

Layer 1 Execution Plan — The Lamarckian Society (analytical core)

Created 2026-07-04. Scope: blueprint §7 build-order steps 1–4 (scaffold → Layer 1 complete, laptop-reproducible). Layer 2 is explicitly out of scope here and gated on Layer 1's scientific-validation tests passing.

Source of truth: lamarckian-society-technical-blueprint-v1.md. Where it is silent I record a decision below rather than improvise silently.

---

Design decisions to pin before coding (blueprint leaves these open)

These are the only places the spec is genuinely underdetermined. Recommendation given for each; flagged for sign-off.

1. Config framework — recommend thin pydantic + PyYAML, not Hydra. Blueprint says "Hydra or a thin equivalent." Hydra is heavyweight and its global-state/os.chdir behaviour fights the "pass rng explicitly, results are a pure function of resolved config" requirement. A thin loader (pydantic models for validation + a small sweep-expander) is dependency-light, aligns with the stdlib-first preference, and makes the "write resolved config beside results" contract trivial. Decision: thin pydantic loader.

2. Selection fitness f_i (the reality-anchored score). Blueprint: "fitness is predictive accuracy against p*" but gives no formula for the discrete model. Decision: f_i = p*_eff_i (truth frequency = fitness; reality-anchored by construction). Post-selection distribution p'_i ∝ p_i^(1−α) · f_i, matching the blueprint's stated w_i ∝ f_i·(p_i)^(−α) with α=0 recovering fitness-proportional greedy. Document as a modelling choice; select-then-sample vs sample-then-select is the documented robustness switch (§2.2).

3. tail_mask definition. Two knobs exist (tail_frac, tail_threshold). Decision: the metric-bearing tail set is the §2.3 definition {i : p*_i < tail_threshold}. tail_frac only drives the twocomponent construction (fraction of items placed in the low-mass component). Document that for zipf, tail_frac is unused.

4. TrueDist type. Decision: a frozen dataclass TrueDist(p_star: np.ndarray, regions: np.ndarray, tail_mask: np.ndarray) — immutable, so p* cannot be mutated in place (except the deliberate re-mint path, which produces a new object).

5. Region partition. Decision: contiguous blocks; require K % R == 0 (assert with a clear error) to keep per-region math clean for v1.

6. Re-mint semantics (E6). Decision: re-mint replaces p_star_eff with a new TrueDist built from current p_t (fresh tail_mask recomputed on p_t), and the original TrueDist is retained only for the KL-to-original metric, never for grounding. This is the irreversibility the experiment tests.

---

Phase 0 — Scaffold & environment

• Install uv (curl -LsSf https://astral.sh/uv/install.sh | sh; lands in ~/.local/bin, no sudo).
• Reproducibility path = uv venv from a committed, hash-pinned uv.lock (GG decision, 2026-07-04; no Apptainer/Docker for Layer 1). pyproject.toml (Python ≥3.11; deps: numpy, scipy, pandas, pyarrow, matplotlib, pydantic, pyyaml; dev: pytest). Commit uv.lock. Create .venv via uv sync.
• Repo layout per §5: src/knowledge/, configs/layer1/, figures/, results/ (gitignored), tests/, paper/. Add src/lamarckian/ package root or make knowledge importable (decide package name — recommend src/lamarckian/knowledge/... with src-layout).
• .gitignore (.venv/, results/, __pycache__/, *.parquet under results but keep hashes).
• Seeding util lamarckian/utils/seeding.py: master seed → np.random.SeedSequence(seed).spawn(n) → per-replicate np.random.default_rng(child). No global RNG anywhere.
• Config loader lamarckian/config.py: pydantic schema mirroring the §2.7 YAML, a load_config(path), a expand_sweeps(cfg) -> list[ResolvedConfig], and write_resolved(cfg, dir).
• Manifest util: write_manifest(dir, results_df) recording lib versions, master seed, git rev-parse HEAD, content hash of results.parquet.
• Makefile targets (env, test, layer1, figures, clean) + pytest skeleton. Gate: make test green on a trivial test.
• Move lamarckian-society-technical-blueprint-v1.md → paper/blueprint.md per §5 (confirm with GG first — it's referenced by name elsewhere).

Phase 1 — Core primitives + null model + VALIDATION GATE

Implement to the normative signatures in §2.7. Order chosen so each piece is unit-testable before the next depends on it.

• knowledge/truth.py::make_true_distribution → TrueDist. Support tail ∈ {zipf, twocomponent}. Unit tests: normalisation, region block sizes, tail_mask matches threshold, determinism from seed.
• knowledge/metrics.py: forward_kl (with eps floor, logged), heterozygosity, tail_mass, support_size, plus per-region variants. Unit tests on hand-computed small vectors.
• knowledge/step.py::generation_step — null path first (single teacher, m=0, selection none): c ~ Multinomial(n, p_t), p_{t+1}=c/n. Exactly neutral Wright–Fisher.
• knowledge/lineage.py::run_lineage(cfg, seed) → tidy per-generation DataFrame (all §2.3 metrics, global + per-region).
• tests/test_scientific_validation.py — the spine:
  • Pred. 1 heterozygosity decay: mean H_t over replicates matches H₀(1−1/n)^t. Prefer testing the full deterministic mean-recursion trajectory (subsumes the fixed point), within Monte-Carlo CI.
  • Pred. 2 fixation probability = initial frequency (long runs, statistical tolerance).
• HARD GATE: do not proceed until Pred. 1–2 pass. If drift ≠ analytic decay, the harness is wrong — fix here.

Phase 2 — Grounding + E1 + E2 (the headline)

• knowledge/step.py::structured_multinomial(m_vector, p_star, regions, policy, rng) — per-region immigration draws from p* restricted+renormalised to each region; uniform spreads m evenly, matched concentrates on exercised regions. Returns length-K counts.
• Extend generation_step with grounding (pooled draw, g = m/(n+m)).
• Pred. 3 validation — exact equilibrium H_eq = H*·m(2n+m−1)/(n+2nm+m²): run to stationarity (burn-in + late-generation + replicate averaging), assert <0.1% rel. error vs closed form across an m grid. Also assert the m→0 and m→∞ limits.
• Pred. 4 validation — tail-persistence: item of freq p*_i maintained iff m·p*_i ≳ 1; verify the survival transition location statistically.
• knowledge/experiment.py::run_experiment(cfg) — sweep grid × n_replicates; long-form results + CIs; write results.parquet + resolved_config.yaml + manifest.json.
• E1 config + run: m=0, single teacher, no selection. Expect H geometric decay, support→1, KL diverges, tail-first loss.
• E2 config + run: sweep g, single teacher, uniform grounding, no selection. Locate critical g* (transition in tail mass / support, since H is smooth in m — the sharp threshold is in discrete tail survival). Report g* with CI. This is the load-bearing result.
• figures/plot_E1.py, plot_E2.py — read results.parquet only.

Phase 3 — E3–E6

• E3 region-matched grounding. Fixed total m; uniform vs matched; one designated inherited-but-unwatered region with a rare tail. Expect uniform lets that region's tail collapse; matched holds it. Per-region metrics essential. plot_E3.py.
• §2.7.1 correlated-teacher construction — knowledge/teachers.py:
  • make_retention_matrix(T, K_T, rho, q, rng) — shared-switch exchangeable Bernoulli.
  • make_correlated_teachers(...) — retention→distributions (head kept at p*; tail at p*_i if retained else tail_floor; renormalise). region_specialisation option.
  • Pred. 5 validation: make_retention_matrix reproduces marginal q, pairwise ρ, and union coverage U(K_T,ρ,q)=T[ρq+(1−ρ)(1−(1−q)^K_T)] to 3 decimals over a (ρ,q) grid.
• E4 multi-teacher decorrelation. Sweep K_T∈{1,2,3,5}, ρ∈[0,1] at fixed q, matched budget (n/K_T each). Report both union U and post-distillation surviving coverage; show their gap shrinks as g rises. plot_E4.py (coverage surface over (K_T,ρ)).
• E5 QD vs greedy. apply_selection (none/greedy/qd, pinned fitness form). Sweep novelty α. Expect greedy→fixation (H→0), qd holds H plateau + re-introduces tails. plot_E5.py.
• E6 re-mint gate. Re-mint at high vs low H; track KL to original truth. Expect collapsed re-mint locks KL high forever; gated (high-H) does not. plot_E6.py.

Phase 4 — Reproducibility polish (Layer 1 slice)

• configs/layer1/E1..E6.yaml all committed with explicit params (no magic numbers in code).
• paper/figure_manifest.md — the §6 claim→experiment→figure rows for Layer 1.
• make layer1 runs E1–E6; make figures regenerates all figures from committed parquet.
• Full test_correctness.py (shapes, normalisation, determinism) + test_scientific_validation.py (Pred. 1–5) green in CI.
• reproduce.sh (uv sync → make test → make layer1 → make figures → write REPRODUCED.md diffing committed result hashes) + README.md reproduce section. No container — the committed uv.lock is the reproducibility source of truth (per GG, 2026-07-04); a Dockerfile may later wrap the same lockfile for Layer 2's GPU work.

---

Definition of done (Layer 1)

Every Layer-1 row of blueprint §6 has a committed figure produced by make figures from committed results; all §2.4 analytic checks (Pred. 1–5) pass; make layer1 && make figures reproduces from a clean .venv. Then — and only then — Layer 2 may begin.

Falsifier watch (report honestly if hit)

• E2 tail mass flat in g, or only stabilises as g→1 → multigenerational thesis refuted.
• E3 uniform protects as well as matched → region-matching claim dies.
• E4 no surviving-coverage benefit at matched budget → recombination claim dies.
• E5 qd ≤ greedy stationary H → QD does no work.
• E6 collapsed lineage recovers original-truth tails after re-mint → irreversibility overstated.

Review — progress log

2026-07-04 — Phases 0 & 1 complete; hard gate PASSED.

• Reorg: docs → paper/ (blueprint.md, the-lamarckian-society-v4.md). src-layout under src/knowledge/.
• A pre-existing tests/test_scientific_validation.py (author-supplied, 22 KB) turned out to hard-specify the package contract — implemented to it rather than inventing interfaces. Key contracts it locked (now honoured): package imports as knowledge.*; run_lineage(cfg_dict, seed) returns a tidy per-gen frame with a heterozygosity column, rows 0..T; p_0 initialises uniform (H_0=1−1/K); metrics.heterozygosity and teachers.make_retention_matrix match the reference to 1e-12 / closed form.
• Env: uv 0.11.26 installed; pyproject.toml + uv.lock committed; numpy 2.5, pandas 3.0, scipy 1.18, pydantic 2.13, pytest 9.1.
• Modules written: metrics, seeding, config (dataclasses + from_dict), truth, teachers, step, lineage. Config is dataclass-based (not pydantic) — the conformance test passes a raw dict; dataclasses validate cleanly and stay stdlib-simple. Pydantic still a dep for the Phase-2 YAML/experiment layer.
• Results: make test green — 68 passed (48 scientific-validation, 20 correctness). Conformance tests RAN (not skipped): Pred. 1 (neutral decay), Pred. 3 (exact H_eq), Pred. 5 (union coverage) all pass against the real package. The Pred. 1–2 hard gate is passed, and grounding already conforms to the exact equilibrium.

Design decisions #1 (dataclasses now / pydantic at YAML layer), #2 (fitness f_i=p*_i), #3 (threshold tail_mask), #6 (re-mint discard) all implemented as planned. Region design: each region an identical 1/R-mass block (symmetric; reduces to global Zipf at R=1, matching the reference).

2026-07-04 — Phase 2 complete (E1 + E2).

• experiment.py: sweep expansion (Cartesian grid; special-cases g→m), paired replicate seeds (shared across grid points), output contract (results.parquet + resolved_config.yaml + manifest.json with lib versions + git commit + sha256). CLI python -m knowledge.experiment <cfg>.
• E1 (null collapse) — reproduces tail-first collapse: H geometric decay matches H₀(1−1/n)ᵗ within CI; tail items die ~10× faster than head items; support 500→1; forward-KL diverges. Figure results/E1/E1.png.
• E2 (headline) — H_sim tracks the exact H_eq closed form across the sweep; phase boundary at g* ≪ 1: g=0.005 (m=1 real sample vs n=200) → 68% of truth H; g=0.05 → 96%. g=0 slides to ~0.10 over 500 gens. Figure results/E2/E2.png. Headline result achieved.
• Metric subtlety found & fixed: aggregate tail_mass is a drift martingale (mean-conserved), so it's a poor collapse indicator. Added tail_support/head_support/tail_frac_alive/head_frac_alive; E1 & E2 figures now use tail-item survival, which is honest and monotone.
• E2 extended 300→500 generations (GG-approved) so the g=0 arm visibly approaches 0 while g>0 arms sit on plateaus.
• Makefile layer1/figures wired to E1–E2. make test still green (68).

Discovered during work

• E2 grounding policy vs. the analytic H_eq: Pred. 3's closed form is derived for plain immigration Multinomial(m, p*). Implemented as policy="proportional", and every policy reduces to it at R=1. E2 should therefore run at R=1 (or proportional) so the phase-boundary sweep tracks the exact H_eq; region structure is E3's concern. Decide E2's init (uniform vs truth) when building its config.
• init: {uniform|truth} added to TruthCfg (uniform default, mandated by the decay conformance test). E1/E2 may want truth start for a clean "tail collapses from the truth" story — revisit in Phase 2.

Potential agents

(none proposed yet)