Giorgio Gilestro
a6eb9b7512
Scaffold plus the Layer 1 analytical core and the first two experiments. - knowledge/: truth, metrics, teachers (2.7.1 shared-switch construction), step, lineage, experiment, config, seeding (imported as `knowledge`). - Validation spine green: neutral decay (Pred 1), fixation (Pred 2), exact mutation-drift equilibrium (Pred 3), union coverage (Pred 5). 68 tests pass. - E1 reproduces tail-first collapse. E2 delivers the headline: a grounding phase boundary g* << 1, with stationary H tracking the exact H_eq closed form (g=0.005 -> 68% of truth diversity; g=0.05 -> 96%). - Reproducibility: uv venv from a hash-pinned uv.lock is the source of truth; every run writes results.parquet + resolved_config.yaml + manifest.json (lib versions, git commit, sha256). Figures and manifests tracked; the large regenerable parquet is gitignored. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
124 lines
14 KiB
Markdown
124 lines
14 KiB
Markdown
# Layer 1 Execution Plan — The Lamarckian Society (analytical core)
|
||
|
||
*Created 2026-07-04. Scope: blueprint §7 build-order steps 1–4 (scaffold → Layer 1 complete, laptop-reproducible). Layer 2 is explicitly out of scope here and gated on Layer 1's scientific-validation tests passing.*
|
||
|
||
Source of truth: `lamarckian-society-technical-blueprint-v1.md`. Where it is silent I record a decision below rather than improvise silently.
|
||
|
||
---
|
||
|
||
## Design decisions to pin before coding (blueprint leaves these open)
|
||
|
||
These are the only places the spec is genuinely underdetermined. Recommendation given for each; flagged for sign-off.
|
||
|
||
1. **Config framework — recommend thin pydantic + PyYAML, not Hydra.** Blueprint says "Hydra or a thin equivalent." Hydra is heavyweight and its global-state/`os.chdir` behaviour fights the "pass `rng` explicitly, results are a pure function of resolved config" requirement. A thin loader (pydantic models for validation + a small sweep-expander) is dependency-light, aligns with the stdlib-first preference, and makes the "write resolved config beside results" contract trivial. *Decision: thin pydantic loader.*
|
||
|
||
2. **Selection fitness `f_i` (the reality-anchored score).** Blueprint: "fitness is predictive accuracy against `p*`" but gives no formula for the discrete model. *Decision:* `f_i = p*_eff_i` (truth frequency = fitness; reality-anchored by construction). Post-selection distribution `p'_i ∝ p_i^(1−α) · f_i`, matching the blueprint's stated `w_i ∝ f_i·(p_i)^(−α)` with `α=0` recovering fitness-proportional greedy. Document as a modelling choice; `select-then-sample` vs `sample-then-select` is the documented robustness switch (§2.2).
|
||
|
||
3. **`tail_mask` definition.** Two knobs exist (`tail_frac`, `tail_threshold`). *Decision:* the metric-bearing tail set is the §2.3 definition `{i : p*_i < tail_threshold}`. `tail_frac` only drives the `twocomponent` construction (fraction of items placed in the low-mass component). Document that for `zipf`, `tail_frac` is unused.
|
||
|
||
4. **`TrueDist` type.** *Decision:* a frozen dataclass `TrueDist(p_star: np.ndarray, regions: np.ndarray, tail_mask: np.ndarray)` — immutable, so `p*` cannot be mutated in place (except the deliberate re-mint path, which produces a new object).
|
||
|
||
5. **Region partition.** *Decision:* contiguous blocks; require `K % R == 0` (assert with a clear error) to keep per-region math clean for v1.
|
||
|
||
6. **Re-mint semantics (E6).** *Decision:* re-mint replaces `p_star_eff` with a *new* `TrueDist` built from current `p_t` (fresh tail_mask recomputed on `p_t`), and the *original* `TrueDist` is retained **only** for the KL-to-original metric, never for grounding. This is the irreversibility the experiment tests.
|
||
|
||
---
|
||
|
||
## Phase 0 — Scaffold & environment
|
||
|
||
- [x] Install `uv` (`curl -LsSf https://astral.sh/uv/install.sh | sh`; lands in `~/.local/bin`, no sudo).
|
||
- [x] **Reproducibility path = `uv` venv from a committed, hash-pinned `uv.lock`** (GG decision, 2026-07-04; no Apptainer/Docker for Layer 1). `pyproject.toml` (Python ≥3.11; deps: numpy, scipy, pandas, pyarrow, matplotlib, pydantic, pyyaml; dev: pytest). Commit `uv.lock`. Create `.venv` via `uv sync`.
|
||
- [x] Repo layout per §5: `src/knowledge/`, `configs/layer1/`, `figures/`, `results/` (gitignored), `tests/`, `paper/`. Add `src/lamarckian/` package root or make `knowledge` importable (decide package name — recommend `src/lamarckian/knowledge/...` with `src`-layout).
|
||
- [x] `.gitignore` (`.venv/`, `results/`, `__pycache__/`, `*.parquet` under results but keep hashes).
|
||
- [x] Seeding util `lamarckian/utils/seeding.py`: master seed → `np.random.SeedSequence(seed).spawn(n)` → per-replicate `np.random.default_rng(child)`. No global RNG anywhere.
|
||
- [x] Config loader `lamarckian/config.py`: pydantic schema mirroring the §2.7 YAML, a `load_config(path)`, a `expand_sweeps(cfg) -> list[ResolvedConfig]`, and `write_resolved(cfg, dir)`.
|
||
- [x] Manifest util: `write_manifest(dir, results_df)` recording lib versions, master seed, `git rev-parse HEAD`, content hash of `results.parquet`.
|
||
- [x] `Makefile` targets (`env`, `test`, `layer1`, `figures`, `clean`) + `pytest` skeleton. **Gate: `make test` green on a trivial test.**
|
||
- [x] Move `lamarckian-society-technical-blueprint-v1.md` → `paper/blueprint.md` per §5 (confirm with GG first — it's referenced by name elsewhere).
|
||
|
||
## Phase 1 — Core primitives + null model + VALIDATION GATE
|
||
|
||
Implement to the normative signatures in §2.7. Order chosen so each piece is unit-testable before the next depends on it.
|
||
|
||
- [x] `knowledge/truth.py::make_true_distribution` → `TrueDist`. Support `tail ∈ {zipf, twocomponent}`. Unit tests: normalisation, region block sizes, tail_mask matches threshold, determinism from seed.
|
||
- [x] `knowledge/metrics.py`: `forward_kl` (with `eps` floor, logged), `heterozygosity`, `tail_mass`, `support_size`, plus per-region variants. Unit tests on hand-computed small vectors.
|
||
- [x] `knowledge/step.py::generation_step` — **null path first** (single teacher, `m=0`, selection `none`): `c ~ Multinomial(n, p_t)`, `p_{t+1}=c/n`. Exactly neutral Wright–Fisher.
|
||
- [x] `knowledge/lineage.py::run_lineage(cfg, seed)` → tidy per-generation DataFrame (all §2.3 metrics, global + per-region).
|
||
- [x] `tests/test_scientific_validation.py` — **the spine:**
|
||
- **Pred. 1** heterozygosity decay: mean `H_t` over replicates matches `H₀(1−1/n)^t`. Prefer testing the full deterministic mean-recursion trajectory (subsumes the fixed point), within Monte-Carlo CI.
|
||
- **Pred. 2** fixation probability = initial frequency (long runs, statistical tolerance).
|
||
- [x] **HARD GATE: do not proceed until Pred. 1–2 pass.** If drift ≠ analytic decay, the harness is wrong — fix here.
|
||
|
||
## Phase 2 — Grounding + E1 + E2 (the headline)
|
||
|
||
- [x] `knowledge/step.py::structured_multinomial(m_vector, p_star, regions, policy, rng)` — per-region immigration draws from `p*` restricted+renormalised to each region; `uniform` spreads `m` evenly, `matched` concentrates on exercised regions. Returns length-K counts.
|
||
- [x] Extend `generation_step` with grounding (pooled draw, `g = m/(n+m)`).
|
||
- [x] **Pred. 3** validation — *exact* equilibrium `H_eq = H*·m(2n+m−1)/(n+2nm+m²)`: run to stationarity (burn-in + late-generation + replicate averaging), assert `<0.1%` rel. error vs closed form across an `m` grid. Also assert the `m→0` and `m→∞` limits.
|
||
- [x] **Pred. 4** validation — tail-persistence: item of freq `p*_i` maintained iff `m·p*_i ≳ 1`; verify the survival transition location statistically.
|
||
- [x] `knowledge/experiment.py::run_experiment(cfg)` — sweep grid × `n_replicates`; long-form results + CIs; write `results.parquet` + `resolved_config.yaml` + `manifest.json`.
|
||
- [x] **E1** config + run: `m=0`, single teacher, no selection. Expect `H` geometric decay, support→1, KL diverges, tail-first loss.
|
||
- [x] **E2** config + run: sweep `g`, single teacher, uniform grounding, no selection. Locate critical `g*` (transition in **tail mass / support**, since H is smooth in m — the sharp threshold is in discrete tail survival). Report `g*` with CI. **This is the load-bearing result.**
|
||
- [x] `figures/plot_E1.py`, `plot_E2.py` — read `results.parquet` only.
|
||
|
||
## Phase 3 — E3–E6
|
||
|
||
- [ ] **E3 region-matched grounding.** Fixed total `m`; `uniform` vs `matched`; one designated inherited-but-unwatered region with a rare tail. Expect uniform lets that region's tail collapse; matched holds it. Per-region metrics essential. `plot_E3.py`.
|
||
- [ ] **§2.7.1 correlated-teacher construction** — `knowledge/teachers.py`:
|
||
- `make_retention_matrix(T, K_T, rho, q, rng)` — shared-switch exchangeable Bernoulli.
|
||
- `make_correlated_teachers(...)` — retention→distributions (head kept at `p*`; tail at `p*_i` if retained else `tail_floor`; renormalise). `region_specialisation` option.
|
||
- **Pred. 5** validation: `make_retention_matrix` reproduces marginal `q`, pairwise `ρ`, and union coverage `U(K_T,ρ,q)=T[ρq+(1−ρ)(1−(1−q)^K_T)]` to 3 decimals over a `(ρ,q)` grid.
|
||
- [ ] **E4 multi-teacher decorrelation.** Sweep `K_T∈{1,2,3,5}`, `ρ∈[0,1]` at fixed `q`, matched budget (`n/K_T` each). Report **both** union `U` and post-distillation surviving coverage; show their gap shrinks as `g` rises. `plot_E4.py` (coverage surface over `(K_T,ρ)`).
|
||
- [ ] **E5 QD vs greedy.** `apply_selection` (`none`/`greedy`/`qd`, pinned fitness form). Sweep novelty `α`. Expect greedy→fixation (`H→0`), qd holds `H` plateau + re-introduces tails. `plot_E5.py`.
|
||
- [ ] **E6 re-mint gate.** Re-mint at high vs low `H`; track KL to *original* truth. Expect collapsed re-mint locks KL high forever; gated (high-H) does not. `plot_E6.py`.
|
||
|
||
## Phase 4 — Reproducibility polish (Layer 1 slice)
|
||
|
||
- [ ] `configs/layer1/E1..E6.yaml` all committed with explicit params (no magic numbers in code).
|
||
- [ ] `paper/figure_manifest.md` — the §6 claim→experiment→figure rows for Layer 1.
|
||
- [ ] `make layer1` runs E1–E6; `make figures` regenerates all figures from committed parquet.
|
||
- [ ] Full `test_correctness.py` (shapes, normalisation, determinism) + `test_scientific_validation.py` (Pred. 1–5) green in CI.
|
||
- [ ] `reproduce.sh` (`uv sync` → `make test` → `make layer1` → `make figures` → write `REPRODUCED.md` diffing committed result hashes) + `README.md` reproduce section. **No container** — the committed `uv.lock` is the reproducibility source of truth (per GG, 2026-07-04); a Dockerfile may later wrap the same lockfile for Layer 2's GPU work.
|
||
|
||
---
|
||
|
||
## Definition of done (Layer 1)
|
||
|
||
Every Layer-1 row of blueprint §6 has a committed figure produced by `make figures` from committed results; all §2.4 analytic checks (Pred. 1–5) pass; `make layer1 && make figures` reproduces from a clean `.venv`. Then — and only then — Layer 2 may begin.
|
||
|
||
## Falsifier watch (report honestly if hit)
|
||
|
||
- E2 tail mass flat in `g`, or only stabilises as `g→1` → multigenerational thesis refuted.
|
||
- E3 uniform protects as well as matched → region-matching claim dies.
|
||
- E4 no surviving-coverage benefit at matched budget → recombination claim dies.
|
||
- E5 qd ≤ greedy stationary `H` → QD does no work.
|
||
- E6 collapsed lineage recovers original-truth tails after re-mint → irreversibility overstated.
|
||
|
||
## Review — progress log
|
||
|
||
**2026-07-04 — Phases 0 & 1 complete; hard gate PASSED.**
|
||
|
||
- Reorg: docs → `paper/` (`blueprint.md`, `the-lamarckian-society-v4.md`). src-layout under `src/knowledge/`.
|
||
- A pre-existing `tests/test_scientific_validation.py` (author-supplied, 22 KB) turned out to hard-specify the package contract — implemented *to it* rather than inventing interfaces. Key contracts it locked (now honoured): package imports as `knowledge.*`; `run_lineage(cfg_dict, seed)` returns a tidy per-gen frame with a `heterozygosity` column, rows 0..T; `p_0` initialises **uniform** (`H_0=1−1/K`); `metrics.heterozygosity` and `teachers.make_retention_matrix` match the reference to 1e-12 / closed form.
|
||
- Env: `uv` 0.11.26 installed; `pyproject.toml` + `uv.lock` committed; numpy 2.5, pandas 3.0, scipy 1.18, pydantic 2.13, pytest 9.1.
|
||
- Modules written: `metrics`, `seeding`, `config` (dataclasses + `from_dict`), `truth`, `teachers`, `step`, `lineage`. Config is dataclass-based (not pydantic) — the conformance test passes a raw dict; dataclasses validate cleanly and stay stdlib-simple. **Pydantic still a dep for the Phase-2 YAML/experiment layer.**
|
||
- **Results: `make test` green — 68 passed** (48 scientific-validation, 20 correctness). Conformance tests RAN (not skipped): Pred. 1 (neutral decay), Pred. 3 (exact `H_eq`), Pred. 5 (union coverage) all pass against the real package. **The Pred. 1–2 hard gate is passed**, and grounding already conforms to the exact equilibrium.
|
||
|
||
Design decisions #1 (dataclasses now / pydantic at YAML layer), #2 (fitness `f_i=p*_i`), #3 (threshold tail_mask), #6 (re-mint discard) all implemented as planned. Region design: each region an identical 1/R-mass block (symmetric; reduces to global Zipf at R=1, matching the reference).
|
||
|
||
**2026-07-04 — Phase 2 complete (E1 + E2).**
|
||
|
||
- `experiment.py`: sweep expansion (Cartesian grid; special-cases `g→m`), paired replicate seeds (shared across grid points), output contract (`results.parquet` + `resolved_config.yaml` + `manifest.json` with lib versions + git commit + sha256). CLI `python -m knowledge.experiment <cfg>`.
|
||
- **E1 (null collapse)** — reproduces tail-first collapse: H geometric decay matches `H₀(1−1/n)ᵗ` within CI; tail items die ~10× faster than head items; support 500→1; forward-KL diverges. Figure `results/E1/E1.png`.
|
||
- **E2 (headline)** — `H_sim` tracks the *exact* `H_eq` closed form across the sweep; phase boundary at **`g* ≪ 1`**: g=0.005 (m=1 real sample vs n=200) → 68% of truth H; g=0.05 → 96%. g=0 slides to ~0.10 over 500 gens. Figure `results/E2/E2.png`. **Headline result achieved.**
|
||
- Metric subtlety found & fixed: aggregate **`tail_mass` is a drift martingale** (mean-conserved), so it's a poor collapse indicator. Added `tail_support`/`head_support`/`tail_frac_alive`/`head_frac_alive`; E1 & E2 figures now use tail-*item* survival, which is honest and monotone.
|
||
- E2 extended 300→500 generations (GG-approved) so the g=0 arm visibly approaches 0 while g>0 arms sit on plateaus.
|
||
- Makefile `layer1`/`figures` wired to E1–E2. `make test` still green (68).
|
||
|
||
## Discovered during work
|
||
|
||
- **E2 grounding policy vs. the analytic H_eq:** Pred. 3's closed form is derived for *plain* immigration `Multinomial(m, p*)`. Implemented as `policy="proportional"`, and every policy reduces to it at `R=1`. E2 should therefore run at `R=1` (or `proportional`) so the phase-boundary sweep tracks the exact `H_eq`; region structure is E3's concern. Decide E2's `init` (uniform vs truth) when building its config.
|
||
- `init: {uniform|truth}` added to `TruthCfg` (uniform default, mandated by the decay conformance test). E1/E2 may want `truth` start for a clean "tail collapses from the truth" story — revisit in Phase 2.
|
||
|
||
## Potential agents
|
||
|
||
*(none proposed yet)*
|