Giorgio Gilestro
62c68d6c8c
Enter the Lamarckian society with a robust theoretical frame. The single-
locus, fixed-p* model can only express recovery toward a ceiling; the
society's load-bearing claim is vertical -- capability that EXCEEDS any
component. Generalize knowledge to a distribution over genotypes (L
biallelic loci, K=2^L, additive fitness = # correct loci), reusing all the
K-mode machinery. The one new operator is recombination: free recombination
sends p -> product of per-locus marginals (linkage equilibrium).
E8 (star, kind: society) -- the vertical claim / Fisher-Muller: decorrelated
PARENTS (specialists, expert on their loci, agnostic elsewhere) are
recombined; sexual merge assembles a genotype fitter than any parent,
climbing to the optimum (12/12, a genotype no parent had) as parent count
grows and rho->0, while the best single parent (~8.7) and the mean-mixture
"model soup" (~11.6) plateau below. Reuses make_retention_matrix (locus
mastery replaces tail-item retention).
E7 (kind: genotype_lineage) -- the advantage of sex: a single population
adapts toward the optimum; the sexual lineage adapts faster than asexual
(clonal interference) by keeping loci in linkage equilibrium (LD->0 vs LD
spike). Honest scope: a speed advantage, not a permanent Muller's-ratchet
gap (subtle to force); E8 carries the headline.
Metaphor shift (per GG): the society is sexual reproduction with UNBOUNDED
parents, not teacher->pupil. Teacher->pupil caps at the ceiling; n-parent
recombination is combinatorial and generative, and unlike biology there is
no two-parent limit. Collapse = asexual degradation; the cure = sex. This
unifies E4 (merge != average) + E6 (irreversibility) under evolution-of-sex
theory and reaches ground Riis's single-locus n-grams cannot.
New: knowledge/{genotype,genotype_lineage,society}.py, configs/layer1/{E7,
E8}.yaml, figures/plot_{E7,E8}.py, READMEs, tests/test_genotype.py (+7).
experiment.py dispatch (kind in {genotype_lineage, society}); make layer1
wired. 112 tests green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
310 lines
29 KiB
Markdown
310 lines
29 KiB
Markdown
# Layer 1 Execution Plan — The Lamarckian Society (analytical core)
|
||
|
||
*Created 2026-07-04. Scope: blueprint §7 build-order steps 1–4 (scaffold → Layer 1 complete, laptop-reproducible). Layer 2 is explicitly out of scope here and gated on Layer 1's scientific-validation tests passing.*
|
||
|
||
Source of truth: `lamarckian-society-technical-blueprint-v1.md`. Where it is silent I record a decision below rather than improvise silently.
|
||
|
||
---
|
||
|
||
## Design decisions to pin before coding (blueprint leaves these open)
|
||
|
||
These are the only places the spec is genuinely underdetermined. Recommendation given for each; flagged for sign-off.
|
||
|
||
1. **Config framework — recommend thin pydantic + PyYAML, not Hydra.** Blueprint says "Hydra or a thin equivalent." Hydra is heavyweight and its global-state/`os.chdir` behaviour fights the "pass `rng` explicitly, results are a pure function of resolved config" requirement. A thin loader (pydantic models for validation + a small sweep-expander) is dependency-light, aligns with the stdlib-first preference, and makes the "write resolved config beside results" contract trivial. *Decision: thin pydantic loader.*
|
||
|
||
2. **Selection fitness `f_i` (the reality-anchored score).** Blueprint: "fitness is predictive accuracy against `p*`" but gives no formula for the discrete model. *Decision:* `f_i = p*_eff_i` (truth frequency = fitness; reality-anchored by construction). Post-selection distribution `p'_i ∝ p_i^(1−α) · f_i`, matching the blueprint's stated `w_i ∝ f_i·(p_i)^(−α)` with `α=0` recovering fitness-proportional greedy. Document as a modelling choice; `select-then-sample` vs `sample-then-select` is the documented robustness switch (§2.2).
|
||
|
||
3. **`tail_mask` definition.** Two knobs exist (`tail_frac`, `tail_threshold`). *Decision:* the metric-bearing tail set is the §2.3 definition `{i : p*_i < tail_threshold}`. `tail_frac` only drives the `twocomponent` construction (fraction of items placed in the low-mass component). Document that for `zipf`, `tail_frac` is unused.
|
||
|
||
4. **`TrueDist` type.** *Decision:* a frozen dataclass `TrueDist(p_star: np.ndarray, regions: np.ndarray, tail_mask: np.ndarray)` — immutable, so `p*` cannot be mutated in place (except the deliberate re-mint path, which produces a new object).
|
||
|
||
5. **Region partition.** *Decision:* contiguous blocks; require `K % R == 0` (assert with a clear error) to keep per-region math clean for v1.
|
||
|
||
6. **Re-mint semantics (E6).** *Decision:* re-mint replaces `p_star_eff` with a *new* `TrueDist` built from current `p_t` (fresh tail_mask recomputed on `p_t`), and the *original* `TrueDist` is retained **only** for the KL-to-original metric, never for grounding. This is the irreversibility the experiment tests.
|
||
|
||
---
|
||
|
||
## Phase 0 — Scaffold & environment
|
||
|
||
- [x] Install `uv` (`curl -LsSf https://astral.sh/uv/install.sh | sh`; lands in `~/.local/bin`, no sudo).
|
||
- [x] **Reproducibility path = `uv` venv from a committed, hash-pinned `uv.lock`** (GG decision, 2026-07-04; no Apptainer/Docker for Layer 1). `pyproject.toml` (Python ≥3.11; deps: numpy, scipy, pandas, pyarrow, matplotlib, pydantic, pyyaml; dev: pytest). Commit `uv.lock`. Create `.venv` via `uv sync`.
|
||
- [x] Repo layout per §5: `src/knowledge/`, `configs/layer1/`, `figures/`, `results/` (gitignored), `tests/`, `paper/`. Add `src/lamarckian/` package root or make `knowledge` importable (decide package name — recommend `src/lamarckian/knowledge/...` with `src`-layout).
|
||
- [x] `.gitignore` (`.venv/`, `results/`, `__pycache__/`, `*.parquet` under results but keep hashes).
|
||
- [x] Seeding util `lamarckian/utils/seeding.py`: master seed → `np.random.SeedSequence(seed).spawn(n)` → per-replicate `np.random.default_rng(child)`. No global RNG anywhere.
|
||
- [x] Config loader `lamarckian/config.py`: pydantic schema mirroring the §2.7 YAML, a `load_config(path)`, a `expand_sweeps(cfg) -> list[ResolvedConfig]`, and `write_resolved(cfg, dir)`.
|
||
- [x] Manifest util: `write_manifest(dir, results_df)` recording lib versions, master seed, `git rev-parse HEAD`, content hash of `results.parquet`.
|
||
- [x] `Makefile` targets (`env`, `test`, `layer1`, `figures`, `clean`) + `pytest` skeleton. **Gate: `make test` green on a trivial test.**
|
||
- [x] Move `lamarckian-society-technical-blueprint-v1.md` → `paper/blueprint.md` per §5 (confirm with GG first — it's referenced by name elsewhere).
|
||
|
||
## Phase 1 — Core primitives + null model + VALIDATION GATE
|
||
|
||
Implement to the normative signatures in §2.7. Order chosen so each piece is unit-testable before the next depends on it.
|
||
|
||
- [x] `knowledge/truth.py::make_true_distribution` → `TrueDist`. Support `tail ∈ {zipf, twocomponent}`. Unit tests: normalisation, region block sizes, tail_mask matches threshold, determinism from seed.
|
||
- [x] `knowledge/metrics.py`: `forward_kl` (with `eps` floor, logged), `heterozygosity`, `tail_mass`, `support_size`, plus per-region variants. Unit tests on hand-computed small vectors.
|
||
- [x] `knowledge/step.py::generation_step` — **null path first** (single teacher, `m=0`, selection `none`): `c ~ Multinomial(n, p_t)`, `p_{t+1}=c/n`. Exactly neutral Wright–Fisher.
|
||
- [x] `knowledge/lineage.py::run_lineage(cfg, seed)` → tidy per-generation DataFrame (all §2.3 metrics, global + per-region).
|
||
- [x] `tests/test_scientific_validation.py` — **the spine:**
|
||
- **Pred. 1** heterozygosity decay: mean `H_t` over replicates matches `H₀(1−1/n)^t`. Prefer testing the full deterministic mean-recursion trajectory (subsumes the fixed point), within Monte-Carlo CI.
|
||
- **Pred. 2** fixation probability = initial frequency (long runs, statistical tolerance).
|
||
- [x] **HARD GATE: do not proceed until Pred. 1–2 pass.** If drift ≠ analytic decay, the harness is wrong — fix here.
|
||
|
||
## Phase 2 — Grounding + E1 + E2 (the headline)
|
||
|
||
- [x] `knowledge/step.py::structured_multinomial(m_vector, p_star, regions, policy, rng)` — per-region immigration draws from `p*` restricted+renormalised to each region; `uniform` spreads `m` evenly, `matched` concentrates on exercised regions. Returns length-K counts.
|
||
- [x] Extend `generation_step` with grounding (pooled draw, `g = m/(n+m)`).
|
||
- [x] **Pred. 3** validation — *exact* equilibrium `H_eq = H*·m(2n+m−1)/(n+2nm+m²)`: run to stationarity (burn-in + late-generation + replicate averaging), assert `<0.1%` rel. error vs closed form across an `m` grid. Also assert the `m→0` and `m→∞` limits.
|
||
- [x] **Pred. 4** validation — tail-persistence: item of freq `p*_i` maintained iff `m·p*_i ≳ 1`; verify the survival transition location statistically.
|
||
- [x] `knowledge/experiment.py::run_experiment(cfg)` — sweep grid × `n_replicates`; long-form results + CIs; write `results.parquet` + `resolved_config.yaml` + `manifest.json`.
|
||
- [x] **E1** config + run: `m=0`, single teacher, no selection. Expect `H` geometric decay, support→1, KL diverges, tail-first loss.
|
||
- [x] **E2** config + run: sweep `g`, single teacher, uniform grounding, no selection. Locate critical `g*` (transition in **tail mass / support**, since H is smooth in m — the sharp threshold is in discrete tail survival). Report `g*` with CI. **This is the load-bearing result.**
|
||
- [x] `figures/plot_E1.py`, `plot_E2.py` — read `results.parquet` only.
|
||
|
||
## Phase 3 — E3–E6
|
||
|
||
- [x] **E3 region-matched grounding.** Fixed total `m`; `uniform` vs `matched`; one designated inherited-but-unwatered region with a rare tail. Expect uniform lets that region's tail collapse; matched holds it. Per-region metrics essential. `plot_E3.py`.
|
||
- [x] **§2.7.1 correlated-teacher construction** — `knowledge/teachers.py`:
|
||
- `make_retention_matrix(T, K_T, rho, q, rng)` — shared-switch exchangeable Bernoulli.
|
||
- `make_correlated_teachers(...)` — retention→distributions (head kept at `p*`; tail at `p*_i` if retained else `tail_floor`; renormalise). `region_specialisation` option.
|
||
- **Pred. 5** validation: `make_retention_matrix` reproduces marginal `q`, pairwise `ρ`, and union coverage `U(K_T,ρ,q)=T[ρq+(1−ρ)(1−(1−q)^K_T)]` to 3 decimals over a `(ρ,q)` grid.
|
||
- [x] **E4 multi-teacher decorrelation.** Sweep `K_T∈{1,2,3,5}`, `ρ∈[0,1]` at fixed `q`, matched budget (`n/K_T` each). Report **both** union `U` and post-distillation surviving coverage; show their gap shrinks as `g` rises. `plot_E4.py` (coverage surface over `(K_T,ρ)`).
|
||
- [x] **E5 QD vs greedy.** `apply_selection` (`none`/`greedy`/`qd`, pinned fitness form). Sweep novelty `α`. Expect greedy→fixation (`H→0`), qd holds `H` plateau + re-introduces tails. `plot_E5.py`.
|
||
- [x] **E6 re-mint gate.** Re-mint at high vs low `H`; track KL to *original* truth. Expect collapsed re-mint locks KL high forever; gated (high-H) does not. `plot_E6.py`.
|
||
|
||
## Phase 4 — Reproducibility polish (Layer 1 slice)
|
||
|
||
- [x] `configs/layer1/E1..E6.yaml` all committed with explicit params (no magic numbers in code).
|
||
- [x] `paper/figure_manifest.md` — the §6 claim→experiment→figure rows for Layer 1.
|
||
- [x] `make layer1` runs E1–E6; `make figures` regenerates all figures from committed parquet.
|
||
- [x] Full `test_correctness.py` (shapes, normalisation, determinism) + `test_scientific_validation.py` (Pred. 1–5) green in CI.
|
||
- [x] `reproduce.sh` (`uv sync` → `make test` → `make layer1` → `make figures` → write `REPRODUCED.md` diffing committed result hashes) + `README.md` reproduce section. **No container** — the committed `uv.lock` is the reproducibility source of truth (per GG, 2026-07-04); a Dockerfile may later wrap the same lockfile for Layer 2's GPU work.
|
||
|
||
---
|
||
|
||
## Definition of done (Layer 1)
|
||
|
||
Every Layer-1 row of blueprint §6 has a committed figure produced by `make figures` from committed results; all §2.4 analytic checks (Pred. 1–5) pass; `make layer1 && make figures` reproduces from a clean `.venv`. Then — and only then — Layer 2 may begin.
|
||
|
||
## Falsifier watch (report honestly if hit)
|
||
|
||
- E2 tail mass flat in `g`, or only stabilises as `g→1` → multigenerational thesis refuted.
|
||
- E3 uniform protects as well as matched → region-matching claim dies.
|
||
- E4 no surviving-coverage benefit at matched budget → recombination claim dies.
|
||
- E5 qd ≤ greedy stationary `H` → QD does no work.
|
||
- E6 collapsed lineage recovers original-truth tails after re-mint → irreversibility overstated.
|
||
|
||
## Review — progress log
|
||
|
||
**2026-07-04 — Phases 0 & 1 complete; hard gate PASSED.**
|
||
|
||
- Reorg: docs → `paper/` (`blueprint.md`, `the-lamarckian-society-v4.md`). src-layout under `src/knowledge/`.
|
||
- A pre-existing `tests/test_scientific_validation.py` (author-supplied, 22 KB) turned out to hard-specify the package contract — implemented *to it* rather than inventing interfaces. Key contracts it locked (now honoured): package imports as `knowledge.*`; `run_lineage(cfg_dict, seed)` returns a tidy per-gen frame with a `heterozygosity` column, rows 0..T; `p_0` initialises **uniform** (`H_0=1−1/K`); `metrics.heterozygosity` and `teachers.make_retention_matrix` match the reference to 1e-12 / closed form.
|
||
- Env: `uv` 0.11.26 installed; `pyproject.toml` + `uv.lock` committed; numpy 2.5, pandas 3.0, scipy 1.18, pydantic 2.13, pytest 9.1.
|
||
- Modules written: `metrics`, `seeding`, `config` (dataclasses + `from_dict`), `truth`, `teachers`, `step`, `lineage`. Config is dataclass-based (not pydantic) — the conformance test passes a raw dict; dataclasses validate cleanly and stay stdlib-simple. **Pydantic still a dep for the Phase-2 YAML/experiment layer.**
|
||
- **Results: `make test` green — 68 passed** (48 scientific-validation, 20 correctness). Conformance tests RAN (not skipped): Pred. 1 (neutral decay), Pred. 3 (exact `H_eq`), Pred. 5 (union coverage) all pass against the real package. **The Pred. 1–2 hard gate is passed**, and grounding already conforms to the exact equilibrium.
|
||
|
||
Design decisions #1 (dataclasses now / pydantic at YAML layer), #2 (fitness `f_i=p*_i`), #3 (threshold tail_mask), #6 (re-mint discard) all implemented as planned. Region design: each region an identical 1/R-mass block (symmetric; reduces to global Zipf at R=1, matching the reference).
|
||
|
||
**2026-07-04 — Phase 2 complete (E1 + E2).**
|
||
|
||
- `experiment.py`: sweep expansion (Cartesian grid; special-cases `g→m`), paired replicate seeds (shared across grid points), output contract (`results.parquet` + `resolved_config.yaml` + `manifest.json` with lib versions + git commit + sha256). CLI `python -m knowledge.experiment <cfg>`.
|
||
- **E1 (null collapse)** — reproduces tail-first collapse: H geometric decay matches `H₀(1−1/n)ᵗ` within CI; tail items die ~10× faster than head items; support 500→1; forward-KL diverges. Figure `results/E1/E1.png`.
|
||
- **E2 (headline)** — `H_sim` tracks the *exact* `H_eq` closed form across the sweep; phase boundary at **`g* ≪ 1`**: g=0.005 (m=1 real sample vs n=200) → 68% of truth H; g=0.05 → 96%. g=0 slides to ~0.10 over 500 gens. Figure `results/E2/E2.png`. **Headline result achieved.**
|
||
- Metric subtlety found & fixed: aggregate **`tail_mass` is a drift martingale** (mean-conserved), so it's a poor collapse indicator. Added `tail_support`/`head_support`/`tail_frac_alive`/`head_frac_alive`; E1 & E2 figures now use tail-*item* survival, which is honest and monotone.
|
||
- E2 extended 300→500 generations (GG-approved) so the g=0 arm visibly approaches 0 while g>0 arms sit on plateaus.
|
||
- Makefile `layer1`/`figures` wired to E1–E2. `make test` still green (68).
|
||
|
||
**2026-07-04 — Phase 3 complete (E3–E6) + E2 analysis add-ons.**
|
||
|
||
- **E3** region-matched grounding: added `grounding.exercised` knob + per-region `tailalive_region_r`. Target region tail survival 0.49 (matched) vs 0.07 (uniform). Note: per-region *H* is mass-confounded — used tail-item survival instead.
|
||
- **E4** multi-teacher recombination: bespoke `run_coverage` runner (`kind: coverage`). Union coverage matches `U(K_T,ρ,q)` exactly. **Key finding (GG-approved): mean-mixture distillation gives NO surviving benefit (conservation law — dilution cancels the union gain); max-merge (M2N2-style) does.** E4 reports both. In CLAUDE.md.
|
||
- **E5** QD vs greedy: greedy → H≈0.01 (fixation); qd holds H 0.48–0.88 rising with α. qd ≫ greedy.
|
||
- **E6** re-mint gate: added `arm` multi-override sweep type. Re-mint while collapsed → KL-to-original diverges (lock-in) + accelerates H collapse; diversity gate (H≥0.75) blocks it → bounded; healthy re-mint harmless.
|
||
- **E2 analysis add-ons** (companion work order `tasks/workorder-E2-analysis-addons.md`, verified): new `analysis.py` (`reduce_to_stationary`, `critical_grounding` bootstrap CI) — real E2 **g*=0.048, CI [0.047,0.050]**; `metrics.tail_band_metrics` + per-band lineage logging; `tests/test_analysis.py` reproduces the work order's verified numbers exactly. E2 figure rebuilt 2×2. **Deviation:** used truth-mass-weighted tail coverage instead of raw `tail_mass` (a drift martingale).
|
||
- All six figures regenerate via `make figures`; **71 tests green**.
|
||
|
||
---
|
||
|
||
# Layer 1.5 — Architecture-general neural existence proof (RNN/VAE/MLP + synthetic/MNIST)
|
||
|
||
*Created 2026-07-04. Plan: `~/.claude/plans/we-are-going-to-cheerful-fog.md`. Re-scopes Layer 2:
|
||
build a cheap, architecture-general neural collapse proof in real trained weights on a
|
||
fully-synthetic sandbox (exact known `p*`) before the LLM rung. Locked decisions: exact-oracle
|
||
categorical token sequences; Histogram+RNN+VAE+MLP; real MNIST as secondary confirmation; LLM +
|
||
C3 vertical claim deferred.*
|
||
|
||
## Progress log
|
||
|
||
**2026-07-04 — Stages A, B, plumbing complete.**
|
||
|
||
- **Env:** installed `uv` 0.11.26 (`~/.local/bin`); `/home` was 100% full — GG approved clearing
|
||
pip/yay/browser caches (~10 GB freed). Base venv synced; 71 Layer-1 tests green.
|
||
- **Stage A (scaffold, pure NumPy):** `src/neural/` — `config.py` (frozen dataclasses reusing
|
||
`knowledge.config` GroundingCfg/RemintCfg/MetricsCfg/_sub), `synthetic.py` (mode-truth via
|
||
`make_true_distribution`; lossless identity + stochastic style token grammar), `oracle.py`
|
||
(`ExactOracle` zero-error + `measure_distribution`), `models.py` (`GenerativeModel` protocol +
|
||
`HistogramModel` bridge), `evaluate.py` (reuses `knowledge.metrics`, Layer-1 row schema),
|
||
`generation_loop.py` (`run_generative_lineage`, reuses `allocate_m`/`structured_multinomial`).
|
||
15 correctness tests green.
|
||
- **Stage B — HARD GATE PASSED:** `tests/test_neural_validation.py` — histogram lineage reproduces
|
||
Pred. 1 (neutral decay, <3% rel err), Pred. 3 (exact `H_eq`, <5%), and tracks Layer-1
|
||
`run_lineage` directly (<3%). The neural plumbing reproduces the analytic core.
|
||
- **Plumbing:** `neural/experiment.py` (`run_and_save` dispatch on `kind`, reuses `_apply_param`
|
||
g→m, paired seeds); extended `knowledge.experiment.save_artifacts` (optional `extra_libs`,
|
||
`extra_manifest`, injectable `grid`; skips missing libs — backward compatible). `configs/neural/bridge.yaml`,
|
||
Makefile `neural`/`env-neural`/`layer2` targets, `.gitignore`. (Experiments are named
|
||
descriptively — `bridge`, `collapse`, `grounding`, `architectures` — not by code.)
|
||
- **`bridge` result (17s):** neural **g\* = 0.0474, CI [0.045, 0.052]** — reproduces Layer-1 E2's
|
||
g\*=0.048 essentially exactly (g=0.005→67% of H*, g=0.05→96%). **89 tests green.**
|
||
|
||
**2026-07-04 — Stage C: torch models + collapse/grounding/architectures.**
|
||
|
||
- **Env:** torch **2.12.1+cu130** (default PyPI wheel ships CUDA 13, matches RTX A4000 driver;
|
||
no custom index needed, cp314 wheels exist). `--extra neural` = torch only; `--extra mnist` =
|
||
torchvision (later). `UV_CACHE_DIR=/tmp` during install (RAM-backed) to spare `/home`.
|
||
- **Models:** `torch_models.py` (RNNGenerator, autoregressive GRU), `torch_mlp.py` (autoregressive
|
||
MLP, causal-masked), `torch_vae.py` (sequence VAE), `train.py` (determinism flags + device/seed
|
||
helpers derived from the SeedSequence stream). `tests/test_neural_torch.py` (torch-gated): gen-0
|
||
fidelity (rnn+mlp) + dry-collapse/grounded-holds. **92 tests green.**
|
||
- **Validated regime:** K=256, n=200, zipf_s=1.3, RNN hidden=128/epochs=25. RNN gen-0 fidelity
|
||
KL(p*‖p̂)=0.008, 64/64 (or 256/256) modes recovered. MLP fidelity KL=0.011. **VAE does NOT clear
|
||
the gen-0 gate** on the Zipf-codeword task (KL≈0.8; prior-hole mismatch — sampling z~N(0,I) misses
|
||
the aggregate posterior) → excluded from `architectures` to avoid confounding collapse with underfitting.
|
||
- **`collapse` (in weights):** dry RNN lineage collapses — forward-KL rises to ~2.2 vs grounded
|
||
~1.4; grounding lifts tail survival (tailalive 0.31 dry → 0.50 at g=0.02). Sign confirmed.
|
||
- **`grounding` (neural phase boundary):** stationary H hovers 80–91% of H* and is **noisy / non-monotonic**
|
||
at 5 reps — no crisp g*. **KEY FINDING:** the neural models' smoothing inductive bias *partially
|
||
resists* H-collapse (dry H stays ~83% of H*), so **forward-KL and tail survival are the sharp
|
||
neural collapse metrics, not H** (mirrors Layer-1's "H is smooth; the threshold lives in tail
|
||
survival"). `grounding` needs (a) forward-KL as the phase metric, (b) more reps (≥10), and/or (c) a
|
||
stronger-collapse regime for a clean neural g*.
|
||
- **`architectures` (architecture-generality) — clean result:** collapse + grounding-rescue appear in ALL three
|
||
model classes (dry→grounded forward-KL: histogram 6.2→4.6, MLP 4.8→1.3, RNN 3.8→1.1; tailalive
|
||
RNN 0.41→0.64, MLP 0.07→0.20). The WF operator is architecture-general. Bonus: neural smoothing
|
||
lets RNN/MLP retain *more* tail than the exact histogram under grounding (they generalise to
|
||
unseen codewords) — an inductive-bias finding worth the write-up.
|
||
|
||
**2026-07-04 — `recombination` (load-bearing E4 replication).**
|
||
|
||
- `recombine.py` mirrors `run_coverage` but trains K_T specialist RNNs on assignments from the
|
||
exact shared-switch retention construction (K_T/rho/q clean; union matches the closed form), then
|
||
recombines the *measured* teacher distributions two ways: `mean` (naive pooling) vs `max`
|
||
(oracle-guided union / M2N2-style), each followed by size-n resampling. The neural merge is the
|
||
per-mode max over teacher distributions (oracle-guided), NOT weight-averaging of RNNs.
|
||
- **Result (8 reps):** at rho=0, **union rises 0.49→0.96** (supply matches closed form); analytic
|
||
**surviving_max rises 0.043→0.087 while surviving_mean stays flat ~0.045** — the conservation law
|
||
(averaging cancels the union gain; max-merge realises it). At rho=1 (identical teachers) union AND
|
||
max are flat — more identical teachers buy nothing. **The "merge, don't average" lesson holds in
|
||
the neural setting.** Trained-weight columns show the same signs but noisier: neural smoothing
|
||
inflates baseline survival and the deep tail barely clears n=200 resampling (compresses magnitude)
|
||
— the expected inductive-bias caveat. torch-gated test added. **93 tests green.**
|
||
|
||
**2026-07-05 — `grounding` refinement + figure (honest reframing).**
|
||
|
||
- Re-ran at **18 reps** (n_eval 15000, 30 gens, g grid refined to 9 points). Falsifier pinned in
|
||
the config *before* running.
|
||
- **forward-KL is the operative neural collapse metric — NOT H or tail-survival.** The RNN's
|
||
smoothing keeps spurious tail modes alive, so `tail_truth_mass_alive` is flat/**non-monotone**
|
||
in g (dry 0.54 > most grounded) and H stays 0.77–0.85 of H\*. Stationary **forward-KL** falls
|
||
monotonically (dry 2.08 → g=0.2: 0.75), significant at g≥0.05 (paired t→3.3; 89% of lineages
|
||
improve at g=0.2). Refines the earlier "forward-KL AND tail survival" note.
|
||
- **The sharp `g*≪1` is an exact-operator feature, softened by neural smoothing.** Median-recovery
|
||
grounding (half the KL gap closed) g≈**0.04** (bootstrap CI [0.004, 0.116]) — echoes Layer-1's
|
||
0.048 — but full (95%) recovery needs g≈0.19. Quantitative `g*≪1` is carried by the histogram
|
||
**bridge** (0.047); the RNN confirms the SIGN and softens the sharpness (blueprint §3.5 met).
|
||
- **Honest note:** the pre-registered 95%-of-H\*/tail-survival falsifier is *not* met — because
|
||
those are the wrong metrics for a smoothing model, not because grounding fails. Reported as such.
|
||
- **Robustness fix:** a fully-degenerate RNN can emit only invalid codewords → `measure_distribution`
|
||
now returns a terminal-collapse sentinel (fixation on the dominant mode) instead of crashing a long
|
||
sweep. Edge-case test added. `figures/plot_grounding.py` written (4-panel, states its own verdict),
|
||
wired into `make figures` (glob all `plot_*.py` except `plot_E[1-6]`).
|
||
|
||
**2026-07-05 — neural figures (all five).**
|
||
|
||
- `figures/plot_{bridge,collapse,grounding,architectures,recombination}.py`, each a pure function
|
||
of its committed bundle (reuse `figures/_figlib.py`), wired into `make figures` (glob all
|
||
`plot_*.py` except `plot_E[1-6]` / `_*`). **bridge**: neural histogram runner sits exactly on the
|
||
exact `H_eq` curve, g*=0.047 (HARD-GATE visual). **collapse**: dry GRU forward-KL climbs, grounded
|
||
held; H barely moves. **architectures**: grouped bars — forward-KL falls / tail survival rises with
|
||
grounding across histogram/GRU/MLP. **recombination**: union matches closed form; max-merge rises
|
||
while mean-distill stays flat (analytic + trained + rho=1 control). **grounding**: the reframed
|
||
4-panel (forward-KL phase boundary, recovery, metric-choice).
|
||
|
||
**2026-07-05 — real-MNIST external-validity tier (`mnist_collapse`).**
|
||
|
||
- New image tier plugged into the existing contract (metrics/grounding/output are data-agnostic and
|
||
reused verbatim): `mnist_data.py` (load, per-class thickness bins, mode = class×thickness bijection,
|
||
`MnistSampler`), `mnist_oracle.py` (frozen CNN + deterministic thickness = `ClassifierOracle`,
|
||
confusion matrix), `mnist_vae.py` (`ConvVAEGenerator`), `mnist_loop.py` (`run_mnist_lineage`), plus
|
||
`kind=mnist_lineage` dispatch in `experiment.py`, `configs/neural/mnist_collapse.yaml`,
|
||
`figures/plot_mnist.py`, `figures/mnist_montage.py` (eyeball diagnostic), `MnistCfg`/`OracleCfg`.
|
||
- **Gates:** CNN mode accuracy **98.5%** (30×30 confusion matrix in the manifest = noise floor);
|
||
VAE gen-0 recovers full 30/30 support (over-smooths freq, KL≈0.5, no prior hole).
|
||
- **Result (4 reps):** dry (g=0) VAE **collapses to a single mode** (KL 0.5→18, support 30→1, tail
|
||
1.0→0.06, H→0); **g=0.1 holds all 30 modes** (KL≈0.6, full tail, H≈0.9). Collapse + grounding-rescue
|
||
confirmed on real images. VAE needs ~10% grounding vs synthetic ~5% (cf. the `grounding` finding).
|
||
**99 tests green** (+5 torchvision-gated). `make mnist` / `make env-mnist` (kept out of `make neural`).
|
||
|
||
**2026-07-05 — learning kernel (Layer-1 extension) + Riis positioning.**
|
||
|
||
- Prompted by revisiting Layer 1 vs 1.5 and the Riis 2026 paper (arXiv:2604.08554). Added
|
||
`knowledge/kernel.py` (`LearningKernelCfg`: reset `u` = smoothing, temperature `τ` = sharpening,
|
||
floor `ε`), wired into `step.generation_step` (post-refit) / `StepCtx` / `DynamicsCfg` — **identity
|
||
by default, so the 68 Layer-1 scientific-validation + correctness tests are unchanged.**
|
||
- **Result:** neutral Wright–Fisher fails BOTH neural models, oppositely. VAE regime (n=6000,K=30):
|
||
neutral is inert, sharpening `τ=0.8` reproduces the collapse-to-one-mode. RNN regime (n=200,K=256):
|
||
neutral → H=0, mutation `u=0.006` reproduces the H-floor (~0.68). Uniform-mutation overshoots the
|
||
RNN's KL → its prior is truth-like, not uniform (honest caveat, future refinement).
|
||
- `configs/layer1/kernel_{sharpen,smooth}.yaml`, `figures/plot_kernel.py` (overlays analytic arms vs
|
||
the committed neural endpoints), READMEs, `tests/test_kernel.py` (+6). Wired into `make layer1`.
|
||
- **Strategic (see CLAUDE.md finding):** concede "collapse=drift" to Riis (prior art; cite); his
|
||
mixed environment retains OLD SYNTHETIC (no real-data injection) → pessimistic, no g* that prevents
|
||
collapse. Our defensible novelty: recombination "merge-don't-average" (flagship), the learning-kernel
|
||
axis (he flags as future work), grounding-as-immigration, real-weights+MNIST breadth, and the
|
||
Lamarckian society + vertical/cumulative C3 claim (not yet run). Reposition: from "collapse is drift"
|
||
to a population-genetic CONTROL THEORY for sustaining open-ended knowledge.
|
||
|
||
**2026-07-05 — multi-locus society frame (E7/E8): raised the ceiling to enter the society.**
|
||
|
||
- Prompted by "enter the society with a robust theoretical frame." The single-locus fixed-`p*` model
|
||
can't express "exceeding" a ceiling. Generalized knowledge to a distribution over **genotypes**
|
||
(`knowledge/genotype.py`: `L` biallelic loci, `K=2^L`, additive fitness, recombination = product of
|
||
per-locus marginals). Reuses all K-mode machinery + `make_retention_matrix` (locus mastery).
|
||
- **E8 (star, `kind: society`, `knowledge/society.py`) — the vertical claim:** decorrelated parents
|
||
recombined; **sexual merge reaches the optimum (12/12, a genotype no parent had)** as parent count
|
||
grows / `ρ→0`, while best-parent (~8.7) and mean-mixture soup (~11.6) plateau. `configs/layer1/E8.yaml`,
|
||
`plot_E8.py`, README. The Fisher–Muller effect for AI.
|
||
- **E7 (`kind: genotype_lineage`, `knowledge/genotype_lineage.py`) — advantage of sex:** sexual lineage
|
||
adapts faster than asexual (LD→0 vs LD spike). Honest: a speed advantage, not a permanent ratchet gap.
|
||
- **Metaphor shift (GG):** sexual reproduction with **unbounded parents**, not teacher→pupil (which caps
|
||
at the ceiling). Collapse = asexual degradation; cure = sex, no parent limit. Unifies E4+E6 under
|
||
evolution-of-sex theory; beyond Riis's single-locus n-grams. `tests/test_genotype.py` (+7).
|
||
Experiment dispatch (`kind` in {genotype_lineage, society}) + `make layer1` wired.
|
||
|
||
## Remaining (all optional / next)
|
||
|
||
- [ ] **The dynamic society:** an evolving *population of parents* (specialists) that ground +
|
||
recombine + select over generations — capability climbing while specialties are re-earned (the full
|
||
C3, dynamic). E7/E8 give the static + single-population pieces; the multi-parent *lineage* is next.
|
||
- [ ] **NK/epistasis landscape** (sign epistasis can make recombination harmful — the honest limit of
|
||
"sex always helps"); **multi-allelic loci**. Deepens the frame.
|
||
- [ ] **Learning-kernel refinement:** truth-like smoothing prior (`prior="truth"`) + measurement floor
|
||
for a quantitative RNN match; **multi-locus / linkage** modes (class×style) as the rigorous home for
|
||
recombination. Both enrich predictive power and separate us further from Riis's single-locus n-grams.
|
||
- [ ] **The Lamarckian society experiments** (multi-agent grounding + decorrelated specialists +
|
||
recombination + QD-selection + re-mint) and the **vertical/cumulative C3 claim** — the highest-ceiling,
|
||
wholly-novel frame; not yet entered.
|
||
- [ ] **`region_matched`** grounding (R>1), **`remint`** re-mint gate (optional).
|
||
- [ ] **VAE fidelity:** fix the prior-hole mismatch (KL-annealing / free-bits / larger latent) so it
|
||
clears the gen-0 gate, then add to `architectures`. Or document as a known limitation.
|
||
- [ ] Real-MNIST secondary tier (`ClassifierOracle` + confusion matrix; `--extra mnist`).
|
||
- [ ] `figures/plot_<name>.py` (reuse `figures/_figlib.py`); wire into `make figures`.
|
||
|
||
## Discovered during work
|
||
|
||
- **E2 grounding policy vs. the analytic H_eq:** Pred. 3's closed form is derived for *plain* immigration `Multinomial(m, p*)`. Implemented as `policy="proportional"`, and every policy reduces to it at `R=1`. E2 should therefore run at `R=1` (or `proportional`) so the phase-boundary sweep tracks the exact `H_eq`; region structure is E3's concern. Decide E2's `init` (uniform vs truth) when building its config.
|
||
- `init: {uniform|truth}` added to `TruthCfg` (uniform default, mandated by the decay conformance test). E1/E2 may want `truth` start for a clean "tail collapses from the truth" story — revisit in Phase 2.
|
||
|
||
## Potential agents
|
||
|
||
*(none proposed yet)*
|