Layer 1 complete: E3-E6 + E2 analysis add-ons
Finishes the Layer 1 analytical core. All six experiments run with honest, publication-quality figures; 71 tests green. - E3 region-matched grounding: `grounding.exercised` knob + per-region tail survival. Matched holds the exercised region's tail (0.49) where uniform spreads thin and lets it collapse (0.07). - E4 multi-teacher recombination: `run_coverage` runner. Union coverage matches U(K_T,rho,q) exactly. Finding: mean-mixture distillation shows NO surviving benefit (a conservation law — 1/K_T dilution cancels the union gain); a union-preserving max-merge (M2N2-style) does. E4 reports both operators. - E5 QD vs greedy: greedy drives fixation (H~0.01); QD holds H at 0.48-0.88, rising with the novelty exponent. - E6 re-mint gate: `arm` multi-override sweep. Re-minting a collapsed lineage locks in divergence of KL-to-original; gating on diversity prevents it. - E2 analysis add-ons (from the companion work order, numbers verified): new analysis.py (reduce_to_stationary, critical_grounding with bootstrap CI -> g*=0.048, 95% CI [0.047,0.050]); tail_band_metrics + per-band logging; the E2 figure rebuilt as a 2x2 (defined g*+CI, g=0 flagged as a finite-time artifact, tail item-vs-mass, per-rarity-band panel). Uses truth-mass-weighted tail coverage rather than the raw (martingale) tail_mass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
a6eb9b7512
commit
1721d047fa
42 changed files with 1938 additions and 135 deletions
|
|
@ -62,22 +62,22 @@ Implement to the normative signatures in §2.7. Order chosen so each piece is un
|
|||
|
||||
## Phase 3 — E3–E6
|
||||
|
||||
- [ ] **E3 region-matched grounding.** Fixed total `m`; `uniform` vs `matched`; one designated inherited-but-unwatered region with a rare tail. Expect uniform lets that region's tail collapse; matched holds it. Per-region metrics essential. `plot_E3.py`.
|
||||
- [ ] **§2.7.1 correlated-teacher construction** — `knowledge/teachers.py`:
|
||||
- [x] **E3 region-matched grounding.** Fixed total `m`; `uniform` vs `matched`; one designated inherited-but-unwatered region with a rare tail. Expect uniform lets that region's tail collapse; matched holds it. Per-region metrics essential. `plot_E3.py`.
|
||||
- [x] **§2.7.1 correlated-teacher construction** — `knowledge/teachers.py`:
|
||||
- `make_retention_matrix(T, K_T, rho, q, rng)` — shared-switch exchangeable Bernoulli.
|
||||
- `make_correlated_teachers(...)` — retention→distributions (head kept at `p*`; tail at `p*_i` if retained else `tail_floor`; renormalise). `region_specialisation` option.
|
||||
- **Pred. 5** validation: `make_retention_matrix` reproduces marginal `q`, pairwise `ρ`, and union coverage `U(K_T,ρ,q)=T[ρq+(1−ρ)(1−(1−q)^K_T)]` to 3 decimals over a `(ρ,q)` grid.
|
||||
- [ ] **E4 multi-teacher decorrelation.** Sweep `K_T∈{1,2,3,5}`, `ρ∈[0,1]` at fixed `q`, matched budget (`n/K_T` each). Report **both** union `U` and post-distillation surviving coverage; show their gap shrinks as `g` rises. `plot_E4.py` (coverage surface over `(K_T,ρ)`).
|
||||
- [ ] **E5 QD vs greedy.** `apply_selection` (`none`/`greedy`/`qd`, pinned fitness form). Sweep novelty `α`. Expect greedy→fixation (`H→0`), qd holds `H` plateau + re-introduces tails. `plot_E5.py`.
|
||||
- [ ] **E6 re-mint gate.** Re-mint at high vs low `H`; track KL to *original* truth. Expect collapsed re-mint locks KL high forever; gated (high-H) does not. `plot_E6.py`.
|
||||
- [x] **E4 multi-teacher decorrelation.** Sweep `K_T∈{1,2,3,5}`, `ρ∈[0,1]` at fixed `q`, matched budget (`n/K_T` each). Report **both** union `U` and post-distillation surviving coverage; show their gap shrinks as `g` rises. `plot_E4.py` (coverage surface over `(K_T,ρ)`).
|
||||
- [x] **E5 QD vs greedy.** `apply_selection` (`none`/`greedy`/`qd`, pinned fitness form). Sweep novelty `α`. Expect greedy→fixation (`H→0`), qd holds `H` plateau + re-introduces tails. `plot_E5.py`.
|
||||
- [x] **E6 re-mint gate.** Re-mint at high vs low `H`; track KL to *original* truth. Expect collapsed re-mint locks KL high forever; gated (high-H) does not. `plot_E6.py`.
|
||||
|
||||
## Phase 4 — Reproducibility polish (Layer 1 slice)
|
||||
|
||||
- [ ] `configs/layer1/E1..E6.yaml` all committed with explicit params (no magic numbers in code).
|
||||
- [ ] `paper/figure_manifest.md` — the §6 claim→experiment→figure rows for Layer 1.
|
||||
- [ ] `make layer1` runs E1–E6; `make figures` regenerates all figures from committed parquet.
|
||||
- [ ] Full `test_correctness.py` (shapes, normalisation, determinism) + `test_scientific_validation.py` (Pred. 1–5) green in CI.
|
||||
- [ ] `reproduce.sh` (`uv sync` → `make test` → `make layer1` → `make figures` → write `REPRODUCED.md` diffing committed result hashes) + `README.md` reproduce section. **No container** — the committed `uv.lock` is the reproducibility source of truth (per GG, 2026-07-04); a Dockerfile may later wrap the same lockfile for Layer 2's GPU work.
|
||||
- [x] `configs/layer1/E1..E6.yaml` all committed with explicit params (no magic numbers in code).
|
||||
- [x] `paper/figure_manifest.md` — the §6 claim→experiment→figure rows for Layer 1.
|
||||
- [x] `make layer1` runs E1–E6; `make figures` regenerates all figures from committed parquet.
|
||||
- [x] Full `test_correctness.py` (shapes, normalisation, determinism) + `test_scientific_validation.py` (Pred. 1–5) green in CI.
|
||||
- [x] `reproduce.sh` (`uv sync` → `make test` → `make layer1` → `make figures` → write `REPRODUCED.md` diffing committed result hashes) + `README.md` reproduce section. **No container** — the committed `uv.lock` is the reproducibility source of truth (per GG, 2026-07-04); a Dockerfile may later wrap the same lockfile for Layer 2's GPU work.
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -114,6 +114,15 @@ Design decisions #1 (dataclasses now / pydantic at YAML layer), #2 (fitness `f_i
|
|||
- E2 extended 300→500 generations (GG-approved) so the g=0 arm visibly approaches 0 while g>0 arms sit on plateaus.
|
||||
- Makefile `layer1`/`figures` wired to E1–E2. `make test` still green (68).
|
||||
|
||||
**2026-07-04 — Phase 3 complete (E3–E6) + E2 analysis add-ons.**
|
||||
|
||||
- **E3** region-matched grounding: added `grounding.exercised` knob + per-region `tailalive_region_r`. Target region tail survival 0.49 (matched) vs 0.07 (uniform). Note: per-region *H* is mass-confounded — used tail-item survival instead.
|
||||
- **E4** multi-teacher recombination: bespoke `run_coverage` runner (`kind: coverage`). Union coverage matches `U(K_T,ρ,q)` exactly. **Key finding (GG-approved): mean-mixture distillation gives NO surviving benefit (conservation law — dilution cancels the union gain); max-merge (M2N2-style) does.** E4 reports both. In CLAUDE.md.
|
||||
- **E5** QD vs greedy: greedy → H≈0.01 (fixation); qd holds H 0.48–0.88 rising with α. qd ≫ greedy.
|
||||
- **E6** re-mint gate: added `arm` multi-override sweep type. Re-mint while collapsed → KL-to-original diverges (lock-in) + accelerates H collapse; diversity gate (H≥0.75) blocks it → bounded; healthy re-mint harmless.
|
||||
- **E2 analysis add-ons** (companion work order `tasks/workorder-E2-analysis-addons.md`, verified): new `analysis.py` (`reduce_to_stationary`, `critical_grounding` bootstrap CI) — real E2 **g*=0.048, CI [0.047,0.050]**; `metrics.tail_band_metrics` + per-band lineage logging; `tests/test_analysis.py` reproduces the work order's verified numbers exactly. E2 figure rebuilt 2×2. **Deviation:** used truth-mass-weighted tail coverage instead of raw `tail_mass` (a drift martingale).
|
||||
- All six figures regenerate via `make figures`; **71 tests green**.
|
||||
|
||||
## Discovered during work
|
||||
|
||||
- **E2 grounding policy vs. the analytic H_eq:** Pred. 3's closed form is derived for *plain* immigration `Multinomial(m, p*)`. Implemented as `policy="proportional"`, and every policy reduces to it at `R=1`. E2 should therefore run at `R=1` (or `proportional`) so the phase-boundary sweep tracks the exact `H_eq`; region structure is E3's concern. Decide E2's `init` (uniform vs truth) when building its config.
|
||||
|
|
|
|||
253
tasks/workorder-E2-analysis-addons.md
Normal file
253
tasks/workorder-E2-analysis-addons.md
Normal file
|
|
@ -0,0 +1,253 @@
|
|||
# Work order: E2 analysis add-ons
|
||||
|
||||
*Increment to the Lamarckian-Society Layer-1 build. Assumes the blueprint conventions
|
||||
(seeded RNG, config-driven, results in tidy DataFrames, figures are pure functions of
|
||||
`results.parquet`). All code below was written and tested against NumPy 2.x / pandas 3.x
|
||||
before hand-off; the "verified numbers" section gives the expected outputs so you can
|
||||
confirm your wiring reproduces them.*
|
||||
|
||||
## Context
|
||||
|
||||
E1 and E2 are validated: E1 reproduces the analytic geometric decay `H0(1-1/n)^t`
|
||||
(tail-first, support collapse, KL divergence); E2's stationary simulation points lie on
|
||||
the exact closed form `H_eq = H* * m(2n+m-1)/(n+2nm+m^2)`. Three refinements make the E2
|
||||
figure publication-honest. None change the dynamics; they are analysis + plotting only.
|
||||
|
||||
1. **Operational `g*` with a bootstrap CI.** The middle E2 panel is a *smooth saturating*
|
||||
curve, not a sharp transition, so "critical g*" must be *defined*, not asserted:
|
||||
`g*` = the grounding fraction at which stationary `H` first reaches `frac`×`H*`
|
||||
(default `frac=0.95`). Report it with a percentile-bootstrap CI over replicates.
|
||||
2. **Tail-mass companion curve.** The right E2 panel plots fraction of tail *items* alive,
|
||||
which saturates low (~6%) because the deep Zipf tail is unrescuable at any feasible
|
||||
grounding. Add the frequency-weighted **tail-mass-retained** curve alongside it: mass
|
||||
is carried by the shallow tail and *is* rescued, so it looks healthy where item-count
|
||||
does not. Both are correct; showing both is the honest picture. (`tail_mass` is already
|
||||
a logged metric — this is a second series, no new storage.)
|
||||
3. **`g=0` point is a finite-time artifact.** In the middle panel it sits at `H≈0.10`,
|
||||
above the exact `H_eq(0)=0`, because at `g=0` there is no stationary state (still
|
||||
collapsing at the last generation). Either exclude `g=0` from the stationary fit or
|
||||
annotate that its true value is 0; don't let it read as a data–theory miss.
|
||||
|
||||
**Optional but recommended (turns point 2 into a mechanism panel):** a per-rarity-band
|
||||
tail metric showing the deep band stays dead while the shallow band recovers — the
|
||||
`m·p*_i ≳ 1` threshold acting band-wise, and the direct motivation for E4/E6.
|
||||
|
||||
---
|
||||
|
||||
## 1. New module: `src/knowledge/analysis.py`
|
||||
|
||||
Post-hoc analysis of E2 results. Pure NumPy/pandas, seeded, deterministic.
|
||||
|
||||
```python
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
|
||||
def reduce_to_stationary(df, value_col="heterozygosity", sweep_col="g",
|
||||
replicate_col="seed", gen_col="generation", last_frac=0.33):
|
||||
"""Per-generation frame -> one stationary value per (sweep, replicate), averaging
|
||||
`value_col` over the final `last_frac` of generations. Use for any logged metric
|
||||
(heterozygosity, tail_mass, ...)."""
|
||||
rows = []
|
||||
for (gval, rep), sub in df.groupby([sweep_col, replicate_col]):
|
||||
v = sub.sort_values(gen_col)[value_col].to_numpy()
|
||||
k = max(1, int(round(last_frac * v.size)))
|
||||
rows.append({sweep_col: gval, replicate_col: rep, value_col: v[-k:].mean()})
|
||||
return pd.DataFrame(rows)
|
||||
|
||||
|
||||
def _interp_crossing(g, H, target):
|
||||
"""First upward crossing of `target` by the (monotone-ish) curve H(g), by linear
|
||||
interpolation between grid points. Returns (g_star, status) with status in
|
||||
{'ok', 'below_grid', 'above_grid'}."""
|
||||
g = np.asarray(g, float); H = np.asarray(H, float)
|
||||
o = np.argsort(g); g, H = g[o], H[o]
|
||||
if H[0] >= target:
|
||||
return g[0], "below_grid" # already above at smallest g swept
|
||||
idx = np.where(H >= target)[0]
|
||||
if idx.size == 0:
|
||||
return g[-1], "above_grid" # never reaches target within swept range
|
||||
i = idx[0]
|
||||
g0, g1, H0, H1 = g[i - 1], g[i], H[i - 1], H[i]
|
||||
if H1 == H0:
|
||||
return g1, "ok"
|
||||
return g0 + (target - H0) * (g1 - g0) / (H1 - H0), "ok"
|
||||
|
||||
|
||||
def critical_grounding(stationary_df, H_star, frac=0.95, sweep_col="g",
|
||||
value_col="heterozygosity", n_boot=2000,
|
||||
ci=(2.5, 97.5), seed=0):
|
||||
"""Operational critical grounding fraction g*: the g at which stationary
|
||||
heterozygosity first reaches `frac` * `H_star`, with a percentile-bootstrap CI
|
||||
over replicates.
|
||||
|
||||
`stationary_df`: one row per (sweep_col, replicate_col) with the stationary value
|
||||
(e.g. the output of reduce_to_stationary). `H_star`: heterozygosity of the truth,
|
||||
= metrics.heterozygosity(p_star). Returns a dict with g_star (point estimate on the
|
||||
replicate means), ci_low, ci_high, status, target_H, frac, n_boot.
|
||||
|
||||
Note: 'status' flags right/left censoring. If the sweep does not bracket the target,
|
||||
widen the g grid rather than trusting a censored g*."""
|
||||
target = frac * H_star
|
||||
gs = np.sort(stationary_df[sweep_col].unique())
|
||||
by_g = {gv: stationary_df.loc[stationary_df[sweep_col] == gv, value_col].to_numpy()
|
||||
for gv in gs}
|
||||
mean_H = np.array([by_g[gv].mean() for gv in gs])
|
||||
g_star, status = _interp_crossing(gs, mean_H, target)
|
||||
|
||||
rng = np.random.default_rng(seed)
|
||||
boots = np.empty(n_boot)
|
||||
for b in range(n_boot):
|
||||
Hb = np.array([rng.choice(by_g[gv], by_g[gv].size, replace=True).mean()
|
||||
for gv in gs])
|
||||
boots[b], _ = _interp_crossing(gs, Hb, target)
|
||||
lo, hi = np.percentile(boots, ci)
|
||||
return {"g_star": float(g_star), "ci_low": float(lo), "ci_high": float(hi),
|
||||
"status": status, "target_H": float(target), "frac": frac,
|
||||
"n_boot": n_boot}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. New online metric in `src/knowledge/metrics.py` (optional band panel)
|
||||
|
||||
Compute this each generation from the current `p` and log the per-band arrays exactly
|
||||
like the existing per-region metrics (e.g. columns `tail_frac_alive_band{b}` and
|
||||
`tail_mass_alive_band{b}`, or long form). Both returned quantities are bounded in [0,1]
|
||||
— do **not** use a raw mass ratio (tiny deep-band denominators make it explode).
|
||||
|
||||
```python
|
||||
def tail_band_metrics(p, p_star, tail_mask, n_bands=4, alive_eps=1e-9):
|
||||
"""Stratify the tail into `n_bands` equal-count rarity bands (band 0 = rarest /
|
||||
deepest). Return two BOUNDED [0,1] arrays of length n_bands:
|
||||
frac_alive[b] fraction of band-b items with p > alive_eps
|
||||
truth_mass_alive[b] share of band-b's TRUE mass (sum p_star) carried by
|
||||
still-alive items
|
||||
Averaged over replicates, deeper bands sit strictly below shallower ones and the
|
||||
gap narrows as grounding rises -- the per-item m*p*_i >~ 1 survival threshold made
|
||||
visible (blueprint prediction 4). Single-run values are noisy; always average over
|
||||
replicates before plotting."""
|
||||
import numpy as np
|
||||
idx = np.where(tail_mask)[0]
|
||||
order = idx[np.argsort(p_star[idx])] # rarest first
|
||||
bands = np.array_split(order, n_bands)
|
||||
frac_alive = np.empty(n_bands)
|
||||
truth_mass_alive = np.empty(n_bands)
|
||||
for b, items in enumerate(bands):
|
||||
alive = p[items] > alive_eps
|
||||
frac_alive[b] = alive.mean()
|
||||
ps = p_star[items]
|
||||
truth_mass_alive[b] = ps[alive].sum() / ps.sum() if ps.sum() > 0 else np.nan
|
||||
return frac_alive, truth_mass_alive
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Figure updates: `figures/plot_E2.py`
|
||||
|
||||
- **Middle panel:** call `critical_grounding(reduce_to_stationary(df), H_star, frac=0.95)`
|
||||
and draw a vertical line/marker at `g_star` with a shaded CI band; annotate
|
||||
`g* ≈ {g_star:.3f} (95% CI [...])`. Also report `frac=0.90` in the caption so the
|
||||
"3–5%" range is explicit. Soften the title from "critical grounding ratio" to
|
||||
e.g. "grounding saturates by g* ≈ 0.05 (95% of H*)".
|
||||
- **Right panel:** add stationary **tail-mass-retained** vs g as a second series
|
||||
(`reduce_to_stationary(df, value_col="tail_mass")`), on a twin axis if scales differ,
|
||||
so mass-healthy vs items-poor is visible in one panel.
|
||||
- **`g=0`:** drop it from the stationary fit *or* mark it hollow with a note
|
||||
"pre-convergence; true H_eq(0)=0".
|
||||
- **Optional band panel:** if `tail_band_metrics` is logged, add a small-multiples or
|
||||
grouped-bar panel of `frac_alive` per band across the g sweep (replicate-averaged).
|
||||
|
||||
---
|
||||
|
||||
## 4. Tests to add (`tests/test_analysis.py`)
|
||||
|
||||
```python
|
||||
import numpy as np, pandas as pd, pytest
|
||||
from knowledge.analysis import reduce_to_stationary, critical_grounding
|
||||
from knowledge.metrics import tail_band_metrics
|
||||
|
||||
|
||||
def _H_eq(n, m, Hs): return Hs * m * (2 * n + m - 1) / (n + 2 * n * m + m * m)
|
||||
|
||||
|
||||
def _synthetic_E2(n=200, Hs=0.95, reps=100, seed=1):
|
||||
gs = [0.0, 0.005, 0.01, 0.02, 0.05, 0.1, 0.2, 0.4]
|
||||
rng = np.random.default_rng(seed); rows = []
|
||||
for g in gs:
|
||||
m = 0 if g == 0 else int(round(g * n / (1 - g)))
|
||||
Htrue = 0.10 if g == 0 else _H_eq(n, m, Hs) # g=0: finite-time artifact
|
||||
for s in range(reps):
|
||||
rows.append({"g": g, "seed": s, "heterozygosity": Htrue + rng.normal(0, 0.004)})
|
||||
return pd.DataFrame(rows)
|
||||
|
||||
|
||||
def test_critical_grounding_matches_known_crossing():
|
||||
stat = _synthetic_E2()
|
||||
r = critical_grounding(stat, H_star=0.95, frac=0.95, seed=7)
|
||||
assert r["status"] == "ok"
|
||||
assert 0.03 < r["g_star"] < 0.07 # ~0.047 for frac=0.95
|
||||
assert r["ci_low"] <= r["g_star"] <= r["ci_high"]
|
||||
r90 = critical_grounding(stat, H_star=0.95, frac=0.90, seed=7)
|
||||
assert r90["g_star"] < r["g_star"] # lower bar -> smaller g*
|
||||
|
||||
|
||||
def test_reduce_to_stationary_recovers_plateau():
|
||||
# constant plateau + noise -> mean ~ plateau
|
||||
rng = np.random.default_rng(0); rows = []
|
||||
for g, plateau in [(0.02, 0.843), (0.05, 0.908)]:
|
||||
for s in range(20):
|
||||
for t in range(300):
|
||||
v = (0.95 if t < 50 else plateau) + rng.normal(0, 0.003)
|
||||
rows.append({"g": g, "seed": s, "generation": t, "heterozygosity": v})
|
||||
red = reduce_to_stationary(pd.DataFrame(rows), last_frac=0.33)
|
||||
means = red.groupby("g")["heterozygosity"].mean()
|
||||
assert means[0.02] == pytest.approx(0.843, abs=0.01)
|
||||
assert means[0.05] == pytest.approx(0.908, abs=0.01)
|
||||
|
||||
|
||||
def test_tail_band_deep_below_shallow():
|
||||
# grounded dynamics on a Zipf tail: deepest band <= shallowest band, replicate-avg
|
||||
K = 1000; s = 1.1
|
||||
w = 1.0 / np.arange(1, K + 1) ** s; p_star = w / w.sum()
|
||||
tail_mask = p_star < 1e-3
|
||||
n, m = 200, 10
|
||||
FA = np.zeros(4)
|
||||
for r in range(20):
|
||||
rng = np.random.default_rng(1000 + r); p = p_star.copy()
|
||||
for _ in range(600):
|
||||
c = rng.multinomial(n, p) + rng.multinomial(m, p_star); p = c / c.sum()
|
||||
fa, _ = tail_band_metrics(p, p_star, tail_mask, n_bands=4)
|
||||
FA += fa
|
||||
FA /= 20
|
||||
assert FA[0] <= FA[-1] # deepest no better than shallowest
|
||||
assert FA[-1] > FA[0] # and strictly worse on average
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Verified numbers (expected outputs — confirm your wiring reproduces these)
|
||||
|
||||
On the synthetic E2 set (n=200, H*=0.95, exact H_eq + N(0,0.004) noise, 100 reps):
|
||||
|
||||
| frac | g* (point) | 95% CI (tight, low-noise synthetic) |
|
||||
|------|-----------|--------------------------------------|
|
||||
| 0.90 | ~0.026 | ~[0.025, 0.026] |
|
||||
| 0.95 | ~0.047 | ~[0.047, 0.048] |
|
||||
| 0.99 | ~0.25 | ~[0.22, 0.27] |
|
||||
|
||||
(Real E2 replicate spread will widen these CIs — that is expected and correct.)
|
||||
|
||||
`reduce_to_stationary` on a plateau frame recovers 0.843 (g=0.02) and 0.908 (g=0.05).
|
||||
|
||||
`tail_band_metrics`, grounded Zipf tail (K=1000, tail=p*<1e-3, 889 tail items), 40 reps,
|
||||
`frac_alive` per band [0=deepest .. 3=shallowest]:
|
||||
|
||||
| g | band0 | band1 | band2 | band3 |
|
||||
|------|-------|-------|-------|-------|
|
||||
| 0.0 | 0.000 | 0.000 | 0.000 | 0.000 |
|
||||
| 0.01 | 0.002 | 0.002 | 0.004 | 0.010 |
|
||||
| 0.05 | 0.004 | 0.009 | 0.012 | 0.033 |
|
||||
| 0.20 | 0.015 | 0.022 | 0.032 | 0.075 |
|
||||
|
||||
Monotone (deeper = worse) at every g>0; all bands rise with g; deep band lags throughout.
|
||||
Loading…
Add table
Add a link
Reference in a new issue