second review round: tempered claims, robust statistics, corrected technical statements

Analyses (figures/stats_llm_epistasis.py, committed + reproducible):
condition-clustered bootstrap CIs (functional measures exclude zero:
dis_raw [+0.04,+0.69], conf-weighted [+0.02,+0.68]; gradient alignment
[-0.59,-0.06]; geometry straddles zero), PAIRED predictor contrasts (not
individually significant — stated), leave-one-condition-out held-out
prediction (functional replicates, geometry ~0, performance baseline
unstable), three outcome references (ordering sensitive to reference —
reported, with the mechanism), between/within-axis decomposition
(within-conflict identification impossible by design; the compat axis
identifies), and seed-level paired reliability (routing/directed beat
soup 3/3 seeds incl. one catastrophic soup failure; CI-width fragility
claim withdrawn).

Renames and corrections: "decisive experiment" -> "controlled predictive
test"; "operational epistasis" -> "confidence-weighted functional
conflict (proposed proxy)"; "functional by construction" -> "controls a
major source of coordinate mismatch / conflict-associated" (module,
configs, READMEs, figures); SI proposition's "chord" defined precisely
(endpoint-loss interpolation, invariant) vs the path (not invariant) +
no-global-optimality caveat (removable = lower bound, residual = upper);
snowball count != performance cliff distinction added; claims table
gains four rows (grid finding / weighting NOT supported / functional-vs-
all-geometry not established / operator choice open); §1 ladder states
the prediction rung as a bounded small-model result.

paper/response-to-review-2.md: point-by-point, opening with the
bookkeeping correction (E13b/c were in the reviewed draft — revised
interpretation, not new results). READMEs rewritten around the four
analyses with the chronology (prospective/adaptive/post-hoc) disclosed.
151 tests green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkRLcc18rwT2Lysu6PbG7v

results/llm_epistasis/README.md

@@ -1,52 +1,86 @@
-# The decisive experiment — predicting merge failure BEFORE merging
+# A controlled predictive test — pre-merge functional conflict and merge penalty
 
-The external review's bar (2026-08-11): population-genetic quantities must *predict*, not
-re-describe — forecast merge success **pre-merge** and beat existing predictors. Design: 39 parent
-pairs (0.5B LoRA children of one frozen base, 3 seeds) on **three axes decorrelated by
-construction** — `conflict` (contradictory conventions on shared ambiguous prompts, private budgets
-fixed), `compat` (the control: **same shared prompts, same convention** — task overlap *without*
-conflict; added after the first grid exposed a confound, see below), and `duration` (weight
-divergence with zero conflict, 1→12 epochs). Primary outcome (pre-registered):
-**merge penalty** = parent potential − merged achieved (the hybrid-load analogue). Figure:
+*(Renamed from "the decisive experiment" after the second external review: this is a **small-model
+controlled predictive test**, not the decisive experiment delivered. The operator-choice component is
+unfinished, the epistasis-specific refinement did not outperform plain disagreement, and
+generalisation beyond this constructed task grid is open.)*
+
+**Question.** Can merge failure be predicted **before merging**, and by what kind of measure? Design:
+39 rows = **13 conditions × 3 seeds** (0.5B LoRA children of one frozen base; parents are retrained
+per condition × seed but share task-data seeds across conditions within a seed, so rows are *not*
+independent — all uncertainty below is condition-clustered). Three axes decorrelated by construction:
+`conflict` (contradictory conventions on shared ambiguous prompts, private budgets fixed), `compat`
+(same shared prompts, same convention — overlap and volume without conflict), `duration` (weight
+divergence with zero conflict). Primary outcome (pre-registered): **merge penalty vs oracle parent
+potential** (per-component best of the parents); also reported vs best-parent and mean-parent
+references. All robust statistics reproduce via `figures/stats_llm_epistasis.py`; figure:
 `llm_epistasis.png`.
 
-### The league table (Spearman ρ vs merge penalty, full three-axis pool, n = 39)
-| pre-merge predictor | ρ | p | reading |
+### The predictors (measured pre-merge)
+- **Confidence-weighted functional conflict** (`epi_conf`) — *a proposed proxy for merge-relevant
+  interactions* (previously "operational epistasis"; renamed because bilateral confident
+  contradiction measures incompatible endpoint behaviour, not non-additive interaction effects in the
+  strict biological sense).
+- **Raw functional disagreement** (`dis_raw`) — the unweighted rate.
+- **Gradient alignment** at the shared base (the ML-literature predictor, cf. 2601.22285);
+  **LoRA-delta cosine / L2** (weight geometry, exact in r-space); **cross-family accuracy**
+  (performance baseline).
+
+### The supported conclusion (stated conditionally)
+> **Across this controlled task grid, pre-merge functional disagreement predicted merge penalties,
+> whereas LoRA-delta cosine and L2 showed no statistically detectable association.** Gradient
+> alignment carried intermediate signal, so the result is not a clean functional-versus-all-geometric
+> divide, and only these selected baselines were tested.
+
+| predictor | ρ (primary) | clustered 95% CI | held-out (LOCO) ρ |
 |---|---|---|---|
-| raw functional disagreement (`dis_raw`) | **+0.460** | 0.003 | predicts |
-| operational epistasis (`epi_conf`, confidence-weighted) | **+0.446** | 0.004 | predicts |
-| gradient alignment at the base (cf. 2601.22285) | −0.347 | 0.03 | weakly informative |
-| LoRA-delta L2 distance (geometry) | +0.165 | 0.32 | uninformative |
-| LoRA-delta cosine (geometry) | +0.030 | 0.86 | uninformative |
-| cross-family accuracy (performance) | −0.005 | 0.98 | uninformative |
+| raw functional disagreement | +0.460 | [+0.04, +0.69] | +0.396 (p=0.013) |
+| confidence-weighted functional conflict | +0.446 | [+0.02, +0.68] | +0.352 (p=0.028) |
+| gradient alignment | −0.347 | [−0.59, −0.06] | +0.10 (n.s.) |
+| delta L2 (geometry) | +0.165 | [−0.27, +0.58] | −0.09 (n.s.) |
+| delta cosine (geometry) | +0.030 | [−0.46, +0.51] | −0.14 (n.s.) |
+| cross-family accuracy | −0.005 | [−0.29, +0.31] | −0.435 (unstable out-of-sample) |
 
-**Headline: functional conflict, measured before merging, predicts merge failure; weight geometry
-does not.** The duration axis spans the same weight-divergence range as the conflict axis
-(L2 ≈ 2.4–4.0) at ~zero penalty, and the compat axis adds the same *data volumes* and overlap at
-~zero penalty — so both geometric predictors collapse once overlap and volume are controlled.
+**Paired comparisons are not individually significant** at this sample size (e.g.
+|ρ(dis_raw)| − |ρ(delta_cos)| = +0.23, CI [−0.23, +0.59]): the honest statement is that the
+functional measures are *individually detectably informative* (CIs exclude zero; held-out
+replication) while the geometry measures are *not distinguishable from zero* — not that functional
+significantly beats geometry head-to-head.
 
-### The control that did the work (`llm_epistasis_compat/`)
-In the first grid (conflict + duration only), `delta_cos` scored ρ = +0.60 — apparently the best
-predictor. That was an **artifact**: every shared-data pair in that pool was a conflicted pair, so
-geometry could win as a mere task-overlap/volume detector. The `compat` axis (overlap without
-conflict) exposes it: penalty ≈ 0.005 there, and the geometry correlations collapse (+0.60 → +0.03).
-The functional measures behave correctly on the control — parents trained on the same convention
-*agree* on the shared prompts (epi_conf: 0.46 conflict vs 0.23 compat, a 2× contrast; raw
-disagreement 0.72 vs 0.47, only 1.5× — the confidence weighting removes complementation noise from
-the *level*, giving the cleaner axis separation).
+### What the decomposition shows (and its limits)
+- **Between vs within.** Much of the pooled correlation is axis discrimination (mean penalty:
+  conflict 0.061 vs compat 0.005 / duration 0.011). Within the conflict axis (n=15) the functional
+  measures still track (+0.59/+0.62) — but so does delta-L2 (+0.71), because *within that axis*
+  conflict fraction, added-data volume, and delta growth are collinear: **within-axis identification
+  is impossible by design; the identification comes from the control axes**, where the same volumes
+  and L2 ranges occur with ~zero penalty.
+- **Outcome-reference sensitivity.** Under the *best-parent* reference the ordering changes
+  (delta-L2 +0.48 vs functional +0.34): that reference inherits parent-strength trends that track
+  training volume — which geometry also tracks — coupling predictor and outcome through the
+  reference rather than through merge damage. We keep the pre-registered oracle-potential primary,
+  and report the sensitivity rather than hide it.
+- **The compat control did the decisive work** — and it *first broke our own experiment*: in the
+  initial two-axis grid the best predictor was delta-cosine (ρ=+0.60), an overlap/volume artifact
+  that the control exposed (collapse to +0.03).
 
-### Honest riders (pre-registered falsifier status)
-1. The internal prediction that confidence weighting would beat raw disagreement **as a rank
-   predictor is not confirmed**: `epi_conf` and `dis_raw` are statistically indistinguishable at
-   n = 39 (the weighting does improve the conflict-vs-compat *contrast* in levels). The paper reports
-   the functional-vs-geometric verdict, not a win for the refinement.
-2. Correlations are moderate (|ρ| ≈ 0.45), bounded by 0.5B merge-outcome noise (soup merges carry
-   large intrinsic seed variance — see `llm_moe_hard_seeds`); read as signs and ordering, not
-   magnitudes. 7B replication is the natural firm-up.
-3. Gradient alignment carries real signal (it differentiates conflicting conventions at the base) but
-   less than the functional measures in this design.
+### Chronology (adaptive, not wholly prospective)
+1. Hypotheses, predictors, primary outcome, and falsifiers written into `configs/llm/epistasis.yaml`
+   **before** the first grid ran (conflict + duration axes).
+2. The first grid's geometry result prompted the `compat` control axis
+   (`configs/llm/epistasis_compat.yaml`, pre-registered readings in its comments) — an **amendment
+   after seeing data**, run on the same seeds; no conflict/duration rows were re-run or altered.
+3. The clustered-bootstrap / LOCO / multi-reference analyses were added at the second review's
+   request, after all data was collected.
 
-**Bottom line for the paper:** the framework's claim — *epistasis (functional conflict), not
-divergence, sets merge compatibility* — survives its designed falsification test at this tier: the
-operational conflict measures predict, the divergence measures do not, and the case was made honest
-by a control that first *broke our own experiment's* favourite-looking geometric predictor.
+### Pre-registered internal prediction: NOT confirmed
+Confidence weighting was predicted to beat raw disagreement as a rank predictor. It does not (paired
+Δ|ρ| = −0.02, CI [−0.13, +0.06]). The weighting does double the conflict-vs-compat *level* contrast
+(2.0× vs 1.5×), but the present evidence favours **functional disagreement generally, not the
+DMI-specific refinement**. Accordingly: the population-genetic framework motivated the measurement
+and the controls; their success does **not** validate the specifically population-genetic mechanism.
+
+### Open (unchanged by this experiment)
+Budget-matched operator choice (the soup-vs-route gap readout is noise-dominated at 0.5B);
+generalisation to unfamiliar conflict structures and real task pairs; 7B replication; whether any
+measured quantity deserves the name *epistasis* (non-additive interaction of combinations) rather
+than *conflict*.

results/llm_merge_seeds/README.md

@@ -18,18 +18,22 @@ the single-seed run — decisive at 7B) and the **worst-family signature is unam
 vs ≤0.16 for any parent — only recombined models are competent everywhere.
 
 ### (B) Union vs fusion, hard benchmark, 3 seeds (`llm_moe_hard_seeds/`)
-Routing (union) 0.274 ± 0.026 overall / 0.238 ± 0.024 worst-family; fusion soup 0.174 ± 0.102 / 0.088
-± 0.093; ties similar; best specialist 0.199 ± 0.026. Union beats fusion on both metrics — **and a new
-finding: fusion is seed-FRAGILE on hard tasks (CI ±0.10) while routing is seed-stable (±0.026).**
-Averaging's outcome depends on which specialist minima the seeds happened to find; selection-based
-recombination is reliable. (Learned router still = oracle: lexically distinct families, known rider.)
+Routing (union) beats fusion **in every seed** (3/3 paired, both metrics; e.g. overall per seed:
+routing 0.258/0.300/0.262 vs soup 0.212/0.238/0.071). The paired per-seed values also show *why*:
+seed 3's soup failed catastrophically (0.071 overall, 0.000 worst-family) while routing was unaffected
+(0.262/0.225). Seed-level sd: soup 0.090 vs routing 0.023 — suggestive of a genuine variance
+difference, but with 3 seeds this is an observation, not an estimate; the robust statement is the
+3/3 paired ordering plus the observed catastrophic-failure mode that routing is structurally immune
+to. (Learned router still = oracle: lexically distinct families, known rider.)
 
 ### (C) Directed offspring selection, hard, 3 seeds (`llm_directed_hard_seeds/`)
-directed_overall 0.221 ± 0.026 (> soup 0.174 ± 0.102 and > best specialist); directed_balanced
-worst-family 0.158 ± 0.036 (> soup 0.088 ± 0.093). Directed selection both beats and **stabilises**
-the a-priori soup; per-input routing (B) remains above any single global blend, as before.
+directed_overall beats the a-priori soup **in every seed** (3/3 paired; 0.225/0.242/0.196 vs
+0.212/0.238/0.071 — including rescuing soup's catastrophic seed); directed_balanced worst-family
+0.158 vs soup 0.088. Directed selection both beats and stabilises the blend (same 3-seed caveat as
+panel B); per-input routing remains above any single global blend, as before.
 
-**Read together:** all three recombination claims hold under seed replication, and the operator
-ordering (route > directed-select > soup, on headroom tasks) is not only a mean effect but a
-*variance* effect — the union/selection operators are the reliable ones. Base: Qwen2.5-0.5B-Instruct;
+**Read together:** all three recombination claims hold under seed replication with consistent paired
+ordering (route > directed-select > soup in every seed on headroom tasks), and the per-seed values
+surface a failure mode — occasional catastrophic soup merges — that the union/selection operators
+avoided in every observed case. Variance *estimates* await more seeds. Base: Qwen2.5-0.5B-Instruct;
 statistical (per-seed) reproducibility per blueprint §4.

results/llm_moe_hard_seeds/README.md

@@ -1,7 +1,7 @@
 # Multi-seed union-vs-fusion, hard benchmark (0.5B, 3 seeds)
 
-Part of the multi-seed firm-up; full legend, table, and the fusion-fragility finding in
-`results/llm_merge_seeds/README.md` (panel B of its `llm_seeds.png`). Headline: union/routing
-0.274 ± 0.026 > fusion 0.174 ± 0.102 overall (worst-family 0.238 vs 0.088), and fusion's ±0.10 CI vs
-routing's ±0.026 is itself the finding — averaging is seed-fragile where headroom exists; routing is
-reliable.
+Part of the multi-seed firm-up; full legend and per-seed table in `results/llm_merge_seeds/README.md`
+(panel B of its `llm_seeds.png`). Headline: routing beats fusion in every seed (3/3 paired, both
+metrics), and one seed exhibited a catastrophic soup failure (0.071 overall / 0.000 worst-family) that
+routing was immune to (0.262/0.225). With 3 seeds the variance contrast (sd 0.090 vs 0.023) is an
+observation, not an estimate.

results/llm_speciation/README.md

@@ -1,8 +1,10 @@
 # LLM-tier model speciation — conflict provokes isolation; duration alone does not
 
-E13 carried into language-model weights (0.5B Qwen, LoRA children of one frozen base — which shares
-its coordinate system with both children, so **there is no permutation/rescaling ambiguity by
-construction: every merge failure here is functional**). Two knobs, pre-registered readings in the
+E13 carried into language-model weights (0.5B Qwen, LoRA children of one frozen base — the shared
+base **controls a major source of coordinate mismatch**, allowing a cleaner test of
+conflict-associated merge failure; averaging can still fail for non-conflict reasons — nonlinear
+interaction, scaling, capacity — so failures here are *conflict-associated*, not automatically
+functional). Two knobs, pre-registered readings in the
 configs; figure `llm_speciation.png` (3 panels; panel B from `results/llm_speciation_add/`).
 
 **Design.** Child A: private family `strings`; child B: private family `arith`; shared **ambiguous