second review round: tempered claims, robust statistics, corrected technical statements

Analyses (figures/stats_llm_epistasis.py, committed + reproducible):
condition-clustered bootstrap CIs (functional measures exclude zero:
dis_raw [+0.04,+0.69], conf-weighted [+0.02,+0.68]; gradient alignment
[-0.59,-0.06]; geometry straddles zero), PAIRED predictor contrasts (not
individually significant — stated), leave-one-condition-out held-out
prediction (functional replicates, geometry ~0, performance baseline
unstable), three outcome references (ordering sensitive to reference —
reported, with the mechanism), between/within-axis decomposition
(within-conflict identification impossible by design; the compat axis
identifies), and seed-level paired reliability (routing/directed beat
soup 3/3 seeds incl. one catastrophic soup failure; CI-width fragility
claim withdrawn).

Renames and corrections: "decisive experiment" -> "controlled predictive
test"; "operational epistasis" -> "confidence-weighted functional
conflict (proposed proxy)"; "functional by construction" -> "controls a
major source of coordinate mismatch / conflict-associated" (module,
configs, READMEs, figures); SI proposition's "chord" defined precisely
(endpoint-loss interpolation, invariant) vs the path (not invariant) +
no-global-optimality caveat (removable = lower bound, residual = upper);
snowball count != performance cliff distinction added; claims table
gains four rows (grid finding / weighting NOT supported / functional-vs-
all-geometry not established / operator choice open); §1 ladder states
the prediction rung as a bounded small-model result.

paper/response-to-review-2.md: point-by-point, opening with the
bookkeeping correction (E13b/c were in the reviewed draft — revised
interpretation, not new results). READMEs rewritten around the four
analyses with the chronology (prospective/adaptive/post-hoc) disclosed.
151 tests green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkRLcc18rwT2Lysu6PbG7v

results/llm_epistasis/README.md

@@ -1,52 +1,86 @@
-# The decisive experiment — predicting merge failure BEFORE merging
+# A controlled predictive test — pre-merge functional conflict and merge penalty
 
-The external review's bar (2026-08-11): population-genetic quantities must *predict*, not
-re-describe — forecast merge success **pre-merge** and beat existing predictors. Design: 39 parent
-pairs (0.5B LoRA children of one frozen base, 3 seeds) on **three axes decorrelated by
-construction** — `conflict` (contradictory conventions on shared ambiguous prompts, private budgets
-fixed), `compat` (the control: **same shared prompts, same convention** — task overlap *without*
-conflict; added after the first grid exposed a confound, see below), and `duration` (weight
-divergence with zero conflict, 1→12 epochs). Primary outcome (pre-registered):
-**merge penalty** = parent potential − merged achieved (the hybrid-load analogue). Figure:
+*(Renamed from "the decisive experiment" after the second external review: this is a **small-model
+controlled predictive test**, not the decisive experiment delivered. The operator-choice component is
+unfinished, the epistasis-specific refinement did not outperform plain disagreement, and
+generalisation beyond this constructed task grid is open.)*
+
+**Question.** Can merge failure be predicted **before merging**, and by what kind of measure? Design:
+39 rows = **13 conditions × 3 seeds** (0.5B LoRA children of one frozen base; parents are retrained
+per condition × seed but share task-data seeds across conditions within a seed, so rows are *not*
+independent — all uncertainty below is condition-clustered). Three axes decorrelated by construction:
+`conflict` (contradictory conventions on shared ambiguous prompts, private budgets fixed), `compat`
+(same shared prompts, same convention — overlap and volume without conflict), `duration` (weight
+divergence with zero conflict). Primary outcome (pre-registered): **merge penalty vs oracle parent
+potential** (per-component best of the parents); also reported vs best-parent and mean-parent
+references. All robust statistics reproduce via `figures/stats_llm_epistasis.py`; figure:
 `llm_epistasis.png`.
 
-### The league table (Spearman ρ vs merge penalty, full three-axis pool, n = 39)
-| pre-merge predictor | ρ | p | reading |
+### The predictors (measured pre-merge)
+- **Confidence-weighted functional conflict** (`epi_conf`) — *a proposed proxy for merge-relevant
+  interactions* (previously "operational epistasis"; renamed because bilateral confident
+  contradiction measures incompatible endpoint behaviour, not non-additive interaction effects in the
+  strict biological sense).
+- **Raw functional disagreement** (`dis_raw`) — the unweighted rate.
+- **Gradient alignment** at the shared base (the ML-literature predictor, cf. 2601.22285);
+  **LoRA-delta cosine / L2** (weight geometry, exact in r-space); **cross-family accuracy**
+  (performance baseline).
+
+### The supported conclusion (stated conditionally)
+> **Across this controlled task grid, pre-merge functional disagreement predicted merge penalties,
+> whereas LoRA-delta cosine and L2 showed no statistically detectable association.** Gradient
+> alignment carried intermediate signal, so the result is not a clean functional-versus-all-geometric
+> divide, and only these selected baselines were tested.
+
+| predictor | ρ (primary) | clustered 95% CI | held-out (LOCO) ρ |
 |---|---|---|---|
-| raw functional disagreement (`dis_raw`) | **+0.460** | 0.003 | predicts |
-| operational epistasis (`epi_conf`, confidence-weighted) | **+0.446** | 0.004 | predicts |
-| gradient alignment at the base (cf. 2601.22285) | −0.347 | 0.03 | weakly informative |
-| LoRA-delta L2 distance (geometry) | +0.165 | 0.32 | uninformative |
-| LoRA-delta cosine (geometry) | +0.030 | 0.86 | uninformative |
-| cross-family accuracy (performance) | −0.005 | 0.98 | uninformative |
+| raw functional disagreement | +0.460 | [+0.04, +0.69] | +0.396 (p=0.013) |
+| confidence-weighted functional conflict | +0.446 | [+0.02, +0.68] | +0.352 (p=0.028) |
+| gradient alignment | −0.347 | [−0.59, −0.06] | +0.10 (n.s.) |
+| delta L2 (geometry) | +0.165 | [−0.27, +0.58] | −0.09 (n.s.) |
+| delta cosine (geometry) | +0.030 | [−0.46, +0.51] | −0.14 (n.s.) |
+| cross-family accuracy | −0.005 | [−0.29, +0.31] | −0.435 (unstable out-of-sample) |
 
-**Headline: functional conflict, measured before merging, predicts merge failure; weight geometry
-does not.** The duration axis spans the same weight-divergence range as the conflict axis
-(L2 ≈ 2.4–4.0) at ~zero penalty, and the compat axis adds the same *data volumes* and overlap at
-~zero penalty — so both geometric predictors collapse once overlap and volume are controlled.
+**Paired comparisons are not individually significant** at this sample size (e.g.
+|ρ(dis_raw)| − |ρ(delta_cos)| = +0.23, CI [−0.23, +0.59]): the honest statement is that the
+functional measures are *individually detectably informative* (CIs exclude zero; held-out
+replication) while the geometry measures are *not distinguishable from zero* — not that functional
+significantly beats geometry head-to-head.
 
-### The control that did the work (`llm_epistasis_compat/`)
-In the first grid (conflict + duration only), `delta_cos` scored ρ = +0.60 — apparently the best
-predictor. That was an **artifact**: every shared-data pair in that pool was a conflicted pair, so
-geometry could win as a mere task-overlap/volume detector. The `compat` axis (overlap without
-conflict) exposes it: penalty ≈ 0.005 there, and the geometry correlations collapse (+0.60 → +0.03).
-The functional measures behave correctly on the control — parents trained on the same convention
-*agree* on the shared prompts (epi_conf: 0.46 conflict vs 0.23 compat, a 2× contrast; raw
-disagreement 0.72 vs 0.47, only 1.5× — the confidence weighting removes complementation noise from
-the *level*, giving the cleaner axis separation).
+### What the decomposition shows (and its limits)
+- **Between vs within.** Much of the pooled correlation is axis discrimination (mean penalty:
+  conflict 0.061 vs compat 0.005 / duration 0.011). Within the conflict axis (n=15) the functional
+  measures still track (+0.59/+0.62) — but so does delta-L2 (+0.71), because *within that axis*
+  conflict fraction, added-data volume, and delta growth are collinear: **within-axis identification
+  is impossible by design; the identification comes from the control axes**, where the same volumes
+  and L2 ranges occur with ~zero penalty.
+- **Outcome-reference sensitivity.** Under the *best-parent* reference the ordering changes
+  (delta-L2 +0.48 vs functional +0.34): that reference inherits parent-strength trends that track
+  training volume — which geometry also tracks — coupling predictor and outcome through the
+  reference rather than through merge damage. We keep the pre-registered oracle-potential primary,
+  and report the sensitivity rather than hide it.
+- **The compat control did the decisive work** — and it *first broke our own experiment*: in the
+  initial two-axis grid the best predictor was delta-cosine (ρ=+0.60), an overlap/volume artifact
+  that the control exposed (collapse to +0.03).
 
-### Honest riders (pre-registered falsifier status)
-1. The internal prediction that confidence weighting would beat raw disagreement **as a rank
-   predictor is not confirmed**: `epi_conf` and `dis_raw` are statistically indistinguishable at
-   n = 39 (the weighting does improve the conflict-vs-compat *contrast* in levels). The paper reports
-   the functional-vs-geometric verdict, not a win for the refinement.
-2. Correlations are moderate (|ρ| ≈ 0.45), bounded by 0.5B merge-outcome noise (soup merges carry
-   large intrinsic seed variance — see `llm_moe_hard_seeds`); read as signs and ordering, not
-   magnitudes. 7B replication is the natural firm-up.
-3. Gradient alignment carries real signal (it differentiates conflicting conventions at the base) but
-   less than the functional measures in this design.
+### Chronology (adaptive, not wholly prospective)
+1. Hypotheses, predictors, primary outcome, and falsifiers written into `configs/llm/epistasis.yaml`
+   **before** the first grid ran (conflict + duration axes).
+2. The first grid's geometry result prompted the `compat` control axis
+   (`configs/llm/epistasis_compat.yaml`, pre-registered readings in its comments) — an **amendment
+   after seeing data**, run on the same seeds; no conflict/duration rows were re-run or altered.
+3. The clustered-bootstrap / LOCO / multi-reference analyses were added at the second review's
+   request, after all data was collected.
 
-**Bottom line for the paper:** the framework's claim — *epistasis (functional conflict), not
-divergence, sets merge compatibility* — survives its designed falsification test at this tier: the
-operational conflict measures predict, the divergence measures do not, and the case was made honest
-by a control that first *broke our own experiment's* favourite-looking geometric predictor.
+### Pre-registered internal prediction: NOT confirmed
+Confidence weighting was predicted to beat raw disagreement as a rank predictor. It does not (paired
+Δ|ρ| = −0.02, CI [−0.13, +0.06]). The weighting does double the conflict-vs-compat *level* contrast
+(2.0× vs 1.5×), but the present evidence favours **functional disagreement generally, not the
+DMI-specific refinement**. Accordingly: the population-genetic framework motivated the measurement
+and the controls; their success does **not** validate the specifically population-genetic mechanism.
+
+### Open (unchanged by this experiment)
+Budget-matched operator choice (the soup-vs-route gap readout is noise-dominated at 0.5B);
+generalisation to unfamiliar conflict structures and real task pairs; 7B replication; whether any
+measured quantity deserves the name *epistasis* (non-additive interaction of combinations) rather
+than *conflict*.