# A controlled predictive test — pre-merge functional conflict and merge penalty

*(Renamed from "the decisive experiment" after the second external review: this is a **small-model
controlled predictive test**, not the decisive experiment delivered. The operator-choice component is
unfinished, the epistasis-specific refinement did not outperform plain disagreement, and
generalisation beyond this constructed task grid is open.)*

**Question.** Can merge failure be predicted **before merging**, and by what kind of measure? Design:
39 rows = **13 conditions × 3 seeds** (0.5B LoRA children of one frozen base; parents are retrained
per condition × seed but share task-data seeds across conditions within a seed, so rows are *not*
independent — all uncertainty below is condition-clustered). Three axes decorrelated by construction:
`conflict` (contradictory conventions on shared ambiguous prompts, private budgets fixed), `compat`
(same shared prompts, same convention — overlap and volume without conflict), `duration` (weight
divergence with zero conflict). Primary outcome (pre-registered): **merge penalty vs oracle parent
potential** (per-component best of the parents); also reported vs best-parent and mean-parent
references. All robust statistics reproduce via `figures/stats_llm_epistasis.py`; figure:
`llm_epistasis.png`.

### The predictors (measured pre-merge)
- **Confidence-weighted functional conflict** (`epi_conf`) — *a proposed proxy for merge-relevant
  interactions* (previously "operational epistasis"; renamed because bilateral confident
  contradiction measures incompatible endpoint behaviour, not non-additive interaction effects in the
  strict biological sense).
- **Raw functional disagreement** (`dis_raw`) — the unweighted rate.
- **Gradient alignment** at the shared base (the ML-literature predictor, cf. 2601.22285);
  **LoRA-delta cosine / L2** (weight geometry, exact in r-space); **cross-family accuracy**
  (performance baseline).

### The supported conclusion (stated conditionally)
> **Across this controlled task grid, pre-merge functional disagreement predicted merge penalties,
> whereas LoRA-delta cosine and L2 showed no statistically detectable association.** Gradient
> alignment carried intermediate signal, so the result is not a clean functional-versus-all-geometric
> divide, and only these selected baselines were tested.

| predictor | ρ (primary) | clustered 95% CI | held-out (LOCO) ρ |
|---|---|---|---|
| raw functional disagreement | +0.460 | [+0.04, +0.69] | +0.396 (p=0.013) |
| confidence-weighted functional conflict | +0.446 | [+0.02, +0.68] | +0.352 (p=0.028) |
| gradient alignment | −0.347 | [−0.59, −0.06] | +0.10 (n.s.) |
| delta L2 (geometry) | +0.165 | [−0.27, +0.58] | −0.09 (n.s.) |
| delta cosine (geometry) | +0.030 | [−0.46, +0.51] | −0.14 (n.s.) |
| cross-family accuracy | −0.005 | [−0.29, +0.31] | −0.435 (unstable out-of-sample) |

**Paired comparisons are not individually significant** at this sample size (e.g.
|ρ(dis_raw)| − |ρ(delta_cos)| = +0.23, CI [−0.23, +0.59]): the honest statement is that the
functional measures are *individually detectably informative* (CIs exclude zero; held-out
replication) while the geometry measures are *not distinguishable from zero* — not that functional
significantly beats geometry head-to-head.

### What the decomposition shows (and its limits)
- **Between vs within.** Much of the pooled correlation is axis discrimination (mean penalty:
  conflict 0.061 vs compat 0.005 / duration 0.011). Within the conflict axis (n=15) the functional
  measures still track (+0.59/+0.62) — but so does delta-L2 (+0.71), because *within that axis*
  conflict fraction, added-data volume, and delta growth are collinear: **within-axis identification
  is impossible by design; the identification comes from the control axes**, where the same volumes
  and L2 ranges occur with ~zero penalty.
- **Outcome-reference sensitivity.** Under the *best-parent* reference the ordering changes
  (delta-L2 +0.48 vs functional +0.34): that reference inherits parent-strength trends that track
  training volume — which geometry also tracks — coupling predictor and outcome through the
  reference rather than through merge damage. We keep the pre-registered oracle-potential primary,
  and report the sensitivity rather than hide it.
- **The compat control did the decisive work** — and it *first broke our own experiment*: in the
  initial two-axis grid the best predictor was delta-cosine (ρ=+0.60), an overlap/volume artifact
  that the control exposed (collapse to +0.03).

### Chronology (adaptive, not wholly prospective)
1. Hypotheses, predictors, primary outcome, and falsifiers written into `configs/llm/epistasis.yaml`
   **before** the first grid ran (conflict + duration axes).
2. The first grid's geometry result prompted the `compat` control axis
   (`configs/llm/epistasis_compat.yaml`, pre-registered readings in its comments) — an **amendment
   after seeing data**, run on the same seeds; no conflict/duration rows were re-run or altered.
3. The clustered-bootstrap / LOCO / multi-reference analyses were added at the second review's
   request, after all data was collected.

### Pre-registered internal prediction: NOT confirmed
Confidence weighting was predicted to beat raw disagreement as a rank predictor. It does not (paired
Δ|ρ| = −0.02, CI [−0.13, +0.06]). The weighting does double the conflict-vs-compat *level* contrast
(2.0× vs 1.5×), but the present evidence favours **functional disagreement generally, not the
DMI-specific refinement**. Accordingly: the population-genetic framework motivated the measurement
and the controls; their success does **not** validate the specifically population-genetic mechanism.

### Open (unchanged by this experiment)
Budget-matched operator choice (the soup-vs-route gap readout is noise-dominated at 0.5B);
generalisation to unfamiliar conflict structures and real task pairs; 7B replication; whether any
measured quantity deserves the name *epistasis* (non-additive interaction of combinations) rather
than *conflict*.