MachineSex/results/llm_epistasis

A controlled predictive test — pre-merge functional conflict and merge penalty

(Renamed from "the decisive experiment" after the second external review: this is a small-model controlled predictive test, not the decisive experiment delivered. The operator-choice component is unfinished, the epistasis-specific refinement did not outperform plain disagreement, and generalisation beyond this constructed task grid is open.)

Question. Can merge failure be predicted before merging, and by what kind of measure? Design: 39 rows = 13 conditions × 3 seeds (0.5B LoRA children of one frozen base; parents are retrained per condition × seed but share task-data seeds across conditions within a seed, so rows are not independent — all uncertainty below is condition-clustered). Three axes decorrelated by construction: conflict (contradictory conventions on shared ambiguous prompts, private budgets fixed), compat (same shared prompts, same convention — overlap and volume without conflict), duration (weight divergence with zero conflict). Primary outcome (pre-registered): merge penalty vs oracle parent potential (per-component best of the parents); also reported vs best-parent and mean-parent references. All robust statistics reproduce via figures/stats_llm_epistasis.py; figure: llm_epistasis.png.

The predictors (measured pre-merge)

• Confidence-weighted functional conflict (epi_conf) — a proposed proxy for merge-relevant interactions (previously "operational epistasis"; renamed because bilateral confident contradiction measures incompatible endpoint behaviour, not non-additive interaction effects in the strict biological sense).
• Raw functional disagreement (dis_raw) — the unweighted rate.
• Gradient alignment at the shared base (the ML-literature predictor, cf. 2601.22285); LoRA-delta cosine / L2 (weight geometry, exact in r-space); cross-family accuracy (performance baseline).

The supported conclusion (stated conditionally)

Across this controlled task grid, pre-merge functional disagreement predicted merge penalties, whereas LoRA-delta cosine and L2 showed no statistically detectable association. Gradient alignment carried intermediate signal, so the result is not a clean functional-versus-all-geometric divide, and only these selected baselines were tested.

predictor	ρ (primary)	clustered 95% CI	held-out (LOCO) ρ
raw functional disagreement	+0.460	[+0.04, +0.69]	+0.396 (p=0.013)
confidence-weighted functional conflict	+0.446	[+0.02, +0.68]	+0.352 (p=0.028)
gradient alignment	−0.347	[−0.59, −0.06]	+0.10 (n.s.)
delta L2 (geometry)	+0.165	[−0.27, +0.58]	−0.09 (n.s.)
delta cosine (geometry)	+0.030	[−0.46, +0.51]	−0.14 (n.s.)
cross-family accuracy	−0.005	[−0.29, +0.31]	−0.435 (unstable out-of-sample)

Paired comparisons are not individually significant at this sample size (e.g. |ρ(dis_raw)| − |ρ(delta_cos)| = +0.23, CI [−0.23, +0.59]): the honest statement is that the functional measures are individually detectably informative (CIs exclude zero; held-out replication) while the geometry measures are not distinguishable from zero — not that functional significantly beats geometry head-to-head.

What the decomposition shows (and its limits)

• Between vs within. Much of the pooled correlation is axis discrimination (mean penalty: conflict 0.061 vs compat 0.005 / duration 0.011). Within the conflict axis (n=15) the functional measures still track (+0.59/+0.62) — but so does delta-L2 (+0.71), because within that axis conflict fraction, added-data volume, and delta growth are collinear: within-axis identification is impossible by design; the identification comes from the control axes, where the same volumes and L2 ranges occur with ~zero penalty.
• Outcome-reference sensitivity. Under the best-parent reference the ordering changes (delta-L2 +0.48 vs functional +0.34): that reference inherits parent-strength trends that track training volume — which geometry also tracks — coupling predictor and outcome through the reference rather than through merge damage. We keep the pre-registered oracle-potential primary, and report the sensitivity rather than hide it.
• The compat control did the decisive work — and it first broke our own experiment: in the initial two-axis grid the best predictor was delta-cosine (ρ=+0.60), an overlap/volume artifact that the control exposed (collapse to +0.03).

Chronology (adaptive, not wholly prospective)

1. Hypotheses, predictors, primary outcome, and falsifiers written into configs/llm/epistasis.yaml before the first grid ran (conflict + duration axes).
2. The first grid's geometry result prompted the compat control axis (configs/llm/epistasis_compat.yaml, pre-registered readings in its comments) — an amendment after seeing data, run on the same seeds; no conflict/duration rows were re-run or altered.
3. The clustered-bootstrap / LOCO / multi-reference analyses were added at the second review's request, after all data was collected.

Pre-registered internal prediction: NOT confirmed

Confidence weighting was predicted to beat raw disagreement as a rank predictor. It does not (paired Δ|ρ| = −0.02, CI [−0.13, +0.06]). The weighting does double the conflict-vs-compat level contrast (2.0× vs 1.5×), but the present evidence favours functional disagreement generally, not the DMI-specific refinement. Accordingly: the population-genetic framework motivated the measurement and the controls; their success does not validate the specifically population-genetic mechanism.

Open (unchanged by this experiment)

Budget-matched operator choice (the soup-vs-route gap readout is noise-dominated at 0.5B); generalisation to unfamiliar conflict structures and real task pairs; 7B replication; whether any measured quantity deserves the name epistasis (non-additive interaction of combinations) rather than conflict.