Giorgio Gilestro
287d2326cc
The decisive experiment from the external review. 39 LoRA parent pairs (0.5B, 3 seeds) on three axes decorrelated by construction: conflict (contradictory conventions on shared prompts, private budgets fixed), compat (same prompts, SAME convention — overlap without conflict), and duration (weight divergence, zero conflict). Six pre-merge predictors; primary outcome = merge penalty (parent potential − merged achieved). League table (Spearman vs penalty, n=39): functional measures predict (dis_raw +0.460, epi_conf +0.446, p<0.005); geometry collapses (delta_cos +0.03, delta_l2 +0.17 n.s.); gradient alignment weak (−0.35); performance ~0. The first grid's apparent geometry win (+0.60) was an overlap/volume artifact — the compat control axis (added for exactly this) exposed and killed it: same overlap and data volume, zero penalty. Honest riders in the README: confidence weighting does not beat raw disagreement as a rank predictor (pre-registered internal prediction not confirmed; it does double the conflict/compat level contrast), and |rho|~0.45 is bounded by 0.5B merge-outcome noise (7B is the firm-up). Also: micro-batched gradient accumulation (OOM fix on the shared 16GB GPU), exact r-space LoRA-delta geometry (brute-force-verified test, 151 green), systemd-run runbook lesson (tmux dies with the SSH session scope on this box). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkRLcc18rwT2Lysu6PbG7v |
||
|---|---|---|
| .. | ||
| llm_epistasis.pdf | ||
| llm_epistasis.png | ||
| manifest.json | ||
| README.md | ||
| resolved_config.yaml | ||
The decisive experiment — predicting merge failure BEFORE merging
The external review's bar (2026-08-11): population-genetic quantities must predict, not
re-describe — forecast merge success pre-merge and beat existing predictors. Design: 39 parent
pairs (0.5B LoRA children of one frozen base, 3 seeds) on three axes decorrelated by
construction — conflict (contradictory conventions on shared ambiguous prompts, private budgets
fixed), compat (the control: same shared prompts, same convention — task overlap without
conflict; added after the first grid exposed a confound, see below), and duration (weight
divergence with zero conflict, 1→12 epochs). Primary outcome (pre-registered):
merge penalty = parent potential − merged achieved (the hybrid-load analogue). Figure:
llm_epistasis.png.
The league table (Spearman ρ vs merge penalty, full three-axis pool, n = 39)
| pre-merge predictor | ρ | p | reading |
|---|---|---|---|
raw functional disagreement (dis_raw) |
+0.460 | 0.003 | predicts |
operational epistasis (epi_conf, confidence-weighted) |
+0.446 | 0.004 | predicts |
| gradient alignment at the base (cf. 2601.22285) | −0.347 | 0.03 | weakly informative |
| LoRA-delta L2 distance (geometry) | +0.165 | 0.32 | uninformative |
| LoRA-delta cosine (geometry) | +0.030 | 0.86 | uninformative |
| cross-family accuracy (performance) | −0.005 | 0.98 | uninformative |
Headline: functional conflict, measured before merging, predicts merge failure; weight geometry does not. The duration axis spans the same weight-divergence range as the conflict axis (L2 ≈ 2.4–4.0) at ~zero penalty, and the compat axis adds the same data volumes and overlap at ~zero penalty — so both geometric predictors collapse once overlap and volume are controlled.
The control that did the work (llm_epistasis_compat/)
In the first grid (conflict + duration only), delta_cos scored ρ = +0.60 — apparently the best
predictor. That was an artifact: every shared-data pair in that pool was a conflicted pair, so
geometry could win as a mere task-overlap/volume detector. The compat axis (overlap without
conflict) exposes it: penalty ≈ 0.005 there, and the geometry correlations collapse (+0.60 → +0.03).
The functional measures behave correctly on the control — parents trained on the same convention
agree on the shared prompts (epi_conf: 0.46 conflict vs 0.23 compat, a 2× contrast; raw
disagreement 0.72 vs 0.47, only 1.5× — the confidence weighting removes complementation noise from
the level, giving the cleaner axis separation).
Honest riders (pre-registered falsifier status)
- The internal prediction that confidence weighting would beat raw disagreement as a rank
predictor is not confirmed:
epi_confanddis_raware statistically indistinguishable at n = 39 (the weighting does improve the conflict-vs-compat contrast in levels). The paper reports the functional-vs-geometric verdict, not a win for the refinement. - Correlations are moderate (|ρ| ≈ 0.45), bounded by 0.5B merge-outcome noise (soup merges carry
large intrinsic seed variance — see
llm_moe_hard_seeds); read as signs and ordering, not magnitudes. 7B replication is the natural firm-up. - Gradient alignment carries real signal (it differentiates conflicting conventions at the base) but less than the functional measures in this design.
Bottom line for the paper: the framework's claim — epistasis (functional conflict), not divergence, sets merge compatibility — survives its designed falsification test at this tier: the operational conflict measures predict, the divergence measures do not, and the case was made honest by a control that first broke our own experiment's favourite-looking geometric predictor.