Giorgio Gilestro
a40ace1821
Analyses (figures/stats_llm_epistasis.py, committed + reproducible): condition-clustered bootstrap CIs (functional measures exclude zero: dis_raw [+0.04,+0.69], conf-weighted [+0.02,+0.68]; gradient alignment [-0.59,-0.06]; geometry straddles zero), PAIRED predictor contrasts (not individually significant — stated), leave-one-condition-out held-out prediction (functional replicates, geometry ~0, performance baseline unstable), three outcome references (ordering sensitive to reference — reported, with the mechanism), between/within-axis decomposition (within-conflict identification impossible by design; the compat axis identifies), and seed-level paired reliability (routing/directed beat soup 3/3 seeds incl. one catastrophic soup failure; CI-width fragility claim withdrawn). Renames and corrections: "decisive experiment" -> "controlled predictive test"; "operational epistasis" -> "confidence-weighted functional conflict (proposed proxy)"; "functional by construction" -> "controls a major source of coordinate mismatch / conflict-associated" (module, configs, READMEs, figures); SI proposition's "chord" defined precisely (endpoint-loss interpolation, invariant) vs the path (not invariant) + no-global-optimality caveat (removable = lower bound, residual = upper); snowball count != performance cliff distinction added; claims table gains four rows (grid finding / weighting NOT supported / functional-vs- all-geometry not established / operator choice open); §1 ladder states the prediction rung as a bounded small-model result. paper/response-to-review-2.md: point-by-point, opening with the bookkeeping correction (E13b/c were in the reviewed draft — revised interpretation, not new results). READMEs rewritten around the four analyses with the chronology (prospective/adaptive/post-hoc) disclosed. 151 tests green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkRLcc18rwT2Lysu6PbG7v
86 lines
6 KiB
Markdown
86 lines
6 KiB
Markdown
# A controlled predictive test — pre-merge functional conflict and merge penalty
|
||
|
||
*(Renamed from "the decisive experiment" after the second external review: this is a **small-model
|
||
controlled predictive test**, not the decisive experiment delivered. The operator-choice component is
|
||
unfinished, the epistasis-specific refinement did not outperform plain disagreement, and
|
||
generalisation beyond this constructed task grid is open.)*
|
||
|
||
**Question.** Can merge failure be predicted **before merging**, and by what kind of measure? Design:
|
||
39 rows = **13 conditions × 3 seeds** (0.5B LoRA children of one frozen base; parents are retrained
|
||
per condition × seed but share task-data seeds across conditions within a seed, so rows are *not*
|
||
independent — all uncertainty below is condition-clustered). Three axes decorrelated by construction:
|
||
`conflict` (contradictory conventions on shared ambiguous prompts, private budgets fixed), `compat`
|
||
(same shared prompts, same convention — overlap and volume without conflict), `duration` (weight
|
||
divergence with zero conflict). Primary outcome (pre-registered): **merge penalty vs oracle parent
|
||
potential** (per-component best of the parents); also reported vs best-parent and mean-parent
|
||
references. All robust statistics reproduce via `figures/stats_llm_epistasis.py`; figure:
|
||
`llm_epistasis.png`.
|
||
|
||
### The predictors (measured pre-merge)
|
||
- **Confidence-weighted functional conflict** (`epi_conf`) — *a proposed proxy for merge-relevant
|
||
interactions* (previously "operational epistasis"; renamed because bilateral confident
|
||
contradiction measures incompatible endpoint behaviour, not non-additive interaction effects in the
|
||
strict biological sense).
|
||
- **Raw functional disagreement** (`dis_raw`) — the unweighted rate.
|
||
- **Gradient alignment** at the shared base (the ML-literature predictor, cf. 2601.22285);
|
||
**LoRA-delta cosine / L2** (weight geometry, exact in r-space); **cross-family accuracy**
|
||
(performance baseline).
|
||
|
||
### The supported conclusion (stated conditionally)
|
||
> **Across this controlled task grid, pre-merge functional disagreement predicted merge penalties,
|
||
> whereas LoRA-delta cosine and L2 showed no statistically detectable association.** Gradient
|
||
> alignment carried intermediate signal, so the result is not a clean functional-versus-all-geometric
|
||
> divide, and only these selected baselines were tested.
|
||
|
||
| predictor | ρ (primary) | clustered 95% CI | held-out (LOCO) ρ |
|
||
|---|---|---|---|
|
||
| raw functional disagreement | +0.460 | [+0.04, +0.69] | +0.396 (p=0.013) |
|
||
| confidence-weighted functional conflict | +0.446 | [+0.02, +0.68] | +0.352 (p=0.028) |
|
||
| gradient alignment | −0.347 | [−0.59, −0.06] | +0.10 (n.s.) |
|
||
| delta L2 (geometry) | +0.165 | [−0.27, +0.58] | −0.09 (n.s.) |
|
||
| delta cosine (geometry) | +0.030 | [−0.46, +0.51] | −0.14 (n.s.) |
|
||
| cross-family accuracy | −0.005 | [−0.29, +0.31] | −0.435 (unstable out-of-sample) |
|
||
|
||
**Paired comparisons are not individually significant** at this sample size (e.g.
|
||
|ρ(dis_raw)| − |ρ(delta_cos)| = +0.23, CI [−0.23, +0.59]): the honest statement is that the
|
||
functional measures are *individually detectably informative* (CIs exclude zero; held-out
|
||
replication) while the geometry measures are *not distinguishable from zero* — not that functional
|
||
significantly beats geometry head-to-head.
|
||
|
||
### What the decomposition shows (and its limits)
|
||
- **Between vs within.** Much of the pooled correlation is axis discrimination (mean penalty:
|
||
conflict 0.061 vs compat 0.005 / duration 0.011). Within the conflict axis (n=15) the functional
|
||
measures still track (+0.59/+0.62) — but so does delta-L2 (+0.71), because *within that axis*
|
||
conflict fraction, added-data volume, and delta growth are collinear: **within-axis identification
|
||
is impossible by design; the identification comes from the control axes**, where the same volumes
|
||
and L2 ranges occur with ~zero penalty.
|
||
- **Outcome-reference sensitivity.** Under the *best-parent* reference the ordering changes
|
||
(delta-L2 +0.48 vs functional +0.34): that reference inherits parent-strength trends that track
|
||
training volume — which geometry also tracks — coupling predictor and outcome through the
|
||
reference rather than through merge damage. We keep the pre-registered oracle-potential primary,
|
||
and report the sensitivity rather than hide it.
|
||
- **The compat control did the decisive work** — and it *first broke our own experiment*: in the
|
||
initial two-axis grid the best predictor was delta-cosine (ρ=+0.60), an overlap/volume artifact
|
||
that the control exposed (collapse to +0.03).
|
||
|
||
### Chronology (adaptive, not wholly prospective)
|
||
1. Hypotheses, predictors, primary outcome, and falsifiers written into `configs/llm/epistasis.yaml`
|
||
**before** the first grid ran (conflict + duration axes).
|
||
2. The first grid's geometry result prompted the `compat` control axis
|
||
(`configs/llm/epistasis_compat.yaml`, pre-registered readings in its comments) — an **amendment
|
||
after seeing data**, run on the same seeds; no conflict/duration rows were re-run or altered.
|
||
3. The clustered-bootstrap / LOCO / multi-reference analyses were added at the second review's
|
||
request, after all data was collected.
|
||
|
||
### Pre-registered internal prediction: NOT confirmed
|
||
Confidence weighting was predicted to beat raw disagreement as a rank predictor. It does not (paired
|
||
Δ|ρ| = −0.02, CI [−0.13, +0.06]). The weighting does double the conflict-vs-compat *level* contrast
|
||
(2.0× vs 1.5×), but the present evidence favours **functional disagreement generally, not the
|
||
DMI-specific refinement**. Accordingly: the population-genetic framework motivated the measurement
|
||
and the controls; their success does **not** validate the specifically population-genetic mechanism.
|
||
|
||
### Open (unchanged by this experiment)
|
||
Budget-matched operator choice (the soup-vs-route gap readout is noise-dominated at 0.5B);
|
||
generalisation to unfamiliar conflict structures and real task pairs; 7B replication; whether any
|
||
measured quantity deserves the name *epistasis* (non-additive interaction of combinations) rather
|
||
than *conflict*.
|