MachineSex/paper/response-to-review-2.md

# Response to the second review

*This response accompanies a further revision. Every number below is reproduced by a committed script
(`figures/stats_llm_epistasis.py`) from committed artifacts; the revised documents are
`results/llm_epistasis/README.md` (rewritten around your four analyses), the manuscript, and
`paper/si-notes.md`.*

---

## 0. A correction first (your §7)

You are right, and we apologise for the bookkeeping error: the draft you reviewed **already
contained** the full-symmetry alignment experiment, the conflict residual, and the
compatible-specialisation null. Our previous letter's "new since the review" conflated three things
that we now state separately: **new results** (the LLM-tier speciation runs, the multi-seed
replication, and the controlled predictive test with its control axis), **new controls and analyses**
(the compatible-overlap axis; the robust statistics in this letter), and **revised interpretation**
(everything about E13b/c, which was experimental content you had already seen and whose *presentation*
we changed). The experimental delta attributable to the review process is the first and second
categories only.

## 1. Your two-conclusion distinction — adopted as the evidential boundary

We accept the boundary exactly as you drew it:

- **Demonstrated:** a small-model, controlled predictive test in which pre-merge functional
  disagreement predicted merge penalties where the selected weight-space measures did not.
- **Not demonstrated:** that an epistasis-specific predictor adds value beyond ordinary functional
  disagreement, or that the prediction improves operator choice.

The experiment is now titled a **"controlled predictive test"** everywhere ("decisive experiment
delivered" is gone), and the manuscript's §1 ladder describes the prediction rung in your conditional
formulation, with its four boundary clauses stated in place: constructed grid, small scale, refinement
not superior, operator choice open. The measure itself is renamed **"confidence-weighted functional
conflict — a proposed proxy for merge-relevant interactions"**; we accept that bilateral confident
contradiction measures incompatible endpoint behaviour, not non-additive interaction in the strict
biological sense, and the paper no longer calls any measured quantity "epistasis." Your distinction —
*the framework motivated the measurement and controls* vs *their success validates the specifically
population-genetic mechanism* — is adopted verbatim; we claim the former.

## 2. The four analyses — run

**(1) Direct predictor comparison.** Condition-clustered bootstrap (13 clusters, B = 4000), 95% CIs
for each predictor's ρ against the pre-registered primary outcome:

| predictor | ρ | clustered 95% CI |
|---|---|---|
| raw functional disagreement | +0.460 | [+0.04, +0.69] |
| confidence-weighted functional conflict | +0.446 | [+0.02, +0.68] |
| gradient alignment | −0.347 | [−0.59, −0.06] |
| delta L2 | +0.165 | [−0.27, +0.58] |
| delta cosine | +0.030 | [−0.46, +0.51] |
| cross-family accuracy | −0.005 | [−0.29, +0.31] |

**Paired contrasts are not individually significant** (e.g. |ρ(dis_raw)| − |ρ(delta_cos)| = +0.23,
CI [−0.23, +0.59]). Held-out prediction (leave-one-condition-out linear fits): functional measures
replicate (dis_raw ρ = +0.396, p = 0.013; conf-weighted +0.352, p = 0.028); geometry ≈ 0; the
performance baseline is unstable out-of-sample (−0.435). So the supported statement — now the
conclusion in the README, the figure title, and the manuscript — is yours: *across this controlled
grid, functional disagreement showed a detectable, held-out-robust association with merge penalty;
LoRA-delta cosine and L2 showed no statistically detectable association; gradient alignment carried
intermediate signal (its CI excludes zero), so this is not a clean functional-versus-all-geometric
divide; head-to-head predictor differences are not individually significant; only these baselines were
tested.* "Weight divergence does not predict merge failure" has been removed as over-broad.

**(2) Sample structure.** 39 rows = **13 conditions × 3 seeds**. Parents are retrained per
condition × seed, but children share task-data seeds across conditions within a seed (e.g. the
duration-3 parent and the conflict-0 parent are trained on essentially the same data), so rows are not
independent — hence clustering by condition in all uncertainty estimates, disclosed in the README.

**(3) Between- vs within-axis.** You were right that the pooled correlation is substantially axis
discrimination (mean penalty: conflict 0.061 vs compat 0.005 / duration 0.011). Within the conflict
axis (n = 15): functional measures +0.59/+0.62 — but delta-L2 is +0.71 there, because *within that
axis* conflict fraction, added-data volume, and delta growth are collinear: **within-axis
identification is impossible by design**, and we now say so; the identification comes from the control
axes, where the same volumes and L2 ranges occur at ~zero penalty. Scatterplots are coloured by axis
in the figure. Predicting penalties for a held-out *conflict mechanism* (not just held-out conditions
of the same mechanism) is listed as the next test — we agree it is the more valuable one.

**(4) Outcome references.** Reported under all three: oracle parent potential (pre-registered
primary), best parent, and mean parent. The ordering is **sensitive to the reference** — under the
best-parent reference, delta-L2 correlates comparably to the functional measures (+0.48 vs +0.34).
Our reading, stated in the README rather than hidden: that reference inherits parent-strength trends
that track training volume, which geometry also tracks, coupling predictor and outcome through the
reference rather than through merge damage. On structural coupling between the disagreement predictor
and the oracle-potential outcome: probe and test sets are disjoint by construction, but we agree
definitional coupling through parental complementarity cannot be fully excluded, which is one more
reason all three references are now on the table.

## 3. Hypothesis labels

Your table is adopted essentially as written — the manuscript's claims-at-a-glance table now carries:
pre-merge disagreement predicts penalty (*empirical within the controlled grid*, with the boundary
clauses in the limits column); confidence weighting improves rank prediction (***not supported***);
functional beats all geometry (*not established — selected baselines only*); operator choice (*open*);
cliff, snowball, emergent DMIs (*hypotheses*, unchanged). We also added your snowball distinction
where the snowball is discussed: super-linear growth in incompatibility *count* does not by itself
entail a sharp *performance* cliff — that needs the count→effect-size→performance link, which the
analytic model supplies under its assumptions and any neural test must establish separately.

## 4. The two technical statements — corrected

**"Endpoints and chord are invariant."** Accepted; the ambiguity was ours. The SI proposition now
defines "chord" precisely as the α-linear interpolation **of the endpoint loss values** — the barrier
baseline, a function of endpoints only, which *is* invariant — and states explicitly that the
**weight-space interpolation path is generally not invariant** (that being precisely why alignment can
lower a barrier). It also now carries your second point: exact recovery of a permuted-and-rescaled
copy validates a special case and does not establish global optimality of the alignment for
independently trained networks — so the "removable" share is a lower bound and the "residual" an upper
bound, stated wherever the decomposition is used.

**"Frozen base pins the coordinate system, so failure is functional by construction."** Accepted, and
your replacement wording adopted verbatim across the module, configs, READMEs, and figure: *the shared
frozen base controls a major source of coordinate mismatch, allowing a cleaner test of
conflict-associated merging failure* — with the boundary stated (failures of delta-averaging can still
reflect nonlinear interaction, scaling, or capacity). "Conflict-associated," not "functional by
construction," throughout.

## 5. Chronology and reliability

**Chronology (now a section of the README).** Prospective: hypotheses, predictors, primary outcome and
falsifiers were in the config before the first grid ran. Adaptive: the compatible-overlap control was
added *after* geometry appeared to win, with its own pre-stated readings, run on the same seeds; no
existing rows were re-run or altered. Post hoc: the clustered-bootstrap/LOCO/multi-reference analyses
were added at your request after all data was collected. We agree this is transparent adaptive
experimentation, not wholly prospective confirmation, and it is labelled as such.

**Reliability.** The CI-width claim is withdrawn. The seed-level statement now reads: routing beat the
soup **in every seed** (3/3 paired, both metrics), directed selection beat the soup 3/3, and one seed
exhibited a catastrophic soup failure (0.071 overall, 0.000 worst-family) to which routing was immune
(0.262/0.225); seed-level sds (0.090 vs 0.023) are reported as an observation, with the explicit note
that three seeds do not support a variance estimate.

## 6. On your bottom line

We accept your formulation as the paper's claim for this line of work — it now closes the relevant
results section nearly verbatim: *we separated overlap, divergence, and conflict experimentally; in
this controlled setting, functional disagreement predicted merging damage when simple weight-distance
measures did not; the proposed epistasis refinement and the emergent-speciation mechanism remain
unconfirmed.* And we take the redirection about what would count next: not 0.5B → 7B alone, but
**generalisation to unfamiliar conflict structures** (a held-out conflict mechanism, and real rather
than constructed task pairs) and **a demonstrably better budget-matched merging decision**. Those two
now head the open-problems list, above the scale replication.

We would welcome a third pass if you have the appetite — particularly on whether the README's
conditional conclusion and the chronology section read at the right strength.