Giorgio Gilestro
6b5591c92f
The five priority fixes, in the PNAS draft and propagated to the
long-form document and results documentation:
1. The averaging proposition now proves what it claims: a FIRST-ORDER
cancellation of the multi-parent retention gain under output-mean
inheritance in the rare-item regime (n·p/K << 1), with the convexity
boundary stated (averaging's variance reduction can reduce extinction
outside that regime — the reviewer's argument) and the union
operator's renormalisation + oracle requirement explicit. "Adding
parents cannot help" deleted everywhere.
2. Grounding: g*~=0.05 restated as an operational threshold (equilibrium
smooth in g — no phase transition); m·p floor restated as
1−exp(−m·p) per-batch observation probability with
retention/occupancy/reintroduction distinguished; the deep-tail rule
de-categoricalised (stratified sampling; recombination recovers only
what parents retain).
3. Grounded INHERITANCE (data channel) separated from grounded
EVALUATION (fitness channel) in the society section; retitled to
"complementary contributions"; general joint necessity disclaimed.
Table 1 + v6 ledger updated.
4. Alignment contradiction removed everywhere ("cannot be an alignment
failure" -> the reviewer's formulation); abstract says "remaining
after permutation-and-rescaling alignment"; group = search space,
control recovery != global optimality; "specialisation is merge-safe"
-> "do not treat divergence/specialisation alone as evidence of
incompatibility".
5. Significance headline matched to the bounded evidence; seed-
dependence sensitivity added (per-seed rho stable +0.37..+0.53 for
functional measures, ~0 for geometry, gradient alignment
seed-UNSTABLE −0.11..−0.55 — reported as its own caveat; LOSO ranges
in stats script).
Presentation: review-process meta-language stripped; "exact" reserved
for closed forms ("analytic model" labels); headroom rule qualitative;
directed-sex phrasing per review; ratchet = consequence-level
correspondence; compact results table (Table 2) added. Response letter:
paper/response-to-review-3.md. Both PDFs rebuilt; 151 tests green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BkRLcc18rwT2Lysu6PbG7v
87 lines
6.3 KiB
Markdown
87 lines
6.3 KiB
Markdown
# E13 — Real-weight model speciation (the alignment residual, now modulo the full symmetry group)
|
||
|
||
**Claim tested.** E12 predicts model *speciation* analytically: as two lineages diverge, recombination
|
||
(merging) fails, via Bateson–Dobzhansky–Muller incompatibilities. E13 confirms it in **real trained
|
||
weights**, separating the part of the merge barrier that is a mere **coordinate artefact** (removable
|
||
by re-coordinating hidden units) from the **residual** that no alignment can remove — the true
|
||
reproductive-isolation signal.
|
||
|
||
**E13c hardening (2026 PNAS campaign).** Recent work shows symmetry groups *richer than permutations*
|
||
remove more of the barrier between independently trained transformers (arXiv:2606.23607;
|
||
neuron-identifiability LMC). We therefore align modulo the **full function-preserving unit symmetry
|
||
group of a plain ReLU MLP** — per-unit positive rescaling (scale canonicalisation, exact) *composed
|
||
with* Git Re-Basin permutation matching (`neural/rebasin.py`; the sanity gate recovers a permuted
|
||
**and rescaled** copy to exact weight identity). Both residuals are reported: `residual` (permutation
|
||
only) and `residual_scale` (full group).
|
||
|
||
**Setup.** No-BatchNorm MLPs (784–512–512–10) on MNIST. Children forked/trained per condition;
|
||
weight-average merge; linear-mode-connectivity error barrier before/after alignment; midpoint
|
||
(merged-model) accuracy recorded alongside. 3 replicates (decomposition/cliff), 4 (emergent).
|
||
Statistically reproducible (seeded); the alignment itself is deterministic NumPy/scipy.
|
||
|
||
### Results — the decomposition (mean over divergence, reps)
|
||
| condition | naive barrier | residual (permutation) | **residual (full symmetry group)** | merged acc |
|
||
|---|---|---|---|---|
|
||
| `shared` (same task, shared fork) | 0.000 | 0.000 | **0.000** | 0.964 |
|
||
| `independent` (same task, different init) | 0.044 | 0.001 | **0.001** | 0.960 (= parents) |
|
||
| `conflict` (contradictory label maps) | 0.502 | 0.502 | **0.497** | **0.037 (inviable)** |
|
||
|
||
- **`independent`**: the barrier is a coordinate artefact — permutations already remove ~98%, and the
|
||
full symmetry group confirms (residual 0.001). The aligned merge performs **at parent level**
|
||
(0.960): same species, different basis.
|
||
- **`conflict`**: permutation+rescaling alignment removes essentially nothing (0.502 → 0.497), and the
|
||
hybrid is functionally dead (accuracy 0.037). The tested alignment removes the same-task barrier but
|
||
leaves the conflict-associated barrier largely unchanged — supporting a functional-conflict
|
||
interpretation without proving optimal alignment (control recovery validates a special case;
|
||
removable = lower bound, residual = upper bound).
|
||
- Formal floor (SI note S1, `paper/si-notes.md`): for label maps conflicting on mass `μ(S)`, *any*
|
||
single merged model errs at rate ≥ `μ(S)/2` against at least one parent, under *any* alignment
|
||
group and merge operator — hybrid disadvantage is information-theoretic, and endpoints/chord are
|
||
invariant to all function-preserving transformations.
|
||
|
||
### Results — the isolation cliff (`speciation_real_cliff/`)
|
||
Sweeping the fraction of conflicting classes (residual = full-symmetry alignment; `t_div=800`):
|
||
|
||
| conflict fraction | 0.0 | 0.2 | 0.4 | 0.6 | 0.8 | 1.0 |
|
||
|---|---|---|---|---|---|---|
|
||
| residual barrier | 0.000 | 0.122 | 0.187 | 0.278 | 0.406 | 0.506 |
|
||
| **merged (hybrid) accuracy** | 0.968 | 0.764 | 0.586 | 0.396 | 0.199 | 0.034 |
|
||
|
||
`residual_scale ≈ residual` at every point (±0.005): the cliff is functional isolation under the full
|
||
symmetry group. Read as **hybrid fitness**, the merged model's accuracy falls 0.97 → 0.03 — the
|
||
real-weight image of E12's *compatible → outbreeding depression → hybrid inviability* trajectory.
|
||
|
||
### Results — emergent divergence does NOT speciate (`speciation_real_emergent/`, E13b)
|
||
The conflict condition *imposes* contradiction; a true BDM incompatibility is *emergent*. Two
|
||
pre-registered conditions with **no conflicting training signal anywhere**: `disjoint` (child A trains
|
||
only on classes 0–4, child B on 5–9) and `augment` (same labels, inputs rolled ±3 px), swept to
|
||
`t_div = 3200` (children trained 6.4× longer than the shared base):
|
||
|
||
- **Residual barrier = 0.000 at every divergence, both conditions** (naive barrier is 0 too — the
|
||
children never leave the shared basin).
|
||
- The `disjoint` parents decay to 0.535/0.474 on the full task (each forgets the other's classes),
|
||
while the **merged model holds ≈ 0.955 at every divergence** — a sustained ~40-point
|
||
**Fisher–Muller rescue** of two catastrophically-forgetting specialists, at zero barrier.
|
||
`augment` shows the same shape (parents 0.65/0.73, merge ≈ 0.90).
|
||
|
||
**Honest conclusion (the pre-registered second reading):** in this regime — shared ancestry, same
|
||
architecture, compatible tasks, divergence up to 3200 steps — **model speciation does not emerge
|
||
spontaneously; reproductive isolation requires functional conflict.** Trained networks are *more*
|
||
merge-compatible than the biological analogy predicts, and the design rule sharpens: *merge freely
|
||
across divergently-specialised lineages of shared ancestry — the danger is conflicting conventions,
|
||
not specialisation per se.* Scope caveat: small MLPs, one fork depth; whether long-horizon
|
||
over-specialisation at LLM scale erodes mergeability (as the expert-training-duration literature
|
||
suggests, arXiv:2607.11997) is exactly the Phase-3 `llm_speciation` question.
|
||
|
||
### Positioning
|
||
Git Re-Basin / Entezari (barriers as coordinate artefacts), the richer-symmetry LMC results
|
||
(2606.23607 and neuron-identifiability, 2026), Frankle (fork instability), Pari 2024 (route don't
|
||
fuse), Zhou 2026 / 2601.22285 (predict mergeability from divergence/geometry), 2603.09463
|
||
(merge-collapse capacity theory). E13's contribution is the synthesis they lack: a controlled
|
||
decomposition where alignment — *modulo the full symmetry group* — cleanly partitions the merge
|
||
barrier into a removable coordinate artefact and a **functional reproductive-isolation residual** that
|
||
rises with task conflict, is absent under compatible specialisation, and carries an
|
||
information-theoretic floor. **Falsifiers (none triggered):** alignment failing on `independent`
|
||
(would invalidate the residual); conflict showing no residual; the richer symmetry group dissolving
|
||
the conflict residual (it removed 0.005 of 0.502); emergent conditions showing residual attributable
|
||
to alignment failure.
|