Giorgio Gilestro
3b9f4f7893
One self-contained README.md per results/ figure folder (Layer 1 E1-E6 and Layer 1.5 bridge/collapse/grounding/architectures/recombination): plain-language claim, setup, a compact symbol glossary, a panel-by-panel walkthrough, and the takeaway + falsifier. Auto-renders when browsing the folder; carries the honest caveats (grounding's ruler reframing, the excluded VAE). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
36 lines
2.8 KiB
Markdown
36 lines
2.8 KiB
Markdown
# recombination — "merge, don't average" holds in real neural weights (E4 in the flesh)
|
||
|
||
**Claim tested:** the E4 finding — complementary specialists can rebuild the rare tail, but only if you
|
||
*merge* them rather than average them — was proven in math. Does it survive in trained neural nets?
|
||
|
||
**Setup (Layer 1.5).** `K_T` specialist **GRUs** are each trained on a different slice of the rare
|
||
tail (slices set by the exact shared-switch construction, so teacher count `K_T`, correlation `ρ` and
|
||
retention `q` stay clean knobs and the construction-level union matches the closed form). The pupil then
|
||
recombines the teachers' *measured* distributions two ways: **mean** (naive pooling) vs
|
||
**oracle-guided max-merge** (per mode, keep the strongest teacher — a union-preserving merge). `K = 256`,
|
||
`n = 200`, `q = 0.5`, swept `K_T ∈ {1,2,3,5}` × `ρ ∈ {0, 1}`, 8 repeats.
|
||
|
||
### Symbols
|
||
- **`K_T`** number of specialist teachers; **`ρ`** how correlated their retained tails are (0 = complementary, 1 = identical clones).
|
||
- **union coverage** — tail covered by ≥1 teacher (the raw supply). **surviving coverage** — what remains after the pupil resamples.
|
||
- **mean-distill** — average/pool the teachers. **max-merge** — keep each mode's strongest teacher (union-preserving).
|
||
- **target vs trained** — coverage computed from the *assigned* distributions vs from the *trained GRU* outputs.
|
||
|
||
### The four panels
|
||
1. **Supply.** Union coverage vs `K_T`: at `ρ = 0` it climbs 0.49 → 0.96 and matches the closed form
|
||
`U(K_T, ρ, q)`; at `ρ = 1` (clones) it is flat. Diverse teachers supply more tail.
|
||
2. **Analytic teachers (`ρ = 0`).** Surviving coverage vs `K_T`: **max-merge (green) rises**
|
||
0.043 → 0.087 while **mean-distill (red) stays flat ~0.045** — the conservation law from E4,
|
||
reproduced on the assigned distributions.
|
||
3. **Trained GRU teachers (`ρ = 0`).** The same comparison on *real trained weights*: same signs —
|
||
max-merge above mean-distill — but **compressed and noisier** (the GRU's smoothing inflates the
|
||
baseline and the deep tail barely clears `n = 200` resampling). The honest inductive-bias caveat.
|
||
4. **Control (`ρ = 1`).** Identical teachers: union, max and mean are all flat — **more clones buy
|
||
nothing.** Decorrelation is what the benefit needs.
|
||
|
||
### Takeaway
|
||
"Merge, don't average" is not a mathematical artefact — in trained neural nets, a union-preserving
|
||
**max-merge realises the multi-teacher tail benefit while naive averaging conserves the collapse.**
|
||
This is load-bearing for the paper's recombination claim (and points at merging over distillation for
|
||
the deferred LLM layer). **Falsifier (not triggered):** if mean-distill had also risen with `K_T`, or
|
||
max never beat mean, the recombination lesson would have died in real weights.
|