diff --git a/results/E1/README.md b/results/E1/README.md new file mode 100644 index 0000000..8414a11 --- /dev/null +++ b/results/E1/README.md @@ -0,0 +1,32 @@ +# E1 — Distillation without grounding collapses, tail-first + +**Claim tested:** if a model is trained only on the previous model's output, generation after +generation, does it lose knowledge — and does the *rare* knowledge go first? + +**Setup (Layer 1, pure math).** A "population" of `K = 500` items with a fixed true frequency +`p*` shaped like a Zipf curve (a few common items, a long tail of rare ones). Each generation we +draw `n = 100` samples from the current model and refit — **no real data is ever added** (`g = 0`). +Run for 600 generations, averaged over 100 independent repeats. + +### Symbols +- **`p*`** — the true frequencies (fixed reality). **`p_t`** — the model's frequencies at generation *t* (drifts). +- **`H`** heterozygosity = diversity (1 = everything equally likely, 0 = one item left). **`H*`** = diversity of the truth. +- **forward-KL** `D(p*‖p_t)` — how far the model has drifted from truth (0 = perfect, grows without bound as the tail is forgotten). +- **support** = how many items still have any probability. **head/tail** = common/rare items. + +### The three panels +1. **Geometric decay.** Blue = the simulated diversity `H`; black dashed = the exact textbook law + `H₀·(1 − 1/n)^t`. They sit on top of each other — the loss of diversity is *exactly* the + population-genetics drift law, not an approximation. (This is the validation gate: if these two + curves disagreed, the simulator would be wrong.) +2. **Tail dies first** (log axis). Red = fraction of *rare* (tail) items still alive; green = + fraction of *common* (head) items still alive. The red curve plunges far faster — rare knowledge + is lost roughly an order of magnitude sooner than common knowledge. +3. **Collapse.** Purple (left axis, log) = number of distinct items surviving, falling from 500 + toward ~1 (everything collapses onto a single dominant item). Orange (right axis) = forward-KL to + truth, diverging as the tail vanishes. + +### Takeaway +Unchecked model-on-model training is a ratchet: diversity decays on a precise mathematical schedule, +and the rare tail is destroyed first. **Falsifier (not triggered):** if `H` had stayed flat, the +whole thesis would fail. It didn't. diff --git a/results/E2/README.md b/results/E2/README.md new file mode 100644 index 0000000..a533fb4 --- /dev/null +++ b/results/E2/README.md @@ -0,0 +1,35 @@ +# E2 — A tiny dose of real data rescues diversity (the headline) + +**Claim tested:** how much *real* data must you mix back in each generation to stop collapse — a +lot, or a little? + +**Setup (Layer 1, pure math).** `K = 1000` items, Zipf truth `p*`, `n = 200` inherited samples per +generation, 500 generations, 100 repeats. Each generation we also mix in `m` fresh **real** samples +drawn from `p*`. The knob swept is the **grounding fraction** `g = m/(n+m)` — the share of the +training pool that is real — across `g ∈ {0, 0.005, 0.01, 0.02, 0.05, 0.1, 0.2, 0.4}`. + +### Symbols +- **`g`** grounding fraction (share of real data); **`g*`** the *critical* value that restores diversity. +- **`p*`** truth, **`p_t`** model, **`H`** diversity, **`H*`** truth's diversity. +- **grounding = "immigration"** in the genetics analogy: real samples are migrants that re-introduce alleles drift keeps killing. + +### The four panels +1. **Trajectories.** Diversity `H` over generations, one line per `g` (dark = dry, bright = more + grounding). `g = 0` slides toward 0; any `g > 0` levels off on a plateau — the collapse is + arrested. +2. **The phase boundary (the headline).** Dots = stationary diversity vs `g`; the black dashed curve + is the *exact* closed-form equilibrium `H_eq`; the red line marks the critical + **`g* ≈ 0.048` (95% CI [0.047, 0.050])** where `H` reaches 95% of the truth's diversity. Only + **~5% real data** buys back essentially all the diversity. The `g = 0` point is drawn hollow (it + is still sliding — its true equilibrium is 0). +3. **Tail coverage.** Fraction of the rare tail retained, by item-count (red) and truth-mass-weighted + (purple). Both rise with `g` but stay modest at feasible grounding: a little grounding restores + *diversity* long before it restores the *deep* tail — which motivates E4 (recombination) and E6. +4. **Per-rarity band.** The tail split into rarity bands (bright = shallowest, dark = deepest). Deep + bands lag: an item survives only once `m·p*ᵢ ≳ 1` (enough real samples per generation to land it + at least once). The sharp threshold lives here, in discrete item survival — not in the smooth `H`. + +### Takeaway +There is a **critical grounding fraction `g* ≪ 1`**: a small, constant trickle of reality +indefinitely holds off collapse. **Falsifier (not triggered):** if diversity had only recovered as +`g → 1`, the practical thesis would die. It recovers at `g ≈ 0.05`. diff --git a/results/E3/README.md b/results/E3/README.md new file mode 100644 index 0000000..7724fdb --- /dev/null +++ b/results/E3/README.md @@ -0,0 +1,33 @@ +# E3 — Grounding must overlap the content it protects + +**Claim tested:** if your budget of real data is fixed, does it matter *where* you spend it? Is it +enough to sprinkle real data uniformly, or must it target the knowledge you care about? + +**Setup (Layer 1, pure math).** `K = 1000` items divided into `R = 10` disjoint **regions** (think +subject areas). `n = 200`, 400 generations, 100 repeats. The **same total** real-data budget is +spent two ways: **uniform** (spread evenly over all 10 regions) vs **matched** (concentrated on the +"exercised" region we want to protect). One region is designated exercised (region 0 here) and has a +rare tail we track. + +### Symbols +- **region** — a block of related items; a stand-in for a task family / subject area. +- **matched vs uniform** — real data aimed at the exercised region vs spread evenly, at equal total budget. +- **tail items alive** — how many rare items in a region still have any probability. + +### The two panels +1. **Target region over time.** Rare-item survival in the exercised region, generation by generation: + blue = matched, red = uniform (bands = 95% CI). Matched **holds** the region's tail alive; uniform + spreads too thin and lets it **collapse**. +2. **Every region at steady state.** Bar chart of stationary tail survival per region, matched (blue) + vs uniform (red); the dotted line marks the exercised region. Matched wins big *there* — at the + cost of the regions it deliberately ignores. Uniform is mediocre everywhere. + +*(Note: per-region diversity `H` is confounded by how much probability mass sits in a region, so this +figure uses the honest, mass-independent metric — tail-item survival.)* + +### Takeaway +Reality checks only protect what they actually cover. To keep a capability alive you must ground +**on that capability**, not on data in general — grounding is local, not a global tonic. Key numbers: +exercised-region tail survival ≈ **0.49 (matched)** vs **0.07 (uniform)** at equal budget. +**Falsifier (not triggered):** if uniform had protected the region as well as matched, the +targeting claim would die. diff --git a/results/E4/README.md b/results/E4/README.md new file mode 100644 index 0000000..cdfed77 --- /dev/null +++ b/results/E4/README.md @@ -0,0 +1,33 @@ +# E4 — Recombination supplies the rare tail; only a union-preserving *merge* realises it + +**Claim tested:** if several specialist models each remember a different slice of the rare tail, can +combining them reconstruct the whole tail? And does *how* you combine them matter? + +**Setup (Layer 1, pure math).** `K = 500` items. We build `K_T` teacher models that each retain the +rare tail with probability `q`, and we control how **correlated** their retained tails are with a +single knob `ρ` (rho): `ρ = 0` = fully complementary teachers, `ρ = 1` = identical teachers. Swept: +`K_T ∈ {1,2,3,5}`, `ρ ∈ {0, 0.25, 0.5, 0.75, 1}`, grounding `g ∈ {0, 0.02, 0.05}`, 200 repeats. + +### Symbols +- **`K_T`** number of teacher models; **`ρ`** how correlated their retained tails are (0 = diverse, 1 = clones). +- **union coverage** — fraction of the tail covered by *at least one* teacher (the raw supply). +- **surviving coverage** — fraction that actually survives in the pupil after it retrains on the combination. +- **mean-distill** — pupil trained on the pooled/averaged teacher outputs. **max-merge** — keep, per item, the strongest teacher (a union-preserving merge, à la M2N2). + +### The three panels +1. **Supply.** Union tail coverage vs `ρ`, one curve per `K_T`; solid lines are the exact closed form + `U(K_T, ρ, q)`. More teachers and more diversity (lower `ρ`) supply more of the tail — and the + simulation matches the formula exactly. +2. **Realisation.** Surviving coverage vs `ρ`. **Solid = max-merge rises** with more/diverse teachers; + **dashed = mean-distill stays flat.** Averaging dilutes each teacher's rare items back below the + survival threshold — the gain is supplied but not realised. +3. **The benefit needs the right operator (`ρ = 0`).** Surviving coverage vs `K_T` under both + operators. Max-merge climbs with teacher count; mean-distill is flat — a **conservation law**: + averaging's `1/K_T` dilution exactly cancels the union gain. + +### Takeaway — "merge, don't average" +Complementary specialists *contain* enough to rebuild the tail, but **naive multi-teacher distillation +(averaging) throws it away**; you must combine them with a union-preserving merge. This is load-bearing +for the paper's recombination claim and is re-tested in real neural weights in `results/recombination/`. +**Falsifier (not triggered):** if mean-distill had also risen with `K_T`, or max never beat mean, the +recombination story would collapse into "just average your models." diff --git a/results/E5/README.md b/results/E5/README.md new file mode 100644 index 0000000..386ed29 --- /dev/null +++ b/results/E5/README.md @@ -0,0 +1,32 @@ +# E5 — Quality-diversity selection preserves diversity where greedy selection destroys it + +**Claim tested:** if each generation you *select* which outputs to keep, does chasing the "best" +outputs (greedy) accelerate collapse — and does rewarding novelty instead prevent it? + +**Setup (Layer 1, pure math).** `K = 500` items, Zipf truth, `n = 200`, 400 generations, 100 repeats, +all arms given the same grounding. Three selection modes: **none** (grounding only, no selection), +**greedy** (keep the fittest — highest-`p*` — items), and **quality-diversity (QD)** (a novelty +bonus `w_i ∝ f_i · p_i^{-α}` that up-weights rare items). The novelty exponent `α` is swept over +`{0.5, 1, 2}`. + +### Symbols +- **greedy** — select toward the fittest/most-probable items (directional pressure). +- **QD (quality-diversity)** — select for fitness *and* novelty; `α` = strength of the novelty bonus. +- **`H`** diversity; **support** = number of distinct items surviving. + +### The three panels +1. **Diversity trajectories.** `H` over generations: red = greedy (crashes toward ~0, i.e. fixation + on a few items); orange/blue = QD at `α = 1, 2` (holds a high plateau); green = none (reference). + Greedy selection is a *second* collapse engine on top of drift. +2. **Novelty dose–response.** Stationary `H` vs the novelty exponent `α` for QD (orange dots), with + greedy (red dashed) and none (green dashed) as reference lines. QD sits above greedy for **every** + `α`, and rises as the novelty bonus strengthens. +3. **Surviving items per arm.** Stationary support (number of distinct items alive) as bars: greedy is + lowest; QD arms keep progressively more items alive as `α` grows; none is the reference. + +### Takeaway +Optimising only for "what looks best" (greedy) collapses the population onto a handful of winners; a +novelty-rewarding, quality-diversity objective actively **re-introduces and holds the tail**. Key +numbers: greedy `H ≈ 0.01` (near-total fixation) vs QD `H ≈ 0.48–0.88` rising with `α`. +**Falsifier (not triggered):** if QD's stationary `H` had been ≤ greedy's, quality-diversity would be +doing no work. diff --git a/results/E6/README.md b/results/E6/README.md new file mode 100644 index 0000000..3a67d3a --- /dev/null +++ b/results/E6/README.md @@ -0,0 +1,34 @@ +# E6 — Re-minting is irreversible; gate it on diversity + +**Claim tested:** what happens if you "re-baseline" — declare the current model's output to be the new +ground truth and throw away the original? If you do this while the model is already collapsed, is the +damage permanent? And can a simple safeguard prevent it? + +**Setup (Layer 1, pure math).** `K = 500`, `n = 200`, 400 generations, 100 repeats. **Re-minting** +periodically freezes the current distribution as the new grounding reference and *discards the +original truth* (it survives only as a yardstick for measuring drift). Four arms: +- **healthy re-mint** — generous grounding (`m = 60`), re-mint while still diverse; +- **collapsed re-mint (ungated)** — starved grounding (`m = 1`), re-mint anyway; +- **collapsed + diversity gate** — same starvation, but only re-mint if diversity `H ≥ 0.75`; +- **collapsed, no re-mint** — the baseline. + +### Symbols +- **re-mint** — adopt the current model's output as the new "reality" and discard the original truth (a founder event). +- **diversity gate** — refuse to re-mint while `H` is below a threshold (here 0.75). +- **forward-KL to ORIGINAL truth** — how far the lineage has drifted from the *real* original, even after it changed its own reference. + +### The two panels +1. **Lock-in.** Forward-KL to the *original* truth over generations; dotted verticals mark re-mint + events. Red (collapsed, ungated) **jumps up at each re-mint and never comes back** — once the + original tails are gone, re-baselining onto the impoverished distribution makes the loss permanent + (they can no longer be grounded back). Green (healthy) and blue (gated) stay low; grey (baseline) + is the reference. +2. **What the gate reads.** Diversity `H` over generations, same colour key, with the gate threshold + (`H = 0.75`, dashed). The gated arm simply **refuses to re-mint while below the line**, so it never + locks in a collapsed state; the ungated collapsed arm re-mints into the floor. + +### Takeaway +Re-minting a collapsed model **crystallises** the collapse — it is a one-way door. A trivial +safeguard (only re-baseline when diversity is still high) preserves recoverability; re-minting a +healthy model is harmless. **Falsifier (not triggered):** if the collapsed lineage had recovered its +original tails after re-minting, the irreversibility claim would be overstated. diff --git a/results/architectures/README.md b/results/architectures/README.md new file mode 100644 index 0000000..89f1236 --- /dev/null +++ b/results/architectures/README.md @@ -0,0 +1,32 @@ +# architectures — collapse and rescue are architecture-general + +**Claim tested:** is model collapse (and its cure, grounding) a quirk of one model type, or does the +same signature appear across genuinely different neural architectures? + +**Setup (Layer 1.5).** The identical generational loop is run with three different generative models — +an **exact histogram** (no neural net), an **autoregressive GRU** (recurrent), and a **causal-masked +MLP** (feed-forward) — each a distinct "inductive bias." `K = 256` modes, `n = 200`, 22 generations, +5 repeats, compared at **dry (`g = 0`)** vs **grounded (`g = 0.05`)**. + +### Symbols +- **inductive bias** — the built-in assumptions of a model type (a histogram has none; a GRU and an MLP smooth differently). +- **forward-KL** distance from truth; **tail items alive** — how many rare modes still appear. +- **dry** = no grounding; **grounded** = 5% real data mixed in. + +### The three panels +1. **Trajectories.** Forward-KL over generations, coloured by architecture; **solid = dry** (climbs, + collapse) vs **dashed = grounded** (held down). The dry-up / grounded-down gap appears in **every** + architecture. +2. **Stationary forward-KL (grouped bars).** For each architecture, dry (red) vs grounded (green). + Divergence **falls with grounding across all three** — histogram, GRU, MLP. +3. **Tail-item survival (grouped bars).** Same grouping. Survival **rises with grounding across all + three.** (Note the histogram's bars are tiny: with no smoothing it drops rare modes outright, + whereas the GRU/MLP keep some alive — an inductive-bias difference, not a contradiction.) + +### Takeaway +The Wright–Fisher collapse operator and the grounding rescue are **not artefacts of one model** — they +show up in an exact counter, a recurrent net, and a feed-forward net alike. This is the +architecture-generality claim of Layer 1.5. **Falsifier (not triggered):** if the signs had appeared +only for the histogram, collapse would be a property of the idealised operator, not of trained models. +*(A VAE was also implemented but fails the generation-0 fidelity check on this task, so it is excluded +to avoid confusing underfitting with collapse — documented as a known limitation.)* diff --git a/results/bridge/README.md b/results/bridge/README.md new file mode 100644 index 0000000..76a8da2 --- /dev/null +++ b/results/bridge/README.md @@ -0,0 +1,32 @@ +# bridge — the histogram model reproduces Layer-1 E2 exactly (the HARD GATE) + +**Claim tested (a plumbing check, not science):** does the neural experiment harness, when run with a +*trivial* model, reproduce the Layer-1 math exactly? If not, no later neural result could be trusted. + +**Setup (Layer 1.5).** Same generational loop as every neural experiment — each generation, draw the +parent's samples, optionally mix in real data, retrain, measure — but the "model" is a **histogram**: +it just counts which modes appeared and resamples them (no neural net, no smoothing). This reduces the +neural setup *exactly* back to Wright–Fisher drift. `K = 200` modes, `n = 200`, 200 generations, +60 repeats, grounding swept `g ∈ {0, …, 0.4}`. + +### Symbols +- **mode** = one of the `K` items (Layer-1.5 word for "item"); read off each generated sequence by a zero-error oracle. +- **`g`** grounding fraction, **`g*`** its critical value, **`H`** diversity, **`H_eq`** the exact closed-form equilibrium diversity. + +### The four panels +1. **Trajectories.** Diversity `H` per `g`. `g = 0` collapses; `g > 0` plateaus — the E2 picture, + now produced by the *neural runner*. +2. **Bridge = Layer 1 (the gate).** Dots = the neural histogram runner's stationary `H` vs `g`; the + black dashed curve = the exact `H_eq` closed form from Layer 1. The dots sit **on** the curve, and + the recovered critical grounding is **`g* = 0.047` (CI [0.045, 0.052])** — matching Layer-1's + 0.048. This equality is what licenses every trained-model result to be read against the analytic + core. +3. **Tail survival.** Fraction of the rare tail retained vs `g` (item-count red, truth-mass purple) — + rises with grounding, deep tail lags, exactly as in E2. +4. **Per-rarity band.** Survival by rarity band vs `g`; deep bands need more grounding + (`m·p*ᵢ ≳ 1`). + +### Takeaway +The harness is faithful: with a memoryless model it reproduces Layer 1 to the decimal. **This is a +gate, not a finding** — passing it is the precondition for the RNN/MLP experiments +(`collapse`, `grounding`, `architectures`, `recombination`), where the *model* is what changes. diff --git a/results/collapse/README.md b/results/collapse/README.md new file mode 100644 index 0000000..d20b77c --- /dev/null +++ b/results/collapse/README.md @@ -0,0 +1,32 @@ +# collapse — model collapse in REAL neural weights, arrested by grounding + +**Claim tested:** does the collapse we proved in math actually happen in a *trained neural network* — +and does a little grounding stop it? + +**Setup (Layer 1.5).** The model is now a small **autoregressive GRU** (a recurrent net, ~128 hidden +units). Each generation a **fresh** GRU is trained from scratch, by ordinary next-token prediction, on +the previous GRU's own generated sequences (plus any real samples). `K = 256` modes, `n = 200`, +25 generations, grounding `g ∈ {0, 0.02, 0.05, 0.1}`, 5 repeats. A generation-0 fidelity check +confirms the GRU reproduces the truth almost perfectly (KL ≈ 0.008) before any collapse is measured. + +### Symbols +- **GRU** — a small recurrent neural net that emits sequences token by token; retrained each generation on the prior generation's output. +- **forward-KL** `D(p*‖p_t)` — distance from truth; the metric that actually sees neural collapse. +- **`H`** diversity; **tail items alive** — how many rare modes still appear. + +### The four panels +1. **Collapse in weights.** Forward-KL over generations, one line per `g`. The dry lineage (`g = 0`) + **climbs** (drifts from truth) toward ~2.3; grounded lineages are held lower. Collapse is real in + trained weights. +2. **`H` barely moves.** Diversity over generations sits near `H*` for all arms — the GRU's smoothing + bias keeps spurious spread alive, so **diversity `H` hides the collapse**. (This is why forward-KL, + not `H`, is the operative neural metric — see `grounding`.) +3. **Stationary divergence vs `g`.** End-state forward-KL falls as grounding rises: more real data → + closer to truth. +4. **Tail survival vs `g`.** Fraction of rare modes alive rises with grounding. + +### Takeaway +The core phenomenon transfers from math to real neural nets: **a GRU trained on its own output drifts +from truth, and grounding arrests it** — the *sign* Layer-1 predicts (blueprint §3.5). Note the +honest caveat surfaced here and developed in `grounding`: **diversity `H` is the wrong ruler for a +neural net** (smoothing keeps it high even during collapse); distance-from-truth is the right one. diff --git a/results/grounding/README.md b/results/grounding/README.md new file mode 100644 index 0000000..81cf114 --- /dev/null +++ b/results/grounding/README.md @@ -0,0 +1,38 @@ +# grounding — the grounding response in real weights (and why the ruler matters) + +**Claim tested:** does the E2 result — a small dose of real data (`g* ≈ 0.05`) rescues diversity — +reproduce in a trained GRU? The honest answer reframes the question. + +**Setup (Layer 1.5).** Autoregressive GRU, `K = 256` modes, `n = 200`, 30 generations, grounding +swept over 9 values `g ∈ {0, 0.005, …, 0.2}`, **18 repeats** (many repeats are needed because each +lineage's fate is genuinely noisy under `n = 200` drift). The falsifier was pinned in the config +*before* running. + +### Symbols +- **`g`** grounding fraction (share of real data), **`g*`** its critical value. +- **forward-KL** distance from truth (the operative neural collapse metric here). +- **tail survival** `tail_truth_mass_alive` — truth-weighted fraction of the rare tail retained. **`H`** diversity. +- **recovery fraction** — how much of the achievable forward-KL reduction a given `g` has bought (0 = dry, 1 = best observed). + +### The four panels +1. **Trajectories.** Forward-KL over generations per `g`: grounding suppresses the climb. +2. **Phase boundary.** Stationary forward-KL vs `g` **falls monotonically** (dry ≈ 2.08 → `g = 0.2` + ≈ 0.75); the effect is statistically significant (paired *t* up to 3.3; 89% of lineages improve at + `g = 0.2`). **The SIGN is confirmed.** +3. **Recovery curve.** Fraction of the divergence gap closed vs `g`. **Half the gap closes by a + median-recovery grounding of `g ≈ 0.04`** (CI [0.004, 0.116]) — a striking echo of Layer-1's 0.048 + (black dashed) — **but full recovery needs `g ≈ 0.19`**, far more than the exact histogram: the + GRU's smoothing both caps the collapse and slows the rescue. +4. **Why forward-KL (the key methodological panel).** Normalised responses of three rulers vs `g`: + `H/H*` (flat ~0.8) and **tail survival (flat / non-monotone — dry is as high as grounded!)** both + fail to register the effect, while **forward-KL recovery** responds cleanly. A smoothing model keeps + *spurious* tail support alive, so counting surviving modes is misleading; only distance-from-truth + is honest. + +### Takeaway (an honest reframing) +Two results: **(1)** the operative neural collapse metric is **forward-KL**, not `H` or tail-survival — +smoothing decouples "modes alive" from "close to truth." **(2)** The *sharp* threshold `g* ≪ 1` is a +property of the exact operator, carried quantitatively by the histogram **bridge** (`g* = 0.047`); the +trained GRU confirms grounding's **direction** and **softens** its sharpness. The pre-registered +95%-of-`H*`/tail falsifier is *not* met — but because those are the wrong rulers for a smoothing +model, not because grounding fails; the blueprint §3.5 directional claim holds robustly. diff --git a/results/recombination/README.md b/results/recombination/README.md new file mode 100644 index 0000000..943e701 --- /dev/null +++ b/results/recombination/README.md @@ -0,0 +1,36 @@ +# recombination — "merge, don't average" holds in real neural weights (E4 in the flesh) + +**Claim tested:** the E4 finding — complementary specialists can rebuild the rare tail, but only if you +*merge* them rather than average them — was proven in math. Does it survive in trained neural nets? + +**Setup (Layer 1.5).** `K_T` specialist **GRUs** are each trained on a different slice of the rare +tail (slices set by the exact shared-switch construction, so teacher count `K_T`, correlation `ρ` and +retention `q` stay clean knobs and the construction-level union matches the closed form). The pupil then +recombines the teachers' *measured* distributions two ways: **mean** (naive pooling) vs +**oracle-guided max-merge** (per mode, keep the strongest teacher — a union-preserving merge). `K = 256`, +`n = 200`, `q = 0.5`, swept `K_T ∈ {1,2,3,5}` × `ρ ∈ {0, 1}`, 8 repeats. + +### Symbols +- **`K_T`** number of specialist teachers; **`ρ`** how correlated their retained tails are (0 = complementary, 1 = identical clones). +- **union coverage** — tail covered by ≥1 teacher (the raw supply). **surviving coverage** — what remains after the pupil resamples. +- **mean-distill** — average/pool the teachers. **max-merge** — keep each mode's strongest teacher (union-preserving). +- **target vs trained** — coverage computed from the *assigned* distributions vs from the *trained GRU* outputs. + +### The four panels +1. **Supply.** Union coverage vs `K_T`: at `ρ = 0` it climbs 0.49 → 0.96 and matches the closed form + `U(K_T, ρ, q)`; at `ρ = 1` (clones) it is flat. Diverse teachers supply more tail. +2. **Analytic teachers (`ρ = 0`).** Surviving coverage vs `K_T`: **max-merge (green) rises** + 0.043 → 0.087 while **mean-distill (red) stays flat ~0.045** — the conservation law from E4, + reproduced on the assigned distributions. +3. **Trained GRU teachers (`ρ = 0`).** The same comparison on *real trained weights*: same signs — + max-merge above mean-distill — but **compressed and noisier** (the GRU's smoothing inflates the + baseline and the deep tail barely clears `n = 200` resampling). The honest inductive-bias caveat. +4. **Control (`ρ = 1`).** Identical teachers: union, max and mean are all flat — **more clones buy + nothing.** Decorrelation is what the benefit needs. + +### Takeaway +"Merge, don't average" is not a mathematical artefact — in trained neural nets, a union-preserving +**max-merge realises the multi-teacher tail benefit while naive averaging conserves the collapse.** +This is load-bearing for the paper's recombination claim (and points at merging over distillation for +the deferred LLM layer). **Falsifier (not triggered):** if mean-distill had also risen with `K_T`, or +max never beat mean, the recombination lesson would have died in real weights.