recombination: reproduce the E4 "merge, don't average" finding in real weights

src/neural/recombine.py mirrors Layer-1 run_coverage but trains K_T specialist RNNs on
assignments from the exact shared-switch retention construction (K_T/rho/q clean; union
matches the closed form), then recombines the measured teacher distributions two ways:
mean (naive pooling) vs oracle-guided max-merge (per-mode strongest teacher, M2N2-style),
each followed by size-n resampling.

Result (8 reps): at rho=0, union rises 0.49->0.96 (supply matches closed form); analytic
surviving_max rises 0.043->0.087 while surviving_mean stays flat ~0.045 — the conservation
law (averaging cancels the union gain, max-merge realises it). At rho=1 (identical
teachers) union and max are flat. The lesson holds in the neural setting; trained-weight
columns show the same signs but noisier (smoothing inflates baseline; deep tail barely
clears n=200 resampling). torch-gated test added. 93 tests green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Giorgio Gilestro 2026-07-04 21:49:44 +01:00
parent aca7b394a3
commit d22dd9d535
7 changed files with 263 additions and 7 deletions

View file

@ -0,0 +1,50 @@
experiment: recombination
kind: recombination
seed: 20260704
n_replicates: 8
# (load-bearing; maps to Layer-1 E4 / blueprint C2+C4): does the recombination finding hold
# in REAL weights? K_T specialist RNNs are each TRAINED on samples from their assigned mode
# subset (assignments from the exact shared-switch retention construction, so K_T/rho/q are
# clean knobs and union_coverage matches the closed form U(K_T,rho,q)). The pupil then
# recombines the trained teacher distributions two ways: mean (pool outputs — naive multi-
# teacher distillation) vs max (oracle-guided union, M2N2-style), each followed by size-n
# resampling. Expect (per Layer-1 E4): union rises with K_T and (1-rho); surviving_max rises
# with K_T while surviving_mean stays FLAT (the conservation law); at rho=1 (identical
# teachers) more teachers buy nothing. Falsifier: surviving_mean also rises with K_T, or max
# does not beat mean -> the "merge, don't average" lesson dies in real weights.
synthetic:
K: 256
R: 1
tail: zipf
zipf_s: 1.3
tail_frac: 0.5
tail_threshold: 1.0e-3
style_len: 3
style_vocab: 5
id_base: 2
model:
kind: rnn
hidden: 128
embed: 24
epochs: 22
lr: 2.0e-3
batch_size: 256
n_eval: 12000
coverage:
n: 200 # pupil resample size (drift); tail survives iff mixture mass > ~1/n
q: 0.5 # marginal tail retention per teacher
retain_thresh: 1.0e-3
region_specialisation: false
sweep:
- param: K_T
values: [1, 2, 3, 5]
- param: rho
values: [0.0, 1.0] # decorrelated vs identical teachers (the control)
output:
dir: results/recombination