Giorgio Gilestro
d22dd9d535
src/neural/recombine.py mirrors Layer-1 run_coverage but trains K_T specialist RNNs on assignments from the exact shared-switch retention construction (K_T/rho/q clean; union matches the closed form), then recombines the measured teacher distributions two ways: mean (naive pooling) vs oracle-guided max-merge (per-mode strongest teacher, M2N2-style), each followed by size-n resampling. Result (8 reps): at rho=0, union rises 0.49->0.96 (supply matches closed form); analytic surviving_max rises 0.043->0.087 while surviving_mean stays flat ~0.045 — the conservation law (averaging cancels the union gain, max-merge realises it). At rho=1 (identical teachers) union and max are flat. The lesson holds in the neural setting; trained-weight columns show the same signs but noisier (smoothing inflates baseline; deep tail barely clears n=200 resampling). torch-gated test added. 93 tests green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
50 lines
1.6 KiB
YAML
50 lines
1.6 KiB
YAML
experiment: recombination
|
|
kind: recombination
|
|
seed: 20260704
|
|
n_replicates: 8
|
|
|
|
# (load-bearing; maps to Layer-1 E4 / blueprint C2+C4): does the recombination finding hold
|
|
# in REAL weights? K_T specialist RNNs are each TRAINED on samples from their assigned mode
|
|
# subset (assignments from the exact shared-switch retention construction, so K_T/rho/q are
|
|
# clean knobs and union_coverage matches the closed form U(K_T,rho,q)). The pupil then
|
|
# recombines the trained teacher distributions two ways: mean (pool outputs — naive multi-
|
|
# teacher distillation) vs max (oracle-guided union, M2N2-style), each followed by size-n
|
|
# resampling. Expect (per Layer-1 E4): union rises with K_T and (1-rho); surviving_max rises
|
|
# with K_T while surviving_mean stays FLAT (the conservation law); at rho=1 (identical
|
|
# teachers) more teachers buy nothing. Falsifier: surviving_mean also rises with K_T, or max
|
|
# does not beat mean -> the "merge, don't average" lesson dies in real weights.
|
|
|
|
synthetic:
|
|
K: 256
|
|
R: 1
|
|
tail: zipf
|
|
zipf_s: 1.3
|
|
tail_frac: 0.5
|
|
tail_threshold: 1.0e-3
|
|
style_len: 3
|
|
style_vocab: 5
|
|
id_base: 2
|
|
|
|
model:
|
|
kind: rnn
|
|
hidden: 128
|
|
embed: 24
|
|
epochs: 22
|
|
lr: 2.0e-3
|
|
batch_size: 256
|
|
n_eval: 12000
|
|
|
|
coverage:
|
|
n: 200 # pupil resample size (drift); tail survives iff mixture mass > ~1/n
|
|
q: 0.5 # marginal tail retention per teacher
|
|
retain_thresh: 1.0e-3
|
|
region_specialisation: false
|
|
|
|
sweep:
|
|
- param: K_T
|
|
values: [1, 2, 3, 5]
|
|
- param: rho
|
|
values: [0.0, 1.0] # decorrelated vs identical teachers (the control)
|
|
|
|
output:
|
|
dir: results/recombination
|