Giorgio Gilestro
84124de143
Clarity pass over the main text (36-item audit), Discussion rewrite and cut, acknowledgements, Souly et al. as ref 62, lettered SI panels, model section moved under Results; plus the untracked curriculum/society/compose/smol configs, runners, figures, stats and tests that the SI already cites. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y64o8FKP7rCuXzC48pxpMm |
||
|---|---|---|
| .. | ||
| manifest.json | ||
| mnist_collapse.pdf | ||
| mnist_collapse.png | ||
| mnist_montage.pdf | ||
| mnist_montage.png | ||
| README.md | ||
| resolved_config.yaml | ||
mnist_collapse — collapse and grounding-rescue on REAL MNIST images (external validity)
Claim tested: everything so far used a synthetic sandbox with a zero-error decoder oracle. Do model collapse and its rescue by grounding also appear on real images with a classifier oracle — i.e. is the effect real, not a synthetic artefact?
Setup (Layer 1.5, real-data tier). The generative model is a convolutional VAE (the model
in which generative collapse was first observed). Each generation a fresh VAE is trained from
scratch on the previous VAE's own generated digits, plus a fraction g of fresh real MNIST
images (grounding). K = 30 modes = digit class × stroke-thickness bin (S=3), Zipf-resampled so
the rarest ~18 modes form a real tail. The oracle is a frozen CNN (digit class) + deterministic
thickness bin; its mode accuracy ≈ 98.5% (recorded in manifest.json with the full 30×30
confusion matrix) is the measurement-noise floor. Two arms — dry (g = 0) vs grounded (g = 0.1) —
n = 6000 images/generation, 15 generations, 4 replicates.
Symbols
- mode = (digit class, stroke-thickness bin);
p*= Zipf truth over the 30 modes;p̂= the VAE's oracle-measured mode distribution. g= grounding fraction (share of real MNIST images each generation). forward-KL = distance from truth; support = distinct modes alive;H= diversity; tail truth-mass alive = fraction of the rare tail retained.
The four panels (dry = red, grounded = green; band = 95% CI over 4 reps)
- Forward-KL. Dry climbs from ~0.5 to ~18 (the VAE drifts far from truth); grounded stays near the floor. Collapse is real on images.
- Support. Dry collapses from all 30 modes to ~1 (the VAE ends up emitting a single blurry mode); grounded holds all 30.
- Tail truth-mass alive. Dry's rare tail is wiped out (→ 0.06); grounded keeps the whole tail.
- Heterozygosity. Dry diversity → 0; grounded holds
H ≈ 0.9.
See mnist_montage.png for the eyeball version: gen-0 digits are varied and recognisable; by
gen 12–15 the dry lineage has degenerated into one blurry blob.
Takeaway
Model collapse and its arrest by a small dose of real data reproduce on real MNIST images with a
learned classifier oracle — external validity for the whole Layer-1.5 story. Note the VAE needs
~10% grounding here (vs ~5% for the synthetic histogram), consistent with the grounding finding
that trained neural models need somewhat more grounding than the exact operator. This is
confirmation-only (signs, not magnitudes; blueprint §3.5) — the exact synthetic oracle remains
the anchor for every quantitative claim, and the oracle confusion matrix is the recorded noise floor.
Falsifier (not triggered): if the dry VAE had shown no diversity loss, or grounding had failed to
arrest it, the external-validity claim would fail.