Giorgio Gilestro
6f8cef1ac5
Removed from main (all preserved on the dev branch): the arXiv build and
its sources, design documents (blueprint, results summary, review responses,
essay drafts), tasks/ and CLAUDE.md, the cover letter and reference tooling,
two unused manuscript figures, and every experiment that feeds no figure or
number in the paper: the collapse null, the sexual-vs-asexual lineage, the
NK speciation variant, the 0.5B single-seed LLM prototypes, the compose and
society experiments with their calibration and pilot runs, and their
configs, runners, tests, figure scripts and PBS jobs. Their result bundles
are moved to results/_archive/ (ignored) so the parquets stay on disk.
Also: plot_llm_speciation reads the s{seed}/ layout; the mating-breadth
plot writes under its bundle name; Makefile targets reduced to the kept
experiments; REPRODUCING.md and README point to dev for the rest.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y64o8FKP7rCuXzC48pxpMm
|
||
|---|---|---|
| .. | ||
| fig2_mnist_collapse.pdf | ||
| fig2_mnist_collapse.png | ||
| manifest.json | ||
| mnist_montage.pdf | ||
| mnist_montage.png | ||
| README.md | ||
| resolved_config.yaml | ||
mnist_collapse — collapse and grounding-rescue on REAL MNIST images (external validity)
Claim tested: everything so far used a synthetic sandbox with a zero-error decoder oracle. Do model collapse and its rescue by grounding also appear on real images with a classifier oracle — i.e. is the effect real, not a synthetic artefact?
Setup (Layer 1.5, real-data tier). The generative model is a convolutional VAE (the model
in which generative collapse was first observed). Each generation a fresh VAE is trained from
scratch on the previous VAE's own generated digits, plus a fraction g of fresh real MNIST
images (grounding). K = 30 modes = digit class × stroke-thickness bin (S=3), Zipf-resampled so
the rarest ~18 modes form a real tail. The oracle is a frozen CNN (digit class) + deterministic
thickness bin; its mode accuracy ≈ 98.5% (recorded in manifest.json with the full 30×30
confusion matrix) is the measurement-noise floor. Two arms — dry (g = 0) vs grounded (g = 0.1) —
n = 6000 images/generation, 15 generations, 4 replicates.
Symbols
- mode = (digit class, stroke-thickness bin);
p*= Zipf truth over the 30 modes;p̂= the VAE's oracle-measured mode distribution. g= grounding fraction (share of real MNIST images each generation). forward-KL = distance from truth; support = distinct modes alive;H= diversity; tail truth-mass alive = fraction of the rare tail retained.
The four panels (dry = red, grounded = green; band = 95% CI over 4 reps)
- Forward-KL. Dry climbs from ~0.5 to ~18 (the VAE drifts far from truth); grounded stays near the floor. Collapse is real on images.
- Support. Dry collapses from all 30 modes to ~1 (the VAE ends up emitting a single blurry mode); grounded holds all 30.
- Tail truth-mass alive. Dry's rare tail is wiped out (→ 0.06); grounded keeps the whole tail.
- Heterozygosity. Dry diversity → 0; grounded holds
H ≈ 0.9.
See mnist_montage.png for the eyeball version: gen-0 digits are varied and recognisable; by
gen 12–15 the dry lineage has degenerated into one blurry blob.
Takeaway
Model collapse and its arrest by a small dose of real data reproduce on real MNIST images with a
learned classifier oracle — external validity for the whole Layer-1.5 story. Note the VAE needs
~10% grounding here (vs ~5% for the synthetic histogram), consistent with the grounding finding
that trained neural models need somewhat more grounding than the exact operator. This is
confirmation-only (signs, not magnitudes; blueprint §3.5) — the exact synthetic oracle remains
the anchor for every quantitative claim, and the oracle confusion matrix is the recorded noise floor.
Falsifier (not triggered): if the dry VAE had shown no diversity loss, or grounding had failed to
arrest it, the external-validity claim would fail.