giorgio/MachineSex
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

neural: real-MNIST external-validity tier (collapse + grounding)

Browse source 
Confirms model collapse and its arrest by grounding on REAL images, not
just the synthetic sandbox. A conv VAE (the canonical generative-collapse
model) is retrained each generation on its own generated digits, with a
fraction g of fresh real MNIST mixed in. Modes = digit class x stroke-
thickness bin (K=30, Zipf, ~18 tail modes); the oracle is a frozen CNN +
deterministic thickness at 98.5% mode accuracy (30x30 confusion matrix
recorded in the manifest as the measurement-noise floor).

Result (4 reps): dry (g=0) collapses to a single mode -- forward-KL
0.5->18, support 30->1, tail 1.0->0.06, H->0 -- while 10% grounding holds
all 30 modes (KL~0.6, full tail, H~0.9). Signs, not magnitudes (blueprint
3.5); the exact synthetic oracle stays the quantitative anchor. The VAE
needs ~10% grounding vs the synthetic histogram's ~5%, consistent with the
grounding finding that trained nets need more than the exact operator.

Plugs into the existing data-agnostic contract (metrics/grounding/output
reused verbatim): mnist_data (thickness bins, class x thickness bijection,
MnistSampler), mnist_oracle (ClassifierOracle + confusion matrix),
mnist_vae (ConvVAEGenerator), mnist_loop (run_mnist_lineage), kind=
mnist_lineage dispatch, MnistCfg/OracleCfg. Figures: plot_mnist (parquet-
only) + mnist_montage (eyeball diagnostic showing digits degenerate to one
blurry mode). make mnist / make env-mnist, kept out of the make neural
loop. 99 tests green (+5 torchvision-gated).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Giorgio Gilestro 2026-07-05 09:19:36 +01:00
parent 3b9f4f7893
commit 79bbc45f41
21 changed files with 2200 additions and 10 deletions
Download patch file Download diff file Expand all files Collapse all files

39
results/mnist_collapse/README.md Normal file
View file

@ -0,0 +1,39 @@
# mnist_collapse — collapse and grounding-rescue on REAL MNIST images (external validity)
**Claim tested:** everything so far used a *synthetic* sandbox with a zero-error decoder oracle. Do
model collapse and its rescue by grounding also appear on **real images** with a **classifier**
oracle — i.e. is the effect real, not a synthetic artefact?
**Setup (Layer 1.5, real-data tier).** The generative model is a **convolutional VAE** (the model
in which generative collapse was first observed). Each generation a **fresh** VAE is trained from
scratch on the previous VAE's own generated digits, plus a fraction `g` of fresh **real** MNIST
images (grounding). `K = 30` **modes** = digit class × stroke-thickness bin (S=3), Zipf-resampled so
the rarest ~18 modes form a real tail. The **oracle** is a frozen CNN (digit class) + deterministic
thickness bin; its **mode accuracy ≈ 98.5%** (recorded in `manifest.json` with the full 30×30
confusion matrix) is the measurement-noise floor. Two arms — dry (`g = 0`) vs grounded (`g = 0.1`) —
`n = 6000` images/generation, 15 generations, 4 replicates.
### Symbols
- **mode** = (digit class, stroke-thickness bin); **`p*`** = Zipf truth over the 30 modes; **`p̂`** = the VAE's oracle-measured mode distribution.
- **`g`** = grounding fraction (share of real MNIST images each generation). **forward-KL** = distance from truth; **support** = distinct modes alive; **`H`** = diversity; **tail truth-mass alive** = fraction of the rare tail retained.
### The four panels (dry = red, grounded = green; band = 95% CI over 4 reps)
1. **Forward-KL.** Dry climbs from ~0.5 to **~18** (the VAE drifts far from truth); grounded stays
near the floor. Collapse is real on images.
2. **Support.** Dry collapses from all 30 modes to **~1** (the VAE ends up emitting a single blurry
mode); grounded holds all 30.
3. **Tail truth-mass alive.** Dry's rare tail is wiped out (→ 0.06); grounded keeps the whole tail.
4. **Heterozygosity.** Dry diversity → 0; grounded holds `H ≈ 0.9`.
See **`mnist_montage.png`** for the eyeball version: gen-0 digits are varied and recognisable; by
gen 12–15 the dry lineage has degenerated into one blurry blob.
### Takeaway
Model collapse and its arrest by a small dose of real data **reproduce on real MNIST images with a
learned classifier oracle** — external validity for the whole Layer-1.5 story. Note the VAE needs
~10% grounding here (vs ~5% for the synthetic histogram), consistent with the `grounding` finding
that trained neural models need somewhat more grounding than the exact operator. This is
**confirmation-only** (signs, not magnitudes; blueprint §3.5) — the exact synthetic oracle remains
the anchor for every quantitative claim, and the oracle confusion matrix is the recorded noise floor.
**Falsifier (not triggered):** if the dry VAE had shown no diversity loss, or grounding had failed to
arrest it, the external-validity claim would fail.

984
results/mnist_collapse/manifest.json Normal file
View file

@ -0,0 +1,984 @@
{
"experiment": "mnist_collapse",
"master_seed": 20260705,
"git_commit": "3b9f4f789358a442d510bf2d609aab0986834ac5",
"python": "3.14.5",
"libraries": {
"numpy": "2.5.0",
"scipy": "1.18.0",
"pandas": "3.0.3",
"pyarrow": "24.0.0",
"torch": "2.12.1",
"torchvision": "0.27.1"
},
"rows": 128,
"results_sha256": "d808003e0e01247923996c94eff18c430308448e23bb5d102aa860701145131e",
"layer": "1.5",
"tier": "mnist",
"model_kind": "convvae",
"oracle_ckpt_sha256": "7ff2cc619861daacf8168277c83cde29483554c92d732a24db9fdaa446fa9ff8",
"oracle_mode_accuracy": 0.9847,
"oracle_class_accuracy": 0.9847,
"confusion_matrix": [
[
365,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
1,
0,
0,
0,
1,
0,
0,
1,
0,
1,
0,
0,
0,
0,
0
],
[
0,
302,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
2,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
[
0,
0,
307,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
366,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
1,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
0,
373,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
0,
0,
390,
0,
2,
1,
0,
0,
0,
0,
0,
0,
0,
0,
0,
1,
1,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
[
1,
0,
0,
0,
0,
1,
335,
0,
0,
1,
0,
0,
0,
1,
0,
0,
0,
0,
0,
0,
0,
1,
0,
1,
1,
0,
0,
0,
0,
0
],
[
1,
0,
0,
0,
0,
0,
0,
308,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
1,
0,
0,
0,
0,
0,
0
],
[
0,
0,
1,
0,
0,
1,
0,
0,
376,
0,
0,
2,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
0,
0,
0,
0,
0,
0,
327,
0,
0,
0,
0,
0,
5,
0,
0,
0,
0,
0,
1,
0,
0,
0,
0,
0,
0,
0,
0
],
[
1,
0,
0,
0,
0,
0,
1,
0,
0,
0,
303,
0,
0,
0,
0,
0,
2,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
368,
0,
0,
0,
0,
0,
1,
0,
0,
0,
0,
0,
0,
0,
0,
1,
0,
0,
0
],
[
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
326,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
4,
0,
0
],
[
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
324,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
1,
0
],
[
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
324,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
3
],
[
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
269,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
0,
0,
0,
0,
0,
0,
1,
0,
0,
0,
0,
0,
0,
299,
0,
0,
1,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
[
0,
2,
0,
0,
0,
0,
0,
0,
0,
0,
1,
0,
0,
0,
0,
0,
0,
318,
0,
1,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
[
3,
0,
0,
1,
0,
1,
0,
0,
0,
0,
0,
0,
2,
1,
0,
3,
0,
0,
247,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
[
3,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
1,
0,
0,
2,
0,
0,
315,
0,
0,
0,
0,
1,
0,
0,
0,
0,
0
],
[
0,
0,
1,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
1,
0,
0,
2,
0,
0,
374,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
1,
0,
0,
4,
0,
0,
2,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
327,
0,
0,
1,
0,
0,
0,
0,
0
],
[
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
335,
0,
1,
0,
0,
0,
2,
0
],
[
0,
1,
0,
0,
0,
0,
1,
2,
1,
0,
1,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
349,
0,
0,
0,
0,
0,
0
],
[
1,
0,
0,
0,
0,
0,
4,
0,
0,
1,
1,
0,
1,
0,
0,
4,
1,
0,
1,
2,
0,
0,
0,
0,
280,
0,
0,
0,
0,
0
],
[
2,
0,
0,
0,
0,
0,
0,
0,
0,
0,
2,
1,
0,
0,
1,
0,
2,
2,
0,
0,
0,
0,
0,
0,
0,
347,
0,
0,
0,
5
],
[
0,
0,
1,
0,
0,
1,
0,
0,
1,
0,
0,
5,
0,
0,
0,
0,
0,
1,
0,
0,
2,
0,
0,
2,
0,
0,
297,
0,
0,
6
],
[
0,
0,
0,
0,
2,
0,
0,
0,
0,
0,
0,
0,
1,
1,
0,
1,
0,
0,
0,
0,
0,
1,
0,
0,
1,
0,
0,
268,
0,
0
],
[
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
2,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
365,
0
],
[
0,
1,
1,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
2,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
363
]
]
}

BIN
results/mnist_collapse/mnist_collapse.pdf Normal file
View file

Binary file not shown.

BIN
results/mnist_collapse/mnist_collapse.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 278 KiB

BIN
results/mnist_collapse/mnist_montage.pdf Normal file
View file

Binary file not shown.

BIN
results/mnist_collapse/mnist_montage.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 182 KiB

110
results/mnist_collapse/resolved_config.yaml Normal file
View file

@ -0,0 +1,110 @@
experiment: mnist_collapse
seed: 20260705
n_replicates: 4
source_config:
experiment: mnist_collapse
kind: mnist_lineage
seed: 20260705
n_replicates: 4
mnist:
K: 30
n_classes: 10
style_bins: 3
R: 1
tail: zipf
zipf_s: 1.5
tail_threshold: 0.01
init: truth
data_root: data
model:
kind: convvae
latent: 32
epochs: 30
lr: 0.001
batch_size: 256
beta: 1.0
n_eval: 10000
oracle:
epochs: 5
lr: 0.001
batch_size: 256
cache: models/mnist_cnn.pt
dynamics:
n: 6000
grounding:
m: 0
policy: proportional
generations: 15
metrics:
kl_floor: 1.0e-09
support_eps: 1.0e-09
sweep:
- param: g
values:
- 0.0
- 0.1
output:
dir: results/mnist_collapse
grid:
- label:
g: 0.0
m: 0
lineage_cfg:
mnist:
K: 30
n_classes: 10
style_bins: 3
R: 1
tail: zipf
zipf_s: 1.5
tail_threshold: 0.01
init: truth
data_root: data
model:
kind: convvae
latent: 32
epochs: 30
lr: 0.001
batch_size: 256
beta: 1.0
n_eval: 10000
dynamics:
n: 6000
grounding:
m: 0
policy: proportional
generations: 15
metrics:
kl_floor: 1.0e-09
support_eps: 1.0e-09
- label:
g: 0.1
m: 667
lineage_cfg:
mnist:
K: 30
n_classes: 10
style_bins: 3
R: 1
tail: zipf
zipf_s: 1.5
tail_threshold: 0.01
init: truth
data_root: data
model:
kind: convvae
latent: 32
epochs: 30
lr: 0.001
batch_size: 256
beta: 1.0
n_eval: 10000
dynamics:
n: 6000
grounding:
m: 667
policy: proportional
generations: 15
metrics:
kl_floor: 1.0e-09
support_eps: 1.0e-09