giorgio/MachineSex
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

neural: real-MNIST external-validity tier (collapse + grounding)

Browse source 
Confirms model collapse and its arrest by grounding on REAL images, not
just the synthetic sandbox. A conv VAE (the canonical generative-collapse
model) is retrained each generation on its own generated digits, with a
fraction g of fresh real MNIST mixed in. Modes = digit class x stroke-
thickness bin (K=30, Zipf, ~18 tail modes); the oracle is a frozen CNN +
deterministic thickness at 98.5% mode accuracy (30x30 confusion matrix
recorded in the manifest as the measurement-noise floor).

Result (4 reps): dry (g=0) collapses to a single mode -- forward-KL
0.5->18, support 30->1, tail 1.0->0.06, H->0 -- while 10% grounding holds
all 30 modes (KL~0.6, full tail, H~0.9). Signs, not magnitudes (blueprint
3.5); the exact synthetic oracle stays the quantitative anchor. The VAE
needs ~10% grounding vs the synthetic histogram's ~5%, consistent with the
grounding finding that trained nets need more than the exact operator.

Plugs into the existing data-agnostic contract (metrics/grounding/output
reused verbatim): mnist_data (thickness bins, class x thickness bijection,
MnistSampler), mnist_oracle (ClassifierOracle + confusion matrix),
mnist_vae (ConvVAEGenerator), mnist_loop (run_mnist_lineage), kind=
mnist_lineage dispatch, MnistCfg/OracleCfg. Figures: plot_mnist (parquet-
only) + mnist_montage (eyeball diagnostic showing digits degenerate to one
blurry mode). make mnist / make env-mnist, kept out of the make neural
loop. 99 tests green (+5 torchvision-gated).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Giorgio Gilestro 2026-07-05 09:19:36 +01:00
parent 3b9f4f7893
commit 79bbc45f41
21 changed files with 2200 additions and 10 deletions
Download patch file Download diff file Expand all files Collapse all files

13
Makefile
View file

@ -1,7 +1,7 @@
# Layer 1 + Layer 1.5 automation. The uv venv (built from the committed uv.lock) is the
# reproducibility source of truth; every target runs inside it via `uv run`.
.PHONY: env env-neural test layer1 layer2 neural figures clean
.PHONY: env env-neural env-mnist test layer1 layer2 neural mnist figures clean
env: ## build .venv from the committed lockfile
uv sync --extra dev
@ -9,14 +9,21 @@ env: ## build .venv from the committed lockfile
env-neural: ## add the Layer 1.5 torch stack (GPU; Stage C onward)
uv sync --extra dev --extra neural
env-mnist: ## add torchvision for the real-MNIST confirmation tier
uv sync --extra dev --extra neural --extra mnist
test: ## correctness tests + scientific-validation tests (the spine of trust)
uv run pytest
layer1: ## run experiments E1-E6
for e in E1 E2 E3 E4 E5 E6; do uv run python -m knowledge.experiment configs/layer1/$$e.yaml; done
neural: ## run Layer 1.5 neural experiments (N-series)
for c in configs/neural/*.yaml; do uv run python -m neural.experiment $$c; done
neural: ## run Layer 1.5 synthetic neural experiments (excludes the heavy MNIST tier)
for c in configs/neural/*.yaml; do case "$$c" in *mnist*) ;; \
*) uv run python -m neural.experiment "$$c" ;; esac; done
mnist: ## run the real-MNIST confirmation tier (needs env-mnist; downloads MNIST)
uv run python -m neural.experiment configs/neural/mnist_collapse.yaml
layer2: neural ## alias: Layer 1.5 is the current Layer-2 deliverable (LLM rung deferred)