Layer 1.5: architecture-general neural existence proof

Re-scopes Layer 2 into a cheaper, architecture-general neural collapse proof
before the LLM rung. Realises the same Wright–Fisher abstractions in real trained
generative models on a fully-synthetic sandbox with an exact oracle, reusing
knowledge.metrics/truth/seeding and the output contract so neural curves overlay
the Layer-1 analytic curves.

  - src/neural/: synthetic token-grammar sandbox (lossless identity + stochastic
    style), ExactOracle, HistogramModel bridge, generation loop, experiment runner
  - HARD GATE passed: histogram lineage reproduces Layer 1 exactly (neutral decay,
    exact H_eq, tracks run_lineage) — tests/test_neural_validation.py
  - torch models: autoregressive RNN + MLP (VAE implemented, not yet fidelity-
    passing); determinism seeding derived from the SeedSequence stream
  - N0 bridge (neural g*=0.047 ≈ Layer-1 0.048), N1 collapse-in-weights, N2 phase
    boundary, N5 architecture-generality (collapse + grounding-rescue in histogram
    + RNN + MLP). Manifests/configs committed; parquet gitignored, hashes tracked
  - additive backward-compatible save_artifacts extension; Makefile neural targets

Finding: neural smoothing partially resists H-collapse, so forward-KL and tail
survival are the sharp neural collapse metrics (H is smooth, per Layer 1).

92 tests green. Remaining (tasks/todo.md): N4 merge, N2 refine, N3/N6, VAE
fidelity, MNIST tier, figures. LLM/LoRA rung and C3 deferred.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

tasks/todo.md

@@ -123,6 +123,79 @@ Design decisions #1 (dataclasses now / pydantic at YAML layer), #2 (fitness `f_i
 - **E2 analysis add-ons** (companion work order `tasks/workorder-E2-analysis-addons.md`, verified): new `analysis.py` (`reduce_to_stationary`, `critical_grounding` bootstrap CI) — real E2 **g*=0.048, CI [0.047,0.050]**; `metrics.tail_band_metrics` + per-band lineage logging; `tests/test_analysis.py` reproduces the work order's verified numbers exactly. E2 figure rebuilt 2×2. **Deviation:** used truth-mass-weighted tail coverage instead of raw `tail_mass` (a drift martingale).
 - All six figures regenerate via `make figures`; **71 tests green**.
 
+---
+
+# Layer 1.5 — Architecture-general neural existence proof (RNN/VAE/MLP + synthetic/MNIST)
+
+*Created 2026-07-04. Plan: `~/.claude/plans/we-are-going-to-cheerful-fog.md`. Re-scopes Layer 2:
+build a cheap, architecture-general neural collapse proof in real trained weights on a
+fully-synthetic sandbox (exact known `p*`) before the LLM rung. Locked decisions: exact-oracle
+categorical token sequences; Histogram+RNN+VAE+MLP; real MNIST as secondary confirmation; LLM +
+C3 vertical claim deferred.*
+
+## Progress log
+
+**2026-07-04 — Stages A, B, plumbing complete.**
+
+- **Env:** installed `uv` 0.11.26 (`~/.local/bin`); `/home` was 100% full — GG approved clearing
+  pip/yay/browser caches (~10 GB freed). Base venv synced; 71 Layer-1 tests green.
+- **Stage A (scaffold, pure NumPy):** `src/neural/` — `config.py` (frozen dataclasses reusing
+  `knowledge.config` GroundingCfg/RemintCfg/MetricsCfg/_sub), `synthetic.py` (mode-truth via
+  `make_true_distribution`; lossless identity + stochastic style token grammar), `oracle.py`
+  (`ExactOracle` zero-error + `measure_distribution`), `models.py` (`GenerativeModel` protocol +
+  `HistogramModel` bridge), `evaluate.py` (reuses `knowledge.metrics`, Layer-1 row schema),
+  `generation_loop.py` (`run_generative_lineage`, reuses `allocate_m`/`structured_multinomial`).
+  15 correctness tests green.
+- **Stage B — HARD GATE PASSED:** `tests/test_neural_validation.py` — histogram lineage reproduces
+  Pred. 1 (neutral decay, <3% rel err), Pred. 3 (exact `H_eq`, <5%), and tracks Layer-1
+  `run_lineage` directly (<3%). The neural plumbing reproduces the analytic core.
+- **Plumbing:** `neural/experiment.py` (`run_and_save` dispatch on `kind`, reuses `_apply_param`
+  g→m, paired seeds); extended `knowledge.experiment.save_artifacts` (optional `extra_libs`,
+  `extra_manifest`, injectable `grid`; skips missing libs — backward compatible). `configs/neural/N0.yaml`,
+  Makefile `neural`/`env-neural`/`layer2` targets, `.gitignore`.
+- **N0 result (bridge, 17s):** neural **g\* = 0.0474, CI [0.045, 0.052]** — reproduces Layer-1 E2's
+  g\*=0.048 essentially exactly (g=0.005→67% of H*, g=0.05→96%). **89 tests green.**
+
+**2026-07-04 — Stage C: torch models + N1/N2/N5.**
+
+- **Env:** torch **2.12.1+cu130** (default PyPI wheel ships CUDA 13, matches RTX A4000 driver;
+  no custom index needed, cp314 wheels exist). `--extra neural` = torch only; `--extra mnist` =
+  torchvision (later). `UV_CACHE_DIR=/tmp` during install (RAM-backed) to spare `/home`.
+- **Models:** `torch_models.py` (RNNGenerator, autoregressive GRU), `torch_mlp.py` (autoregressive
+  MLP, causal-masked), `torch_vae.py` (sequence VAE), `train.py` (determinism flags + device/seed
+  helpers derived from the SeedSequence stream). `tests/test_neural_torch.py` (torch-gated): gen-0
+  fidelity (rnn+mlp) + dry-collapse/grounded-holds. **92 tests green.**
+- **Validated regime:** K=256, n=200, zipf_s=1.3, RNN hidden=128/epochs=25. RNN gen-0 fidelity
+  KL(p*‖p̂)=0.008, 64/64 (or 256/256) modes recovered. MLP fidelity KL=0.011. **VAE does NOT clear
+  the gen-0 gate** on the Zipf-codeword task (KL≈0.8; prior-hole mismatch — sampling z~N(0,I) misses
+  the aggregate posterior) → excluded from N5 to avoid confounding collapse with underfitting.
+- **N1 (collapse in weights):** dry RNN lineage collapses — forward-KL rises to ~2.2 vs grounded
+  ~1.4; grounding lifts tail survival (tailalive 0.31 dry → 0.50 at g=0.02). Sign confirmed.
+- **N2 (neural phase boundary):** stationary H hovers 80–91% of H* and is **noisy / non-monotonic**
+  at 5 reps — no crisp g*. **KEY FINDING:** the neural models' smoothing inductive bias *partially
+  resists* H-collapse (dry H stays ~83% of H*), so **forward-KL and tail survival are the sharp
+  neural collapse metrics, not H** (mirrors Layer-1's "H is smooth; the threshold lives in tail
+  survival"). N2 needs (a) forward-KL as the phase metric, (b) more reps (≥10), and/or (c) a
+  stronger-collapse regime for a clean neural g*.
+- **N5 (architecture-generality) — clean result:** collapse + grounding-rescue appear in ALL three
+  model classes (dry→grounded forward-KL: histogram 6.2→4.6, MLP 4.8→1.3, RNN 3.8→1.1; tailalive
+  RNN 0.41→0.64, MLP 0.07→0.20). The WF operator is architecture-general. Bonus: neural smoothing
+  lets RNN/MLP retain *more* tail than the exact histogram under grounding (they generalise to
+  unseen codewords) — an inductive-bias finding worth the write-up.
+
+## Remaining
+
+- [ ] **N2 refinement:** re-run with forward-KL as the phase metric + ≥10 reps (and/or smaller n)
+  for a clean neural g*. Pin the falsifier ("g* ≪ 1 exists") before re-running.
+- [ ] **N4 (load-bearing):** `recombine.py` — mean-mixture vs union-preserving merge. The neural
+  merge MUST be **oracle-guided mixture sampling** (sample from the teacher strongest on each mode),
+  NOT weight-averaging of recurrent nets (flag #7). Reproduce the E4 "mean flat, max rises" finding.
+- [ ] **N3** region-matched grounding (R>1), **N6** re-mint gate (optional).
+- [ ] **VAE fidelity:** fix the prior-hole mismatch (KL-annealing / free-bits / larger latent) so it
+  clears the gen-0 gate, then add to N5. Or document as a known limitation.
+- [ ] Real-MNIST secondary tier (`ClassifierOracle` + confusion matrix; `--extra mnist`).
+- [ ] `figures/plot_N*.py` (reuse `figures/_figlib.py`); wire into `make figures`.
+
 ## Discovered during work
 
 - **E2 grounding policy vs. the analytic H_eq:** Pred. 3's closed form is derived for *plain* immigration `Multinomial(m, p*)`. Implemented as `policy="proportional"`, and every policy reduces to it at `R=1`. E2 should therefore run at `R=1` (or `proportional`) so the phase-boundary sweep tracks the exact `H_eq`; region structure is E3's concern. Decide E2's `init` (uniform vs truth) when building its config.