Layer 1.5: architecture-general neural existence proof

Re-scopes Layer 2 into a cheaper, architecture-general neural collapse proof
before the LLM rung. Realises the same Wright–Fisher abstractions in real trained
generative models on a fully-synthetic sandbox with an exact oracle, reusing
knowledge.metrics/truth/seeding and the output contract so neural curves overlay
the Layer-1 analytic curves.

  - src/neural/: synthetic token-grammar sandbox (lossless identity + stochastic
    style), ExactOracle, HistogramModel bridge, generation loop, experiment runner
  - HARD GATE passed: histogram lineage reproduces Layer 1 exactly (neutral decay,
    exact H_eq, tracks run_lineage) — tests/test_neural_validation.py
  - torch models: autoregressive RNN + MLP (VAE implemented, not yet fidelity-
    passing); determinism seeding derived from the SeedSequence stream
  - N0 bridge (neural g*=0.047 ≈ Layer-1 0.048), N1 collapse-in-weights, N2 phase
    boundary, N5 architecture-generality (collapse + grounding-rescue in histogram
    + RNN + MLP). Manifests/configs committed; parquet gitignored, hashes tracked
  - additive backward-compatible save_artifacts extension; Makefile neural targets

Finding: neural smoothing partially resists H-collapse, so forward-KL and tail
survival are the sharp neural collapse metrics (H is smooth, per Layer 1).

92 tests green. Remaining (tasks/todo.md): N4 merge, N2 refine, N3/N6, VAE
fidelity, MNIST tier, figures. LLM/LoRA rung and C3 deferred.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

src/neural/oracle.py

@@ -0,0 +1,82 @@
+"""The oracle — the neural analogue of Layer 1's "reality's no".
+
+An oracle maps an observation to the mode it belongs to. For the fully-synthetic sandbox
+the oracle is **exact** (it decodes the lossless identity segment), so the measured mode
+distribution ``p_hat`` carries zero measurement noise — this is what lets a trained model's
+collapse be read directly against the known ``p*``. (The MNIST tier will add a
+``ClassifierOracle`` wrapping a frozen network plus its confusion matrix; that arrives with
+Stage C and is confirmation-only.)
+"""
+
+from __future__ import annotations
+
+from typing import Protocol, runtime_checkable
+
+import numpy as np
+
+from .config import SyntheticCfg
+
+
+@runtime_checkable
+class Oracle(Protocol):
+    """Adjudicates which mode an observation belongs to."""
+
+    def classify(self, X: np.ndarray) -> np.ndarray:
+        """Return the length-``n`` mode index for each row of ``X``."""
+        ...
+
+
+class ExactOracle:
+    """Zero-error oracle for the synthetic sandbox: decodes the identity segment.
+
+    Args:
+        cfg (SyntheticCfg): The synthetic configuration whose grammar produced ``X``.
+    """
+
+    def __init__(self, cfg: SyntheticCfg) -> None:
+        self.cfg = cfg
+        self.id_len = cfg.id_len
+        self.base = cfg.id_base
+        self.K = cfg.K
+        # Positional weights for base-`base` decoding, most-significant digit first.
+        self._weights = self.base ** np.arange(self.id_len - 1, -1, -1, dtype=np.int64)
+
+    def classify(self, X: np.ndarray) -> np.ndarray:
+        """Decode mode indices from the identity segment of each observation.
+
+        Args:
+            X (np.ndarray): Token sequences of shape ``(n, seq_len)``.
+
+        Returns:
+            np.ndarray: Length-``n`` decoded indices. Grammar-valid data always decodes into
+            ``[0, K)``; a *neural* model may emit an invalid codeword that decodes to
+            ``>= K`` — such samples are dropped by :func:`measure_distribution` rather than
+            being clipped onto a real mode (which would bias ``p_hat``).
+        """
+        X = np.asarray(X, dtype=np.int64)
+        ident = X[:, : self.id_len]
+        return ident @ self._weights
+
+
+def measure_distribution(X: np.ndarray, oracle: Oracle, K: int) -> np.ndarray:
+    """Measure the empirical mode distribution ``p_hat`` of a sample.
+
+    This is the neural readout of ``p_t``: classify every observation and normalise the
+    mode histogram. Modes absent from ``X`` receive zero mass (support shrinks exactly as
+    in Layer 1). Invalid codewords (decoded index ``>= K``, only producible by a neural
+    model) are dropped, so ``p_hat`` is renormalised over grammar-valid samples.
+
+    Args:
+        X (np.ndarray): Token sequences of shape ``(n, seq_len)``.
+        oracle (Oracle): The mode adjudicator.
+        K (int): Number of modes.
+
+    Returns:
+        np.ndarray: Length-``K`` probability vector summing to 1.
+    """
+    modes = oracle.classify(X)
+    counts = np.bincount(modes, minlength=K)[:K].astype(float)
+    total = counts.sum()
+    if total <= 0:
+        raise ValueError("measure_distribution received an empty sample")
+    return counts / total