Layer 1.5: architecture-general neural existence proof

Re-scopes Layer 2 into a cheaper, architecture-general neural collapse proof
before the LLM rung. Realises the same Wright–Fisher abstractions in real trained
generative models on a fully-synthetic sandbox with an exact oracle, reusing
knowledge.metrics/truth/seeding and the output contract so neural curves overlay
the Layer-1 analytic curves.

  - src/neural/: synthetic token-grammar sandbox (lossless identity + stochastic
    style), ExactOracle, HistogramModel bridge, generation loop, experiment runner
  - HARD GATE passed: histogram lineage reproduces Layer 1 exactly (neutral decay,
    exact H_eq, tracks run_lineage) — tests/test_neural_validation.py
  - torch models: autoregressive RNN + MLP (VAE implemented, not yet fidelity-
    passing); determinism seeding derived from the SeedSequence stream
  - N0 bridge (neural g*=0.047 ≈ Layer-1 0.048), N1 collapse-in-weights, N2 phase
    boundary, N5 architecture-generality (collapse + grounding-rescue in histogram
    + RNN + MLP). Manifests/configs committed; parquet gitignored, hashes tracked
  - additive backward-compatible save_artifacts extension; Makefile neural targets

Finding: neural smoothing partially resists H-collapse, so forward-KL and tail
survival are the sharp neural collapse metrics (H is smooth, per Layer 1).

92 tests green. Remaining (tasks/todo.md): N4 merge, N2 refine, N3/N6, VAE
fidelity, MNIST tier, figures. LLM/LoRA rung and C3 deferred.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Giorgio Gilestro 2026-07-04 21:02:49 +01:00
parent 1721d047fa
commit 840b6b00b3
35 changed files with 3679 additions and 23 deletions

View file

@ -211,11 +211,28 @@ def _content_hash(path: Path) -> str:
return h.hexdigest()
def save_artifacts(cfg: dict, df: pd.DataFrame, out_dir: Path) -> None:
def save_artifacts(cfg: dict, df: pd.DataFrame, out_dir: Path,
extra_libs: tuple[str, ...] = (),
extra_manifest: dict | None = None,
grid: list | None = None) -> None:
"""Write the reproducibility output contract (blueprint 2.7 / 4).
Writes ``results.parquet``, ``resolved_config.yaml`` (the fully-expanded config), and
``manifest.json`` (library versions, master seed, git commit, content hash).
Args:
cfg (dict): The parsed experiment config.
df (pd.DataFrame): The long-form results.
out_dir (Path): Output directory.
extra_libs (tuple[str, ...]): Extra library names to record versions for (e.g.
``torch``, ``torchvision`` for Layer 1.5). Missing libraries are skipped, so a
caller can pass optional deps unconditionally.
extra_manifest (dict | None): Extra key/value pairs to merge into the manifest
(e.g. model architecture, oracle checkpoint hash, determinism flags).
grid (list | None): Pre-expanded ``[{label, lineage_cfg}, ...]`` to record in the
resolved config. If None, it is computed via ``expand_sweeps`` for the Layer-1
``lineage`` kind (a caller with a different schema, e.g. Layer 1.5, passes its
own expanded grid here).
"""
out_dir.mkdir(parents=True, exist_ok=True)
results_path = out_dir / "results.parquet"
@ -227,24 +244,32 @@ def save_artifacts(cfg: dict, df: pd.DataFrame, out_dir: Path) -> None:
"n_replicates": cfg["n_replicates"],
"source_config": cfg,
}
if cfg.get("kind", "lineage") == "lineage":
if grid is not None:
resolved["grid"] = grid
elif cfg.get("kind", "lineage") == "lineage":
resolved["grid"] = [
{"label": label, "lineage_cfg": lineage_cfg}
for label, lineage_cfg in expand_sweeps(cfg)
]
(out_dir / "resolved_config.yaml").write_text(yaml.safe_dump(resolved, sort_keys=False))
libraries: dict[str, str] = {}
for lib in ("numpy", "scipy", "pandas", "pyarrow") + tuple(extra_libs):
try:
libraries[lib] = version(lib)
except Exception: # optional dep not installed -> omit rather than crash
pass
manifest = {
"experiment": cfg["experiment"],
"master_seed": cfg["seed"],
"git_commit": _git_commit(),
"python": sys.version.split()[0],
"libraries": {
lib: version(lib) for lib in ("numpy", "scipy", "pandas", "pyarrow")
},
"libraries": libraries,
"rows": int(len(df)),
"results_sha256": _content_hash(results_path),
}
if extra_manifest:
manifest.update(extra_manifest)
(out_dir / "manifest.json").write_text(json.dumps(manifest, indent=2))