Layer 1.5: architecture-general neural existence proof

Re-scopes Layer 2 into a cheaper, architecture-general neural collapse proof
before the LLM rung. Realises the same Wright–Fisher abstractions in real trained
generative models on a fully-synthetic sandbox with an exact oracle, reusing
knowledge.metrics/truth/seeding and the output contract so neural curves overlay
the Layer-1 analytic curves.

  - src/neural/: synthetic token-grammar sandbox (lossless identity + stochastic
    style), ExactOracle, HistogramModel bridge, generation loop, experiment runner
  - HARD GATE passed: histogram lineage reproduces Layer 1 exactly (neutral decay,
    exact H_eq, tracks run_lineage) — tests/test_neural_validation.py
  - torch models: autoregressive RNN + MLP (VAE implemented, not yet fidelity-
    passing); determinism seeding derived from the SeedSequence stream
  - N0 bridge (neural g*=0.047 ≈ Layer-1 0.048), N1 collapse-in-weights, N2 phase
    boundary, N5 architecture-generality (collapse + grounding-rescue in histogram
    + RNN + MLP). Manifests/configs committed; parquet gitignored, hashes tracked
  - additive backward-compatible save_artifacts extension; Makefile neural targets

Finding: neural smoothing partially resists H-collapse, so forward-KL and tail
survival are the sharp neural collapse metrics (H is smooth, per Layer 1).

92 tests green. Remaining (tasks/todo.md): N4 merge, N2 refine, N3/N6, VAE
fidelity, MNIST tier, figures. LLM/LoRA rung and C3 deferred.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Giorgio Gilestro 2026-07-04 21:02:49 +01:00
parent 1721d047fa
commit 840b6b00b3
35 changed files with 3679 additions and 23 deletions

View file

@ -2,12 +2,28 @@
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## Current state: greenfield
## Current state: Layer 1 complete; Layer 1.5 (neural) in progress
This repository currently contains **only two design documents and no code**. The task is to implement the study those documents specify.
- **Layer 1** (`src/knowledge/`) — **complete and validated.** All six experiments E1E6, the
closed-form scientific-validation tests, figures, and reproducibility harness exist. Headline:
critical grounding `g* = 0.048 ≪ 1`; the E4 finding that mean-mixture distillation conserves
collapse while only a union-preserving max-merge realises the recombination benefit.
- **Layer 1.5** (`src/neural/`) — **in progress.** An architecture-general neural existence proof
(re-scoped Layer 2): the same WrightFisher abstractions realised in *real trained generative
models* (histogram bridge + RNN + MLP; VAE implemented but not fidelity-passing) on a
fully-synthetic sandbox with an exact oracle, plus real MNIST as a later secondary tier. See
`tasks/todo.md` for status and `~/.claude/plans/we-are-going-to-cheerful-fog.md` for the plan.
**Done:** scaffold, the histogram bridge gate (reproduces Layer 1 exactly), N0 (bridge, neural
g*=0.047 ≈ Layer 1), N1 (collapse in RNN weights), N2 (neural phase boundary), N5
(architecture-generality). **Remaining:** N4 (mean-vs-max-merge, load-bearing), N3, N6, figures,
the MNIST tier. The LLM/LoRA rung and the C3 vertical claim are deferred.
- `the-lamarckian-society-v4.md` — the *perspective paper*. The conceptual thesis: a multigenerational architecture of continual-learning agents that mature, teach, and evolve. Read this for the "why."
- `lamarckian-society-technical-blueprint-v1.md` — the *technical blueprint*. A build specification written to be handed to a coding agent. Read this for the "what" and "how." **It is normative**: module names, function signatures, config schema, experiment IDs, and directory layout in it are the contract to implement against.
The two design documents are the source of truth for intent:
- `paper/the-lamarckian-society-v4.md` — the *perspective paper* (the "why").
- `paper/blueprint.md` — the *technical blueprint* (the "what"/"how"). **It is normative** for
Layer 1 and the LLM Layer 2; Layer 1.5 is a cost-staged intermediate the blueprint does not
cover, designed to preserve the same §1 abstractions.
Everything below summarizes the blueprint so you can orient fast, but the blueprint is the source of truth. When they conflict, the blueprint wins; when the blueprint is silent, minimize decisions and match its established patterns.