Giorgio Gilestro
a6eb9b7512
Scaffold plus the Layer 1 analytical core and the first two experiments. - knowledge/: truth, metrics, teachers (2.7.1 shared-switch construction), step, lineage, experiment, config, seeding (imported as `knowledge`). - Validation spine green: neutral decay (Pred 1), fixation (Pred 2), exact mutation-drift equilibrium (Pred 3), union coverage (Pred 5). 68 tests pass. - E1 reproduces tail-first collapse. E2 delivers the headline: a grounding phase boundary g* << 1, with stationary H tracking the exact H_eq closed form (g=0.005 -> 68% of truth diversity; g=0.05 -> 96%). - Reproducibility: uv venv from a hash-pinned uv.lock is the source of truth; every run writes results.parquet + resolved_config.yaml + manifest.json (lib versions, git commit, sha256). Figures and manifests tracked; the large regenerable parquet is gitignored. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
93 lines
10 KiB
Markdown
93 lines
10 KiB
Markdown
# CLAUDE.md
|
||
|
||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||
|
||
## Current state: greenfield
|
||
|
||
This repository currently contains **only two design documents and no code**. The task is to implement the study those documents specify.
|
||
|
||
- `the-lamarckian-society-v4.md` — the *perspective paper*. The conceptual thesis: a multigenerational architecture of continual-learning agents that mature, teach, and evolve. Read this for the "why."
|
||
- `lamarckian-society-technical-blueprint-v1.md` — the *technical blueprint*. A build specification written to be handed to a coding agent. Read this for the "what" and "how." **It is normative**: module names, function signatures, config schema, experiment IDs, and directory layout in it are the contract to implement against.
|
||
|
||
Everything below summarizes the blueprint so you can orient fast, but the blueprint is the source of truth. When they conflict, the blueprint wins; when the blueprint is silent, minimize decisions and match its established patterns.
|
||
|
||
## The one idea you must hold in your head
|
||
|
||
**Knowledge transmission across agent generations is modelled *literally* as a Wright–Fisher population-genetics process** — not by analogy. A model's knowledge is a distribution `p_t` over `K` discrete items on a simplex; a fixed true distribution `p*` has a rare **tail**; each generational step is "sample from parent (drift) + mix in fresh real samples (immigration/grounding) + refit." Model collapse = loss of rare alleles under drift. Every experiment is a manipulation of this single process.
|
||
|
||
The population-genetics dictionary in blueprint §1 is the spine. Keep its abstractions **identical across both layers** — this is a hard requirement, because it is the only thing that lets a Layer-2 neural result count as confirming a Layer-1 analytic prediction:
|
||
|
||
| Abstraction | Layer 1 (analytic) | Layer 2 (neural) |
|
||
|---|---|---|
|
||
| region | disjoint block of the `K` items | task family (e.g. string ops, recursion) |
|
||
| rarity / tail | low `p*` items | low-frequency task types |
|
||
| grounding fraction `g` | `m/(n+m)` real-vs-inherited samples | proportion of verifier-passed items in pupil's training mix |
|
||
| decorrelation `ρ` | shared retained-tail correlation between teachers | LoRA specialists on disjoint task families |
|
||
| diversity `H` | heterozygosity `1 − Σ pᵢ²` | solution diversity of generated code |
|
||
| reality's "no" | grounding against `p*` | execution-based unit-test verifier |
|
||
|
||
## Two layers, staged by cost
|
||
|
||
- **Layer 1 — analytical core** (`src/knowledge/`). Pure NumPy/SciPy Wright–Fisher simulator. Laptop, minutes, no GPU. Carries the paper's quantitative claims. **Three of the five §2.4 predictions are closed-form**, so validation is an **exact** test, not a vibe check — these become `<0.1%`-tolerance assertions in `test_scientific_validation.py`:
|
||
- **Pred. 1** — neutral heterozygosity decay: `E[Hₜ] = H₀(1 − 1/n)ᵗ`.
|
||
- **Pred. 3** — *exact* mutation–drift equilibrium for the implemented immigration model: `H_eq = H* · m(2n+m−1)/(n+2nm+m²)`, with `H* = 1 − Σ(p*ᵢ)²`. The textbook `θ/(1+θ)` (θ=2m) is only the rare-immigrant limit. **Critical nuance: H is *smooth* in m — the sharp phase threshold lives in discrete tail-item survival (Pred. 4: an item survives iff `m·p*ᵢ ≳ 1`), not in H.** Do not describe E2 as a discontinuity in H.
|
||
- **Pred. 5** — closed-form recombination benefit: `U(K_T, ρ, q) = T[ρq + (1−ρ)(1−(1−q)^K_T)]` (expected tail items retained by ≥1 of K_T teachers).
|
||
- **Layer 2 — neural existence proof** (`src/neural/`). Small open-weight models (default OLMo-2-1B / SmolLM2-1.7B, fallback Qwen2.5-1.5B-Instruct; pin the HF revision hash, never track `main`), LoRA specialisation, distillation/merging across 2–3 generations, program-synthesis-with-unit-tests as the verifier. One consumer GPU. Only needs to show the **sign** of three effects, not precise magnitudes.
|
||
|
||
## Experiments and their falsifiers
|
||
|
||
Each experiment is one config file → one runner invocation → one `results.parquet` → one figure. Every experiment has a **falsifier** — an outcome that would refute the corresponding claim. The design is built to be able to kill the thesis; preserve that.
|
||
|
||
- Layer 1: **E1** reproduce collapse (null), **E2** grounding phase boundary (headline: is there a critical `g* ≪ 1`?), **E3** region-matched grounding, **E4** multi-teacher decorrelation, **E5** quality-diversity vs. greedy selection, **E6** re-minting gate / irreversibility.
|
||
- Layer 2: **C1** dry vs. grounded, **C2** one vs. N complementary teachers at matched budget, **C3** the vertical claim (general knowledge climbs while each specialty is re-earned and *exceeded* — this is load-bearing, prioritize it), **C4** distillation vs. merging (optional).
|
||
|
||
Blueprint §6 is the claim→experiment→figure→falsifier traceability matrix and is the definition of done.
|
||
|
||
## The one non-obvious implementation piece: the correlated-teacher construction (§2.7.1)
|
||
|
||
E4's whole purpose is to isolate the effect of teacher **decorrelation ρ**, so ρ must be a *directly constructed, independently-swept* knob — **never** an emergent quantity you get by tuning drift (that ρ would be confounded with `n`, `m`, tail size, and generation count, i.e. with the very drift E4 holds fixed). The construction is a **shared-switch exchangeable Bernoulli**: for each of the `T` tail items, draw a shared switch `z~Bern(ρ)`, a shared retention `s~Bern(q)`, and per-teacher independent `u⁽ᵏ⁾~Bern(q)`; set teacher k's retention `r⁽ᵏ⁾ = s if z else u⁽ᵏ⁾`. This yields exact marginal retention `q` and exact pairwise correlation `ρ` (provable: Cov = ρq(1−q), Var = q(1−q)), and is exchangeable so ρ is a single scalar. `make_retention_matrix(T, K_T, rho, q, rng)` returns the `(K_T, T)` binary matrix; `make_correlated_teachers` maps it to distributions (head items always kept at `p*`; tail item kept at `p*ᵢ` if retained, else `tail_floor`; renormalise so dropped-tail mass flows to survivors). The exact-construction path is **preferred** for E4; the drift-based path exists only as a realism cross-check. `region_specialisation=True` forces full retention of a teacher's home-region tails and applies the ρ construction only off-home.
|
||
|
||
**E4 reports two coverages, and their gap is a result, not noise:** the construction-level union `U(K_T,ρ,q)` (must match the closed form exactly) and the post-distillation *surviving* coverage after the pupil's size-`n` resampling. A tail item present in the mixture only survives if its mixture mass clears `~1/n` (Pred. 4) — so the gap is precisely "the tail recombination *supplied* but drift *re-erased* because grounding was too thin," which ties E4 back to E2/E3.
|
||
|
||
## Build order (blueprint §7) — respect the gate
|
||
|
||
1. Scaffold: repo layout (§5), container, pytest skeleton, config system, seeding utils. `make test` green.
|
||
2. Layer 1 core + null model + `test_scientific_validation.py` against §2.4 predictions 1–2.
|
||
**HARD GATE: do not proceed until simulated drift matches the analytic heterozygosity decay `E[Hₜ] = H₀(1 − 1/n)ᵗ`.**
|
||
3. Layer 1 grounding + E1–E2 (the headline result).
|
||
4. Layer 1 E3–E6. Layer 1 is now a complete laptop-reproducible paper on its own.
|
||
5. Layer 2 scaffold + verifier (test determinism & sandbox isolation *before* any training).
|
||
6. Layer 2 C1 + C3.
|
||
7. Layer 2 C2 (+ C4 if compute allows).
|
||
8. Reproduction pass.
|
||
|
||
**Do not start Layer 2 until Layer 1's scientific-validation tests pass.**
|
||
|
||
## Prescribed structure and commands (do not yet exist — create per blueprint §4–5)
|
||
|
||
Target module interfaces are given with **normative names** in blueprint §2.7 (Layer 1) and §3.6 (Layer 2); downstream scripts depend on these signatures, so implement to them exactly. Target repo layout is §5. Planned automation:
|
||
|
||
```
|
||
make env # uv sync -> .venv from committed uv.lock
|
||
make test # correctness tests + scientific-validation tests
|
||
make layer1 # run E1–E6
|
||
make layer2 # run C1–C3 (C4 optional)
|
||
make figures # regenerate every figure from committed results.parquet
|
||
make all
|
||
./reproduce.sh # uv sync → test → run all at committed seeds → regen figures → REPRODUCED.md
|
||
```
|
||
|
||
Single-experiment run pattern: one YAML config per experiment under `configs/layer1/EX.yaml` or `configs/layer2/CX.yaml`, fed to the experiment runner. Figures are regenerated separately by `figures/plot_EX.py` reading **only** `results.parquet` (no re-simulation).
|
||
|
||
## Non-negotiable engineering standard (blueprint §4)
|
||
|
||
- **Reproducibility is a hard requirement, not a preference (this is a paper).** The environment is a **`uv` venv built from a committed, hash-pinned `uv.lock`** — that lockfile is the source of truth for "it runs" (Apptainer is dropped; a Dockerfile may later wrap the same lockfile for Layer 2's GPU work). Layer 1 is bitwise-reproducible from a single master seed; Layer 2 is *statistically* reproducible (document residual GPU non-determinism, set determinism flags, report per-seed points).
|
||
- **Seeding:** one master seed in config → derive all sub-seeds via `np.random.SeedSequence.spawn`. Never touch global RNG state; pass `rng` explicitly everywhere. Results are a pure function of the resolved config.
|
||
- **No magic numbers in code.** Every parameter lives in a YAML resolved at run time; the *resolved* config (after sweep expansion) is written next to results. Sweeps are declared in config, not hard-coded.
|
||
- **Output contract for every run:** `results.parquet` (long form) + `resolved_config.yaml` + `manifest.json` (library/CUDA versions, seed, git commit, model revision hashes, content hash of results). Every figure must be a pure function of a committed results artifact.
|
||
- **Scientific-validation tests are the spine of trust.** They assert the simulator reproduces the §2.4 closed forms within tolerance; if they fail, the *science* is wrong, not just the code. Keep them.
|
||
- **Open science end-to-end:** open-weight models only, permissive/open tooling (uv, MLflow or plain versioned Parquet — avoid closed SaaS trackers), `results/` gitignored but hashes tracked.
|
||
|
||
## Stack
|
||
|
||
Python ≥ 3.11. Layer 1: NumPy, SciPy, pandas, matplotlib — no GPU, no heavy deps. Layer 2: PyTorch, HF `transformers` + `peft` (LoRA), `datasets`, optional `vllm`; sandboxed `subprocess` verifier. Config via a thin pydantic + PyYAML loader (not Hydra — its global state/`chdir` fights the pure-function-of-resolved-config contract). Env via a **`uv` venv from a committed `uv.lock`** — the lockfile is the reproducibility source of truth; Layer 1 needs no container.
|