Giorgio Gilestro
39f6c9f4df
The easy task families saturated 7B (strings & arith at 1.00), so the earlier
7B nulls — moe: fusion 0.87 > union 0.84; directed ~= soup — could not separate
"refinements don't help at scale" from "tasks too easy at 7B". Adds a hard task
variant (hard: true in tasks.py: multi-step lists, Caesar ciphers / letter
transforms, multi-step & larger arithmetic; same family labels and answer
formats, threaded through make_tasks/train_specialist/runners; hard specialists
cache separately as spec_*_hard) and re-runs both experiments at 7B on Imperial
CX3 (one L40S, 24 min, unsaturated: arith ~0.48, strings 0.67, lists 0.34).
Both nulls flip back to the 0.5B ordering:
- Union beats fusion again: routing 0.500 > fusion 0.40 (soup 0.392 / ties
0.400), the same 10-pt margin as 0.5B. Fusion dilutes the fragile strings
specialist so hard (0.665 -> soup 0.300) that soup even trails the best single
specialist (0.425); routing keeps it intact (0.670).
- Directed selection beats soup again: 0.492 > 0.392 (+10 pts), recovering most
of routing's benefit from one deployable merged model (lifts strings to 0.630).
Correction to the earlier interpretation: the llm_moe_hpc "regime flip" and the
llm_directed_hpc "no headroom" null were driven by TASK SATURATION, not base
capability. The operative variable is headroom — "merge, don't average" (union >
fusion) and "directed sex" (selection > single blend) hold whenever there is room
to lose to dilution: a weak base (0.5B) OR hard tasks at a strong base (7B-hard).
Fusion only wins in the degenerate corner where easy tasks let a strong base
compose to the 1.00 ceiling. Vindicates E8's max > mean in real 7B weights once
saturation is controlled.
Default (easy) task behaviour is unchanged (hard defaults False). +1 hard-task
test (131 green). Excludes the 0.5B smoke bundle (a pipeline gate, not a
deliverable). Results in results/llm_{moe,directed}_hard_hpc/ (parquet gitignored).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|---|---|---|
| configs | ||
| figures | ||
| hpc | ||
| paper | ||
| results | ||
| src | ||
| tasks | ||
| tests | ||
| .gitignore | ||
| CLAUDE.md | ||
| Makefile | ||
| pyproject.toml | ||
| README.md | ||
| uv.lock | ||
The Lamarckian Society — Layer 1 (analytical core)
A parametric population-genetics model of knowledge transmission across generations of
learning agents. Knowledge transmission is modelled literally as a Wright–Fisher
process (not by analogy): a model's knowledge is a distribution p_t over K discrete
items; a fixed true distribution p* has a rare tail; each generational step is
"sample from the parent (drift) + mix in fresh real samples (grounding/immigration) +
refit." Model collapse is the loss of rare alleles under drift.
See paper/blueprint.md (the normative build spec),
paper/the-lamarckian-society-v5.md (the perspective paper), and
paper/results-summary.md (a summary of all results).
Reproduce
Environment is a uv venv built from the committed, hash-pinned uv.lock — that
lockfile is the single source of truth for "it runs" (Layer 1 is pure NumPy/SciPy and
bitwise-reproducible from a seed; no container needed).
# one-time: install uv (https://astral.sh/uv)
curl -LsSf https://astral.sh/uv/install.sh | sh
uv sync # build .venv from uv.lock
make test # correctness + scientific-validation tests (the spine of trust)
make layer1 # run experiments E1–E6
make figures # regenerate figures from committed results
Layout
src/knowledge/ Layer 1 package (imported as `knowledge`)
configs/layer1/ one YAML per experiment (E1..E6)
figures/ plot_EX.py — read results.parquet only
tests/ test_correctness.py + test_scientific_validation.py (analytic checks)
paper/ blueprint.md, perspective paper, figure_manifest.md
results/ written artifacts (gitignored; hashes tracked in manifest.json)