Giorgio Gilestro
6f8cef1ac5
Removed from main (all preserved on the dev branch): the arXiv build and
its sources, design documents (blueprint, results summary, review responses,
essay drafts), tasks/ and CLAUDE.md, the cover letter and reference tooling,
two unused manuscript figures, and every experiment that feeds no figure or
number in the paper: the collapse null, the sexual-vs-asexual lineage, the
NK speciation variant, the 0.5B single-seed LLM prototypes, the compose and
society experiments with their calibration and pilot runs, and their
configs, runners, tests, figure scripts and PBS jobs. Their result bundles
are moved to results/_archive/ (ignored) so the parquets stay on disk.
Also: plot_llm_speciation reads the s{seed}/ layout; the mating-breadth
plot writes under its bundle name; Makefile targets reduced to the kept
experiments; REPRODUCING.md and README point to dev for the rest.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y64o8FKP7rCuXzC48pxpMm
|
||
|---|---|---|
| .. | ||
| llm_seeds.pdf | ||
| llm_seeds.png | ||
| manifest.json | ||
| README.md | ||
| resolved_config.yaml | ||
Multi-seed LLM recombination (0.5B) — the claims with error bars
PNAS work-order Phase 3: removes the "one seed" objection on the three LLM recombination claims.
Protocol: test sets fixed (seed 1000+i per family), training seed varied (specialists cache
per-seed as spec_<family>[_hard]_s<seed>), so across-seed variance is training variance only.
Figure: llm_seeds.png (this dir) aggregates all three experiments, 95% CI over seeds.
(A) Fisher–Muller, easy benchmark, 5 seeds (llm_merge_seeds)
| model | overall | worst-family |
|---|---|---|
| merge_ties | 0.647 ± 0.027 | 0.282 ± 0.020 |
| merge_soup | 0.632 ± 0.042 | 0.278 ± 0.028 |
| best specialist (strings) | 0.592 ± 0.009 | 0.078 ± 0.011 |
| base | 0.277 | 0.150 |
Both merges beat every specialist overall (ties: non-overlapping CIs; soup: marginal at 0.5B, as in the single-seed run — decisive at 7B) and the worst-family signature is unambiguous: merges ≈0.28 vs ≤0.16 for any parent — only recombined models are competent everywhere.
(B) Union vs fusion, hard benchmark, 3 seeds (llm_moe_hard_seeds/)
Routing (union) beats fusion in every seed (3/3 paired, both metrics; e.g. overall per seed: routing 0.258/0.300/0.262 vs soup 0.212/0.238/0.071). The paired per-seed values also show why: seed 3's soup failed catastrophically (0.071 overall, 0.000 worst-family) while routing was unaffected (0.262/0.225). Seed-level sd: soup 0.090 vs routing 0.023 — suggestive of a genuine variance difference, but with 3 seeds this is an observation, not an estimate; the robust statement is the 3/3 paired ordering plus the observed catastrophic-failure mode that routing is structurally immune to. (Learned router still = oracle: lexically distinct families, known rider.)
(C) Directed offspring selection, hard, 3 seeds (llm_directed_hard_seeds/)
directed_overall beats the a-priori soup in every seed (3/3 paired; 0.225/0.242/0.196 vs 0.212/0.238/0.071 — including rescuing soup's catastrophic seed); directed_balanced worst-family 0.158 vs soup 0.088. Directed selection both beats and stabilises the blend (same 3-seed caveat as panel B); per-input routing remains above any single global blend, as before.
Read together: all three recombination claims hold under seed replication with consistent paired ordering (route > directed-select > soup in every seed on headroom tasks), and the per-seed values surface a failure mode — occasional catastrophic soup merges — that the union/selection operators avoided in every observed case. Variance estimates await more seeds. Base: Qwen2.5-0.5B-Instruct; statistical (per-seed) reproducibility per blueprint §4.