society: multi-locus recombination frame — the vertical claim (E7/E8)

Enter the Lamarckian society with a robust theoretical frame. The single-
locus, fixed-p* model can only express recovery toward a ceiling; the
society's load-bearing claim is vertical -- capability that EXCEEDS any
component. Generalize knowledge to a distribution over genotypes (L
biallelic loci, K=2^L, additive fitness = # correct loci), reusing all the
K-mode machinery. The one new operator is recombination: free recombination
sends p -> product of per-locus marginals (linkage equilibrium).

E8 (star, kind: society) -- the vertical claim / Fisher-Muller: decorrelated
PARENTS (specialists, expert on their loci, agnostic elsewhere) are
recombined; sexual merge assembles a genotype fitter than any parent,
climbing to the optimum (12/12, a genotype no parent had) as parent count
grows and rho->0, while the best single parent (~8.7) and the mean-mixture
"model soup" (~11.6) plateau below. Reuses make_retention_matrix (locus
mastery replaces tail-item retention).

E7 (kind: genotype_lineage) -- the advantage of sex: a single population
adapts toward the optimum; the sexual lineage adapts faster than asexual
(clonal interference) by keeping loci in linkage equilibrium (LD->0 vs LD
spike). Honest scope: a speed advantage, not a permanent Muller's-ratchet
gap (subtle to force); E8 carries the headline.

Metaphor shift (per GG): the society is sexual reproduction with UNBOUNDED
parents, not teacher->pupil. Teacher->pupil caps at the ceiling; n-parent
recombination is combinatorial and generative, and unlike biology there is
no two-parent limit. Collapse = asexual degradation; the cure = sex. This
unifies E4 (merge != average) + E6 (irreversibility) under evolution-of-sex
theory and reaches ground Riis's single-locus n-grams cannot.

New: knowledge/{genotype,genotype_lineage,society}.py, configs/layer1/{E7,
E8}.yaml, figures/plot_{E7,E8}.py, READMEs, tests/test_genotype.py (+7).
experiment.py dispatch (kind in {genotype_lineage, society}); make layer1
wired. 112 tests green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Giorgio Gilestro 2026-07-05 10:51:41 +01:00
parent 871bc39ec6
commit 62c68d6c8c
22 changed files with 879 additions and 3 deletions

View file

@ -80,6 +80,8 @@ E4's whole purpose is to isolate the effect of teacher **decorrelation ρ**, so
**Strategic positioning vs Riis 2026 (arXiv:2604.08554, "Drift and selection in LLM text ecosystems").** Riis independently formalizes **collapse = WrightFisher drift** (his Thm 1) with n-gram agents: minority-mass martingale, rare-first extinction, single-token dropout ≈ αe^{α}, de Bruijn-polytope fixed points, plus descriptive-vs-normative *selection* (Thm 2). **Concede as prior art:** "collapse is literally WrightFisher", the martingale, rare-first loss, the WF/effective-population formalism — cite him; do **not** frame these as our contribution. **Crucial distinction that protects us:** his "mixed environment" *retains the lineage's own old synthetic tokens* — there is **no injection of fresh real data from a fixed `p*`**, so his headline is *pessimistic* (Thm 1c: extinction is independent of α — retention only changes speed). Our **grounding is immigration from a non-drifting external truth**, giving a stationary `H_eq>0` and a critical `g*≪1` that *prevents* collapse — the mechanism his closed loop lacks. **Our defensible novelty, ranked:** (1) **recombination + "merge, don't average" conservation law** (E4) — he has no model-merging operator; flagship; (2) **the learning-kernel / estimator-bias axis** — he *explicitly names it as future work*; we now build+measure it; (3) grounding threshold (solid anchor, but immigrationdrift balance is classic — not a flagship); (4) architecture-generality in real weights + MNIST; (5) **the Lamarckian society + the vertical/cumulative C3 claim — wholly ours, not yet run.** Reposition the paper from *"collapse is drift"* (now contested) to **a population-genetic *control theory* for sustaining open-ended knowledge**: drift is the diagnosed disease (cite Riis), our contribution is the engineered remedies and their integration.
**Finding (2026-07-05, E7/E8 — the multi-locus society frame; raises the ceiling).** To express the *vertical* claim (capability that *exceeds* any component), knowledge is generalized from a single-locus fixed-`p*` distribution to a distribution over **genotypes** (`L` biallelic loci, `K=2^L`; fitness = # correct loci; reuses all the K-mode machinery). The one new operator is **recombination** (`knowledge/genotype.py`): free recombination sends `p → ⊗ per-locus marginals` (linkage equilibrium). Two experiments, both analytic. **E8 (the star, `kind: society`) — the vertical claim / FisherMuller:** decorrelated *parents* (specialists, expert on their loci, agnostic elsewhere) are recombined; **sexual merge assembles a genotype fitter than any parent, climbing to the optimum (12/12) as parent count grows and `ρ→0`, while the best single parent (~8.7) and the mean-mixture "model soup" (~11.6) plateau below.** Clean, dramatic, 40 reps; reuses `make_retention_matrix` (locus mastery replaces tail-item retention). **E7 (`kind: genotype_lineage`) — the advantage of sex:** a single population adapting toward the optimum; the sexual lineage adapts *faster* (clonal interference slows the asexual one) by keeping loci in linkage equilibrium (LD→0 vs LD spike). Honest scope: a **speed** advantage, not a permanent Muller's-ratchet gap (the single-population ratchet is subtle to force; E8 carries the headline). **Metaphor shift (GG, 2026-07-05):** the society is framed as **sexual reproduction with unbounded parents**, *not* teacher→pupil — teacher→pupil caps at the ceiling (recovery), n-parent recombination is combinatorial and *generative* (exceeds any parent), and unlike biology there is no two-parent limit. Collapse = asexual degradation; the cure = sex. This unifies E4 (merge≠average) + E6 (irreversibility) under evolution-of-sex theory and stakes ground Riis's single-locus n-grams cannot reach. Scope is bounded: fixed combinatorial space (`L≤12`, "effectively open-ended relative to n"), additive fitness (NK/epistasis is an optional extension).
## Build order (blueprint §7) — respect the gate
1. Scaffold: repo layout (§5), container, pytest skeleton, config system, seeding utils. `make test` green.

View file

@ -16,7 +16,7 @@ test: ## correctness tests + scientific-validation tests (the spine
uv run pytest
layer1: ## run experiments E1-E6 + the learning-kernel bridge (analytic)
for e in E1 E2 E3 E4 E5 E6 kernel_sharpen kernel_smooth; do uv run python -m knowledge.experiment configs/layer1/$$e.yaml; done
for e in E1 E2 E3 E4 E5 E6 E7 E8 kernel_sharpen kernel_smooth; do uv run python -m knowledge.experiment configs/layer1/$$e.yaml; done
neural: ## run Layer 1.5 synthetic neural experiments (excludes the heavy MNIST tier)
for c in configs/neural/*.yaml; do case "$$c" in *mnist*) ;; \

29
configs/layer1/E7.yaml Normal file
View file

@ -0,0 +1,29 @@
experiment: E7
kind: genotype_lineage
seed: 20260705
n_replicates: 20
# (The advantage of sex — the dynamic mechanism behind E8): a single population adapts from all-wrong
# toward a multi-locus optimum under selection + drift + mutation. Beneficial alleles arise in
# different sub-lineages; recombination reassorts them into one genotype, while an asexual lineage
# suffers clonal interference (the alleles compete and cannot combine). Expect the SEXUAL lineage
# (recomb_rate=1) to climb toward the optimum faster than the ASEXUAL one (recomb_rate=0) — the
# classical advantage of sex, and the reason a lone model lineage cannot do what a recombining
# society can. Honest scope: a SPEED advantage, not a dramatic permanent gap (the single-population
# ratchet is subtle); E8 carries the headline. Falsifier: sexual adapts no faster than asexual.
genotype:
L: 12
n: 150 # population/resample size (drift strength)
mu: 0.02 # per-locus mutation (flip) rate
base: 1.3 # multiplicative selection: fitness weight = base^(#correct loci)
recomb_rate: 0.0 # overwritten per arm by the sweep
init: wrong # start all-wrong (load L); adapt upward
generations: 120
sweep:
- param: genotype.recomb_rate
values: [0.0, 1.0] # asexual vs sexual
output: {dir: results/E7}

29
configs/layer1/E8.yaml Normal file
View file

@ -0,0 +1,29 @@
experiment: E8
kind: society
seed: 20260705
n_replicates: 40
# (The vertical claim / Fisher-Muller — the society headline): can an offspring recombined from
# MANY decorrelated parents be fitter than ANY parent? Each parent is a specialist: confident-
# correct (hi) on the loci it has mastered, agnostic (~0.5) elsewhere; which loci each masters comes
# from the exact shared-switch construction, so parent count K_T and decorrelation rho are clean
# knobs. Deployed capability = fitness of the MODE genotype. Compare best single parent vs mean-
# mixture ("model soup", combine-but-don't-recombine) vs sexual recombination (assemble the best
# allele of each locus across all parents). Expect: sexual climbs to the optimum (=L, a genotype NO
# parent had) as K_T grows and rho->0, while best-parent and average plateau far below. Unlike
# biological sex there is no two-parent limit. Falsifier: sexual never exceeds the best parent, or
# averaging matches sexual.
society:
L: 12 # loci; the optimum (all-correct) has fitness 12 and no parent possesses it
q: 0.5 # fraction of loci each parent masters (marginal mastery)
hi: 0.9 # correct-allele prob on a mastered locus (confident expert)
lo: 0.45 # correct-allele prob on an unmastered locus (agnostic, slightly wrong)
sweep:
- param: K_T
values: [1, 2, 3, 5, 8, 12] # number of parents (unbounded; grows the recombinant reach)
- param: rho
values: [0.0, 0.5, 1.0] # decorrelated -> identical parents (the control)
output: {dir: results/E8}

62
figures/plot_E7.py Normal file
View file

@ -0,0 +1,62 @@
"""E7 figure — the advantage of sex: recombination adapts faster than clonal reproduction.
The dynamic mechanism behind E8. A single population adapts from all-wrong toward a multi-locus
optimum under selection + drift + mutation. Beneficial alleles arise in different sub-lineages;
sexual recombination reassorts them into one genotype, while an asexual lineage suffers clonal
interference. The sexual lineage climbs faster the classical advantage of sex (an honest *speed*
advantage; both eventually plateau near the optimum in this tractable regime).
Two panels: (A) mean-fitness adaptation curves, asexual vs sexual, over generations; (B) linkage
disequilibrium over generations asexual holds beneficial alleles in disequilibrium (scattered
across genotypes) while sexual drives it to ~0 (assembled), the mechanism of the speed gap.
Usage: python figures/plot_E7.py [results/E7]
"""
from __future__ import annotations
import sys
from pathlib import Path
import matplotlib.pyplot as plt
sys.path.insert(0, str(Path(__file__).parent))
from _figlib import load_bundle, mean_ci, savefig # noqa: E402
def main(results_dir: str = "results/E7") -> None:
df, cfg = load_bundle(results_dir)
L = cfg["genotype"]["L"]
arms = [(0.0, "#7f7f7f", "asexual (clonal)"), (1.0, "#d62728", "sexual (recombining)")]
fig, axes = plt.subplots(1, 2, figsize=(13, 5))
ax = axes[0]
for rate, c, lab in arms:
sub = df[df["recomb_rate"] == rate]
g, m, ci = mean_ci(sub, "generation", "mean_fitness")
ax.plot(g, m, "-", color=c, lw=1.8, label=lab)
ax.fill_between(g, m - ci, m + ci, color=c, alpha=0.2)
ax.axhline(L, ls=":", color="green", lw=1, label=f"optimum ($L$={L})")
ax.set(xlabel="generation", ylabel="mean fitness (# correct loci)",
title="Advantage of sex: recombination adapts faster\n(clonal interference slows the asexual lineage)")
ax.legend(frameon=False, fontsize=9)
ax = axes[1]
for rate, c, lab in arms:
sub = df[df["recomb_rate"] == rate]
g, m, ci = mean_ci(sub, "generation", "ld")
ax.plot(g, m, "-", color=c, lw=1.8, label=lab)
ax.fill_between(g, m - ci, m + ci, color=c, alpha=0.2)
ax.set(xlabel="generation", ylabel="mean linkage disequilibrium |D|",
title="Mechanism: asexual scatters beneficial alleles (LD>0);\nsexual assembles them (LD→0)")
ax.legend(frameon=False, fontsize=9)
fig.suptitle("E7 — the advantage of sex: recombination reassorts beneficial alleles that arose "
"in different lineages", y=1.02, fontsize=12)
fig.tight_layout()
savefig(fig, results_dir, "E7")
if __name__ == "__main__":
main(*sys.argv[1:])

68
figures/plot_E8.py Normal file
View file

@ -0,0 +1,68 @@
"""E8 figure — the vertical claim: n-parent recombination exceeds any parent (FisherMuller).
The society headline. Decorrelated *parents* are specialists (expert on some loci, agnostic on the
rest); an *offspring* recombined from all of them can be fitter than any parent capability that
*exceeds* every component, not just recovers a ceiling. Unlike biological sex there is no two-parent
limit, so capability climbs toward the optimum as the parent pool grows and decorrelates.
Two panels: (A) deployed capability (mode-genotype fitness) vs parent count at ρ=0 sexual
recombination reaches the optimum (a genotype no parent had) while the best single parent and the
mean-mixture "model soup" plateau below; (B) the decorrelation control sexual capability vs parent
count for ρ {0, 0.5, 1}: decorrelated parents (ρ=0) climb to the optimum, identical parents (ρ=1)
buy nothing. Reads only the committed bundle.
Usage: python figures/plot_E8.py [results/E8]
"""
from __future__ import annotations
import sys
from pathlib import Path
import matplotlib.pyplot as plt
import numpy as np
sys.path.insert(0, str(Path(__file__).parent))
from _figlib import load_bundle, mean_ci, savefig # noqa: E402
def main(results_dir: str = "results/E8") -> None:
df, cfg = load_bundle(results_dir)
L = cfg["society"]["L"]
rhos = sorted(df["rho"].unique())
fig, axes = plt.subplots(1, 2, figsize=(13, 5))
# Panel A: best-parent vs average vs sexual, at rho=0.
ax = axes[0]
d0 = df[df["rho"] == 0.0]
for col, c, lab in [("best_parent", "#7f7f7f", "best single parent"),
("average", "#1f77b4", "average (model soup)"),
("sexual", "#d62728", "sexual recombination")]:
k, m, ci = mean_ci(d0, "K_T", col)
ax.errorbar(k, m, yerr=ci, fmt="-o", color=c, capsize=3, label=lab)
ax.axhline(L, ls=":", color="green", lw=1, label=f"optimum ($L$={L})")
ax.set(xlabel="number of parents $K_T$", ylabel="deployed capability (mode fitness)",
title="Recombination exceeds any parent (ρ=0):\nsexual reaches the optimum; soup & best-parent plateau")
ax.legend(frameon=False, fontsize=9)
# Panel B: sexual capability vs K_T for each rho (decorrelation control).
ax = axes[1]
colors = plt.cm.viridis(np.linspace(0, 0.8, len(rhos)))
for rho, c in zip(rhos, colors):
sub = df[df["rho"] == rho]
k, m, ci = mean_ci(sub, "K_T", "sexual")
ax.errorbar(k, m, yerr=ci, fmt="-o", color=c, capsize=3, label=fr"ρ={rho:g}")
ax.axhline(L, ls=":", color="green", lw=1, label=f"optimum ($L$={L})")
ax.set(xlabel="number of parents $K_T$", ylabel="sexual-recombination capability",
title="Decorrelation is the fuel:\nρ=0 climbs to the optimum; ρ=1 (clones) buy nothing")
ax.legend(frameon=False, fontsize=9)
fig.suptitle("E8 — the vertical claim: an offspring recombined from many decorrelated parents "
"is fitter than any parent (FisherMuller; no two-parent limit)", y=1.02, fontsize=12)
fig.tight_layout()
savefig(fig, results_dir, "E8")
if __name__ == "__main__":
main(*sys.argv[1:])

BIN
results/E7/E7.pdf Normal file

Binary file not shown.

BIN
results/E7/E7.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 157 KiB

31
results/E7/README.md Normal file
View file

@ -0,0 +1,31 @@
# E7 — the advantage of sex: recombination adapts faster than clonal reproduction
**Claim tested.** The dynamic mechanism behind E8: *why* can a recombining society reach capability a
lone lineage cannot? Because recombination reassorts beneficial variants that arise in different
sub-lineages, while an asexual (clonal) lineage suffers **clonal interference** — the variants compete
and cannot combine.
**Setup.** A single population (distribution over `2^L` genotypes, `L=12`) adapts from **all-wrong**
toward the multi-locus optimum under the composed step: selection (fitness-proportional) + drift
(resample `n=150`) + mutation (per-locus flips, `μ=0.02`) + recombination. Two arms — **asexual**
(`recomb_rate=0`) vs **sexual** (`recomb_rate=1`). 20 replicates.
### Symbols
- **asexual/clonal** = offspring are whole-genotype copies (Layer-1's regime) · **sexual** = loci reassorted across the population each generation.
- **fitness** = number of correct loci (optimum = `L`) · **linkage disequilibrium |D|** = how far the loci are from statistical independence (correct alleles scattered across different genotypes).
### The two panels
1. **Advantage of sex.** Mean fitness over generations: the **sexual lineage (red) climbs faster**
than the asexual one (grey) through the adaptation phase (gen ~1035). *Honest scope:* both plateau
near the optimum by gen ~40 in this tractable regime — this is a **speed** advantage, not a
permanent gap (the single-population Muller's ratchet is subtle to force; E8 carries the headline).
2. **Mechanism.** Linkage disequilibrium over generations: the asexual lineage spikes to `|D|≈0.04`
during adaptation (beneficial alleles held apart, scattered across genotypes), while the sexual
lineage stays at `|D|≈0` — it *assembles* them. The LD gap is exactly why sexual adapts faster.
### Takeaway
Recombination's advantage is real and classical: it combines good ideas that arose independently,
which clonal reproduction cannot. This is the population-level reason a single evolving model lineage
degrades or stalls where a recombining **society** climbs — and it grounds the E8 vertical result in
the evolution-of-sex theory. **Falsifier (not triggered):** if the sexual lineage adapted no faster
than the asexual one (and kept the same LD), recombination would do no work.

14
results/E7/manifest.json Normal file
View file

@ -0,0 +1,14 @@
{
"experiment": "E7",
"master_seed": 20260705,
"git_commit": "871bc39ec6628f82aed75d007fdf675880eebc97",
"python": "3.14.5",
"libraries": {
"numpy": "2.5.0",
"scipy": "1.18.0",
"pandas": "3.0.3",
"pyarrow": "24.0.0"
},
"rows": 4840,
"results_sha256": "4836cd7045ad5419554e7edc65e4e12b23877a551f38969d65e09a4f77715faa"
}

View file

@ -0,0 +1,23 @@
experiment: E7
seed: 20260705
n_replicates: 20
source_config:
experiment: E7
kind: genotype_lineage
seed: 20260705
n_replicates: 20
genotype:
L: 12
n: 150
mu: 0.02
base: 1.3
recomb_rate: 0.0
init: wrong
generations: 120
sweep:
- param: genotype.recomb_rate
values:
- 0.0
- 1.0
output:
dir: results/E7

BIN
results/E8/E8.pdf Normal file

Binary file not shown.

BIN
results/E8/E8.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 167 KiB

35
results/E8/README.md Normal file
View file

@ -0,0 +1,35 @@
# E8 — the vertical claim: n-parent recombination exceeds any parent (FisherMuller)
**Claim tested.** The society's headline, and the thing no fixed-`p*` model could express: can an
offspring recombined from **many decorrelated parents** be *fitter than any parent* — capability that
**exceeds** every component, not merely recovers a ceiling?
**Setup.** A capability is a **genotype** of `L=12` biallelic loci; fitness = number of correct
loci; the optimum (all-correct, fitness 12) is a genotype **no parent possesses**. Each *parent* is a
specialist: confident-correct (`hi=0.9`) on the loci it has mastered, agnostic (`lo=0.45`) on the
rest. Which loci each masters comes from the exact shared-switch construction, so **parent count
`K_T`** and **decorrelation `ρ`** are clean, independently-swept knobs. Deployed capability = fitness
of the **mode** (most-probable) genotype — what you would ship. 40 replicates.
### Symbols
- **parent** = a specialist model; **offspring** = the recombined model; **`K_T`** = number of parents (unbounded — biological sex is stuck at 2; model merging is not).
- **`ρ`** = correlation of which loci parents master (0 = complementary, 1 = identical clones).
- **best parent** = fittest single specialist · **average** = mean-mixture "model soup" (combine, don't recombine) · **sexual** = union-preserving recombination (assemble the best allele of each locus).
### The two panels
1. **Recombination exceeds any parent (ρ=0).** Deployed capability vs `K_T`: **sexual (red) climbs
to the optimum (12)** as parents accumulate — a genotype none of them had — while the **best single
parent (grey) plateaus at ~8.7** and the **model soup (blue) reaches ~11.6** but is beaten by
sexual at every `K_T` (and badly at small `K_T`: at 2 parents, sexual 9.0 vs soup 7.2 vs best 6.9).
2. **Decorrelation is the fuel.** Sexual capability vs `K_T` for `ρ ∈ {0, 0.5, 1}`: decorrelated
parents (`ρ=0`) climb to the optimum; identical clones (`ρ=1`) buy nothing (flat at ~6). The
benefit is *combinatorial reach across complementary parents*, not merely "more models".
### Takeaway
This is the FisherMuller effect for AI: recombination assembles beneficial variants that live in
*different* parents into an offspring fitter than any of them. It is the rigorous, un-preempted core
of the Lamarckian society — collapse is asexual degradation; the cure is **sex, with no parent
limit.** It reframes model merging from "averaging weights" to "meiotic reassortment", and it is the
mechanism by which general capability can *climb while each specialty is re-earned and exceeded.* The
dynamic version (why a lone lineage cannot do this) is E7. **Falsifier (not triggered):** if sexual
never exceeded the best parent, or averaging matched it, the vertical claim would fail.

14
results/E8/manifest.json Normal file
View file

@ -0,0 +1,14 @@
{
"experiment": "E8",
"master_seed": 20260705,
"git_commit": "871bc39ec6628f82aed75d007fdf675880eebc97",
"python": "3.14.5",
"libraries": {
"numpy": "2.5.0",
"scipy": "1.18.0",
"pandas": "3.0.3",
"pyarrow": "24.0.0"
},
"rows": 720,
"results_sha256": "fec5c4fcaa86999291dd77b015cd650ad8c28cb62f67eab54ca5be04bc18a711"
}

View file

@ -0,0 +1,29 @@
experiment: E8
seed: 20260705
n_replicates: 40
source_config:
experiment: E8
kind: society
seed: 20260705
n_replicates: 40
society:
L: 12
q: 0.5
hi: 0.9
lo: 0.45
sweep:
- param: K_T
values:
- 1
- 2
- 3
- 5
- 8
- 12
- param: rho
values:
- 0.0
- 0.5
- 1.0
output:
dir: results/E8

View file

@ -127,6 +127,46 @@ def run_experiment(cfg: dict) -> pd.DataFrame:
return out
_GENOTYPE_KEYS = ("genotype", "generations")
def run_genotype_experiment(cfg: dict) -> pd.DataFrame:
"""Run a genotype lineage across a sweep x replicates (E7, advantage of sex).
Mirrors ``run_experiment`` (paired replicate seeds) but assembles the base from the
``genotype``/``generations`` blocks and calls ``run_genotype_lineage``. Sweeps use the same
dotted-path ``_apply_param`` (e.g. ``genotype.recomb_rate`` for asexual vs sexual).
"""
from .genotype_lineage import run_genotype_lineage
base = {k: copy.deepcopy(cfg[k]) for k in _GENOTYPE_KEYS if k in cfg}
sweeps = cfg.get("sweep", [])
if isinstance(sweeps, dict):
sweeps = [sweeps]
params = [s["param"] for s in sweeps]
value_lists = [list(s["values"]) for s in sweeps]
combos = [({}, base)] if not sweeps else []
for values in itertools.product(*value_lists):
lin = copy.deepcopy(base)
label: dict = {}
for param, val in zip(params, values):
label.update(_apply_param(lin, param, val))
combos.append((label, lin))
seeds = spawn_seeds(int(cfg["seed"]), int(cfg["n_replicates"]))
frames: list[pd.DataFrame] = []
for label, lin in combos:
for rep, ss in enumerate(seeds):
df = run_genotype_lineage(lin, int(ss.generate_state(1)[0]))
for col, val in label.items():
df[col] = val
df["replicate"] = rep
frames.append(df)
out = pd.concat(frames, ignore_index=True)
out.insert(0, "experiment", cfg["experiment"])
return out
def run_coverage(cfg: dict) -> pd.DataFrame:
"""E4 runner: multi-teacher recombination coverage (blueprint 2.5-E4 / 2.7.1).
@ -278,7 +318,16 @@ def run_and_save(config_path: str | Path) -> Path:
config_path = Path(config_path)
cfg = yaml.safe_load(config_path.read_text())
out_dir = Path(cfg.get("output", {}).get("dir", f"results/{cfg['experiment']}"))
df = run_coverage(cfg) if cfg.get("kind") == "coverage" else run_experiment(cfg)
kind = cfg.get("kind", "lineage")
if kind == "coverage":
df = run_coverage(cfg)
elif kind == "genotype_lineage":
df = run_genotype_experiment(cfg) # E7: advantage of sex
elif kind == "society":
from .society import run_society # E8: multi-parent recombination
df = run_society(cfg)
else:
df = run_experiment(cfg)
save_artifacts(cfg, df, out_dir)
return out_dir

202
src/knowledge/genotype.py Normal file
View file

@ -0,0 +1,202 @@
"""Multi-locus genotype space + recombination — the theoretical frame for the society.
The single-locus, fixed-`p*` model can only express *recovery toward a ceiling*. The society's
vertical claim capability that *exceeds* any component needs combinatorial structure. A
**genotype** is `L` biallelic loci (allele 1 = "correct", 0 = "wrong"); a model's knowledge is a
distribution over the `2^L` genotypes (a K-vector with `K = 2^L`, so all of Layer 1's K-mode
machinery drift, grounding, selection, metrics applies unchanged). Fitness is additive (the
number of correct loci); the optimum is the all-correct genotype.
The one genuinely new operator is **recombination**. Free recombination replaces the joint genotype
distribution by the product of its per-locus marginals (linkage equilibrium / Robbins proportions)
the population-level image of meiotic reassortment. This is what halts Muller's ratchet (it
reconstitutes low-load genotypes from complementary high-load parents) and drives the FisherMuller
effect (it assembles beneficial alleles that arose in different lineages into a genotype fitter than
any parent). `rate=0` is asexual/clonal (Layer 1's regime); `rate=1` is free recombination.
"""
from __future__ import annotations
import numpy as np
from .metrics import heterozygosity
def genotype_bits(L: int) -> np.ndarray:
"""Return the ``(2^L, L)`` matrix of genotype bit-vectors (row g = g in binary, MSB first).
Args:
L (int): Number of biallelic loci.
Returns:
np.ndarray: ``(2^L, L)`` int8 array; ``bits[g, l]`` is allele of locus ``l`` in genotype ``g``.
"""
g = np.arange(1 << L, dtype=np.int64)
shifts = np.arange(L - 1, -1, -1, dtype=np.int64)
return ((g[:, None] >> shifts[None, :]) & 1).astype(np.int8)
def additive_fitness(L: int) -> np.ndarray:
"""Additive fitness vector: ``f(g)`` = number of correct (allele-1) loci in genotype ``g``.
Args:
L (int): Number of loci.
Returns:
np.ndarray: Length-``2^L`` fitness vector in ``{0, 1, ..., L}``; optimum = all-ones genotype.
"""
return genotype_bits(L).sum(axis=1).astype(float)
def locus_marginals(p: np.ndarray, L: int) -> np.ndarray:
"""Per-locus frequency of the correct (allele-1) variant under distribution ``p``.
Args:
p (np.ndarray): Genotype distribution (length ``2^L``, sums to 1).
L (int): Number of loci.
Returns:
np.ndarray: Length-``L`` array; entry ``l`` is ``P(locus l is correct)``.
"""
return genotype_bits(L).T.astype(float) @ np.asarray(p, dtype=float) # (L, 2^L) @ (2^L,)
def linkage_equilibrium(marginals: np.ndarray) -> np.ndarray:
"""Build the product (linkage-equilibrium) genotype distribution from per-locus marginals.
``p_LE(g) = _l [q_l if g_l==1 else (1q_l)]`` the joint under free recombination, where the
loci are statistically independent (Robbins proportions).
Args:
marginals (np.ndarray): Length-``L`` correct-allele frequencies ``q_l``.
Returns:
np.ndarray: Length-``2^L`` product distribution (sums to 1).
"""
q = np.asarray(marginals, dtype=float)
L = q.size
bits = genotype_bits(L) # (2^L, L)
per = np.where(bits == 1, q[None, :], 1.0 - q[None, :]) # (2^L, L)
p = per.prod(axis=1)
s = p.sum()
return p / s if s > 0 else p
def recombine(p: np.ndarray, L: int, rate: float) -> np.ndarray:
"""Apply recombination at ``rate`` to a genotype distribution.
``rate=0`` returns ``p`` unchanged (asexual / clonal); ``rate=1`` returns the full
linkage-equilibrium product of ``p``'s marginals (free recombination); intermediate rates
interpolate ``(1rate)·p + rate·p_LE``.
Args:
p (np.ndarray): Genotype distribution (length ``2^L``).
L (int): Number of loci.
rate (float): Recombination rate in ``[0, 1]``.
Returns:
np.ndarray: The recombined distribution (sums to 1).
"""
p = np.asarray(p, dtype=float)
if rate <= 0.0:
return p
p_le = linkage_equilibrium(locus_marginals(p, L))
out = p_le if rate >= 1.0 else (1.0 - rate) * p + rate * p_le
s = out.sum()
return out / s if s > 0 else out
def recombine_teachers(teachers: list[np.ndarray], L: int, rate: float) -> np.ndarray:
"""Merge ``K_T`` teacher genotype distributions (the multi-teacher / FisherMuller operator).
``rate=0`` = clonal mean-mixture (``mean(teachers)`` keeps genotypes intact, cannot create a
genotype no teacher had). ``rate=1`` = sexual merge: the linkage-equilibrium product of the
*pooled* per-locus marginals, which assembles the best allele of each locus across teachers into
genotypes none of them held the FisherMuller effect and "merge, don't average" at the genotype
level.
Args:
teachers (list[np.ndarray]): ``K_T`` genotype distributions (each length ``2^L``).
L (int): Number of loci.
rate (float): Recombination rate in ``[0, 1]``.
Returns:
np.ndarray: The merged genotype distribution (sums to 1).
"""
stack = np.asarray(teachers, dtype=float)
mean_mix = stack.mean(axis=0)
if rate <= 0.0:
return mean_mix
pooled_marginals = locus_marginals(mean_mix, L) # pooled per-locus correct-allele freq
p_le = linkage_equilibrium(pooled_marginals)
out = p_le if rate >= 1.0 else (1.0 - rate) * mean_mix + rate * p_le
s = out.sum()
return out / s if s > 0 else out
def mutate(p: np.ndarray, L: int, mu: float) -> np.ndarray:
"""Apply per-locus symmetric mutation at rate ``mu`` to a genotype distribution.
Each locus independently flips with probability ``mu``. At the distribution level this convolves
``p`` with the per-locus flip kernel; implemented as ``L`` independent locus mixings. ``mu=0`` is
a no-op. Provides the mutational pressure that Muller's ratchet grinds against.
Args:
p (np.ndarray): Genotype distribution (length ``2^L``).
L (int): Number of loci.
mu (float): Per-locus flip probability in ``[0, 0.5]``.
Returns:
np.ndarray: The mutated distribution (sums to 1).
"""
p = np.asarray(p, dtype=float)
if mu <= 0.0:
return p
q = p.reshape([2] * L) if L > 0 else p
for axis in range(L):
flipped = np.flip(q, axis=axis)
q = (1.0 - mu) * q + mu * flipped
out = q.reshape(-1)
s = out.sum()
return out / s if s > 0 else out
def locus_metrics(p: np.ndarray, L: int, fitness: np.ndarray, alive_eps: float = 1e-9) -> dict:
"""Genotype-aware metrics for one generation (the multi-locus analogue of the K-mode row).
Args:
p (np.ndarray): Genotype distribution (length ``2^L``).
L (int): Number of loci.
fitness (np.ndarray): Length-``2^L`` fitness (typically :func:`additive_fitness`).
alive_eps (float): A genotype/allele is "present" if its probability exceeds this.
Returns:
dict: ``mean_fitness``, ``best_fitness`` (max fitness of any present genotype),
``opt_freq`` (mass on the all-correct optimum), ``min_load`` (fewest wrong loci among present
genotypes = the Muller-ratchet state), ``mean_locus_correct`` (mean per-locus correct-allele
freq), ``locus_H`` (mean per-locus heterozygosity), and ``ld`` (mean pairwise linkage
disequilibrium |D|).
"""
p = np.asarray(p, dtype=float)
present = p > alive_eps
q = locus_marginals(p, L) # per-locus correct freq
bits = genotype_bits(L)
row = {
"mean_fitness": float(fitness @ p),
"best_fitness": float(fitness[present].max()) if present.any() else 0.0,
"opt_freq": float(p[-1]), # genotype 2^L-1 = all-ones optimum
"min_load": int(L - fitness[present].max()) if present.any() else L,
"mean_locus_correct": float(q.mean()),
"locus_H": float(np.mean([heterozygosity(np.array([1 - qi, qi])) for qi in q])),
}
# Mean pairwise linkage disequilibrium |D_ij| = |P(1_i,1_j) - q_i q_j| over locus pairs.
if L >= 2:
ds = []
for i in range(L):
for j in range(i + 1, L):
p_ij = float(p[(bits[:, i] == 1) & (bits[:, j] == 1)].sum())
ds.append(abs(p_ij - q[i] * q[j]))
row["ld"] = float(np.mean(ds))
else:
row["ld"] = 0.0
return row

View file

@ -0,0 +1,82 @@
"""Single-population genotype evolution — the advantage of sex (E7).
A population (distribution over the ``2^L`` genotypes) adapts toward a multi-locus optimum under
the composed generational step: **selection** (fitness-proportional, favouring correct alleles) +
**drift** (finite resample of ``n``) + **mutation** (per-locus flips) + **recombination** (asexual
``rate=0`` vs sexual ``rate>0``). Recombination reassorts beneficial alleles that arise in different
sub-lineages into one genotype; without it (asexual) those alleles suffer *clonal interference* and
adaptation is slower. So a sexual lineage climbs toward the optimum faster than an asexual one the
classical advantage of sex, and the dynamic counterpart of E8's one-shot multi-parent assembly.
Reuses ``step.apply_selection`` (fitness = number of correct loci) and the ``genotype`` operators;
emits the same tidy per-generation DataFrame contract as ``run_lineage`` (with genotype-aware
columns from ``genotype.locus_metrics``).
"""
from __future__ import annotations
from typing import Any, Mapping
import numpy as np
import pandas as pd
from .genotype import additive_fitness, locus_metrics, mutate, recombine
from .step import apply_selection
def run_genotype_lineage(cfg: Mapping[str, Any], seed: int) -> pd.DataFrame:
"""Run one genotype lineage and return per-generation genotype metrics.
Args:
cfg (Mapping): Config with a ``genotype`` block (``L``; drift ``n``; mutation ``mu``;
selection ``base`` for multiplicative fitness ``base^#correct``; recombination
``recomb_rate``; ``init`` in {``wrong``, ``uniform``, ``optimum``}) and
``generations``.
seed (int): Replicate seed; the run is a pure function of (cfg, seed).
Returns:
pd.DataFrame: One row per generation 0..T with ``generation`` plus the
``genotype.locus_metrics`` columns (``mean_fitness``, ``best_fitness``, ``opt_freq``,
``min_load``, ``mean_locus_correct``, ``locus_H``, ``ld``).
"""
g = cfg["genotype"]
L = int(g["L"])
n = int(g["n"])
mu = float(g.get("mu", 0.0))
base = float(g.get("base", 1.0))
rate = float(g.get("recomb_rate", 0.0))
generations = int(cfg.get("generations", 100))
K = 1 << L
report_fitness = additive_fitness(L) # # correct loci (0..L), for metrics
sel_fitness = base ** report_fitness # multiplicative selection weight
init = g.get("init", "wrong")
p = np.zeros(K)
if init == "wrong":
p[0] = 1.0 # all-wrong genotype (load L); adapt upward
elif init == "optimum":
p[-1] = 1.0 # all-correct (for degradation studies)
elif init == "uniform":
p[:] = 1.0 / K
else:
raise ValueError(f"unknown genotype init {init!r} (expected wrong|optimum|uniform)")
rng = np.random.default_rng(seed)
rows: list[dict] = []
def record(t: int) -> None:
row = {"generation": t}
row.update(locus_metrics(p, L, report_fitness, alive_eps=1.0 / n))
rows.append(row)
record(0)
for t in range(1, generations + 1):
p = apply_selection(p, sel_fitness, "greedy", 0.0) # fitness-proportional selection
counts = rng.multinomial(n, p) # drift
p = counts / counts.sum()
p = mutate(p, L, mu) # per-locus mutation
p = recombine(p, L, rate) # asexual (0) vs sexual (>0)
record(t)
return pd.DataFrame(rows)

95
src/knowledge/society.py Normal file
View file

@ -0,0 +1,95 @@
"""Multi-parent recombination — the FisherMuller vertical claim (E8).
The society's headline: **an offspring recombined from many decorrelated parents can be fitter
than any parent** capability that *exceeds* every component, not just recovers a ceiling. This
is the FisherMuller effect, and unlike biological sex it has **no two-parent limit**: an offspring
here can have arbitrarily many parents (``recombine_teachers`` pools per-locus marginals over all of
them).
Each parent is a *specialist*: confident-correct on the loci it has mastered, agnostic (~0.5) on the
rest the realistic picture of an expert. Which loci each parent masters comes from the exact
shared-switch construction (``teachers.make_retention_matrix``), so parent count ``K_T`` and
decorrelation ``rho`` are clean, independently-swept knobs (mastery of a *locus* replaces retention
of a *tail item*). Three deployable capabilities are compared, all as the fitness of the *mode*
(most-probable) genotype what you would actually ship:
* **best_parent** the single fittest specialist (no combination).
* **average** the mean-mixture "model soup" (combine, but don't recombine loci).
* **sexual** the union-preserving recombination (assemble the best allele of each locus).
Sexual climbs to the optimum as parents accumulate and decorrelate; the other two plateau.
"""
from __future__ import annotations
import itertools
import numpy as np
import pandas as pd
from .genotype import additive_fitness, linkage_equilibrium, recombine_teachers
from .seeding import spawn_seeds
from .teachers import make_retention_matrix
def make_specialist(mastery: np.ndarray, hi: float, lo: float) -> np.ndarray:
"""Build a specialist's genotype distribution from its per-locus mastery mask.
Args:
mastery (np.ndarray): Length-``L`` bool; True where this parent is expert.
hi (float): Correct-allele probability on mastered loci (confident).
lo (float): Correct-allele probability on unmastered loci (agnostic, ~0.5).
Returns:
np.ndarray: Length-``2^L`` genotype distribution (product / linkage-equilibrium form).
"""
q = np.where(np.asarray(mastery, dtype=bool), hi, lo)
return linkage_equilibrium(q)
def _mode_fitness(p: np.ndarray, fitness: np.ndarray) -> float:
"""Fitness of the most-probable (deployed consensus) genotype."""
return float(fitness[int(np.argmax(p))])
def run_society(cfg: dict) -> pd.DataFrame:
"""Sweep parent count x decorrelation; compare best-parent vs average vs sexual capability.
Args:
cfg (dict): Parsed experiment config with a ``society`` block (``L``, ``q`` mastery
fraction, ``hi``, ``lo``), a ``sweep`` (``K_T`` x ``rho``), ``seed``, ``n_replicates``.
Returns:
pd.DataFrame: One row per (K_T, rho, replicate) with ``best_parent``, ``average``,
``sexual`` (mode-genotype fitness, 0..L) and ``L``.
"""
soc = cfg["society"]
L = int(soc["L"])
q = float(soc.get("q", 0.5))
hi, lo = float(soc.get("hi", 0.9)), float(soc.get("lo", 0.45))
fitness = additive_fitness(L)
sweeps = cfg["sweep"]
if isinstance(sweeps, dict):
sweeps = [sweeps]
params = [s["param"] for s in sweeps]
value_lists = [list(s["values"]) for s in sweeps]
seeds = spawn_seeds(int(cfg["seed"]), int(cfg["n_replicates"]))
rows: list[dict] = []
for combo in itertools.product(*value_lists):
d = dict(zip(params, combo))
K_T, rho = int(d["K_T"]), float(d["rho"])
for rep, ss in enumerate(seeds):
child = int(ss.generate_state(1)[0])
rng = np.random.default_rng(child)
mastery = make_retention_matrix(L, K_T, rho, q, rng).astype(bool) # (K_T, L)
parents = [make_specialist(mastery[k], hi, lo) for k in range(K_T)]
rows.append({
"experiment": cfg["experiment"], "K_T": K_T, "rho": rho, "q": q, "L": L,
"replicate": rep,
"best_parent": max(_mode_fitness(p, fitness) for p in parents),
"average": _mode_fitness(recombine_teachers(parents, L, 0.0), fitness),
"sexual": _mode_fitness(recombine_teachers(parents, L, 1.0), fitness),
})
return pd.DataFrame(rows)

View file

@ -264,8 +264,30 @@ C3 vertical claim deferred.*
Lamarckian society + vertical/cumulative C3 claim (not yet run). Reposition: from "collapse is drift"
to a population-genetic CONTROL THEORY for sustaining open-ended knowledge.
## Remaining (all optional)
**2026-07-05 — multi-locus society frame (E7/E8): raised the ceiling to enter the society.**
- Prompted by "enter the society with a robust theoretical frame." The single-locus fixed-`p*` model
can't express "exceeding" a ceiling. Generalized knowledge to a distribution over **genotypes**
(`knowledge/genotype.py`: `L` biallelic loci, `K=2^L`, additive fitness, recombination = product of
per-locus marginals). Reuses all K-mode machinery + `make_retention_matrix` (locus mastery).
- **E8 (star, `kind: society`, `knowledge/society.py`) — the vertical claim:** decorrelated parents
recombined; **sexual merge reaches the optimum (12/12, a genotype no parent had)** as parent count
grows / `ρ→0`, while best-parent (~8.7) and mean-mixture soup (~11.6) plateau. `configs/layer1/E8.yaml`,
`plot_E8.py`, README. The FisherMuller effect for AI.
- **E7 (`kind: genotype_lineage`, `knowledge/genotype_lineage.py`) — advantage of sex:** sexual lineage
adapts faster than asexual (LD→0 vs LD spike). Honest: a speed advantage, not a permanent ratchet gap.
- **Metaphor shift (GG):** sexual reproduction with **unbounded parents**, not teacher→pupil (which caps
at the ceiling). Collapse = asexual degradation; cure = sex, no parent limit. Unifies E4+E6 under
evolution-of-sex theory; beyond Riis's single-locus n-grams. `tests/test_genotype.py` (+7).
Experiment dispatch (`kind` in {genotype_lineage, society}) + `make layer1` wired.
## Remaining (all optional / next)
- [ ] **The dynamic society:** an evolving *population of parents* (specialists) that ground +
recombine + select over generations — capability climbing while specialties are re-earned (the full
C3, dynamic). E7/E8 give the static + single-population pieces; the multi-parent *lineage* is next.
- [ ] **NK/epistasis landscape** (sign epistasis can make recombination harmful — the honest limit of
"sex always helps"); **multi-allelic loci**. Deepens the frame.
- [ ] **Learning-kernel refinement:** truth-like smoothing prior (`prior="truth"`) + measurement floor
for a quantitative RNN match; **multi-locus / linkage** modes (class×style) as the rigorous home for
recombination. Both enrich predictive power and separate us further from Riis's single-locus n-grams.

90
tests/test_genotype.py Normal file
View file

@ -0,0 +1,90 @@
"""Multi-locus / recombination tests (pure NumPy) — the spine of the society frame.
Cover the genotype algebra (bits, additive fitness, linkage equilibrium), the recombination
operators (identity at rate 0, product-of-marginals at rate 1), and the two headline behaviours:
E8 sexual recombination of decorrelated parents *exceeds* the best parent; E7 a sexual lineage
adapts faster than an asexual one.
"""
from __future__ import annotations
import numpy as np
from knowledge.genotype import (
additive_fitness, genotype_bits, linkage_equilibrium, locus_marginals, mutate,
recombine, recombine_teachers,
)
from knowledge.genotype_lineage import run_genotype_lineage
from knowledge.society import make_specialist, run_society
from knowledge.teachers import make_retention_matrix
def test_genotype_bits_and_additive_fitness():
bits = genotype_bits(3)
assert bits.shape == (8, 3)
assert np.array_equal(bits[0], [0, 0, 0]) and np.array_equal(bits[7], [1, 1, 1])
f = additive_fitness(3)
assert f[0] == 0 and f[7] == 3 and f[1] == 1 # MSB-first: g=1 -> [0,0,1]
def test_linkage_equilibrium_is_product_and_normalised():
q = np.array([0.9, 0.5, 0.2])
p = linkage_equilibrium(q)
assert np.isclose(p.sum(), 1.0)
# P(all-correct) = ∏ q; P(all-wrong) = ∏ (1-q)
assert np.isclose(p[-1], np.prod(q)) and np.isclose(p[0], np.prod(1 - q))
assert np.allclose(locus_marginals(p, 3), q) # marginals round-trip
def test_recombine_identity_and_free():
rng = np.random.default_rng(0)
p = rng.dirichlet(np.ones(16)) # L=4
assert np.array_equal(recombine(p, 4, 0.0), p) # rate 0 = asexual (unchanged)
free = recombine(p, 4, 1.0) # rate 1 = product of marginals
assert np.allclose(free, linkage_equilibrium(locus_marginals(p, 4)))
assert np.isclose(free.sum(), 1.0)
def test_mutate_normalises_and_spreads():
L = 4
p = np.zeros(1 << L); p[0] = 1.0 # point mass on all-wrong
out = mutate(p, L, 0.1)
assert np.isclose(out.sum(), 1.0)
assert out[0] < 1.0 and (out > 0).sum() > 1 # mass leaks to neighbours
def test_recombine_teachers_assembles_optimum_only_sexually():
# Two complementary parents: one masters the low loci, the other the high loci. Sexual merge
# assembles the all-correct optimum (a genotype NEITHER parent has as its mode); clonal cannot.
L = 8
lo = np.array([1, 1, 1, 1, 0, 0, 0, 0], dtype=bool)
parents = [make_specialist(lo, 0.9, 0.45), make_specialist(~lo, 0.9, 0.45)]
f = additive_fitness(L)
best_parent = max(f[int(np.argmax(p))] for p in parents)
sexual = f[int(np.argmax(recombine_teachers(parents, L, 1.0)))]
clonal = f[int(np.argmax(recombine_teachers(parents, L, 0.0)))]
assert sexual == L # sexual assembles the optimum
assert sexual > best_parent # ... which exceeds either parent
assert clonal <= best_parent + 0 # clonal (soup) does not assemble it
def test_e8_sexual_exceeds_best_parent_and_soup():
cfg = {"experiment": "e8t", "seed": 1, "n_replicates": 8,
"society": {"L": 10, "q": 0.5, "hi": 0.9, "lo": 0.45},
"sweep": [{"param": "K_T", "values": [8]}, {"param": "rho", "values": [0.0]}]}
df = run_society(cfg)
m = df.mean(numeric_only=True)
assert m["sexual"] > m["best_parent"] + 1.5 # clearly exceeds the best parent
assert m["sexual"] >= m["average"] # and is at least as good as the soup
def test_e7_sexual_adapts_at_least_as_fast():
base = {"genotype": {"L": 10, "n": 150, "mu": 0.02, "base": 1.3, "init": "wrong"},
"generations": 25}
asex = run_genotype_lineage({**base, "genotype": {**base["genotype"], "recomb_rate": 0.0}}, 0)
sex = run_genotype_lineage({**base, "genotype": {**base["genotype"], "recomb_rate": 1.0}}, 0)
mid = 12
a = asex[asex["generation"] == mid]["mean_fitness"].iloc[0]
s = sex[sex["generation"] == mid]["mean_fitness"].iloc[0]
assert s >= a - 1e-9 # sexual adapts at least as fast mid-run
assert sex["ld"].max() < asex["ld"].max() # ... by keeping loci in linkage equilibrium