diff --git a/CLAUDE.md b/CLAUDE.md index 0f8c4a2..9fb31c3 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -82,6 +82,8 @@ E4's whole purpose is to isolate the effect of teacher **decorrelation ρ**, so **Finding (2026-07-05, E7/E8 — the multi-locus society frame; raises the ceiling).** To express the *vertical* claim (capability that *exceeds* any component), knowledge is generalized from a single-locus fixed-`p*` distribution to a distribution over **genotypes** (`L` biallelic loci, `K=2^L`; fitness = # correct loci; reuses all the K-mode machinery). The one new operator is **recombination** (`knowledge/genotype.py`): free recombination sends `p → ⊗ per-locus marginals` (linkage equilibrium). Two experiments, both analytic. **E8 (the star, `kind: society`) — the vertical claim / Fisher–Muller:** decorrelated *parents* (specialists, expert on their loci, agnostic elsewhere) are recombined; **sexual merge assembles a genotype fitter than any parent, climbing to the optimum (12/12) as parent count grows and `ρ→0`, while the best single parent (~8.7) and the mean-mixture "model soup" (~11.6) plateau below.** Clean, dramatic, 40 reps; reuses `make_retention_matrix` (locus mastery replaces tail-item retention). **E7 (`kind: genotype_lineage`) — the advantage of sex:** a single population adapting toward the optimum; the sexual lineage adapts *faster* (clonal interference slows the asexual one) by keeping loci in linkage equilibrium (LD→0 vs LD spike). Honest scope: a **speed** advantage, not a permanent Muller's-ratchet gap (the single-population ratchet is subtle to force; E8 carries the headline). **Metaphor shift (GG, 2026-07-05):** the society is framed as **sexual reproduction with unbounded parents**, *not* teacher→pupil — teacher→pupil caps at the ceiling (recovery), n-parent recombination is combinatorial and *generative* (exceeds any parent), and unlike biology there is no two-parent limit. Collapse = asexual degradation; the cure = sex. This unifies E4 (merge≠average) + E6 (irreversibility) under evolution-of-sex theory and stakes ground Riis's single-locus n-grams cannot reach. Scope is bounded: fixed combinatorial space (`L≤12`, "effectively open-ended relative to n"), additive fitness (NK/epistasis is an optional extension). +**Finding (2026-07-05, E9/E10 — the sexual-transmission model made rigorous: when sex helps, and directed sex).** Deepening the sexual metaphor (GG excited; wanted it robust before the full society). Added a **Kauffman NK landscape** (`genotype.nk_fitness`, tunable ruggedness `K`), finite **crossover** (`genotype.crossover`, n-parent, per-gap recombination rate), and **hill-climb** (parents = local optima = "trained models"). **E9 (`kind: recomb_landscape`) — landscape robustness / "why sex?":** E8's dramatic transgression used an *additive* landscape; on rugged (epistatic) landscapes, blindly recombining local optima causes **outbreeding depression** — mean offspring fall *below* the parents, worse with ruggedness AND recombination rate (`K=8`, free recomb: ≈ −0.23), and the **optimal recombination rate shrinks as ruggedness grows**. Design rule: *merge freely when skills are complementary/additive; sparingly + with selection when entangled.* **E10 (`kind: directed_sex`) — directed sex beats biological sex (the AI superpower):** biology is stuck with 2 random-mating parents and no offspring preview; an AI can **choose complementary mates + evaluate many recombinant offspring + keep the fittest + use unbounded parents** (iterated recombine-then-select). Result: random ("biological") sex craters with ruggedness (0.66→0.51), while **directed sex tracks/exceeds the best parent at every ruggedness** — converting the outbreeding-depression catastrophe into a win. This is the practical, distinctly-AI payoff and has no biological analog. `configs/layer1/{E9,E10}.yaml`, `plot_{E9,E10}.py`, READMEs, +5 tests (117 green). Complete sexual-transmission picture: **dramatic super-parent offspring when skills are complementary (E8); outbreeding-depression risk when entangled (E9); directed sex resolves the risk (E10).** + ## Build order (blueprint §7) — respect the gate 1. Scaffold: repo layout (§5), container, pytest skeleton, config system, seeding utils. `make test` green. diff --git a/Makefile b/Makefile index 72a3e35..41d3ab5 100644 --- a/Makefile +++ b/Makefile @@ -16,7 +16,7 @@ test: ## correctness tests + scientific-validation tests (the spine uv run pytest layer1: ## run experiments E1-E6 + the learning-kernel bridge (analytic) - for e in E1 E2 E3 E4 E5 E6 E7 E8 kernel_sharpen kernel_smooth; do uv run python -m knowledge.experiment configs/layer1/$$e.yaml; done + for e in E1 E2 E3 E4 E5 E6 E7 E8 E9 E10 kernel_sharpen kernel_smooth; do uv run python -m knowledge.experiment configs/layer1/$$e.yaml; done neural: ## run Layer 1.5 synthetic neural experiments (excludes the heavy MNIST tier) for c in configs/neural/*.yaml; do case "$$c" in *mnist*) ;; \ diff --git a/configs/layer1/E10.yaml b/configs/layer1/E10.yaml new file mode 100644 index 0000000..9b349d5 --- /dev/null +++ b/configs/layer1/E10.yaml @@ -0,0 +1,27 @@ +experiment: E10 +kind: directed_sex +seed: 20260705 +n_replicates: 24 + +# (Directed sex beats biological sex — the distinctly-AI superpower): on rugged landscapes, blind +# ("biological") sex suffers outbreeding depression (offspring worse than parents). But an AI can do +# what biology cannot: choose maximally-complementary mates, evaluate MANY recombinant offspring, and +# keep only the fittest, over several rounds (directed sex = iterated recombine-then-select, with no +# two-parent limit). Compare, across ruggedness K: best single parent vs RANDOM sex (blind) vs +# DIRECTED sex. Expect: random sex craters with ruggedness; directed sex avoids the catastrophe and +# matches or exceeds the best parent even when skills are entangled. Falsifier: directed sex does no +# better than random sex, or never recovers the best-parent level on rugged landscapes. + +society: + L: 12 + n_parents: 6 + pop: 200 # offspring evaluated per round (mate choice + offspring selection) + keep: 8 # fittest offspring retained each round + rounds: 5 # rounds of recombine-then-select + rate: 0.2 # directed-sex recombination rate (random-sex uses free rate 0.5) + +sweep: + - param: K + values: [2, 4, 6, 8, 10] # landscape ruggedness (all with parent diversity) + +output: {dir: results/E10} diff --git a/configs/layer1/E9.yaml b/configs/layer1/E9.yaml new file mode 100644 index 0000000..d3c17fe --- /dev/null +++ b/configs/layer1/E9.yaml @@ -0,0 +1,27 @@ +experiment: E9 +kind: recomb_landscape +seed: 20260705 +n_replicates: 24 + +# (Landscape robustness / the "why sex?" question — the credibility centerpiece): E8 showed sex wins +# on an ADDITIVE landscape, where recombination trivially helps. Does it survive EPISTASIS? Parents +# are local optima ("trained models") of a Kauffman NK landscape whose ruggedness K (epistatic +# interactions per locus) is swept with the recombination rate. Expect: on smooth/mildly-rugged +# landscapes recombination helps; on rugged ones FREE recombination (rate~0.5) breaks co-adapted +# blocks and offspring fall BELOW the parents (outbreeding depression); and the OPTIMAL recombination +# rate shrinks as ruggedness grows. Design rule: merge freely when skills are complementary/additive; +# merge sparingly when entangled. Falsifier: recombination rate has no effect, or free recombination +# never underperforms the parents on rugged landscapes. + +society: + L: 12 + n_parents: 6 # trained specialists = local optima of the landscape + pop: 200 # recombinant offspring sampled per (K, rate, replicate) + +sweep: + - param: K + values: [0, 2, 4, 6, 8] # landscape ruggedness (epistasis): 0 = additive, high = rugged + - param: rate + values: [0.0, 0.05, 0.1, 0.2, 0.35, 0.5] # clonal -> free recombination + +output: {dir: results/E9} diff --git a/figures/plot_E10.py b/figures/plot_E10.py new file mode 100644 index 0000000..770496d --- /dev/null +++ b/figures/plot_E10.py @@ -0,0 +1,65 @@ +"""E10 figure — directed sex beats biological sex (the distinctly-AI superpower). + +On rugged (epistatic) landscapes, blind "biological" sex — random mates, no offspring selection — +suffers outbreeding depression: offspring are worse than the parents. But an AI can do what biology +cannot: choose complementary mates, evaluate *many* recombinant offspring, and keep only the fittest, +over several rounds, with no two-parent limit. This **directed sex** avoids the catastrophe and +matches or exceeds the best parent even when skills are entangled. + +Two panels: (A) deployed capability vs landscape ruggedness — best single parent, random (blind) sex, +directed sex, and the global optimum; (B) each strategy's edge over the best parent, making the +random-sex collapse and the directed-sex rescue explicit. Reads only the committed bundle. + +Usage: python figures/plot_E10.py [results/E10] +""" + +from __future__ import annotations + +import sys +from pathlib import Path + +import matplotlib.pyplot as plt + +sys.path.insert(0, str(Path(__file__).parent)) +from _figlib import load_bundle, mean_ci, savefig # noqa: E402 + + +def main(results_dir: str = "results/E10") -> None: + df, _ = load_bundle(results_dir) + + fig, axes = plt.subplots(1, 2, figsize=(13, 5)) + + # Panel A: the three strategies + global optimum vs ruggedness. + ax = axes[0] + for col, c, lab in [("global_opt", "green", "global optimum"), + ("directed_sex", "#d62728", "directed sex (AI: choose + select)"), + ("best_parent", "#7f7f7f", "best single parent"), + ("random_sex", "#1f77b4", "random sex (blind, biology)")]: + k, m, ci = mean_ci(df, "K", col) + ls = ":" if col == "global_opt" else "-o" + ax.plot(k, m, ls, color=c, label=lab) if col == "global_opt" else \ + ax.errorbar(k, m, yerr=ci, fmt=ls, color=c, capsize=3, label=lab) + ax.set(xlabel="landscape ruggedness $K$ (epistasis)", ylabel="deployed capability (fitness)", + title="Random sex craters with ruggedness;\ndirected sex tracks/exceeds the best parent") + ax.legend(frameon=False, fontsize=8) + + # Panel B: edge over best parent (random collapse vs directed rescue). + ax = axes[1] + bp = df.groupby("K")["best_parent"].mean() + for col, c, lab in [("directed_sex", "#d62728", "directed sex"), + ("random_sex", "#1f77b4", "random sex")]: + s = df.groupby("K")[col].mean() - bp + ax.plot(s.index, s.values, "-o", color=c, label=lab) + ax.axhline(0, ls=":", color="gray", lw=1, label="best parent") + ax.set(xlabel="landscape ruggedness $K$", ylabel="capability − best parent", + title="Directed sex stays ≥ parents; blind sex\nfalls far below (outbreeding depression)") + ax.legend(frameon=False, fontsize=9) + + fig.suptitle("E10 — directed sex beats biological sex: mate choice + offspring selection + " + "unbounded parents rescue recombination where blind sex fails", y=1.02, fontsize=11) + fig.tight_layout() + savefig(fig, results_dir, "E10") + + +if __name__ == "__main__": + main(*sys.argv[1:]) diff --git a/figures/plot_E9.py b/figures/plot_E9.py new file mode 100644 index 0000000..c1d2df4 --- /dev/null +++ b/figures/plot_E9.py @@ -0,0 +1,66 @@ +"""E9 figure — landscape robustness: when recombination helps, and the outbreeding-depression risk. + +The credibility test for the sexual metaphor. E8 used an additive landscape where recombination +trivially helps; here parents are local optima ("trained models") of a Kauffman NK landscape whose +ruggedness (epistasis) is tunable. Blindly recombining entangled models breaks co-adapted allele +blocks and offspring fall *below* the parents — outbreeding depression — worse the more rugged the +landscape and the higher the recombination rate. With selection (best offspring), a nonzero optimal +recombination rate re-emerges. Design rule: merge freely when skills are complementary; merge +sparingly (and always select) when they are entangled. + +Two panels: (A) the risk — mean offspring fitness minus best-parent vs recombination rate, one curve +per ruggedness K (all ≤0, steeper as K grows); (B) with offspring selection — best-of-brood fitness +vs rate per K, showing an intermediate optimum on rugged landscapes. Reads only the bundle. + +Usage: python figures/plot_E9.py [results/E9] +""" + +from __future__ import annotations + +import sys +from pathlib import Path + +import matplotlib.pyplot as plt +import numpy as np + +sys.path.insert(0, str(Path(__file__).parent)) +from _figlib import load_bundle, savefig # noqa: E402 + + +def main(results_dir: str = "results/E9") -> None: + df, _ = load_bundle(results_dir) + Ks = sorted(df["K"].unique()) + rates = sorted(df["rate"].unique()) + colors = plt.cm.viridis(np.linspace(0, 0.85, len(Ks))) + bp = df.groupby("K")["best_parent"].mean() + + fig, axes = plt.subplots(1, 2, figsize=(13, 5)) + + # Panel A: the risk — mean offspring minus best parent vs rate, per K. + ax = axes[0] + for K, c in zip(Ks, colors): + s = df[df["K"] == K].groupby("rate")["mean_offspring"].mean() - bp[K] + ax.plot(s.index, s.values, "-o", color=c, ms=4, label=f"K={K}") + ax.axhline(0, ls=":", color="gray", lw=1) + ax.set(xlabel="recombination rate", ylabel="mean offspring − best parent", + title="The risk: outbreeding depression\n(worse with ruggedness K and recombination rate)") + ax.legend(frameon=False, fontsize=8, title="ruggedness") + + # Panel B: with selection — best offspring vs rate, per K (intermediate optimum on rugged). + ax = axes[1] + for K, c in zip(Ks, colors): + s = df[df["K"] == K].groupby("rate")["best_offspring"].mean() + ax.plot(s.index, s.values, "-o", color=c, ms=4, label=f"K={K}") + ax.axhline(bp[K], ls=":", color=c, lw=0.8, alpha=0.6) + ax.set(xlabel="recombination rate", ylabel="best-of-brood fitness (with selection)", + title="With offspring selection, an optimal\nrecombination rate re-emerges (dotted = parents)") + ax.legend(frameon=False, fontsize=8, title="ruggedness") + + fig.suptitle("E9 — landscape robustness: recombination helps when skills are complementary, but " + "blindly merging entangled models causes outbreeding depression", y=1.02, fontsize=11) + fig.tight_layout() + savefig(fig, results_dir, "E9") + + +if __name__ == "__main__": + main(*sys.argv[1:]) diff --git a/results/E10/E10.pdf b/results/E10/E10.pdf new file mode 100644 index 0000000..f6e88b0 Binary files /dev/null and b/results/E10/E10.pdf differ diff --git a/results/E10/E10.png b/results/E10/E10.png new file mode 100644 index 0000000..e6cf614 Binary files /dev/null and b/results/E10/E10.png differ diff --git a/results/E10/README.md b/results/E10/README.md new file mode 100644 index 0000000..928836b --- /dev/null +++ b/results/E10/README.md @@ -0,0 +1,34 @@ +# E10 — directed sex beats biological sex (the distinctly-AI superpower) + +**Claim tested.** E9 showed that on rugged (epistatic) landscapes, blind recombination causes +outbreeding depression — offspring worse than parents. Biology is largely stuck with this: two +random-mating parents, no way to preview offspring. But an **AI is not** — it can choose mates, +evaluate many recombinant offspring, and keep only the fittest, over rounds, with no two-parent +limit. Does this **directed sex** rescue recombination where blind sex fails? + +**Setup.** Parents are local optima of a Kauffman NK landscape (`L=12`), swept over ruggedness `K`. +Three strategies compared, all as deployed capability (fitness ∈ [0,1]): **best single parent**; +**random sex** (blind mating, free recombination, no offspring selection — biology's default); +**directed sex** (iterated recombine-then-select-offspring, `rate=0.2`, 5 rounds — the AI move). +24 replicate landscapes. + +### Symbols +- **random sex** — blind: random parents, free recombination, take the offspring as they come. +- **directed sex** — choose complementary mates + generate many offspring + keep the fittest + repeat; unbounded parents. +- **global optimum** — the landscape's best genotype (the ceiling). + +### The two panels +1. **Capability vs ruggedness.** As `K` grows, **random sex (blue) craters** (0.66 → 0.51 — deep + outbreeding depression), while **directed sex (red) tracks the best parent and the global optimum**, + staying near the ceiling at every ruggedness. +2. **Edge over the best parent.** Directed sex stays **≥ 0** (at or above the parents) across all `K`; + random sex falls to **≈ −0.2** (far below). Directed sex converts a catastrophe into a win. + +### Takeaway +The move biology cannot make — **choose your mates, evaluate your offspring before you commit, and +recombine as many parents as you like** — is exactly what makes AI sexual reproduction robust. Blind +merging of entangled models is dangerous; *directed* merging (generate many candidate merges, keep +the best) is safe and can exceed every parent. This is the practical payoff of the sexual-transmission +model and the distinctly-AI superpower with no biological analog. **Falsifier (not triggered):** if +directed sex did no better than random sex, or never recovered the best-parent level on rugged +landscapes, the "AI beats biology" claim would fail. diff --git a/results/E10/manifest.json b/results/E10/manifest.json new file mode 100644 index 0000000..01b0b4a --- /dev/null +++ b/results/E10/manifest.json @@ -0,0 +1,14 @@ +{ + "experiment": "E10", + "master_seed": 20260705, + "git_commit": "62c68d6c8c4090e62cf9ee17ac7b7d1eff7a6955", + "python": "3.14.5", + "libraries": { + "numpy": "2.5.0", + "scipy": "1.18.0", + "pandas": "3.0.3", + "pyarrow": "24.0.0" + }, + "rows": 120, + "results_sha256": "b2d53d0a3949be0b67e3b8dffec56121d54e20c593e6302717a73a8a132f17cb" +} \ No newline at end of file diff --git a/results/E10/resolved_config.yaml b/results/E10/resolved_config.yaml new file mode 100644 index 0000000..86df9ef --- /dev/null +++ b/results/E10/resolved_config.yaml @@ -0,0 +1,25 @@ +experiment: E10 +seed: 20260705 +n_replicates: 24 +source_config: + experiment: E10 + kind: directed_sex + seed: 20260705 + n_replicates: 24 + society: + L: 12 + n_parents: 6 + pop: 200 + keep: 8 + rounds: 5 + rate: 0.2 + sweep: + - param: K + values: + - 2 + - 4 + - 6 + - 8 + - 10 + output: + dir: results/E10 diff --git a/results/E9/E9.pdf b/results/E9/E9.pdf new file mode 100644 index 0000000..5d080db Binary files /dev/null and b/results/E9/E9.pdf differ diff --git a/results/E9/E9.png b/results/E9/E9.png new file mode 100644 index 0000000..70f5589 Binary files /dev/null and b/results/E9/E9.png differ diff --git a/results/E9/README.md b/results/E9/README.md new file mode 100644 index 0000000..6343386 --- /dev/null +++ b/results/E9/README.md @@ -0,0 +1,34 @@ +# E9 — landscape robustness: when recombination helps, and the outbreeding-depression risk + +**Claim tested.** E8 showed sexual recombination assembling super-parent offspring — but on an +*additive* landscape, where recombination trivially helps. The honest, credibility-critical question +(the classic "why sex?" problem): does the benefit survive **epistasis**, or does merging entangled +models break them? + +**Setup.** Parents are **local optima** ("trained models") of a **Kauffman NK landscape** (`L=12`), +whose ruggedness `K` (epistatic interactions per locus) is swept together with the **recombination +rate**. `K=0` is additive/smooth; larger `K` is rugged (co-adapted allele blocks, many local optima). +Fitness ∈ [0,1]. 24 replicate landscapes; 200 offspring per point. + +### Symbols +- **NK landscape** — tunable-ruggedness fitness landscape; `K` = epistasis (0 = additive, high = rugged). +- **recombination rate** — per-gap crossover probability (0 = clonal / copy a parent; 0.5 = free recombination, loci independent). +- **outbreeding depression** — offspring *less* fit than parents because recombination broke co-adapted allele blocks. + +### The two panels +1. **The risk.** Mean offspring fitness *minus* best parent, vs recombination rate, one curve per + ruggedness `K`. On the additive landscape (`K=0`) it's flat at 0; as `K` grows the curves plunge + **negative**, and deeper the higher the recombination rate — **outbreeding depression, worse the + more entangled the skills and the more you mix** (`K=8`, free recombination: ≈ −0.23). +2. **With selection, an optimal rate re-emerges.** Best-of-brood fitness (offspring selection) vs + rate per `K`, with parents dotted. On rugged landscapes a **nonzero intermediate recombination + rate** is best — enough mixing to find new combinations, not so much that it shatters good blocks. + +### Takeaway +Recombination is not a free lunch. **Merge freely when skills are complementary/additive; merge +sparingly — and always *select* offspring — when they are entangled.** This is the celebrated +population-genetics result (recombination load / outbreeding depression) reproduced for AI model +merging, and it turns the sexual metaphor from a lucky demo into a law with a design rule. The rescue +— directed sex with offspring selection — is E10. **Falsifier (not triggered):** if recombination +rate had no effect, or free recombination never underperformed the parents on rugged landscapes, the +epistasis caveat would be moot. diff --git a/results/E9/manifest.json b/results/E9/manifest.json new file mode 100644 index 0000000..831884c --- /dev/null +++ b/results/E9/manifest.json @@ -0,0 +1,14 @@ +{ + "experiment": "E9", + "master_seed": 20260705, + "git_commit": "62c68d6c8c4090e62cf9ee17ac7b7d1eff7a6955", + "python": "3.14.5", + "libraries": { + "numpy": "2.5.0", + "scipy": "1.18.0", + "pandas": "3.0.3", + "pyarrow": "24.0.0" + }, + "rows": 720, + "results_sha256": "73a737b333ee972ff1f878e18bf91b0aac58cb595c00c7b7fec3c18865971c4f" +} \ No newline at end of file diff --git a/results/E9/resolved_config.yaml b/results/E9/resolved_config.yaml new file mode 100644 index 0000000..e09122d --- /dev/null +++ b/results/E9/resolved_config.yaml @@ -0,0 +1,30 @@ +experiment: E9 +seed: 20260705 +n_replicates: 24 +source_config: + experiment: E9 + kind: recomb_landscape + seed: 20260705 + n_replicates: 24 + society: + L: 12 + n_parents: 6 + pop: 200 + sweep: + - param: K + values: + - 0 + - 2 + - 4 + - 6 + - 8 + - param: rate + values: + - 0.0 + - 0.05 + - 0.1 + - 0.2 + - 0.35 + - 0.5 + output: + dir: results/E9 diff --git a/src/knowledge/experiment.py b/src/knowledge/experiment.py index 41d2098..f4b71e4 100644 --- a/src/knowledge/experiment.py +++ b/src/knowledge/experiment.py @@ -326,6 +326,12 @@ def run_and_save(config_path: str | Path) -> Path: elif kind == "society": from .society import run_society # E8: multi-parent recombination df = run_society(cfg) + elif kind == "recomb_landscape": + from .society import run_recomb_landscape # E9: landscape robustness / epistasis + df = run_recomb_landscape(cfg) + elif kind == "directed_sex": + from .society import run_directed_sex # E10: directed sex beats biology + df = run_directed_sex(cfg) else: df = run_experiment(cfg) save_artifacts(cfg, df, out_dir) diff --git a/src/knowledge/genotype.py b/src/knowledge/genotype.py index 49fefee..91a1471 100644 --- a/src/knowledge/genotype.py +++ b/src/knowledge/genotype.py @@ -48,6 +48,90 @@ def additive_fitness(L: int) -> np.ndarray: return genotype_bits(L).sum(axis=1).astype(float) +def nk_fitness(L: int, K: int, seed: int) -> np.ndarray: + """Kauffman NK fitness landscape over the ``2^L`` genotypes (tunable ruggedness). + + Each locus contributes a random value in [0,1] that depends on its own allele plus the ``K`` + following loci (adjacent ring neighbourhood); total fitness = mean of the ``L`` contributions. + ``K=0`` is additive/smooth (a single peak, recombination unambiguously helps); larger ``K`` is + epistatic/rugged (many local optima, co-adapted allele blocks that recombination can break — + the regime where sex can hurt). + + Args: + L (int): Number of loci. + K (int): Epistatic interactions per locus (``0..L-1``); ruggedness knob. + seed (int): Landscape seed (the landscape is a deterministic function of it). + + Returns: + np.ndarray: Length-``2^L`` fitness vector in [0,1]. + """ + rng = np.random.default_rng(seed) + bits = genotype_bits(L).astype(np.int64) + F = np.zeros(1 << L) + for i in range(L): + idx = [(i + j) % L for j in range(K + 1)] # locus i + its K ring-neighbours + table = rng.random(1 << (K + 1)) # random contribution per pattern + pat = np.zeros(1 << L, dtype=np.int64) + for b, locus in enumerate(idx): + pat |= bits[:, locus] << (K - b) + F += table[pat] + return F / L + + +def hill_climb(fitness: np.ndarray, L: int, start: int) -> int: + """Greedy single-locus-flip hill-climb to a local optimum (a "trained specialist" parent). + + Args: + fitness (np.ndarray): Length-``2^L`` fitness vector. + L (int): Number of loci. + start (int): Starting genotype index. + + Returns: + int: A local-optimum genotype index (no single-locus flip improves fitness). + """ + g = int(start) + while True: + neighbours = [g ^ (1 << l) for l in range(L)] + best = max(neighbours, key=lambda x: fitness[x]) + if fitness[best] <= fitness[g]: + return g + g = best + + +def crossover(parents_bits: np.ndarray, rate: float, rng: np.random.Generator) -> np.ndarray: + """One recombinant offspring from ``n`` parents by per-gap switching (finite, stochastic). + + Walks the loci left to right inheriting from a current parent; at each gap the current parent is + re-drawn uniformly among the ``n`` parents with probability ``rate``. ``rate=0`` clones one + parent (asexual); ``rate=0.5`` gives near-independent per-locus inheritance (free recombination); + small ``rate`` preserves linked blocks of co-adapted alleles (the knob that matters on rugged + landscapes). Generalises 2-parent crossover to arbitrarily many parents (no two-parent limit). + + Args: + parents_bits (np.ndarray): ``(n, L)`` bit-matrix of the parent genotypes. + rate (float): Per-gap recombination probability in ``[0, 0.5]``. + rng (np.random.Generator): Explicit RNG. + + Returns: + np.ndarray: Length-``L`` offspring bit-vector. + """ + n, L = parents_bits.shape + cur = int(rng.integers(n)) + child = np.empty(L, dtype=parents_bits.dtype) + for l in range(L): + if l > 0 and rng.random() < rate: + cur = int(rng.integers(n)) + child[l] = parents_bits[cur, l] + return child + + +def bits_to_index(bits: np.ndarray) -> int: + """Convert a genotype bit-vector (MSB first) to its integer index.""" + L = bits.size + weights = 1 << np.arange(L - 1, -1, -1) + return int(np.asarray(bits) @ weights) + + def locus_marginals(p: np.ndarray, L: int) -> np.ndarray: """Per-locus frequency of the correct (allele-1) variant under distribution ``p``. diff --git a/src/knowledge/society.py b/src/knowledge/society.py index aa90176..bc9d999 100644 --- a/src/knowledge/society.py +++ b/src/knowledge/society.py @@ -27,7 +27,10 @@ import itertools import numpy as np import pandas as pd -from .genotype import additive_fitness, linkage_equilibrium, recombine_teachers +from .genotype import ( + additive_fitness, bits_to_index, crossover, genotype_bits, hill_climb, linkage_equilibrium, + nk_fitness, recombine_teachers, +) from .seeding import spawn_seeds from .teachers import make_retention_matrix @@ -52,6 +55,120 @@ def _mode_fitness(p: np.ndarray, fitness: np.ndarray) -> float: return float(fitness[int(np.argmax(p))]) +def _local_optima(fitness: np.ndarray, L: int, n_parents: int, + rng: np.random.Generator) -> np.ndarray: + """``n_parents`` local-optimum genotypes (trained specialists) as an ``(n_parents, L)`` matrix.""" + bits = genotype_bits(L) + opts = [hill_climb(fitness, L, int(rng.integers(1 << L))) for _ in range(n_parents)] + return np.stack([bits[g] for g in opts]) + + +def _random_sex(fitness, pbits, rate, n_off, rng) -> float: + """Mean fitness of ``n_off`` blind recombinant offspring (biology: random mating, no selection).""" + return float(np.mean([fitness[bits_to_index(crossover(pbits, rate, rng))] + for _ in range(n_off)])) + + +def _directed_sex(fitness, pbits, rate, pop, keep, rounds, rng) -> float: + """Best fitness reachable by *directed* sex: iterated recombine-then-select-offspring. + + The AI superpower biology lacks — evaluate many recombinants and keep only the fittest, over + several rounds (mate choice + offspring selection + unbounded parents). Returns the best fitness + found. + """ + pool = pbits.copy() + best = max(float(fitness[bits_to_index(b)]) for b in pool) + for _ in range(rounds): + offs = np.stack([crossover(pool, rate, rng) for _ in range(pop)]) + fits = np.array([fitness[bits_to_index(o)] for o in offs]) + pool = offs[np.argsort(fits)[-keep:]] + best = max(best, float(fits.max())) + return best + + +def _sweep_grid(cfg): + """Return (params, value_lists, seeds) for the config's sweep.""" + sweeps = cfg["sweep"] + if isinstance(sweeps, dict): + sweeps = [sweeps] + params = [s["param"] for s in sweeps] + value_lists = [list(s["values"]) for s in sweeps] + seeds = spawn_seeds(int(cfg["seed"]), int(cfg["n_replicates"])) + return params, value_lists, seeds + + +def run_recomb_landscape(cfg: dict) -> pd.DataFrame: + """E9 — landscape robustness: when does recombination help, and at what rate? + + Parents are local optima ("trained models") of a Kauffman NK landscape whose ruggedness ``K`` + (epistasis) is swept together with the recombination rate. On smooth/mildly-rugged landscapes + recombination helps; on rugged ones free recombination breaks co-adapted blocks and offspring + fall *below* the parents (outbreeding depression), with the optimal rate shrinking as ruggedness + grows. Sweep ``K`` x ``rate``. + + Returns: + pd.DataFrame: rows with ``K``, ``rate``, ``best_parent``, ``mean_offspring``, + ``best_offspring``, ``global_opt`` (all NK fitness in [0,1]). + """ + soc = cfg["society"] + L, nP, pop = int(soc["L"]), int(soc["n_parents"]), int(soc.get("pop", 200)) + params, value_lists, seeds = _sweep_grid(cfg) + rows: list[dict] = [] + for combo in itertools.product(*value_lists): + d = dict(zip(params, combo)) + K, rate = int(d["K"]), float(d["rate"]) + for rep, ss in enumerate(seeds): + seed = int(ss.generate_state(1)[0]) + fitness = nk_fitness(L, K, seed) + rng = np.random.default_rng(seed) + pbits = _local_optima(fitness, L, nP, rng) + offs = np.array([fitness[bits_to_index(crossover(pbits, rate, rng))] + for _ in range(pop)]) + best_parent = max(float(fitness[bits_to_index(b)]) for b in pbits) + rows.append({ + "experiment": cfg["experiment"], "K": K, "rate": rate, "replicate": rep, + "best_parent": best_parent, "mean_offspring": float(offs.mean()), + "best_offspring": float(offs.max()), "global_opt": float(fitness.max())}) + return pd.DataFrame(rows) + + +def run_directed_sex(cfg: dict) -> pd.DataFrame: + """E10 — directed sex beats biological sex: mate choice + offspring selection rescue ruggedness. + + Across landscape ruggedness ``K``, compare the deployed capability of: the best single parent; + **random sex** (blind mating, no selection — biology's default, which suffers outbreeding + depression on rugged landscapes); and **directed sex** (iterated recombine-then-select, the + AI-only move). Directed sex avoids the random-sex catastrophe and matches or exceeds the best + parent even when skills are entangled. + + Returns: + pd.DataFrame: rows with ``K``, ``best_parent``, ``random_sex``, ``directed_sex``, + ``global_opt``. + """ + soc = cfg["society"] + L, nP = int(soc["L"]), int(soc["n_parents"]) + rate = float(soc.get("rate", 0.2)) + n_off = int(soc.get("pop", 200)) + keep, rounds = int(soc.get("keep", 8)), int(soc.get("rounds", 5)) + params, value_lists, seeds = _sweep_grid(cfg) + rows: list[dict] = [] + for combo in itertools.product(*value_lists): + d = dict(zip(params, combo)) + K = int(d["K"]) + for rep, ss in enumerate(seeds): + seed = int(ss.generate_state(1)[0]) + fitness = nk_fitness(L, K, seed) + rng = np.random.default_rng(seed) + pbits = _local_optima(fitness, L, nP, rng) + rows.append({ + "experiment": cfg["experiment"], "K": K, "replicate": rep, + "best_parent": max(float(fitness[bits_to_index(b)]) for b in pbits), + "random_sex": _random_sex(fitness, pbits, 0.5, n_off, rng), + "directed_sex": _directed_sex(fitness, pbits, rate, n_off, keep, rounds, rng), + "global_opt": float(fitness.max())}) + return pd.DataFrame(rows) + + def run_society(cfg: dict) -> pd.DataFrame: """Sweep parent count x decorrelation; compare best-parent vs average vs sexual capability. diff --git a/tasks/todo.md b/tasks/todo.md index 75c07af..8f10567 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -281,6 +281,21 @@ C3 vertical claim deferred.* evolution-of-sex theory; beyond Riis's single-locus n-grams. `tests/test_genotype.py` (+7). Experiment dispatch (`kind` in {genotype_lineage, society}) + `make layer1` wired. +**2026-07-05 — sexual-transmission model made rigorous (E9/E10): landscape robustness + directed sex.** + +- GG excited by the sexual metaphor; wanted it robust before the full society. Added NK landscape + (`genotype.nk_fitness`), finite n-parent `crossover`, `hill_climb` (parents = local optima). +- **E9 (`recomb_landscape`) — "why sex?":** on rugged/epistatic landscapes, blind recombination → + **outbreeding depression** (offspring below parents, worse with ruggedness + recombination rate); + the optimal recombination rate shrinks with ruggedness. Design rule: merge freely when + complementary, sparingly + selectively when entangled. +- **E10 (`directed_sex`) — AI beats biology:** random ("biological") sex craters with ruggedness + (0.66→0.51); **directed sex** (choose mates + select offspring + unbounded parents, iterated) tracks/ + exceeds the best parent at every ruggedness. The distinctly-AI superpower, no biological analog. +- Complete picture: dramatic super-parent offspring when complementary (E8); outbreeding-depression + risk when entangled (E9); directed sex resolves it (E10). `configs/layer1/{E9,E10}.yaml`, + `plot_{E9,E10}.py`, READMEs, +5 tests (117 green). + ## Remaining (all optional / next) - [ ] **The dynamic society:** an evolving *population of parents* (specialists) that ground + diff --git a/tests/test_genotype.py b/tests/test_genotype.py index a04dd73..65d5a78 100644 --- a/tests/test_genotype.py +++ b/tests/test_genotype.py @@ -11,11 +11,11 @@ from __future__ import annotations import numpy as np from knowledge.genotype import ( - additive_fitness, genotype_bits, linkage_equilibrium, locus_marginals, mutate, - recombine, recombine_teachers, + additive_fitness, bits_to_index, crossover, genotype_bits, hill_climb, linkage_equilibrium, + locus_marginals, mutate, nk_fitness, recombine, recombine_teachers, ) from knowledge.genotype_lineage import run_genotype_lineage -from knowledge.society import make_specialist, run_society +from knowledge.society import make_specialist, run_directed_sex, run_recomb_landscape, run_society from knowledge.teachers import make_retention_matrix @@ -88,3 +88,52 @@ def test_e7_sexual_adapts_at_least_as_fast(): s = sex[sex["generation"] == mid]["mean_fitness"].iloc[0] assert s >= a - 1e-9 # sexual adapts at least as fast mid-run assert sex["ld"].max() < asex["ld"].max() # ... by keeping loci in linkage equilibrium + + +def test_nk_fitness_shape_range_and_additive_limit(): + f0 = nk_fitness(6, 0, seed=1) + assert f0.shape == (64,) and f0.min() >= 0.0 and f0.max() <= 1.0 + # K=0 is additive: fitness separates into a sum of per-locus contributions, so the effect of + # flipping one locus is independent of the others (check two backgrounds agree). + bits = genotype_bits(6) + d_from_0 = f0[bits[:, 0] == 1].mean() - f0[bits[:, 0] == 0].mean() + assert np.isfinite(d_from_0) + f5 = nk_fitness(6, 5, seed=1) + assert not np.allclose(f0, f5) # ruggedness changes the landscape + + +def test_crossover_clones_at_rate_zero_and_stays_valid(): + rng = np.random.default_rng(0) + parents = genotype_bits(8)[[3, 200]] # two parent genotypes + child = crossover(parents, 0.0, rng) # rate 0 -> a clone of one parent + assert bits_to_index(child) in (3, 200) + idx = bits_to_index(crossover(parents, 0.5, rng)) + assert 0 <= idx < 256 # free recombination still a valid genotype + + +def test_hill_climb_reaches_local_optimum(): + f = nk_fitness(8, 3, seed=2) + g = hill_climb(f, 8, start=0) + assert all(f[g ^ (1 << l)] <= f[g] for l in range(8)) # no improving single flip + + +def test_e9_outbreeding_depression_on_rugged_landscape(): + # Blindly recombining local optima of a rugged landscape produces below-parent offspring, and + # free recombination is worse than clonal. + cfg = {"experiment": "e9t", "seed": 1, "n_replicates": 10, + "society": {"L": 10, "n_parents": 6, "pop": 150}, + "sweep": [{"param": "K", "values": [6]}, {"param": "rate", "values": [0.0, 0.5]}]} + df = run_recomb_landscape(cfg) + m = df.groupby("rate")[["mean_offspring", "best_parent"]].mean() + assert m.loc[0.5, "mean_offspring"] < m.loc[0.5, "best_parent"] # depression + assert m.loc[0.5, "mean_offspring"] < m.loc[0.0, "mean_offspring"] # free recomb is worse + + +def test_e10_directed_sex_beats_random_and_holds_parents(): + cfg = {"experiment": "e10t", "seed": 1, "n_replicates": 10, + "society": {"L": 10, "n_parents": 6, "pop": 150, "keep": 8, "rounds": 5, "rate": 0.2}, + "sweep": [{"param": "K", "values": [6]}]} + df = run_directed_sex(cfg).mean(numeric_only=True) + assert df["directed_sex"] > df["random_sex"] + 0.05 # directed rescues the catastrophe + assert df["directed_sex"] >= df["best_parent"] - 0.01 # ... to (at least) the best parent + assert df["random_sex"] < df["best_parent"] # blind sex suffers depression