llm_directed: directed sex (breed offspring + select on verifier) — E10 in real weights

Adds the "directed sex" operator (E10) the moe regime-flip pointed to: don't
commit to one a-priori blend — breed a population of recombinant offspring
(specialists merged at Dirichlet-sampled weights), score each on a held-out
validation split with the verifier, and keep the fittest, reported on a fresh
test split. Two breeding objectives: best-overall and best-worst-family.
src/llm/directed.py + kind llm_directed, reusing the cached specialists.

Result — refinements pay off in proportion to how far the uniform soup is from
optimal:
- 0.5B (soup dilutes): directed selection beats soup on the bred objective —
  directed_overall 0.69 > soup 0.64; directed_balanced worst-family 0.37 > 0.26.
  Riders: single-objective selection trades off the other axis (overall-breed
  tanks lists to 0.17); a global blend still trails per-input routing (0.74).
- 7B (Imperial CX3, soup already composes to ceiling on near-saturated families,
  strings/arith 1.00): directed ~= soup (0.868 ~ 0.873, marginally below via a
  val/test overfit gap) — no fitter offspring to breed.

Through-line across all four LLM runs: "merge, don't average" and its refinements
(routing, directed selection) are weak-base / suboptimal-default phenomena — they
help at 0.5B and are inert at 7B. Honest limitation kept in the writeup: the 7B
families are near-saturated, which caps the headroom; a harder unsaturated
benchmark is the fair next test.

Also folds in the two llm_moe local manifest/config files missed in 8da0dac.
+3 directed unit tests (130 green). Results in results/llm_directed{,_hpc}/
(parquet gitignored).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

CLAUDE.md

@@ -95,6 +95,8 @@ E4's whole purpose is to isolate the effect of teacher **decorrelation ρ**, so
 
 **Finding (2026-07-05, `llm_moe_hpc` — the regime *flips* at 7B; "merge, don't average" is a weak-base law).** Re-ran `llm_moe` at **Qwen2.5-7B-Instruct** (L40S, 9 min, reusing the cached 7B specialists). **The union-vs-fusion ordering inverts:** at 0.5B union won (routing 0.74 > soup 0.64); at 7B **fusion wins — soup 0.87 > routing 0.84 > max_merge 0.78.** Mechanism, and it's the deep point: **routing *selects* one intact specialist so it is capped at the best parent per family** (lists 0.57 = spec_lists, strings 0.97 = spec_strings), whereas **fusion *blends* deltas and, at a capable base, composes *beyond* any parent** (soup lists 0.62 > spec 0.57, strings 1.00 > spec 0.97). Selection can't synthesise something better than its best component; averaging-that-composes can. So the E4/E8 "merge, don't average" law is **regime-dependent — a weak-parent / small-model law, not universal**: union wins exactly when averaging *dilutes* (0.5B), fusion wins once the base has headroom to *compose* (7B). This refines rather than contradicts E8 (whose additive-landscape `max>mean` assumed no compositional headroom). The operator to actually want is **fusion-that-composes + selection over recombinant offspring** = the "directed sex" ideal (E10), the natural next experiment. `results/llm_moe_hpc/` (README + regime-aware figure title). Riders unchanged: learned router trivially perfect (lexical families), `max_merge` the weakest union (not input-adaptive).
 
+**Finding (2026-07-05, `llm_directed` — directed sex in weights; refinements pay off only when the default blend is suboptimal).** E10 in real LLM weights (`src/llm/directed.py`, `kind: llm_directed`): breed a *population* of recombinant offspring (specialists merged at Dirichlet-sampled weights), score each against the verifier on a held-out **validation** split, keep the fittest — reported on a fresh **test** split (no selection-on-test leakage). Two breeding objectives (best-overall, best-worst-family). **The value scales with how far the uniform soup is from optimal, giving a clean regime split:** **0.5B** — soup dilutes, so directed selection beats it (`directed_overall` 0.69 > soup 0.64; `directed_balanced` worst-family 0.37 > soup 0.26), though single-objective selection trades off the other axis (breeding for overall tanks the rare `lists` to 0.17) and a *global* blend still trails per-input **routing** (0.74). **7B** — soup already *composes* to the ceiling on these near-saturated families (strings & arith at 1.00), so directed selection finds nothing better: **directed 0.868 ≈ soup 0.873** (marginally below, a val/test overfit gap). **Honest limitation:** the 7B families are near-saturated (2/3 at 1.00), which structurally caps the headroom — this run can't separate "directed sex doesn't help at scale" from "these tasks are too easy at 7B"; a *harder, unsaturated* benchmark is the fair next test. **Through-line across all four LLM runs:** "merge, don't average" and its refinements (routing, directed selection) are **weak-base / suboptimal-default** phenomena — they pay off at 0.5B (soup far from optimal) and are inert at 7B (soup near-optimal on saturated tasks). `configs/llm/{directed,directed_hpc}.yaml`, `plot_llm_directed.py`, `results/llm_directed{,_hpc}/`, `hpc/llm_directed.pbs`, +3 tests (130 green).
+
 ## Build order (blueprint §7) — respect the gate
 
 1. Scaffold: repo layout (§5), container, pytest skeleton, config system, seeding utils. `make test` green.

Makefile

@@ -28,9 +28,10 @@ mnist:             ## run the real-MNIST confirmation tier (needs env-mnist; dow
 env-llm:           ## add the LLM stack for the Layer-2 prototype (GPU; transformers/peft)
 	uv sync --extra dev --extra neural --extra llm
 
-llm:               ## run the LLM prototypes: merge (fusion) + moe (union) recombination
+llm:               ## run the LLM prototypes: merge (fusion) + moe (union) + directed (offspring select)
 	uv run python -m llm.experiment configs/llm/merge.yaml
 	uv run python -m llm.experiment configs/llm/moe.yaml
+	uv run python -m llm.experiment configs/llm/directed.yaml
 
 layer2: neural     ## alias: Layer 1.5 is the current Layer-2 deliverable (LLM rung deferred)
 

configs/llm/directed.yaml

@@ -0,0 +1,25 @@
+experiment: llm_directed
+kind: llm_directed
+seed: 1
+n_replicates: 1
+
+# Layer 2 / LLM — DIRECTED SEX (E10) in weight space: breed many recombinant offspring and keep the
+# fittest. llm_moe showed fusion can COMPOSE beyond the parents (7B) but the right blend is unknown and
+# base-dependent, while pure routing is capped at the best parent. Directed sex resolves both: sample a
+# population of weighted merges of the specialists, score each on a held-out VALIDATION split with the
+# verifier (grounding), and select the winner — reported on a fresh TEST split (no selection-on-test
+# leakage). Two objectives: best-overall and best-worst-family (raw capability vs the balanced
+# generalist). Prediction: directed selection beats the single a-priori soup and every parent, at
+# either scale. Falsifier: directed offspring ≤ uniform soup on test.
+
+base_model: Qwen/Qwen2.5-0.5B-Instruct   # reuses the cached llm_merge specialists
+families: [lists, strings, arith]
+n_train: 700          # only if cached specialists are absent
+n_val: 80             # held-out split the verifier selects on
+n_test: 100           # fresh split winners are reported on
+n_candidates: 16      # offspring population size
+concentration: 0.5    # Dirichlet concentration (<1 = sparser, specialist-dominant blends)
+epochs: 3
+lora: {r: 16, alpha: 32}
+
+output: {dir: results/llm_directed}

configs/llm/directed_hpc.yaml

@@ -0,0 +1,21 @@
+experiment: llm_directed_hpc
+kind: llm_directed
+seed: 1
+n_replicates: 1
+
+# Scaled directed-sex run for an L40S (48 GB). At 7B the uniform soup already composes (0.87, beating
+# every specialist), so the sharp question is whether SEARCHING the recombination-weight space +
+# selecting on the verifier can push past even that strong a-priori blend — i.e. does offspring
+# selection still add headroom once fusion is already strong? Reuses the cached 7B specialists.
+
+base_model: Qwen/Qwen2.5-7B-Instruct
+families: [lists, strings, arith]
+n_train: 800          # only if cached specialists are absent (fresh on the node)
+n_val: 100
+n_test: 200
+n_candidates: 24
+concentration: 0.5
+epochs: 3
+lora: {r: 16, alpha: 32}
+
+output: {dir: results/llm_directed_hpc}

figures/plot_llm_directed.py

@@ -0,0 +1,94 @@
+"""llm_directed figure — directed sex in weight space (E10): breed offspring, select the fittest.
+
+A population of recombinant offspring (weighted merges of the specialists) is scored on a held-out
+validation split by the verifier; the winners (best-overall, best-worst-family) are reported on a
+fresh test split against the uniform-soup blend and the best single specialist. Two panels: (A)
+per-family accuracy — directed offspring (green) vs soup (orange) vs specialists (blue); (B) overall
+vs worst-family, with the best-specialist bar as the parent ceiling. The suptitle reports whether
+directed selection beat the single a-priori soup. Reads only the committed bundle.
+
+Usage:  python figures/plot_llm_directed.py [results/llm_directed]
+"""
+
+from __future__ import annotations
+
+import sys
+from pathlib import Path
+
+import matplotlib.pyplot as plt
+import numpy as np
+
+sys.path.insert(0, str(Path(__file__).parent))
+from _figlib import load_bundle, savefig  # noqa: E402
+
+_FAMS = ["lists", "strings", "arith"]
+_DIRECTED = {"directed_overall": "directed:overall", "directed_balanced": "directed:balanced"}
+
+
+def _acc(df, model, metric):
+    r = df[(df["model"] == model) & (df["metric"] == metric)]["accuracy"]
+    return float(r.iloc[0]) if len(r) else float("nan")
+
+
+def main(results_dir: str = "results/llm_directed") -> None:
+    df, cfg = load_bundle(results_dir)
+    present = set(df["model"].unique())
+    specialists = sorted(m for m in present if m.startswith("spec_"))
+    directed = [m for m in _DIRECTED if m in present]
+    soup = ["merge_soup"] if "merge_soup" in present else []
+    models = ["base"] + specialists + soup + directed
+    labels = {"base": "base", **{s: s.replace("spec_", "spec:") for s in specialists},
+              "merge_soup": "soup (uniform)", **_DIRECTED}
+    colors = {"base": "#7f7f7f", **{s: "#1f77b4" for s in specialists},
+              "merge_soup": "#ff7f0e", **{m: "#2ca02c" for m in directed}}
+
+    fig, axes = plt.subplots(1, 2, figsize=(13, 5))
+
+    # Panel A: per-family accuracy.
+    ax = axes[0]
+    x = np.arange(len(_FAMS))
+    w = 0.8 / len(models)
+    for i, mdl in enumerate(models):
+        vals = [_acc(df, mdl, f) for f in _FAMS]
+        ax.bar(x + (i - (len(models) - 1) / 2) * w, vals, w, label=labels[mdl], color=colors[mdl],
+               alpha=0.9 if mdl in directed else 0.65)
+    ax.set_xticks(x); ax.set_xticklabels(_FAMS)
+    ax.set(ylabel="accuracy", title="Per-family: directed offspring (green), selected on the\n"
+           "verifier, vs the single uniform soup (orange) and the parents")
+    ax.legend(frameon=False, fontsize=8, ncol=2)
+
+    # Panel B: overall vs worst-family, with the best-specialist ceiling.
+    ax = axes[1]
+    x2 = np.arange(len(models))
+    for off, metric, hatch, lab in [(-0.2, "overall", "", "overall"),
+                                    (0.2, "worst_family", "//", "worst family")]:
+        ax.bar(x2 + off, [_acc(df, m, metric) for m in models], 0.38,
+               color=[colors[m] for m in models], hatch=hatch, alpha=0.85, label=lab,
+               edgecolor="white")
+    ax.set_xticks(x2); ax.set_xticklabels([labels[m] for m in models], rotation=25, ha="right",
+                                          fontsize=8)
+    if specialists:
+        ceil = max(_acc(df, s, "overall") for s in specialists)
+        ax.axhline(ceil, ls=":", c="#1f77b4", lw=1, alpha=0.7)      # best-parent ceiling
+    ax.set(ylabel="accuracy", title="Overall (solid) vs worst-family (hatched):\n"
+           "directed offspring vs soup vs the best parent (dotted)")
+    ax.legend(frameon=False, fontsize=9)
+
+    best_dir = max([_acc(df, m, "overall") for m in directed], default=float("nan"))
+    soup_o = _acc(df, "merge_soup", "overall")
+    best_spec = max([_acc(df, s, "overall") for s in specialists], default=float("nan"))
+    if best_dir > soup_o + 0.005:
+        verdict = f"directed {best_dir:.2f} > soup {soup_o:.2f} overall"
+    elif best_dir > soup_o - 0.005:
+        verdict = f"directed {best_dir:.2f} ≈ soup {soup_o:.2f} overall"
+    else:
+        verdict = f"directed {best_dir:.2f} < soup {soup_o:.2f} overall"
+    verdict += f" (best parent {best_spec:.2f})"
+    fig.suptitle(f"llm_directed — directed sex (breed offspring + select on the verifier): {verdict} "
+                 f"({cfg['base_model'].split('/')[-1]})", y=1.0, fontsize=12)
+    fig.tight_layout()
+    savefig(fig, results_dir, "llm_directed")
+
+
+if __name__ == "__main__":
+    main(*sys.argv[1:])

hpc/llm_directed.pbs

@@ -0,0 +1,22 @@
+#!/bin/bash
+# Directed sex (llm_directed) on an L40S (46 GB) — breed recombinant offspring + select on the verifier.
+# Reuses the cached 7B specialist adapters (models/llm/spec_*) if present. Same env as the other LLM
+# jobs (see hpc/README.md): uv sync --extra dev --extra neural --extra llm + pre-download the 7B base.
+#   submit:  qsub hpc/llm_directed.pbs      status: qstat -u $USER
+#PBS -l select=1:ncpus=8:mem=64gb:ngpus=1:gpu_type=L40S
+#PBS -l walltime=02:00:00
+#PBS -N lam_llm_directed
+
+cd "$PBS_O_WORKDIR"
+export HF_HOME="$EPHEMERAL/hf_cache"
+export TOKENIZERS_PARALLELISM=false
+export UV_CACHE_DIR="$EPHEMERAL/uvcache"
+
+source .venv/bin/activate
+nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv,noheader
+
+python -m llm.experiment configs/llm/directed_hpc.yaml
+
+# results/llm_directed_hpc/ written in-place (parquet gitignored). Sync back to plot:
+#   rsync -avz hpc:'…/LamarckianAI/results/llm_directed_hpc/' results/llm_directed_hpc/
+echo "done: $(date)"

results/llm_directed/README.md

@@ -0,0 +1,50 @@
+# llm_directed — directed sex in weight space: breed offspring + select on the verifier (E10, 0.5B)
+
+**Claim tested.** `llm_moe` left a clean gap: fusion can *compose* beyond the parents but the right
+blend is unknown and base-dependent, while pure routing is capped at the best parent. E10's answer is
+**directed sex** — biology can't preview offspring, an AI can: breed a *population* of recombinant
+offspring (the specialists merged at many different weights), score each against the verifier
+("reality") on a held-out validation split, and keep the fittest. Selection replaces betting on one
+a-priori blend. Two breeding objectives: best validation **overall**, and best validation
+**worst-family** (raw capability vs the balanced generalist).
+
+**Setup.** Base **Qwen2.5-0.5B-Instruct**, the three cached `llm_merge` specialists, **16 offspring**
+(Dirichlet-weighted merges, concentration 0.5, pinning candidate 0 = uniform soup for reference),
+scored on an **80-task/family validation** split, winners reported on a **fresh 100-task/family test**
+split (no selection-on-test leakage). Seed 1.
+
+### Results (test accuracy)
+| model | lists | strings | arith | overall | worst-family |
+|---|---|---|---|---|---|
+| best specialist (strings) | 0.08 | 1.00 | 0.80 | 0.63 | 0.08 |
+| merge_soup (uniform, candidate 0) | 0.26 | 0.74 | 0.91 | 0.64 | 0.26 |
+| **directed_overall** (bred for overall) | 0.17 | 0.99 | 0.92 | **0.69** | 0.17 |
+| **directed_balanced** (bred for worst-family) | 0.37 | 0.37 | 0.79 | 0.51 | **0.37** |
+
+### What holds, and the honest cost
+- **Offspring selection beats the single a-priori blend — on the objective you breed for.**
+  `directed_overall` reaches **0.69 overall > soup 0.64** (and > best parent 0.63); `directed_balanced`
+  reaches **0.37 worst-family > soup 0.26**. Searching the recombination-weight space and letting the
+  verifier choose beats committing to uniform averaging — the E10 "preview and keep the fittest" claim,
+  in real weights.
+- **Single-objective selection trades off the other axis (honest).** Breeding for *overall* on
+  lexically-imbalanced families finds a strings+arith-heavy blend that sacrifices the rare `lists`
+  skill (0.17, below soup's 0.26); breeding for *balance* lifts worst-family to 0.37 but costs overall.
+  Directed sex gives *control* over what you breed for — it does not hand you both for free.
+- **A global blend still trails per-input routing at a weak base.** At 0.5B the best directed *global*
+  merge (0.69 / 0.43-max) does not beat `llm_moe`'s per-input **routing** (0.74 / 0.43): when the base
+  is weak, adapting the recombination *per input* beats any one fixed blend, however well selected. So
+  directed sex over blends beats *averaging*, not *routing* — combining the two (route, then select
+  among routed+blended offspring) is the natural next operator.
+
+### Takeaway
+Directed sex — breed a population, select on the verifier — is confirmed in real LLM weights: it beats
+the single uniform soup on whichever objective it optimises, the distinctly-AI advantage (offspring
+preview + unbounded candidates) that biology lacks. The honest scope at 0.5B: selection buys one axis
+at the other's expense, and a single global blend can't yet beat per-input routing. Whether searching
+blends + selection can exceed even the *strong* 7B soup (which routing could not) is answered by
+**`results/llm_directed_hpc/`: it can't — directed ≈ soup (0.868 ≈ 0.873)** because the 7B soup already
+composes to the ceiling on these near-saturated families, leaving no fitter offspring to breed. So
+directed sex helps exactly when the default blend is *suboptimal* (0.5B), and is inert when it is
+already near-optimal (7B). **Falsifier (not triggered at 0.5B):** directed offspring ≤ uniform soup on
+their bred objective — instead each beat it.

results/llm_directed/manifest.json

@@ -0,0 +1,25 @@
+{
+  "experiment": "llm_directed",
+  "master_seed": 1,
+  "git_commit": "8da0dac00713fb9708804b4696a847a3767758d5",
+  "python": "3.14.5",
+  "libraries": {
+    "numpy": "2.5.0",
+    "scipy": "1.18.0",
+    "pandas": "3.0.3",
+    "pyarrow": "24.0.0",
+    "torch": "2.12.1",
+    "transformers": "5.13.0",
+    "peft": "0.19.1"
+  },
+  "rows": 35,
+  "results_sha256": "143b86833cb320c7c6c693f0b5394f381a94a69277cc31ba5e0977e5ea6ffd78",
+  "layer": "2",
+  "tier": "llm",
+  "base_model": "Qwen/Qwen2.5-0.5B-Instruct",
+  "directed": {
+    "n_candidates": 16,
+    "concentration": 0.5,
+    "n_val": 80
+  }
+}

results/llm_directed/resolved_config.yaml

@@ -0,0 +1,24 @@
+experiment: llm_directed
+seed: 1
+n_replicates: 1
+source_config:
+  experiment: llm_directed
+  kind: llm_directed
+  seed: 1
+  n_replicates: 1
+  base_model: Qwen/Qwen2.5-0.5B-Instruct
+  families:
+  - lists
+  - strings
+  - arith
+  n_train: 700
+  n_val: 80
+  n_test: 100
+  n_candidates: 16
+  concentration: 0.5
+  epochs: 3
+  lora:
+    r: 16
+    alpha: 32
+  output:
+    dir: results/llm_directed

results/llm_directed_hpc/README.md

@@ -0,0 +1,49 @@
+# llm_directed_hpc — directed sex at scale (7B): no headroom once the soup already composes
+
+**Claim tested.** At 0.5B, breeding offspring and selecting on the verifier beat the uniform soup
+(`llm_directed`: directed 0.69 > soup 0.64). But `llm_merge_hpc` showed the 7B soup already *composes*
+to 0.87, beating every specialist. So the honest question: does searching the recombination-weight
+space + selecting on the verifier find anything better than that strong default — or is there simply no
+headroom left? Run on one **L40S (46 GB)** GPU of Imperial's CX3 HPC, 9 min walltime, 24 offspring,
+reusing the cached 7B specialists.
+
+**Setup.** Base **Qwen2.5-7B-Instruct**, cached 7B specialists, **24 offspring** (Dirichlet-weighted
+merges, concentration 0.5, candidate 0 = uniform soup), scored on a **100-task/family validation**
+split, winners reported on a **fresh 200-task/family test** split. Seed 1.
+
+### Results (test accuracy)
+| model | lists | strings | arith | overall | worst-family |
+|---|---|---|---|---|---|
+| best specialist (lists) | 0.57 | 0.74 | 1.00 | 0.77 | 0.57 |
+| merge_soup (uniform, candidate 0) | 0.62 | 1.00 | 1.00 | **0.873** | 0.625 |
+| directed_overall (bred for overall) | 0.62 | 1.00 | 1.00 | 0.868 | 0.615 |
+| directed_balanced (bred for worst-family) | 0.62 | 1.00 | 1.00 | 0.868 | 0.615 |
+
+### The finding: directed selection is inert once the default is already near-optimal
+- **Directed ≈ soup, and marginally below it (0.868 vs 0.873).** Both breeding objectives converged to
+  a near-soup blend, and the validation-selected winner is a *hair* worse than the plain uniform soup
+  on the held-out test set — a small val/test generalisation gap (selection overfits the 100-task/
+  family validation split). Searching 24 offspring found nothing that beats candidate 0.
+- **Why: no headroom.** The 7B soup already *composes* to the ceiling on these families — strings and
+  arith are saturated at **1.00**, and lists (0.62) is the only slack, itself already above every
+  specialist. When the default blend is already optimal, there is no fitter offspring to breed, so
+  selection can only match it (or lose slightly to val noise).
+- **Honest limitation.** These task families are *near-saturated* at 7B (2 of 3 at 1.00), which
+  structurally caps the headroom any recombination refinement could exploit. A fair test of directed
+  sex at scale needs a **harder, unsaturated** benchmark where the optimal blend is non-trivial — this
+  run cannot distinguish "directed sex doesn't help at scale" from "these tasks are too easy at 7B."
+
+### Takeaway — the through-line across all four LLM runs
+The value of every recombination *refinement* (routing, directed selection) scales with **how far the
+default uniform soup is from optimal**:
+- **0.5B** — soup *dilutes* (far from optimal): routing beats soup (0.74 > 0.64), directed selection
+  beats soup (0.69 > 0.64). Refinements pay off.
+- **7B** — soup *composes* to near-ceiling on saturated tasks: routing < soup (0.84 < 0.87), directed
+  ≈ soup (0.868 ≈ 0.873). No headroom; refinements are inert.
+
+So "merge, don't average" and its refinements are **weak-base / suboptimal-default** phenomena. The
+open question a capable base leaves is whether directed sex helps on *hard, unsaturated* tasks at scale
+— the natural next benchmark. **Falsifier for this run (triggered as a null, reported honestly):**
+directed offspring failed to exceed the uniform soup at 7B; here they tied/marginally trailed it
+because the soup was already optimal on near-saturated families. Provenance in `manifest.json`
+(L40S, torch 2.12.1 / transformers 5.13.0 / peft 0.19.1; `git_commit: null` — rsync'd node copy).

results/llm_directed_hpc/manifest.json

@@ -0,0 +1,25 @@
+{
+  "experiment": "llm_directed_hpc",
+  "master_seed": 1,
+  "git_commit": null,
+  "python": "3.11.13",
+  "libraries": {
+    "numpy": "2.4.6",
+    "scipy": "1.17.1",
+    "pandas": "3.0.3",
+    "pyarrow": "24.0.0",
+    "torch": "2.12.1",
+    "transformers": "5.13.0",
+    "peft": "0.19.1"
+  },
+  "rows": 35,
+  "results_sha256": "5bb2aef441ac8e00d3e6f8de02f74686415c367097b07f0688bbd687eb93d25d",
+  "layer": "2",
+  "tier": "llm",
+  "base_model": "Qwen/Qwen2.5-7B-Instruct",
+  "directed": {
+    "n_candidates": 24,
+    "concentration": 0.5,
+    "n_val": 100
+  }
+}

results/llm_directed_hpc/resolved_config.yaml

@@ -0,0 +1,24 @@
+experiment: llm_directed_hpc
+seed: 1
+n_replicates: 1
+source_config:
+  experiment: llm_directed_hpc
+  kind: llm_directed
+  seed: 1
+  n_replicates: 1
+  base_model: Qwen/Qwen2.5-7B-Instruct
+  families:
+  - lists
+  - strings
+  - arith
+  n_train: 800
+  n_val: 100
+  n_test: 200
+  n_candidates: 24
+  concentration: 0.5
+  epochs: 3
+  lora:
+    r: 16
+    alpha: 32
+  output:
+    dir: results/llm_directed_hpc

results/llm_moe/manifest.json

@@ -0,0 +1,27 @@
+{
+  "experiment": "llm_moe",
+  "master_seed": 1,
+  "git_commit": "585264d0b42f0e829229611bd83b08f5a5e418b7",
+  "python": "3.14.5",
+  "libraries": {
+    "numpy": "2.5.0",
+    "scipy": "1.18.0",
+    "pandas": "3.0.3",
+    "pyarrow": "24.0.0",
+    "torch": "2.12.1",
+    "transformers": "5.13.0",
+    "peft": "0.19.1"
+  },
+  "rows": 47,
+  "results_sha256": "3e73caaeae3b5d77ded3ba956af4b767c0d4025a41a7ae1841806ce74b78045e",
+  "layer": "2",
+  "tier": "llm",
+  "base_model": "Qwen/Qwen2.5-0.5B-Instruct",
+  "operators": [
+    "soup",
+    "ties",
+    "moe_oracle",
+    "moe_learned",
+    "max_merge"
+  ]
+}

results/llm_moe/resolved_config.yaml

@@ -0,0 +1,28 @@
+experiment: llm_moe
+seed: 1
+n_replicates: 1
+source_config:
+  experiment: llm_moe
+  kind: llm_moe
+  seed: 1
+  n_replicates: 1
+  base_model: Qwen/Qwen2.5-0.5B-Instruct
+  families:
+  - lists
+  - strings
+  - arith
+  n_train: 700
+  n_test: 100
+  n_route: 32
+  epochs: 3
+  lora:
+    r: 16
+    alpha: 32
+  operators:
+  - soup
+  - ties
+  - moe_oracle
+  - moe_learned
+  - max_merge
+  output:
+    dir: results/llm_moe

src/llm/directed.py

@@ -0,0 +1,72 @@
+"""Directed sex in weight space — recombinant offspring + selection on the verifier (E10 in real LLMs).
+
+`llm_merge` blends the specialists with *one* fixed rule (uniform soup, or ties); `llm_moe` *selects*
+one intact specialist per input. Both commit to a single recombination *a priori*. Biology can't
+preview offspring; an AI can — evaluate many recombinants and keep the fittest. This is E10's
+"directed sex": generate a **population** of offspring by recombining the parents at *different* mixing
+weights, score each against the verifier ("reality") on a held-out validation split, and select the
+best. It unifies the two regimes `llm_moe` exposed — fusion *composes* beyond the parents (so we want
+blends, not pure selection), but the *right* blend is unknown and base-dependent (so we search it and
+let grounding choose), instead of betting on uniform averaging.
+
+This module holds the pure, testable pieces — sampling a diverse population of simplex-ish merge
+weights, and selecting winners from validation scores. The weight-space recombination + evaluation
+loop lives in :func:`llm.experiment.run_directed_experiment` (it needs the loaded PEFT model).
+"""
+
+from __future__ import annotations
+
+import numpy as np
+
+
+def sample_merge_weights(k: int, n: int, rng: np.random.Generator, *, concentration: float = 0.5,
+                         scale_lo: float = 1.0, scale_hi: float | None = None) -> np.ndarray:
+    """Sample ``n`` diverse recombination-weight vectors over ``k`` parents (the offspring genotypes).
+
+    Each row is a Dirichlet draw (direction on the simplex) times a random total magnitude, spanning
+    from soup-like (total ≈ 1, balanced blend) to task-arithmetic-like (total ≈ ``k``, additive).
+    ``concentration < 1`` biases toward *sparse* mixes (one or two parents dominant) for real
+    diversity — the point of previewing many offspring rather than one average.
+
+    The first two rows are pinned to the canonical baselines for coverage: uniform **soup**
+    (``1/k`` each) and unit **task-arithmetic** (``1`` each); the remaining ``n-2`` are random.
+
+    Args:
+        k (int): number of parents (specialists).
+        n (int): population size (candidates). Must be ≥ 2.
+        rng (np.random.Generator): explicit RNG (seeded upstream via SeedSequence).
+        concentration (float): Dirichlet concentration; < 1 → sparser, specialist-dominant blends.
+        scale_lo (float): minimum total weight magnitude.
+        scale_hi (float | None): maximum total weight magnitude (defaults to ``k``).
+
+    Returns:
+        np.ndarray: ``(n, k)`` float32 merge-weight vectors.
+    """
+    if n < 2:
+        raise ValueError("need at least 2 candidates (soup + task_arith baselines)")
+    hi = float(k) if scale_hi is None else float(scale_hi)
+    out = np.empty((n, k), dtype=np.float32)
+    out[0] = np.full(k, 1.0 / k)                           # uniform soup
+    out[1] = np.ones(k)                                    # task arithmetic
+    for i in range(2, n):
+        direction = rng.dirichlet(np.full(k, concentration))
+        total = rng.uniform(scale_lo, hi)
+        out[i] = direction * total
+    return out
+
+
+def select_winners(val_overall: np.ndarray, val_worst: np.ndarray) -> dict:
+    """Pick the offspring that maximise validation *overall* and validation *worst-family* accuracy.
+
+    Two selection objectives = two things directed sex can breed for: raw capability, or balance
+    across skills (the Fisher–Muller generalist). Selection is on validation only; the winners are
+    then reported on a fresh test split (no selection-on-test leakage).
+
+    Args:
+        val_overall (np.ndarray): per-candidate validation overall accuracy.
+        val_worst (np.ndarray): per-candidate validation worst-family accuracy.
+
+    Returns:
+        dict: ``{"overall": idx, "balanced": idx}`` candidate indices.
+    """
+    return {"overall": int(np.argmax(val_overall)), "balanced": int(np.argmax(val_worst))}

src/llm/experiment.py

@@ -22,6 +22,7 @@ import yaml
 
 from knowledge.experiment import save_artifacts
 
+from .directed import sample_merge_weights, select_winners
 from .evaluate import evaluate, generate, load_model
 from .merge import load_specialists, make_merge
 from .moe import build_max_merge, embed_prompts, learned_routes, moe_generate
@@ -184,7 +185,74 @@ def run_moe_experiment(cfg: dict) -> pd.DataFrame:
     return pd.DataFrame(rows)
 
 
-_RUNNERS = {"llm_merge": run_merge_experiment, "llm_moe": run_moe_experiment}
+def run_directed_experiment(cfg: dict) -> pd.DataFrame:
+    """Directed sex (E10) in weight space: breed many recombinant offspring, select the fittest.
+
+    Generates a population of weighted merges of the specialists, scores each on a held-out
+    *validation* split with the verifier (grounding = "reality that says no"), and keeps the two
+    winners — best validation *overall* and best validation *worst-family* — reporting them on a
+    fresh *test* split alongside the uniform-soup and best-specialist baselines. The claim (E10): an
+    AI can preview offspring and keep the fittest, so directed selection over recombinants beats both
+    the single a-priori blend (soup) and any parent, at either scale.
+    """
+    import torch
+
+    name = cfg["experiment"]
+    base = cfg["base_model"]
+    fams = list(cfg.get("families", list(FAMILIES)))
+    n_val = int(cfg.get("n_val", 80))
+    n_cand = int(cfg.get("n_candidates", 16))
+    conc = float(cfg.get("concentration", 0.5))
+    seed = int(cfg["seed"])
+    rows: list[dict] = []
+
+    test = test_of(cfg, fams)
+    val = sum([make_tasks(f, n_val, seed=3000 + i) for i, f in enumerate(fams)], [])
+
+    # base + specialists (parents), scored on test
+    m, tok = load_model(base)
+    rows += _rows(name, "base", "base", evaluate(m, tok, test))
+    del m; torch.cuda.empty_cache()
+    dirs = _load_or_train_specialists(cfg, base, fams, name, rows)
+    k = len(dirs)
+
+    # one base with all specialists; breed a population of weighted-merge offspring
+    model, tok = load_specialists(base, dirs)
+    rng = np.random.default_rng(seed)                      # Layer-2 statistical reproducibility
+    weights = sample_merge_weights(k, n_cand, rng, concentration=conc)
+    adapters = [f"a{i}" for i in range(k)]
+
+    val_overall = np.empty(n_cand)
+    val_worst = np.empty(n_cand)
+    for i in range(n_cand):
+        cname = f"cand{i}"
+        model.add_weighted_adapter(adapters, weights[i].tolist(), cname, combination_type="linear")
+        model.set_adapter(cname)
+        acc = evaluate(model, tok, val)                    # selection signal (validation only)
+        val_overall[i] = acc["overall"]
+        val_worst[i] = min(acc[f] for f in fams)
+
+    winners = select_winners(val_overall, val_worst)       # {"overall": idx, "balanced": idx}
+
+    def _test_acc(cname: str) -> dict:
+        model.set_adapter(cname)
+        outs = generate(model, tok, [t.prompt for t in test])
+        corr = np.array([verify(o, t) for o, t in zip(outs, test)])
+        famv = np.array([t.family for t in test])
+        acc = {"overall": float(corr.mean())}
+        acc.update({f: float(corr[famv == f].mean()) for f in fams})
+        return acc
+
+    # uniform soup baseline is candidate 0 by construction; report it on test for direct comparison
+    rows += _rows(name, "merge_soup", "merge", _test_acc("cand0"))
+    for label, idx in winners.items():
+        rows += _rows(name, f"directed_{label}", "directed", _test_acc(f"cand{idx}"))
+
+    return pd.DataFrame(rows)
+
+
+_RUNNERS = {"llm_merge": run_merge_experiment, "llm_moe": run_moe_experiment,
+            "llm_directed": run_directed_experiment}
 
 
 def run_and_save(config_path: str | Path) -> Path:
@@ -200,6 +268,10 @@ def run_and_save(config_path: str | Path) -> Path:
     extra = {"layer": "2", "tier": "llm", "base_model": cfg["base_model"]}
     if kind == "llm_moe":
         extra["operators"] = list(cfg.get("operators", []))
+    if kind == "llm_directed":
+        extra["directed"] = {"n_candidates": int(cfg.get("n_candidates", 16)),
+                             "concentration": float(cfg.get("concentration", 0.5)),
+                             "n_val": int(cfg.get("n_val", 80))}
     save_artifacts(cfg, df, out_dir, extra_libs=("torch", "transformers", "peft"),
                    extra_manifest=extra, grid=None)
     return out_dir

tasks/todo.md

@@ -367,3 +367,16 @@ C3 vertical claim deferred.*
   spec 0.57). So "merge, don't average" is a **weak-base law**, not universal — union wins under
   dilution (0.5B), fusion wins under composition (7B). Refines E8. Next: fusion + offspring-selection
   (directed sex). `results/llm_moe_hpc/` README + regime-aware figure.
+
+**2026-07-05 — Directed sex (`llm_directed`): E10 in weights = breed offspring + select on verifier.**
+- `src/llm/directed.py`: sample a population of Dirichlet-weighted merges, score on a held-out VAL
+  split, keep the best-overall + best-worst-family, report on a fresh TEST split. `kind: llm_directed`.
+- **0.5B:** directed selection beats the single a-priori soup on the bred objective — directed_overall
+  0.69 > soup 0.64; directed_balanced worst-family 0.37 > soup 0.26. Riders: single-objective selection
+  trades off the other axis (overall-breed tanks lists 0.17); a global blend still trails per-input
+  routing (0.74). **7B (CX3 L40S, 9 min):** directed ≈ soup (0.868 ≈ 0.873) — soup already composes to
+  ceiling on near-saturated families (strings/arith 1.00), no fitter offspring to breed.
+- **Through-line:** recombination refinements pay off ∝ how suboptimal the default soup is — big at
+  0.5B, nil at 7B. Honest limit: 7B families near-saturated; a harder benchmark is the fair next test.
+- `configs/llm/{directed,directed_hpc}.yaml`, `figures/plot_llm_directed.py`, READMEs, `hpc/llm_directed.pbs`,
+  Makefile `llm` target, +3 tests (130 green). `results/llm_directed{,_hpc}/`.

tests/test_llm.py

@@ -10,6 +10,7 @@ from __future__ import annotations
 
 import numpy as np
 
+from llm.directed import sample_merge_weights, select_winners
 from llm.moe import learned_routes
 from llm.tasks import FAMILIES, make_tasks, verify
 
@@ -66,3 +67,28 @@ def test_learned_router_is_cosine_scale_invariant():
     test_emb = np.array([[10.0, 0.0], [0.0, 0.01]])            # very different magnitudes
     routes = learned_routes(train_emb, train_fam, test_emb, fams)
     assert [fams[r] for r in routes] == ["a", "b"]
+
+
+def test_merge_weights_population_pins_baselines_and_diversifies():
+    # The offspring population must contain the two canonical baselines (uniform soup, unit task-arith)
+    # and be diverse + reproducible for the rest.
+    rng = np.random.default_rng(0)
+    w = sample_merge_weights(3, 16, rng)
+    assert w.shape == (16, 3)
+    assert np.allclose(w[0], 1 / 3)                             # candidate 0 = uniform soup
+    assert np.allclose(w[1], 1.0)                              # candidate 1 = task arithmetic
+    assert np.unique(w[2:].round(3), axis=0).shape[0] > 5      # the random offspring are diverse
+    assert np.allclose(sample_merge_weights(3, 16, np.random.default_rng(0)), w)  # deterministic
+
+
+def test_select_winners_picks_argmax_per_objective():
+    val_overall = np.array([0.5, 0.9, 0.7])
+    val_worst = np.array([0.4, 0.1, 0.6])                       # a different candidate is most balanced
+    w = select_winners(val_overall, val_worst)
+    assert w == {"overall": 1, "balanced": 2}
+
+
+def test_merge_weights_requires_two_candidates():
+    import pytest
+    with pytest.raises(ValueError):
+        sample_merge_weights(3, 1, np.random.default_rng(0))