llm_directed: directed sex (breed offspring + select on verifier) — E10 in real weights

Adds the "directed sex" operator (E10) the moe regime-flip pointed to: don't
commit to one a-priori blend — breed a population of recombinant offspring
(specialists merged at Dirichlet-sampled weights), score each on a held-out
validation split with the verifier, and keep the fittest, reported on a fresh
test split. Two breeding objectives: best-overall and best-worst-family.
src/llm/directed.py + kind llm_directed, reusing the cached specialists.

Result — refinements pay off in proportion to how far the uniform soup is from
optimal:
- 0.5B (soup dilutes): directed selection beats soup on the bred objective —
  directed_overall 0.69 > soup 0.64; directed_balanced worst-family 0.37 > 0.26.
  Riders: single-objective selection trades off the other axis (overall-breed
  tanks lists to 0.17); a global blend still trails per-input routing (0.74).
- 7B (Imperial CX3, soup already composes to ceiling on near-saturated families,
  strings/arith 1.00): directed ~= soup (0.868 ~ 0.873, marginally below via a
  val/test overfit gap) — no fitter offspring to breed.

Through-line across all four LLM runs: "merge, don't average" and its refinements
(routing, directed selection) are weak-base / suboptimal-default phenomena — they
help at 0.5B and are inert at 7B. Honest limitation kept in the writeup: the 7B
families are near-saturated, which caps the headroom; a harder unsaturated
benchmark is the fair next test.

Also folds in the two llm_moe local manifest/config files missed in 8da0dac.
+3 directed unit tests (130 green). Results in results/llm_directed{,_hpc}/
(parquet gitignored).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

results/llm_directed/README.md

@@ -0,0 +1,50 @@
+# llm_directed — directed sex in weight space: breed offspring + select on the verifier (E10, 0.5B)
+
+**Claim tested.** `llm_moe` left a clean gap: fusion can *compose* beyond the parents but the right
+blend is unknown and base-dependent, while pure routing is capped at the best parent. E10's answer is
+**directed sex** — biology can't preview offspring, an AI can: breed a *population* of recombinant
+offspring (the specialists merged at many different weights), score each against the verifier
+("reality") on a held-out validation split, and keep the fittest. Selection replaces betting on one
+a-priori blend. Two breeding objectives: best validation **overall**, and best validation
+**worst-family** (raw capability vs the balanced generalist).
+
+**Setup.** Base **Qwen2.5-0.5B-Instruct**, the three cached `llm_merge` specialists, **16 offspring**
+(Dirichlet-weighted merges, concentration 0.5, pinning candidate 0 = uniform soup for reference),
+scored on an **80-task/family validation** split, winners reported on a **fresh 100-task/family test**
+split (no selection-on-test leakage). Seed 1.
+
+### Results (test accuracy)
+| model | lists | strings | arith | overall | worst-family |
+|---|---|---|---|---|---|
+| best specialist (strings) | 0.08 | 1.00 | 0.80 | 0.63 | 0.08 |
+| merge_soup (uniform, candidate 0) | 0.26 | 0.74 | 0.91 | 0.64 | 0.26 |
+| **directed_overall** (bred for overall) | 0.17 | 0.99 | 0.92 | **0.69** | 0.17 |
+| **directed_balanced** (bred for worst-family) | 0.37 | 0.37 | 0.79 | 0.51 | **0.37** |
+
+### What holds, and the honest cost
+- **Offspring selection beats the single a-priori blend — on the objective you breed for.**
+  `directed_overall` reaches **0.69 overall > soup 0.64** (and > best parent 0.63); `directed_balanced`
+  reaches **0.37 worst-family > soup 0.26**. Searching the recombination-weight space and letting the
+  verifier choose beats committing to uniform averaging — the E10 "preview and keep the fittest" claim,
+  in real weights.
+- **Single-objective selection trades off the other axis (honest).** Breeding for *overall* on
+  lexically-imbalanced families finds a strings+arith-heavy blend that sacrifices the rare `lists`
+  skill (0.17, below soup's 0.26); breeding for *balance* lifts worst-family to 0.37 but costs overall.
+  Directed sex gives *control* over what you breed for — it does not hand you both for free.
+- **A global blend still trails per-input routing at a weak base.** At 0.5B the best directed *global*
+  merge (0.69 / 0.43-max) does not beat `llm_moe`'s per-input **routing** (0.74 / 0.43): when the base
+  is weak, adapting the recombination *per input* beats any one fixed blend, however well selected. So
+  directed sex over blends beats *averaging*, not *routing* — combining the two (route, then select
+  among routed+blended offspring) is the natural next operator.
+
+### Takeaway
+Directed sex — breed a population, select on the verifier — is confirmed in real LLM weights: it beats
+the single uniform soup on whichever objective it optimises, the distinctly-AI advantage (offspring
+preview + unbounded candidates) that biology lacks. The honest scope at 0.5B: selection buys one axis
+at the other's expense, and a single global blend can't yet beat per-input routing. Whether searching
+blends + selection can exceed even the *strong* 7B soup (which routing could not) is answered by
+**`results/llm_directed_hpc/`: it can't — directed ≈ soup (0.868 ≈ 0.873)** because the 7B soup already
+composes to the ceiling on these near-saturated families, leaving no fitter offspring to breed. So
+directed sex helps exactly when the default blend is *suboptimal* (0.5B), and is inert when it is
+already near-optimal (7B). **Falsifier (not triggered at 0.5B):** directed offspring ≤ uniform soup on
+their bred objective — instead each beat it.

results/llm_directed/manifest.json

@@ -0,0 +1,25 @@
+{
+  "experiment": "llm_directed",
+  "master_seed": 1,
+  "git_commit": "8da0dac00713fb9708804b4696a847a3767758d5",
+  "python": "3.14.5",
+  "libraries": {
+    "numpy": "2.5.0",
+    "scipy": "1.18.0",
+    "pandas": "3.0.3",
+    "pyarrow": "24.0.0",
+    "torch": "2.12.1",
+    "transformers": "5.13.0",
+    "peft": "0.19.1"
+  },
+  "rows": 35,
+  "results_sha256": "143b86833cb320c7c6c693f0b5394f381a94a69277cc31ba5e0977e5ea6ffd78",
+  "layer": "2",
+  "tier": "llm",
+  "base_model": "Qwen/Qwen2.5-0.5B-Instruct",
+  "directed": {
+    "n_candidates": 16,
+    "concentration": 0.5,
+    "n_val": 80
+  }
+}

results/llm_directed/resolved_config.yaml

@@ -0,0 +1,24 @@
+experiment: llm_directed
+seed: 1
+n_replicates: 1
+source_config:
+  experiment: llm_directed
+  kind: llm_directed
+  seed: 1
+  n_replicates: 1
+  base_model: Qwen/Qwen2.5-0.5B-Instruct
+  families:
+  - lists
+  - strings
+  - arith
+  n_train: 700
+  n_val: 80
+  n_test: 100
+  n_candidates: 16
+  concentration: 0.5
+  epochs: 3
+  lora:
+    r: 16
+    alpha: 32
+  output:
+    dir: results/llm_directed