hard benchmark: the 7B "fusion wins / no headroom" results were saturation artefacts

The easy task families saturated 7B (strings & arith at 1.00), so the earlier
7B nulls — moe: fusion 0.87 > union 0.84; directed ~= soup — could not separate
"refinements don't help at scale" from "tasks too easy at 7B". Adds a hard task
variant (hard: true in tasks.py: multi-step lists, Caesar ciphers / letter
transforms, multi-step & larger arithmetic; same family labels and answer
formats, threaded through make_tasks/train_specialist/runners; hard specialists
cache separately as spec_*_hard) and re-runs both experiments at 7B on Imperial
CX3 (one L40S, 24 min, unsaturated: arith ~0.48, strings 0.67, lists 0.34).

Both nulls flip back to the 0.5B ordering:
- Union beats fusion again: routing 0.500 > fusion 0.40 (soup 0.392 / ties
  0.400), the same 10-pt margin as 0.5B. Fusion dilutes the fragile strings
  specialist so hard (0.665 -> soup 0.300) that soup even trails the best single
  specialist (0.425); routing keeps it intact (0.670).
- Directed selection beats soup again: 0.492 > 0.392 (+10 pts), recovering most
  of routing's benefit from one deployable merged model (lifts strings to 0.630).

Correction to the earlier interpretation: the llm_moe_hpc "regime flip" and the
llm_directed_hpc "no headroom" null were driven by TASK SATURATION, not base
capability. The operative variable is headroom — "merge, don't average" (union >
fusion) and "directed sex" (selection > single blend) hold whenever there is room
to lose to dilution: a weak base (0.5B) OR hard tasks at a strong base (7B-hard).
Fusion only wins in the degenerate corner where easy tasks let a strong base
compose to the 1.00 ceiling. Vindicates E8's max > mean in real 7B weights once
saturation is controlled.

Default (easy) task behaviour is unchanged (hard defaults False). +1 hard-task
test (131 green). Excludes the 0.5B smoke bundle (a pipeline gate, not a
deliverable). Results in results/llm_{moe,directed}_hard_hpc/ (parquet gitignored).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

results/llm_directed_hard_hpc/README.md

@@ -0,0 +1,42 @@
+# llm_directed_hard_hpc — directed sex on HARD (unsaturated) tasks at 7B: offspring selection helps again
+
+**Claim tested.** `llm_directed_hpc` found directed selection ≈ soup at 7B (0.868 ≈ 0.873) and flagged
+the honest caveat that the easy families were *saturated* (no headroom for selection to exploit). This
+run re-runs directed sex on the **hard task variant**, where the uniform soup is far from the ceiling —
+the regime in which breeding + selection can actually improve on the default blend. One **L40S (46 GB)**
+GPU, Imperial CX3; reuses the `spec_*_hard` specialists trained by `llm_moe_hard_hpc`.
+
+**Setup.** Base **Qwen2.5-7B-Instruct**, `hard: true`, **24 offspring** (Dirichlet-weighted merges),
+100-task/family validation split (selection), 200-task/family test split (report). Seed 1.
+
+### Results (test accuracy — unsaturated)
+| model | lists | strings | arith | overall | worst-family |
+|---|---|---|---|---|---|
+| best specialist (strings) | 0.155 | 0.665 | 0.455 | 0.425 | 0.155 |
+| merge_soup (uniform, candidate 0) | 0.390 | 0.300 | 0.485 | 0.392 | 0.300 |
+| **directed_overall** | 0.380 | 0.630 | 0.465 | **0.492** | **0.380** |
+| **directed_balanced** | 0.380 | 0.630 | 0.465 | **0.492** | **0.380** |
+
+### The finding: the 7B "no headroom" null was also saturation
+- **Directed selection beats the uniform soup by +10 points (0.492 > 0.392).** On hard tasks there *is*
+  a better blend than uniform averaging, and breeding 24 offspring + selecting on the verifier finds it
+  — recovering most of the routing-level performance (0.492 vs routing 0.500) from a single deployable
+  merged model. The `llm_directed_hpc` null (directed ≈ soup at 7B) was a saturation artefact, exactly
+  as that run's honest caveat predicted.
+- **Selection repairs fusion's dilution.** The winning offspring lifts strings from soup's diluted
+  0.300 back to **0.630** (near the 0.665 specialist) while keeping lists' composition gain (0.380) —
+  i.e. it finds a blend that composes where composition helps and avoids diluting the fragile skill.
+  Both breeding objectives converged to the same winner (overall = balanced), which also improves
+  worst-family (0.380 > soup 0.300).
+- **Directed ≈ routing here.** A *single* searched-and-selected merged model (0.492) matches the
+  per-input router (0.500) on hard tasks — offspring selection buys most of routing's benefit without
+  needing a router at inference.
+
+### Takeaway
+On unsaturated tasks, directed sex (breed offspring + select on the verifier) beats the single a-priori
+soup at 7B, resolving the earlier null: it was task saturation, not scale, that made selection inert.
+Together with `llm_moe_hard_hpc` this completes the correction — **both** "merge, don't average" (union
+> fusion) and "directed sex" (selection > single blend) are **headroom** phenomena that hold at 7B once
+the tasks are hard enough to leave room, not weak-base-only effects. **Falsifier (not triggered):**
+directed offspring ≤ uniform soup — instead they beat it by 10 points. Provenance in `manifest.json`
+(`hard: true`, L40S, torch 2.12.1 / transformers 5.13.0 / peft 0.19.1).

results/llm_directed_hard_hpc/manifest.json

@@ -0,0 +1,26 @@
+{
+  "experiment": "llm_directed_hard_hpc",
+  "master_seed": 1,
+  "git_commit": null,
+  "python": "3.11.13",
+  "libraries": {
+    "numpy": "2.4.6",
+    "scipy": "1.17.1",
+    "pandas": "3.0.3",
+    "pyarrow": "24.0.0",
+    "torch": "2.12.1",
+    "transformers": "5.13.0",
+    "peft": "0.19.1"
+  },
+  "rows": 35,
+  "results_sha256": "a5e45785a8701c5e1cdcfc64bc1482dffe4e68f8c6978e9d07dc8f04d40fbab6",
+  "layer": "2",
+  "tier": "llm",
+  "base_model": "Qwen/Qwen2.5-7B-Instruct",
+  "hard": true,
+  "directed": {
+    "n_candidates": 24,
+    "concentration": 0.5,
+    "n_val": 100
+  }
+}

results/llm_directed_hard_hpc/resolved_config.yaml

@@ -0,0 +1,25 @@
+experiment: llm_directed_hard_hpc
+seed: 1
+n_replicates: 1
+source_config:
+  experiment: llm_directed_hard_hpc
+  kind: llm_directed
+  seed: 1
+  n_replicates: 1
+  base_model: Qwen/Qwen2.5-7B-Instruct
+  hard: true
+  families:
+  - lists
+  - strings
+  - arith
+  n_train: 800
+  n_val: 100
+  n_test: 200
+  n_candidates: 24
+  concentration: 0.5
+  epochs: 3
+  lora:
+    r: 16
+    alpha: 32
+  output:
+    dir: results/llm_directed_hard_hpc