hard benchmark: the 7B "fusion wins / no headroom" results were saturation artefacts

The easy task families saturated 7B (strings & arith at 1.00), so the earlier
7B nulls — moe: fusion 0.87 > union 0.84; directed ~= soup — could not separate
"refinements don't help at scale" from "tasks too easy at 7B". Adds a hard task
variant (hard: true in tasks.py: multi-step lists, Caesar ciphers / letter
transforms, multi-step & larger arithmetic; same family labels and answer
formats, threaded through make_tasks/train_specialist/runners; hard specialists
cache separately as spec_*_hard) and re-runs both experiments at 7B on Imperial
CX3 (one L40S, 24 min, unsaturated: arith ~0.48, strings 0.67, lists 0.34).

Both nulls flip back to the 0.5B ordering:
- Union beats fusion again: routing 0.500 > fusion 0.40 (soup 0.392 / ties
  0.400), the same 10-pt margin as 0.5B. Fusion dilutes the fragile strings
  specialist so hard (0.665 -> soup 0.300) that soup even trails the best single
  specialist (0.425); routing keeps it intact (0.670).
- Directed selection beats soup again: 0.492 > 0.392 (+10 pts), recovering most
  of routing's benefit from one deployable merged model (lifts strings to 0.630).

Correction to the earlier interpretation: the llm_moe_hpc "regime flip" and the
llm_directed_hpc "no headroom" null were driven by TASK SATURATION, not base
capability. The operative variable is headroom — "merge, don't average" (union >
fusion) and "directed sex" (selection > single blend) hold whenever there is room
to lose to dilution: a weak base (0.5B) OR hard tasks at a strong base (7B-hard).
Fusion only wins in the degenerate corner where easy tasks let a strong base
compose to the 1.00 ceiling. Vindicates E8's max > mean in real 7B weights once
saturation is controlled.

Default (easy) task behaviour is unchanged (hard defaults False). +1 hard-task
test (131 green). Excludes the 0.5B smoke bundle (a pipeline gate, not a
deliverable). Results in results/llm_{moe,directed}_hard_hpc/ (parquet gitignored).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

CLAUDE.md

@@ -97,6 +97,8 @@ E4's whole purpose is to isolate the effect of teacher **decorrelation ρ**, so
 
 **Finding (2026-07-05, `llm_directed` — directed sex in weights; refinements pay off only when the default blend is suboptimal).** E10 in real LLM weights (`src/llm/directed.py`, `kind: llm_directed`): breed a *population* of recombinant offspring (specialists merged at Dirichlet-sampled weights), score each against the verifier on a held-out **validation** split, keep the fittest — reported on a fresh **test** split (no selection-on-test leakage). Two breeding objectives (best-overall, best-worst-family). **The value scales with how far the uniform soup is from optimal, giving a clean regime split:** **0.5B** — soup dilutes, so directed selection beats it (`directed_overall` 0.69 > soup 0.64; `directed_balanced` worst-family 0.37 > soup 0.26), though single-objective selection trades off the other axis (breeding for overall tanks the rare `lists` to 0.17) and a *global* blend still trails per-input **routing** (0.74). **7B** — soup already *composes* to the ceiling on these near-saturated families (strings & arith at 1.00), so directed selection finds nothing better: **directed 0.868 ≈ soup 0.873** (marginally below, a val/test overfit gap). **Honest limitation:** the 7B families are near-saturated (2/3 at 1.00), which structurally caps the headroom — this run can't separate "directed sex doesn't help at scale" from "these tasks are too easy at 7B"; a *harder, unsaturated* benchmark is the fair next test. **Through-line across all four LLM runs:** "merge, don't average" and its refinements (routing, directed selection) are **weak-base / suboptimal-default** phenomena — they pay off at 0.5B (soup far from optimal) and are inert at 7B (soup near-optimal on saturated tasks). `configs/llm/{directed,directed_hpc}.yaml`, `plot_llm_directed.py`, `results/llm_directed{,_hpc}/`, `hpc/llm_directed.pbs`, +3 tests (130 green).
 
+**Finding (2026-07-05, HARD benchmark `llm_moe_hard_hpc` + `llm_directed_hard_hpc` — the 7B "fusion wins / no headroom" results were SATURATION artefacts; the law is HEADROOM, not base-size).** The easy families saturated 7B (strings & arith at 1.00), so the 7B nulls (moe: fusion 0.87 > union 0.84; directed ≈ soup) couldn't separate "refinements don't help at scale" from "tasks too easy." Built a **hard task variant** (`hard: true` in `tasks.py`: multi-step lists, Caesar ciphers / letter transforms, multi-step & larger arithmetic — same family labels & answer formats, threaded through `make_tasks`/`train_specialist`/runners; hard specialists cache separately as `spec_*_hard`) and re-ran both at 7B on Imperial CX3 (one L40S, 24 min, unsaturated: arith ≈0.48, strings 0.67, lists 0.34). **Both nulls flip back to the 0.5B ordering:** (1) **union beats fusion again — routing 0.500 > fusion 0.40** (soup 0.392/ties 0.400), the *same* 10-pt margin as 0.5B; fusion dilutes the fragile strings-specialist so hard (0.665 → soup 0.300) that soup even **trails the best single specialist** (0.425), while routing keeps it (0.670). (2) **directed selection beats soup again — 0.492 > 0.392** (+10 pts), recovering most of routing's benefit from one deployable merged model (lifts strings back to 0.630). **Correction to the earlier interpretation:** the `llm_moe_hpc` "regime flip" (fusion wins at 7B) and `llm_directed_hpc` "no headroom" were both driven by **task saturation, not base capability**. The operative variable is **headroom**: "merge, don't average" (union > fusion) and "directed sex" (selection > single blend) hold whenever there's room to lose to dilution — weak base (0.5B) *or* hard tasks at a strong base (7B-hard); fusion only wins in the degenerate corner where easy tasks let a strong base compose to the 1.00 ceiling. This vindicates E8's `max > mean` in real 7B weights once saturation is controlled. `configs/llm/{moe_hard,moe_hard_hpc,directed_hard_hpc}.yaml`, `hpc/llm_hard.pbs`, `results/llm_{moe,directed}_hard_hpc/`, +1 hard-task test (131 green).
+
 ## Build order (blueprint §7) — respect the gate
 
 1. Scaffold: repo layout (§5), container, pytest skeleton, config system, seeding utils. `make test` green.

configs/llm/directed_hard_hpc.yaml

@@ -0,0 +1,23 @@
+experiment: llm_directed_hard_hpc
+kind: llm_directed
+seed: 1
+n_replicates: 1
+
+# Directed sex on the HARD benchmark (L40S). On the easy families the 7B soup already composed to the
+# ceiling, so offspring selection had no headroom (directed ≈ soup). This run uses the harder task
+# variant (hard: true), where the uniform soup should be well below saturation — the regime in which
+# breeding + selection can actually improve on the default blend. Reuses the spec_*_hard specialists
+# trained by moe_hard_hpc if present. The fair test the easy 7B null could not provide.
+
+base_model: Qwen/Qwen2.5-7B-Instruct
+hard: true
+families: [lists, strings, arith]
+n_train: 800
+n_val: 100
+n_test: 200
+n_candidates: 24
+concentration: 0.5
+epochs: 3
+lora: {r: 16, alpha: 32}
+
+output: {dir: results/llm_directed_hard_hpc}

configs/llm/moe_hard.yaml

@@ -0,0 +1,21 @@
+experiment: llm_moe_hard
+kind: llm_moe
+seed: 1
+n_replicates: 1
+
+# Local 0.5B SMOKE for the hard benchmark: confirms the harder task variant (hard: true) runs
+# end-to-end — trains fresh hard specialists (cached as spec_*_hard), the verifier still scores them,
+# routing/fusion operators execute. 0.5B will score low on these harder tasks (that is expected; the
+# calibrated comparison is the 7B moe_hard_hpc run). Kept small for speed.
+
+base_model: Qwen/Qwen2.5-0.5B-Instruct
+hard: true
+families: [lists, strings, arith]
+n_train: 400
+n_test: 80
+n_route: 32
+epochs: 3
+lora: {r: 16, alpha: 32}
+operators: [soup, ties, moe_oracle, moe_learned, max_merge]
+
+output: {dir: results/llm_moe_hard}

configs/llm/moe_hard_hpc.yaml

@@ -0,0 +1,24 @@
+experiment: llm_moe_hard_hpc
+kind: llm_moe
+seed: 1
+n_replicates: 1
+
+# The HARD-benchmark firm-up (L40S). The easy families saturated 7B at 1.00 (strings/arith), so the
+# 7B moe/directed nulls could not distinguish "refinements don't help at scale" from "tasks too easy".
+# This run uses the HARDER task variant (hard: true) — longer lists + multi-step ops, Caesar ciphers /
+# letter transforms, multi-step & larger arithmetic — so 7B is NOT saturated and routing vs fusion has
+# real headroom to separate. Trains fresh HARD specialists (cached as spec_*_hard). Same operators as
+# moe_hpc; the question: does the 0.5B ordering (union > fusion) reappear at 7B once fusion can no
+# longer trivially compose to the ceiling?
+
+base_model: Qwen/Qwen2.5-7B-Instruct
+hard: true
+families: [lists, strings, arith]
+n_train: 800
+n_test: 200
+n_route: 48
+epochs: 3
+lora: {r: 16, alpha: 32}
+operators: [soup, ties, moe_oracle, moe_learned, max_merge]
+
+output: {dir: results/llm_moe_hard_hpc}

hpc/llm_hard.pbs

@@ -0,0 +1,26 @@
+#!/bin/bash
+# The HARD-benchmark firm-up on an L40S (46 GB): runs moe_hard_hpc then directed_hard_hpc in one job.
+# moe trains the hard specialists (cached spec_*_hard); directed reuses them — so training happens once
+# and both experiments share it. Harder tasks (multi-step lists, Caesar ciphers, multi-step arith) keep
+# 7B off saturation, giving routing/fusion/selection real headroom to separate — the fair test the easy
+# 7B runs could not provide. Same env as the other LLM jobs (see hpc/README.md).
+#   submit:  qsub hpc/llm_hard.pbs      status: qstat -u $USER
+#PBS -l select=1:ncpus=8:mem=64gb:ngpus=1:gpu_type=L40S
+#PBS -l walltime=01:00:00
+#PBS -N lam_llm_hard
+
+cd "$PBS_O_WORKDIR"
+export HF_HOME="$EPHEMERAL/hf_cache"
+export TOKENIZERS_PARALLELISM=false
+export UV_CACHE_DIR="$EPHEMERAL/uvcache"
+
+source .venv/bin/activate
+nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv,noheader
+
+python -m llm.experiment configs/llm/moe_hard_hpc.yaml        # trains spec_*_hard + fusion/union
+python -m llm.experiment configs/llm/directed_hard_hpc.yaml   # reuses spec_*_hard + offspring select
+
+# results/{llm_moe_hard_hpc,llm_directed_hard_hpc}/ written in-place (parquet gitignored). Sync back:
+#   rsync -avz hpc:'…/results/llm_moe_hard_hpc/' results/llm_moe_hard_hpc/
+#   rsync -avz hpc:'…/results/llm_directed_hard_hpc/' results/llm_directed_hard_hpc/
+echo "done: $(date)"

results/llm_directed_hard_hpc/README.md

@@ -0,0 +1,42 @@
+# llm_directed_hard_hpc — directed sex on HARD (unsaturated) tasks at 7B: offspring selection helps again
+
+**Claim tested.** `llm_directed_hpc` found directed selection ≈ soup at 7B (0.868 ≈ 0.873) and flagged
+the honest caveat that the easy families were *saturated* (no headroom for selection to exploit). This
+run re-runs directed sex on the **hard task variant**, where the uniform soup is far from the ceiling —
+the regime in which breeding + selection can actually improve on the default blend. One **L40S (46 GB)**
+GPU, Imperial CX3; reuses the `spec_*_hard` specialists trained by `llm_moe_hard_hpc`.
+
+**Setup.** Base **Qwen2.5-7B-Instruct**, `hard: true`, **24 offspring** (Dirichlet-weighted merges),
+100-task/family validation split (selection), 200-task/family test split (report). Seed 1.
+
+### Results (test accuracy — unsaturated)
+| model | lists | strings | arith | overall | worst-family |
+|---|---|---|---|---|---|
+| best specialist (strings) | 0.155 | 0.665 | 0.455 | 0.425 | 0.155 |
+| merge_soup (uniform, candidate 0) | 0.390 | 0.300 | 0.485 | 0.392 | 0.300 |
+| **directed_overall** | 0.380 | 0.630 | 0.465 | **0.492** | **0.380** |
+| **directed_balanced** | 0.380 | 0.630 | 0.465 | **0.492** | **0.380** |
+
+### The finding: the 7B "no headroom" null was also saturation
+- **Directed selection beats the uniform soup by +10 points (0.492 > 0.392).** On hard tasks there *is*
+  a better blend than uniform averaging, and breeding 24 offspring + selecting on the verifier finds it
+  — recovering most of the routing-level performance (0.492 vs routing 0.500) from a single deployable
+  merged model. The `llm_directed_hpc` null (directed ≈ soup at 7B) was a saturation artefact, exactly
+  as that run's honest caveat predicted.
+- **Selection repairs fusion's dilution.** The winning offspring lifts strings from soup's diluted
+  0.300 back to **0.630** (near the 0.665 specialist) while keeping lists' composition gain (0.380) —
+  i.e. it finds a blend that composes where composition helps and avoids diluting the fragile skill.
+  Both breeding objectives converged to the same winner (overall = balanced), which also improves
+  worst-family (0.380 > soup 0.300).
+- **Directed ≈ routing here.** A *single* searched-and-selected merged model (0.492) matches the
+  per-input router (0.500) on hard tasks — offspring selection buys most of routing's benefit without
+  needing a router at inference.
+
+### Takeaway
+On unsaturated tasks, directed sex (breed offspring + select on the verifier) beats the single a-priori
+soup at 7B, resolving the earlier null: it was task saturation, not scale, that made selection inert.
+Together with `llm_moe_hard_hpc` this completes the correction — **both** "merge, don't average" (union
+> fusion) and "directed sex" (selection > single blend) are **headroom** phenomena that hold at 7B once
+the tasks are hard enough to leave room, not weak-base-only effects. **Falsifier (not triggered):**
+directed offspring ≤ uniform soup — instead they beat it by 10 points. Provenance in `manifest.json`
+(`hard: true`, L40S, torch 2.12.1 / transformers 5.13.0 / peft 0.19.1).

results/llm_directed_hard_hpc/manifest.json

@@ -0,0 +1,26 @@
+{
+  "experiment": "llm_directed_hard_hpc",
+  "master_seed": 1,
+  "git_commit": null,
+  "python": "3.11.13",
+  "libraries": {
+    "numpy": "2.4.6",
+    "scipy": "1.17.1",
+    "pandas": "3.0.3",
+    "pyarrow": "24.0.0",
+    "torch": "2.12.1",
+    "transformers": "5.13.0",
+    "peft": "0.19.1"
+  },
+  "rows": 35,
+  "results_sha256": "a5e45785a8701c5e1cdcfc64bc1482dffe4e68f8c6978e9d07dc8f04d40fbab6",
+  "layer": "2",
+  "tier": "llm",
+  "base_model": "Qwen/Qwen2.5-7B-Instruct",
+  "hard": true,
+  "directed": {
+    "n_candidates": 24,
+    "concentration": 0.5,
+    "n_val": 100
+  }
+}

results/llm_directed_hard_hpc/resolved_config.yaml

@@ -0,0 +1,25 @@
+experiment: llm_directed_hard_hpc
+seed: 1
+n_replicates: 1
+source_config:
+  experiment: llm_directed_hard_hpc
+  kind: llm_directed
+  seed: 1
+  n_replicates: 1
+  base_model: Qwen/Qwen2.5-7B-Instruct
+  hard: true
+  families:
+  - lists
+  - strings
+  - arith
+  n_train: 800
+  n_val: 100
+  n_test: 200
+  n_candidates: 24
+  concentration: 0.5
+  epochs: 3
+  lora:
+    r: 16
+    alpha: 32
+  output:
+    dir: results/llm_directed_hard_hpc

results/llm_moe_hard_hpc/README.md

@@ -0,0 +1,45 @@
+# llm_moe_hard_hpc — union vs fusion on HARD (unsaturated) tasks at 7B: the flip was a saturation artefact
+
+**Claim tested.** `llm_moe_hpc` found fusion beating union at 7B (soup 0.87 > routing 0.84) and read it
+as "a capable base lets averaging compose." But the easy families were *saturated* (strings & arith at
+1.00), so that flip could have been an artefact of no headroom rather than base capability. This run
+re-runs the union-vs-fusion contrast on the **hard task variant** (multi-step lists, Caesar ciphers /
+letter transforms, multi-step & larger arithmetic), where 7B is **not** saturated. One **L40S (46 GB)**
+GPU, Imperial CX3.
+
+**Setup.** Base **Qwen2.5-7B-Instruct**, `hard: true`, fresh hard specialists (cached `spec_*_hard`),
+200 test tasks/family, seed 1. Same five operators as `llm_moe_hpc`.
+
+### Results (accuracy — nothing saturated; arith ≈ 0.48, strings 0.67, lists 0.34)
+| operator | lists | strings | arith | overall | worst-family | router |
+|---|---|---|---|---|---|---|
+| best specialist (strings) | 0.155 | 0.665 | 0.455 | 0.425 | 0.155 | — |
+| fuse: soup | 0.390 | 0.300 | 0.485 | 0.392 | 0.300 | — |
+| fuse: ties | 0.385 | 0.330 | 0.485 | 0.400 | 0.330 | — |
+| **route: oracle** | 0.335 | **0.670** | 0.495 | **0.500** | 0.335 | 1.00 |
+| **route: learned** | 0.335 | 0.670 | 0.495 | **0.500** | 0.335 | 1.00 |
+| max-merge | 0.215 | 0.195 | 0.480 | 0.297 | 0.195 | — |
+
+### The finding: the 7B "fusion wins" flip was saturation, not capability
+- **Union beats fusion again — decisively.** Routing **0.500 > fusion 0.40 (soup 0.392 / ties 0.400)**,
+  a 10-point margin, the *same* ordering as 0.5B. The easy-task flip (fusion > union at 7B) does **not**
+  survive once the tasks are hard enough to leave headroom.
+- **Fusion dilutes so badly it loses to the best single specialist.** On hard tasks the strings skill
+  (Caesar ciphers etc.) is fragile: the strings-specialist scores **0.665**, but soup washes it out to
+  **0.300** — so soup (0.392 overall) even trails the best *single* specialist (0.425). Routing keeps
+  the specialist intact (strings 0.670) and wins. Dilution is severe exactly when the specialist's
+  contribution is hard-won.
+- **So "merge, don't average" is a HEADROOM law, not a base-size law.** Union > fusion whenever there
+  is room to lose to dilution — a weak base (0.5B) *or* hard tasks at a strong base (7B-hard). Fusion
+  only wins in the degenerate corner where the tasks are so easy the strong base composes to the 1.00
+  ceiling (7B-easy). This corrects the `llm_moe_hpc` interpretation: base capability was a confound;
+  the operative variable is task headroom.
+- **Riders unchanged.** Learned router still perfect (1.00, lexical families); `max_merge` still the
+  weakest union (0.297, not input-adaptive).
+
+### Takeaway
+On genuinely hard, unsaturated tasks, the union operator (routing) beats fusion at 7B by the same
+margin it does at 0.5B — E8's `max > mean` in real weights, robust across scale once you control for
+saturation. The `llm_moe_hpc` flip is re-read as a saturation artefact. **Falsifier (not triggered):**
+fusion matching/beating routing on hard tasks — instead fusion diluted below even the best specialist.
+Provenance in `manifest.json` (`hard: true`, L40S, torch 2.12.1 / transformers 5.13.0 / peft 0.19.1).

results/llm_moe_hard_hpc/manifest.json

@@ -0,0 +1,28 @@
+{
+  "experiment": "llm_moe_hard_hpc",
+  "master_seed": 1,
+  "git_commit": null,
+  "python": "3.11.13",
+  "libraries": {
+    "numpy": "2.4.6",
+    "scipy": "1.17.1",
+    "pandas": "3.0.3",
+    "pyarrow": "24.0.0",
+    "torch": "2.12.1",
+    "transformers": "5.13.0",
+    "peft": "0.19.1"
+  },
+  "rows": 47,
+  "results_sha256": "a4e7c37e7cd900b56aa24ffc827e38e9a9d5cbb44dcff8d83cf010e992ed5c8a",
+  "layer": "2",
+  "tier": "llm",
+  "base_model": "Qwen/Qwen2.5-7B-Instruct",
+  "hard": true,
+  "operators": [
+    "soup",
+    "ties",
+    "moe_oracle",
+    "moe_learned",
+    "max_merge"
+  ]
+}

results/llm_moe_hard_hpc/resolved_config.yaml

@@ -0,0 +1,29 @@
+experiment: llm_moe_hard_hpc
+seed: 1
+n_replicates: 1
+source_config:
+  experiment: llm_moe_hard_hpc
+  kind: llm_moe
+  seed: 1
+  n_replicates: 1
+  base_model: Qwen/Qwen2.5-7B-Instruct
+  hard: true
+  families:
+  - lists
+  - strings
+  - arith
+  n_train: 800
+  n_test: 200
+  n_route: 48
+  epochs: 3
+  lora:
+    r: 16
+    alpha: 32
+  operators:
+  - soup
+  - ties
+  - moe_oracle
+  - moe_learned
+  - max_merge
+  output:
+    dir: results/llm_moe_hard_hpc

src/llm/experiment.py

@@ -51,9 +51,11 @@ def run_merge_experiment(cfg: dict) -> pd.DataFrame:
     lora = cfg.get("lora", {})
     merges = cfg.get("merges", ["soup", "ties"])
     seed = int(cfg["seed"])
+    hard = bool(cfg.get("hard", False))
+    suffix = "_hard" if hard else ""                        # hard specialists cache separately
     adapters_root = Path(cfg.get("adapters_dir", "models/llm"))
 
-    test = sum([make_tasks(f, n_test, seed=1000 + i) for i, f in enumerate(fams)], [])
+    test = sum([make_tasks(f, n_test, seed=1000 + i, hard=hard) for i, f in enumerate(fams)], [])
     rows: list[dict] = []
 
     # base
@@ -64,8 +66,8 @@ def run_merge_experiment(cfg: dict) -> pd.DataFrame:
     # one specialist per family
     dirs = []
     for i, f in enumerate(fams):
-        d = str(adapters_root / f"spec_{f}")
-        train_specialist(base, f, d, n_train=n_train, epochs=epochs, seed=seed + i,
+        d = str(adapters_root / f"spec_{f}{suffix}")
+        train_specialist(base, f, d, n_train=n_train, epochs=epochs, seed=seed + i, hard=hard,
                          r=int(lora.get("r", 16)), alpha=int(lora.get("alpha", 32)))
         dirs.append(d)
         m, tok = load_model(base, adapter_dir=d)
@@ -96,12 +98,14 @@ def _load_or_train_specialists(cfg: dict, base: str, fams: list[str], name: str,
     epochs = int(cfg.get("epochs", 3))
     lora = cfg.get("lora", {})
     seed = int(cfg["seed"])
+    hard = bool(cfg.get("hard", False))
+    suffix = "_hard" if hard else ""                        # hard specialists cache separately
     adapters_root = Path(cfg.get("adapters_dir", "models/llm"))
     dirs: list[str] = []
     for i, f in enumerate(fams):
-        d = str(adapters_root / f"spec_{f}")
+        d = str(adapters_root / f"spec_{f}{suffix}")
         if not (Path(d) / "adapter_config.json").exists():   # reuse across llm_merge / llm_moe runs
-            train_specialist(base, f, d, n_train=n_train, epochs=epochs, seed=seed + i,
+            train_specialist(base, f, d, n_train=n_train, epochs=epochs, seed=seed + i, hard=hard,
                              r=int(lora.get("r", 16)), alpha=int(lora.get("alpha", 32)))
         dirs.append(d)
         m, tok = load_model(base, adapter_dir=d)
@@ -113,7 +117,8 @@ def _load_or_train_specialists(cfg: dict, base: str, fams: list[str], name: str,
 def test_of(cfg: dict, fams: list[str]) -> list:
     """The held-out mixed test set (one seed offset per family), shared by both LLM runners."""
     n_test = int(cfg.get("n_test", 100))
-    return sum([make_tasks(f, n_test, seed=1000 + i) for i, f in enumerate(fams)], [])
+    hard = bool(cfg.get("hard", False))
+    return sum([make_tasks(f, n_test, seed=1000 + i, hard=hard) for i, f in enumerate(fams)], [])
 
 
 def run_moe_experiment(cfg: dict) -> pd.DataFrame:
@@ -160,7 +165,8 @@ def run_moe_experiment(cfg: dict) -> pd.DataFrame:
     if "moe_oracle" in ops:
         routes["moe_oracle"] = true_idx
     if "moe_learned" in ops:
-        route_train = sum([make_tasks(f, n_route, seed=2000 + i) for i, f in enumerate(fams)], [])
+        route_train = sum([make_tasks(f, n_route, seed=2000 + i, hard=bool(cfg.get("hard", False)))
+                           for i, f in enumerate(fams)], [])
         tr_emb = embed_prompts(model, tok, [t.prompt for t in route_train])
         te_emb = embed_prompts(model, tok, prompts)
         tr_fam = np.array([t.family for t in route_train])
@@ -207,7 +213,8 @@ def run_directed_experiment(cfg: dict) -> pd.DataFrame:
     rows: list[dict] = []
 
     test = test_of(cfg, fams)
-    val = sum([make_tasks(f, n_val, seed=3000 + i) for i, f in enumerate(fams)], [])
+    val = sum([make_tasks(f, n_val, seed=3000 + i, hard=bool(cfg.get("hard", False)))
+               for i, f in enumerate(fams)], [])
 
     # base + specialists (parents), scored on test
     m, tok = load_model(base)
@@ -265,7 +272,8 @@ def run_and_save(config_path: str | Path) -> Path:
     if kind not in _RUNNERS:
         raise ValueError(f"unknown LLM experiment kind {kind!r} (expected one of {list(_RUNNERS)})")
     df = _RUNNERS[kind](cfg)
-    extra = {"layer": "2", "tier": "llm", "base_model": cfg["base_model"]}
+    extra = {"layer": "2", "tier": "llm", "base_model": cfg["base_model"],
+             "hard": bool(cfg.get("hard", False))}
     if kind == "llm_moe":
         extra["operators"] = list(cfg.get("operators", []))
     if kind == "llm_directed":

src/llm/specialise.py

@@ -29,9 +29,12 @@ def _encode(tok, task, device):
 
 def train_specialist(base_name: str, family: str, out_dir: str, *, n_train: int = 600,
                      epochs: int = 3, lr: float = 2e-4, batch_size: int = 8, r: int = 16,
-                     alpha: int = 32, seed: int = 0, device: str = "cuda") -> str:
+                     alpha: int = 32, seed: int = 0, device: str = "cuda", hard: bool = False) -> str:
     """Fine-tune a LoRA specialist on ``family`` and save the adapter to ``out_dir``.
 
+    Args:
+        hard (bool): train on the harder task variant (must match the eval difficulty).
+
     Returns the adapter directory path.
     """
     import torch
@@ -49,7 +52,7 @@ def train_specialist(base_name: str, family: str, out_dir: str, *, n_train: int
     model = get_peft_model(model, lora)
     model.train()
 
-    tasks = make_tasks(family, n_train, seed=seed)
+    tasks = make_tasks(family, n_train, seed=seed, hard=hard)
     encoded = [_encode(tok, t, device) for t in tasks]
     opt = torch.optim.AdamW([p for p in model.parameters() if p.requires_grad], lr=lr)
     rng = np.random.default_rng(seed)

src/llm/tasks.py

@@ -10,6 +10,11 @@ decorrelated from the others (the precondition for recombination to have anythin
 Each task is a natural-language instruction + input with a single canonical answer; the verifier
 normalises the model's output and checks exact equality. This is the LLM analogue of the synthetic
 sandbox's exact oracle — correctness is the "reality that can say no" (grounding / fitness).
+
+A ``hard=True`` variant of each family (longer inputs + multi-step / cipher operations) exists so a
+*capable* base (7B) is **not saturated** at 1.00 — the regime where recombination refinements
+(routing, offspring selection) have headroom to matter. Same family labels and answer formats (int
+list / integer / lowercase word), so the verifier and the whole experiment machinery are unchanged.
 """
 
 from __future__ import annotations
@@ -37,7 +42,8 @@ def _fmt_list(xs) -> str:
     return "[" + ", ".join(str(int(x)) for x in xs) + "]"
 
 
-def _list_task(rng) -> Task:
+def _list_task(rng, hard: bool = False) -> Task:
+    if not hard:
         n = int(rng.integers(6, 9))                        # longer lists (harder)
         xs = rng.integers(0, 20, size=n).tolist()
         op = rng.choice(["sort ascending", "sort descending", "reverse",
@@ -64,9 +70,44 @@ def _list_task(rng) -> Task:
         want = "the resulting list" if ans.startswith("[") else "the single number"
         return Task("lists", f"Given the list {_fmt_list(xs)}, compute its {op}. "
                     f"Output only {want} and nothing else.", ans)
+    # hard: longer lists + MULTI-STEP operations (a capable base is not saturated here)
+    n = int(rng.integers(10, 15))
+    xs = rng.integers(0, 30, size=n).tolist()
+    op = rng.choice(["sort ascending then the element at index 2 (0-based)",
+                     "sum of the elements at even indices (0-based)",
+                     "the third smallest value",
+                     "the prefix sums (running totals) as a list",
+                     "sort descending then the sum of the first three values",
+                     "the product of the two smallest values",
+                     "the values strictly greater than 15, preserving order"])
+    if op.startswith("sort ascending then the element"):
+        ans = str(sorted(xs)[2])
+    elif op.startswith("sum of the elements at even"):
+        ans = str(sum(xs[::2]))
+    elif op == "the third smallest value":
+        ans = str(sorted(xs)[2])
+    elif op.startswith("the prefix sums"):
+        acc, out = 0, []
+        for x in xs:
+            acc += x; out.append(acc)
+        ans = _fmt_list(out)
+    elif op.startswith("sort descending then the sum"):
+        ans = str(sum(sorted(xs, reverse=True)[:3]))
+    elif op.startswith("the product of the two smallest"):
+        s = sorted(xs); ans = str(s[0] * s[1])
+    else:
+        ans = _fmt_list([x for x in xs if x > 15])
+    want = "the resulting list" if ans.startswith("[") else "the single number"
+    return Task("lists", f"Given the list {_fmt_list(xs)}, compute {op}. "
+                f"Output only {want} and nothing else.", ans)
 
 
-def _string_task(rng) -> Task:
+def _caesar(w: str, k: int) -> str:
+    return "".join(chr((ord(c) - 97 + k) % 26 + 97) for c in w)
+
+
+def _string_task(rng, hard: bool = False) -> Task:
+    if not hard:
         w = str(rng.choice(_WORDS))
         op = rng.choice(["reverse", "uppercase", "count of the letter 'a'", "count of vowels",
                          "remove all vowels", "first three characters", "last three characters"])
@@ -86,9 +127,31 @@ def _string_task(rng) -> Task:
             ans = w[-3:]
         return Task("strings", f"Given the word \"{w}\", compute its {op}. "
                     f"Output only the answer and nothing else.", ans)
+    # hard: longer words (two concatenated) + cipher / multi-step transforms
+    w = str(rng.choice(_WORDS)) + str(rng.choice(_WORDS))
+    op = rng.choice(["caesar", "sortletters", "devowel", "mostfreq", "distinct", "revdevowel"])
+    if op == "caesar":
+        ans = _caesar(w, 3)
+        instr = "shift each letter forward by 3 in the alphabet, wrapping z to a (a Caesar cipher)"
+    elif op == "sortletters":
+        ans = "".join(sorted(w)); instr = "its letters sorted in alphabetical order"
+    elif op == "devowel":
+        ans = "".join("x" if c in "aeiou" else c for c in w)
+        instr = "the word with every vowel replaced by the letter x"
+    elif op == "mostfreq":
+        ans = min(set(w), key=lambda c: (-w.count(c), c))
+        instr = "the letter that occurs most often (on a tie, the one earliest in the alphabet)"
+    elif op == "distinct":
+        ans = str(len(set(w))); instr = "the number of distinct letters"
+    else:
+        ans = "".join(c for c in w[::-1] if c not in "aeiou")
+        instr = "the word reversed and then with all vowels removed"
+    return Task("strings", f"Given the word \"{w}\", compute {instr}. "
+                f"Output only the answer and nothing else.", ans)
 
 
-def _arith_task(rng) -> Task:
+def _arith_task(rng, hard: bool = False) -> Task:
+    if not hard:
         kind = rng.choice(["mul", "sub", "mul2", "seq"])   # multiplication-heavy (harder)
         if kind == "mul":
             a, b = rng.integers(11, 40, size=2).tolist(); ans = str(a * b)
@@ -104,15 +167,48 @@ def _arith_task(rng) -> Task:
             seq = [start + i * step for i in range(4)]; ans = str(seq[-1] + step)
             q = f"What comes next in the sequence {', '.join(map(str, seq))}?"
         return Task("arith", q + " Output only the number and nothing else.", ans)
+    # hard: multi-step / larger operands (a capable base is not saturated here)
+    kind = rng.choice(["madd", "mod", "diffsq", "bigmul", "rangesum", "gcd"])
+    if kind == "madd":
+        a, b = rng.integers(11, 40, size=2).tolist(); c = int(rng.integers(10, 100))
+        ans = str(a * b + c); q = f"What is {a} * {b} + {c}?"
+    elif kind == "mod":
+        a = int(rng.integers(100, 1000)); b = int(rng.integers(3, 20))
+        ans = str(a % b); q = f"What is {a} mod {b} (the remainder of {a} divided by {b})?"
+    elif kind == "diffsq":
+        a, b = sorted(rng.integers(10, 30, size=2).tolist(), reverse=True)
+        ans = str(a * a - b * b); q = f"What is {a}^2 - {b}^2?"
+    elif kind == "bigmul":
+        a = int(rng.integers(100, 1000)); b = int(rng.integers(2, 10))
+        ans = str(a * b); q = f"What is {a} * {b}?"
+    elif kind == "rangesum":
+        a = int(rng.integers(1, 20)); b = int(rng.integers(25, 55))
+        ans = str(sum(range(a, b + 1)))
+        q = f"What is the sum of all integers from {a} to {b} inclusive?"
+    else:
+        import math
+        a, b = rng.integers(6, 60, size=2).tolist()
+        ans = str(math.gcd(a, b)); q = f"What is the greatest common divisor of {a} and {b}?"
+    return Task("arith", q + " Output only the number and nothing else.", ans)
 
 
 _GEN = {"lists": _list_task, "strings": _string_task, "arith": _arith_task}
 
 
-def make_tasks(family: str, n: int, seed: int) -> list[Task]:
-    """Generate ``n`` unique-ish tasks for a family (deterministic in ``seed``)."""
+def make_tasks(family: str, n: int, seed: int, hard: bool = False) -> list[Task]:
+    """Generate ``n`` unique-ish tasks for a family (deterministic in ``seed``).
+
+    Args:
+        family (str): one of :data:`FAMILIES`.
+        n (int): number of tasks.
+        seed (int): RNG seed (deterministic output).
+        hard (bool): use the harder multi-step / cipher variant (same family label + answer format).
+
+    Returns:
+        list[Task]: the generated tasks.
+    """
     rng = np.random.default_rng(seed)
-    return [_GEN[family](rng) for _ in range(n)]
+    return [_GEN[family](rng, hard) for _ in range(n)]
 
 
 def _normalise(s: str) -> str:

tasks/todo.md

@@ -380,3 +380,16 @@ C3 vertical claim deferred.*
   0.5B, nil at 7B. Honest limit: 7B families near-saturated; a harder benchmark is the fair next test.
 - `configs/llm/{directed,directed_hpc}.yaml`, `figures/plot_llm_directed.py`, READMEs, `hpc/llm_directed.pbs`,
   Makefile `llm` target, +3 tests (130 green). `results/llm_directed{,_hpc}/`.
+
+**2026-07-05 — HARD benchmark: the 7B "fusion wins / no headroom" nulls were SATURATION artefacts.** ✅
+- Easy families saturated 7B (strings/arith 1.00), confounding the moe/directed 7B nulls. Built a hard
+  task variant (`hard: true`: multi-step lists, Caesar ciphers, multi-step/larger arith) threaded through
+  make_tasks/train_specialist/runners; hard specialists cache as `spec_*_hard`. Ran both at 7B on CX3
+  (one L40S, 24 min, unsaturated).
+- **Both nulls flip back to the 0.5B ordering:** union/routing 0.500 > fusion 0.40 (soup dilutes the
+  strings specialist 0.665→0.300, below even the best single specialist 0.425); directed selection 0.492
+  > soup 0.392 (+10 pts). **The operative variable is HEADROOM, not base-size** — "merge, don't average"
+  and "directed sex" hold whenever there's room to lose to dilution (weak base OR hard tasks); fusion only
+  wins where easy tasks let a strong base compose to ceiling. Vindicates E8 max>mean at 7B.
+- `configs/llm/{moe_hard,moe_hard_hpc,directed_hard_hpc}.yaml`, `hpc/llm_hard.pbs`, hard READMEs+figures,
+  +1 test (131 green). `results/llm_{moe,directed}_hard_hpc/`.

tests/test_llm.py

@@ -25,6 +25,21 @@ def test_make_tasks_wellformed_and_deterministic():
     assert [t.answer for t in a] == [t.answer for t in b]      # deterministic in the seed
 
 
+def test_hard_tasks_wellformed_verifiable_and_distinct():
+    # The hard variant must stay well-formed, self-verifying (canonical answer passes its own verifier),
+    # and genuinely different from the easy variant (harder content, same family labels + answer format).
+    for fam in FAMILIES:
+        hard = make_tasks(fam, 30, seed=7, hard=True)
+        assert len(hard) == 30 and all(t.family == fam for t in hard)
+        assert all(t.prompt and t.answer for t in hard)
+        assert all(verify(t.answer, t) for t in hard)              # canonical answers verify
+        easy = make_tasks(fam, 30, seed=7, hard=False)
+        assert [t.prompt for t in hard] != [t.prompt for t in easy]  # hard != easy
+    # a Caesar-cipher answer is a real transform of the input (not the identity)
+    caesars = [t for t in make_tasks("strings", 60, seed=2, hard=True) if "Caesar" in t.prompt]
+    assert caesars and any(t.answer not in t.prompt for t in caesars)
+
+
 def test_verifier_accepts_correct_including_verbose():
     tasks = make_tasks("lists", 40, seed=1) + make_tasks("arith", 40, seed=2)
     assert all(verify(t.answer, t) for t in tasks)              # the canonical answer verifies