diff --git a/CLAUDE.md b/CLAUDE.md index b950df4..f60976b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -97,6 +97,8 @@ E4's whole purpose is to isolate the effect of teacher **decorrelation ρ**, so **Finding (2026-07-05, `llm_directed` — directed sex in weights; refinements pay off only when the default blend is suboptimal).** E10 in real LLM weights (`src/llm/directed.py`, `kind: llm_directed`): breed a *population* of recombinant offspring (specialists merged at Dirichlet-sampled weights), score each against the verifier on a held-out **validation** split, keep the fittest — reported on a fresh **test** split (no selection-on-test leakage). Two breeding objectives (best-overall, best-worst-family). **The value scales with how far the uniform soup is from optimal, giving a clean regime split:** **0.5B** — soup dilutes, so directed selection beats it (`directed_overall` 0.69 > soup 0.64; `directed_balanced` worst-family 0.37 > soup 0.26), though single-objective selection trades off the other axis (breeding for overall tanks the rare `lists` to 0.17) and a *global* blend still trails per-input **routing** (0.74). **7B** — soup already *composes* to the ceiling on these near-saturated families (strings & arith at 1.00), so directed selection finds nothing better: **directed 0.868 ≈ soup 0.873** (marginally below, a val/test overfit gap). **Honest limitation:** the 7B families are near-saturated (2/3 at 1.00), which structurally caps the headroom — this run can't separate "directed sex doesn't help at scale" from "these tasks are too easy at 7B"; a *harder, unsaturated* benchmark is the fair next test. **Through-line across all four LLM runs:** "merge, don't average" and its refinements (routing, directed selection) are **weak-base / suboptimal-default** phenomena — they pay off at 0.5B (soup far from optimal) and are inert at 7B (soup near-optimal on saturated tasks). `configs/llm/{directed,directed_hpc}.yaml`, `plot_llm_directed.py`, `results/llm_directed{,_hpc}/`, `hpc/llm_directed.pbs`, +3 tests (130 green). +**Finding (2026-07-05, HARD benchmark `llm_moe_hard_hpc` + `llm_directed_hard_hpc` — the 7B "fusion wins / no headroom" results were SATURATION artefacts; the law is HEADROOM, not base-size).** The easy families saturated 7B (strings & arith at 1.00), so the 7B nulls (moe: fusion 0.87 > union 0.84; directed ≈ soup) couldn't separate "refinements don't help at scale" from "tasks too easy." Built a **hard task variant** (`hard: true` in `tasks.py`: multi-step lists, Caesar ciphers / letter transforms, multi-step & larger arithmetic — same family labels & answer formats, threaded through `make_tasks`/`train_specialist`/runners; hard specialists cache separately as `spec_*_hard`) and re-ran both at 7B on Imperial CX3 (one L40S, 24 min, unsaturated: arith ≈0.48, strings 0.67, lists 0.34). **Both nulls flip back to the 0.5B ordering:** (1) **union beats fusion again — routing 0.500 > fusion 0.40** (soup 0.392/ties 0.400), the *same* 10-pt margin as 0.5B; fusion dilutes the fragile strings-specialist so hard (0.665 → soup 0.300) that soup even **trails the best single specialist** (0.425), while routing keeps it (0.670). (2) **directed selection beats soup again — 0.492 > 0.392** (+10 pts), recovering most of routing's benefit from one deployable merged model (lifts strings back to 0.630). **Correction to the earlier interpretation:** the `llm_moe_hpc` "regime flip" (fusion wins at 7B) and `llm_directed_hpc` "no headroom" were both driven by **task saturation, not base capability**. The operative variable is **headroom**: "merge, don't average" (union > fusion) and "directed sex" (selection > single blend) hold whenever there's room to lose to dilution — weak base (0.5B) *or* hard tasks at a strong base (7B-hard); fusion only wins in the degenerate corner where easy tasks let a strong base compose to the 1.00 ceiling. This vindicates E8's `max > mean` in real 7B weights once saturation is controlled. `configs/llm/{moe_hard,moe_hard_hpc,directed_hard_hpc}.yaml`, `hpc/llm_hard.pbs`, `results/llm_{moe,directed}_hard_hpc/`, +1 hard-task test (131 green). + ## Build order (blueprint §7) — respect the gate 1. Scaffold: repo layout (§5), container, pytest skeleton, config system, seeding utils. `make test` green. diff --git a/configs/llm/directed_hard_hpc.yaml b/configs/llm/directed_hard_hpc.yaml new file mode 100644 index 0000000..b384886 --- /dev/null +++ b/configs/llm/directed_hard_hpc.yaml @@ -0,0 +1,23 @@ +experiment: llm_directed_hard_hpc +kind: llm_directed +seed: 1 +n_replicates: 1 + +# Directed sex on the HARD benchmark (L40S). On the easy families the 7B soup already composed to the +# ceiling, so offspring selection had no headroom (directed ≈ soup). This run uses the harder task +# variant (hard: true), where the uniform soup should be well below saturation — the regime in which +# breeding + selection can actually improve on the default blend. Reuses the spec_*_hard specialists +# trained by moe_hard_hpc if present. The fair test the easy 7B null could not provide. + +base_model: Qwen/Qwen2.5-7B-Instruct +hard: true +families: [lists, strings, arith] +n_train: 800 +n_val: 100 +n_test: 200 +n_candidates: 24 +concentration: 0.5 +epochs: 3 +lora: {r: 16, alpha: 32} + +output: {dir: results/llm_directed_hard_hpc} diff --git a/configs/llm/moe_hard.yaml b/configs/llm/moe_hard.yaml new file mode 100644 index 0000000..a2991cf --- /dev/null +++ b/configs/llm/moe_hard.yaml @@ -0,0 +1,21 @@ +experiment: llm_moe_hard +kind: llm_moe +seed: 1 +n_replicates: 1 + +# Local 0.5B SMOKE for the hard benchmark: confirms the harder task variant (hard: true) runs +# end-to-end — trains fresh hard specialists (cached as spec_*_hard), the verifier still scores them, +# routing/fusion operators execute. 0.5B will score low on these harder tasks (that is expected; the +# calibrated comparison is the 7B moe_hard_hpc run). Kept small for speed. + +base_model: Qwen/Qwen2.5-0.5B-Instruct +hard: true +families: [lists, strings, arith] +n_train: 400 +n_test: 80 +n_route: 32 +epochs: 3 +lora: {r: 16, alpha: 32} +operators: [soup, ties, moe_oracle, moe_learned, max_merge] + +output: {dir: results/llm_moe_hard} diff --git a/configs/llm/moe_hard_hpc.yaml b/configs/llm/moe_hard_hpc.yaml new file mode 100644 index 0000000..b2b9f23 --- /dev/null +++ b/configs/llm/moe_hard_hpc.yaml @@ -0,0 +1,24 @@ +experiment: llm_moe_hard_hpc +kind: llm_moe +seed: 1 +n_replicates: 1 + +# The HARD-benchmark firm-up (L40S). The easy families saturated 7B at 1.00 (strings/arith), so the +# 7B moe/directed nulls could not distinguish "refinements don't help at scale" from "tasks too easy". +# This run uses the HARDER task variant (hard: true) — longer lists + multi-step ops, Caesar ciphers / +# letter transforms, multi-step & larger arithmetic — so 7B is NOT saturated and routing vs fusion has +# real headroom to separate. Trains fresh HARD specialists (cached as spec_*_hard). Same operators as +# moe_hpc; the question: does the 0.5B ordering (union > fusion) reappear at 7B once fusion can no +# longer trivially compose to the ceiling? + +base_model: Qwen/Qwen2.5-7B-Instruct +hard: true +families: [lists, strings, arith] +n_train: 800 +n_test: 200 +n_route: 48 +epochs: 3 +lora: {r: 16, alpha: 32} +operators: [soup, ties, moe_oracle, moe_learned, max_merge] + +output: {dir: results/llm_moe_hard_hpc} diff --git a/hpc/llm_hard.pbs b/hpc/llm_hard.pbs new file mode 100644 index 0000000..b57ae6a --- /dev/null +++ b/hpc/llm_hard.pbs @@ -0,0 +1,26 @@ +#!/bin/bash +# The HARD-benchmark firm-up on an L40S (46 GB): runs moe_hard_hpc then directed_hard_hpc in one job. +# moe trains the hard specialists (cached spec_*_hard); directed reuses them — so training happens once +# and both experiments share it. Harder tasks (multi-step lists, Caesar ciphers, multi-step arith) keep +# 7B off saturation, giving routing/fusion/selection real headroom to separate — the fair test the easy +# 7B runs could not provide. Same env as the other LLM jobs (see hpc/README.md). +# submit: qsub hpc/llm_hard.pbs status: qstat -u $USER +#PBS -l select=1:ncpus=8:mem=64gb:ngpus=1:gpu_type=L40S +#PBS -l walltime=01:00:00 +#PBS -N lam_llm_hard + +cd "$PBS_O_WORKDIR" +export HF_HOME="$EPHEMERAL/hf_cache" +export TOKENIZERS_PARALLELISM=false +export UV_CACHE_DIR="$EPHEMERAL/uvcache" + +source .venv/bin/activate +nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv,noheader + +python -m llm.experiment configs/llm/moe_hard_hpc.yaml # trains spec_*_hard + fusion/union +python -m llm.experiment configs/llm/directed_hard_hpc.yaml # reuses spec_*_hard + offspring select + +# results/{llm_moe_hard_hpc,llm_directed_hard_hpc}/ written in-place (parquet gitignored). Sync back: +# rsync -avz hpc:'…/results/llm_moe_hard_hpc/' results/llm_moe_hard_hpc/ +# rsync -avz hpc:'…/results/llm_directed_hard_hpc/' results/llm_directed_hard_hpc/ +echo "done: $(date)" diff --git a/results/llm_directed_hard_hpc/README.md b/results/llm_directed_hard_hpc/README.md new file mode 100644 index 0000000..386885a --- /dev/null +++ b/results/llm_directed_hard_hpc/README.md @@ -0,0 +1,42 @@ +# llm_directed_hard_hpc — directed sex on HARD (unsaturated) tasks at 7B: offspring selection helps again + +**Claim tested.** `llm_directed_hpc` found directed selection ≈ soup at 7B (0.868 ≈ 0.873) and flagged +the honest caveat that the easy families were *saturated* (no headroom for selection to exploit). This +run re-runs directed sex on the **hard task variant**, where the uniform soup is far from the ceiling — +the regime in which breeding + selection can actually improve on the default blend. One **L40S (46 GB)** +GPU, Imperial CX3; reuses the `spec_*_hard` specialists trained by `llm_moe_hard_hpc`. + +**Setup.** Base **Qwen2.5-7B-Instruct**, `hard: true`, **24 offspring** (Dirichlet-weighted merges), +100-task/family validation split (selection), 200-task/family test split (report). Seed 1. + +### Results (test accuracy — unsaturated) +| model | lists | strings | arith | overall | worst-family | +|---|---|---|---|---|---| +| best specialist (strings) | 0.155 | 0.665 | 0.455 | 0.425 | 0.155 | +| merge_soup (uniform, candidate 0) | 0.390 | 0.300 | 0.485 | 0.392 | 0.300 | +| **directed_overall** | 0.380 | 0.630 | 0.465 | **0.492** | **0.380** | +| **directed_balanced** | 0.380 | 0.630 | 0.465 | **0.492** | **0.380** | + +### The finding: the 7B "no headroom" null was also saturation +- **Directed selection beats the uniform soup by +10 points (0.492 > 0.392).** On hard tasks there *is* + a better blend than uniform averaging, and breeding 24 offspring + selecting on the verifier finds it + — recovering most of the routing-level performance (0.492 vs routing 0.500) from a single deployable + merged model. The `llm_directed_hpc` null (directed ≈ soup at 7B) was a saturation artefact, exactly + as that run's honest caveat predicted. +- **Selection repairs fusion's dilution.** The winning offspring lifts strings from soup's diluted + 0.300 back to **0.630** (near the 0.665 specialist) while keeping lists' composition gain (0.380) — + i.e. it finds a blend that composes where composition helps and avoids diluting the fragile skill. + Both breeding objectives converged to the same winner (overall = balanced), which also improves + worst-family (0.380 > soup 0.300). +- **Directed ≈ routing here.** A *single* searched-and-selected merged model (0.492) matches the + per-input router (0.500) on hard tasks — offspring selection buys most of routing's benefit without + needing a router at inference. + +### Takeaway +On unsaturated tasks, directed sex (breed offspring + select on the verifier) beats the single a-priori +soup at 7B, resolving the earlier null: it was task saturation, not scale, that made selection inert. +Together with `llm_moe_hard_hpc` this completes the correction — **both** "merge, don't average" (union +> fusion) and "directed sex" (selection > single blend) are **headroom** phenomena that hold at 7B once +the tasks are hard enough to leave room, not weak-base-only effects. **Falsifier (not triggered):** +directed offspring ≤ uniform soup — instead they beat it by 10 points. Provenance in `manifest.json` +(`hard: true`, L40S, torch 2.12.1 / transformers 5.13.0 / peft 0.19.1). diff --git a/results/llm_directed_hard_hpc/llm_directed.pdf b/results/llm_directed_hard_hpc/llm_directed.pdf new file mode 100644 index 0000000..c9316c4 Binary files /dev/null and b/results/llm_directed_hard_hpc/llm_directed.pdf differ diff --git a/results/llm_directed_hard_hpc/llm_directed.png b/results/llm_directed_hard_hpc/llm_directed.png new file mode 100644 index 0000000..f2c555f Binary files /dev/null and b/results/llm_directed_hard_hpc/llm_directed.png differ diff --git a/results/llm_directed_hard_hpc/manifest.json b/results/llm_directed_hard_hpc/manifest.json new file mode 100644 index 0000000..2fb7b65 --- /dev/null +++ b/results/llm_directed_hard_hpc/manifest.json @@ -0,0 +1,26 @@ +{ + "experiment": "llm_directed_hard_hpc", + "master_seed": 1, + "git_commit": null, + "python": "3.11.13", + "libraries": { + "numpy": "2.4.6", + "scipy": "1.17.1", + "pandas": "3.0.3", + "pyarrow": "24.0.0", + "torch": "2.12.1", + "transformers": "5.13.0", + "peft": "0.19.1" + }, + "rows": 35, + "results_sha256": "a5e45785a8701c5e1cdcfc64bc1482dffe4e68f8c6978e9d07dc8f04d40fbab6", + "layer": "2", + "tier": "llm", + "base_model": "Qwen/Qwen2.5-7B-Instruct", + "hard": true, + "directed": { + "n_candidates": 24, + "concentration": 0.5, + "n_val": 100 + } +} \ No newline at end of file diff --git a/results/llm_directed_hard_hpc/resolved_config.yaml b/results/llm_directed_hard_hpc/resolved_config.yaml new file mode 100644 index 0000000..4949ae0 --- /dev/null +++ b/results/llm_directed_hard_hpc/resolved_config.yaml @@ -0,0 +1,25 @@ +experiment: llm_directed_hard_hpc +seed: 1 +n_replicates: 1 +source_config: + experiment: llm_directed_hard_hpc + kind: llm_directed + seed: 1 + n_replicates: 1 + base_model: Qwen/Qwen2.5-7B-Instruct + hard: true + families: + - lists + - strings + - arith + n_train: 800 + n_val: 100 + n_test: 200 + n_candidates: 24 + concentration: 0.5 + epochs: 3 + lora: + r: 16 + alpha: 32 + output: + dir: results/llm_directed_hard_hpc diff --git a/results/llm_moe_hard_hpc/README.md b/results/llm_moe_hard_hpc/README.md new file mode 100644 index 0000000..15c4e3d --- /dev/null +++ b/results/llm_moe_hard_hpc/README.md @@ -0,0 +1,45 @@ +# llm_moe_hard_hpc — union vs fusion on HARD (unsaturated) tasks at 7B: the flip was a saturation artefact + +**Claim tested.** `llm_moe_hpc` found fusion beating union at 7B (soup 0.87 > routing 0.84) and read it +as "a capable base lets averaging compose." But the easy families were *saturated* (strings & arith at +1.00), so that flip could have been an artefact of no headroom rather than base capability. This run +re-runs the union-vs-fusion contrast on the **hard task variant** (multi-step lists, Caesar ciphers / +letter transforms, multi-step & larger arithmetic), where 7B is **not** saturated. One **L40S (46 GB)** +GPU, Imperial CX3. + +**Setup.** Base **Qwen2.5-7B-Instruct**, `hard: true`, fresh hard specialists (cached `spec_*_hard`), +200 test tasks/family, seed 1. Same five operators as `llm_moe_hpc`. + +### Results (accuracy — nothing saturated; arith ≈ 0.48, strings 0.67, lists 0.34) +| operator | lists | strings | arith | overall | worst-family | router | +|---|---|---|---|---|---|---| +| best specialist (strings) | 0.155 | 0.665 | 0.455 | 0.425 | 0.155 | — | +| fuse: soup | 0.390 | 0.300 | 0.485 | 0.392 | 0.300 | — | +| fuse: ties | 0.385 | 0.330 | 0.485 | 0.400 | 0.330 | — | +| **route: oracle** | 0.335 | **0.670** | 0.495 | **0.500** | 0.335 | 1.00 | +| **route: learned** | 0.335 | 0.670 | 0.495 | **0.500** | 0.335 | 1.00 | +| max-merge | 0.215 | 0.195 | 0.480 | 0.297 | 0.195 | — | + +### The finding: the 7B "fusion wins" flip was saturation, not capability +- **Union beats fusion again — decisively.** Routing **0.500 > fusion 0.40 (soup 0.392 / ties 0.400)**, + a 10-point margin, the *same* ordering as 0.5B. The easy-task flip (fusion > union at 7B) does **not** + survive once the tasks are hard enough to leave headroom. +- **Fusion dilutes so badly it loses to the best single specialist.** On hard tasks the strings skill + (Caesar ciphers etc.) is fragile: the strings-specialist scores **0.665**, but soup washes it out to + **0.300** — so soup (0.392 overall) even trails the best *single* specialist (0.425). Routing keeps + the specialist intact (strings 0.670) and wins. Dilution is severe exactly when the specialist's + contribution is hard-won. +- **So "merge, don't average" is a HEADROOM law, not a base-size law.** Union > fusion whenever there + is room to lose to dilution — a weak base (0.5B) *or* hard tasks at a strong base (7B-hard). Fusion + only wins in the degenerate corner where the tasks are so easy the strong base composes to the 1.00 + ceiling (7B-easy). This corrects the `llm_moe_hpc` interpretation: base capability was a confound; + the operative variable is task headroom. +- **Riders unchanged.** Learned router still perfect (1.00, lexical families); `max_merge` still the + weakest union (0.297, not input-adaptive). + +### Takeaway +On genuinely hard, unsaturated tasks, the union operator (routing) beats fusion at 7B by the same +margin it does at 0.5B — E8's `max > mean` in real weights, robust across scale once you control for +saturation. The `llm_moe_hpc` flip is re-read as a saturation artefact. **Falsifier (not triggered):** +fusion matching/beating routing on hard tasks — instead fusion diluted below even the best specialist. +Provenance in `manifest.json` (`hard: true`, L40S, torch 2.12.1 / transformers 5.13.0 / peft 0.19.1). diff --git a/results/llm_moe_hard_hpc/llm_moe.pdf b/results/llm_moe_hard_hpc/llm_moe.pdf new file mode 100644 index 0000000..9a93933 Binary files /dev/null and b/results/llm_moe_hard_hpc/llm_moe.pdf differ diff --git a/results/llm_moe_hard_hpc/llm_moe.png b/results/llm_moe_hard_hpc/llm_moe.png new file mode 100644 index 0000000..3684fdf Binary files /dev/null and b/results/llm_moe_hard_hpc/llm_moe.png differ diff --git a/results/llm_moe_hard_hpc/manifest.json b/results/llm_moe_hard_hpc/manifest.json new file mode 100644 index 0000000..eee66cf --- /dev/null +++ b/results/llm_moe_hard_hpc/manifest.json @@ -0,0 +1,28 @@ +{ + "experiment": "llm_moe_hard_hpc", + "master_seed": 1, + "git_commit": null, + "python": "3.11.13", + "libraries": { + "numpy": "2.4.6", + "scipy": "1.17.1", + "pandas": "3.0.3", + "pyarrow": "24.0.0", + "torch": "2.12.1", + "transformers": "5.13.0", + "peft": "0.19.1" + }, + "rows": 47, + "results_sha256": "a4e7c37e7cd900b56aa24ffc827e38e9a9d5cbb44dcff8d83cf010e992ed5c8a", + "layer": "2", + "tier": "llm", + "base_model": "Qwen/Qwen2.5-7B-Instruct", + "hard": true, + "operators": [ + "soup", + "ties", + "moe_oracle", + "moe_learned", + "max_merge" + ] +} \ No newline at end of file diff --git a/results/llm_moe_hard_hpc/resolved_config.yaml b/results/llm_moe_hard_hpc/resolved_config.yaml new file mode 100644 index 0000000..839a86d --- /dev/null +++ b/results/llm_moe_hard_hpc/resolved_config.yaml @@ -0,0 +1,29 @@ +experiment: llm_moe_hard_hpc +seed: 1 +n_replicates: 1 +source_config: + experiment: llm_moe_hard_hpc + kind: llm_moe + seed: 1 + n_replicates: 1 + base_model: Qwen/Qwen2.5-7B-Instruct + hard: true + families: + - lists + - strings + - arith + n_train: 800 + n_test: 200 + n_route: 48 + epochs: 3 + lora: + r: 16 + alpha: 32 + operators: + - soup + - ties + - moe_oracle + - moe_learned + - max_merge + output: + dir: results/llm_moe_hard_hpc diff --git a/src/llm/experiment.py b/src/llm/experiment.py index 6b73c92..20ccf06 100644 --- a/src/llm/experiment.py +++ b/src/llm/experiment.py @@ -51,9 +51,11 @@ def run_merge_experiment(cfg: dict) -> pd.DataFrame: lora = cfg.get("lora", {}) merges = cfg.get("merges", ["soup", "ties"]) seed = int(cfg["seed"]) + hard = bool(cfg.get("hard", False)) + suffix = "_hard" if hard else "" # hard specialists cache separately adapters_root = Path(cfg.get("adapters_dir", "models/llm")) - test = sum([make_tasks(f, n_test, seed=1000 + i) for i, f in enumerate(fams)], []) + test = sum([make_tasks(f, n_test, seed=1000 + i, hard=hard) for i, f in enumerate(fams)], []) rows: list[dict] = [] # base @@ -64,8 +66,8 @@ def run_merge_experiment(cfg: dict) -> pd.DataFrame: # one specialist per family dirs = [] for i, f in enumerate(fams): - d = str(adapters_root / f"spec_{f}") - train_specialist(base, f, d, n_train=n_train, epochs=epochs, seed=seed + i, + d = str(adapters_root / f"spec_{f}{suffix}") + train_specialist(base, f, d, n_train=n_train, epochs=epochs, seed=seed + i, hard=hard, r=int(lora.get("r", 16)), alpha=int(lora.get("alpha", 32))) dirs.append(d) m, tok = load_model(base, adapter_dir=d) @@ -96,12 +98,14 @@ def _load_or_train_specialists(cfg: dict, base: str, fams: list[str], name: str, epochs = int(cfg.get("epochs", 3)) lora = cfg.get("lora", {}) seed = int(cfg["seed"]) + hard = bool(cfg.get("hard", False)) + suffix = "_hard" if hard else "" # hard specialists cache separately adapters_root = Path(cfg.get("adapters_dir", "models/llm")) dirs: list[str] = [] for i, f in enumerate(fams): - d = str(adapters_root / f"spec_{f}") + d = str(adapters_root / f"spec_{f}{suffix}") if not (Path(d) / "adapter_config.json").exists(): # reuse across llm_merge / llm_moe runs - train_specialist(base, f, d, n_train=n_train, epochs=epochs, seed=seed + i, + train_specialist(base, f, d, n_train=n_train, epochs=epochs, seed=seed + i, hard=hard, r=int(lora.get("r", 16)), alpha=int(lora.get("alpha", 32))) dirs.append(d) m, tok = load_model(base, adapter_dir=d) @@ -113,7 +117,8 @@ def _load_or_train_specialists(cfg: dict, base: str, fams: list[str], name: str, def test_of(cfg: dict, fams: list[str]) -> list: """The held-out mixed test set (one seed offset per family), shared by both LLM runners.""" n_test = int(cfg.get("n_test", 100)) - return sum([make_tasks(f, n_test, seed=1000 + i) for i, f in enumerate(fams)], []) + hard = bool(cfg.get("hard", False)) + return sum([make_tasks(f, n_test, seed=1000 + i, hard=hard) for i, f in enumerate(fams)], []) def run_moe_experiment(cfg: dict) -> pd.DataFrame: @@ -160,7 +165,8 @@ def run_moe_experiment(cfg: dict) -> pd.DataFrame: if "moe_oracle" in ops: routes["moe_oracle"] = true_idx if "moe_learned" in ops: - route_train = sum([make_tasks(f, n_route, seed=2000 + i) for i, f in enumerate(fams)], []) + route_train = sum([make_tasks(f, n_route, seed=2000 + i, hard=bool(cfg.get("hard", False))) + for i, f in enumerate(fams)], []) tr_emb = embed_prompts(model, tok, [t.prompt for t in route_train]) te_emb = embed_prompts(model, tok, prompts) tr_fam = np.array([t.family for t in route_train]) @@ -207,7 +213,8 @@ def run_directed_experiment(cfg: dict) -> pd.DataFrame: rows: list[dict] = [] test = test_of(cfg, fams) - val = sum([make_tasks(f, n_val, seed=3000 + i) for i, f in enumerate(fams)], []) + val = sum([make_tasks(f, n_val, seed=3000 + i, hard=bool(cfg.get("hard", False))) + for i, f in enumerate(fams)], []) # base + specialists (parents), scored on test m, tok = load_model(base) @@ -265,7 +272,8 @@ def run_and_save(config_path: str | Path) -> Path: if kind not in _RUNNERS: raise ValueError(f"unknown LLM experiment kind {kind!r} (expected one of {list(_RUNNERS)})") df = _RUNNERS[kind](cfg) - extra = {"layer": "2", "tier": "llm", "base_model": cfg["base_model"]} + extra = {"layer": "2", "tier": "llm", "base_model": cfg["base_model"], + "hard": bool(cfg.get("hard", False))} if kind == "llm_moe": extra["operators"] = list(cfg.get("operators", [])) if kind == "llm_directed": diff --git a/src/llm/specialise.py b/src/llm/specialise.py index 89625fa..13a26c8 100644 --- a/src/llm/specialise.py +++ b/src/llm/specialise.py @@ -29,9 +29,12 @@ def _encode(tok, task, device): def train_specialist(base_name: str, family: str, out_dir: str, *, n_train: int = 600, epochs: int = 3, lr: float = 2e-4, batch_size: int = 8, r: int = 16, - alpha: int = 32, seed: int = 0, device: str = "cuda") -> str: + alpha: int = 32, seed: int = 0, device: str = "cuda", hard: bool = False) -> str: """Fine-tune a LoRA specialist on ``family`` and save the adapter to ``out_dir``. + Args: + hard (bool): train on the harder task variant (must match the eval difficulty). + Returns the adapter directory path. """ import torch @@ -49,7 +52,7 @@ def train_specialist(base_name: str, family: str, out_dir: str, *, n_train: int model = get_peft_model(model, lora) model.train() - tasks = make_tasks(family, n_train, seed=seed) + tasks = make_tasks(family, n_train, seed=seed, hard=hard) encoded = [_encode(tok, t, device) for t in tasks] opt = torch.optim.AdamW([p for p in model.parameters() if p.requires_grad], lr=lr) rng = np.random.default_rng(seed) diff --git a/src/llm/tasks.py b/src/llm/tasks.py index 2294ac6..d450340 100644 --- a/src/llm/tasks.py +++ b/src/llm/tasks.py @@ -10,6 +10,11 @@ decorrelated from the others (the precondition for recombination to have anythin Each task is a natural-language instruction + input with a single canonical answer; the verifier normalises the model's output and checks exact equality. This is the LLM analogue of the synthetic sandbox's exact oracle — correctness is the "reality that can say no" (grounding / fitness). + +A ``hard=True`` variant of each family (longer inputs + multi-step / cipher operations) exists so a +*capable* base (7B) is **not saturated** at 1.00 — the regime where recombination refinements +(routing, offspring selection) have headroom to matter. Same family labels and answer formats (int +list / integer / lowercase word), so the verifier and the whole experiment machinery are unchanged. """ from __future__ import annotations @@ -37,82 +42,173 @@ def _fmt_list(xs) -> str: return "[" + ", ".join(str(int(x)) for x in xs) + "]" -def _list_task(rng) -> Task: - n = int(rng.integers(6, 9)) # longer lists (harder) - xs = rng.integers(0, 20, size=n).tolist() - op = rng.choice(["sort ascending", "sort descending", "reverse", - "second largest value", "sum of the two largest values", - "sorted then reversed", "unique values preserving order"]) - if op == "sort ascending": - ans = _fmt_list(sorted(xs)) - elif op == "sort descending": - ans = _fmt_list(sorted(xs, reverse=True)) - elif op == "reverse": - ans = _fmt_list(list(reversed(xs))) - elif op == "second largest value": - ans = str(sorted(xs, reverse=True)[1]) - elif op == "sum of the two largest values": - ans = str(sum(sorted(xs, reverse=True)[:2])) - elif op == "sorted then reversed": - ans = _fmt_list(sorted(xs, reverse=True)) - else: - seen, out = set(), [] +def _list_task(rng, hard: bool = False) -> Task: + if not hard: + n = int(rng.integers(6, 9)) # longer lists (harder) + xs = rng.integers(0, 20, size=n).tolist() + op = rng.choice(["sort ascending", "sort descending", "reverse", + "second largest value", "sum of the two largest values", + "sorted then reversed", "unique values preserving order"]) + if op == "sort ascending": + ans = _fmt_list(sorted(xs)) + elif op == "sort descending": + ans = _fmt_list(sorted(xs, reverse=True)) + elif op == "reverse": + ans = _fmt_list(list(reversed(xs))) + elif op == "second largest value": + ans = str(sorted(xs, reverse=True)[1]) + elif op == "sum of the two largest values": + ans = str(sum(sorted(xs, reverse=True)[:2])) + elif op == "sorted then reversed": + ans = _fmt_list(sorted(xs, reverse=True)) + else: + seen, out = set(), [] + for x in xs: + if x not in seen: + seen.add(x); out.append(x) + ans = _fmt_list(out) + want = "the resulting list" if ans.startswith("[") else "the single number" + return Task("lists", f"Given the list {_fmt_list(xs)}, compute its {op}. " + f"Output only {want} and nothing else.", ans) + # hard: longer lists + MULTI-STEP operations (a capable base is not saturated here) + n = int(rng.integers(10, 15)) + xs = rng.integers(0, 30, size=n).tolist() + op = rng.choice(["sort ascending then the element at index 2 (0-based)", + "sum of the elements at even indices (0-based)", + "the third smallest value", + "the prefix sums (running totals) as a list", + "sort descending then the sum of the first three values", + "the product of the two smallest values", + "the values strictly greater than 15, preserving order"]) + if op.startswith("sort ascending then the element"): + ans = str(sorted(xs)[2]) + elif op.startswith("sum of the elements at even"): + ans = str(sum(xs[::2])) + elif op == "the third smallest value": + ans = str(sorted(xs)[2]) + elif op.startswith("the prefix sums"): + acc, out = 0, [] for x in xs: - if x not in seen: - seen.add(x); out.append(x) + acc += x; out.append(acc) ans = _fmt_list(out) + elif op.startswith("sort descending then the sum"): + ans = str(sum(sorted(xs, reverse=True)[:3])) + elif op.startswith("the product of the two smallest"): + s = sorted(xs); ans = str(s[0] * s[1]) + else: + ans = _fmt_list([x for x in xs if x > 15]) want = "the resulting list" if ans.startswith("[") else "the single number" - return Task("lists", f"Given the list {_fmt_list(xs)}, compute its {op}. " + return Task("lists", f"Given the list {_fmt_list(xs)}, compute {op}. " f"Output only {want} and nothing else.", ans) -def _string_task(rng) -> Task: - w = str(rng.choice(_WORDS)) - op = rng.choice(["reverse", "uppercase", "count of the letter 'a'", "count of vowels", - "remove all vowels", "first three characters", "last three characters"]) - if op == "reverse": - ans = w[::-1] - elif op == "uppercase": - ans = w.upper() - elif op == "count of the letter 'a'": - ans = str(w.count("a")) - elif op == "count of vowels": - ans = str(sum(c in "aeiou" for c in w)) - elif op == "remove all vowels": - ans = "".join(c for c in w if c not in "aeiou") - elif op == "first three characters": - ans = w[:3] +def _caesar(w: str, k: int) -> str: + return "".join(chr((ord(c) - 97 + k) % 26 + 97) for c in w) + + +def _string_task(rng, hard: bool = False) -> Task: + if not hard: + w = str(rng.choice(_WORDS)) + op = rng.choice(["reverse", "uppercase", "count of the letter 'a'", "count of vowels", + "remove all vowels", "first three characters", "last three characters"]) + if op == "reverse": + ans = w[::-1] + elif op == "uppercase": + ans = w.upper() + elif op == "count of the letter 'a'": + ans = str(w.count("a")) + elif op == "count of vowels": + ans = str(sum(c in "aeiou" for c in w)) + elif op == "remove all vowels": + ans = "".join(c for c in w if c not in "aeiou") + elif op == "first three characters": + ans = w[:3] + else: + ans = w[-3:] + return Task("strings", f"Given the word \"{w}\", compute its {op}. " + f"Output only the answer and nothing else.", ans) + # hard: longer words (two concatenated) + cipher / multi-step transforms + w = str(rng.choice(_WORDS)) + str(rng.choice(_WORDS)) + op = rng.choice(["caesar", "sortletters", "devowel", "mostfreq", "distinct", "revdevowel"]) + if op == "caesar": + ans = _caesar(w, 3) + instr = "shift each letter forward by 3 in the alphabet, wrapping z to a (a Caesar cipher)" + elif op == "sortletters": + ans = "".join(sorted(w)); instr = "its letters sorted in alphabetical order" + elif op == "devowel": + ans = "".join("x" if c in "aeiou" else c for c in w) + instr = "the word with every vowel replaced by the letter x" + elif op == "mostfreq": + ans = min(set(w), key=lambda c: (-w.count(c), c)) + instr = "the letter that occurs most often (on a tie, the one earliest in the alphabet)" + elif op == "distinct": + ans = str(len(set(w))); instr = "the number of distinct letters" else: - ans = w[-3:] - return Task("strings", f"Given the word \"{w}\", compute its {op}. " + ans = "".join(c for c in w[::-1] if c not in "aeiou") + instr = "the word reversed and then with all vowels removed" + return Task("strings", f"Given the word \"{w}\", compute {instr}. " f"Output only the answer and nothing else.", ans) -def _arith_task(rng) -> Task: - kind = rng.choice(["mul", "sub", "mul2", "seq"]) # multiplication-heavy (harder) - if kind == "mul": - a, b = rng.integers(11, 40, size=2).tolist(); ans = str(a * b) - q = f"What is {a} * {b}?" - elif kind == "sub": - a, b = sorted(rng.integers(100, 1000, size=2).tolist(), reverse=True); ans = str(a - b) - q = f"What is {a} - {b}?" - elif kind == "mul2": - a = int(rng.integers(11, 100)); b = int(rng.integers(3, 10)); ans = str(a * b) - q = f"What is {a} * {b}?" +def _arith_task(rng, hard: bool = False) -> Task: + if not hard: + kind = rng.choice(["mul", "sub", "mul2", "seq"]) # multiplication-heavy (harder) + if kind == "mul": + a, b = rng.integers(11, 40, size=2).tolist(); ans = str(a * b) + q = f"What is {a} * {b}?" + elif kind == "sub": + a, b = sorted(rng.integers(100, 1000, size=2).tolist(), reverse=True); ans = str(a - b) + q = f"What is {a} - {b}?" + elif kind == "mul2": + a = int(rng.integers(11, 100)); b = int(rng.integers(3, 10)); ans = str(a * b) + q = f"What is {a} * {b}?" + else: + start = int(rng.integers(2, 12)); step = int(rng.integers(3, 12)) + seq = [start + i * step for i in range(4)]; ans = str(seq[-1] + step) + q = f"What comes next in the sequence {', '.join(map(str, seq))}?" + return Task("arith", q + " Output only the number and nothing else.", ans) + # hard: multi-step / larger operands (a capable base is not saturated here) + kind = rng.choice(["madd", "mod", "diffsq", "bigmul", "rangesum", "gcd"]) + if kind == "madd": + a, b = rng.integers(11, 40, size=2).tolist(); c = int(rng.integers(10, 100)) + ans = str(a * b + c); q = f"What is {a} * {b} + {c}?" + elif kind == "mod": + a = int(rng.integers(100, 1000)); b = int(rng.integers(3, 20)) + ans = str(a % b); q = f"What is {a} mod {b} (the remainder of {a} divided by {b})?" + elif kind == "diffsq": + a, b = sorted(rng.integers(10, 30, size=2).tolist(), reverse=True) + ans = str(a * a - b * b); q = f"What is {a}^2 - {b}^2?" + elif kind == "bigmul": + a = int(rng.integers(100, 1000)); b = int(rng.integers(2, 10)) + ans = str(a * b); q = f"What is {a} * {b}?" + elif kind == "rangesum": + a = int(rng.integers(1, 20)); b = int(rng.integers(25, 55)) + ans = str(sum(range(a, b + 1))) + q = f"What is the sum of all integers from {a} to {b} inclusive?" else: - start = int(rng.integers(2, 12)); step = int(rng.integers(3, 12)) - seq = [start + i * step for i in range(4)]; ans = str(seq[-1] + step) - q = f"What comes next in the sequence {', '.join(map(str, seq))}?" + import math + a, b = rng.integers(6, 60, size=2).tolist() + ans = str(math.gcd(a, b)); q = f"What is the greatest common divisor of {a} and {b}?" return Task("arith", q + " Output only the number and nothing else.", ans) _GEN = {"lists": _list_task, "strings": _string_task, "arith": _arith_task} -def make_tasks(family: str, n: int, seed: int) -> list[Task]: - """Generate ``n`` unique-ish tasks for a family (deterministic in ``seed``).""" +def make_tasks(family: str, n: int, seed: int, hard: bool = False) -> list[Task]: + """Generate ``n`` unique-ish tasks for a family (deterministic in ``seed``). + + Args: + family (str): one of :data:`FAMILIES`. + n (int): number of tasks. + seed (int): RNG seed (deterministic output). + hard (bool): use the harder multi-step / cipher variant (same family label + answer format). + + Returns: + list[Task]: the generated tasks. + """ rng = np.random.default_rng(seed) - return [_GEN[family](rng) for _ in range(n)] + return [_GEN[family](rng, hard) for _ in range(n)] def _normalise(s: str) -> str: diff --git a/tasks/todo.md b/tasks/todo.md index 0cfef1d..2963b0e 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -380,3 +380,16 @@ C3 vertical claim deferred.* 0.5B, nil at 7B. Honest limit: 7B families near-saturated; a harder benchmark is the fair next test. - `configs/llm/{directed,directed_hpc}.yaml`, `figures/plot_llm_directed.py`, READMEs, `hpc/llm_directed.pbs`, Makefile `llm` target, +3 tests (130 green). `results/llm_directed{,_hpc}/`. + +**2026-07-05 — HARD benchmark: the 7B "fusion wins / no headroom" nulls were SATURATION artefacts.** ✅ +- Easy families saturated 7B (strings/arith 1.00), confounding the moe/directed 7B nulls. Built a hard + task variant (`hard: true`: multi-step lists, Caesar ciphers, multi-step/larger arith) threaded through + make_tasks/train_specialist/runners; hard specialists cache as `spec_*_hard`. Ran both at 7B on CX3 + (one L40S, 24 min, unsaturated). +- **Both nulls flip back to the 0.5B ordering:** union/routing 0.500 > fusion 0.40 (soup dilutes the + strings specialist 0.665→0.300, below even the best single specialist 0.425); directed selection 0.492 + > soup 0.392 (+10 pts). **The operative variable is HEADROOM, not base-size** — "merge, don't average" + and "directed sex" hold whenever there's room to lose to dilution (weak base OR hard tasks); fusion only + wins where easy tasks let a strong base compose to ceiling. Vindicates E8 max>mean at 7B. +- `configs/llm/{moe_hard,moe_hard_hpc,directed_hard_hpc}.yaml`, `hpc/llm_hard.pbs`, hard READMEs+figures, + +1 test (131 green). `results/llm_{moe,directed}_hard_hpc/`. diff --git a/tests/test_llm.py b/tests/test_llm.py index 77b1dd3..acf5c75 100644 --- a/tests/test_llm.py +++ b/tests/test_llm.py @@ -25,6 +25,21 @@ def test_make_tasks_wellformed_and_deterministic(): assert [t.answer for t in a] == [t.answer for t in b] # deterministic in the seed +def test_hard_tasks_wellformed_verifiable_and_distinct(): + # The hard variant must stay well-formed, self-verifying (canonical answer passes its own verifier), + # and genuinely different from the easy variant (harder content, same family labels + answer format). + for fam in FAMILIES: + hard = make_tasks(fam, 30, seed=7, hard=True) + assert len(hard) == 30 and all(t.family == fam for t in hard) + assert all(t.prompt and t.answer for t in hard) + assert all(verify(t.answer, t) for t in hard) # canonical answers verify + easy = make_tasks(fam, 30, seed=7, hard=False) + assert [t.prompt for t in hard] != [t.prompt for t in easy] # hard != easy + # a Caesar-cipher answer is a real transform of the input (not the identity) + caesars = [t for t in make_tasks("strings", 60, seed=2, hard=True) if "Caesar" in t.prompt] + assert caesars and any(t.answer not in t.prompt for t in caesars) + + def test_verifier_accepts_correct_including_verbose(): tasks = make_tasks("lists", 40, seed=1) + make_tasks("arith", 40, seed=2) assert all(verify(t.answer, t) for t in tasks) # the canonical answer verifies