diff --git a/CLAUDE.md b/CLAUDE.md index 0e2d8cd..b950df4 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -95,6 +95,8 @@ E4's whole purpose is to isolate the effect of teacher **decorrelation ρ**, so **Finding (2026-07-05, `llm_moe_hpc` — the regime *flips* at 7B; "merge, don't average" is a weak-base law).** Re-ran `llm_moe` at **Qwen2.5-7B-Instruct** (L40S, 9 min, reusing the cached 7B specialists). **The union-vs-fusion ordering inverts:** at 0.5B union won (routing 0.74 > soup 0.64); at 7B **fusion wins — soup 0.87 > routing 0.84 > max_merge 0.78.** Mechanism, and it's the deep point: **routing *selects* one intact specialist so it is capped at the best parent per family** (lists 0.57 = spec_lists, strings 0.97 = spec_strings), whereas **fusion *blends* deltas and, at a capable base, composes *beyond* any parent** (soup lists 0.62 > spec 0.57, strings 1.00 > spec 0.97). Selection can't synthesise something better than its best component; averaging-that-composes can. So the E4/E8 "merge, don't average" law is **regime-dependent — a weak-parent / small-model law, not universal**: union wins exactly when averaging *dilutes* (0.5B), fusion wins once the base has headroom to *compose* (7B). This refines rather than contradicts E8 (whose additive-landscape `max>mean` assumed no compositional headroom). The operator to actually want is **fusion-that-composes + selection over recombinant offspring** = the "directed sex" ideal (E10), the natural next experiment. `results/llm_moe_hpc/` (README + regime-aware figure title). Riders unchanged: learned router trivially perfect (lexical families), `max_merge` the weakest union (not input-adaptive). +**Finding (2026-07-05, `llm_directed` — directed sex in weights; refinements pay off only when the default blend is suboptimal).** E10 in real LLM weights (`src/llm/directed.py`, `kind: llm_directed`): breed a *population* of recombinant offspring (specialists merged at Dirichlet-sampled weights), score each against the verifier on a held-out **validation** split, keep the fittest — reported on a fresh **test** split (no selection-on-test leakage). Two breeding objectives (best-overall, best-worst-family). **The value scales with how far the uniform soup is from optimal, giving a clean regime split:** **0.5B** — soup dilutes, so directed selection beats it (`directed_overall` 0.69 > soup 0.64; `directed_balanced` worst-family 0.37 > soup 0.26), though single-objective selection trades off the other axis (breeding for overall tanks the rare `lists` to 0.17) and a *global* blend still trails per-input **routing** (0.74). **7B** — soup already *composes* to the ceiling on these near-saturated families (strings & arith at 1.00), so directed selection finds nothing better: **directed 0.868 ≈ soup 0.873** (marginally below, a val/test overfit gap). **Honest limitation:** the 7B families are near-saturated (2/3 at 1.00), which structurally caps the headroom — this run can't separate "directed sex doesn't help at scale" from "these tasks are too easy at 7B"; a *harder, unsaturated* benchmark is the fair next test. **Through-line across all four LLM runs:** "merge, don't average" and its refinements (routing, directed selection) are **weak-base / suboptimal-default** phenomena — they pay off at 0.5B (soup far from optimal) and are inert at 7B (soup near-optimal on saturated tasks). `configs/llm/{directed,directed_hpc}.yaml`, `plot_llm_directed.py`, `results/llm_directed{,_hpc}/`, `hpc/llm_directed.pbs`, +3 tests (130 green). + ## Build order (blueprint §7) — respect the gate 1. Scaffold: repo layout (§5), container, pytest skeleton, config system, seeding utils. `make test` green. diff --git a/Makefile b/Makefile index 9141e24..f8982c9 100644 --- a/Makefile +++ b/Makefile @@ -28,9 +28,10 @@ mnist: ## run the real-MNIST confirmation tier (needs env-mnist; dow env-llm: ## add the LLM stack for the Layer-2 prototype (GPU; transformers/peft) uv sync --extra dev --extra neural --extra llm -llm: ## run the LLM prototypes: merge (fusion) + moe (union) recombination +llm: ## run the LLM prototypes: merge (fusion) + moe (union) + directed (offspring select) uv run python -m llm.experiment configs/llm/merge.yaml uv run python -m llm.experiment configs/llm/moe.yaml + uv run python -m llm.experiment configs/llm/directed.yaml layer2: neural ## alias: Layer 1.5 is the current Layer-2 deliverable (LLM rung deferred) diff --git a/configs/llm/directed.yaml b/configs/llm/directed.yaml new file mode 100644 index 0000000..6500f4c --- /dev/null +++ b/configs/llm/directed.yaml @@ -0,0 +1,25 @@ +experiment: llm_directed +kind: llm_directed +seed: 1 +n_replicates: 1 + +# Layer 2 / LLM — DIRECTED SEX (E10) in weight space: breed many recombinant offspring and keep the +# fittest. llm_moe showed fusion can COMPOSE beyond the parents (7B) but the right blend is unknown and +# base-dependent, while pure routing is capped at the best parent. Directed sex resolves both: sample a +# population of weighted merges of the specialists, score each on a held-out VALIDATION split with the +# verifier (grounding), and select the winner — reported on a fresh TEST split (no selection-on-test +# leakage). Two objectives: best-overall and best-worst-family (raw capability vs the balanced +# generalist). Prediction: directed selection beats the single a-priori soup and every parent, at +# either scale. Falsifier: directed offspring ≤ uniform soup on test. + +base_model: Qwen/Qwen2.5-0.5B-Instruct # reuses the cached llm_merge specialists +families: [lists, strings, arith] +n_train: 700 # only if cached specialists are absent +n_val: 80 # held-out split the verifier selects on +n_test: 100 # fresh split winners are reported on +n_candidates: 16 # offspring population size +concentration: 0.5 # Dirichlet concentration (<1 = sparser, specialist-dominant blends) +epochs: 3 +lora: {r: 16, alpha: 32} + +output: {dir: results/llm_directed} diff --git a/configs/llm/directed_hpc.yaml b/configs/llm/directed_hpc.yaml new file mode 100644 index 0000000..528dd5e --- /dev/null +++ b/configs/llm/directed_hpc.yaml @@ -0,0 +1,21 @@ +experiment: llm_directed_hpc +kind: llm_directed +seed: 1 +n_replicates: 1 + +# Scaled directed-sex run for an L40S (48 GB). At 7B the uniform soup already composes (0.87, beating +# every specialist), so the sharp question is whether SEARCHING the recombination-weight space + +# selecting on the verifier can push past even that strong a-priori blend — i.e. does offspring +# selection still add headroom once fusion is already strong? Reuses the cached 7B specialists. + +base_model: Qwen/Qwen2.5-7B-Instruct +families: [lists, strings, arith] +n_train: 800 # only if cached specialists are absent (fresh on the node) +n_val: 100 +n_test: 200 +n_candidates: 24 +concentration: 0.5 +epochs: 3 +lora: {r: 16, alpha: 32} + +output: {dir: results/llm_directed_hpc} diff --git a/figures/plot_llm_directed.py b/figures/plot_llm_directed.py new file mode 100644 index 0000000..d5dd3f3 --- /dev/null +++ b/figures/plot_llm_directed.py @@ -0,0 +1,94 @@ +"""llm_directed figure — directed sex in weight space (E10): breed offspring, select the fittest. + +A population of recombinant offspring (weighted merges of the specialists) is scored on a held-out +validation split by the verifier; the winners (best-overall, best-worst-family) are reported on a +fresh test split against the uniform-soup blend and the best single specialist. Two panels: (A) +per-family accuracy — directed offspring (green) vs soup (orange) vs specialists (blue); (B) overall +vs worst-family, with the best-specialist bar as the parent ceiling. The suptitle reports whether +directed selection beat the single a-priori soup. Reads only the committed bundle. + +Usage: python figures/plot_llm_directed.py [results/llm_directed] +""" + +from __future__ import annotations + +import sys +from pathlib import Path + +import matplotlib.pyplot as plt +import numpy as np + +sys.path.insert(0, str(Path(__file__).parent)) +from _figlib import load_bundle, savefig # noqa: E402 + +_FAMS = ["lists", "strings", "arith"] +_DIRECTED = {"directed_overall": "directed:overall", "directed_balanced": "directed:balanced"} + + +def _acc(df, model, metric): + r = df[(df["model"] == model) & (df["metric"] == metric)]["accuracy"] + return float(r.iloc[0]) if len(r) else float("nan") + + +def main(results_dir: str = "results/llm_directed") -> None: + df, cfg = load_bundle(results_dir) + present = set(df["model"].unique()) + specialists = sorted(m for m in present if m.startswith("spec_")) + directed = [m for m in _DIRECTED if m in present] + soup = ["merge_soup"] if "merge_soup" in present else [] + models = ["base"] + specialists + soup + directed + labels = {"base": "base", **{s: s.replace("spec_", "spec:") for s in specialists}, + "merge_soup": "soup (uniform)", **_DIRECTED} + colors = {"base": "#7f7f7f", **{s: "#1f77b4" for s in specialists}, + "merge_soup": "#ff7f0e", **{m: "#2ca02c" for m in directed}} + + fig, axes = plt.subplots(1, 2, figsize=(13, 5)) + + # Panel A: per-family accuracy. + ax = axes[0] + x = np.arange(len(_FAMS)) + w = 0.8 / len(models) + for i, mdl in enumerate(models): + vals = [_acc(df, mdl, f) for f in _FAMS] + ax.bar(x + (i - (len(models) - 1) / 2) * w, vals, w, label=labels[mdl], color=colors[mdl], + alpha=0.9 if mdl in directed else 0.65) + ax.set_xticks(x); ax.set_xticklabels(_FAMS) + ax.set(ylabel="accuracy", title="Per-family: directed offspring (green), selected on the\n" + "verifier, vs the single uniform soup (orange) and the parents") + ax.legend(frameon=False, fontsize=8, ncol=2) + + # Panel B: overall vs worst-family, with the best-specialist ceiling. + ax = axes[1] + x2 = np.arange(len(models)) + for off, metric, hatch, lab in [(-0.2, "overall", "", "overall"), + (0.2, "worst_family", "//", "worst family")]: + ax.bar(x2 + off, [_acc(df, m, metric) for m in models], 0.38, + color=[colors[m] for m in models], hatch=hatch, alpha=0.85, label=lab, + edgecolor="white") + ax.set_xticks(x2); ax.set_xticklabels([labels[m] for m in models], rotation=25, ha="right", + fontsize=8) + if specialists: + ceil = max(_acc(df, s, "overall") for s in specialists) + ax.axhline(ceil, ls=":", c="#1f77b4", lw=1, alpha=0.7) # best-parent ceiling + ax.set(ylabel="accuracy", title="Overall (solid) vs worst-family (hatched):\n" + "directed offspring vs soup vs the best parent (dotted)") + ax.legend(frameon=False, fontsize=9) + + best_dir = max([_acc(df, m, "overall") for m in directed], default=float("nan")) + soup_o = _acc(df, "merge_soup", "overall") + best_spec = max([_acc(df, s, "overall") for s in specialists], default=float("nan")) + if best_dir > soup_o + 0.005: + verdict = f"directed {best_dir:.2f} > soup {soup_o:.2f} overall" + elif best_dir > soup_o - 0.005: + verdict = f"directed {best_dir:.2f} ≈ soup {soup_o:.2f} overall" + else: + verdict = f"directed {best_dir:.2f} < soup {soup_o:.2f} overall" + verdict += f" (best parent {best_spec:.2f})" + fig.suptitle(f"llm_directed — directed sex (breed offspring + select on the verifier): {verdict} " + f"({cfg['base_model'].split('/')[-1]})", y=1.0, fontsize=12) + fig.tight_layout() + savefig(fig, results_dir, "llm_directed") + + +if __name__ == "__main__": + main(*sys.argv[1:]) diff --git a/hpc/llm_directed.pbs b/hpc/llm_directed.pbs new file mode 100644 index 0000000..bd55173 --- /dev/null +++ b/hpc/llm_directed.pbs @@ -0,0 +1,22 @@ +#!/bin/bash +# Directed sex (llm_directed) on an L40S (46 GB) — breed recombinant offspring + select on the verifier. +# Reuses the cached 7B specialist adapters (models/llm/spec_*) if present. Same env as the other LLM +# jobs (see hpc/README.md): uv sync --extra dev --extra neural --extra llm + pre-download the 7B base. +# submit: qsub hpc/llm_directed.pbs status: qstat -u $USER +#PBS -l select=1:ncpus=8:mem=64gb:ngpus=1:gpu_type=L40S +#PBS -l walltime=02:00:00 +#PBS -N lam_llm_directed + +cd "$PBS_O_WORKDIR" +export HF_HOME="$EPHEMERAL/hf_cache" +export TOKENIZERS_PARALLELISM=false +export UV_CACHE_DIR="$EPHEMERAL/uvcache" + +source .venv/bin/activate +nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv,noheader + +python -m llm.experiment configs/llm/directed_hpc.yaml + +# results/llm_directed_hpc/ written in-place (parquet gitignored). Sync back to plot: +# rsync -avz hpc:'…/LamarckianAI/results/llm_directed_hpc/' results/llm_directed_hpc/ +echo "done: $(date)" diff --git a/results/llm_directed/README.md b/results/llm_directed/README.md new file mode 100644 index 0000000..bc6a1b0 --- /dev/null +++ b/results/llm_directed/README.md @@ -0,0 +1,50 @@ +# llm_directed — directed sex in weight space: breed offspring + select on the verifier (E10, 0.5B) + +**Claim tested.** `llm_moe` left a clean gap: fusion can *compose* beyond the parents but the right +blend is unknown and base-dependent, while pure routing is capped at the best parent. E10's answer is +**directed sex** — biology can't preview offspring, an AI can: breed a *population* of recombinant +offspring (the specialists merged at many different weights), score each against the verifier +("reality") on a held-out validation split, and keep the fittest. Selection replaces betting on one +a-priori blend. Two breeding objectives: best validation **overall**, and best validation +**worst-family** (raw capability vs the balanced generalist). + +**Setup.** Base **Qwen2.5-0.5B-Instruct**, the three cached `llm_merge` specialists, **16 offspring** +(Dirichlet-weighted merges, concentration 0.5, pinning candidate 0 = uniform soup for reference), +scored on an **80-task/family validation** split, winners reported on a **fresh 100-task/family test** +split (no selection-on-test leakage). Seed 1. + +### Results (test accuracy) +| model | lists | strings | arith | overall | worst-family | +|---|---|---|---|---|---| +| best specialist (strings) | 0.08 | 1.00 | 0.80 | 0.63 | 0.08 | +| merge_soup (uniform, candidate 0) | 0.26 | 0.74 | 0.91 | 0.64 | 0.26 | +| **directed_overall** (bred for overall) | 0.17 | 0.99 | 0.92 | **0.69** | 0.17 | +| **directed_balanced** (bred for worst-family) | 0.37 | 0.37 | 0.79 | 0.51 | **0.37** | + +### What holds, and the honest cost +- **Offspring selection beats the single a-priori blend — on the objective you breed for.** + `directed_overall` reaches **0.69 overall > soup 0.64** (and > best parent 0.63); `directed_balanced` + reaches **0.37 worst-family > soup 0.26**. Searching the recombination-weight space and letting the + verifier choose beats committing to uniform averaging — the E10 "preview and keep the fittest" claim, + in real weights. +- **Single-objective selection trades off the other axis (honest).** Breeding for *overall* on + lexically-imbalanced families finds a strings+arith-heavy blend that sacrifices the rare `lists` + skill (0.17, below soup's 0.26); breeding for *balance* lifts worst-family to 0.37 but costs overall. + Directed sex gives *control* over what you breed for — it does not hand you both for free. +- **A global blend still trails per-input routing at a weak base.** At 0.5B the best directed *global* + merge (0.69 / 0.43-max) does not beat `llm_moe`'s per-input **routing** (0.74 / 0.43): when the base + is weak, adapting the recombination *per input* beats any one fixed blend, however well selected. So + directed sex over blends beats *averaging*, not *routing* — combining the two (route, then select + among routed+blended offspring) is the natural next operator. + +### Takeaway +Directed sex — breed a population, select on the verifier — is confirmed in real LLM weights: it beats +the single uniform soup on whichever objective it optimises, the distinctly-AI advantage (offspring +preview + unbounded candidates) that biology lacks. The honest scope at 0.5B: selection buys one axis +at the other's expense, and a single global blend can't yet beat per-input routing. Whether searching +blends + selection can exceed even the *strong* 7B soup (which routing could not) is answered by +**`results/llm_directed_hpc/`: it can't — directed ≈ soup (0.868 ≈ 0.873)** because the 7B soup already +composes to the ceiling on these near-saturated families, leaving no fitter offspring to breed. So +directed sex helps exactly when the default blend is *suboptimal* (0.5B), and is inert when it is +already near-optimal (7B). **Falsifier (not triggered at 0.5B):** directed offspring ≤ uniform soup on +their bred objective — instead each beat it. diff --git a/results/llm_directed/llm_directed.pdf b/results/llm_directed/llm_directed.pdf new file mode 100644 index 0000000..ac22137 Binary files /dev/null and b/results/llm_directed/llm_directed.pdf differ diff --git a/results/llm_directed/llm_directed.png b/results/llm_directed/llm_directed.png new file mode 100644 index 0000000..6d7e713 Binary files /dev/null and b/results/llm_directed/llm_directed.png differ diff --git a/results/llm_directed/manifest.json b/results/llm_directed/manifest.json new file mode 100644 index 0000000..71bb9e7 --- /dev/null +++ b/results/llm_directed/manifest.json @@ -0,0 +1,25 @@ +{ + "experiment": "llm_directed", + "master_seed": 1, + "git_commit": "8da0dac00713fb9708804b4696a847a3767758d5", + "python": "3.14.5", + "libraries": { + "numpy": "2.5.0", + "scipy": "1.18.0", + "pandas": "3.0.3", + "pyarrow": "24.0.0", + "torch": "2.12.1", + "transformers": "5.13.0", + "peft": "0.19.1" + }, + "rows": 35, + "results_sha256": "143b86833cb320c7c6c693f0b5394f381a94a69277cc31ba5e0977e5ea6ffd78", + "layer": "2", + "tier": "llm", + "base_model": "Qwen/Qwen2.5-0.5B-Instruct", + "directed": { + "n_candidates": 16, + "concentration": 0.5, + "n_val": 80 + } +} \ No newline at end of file diff --git a/results/llm_directed/resolved_config.yaml b/results/llm_directed/resolved_config.yaml new file mode 100644 index 0000000..103fe47 --- /dev/null +++ b/results/llm_directed/resolved_config.yaml @@ -0,0 +1,24 @@ +experiment: llm_directed +seed: 1 +n_replicates: 1 +source_config: + experiment: llm_directed + kind: llm_directed + seed: 1 + n_replicates: 1 + base_model: Qwen/Qwen2.5-0.5B-Instruct + families: + - lists + - strings + - arith + n_train: 700 + n_val: 80 + n_test: 100 + n_candidates: 16 + concentration: 0.5 + epochs: 3 + lora: + r: 16 + alpha: 32 + output: + dir: results/llm_directed diff --git a/results/llm_directed_hpc/README.md b/results/llm_directed_hpc/README.md new file mode 100644 index 0000000..b3bcac8 --- /dev/null +++ b/results/llm_directed_hpc/README.md @@ -0,0 +1,49 @@ +# llm_directed_hpc — directed sex at scale (7B): no headroom once the soup already composes + +**Claim tested.** At 0.5B, breeding offspring and selecting on the verifier beat the uniform soup +(`llm_directed`: directed 0.69 > soup 0.64). But `llm_merge_hpc` showed the 7B soup already *composes* +to 0.87, beating every specialist. So the honest question: does searching the recombination-weight +space + selecting on the verifier find anything better than that strong default — or is there simply no +headroom left? Run on one **L40S (46 GB)** GPU of Imperial's CX3 HPC, 9 min walltime, 24 offspring, +reusing the cached 7B specialists. + +**Setup.** Base **Qwen2.5-7B-Instruct**, cached 7B specialists, **24 offspring** (Dirichlet-weighted +merges, concentration 0.5, candidate 0 = uniform soup), scored on a **100-task/family validation** +split, winners reported on a **fresh 200-task/family test** split. Seed 1. + +### Results (test accuracy) +| model | lists | strings | arith | overall | worst-family | +|---|---|---|---|---|---| +| best specialist (lists) | 0.57 | 0.74 | 1.00 | 0.77 | 0.57 | +| merge_soup (uniform, candidate 0) | 0.62 | 1.00 | 1.00 | **0.873** | 0.625 | +| directed_overall (bred for overall) | 0.62 | 1.00 | 1.00 | 0.868 | 0.615 | +| directed_balanced (bred for worst-family) | 0.62 | 1.00 | 1.00 | 0.868 | 0.615 | + +### The finding: directed selection is inert once the default is already near-optimal +- **Directed ≈ soup, and marginally below it (0.868 vs 0.873).** Both breeding objectives converged to + a near-soup blend, and the validation-selected winner is a *hair* worse than the plain uniform soup + on the held-out test set — a small val/test generalisation gap (selection overfits the 100-task/ + family validation split). Searching 24 offspring found nothing that beats candidate 0. +- **Why: no headroom.** The 7B soup already *composes* to the ceiling on these families — strings and + arith are saturated at **1.00**, and lists (0.62) is the only slack, itself already above every + specialist. When the default blend is already optimal, there is no fitter offspring to breed, so + selection can only match it (or lose slightly to val noise). +- **Honest limitation.** These task families are *near-saturated* at 7B (2 of 3 at 1.00), which + structurally caps the headroom any recombination refinement could exploit. A fair test of directed + sex at scale needs a **harder, unsaturated** benchmark where the optimal blend is non-trivial — this + run cannot distinguish "directed sex doesn't help at scale" from "these tasks are too easy at 7B." + +### Takeaway — the through-line across all four LLM runs +The value of every recombination *refinement* (routing, directed selection) scales with **how far the +default uniform soup is from optimal**: +- **0.5B** — soup *dilutes* (far from optimal): routing beats soup (0.74 > 0.64), directed selection + beats soup (0.69 > 0.64). Refinements pay off. +- **7B** — soup *composes* to near-ceiling on saturated tasks: routing < soup (0.84 < 0.87), directed + ≈ soup (0.868 ≈ 0.873). No headroom; refinements are inert. + +So "merge, don't average" and its refinements are **weak-base / suboptimal-default** phenomena. The +open question a capable base leaves is whether directed sex helps on *hard, unsaturated* tasks at scale +— the natural next benchmark. **Falsifier for this run (triggered as a null, reported honestly):** +directed offspring failed to exceed the uniform soup at 7B; here they tied/marginally trailed it +because the soup was already optimal on near-saturated families. Provenance in `manifest.json` +(L40S, torch 2.12.1 / transformers 5.13.0 / peft 0.19.1; `git_commit: null` — rsync'd node copy). diff --git a/results/llm_directed_hpc/llm_directed.pdf b/results/llm_directed_hpc/llm_directed.pdf new file mode 100644 index 0000000..16d8206 Binary files /dev/null and b/results/llm_directed_hpc/llm_directed.pdf differ diff --git a/results/llm_directed_hpc/llm_directed.png b/results/llm_directed_hpc/llm_directed.png new file mode 100644 index 0000000..b49f154 Binary files /dev/null and b/results/llm_directed_hpc/llm_directed.png differ diff --git a/results/llm_directed_hpc/manifest.json b/results/llm_directed_hpc/manifest.json new file mode 100644 index 0000000..8f9a779 --- /dev/null +++ b/results/llm_directed_hpc/manifest.json @@ -0,0 +1,25 @@ +{ + "experiment": "llm_directed_hpc", + "master_seed": 1, + "git_commit": null, + "python": "3.11.13", + "libraries": { + "numpy": "2.4.6", + "scipy": "1.17.1", + "pandas": "3.0.3", + "pyarrow": "24.0.0", + "torch": "2.12.1", + "transformers": "5.13.0", + "peft": "0.19.1" + }, + "rows": 35, + "results_sha256": "5bb2aef441ac8e00d3e6f8de02f74686415c367097b07f0688bbd687eb93d25d", + "layer": "2", + "tier": "llm", + "base_model": "Qwen/Qwen2.5-7B-Instruct", + "directed": { + "n_candidates": 24, + "concentration": 0.5, + "n_val": 100 + } +} \ No newline at end of file diff --git a/results/llm_directed_hpc/resolved_config.yaml b/results/llm_directed_hpc/resolved_config.yaml new file mode 100644 index 0000000..a3a65e2 --- /dev/null +++ b/results/llm_directed_hpc/resolved_config.yaml @@ -0,0 +1,24 @@ +experiment: llm_directed_hpc +seed: 1 +n_replicates: 1 +source_config: + experiment: llm_directed_hpc + kind: llm_directed + seed: 1 + n_replicates: 1 + base_model: Qwen/Qwen2.5-7B-Instruct + families: + - lists + - strings + - arith + n_train: 800 + n_val: 100 + n_test: 200 + n_candidates: 24 + concentration: 0.5 + epochs: 3 + lora: + r: 16 + alpha: 32 + output: + dir: results/llm_directed_hpc diff --git a/results/llm_moe/manifest.json b/results/llm_moe/manifest.json new file mode 100644 index 0000000..c496210 --- /dev/null +++ b/results/llm_moe/manifest.json @@ -0,0 +1,27 @@ +{ + "experiment": "llm_moe", + "master_seed": 1, + "git_commit": "585264d0b42f0e829229611bd83b08f5a5e418b7", + "python": "3.14.5", + "libraries": { + "numpy": "2.5.0", + "scipy": "1.18.0", + "pandas": "3.0.3", + "pyarrow": "24.0.0", + "torch": "2.12.1", + "transformers": "5.13.0", + "peft": "0.19.1" + }, + "rows": 47, + "results_sha256": "3e73caaeae3b5d77ded3ba956af4b767c0d4025a41a7ae1841806ce74b78045e", + "layer": "2", + "tier": "llm", + "base_model": "Qwen/Qwen2.5-0.5B-Instruct", + "operators": [ + "soup", + "ties", + "moe_oracle", + "moe_learned", + "max_merge" + ] +} \ No newline at end of file diff --git a/results/llm_moe/resolved_config.yaml b/results/llm_moe/resolved_config.yaml new file mode 100644 index 0000000..e4a584e --- /dev/null +++ b/results/llm_moe/resolved_config.yaml @@ -0,0 +1,28 @@ +experiment: llm_moe +seed: 1 +n_replicates: 1 +source_config: + experiment: llm_moe + kind: llm_moe + seed: 1 + n_replicates: 1 + base_model: Qwen/Qwen2.5-0.5B-Instruct + families: + - lists + - strings + - arith + n_train: 700 + n_test: 100 + n_route: 32 + epochs: 3 + lora: + r: 16 + alpha: 32 + operators: + - soup + - ties + - moe_oracle + - moe_learned + - max_merge + output: + dir: results/llm_moe diff --git a/src/llm/directed.py b/src/llm/directed.py new file mode 100644 index 0000000..a9cfbbf --- /dev/null +++ b/src/llm/directed.py @@ -0,0 +1,72 @@ +"""Directed sex in weight space — recombinant offspring + selection on the verifier (E10 in real LLMs). + +`llm_merge` blends the specialists with *one* fixed rule (uniform soup, or ties); `llm_moe` *selects* +one intact specialist per input. Both commit to a single recombination *a priori*. Biology can't +preview offspring; an AI can — evaluate many recombinants and keep the fittest. This is E10's +"directed sex": generate a **population** of offspring by recombining the parents at *different* mixing +weights, score each against the verifier ("reality") on a held-out validation split, and select the +best. It unifies the two regimes `llm_moe` exposed — fusion *composes* beyond the parents (so we want +blends, not pure selection), but the *right* blend is unknown and base-dependent (so we search it and +let grounding choose), instead of betting on uniform averaging. + +This module holds the pure, testable pieces — sampling a diverse population of simplex-ish merge +weights, and selecting winners from validation scores. The weight-space recombination + evaluation +loop lives in :func:`llm.experiment.run_directed_experiment` (it needs the loaded PEFT model). +""" + +from __future__ import annotations + +import numpy as np + + +def sample_merge_weights(k: int, n: int, rng: np.random.Generator, *, concentration: float = 0.5, + scale_lo: float = 1.0, scale_hi: float | None = None) -> np.ndarray: + """Sample ``n`` diverse recombination-weight vectors over ``k`` parents (the offspring genotypes). + + Each row is a Dirichlet draw (direction on the simplex) times a random total magnitude, spanning + from soup-like (total ≈ 1, balanced blend) to task-arithmetic-like (total ≈ ``k``, additive). + ``concentration < 1`` biases toward *sparse* mixes (one or two parents dominant) for real + diversity — the point of previewing many offspring rather than one average. + + The first two rows are pinned to the canonical baselines for coverage: uniform **soup** + (``1/k`` each) and unit **task-arithmetic** (``1`` each); the remaining ``n-2`` are random. + + Args: + k (int): number of parents (specialists). + n (int): population size (candidates). Must be ≥ 2. + rng (np.random.Generator): explicit RNG (seeded upstream via SeedSequence). + concentration (float): Dirichlet concentration; < 1 → sparser, specialist-dominant blends. + scale_lo (float): minimum total weight magnitude. + scale_hi (float | None): maximum total weight magnitude (defaults to ``k``). + + Returns: + np.ndarray: ``(n, k)`` float32 merge-weight vectors. + """ + if n < 2: + raise ValueError("need at least 2 candidates (soup + task_arith baselines)") + hi = float(k) if scale_hi is None else float(scale_hi) + out = np.empty((n, k), dtype=np.float32) + out[0] = np.full(k, 1.0 / k) # uniform soup + out[1] = np.ones(k) # task arithmetic + for i in range(2, n): + direction = rng.dirichlet(np.full(k, concentration)) + total = rng.uniform(scale_lo, hi) + out[i] = direction * total + return out + + +def select_winners(val_overall: np.ndarray, val_worst: np.ndarray) -> dict: + """Pick the offspring that maximise validation *overall* and validation *worst-family* accuracy. + + Two selection objectives = two things directed sex can breed for: raw capability, or balance + across skills (the Fisher–Muller generalist). Selection is on validation only; the winners are + then reported on a fresh test split (no selection-on-test leakage). + + Args: + val_overall (np.ndarray): per-candidate validation overall accuracy. + val_worst (np.ndarray): per-candidate validation worst-family accuracy. + + Returns: + dict: ``{"overall": idx, "balanced": idx}`` candidate indices. + """ + return {"overall": int(np.argmax(val_overall)), "balanced": int(np.argmax(val_worst))} diff --git a/src/llm/experiment.py b/src/llm/experiment.py index 66fb24c..6b73c92 100644 --- a/src/llm/experiment.py +++ b/src/llm/experiment.py @@ -22,6 +22,7 @@ import yaml from knowledge.experiment import save_artifacts +from .directed import sample_merge_weights, select_winners from .evaluate import evaluate, generate, load_model from .merge import load_specialists, make_merge from .moe import build_max_merge, embed_prompts, learned_routes, moe_generate @@ -184,7 +185,74 @@ def run_moe_experiment(cfg: dict) -> pd.DataFrame: return pd.DataFrame(rows) -_RUNNERS = {"llm_merge": run_merge_experiment, "llm_moe": run_moe_experiment} +def run_directed_experiment(cfg: dict) -> pd.DataFrame: + """Directed sex (E10) in weight space: breed many recombinant offspring, select the fittest. + + Generates a population of weighted merges of the specialists, scores each on a held-out + *validation* split with the verifier (grounding = "reality that says no"), and keeps the two + winners — best validation *overall* and best validation *worst-family* — reporting them on a + fresh *test* split alongside the uniform-soup and best-specialist baselines. The claim (E10): an + AI can preview offspring and keep the fittest, so directed selection over recombinants beats both + the single a-priori blend (soup) and any parent, at either scale. + """ + import torch + + name = cfg["experiment"] + base = cfg["base_model"] + fams = list(cfg.get("families", list(FAMILIES))) + n_val = int(cfg.get("n_val", 80)) + n_cand = int(cfg.get("n_candidates", 16)) + conc = float(cfg.get("concentration", 0.5)) + seed = int(cfg["seed"]) + rows: list[dict] = [] + + test = test_of(cfg, fams) + val = sum([make_tasks(f, n_val, seed=3000 + i) for i, f in enumerate(fams)], []) + + # base + specialists (parents), scored on test + m, tok = load_model(base) + rows += _rows(name, "base", "base", evaluate(m, tok, test)) + del m; torch.cuda.empty_cache() + dirs = _load_or_train_specialists(cfg, base, fams, name, rows) + k = len(dirs) + + # one base with all specialists; breed a population of weighted-merge offspring + model, tok = load_specialists(base, dirs) + rng = np.random.default_rng(seed) # Layer-2 statistical reproducibility + weights = sample_merge_weights(k, n_cand, rng, concentration=conc) + adapters = [f"a{i}" for i in range(k)] + + val_overall = np.empty(n_cand) + val_worst = np.empty(n_cand) + for i in range(n_cand): + cname = f"cand{i}" + model.add_weighted_adapter(adapters, weights[i].tolist(), cname, combination_type="linear") + model.set_adapter(cname) + acc = evaluate(model, tok, val) # selection signal (validation only) + val_overall[i] = acc["overall"] + val_worst[i] = min(acc[f] for f in fams) + + winners = select_winners(val_overall, val_worst) # {"overall": idx, "balanced": idx} + + def _test_acc(cname: str) -> dict: + model.set_adapter(cname) + outs = generate(model, tok, [t.prompt for t in test]) + corr = np.array([verify(o, t) for o, t in zip(outs, test)]) + famv = np.array([t.family for t in test]) + acc = {"overall": float(corr.mean())} + acc.update({f: float(corr[famv == f].mean()) for f in fams}) + return acc + + # uniform soup baseline is candidate 0 by construction; report it on test for direct comparison + rows += _rows(name, "merge_soup", "merge", _test_acc("cand0")) + for label, idx in winners.items(): + rows += _rows(name, f"directed_{label}", "directed", _test_acc(f"cand{idx}")) + + return pd.DataFrame(rows) + + +_RUNNERS = {"llm_merge": run_merge_experiment, "llm_moe": run_moe_experiment, + "llm_directed": run_directed_experiment} def run_and_save(config_path: str | Path) -> Path: @@ -200,6 +268,10 @@ def run_and_save(config_path: str | Path) -> Path: extra = {"layer": "2", "tier": "llm", "base_model": cfg["base_model"]} if kind == "llm_moe": extra["operators"] = list(cfg.get("operators", [])) + if kind == "llm_directed": + extra["directed"] = {"n_candidates": int(cfg.get("n_candidates", 16)), + "concentration": float(cfg.get("concentration", 0.5)), + "n_val": int(cfg.get("n_val", 80))} save_artifacts(cfg, df, out_dir, extra_libs=("torch", "transformers", "peft"), extra_manifest=extra, grid=None) return out_dir diff --git a/tasks/todo.md b/tasks/todo.md index 19a1198..0cfef1d 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -367,3 +367,16 @@ C3 vertical claim deferred.* spec 0.57). So "merge, don't average" is a **weak-base law**, not universal — union wins under dilution (0.5B), fusion wins under composition (7B). Refines E8. Next: fusion + offspring-selection (directed sex). `results/llm_moe_hpc/` README + regime-aware figure. + +**2026-07-05 — Directed sex (`llm_directed`): E10 in weights = breed offspring + select on verifier.** +- `src/llm/directed.py`: sample a population of Dirichlet-weighted merges, score on a held-out VAL + split, keep the best-overall + best-worst-family, report on a fresh TEST split. `kind: llm_directed`. +- **0.5B:** directed selection beats the single a-priori soup on the bred objective — directed_overall + 0.69 > soup 0.64; directed_balanced worst-family 0.37 > soup 0.26. Riders: single-objective selection + trades off the other axis (overall-breed tanks lists 0.17); a global blend still trails per-input + routing (0.74). **7B (CX3 L40S, 9 min):** directed ≈ soup (0.868 ≈ 0.873) — soup already composes to + ceiling on near-saturated families (strings/arith 1.00), no fitter offspring to breed. +- **Through-line:** recombination refinements pay off ∝ how suboptimal the default soup is — big at + 0.5B, nil at 7B. Honest limit: 7B families near-saturated; a harder benchmark is the fair next test. +- `configs/llm/{directed,directed_hpc}.yaml`, `figures/plot_llm_directed.py`, READMEs, `hpc/llm_directed.pbs`, + Makefile `llm` target, +3 tests (130 green). `results/llm_directed{,_hpc}/`. diff --git a/tests/test_llm.py b/tests/test_llm.py index c02a327..77b1dd3 100644 --- a/tests/test_llm.py +++ b/tests/test_llm.py @@ -10,6 +10,7 @@ from __future__ import annotations import numpy as np +from llm.directed import sample_merge_weights, select_winners from llm.moe import learned_routes from llm.tasks import FAMILIES, make_tasks, verify @@ -66,3 +67,28 @@ def test_learned_router_is_cosine_scale_invariant(): test_emb = np.array([[10.0, 0.0], [0.0, 0.01]]) # very different magnitudes routes = learned_routes(train_emb, train_fam, test_emb, fams) assert [fams[r] for r in routes] == ["a", "b"] + + +def test_merge_weights_population_pins_baselines_and_diversifies(): + # The offspring population must contain the two canonical baselines (uniform soup, unit task-arith) + # and be diverse + reproducible for the rest. + rng = np.random.default_rng(0) + w = sample_merge_weights(3, 16, rng) + assert w.shape == (16, 3) + assert np.allclose(w[0], 1 / 3) # candidate 0 = uniform soup + assert np.allclose(w[1], 1.0) # candidate 1 = task arithmetic + assert np.unique(w[2:].round(3), axis=0).shape[0] > 5 # the random offspring are diverse + assert np.allclose(sample_merge_weights(3, 16, np.random.default_rng(0)), w) # deterministic + + +def test_select_winners_picks_argmax_per_objective(): + val_overall = np.array([0.5, 0.9, 0.7]) + val_worst = np.array([0.4, 0.1, 0.6]) # a different candidate is most balanced + w = select_winners(val_overall, val_worst) + assert w == {"overall": 1, "balanced": 2} + + +def test_merge_weights_requires_two_candidates(): + import pytest + with pytest.raises(ValueError): + sample_merge_weights(3, 1, np.random.default_rng(0))