diff --git a/CLAUDE.md b/CLAUDE.md index 1476e26..0e2d8cd 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -91,6 +91,10 @@ E4's whole purpose is to isolate the effect of teacher **decorrelation ρ**, so **Finding (2026-07-05, `llm_merge_hpc` — the 7B firm-up on Imperial CX3; the marginal sign becomes decisive).** Re-ran `llm_merge` at a capable base — **Qwen2.5-7B-Instruct**, 200 test tasks/family, one **L40S (46 GB)** GPU, 8 min walltime — via the `/imperial-hpc` runbook (see `memory/hpc-setup.md`). **Both merges reach 0.87 overall, decisively above the best single specialist (0.77) and above every specialist on every family; worst-family 0.62 vs ≤0.57 for any specialist.** The two 0.5 B caveats are resolved: (1) the strict Fisher–Muller "exceeds every parent overall" claim is now clean (+10 points, not marginal); (2) the **dilution vanishes** — at 7 B the merge *beats* the lists-specialist on lists (0.62 > 0.57), where at 0.5 B averaging diluted it (0.43 → 0.26). **Interpretation: dilution is a small-model artefact; a capable base has enough headroom that weight-space averaging composes rather than dilutes** — the "merge, don't average" concern (E4) softens once parents are strong (soup ≈ ties at K=3). Results synced to `results/llm_merge_hpc/` (README legend + data-driven figure title). The natural refinement is *module-level* union-preserving recombination (MoE-expert / adapter-union merge, the real-weight image of E8's max-merge) rather than delta-averaging. +**Finding (2026-07-05, `llm_moe` — the union operator in real weights; E8's `max` vs `mean`, 0.5B).** Added the *union-preserving* recombination operator that `llm_merge` lacked (`src/llm/moe.py`, `kind: llm_moe`): never average the parents — keep each specialist LoRA intact and **select** the right one per prompt (MoE **router**: `oracle`, or `learned` = training-free nearest-centroid over the *base* model's own prompt embeddings) or per module (`max_merge` = winner-take-all by delta-norm). Reuses the cached `llm_merge` specialists (no retraining). **Result (0.5B, seed 1):** **routing wins decisively over fusion — overall 0.74 / worst-family 0.43 vs soup 0.64/0.26** — and recovers *each* specialist's own-family peak exactly (no dilution: fusion diluted the lists-specialist 0.43→0.26, routing keeps 0.43). This is E8's `max`(union) > `mean`(average) in real LLM weights. **Two honest riders:** (1) the learned router is *trivially perfect* (1.00) because the three families are lexically distinct — routing's win here rests partly on the routing problem being easy (ambiguous/overlapping skills would make the router the bottleneck — the interesting next failure mode); (2) **router-free `max_merge` is a poor union (0.46)** — static per-module winner-take-all isn't input-adaptive, so it collapses toward the strongest-norm modules; the union benefit needs *routing*, not weight surgery. `configs/llm/moe.yaml`, `plot_llm_moe.py`, `results/llm_moe/README.md`, +2 router tests (127 green). The regime question — does routing still beat fusion once a capable base lets fusion *compose* rather than dilute (7B soup already beats its specialists)? — is the `llm_moe_hpc` 7B run below. + +**Finding (2026-07-05, `llm_moe_hpc` — the regime *flips* at 7B; "merge, don't average" is a weak-base law).** Re-ran `llm_moe` at **Qwen2.5-7B-Instruct** (L40S, 9 min, reusing the cached 7B specialists). **The union-vs-fusion ordering inverts:** at 0.5B union won (routing 0.74 > soup 0.64); at 7B **fusion wins — soup 0.87 > routing 0.84 > max_merge 0.78.** Mechanism, and it's the deep point: **routing *selects* one intact specialist so it is capped at the best parent per family** (lists 0.57 = spec_lists, strings 0.97 = spec_strings), whereas **fusion *blends* deltas and, at a capable base, composes *beyond* any parent** (soup lists 0.62 > spec 0.57, strings 1.00 > spec 0.97). Selection can't synthesise something better than its best component; averaging-that-composes can. So the E4/E8 "merge, don't average" law is **regime-dependent — a weak-parent / small-model law, not universal**: union wins exactly when averaging *dilutes* (0.5B), fusion wins once the base has headroom to *compose* (7B). This refines rather than contradicts E8 (whose additive-landscape `max>mean` assumed no compositional headroom). The operator to actually want is **fusion-that-composes + selection over recombinant offspring** = the "directed sex" ideal (E10), the natural next experiment. `results/llm_moe_hpc/` (README + regime-aware figure title). Riders unchanged: learned router trivially perfect (lexical families), `max_merge` the weakest union (not input-adaptive). + ## Build order (blueprint §7) — respect the gate 1. Scaffold: repo layout (§5), container, pytest skeleton, config system, seeding utils. `make test` green. diff --git a/Makefile b/Makefile index 8ca946b..9141e24 100644 --- a/Makefile +++ b/Makefile @@ -28,8 +28,9 @@ mnist: ## run the real-MNIST confirmation tier (needs env-mnist; dow env-llm: ## add the LLM stack for the Layer-2 prototype (GPU; transformers/peft) uv sync --extra dev --extra neural --extra llm -llm: ## run the LLM merge prototype (needs env-llm; downloads a small base model) +llm: ## run the LLM prototypes: merge (fusion) + moe (union) recombination uv run python -m llm.experiment configs/llm/merge.yaml + uv run python -m llm.experiment configs/llm/moe.yaml layer2: neural ## alias: Layer 1.5 is the current Layer-2 deliverable (LLM rung deferred) diff --git a/configs/llm/moe.yaml b/configs/llm/moe.yaml new file mode 100644 index 0000000..0cd7597 --- /dev/null +++ b/configs/llm/moe.yaml @@ -0,0 +1,25 @@ +experiment: llm_moe +kind: llm_moe +seed: 1 +n_replicates: 1 + +# Layer 2 / LLM — module-level, UNION-PRESERVING recombination (the real-weight image of E8's *max*). +# Reuses the specialist adapters trained by configs/llm/merge.yaml (models/llm/spec_*) and contrasts +# two families of recombination operator on the same held-out mixed test set: +# FUSION (blend the deltas): soup = mean(Δ_k); ties = sign-reconciled union. +# UNION (never average): moe_oracle / moe_learned = keep every specialist intact and ROUTE each +# prompt to one (MoE-over-experts); max_merge = per-module winner-take-all. +# Prediction (E8, "merge don't average"): union beats fusion exactly where fusion DILUTES — pronounced +# at a weak base (0.5B), narrowing once a capable base lets fusion compose (7B). Falsifier: fusion +# matches or beats the routing ceiling (moe_oracle) at 0.5B, i.e. averaging never dilutes. + +base_model: Qwen/Qwen2.5-0.5B-Instruct # reuses the same cached specialists as llm_merge +families: [lists, strings, arith] +n_train: 700 # only used if the cached specialists are absent +n_test: 100 +n_route: 32 # labelled prompts per family for the learned router's centroids +epochs: 3 +lora: {r: 16, alpha: 32} +operators: [soup, ties, moe_oracle, moe_learned, max_merge] + +output: {dir: results/llm_moe} diff --git a/configs/llm/moe_hpc.yaml b/configs/llm/moe_hpc.yaml new file mode 100644 index 0000000..3abb962 --- /dev/null +++ b/configs/llm/moe_hpc.yaml @@ -0,0 +1,21 @@ +experiment: llm_moe_hpc +kind: llm_moe +seed: 1 +n_replicates: 1 + +# Scaled version of configs/llm/moe.yaml for an L40S (48 GB). At 7B the llm_merge fusion baseline +# (soup) already *composed* rather than diluted (merge 0.87 > best specialist 0.77, no dilution), so +# the sharp question here is whether the UNION operators (routing / max-merge) still add anything once +# the base is capable — i.e. does "merge, don't average" still bite at scale, or does a strong base +# make fusion and union converge? Either way is a reportable regime result. + +base_model: Qwen/Qwen2.5-7B-Instruct +families: [lists, strings, arith] +n_train: 800 # only used if the cached specialists are absent (fresh on the HPC node) +n_test: 200 +n_route: 48 +epochs: 3 +lora: {r: 16, alpha: 32} +operators: [soup, ties, moe_oracle, moe_learned, max_merge] + +output: {dir: results/llm_moe_hpc} diff --git a/figures/plot_llm_moe.py b/figures/plot_llm_moe.py new file mode 100644 index 0000000..e1b298f --- /dev/null +++ b/figures/plot_llm_moe.py @@ -0,0 +1,97 @@ +"""llm_moe figure — union-preserving recombination (route / max-merge) vs fusion (soup / ties). + +The real-weight image of E8's *max*: keep every specialist intact and *select* (route per prompt, or +per module) instead of averaging the deltas. Two panels: (A) per-family accuracy for the base, each +specialist, the fusion merges, and the union operators — the union operators should match the best +specialist on every family (they *are* that specialist there), while fusion may dilute or compose; +(B) overall vs worst-family, fusion vs union, with the routing ceiling (moe_oracle) marked. The +suptitle reports whether union beats fusion (dilution regime) or they converge (composition regime). +Reads only the committed bundle. + +Usage: python figures/plot_llm_moe.py [results/llm_moe] +""" + +from __future__ import annotations + +import sys +from pathlib import Path + +import matplotlib.pyplot as plt +import numpy as np + +sys.path.insert(0, str(Path(__file__).parent)) +from _figlib import load_bundle, savefig # noqa: E402 + +_FAMS = ["lists", "strings", "arith"] +_FUSION = {"merge_soup": "fuse:soup", "merge_ties": "fuse:ties"} +_UNION = {"moe_oracle": "route:oracle", "moe_learned": "route:learned", "max_merge": "max-merge"} + + +def _acc(df, model, metric): + r = df[(df["model"] == model) & (df["metric"] == metric)]["accuracy"] + return float(r.iloc[0]) if len(r) else float("nan") + + +def main(results_dir: str = "results/llm_moe") -> None: + df, cfg = load_bundle(results_dir) + present = set(df["model"].unique()) + specialists = sorted(m for m in present if m.startswith("spec_")) + fusion = [m for m in _FUSION if m in present] + union = [m for m in _UNION if m in present] + models = ["base"] + specialists + fusion + union + labels = {"base": "base", **{s: s.replace("spec_", "spec:") for s in specialists}, + **_FUSION, **_UNION} + colors = {"base": "#7f7f7f", **{s: "#1f77b4" for s in specialists}, + **{m: "#ff7f0e" for m in fusion}, **{m: "#2ca02c" for m in union}} + + fig, axes = plt.subplots(1, 2, figsize=(14, 5)) + + # Panel A: per-family accuracy, grouped by model. + ax = axes[0] + x = np.arange(len(_FAMS)) + w = 0.8 / len(models) + for i, mdl in enumerate(models): + vals = [_acc(df, mdl, f) for f in _FAMS] + ax.bar(x + (i - (len(models) - 1) / 2) * w, vals, w, label=labels[mdl], color=colors[mdl], + alpha=0.9 if (mdl in union or mdl in fusion) else 0.65) + ax.set_xticks(x); ax.set_xticklabels(_FAMS) + ax.set(ylabel="accuracy", title="Per-family: fusion (orange) blends the deltas; union (green)\n" + "keeps each specialist intact and selects — no dilution") + ax.legend(frameon=False, fontsize=8, ncol=2) + + # Panel B: overall vs worst-family, per model. + ax = axes[1] + x2 = np.arange(len(models)) + for off, metric, hatch, lab in [(-0.2, "overall", "", "overall"), + (0.2, "worst_family", "//", "worst family")]: + ax.bar(x2 + off, [_acc(df, m, metric) for m in models], 0.38, + color=[colors[m] for m in models], hatch=hatch, alpha=0.85, label=lab, + edgecolor="white") + ax.set_xticks(x2); ax.set_xticklabels([labels[m] for m in models], rotation=25, ha="right", + fontsize=8) + ax.set(ylabel="accuracy", title="Overall (solid) vs worst-family (hatched):\nfusion vs union " + "recombination") + # Mark the routing ceiling (oracle) if present. + if "moe_oracle" in present: + ceil = _acc(df, "moe_oracle", "overall") + ax.axhline(ceil, ls=":", c="#2ca02c", lw=1, alpha=0.7) + ax.legend(frameon=False, fontsize=9) + + best_fuse = max([_acc(df, m, "overall") for m in fusion], default=float("nan")) + best_union = max([_acc(df, m, "overall") for m in union], default=float("nan")) + router = _acc(df, "moe_learned", "router_acc") if "moe_learned" in present else float("nan") + if best_union > best_fuse + 0.01: + verdict = f"union {best_union:.2f} > fusion {best_fuse:.2f} overall (fusion dilutes)" + elif best_fuse > best_union + 0.01: + verdict = f"fusion {best_fuse:.2f} > union {best_union:.2f} overall (strong base composes)" + else: + verdict = f"union ≈ fusion ({best_union:.2f} vs {best_fuse:.2f}) overall" + rtxt = f"; learned router {router:.2f}" if router == router else "" + fig.suptitle(f"llm_moe — module-level union vs fusion recombination: {verdict}{rtxt} " + f"({cfg['base_model'].split('/')[-1]})", y=1.0, fontsize=12) + fig.tight_layout() + savefig(fig, results_dir, "llm_moe") + + +if __name__ == "__main__": + main(*sys.argv[1:]) diff --git a/hpc/llm_moe.pbs b/hpc/llm_moe.pbs new file mode 100644 index 0000000..671c2fe --- /dev/null +++ b/hpc/llm_moe.pbs @@ -0,0 +1,23 @@ +#!/bin/bash +# Module-level union-preserving recombination (llm_moe) on an L40S (46 GB) — route/max-merge vs fusion. +# Reuses the specialist adapters trained by the llm_merge_hpc run if models/llm/spec_* is present on +# the node; otherwise trains them fresh. Same env as hpc/llm_merge.pbs (see hpc/README.md): +# `uv sync --extra dev --extra neural --extra llm` on the login node + pre-download the 7B base. +# submit: qsub hpc/llm_moe.pbs status: qstat -u $USER +#PBS -l select=1:ncpus=8:mem=64gb:ngpus=1:gpu_type=L40S +#PBS -l walltime=02:00:00 +#PBS -N lam_llm_moe + +cd "$PBS_O_WORKDIR" +export HF_HOME="$EPHEMERAL/hf_cache" +export TOKENIZERS_PARALLELISM=false +export UV_CACHE_DIR="$EPHEMERAL/uvcache" + +source .venv/bin/activate +nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv,noheader + +python -m llm.experiment configs/llm/moe_hpc.yaml + +# results/llm_moe_hpc/ written in-place (parquet gitignored). Sync back to plot: +# rsync -avz hpc:'…/LamarckianAI/results/llm_moe_hpc/' results/llm_moe_hpc/ +echo "done: $(date)" diff --git a/results/llm_moe/README.md b/results/llm_moe/README.md new file mode 100644 index 0000000..a47b84b --- /dev/null +++ b/results/llm_moe/README.md @@ -0,0 +1,58 @@ +# llm_moe — union-preserving recombination (route / max-merge) vs fusion (the real-weight E8 *max*) + +**Claim tested.** E8 (analytic) found that the recombination *operator* matters: a **union** over +parents (`max`) assembles a child fitter than any parent, while an **average** (`mean`, the model +soup) conserves collapse. `llm_merge` showed fusion (soup/ties, which *average* the LoRA deltas). This +experiment adds the **union** operator to real LLM weights — never average the parents, keep each +specialist intact and **select** the right one per input (a Mixture-of-Experts *router*) or per module +(winner-take-all) — and asks whether union beats fusion, and *where*. + +**Setup.** Base **Qwen2.5-0.5B-Instruct**, reusing the three cached LoRA specialists from `llm_merge` +(disjoint families `lists`/`strings`/`arith`, exact-match verifier), 100 test tasks/family, seed 1. +Five recombination operators on the same test set: +- **Fusion** (blend the deltas): `soup` = mean(Δₖ); `ties` = sign-reconciled union. +- **Union** (never average): `route:oracle`/`route:learned` keep all adapters live and route each + prompt to one specialist (MoE); `max-merge` builds one adapter taking, per module, the specialist + with the largest-norm delta. The learned router is training-free — nearest-centroid over the *base* + model's own prompt embeddings; its routing accuracy is reported. + +### Results (accuracy) +| operator | lists | strings | arith | overall | worst-family | router | +|---|---|---|---|---|---|---| +| base | 0.15 | 0.15 | 0.53 | 0.28 | 0.15 | — | +| best specialist (strings) | 0.08 | 1.00 | 0.80 | 0.63 | 0.08 | — | +| fuse: soup | 0.26 | 0.74 | 0.91 | 0.64 | 0.26 | — | +| fuse: ties | 0.23 | 0.71 | 0.90 | 0.61 | 0.23 | — | +| **route: oracle** | 0.43 | 1.00 | 0.78 | **0.74** | **0.43** | 1.00 | +| **route: learned** | 0.43 | 1.00 | 0.78 | **0.74** | **0.43** | **1.00** | +| max-merge | 0.18 | 0.34 | 0.87 | 0.46 | 0.18 | — | + +### What holds +- **Union (routing) beats fusion at a weak base — decisively.** Routing reaches **0.74 overall / + 0.43 worst-family**, above both fusion merges (soup 0.64/0.26) and every specialist. It recovers + *each* specialist's own-family peak exactly (lists 0.43, strings 1.00, arith 0.78) because it *is* + that specialist there — **no dilution**. This is E8's `max` (union) beating `mean` (average) in real + LLM weights: exactly where fusion diluted the lists-specialist (0.43→0.26), routing keeps 0.43. +- **The learned router is perfect here (1.00) — stated as a caveat, not a triumph.** The three families + are lexically distinct, so nearest-centroid routing over base embeddings is trivially easy; + `route:learned` equals `route:oracle`. Routing's advantage on *these* tasks therefore rests partly on + the routing problem being easy — the honest scope. On overlapping/ambiguous skills the router would + be the bottleneck, and that is the interesting failure mode to probe next. +- **Static per-module `max-merge` is a poor union (0.46) — an informative negative.** Picking, per + module, the largest-norm specialist delta is *not* input-adaptive: it collapses toward whichever + specialist dominates the weight norms (arith 0.87, but lists 0.18, strings 0.34). The union benefit + needs **routing** (input-adaptive selection), not weight-space surgery — "keep the parents whole" + only pays off if you also *choose* the right parent per input. + +### Takeaway +Adds the union half of E8's operator dichotomy to real LLM weights and confirms its sign at a weak +base: **route-don't-average > average**, with no dilution, mirroring the analytic `max > mean`. Two +honest riders — the learned router is trivially good because the families are lexically separable, and +the router-free `max-merge` union fails because it isn't input-adaptive. The regime question the 0.5B +result raises — *does routing still help once a capable base lets fusion **compose** rather than +dilute?* (`llm_merge_hpc` showed 7B soup already beats its specialists with no dilution) — is answered +by **`results/llm_moe_hpc/`: the ordering flips.** At 7B fusion wins (soup 0.87 > routing 0.84), +because routing is capped at the best parent while fusion composes beyond it. So "merge, don't average" +is a **weak-base law**: union wins here (0.5B, dilution regime), fusion wins there (7B, composition +regime). **Falsifier (not triggered at 0.5B):** fusion matching the routing ceiling, i.e. averaging +never diluting. diff --git a/results/llm_moe/llm_moe.pdf b/results/llm_moe/llm_moe.pdf new file mode 100644 index 0000000..515d356 Binary files /dev/null and b/results/llm_moe/llm_moe.pdf differ diff --git a/results/llm_moe/llm_moe.png b/results/llm_moe/llm_moe.png new file mode 100644 index 0000000..fef5929 Binary files /dev/null and b/results/llm_moe/llm_moe.png differ diff --git a/results/llm_moe_hpc/README.md b/results/llm_moe_hpc/README.md new file mode 100644 index 0000000..7191090 --- /dev/null +++ b/results/llm_moe_hpc/README.md @@ -0,0 +1,52 @@ +# llm_moe_hpc — union vs fusion recombination at scale (7B, Imperial CX3): the regime *flips* + +**Claim tested.** The 0.5B `llm_moe` found that a **union** operator (route/select, never average) beats +**fusion** (soup/ties, average the deltas), because at a weak base averaging *dilutes*. But +`llm_merge_hpc` showed that at 7B fusion stops diluting and starts *composing* (soup 0.87 beat every +specialist). So the sharp question: **does routing still beat fusion once the base is capable — or does +a strong base invert the ordering?** Run on one **L40S (46 GB)** GPU of Imperial's CX3 HPC, 9 min +walltime, reusing the cached 7B specialists. + +**Setup.** Base **Qwen2.5-7B-Instruct**, the three cached LoRA specialists from `llm_merge_hpc` +(`lists`/`strings`/`arith`, exact-match verifier), 200 test tasks/family, seed 1. Same five operators +as the 0.5B run: fusion (`soup`, `ties`) vs union (`route:oracle`, `route:learned`, `max-merge`). + +### Results (accuracy) +| operator | lists | strings | arith | overall | worst-family | router | +|---|---|---|---|---|---|---| +| base | 0.46 | 0.69 | 1.00 | 0.71 | 0.46 | — | +| best specialist (lists) | 0.57 | 0.74 | 1.00 | 0.77 | 0.57 | — | +| **fuse: soup** | **0.62** | **1.00** | 1.00 | **0.87** | **0.62** | — | +| fuse: ties | 0.62 | 1.00 | 0.99 | 0.87 | 0.62 | — | +| route: oracle | 0.57 | 0.97 | 0.96 | 0.84 | 0.57 | 1.00 | +| route: learned | 0.57 | 0.97 | 0.96 | 0.84 | 0.57 | 1.00 | +| max-merge | 0.48 | 0.90 | 0.96 | 0.78 | 0.48 | — | + +### The finding: "merge, don't average" is regime-dependent, and inverts at scale +- **The ordering flips.** At 0.5B, union > fusion (routing 0.74 > soup 0.64). At 7B, **fusion > union** + (soup **0.87** > routing 0.84 > max-merge 0.78). The exact opposite winner. +- **Why: routing is capped at the best parent; fusion can *exceed* it.** Routing *selects* one intact + specialist, so per family it can only reach that specialist's own score (lists 0.57 = spec_lists, + strings 0.97 = spec_strings). Fusion *blends* the deltas — and at a capable base the blend + **composes beyond any parent**: soup scores lists 0.62 (> spec_lists 0.57) and strings 1.00 + (> spec_strings 0.97). Selection cannot synthesise something better than its best component; + averaging, when it composes rather than dilutes, can. So fusion's worst-family (0.62) also beats + routing's (0.57). +- **The regime boundary is dilution.** Union wins exactly when averaging *dilutes* (weak base, 0.5B); + fusion wins once the base has enough headroom that averaging *composes* (7B). "Merge, don't average" + (E4/E8) is therefore a **small-model / weak-parent** law, not a universal one — a genuine refinement + of the analytic claim, not a contradiction of it (E8's additive-landscape `max > mean` assumed no + such compositional headroom). +- **Riders unchanged.** The learned router is still perfect (1.00, lexically-distinct families), and + `max-merge` remains the weakest union (0.78) — static per-module winner-take-all is not + input-adaptive. + +### Takeaway +A clean, honest regime result: **route-don't-average wins at a weak base; average-that-composes wins at +a strong one.** Pure selection (routing) never dilutes but is bounded by the best parent; fusion risks +dilution but, given a capable base, transcends the parents — which is what the Fisher–Muller "exceed +every parent" claim actually needs at scale. The operator to want is therefore *fusion that composes +plus selection over candidates* — the "directed sex" ideal (offspring selection over recombinants), +the natural next experiment. **Falsifier for this run (not triggered):** routing beating fusion at 7B, +i.e. dilution persisting at scale — instead it inverted. Provenance in `manifest.json` (L40S, torch +2.12.1 / transformers 5.13.0 / peft 0.19.1; `git_commit: null` — produced on an rsync'd node copy). diff --git a/results/llm_moe_hpc/llm_moe.pdf b/results/llm_moe_hpc/llm_moe.pdf new file mode 100644 index 0000000..49f5590 Binary files /dev/null and b/results/llm_moe_hpc/llm_moe.pdf differ diff --git a/results/llm_moe_hpc/llm_moe.png b/results/llm_moe_hpc/llm_moe.png new file mode 100644 index 0000000..fa53c3b Binary files /dev/null and b/results/llm_moe_hpc/llm_moe.png differ diff --git a/results/llm_moe_hpc/manifest.json b/results/llm_moe_hpc/manifest.json new file mode 100644 index 0000000..3f5d1ea --- /dev/null +++ b/results/llm_moe_hpc/manifest.json @@ -0,0 +1,27 @@ +{ + "experiment": "llm_moe_hpc", + "master_seed": 1, + "git_commit": null, + "python": "3.11.13", + "libraries": { + "numpy": "2.4.6", + "scipy": "1.17.1", + "pandas": "3.0.3", + "pyarrow": "24.0.0", + "torch": "2.12.1", + "transformers": "5.13.0", + "peft": "0.19.1" + }, + "rows": 47, + "results_sha256": "bd91413c4cf174980c222dca5f435cb6957dee775d2f27bfb7a30dedc0f36c9f", + "layer": "2", + "tier": "llm", + "base_model": "Qwen/Qwen2.5-7B-Instruct", + "operators": [ + "soup", + "ties", + "moe_oracle", + "moe_learned", + "max_merge" + ] +} \ No newline at end of file diff --git a/results/llm_moe_hpc/resolved_config.yaml b/results/llm_moe_hpc/resolved_config.yaml new file mode 100644 index 0000000..a1b73aa --- /dev/null +++ b/results/llm_moe_hpc/resolved_config.yaml @@ -0,0 +1,28 @@ +experiment: llm_moe_hpc +seed: 1 +n_replicates: 1 +source_config: + experiment: llm_moe_hpc + kind: llm_moe + seed: 1 + n_replicates: 1 + base_model: Qwen/Qwen2.5-7B-Instruct + families: + - lists + - strings + - arith + n_train: 800 + n_test: 200 + n_route: 48 + epochs: 3 + lora: + r: 16 + alpha: 32 + operators: + - soup + - ties + - moe_oracle + - moe_learned + - max_merge + output: + dir: results/llm_moe_hpc diff --git a/src/llm/experiment.py b/src/llm/experiment.py index a39114d..66fb24c 100644 --- a/src/llm/experiment.py +++ b/src/llm/experiment.py @@ -24,12 +24,13 @@ from knowledge.experiment import save_artifacts from .evaluate import evaluate, generate, load_model from .merge import load_specialists, make_merge +from .moe import build_max_merge, embed_prompts, learned_routes, moe_generate from .specialise import train_specialist from .tasks import FAMILIES, make_tasks, verify def _rows(experiment: str, model: str, kind: str, acc: dict) -> list[dict]: - worst = min(acc[f] for f in FAMILIES) + worst = min(acc[f] for f in acc if f in FAMILIES) out = [{"experiment": experiment, "model": model, "kind": kind, "metric": k, "accuracy": v} for k, v in acc.items()] out.append({"experiment": experiment, "model": model, "kind": kind, @@ -85,16 +86,122 @@ def run_merge_experiment(cfg: dict) -> pd.DataFrame: return pd.DataFrame(rows) +def _load_or_train_specialists(cfg: dict, base: str, fams: list[str], name: str, + rows: list[dict]) -> list[str]: + """Reuse cached specialist adapters if present, else train one per family. Appends spec rows.""" + import torch + + n_train = int(cfg.get("n_train", 700)) + epochs = int(cfg.get("epochs", 3)) + lora = cfg.get("lora", {}) + seed = int(cfg["seed"]) + adapters_root = Path(cfg.get("adapters_dir", "models/llm")) + dirs: list[str] = [] + for i, f in enumerate(fams): + d = str(adapters_root / f"spec_{f}") + if not (Path(d) / "adapter_config.json").exists(): # reuse across llm_merge / llm_moe runs + train_specialist(base, f, d, n_train=n_train, epochs=epochs, seed=seed + i, + r=int(lora.get("r", 16)), alpha=int(lora.get("alpha", 32))) + dirs.append(d) + m, tok = load_model(base, adapter_dir=d) + rows += _rows(name, f"spec_{f}", "specialist", evaluate(m, tok, test_of(cfg, fams))) + del m; torch.cuda.empty_cache() + return dirs + + +def test_of(cfg: dict, fams: list[str]) -> list: + """The held-out mixed test set (one seed offset per family), shared by both LLM runners.""" + n_test = int(cfg.get("n_test", 100)) + return sum([make_tasks(f, n_test, seed=1000 + i) for i, f in enumerate(fams)], []) + + +def run_moe_experiment(cfg: dict) -> pd.DataFrame: + """Contrast union-preserving recombination (route / max-merge) with fusion (soup / ties). + + Reuses the trained specialist adapters and evaluates, per operator, on the held-out mixed test + set. Union operators keep each specialist intact and *select* (per prompt via a router, or per + module via winner-take-all); fusion operators blend the deltas. The real-weight image of E8's + ``max`` vs the ``mean`` baseline. Returns long-form accuracies (+ a ``router_acc`` fidelity row + for each routed operator). + """ + import torch + + name = cfg["experiment"] + base = cfg["base_model"] + fams = list(cfg.get("families", list(FAMILIES))) + ops = list(cfg.get("operators", ["soup", "ties", "moe_oracle", "moe_learned", "max_merge"])) + n_route = int(cfg.get("n_route", 32)) + rows: list[dict] = [] + + test = test_of(cfg, fams) + prompts = [t.prompt for t in test] + fam = np.array([t.family for t in test]) + true_idx = np.array([fams.index(t.family) for t in test]) + + # base + specialists (parents) + m, tok = load_model(base) + rows += _rows(name, "base", "base", evaluate(m, tok, test)) + del m; torch.cuda.empty_cache() + dirs = _load_or_train_specialists(cfg, base, fams, name, rows) + k = len(dirs) + + # all specialists on one base for the recombination operators + model, tok = load_specialists(base, dirs) + + def _score(outs: list[str]) -> dict: + corr = np.array([verify(o, t) for o, t in zip(outs, test)]) + acc = {"overall": float(corr.mean())} + acc.update({f: float(corr[fam == f].mean()) for f in fams}) + return acc + + # precompute router assignments once, from the base model's own prompt embeddings + routes: dict[str, np.ndarray] = {} + if "moe_oracle" in ops: + routes["moe_oracle"] = true_idx + if "moe_learned" in ops: + route_train = sum([make_tasks(f, n_route, seed=2000 + i) for i, f in enumerate(fams)], []) + tr_emb = embed_prompts(model, tok, [t.prompt for t in route_train]) + te_emb = embed_prompts(model, tok, prompts) + tr_fam = np.array([t.family for t in route_train]) + routes["moe_learned"] = learned_routes(tr_emb, tr_fam, te_emb, fams) + + for op in ops: + if op in ("soup", "ties", "task_arith"): + make_merge(model, k, op, op) + rows += _rows(name, f"merge_{op}", "merge", _score(generate(model, tok, prompts))) + elif op in ("moe_oracle", "moe_learned"): + r = routes[op] + rows += _rows(name, op, "moe", _score(moe_generate(model, tok, prompts, r, k))) + rows.append({"experiment": name, "model": op, "kind": "moe", + "metric": "router_acc", "accuracy": float((r == true_idx).mean())}) + elif op == "max_merge": + build_max_merge(model, k, "max_merge") + rows += _rows(name, "max_merge", "moe", _score(generate(model, tok, prompts))) + else: + raise ValueError(f"unknown operator {op!r} (soup|ties|task_arith|moe_oracle|" + f"moe_learned|max_merge)") + + return pd.DataFrame(rows) + + +_RUNNERS = {"llm_merge": run_merge_experiment, "llm_moe": run_moe_experiment} + + def run_and_save(config_path: str | Path) -> Path: - """Load an LLM experiment YAML, run it, and write the artifact triple.""" + """Load an LLM experiment YAML, run it (dispatch on ``kind``), and write the artifact triple.""" config_path = Path(config_path) cfg = yaml.safe_load(config_path.read_text()) out_dir = Path(cfg.get("output", {}).get("dir", f"results/{cfg['experiment']}")) cfg.setdefault("n_replicates", 1) - df = run_merge_experiment(cfg) + kind = cfg.get("kind", "llm_merge") + if kind not in _RUNNERS: + raise ValueError(f"unknown LLM experiment kind {kind!r} (expected one of {list(_RUNNERS)})") + df = _RUNNERS[kind](cfg) + extra = {"layer": "2", "tier": "llm", "base_model": cfg["base_model"]} + if kind == "llm_moe": + extra["operators"] = list(cfg.get("operators", [])) save_artifacts(cfg, df, out_dir, extra_libs=("torch", "transformers", "peft"), - extra_manifest={"layer": "2", "tier": "llm", "base_model": cfg["base_model"]}, - grid=None) + extra_manifest=extra, grid=None) return out_dir diff --git a/src/llm/moe.py b/src/llm/moe.py new file mode 100644 index 0000000..4dcbdac --- /dev/null +++ b/src/llm/moe.py @@ -0,0 +1,154 @@ +"""Module-level, union-preserving recombination of specialists — the real-weight image of E8's *max*. + +`merge.py` fuses specialist adapters into one set of weights (``soup`` averages the deltas, ``ties`` +sign-reconciles them). Fusion can *dilute*: at a weak base, averaging a specialist's delta by ``1/K`` +erased its peak (the 0.5B ``llm_merge`` result). The alternative recombination operator — the one E8 +calls *max* / union-preserving — never averages the parents at all. It keeps every specialist adapter +**intact** and, per input (or per module), **selects** the parent that owns that skill. Nothing is +diluted because nothing is blended; the child is the *union* of the parents' capabilities. + +Two selection operators, both reusing the already-trained specialist adapters (no retraining): + +* **route** — a Mixture-of-Experts over the specialists: a cheap router assigns each prompt to one + specialist adapter, which then answers it. Router variants: ``oracle`` (route by the known task + family — the ceiling of routing) and ``learned`` (nearest-centroid over the *base* model's own + prompt embeddings — an honest, training-free router; its accuracy is reported). +* **max_merge** — a router-free static union: build a single adapter that, per LoRA module, copies the + delta from the specialist whose delta has the largest norm there (winner-take-all per module). The + literal weight-space image of E8's element-wise ``max`` over teachers. + +The experiment (``kind: llm_moe``) contrasts these union operators against the fusion baselines +(soup/ties) at both scales — the prediction being that union wins where fusion dilutes (0.5B) and the +gap narrows once a capable base lets fusion *compose* rather than dilute (7B): the regime boundary of +"merge, don't average" in real LLM weights. +""" + +from __future__ import annotations + +import numpy as np + +from .evaluate import generate + + +def embed_prompts(model, tok, prompts: list[str], batch_size: int = 32, + device: str = "cuda") -> np.ndarray: + """Mean last-hidden-state embedding of each prompt from the **base** model (adapters disabled). + + The router must see a parent-agnostic representation, so generation happens with + ``model.disable_adapter()`` — the embedding is the frozen base model's, not any specialist's. + + Args: + model: a PEFT model with specialist adapters loaded. + tok: the matching tokenizer (left-padded). + prompts (list[str]): raw user prompts. + batch_size (int): forward-pass batch size. + device (str): compute device. + + Returns: + np.ndarray: ``(len(prompts), hidden)`` float32 embeddings. + """ + import torch + + embs: list[np.ndarray] = [] + with model.disable_adapter(): # parent-agnostic base representation + for i in range(0, len(prompts), batch_size): + chunk = prompts[i:i + batch_size] + msgs = [[{"role": "user", "content": p}] for p in chunk] + enc = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt", + return_dict=True, padding=True).to(device) + with torch.no_grad(): + out = model(**enc, output_hidden_states=True) + h = out.hidden_states[-1] # (B, T, H) + mask = enc["attention_mask"].unsqueeze(-1) # (B, T, 1) + mean = (h * mask).sum(1) / mask.sum(1).clamp(min=1) + embs.append(mean.float().cpu().numpy()) + return np.concatenate(embs, axis=0) + + +def learned_routes(train_emb: np.ndarray, train_fam: np.ndarray, test_emb: np.ndarray, + fams: list[str]) -> np.ndarray: + """Nearest-centroid router: assign each test prompt to the family whose train centroid is closest. + + A training-free expert selector — the "regulatory gene" that decides which specialist to express. + + Args: + train_emb (np.ndarray): ``(n_train, H)`` base embeddings of labelled train prompts. + train_fam (np.ndarray): family label per train prompt. + test_emb (np.ndarray): ``(n_test, H)`` base embeddings of test prompts. + fams (list[str]): family order (index i ↔ adapter ``a{i}``). + + Returns: + np.ndarray: ``(n_test,)`` int expert index in ``[0, len(fams))``. + """ + centroids = np.stack([train_emb[train_fam == f].mean(0) for f in fams]) # (K, H) + # Cosine distance is scale-robust for hidden states. + c = centroids / (np.linalg.norm(centroids, axis=1, keepdims=True) + 1e-8) + t = test_emb / (np.linalg.norm(test_emb, axis=1, keepdims=True) + 1e-8) + sims = t @ c.T # (n_test, K) + return sims.argmax(1) + + +def moe_generate(model, tok, prompts: list[str], routes: np.ndarray, k: int, + **gen_kw) -> list[str]: + """Generate each prompt with its routed specialist adapter active (grouped by expert for speed). + + Args: + model: PEFT model with adapters ``a0..a{k-1}`` loaded. + tok: tokenizer. + prompts (list[str]): prompts to answer. + routes (np.ndarray): expert index per prompt (from an oracle or learned router). + k (int): number of experts. + **gen_kw: forwarded to :func:`llm.evaluate.generate`. + + Returns: + list[str]: completions, in the original prompt order. + """ + outs: list[str] = [""] * len(prompts) + for expert in range(k): + idx = np.nonzero(routes == expert)[0] + if len(idx) == 0: + continue + model.set_adapter(f"a{expert}") + completions = generate(model, tok, [prompts[i] for i in idx], **gen_kw) + for j, i in enumerate(idx): + outs[i] = completions[j] + return outs + + +def build_max_merge(model, k: int, name: str): + """Add a router-free union adapter ``name``: per LoRA module, keep the largest-norm specialist delta. + + For each LoRA-adapted module, the effective delta of specialist ``a_i`` is ``B_i @ A_i`` (scaled). + We select, per module, the specialist whose delta has the largest Frobenius norm and copy its + ``A``/``B`` into a fresh adapter — a deterministic winner-take-all union (E8's element-wise max at + the granularity of a module). No averaging, so no dilution. + + Args: + model: PEFT model with adapters ``a0..a{k-1}``. + k (int): number of specialists. + name (str): name of the new merged adapter (created as a copy of ``a0`` then overwritten). + + Returns: + The model, with adapter ``name`` added and set active. + """ + import torch + from peft.tuners.lora import LoraLayer + + # Seed the new adapter from a0's config, then overwrite its weights per module. + model.add_weighted_adapter([f"a{i}" for i in range(k)], [1.0] + [0.0] * (k - 1), name, + combination_type="linear") + with torch.no_grad(): + for module in model.modules(): + if not isinstance(module, LoraLayer) or name not in module.lora_A: + continue + norms = [] + for i in range(k): + a, b = f"a{i}", f"a{i}" + delta = module.lora_B[a].weight @ module.lora_A[a].weight + norms.append(float(delta.norm()) * module.scaling.get(f"a{i}", 1.0)) + win = int(np.argmax(norms)) + module.lora_A[name].weight.copy_(module.lora_A[f"a{win}"].weight) + module.lora_B[name].weight.copy_(module.lora_B[f"a{win}"].weight) + module.scaling[name] = module.scaling.get(f"a{win}", 1.0) + model.set_adapter(name) + return model diff --git a/tasks/todo.md b/tasks/todo.md index bc25646..19a1198 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -352,3 +352,18 @@ C3 vertical claim deferred.* and dilution VANISHES (merge 0.62 > lists-spec 0.57 on lists) — dilution was a small-model artefact. - `results/llm_merge_hpc/` (README legend, data-driven figure title). Next refinement: module-level union-preserving recombination (MoE-expert/adapter-union = real-weight E8 max-merge), not delta-avg. + +**2026-07-05 — MoE-expert / union recombination (`llm_moe`): E8's `max` vs `mean` in real weights.** +- `src/llm/moe.py`: router (oracle + training-free nearest-centroid over base embeddings) + MoE + generate + router-free per-module `max_merge`. `kind: llm_moe` reuses the cached specialists. +- **0.5B result:** routing beats fusion decisively — overall 0.74/worst 0.43 vs soup 0.64/0.26, no + dilution (recovers each specialist's own-family peak). Riders: learned router trivially perfect + (1.00, lexically-separable families) and static `max_merge` a poor union (0.46, not input-adaptive). +- `configs/llm/{moe,moe_hpc}.yaml`, `figures/plot_llm_moe.py`, README, +2 tests (127 green), + `hpc/llm_moe.pbs`. +- **7B firm-up (`llm_moe_hpc`, CX3 L40S, 9 min): the ordering FLIPS.** At 7B fusion wins — + soup 0.87 > routing 0.84 > max_merge 0.78 (0.5B had routing 0.74 > soup 0.64). Routing is capped at + the best parent per family; fusion *composes beyond* it at a capable base (soup lists 0.62 > + spec 0.57). So "merge, don't average" is a **weak-base law**, not universal — union wins under + dilution (0.5B), fusion wins under composition (7B). Refines E8. Next: fusion + offspring-selection + (directed sex). `results/llm_moe_hpc/` README + regime-aware figure. diff --git a/tests/test_llm.py b/tests/test_llm.py index f6443a6..c02a327 100644 --- a/tests/test_llm.py +++ b/tests/test_llm.py @@ -8,6 +8,9 @@ accepts correct answers (including verbose model phrasings) and rejects wrong on from __future__ import annotations +import numpy as np + +from llm.moe import learned_routes from llm.tasks import FAMILIES, make_tasks, verify @@ -37,3 +40,29 @@ def test_verifier_rejects_wrong(): assert not verify(wrong, t) lt = next(x for x in make_tasks("lists", 30, seed=6) if x.answer.startswith("[")) assert not verify("[9, 9, 9]", lt) or lt.answer == "[9, 9, 9]" + + +def test_learned_router_assigns_nearest_centroid(): + # Three well-separated families in a 4-D "embedding" space; the nearest-centroid router + # (the MoE expert-selection gene) must route each test prompt to its own family's specialist. + rng = np.random.default_rng(0) + fams = ["lists", "strings", "arith"] + anchors = {"lists": [5, 0, 0, 0], "strings": [0, 5, 0, 0], "arith": [0, 0, 5, 0]} + train_emb = np.array([anchors[f] for f in fams for _ in range(8)], dtype=float) + train_emb += rng.normal(scale=0.1, size=train_emb.shape) + train_fam = np.array([f for f in fams for _ in range(8)]) + test_fam = np.array(["arith", "lists", "strings", "arith"]) + test_emb = np.array([anchors[f] for f in test_fam], dtype=float) + rng.normal(scale=0.1, size=(4, 4)) + routes = learned_routes(train_emb, train_fam, test_emb, fams) + assert [fams[r] for r in routes] == list(test_fam) # each routed to its own family + + +def test_learned_router_is_cosine_scale_invariant(): + # Cosine routing must ignore prompt-embedding magnitude (long vs short prompts): a test point on a + # family's ray routes there regardless of its norm. + fams = ["a", "b"] + train_emb = np.array([[1.0, 0.0], [1.0, 0.0], [0.0, 1.0], [0.0, 1.0]]) + train_fam = np.array(["a", "a", "b", "b"]) + test_emb = np.array([[10.0, 0.0], [0.0, 0.01]]) # very different magnitudes + routes = learned_routes(train_emb, train_fam, test_emb, fams) + assert [fams[r] for r in routes] == ["a", "b"]