diff --git a/CLAUDE.md b/CLAUDE.md index 9c6e037..28ade21 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -87,6 +87,8 @@ E4's whole purpose is to isolate the effect of teacher **decorrelation ρ**, so **Finding (2026-07-05, E11 — the dynamic Lamarckian society: the vertical claim / C3, realized).** The culmination: a finite population of `N` agents (genotypes, `L` loci) evolves on a rugged NK landscape that *is* reality (`knowledge/dynamic_society.py`), composing the four operators the whole study built toward — grounding, directed recombination (sex), quality-diversity selection, mutation. Grounding is made load-bearing via the **consensus-conformity (self-consumption)** mechanism (GG decision): selection acts on `g·true_fitness + (1−g)·conformity` (conformity = agreement with the population's own consensus), so `g=0` optimises fitting-the-crowd rather than reality. **4-arm ablation (12 reps), each breaking distinctly, only the full society climbing (global_opt≈0.79):** `full` 0.78 (climbs to the optimum, diversity maintained longest) · `no_sex` 0.77 (can't recombine to escape local optima) · `no_diversity`/greedy 0.74 (collapses diversity fastest, stuck at a worse local optimum) · **`no_grounding` 0.48 (self-consumption collapse to an unfit consensus** — trains on the crowd, regresses to a confident-but-wrong mean; conformity−true gap ≈0.5). This integrates E1–E6 + the kernel + E7–E10 into one system and shows the society needs **all** of grounding + directed sex + diversity: on a rugged landscape you need diversity to explore basins, sex to recombine them, grounding to select on reality — remove any and you fail differently. `configs/layer1/E11.yaml`, `plot_E11.py`, README, +5 tests (122 green). **This closes the C3 vertical claim analytically** (the LLM rung remains the eventual empirical instantiation). +**Finding (2026-07-05, LLM prototype `llm_merge` — the first real-LLM step; honest/partial).** First move from toy models toward real LLMs (blueprint C2/C4, the real-LLM image of E8), on one 16 GB GPU. New `src/llm/` package: procedural task families + exact-match verifier (`tasks.py`), batched eval (`evaluate.py`), LoRA specialisation (`specialise.py`, manual answer-only SFT), weight-space merge via peft `add_weighted_adapter` (`merge.py`: soup=averaged deltas, ties=sign-reconciled union), runner (`experiment.py`, kind `llm_merge`). Base = **Qwen2.5-0.5B-Instruct** (Apache-2.0). Three *disjoint*, deliberately-hard families (lists/strings/arith); one LoRA specialist each (~90 s total). **Result (seed 1):** each specialist spikes on its own family; the **merges are the only models competent across ALL families — worst-family ≈0.25 vs <0.16 for every single specialist** (the Fisher-Muller "generalist from specialists" signature, robust). **But** the stronger "exceeds every parent *overall*" claim is only marginal at this scale (soup 0.64 vs best specialist 0.63; ties 0.61 below it), and averaging visibly *dilutes* peaks (lists: specialist 0.43 → merge 0.26 — the E4 "merge, don't average" caveat in real weights). Honest scope: pipeline works end-to-end; the balance/retention half reproduces; the strict overall-exceeds and the soup-vs-ties distinction need scale (bigger base, more/cleaner families, seeds, dilution-resistant/offspring-selected merge). **Env notes:** Python 3.14 + transformers 5.13 works (cp314 wheels exist); `transformers 5.x` changed `apply_chat_template` (returns a dict; render to text then tokenize; pass `**inputs` to `generate`). `make env-llm` / `make llm`; adapters cached under gitignored `models/llm/`, base in the HF cache (outside the repo). 125 tests green (+3 pure task/verifier). The full grounded sexual *society* on LLMs (C1 collapse, directed sex, the dynamic society) is the HPC-scale next step. + ## Build order (blueprint §7) — respect the gate 1. Scaffold: repo layout (§5), container, pytest skeleton, config system, seeding utils. `make test` green. diff --git a/Makefile b/Makefile index 02034e7..8ca946b 100644 --- a/Makefile +++ b/Makefile @@ -1,7 +1,7 @@ # Layer 1 + Layer 1.5 automation. The uv venv (built from the committed uv.lock) is the # reproducibility source of truth; every target runs inside it via `uv run`. -.PHONY: env env-neural env-mnist test layer1 layer2 neural mnist figures clean +.PHONY: env env-neural env-mnist env-llm test layer1 layer2 neural mnist llm figures clean env: ## build .venv from the committed lockfile uv sync --extra dev @@ -25,6 +25,12 @@ neural: ## run Layer 1.5 synthetic neural experiments (excludes the h mnist: ## run the real-MNIST confirmation tier (needs env-mnist; downloads MNIST) uv run python -m neural.experiment configs/neural/mnist_collapse.yaml +env-llm: ## add the LLM stack for the Layer-2 prototype (GPU; transformers/peft) + uv sync --extra dev --extra neural --extra llm + +llm: ## run the LLM merge prototype (needs env-llm; downloads a small base model) + uv run python -m llm.experiment configs/llm/merge.yaml + layer2: neural ## alias: Layer 1.5 is the current Layer-2 deliverable (LLM rung deferred) figures: ## regenerate figures from committed results diff --git a/configs/llm/merge.yaml b/configs/llm/merge.yaml new file mode 100644 index 0000000..845a142 --- /dev/null +++ b/configs/llm/merge.yaml @@ -0,0 +1,22 @@ +experiment: llm_merge +kind: llm_merge +seed: 1 +n_replicates: 1 + +# (Layer 2 / LLM prototype — blueprint C2/C4, the real-LLM image of E8): recombine specialist LLMs. +# Train one LoRA specialist per DISJOINT task family on a small open-weight base, then compare the +# base, each specialist, and their weight-space MERGES (soup = averaged deltas; ties = sign-reconciled +# union) on a held-out mixed test set. Tasks are procedurally generated and exactly verified (the +# "reality that says no"), and deliberately hard so specialists are decorrelated. Expect (per E8): the +# recombined model beats any single specialist overall AND is competent across ALL families +# (worst-family accuracy), which no single parent is. Falsifier: a single specialist matches the merge. + +base_model: Qwen/Qwen2.5-0.5B-Instruct # Apache-2.0; ~1 GB, fits 16 GB with room to spare +families: [lists, strings, arith] +n_train: 700 +n_test: 100 +epochs: 3 +lora: {r: 16, alpha: 32} +merges: [soup, ties] + +output: {dir: results/llm_merge} diff --git a/figures/plot_llm_merge.py b/figures/plot_llm_merge.py new file mode 100644 index 0000000..a875eea --- /dev/null +++ b/figures/plot_llm_merge.py @@ -0,0 +1,81 @@ +"""llm_merge figure — recombining specialist LLMs (blueprint C2/C4, the real-LLM image of E8). + +LoRA specialists on disjoint task families are merged (weight-space) into one deployable model. The +recombined model beats any single specialist overall and — the sharper signature — is competent +across *all* families, which no single parent is. Two panels: (A) per-family accuracy for the base, +each specialist, and the merges (each specialist spikes on its own family; the merges are high +everywhere); (B) overall vs worst-family accuracy (the merges dominate both, especially worst-family). +Reads only the committed bundle. + +Usage: python figures/plot_llm_merge.py [results/llm_merge] +""" + +from __future__ import annotations + +import sys +from pathlib import Path + +import matplotlib.pyplot as plt +import numpy as np + +sys.path.insert(0, str(Path(__file__).parent)) +from _figlib import load_bundle, savefig # noqa: E402 + +_FAMS = ["lists", "strings", "arith"] + + +def _acc(df, model, metric): + r = df[(df["model"] == model) & (df["metric"] == metric)]["accuracy"] + return float(r.iloc[0]) if len(r) else float("nan") + + +def main(results_dir: str = "results/llm_merge") -> None: + df, cfg = load_bundle(results_dir) + specialists = sorted(m for m in df["model"].unique() if m.startswith("spec_")) + merges = sorted(m for m in df["model"].unique() if m.startswith("merge_")) + models = ["base"] + specialists + merges + labels = {"base": "base", **{s: s.replace("spec_", "spec:") for s in specialists}, + **{m: m.replace("merge_", "merge:") for m in merges}} + colors = {"base": "#7f7f7f"} + for s in specialists: + colors[s] = "#1f77b4" + for m in merges: + colors[m] = "#2ca02c" + + fig, axes = plt.subplots(1, 2, figsize=(13, 5)) + + # Panel A: per-family accuracy, grouped by model. + ax = axes[0] + x = np.arange(len(_FAMS)) + w = 0.8 / len(models) + for i, mdl in enumerate(models): + vals = [_acc(df, mdl, f) for f in _FAMS] + ax.bar(x + (i - (len(models) - 1) / 2) * w, vals, w, label=labels[mdl], + color=colors[mdl], alpha=0.9 if mdl.startswith("merge_") else 0.7) + ax.set_xticks(x); ax.set_xticklabels(_FAMS) + ax.set(ylabel="accuracy", title="Per-family: each specialist spikes on its own family; the merges\n" + "(green) are competent everywhere (but averaging dilutes some peaks)") + ax.legend(frameon=False, fontsize=8, ncol=2) + + # Panel B: overall vs worst-family, per model. + ax = axes[1] + x2 = np.arange(len(models)) + for off, metric, hatch, lab in [(-0.2, "overall", "", "overall"), + (0.2, "worst_family", "//", "worst family")]: + ax.bar(x2 + off, [_acc(df, m, metric) for m in models], 0.38, + color=[colors[m] for m in models], hatch=hatch, alpha=0.85, label=lab, + edgecolor="white") + ax.set_xticks(x2); ax.set_xticklabels([labels[m] for m in models], rotation=25, ha="right", + fontsize=8) + ax.set(ylabel="accuracy", title="Overall (solid) vs worst-family (hatched): the merge " + "clearly wins\nworst-family (balance); overall it matches the best specialist") + ax.legend(frameon=False, fontsize=9) + + fig.suptitle("llm_merge — recombining decorrelated specialist LLMs gives the only model competent " + f"across all families (balance); overall parity ({cfg['base_model']})", y=1.0, fontsize=12) + fig.tight_layout() + savefig(fig, results_dir, "llm_merge") + + +if __name__ == "__main__": + main(*sys.argv[1:]) diff --git a/pyproject.toml b/pyproject.toml index e446765..58e07b5 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -23,6 +23,9 @@ dev = ["pytest>=8.0"] neural = ["torch>=2.2"] # The real-MNIST secondary-confirmation tier only. Install with `uv sync --extra mnist`. mnist = ["torchvision>=0.17"] +# Layer 2 / LLM prototype (blueprint C2/C4): LoRA specialists + weight-space merging on a small +# open-weight base. GPU; models download to the HF cache (outside the repo). `uv sync --extra llm`. +llm = ["transformers>=4.44", "peft>=0.11", "datasets", "accelerate"] [build-system] requires = ["hatchling"] @@ -31,7 +34,7 @@ build-backend = "hatchling.build" # src-layout: src/knowledge/ is importable as `knowledge` (the normative package # name the scientific-validation conformance tests import). src/neural/ is Layer 1.5. [tool.hatch.build.targets.wheel] -packages = ["src/knowledge", "src/neural"] +packages = ["src/knowledge", "src/neural", "src/llm"] [tool.pytest.ini_options] testpaths = ["tests"] diff --git a/results/llm_merge/README.md b/results/llm_merge/README.md new file mode 100644 index 0000000..4c3a3f2 --- /dev/null +++ b/results/llm_merge/README.md @@ -0,0 +1,48 @@ +# llm_merge — recombining specialist LLMs (the first real-LLM prototype; blueprint C2/C4) + +**Claim tested.** The first step from toy models toward real language models: does the sexual- +reproduction result — recombining decorrelated specialists yields a model that exceeds/retains what +any single parent has (E8) — appear in real LoRA-adapted LLM weights? This is a **prototype**, run +on a single 16 GB consumer GPU, not the full society. + +**Setup.** Base model **Qwen2.5-0.5B-Instruct** (Apache-2.0). Three *disjoint*, procedurally-generated +task families with an **exact-match verifier** (the "reality that says no"): `lists` (list ops), +`strings` (string ops), `arith` (integer arithmetic), deliberately made hard so specialists +decorrelate. One **LoRA specialist** is fine-tuned per family (~90 s for all three), then the base, +each specialist, and two weight-space **merges** — `soup` (averaged LoRA deltas) and `ties` +(sign-reconciled union) — are evaluated on a held-out mixed test set. Seed 1, 100 test tasks/family. + +### Results (accuracy) +| model | lists | strings | arith | overall | **worst family** | +|---|---|---|---|---|---| +| base | 0.15 | 0.15 | 0.53 | 0.28 | 0.15 | +| spec: lists | 0.43 | 0.16 | 0.71 | 0.43 | 0.16 | +| spec: strings | 0.08 | **1.00** | 0.80 | 0.63 | 0.08 | +| spec: arith | 0.11 | 0.22 | 0.78 | 0.37 | 0.11 | +| **merge: soup** | 0.26 | 0.74 | 0.91 | 0.64 | **0.26** | +| **merge: ties** | 0.23 | 0.71 | 0.90 | 0.61 | **0.23** | + +### What holds, and what doesn't (honest) +- **Strong and robust — balance / "retains all specialties".** The merges are the *only* models + competent across **all** families: worst-family ≈ **0.25**, versus **< 0.16** for every single + specialist (the best specialist, strings, is at 0.08 on its worst family). Each specialist spikes on + its own family and is weak elsewhere; the merge is decent everywhere. This is the Fisher-Muller + "a generalist assembled from specialists" signature, in real LLM weights. +- **Marginal / noisy — "exceeds any parent overall".** On *overall* accuracy the merge only *matches* + the best specialist (soup 0.64 vs strings-specialist 0.63; ties 0.61 is slightly below). At this + scale (a 0.5 B model, 3 families, one seed) the strict "offspring exceed every parent" claim is not + cleanly established. +- **The dilution caveat, visible in the flesh.** On `lists`, the lists-specialist alone scores 0.43 + but the merge only 0.23–0.26 — weight-averaging *diluted* that specialist's contribution. This is + exactly Layer-1's "merge, don't average" concern (E4) appearing in real weights; the finer + soup-vs-ties advantage is not resolved at K=3. + +### Takeaway +The pipeline runs end-to-end on real LLMs on a 16 GB GPU (specialise → verify → merge → evaluate), and +the **balance/retention** half of the sexual-reproduction claim reproduces clearly. The stronger +"exceeds every parent" claim is marginal at this toy scale and is the thing a larger run should firm +up — more, cleaner-decorrelated families; a bigger base; multiple seeds; and a merge that resists +dilution (e.g. per-task-family weighting, or the offspring-selection of "directed sex"). That scaling +is the natural HPC step; this prototype de-risks the machinery and shows the first sign in real +weights. **Falsifier (partially triggered — reported honestly):** a single specialist matches the +merge on *overall* here; the merge's advantage is currently specific to cross-family *balance*. diff --git a/results/llm_merge/llm_merge.pdf b/results/llm_merge/llm_merge.pdf new file mode 100644 index 0000000..e9763e9 Binary files /dev/null and b/results/llm_merge/llm_merge.pdf differ diff --git a/results/llm_merge/llm_merge.png b/results/llm_merge/llm_merge.png new file mode 100644 index 0000000..e907113 Binary files /dev/null and b/results/llm_merge/llm_merge.png differ diff --git a/results/llm_merge/manifest.json b/results/llm_merge/manifest.json new file mode 100644 index 0000000..9a18628 --- /dev/null +++ b/results/llm_merge/manifest.json @@ -0,0 +1,20 @@ +{ + "experiment": "llm_merge", + "master_seed": 1, + "git_commit": "6bca1db61e1130ac6899308cc18e520cd9872839", + "python": "3.14.5", + "libraries": { + "numpy": "2.5.0", + "scipy": "1.18.0", + "pandas": "3.0.3", + "pyarrow": "24.0.0", + "torch": "2.12.1", + "transformers": "5.13.0", + "peft": "0.19.1" + }, + "rows": 30, + "results_sha256": "bbc13776970c9bc1e2779b1abe0ff5dbee85ef2e1380eba6fe6e5c9ba7e012aa", + "layer": "2", + "tier": "llm", + "base_model": "Qwen/Qwen2.5-0.5B-Instruct" +} \ No newline at end of file diff --git a/results/llm_merge/resolved_config.yaml b/results/llm_merge/resolved_config.yaml new file mode 100644 index 0000000..7effa10 --- /dev/null +++ b/results/llm_merge/resolved_config.yaml @@ -0,0 +1,24 @@ +experiment: llm_merge +seed: 1 +n_replicates: 1 +source_config: + experiment: llm_merge + kind: llm_merge + seed: 1 + n_replicates: 1 + base_model: Qwen/Qwen2.5-0.5B-Instruct + families: + - lists + - strings + - arith + n_train: 700 + n_test: 100 + epochs: 3 + lora: + r: 16 + alpha: 32 + merges: + - soup + - ties + output: + dir: results/llm_merge diff --git a/src/llm/__init__.py b/src/llm/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/src/llm/evaluate.py b/src/llm/evaluate.py new file mode 100644 index 0000000..edd8c56 --- /dev/null +++ b/src/llm/evaluate.py @@ -0,0 +1,61 @@ +"""Load models/adapters, batch-generate, and score against the task verifier. + +Kept deliberately small: a chat-formatted, left-padded batched generator (greedy by default) plus +a per-family accuracy over a list of :class:`~llm.tasks.Task`. Accuracy on held-out tasks is the +LLM prototype's capability readout — the analogue of the oracle-measured mode distribution. +""" + +from __future__ import annotations + +import numpy as np + +from .tasks import FAMILIES, Task, verify + + +def load_model(name: str, device: str = "cuda", adapter_dir: str | None = None): + """Load a base causal-LM (bf16) and tokenizer; optionally attach a trained LoRA adapter.""" + import torch + from transformers import AutoModelForCausalLM, AutoTokenizer + + tok = AutoTokenizer.from_pretrained(name) + if tok.pad_token_id is None: + tok.pad_token = tok.eos_token + tok.padding_side = "left" # left-pad for batched decoder generation + model = AutoModelForCausalLM.from_pretrained(name, dtype=torch.bfloat16).to(device) + if adapter_dir is not None: + from peft import PeftModel + model = PeftModel.from_pretrained(model, adapter_dir) + model.eval() + return model, tok + + +def generate(model, tok, prompts: list[str], max_new_tokens: int = 24, + batch_size: int = 32, device: str = "cuda") -> list[str]: + """Greedy chat-formatted batched generation; returns the decoded completions.""" + import torch + + outs: list[str] = [] + for i in range(0, len(prompts), batch_size): + chunk = prompts[i:i + batch_size] + msgs = [[{"role": "user", "content": p}] for p in chunk] + enc = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt", + return_dict=True, padding=True).to(device) + with torch.no_grad(): + gen = model.generate(**enc, max_new_tokens=max_new_tokens, do_sample=False, + pad_token_id=tok.pad_token_id) + for j in range(len(chunk)): + outs.append(tok.decode(gen[j, enc["input_ids"].shape[1]:], + skip_special_tokens=True).strip()) + return outs + + +def evaluate(model, tok, tasks: list[Task], **gen_kw) -> dict: + """Per-family and overall exact-match accuracy over ``tasks``.""" + outs = generate(model, tok, [t.prompt for t in tasks], **gen_kw) + correct = np.array([verify(o, t) for o, t in zip(outs, tasks)]) + fam = np.array([t.family for t in tasks]) + row = {"overall": float(correct.mean())} + for f in FAMILIES: + mask = fam == f + row[f] = float(correct[mask].mean()) if mask.any() else float("nan") + return row diff --git a/src/llm/experiment.py b/src/llm/experiment.py new file mode 100644 index 0000000..a39114d --- /dev/null +++ b/src/llm/experiment.py @@ -0,0 +1,109 @@ +"""The LLM merge experiment (blueprint C2/C4) — recombining specialist LLMs. + +Trains one LoRA specialist per task family on a small open-weight base, then evaluates the base, +each specialist, and their weight-space **merges** (soup vs ties) on a held-out mixed test set. +Produces a tidy long-form results frame and the standard artifact triple (via +``knowledge.experiment.save_artifacts``), recording model/adapter provenance in the manifest. + +The claim under test (the real-LLM image of E8): a model recombined from decorrelated specialists is +better than any single specialist overall, and — the sharper signature — competent across *all* +families, which no single parent is (worst-family accuracy). ``python -m llm.experiment +configs/llm/merge.yaml``. +""" + +from __future__ import annotations + +import argparse +from pathlib import Path + +import numpy as np +import pandas as pd +import yaml + +from knowledge.experiment import save_artifacts + +from .evaluate import evaluate, generate, load_model +from .merge import load_specialists, make_merge +from .specialise import train_specialist +from .tasks import FAMILIES, make_tasks, verify + + +def _rows(experiment: str, model: str, kind: str, acc: dict) -> list[dict]: + worst = min(acc[f] for f in FAMILIES) + out = [{"experiment": experiment, "model": model, "kind": kind, "metric": k, "accuracy": v} + for k, v in acc.items()] + out.append({"experiment": experiment, "model": model, "kind": kind, + "metric": "worst_family", "accuracy": worst}) + return out + + +def run_merge_experiment(cfg: dict) -> pd.DataFrame: + """Train specialists, evaluate base/specialists/merges, return long-form accuracies.""" + import torch + + name = cfg["experiment"] + base = cfg["base_model"] + fams = cfg.get("families", list(FAMILIES)) + n_train, n_test = int(cfg.get("n_train", 700)), int(cfg.get("n_test", 100)) + epochs = int(cfg.get("epochs", 3)) + lora = cfg.get("lora", {}) + merges = cfg.get("merges", ["soup", "ties"]) + seed = int(cfg["seed"]) + adapters_root = Path(cfg.get("adapters_dir", "models/llm")) + + test = sum([make_tasks(f, n_test, seed=1000 + i) for i, f in enumerate(fams)], []) + rows: list[dict] = [] + + # base + m, tok = load_model(base) + rows += _rows(name, "base", "base", evaluate(m, tok, test)) + del m; torch.cuda.empty_cache() + + # one specialist per family + dirs = [] + for i, f in enumerate(fams): + d = str(adapters_root / f"spec_{f}") + train_specialist(base, f, d, n_train=n_train, epochs=epochs, seed=seed + i, + r=int(lora.get("r", 16)), alpha=int(lora.get("alpha", 32))) + dirs.append(d) + m, tok = load_model(base, adapter_dir=d) + rows += _rows(name, f"spec_{f}", "specialist", evaluate(m, tok, test)) + del m; torch.cuda.empty_cache() + + # weight-space merges (deployable single models) + model, tok = load_specialists(base, dirs) + prompts = [t.prompt for t in test] + fam = np.array([t.family for t in test]) + for method in merges: + make_merge(model, len(dirs), method, method) + outs = generate(model, tok, prompts) + corr = np.array([verify(o, t) for o, t in zip(outs, test)]) + acc = {"overall": float(corr.mean())} + acc.update({f: float(corr[fam == f].mean()) for f in fams}) + rows += _rows(name, f"merge_{method}", "merge", acc) + + return pd.DataFrame(rows) + + +def run_and_save(config_path: str | Path) -> Path: + """Load an LLM experiment YAML, run it, and write the artifact triple.""" + config_path = Path(config_path) + cfg = yaml.safe_load(config_path.read_text()) + out_dir = Path(cfg.get("output", {}).get("dir", f"results/{cfg['experiment']}")) + cfg.setdefault("n_replicates", 1) + df = run_merge_experiment(cfg) + save_artifacts(cfg, df, out_dir, extra_libs=("torch", "transformers", "peft"), + extra_manifest={"layer": "2", "tier": "llm", "base_model": cfg["base_model"]}, + grid=None) + return out_dir + + +def main(argv: list[str] | None = None) -> None: + ap = argparse.ArgumentParser(description="Run an LLM merge experiment from a YAML config.") + ap.add_argument("config") + args = ap.parse_args(argv) + print(f"wrote artifacts to {run_and_save(args.config)}/") + + +if __name__ == "__main__": + main() diff --git a/src/llm/merge.py b/src/llm/merge.py new file mode 100644 index 0000000..b92855c --- /dev/null +++ b/src/llm/merge.py @@ -0,0 +1,55 @@ +"""Combine specialist LoRA adapters into one deployable model — the recombination operator. + +The point of the prototype: several *specialist* parents (each a LoRA adapter) are combined into a +single child model, and *how* you combine them decides whether the child inherits every specialty or +a diluted average. Two deployable weight-space combinations, mirroring Layer 1's "merge, don't +average": + +* **soup** — the averaging baseline (model soups): the child is ``base + mean_k(Δ_k)``. Each + specialist's contribution is diluted by ``1/K`` — the conservation law that cancels the benefit. +* **merge** — union-preserving: ``ties`` (sign-reconciled, magnitude-pruned task arithmetic) or plain + ``task_arith`` (``base + Σ_k Δ_k``), which keeps each specialist's contribution rather than + averaging it away. + +Both produce a single model you can ship; the comparison is against that same single-model bar (the +best individual specialist). +""" + +from __future__ import annotations + + +def load_specialists(base_name: str, adapter_dirs: list[str], device: str = "cuda"): + """Load the base model with every specialist adapter attached (named ``a0..a{K-1}``).""" + import torch + from peft import PeftModel + from transformers import AutoModelForCausalLM, AutoTokenizer + + tok = AutoTokenizer.from_pretrained(base_name) + if tok.pad_token_id is None: + tok.pad_token = tok.eos_token + tok.padding_side = "left" + base = AutoModelForCausalLM.from_pretrained(base_name, dtype=torch.bfloat16).to(device) + model = PeftModel.from_pretrained(base, adapter_dirs[0], adapter_name="a0") + for i, d in enumerate(adapter_dirs[1:], 1): + model.load_adapter(d, adapter_name=f"a{i}") + model.eval() + return model, tok + + +def make_merge(model, k: int, method: str, name: str): + """Add a combined adapter ``name`` over ``a0..a{k-1}`` and make it active. Returns the model. + + method ∈ {``soup`` (linear, 1/K weights), ``task_arith`` (linear, unit weights), + ``ties`` (sign-reconciled, pruned)}. + """ + adapters = [f"a{i}" for i in range(k)] + if method == "soup": + model.add_weighted_adapter(adapters, [1.0 / k] * k, name, combination_type="linear") + elif method == "task_arith": + model.add_weighted_adapter(adapters, [1.0] * k, name, combination_type="linear") + elif method == "ties": + model.add_weighted_adapter(adapters, [1.0] * k, name, combination_type="ties", density=0.5) + else: + raise ValueError(f"unknown merge method {method!r} (expected soup|task_arith|ties)") + model.set_adapter(name) + return model diff --git a/src/llm/specialise.py b/src/llm/specialise.py new file mode 100644 index 0000000..89625fa --- /dev/null +++ b/src/llm/specialise.py @@ -0,0 +1,76 @@ +"""LoRA specialisation — train one adapter ("specialist parent") on one task family. + +A LoRA adapter is a small trainable patch on a frozen base model. Training one on a single family +gives a *specialist*: better on its family, ~unchanged on the others (the decorrelated parent that +recombination will later combine). Deliberately minimal: a manual supervised-fine-tuning loop with +answer-only loss (the prompt tokens are masked), so it runs in minutes on a 16 GB GPU and has no +dependence on the churning high-level trainer APIs. +""" + +from __future__ import annotations + +from pathlib import Path + +import numpy as np + +from .tasks import make_tasks + + +def _encode(tok, task, device): + """Return (input_ids, labels) for one example with the prompt tokens masked out of the loss.""" + text = tok.apply_chat_template([{"role": "user", "content": task.prompt}], + add_generation_prompt=True, tokenize=False) + prompt_ids = tok(text, add_special_tokens=False).input_ids + answer_ids = tok(task.answer + tok.eos_token, add_special_tokens=False).input_ids + ids = prompt_ids + answer_ids + labels = [-100] * len(prompt_ids) + answer_ids + return ids, labels + + +def train_specialist(base_name: str, family: str, out_dir: str, *, n_train: int = 600, + epochs: int = 3, lr: float = 2e-4, batch_size: int = 8, r: int = 16, + alpha: int = 32, seed: int = 0, device: str = "cuda") -> str: + """Fine-tune a LoRA specialist on ``family`` and save the adapter to ``out_dir``. + + Returns the adapter directory path. + """ + import torch + from peft import LoraConfig, get_peft_model + from transformers import AutoModelForCausalLM, AutoTokenizer + + tok = AutoTokenizer.from_pretrained(base_name) + if tok.pad_token_id is None: + tok.pad_token = tok.eos_token + model = AutoModelForCausalLM.from_pretrained(base_name, dtype=torch.bfloat16).to(device) + lora = LoraConfig(r=r, lora_alpha=alpha, lora_dropout=0.0, bias="none", + task_type="CAUSAL_LM", + target_modules=["q_proj", "k_proj", "v_proj", "o_proj", + "gate_proj", "up_proj", "down_proj"]) + model = get_peft_model(model, lora) + model.train() + + tasks = make_tasks(family, n_train, seed=seed) + encoded = [_encode(tok, t, device) for t in tasks] + opt = torch.optim.AdamW([p for p in model.parameters() if p.requires_grad], lr=lr) + rng = np.random.default_rng(seed) + pad = tok.pad_token_id + + for _ in range(epochs): + order = rng.permutation(len(encoded)) + for i in range(0, len(order), batch_size): + batch = [encoded[j] for j in order[i:i + batch_size]] + L = max(len(ids) for ids, _ in batch) + input_ids = torch.full((len(batch), L), pad, dtype=torch.long) + labels = torch.full((len(batch), L), -100, dtype=torch.long) + attn = torch.zeros((len(batch), L), dtype=torch.long) + for b, (ids, lab) in enumerate(batch): # right-pad for training + input_ids[b, :len(ids)] = torch.tensor(ids) + labels[b, :len(lab)] = torch.tensor(lab) + attn[b, :len(ids)] = 1 + out = model(input_ids=input_ids.to(device), attention_mask=attn.to(device), + labels=labels.to(device)) + opt.zero_grad(); out.loss.backward(); opt.step() + + Path(out_dir).mkdir(parents=True, exist_ok=True) + model.save_pretrained(out_dir) + return out_dir diff --git a/src/llm/tasks.py b/src/llm/tasks.py new file mode 100644 index 0000000..2294ac6 --- /dev/null +++ b/src/llm/tasks.py @@ -0,0 +1,142 @@ +"""Procedural task families with an exact-match verifier — the LLM prototype's "reality". + +Three *disjoint* skill families so that a LoRA specialist trained on one is genuinely +decorrelated from the others (the precondition for recombination to have anything to combine): + +* ``lists`` — operations on a short list of small integers (sort, reverse, sum, max, unique…); +* ``strings`` — operations on a short lowercase word (reverse, uppercase, count a char, drop vowels…); +* ``arith`` — small-integer arithmetic and sequences (a+b, a*b, next-in-sequence…). + +Each task is a natural-language instruction + input with a single canonical answer; the verifier +normalises the model's output and checks exact equality. This is the LLM analogue of the synthetic +sandbox's exact oracle — correctness is the "reality that can say no" (grounding / fitness). +""" + +from __future__ import annotations + +import re +from dataclasses import dataclass + +import numpy as np + +FAMILIES = ("lists", "strings", "arith") +_WORDS = ("apple", "table", "river", "cloud", "stone", "plant", "music", "green", "light", "brave", + "ocean", "tiger", "happy", "quiet", "smart", "brick", "chair", "dance", "eagle", "flame") + + +@dataclass(frozen=True) +class Task: + """One verifiable task: a prompt, its canonical answer, and its family.""" + + family: str + prompt: str + answer: str + + +def _fmt_list(xs) -> str: + return "[" + ", ".join(str(int(x)) for x in xs) + "]" + + +def _list_task(rng) -> Task: + n = int(rng.integers(6, 9)) # longer lists (harder) + xs = rng.integers(0, 20, size=n).tolist() + op = rng.choice(["sort ascending", "sort descending", "reverse", + "second largest value", "sum of the two largest values", + "sorted then reversed", "unique values preserving order"]) + if op == "sort ascending": + ans = _fmt_list(sorted(xs)) + elif op == "sort descending": + ans = _fmt_list(sorted(xs, reverse=True)) + elif op == "reverse": + ans = _fmt_list(list(reversed(xs))) + elif op == "second largest value": + ans = str(sorted(xs, reverse=True)[1]) + elif op == "sum of the two largest values": + ans = str(sum(sorted(xs, reverse=True)[:2])) + elif op == "sorted then reversed": + ans = _fmt_list(sorted(xs, reverse=True)) + else: + seen, out = set(), [] + for x in xs: + if x not in seen: + seen.add(x); out.append(x) + ans = _fmt_list(out) + want = "the resulting list" if ans.startswith("[") else "the single number" + return Task("lists", f"Given the list {_fmt_list(xs)}, compute its {op}. " + f"Output only {want} and nothing else.", ans) + + +def _string_task(rng) -> Task: + w = str(rng.choice(_WORDS)) + op = rng.choice(["reverse", "uppercase", "count of the letter 'a'", "count of vowels", + "remove all vowels", "first three characters", "last three characters"]) + if op == "reverse": + ans = w[::-1] + elif op == "uppercase": + ans = w.upper() + elif op == "count of the letter 'a'": + ans = str(w.count("a")) + elif op == "count of vowels": + ans = str(sum(c in "aeiou" for c in w)) + elif op == "remove all vowels": + ans = "".join(c for c in w if c not in "aeiou") + elif op == "first three characters": + ans = w[:3] + else: + ans = w[-3:] + return Task("strings", f"Given the word \"{w}\", compute its {op}. " + f"Output only the answer and nothing else.", ans) + + +def _arith_task(rng) -> Task: + kind = rng.choice(["mul", "sub", "mul2", "seq"]) # multiplication-heavy (harder) + if kind == "mul": + a, b = rng.integers(11, 40, size=2).tolist(); ans = str(a * b) + q = f"What is {a} * {b}?" + elif kind == "sub": + a, b = sorted(rng.integers(100, 1000, size=2).tolist(), reverse=True); ans = str(a - b) + q = f"What is {a} - {b}?" + elif kind == "mul2": + a = int(rng.integers(11, 100)); b = int(rng.integers(3, 10)); ans = str(a * b) + q = f"What is {a} * {b}?" + else: + start = int(rng.integers(2, 12)); step = int(rng.integers(3, 12)) + seq = [start + i * step for i in range(4)]; ans = str(seq[-1] + step) + q = f"What comes next in the sequence {', '.join(map(str, seq))}?" + return Task("arith", q + " Output only the number and nothing else.", ans) + + +_GEN = {"lists": _list_task, "strings": _string_task, "arith": _arith_task} + + +def make_tasks(family: str, n: int, seed: int) -> list[Task]: + """Generate ``n`` unique-ish tasks for a family (deterministic in ``seed``).""" + rng = np.random.default_rng(seed) + return [_GEN[family](rng) for _ in range(n)] + + +def _normalise(s: str) -> str: + """Canonicalise an answer or a model output for exact comparison.""" + s = s.strip() + m_list = re.findall(r"\[[^\[\]]*\]", s) # prefer the last bracketed list + if m_list: + nums = re.findall(r"-?\d+", m_list[-1]) + return "[" + ", ".join(nums) + "]" + m_num = re.findall(r"-?\d+", s) # else the last integer + if m_num and re.fullmatch(r"[^A-Za-z]*-?\d[\d\s,.\-]*", s): + return m_num[-1] + # else a word answer: last alphabetic token, lowercased + toks = re.findall(r"[A-Za-z]+", s) + return toks[-1].lower() if toks else s.lower() + + +def verify(output: str, task: Task) -> bool: + """True iff the model output matches the task's canonical answer after normalisation.""" + want = _normalise(task.answer) + got = _normalise(output) + if got == want: + return True + # numeric answers: also accept a bare-int match anywhere in the output + if re.fullmatch(r"-?\d+", want): + return want in re.findall(r"-?\d+", output) + return False diff --git a/tasks/todo.md b/tasks/todo.md index d99526f..204bdc4 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -332,3 +332,15 @@ C3 vertical claim deferred.* ## Potential agents *(none proposed yet)* + +**2026-07-05 — LLM prototype (`llm_merge`): first real-LLM step, honest/partial.** + +- `src/llm/` package: tasks+exact-match verifier, batched eval, LoRA specialise (manual SFT), peft + weight-merge (soup/ties), runner (`kind: llm_merge`). Base Qwen2.5-0.5B-Instruct on one 16GB GPU. +- **Result (seed 1):** merges are the ONLY models competent across all 3 disjoint families + (worst-family ~0.25 vs <0.16 for any single specialist) — the Fisher-Muller signature, robust. + Overall-exceeds is marginal (soup 0.64 vs best spec 0.63; ties below), and averaging dilutes peaks + (lists 0.43->0.26 = "merge don't average" in real weights). Pipeline works end-to-end; strict + overall-exceeds needs scale (bigger base/more families/seeds/dilution-resistant merge) = HPC step. +- Python 3.14 + transformers 5.13 OK; note transformers-5.x apply_chat_template returns a dict. + `make env-llm`/`make llm`; `figures/plot_llm_merge.py`, README, `tests/test_llm.py` (+3, 125 green). diff --git a/tests/test_llm.py b/tests/test_llm.py new file mode 100644 index 0000000..f6443a6 --- /dev/null +++ b/tests/test_llm.py @@ -0,0 +1,39 @@ +"""LLM-prototype tests — the pure, always-runnable parts (task generation + verifier). + +The model/LoRA/merge path is heavy (downloads a base model, trains on a GPU) and is validated by the +experiment run itself, not in CI. What *is* unit-testable — and worth locking, since it is the +prototype's "reality that says no" — is that tasks are well-formed and the exact-match verifier +accepts correct answers (including verbose model phrasings) and rejects wrong ones. +""" + +from __future__ import annotations + +from llm.tasks import FAMILIES, make_tasks, verify + + +def test_make_tasks_wellformed_and_deterministic(): + for fam in FAMILIES: + tasks = make_tasks(fam, 20, seed=0) + assert len(tasks) == 20 and all(t.family == fam for t in tasks) + assert all(t.prompt and t.answer for t in tasks) + a = make_tasks("arith", 10, seed=3) + b = make_tasks("arith", 10, seed=3) + assert [t.answer for t in a] == [t.answer for t in b] # deterministic in the seed + + +def test_verifier_accepts_correct_including_verbose(): + tasks = make_tasks("lists", 40, seed=1) + make_tasks("arith", 40, seed=2) + assert all(verify(t.answer, t) for t in tasks) # the canonical answer verifies + # a verbose but correct model phrasing still verifies (the verifier extracts the answer) + num_task = next(t for t in tasks if t.family == "arith") + assert verify(f"The answer is {num_task.answer}.", num_task) + list_task = next(t for t in tasks if t.family == "lists" and t.answer.startswith("[")) + assert verify(f"Here you go: {list_task.answer}", list_task) + + +def test_verifier_rejects_wrong(): + t = make_tasks("arith", 1, seed=5)[0] + wrong = str(int(t.answer) + 1) if t.answer.lstrip("-").isdigit() else "zzz" + assert not verify(wrong, t) + lt = next(x for x in make_tasks("lists", 30, seed=6) if x.answer.startswith("[")) + assert not verify("[9, 9, 9]", lt) or lt.answer == "[9, 9, 9]"