Giorgio Gilestro
5a23ddaf2a
llm_speciation (new kind; src/llm/speciation.py): E13 in LLM weights. LoRA children share the frozen base's coordinates, so merge failure is functional by construction. CONFLICT (ambiguous sort prompts learned under opposite conventions — the BDM structure): function-specific hybrid breakdown — merged coherence 0.02-0.08 falls below BOTH parents (~0.2) on the conflicted function; and in the de-confounded `add` design (private budget fixed, conflict added on top; 3 seeds after a single-seed pilot showed one anomalous point) the merge's private-family accuracy shows NO trend with conflict — the damage is surgical, not global. DURATION (over-trained disjoint specialists, 1->12 epochs): the merge improves (0.84->0.94) and stays above the best parent — the MLP "no emergent isolation" null generalises; relevant to the expert-training-duration report (2607.11997), with the epistasis prediction left to the decisive experiment. Multi-seed firm-up (seeds threaded into specialist caches; `seeds:` list support in the runner; fixed test sets): all three recombination claims hold with CIs — merges beat every specialist (5 seeds, ties 0.647±0.027 > best spec 0.592±0.009; worst-family 0.28 vs <=0.16); union 0.274±0.026 > fusion 0.174±0.102 on hard (3 seeds); directed 0.221±0.026 > soup. NEW finding: fusion is seed-FRAGILE where headroom exists (CI ±0.10) while routing/directed selection are stable (±0.026) — the union/selection operators win on reliability, not just mean. Figures (llm_speciation 3-panel; llm_seeds 3-panel with 95% CI), READMEs, +1 convention test (150 green), make llm-speciation / llm-seeds targets. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkRLcc18rwT2Lysu6PbG7v
88 lines
3.5 KiB
Python
88 lines
3.5 KiB
Python
"""Multi-seed LLM robustness figure — the recombination claims with error bars.
|
||
|
||
Aggregates the three multi-seed 0.5B experiments (fixed test sets, training seed varied) into one
|
||
figure with 95% CIs over seeds:
|
||
|
||
(A) llm_merge_seeds — Fisher–Muller: merged specialists vs the best single specialist, overall and
|
||
worst-family (5 seeds).
|
||
(B) llm_moe_hard_seeds — union (routing) vs fusion (soup/ties) on the hard benchmark (3 seeds).
|
||
(C) llm_directed_hard_seeds — directed offspring selection vs the a-priori soup, hard (3 seeds).
|
||
|
||
Usage: python figures/plot_llm_seeds.py
|
||
"""
|
||
|
||
from __future__ import annotations
|
||
|
||
import sys
|
||
from pathlib import Path
|
||
|
||
import numpy as np
|
||
import matplotlib.pyplot as plt
|
||
|
||
sys.path.insert(0, str(Path(__file__).parent))
|
||
from _figlib import load_bundle, savefig # noqa: E402
|
||
|
||
|
||
def _agg(df, models, metric):
|
||
"""Per-model mean and 95% CI over seeds for one metric."""
|
||
out = []
|
||
for m in models:
|
||
v = df[(df["model"] == m) & (df["metric"] == metric)].groupby("seed")["accuracy"].mean()
|
||
out.append((v.mean(), 1.96 * v.std(ddof=1) / max(1, np.sqrt(len(v)))))
|
||
return out
|
||
|
||
|
||
def _panel(ax, df, models, labels, title):
|
||
x = np.arange(len(models))
|
||
for off, metric, color in ((-0.17, "overall", "#2c7fb8"), (0.17, "worst_family", "#d62728")):
|
||
vals = _agg(df, models, metric)
|
||
ax.bar(x + off, [v for v, _ in vals], 0.34, yerr=[e for _, e in vals],
|
||
capsize=3, color=color, label=metric)
|
||
ax.set_xticks(x); ax.set_xticklabels(labels, fontsize=8)
|
||
ax.set(ylabel="verifier accuracy", ylim=(0, 1.0), title=title)
|
||
ax.legend(frameon=False, fontsize=8)
|
||
|
||
|
||
def _best_spec(df):
|
||
"""Synthesise a best-single-specialist row set per seed (max over spec_* by overall)."""
|
||
specs = sorted(m for m in df["model"].unique() if m.startswith("spec_"))
|
||
rows = []
|
||
for s, sub in df.groupby("seed"):
|
||
ov = {m: sub[(sub["model"] == m) & (sub["metric"] == "overall")]["accuracy"].mean()
|
||
for m in specs}
|
||
best = max(ov, key=ov.get)
|
||
b = sub[sub["model"] == best].copy()
|
||
b["model"] = "best_specialist"
|
||
rows.append(b)
|
||
import pandas as pd
|
||
return pd.concat([df] + rows, ignore_index=True)
|
||
|
||
|
||
def main() -> None:
|
||
fig, axes = plt.subplots(1, 3, figsize=(16, 4.8))
|
||
|
||
df, _ = load_bundle("results/llm_merge_seeds")
|
||
_panel(axes[0], _best_spec(df), ["base", "best_specialist", "merge_soup", "merge_ties"],
|
||
["base", "best\nspecialist", "merge\n(soup)", "merge\n(ties)"],
|
||
"(A) Fisher–Muller with error bars\n(5 seeds, easy benchmark, 0.5B)")
|
||
|
||
df, _ = load_bundle("results/llm_moe_hard_seeds")
|
||
_panel(axes[1], _best_spec(df), ["best_specialist", "merge_soup", "merge_ties", "moe_oracle",
|
||
"moe_learned"],
|
||
["best\nspecialist", "fusion\n(soup)", "fusion\n(ties)", "union\n(route,oracle)",
|
||
"union\n(route,learned)"],
|
||
"(B) union vs fusion, hard benchmark\n(3 seeds, 0.5B)")
|
||
|
||
df, _ = load_bundle("results/llm_directed_hard_seeds")
|
||
_panel(axes[2], df, ["merge_soup", "directed_overall", "directed_balanced"],
|
||
["a-priori soup", "directed\n(overall)", "directed\n(balanced)"],
|
||
"(C) directed offspring selection, hard\n(3 seeds, 0.5B)")
|
||
|
||
fig.suptitle("The LLM recombination claims are seed-robust (fixed test sets; training seed varied; 95% CI)",
|
||
y=1.03, fontsize=12)
|
||
fig.tight_layout()
|
||
savefig(fig, "results/llm_merge_seeds", "llm_seeds")
|
||
|
||
|
||
if __name__ == "__main__":
|
||
main()
|