llm_merge_hpc: the 7B firm-up makes the Fisher-Muller sign decisive

Re-ran the specialist-merge experiment at a capable base (Qwen2.5-7B-Instruct,
200 tests/family) on one L40S GPU of Imperial's CX3 HPC (8 min walltime). The
two caveats the 0.5B prototype left marginal are now resolved:

- "exceeds every parent overall" is clean: both merges 0.87 vs best specialist
  0.77 (+10 pts), and above every specialist on every family.
- dilution vanishes: at 0.5B averaging diluted the lists-specialist
  (0.43->0.26); at 7B the merge beats it (0.62>0.57). Dilution was a
  small-model artefact -- a capable base composes rather than dilutes, which
  softens E4's "merge, don't average" once the parents are strong.

The figure title is now data-driven (reports ">" for 7B, "~" for 0.5B).
Adds the hpc/ smoke job script and the llm_merge walltime trim. Results synced
to results/llm_merge_hpc/ (parquet gitignored per the reproducibility contract).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Giorgio Gilestro 2026-07-05 17:32:51 +01:00
parent 38bb252c20
commit 585264d0b4
12 changed files with 140 additions and 22 deletions

View file

@ -67,12 +67,17 @@ def main(results_dir: str = "results/llm_merge") -> None:
edgecolor="white")
ax.set_xticks(x2); ax.set_xticklabels([labels[m] for m in models], rotation=25, ha="right",
fontsize=8)
ax.set(ylabel="accuracy", title="Overall (solid) vs worst-family (hatched): the merge "
"clearly wins\nworst-family (balance); overall it matches the best specialist")
ax.set(ylabel="accuracy", title="Overall (solid) vs worst-family (hatched):\n"
"the recombined model vs the best single specialist")
ax.legend(frameon=False, fontsize=9)
fig.suptitle("llm_merge — recombining decorrelated specialist LLMs gives the only model competent "
f"across all families (balance); overall parity ({cfg['base_model']})", y=1.0, fontsize=12)
best_spec = max(_acc(df, m, "overall") for m in specialists)
best_merge = max(_acc(df, m, "overall") for m in merges)
verdict = (f"recombined {best_merge:.2f} > best specialist {best_spec:.2f} overall"
if best_merge > best_spec + 0.005 else
f"recombined {best_merge:.2f} ≈ best specialist {best_spec:.2f} overall")
fig.suptitle(f"llm_merge — recombining decorrelated specialist LLMs: {verdict} "
f"({cfg['base_model'].split('/')[-1]})", y=1.0, fontsize=12)
fig.tight_layout()
savefig(fig, results_dir, "llm_merge")