llm_merge_hpc: the 7B firm-up makes the Fisher-Muller sign decisive
Re-ran the specialist-merge experiment at a capable base (Qwen2.5-7B-Instruct, 200 tests/family) on one L40S GPU of Imperial's CX3 HPC (8 min walltime). The two caveats the 0.5B prototype left marginal are now resolved: - "exceeds every parent overall" is clean: both merges 0.87 vs best specialist 0.77 (+10 pts), and above every specialist on every family. - dilution vanishes: at 0.5B averaging diluted the lists-specialist (0.43->0.26); at 7B the merge beats it (0.62>0.57). Dilution was a small-model artefact -- a capable base composes rather than dilutes, which softens E4's "merge, don't average" once the parents are strong. The figure title is now data-driven (reports ">" for 7B, "~" for 0.5B). Adds the hpc/ smoke job script and the llm_merge walltime trim. Results synced to results/llm_merge_hpc/ (parquet gitignored per the reproducibility contract). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
parent
38bb252c20
commit
585264d0b4
12 changed files with 140 additions and 22 deletions
|
|
@ -67,12 +67,17 @@ def main(results_dir: str = "results/llm_merge") -> None:
|
|||
edgecolor="white")
|
||||
ax.set_xticks(x2); ax.set_xticklabels([labels[m] for m in models], rotation=25, ha="right",
|
||||
fontsize=8)
|
||||
ax.set(ylabel="accuracy", title="Overall (solid) vs worst-family (hatched): the merge "
|
||||
"clearly wins\nworst-family (balance); overall it matches the best specialist")
|
||||
ax.set(ylabel="accuracy", title="Overall (solid) vs worst-family (hatched):\n"
|
||||
"the recombined model vs the best single specialist")
|
||||
ax.legend(frameon=False, fontsize=9)
|
||||
|
||||
fig.suptitle("llm_merge — recombining decorrelated specialist LLMs gives the only model competent "
|
||||
f"across all families (balance); overall parity ({cfg['base_model']})", y=1.0, fontsize=12)
|
||||
best_spec = max(_acc(df, m, "overall") for m in specialists)
|
||||
best_merge = max(_acc(df, m, "overall") for m in merges)
|
||||
verdict = (f"recombined {best_merge:.2f} > best specialist {best_spec:.2f} overall"
|
||||
if best_merge > best_spec + 0.005 else
|
||||
f"recombined {best_merge:.2f} ≈ best specialist {best_spec:.2f} overall")
|
||||
fig.suptitle(f"llm_merge — recombining decorrelated specialist LLMs: {verdict} "
|
||||
f"({cfg['base_model'].split('/')[-1]})", y=1.0, fontsize=12)
|
||||
fig.tight_layout()
|
||||
savefig(fig, results_dir, "llm_merge")
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue