Giorgio Gilestro
585264d0b4
Re-ran the specialist-merge experiment at a capable base (Qwen2.5-7B-Instruct, 200 tests/family) on one L40S GPU of Imperial's CX3 HPC (8 min walltime). The two caveats the 0.5B prototype left marginal are now resolved: - "exceeds every parent overall" is clean: both merges 0.87 vs best specialist 0.77 (+10 pts), and above every specialist on every family. - dilution vanishes: at 0.5B averaging diluted the lists-specialist (0.43->0.26); at 7B the merge beats it (0.62>0.57). Dilution was a small-model artefact -- a capable base composes rather than dilutes, which softens E4's "merge, don't average" once the parents are strong. The figure title is now data-driven (reports ">" for 7B, "~" for 0.5B). Adds the hpc/ smoke job script and the llm_merge walltime trim. Results synced to results/llm_merge_hpc/ (parquet gitignored per the reproducibility contract). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
24 lines
1.2 KiB
Bash
24 lines
1.2 KiB
Bash
#!/bin/bash
|
|
# Scaled LLM merge experiment on an L40S (46 GB) — the "firm up the sign" run.
|
|
# Confirmed on CX3 (probe 2026-07-05): L40S driver 580 supports CUDA 13.0, so the committed
|
|
# torch cu130 env works as-is; compute nodes HAVE internet; $TMPDIR is fast local NVMe.
|
|
# Prereqs on the LOGIN node (see hpc/README.md): `uv sync --extra dev --extra neural --extra llm`
|
|
# (cache on $EPHEMERAL) + pre-download the base into $HF_HOME below.
|
|
# submit: qsub hpc/llm_merge.pbs status: qstat -u $USER
|
|
#PBS -l select=1:ncpus=8:mem=64gb:ngpus=1:gpu_type=L40S
|
|
#PBS -l walltime=02:00:00
|
|
#PBS -N lam_llm_merge
|
|
|
|
cd "$PBS_O_WORKDIR"
|
|
export HF_HOME="$EPHEMERAL/hf_cache" # models cached on the 11 TB ephemeral store
|
|
export TOKENIZERS_PARALLELISM=false
|
|
export UV_CACHE_DIR="$EPHEMERAL/uvcache"
|
|
|
|
source .venv/bin/activate # uv-built env (Python 3.14 + torch cu130 + transformers/peft)
|
|
nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv,noheader
|
|
|
|
python -m llm.experiment configs/llm/merge_hpc.yaml
|
|
|
|
# results/llm_merge_hpc/ is written in-place (parquet gitignored). Sync back to plot:
|
|
# rsync -avz hpc:'…/LamarckianAI/results/llm_merge_hpc/' results/llm_merge_hpc/
|
|
echo "done: $(date)"
|