llm_merge_hpc: the 7B firm-up makes the Fisher-Muller sign decisive

Re-ran the specialist-merge experiment at a capable base (Qwen2.5-7B-Instruct,
200 tests/family) on one L40S GPU of Imperial's CX3 HPC (8 min walltime). The
two caveats the 0.5B prototype left marginal are now resolved:

- "exceeds every parent overall" is clean: both merges 0.87 vs best specialist
  0.77 (+10 pts), and above every specialist on every family.
- dilution vanishes: at 0.5B averaging diluted the lists-specialist
  (0.43->0.26); at 7B the merge beats it (0.62>0.57). Dilution was a
  small-model artefact -- a capable base composes rather than dilutes, which
  softens E4's "merge, don't average" once the parents are strong.

The figure title is now data-driven (reports ">" for 7B, "~" for 0.5B).
Adds the hpc/ smoke job script and the llm_merge walltime trim. Results synced
to results/llm_merge_hpc/ (parquet gitignored per the reproducibility contract).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

hpc/llm_merge.pbs

@@ -1,28 +1,24 @@
 #!/bin/bash
-# Scaled LLM merge experiment on an L40S (48 GB) — the "firm up the sign" run.
-# Prereqs (do ONCE on the login node, which has internet — see hpc/README.md):
-#   1) build the env:            uv sync --extra dev --extra neural --extra llm
-#      (if the L40S driver is older than CUDA 13, first pin torch — see README)
-#   2) pre-download the base:    HF_HOME=$HOME/hf_cache uv run python -c \
-#         "from transformers import AutoModelForCausalLM,AutoTokenizer as T; \
-#          [f('Qwen/Qwen2.5-7B-Instruct') for f in (T.from_pretrained, AutoModelForCausalLM.from_pretrained)]"
-#   submit:  qsub hpc/llm_merge.pbs
+# Scaled LLM merge experiment on an L40S (46 GB) — the "firm up the sign" run.
+# Confirmed on CX3 (probe 2026-07-05): L40S driver 580 supports CUDA 13.0, so the committed
+# torch cu130 env works as-is; compute nodes HAVE internet; $TMPDIR is fast local NVMe.
+# Prereqs on the LOGIN node (see hpc/README.md): `uv sync --extra dev --extra neural --extra llm`
+# (cache on $EPHEMERAL) + pre-download the base into $HF_HOME below.
+#   submit:  qsub hpc/llm_merge.pbs      status: qstat -u $USER
 #PBS -l select=1:ncpus=8:mem=64gb:ngpus=1:gpu_type=L40S
-#PBS -l walltime=04:00:00
-#PBS -N lamarckian_llm_merge
+#PBS -l walltime=02:00:00
+#PBS -N lam_llm_merge
 
 cd "$PBS_O_WORKDIR"
-export HF_HOME="$HOME/hf_cache"           # models cached here (pre-downloaded on the login node)
-export HF_HUB_OFFLINE=1                    # compute node has no internet -> use the cache
-                                          # (delete this line if the probe shows internet works)
+export HF_HOME="$EPHEMERAL/hf_cache"       # models cached on the 11 TB ephemeral store
 export TOKENIZERS_PARALLELISM=false
+export UV_CACHE_DIR="$EPHEMERAL/uvcache"
 
-source .venv/bin/activate                 # uv-built env (Python 3.14 + torch + transformers/peft)
-nvidia-smi --query-gpu=name,memory.total --format=csv,noheader
+source .venv/bin/activate                  # uv-built env (Python 3.14 + torch cu130 + transformers/peft)
+nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv,noheader
 
 python -m llm.experiment configs/llm/merge_hpc.yaml
 
-# results/llm_merge_hpc/{results.parquet,resolved_config.yaml,manifest.json} are written in-place.
-# The parquet is gitignored; sync it back to analyse/plot locally, e.g. from your laptop:
-#   rsync -avz <user>@login.hpc.ic.ac.uk:'~/…/results/llm_merge_hpc/' results/llm_merge_hpc/
+# results/llm_merge_hpc/ is written in-place (parquet gitignored). Sync back to plot:
+#   rsync -avz hpc:'…/LamarckianAI/results/llm_merge_hpc/' results/llm_merge_hpc/
 echo "done: $(date)"

hpc/llm_smoke.pbs

@@ -0,0 +1,14 @@
+#!/bin/bash
+# Smoke gate: run the small 0.5B merge experiment on an L40S to confirm the GPU pipeline works
+# on CX3 before the 4 h 7B run. ~5 min.  submit: qsub hpc/llm_smoke.pbs
+#PBS -l select=1:ncpus=8:mem=48gb:ngpus=1:gpu_type=L40S
+#PBS -l walltime=00:20:00
+#PBS -N lam_llm_smoke
+
+cd "$PBS_O_WORKDIR"
+export HF_HOME="$EPHEMERAL/hf_cache"
+export TOKENIZERS_PARALLELISM=false
+source .venv/bin/activate
+nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv,noheader
+python -m llm.experiment configs/llm/merge.yaml   # 0.5B, writes results/llm_merge/
+echo "done: $(date)"