hpc: PBS job scripts for Imperial CX3 (probe + scaled LLM merge run)

Enable running the LLM tier on Imperial's HPC (scheduler: PBS Pro / qsub).
- hpc/probe.pbs: 10-min 1-GPU reconnaissance job resolving the two unknowns
  the RCS docs omit -- compute-node internet access and the L40S driver's
  CUDA version -- plus TMPDIR/disk and available python/cuda modules.
- hpc/llm_merge.pbs: scaled run on an L40S (48 GB), offline HF-cache wired,
  runs configs/llm/merge_hpc.yaml.
- configs/llm/merge_hpc.yaml: Qwen2.5-7B-Instruct (fits the L40S) to reduce
  the noise that left the 0.5B prototype's overall-exceeds sign marginal.
- hpc/README.md: the git-based workflow (login-node uv env + model
  pre-download -> qsub -> rsync results back), confirmed PBS/GPU directives,
  and the code TODOs for the definitive run (multi-seed, more families,
  directed/dilution-resistant merge).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Giorgio Gilestro 2026-07-05 16:02:09 +01:00
parent 809e45a5e0
commit 38bb252c20
4 changed files with 135 additions and 0 deletions

23
hpc/probe.pbs Normal file
View file

@ -0,0 +1,23 @@
#!/bin/bash
# Reconnaissance job — resolves the two unknowns the RCS docs don't cover:
# (a) do compute nodes have internet? (b) what CUDA version does the L40S driver support?
# plus TMPDIR/disk and available python/cuda modules. Cheap: 1 GPU, 10 min.
# submit: qsub hpc/probe.pbs status: qstat -u $USER output: probe.o<jobid>
#PBS -l select=1:ncpus=4:mem=24gb:ngpus=1
#PBS -l walltime=00:10:00
#PBS -N lamarckian_probe
cd "$PBS_O_WORKDIR" || cd "$HOME"
echo "=== node / date ==="; hostname; date
echo; echo "=== GPU ==="; nvidia-smi --query-gpu=name,driver_version,memory.total --format=csv 2>&1
echo "--- max CUDA the driver supports (top-right of nvidia-smi) ---"; nvidia-smi 2>&1 | head -4
echo; echo "=== compute-node internet? (critical) ==="
curl -sS -m 15 -o /dev/null -w "huggingface.co -> HTTP %{http_code}\n" https://huggingface.co 2>&1 \
|| echo ">>> NO internet on compute node (must pre-download models on the login node)"
echo; echo "=== TMPDIR (node-local scratch) ==="; echo "TMPDIR=$TMPDIR"; df -h "$TMPDIR" 2>/dev/null | tail -1
echo "=== \$HOME disk ==="; df -h "$HOME" 2>/dev/null | tail -1
echo; echo "=== python / cuda / conda modules available ==="
module avail 2>&1 | tr ' ' '\n' | grep -iE "^(python|cuda|anaconda|miniforge|gcc)" | sort -u | head -40
echo; echo "=== uv installed? ==="; command -v uv 2>/dev/null || ls -la ~/.local/bin/uv 2>/dev/null || echo "uv not installed (curl -LsSf https://astral.sh/uv/install.sh | sh)"
echo "=== system python3 ==="; python3 --version 2>&1
echo; echo "=== done ==="