Giorgio Gilestro 38bb252c20 hpc: PBS job scripts for Imperial CX3 (probe + scaled LLM merge run)

Enable running the LLM tier on Imperial's HPC (scheduler: PBS Pro / qsub).
- hpc/probe.pbs: 10-min 1-GPU reconnaissance job resolving the two unknowns
  the RCS docs omit -- compute-node internet access and the L40S driver's
  CUDA version -- plus TMPDIR/disk and available python/cuda modules.
- hpc/llm_merge.pbs: scaled run on an L40S (48 GB), offline HF-cache wired,
  runs configs/llm/merge_hpc.yaml.
- configs/llm/merge_hpc.yaml: Qwen2.5-7B-Instruct (fits the L40S) to reduce
  the noise that left the 0.5B prototype's overall-exceeds sign marginal.
- hpc/README.md: the git-based workflow (login-node uv env + model
  pre-download -> qsub -> rsync results back), confirmed PBS/GPU directives,
  and the code TODOs for the definitive run (multi-seed, more families,
  directed/dilution-resistant merge).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

2026-07-05 16:02:09 +01:00

3.7 KiB

Raw Blame History

Running the Lamarckian Society on Imperial's HPC (CX3, PBS Pro)

The LLM experiments are the only part that wants more than a laptop GPU. This directory holds the PBS job scripts for Imperial's CX3 cluster (scheduler: PBS Pro — qsub, not Slurm). The analytic (Layer 1) and small-neural (Layer 1.5) tiers all run locally and need nothing here.

Confirmed facts (Imperial RCS user guide)

Submit / monitor / cancel: qsub <script> · qstat -u $USER (Q=queued, R=running) · qdel <jobid>. Output lands in <script>.o<jobid> (stdout) and .e<jobid> (stderr).
GPU resource line: #PBS -l select=1:ncpus=4:mem=24gb:ngpus=1:gpu_type=L40S (leave :gpu_type=… off for the default). GPUs: L40S 48 GB (default), RTX6000 24 GB, A100 40 GB (scarce). Queue gpu72 (~72 h), up to 8 GPUs/node.
Filesystem: jobs start in $HOME; $PBS_O_WORKDIR = the submit directory; $TMPDIR = fast node-local scratch (copy large inputs in, results out); keep a job under ~100 GB.
Modules: e.g. module load Python/3.12.3-GCCcore-13.3.0 (we use uv instead — see below).

Two unknowns the docs don't cover — resolved by `probe.pbs`

Do compute nodes have internet? If not, models and packages must be fetched on the login node and used offline on the compute node.
What CUDA version does the L40S driver support? Our env ships torch cu13; an older driver needs a pinned torch (cu124/cu121).

Run the probe first: git pull on the login node, then qsub hpc/probe.pbs, then read probe.o<jobid>. It prints the GPU + driver CUDA, the internet test, $TMPDIR/disk, and the available python/cuda modules. (Paste that output back and the real scripts get finalised.)

git clone <this repo>  &&  cd LamarckianAI
curl -LsSf https://astral.sh/uv/install.sh | sh          # uv -> ~/.local/bin (no sudo)
uv sync --extra dev --extra neural --extra llm           # builds .venv (Python 3.14 + torch + transformers/peft)
# If the probe shows the L40S driver is < CUDA 13, pin torch to match first, e.g.:
#   uv pip install --python .venv "torch==2.*" --index-url https://download.pytorch.org/whl/cu124
# Pre-download the base model into a cache the compute node can read:
HF_HOME=$HOME/hf_cache uv run python -c "from transformers import AutoModelForCausalLM, AutoTokenizer as T; \
  n='Qwen/Qwen2.5-7B-Instruct'; T.from_pretrained(n); AutoModelForCausalLM.from_pretrained(n)"

Run the experiment

qsub hpc/llm_merge.pbs            # L40S, ~4 h; runs configs/llm/merge_hpc.yaml
qstat -u $USER                    # watch it

Results are written to results/llm_merge_hpc/ (the .parquet is gitignored). Sync it back to a machine with the plotting env to analyse:

rsync -avz <user>@login.hpc.ic.ac.uk:'~/LamarckianAI/results/llm_merge_hpc/' results/llm_merge_hpc/
python figures/plot_llm_merge.py results/llm_merge_hpc

Notes

Why uv, not the Python module: uv installs its own Python 3.14 and the exact pinned deps, so the HPC env matches the laptop env reproducibly and is independent of the cluster's module set. The only cluster-specific adjustment is the torch CUDA build if the driver is old (above).
HF_HUB_OFFLINE=1 is set in llm_merge.pbs on the assumption compute nodes are offline; delete that line if the probe shows internet works.
The definitive "firm up the sign" run (not yet coded) also wants: several seeds with mean±CI; more task families; and a dilution-resistant / offspring-selected ("directed sex") merge. merge_hpc.yaml only bumps the base model for now — enough to reduce noise, but the code changes are the real fix.

3.7 KiB Raw Blame History