MachineSex/hpc

Running the Lamarckian Society on Imperial's HPC (CX3, PBS Pro)

The LLM experiments are the only part that wants more than a laptop GPU. This directory holds the PBS job scripts for Imperial's CX3 cluster (scheduler: PBS Pro — qsub, not Slurm). The analytic (Layer 1) and small-neural (Layer 1.5) tiers all run locally and need nothing here.

Confirmed facts (Imperial RCS user guide)

• Submit / monitor / cancel: qsub <script> · qstat -u $USER (Q=queued, R=running) · qdel <jobid>. Output lands in <script>.o<jobid> (stdout) and .e<jobid> (stderr).
• GPU resource line: #PBS -l select=1:ncpus=4:mem=24gb:ngpus=1:gpu_type=L40S (leave :gpu_type=… off for the default). GPUs: L40S 48 GB (default), RTX6000 24 GB, A100 40 GB (scarce). Queue gpu72 (~72 h), up to 8 GPUs/node.
• Filesystem: jobs start in $HOME; $PBS_O_WORKDIR = the submit directory; $TMPDIR = fast node-local scratch (copy large inputs in, results out); keep a job under ~100 GB.
• Modules: e.g. module load Python/3.12.3-GCCcore-13.3.0 (we use uv instead — see below).

Two unknowns the docs don't cover — resolved by probe.pbs

1. Do compute nodes have internet? If not, models and packages must be fetched on the login node and used offline on the compute node.
2. What CUDA version does the L40S driver support? Our env ships torch cu13; an older driver needs a pinned torch (cu124/cu121).

Run the probe first: git pull on the login node, then qsub hpc/probe.pbs, then read probe.o<jobid>. It prints the GPU + driver CUDA, the internet test, $TMPDIR/disk, and the available python/cuda modules. (Paste that output back and the real scripts get finalised.)

One-time setup on the LOGIN node (which has internet)

git clone <this repo>  &&  cd LamarckianAI
curl -LsSf https://astral.sh/uv/install.sh | sh          # uv -> ~/.local/bin (no sudo)
uv sync --extra dev --extra neural --extra llm           # builds .venv (Python 3.14 + torch + transformers/peft)
# If the probe shows the L40S driver is < CUDA 13, pin torch to match first, e.g.:
#   uv pip install --python .venv "torch==2.*" --index-url https://download.pytorch.org/whl/cu124
# Pre-download the base model into a cache the compute node can read:
HF_HOME=$HOME/hf_cache uv run python -c "from transformers import AutoModelForCausalLM, AutoTokenizer as T; \
  n='Qwen/Qwen2.5-7B-Instruct'; T.from_pretrained(n); AutoModelForCausalLM.from_pretrained(n)"

Run the experiment

qsub hpc/llm_merge.pbs            # L40S, ~4 h; runs configs/llm/merge_hpc.yaml
qstat -u $USER                    # watch it

Results are written to results/llm_merge_hpc/ (the .parquet is gitignored). Sync it back to a machine with the plotting env to analyse:

rsync -avz <user>@login.hpc.ic.ac.uk:'~/LamarckianAI/results/llm_merge_hpc/' results/llm_merge_hpc/
python figures/plot_llm_merge.py results/llm_merge_hpc

Notes

• Why uv, not the Python module: uv installs its own Python 3.14 and the exact pinned deps, so the HPC env matches the laptop env reproducibly and is independent of the cluster's module set. The only cluster-specific adjustment is the torch CUDA build if the driver is old (above).
• HF_HUB_OFFLINE=1 is set in llm_merge.pbs on the assumption compute nodes are offline; delete that line if the probe shows internet works.
• The definitive "firm up the sign" run (not yet coded) also wants: several seeds with mean±CI; more task families; and a dilution-resistant / offspring-selected ("directed sex") merge. merge_hpc.yaml only bumps the base model for now — enough to reduce noise, but the code changes are the real fix.