MachineSex/hpc/README.md

Running the Lamarckian Society on Imperial's HPC (CX3, PBS Pro)

The LLM experiments are the only part that wants more than a laptop GPU. This directory holds the PBS job scripts for Imperial's CX3 cluster (scheduler: PBS Pro — qsub, not Slurm). The analytic (Layer 1) and small-neural (Layer 1.5) tiers all run locally and need nothing here.

Confirmed facts (Imperial RCS user guide)

• Submit / monitor / cancel: qsub <script> · qstat -u $USER (Q=queued, R=running) · qdel <jobid>. Output lands in <script>.o<jobid> (stdout) and .e<jobid> (stderr).
• GPU resource line: #PBS -l select=1:ncpus=4:mem=24gb:ngpus=1:gpu_type=L40S (leave :gpu_type=… off for the default). GPUs: L40S 48 GB (default), RTX6000 24 GB, A100 40 GB (scarce). Queue gpu72 (~72 h), up to 8 GPUs/node.
• Filesystem: jobs start in $HOME; $PBS_O_WORKDIR = the submit directory; $TMPDIR = fast node-local scratch (copy large inputs in, results out); keep a job under ~100 GB.
• Modules: e.g. module load Python/3.12.3-GCCcore-13.3.0 (we use uv instead — see below).

Two unknowns the docs don't cover — resolved by probe.pbs

1. Do compute nodes have internet? If not, models and packages must be fetched on the login node and used offline on the compute node.
2. What CUDA version does the L40S driver support? Our env ships torch cu13; an older driver needs a pinned torch (cu124/cu121).

Run the probe first: git pull on the login node, then qsub hpc/probe.pbs, then read probe.o<jobid>. It prints the GPU + driver CUDA, the internet test, $TMPDIR/disk, and the available python/cuda modules. (Paste that output back and the real scripts get finalised.)

One-time setup on the LOGIN node (which has internet)

git clone <this repo>  &&  cd LamarckianAI
curl -LsSf https://astral.sh/uv/install.sh | sh          # uv -> ~/.local/bin (no sudo)
uv sync --extra dev --extra neural --extra llm           # builds .venv (Python 3.14 + torch + transformers/peft)
# If the probe shows the L40S driver is < CUDA 13, pin torch to match first, e.g.:
#   uv pip install --python .venv "torch==2.*" --index-url https://download.pytorch.org/whl/cu124
# Pre-download the base model into a cache the compute node can read:
HF_HOME=$HOME/hf_cache uv run python -c "from transformers import AutoModelForCausalLM, AutoTokenizer as T; \
  n='Qwen/Qwen2.5-7B-Instruct'; T.from_pretrained(n); AutoModelForCausalLM.from_pretrained(n)"

Run the experiment

qsub hpc/llm_merge.pbs            # L40S, ~4 h; runs configs/llm/merge_hpc.yaml
qstat -u $USER                    # watch it

Results are written to results/llm_merge_hpc/ (the .parquet is gitignored). Sync it back to a machine with the plotting env to analyse:

rsync -avz <user>@login.hpc.ic.ac.uk:'~/LamarckianAI/results/llm_merge_hpc/' results/llm_merge_hpc/
python figures/plot_llm_merge.py results/llm_merge_hpc

Notes

• Why uv, not the Python module: uv installs its own Python 3.14 and the exact pinned deps, so the HPC env matches the laptop env reproducibly and is independent of the cluster's module set. The only cluster-specific adjustment is the torch CUDA build if the driver is old (above).
• HF_HUB_OFFLINE=1 is set in llm_merge.pbs on the assumption compute nodes are offline; delete that line if the probe shows internet works.
• The definitive "firm up the sign" run (not yet coded) also wants: several seeds with mean±CI; more task families; and a dilution-resistant / offspring-selected ("directed sex") merge. merge_hpc.yaml only bumps the base model for now — enough to reduce noise, but the code changes are the real fix.

Array jobs added 2026-09-11

• hpc/llm_curriculum_controls.pbs — seeds 2–3 of the two declinable-merge controls (forced stop curriculum_v5_stop3, decorrelated curriculum curriculum_v5_decor); ~20 min (stop3) / ~40 min (decor, two arms) per element on one L40S.
• hpc/llm_7b_seeds.pbs — seeds 2–3 of the three 7B runs, chained merge → moe_hard → directed_hard per element so the hard specialists are trained once; ~33 min per seed. Output lands in results/llm_<name>_hpc/s{seed}/ (seed 1 was moved to s1/; figures/_figlib.load_seed_bundles reads either layout).
• Gotcha met today: the 7B base was not in $EPHEMERAL/hf_cache; snapshot_download on the login node took 19 s. Do not detect its completion with pgrep -f snapshot_download from a bash -lc wrapper whose own command line contains that string.