Giorgio Gilestro
38bb252c20
Enable running the LLM tier on Imperial's HPC (scheduler: PBS Pro / qsub). - hpc/probe.pbs: 10-min 1-GPU reconnaissance job resolving the two unknowns the RCS docs omit -- compute-node internet access and the L40S driver's CUDA version -- plus TMPDIR/disk and available python/cuda modules. - hpc/llm_merge.pbs: scaled run on an L40S (48 GB), offline HF-cache wired, runs configs/llm/merge_hpc.yaml. - configs/llm/merge_hpc.yaml: Qwen2.5-7B-Instruct (fits the L40S) to reduce the noise that left the 0.5B prototype's overall-exceeds sign marginal. - hpc/README.md: the git-based workflow (login-node uv env + model pre-download -> qsub -> rsync results back), confirmed PBS/GPU directives, and the code TODOs for the definitive run (multi-seed, more families, directed/dilution-resistant merge). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
64 lines
3.7 KiB
Markdown
64 lines
3.7 KiB
Markdown
# Running the Lamarckian Society on Imperial's HPC (CX3, PBS Pro)
|
|
|
|
The LLM experiments are the only part that wants more than a laptop GPU. This directory holds the
|
|
PBS job scripts for Imperial's **CX3** cluster (scheduler: **PBS Pro** — `qsub`, not Slurm). The
|
|
analytic (Layer 1) and small-neural (Layer 1.5) tiers all run locally and need nothing here.
|
|
|
|
## Confirmed facts (Imperial RCS user guide)
|
|
|
|
- **Submit / monitor / cancel:** `qsub <script>` · `qstat -u $USER` (Q=queued, R=running) · `qdel <jobid>`.
|
|
Output lands in `<script>.o<jobid>` (stdout) and `.e<jobid>` (stderr).
|
|
- **GPU resource line:** `#PBS -l select=1:ncpus=4:mem=24gb:ngpus=1:gpu_type=L40S`
|
|
(leave `:gpu_type=…` off for the default). GPUs: **L40S 48 GB (default)**, RTX6000 24 GB,
|
|
A100 40 GB (scarce). Queue **gpu72** (~72 h), up to 8 GPUs/node.
|
|
- **Filesystem:** jobs start in `$HOME`; `$PBS_O_WORKDIR` = the submit directory; `$TMPDIR` = fast
|
|
node-local scratch (copy large inputs in, results out); keep a job under ~100 GB.
|
|
- **Modules:** e.g. `module load Python/3.12.3-GCCcore-13.3.0` (we use `uv` instead — see below).
|
|
|
|
## Two unknowns the docs don't cover — resolved by `probe.pbs`
|
|
|
|
1. **Do compute nodes have internet?** If not, models and packages must be fetched on the *login*
|
|
node and used offline on the compute node.
|
|
2. **What CUDA version does the L40S driver support?** Our env ships torch **cu13**; an older driver
|
|
needs a pinned torch (cu124/cu121).
|
|
|
|
**Run the probe first:** `git pull` on the login node, then `qsub hpc/probe.pbs`, then read
|
|
`probe.o<jobid>`. It prints the GPU + driver CUDA, the internet test, `$TMPDIR`/disk, and the
|
|
available `python`/`cuda` modules. (Paste that output back and the real scripts get finalised.)
|
|
|
|
## One-time setup on the LOGIN node (which has internet)
|
|
|
|
```bash
|
|
git clone <this repo> && cd LamarckianAI
|
|
curl -LsSf https://astral.sh/uv/install.sh | sh # uv -> ~/.local/bin (no sudo)
|
|
uv sync --extra dev --extra neural --extra llm # builds .venv (Python 3.14 + torch + transformers/peft)
|
|
# If the probe shows the L40S driver is < CUDA 13, pin torch to match first, e.g.:
|
|
# uv pip install --python .venv "torch==2.*" --index-url https://download.pytorch.org/whl/cu124
|
|
# Pre-download the base model into a cache the compute node can read:
|
|
HF_HOME=$HOME/hf_cache uv run python -c "from transformers import AutoModelForCausalLM, AutoTokenizer as T; \
|
|
n='Qwen/Qwen2.5-7B-Instruct'; T.from_pretrained(n); AutoModelForCausalLM.from_pretrained(n)"
|
|
```
|
|
|
|
## Run the experiment
|
|
|
|
```bash
|
|
qsub hpc/llm_merge.pbs # L40S, ~4 h; runs configs/llm/merge_hpc.yaml
|
|
qstat -u $USER # watch it
|
|
```
|
|
Results are written to `results/llm_merge_hpc/` (the `.parquet` is gitignored). Sync it back to a
|
|
machine with the plotting env to analyse:
|
|
```bash
|
|
rsync -avz <user>@login.hpc.ic.ac.uk:'~/LamarckianAI/results/llm_merge_hpc/' results/llm_merge_hpc/
|
|
python figures/plot_llm_merge.py results/llm_merge_hpc
|
|
```
|
|
|
|
## Notes
|
|
|
|
- **Why `uv`, not the Python module:** `uv` installs its own Python 3.14 and the exact pinned deps, so
|
|
the HPC env matches the laptop env reproducibly and is independent of the cluster's module set. The
|
|
only cluster-specific adjustment is the torch CUDA build if the driver is old (above).
|
|
- **`HF_HUB_OFFLINE=1`** is set in `llm_merge.pbs` on the assumption compute nodes are offline; delete
|
|
that line if the probe shows internet works.
|
|
- **The definitive "firm up the sign" run** (not yet coded) also wants: several seeds with mean±CI;
|
|
more task families; and a dilution-resistant / offspring-selected ("directed sex") merge. `merge_hpc.yaml`
|
|
only bumps the base model for now — enough to reduce noise, but the code changes are the real fix.
|