Giorgio Gilestro
84124de143
Clarity pass over the main text (36-item audit), Discussion rewrite and cut, acknowledgements, Souly et al. as ref 62, lettered SI panels, model section moved under Results; plus the untracked curriculum/society/compose/smol configs, runners, figures, stats and tests that the SI already cites. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y64o8FKP7rCuXzC48pxpMm
77 lines
4.5 KiB
Markdown
77 lines
4.5 KiB
Markdown
# Running the Lamarckian Society on Imperial's HPC (CX3, PBS Pro)
|
||
|
||
The LLM experiments are the only part that wants more than a laptop GPU. This directory holds the
|
||
PBS job scripts for Imperial's **CX3** cluster (scheduler: **PBS Pro** — `qsub`, not Slurm). The
|
||
analytic (Layer 1) and small-neural (Layer 1.5) tiers all run locally and need nothing here.
|
||
|
||
## Confirmed facts (Imperial RCS user guide)
|
||
|
||
- **Submit / monitor / cancel:** `qsub <script>` · `qstat -u $USER` (Q=queued, R=running) · `qdel <jobid>`.
|
||
Output lands in `<script>.o<jobid>` (stdout) and `.e<jobid>` (stderr).
|
||
- **GPU resource line:** `#PBS -l select=1:ncpus=4:mem=24gb:ngpus=1:gpu_type=L40S`
|
||
(leave `:gpu_type=…` off for the default). GPUs: **L40S 48 GB (default)**, RTX6000 24 GB,
|
||
A100 40 GB (scarce). Queue **gpu72** (~72 h), up to 8 GPUs/node.
|
||
- **Filesystem:** jobs start in `$HOME`; `$PBS_O_WORKDIR` = the submit directory; `$TMPDIR` = fast
|
||
node-local scratch (copy large inputs in, results out); keep a job under ~100 GB.
|
||
- **Modules:** e.g. `module load Python/3.12.3-GCCcore-13.3.0` (we use `uv` instead — see below).
|
||
|
||
## Two unknowns the docs don't cover — resolved by `probe.pbs`
|
||
|
||
1. **Do compute nodes have internet?** If not, models and packages must be fetched on the *login*
|
||
node and used offline on the compute node.
|
||
2. **What CUDA version does the L40S driver support?** Our env ships torch **cu13**; an older driver
|
||
needs a pinned torch (cu124/cu121).
|
||
|
||
**Run the probe first:** `git pull` on the login node, then `qsub hpc/probe.pbs`, then read
|
||
`probe.o<jobid>`. It prints the GPU + driver CUDA, the internet test, `$TMPDIR`/disk, and the
|
||
available `python`/`cuda` modules. (Paste that output back and the real scripts get finalised.)
|
||
|
||
## One-time setup on the LOGIN node (which has internet)
|
||
|
||
```bash
|
||
git clone <this repo> && cd LamarckianAI
|
||
curl -LsSf https://astral.sh/uv/install.sh | sh # uv -> ~/.local/bin (no sudo)
|
||
uv sync --extra dev --extra neural --extra llm # builds .venv (Python 3.14 + torch + transformers/peft)
|
||
# If the probe shows the L40S driver is < CUDA 13, pin torch to match first, e.g.:
|
||
# uv pip install --python .venv "torch==2.*" --index-url https://download.pytorch.org/whl/cu124
|
||
# Pre-download the base model into a cache the compute node can read:
|
||
HF_HOME=$HOME/hf_cache uv run python -c "from transformers import AutoModelForCausalLM, AutoTokenizer as T; \
|
||
n='Qwen/Qwen2.5-7B-Instruct'; T.from_pretrained(n); AutoModelForCausalLM.from_pretrained(n)"
|
||
```
|
||
|
||
## Run the experiment
|
||
|
||
```bash
|
||
qsub hpc/llm_merge.pbs # L40S, ~4 h; runs configs/llm/merge_hpc.yaml
|
||
qstat -u $USER # watch it
|
||
```
|
||
Results are written to `results/llm_merge_hpc/` (the `.parquet` is gitignored). Sync it back to a
|
||
machine with the plotting env to analyse:
|
||
```bash
|
||
rsync -avz <user>@login.hpc.ic.ac.uk:'~/LamarckianAI/results/llm_merge_hpc/' results/llm_merge_hpc/
|
||
python figures/plot_llm_merge.py results/llm_merge_hpc
|
||
```
|
||
|
||
## Notes
|
||
|
||
- **Why `uv`, not the Python module:** `uv` installs its own Python 3.14 and the exact pinned deps, so
|
||
the HPC env matches the laptop env reproducibly and is independent of the cluster's module set. The
|
||
only cluster-specific adjustment is the torch CUDA build if the driver is old (above).
|
||
- **`HF_HUB_OFFLINE=1`** is set in `llm_merge.pbs` on the assumption compute nodes are offline; delete
|
||
that line if the probe shows internet works.
|
||
- **The definitive "firm up the sign" run** (not yet coded) also wants: several seeds with mean±CI;
|
||
more task families; and a dilution-resistant / offspring-selected ("directed sex") merge. `merge_hpc.yaml`
|
||
only bumps the base model for now — enough to reduce noise, but the code changes are the real fix.
|
||
|
||
## Array jobs added 2026-09-11
|
||
|
||
- `hpc/llm_curriculum_controls.pbs` — seeds 2–3 of the two declinable-merge controls (forced stop
|
||
`curriculum_v5_stop3`, decorrelated curriculum `curriculum_v5_decor`); ~20 min (stop3) / ~40 min
|
||
(decor, two arms) per element on one L40S.
|
||
- `hpc/llm_7b_seeds.pbs` — seeds 2–3 of the three 7B runs, chained merge → moe_hard → directed_hard
|
||
per element so the hard specialists are trained once; ~33 min per seed. Output lands in
|
||
`results/llm_<name>_hpc/s{seed}/` (seed 1 was moved to `s1/`; `figures/_figlib.load_seed_bundles`
|
||
reads either layout).
|
||
- Gotcha met today: the 7B base was not in `$EPHEMERAL/hf_cache`; `snapshot_download` on the login
|
||
node took 19 s. Do not detect its completion with `pgrep -f snapshot_download` from a `bash -lc`
|
||
wrapper whose own command line contains that string.
|