# Running the Lamarckian Society on Imperial's HPC (CX3, PBS Pro)

The LLM experiments are the only part that wants more than a laptop GPU. This directory holds the
PBS job scripts for Imperial's **CX3** cluster (scheduler: **PBS Pro** — `qsub`, not Slurm). The
analytic (Layer 1) and small-neural (Layer 1.5) tiers all run locally and need nothing here.

## Confirmed facts (Imperial RCS user guide)

- **Submit / monitor / cancel:** `qsub <script>` · `qstat -u $USER` (Q=queued, R=running) · `qdel <jobid>`.
  Output lands in `<script>.o<jobid>` (stdout) and `.e<jobid>` (stderr).
- **GPU resource line:** `#PBS -l select=1:ncpus=4:mem=24gb:ngpus=1:gpu_type=L40S`
  (leave `:gpu_type=…` off for the default). GPUs: **L40S 48 GB (default)**, RTX6000 24 GB,
  A100 40 GB (scarce). Queue **gpu72** (~72 h), up to 8 GPUs/node.
- **Filesystem:** jobs start in `$HOME`; `$PBS_O_WORKDIR` = the submit directory; `$TMPDIR` = fast
  node-local scratch (copy large inputs in, results out); keep a job under ~100 GB.
- **Modules:** e.g. `module load Python/3.12.3-GCCcore-13.3.0` (we use `uv` instead — see below).

## Two unknowns the docs don't cover — resolved by `probe.pbs`

1. **Do compute nodes have internet?** If not, models and packages must be fetched on the *login*
   node and used offline on the compute node.
2. **What CUDA version does the L40S driver support?** Our env ships torch **cu13**; an older driver
   needs a pinned torch (cu124/cu121).

**Run the probe first:** `git pull` on the login node, then `qsub hpc/probe.pbs`, then read
`probe.o<jobid>`. It prints the GPU + driver CUDA, the internet test, `$TMPDIR`/disk, and the
available `python`/`cuda` modules. (Paste that output back and the real scripts get finalised.)

## One-time setup on the LOGIN node (which has internet)

```bash
git clone <this repo>  &&  cd LamarckianAI
curl -LsSf https://astral.sh/uv/install.sh | sh          # uv -> ~/.local/bin (no sudo)
uv sync --extra dev --extra neural --extra llm           # builds .venv (Python 3.14 + torch + transformers/peft)
# If the probe shows the L40S driver is < CUDA 13, pin torch to match first, e.g.:
#   uv pip install --python .venv "torch==2.*" --index-url https://download.pytorch.org/whl/cu124
# Pre-download the base model into a cache the compute node can read:
HF_HOME=$HOME/hf_cache uv run python -c "from transformers import AutoModelForCausalLM, AutoTokenizer as T; \
  n='Qwen/Qwen2.5-7B-Instruct'; T.from_pretrained(n); AutoModelForCausalLM.from_pretrained(n)"
```

## Run the experiment

```bash
qsub hpc/llm_merge.pbs            # L40S, ~4 h; runs configs/llm/merge_hpc.yaml
qstat -u $USER                    # watch it
```
Results are written to `results/llm_merge_hpc/` (the `.parquet` is gitignored). Sync it back to a
machine with the plotting env to analyse:
```bash
rsync -avz <user>@login.hpc.ic.ac.uk:'~/LamarckianAI/results/llm_merge_hpc/' results/llm_merge_hpc/
python figures/plot_llm_merge.py results/llm_merge_hpc
```

## Notes

- **Why `uv`, not the Python module:** `uv` installs its own Python 3.14 and the exact pinned deps, so
  the HPC env matches the laptop env reproducibly and is independent of the cluster's module set. The
  only cluster-specific adjustment is the torch CUDA build if the driver is old (above).
- **`HF_HUB_OFFLINE=1`** is set in `llm_merge.pbs` on the assumption compute nodes are offline; delete
  that line if the probe shows internet works.
- **The definitive "firm up the sign" run** (not yet coded) also wants: several seeds with mean±CI;
  more task families; and a dilution-resistant / offspring-selected ("directed sex") merge. `merge_hpc.yaml`
  only bumps the base model for now — enough to reduce noise, but the code changes are the real fix.