hpc: PBS job scripts for Imperial CX3 (probe + scaled LLM merge run)

Enable running the LLM tier on Imperial's HPC (scheduler: PBS Pro / qsub).
- hpc/probe.pbs: 10-min 1-GPU reconnaissance job resolving the two unknowns
  the RCS docs omit -- compute-node internet access and the L40S driver's
  CUDA version -- plus TMPDIR/disk and available python/cuda modules.
- hpc/llm_merge.pbs: scaled run on an L40S (48 GB), offline HF-cache wired,
  runs configs/llm/merge_hpc.yaml.
- configs/llm/merge_hpc.yaml: Qwen2.5-7B-Instruct (fits the L40S) to reduce
  the noise that left the 0.5B prototype's overall-exceeds sign marginal.
- hpc/README.md: the git-based workflow (login-node uv env + model
  pre-download -> qsub -> rsync results back), confirmed PBS/GPU directives,
  and the code TODOs for the definitive run (multi-seed, more families,
  directed/dilution-resistant merge).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

hpc/README.md

@@ -0,0 +1,64 @@
+# Running the Lamarckian Society on Imperial's HPC (CX3, PBS Pro)
+
+The LLM experiments are the only part that wants more than a laptop GPU. This directory holds the
+PBS job scripts for Imperial's **CX3** cluster (scheduler: **PBS Pro** — `qsub`, not Slurm). The
+analytic (Layer 1) and small-neural (Layer 1.5) tiers all run locally and need nothing here.
+
+## Confirmed facts (Imperial RCS user guide)
+
+- **Submit / monitor / cancel:** `qsub <script>` · `qstat -u $USER` (Q=queued, R=running) · `qdel <jobid>`.
+  Output lands in `<script>.o<jobid>` (stdout) and `.e<jobid>` (stderr).
+- **GPU resource line:** `#PBS -l select=1:ncpus=4:mem=24gb:ngpus=1:gpu_type=L40S`
+  (leave `:gpu_type=…` off for the default). GPUs: **L40S 48 GB (default)**, RTX6000 24 GB,
+  A100 40 GB (scarce). Queue **gpu72** (~72 h), up to 8 GPUs/node.
+- **Filesystem:** jobs start in `$HOME`; `$PBS_O_WORKDIR` = the submit directory; `$TMPDIR` = fast
+  node-local scratch (copy large inputs in, results out); keep a job under ~100 GB.
+- **Modules:** e.g. `module load Python/3.12.3-GCCcore-13.3.0` (we use `uv` instead — see below).
+
+## Two unknowns the docs don't cover — resolved by `probe.pbs`
+
+1. **Do compute nodes have internet?** If not, models and packages must be fetched on the *login*
+   node and used offline on the compute node.
+2. **What CUDA version does the L40S driver support?** Our env ships torch **cu13**; an older driver
+   needs a pinned torch (cu124/cu121).
+
+**Run the probe first:** `git pull` on the login node, then `qsub hpc/probe.pbs`, then read
+`probe.o<jobid>`. It prints the GPU + driver CUDA, the internet test, `$TMPDIR`/disk, and the
+available `python`/`cuda` modules. (Paste that output back and the real scripts get finalised.)
+
+## One-time setup on the LOGIN node (which has internet)
+
+```bash
+git clone <this repo>  &&  cd LamarckianAI
+curl -LsSf https://astral.sh/uv/install.sh | sh          # uv -> ~/.local/bin (no sudo)
+uv sync --extra dev --extra neural --extra llm           # builds .venv (Python 3.14 + torch + transformers/peft)
+# If the probe shows the L40S driver is < CUDA 13, pin torch to match first, e.g.:
+#   uv pip install --python .venv "torch==2.*" --index-url https://download.pytorch.org/whl/cu124
+# Pre-download the base model into a cache the compute node can read:
+HF_HOME=$HOME/hf_cache uv run python -c "from transformers import AutoModelForCausalLM, AutoTokenizer as T; \
+  n='Qwen/Qwen2.5-7B-Instruct'; T.from_pretrained(n); AutoModelForCausalLM.from_pretrained(n)"
+```
+
+## Run the experiment
+
+```bash
+qsub hpc/llm_merge.pbs            # L40S, ~4 h; runs configs/llm/merge_hpc.yaml
+qstat -u $USER                    # watch it
+```
+Results are written to `results/llm_merge_hpc/` (the `.parquet` is gitignored). Sync it back to a
+machine with the plotting env to analyse:
+```bash
+rsync -avz <user>@login.hpc.ic.ac.uk:'~/LamarckianAI/results/llm_merge_hpc/' results/llm_merge_hpc/
+python figures/plot_llm_merge.py results/llm_merge_hpc
+```
+
+## Notes
+
+- **Why `uv`, not the Python module:** `uv` installs its own Python 3.14 and the exact pinned deps, so
+  the HPC env matches the laptop env reproducibly and is independent of the cluster's module set. The
+  only cluster-specific adjustment is the torch CUDA build if the driver is old (above).
+- **`HF_HUB_OFFLINE=1`** is set in `llm_merge.pbs` on the assumption compute nodes are offline; delete
+  that line if the probe shows internet works.
+- **The definitive "firm up the sign" run** (not yet coded) also wants: several seeds with mean±CI;
+  more task families; and a dilution-resistant / offspring-selected ("directed sex") merge. `merge_hpc.yaml`
+  only bumps the base model for now — enough to reduce noise, but the code changes are the real fix.