hpc: PBS job scripts for Imperial CX3 (probe + scaled LLM merge run)

Enable running the LLM tier on Imperial's HPC (scheduler: PBS Pro / qsub).
- hpc/probe.pbs: 10-min 1-GPU reconnaissance job resolving the two unknowns
  the RCS docs omit -- compute-node internet access and the L40S driver's
  CUDA version -- plus TMPDIR/disk and available python/cuda modules.
- hpc/llm_merge.pbs: scaled run on an L40S (48 GB), offline HF-cache wired,
  runs configs/llm/merge_hpc.yaml.
- configs/llm/merge_hpc.yaml: Qwen2.5-7B-Instruct (fits the L40S) to reduce
  the noise that left the 0.5B prototype's overall-exceeds sign marginal.
- hpc/README.md: the git-based workflow (login-node uv env + model
  pre-download -> qsub -> rsync results back), confirmed PBS/GPU directives,
  and the code TODOs for the definitive run (multi-seed, more families,
  directed/dilution-resistant merge).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

hpc/README.md

@@ -0,0 +1,64 @@
+# Running the Lamarckian Society on Imperial's HPC (CX3, PBS Pro)
+
+The LLM experiments are the only part that wants more than a laptop GPU. This directory holds the
+PBS job scripts for Imperial's **CX3** cluster (scheduler: **PBS Pro** — `qsub`, not Slurm). The
+analytic (Layer 1) and small-neural (Layer 1.5) tiers all run locally and need nothing here.
+
+## Confirmed facts (Imperial RCS user guide)
+
+- **Submit / monitor / cancel:** `qsub <script>` · `qstat -u $USER` (Q=queued, R=running) · `qdel <jobid>`.
+  Output lands in `<script>.o<jobid>` (stdout) and `.e<jobid>` (stderr).
+- **GPU resource line:** `#PBS -l select=1:ncpus=4:mem=24gb:ngpus=1:gpu_type=L40S`
+  (leave `:gpu_type=…` off for the default). GPUs: **L40S 48 GB (default)**, RTX6000 24 GB,
+  A100 40 GB (scarce). Queue **gpu72** (~72 h), up to 8 GPUs/node.
+- **Filesystem:** jobs start in `$HOME`; `$PBS_O_WORKDIR` = the submit directory; `$TMPDIR` = fast
+  node-local scratch (copy large inputs in, results out); keep a job under ~100 GB.
+- **Modules:** e.g. `module load Python/3.12.3-GCCcore-13.3.0` (we use `uv` instead — see below).
+
+## Two unknowns the docs don't cover — resolved by `probe.pbs`
+
+1. **Do compute nodes have internet?** If not, models and packages must be fetched on the *login*
+   node and used offline on the compute node.
+2. **What CUDA version does the L40S driver support?** Our env ships torch **cu13**; an older driver
+   needs a pinned torch (cu124/cu121).
+
+**Run the probe first:** `git pull` on the login node, then `qsub hpc/probe.pbs`, then read
+`probe.o<jobid>`. It prints the GPU + driver CUDA, the internet test, `$TMPDIR`/disk, and the
+available `python`/`cuda` modules. (Paste that output back and the real scripts get finalised.)
+
+## One-time setup on the LOGIN node (which has internet)
+
+```bash
+git clone <this repo>  &&  cd LamarckianAI
+curl -LsSf https://astral.sh/uv/install.sh | sh          # uv -> ~/.local/bin (no sudo)
+uv sync --extra dev --extra neural --extra llm           # builds .venv (Python 3.14 + torch + transformers/peft)
+# If the probe shows the L40S driver is < CUDA 13, pin torch to match first, e.g.:
+#   uv pip install --python .venv "torch==2.*" --index-url https://download.pytorch.org/whl/cu124
+# Pre-download the base model into a cache the compute node can read:
+HF_HOME=$HOME/hf_cache uv run python -c "from transformers import AutoModelForCausalLM, AutoTokenizer as T; \
+  n='Qwen/Qwen2.5-7B-Instruct'; T.from_pretrained(n); AutoModelForCausalLM.from_pretrained(n)"
+```
+
+## Run the experiment
+
+```bash
+qsub hpc/llm_merge.pbs            # L40S, ~4 h; runs configs/llm/merge_hpc.yaml
+qstat -u $USER                    # watch it
+```
+Results are written to `results/llm_merge_hpc/` (the `.parquet` is gitignored). Sync it back to a
+machine with the plotting env to analyse:
+```bash
+rsync -avz <user>@login.hpc.ic.ac.uk:'~/LamarckianAI/results/llm_merge_hpc/' results/llm_merge_hpc/
+python figures/plot_llm_merge.py results/llm_merge_hpc
+```
+
+## Notes
+
+- **Why `uv`, not the Python module:** `uv` installs its own Python 3.14 and the exact pinned deps, so
+  the HPC env matches the laptop env reproducibly and is independent of the cluster's module set. The
+  only cluster-specific adjustment is the torch CUDA build if the driver is old (above).
+- **`HF_HUB_OFFLINE=1`** is set in `llm_merge.pbs` on the assumption compute nodes are offline; delete
+  that line if the probe shows internet works.
+- **The definitive "firm up the sign" run** (not yet coded) also wants: several seeds with mean±CI;
+  more task families; and a dilution-resistant / offspring-selected ("directed sex") merge. `merge_hpc.yaml`
+  only bumps the base model for now — enough to reduce noise, but the code changes are the real fix.

hpc/llm_merge.pbs

@@ -0,0 +1,28 @@
+#!/bin/bash
+# Scaled LLM merge experiment on an L40S (48 GB) — the "firm up the sign" run.
+# Prereqs (do ONCE on the login node, which has internet — see hpc/README.md):
+#   1) build the env:            uv sync --extra dev --extra neural --extra llm
+#      (if the L40S driver is older than CUDA 13, first pin torch — see README)
+#   2) pre-download the base:    HF_HOME=$HOME/hf_cache uv run python -c \
+#         "from transformers import AutoModelForCausalLM,AutoTokenizer as T; \
+#          [f('Qwen/Qwen2.5-7B-Instruct') for f in (T.from_pretrained, AutoModelForCausalLM.from_pretrained)]"
+#   submit:  qsub hpc/llm_merge.pbs
+#PBS -l select=1:ncpus=8:mem=64gb:ngpus=1:gpu_type=L40S
+#PBS -l walltime=04:00:00
+#PBS -N lamarckian_llm_merge
+
+cd "$PBS_O_WORKDIR"
+export HF_HOME="$HOME/hf_cache"           # models cached here (pre-downloaded on the login node)
+export HF_HUB_OFFLINE=1                    # compute node has no internet -> use the cache
+                                          # (delete this line if the probe shows internet works)
+export TOKENIZERS_PARALLELISM=false
+
+source .venv/bin/activate                 # uv-built env (Python 3.14 + torch + transformers/peft)
+nvidia-smi --query-gpu=name,memory.total --format=csv,noheader
+
+python -m llm.experiment configs/llm/merge_hpc.yaml
+
+# results/llm_merge_hpc/{results.parquet,resolved_config.yaml,manifest.json} are written in-place.
+# The parquet is gitignored; sync it back to analyse/plot locally, e.g. from your laptop:
+#   rsync -avz <user>@login.hpc.ic.ac.uk:'~/…/results/llm_merge_hpc/' results/llm_merge_hpc/
+echo "done: $(date)"

hpc/probe.pbs

@@ -0,0 +1,23 @@
+#!/bin/bash
+# Reconnaissance job — resolves the two unknowns the RCS docs don't cover:
+#   (a) do compute nodes have internet? (b) what CUDA version does the L40S driver support?
+# plus TMPDIR/disk and available python/cuda modules. Cheap: 1 GPU, 10 min.
+#   submit:  qsub hpc/probe.pbs      status: qstat -u $USER      output: probe.o<jobid>
+#PBS -l select=1:ncpus=4:mem=24gb:ngpus=1
+#PBS -l walltime=00:10:00
+#PBS -N lamarckian_probe
+
+cd "$PBS_O_WORKDIR" || cd "$HOME"
+echo "=== node / date ==="; hostname; date
+echo; echo "=== GPU ==="; nvidia-smi --query-gpu=name,driver_version,memory.total --format=csv 2>&1
+echo "--- max CUDA the driver supports (top-right of nvidia-smi) ---"; nvidia-smi 2>&1 | head -4
+echo; echo "=== compute-node internet? (critical) ==="
+curl -sS -m 15 -o /dev/null -w "huggingface.co -> HTTP %{http_code}\n" https://huggingface.co 2>&1 \
+  || echo ">>> NO internet on compute node (must pre-download models on the login node)"
+echo; echo "=== TMPDIR (node-local scratch) ==="; echo "TMPDIR=$TMPDIR"; df -h "$TMPDIR" 2>/dev/null | tail -1
+echo "=== \$HOME disk ==="; df -h "$HOME" 2>/dev/null | tail -1
+echo; echo "=== python / cuda / conda modules available ==="
+module avail 2>&1 | tr ' ' '\n' | grep -iE "^(python|cuda|anaconda|miniforge|gcc)" | sort -u | head -40
+echo; echo "=== uv installed? ==="; command -v uv 2>/dev/null || ls -la ~/.local/bin/uv 2>/dev/null || echo "uv not installed (curl -LsSf https://astral.sh/uv/install.sh | sh)"
+echo "=== system python3 ==="; python3 --version 2>&1
+echo; echo "=== done ==="