Giorgio Gilestro
39f6c9f4df
The easy task families saturated 7B (strings & arith at 1.00), so the earlier
7B nulls — moe: fusion 0.87 > union 0.84; directed ~= soup — could not separate
"refinements don't help at scale" from "tasks too easy at 7B". Adds a hard task
variant (hard: true in tasks.py: multi-step lists, Caesar ciphers / letter
transforms, multi-step & larger arithmetic; same family labels and answer
formats, threaded through make_tasks/train_specialist/runners; hard specialists
cache separately as spec_*_hard) and re-runs both experiments at 7B on Imperial
CX3 (one L40S, 24 min, unsaturated: arith ~0.48, strings 0.67, lists 0.34).
Both nulls flip back to the 0.5B ordering:
- Union beats fusion again: routing 0.500 > fusion 0.40 (soup 0.392 / ties
0.400), the same 10-pt margin as 0.5B. Fusion dilutes the fragile strings
specialist so hard (0.665 -> soup 0.300) that soup even trails the best single
specialist (0.425); routing keeps it intact (0.670).
- Directed selection beats soup again: 0.492 > 0.392 (+10 pts), recovering most
of routing's benefit from one deployable merged model (lifts strings to 0.630).
Correction to the earlier interpretation: the llm_moe_hpc "regime flip" and the
llm_directed_hpc "no headroom" null were driven by TASK SATURATION, not base
capability. The operative variable is headroom — "merge, don't average" (union >
fusion) and "directed sex" (selection > single blend) hold whenever there is room
to lose to dilution: a weak base (0.5B) OR hard tasks at a strong base (7B-hard).
Fusion only wins in the degenerate corner where easy tasks let a strong base
compose to the 1.00 ceiling. Vindicates E8's max > mean in real 7B weights once
saturation is controlled.
Default (easy) task behaviour is unchanged (hard defaults False). +1 hard-task
test (131 green). Excludes the 0.5B smoke bundle (a pipeline gate, not a
deliverable). Results in results/llm_{moe,directed}_hard_hpc/ (parquet gitignored).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|---|---|---|
| .. | ||
| llm_directed.pbs | ||
| llm_hard.pbs | ||
| llm_merge.pbs | ||
| llm_moe.pbs | ||
| llm_smoke.pbs | ||
| probe.pbs | ||
| README.md | ||
Running the Lamarckian Society on Imperial's HPC (CX3, PBS Pro)
The LLM experiments are the only part that wants more than a laptop GPU. This directory holds the
PBS job scripts for Imperial's CX3 cluster (scheduler: PBS Pro — qsub, not Slurm). The
analytic (Layer 1) and small-neural (Layer 1.5) tiers all run locally and need nothing here.
Confirmed facts (Imperial RCS user guide)
- Submit / monitor / cancel:
qsub <script>·qstat -u $USER(Q=queued, R=running) ·qdel <jobid>. Output lands in<script>.o<jobid>(stdout) and.e<jobid>(stderr). - GPU resource line:
#PBS -l select=1:ncpus=4:mem=24gb:ngpus=1:gpu_type=L40S(leave:gpu_type=…off for the default). GPUs: L40S 48 GB (default), RTX6000 24 GB, A100 40 GB (scarce). Queue gpu72 (~72 h), up to 8 GPUs/node. - Filesystem: jobs start in
$HOME;$PBS_O_WORKDIR= the submit directory;$TMPDIR= fast node-local scratch (copy large inputs in, results out); keep a job under ~100 GB. - Modules: e.g.
module load Python/3.12.3-GCCcore-13.3.0(we useuvinstead — see below).
Two unknowns the docs don't cover — resolved by probe.pbs
- Do compute nodes have internet? If not, models and packages must be fetched on the login node and used offline on the compute node.
- What CUDA version does the L40S driver support? Our env ships torch cu13; an older driver needs a pinned torch (cu124/cu121).
Run the probe first: git pull on the login node, then qsub hpc/probe.pbs, then read
probe.o<jobid>. It prints the GPU + driver CUDA, the internet test, $TMPDIR/disk, and the
available python/cuda modules. (Paste that output back and the real scripts get finalised.)
One-time setup on the LOGIN node (which has internet)
git clone <this repo> && cd LamarckianAI
curl -LsSf https://astral.sh/uv/install.sh | sh # uv -> ~/.local/bin (no sudo)
uv sync --extra dev --extra neural --extra llm # builds .venv (Python 3.14 + torch + transformers/peft)
# If the probe shows the L40S driver is < CUDA 13, pin torch to match first, e.g.:
# uv pip install --python .venv "torch==2.*" --index-url https://download.pytorch.org/whl/cu124
# Pre-download the base model into a cache the compute node can read:
HF_HOME=$HOME/hf_cache uv run python -c "from transformers import AutoModelForCausalLM, AutoTokenizer as T; \
n='Qwen/Qwen2.5-7B-Instruct'; T.from_pretrained(n); AutoModelForCausalLM.from_pretrained(n)"
Run the experiment
qsub hpc/llm_merge.pbs # L40S, ~4 h; runs configs/llm/merge_hpc.yaml
qstat -u $USER # watch it
Results are written to results/llm_merge_hpc/ (the .parquet is gitignored). Sync it back to a
machine with the plotting env to analyse:
rsync -avz <user>@login.hpc.ic.ac.uk:'~/LamarckianAI/results/llm_merge_hpc/' results/llm_merge_hpc/
python figures/plot_llm_merge.py results/llm_merge_hpc
Notes
- Why
uv, not the Python module:uvinstalls its own Python 3.14 and the exact pinned deps, so the HPC env matches the laptop env reproducibly and is independent of the cluster's module set. The only cluster-specific adjustment is the torch CUDA build if the driver is old (above). HF_HUB_OFFLINE=1is set inllm_merge.pbson the assumption compute nodes are offline; delete that line if the probe shows internet works.- The definitive "firm up the sign" run (not yet coded) also wants: several seeds with mean±CI;
more task families; and a dilution-resistant / offspring-selected ("directed sex") merge.
merge_hpc.yamlonly bumps the base model for now — enough to reduce noise, but the code changes are the real fix.