Giorgio Gilestro
38bb252c20
Enable running the LLM tier on Imperial's HPC (scheduler: PBS Pro / qsub). - hpc/probe.pbs: 10-min 1-GPU reconnaissance job resolving the two unknowns the RCS docs omit -- compute-node internet access and the L40S driver's CUDA version -- plus TMPDIR/disk and available python/cuda modules. - hpc/llm_merge.pbs: scaled run on an L40S (48 GB), offline HF-cache wired, runs configs/llm/merge_hpc.yaml. - configs/llm/merge_hpc.yaml: Qwen2.5-7B-Instruct (fits the L40S) to reduce the noise that left the 0.5B prototype's overall-exceeds sign marginal. - hpc/README.md: the git-based workflow (login-node uv env + model pre-download -> qsub -> rsync results back), confirmed PBS/GPU directives, and the code TODOs for the definitive run (multi-seed, more families, directed/dilution-resistant merge). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
23 lines
1.5 KiB
Bash
23 lines
1.5 KiB
Bash
#!/bin/bash
|
|
# Reconnaissance job — resolves the two unknowns the RCS docs don't cover:
|
|
# (a) do compute nodes have internet? (b) what CUDA version does the L40S driver support?
|
|
# plus TMPDIR/disk and available python/cuda modules. Cheap: 1 GPU, 10 min.
|
|
# submit: qsub hpc/probe.pbs status: qstat -u $USER output: probe.o<jobid>
|
|
#PBS -l select=1:ncpus=4:mem=24gb:ngpus=1
|
|
#PBS -l walltime=00:10:00
|
|
#PBS -N lamarckian_probe
|
|
|
|
cd "$PBS_O_WORKDIR" || cd "$HOME"
|
|
echo "=== node / date ==="; hostname; date
|
|
echo; echo "=== GPU ==="; nvidia-smi --query-gpu=name,driver_version,memory.total --format=csv 2>&1
|
|
echo "--- max CUDA the driver supports (top-right of nvidia-smi) ---"; nvidia-smi 2>&1 | head -4
|
|
echo; echo "=== compute-node internet? (critical) ==="
|
|
curl -sS -m 15 -o /dev/null -w "huggingface.co -> HTTP %{http_code}\n" https://huggingface.co 2>&1 \
|
|
|| echo ">>> NO internet on compute node (must pre-download models on the login node)"
|
|
echo; echo "=== TMPDIR (node-local scratch) ==="; echo "TMPDIR=$TMPDIR"; df -h "$TMPDIR" 2>/dev/null | tail -1
|
|
echo "=== \$HOME disk ==="; df -h "$HOME" 2>/dev/null | tail -1
|
|
echo; echo "=== python / cuda / conda modules available ==="
|
|
module avail 2>&1 | tr ' ' '\n' | grep -iE "^(python|cuda|anaconda|miniforge|gcc)" | sort -u | head -40
|
|
echo; echo "=== uv installed? ==="; command -v uv 2>/dev/null || ls -la ~/.local/bin/uv 2>/dev/null || echo "uv not installed (curl -LsSf https://astral.sh/uv/install.sh | sh)"
|
|
echo "=== system python3 ==="; python3 --version 2>&1
|
|
echo; echo "=== done ==="
|