Giorgio Gilestro
8086504f44
Queue strategy per GG: one self-contained job per seed (founders inline - no cache races, no dependencies), single L40S, 6 h walltime (measured: pilot ran 31 min/arm on the A4000, so a seed is ~2.5-3.5 h) - short independent jobs that backfill well. Seed 1 runs locally on the A4000 as queue insurance; seeds 2-4 go to CX3 as array elements. 0.5B for the campaign (sign-level, seed-replicated, the paper's established pattern); a single 7B confirm of the headline contrast is the post-campaign step. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkRLcc18rwT2Lysu6PbG7v
25 lines
1.1 KiB
Bash
25 lines
1.1 KiB
Bash
#!/bin/bash
|
|
# The LLM society campaign (C3): 4-arm ablation, one seed per array element on one L40S each.
|
|
# Seed 1 runs locally on GG's A4000 (queue insurance); this array covers seeds 2-4. Each job is
|
|
# self-contained (founders trained inline, adapters under models/, deleted per generation), so a
|
|
# killed element reruns cleanly and elements schedule independently. ~2.5-3.5 h measured-scale
|
|
# runtime; 6 h walltime for backfill-friendly margin.
|
|
# submit: qsub hpc/llm_society.pbs status: qstat -u $USER
|
|
#PBS -l select=1:ncpus=8:mem=64gb:ngpus=1:gpu_type=L40S
|
|
#PBS -l walltime=06:00:00
|
|
#PBS -N lam_society
|
|
#PBS -J 2-4
|
|
|
|
cd "$PBS_O_WORKDIR"
|
|
export HF_HOME="$EPHEMERAL/hf_cache"
|
|
export TOKENIZERS_PARALLELISM=false
|
|
export UV_CACHE_DIR="$EPHEMERAL/uvcache"
|
|
|
|
source .venv/bin/activate
|
|
nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv,noheader
|
|
|
|
python -m llm.experiment "configs/llm/society_campaign_s${PBS_ARRAY_INDEX}.yaml"
|
|
|
|
# results/llm_society_campaign/s${PBS_ARRAY_INDEX}/ written in-place. Sync back:
|
|
# rsync -avz hpc:'.../results/llm_society_campaign/' results/llm_society_campaign/
|
|
echo "done: $(date)"
|