llm_society campaign: 4 seed-configs (N=8, G=10, 4 arms) + CX3 PBS array
Queue strategy per GG: one self-contained job per seed (founders inline - no cache races, no dependencies), single L40S, 6 h walltime (measured: pilot ran 31 min/arm on the A4000, so a seed is ~2.5-3.5 h) - short independent jobs that backfill well. Seed 1 runs locally on the A4000 as queue insurance; seeds 2-4 go to CX3 as array elements. 0.5B for the campaign (sign-level, seed-replicated, the paper's established pattern); a single 7B confirm of the headline contrast is the post-campaign step. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BkRLcc18rwT2Lysu6PbG7v
This commit is contained in:
parent
9e83aa7034
commit
8086504f44
5 changed files with 117 additions and 0 deletions
25
hpc/llm_society.pbs
Normal file
25
hpc/llm_society.pbs
Normal file
|
|
@ -0,0 +1,25 @@
|
|||
#!/bin/bash
|
||||
# The LLM society campaign (C3): 4-arm ablation, one seed per array element on one L40S each.
|
||||
# Seed 1 runs locally on GG's A4000 (queue insurance); this array covers seeds 2-4. Each job is
|
||||
# self-contained (founders trained inline, adapters under models/, deleted per generation), so a
|
||||
# killed element reruns cleanly and elements schedule independently. ~2.5-3.5 h measured-scale
|
||||
# runtime; 6 h walltime for backfill-friendly margin.
|
||||
# submit: qsub hpc/llm_society.pbs status: qstat -u $USER
|
||||
#PBS -l select=1:ncpus=8:mem=64gb:ngpus=1:gpu_type=L40S
|
||||
#PBS -l walltime=06:00:00
|
||||
#PBS -N lam_society
|
||||
#PBS -J 2-4
|
||||
|
||||
cd "$PBS_O_WORKDIR"
|
||||
export HF_HOME="$EPHEMERAL/hf_cache"
|
||||
export TOKENIZERS_PARALLELISM=false
|
||||
export UV_CACHE_DIR="$EPHEMERAL/uvcache"
|
||||
|
||||
source .venv/bin/activate
|
||||
nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv,noheader
|
||||
|
||||
python -m llm.experiment "configs/llm/society_campaign_s${PBS_ARRAY_INDEX}.yaml"
|
||||
|
||||
# results/llm_society_campaign/s${PBS_ARRAY_INDEX}/ written in-place. Sync back:
|
||||
# rsync -avz hpc:'.../results/llm_society_campaign/' results/llm_society_campaign/
|
||||
echo "done: $(date)"
|
||||
Loading…
Add table
Add a link
Reference in a new issue