main: keep only what reproduces the manuscript; everything else lives on dev

Removed from main (all preserved on the dev branch): the arXiv build and
its sources, design documents (blueprint, results summary, review responses,
essay drafts), tasks/ and CLAUDE.md, the cover letter and reference tooling,
two unused manuscript figures, and every experiment that feeds no figure or
number in the paper: the collapse null, the sexual-vs-asexual lineage, the
NK speciation variant, the 0.5B single-seed LLM prototypes, the compose and
society experiments with their calibration and pilot runs, and their
configs, runners, tests, figure scripts and PBS jobs. Their result bundles
are moved to results/_archive/ (ignored) so the parquets stay on disk.

Also: plot_llm_speciation reads the s{seed}/ layout; the mating-breadth
plot writes under its bundle name; Makefile targets reduced to the kept
experiments; REPRODUCING.md and README point to dev for the rest.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y64o8FKP7rCuXzC48pxpMm
This commit is contained in:
Giorgio Gilestro 2026-09-13 17:07:23 +01:00
parent ab3dc10587
commit 6f8cef1ac5
292 changed files with 26 additions and 15590 deletions

View file

@ -1,43 +0,0 @@
#!/bin/bash
# The composition campaign (prereg tasks/prereg-llm-compose-v3.md): one (seed, arm) per array
# element. Seed 1 runs locally on the A4000 as the hedge; this array covers seeds 2-3 x 3 arms.
# Each element trains its own founders (cached per seed under models/, first writer wins via the
# adapter_config.json check) and checkpoints every generation, so a requeued element resumes.
# submit: qsub hpc/llm_compose.pbs status: qstat -u $USER -t
# index -> seed = 2 + (i-1)/3, arm = (dry grounded dry_cat)[(i-1)%3]
#PBS -l select=1:ncpus=8:mem=64gb:ngpus=1:gpu_type=L40S
#PBS -l walltime=08:00:00
#PBS -N lam_compose
#PBS -J 1-6
cd "$PBS_O_WORKDIR"
export HF_HOME="$EPHEMERAL/hf_cache"
export HF_DATASETS_CACHE="$EPHEMERAL/hf_cache/datasets"
export TOKENIZERS_PARALLELISM=false
export UV_CACHE_DIR="$EPHEMERAL/uvcache"
ARMS=(dry grounded dry_cat)
I=$((PBS_ARRAY_INDEX - 1))
SEED=$((2 + I / 3))
ARM=${ARMS[$((I % 3))]}
source .venv/bin/activate
nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv,noheader
echo "seed=$SEED arm=$ARM start=$(date)"
CFG="configs/llm/_gen/compose_s${SEED}_${ARM}.yaml"
mkdir -p configs/llm/_gen
python - "$SEED" "$ARM" "$CFG" <<'EOF'
import sys, yaml
seed, arm, out = int(sys.argv[1]), sys.argv[2], sys.argv[3]
cfg = yaml.safe_load(open("configs/llm/compose_s1.yaml"))
cfg["seed"] = seed
cfg["arms"] = [arm]
cfg["batch_size"] = 32 # L40S has 46 GB; the A4000 setting is 16
cfg["score_batch_size"] = 8
cfg["output"] = {"dir": f"results/llm_compose/s{seed}_{arm}"}
yaml.safe_dump(cfg, open(out, "w"), sort_keys=False)
EOF
python -m llm.experiment "$CFG"
echo "done: $(date)"

View file

@ -1,14 +0,0 @@
#!/bin/bash
# Smoke gate: run the small 0.5B merge experiment on an L40S to confirm the GPU pipeline works
# on CX3 before the 4 h 7B run. ~5 min. submit: qsub hpc/llm_smoke.pbs
#PBS -l select=1:ncpus=8:mem=48gb:ngpus=1:gpu_type=L40S
#PBS -l walltime=00:20:00
#PBS -N lam_llm_smoke
cd "$PBS_O_WORKDIR"
export HF_HOME="$EPHEMERAL/hf_cache"
export TOKENIZERS_PARALLELISM=false
source .venv/bin/activate
nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv,noheader
python -m llm.experiment configs/llm/merge.yaml # 0.5B, writes results/llm_merge/
echo "done: $(date)"

View file

@ -1,25 +0,0 @@
#!/bin/bash
# The LLM society campaign (C3): 4-arm ablation, one seed per array element on one L40S each.
# Seed 1 runs locally on GG's A4000 (queue insurance); this array covers seeds 2-4. Each job is
# self-contained (founders trained inline, adapters under models/, deleted per generation), so a
# killed element reruns cleanly and elements schedule independently. ~2.5-3.5 h measured-scale
# runtime; 6 h walltime for backfill-friendly margin.
# submit: qsub hpc/llm_society.pbs status: qstat -u $USER
#PBS -l select=1:ncpus=8:mem=64gb:ngpus=1:gpu_type=L40S
#PBS -l walltime=06:00:00
#PBS -N lam_society
#PBS -J 2-4
cd "$PBS_O_WORKDIR"
export HF_HOME="$EPHEMERAL/hf_cache"
export TOKENIZERS_PARALLELISM=false
export UV_CACHE_DIR="$EPHEMERAL/uvcache"
source .venv/bin/activate
nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv,noheader
python -m llm.experiment "configs/llm/society_campaign_s${PBS_ARRAY_INDEX}.yaml"
# results/llm_society_campaign/s${PBS_ARRAY_INDEX}/ written in-place. Sync back:
# rsync -avz hpc:'.../results/llm_society_campaign/' results/llm_society_campaign/
echo "done: $(date)"

View file

@ -1,42 +0,0 @@
#!/bin/bash
# The v2 society campaign (prereg tasks/prereg-llm-society-v2.md §9): one (seed, arm) per array
# element on one L40S each, 16 elements = 4 seeds x 4 arms. Each element is self-contained: founders
# are trained inline and cached per seed (the four arm-elements of a seed share them via the
# filesystem; the first to arrive trains, the others wait on the adapter_config.json check), the loop
# checkpoints every generation and resumes, so a killed element is re-queued with the same index and
# picks up where it stopped. ~6 h per element at k_inherit=300 (prereg §9); 8 h walltime.
# submit: qsub hpc/llm_society_v2.pbs status: qstat -u $USER -t
# index -> seed = 1 + (i-1) // 4, arm = (full no_grounding no_sex no_diversity)[(i-1) % 4]
#PBS -l select=1:ncpus=8:mem=64gb:ngpus=1:gpu_type=L40S
#PBS -l walltime=08:00:00
#PBS -N lam_society_v2
#PBS -J 1-16
cd "$PBS_O_WORKDIR"
export HF_HOME="$EPHEMERAL/hf_cache"
export TOKENIZERS_PARALLELISM=false
export UV_CACHE_DIR="$EPHEMERAL/uvcache"
ARMS=(full no_grounding no_sex no_diversity)
I=$((PBS_ARRAY_INDEX - 1))
SEED=$((1 + I / 4))
ARM=${ARMS[$((I % 4))]}
source .venv/bin/activate
nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv,noheader
echo "seed=$SEED arm=$ARM start=$(date)"
# One config per (seed, arm): generated from the seed template so the resolved config is exact.
CFG="configs/llm/_gen/society_v2_s${SEED}_${ARM}.yaml"
mkdir -p configs/llm/_gen
python - "$SEED" "$ARM" "$CFG" <<'EOF'
import sys, yaml
seed, arm, out = int(sys.argv[1]), sys.argv[2], sys.argv[3]
cfg = yaml.safe_load(open(f"configs/llm/society_v2_s{seed}.yaml"))
cfg["arms"] = [arm]
cfg["output"] = {"dir": f"results/llm_society_v2/s{seed}_{arm}"}
yaml.safe_dump(cfg, open(out, "w"), sort_keys=False)
EOF
python -m llm.experiment "$CFG"
echo "done: $(date)"