Giorgio Gilestro
84124de143
Clarity pass over the main text (36-item audit), Discussion rewrite and cut, acknowledgements, Souly et al. as ref 62, lettered SI panels, model section moved under Results; plus the untracked curriculum/society/compose/smol configs, runners, figures, stats and tests that the SI already cites. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y64o8FKP7rCuXzC48pxpMm
31 lines
1.3 KiB
YAML
31 lines
1.3 KiB
YAML
# Generation-0 gate for the composition experiment (prereg v3 §4: C1 base, C3 specialists,
|
|
# C4 replication). Trains the two founders and measures everything at t=0 without iterating.
|
|
#
|
|
# Base choice (C1, measured 2026-09-07): Qwen2.5-1.5B **base, not Instruct**. Zero-shot GSM-Hard
|
|
# program-aided: Instruct-1.5B 0.500, Instruct-3B 0.417, base-3B 0.633 — all far too strong, since a
|
|
# base that already has the skills makes the specialists' contribution vacuous. Base-1.5B scores
|
|
# 0.067, within noise of the 0.059 Llama-2-7B starting point in LoRA Soups (COLING 2025), so the
|
|
# published composition effect has room to appear.
|
|
experiment: llm_compose_gate
|
|
kind: llm_compose
|
|
base_model: Qwen/Qwen2.5-1.5B
|
|
seed: 1
|
|
generations: 0 # gate only: train founders, measure t=0, stop
|
|
arms: [dry]
|
|
n_hard: 150 # composed target (GSM8k-Hard, program-aided) — SE ~0.04
|
|
n_gsm8k: 100 # math own-skill probe
|
|
n_mbpp: 80 # code own-skill probe (execution-verified)
|
|
n_probe: 40 # shared prompts for rho
|
|
k_inherit: 300
|
|
epochs: 3
|
|
conf_gate: 0.85
|
|
g: 0.10
|
|
spec_train: 1200
|
|
spec_epochs: 3
|
|
max_new_tokens: 320
|
|
batch_size: 16
|
|
score_batch_size: 4
|
|
train_batch_size: 2 # 16 GB A4000; the fp32 logit upcast at vocab 152k is the constraint
|
|
train_max_len: 448
|
|
lora: {r: 16, alpha: 32}
|
|
output: {dir: results/llm_compose_gate}
|