Manuscript revision and pending experiment work, snapshot before restructuring

Clarity pass over the main text (36-item audit), Discussion rewrite and cut, acknowledgements, Souly et al. as ref 62, lettered SI panels, model section moved under Results; plus the untracked curriculum/society/compose/smol configs, runners, figures, stats and tests that the SI already cites. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y64o8FKP7rCuXzC48pxpMm
2026-09-13 16:54:09 +01:00 · 2026-09-13 16:54:09 +01:00 · 84124de143
commit 84124de143
parent e4804adabc
450 changed files with 52813 additions and 1202 deletions
--- a/results/llm_moe_hard_hpc/README.md
+++ b/results/llm_moe_hard_hpc/README.md
@ -43,3 +43,34 @@ margin it does at 0.5B — E8's `max > mean` in real weights, robust across scal
 saturation. The `llm_moe_hpc` flip is re-read as a saturation artefact. **Falsifier (not triggered):**
 fusion matching/beating routing on hard tasks — instead fusion diluted below even the best specialist.
 Provenance in `manifest.json` (`hard: true`, L40S, torch 2.12.1 / transformers 5.13.0 / peft 0.19.1).
+
+## Seeds 1–3 (2026-09-11)
+
+Seeds 2–3 were run on CX3 via `hpc/llm_7b_seeds.pbs` (seed 1 above was moved to `s1/`; the bundle
+layout is now `s{seed}/`). Fixed test sets, training seed varied. Per-seed values and mean ± 95% CI
+from `figures/stats_llm_7b_seeds.py`:
+```
+model       metric  n_seeds    s1    s2    s3  mean  ci95
+best_specialist      overall        3 0.425 0.407 0.390 0.407 0.020
+best_specialist worst_family        3 0.155 0.150 0.185 0.163 0.021
+     merge_soup      overall        3 0.392 0.405 0.428 0.408 0.021
+     merge_soup worst_family        3 0.300 0.345 0.340 0.328 0.028
+     merge_ties      overall        3 0.400 0.427 0.438 0.422 0.022
+     merge_ties worst_family        3 0.330 0.385 0.380 0.365 0.034
+     moe_oracle      overall        3 0.500 0.498 0.510 0.503 0.007
+     moe_oracle worst_family        3 0.335 0.360 0.455 0.383 0.072
+    moe_learned      overall        3 0.500 0.498 0.510 0.503 0.007
+    moe_learned worst_family        3 0.335 0.360 0.455 0.383 0.072
+      max_merge      overall        3 0.297 0.342 0.403 0.347 0.061
+      max_merge worst_family        3 0.195 0.175 0.240 0.203 0.038
+
+                    contrast       metric  n_seeds     s1     s2    s3  mean  ci95 sign_agrees
+     moe_oracle − merge_soup      overall        3  0.108  0.093 0.082 0.094 0.015         3/3
+     moe_oracle − merge_soup worst_family        3  0.035  0.015 0.115 0.055 0.060         3/3
+    moe_learned − merge_soup      overall        3  0.108  0.093 0.082 0.094 0.015         3/3
+    moe_learned − merge_soup worst_family        3  0.035  0.015 0.115 0.055 0.060         3/3
+merge_soup − best_specialist      overall        3 -0.033 -0.002 0.038 0.001 0.041         1/3
+merge_soup − best_specialist worst_family        3  0.145  0.195 0.155 0.165 0.030         3/3
+```
+
+Reading: routing beats the weight-average in every seed (+0.094 ± 0.015 overall). The seed-1 observation that the soup fell *below* the best single specialist did not replicate (soup − best specialist overall −0.033, −0.002, +0.038; mean +0.001): over three seeds the soup matches the best parent overall and beats it on worst-family (+0.165, 3/3). `max_merge` remains the weakest union (0.347).
--- a/results/llm_moe_hard_hpc/s1/manifest.json
+++ b/results/llm_moe_hard_hpc/s1/manifest.json
@ -0,0 +1,28 @@
+{
+  "experiment": "llm_moe_hard_hpc",
+  "master_seed": 1,
+  "git_commit": null,
+  "python": "3.11.13",
+  "libraries": {
+    "numpy": "2.4.6",
+    "scipy": "1.17.1",
+    "pandas": "3.0.3",
+    "pyarrow": "24.0.0",
+    "torch": "2.12.1",
+    "transformers": "5.13.0",
+    "peft": "0.19.1"
+  },
+  "rows": 47,
+  "results_sha256": "a4e7c37e7cd900b56aa24ffc827e38e9a9d5cbb44dcff8d83cf010e992ed5c8a",
+  "layer": "2",
+  "tier": "llm",
+  "base_model": "Qwen/Qwen2.5-7B-Instruct",
+  "hard": true,
+  "operators": [
+    "soup",
+    "ties",
+    "moe_oracle",
+    "moe_learned",
+    "max_merge"
+  ]
+}
--- a/results/llm_moe_hard_hpc/s1/resolved_config.yaml
+++ b/results/llm_moe_hard_hpc/s1/resolved_config.yaml
@ -0,0 +1,29 @@
+experiment: llm_moe_hard_hpc
+seed: 1
+n_replicates: 1
+source_config:
+  experiment: llm_moe_hard_hpc
+  kind: llm_moe
+  seed: 1
+  n_replicates: 1
+  base_model: Qwen/Qwen2.5-7B-Instruct
+  hard: true
+  families:
+  - lists
+  - strings
+  - arith
+  n_train: 800
+  n_test: 200
+  n_route: 48
+  epochs: 3
+  lora:
+    r: 16
+    alpha: 32
+  operators:
+  - soup
+  - ties
+  - moe_oracle
+  - moe_learned
+  - max_merge
+  output:
+    dir: results/llm_moe_hard_hpc
--- a/results/llm_moe_hard_hpc/s2/manifest.json
+++ b/results/llm_moe_hard_hpc/s2/manifest.json
@ -0,0 +1,28 @@
+{
+  "experiment": "llm_moe_hard_hpc",
+  "master_seed": 2,
+  "git_commit": null,
+  "python": "3.11.7",
+  "libraries": {
+    "numpy": "2.4.6",
+    "scipy": "1.17.1",
+    "pandas": "3.0.3",
+    "pyarrow": "24.0.0",
+    "torch": "2.12.1",
+    "transformers": "5.16.1",
+    "peft": "0.20.0"
+  },
+  "rows": 47,
+  "results_sha256": "60202401aa7a95c011736322389efec0c078f9fb9f0d0594cfa9002146ae97fb",
+  "layer": "2",
+  "tier": "llm",
+  "base_model": "Qwen/Qwen2.5-7B-Instruct",
+  "hard": true,
+  "operators": [
+    "soup",
+    "ties",
+    "moe_oracle",
+    "moe_learned",
+    "max_merge"
+  ]
+}
--- a/results/llm_moe_hard_hpc/s2/resolved_config.yaml
+++ b/results/llm_moe_hard_hpc/s2/resolved_config.yaml
@ -0,0 +1,29 @@
+experiment: llm_moe_hard_hpc
+seed: 2
+n_replicates: 1
+source_config:
+  experiment: llm_moe_hard_hpc
+  kind: llm_moe
+  seed: 2
+  n_replicates: 1
+  base_model: Qwen/Qwen2.5-7B-Instruct
+  hard: true
+  families:
+  - lists
+  - strings
+  - arith
+  n_train: 800
+  n_test: 200
+  n_route: 48
+  epochs: 3
+  lora:
+    r: 16
+    alpha: 32
+  operators:
+  - soup
+  - ties
+  - moe_oracle
+  - moe_learned
+  - max_merge
+  output:
+    dir: results/llm_moe_hard_hpc/s2
--- a/results/llm_moe_hard_hpc/s3/manifest.json
+++ b/results/llm_moe_hard_hpc/s3/manifest.json
@ -0,0 +1,28 @@
+{
+  "experiment": "llm_moe_hard_hpc",
+  "master_seed": 3,
+  "git_commit": null,
+  "python": "3.11.7",
+  "libraries": {
+    "numpy": "2.4.6",
+    "scipy": "1.17.1",
+    "pandas": "3.0.3",
+    "pyarrow": "24.0.0",
+    "torch": "2.12.1",
+    "transformers": "5.16.1",
+    "peft": "0.20.0"
+  },
+  "rows": 47,
+  "results_sha256": "1a9962f95ed0b0458676682d415fafcaa530a11eefc3833ba46a144e22ee90d5",
+  "layer": "2",
+  "tier": "llm",
+  "base_model": "Qwen/Qwen2.5-7B-Instruct",
+  "hard": true,
+  "operators": [
+    "soup",
+    "ties",
+    "moe_oracle",
+    "moe_learned",
+    "max_merge"
+  ]
+}
--- a/results/llm_moe_hard_hpc/s3/resolved_config.yaml
+++ b/results/llm_moe_hard_hpc/s3/resolved_config.yaml
@ -0,0 +1,29 @@
+experiment: llm_moe_hard_hpc
+seed: 3
+n_replicates: 1
+source_config:
+  experiment: llm_moe_hard_hpc
+  kind: llm_moe
+  seed: 3
+  n_replicates: 1
+  base_model: Qwen/Qwen2.5-7B-Instruct
+  hard: true
+  families:
+  - lists
+  - strings
+  - arith
+  n_train: 800
+  n_test: 200
+  n_route: 48
+  epochs: 3
+  lora:
+    r: 16
+    alpha: 32
+  operators:
+  - soup
+  - ties
+  - moe_oracle
+  - moe_learned
+  - max_merge
+  output:
+    dir: results/llm_moe_hard_hpc/s3