Restructure: descriptive tier and experiment names, paper/manuscript
- paper/pnas -> paper/manuscript (venue-neutral)
- configs/layer1 -> configs/inheritance, src/knowledge -> src/inheritance
(imported as `inheritance`), make layer1 -> make inheritance; layer2 alias dropped
- inheritance and trained-network bundles named after the manuscript figure
they feed (fig2_grounding_sweep, figS3_rebaselining, ...), or descriptively
where they feed none; configs keep their `experiment:` value so parquet
hashes are unchanged, only output.dir moves
- figure scripts, SI figure sources, notebooks, REPRODUCING.md, README and the
SI Methods/tables updated; make clean no longer deletes tracked manifests;
reproduce.sh hashes the s{seed}/ layouts too
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y64o8FKP7rCuXzC48pxpMm
This commit is contained in:
parent
84124de143
commit
ab3dc10587
240 changed files with 477 additions and 476 deletions
33
results/figS5_aimed_grounding/README.md
Normal file
33
results/figS5_aimed_grounding/README.md
Normal file
|
|
@ -0,0 +1,33 @@
|
|||
# E3 — Grounding must overlap the content it protects
|
||||
|
||||
**Claim tested:** if your budget of real data is fixed, does it matter *where* you spend it? Is it
|
||||
enough to sprinkle real data uniformly, or must it target the knowledge you care about?
|
||||
|
||||
**Setup (Layer 1, pure math).** `K = 1000` items divided into `R = 10` disjoint **regions** (think
|
||||
subject areas). `n = 200`, 400 generations, 100 repeats. The **same total** real-data budget is
|
||||
spent two ways: **uniform** (spread evenly over all 10 regions) vs **matched** (concentrated on the
|
||||
"exercised" region we want to protect). One region is designated exercised (region 0 here) and has a
|
||||
rare tail we track.
|
||||
|
||||
### Symbols
|
||||
- **region** — a block of related items; a stand-in for a task family / subject area.
|
||||
- **matched vs uniform** — real data aimed at the exercised region vs spread evenly, at equal total budget.
|
||||
- **tail items alive** — how many rare items in a region still have any probability.
|
||||
|
||||
### The two panels
|
||||
1. **Target region over time.** Rare-item survival in the exercised region, generation by generation:
|
||||
blue = matched, red = uniform (bands = 95% CI). Matched **holds** the region's tail alive; uniform
|
||||
spreads too thin and lets it **collapse**.
|
||||
2. **Every region at steady state.** Bar chart of stationary tail survival per region, matched (blue)
|
||||
vs uniform (red); the dotted line marks the exercised region. Matched wins big *there* — at the
|
||||
cost of the regions it deliberately ignores. Uniform is mediocre everywhere.
|
||||
|
||||
*(Note: per-region diversity `H` is confounded by how much probability mass sits in a region, so this
|
||||
figure uses the honest, mass-independent metric — tail-item survival.)*
|
||||
|
||||
### Takeaway
|
||||
Reality checks only protect what they actually cover. To keep a capability alive you must ground
|
||||
**on that capability**, not on data in general — grounding is local, not a global tonic. Key numbers:
|
||||
exercised-region tail survival ≈ **0.49 (matched)** vs **0.07 (uniform)** at equal budget.
|
||||
**Falsifier (not triggered):** if uniform had protected the region as well as matched, the
|
||||
targeting claim would die.
|
||||
BIN
results/figS5_aimed_grounding/figS5_aimed_grounding.pdf
Normal file
BIN
results/figS5_aimed_grounding/figS5_aimed_grounding.pdf
Normal file
Binary file not shown.
BIN
results/figS5_aimed_grounding/figS5_aimed_grounding.png
Normal file
BIN
results/figS5_aimed_grounding/figS5_aimed_grounding.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 125 KiB |
14
results/figS5_aimed_grounding/manifest.json
Normal file
14
results/figS5_aimed_grounding/manifest.json
Normal file
|
|
@ -0,0 +1,14 @@
|
|||
{
|
||||
"experiment": "E3_region_matched_grounding",
|
||||
"master_seed": 20260704,
|
||||
"git_commit": "a6eb9b75124779375fa1a0b3a64115ecd705b218",
|
||||
"python": "3.14.5",
|
||||
"libraries": {
|
||||
"numpy": "2.5.0",
|
||||
"scipy": "1.18.0",
|
||||
"pandas": "3.0.3",
|
||||
"pyarrow": "24.0.0"
|
||||
},
|
||||
"rows": 80200,
|
||||
"results_sha256": "7ae056faf8db591a913087004068d0e732f10b0244cdc6a78bb72a691f4b8604"
|
||||
}
|
||||
111
results/figS5_aimed_grounding/resolved_config.yaml
Normal file
111
results/figS5_aimed_grounding/resolved_config.yaml
Normal file
|
|
@ -0,0 +1,111 @@
|
|||
experiment: E3_region_matched_grounding
|
||||
seed: 20260704
|
||||
n_replicates: 100
|
||||
source_config:
|
||||
experiment: E3_region_matched_grounding
|
||||
seed: 20260704
|
||||
n_replicates: 100
|
||||
generations: 400
|
||||
truth:
|
||||
K: 1000
|
||||
R: 10
|
||||
tail: zipf
|
||||
zipf_s: 1.1
|
||||
tail_frac: 0.5
|
||||
tail_threshold: 0.001
|
||||
init: truth
|
||||
dynamics:
|
||||
n: 200
|
||||
teachers:
|
||||
K_T: 1
|
||||
rho: 0.0
|
||||
q: 1.0
|
||||
grounding:
|
||||
m: 100
|
||||
policy: uniform
|
||||
exercised:
|
||||
- 0
|
||||
selection:
|
||||
mode: none
|
||||
novelty_alpha: 0.0
|
||||
remint:
|
||||
enabled: false
|
||||
period: null
|
||||
H_gate: null
|
||||
metrics:
|
||||
kl_floor: 1.0e-09
|
||||
support_eps: 1.0e-09
|
||||
sweep:
|
||||
- param: dynamics.grounding.policy
|
||||
values:
|
||||
- uniform
|
||||
- matched
|
||||
output:
|
||||
dir: results/figS5_aimed_grounding
|
||||
grid:
|
||||
- label:
|
||||
policy: uniform
|
||||
lineage_cfg:
|
||||
truth:
|
||||
K: 1000
|
||||
R: 10
|
||||
tail: zipf
|
||||
zipf_s: 1.1
|
||||
tail_frac: 0.5
|
||||
tail_threshold: 0.001
|
||||
init: truth
|
||||
dynamics:
|
||||
n: 200
|
||||
teachers:
|
||||
K_T: 1
|
||||
rho: 0.0
|
||||
q: 1.0
|
||||
grounding:
|
||||
m: 100
|
||||
policy: uniform
|
||||
exercised:
|
||||
- 0
|
||||
selection:
|
||||
mode: none
|
||||
novelty_alpha: 0.0
|
||||
remint:
|
||||
enabled: false
|
||||
period: null
|
||||
H_gate: null
|
||||
generations: 400
|
||||
metrics:
|
||||
kl_floor: 1.0e-09
|
||||
support_eps: 1.0e-09
|
||||
- label:
|
||||
policy: matched
|
||||
lineage_cfg:
|
||||
truth:
|
||||
K: 1000
|
||||
R: 10
|
||||
tail: zipf
|
||||
zipf_s: 1.1
|
||||
tail_frac: 0.5
|
||||
tail_threshold: 0.001
|
||||
init: truth
|
||||
dynamics:
|
||||
n: 200
|
||||
teachers:
|
||||
K_T: 1
|
||||
rho: 0.0
|
||||
q: 1.0
|
||||
grounding:
|
||||
m: 100
|
||||
policy: matched
|
||||
exercised:
|
||||
- 0
|
||||
selection:
|
||||
mode: none
|
||||
novelty_alpha: 0.0
|
||||
remint:
|
||||
enabled: false
|
||||
period: null
|
||||
H_gate: null
|
||||
generations: 400
|
||||
metrics:
|
||||
kl_floor: 1.0e-09
|
||||
support_eps: 1.0e-09
|
||||
Loading…
Add table
Add a link
Reference in a new issue