Giorgio Gilestro
84124de143
Clarity pass over the main text (36-item audit), Discussion rewrite and cut, acknowledgements, Souly et al. as ref 62, lettered SI panels, model section moved under Results; plus the untracked curriculum/society/compose/smol configs, runners, figures, stats and tests that the SI already cites. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y64o8FKP7rCuXzC48pxpMm |
||
|---|---|---|
| .. | ||
| architectures.pdf | ||
| architectures.png | ||
| manifest.json | ||
| README.md | ||
| resolved_config.yaml | ||
architectures — collapse and rescue are architecture-general
Claim tested: is model collapse (and its cure, grounding) a quirk of one model type, or does the same signature appear across genuinely different neural architectures?
Setup (Layer 1.5). The identical generational loop is run with three different generative models —
an exact histogram (no neural net), an autoregressive GRU (recurrent), and a causal-masked
MLP (feed-forward) — each a distinct "inductive bias." K = 256 modes, n = 200, 22 generations,
5 repeats, compared at dry (g = 0) vs grounded (g = 0.05).
Symbols
- inductive bias — the built-in assumptions of a model type (a histogram has none; a GRU and an MLP smooth differently).
- forward-KL distance from truth; tail items alive — how many rare modes still appear.
- dry = no grounding; grounded = 5% real data mixed in.
The three panels
- Trajectories. Forward-KL over generations, coloured by architecture; solid = dry (climbs, collapse) vs dashed = grounded (held down). The dry-up / grounded-down gap appears in every architecture.
- Stationary forward-KL (grouped bars). For each architecture, dry (red) vs grounded (green). Divergence falls with grounding across all three — histogram, GRU, MLP.
- Tail-item survival (grouped bars). Same grouping. Survival rises with grounding across all three. (Note the histogram's bars are tiny: with no smoothing it drops rare modes outright, whereas the GRU/MLP keep some alive — an inductive-bias difference, not a contradiction.)
Takeaway
The Wright–Fisher collapse operator and the grounding rescue are not artefacts of one model — they show up in an exact counter, a recurrent net, and a feed-forward net alike. This is the architecture-generality claim of Layer 1.5. Falsifier (not triggered): if the signs had appeared only for the histogram, collapse would be a property of the idealised operator, not of trained models. (A VAE was also implemented but fails the generation-0 fidelity check on this task, so it is excluded to avoid confusing underfitting with collapse — documented as a known limitation.)