Giorgio Gilestro
84124de143
Clarity pass over the main text (36-item audit), Discussion rewrite and cut, acknowledgements, Souly et al. as ref 62, lettered SI panels, model section moved under Results; plus the untracked curriculum/society/compose/smol configs, runners, figures, stats and tests that the SI already cites. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y64o8FKP7rCuXzC48pxpMm
1393 lines
110 KiB
Text
1393 lines
110 KiB
Text
TY - JOUR
|
||
AU - Laufer, Benjamin
|
||
AU - Oderinwale, Hamidah
|
||
AU - Kleinberg, Jon
|
||
TI - Anatomy of a Machine Learning Ecosystem: 2 Million Models on Hugging Face
|
||
T2 - arXiv
|
||
AN - arXiv:2508.06811
|
||
PY - 2025
|
||
PB - arXiv
|
||
DO - 10.48550/arXiv.2508.06811
|
||
UR - https://arxiv.org/abs/2508.06811
|
||
AB - Many have observed that the development and deployment of generative machine learning (ML) and artificial intelligence (AI) models follow a distinctive pattern in which pre-trained models are adapted and fine-tuned for specific downstream tasks. However, there is limited empirical work that examines the structure of these interactions. This paper analyzes 1.86 million models on Hugging Face, a leading peer production platform for model development. Our study of model family trees -- networks that connect fine-tuned models to their base or parent -- reveals sprawling fine-tuning lineages that vary widely in size and structure. Using an evolutionary biology lens to study ML models, we use model metadata and model cards to measure the genetic similarity and mutation of traits over model families. We find that models tend to exhibit a family resemblance, meaning their genetic markers and traits exhibit more overlap when they belong to the same model family. However, these similarities depart in certain ways from standard models of asexual reproduction, because mutations are fast and directed, such that two `sibling' models tend to exhibit more similarity than parent/child pairs. Further analysis of the directional drifts of these mutations reveals qualitative insights about the open machine learning ecosystem: Licenses counter-intuitively drift from restrictive, commercial licenses towards permissive or copyleft licenses, often in violation of upstream license's terms; models evolve from multi-lingual compatibility towards english-only compatibility; and model cards reduce in length and standardize by turning, more often, to templates and automatically generated text. Overall, this work takes a step toward an empirically grounded understanding of model fine-tuning and suggests that ecological models and methods can yield novel scientific insights.
|
||
N1 - Manuscript reference 1
|
||
ER -
|
||
TY - JOUR
|
||
AU - Horwitz, Eliahu
|
||
AU - Shul, Asaf
|
||
AU - Hoshen, Yedid
|
||
TI - Unsupervised Model Tree Heritage Recovery
|
||
T2 - arXiv
|
||
AN - arXiv:2405.18432
|
||
PY - 2024
|
||
PB - arXiv
|
||
DO - 10.48550/arXiv.2405.18432
|
||
UR - https://arxiv.org/abs/2405.18432
|
||
AB - The number of models shared online has recently skyrocketed, with over one million public models available on Hugging Face. Sharing models allows other users to build on existing models, using them as initialization for fine-tuning, improving accuracy, and saving compute and energy. However, it also raises important intellectual property issues, as fine-tuning may violate the license terms of the original model or that of its training data. A Model Tree, i.e., a tree data structure rooted at a foundation model and having directed edges between a parent model and other models directly fine-tuned from it (children), would settle such disputes by making the model heritage explicit. Unfortunately, current models are not well documented, with most model metadata (e.g., "model cards") not providing accurate information about heritage. In this paper, we introduce the task of Unsupervised Model Tree Heritage Recovery (Unsupervised MoTHer Recovery) for collections of neural networks. For each pair of models, this task requires: i) determining if they are directly related, and ii) establishing the direction of the relationship. Our hypothesis is that model weights encode this information, the challenge is to decode the underlying tree structure given the weights. We discover several properties of model weights that allow us to perform this task. By using these properties, we formulate the MoTHer Recovery task as finding a directed minimal spanning tree. In extensive experiments we demonstrate that our method successfully reconstructs complex Model Trees.
|
||
N1 - Manuscript reference 2
|
||
ER -
|
||
TY - JOUR
|
||
AU - Jiang, Wenxin
|
||
AU - Yasmin, Jerin
|
||
AU - Jones, Jason
|
||
AU - Synovic, Nicholas
|
||
AU - Kuo, Jiashen
|
||
AU - Bielanski, Nathaniel
|
||
AU - Tian, Yuan
|
||
AU - Thiruvathukal, George K.
|
||
AU - Davis, James C.
|
||
TI - PeaTMOSS: A Dataset and Initial Analysis of Pre-Trained Models in Open-Source Software
|
||
T2 - arXiv
|
||
AN - arXiv:2402.00699
|
||
PY - 2024
|
||
PB - arXiv
|
||
DO - 10.48550/arXiv.2402.00699
|
||
UR - https://arxiv.org/abs/2402.00699
|
||
AB - The development and training of deep learning models have become increasingly costly and complex. Consequently, software engineers are adopting pre-trained models (PTMs) for their downstream applications. The dynamics of the PTM supply chain remain largely unexplored, signaling a clear need for structured datasets that document not only the metadata but also the subsequent applications of these models. Without such data, the MSR community cannot comprehensively understand the impact of PTM adoption and reuse. This paper presents the PeaTMOSS dataset, which comprises metadata for 281,638 PTMs and detailed snapshots for all PTMs with over 50 monthly downloads (14,296 PTMs), along with 28,575 open-source software repositories from GitHub that utilize these models. Additionally, the dataset includes 44,337 mappings from 15,129 downstream GitHub repositories to the 2,530 PTMs they use. To enhance the dataset's comprehensiveness, we developed prompts for a large language model to automatically extract model metadata, including the model's training datasets, parameters, and evaluation metrics. Our analysis of this dataset provides the first summary statistics for the PTM supply chain, showing the trend of PTM development and common shortcomings of PTM package documentation. Our example application reveals inconsistencies in software licenses across PTMs and their dependent projects. PeaTMOSS lays the foundation for future research, offering rich opportunities to investigate the PTM supply chain. We outline mining opportunities on PTMs, their downstream usage, and cross-cutting questions.
|
||
N1 - Manuscript reference 3
|
||
ER -
|
||
TY - JOUR
|
||
AU - Yadav, Prateek
|
||
AU - Tam, Derek
|
||
AU - Choshen, Leshem
|
||
AU - Raffel, Colin
|
||
AU - Bansal, Mohit
|
||
TI - TIES-Merging: Resolving Interference When Merging Models
|
||
T2 - arXiv
|
||
AN - arXiv:2306.01708
|
||
PY - 2023
|
||
PB - arXiv
|
||
DO - 10.48550/arXiv.2306.01708
|
||
UR - https://arxiv.org/abs/2306.01708
|
||
AB - Transfer learning - i.e., further fine-tuning a pre-trained model on a downstream task - can confer significant advantages, including improved downstream performance, faster convergence, and better sample efficiency. These advantages have led to a proliferation of task-specific fine-tuned models, which typically can only perform a single task and do not benefit from one another. Recently, model merging techniques have emerged as a solution to combine multiple task-specific models into a single multitask model without performing additional training. However, existing merging methods often ignore the interference between parameters of different models, resulting in large performance drops when merging multiple models. In this paper, we demonstrate that prior merging techniques inadvertently lose valuable information due to two major sources of interference: (a) interference due to redundant parameter values and (b) disagreement on the sign of a given parameter's values across models. To address this, we propose our method, TRIM, ELECT SIGN & MERGE (TIES-Merging), which introduces three novel steps when merging models: (1) resetting parameters that only changed a small amount during fine-tuning, (2) resolving sign conflicts, and (3) merging only the parameters that are in alignment with the final agreed-upon sign. We find that TIES-Merging outperforms several existing methods in diverse settings covering a range of modalities, domains, number of tasks, model sizes, architectures, and fine-tuning settings. We further analyze the impact of different types of interference on model parameters, and highlight the importance of resolving sign interference. Our code is available at https://github.com/prateeky2806/ties-merging
|
||
N1 - Manuscript reference 4
|
||
ER -
|
||
TY - JOUR
|
||
AU - Akiba, Takuya
|
||
AU - Shing, Makoto
|
||
AU - Tang, Yujin
|
||
AU - Sun, Qi
|
||
AU - Ha, David
|
||
TI - Evolutionary optimization of model merging recipes
|
||
T2 - Nature Machine Intelligence
|
||
PY - 2025
|
||
VL - 7
|
||
IS - 2
|
||
PB - Springer Science and Business Media LLC
|
||
DO - 10.1038/s42256-024-00975-8
|
||
UR - http://dx.doi.org/10.1038/s42256-024-00975-8
|
||
AB - Abstract Large language models (LLMs) have become increasingly capable, but their development often requires substantial computational resources. Although model merging has emerged as a cost-effective promising approach for creating new models by combining existing ones, it currently relies on human intuition and domain knowledge, limiting its potential. Here we propose an evolutionary approach that overcomes this limitation by automatically discovering effective combinations of diverse open-source models, harnessing their collective intelligence without requiring extensive additional training data or compute. Our approach operates in both parameter space and data flow space, allowing optimization beyond just the weights of the individual models. This approach even facilitates cross-domain merging, generating models such as a Japanese LLM with math reasoning capabilities. Surprisingly, our Japanese math LLM achieved state-of-the-art performance on a variety of established Japanese LLM benchmarks, even surpassing models with substantially more parameters, despite not being explicitly trained for such tasks. Furthermore, a culturally aware Japanese vision–language model generated through our approach demonstrates its effectiveness in describing Japanese culture-specific content, outperforming previous Japanese vision–language models. This work not only contributes new state-of-the-art models back to the open-source community but also introduces a new paradigm for automated model composition, paving the way for exploring alternative, efficient approaches to foundation model development.
|
||
SP - 195
|
||
EP - 204
|
||
N1 - Manuscript reference 5
|
||
ER -
|
||
TY - JOUR
|
||
AU - Goddard, Charles
|
||
AU - Siriwardhana, Shamane
|
||
AU - Ehghaghi, Malikeh
|
||
AU - Meyers, Luke
|
||
AU - Karpukhin, Vlad
|
||
AU - Benedict, Brian
|
||
AU - McQuade, Mark
|
||
AU - Solawetz, Jacob
|
||
TI - Arcee's MergeKit: A Toolkit for Merging Large Language Models
|
||
T2 - arXiv
|
||
AN - arXiv:2403.13257
|
||
PY - 2024
|
||
PB - arXiv
|
||
DO - 10.48550/arXiv.2403.13257
|
||
UR - https://arxiv.org/abs/2403.13257
|
||
AB - The rapid expansion of the open-source language model landscape presents an opportunity to merge the competencies of these model checkpoints by combining their parameters. Advances in transfer learning, the process of fine-tuning pretrained models for specific tasks, has resulted in the development of vast amounts of task-specific models, typically specialized in individual tasks and unable to utilize each other's strengths. Model merging facilitates the creation of multitask models without the need for additional training, offering a promising avenue for enhancing model performance and versatility. By preserving the intrinsic capabilities of the original models, model merging addresses complex challenges in AI - including the difficulties of catastrophic forgetting and multitask learning. To support this expanding area of research, we introduce MergeKit, a comprehensive, open-source library designed to facilitate the application of model merging strategies. MergeKit offers an extensible framework to efficiently merge models on any hardware, providing utility to researchers and practitioners. To date, thousands of models have been merged by the open-source community, leading to the creation of some of the worlds most powerful open-source model checkpoints, as assessed by the Open LLM Leaderboard. The library is accessible at https://github.com/arcee-ai/MergeKit.
|
||
N1 - Manuscript reference 6
|
||
ER -
|
||
TY - JOUR
|
||
AU - Yang, Enneng
|
||
AU - Shen, Li
|
||
AU - Guo, Guibing
|
||
AU - Wang, Xingwei
|
||
AU - Cao, Xiaochun
|
||
AU - Zhang, Jie
|
||
AU - Tao, Dacheng
|
||
TI - Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities
|
||
T2 - arXiv
|
||
AN - arXiv:2408.07666
|
||
PY - 2024
|
||
PB - arXiv
|
||
DO - 10.48550/arXiv.2408.07666
|
||
UR - https://arxiv.org/abs/2408.07666
|
||
AB - Model merging is an efficient empowerment technique in the machine learning community that does not require the collection of raw training data and does not require expensive computation. As model merging becomes increasingly prevalent across various fields, it is crucial to understand the available model merging techniques comprehensively. However, there is a significant gap in the literature regarding a systematic and thorough review of these techniques. This survey provides a comprehensive overview of model merging methods and theories, their applications in various domains and settings, and future research directions. Specifically, we first propose a new taxonomic approach that exhaustively discusses existing model merging methods. Secondly, we discuss the application of model merging techniques in large language models, multimodal large language models, and more than ten machine learning subfields, including continual learning, multi-task learning, few-shot learning, etc. Finally, we highlight the remaining challenges of model merging and discuss future research directions. A comprehensive list of papers about model merging is available at https://github.com/EnnengYang/Awesome-Model-Merging-Methods-Theories-Applications.
|
||
N1 - Manuscript reference 7
|
||
ER -
|
||
TY - JOUR
|
||
AU - Zhang, Yiqun
|
||
AU - Ye, Peng
|
||
AU - Yang, Xiaocui
|
||
AU - Feng, Shi
|
||
AU - Zhang, Shufei
|
||
AU - Bai, Lei
|
||
AU - Ouyang, Wanli
|
||
AU - Hu, Shuyue
|
||
TI - Nature-Inspired Population-Based Evolution of Large Language Models
|
||
T2 - arXiv
|
||
AN - arXiv:2503.01155
|
||
PY - 2025
|
||
PB - arXiv
|
||
DO - 10.48550/arXiv.2503.01155
|
||
UR - https://arxiv.org/abs/2503.01155
|
||
AB - Evolution, the engine behind the survival and growth of life on Earth, operates through the population-based process of reproduction. Inspired by this principle, this paper formally defines a newly emerging problem -- the population-based evolution of large language models (LLMs) -- and introduces a novel framework. Starting with a population of parent LLMs, our framework enables the population to evolve through four key operations: (i) crossover, merging the weights of different parents to create offspring LLMs, (ii) mutation, introducing small, random changes to model weights to foster diversity, (iii) selection, prioritizing high-performing models, and (iv) succession, transferring the learned experience from parent to offspring LLMs. With only 200 samples per new task, the LLM population evolves rapidly to adapt to the task at hand, without any gradients. Experiments on 12 datasets show that our framework consistently outperforms existing multi-LLM merging and adaptation methods, achieving accuracy gains of up to 54.8% over the best LLM in the initial population. Moreover, our framework allows for the evolution of LLMs across multiple new tasks simultaneously, scaling effectively with populations of up to 40 LLMs, and even zero-shot generalization to unseen held-out tasks. We have open-sourced the code on GitHub and released the weights of 10 parent LLMs, fine-tuned from gemma-2-2b-it, on HuggingFace$, enabling reproduction of our proposed framework using just a single 4090 GPU with 24GB memory, without any performance degradation.
|
||
N1 - Manuscript reference 8
|
||
ER -
|
||
TY - JOUR
|
||
AU - Abrantes, João
|
||
AU - Lange, Robert Tjarko
|
||
AU - Tang, Yujin
|
||
TI - Competition and Attraction Improve Model Fusion
|
||
T2 - arXiv
|
||
AN - arXiv:2508.16204
|
||
PY - 2025
|
||
PB - arXiv
|
||
DO - 10.48550/arXiv.2508.16204
|
||
UR - https://arxiv.org/abs/2508.16204
|
||
AB - Model merging is a powerful technique for integrating the specialized knowledge of multiple machine learning models into a single model. However, existing methods require manually partitioning model parameters into fixed groups for merging, which restricts the exploration of potential combinations and limits performance. To overcome these limitations, we propose Model Merging of Natural Niches (M2N2), an evolutionary algorithm with three key features: (1) dynamic adjustment of merging boundaries to progressively explore a broader range of parameter combinations; (2) a diversity preservation mechanism inspired by the competition for resources in nature, to maintain a population of diverse, high-performing models that are particularly well-suited for merging; and (3) a heuristicbased attraction metric to identify the most promising pairs of models for fusion. Our experimental results demonstrate, for the first time, that model merging can be used to evolve models entirely from scratch. Specifically, we apply M2N2 to evolve MNIST classifiers from scratch and achieve performance comparable to CMA-ES, while being computationally more efficient. Furthermore, M2N2 scales to merge specialized language and image generation models, achieving state-of-the-art performance. Notably, it preserves crucial model capabilities beyond those explicitly optimized by the fitness function, highlighting its robustness and versatility. Our code is available at https://github.com/SakanaAI/natural_niches
|
||
N1 - Manuscript reference 9
|
||
ER -
|
||
TY - JOUR
|
||
AU - Subramaniam, Vighnesh
|
||
AU - Du, Yilun
|
||
AU - Tenenbaum, Joshua B.
|
||
AU - Torralba, Antonio
|
||
AU - Li, Shuang
|
||
AU - Mordatch, Igor
|
||
TI - Multiagent Finetuning: Self Improvement with Diverse Reasoning Chains
|
||
T2 - arXiv
|
||
AN - arXiv:2501.05707
|
||
PY - 2025
|
||
PB - arXiv
|
||
DO - 10.48550/arXiv.2501.05707
|
||
UR - https://arxiv.org/abs/2501.05707
|
||
AB - Large language models (LLMs) have achieved remarkable performance in recent years but are fundamentally limited by the underlying training data. To improve models beyond the training data, recent works have explored how LLMs can be used to generate synthetic data for autonomous self-improvement. However, successive steps of self-improvement can reach a point of diminishing returns. In this work, we propose a complementary approach towards self-improvement where finetuning is applied to a multiagent society of language models. A group of language models, all starting from the same base model, are independently specialized by updating each one using data generated through multiagent interactions among the models. By training each model on independent sets of data, we illustrate how this approach enables specialization across models and diversification over the set of models. As a result, our overall system is able to preserve diverse reasoning chains and autonomously improve over many more rounds of fine-tuning than single-agent self-improvement methods. We quantitatively illustrate the efficacy of the approach across a wide suite of reasoning tasks.
|
||
N1 - Manuscript reference 10
|
||
ER -
|
||
TY - JOUR
|
||
AU - Nvidia
|
||
AU - :
|
||
AU - Adler, Bo
|
||
AU - Agarwal, Niket
|
||
AU - Aithal, Ashwath
|
||
AU - Anh, Dong H.
|
||
AU - Bhattacharya, Pallab
|
||
AU - Brundyn, Annika
|
||
AU - Casper, Jared
|
||
AU - Catanzaro, Bryan
|
||
AU - Clay, Sharon
|
||
AU - Cohen, Jonathan
|
||
AU - Das, Sirshak
|
||
AU - Dattagupta, Ayush
|
||
AU - Delalleau, Olivier
|
||
AU - Derczynski, Leon
|
||
AU - Dong, Yi
|
||
AU - Egert, Daniel
|
||
AU - Evans, Ellie
|
||
AU - Ficek, Aleksander
|
||
AU - Fridman, Denys
|
||
AU - Ghosh, Shaona
|
||
AU - Ginsburg, Boris
|
||
AU - Gitman, Igor
|
||
AU - Grzegorzek, Tomasz
|
||
AU - Hero, Robert
|
||
AU - Huang, Jining
|
||
AU - Jawa, Vibhu
|
||
AU - Jennings, Joseph
|
||
AU - Jhunjhunwala, Aastha
|
||
AU - Kamalu, John
|
||
AU - Khan, Sadaf
|
||
AU - Kuchaiev, Oleksii
|
||
AU - LeGresley, Patrick
|
||
AU - Li, Hui
|
||
AU - Liu, Jiwei
|
||
AU - Liu, Zihan
|
||
AU - Long, Eileen
|
||
AU - Mahabaleshwarkar, Ameya Sunil
|
||
AU - Majumdar, Somshubra
|
||
AU - Maki, James
|
||
AU - Martinez, Miguel
|
||
AU - de Melo, Maer Rodrigues
|
||
AU - Moshkov, Ivan
|
||
AU - Narayanan, Deepak
|
||
AU - Narenthiran, Sean
|
||
AU - Navarro, Jesus
|
||
AU - Nguyen, Phong
|
||
AU - Nitski, Osvald
|
||
AU - Noroozi, Vahid
|
||
AU - Nutheti, Guruprasad
|
||
AU - Parisien, Christopher
|
||
AU - Parmar, Jupinder
|
||
AU - Patwary, Mostofa
|
||
AU - Pawelec, Krzysztof
|
||
AU - Ping, Wei
|
||
AU - Prabhumoye, Shrimai
|
||
AU - Roy, Rajarshi
|
||
AU - Saar, Trisha
|
||
AU - Sabavat, Vasanth Rao Naik
|
||
AU - Satheesh, Sanjeev
|
||
AU - Scowcroft, Jane Polak
|
||
AU - Sewall, Jason
|
||
AU - Shamis, Pavel
|
||
AU - Shen, Gerald
|
||
AU - Shoeybi, Mohammad
|
||
AU - Sizer, Dave
|
||
AU - Smelyanskiy, Misha
|
||
AU - Soares, Felipe
|
||
AU - Sreedhar, Makesh Narsimhan
|
||
AU - Su, Dan
|
||
AU - Subramanian, Sandeep
|
||
AU - Sun, Shengyang
|
||
AU - Toshniwal, Shubham
|
||
AU - Wang, Hao
|
||
AU - Wang, Zhilin
|
||
AU - You, Jiaxuan
|
||
AU - Zeng, Jiaqi
|
||
AU - Zhang, Jimmy
|
||
AU - Zhang, Jing
|
||
AU - Zhang, Vivienne
|
||
AU - Zhang, Yian
|
||
AU - Zhu, Chen
|
||
TI - Nemotron-4 340B Technical Report
|
||
T2 - arXiv
|
||
AN - arXiv:2406.11704
|
||
PY - 2024
|
||
PB - arXiv
|
||
DO - 10.48550/arXiv.2406.11704
|
||
UR - https://arxiv.org/abs/2406.11704
|
||
AB - We release the Nemotron-4 340B model family, including Nemotron-4-340B-Base, Nemotron-4-340B-Instruct, and Nemotron-4-340B-Reward. Our models are open access under the NVIDIA Open Model License Agreement, a permissive model license that allows distribution, modification, and use of the models and its outputs. These models perform competitively to open access models on a wide range of evaluation benchmarks, and were sized to fit on a single DGX H100 with 8 GPUs when deployed in FP8 precision. We believe that the community can benefit from these models in various research studies and commercial applications, especially for generating synthetic data to train smaller language models. Notably, over 98% of data used in our model alignment process is synthetically generated, showcasing the effectiveness of these models in generating synthetic data. To further support open research and facilitate model development, we are also open-sourcing the synthetic data generation pipeline used in our model alignment process.
|
||
N1 - Manuscript reference 11
|
||
ER -
|
||
TY - JOUR
|
||
AU - Abdin, Marah
|
||
AU - Aneja, Jyoti
|
||
AU - Behl, Harkirat
|
||
AU - Bubeck, Sébastien
|
||
AU - Eldan, Ronen
|
||
AU - Gunasekar, Suriya
|
||
AU - Harrison, Michael
|
||
AU - Hewett, Russell J.
|
||
AU - Javaheripi, Mojan
|
||
AU - Kauffmann, Piero
|
||
AU - Lee, James R.
|
||
AU - Lee, Yin Tat
|
||
AU - Li, Yuanzhi
|
||
AU - Liu, Weishung
|
||
AU - Mendes, Caio C. T.
|
||
AU - Nguyen, Anh
|
||
AU - Price, Eric
|
||
AU - de Rosa, Gustavo
|
||
AU - Saarikivi, Olli
|
||
AU - Salim, Adil
|
||
AU - Shah, Shital
|
||
AU - Wang, Xin
|
||
AU - Ward, Rachel
|
||
AU - Wu, Yue
|
||
AU - Yu, Dingli
|
||
AU - Zhang, Cyril
|
||
AU - Zhang, Yi
|
||
TI - Phi-4 Technical Report
|
||
T2 - arXiv
|
||
AN - arXiv:2412.08905
|
||
PY - 2024
|
||
PB - arXiv
|
||
DO - 10.48550/arXiv.2412.08905
|
||
UR - https://arxiv.org/abs/2412.08905
|
||
AB - We present phi-4, a 14-billion parameter language model developed with a training recipe that is centrally focused on data quality. Unlike most language models, where pre-training is based primarily on organic data sources such as web content or code, phi-4 strategically incorporates synthetic data throughout the training process. While previous models in the Phi family largely distill the capabilities of a teacher model (specifically GPT-4), phi-4 substantially surpasses its teacher model on STEM-focused QA capabilities, giving evidence that our data-generation and post-training techniques go beyond distillation. Despite minimal changes to the phi-3 architecture, phi-4 achieves strong performance relative to its size -- especially on reasoning-focused benchmarks -- due to improved data, training curriculum, and innovations in the post-training scheme.
|
||
N1 - Manuscript reference 12
|
||
ER -
|
||
TY - JOUR
|
||
AU - Wang, Yizhong
|
||
AU - Kordi, Yeganeh
|
||
AU - Mishra, Swaroop
|
||
AU - Liu, Alisa
|
||
AU - Smith, Noah A.
|
||
AU - Khashabi, Daniel
|
||
AU - Hajishirzi, Hannaneh
|
||
TI - Self-Instruct: Aligning Language Models with Self-Generated Instructions
|
||
T2 - arXiv
|
||
AN - arXiv:2212.10560
|
||
PY - 2022
|
||
PB - arXiv
|
||
DO - 10.48550/arXiv.2212.10560
|
||
UR - https://arxiv.org/abs/2212.10560
|
||
AB - Large "instruction-tuned" language models (i.e., finetuned to respond to instructions) have demonstrated a remarkable ability to generalize zero-shot to new tasks. Nevertheless, they depend heavily on human-written instruction data that is often limited in quantity, diversity, and creativity, therefore hindering the generality of the tuned model. We introduce Self-Instruct, a framework for improving the instruction-following capabilities of pretrained language models by bootstrapping off their own generations. Our pipeline generates instructions, input, and output samples from a language model, then filters invalid or similar ones before using them to finetune the original model. Applying our method to the vanilla GPT3, we demonstrate a 33% absolute improvement over the original model on Super-NaturalInstructions, on par with the performance of InstructGPT-001, which was trained with private user data and human annotations. For further evaluation, we curate a set of expert-written instructions for novel tasks, and show through human evaluation that tuning GPT3 with Self-Instruct outperforms using existing public instruction datasets by a large margin, leaving only a 5% absolute gap behind InstructGPT-001. Self-Instruct provides an almost annotation-free method for aligning pre-trained language models with instructions, and we release our large synthetic dataset to facilitate future studies on instruction tuning. Our code and data are available at https://github.com/yizhongw/self-instruct.
|
||
N1 - Manuscript reference 13
|
||
ER -
|
||
TY - JOUR
|
||
AU - Thompson, Brian
|
||
AU - Dhaliwal, Mehak Preet
|
||
AU - Frisch, Peter
|
||
AU - Domhan, Tobias
|
||
AU - Federico, Marcello
|
||
TI - A Shocking Amount of the Web is Machine Translated: Insights from Multi-Way Parallelism
|
||
T2 - arXiv
|
||
AN - arXiv:2401.05749
|
||
PY - 2024
|
||
PB - arXiv
|
||
DO - 10.48550/arXiv.2401.05749
|
||
UR - https://arxiv.org/abs/2401.05749
|
||
AB - We show that content on the web is often translated into many languages, and the low quality of these multi-way translations indicates they were likely created using Machine Translation (MT). Multi-way parallel, machine generated content not only dominates the translations in lower resource languages; it also constitutes a large fraction of the total web content in those languages. We also find evidence of a selection bias in the type of content which is translated into many languages, consistent with low quality English content being translated en masse into many lower resource languages, via MT. Our work raises serious concerns about training models such as multilingual large language models on both monolingual and bilingual data scraped from the web.
|
||
N1 - Manuscript reference 14
|
||
ER -
|
||
TY - JOUR
|
||
AU - Liang, Weixin
|
||
AU - Izzo, Zachary
|
||
AU - Zhang, Yaohui
|
||
AU - Lepp, Haley
|
||
AU - Cao, Hancheng
|
||
AU - Zhao, Xuandong
|
||
AU - Chen, Lingjiao
|
||
AU - Ye, Haotian
|
||
AU - Liu, Sheng
|
||
AU - Huang, Zhi
|
||
AU - McFarland, Daniel A.
|
||
AU - Zou, James Y.
|
||
TI - Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews
|
||
T2 - arXiv
|
||
AN - arXiv:2403.07183
|
||
PY - 2024
|
||
PB - arXiv
|
||
DO - 10.48550/arXiv.2403.07183
|
||
UR - https://arxiv.org/abs/2403.07183
|
||
AB - We present an approach for estimating the fraction of text in a large corpus which is likely to be substantially modified or produced by a large language model (LLM). Our maximum likelihood model leverages expert-written and AI-generated reference texts to accurately and efficiently examine real-world LLM-use at the corpus level. We apply this approach to a case study of scientific peer review in AI conferences that took place after the release of ChatGPT: ICLR 2024, NeurIPS 2023, CoRL 2023 and EMNLP 2023. Our results suggest that between 6.5% and 16.9% of text submitted as peer reviews to these conferences could have been substantially modified by LLMs, i.e. beyond spell-checking or minor writing updates. The circumstances in which generated text occurs offer insight into user behavior: the estimated fraction of LLM-generated text is higher in reviews which report lower confidence, were submitted close to the deadline, and from reviewers who are less likely to respond to author rebuttals. We also observe corpus-level trends in generated text which may be too subtle to detect at the individual level, and discuss the implications of such trends on peer review. We call for future interdisciplinary work to examine how LLM use is changing our information and knowledge practices.
|
||
N1 - Manuscript reference 15
|
||
ER -
|
||
TY - JOUR
|
||
AU - Villalobos, Pablo
|
||
AU - Ho, Anson
|
||
AU - Sevilla, Jaime
|
||
AU - Besiroglu, Tamay
|
||
AU - Heim, Lennart
|
||
AU - Hobbhahn, Marius
|
||
TI - Will we run out of data? Limits of LLM scaling based on human-generated data
|
||
T2 - arXiv
|
||
AN - arXiv:2211.04325
|
||
PY - 2022
|
||
PB - arXiv
|
||
DO - 10.48550/arXiv.2211.04325
|
||
UR - https://arxiv.org/abs/2211.04325
|
||
AB - We investigate the potential constraints on LLM scaling posed by the availability of public human-generated text data. We forecast the growing demand for training data based on current trends and estimate the total stock of public human text data. Our findings indicate that if current LLM development trends continue, models will be trained on datasets roughly equal in size to the available stock of public human text data between 2026 and 2032, or slightly earlier if models are overtrained. We explore how progress in language modeling can continue when human-generated text datasets cannot be scaled any further. We argue that synthetic data generation, transfer learning from data-rich domains, and data efficiency improvements might support further progress.
|
||
N1 - Manuscript reference 16
|
||
ER -
|
||
TY - JOUR
|
||
AU - Brinkmann, Levin
|
||
AU - Baumann, Fabian
|
||
AU - Bonnefon, Jean-François
|
||
AU - Derex, Maxime
|
||
AU - Müller, Thomas F.
|
||
AU - Nussberger, Anne-Marie
|
||
AU - Czaplicka, Agnieszka
|
||
AU - Acerbi, Alberto
|
||
AU - Griffiths, Thomas L.
|
||
AU - Henrich, Joseph
|
||
AU - Leibo, Joel Z.
|
||
AU - McElreath, Richard
|
||
AU - Oudeyer, Pierre-Yves
|
||
AU - Stray, Jonathan
|
||
AU - Rahwan, Iyad
|
||
TI - Machine culture
|
||
T2 - Nature Human Behaviour
|
||
PY - 2023
|
||
VL - 7
|
||
IS - 11
|
||
PB - Springer Science and Business Media LLC
|
||
DO - 10.1038/s41562-023-01742-2
|
||
UR - http://dx.doi.org/10.1038/s41562-023-01742-2
|
||
SP - 1855
|
||
EP - 1868
|
||
N1 - Manuscript reference 17
|
||
ER -
|
||
TY - JOUR
|
||
AU - Park, Joon Sung
|
||
AU - O'Brien, Joseph C.
|
||
AU - Cai, Carrie J.
|
||
AU - Morris, Meredith Ringel
|
||
AU - Liang, Percy
|
||
AU - Bernstein, Michael S.
|
||
TI - Generative Agents: Interactive Simulacra of Human Behavior
|
||
T2 - arXiv
|
||
AN - arXiv:2304.03442
|
||
PY - 2023
|
||
PB - arXiv
|
||
DO - 10.48550/arXiv.2304.03442
|
||
UR - https://arxiv.org/abs/2304.03442
|
||
AB - Believable proxies of human behavior can empower interactive applications ranging from immersive environments to rehearsal spaces for interpersonal communication to prototyping tools. In this paper, we introduce generative agents--computational software agents that simulate believable human behavior. Generative agents wake up, cook breakfast, and head to work; artists paint, while authors write; they form opinions, notice each other, and initiate conversations; they remember and reflect on days past as they plan the next day. To enable generative agents, we describe an architecture that extends a large language model to store a complete record of the agent's experiences using natural language, synthesize those memories over time into higher-level reflections, and retrieve them dynamically to plan behavior. We instantiate generative agents to populate an interactive sandbox environment inspired by The Sims, where end users can interact with a small town of twenty five agents using natural language. In an evaluation, these generative agents produce believable individual and emergent social behaviors: for example, starting with only a single user-specified notion that one agent wants to throw a Valentine's Day party, the agents autonomously spread invitations to the party over the next two days, make new acquaintances, ask each other out on dates to the party, and coordinate to show up for the party together at the right time. We demonstrate through ablation that the components of our agent architecture--observation, planning, and reflection--each contribute critically to the believability of agent behavior. By fusing large language models with computational, interactive agents, this work introduces architectural and interaction patterns for enabling believable simulations of human behavior.
|
||
N1 - Manuscript reference 18
|
||
ER -
|
||
TY - JOUR
|
||
AU - Guo, Taicheng
|
||
AU - Chen, Xiuying
|
||
AU - Wang, Yaqi
|
||
AU - Chang, Ruidi
|
||
AU - Pei, Shichao
|
||
AU - Chawla, Nitesh V.
|
||
AU - Wiest, Olaf
|
||
AU - Zhang, Xiangliang
|
||
TI - Large Language Model based Multi-Agents: A Survey of Progress and Challenges
|
||
T2 - arXiv
|
||
AN - arXiv:2402.01680
|
||
PY - 2024
|
||
PB - arXiv
|
||
DO - 10.48550/arXiv.2402.01680
|
||
UR - https://arxiv.org/abs/2402.01680
|
||
AB - Large Language Models (LLMs) have achieved remarkable success across a wide array of tasks. Due to the impressive planning and reasoning abilities of LLMs, they have been used as autonomous agents to do many tasks automatically. Recently, based on the development of using one LLM as a single planning or decision-making agent, LLM-based multi-agent systems have achieved considerable progress in complex problem-solving and world simulation. To provide the community with an overview of this dynamic field, we present this survey to offer an in-depth discussion on the essential aspects of multi-agent systems based on LLMs, as well as the challenges. Our goal is for readers to gain substantial insights on the following questions: What domains and environments do LLM-based multi-agents simulate? How are these agents profiled and how do they communicate? What mechanisms contribute to the growth of agents' capacities? For those interested in delving into this field of study, we also summarize the commonly used datasets or benchmarks for them to have convenient access. To keep researchers updated on the latest studies, we maintain an open-source GitHub repository, dedicated to outlining the research on LLM-based multi-agent systems.
|
||
N1 - Manuscript reference 19
|
||
ER -
|
||
TY - JOUR
|
||
AU - Tomasev, Nenad
|
||
AU - Franklin, Matija
|
||
AU - Leibo, Joel Z.
|
||
AU - Jacobs, Julian
|
||
AU - Cunningham, William A.
|
||
AU - Gabriel, Iason
|
||
AU - Osindero, Simon
|
||
TI - Virtual Agent Economies
|
||
T2 - arXiv
|
||
AN - arXiv:2509.10147
|
||
PY - 2025
|
||
PB - arXiv
|
||
DO - 10.48550/arXiv.2509.10147
|
||
UR - https://arxiv.org/abs/2509.10147
|
||
AB - The rapid adoption of autonomous AI agents is giving rise to a new economic layer where agents transact and coordinate at scales and speeds beyond direct human oversight. We propose the "sandbox economy" as a framework for analyzing this emergent system, characterizing it along two key dimensions: its origins (emergent vs. intentional) and its degree of separateness from the established human economy (permeable vs. impermeable). Our current trajectory points toward a spontaneous emergence of a vast and highly permeable AI agent economy, presenting us with opportunities for an unprecedented degree of coordination as well as significant challenges, including systemic economic risk and exacerbated inequality. Here we discuss a number of possible design choices that may lead to safely steerable AI agent markets. In particular, we consider auction mechanisms for fair resource allocation and preference resolution, the design of AI "mission economies" to coordinate around achieving collective goals, and socio-technical infrastructure needed to ensure trust, safety, and accountability. By doing this, we argue for the proactive design of steerable agent markets to ensure the coming technological shift aligns with humanity's long-term collective flourishing.
|
||
N1 - Manuscript reference 20
|
||
ER -
|
||
TY - JOUR
|
||
AU - Shumailov, Ilia
|
||
AU - Shumaylov, Zakhar
|
||
AU - Zhao, Yiren
|
||
AU - Papernot, Nicolas
|
||
AU - Anderson, Ross
|
||
AU - Gal, Yarin
|
||
TI - AI models collapse when trained on recursively generated data
|
||
T2 - Nature
|
||
PY - 2024
|
||
VL - 631
|
||
IS - 8022
|
||
PB - Springer Science and Business Media LLC
|
||
DO - 10.1038/s41586-024-07566-y
|
||
UR - http://dx.doi.org/10.1038/s41586-024-07566-y
|
||
AB - Abstract Stable diffusion revolutionized image creation from descriptive text. GPT-2 (ref. 1 ), GPT-3(.5) (ref. 2 ) and GPT-4 (ref. 3 ) demonstrated high performance across a variety of language tasks. ChatGPT introduced such language models to the public. It is now clear that generative artificial intelligence (AI) such as large language models (LLMs) is here to stay and will substantially change the ecosystem of online text and images. Here we consider what may happen to GPT-{ n } once LLMs contribute much of the text found online. We find that indiscriminate use of model-generated content in training causes irreversible defects in the resulting models, in which tails of the original content distribution disappear. We refer to this effect as ‘model collapse’ and show that it can occur in LLMs as well as in variational autoencoders (VAEs) and Gaussian mixture models (GMMs). We build theoretical intuition behind the phenomenon and portray its ubiquity among all learned generative models. We demonstrate that it must be taken seriously if we are to sustain the benefits of training from large-scale data scraped from the web. Indeed, the value of data collected about genuine human interactions with systems will be increasingly valuable in the presence of LLM-generated content in data crawled from the Internet.
|
||
SP - 755
|
||
EP - 759
|
||
N1 - Manuscript reference 21
|
||
ER -
|
||
TY - JOUR
|
||
AU - Crutchfield, James P.
|
||
AU - Whalen, Sean
|
||
TI - Structural Drift: The Population Dynamics of Sequential Learning
|
||
T2 - PLoS Computational Biology
|
||
A2 - Bergstrom, Carl T.
|
||
PY - 2012
|
||
VL - 8
|
||
IS - 6
|
||
PB - Public Library of Science (PLoS)
|
||
DO - 10.1371/journal.pcbi.1002510
|
||
UR - http://dx.doi.org/10.1371/journal.pcbi.1002510
|
||
SP - e1002510
|
||
N1 - Manuscript reference 22
|
||
ER -
|
||
TY - JOUR
|
||
AU - Riis, Søren
|
||
TI - Drift and selection in LLM text ecosystems
|
||
T2 - arXiv
|
||
AN - arXiv:2604.08554
|
||
PY - 2026
|
||
PB - arXiv
|
||
DO - 10.48550/arXiv.2604.08554
|
||
UR - https://arxiv.org/abs/2604.08554
|
||
AB - The public text record -- the material from which both people and AI systems now learn -- is increasingly shaped by its own outputs. Generated text enters the public record, later agents learn from it, and the cycle repeats. Here we develop an exactly solvable mathematical framework for this recursive process, based on variable-order $n$-gram agents, and separate two forces acting on the public corpus. The first is drift: unfiltered reuse progressively removes rare forms, and in the infinite-corpus limit we characterise the stable distributions exactly. The second is selection: publication, ranking and verification filter what enters the record, and the outcome depends on what is selected. When publication merely reflects the statistical status quo, the corpus converges to a shallow state in which further lookahead brings no benefit. When publication is normative -- rewarding quality, correctness or novelty -- deeper structure persists, and we establish an optimal upper bound on the resulting divergence from shallow equilibria. The framework therefore identifies when recursive publication compresses public text and when selective filtering sustains richer structure, with implications for the design of AI training corpora.
|
||
N1 - Manuscript reference 23
|
||
ER -
|
||
TY - JOUR
|
||
AU - Benati, Matteo
|
||
AU - Londei, Alessandro
|
||
AU - Lanzieri, Denise
|
||
AU - Loreto, Vittorio
|
||
TI - First-Extinction Law for Resampling Processes
|
||
T2 - arXiv
|
||
AN - arXiv:2509.20101
|
||
PY - 2025
|
||
PB - arXiv
|
||
DO - 10.48550/arXiv.2509.20101
|
||
UR - https://arxiv.org/abs/2509.20101
|
||
AB - Extinction times in resampling processes are fundamental yet often intractable, as previous formulas scale as $2^M$ with the number of states $M$ present in the initial probability distribution. We solve this by treating multinomial updates as independent square-root diffusions of zero drift, yielding a closed-form law for the first-extinction time. We prove that the mean coincides exactly with the Wright-Fisher result of Baxter et al., thereby replacing exponential-cost evaluations with a linear-cost expression, and we validate this result through extensive simulations. Finally, we demonstrate predictive power for model collapse in a simple self-training setup: the onset of collapse coincides with the resampling-driven first-extinction time computed from the model's initial stationary distribution. These results hint to a unified view of resampling extinction dynamics.
|
||
N1 - Manuscript reference 24
|
||
ER -
|
||
TY - JOUR
|
||
AU - Yoon, Youngseok
|
||
AU - Hu, Dainong
|
||
AU - Weissburg, Iain
|
||
AU - Qin, Yao
|
||
AU - Jeong, Haewon
|
||
TI - Model Collapse in the Self-Consuming Chain of Diffusion Finetuning: A Novel Perspective from Quantitative Trait Modeling
|
||
T2 - arXiv
|
||
AN - arXiv:2407.17493
|
||
PY - 2024
|
||
PB - arXiv
|
||
DO - 10.48550/arXiv.2407.17493
|
||
UR - https://arxiv.org/abs/2407.17493
|
||
AB - Model collapse, the severe degradation of generative models when iteratively trained on their own outputs, has gained significant attention in recent years. This paper examines Chain of Diffusion, where a pretrained text-to-image diffusion model is finetuned on its own generated images. We demonstrate that severe image quality degradation was universal and identify CFG scale as the key factor impacting this model collapse. Drawing on an analogy between the Chain of Diffusion and biological evolution, we then introduce a novel theoretical analysis based on quantitative trait modeling from statistical genetics. Our theoretical analysis aligns with empirical observations of the generated images in the Chain of Diffusion. Finally, we propose Reusable Diffusion Finetuning (ReDiFine), a simple yet effective strategy inspired by genetic mutations. It operates robustly across various scenarios without requiring any hyperparameter tuning, making it a plug-and-play solution for reusable image generation.
|
||
N1 - Manuscript reference 25
|
||
ER -
|
||
TY - CHAP
|
||
AU - McCloskey, Michael
|
||
AU - Cohen, Neal J.
|
||
TI - Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem
|
||
BT - Psychology of Learning and Motivation
|
||
PY - 1989
|
||
PB - Elsevier
|
||
DO - 10.1016/s0079-7421(08)60536-8
|
||
UR - http://dx.doi.org/10.1016/s0079-7421(08)60536-8
|
||
SP - 109
|
||
EP - 165
|
||
N1 - Manuscript reference 26
|
||
ER -
|
||
TY - JOUR
|
||
AU - French, R
|
||
TI - Catastrophic forgetting in connectionist networks
|
||
T2 - Trends in Cognitive Sciences
|
||
PY - 1999
|
||
VL - 3
|
||
IS - 4
|
||
PB - Elsevier BV
|
||
DO - 10.1016/s1364-6613(99)01294-2
|
||
UR - http://dx.doi.org/10.1016/s1364-6613(99)01294-2
|
||
SP - 128
|
||
EP - 135
|
||
N1 - Manuscript reference 27
|
||
ER -
|
||
TY - JOUR
|
||
AU - Muller, H.J.
|
||
TI - The relation of recombination to mutational advance
|
||
T2 - Mutation Research
|
||
PY - 1964
|
||
VL - 1
|
||
IS - 1
|
||
PB - Elsevier BV
|
||
DO - 10.1016/0027-5107(64)90047-8
|
||
UR - http://dx.doi.org/10.1016/0027-5107(64)90047-8
|
||
SP - 2
|
||
EP - 9
|
||
N1 - Manuscript reference 28
|
||
ER -
|
||
TY - JOUR
|
||
AU - Yi, Bingji
|
||
AU - Liu, Qiyuan
|
||
AU - Cheng, Yuwei
|
||
AU - Xu, Haifeng
|
||
TI - Escaping Model Collapse via Synthetic Data Verification: Near-term Improvements and Long-term Convergence
|
||
T2 - arXiv
|
||
AN - arXiv:2510.16657
|
||
PY - 2025
|
||
PB - arXiv
|
||
DO - 10.48550/arXiv.2510.16657
|
||
UR - https://arxiv.org/abs/2510.16657
|
||
AB - Synthetic data has been increasingly used to train frontier generative models. However, recent studies raise key concerns that iteratively retraining a generative model on its self-generated synthetic data may keep deteriorating model performance, a phenomenon often coined model collapse. In this paper, we investigate ways to modify the synthetic retraining process to avoid model collapse, and even possibly help reverse the trend from collapse to improvement. Our key finding is that by injecting information through an external synthetic data verifier, whether a human or a better model, synthetic retraining will not cause model collapse. Specifically, we situate our theoretical analysis in the fundamental linear regression setting, showing that verifier-guided retraining can yield near-term improvements, but ultimately drives the parameter estimate to the verifier's "knowledge center" in the long run. Our theory further predicts that, unless the verifier is perfectly reliable, these early gains will plateau and may even reverse. Indeed, our experiments across linear regression, Variational Autoencoders (VAEs) trained on MNIST, and fining-tuning SmolLM2-135M on the XSUM task confirm these theoretical insights.
|
||
N1 - Manuscript reference 29
|
||
ER -
|
||
TY - JOUR
|
||
AU - Gerstgrasser, Matthias
|
||
AU - Schaeffer, Rylan
|
||
AU - Dey, Apratim
|
||
AU - Rafailov, Rafael
|
||
AU - Sleight, Henry
|
||
AU - Hughes, John
|
||
AU - Korbak, Tomasz
|
||
AU - Agrawal, Rajashree
|
||
AU - Pai, Dhruv
|
||
AU - Gromov, Andrey
|
||
AU - Roberts, Daniel A.
|
||
AU - Yang, Diyi
|
||
AU - Donoho, David L.
|
||
AU - Koyejo, Sanmi
|
||
TI - Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data
|
||
T2 - arXiv
|
||
AN - arXiv:2404.01413
|
||
PY - 2024
|
||
PB - arXiv
|
||
DO - 10.48550/arXiv.2404.01413
|
||
UR - https://arxiv.org/abs/2404.01413
|
||
AB - The proliferation of generative models, combined with pretraining on web-scale data, raises a timely question: what happens when these models are trained on their own generated outputs? Recent investigations into model-data feedback loops proposed that such loops would lead to a phenomenon termed model collapse, under which performance progressively degrades with each model-data feedback iteration until fitted models become useless. However, those studies largely assumed that new data replace old data over time, where an arguably more realistic assumption is that data accumulate over time. In this paper, we ask: what effect does accumulating data have on model collapse? We empirically study this question by pretraining sequences of language models on text corpora. We confirm that replacing the original real data by each generation's synthetic data does indeed tend towards model collapse, then demonstrate that accumulating the successive generations of synthetic data alongside the original real data avoids model collapse; these results hold across a range of model sizes, architectures, and hyperparameters. We obtain similar results for deep generative models on other types of real data: diffusion models for molecule conformation generation and variational autoencoders for image generation. To understand why accumulating data can avoid model collapse, we use an analytically tractable framework introduced by prior work in which a sequence of linear models are fit to the previous models' outputs. Previous work used this framework to show that if data are replaced, the test error increases with the number of model-fitting iterations; we extend this argument to prove that if data instead accumulate, the test error has a finite upper bound independent of the number of iterations, meaning model collapse no longer occurs.
|
||
N1 - Manuscript reference 30
|
||
ER -
|
||
TY - JOUR
|
||
AU - Wright, Sewall
|
||
TI - EVOLUTION IN MENDELIAN POPULATIONS
|
||
T2 - Genetics
|
||
PY - 1931
|
||
VL - 16
|
||
IS - 2
|
||
PB - Oxford University Press (OUP)
|
||
DO - 10.1093/genetics/16.2.97
|
||
UR - http://dx.doi.org/10.1093/genetics/16.2.97
|
||
SP - 97
|
||
EP - 159
|
||
N1 - Manuscript reference 31
|
||
ER -
|
||
TY - JOUR
|
||
AU - Mills, L. Scott
|
||
AU - Allendorf, Fred W.
|
||
TI - The One‐Migrant‐per‐Generation Rule in Conservation and Management
|
||
T2 - Conservation Biology
|
||
PY - 1996
|
||
VL - 10
|
||
IS - 6
|
||
PB - Wiley
|
||
DO - 10.1046/j.1523-1739.1996.10061509.x
|
||
UR - http://dx.doi.org/10.1046/j.1523-1739.1996.10061509.x
|
||
AB - In the face of continuing habitat fragmentation and isolation, the optimal level of connectivity between populations has become a central issue in conservation biology. A common rule of thumb holds that one migrant per generation into a subpopulation is sufficient to minimize the loss of polymorphism and heterozygosity within subpopulations while allowing for divergence in allele frequencies among subpopulations. The one‐migrant‐per‐generation rule is based on numerous simplifying assumptions that may not hold in natural populations. We examine the conceptual and theoretical basis of the rule and consider both genetic and nongenetic factors that influence the desired level of connectivity among subpopulations. We conclude that one migrant per generation is a desirable minimum, but it may be inadequate for many natural populations. We suggest that a minimum of 1 and a maximum of 10 migrants per generation would be an appropriate general rule of thumb for genetic purposes, bearing in mind that factors other than genetics may further influence the ideal level of connectivity.
|
||
SP - 1509
|
||
EP - 1518
|
||
N1 - Manuscript reference 32
|
||
ER -
|
||
TY - JOUR
|
||
AU - Jenkin, Fleeming
|
||
TI - [Review of] The Origin of Species
|
||
T2 - The North British Review
|
||
PY - 1867
|
||
VL - 46
|
||
SP - 277
|
||
EP - 318
|
||
N1 - Manuscript reference 33; predates DOIs
|
||
ER -
|
||
TY - JOUR
|
||
AU - Pari, Jyothish
|
||
AU - Jelassi, Samy
|
||
AU - Agrawal, Pulkit
|
||
TI - Collective Model Intelligence Requires Compatible Specialization
|
||
T2 - arXiv
|
||
AN - arXiv:2411.02207
|
||
PY - 2024
|
||
PB - arXiv
|
||
DO - 10.48550/arXiv.2411.02207
|
||
UR - https://arxiv.org/abs/2411.02207
|
||
AB - In this work, we explore the limitations of combining models by averaging intermediate features, referred to as model merging, and propose a new direction for achieving collective model intelligence through what we call compatible specialization. Current methods for model merging, such as parameter and feature averaging, struggle to effectively combine specialized models due to representational divergence during fine-tuning. As models specialize to their individual domains, their internal feature representations become increasingly incompatible, leading to poor performance when attempting to merge them for new tasks. We analyze this phenomenon using centered kernel alignment (CKA) and show that as models specialize, the similarity in their feature space structure diminishes, hindering their capacity for collective use. To address these challenges, we investigate routing-based merging strategies, which offer more flexible methods for combining specialized models by dynamically routing across different layers. This allows us to improve on existing methods by combining features from multiple layers rather than relying on fixed, layer-wise combinations. However, we find that these approaches still face limitations when layers within models are representationally incompatible. Our findings highlight the importance of designing new approaches for model merging that operate on well-defined input and output spaces, similar to how humans communicate through language rather than intermediate neural activations.
|
||
N1 - Manuscript reference 34
|
||
ER -
|
||
TY - BOOK
|
||
AU - Fisher, Ronald A.
|
||
TI - The Genetical Theory of Natural Selection
|
||
CY - Oxford
|
||
PY - 1930
|
||
PB - Clarendon Press
|
||
N1 - Manuscript reference 35; predates DOIs
|
||
ER -
|
||
TY - JOUR
|
||
AU - Muller, H. J.
|
||
TI - Some Genetic Aspects of Sex
|
||
T2 - The American Naturalist
|
||
PY - 1932
|
||
VL - 66
|
||
IS - 703
|
||
PB - University of Chicago Press
|
||
DO - 10.1086/280418
|
||
UR - http://dx.doi.org/10.1086/280418
|
||
SP - 118
|
||
EP - 138
|
||
N1 - Manuscript reference 36
|
||
ER -
|
||
TY - JOUR
|
||
AU - Hu, Edward J.
|
||
AU - Shen, Yelong
|
||
AU - Wallis, Phillip
|
||
AU - Allen-Zhu, Zeyuan
|
||
AU - Li, Yuanzhi
|
||
AU - Wang, Shean
|
||
AU - Wang, Lu
|
||
AU - Chen, Weizhu
|
||
TI - LoRA: Low-Rank Adaptation of Large Language Models
|
||
T2 - arXiv
|
||
AN - arXiv:2106.09685
|
||
PY - 2021
|
||
PB - arXiv
|
||
DO - 10.48550/arXiv.2106.09685
|
||
UR - https://arxiv.org/abs/2106.09685
|
||
AB - An important paradigm of natural language processing consists of large-scale pre-training on general domain data and adaptation to particular tasks or domains. As we pre-train larger models, full fine-tuning, which retrains all model parameters, becomes less feasible. Using GPT-3 175B as an example -- deploying independent instances of fine-tuned models, each with 175B parameters, is prohibitively expensive. We propose Low-Rank Adaptation, or LoRA, which freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture, greatly reducing the number of trainable parameters for downstream tasks. Compared to GPT-3 175B fine-tuned with Adam, LoRA can reduce the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times. LoRA performs on-par or better than fine-tuning in model quality on RoBERTa, DeBERTa, GPT-2, and GPT-3, despite having fewer trainable parameters, a higher training throughput, and, unlike adapters, no additional inference latency. We also provide an empirical investigation into rank-deficiency in language model adaptation, which sheds light on the efficacy of LoRA. We release a package that facilitates the integration of LoRA with PyTorch models and provide our implementations and model checkpoints for RoBERTa, DeBERTa, and GPT-2 at https://github.com/microsoft/LoRA.
|
||
N1 - Manuscript reference 37
|
||
ER -
|
||
TY - JOUR
|
||
AU - Kauffman, Stuart
|
||
AU - Levin, Simon
|
||
TI - Towards a general theory of adaptive walks on rugged landscapes
|
||
T2 - Journal of Theoretical Biology
|
||
PY - 1987
|
||
VL - 128
|
||
IS - 1
|
||
PB - Elsevier BV
|
||
DO - 10.1016/s0022-5193(87)80029-2
|
||
UR - http://dx.doi.org/10.1016/s0022-5193(87)80029-2
|
||
SP - 11
|
||
EP - 45
|
||
N1 - Manuscript reference 38
|
||
ER -
|
||
TY - CHAP
|
||
AU - Templeton, Alan R.
|
||
TI - Coadaptation and outbreeding depression
|
||
BT - Conservation Biology: The Science of Scarcity and Diversity
|
||
A2 - Soulé, Michael E.
|
||
CY - Sunderland, MA
|
||
PY - 1986
|
||
PB - Sinauer Associates
|
||
SP - 105
|
||
EP - 116
|
||
N1 - Manuscript reference 39; predates DOIs
|
||
ER -
|
||
TY - JOUR
|
||
AU - Lehman, Joel
|
||
AU - Stanley, Kenneth O.
|
||
TI - Abandoning Objectives: Evolution Through the Search for Novelty Alone
|
||
T2 - Evolutionary Computation
|
||
PY - 2011
|
||
VL - 19
|
||
IS - 2
|
||
PB - MIT Press - Journals
|
||
DO - 10.1162/evco_a_00025
|
||
UR - http://dx.doi.org/10.1162/evco_a_00025
|
||
AB - In evolutionary computation, the fitness function normally measures progress toward an objective in the search space, effectively acting as an objective function. Through deception, such objective functions may actually prevent the objective from being reached. While methods exist to mitigate deception, they leave the underlying pathology untreated: Objective functions themselves may actively misdirect search toward dead ends. This paper proposes an approach to circumventing deception that also yields a new perspective on open-ended evolution. Instead of either explicitly seeking an objective or modeling natural evolution to capture open-endedness, the idea is to simply search for behavioral novelty. Even in an objective-based problem, such novelty search ignores the objective. Because many points in the search space collapse to a single behavior, the search for novelty is often feasible. Furthermore, because there are only so many simple behaviors, the search for novelty leads to increasing complexity. By decoupling open-ended search from artificial life worlds, the search for novelty is applicable to real world problems. Counterintuitively, in the maze navigation and biped walking tasks in this paper, novelty search significantly outperforms objective-based search, suggesting the strange conclusion that some problems are best solved by methods that ignore the objective. The main lesson is the inherent limitation of the objective-based paradigm and the unexploited opportunity to guide search through other means.
|
||
SP - 189
|
||
EP - 223
|
||
N1 - Manuscript reference 40
|
||
ER -
|
||
TY - JOUR
|
||
AU - Thede, Lukas
|
||
AU - Roth, Karsten
|
||
AU - Bethge, Matthias
|
||
AU - Akata, Zeynep
|
||
AU - Hartvigsen, Tom
|
||
TI - WikiBigEdit: Understanding the Limits of Lifelong Knowledge Editing in LLMs
|
||
T2 - arXiv
|
||
AN - arXiv:2503.05683
|
||
PY - 2025
|
||
PB - arXiv
|
||
DO - 10.48550/arXiv.2503.05683
|
||
UR - https://arxiv.org/abs/2503.05683
|
||
AB - Keeping large language models factually up-to-date is crucial for deployment, yet costly retraining remains a challenge. Knowledge editing offers a promising alternative, but methods are only tested on small-scale or synthetic edit benchmarks. In this work, we aim to bridge research into lifelong knowledge editing to real-world edits at a practically relevant scale. We first introduce WikiBigEdit; a large-scale benchmark of real-world Wikidata edits, built to automatically extend lifelong for future-proof benchmarking. In its first instance, it includes over 500K question-answer pairs for knowledge editing alongside a comprehensive evaluation pipeline. Finally, we use WikiBigEdit to study existing knowledge editing techniques' ability to incorporate large volumes of real-world facts and contrast their capabilities to generic modification techniques such as retrieval augmentation and continual finetuning to acquire a complete picture of the practical extent of current lifelong knowledge editing.
|
||
N1 - Manuscript reference 41
|
||
ER -
|
||
TY - JOUR
|
||
AU - Clemente, Simone
|
||
AU - Houidi, Zied Ben
|
||
AU - Huet, Alexis
|
||
AU - Rossi, Dario
|
||
AU - Franzese, Giulio
|
||
AU - Michiardi, Pietro
|
||
TI - In Praise of Stubbornness: An Empirical Case for Cognitive-Dissonance Aware Continual Update of Knowledge in LLMs
|
||
T2 - arXiv
|
||
AN - arXiv:2502.04390
|
||
PY - 2025
|
||
PB - arXiv
|
||
DO - 10.48550/arXiv.2502.04390
|
||
UR - https://arxiv.org/abs/2502.04390
|
||
AB - Through systematic empirical investigation, we uncover a fundamental and concerning property of Large Language Models: while they can safely learn facts that don't contradict their knowledge, attempting to update facts with contradictory information triggers catastrophic corruption of unrelated knowledge. Unlike humans, who naturally resist contradictory information, these models indiscriminately accept contradictions, leading to devastating interference, destroying up to 80% of unrelated knowledge even when learning as few as 10-100 contradicting facts. To understand whether this interference could be mitigated through selective plasticity, we experiment with targeted network updates, distinguishing between previously used (stubborn) and rarely used (plastic) neurons. We uncover another asymmetry: while sparing frequently-used neurons significantly improves retention of existing knowledge for non-contradictory updates (98% vs 93% with standard updates), contradictory updates trigger catastrophic interference regardless of targeting strategy. This effect which persists across tested model scales (GPT-2 to GPT-J-6B), suggests a fundamental limitation in how neural networks handle contradictions. Finally, we demonstrate that contradictory information can be reliably detected (95%+ accuracy) using simple model features, offering a potential protective mechanism. These findings motivate new architectures that can, like humans, naturally resist contradictions rather than allowing destructive overwrites.
|
||
N1 - Manuscript reference 42
|
||
ER -
|
||
TY - JOUR
|
||
AU - Störk, Julius
|
||
TI - Interference and Retention in Continual Learning
|
||
T2 - arXiv
|
||
AN - arXiv:2607.09202
|
||
PY - 2026
|
||
PB - arXiv
|
||
DO - 10.48550/arXiv.2607.09202
|
||
UR - https://arxiv.org/abs/2607.09202
|
||
AB - Continual learning commonly relies on post-hoc mechanisms such as replay, elastic regularization, or distillation. This work argues that forgetting should instead be modeled directly as interference between tasks. In the frozen-feature regime, forgetting from learning a new task is exactly the interference energy induced on the old task. In deep networks, the same quantity is recovered through path-averaged curvature with minimal additional forward passes. When task supports are disjoint, forgetting can be eliminated structurally and when task supports overlap in conflicting directions, a non-zero distortion floor is unavoidable. The same geometry optimally merges models through task-aware orthogonalization. From this analysis we derive Interference-Gated Functional Allocation (IGFA), a replay-free, Fisher-free method that shares directions when tasks align and protects them when they conflict. Across benchmarks, IGFA achieves lossless retention when tasks are structurally separable and moves unavoidable cost from irreversible forgetting into deferred but recoverable plasticity when they are not. It matches the strongest replay-free structural baselines on dissimilar-task streams and improves on unconditional projection when similarity makes transfer worth preserving.
|
||
N1 - Manuscript reference 43
|
||
ER -
|
||
TY - JOUR
|
||
AU - Barton, N. H.
|
||
TI - A general model for the evolution of recombination
|
||
T2 - Genetical Research
|
||
PY - 1995
|
||
VL - 65
|
||
IS - 2
|
||
PB - Wiley
|
||
DO - 10.1017/s0016672300033140
|
||
UR - http://dx.doi.org/10.1017/s0016672300033140
|
||
AB - Summary A general representation of multilocus selection is extended to allow recombination to depend on genotype. The equations simplify if modifier alleles have small effects on recombination. The evolution of such modifiers only depends on how they alter recombination between the selected loci, and does not involve dominance in modifier effects. The net selection on modifiers can be found explicitly if epistasis is weak relative to recombination. This analysis shows that recombination can be favoured in two ways: because it impedes the response to epistasis which fluctuates in sign, or because it facilitates the response to directional selection. The first mechanism is implausible, because epistasis must change sign over periods of a few generations: faster or slower fluctuations favour reduced recombination. The second mechanism requires weak negative epistasis between favourable alleles, which may either be increasing, or held in check by mutation. The selection ( s i ) on recombination modifiers depends on the reduction in additive variance of log (fitness) due to linkage disequilibria (υ 1 < 0), and on non-additive variance in log (fitness) ( V ′ 2 , V ′ 3 ,.. epistasis between 2, 3.. loci). For unlinked loci and pairwise epistasis, s i = − (υ 1 + 4 V 2 /3)δ r , where δ r is the average increase in recombination caused by the modifier. The approximations are checked against exact calculations for three loci, and against Charlesworth's analyses of mutation/selection balance (1990), and directional selection (1993). The analysis demonstrates a general relation between selection on recombination and observable components of fitness variation, which is open to experimental test.
|
||
SP - 123
|
||
EP - 144
|
||
N1 - Manuscript reference 44
|
||
ER -
|
||
TY - JOUR
|
||
AU - Otto, Sarah Perin
|
||
AU - Feldman, Marcus W.
|
||
TI - Deleterious Mutations, Variable Epistatic Interactions, and the Evolution of Recombination
|
||
T2 - Theoretical Population Biology
|
||
PY - 1997
|
||
VL - 51
|
||
IS - 2
|
||
PB - Elsevier BV
|
||
DO - 10.1006/tpbi.1997.1301
|
||
UR - http://dx.doi.org/10.1006/tpbi.1997.1301
|
||
SP - 134
|
||
EP - 147
|
||
N1 - Manuscript reference 45
|
||
ER -
|
||
TY - JOUR
|
||
AU - Otto, Sarah P.
|
||
AU - Lenormand, Thomas
|
||
TI - Resolving the paradox of sex and recombination
|
||
T2 - Nature Reviews Genetics
|
||
PY - 2002
|
||
VL - 3
|
||
IS - 4
|
||
PB - Springer Science and Business Media LLC
|
||
DO - 10.1038/nrg761
|
||
UR - http://dx.doi.org/10.1038/nrg761
|
||
SP - 252
|
||
EP - 261
|
||
N1 - Manuscript reference 46
|
||
ER -
|
||
TY - JOUR
|
||
AU - Altenberg, Lee
|
||
AU - Feldman, Marcus W
|
||
TI - Selection, Generalized Transmission and the Evolution of Modifier Genes. I. The Reduction Principle
|
||
T2 - Genetics
|
||
PY - 1987
|
||
VL - 117
|
||
IS - 3
|
||
PB - Oxford University Press (OUP)
|
||
DO - 10.1093/genetics/117.3.559
|
||
UR - http://dx.doi.org/10.1093/genetics/117.3.559
|
||
AB - ABSTRACT Modifier gene models are used to explore the evolution of features of organisms, such as the genetic system, that are not directly involved in the determination of fitness. Recent work has shown that a general "reduction principle" holds in models of selectively neutral modifiers of recombination, mutation, and migration. Here we present a framework for models of modifier genes that shows these reduction results to be part of a more general theory, for which recombination and mutation are special cases.—The deterministic forces that affect the genetic composition of a population can be partitioned into two categories: selection and transmission. Selection includes differential viabilities, fertilities, and mating success. Imperfect transmission occurs as a result of such phenomena as recombination, mutation and migration, meiosis, gene conversion, and meiotic drive. Selectively neutral modifier genes affect transmission, and a neutral modifier gene can evolve only by generating association with selected genes whose transmission it affects.–We show that, in randomly mating populations at equilibrium, imperfect transmission of selected genes allows a variance in their marginal fitnesses to be maintained. This variance in the marginal fitnesses of selected genes is what drives the evolution of neutral modifier genes. Populations with a variance in marginal fitnesses at equilibrium are always subject to invasion by modifier genes that bring about perfect transmission of the selected genes. It is also found, within certain constraints, that for modifier genes producing what we call "linear variation" in the transmission processes, a new modifier allele can invade a population at equilibrium if it reduces the level of imperfect transmission acting on the selected genes, and will be expelled if it increases the level of imperfect transmission. Moreover, the strength of the induced selection on the modifier gene is shown to range up to the order of the departure of the genetic system from perfect transmission.
|
||
SP - 559
|
||
EP - 572
|
||
N1 - Manuscript reference 47
|
||
ER -
|
||
TY - JOUR
|
||
AU - Orr, H A
|
||
TI - The population genetics of speciation: the evolution of hybrid incompatibilities.
|
||
T2 - Genetics
|
||
PY - 1995
|
||
VL - 139
|
||
IS - 4
|
||
PB - Oxford University Press (OUP)
|
||
DO - 10.1093/genetics/139.4.1805
|
||
UR - http://dx.doi.org/10.1093/genetics/139.4.1805
|
||
AB - Abstract Speciation often results from the accumulation of "complementary genes," i.e., from genes that, while having no deleterious effect within species, cause inviability or sterility when brought together with genes from another species. Here I model speciation as the accumulation of genic incompatibilities between diverging populations. Several results are obtained. First, and most important, the number of genic incompatibilities between taxa increases much faster than linearly with time. In particular, the probability of speciation increases at least as fast as the square of the time since separation between two taxa. Second, as Muller realized, all hybrid incompatibilities must initially be asymmetric. Third, at loci that have diverged between taxa, evolutionarily derived alleles cause hybrid problems far more often than ancestral alleles. Last, it is "easier" to evolve complex hybrid incompatibilities requiring the simultaneous action of three or more loci than to evolve simple incompatibilities between pairs of genes. These results have several important implications for genetic analyses of speciation.
|
||
SP - 1805
|
||
EP - 1813
|
||
N1 - Manuscript reference 48
|
||
ER -
|
||
TY - JOUR
|
||
AU - Orr, H. Allen
|
||
AU - Turelli, Michael
|
||
TI - THE EVOLUTION OF POSTZYGOTIC ISOLATION: ACCUMULATING DOBZHANSKY-MULLER INCOMPATIBILITIES
|
||
T2 - Evolution
|
||
PY - 2001
|
||
VL - 55
|
||
IS - 6
|
||
PB - Oxford University Press (OUP)
|
||
DO - 10.1111/j.0014-3820.2001.tb00628.x
|
||
UR - http://dx.doi.org/10.1111/j.0014-3820.2001.tb00628.x
|
||
SP - 1085
|
||
EP - 1094
|
||
N1 - Manuscript reference 49
|
||
ER -
|
||
TY - JOUR
|
||
AU - Ainsworth, Samuel K.
|
||
AU - Hayase, Jonathan
|
||
AU - Srinivasa, Siddhartha
|
||
TI - Git Re-Basin: Merging Models modulo Permutation Symmetries
|
||
T2 - arXiv
|
||
AN - arXiv:2209.04836
|
||
PY - 2022
|
||
PB - arXiv
|
||
DO - 10.48550/arXiv.2209.04836
|
||
UR - https://arxiv.org/abs/2209.04836
|
||
AB - The success of deep learning is due in large part to our ability to solve certain massive non-convex optimization problems with relative ease. Though non-convex optimization is NP-hard, simple algorithms -- often variants of stochastic gradient descent -- exhibit surprising effectiveness in fitting large neural networks in practice. We argue that neural network loss landscapes often contain (nearly) a single basin after accounting for all possible permutation symmetries of hidden units a la Entezari et al. 2021. We introduce three algorithms to permute the units of one model to bring them into alignment with a reference model in order to merge the two models in weight space. This transformation produces a functionally equivalent set of weights that lie in an approximately convex basin near the reference model. Experimentally, we demonstrate the single basin phenomenon across a variety of model architectures and datasets, including the first (to our knowledge) demonstration of zero-barrier linear mode connectivity between independently trained ResNet models on CIFAR-10. Additionally, we identify intriguing phenomena relating model width and training time to mode connectivity. Finally, we discuss shortcomings of the linear mode connectivity hypothesis, including a counterexample to the single basin theory.
|
||
N1 - Manuscript reference 50
|
||
ER -
|
||
TY - JOUR
|
||
AU - Li, Tianyi
|
||
AU - Shen, Zhiqiang
|
||
TI - Scaling Linear Mode Connectivity and Merging to Billion Parameter Pretrained Transformers
|
||
T2 - arXiv
|
||
AN - arXiv:2606.23607
|
||
PY - 2026
|
||
PB - arXiv
|
||
DO - 10.48550/arXiv.2606.23607
|
||
UR - https://arxiv.org/abs/2606.23607
|
||
AB - Linear mode connectivity (LMC) provides a promising foundation for understanding and merging independently trained neural networks, but existing methods typically optimize the interpolation path from only one model endpoint, limiting their scalability and effectiveness for large pretrained transformers. We propose a novel and scalable framework for enabling LMC-based model merging to {\em billion-parameter pretrained transformers}. Our method applies properly parameterized functionality-preserving weight transformations to align functionally equivalent solutions, and introduces a dual learning procedure in which both models jointly learn their corresponding transformations toward a shared linear interpolation path. This bidirectional optimization substantially reduces interpolation barriers and enables more reliable merging across large-scale architectures. Empirically, we show that our approach achieves near-zero loss barriers on WikiText for language models with medium-sized parameters, representing, to our knowledge, the first demonstration of near-barrier-free linear connectivity at this scale. In the vision domain, ViT-L maintains above 69\% ImageNet top-1 accuracy throughout the interpolation path, while modern billion-parameter LLMs exhibit only small loss barriers. These results suggest that properly resolving parameter symmetries enables large pretrained Transformers to be connected and merged through simple linear paths with substantially improved interpolation performance. Code: https://github.com/VILA-Lab/Dual-Learned-Matching .
|
||
N1 - Manuscript reference 51
|
||
ER -
|
||
TY - JOUR
|
||
AU - Sharma, Ekansh
|
||
AU - Roy, Daniel M.
|
||
AU - Dziugaite, Gintare Karolina
|
||
TI - The Non-Local Model Merging Problem: Permutation Symmetries and Variance Collapse
|
||
T2 - arXiv
|
||
AN - arXiv:2410.12766
|
||
PY - 2024
|
||
PB - arXiv
|
||
DO - 10.48550/arXiv.2410.12766
|
||
UR - https://arxiv.org/abs/2410.12766
|
||
AB - Model merging aims to efficiently combine the weights of multiple expert models, each trained on a specific task, into a single multi-task model, with strong performance across all tasks. When applied to all but the last layer of weights, existing methods -- such as Task Arithmetic, TIES-merging, and TALL mask merging -- work well to combine expert models obtained by fine-tuning a common foundation model, operating within a "local" neighborhood of the foundation model. This work explores the more challenging scenario of "non-local" merging, which we find arises when an expert model changes significantly during pretraining or where the expert models do not even share a common foundation model. We observe that standard merging techniques often fail to generalize effectively in this non-local setting, even when accounting for permutation symmetries using standard techniques. We identify that this failure is, in part, due to "variance collapse", a phenomenon identified also in the setting of linear mode connectivity by Jordan et al. (2023). To address this, we propose a multi-task technique to re-scale and shift the output activations of the merged model for each task, aligning its output statistics with those of the corresponding task-specific expert models. Our experiments demonstrate that this correction significantly improves the performance of various model merging approaches in non-local settings, providing a strong baseline for future research on this problem.
|
||
N1 - Manuscript reference 52
|
||
ER -
|
||
TY - JOUR
|
||
AU - Kozodoi, Nikita
|
||
AU - Afolabi, Zainab
|
||
AU - Butler, Jack
|
||
TI - Are we Merging the Right Models? Impact of Expert Training Duration on Model Merging for LLMs
|
||
T2 - arXiv
|
||
AN - arXiv:2607.11997
|
||
PY - 2026
|
||
PB - arXiv
|
||
DO - 10.48550/arXiv.2607.11997
|
||
UR - https://arxiv.org/abs/2607.11997
|
||
AB - Multi-task model merging combines separately trained expert models into a single model that handles all tasks without co-training. Standard practice merges experts at their optimal validation loss. We challenge this convention by systematically studying how training duration of domain experts affects the quality of the merged model. We fine-tune experts on five domains (Math, Code, Instruction Following, Multilingual, and Safety) across three model sizes (Qwen 3.5 0.8B, 2B, and 4B), saving checkpoints from 25% to 500% of the optimal training steps and evaluating five merging methods at each duration. Our findings reveal a striking method-dependent pattern: simple averaging degrades sharply with overfitting, while sparsification-based methods achieve their best performance well past the validation optimum. We formalize this through bias-variance decomposition analysis, drawing a parallel to random forests where averaging benefits from high-variance individual learners. These results suggest that training duration and merging method should be chosen jointly rather than independently.
|
||
N1 - Manuscript reference 53
|
||
ER -
|
||
TY - JOUR
|
||
AU - Zhou, Luca
|
||
AU - Zhao, Bo
|
||
AU - Yu, Rose
|
||
AU - Rodolà, Emanuele
|
||
TI - Demystifying Mergeability: Interpretable Properties to Predict Model Merging Success
|
||
T2 - arXiv
|
||
AN - arXiv:2601.22285
|
||
PY - 2026
|
||
PB - arXiv
|
||
DO - 10.48550/arXiv.2601.22285
|
||
UR - https://arxiv.org/abs/2601.22285
|
||
AB - Model merging combines knowledge from separately fine-tuned models, yet the factors driving its success remain poorly understood. While recent work treats mergeability as an intrinsic property of the models, we show with an architecture-agnostic framework that it fundamentally depends on both the merging method and the partner tasks. Using L1-regularized linear optimization over a set of interpretable pairwise metrics (e.g., gradient L_2 distance), we uncover properties correlating with post-merge normalized accuracy across five merging methods. We find architecture- and method-specific variation in success drivers (64.0% average top-5 metric overlap; 79.3% sign agreement), with certain methods, notably TIES, exhibiting distinct ``fingerprints'' that diverge from the broader consensus. Crucially, however, gradient alignment metrics consistently emerge as the most fundamental signals of compatibility. These findings provide a diagnostic foundation for understanding mergeability and motivate future merge-aware fine-tuning strategies.
|
||
N1 - Manuscript reference 54
|
||
ER -
|
||
TY - JOUR
|
||
AU - Scialom, Thomas
|
||
AU - Chakrabarty, Tuhin
|
||
AU - Muresan, Smaranda
|
||
TI - Fine-tuned Language Models are Continual Learners
|
||
T2 - arXiv
|
||
AN - arXiv:2205.12393
|
||
PY - 2022
|
||
PB - arXiv
|
||
DO - 10.48550/arXiv.2205.12393
|
||
UR - https://arxiv.org/abs/2205.12393
|
||
AB - Recent work on large language models relies on the intuition that most natural language processing tasks can be described via natural language instructions. Language models trained on these instructions show strong zero-shot performance on several standard datasets. However, these models even though impressive still perform poorly on a wide range of tasks outside of their respective training and evaluation sets. To address this limitation, we argue that a model should be able to keep extending its knowledge and abilities, without forgetting previous skills. In spite of the limited success of Continual Learning we show that Language Models can be continual learners. We empirically investigate the reason for this success and conclude that Continual Learning emerges from self-supervision pre-training. Our resulting model Continual-T0 (CT0) is able to learn diverse new tasks, while still maintaining good performance on previous tasks, spanning remarkably through 70 datasets in total. Finally, we show that CT0 is able to combine instructions in ways it was never trained for, demonstrating some compositionality.
|
||
N1 - Manuscript reference 55
|
||
ER -
|
||
TY - JOUR
|
||
AU - Ibrahim, Adam
|
||
AU - Thérien, Benjamin
|
||
AU - Gupta, Kshitij
|
||
AU - Richter, Mats L.
|
||
AU - Anthony, Quentin
|
||
AU - Lesort, Timothée
|
||
AU - Belilovsky, Eugene
|
||
AU - Rish, Irina
|
||
TI - Simple and Scalable Strategies to Continually Pre-train Large Language Models
|
||
T2 - arXiv
|
||
AN - arXiv:2403.08763
|
||
PY - 2024
|
||
PB - arXiv
|
||
DO - 10.48550/arXiv.2403.08763
|
||
UR - https://arxiv.org/abs/2403.08763
|
||
AB - Large language models (LLMs) are routinely pre-trained on billions of tokens, only to start the process over again once new data becomes available. A much more efficient solution is to continually pre-train these models, saving significant compute compared to re-training. However, the distribution shift induced by new data typically results in degraded performance on previous data or poor adaptation to the new data. In this work, we show that a simple and scalable combination of learning rate (LR) re-warming, LR re-decaying, and replay of previous data is sufficient to match the performance of fully re-training from scratch on all available data, as measured by the final loss and the average score on several language model (LM) evaluation benchmarks. Specifically, we show this for a weak but realistic distribution shift between two commonly used LLM pre-training datasets (English$\rightarrow$English) and a stronger distribution shift (English$\rightarrow$German) at the $405$M parameter model scale with large dataset sizes (hundreds of billions of tokens). Selecting the weak but realistic shift for larger-scale experiments, we also find that our continual learning strategies match the re-training baseline for a 10B parameter LLM. Our results demonstrate that LLMs can be successfully updated via simple and scalable continual learning strategies, matching the re-training baseline using only a fraction of the compute. Finally, inspired by previous work, we propose alternatives to the cosine learning rate schedule that help circumvent forgetting induced by LR re-warming and that are not bound to a fixed token budget.
|
||
N1 - Manuscript reference 56
|
||
ER -
|
||
TY - JOUR
|
||
AU - ROBINS, ANTHONY
|
||
TI - Catastrophic Forgetting, Rehearsal and Pseudorehearsal
|
||
T2 - Connection Science
|
||
PY - 1995
|
||
VL - 7
|
||
IS - 2
|
||
PB - Informa UK Limited
|
||
DO - 10.1080/09540099550039318
|
||
UR - http://dx.doi.org/10.1080/09540099550039318
|
||
SP - 123
|
||
EP - 146
|
||
N1 - Manuscript reference 57
|
||
ER -
|
||
TY - JOUR
|
||
AU - Shin, Hanul
|
||
AU - Lee, Jung Kwon
|
||
AU - Kim, Jaehong
|
||
AU - Kim, Jiwon
|
||
TI - Continual Learning with Deep Generative Replay
|
||
T2 - arXiv
|
||
AN - arXiv:1705.08690
|
||
PY - 2017
|
||
PB - arXiv
|
||
DO - 10.48550/arXiv.1705.08690
|
||
UR - https://arxiv.org/abs/1705.08690
|
||
AB - Attempts to train a comprehensive artificial intelligence capable of solving multiple tasks have been impeded by a chronic problem called catastrophic forgetting. Although simply replaying all previous data alleviates the problem, it requires large memory and even worse, often infeasible in real world applications where the access to past data is limited. Inspired by the generative nature of hippocampus as a short-term memory system in primate brain, we propose the Deep Generative Replay, a novel framework with a cooperative dual model architecture consisting of a deep generative model ("generator") and a task solving model ("solver"). With only these two models, training data for previous tasks can easily be sampled and interleaved with those for a new task. We test our methods in several sequential learning settings involving image classification tasks.
|
||
N1 - Manuscript reference 58
|
||
ER -
|
||
TY - JOUR
|
||
AU - Feng, Yunzhen
|
||
AU - Dohmatob, Elvis
|
||
AU - Yang, Pu
|
||
AU - Charton, Francois
|
||
AU - Kempe, Julia
|
||
TI - Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verification
|
||
T2 - arXiv
|
||
AN - arXiv:2406.07515
|
||
PY - 2024
|
||
PB - arXiv
|
||
DO - 10.48550/arXiv.2406.07515
|
||
UR - https://arxiv.org/abs/2406.07515
|
||
AB - Large Language Models (LLM) are increasingly trained on data generated by other LLM, either because generated text and images become part of the pre-training corpus, or because synthetized data is used as a replacement for expensive human-annotation. This raises concerns about \emph{model collapse}, a drop in model performance when their training sets include generated data. Considering that it is easier for both humans and machines to tell between good and bad examples than to generate high-quality samples, we investigate the use of verification on synthesized data to prevent model collapse. We provide a theoretical characterization using Gaussian mixtures, linear classifiers, and linear verifiers to derive conditions with measurable proxies to assess whether the verifier can effectively select synthesized data that leads to optimal performance. We experiment with two practical tasks -- computing matrix eigenvalues with transformers and news summarization with LLMs -- which both exhibit model collapse when trained on generated data, and show that verifiers, even imperfect ones, can indeed be harnessed to prevent model collapse and that our proposed proxy measure strongly correlates with performance.
|
||
N1 - Manuscript reference 59
|
||
ER -
|
||
TY - JOUR
|
||
AU - Rusu, Andrei A.
|
||
AU - Rabinowitz, Neil C.
|
||
AU - Desjardins, Guillaume
|
||
AU - Soyer, Hubert
|
||
AU - Kirkpatrick, James
|
||
AU - Kavukcuoglu, Koray
|
||
AU - Pascanu, Razvan
|
||
AU - Hadsell, Raia
|
||
TI - Progressive Neural Networks
|
||
T2 - arXiv
|
||
AN - arXiv:1606.04671
|
||
PY - 2016
|
||
PB - arXiv
|
||
DO - 10.48550/arXiv.1606.04671
|
||
UR - https://arxiv.org/abs/1606.04671
|
||
AB - Learning to solve complex sequences of tasks--while both leveraging transfer and avoiding catastrophic forgetting--remains a key obstacle to achieving human-level intelligence. The progressive networks approach represents a step forward in this direction: they are immune to forgetting and can leverage prior knowledge via lateral connections to previously learned features. We evaluate this architecture extensively on a wide variety of reinforcement learning tasks (Atari and 3D maze games), and show that it outperforms common baselines based on pretraining and finetuning. Using a novel sensitivity measure, we demonstrate that transfer occurs at both low-level sensory and high-level control layers of the learned policy.
|
||
N1 - Manuscript reference 60
|
||
ER -
|
||
TY - JOUR
|
||
AU - Biderman, Dan
|
||
AU - Portes, Jacob
|
||
AU - Ortiz, Jose Javier Gonzalez
|
||
AU - Paul, Mansheej
|
||
AU - Greengard, Philip
|
||
AU - Jennings, Connor
|
||
AU - King, Daniel
|
||
AU - Havens, Sam
|
||
AU - Chiley, Vitaliy
|
||
AU - Frankle, Jonathan
|
||
AU - Blakeney, Cody
|
||
AU - Cunningham, John P.
|
||
TI - LoRA Learns Less and Forgets Less
|
||
T2 - arXiv
|
||
AN - arXiv:2405.09673
|
||
PY - 2024
|
||
PB - arXiv
|
||
DO - 10.48550/arXiv.2405.09673
|
||
UR - https://arxiv.org/abs/2405.09673
|
||
AB - Low-Rank Adaptation (LoRA) is a widely-used parameter-efficient finetuning method for large language models. LoRA saves memory by training only low rank perturbations to selected weight matrices. In this work, we compare the performance of LoRA and full finetuning on two target domains, programming and mathematics. We consider both the instruction finetuning (approximately 100K prompt-response pairs) and continued pretraining (20B unstructured tokens) data regimes. Our results show that, in the standard low-rank settings, LoRA substantially underperforms full finetuning. Nevertheless, LoRA better maintains the base model's performance on tasks outside the target domain. We show that LoRA mitigates forgetting more than common regularization techniques such as weight decay and dropout; it also helps maintain more diverse generations. Finally, we show that full finetuning learns perturbations with a rank that is 10-100X greater than typical LoRA configurations, possibly explaining some of the reported gaps. We conclude by proposing best practices for finetuning with LoRA.
|
||
N1 - Manuscript reference 61
|
||
ER -
|
||
TY - JOUR
|
||
AU - McClelland, James L.
|
||
AU - McNaughton, Bruce L.
|
||
AU - O'Reilly, Randall C.
|
||
TI - Why there are complementary learning systems in the hippocampus and neocortex: Insights from the successes and failures of connectionist models of learning and memory.
|
||
T2 - Psychological Review
|
||
PY - 1995
|
||
VL - 102
|
||
IS - 3
|
||
PB - American Psychological Association (APA)
|
||
DO - 10.1037/0033-295x.102.3.419
|
||
UR - http://dx.doi.org/10.1037/0033-295x.102.3.419
|
||
SP - 419
|
||
EP - 457
|
||
N1 - Manuscript reference 62
|
||
ER -
|
||
TY - JOUR
|
||
AU - Kumaran, Dharshan
|
||
AU - Hassabis, Demis
|
||
AU - McClelland, James L.
|
||
TI - What Learning Systems do Intelligent Agents Need? Complementary Learning Systems Theory Updated
|
||
T2 - Trends in Cognitive Sciences
|
||
PY - 2016
|
||
VL - 20
|
||
IS - 7
|
||
PB - Elsevier BV
|
||
DO - 10.1016/j.tics.2016.05.004
|
||
UR - http://dx.doi.org/10.1016/j.tics.2016.05.004
|
||
SP - 512
|
||
EP - 534
|
||
N1 - Manuscript reference 63
|
||
ER -
|
||
TY - JOUR
|
||
AU - Schwarz, Jonathan
|
||
AU - Luketina, Jelena
|
||
AU - Czarnecki, Wojciech M.
|
||
AU - Grabska-Barwinska, Agnieszka
|
||
AU - Teh, Yee Whye
|
||
AU - Pascanu, Razvan
|
||
AU - Hadsell, Raia
|
||
TI - Progress & Compress: A scalable framework for continual learning
|
||
T2 - arXiv
|
||
AN - arXiv:1805.06370
|
||
PY - 2018
|
||
PB - arXiv
|
||
DO - 10.48550/arXiv.1805.06370
|
||
UR - https://arxiv.org/abs/1805.06370
|
||
AB - We introduce a conceptually simple and scalable framework for continual learning domains where tasks are learned sequentially. Our method is constant in the number of parameters and is designed to preserve performance on previously encountered tasks while accelerating learning progress on subsequent problems. This is achieved by training a network with two components: A knowledge base, capable of solving previously encountered problems, which is connected to an active column that is employed to efficiently learn the current task. After learning a new task, the active column is distilled into the knowledge base, taking care to protect any previously acquired skills. This cycle of active learning (progression) followed by consolidation (compression) requires no architecture growth, no access to or storing of previous data or tasks, and no task-specific parameters. We demonstrate the progress & compress approach on sequential classification of handwritten alphabets as well as two reinforcement learning domains: Atari games and 3D maze navigation.
|
||
N1 - Manuscript reference 64
|
||
ER -
|
||
TY - JOUR
|
||
AU - Ilharco, Gabriel
|
||
AU - Ribeiro, Marco Tulio
|
||
AU - Wortsman, Mitchell
|
||
AU - Gururangan, Suchin
|
||
AU - Schmidt, Ludwig
|
||
AU - Hajishirzi, Hannaneh
|
||
AU - Farhadi, Ali
|
||
TI - Editing Models with Task Arithmetic
|
||
T2 - arXiv
|
||
AN - arXiv:2212.04089
|
||
PY - 2022
|
||
PB - arXiv
|
||
DO - 10.48550/arXiv.2212.04089
|
||
UR - https://arxiv.org/abs/2212.04089
|
||
AB - Changing how pre-trained models behave -- e.g., improving their performance on a downstream task or mitigating biases learned during pre-training -- is a common practice when developing machine learning systems. In this work, we propose a new paradigm for steering the behavior of neural networks, centered around \textit{task vectors}. A task vector specifies a direction in the weight space of a pre-trained model, such that movement in that direction improves performance on the task. We build task vectors by subtracting the weights of a pre-trained model from the weights of the same model after fine-tuning on a task. We show that these task vectors can be modified and combined together through arithmetic operations such as negation and addition, and the behavior of the resulting model is steered accordingly. Negating a task vector decreases performance on the target task, with little change in model behavior on control tasks. Moreover, adding task vectors together can improve performance on multiple tasks at once. Finally, when tasks are linked by an analogy relationship of the form ``A is to B as C is to D", combining task vectors from three of the tasks can improve performance on the fourth, even when no data from the fourth task is used for training. Overall, our experiments with several models, modalities and tasks show that task arithmetic is a simple, efficient and effective way of editing models.
|
||
N1 - Manuscript reference 65
|
||
ER -
|
||
TY - JOUR
|
||
AU - Marczak, Daniel
|
||
AU - Twardowski, Bartłomiej
|
||
AU - Trzciński, Tomasz
|
||
AU - Cygert, Sebastian
|
||
TI - MagMax: Leveraging Model Merging for Seamless Continual Learning
|
||
T2 - arXiv
|
||
AN - arXiv:2407.06322
|
||
PY - 2024
|
||
PB - arXiv
|
||
DO - 10.48550/arXiv.2407.06322
|
||
UR - https://arxiv.org/abs/2407.06322
|
||
AB - This paper introduces a continual learning approach named MagMax, which utilizes model merging to enable large pre-trained models to continuously learn from new data without forgetting previously acquired knowledge. Distinct from traditional continual learning methods that aim to reduce forgetting during task training, MagMax combines sequential fine-tuning with a maximum magnitude weight selection for effective knowledge integration across tasks. Our initial contribution is an extensive examination of model merging techniques, revealing that simple approaches like weight averaging and random weight selection surprisingly hold up well in various continual learning contexts. More importantly, we present MagMax, a novel model-merging strategy that enables continual learning of large pre-trained models for successive tasks. Our thorough evaluation demonstrates the superiority of MagMax in various scenarios, including class- and domain-incremental learning settings. The code is available at this URL: https://github.com/danielm1405/magmax.
|
||
N1 - Manuscript reference 66
|
||
ER -
|
||
TY - JOUR
|
||
AU - Alexandrov, Anton
|
||
AU - Raychev, Veselin
|
||
AU - Müller, Mark Niklas
|
||
AU - Zhang, Ce
|
||
AU - Vechev, Martin
|
||
AU - Toutanova, Kristina
|
||
TI - Mitigating Catastrophic Forgetting in Language Transfer via Model Merging
|
||
T2 - arXiv
|
||
AN - arXiv:2407.08699
|
||
PY - 2024
|
||
PB - arXiv
|
||
DO - 10.48550/arXiv.2407.08699
|
||
UR - https://arxiv.org/abs/2407.08699
|
||
AB - As open-weight large language models (LLMs) achieve ever more impressive performances across a wide range of tasks in English, practitioners aim to adapt these models to different languages. However, such language adaptation is often accompanied by catastrophic forgetting of the base model's capabilities, severely limiting the usefulness of the resulting model. We address this issue by proposing Branch-and-Merge (BaM), a new adaptation method based on iteratively merging multiple models, fine-tuned on a subset of the available training data. BaM is based on the insight that this yields lower magnitude but higher quality weight changes, reducing forgetting of the source domain while maintaining learning on the target domain. We demonstrate in an extensive empirical study on Bulgarian and German that BaM can significantly reduce forgetting while matching or even improving target domain performance compared to both standard continued pretraining and instruction finetuning across different model architectures.
|
||
N1 - Manuscript reference 67
|
||
ER -
|
||
TY - JOUR
|
||
AU - Dziadzio, Sebastian
|
||
AU - Udandarao, Vishaal
|
||
AU - Roth, Karsten
|
||
AU - Prabhu, Ameya
|
||
AU - Akata, Zeynep
|
||
AU - Albanie, Samuel
|
||
AU - Bethge, Matthias
|
||
TI - How to Merge Your Multimodal Models Over Time?
|
||
T2 - arXiv
|
||
AN - arXiv:2412.06712
|
||
PY - 2024
|
||
PB - arXiv
|
||
DO - 10.48550/arXiv.2412.06712
|
||
UR - https://arxiv.org/abs/2412.06712
|
||
AB - Model merging combines multiple expert models - finetuned from a base foundation model on diverse tasks and domains - into a single, more capable model. However, most existing model merging approaches assume that all experts are available simultaneously. In reality, new tasks and domains emerge progressively over time, requiring strategies to integrate the knowledge of expert models as they become available: a process we call temporal model merging. The temporal dimension introduces unique challenges not addressed in prior work, raising new questions such as: when training for a new task, should the expert model start from the merged past experts or from the original base model? Should we merge all models at each time step? Which merging techniques are best suited for temporal merging? Should different strategies be used to initialize the training and deploy the model? To answer these questions, we propose a unified framework called TIME - Temporal Integration of Model Expertise - which defines temporal model merging across three axes: (1) Initialization Phase, (2) Deployment Phase, and (3) Merging Technique. Using TIME, we study temporal model merging across model sizes, compute budgets, and learning horizons on the FoMo-in-Flux benchmark. Our comprehensive suite of experiments across TIME allows us to uncover key insights for temporal model merging, offering a better understanding of current challenges and best practices for effective temporal model merging.
|
||
N1 - Manuscript reference 68
|
||
ER -
|
||
TY - JOUR
|
||
AU - Toneva, Mariya
|
||
AU - Sordoni, Alessandro
|
||
AU - Combes, Remi Tachet des
|
||
AU - Trischler, Adam
|
||
AU - Bengio, Yoshua
|
||
AU - Gordon, Geoffrey J.
|
||
TI - An Empirical Study of Example Forgetting during Deep Neural Network Learning
|
||
T2 - arXiv
|
||
AN - arXiv:1812.05159
|
||
PY - 2018
|
||
PB - arXiv
|
||
DO - 10.48550/arXiv.1812.05159
|
||
UR - https://arxiv.org/abs/1812.05159
|
||
AB - Inspired by the phenomenon of catastrophic forgetting, we investigate the learning dynamics of neural networks as they train on single classification tasks. Our goal is to understand whether a related phenomenon occurs when data does not undergo a clear distributional shift. We define a `forgetting event' to have occurred when an individual training example transitions from being classified correctly to incorrectly over the course of learning. Across several benchmark data sets, we find that: (i) certain examples are forgotten with high frequency, and some not at all; (ii) a data set's (un)forgettable examples generalize across neural architectures; and (iii) based on forgetting dynamics, a significant fraction of examples can be omitted from the training data set while still maintaining state-of-the-art generalization performance.
|
||
N1 - Manuscript reference 69
|
||
ER -
|
||
TY - JOUR
|
||
AU - Kandpal, Nikhil
|
||
AU - Deng, Haikang
|
||
AU - Roberts, Adam
|
||
AU - Wallace, Eric
|
||
AU - Raffel, Colin
|
||
TI - Large Language Models Struggle to Learn Long-Tail Knowledge
|
||
T2 - arXiv
|
||
AN - arXiv:2211.08411
|
||
PY - 2022
|
||
PB - arXiv
|
||
DO - 10.48550/arXiv.2211.08411
|
||
UR - https://arxiv.org/abs/2211.08411
|
||
AB - The Internet contains a wealth of knowledge -- from the birthdays of historical figures to tutorials on how to code -- all of which may be learned by language models. However, while certain pieces of information are ubiquitous on the web, others appear extremely rarely. In this paper, we study the relationship between the knowledge memorized by large language models and the information in pre-training datasets scraped from the web. In particular, we show that a language model's ability to answer a fact-based question relates to how many documents associated with that question were seen during pre-training. We identify these relevant documents by entity linking pre-training datasets and counting documents that contain the same entities as a given question-answer pair. Our results demonstrate strong correlational and causal relationships between accuracy and relevant document count for numerous question answering datasets (e.g., TriviaQA), pre-training corpora (e.g., ROOTS), and model sizes (e.g., 176B parameters). Moreover, while larger models are better at learning long-tail knowledge, we estimate that today's models must be scaled by many orders of magnitude to reach competitive QA performance on questions with little support in the pre-training data. Finally, we show that retrieval-augmentation can reduce the dependence on relevant pre-training information, presenting a promising approach for capturing the long-tail.
|
||
N1 - Manuscript reference 70
|
||
ER -
|
||
TY - JOUR
|
||
AU - Liu, Xialei
|
||
AU - Hu, Yu-Song
|
||
AU - Cao, Xu-Sheng
|
||
AU - Bagdanov, Andrew D.
|
||
AU - Li, Ke
|
||
AU - Cheng, Ming-Ming
|
||
TI - Long-Tailed Class Incremental Learning
|
||
T2 - arXiv
|
||
AN - arXiv:2210.00266
|
||
PY - 2022
|
||
PB - arXiv
|
||
DO - 10.48550/arXiv.2210.00266
|
||
UR - https://arxiv.org/abs/2210.00266
|
||
AB - In class incremental learning (CIL) a model must learn new classes in a sequential manner without forgetting old ones. However, conventional CIL methods consider a balanced distribution for each new task, which ignores the prevalence of long-tailed distributions in the real world. In this work we propose two long-tailed CIL scenarios, which we term ordered and shuffled LT-CIL. Ordered LT-CIL considers the scenario where we learn from head classes collected with more samples than tail classes which have few. Shuffled LT-CIL, on the other hand, assumes a completely random long-tailed distribution for each task. We systematically evaluate existing methods in both LT-CIL scenarios and demonstrate very different behaviors compared to conventional CIL scenarios. Additionally, we propose a two-stage learning baseline with a learnable weight scaling layer for reducing the bias caused by long-tailed distribution in LT-CIL and which in turn also improves the performance of conventional CIL due to the limited exemplars. Our results demonstrate the superior performance (up to 6.44 points in average incremental accuracy) of our approach on CIFAR-100 and ImageNet-Subset. The code is available at https://github.com/xialeiliu/Long-Tailed-CIL
|
||
N1 - Manuscript reference 71
|
||
ER -
|
||
TY - JOUR
|
||
AU - Kotha, Suhas
|
||
AU - Springer, Jacob Mitchell
|
||
AU - Raghunathan, Aditi
|
||
TI - Understanding Catastrophic Forgetting in Language Models via Implicit Inference
|
||
T2 - arXiv
|
||
AN - arXiv:2309.10105
|
||
PY - 2023
|
||
PB - arXiv
|
||
DO - 10.48550/arXiv.2309.10105
|
||
UR - https://arxiv.org/abs/2309.10105
|
||
AB - We lack a systematic understanding of the effects of fine-tuning (via methods such as instruction-tuning or reinforcement learning from human feedback), particularly on tasks outside the narrow fine-tuning distribution. In a simplified scenario, we demonstrate that improving performance on tasks within the fine-tuning data distribution comes at the expense of capabilities on other tasks. We hypothesize that language models implicitly infer the task of the prompt and that fine-tuning skews this inference towards tasks in the fine-tuning distribution. To test this, we propose Conjugate Prompting, which artificially makes the task look farther from the fine-tuning distribution while requiring the same capability, and we find that this recovers some of the pretraining capabilities in our synthetic setup. Since real-world fine-tuning distributions are predominantly English, we apply conjugate prompting to recover pretrained capabilities in LLMs by simply translating the prompts to different languages. This allows us to recover in-context learning abilities lost via instruction tuning, natural reasoning capability lost during code fine-tuning, and, more concerningly, harmful content generation suppressed by safety fine-tuning in chatbots like ChatGPT.
|
||
N1 - Manuscript reference 72
|
||
ER -
|
||
TY - JOUR
|
||
AU - Yu, Le
|
||
AU - Yu, Bowen
|
||
AU - Yu, Haiyang
|
||
AU - Huang, Fei
|
||
AU - Li, Yongbin
|
||
TI - Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch
|
||
T2 - arXiv
|
||
AN - arXiv:2311.03099
|
||
PY - 2023
|
||
PB - arXiv
|
||
DO - 10.48550/arXiv.2311.03099
|
||
UR - https://arxiv.org/abs/2311.03099
|
||
AB - In this paper, we unveil that Language Models (LMs) can acquire new capabilities by assimilating parameters from homologous models without retraining or GPUs. We first introduce DARE to set most delta parameters (i.e., the disparity between fine-tuned and pre-trained parameters) to zeros without affecting the abilities of Supervised Fine-Tuning (SFT) LMs, which randomly Drops delta parameters with a ratio $p$ And REscales the remaining ones by $1 / (1 - p)$ to approximate the original embeddings. Then, we use DARE as a versatile plug-in to sparsify delta parameters of multiple SFT homologous models for mitigating parameter interference and merge them into a single model by parameter fusing. We experiment with encoder- and decoder-based LMs, showing that: (1) SFT delta parameter value ranges are typically small (within 0.002) with extreme redundancy, and DARE can effortlessly eliminate 90% or even 99% of them; (2) DARE can merge multiple task-specific LMs into one LM with diverse capabilities. Notably, this phenomenon is more pronounced in large-scale LMs, where the merged LM reveals the potential to surpass the performance of any source LM, providing a new discovery. We also utilize DARE to create a merged LM that ranks first among models with 7 billion parameters on the Open LLM Leaderboard.
|
||
N1 - Manuscript reference 73
|
||
ER -
|
||
TY - JOUR
|
||
AU - Wortsman, Mitchell
|
||
AU - Ilharco, Gabriel
|
||
AU - Gadre, Samir Yitzhak
|
||
AU - Roelofs, Rebecca
|
||
AU - Gontijo-Lopes, Raphael
|
||
AU - Morcos, Ari S.
|
||
AU - Namkoong, Hongseok
|
||
AU - Farhadi, Ali
|
||
AU - Carmon, Yair
|
||
AU - Kornblith, Simon
|
||
AU - Schmidt, Ludwig
|
||
TI - Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
|
||
T2 - arXiv
|
||
AN - arXiv:2203.05482
|
||
PY - 2022
|
||
PB - arXiv
|
||
DO - 10.48550/arXiv.2203.05482
|
||
UR - https://arxiv.org/abs/2203.05482
|
||
AB - The conventional recipe for maximizing model accuracy is to (1) train multiple models with various hyperparameters and (2) pick the individual model which performs best on a held-out validation set, discarding the remainder. In this paper, we revisit the second step of this procedure in the context of fine-tuning large pre-trained models, where fine-tuned models often appear to lie in a single low error basin. We show that averaging the weights of multiple models fine-tuned with different hyperparameter configurations often improves accuracy and robustness. Unlike a conventional ensemble, we may average many models without incurring any additional inference or memory costs -- we call the results "model soups." When fine-tuning large pre-trained models such as CLIP, ALIGN, and a ViT-G pre-trained on JFT, our soup recipe provides significant improvements over the best model in a hyperparameter sweep on ImageNet. The resulting ViT-G model, which attains 90.94% top-1 accuracy on ImageNet, achieved a new state of the art. Furthermore, we show that the model soup approach extends to multiple image classification and natural language processing tasks, improves out-of-distribution performance, and improves zero-shot performance on new downstream tasks. Finally, we analytically relate the performance similarity of weight-averaging and logit-ensembling to flatness of the loss and confidence of the predictions, and validate this relation empirically. Code is available at https://github.com/mlfoundations/model-soups.
|
||
N1 - Manuscript reference 74
|
||
ER -
|
||
TY - JOUR
|
||
AU - Cao, Yuan
|
||
AU - Ran, Dezhi
|
||
AU - Guo, Yuzhe
|
||
AU - Wu, Mengzhou
|
||
AU - Chen, Simin
|
||
AU - Li, Linyi
|
||
AU - Yang, Wei
|
||
AU - Xie, Tao
|
||
TI - An Empirical Study and Theoretical Explanation on Task-Level Model-Merging Collapse
|
||
T2 - arXiv
|
||
AN - arXiv:2603.09463
|
||
PY - 2026
|
||
PB - arXiv
|
||
DO - 10.48550/arXiv.2603.09463
|
||
UR - https://arxiv.org/abs/2603.09463
|
||
AB - Model merging unifies independently fine-tuned LLMs from the same base, enabling reuse and integration of parallel development efforts without retraining. However, in practice we observe that merging does not always succeed: certain combinations of task-specialist models suffer from catastrophic performance degradation after merging. We refer to this failure mode as merging collapse. Intuitively, collapse arises when the learned representations or parameter adjustments for different tasks are fundamentally incompatible, so that merging forces destructive interference rather than synergy. In this paper, we identify and characterize the phenomenon of task-level merging collapse, where certain task combinations consistently trigger huge performance degradation across all merging methods. Through extensive experiments and statistical analysis, we demonstrate that representational incompatibility between tasks is strongly correlated with merging collapse, while parameter-space conflict metrics show minimal correlation, challenging conventional wisdom in model merging literature. We provide a theoretical explanation on this phenomenon through rate-distortion theory with a dimension-dependent bound, establishing fundamental limits on task mergeability regardless of methodology.
|
||
N1 - Manuscript reference 75
|
||
ER -
|
||
TY - JOUR
|
||
AU - Horoi, Stefan
|
||
AU - Wolf, Guy
|
||
AU - Belilovsky, Eugene
|
||
AU - Dziugaite, Gintare Karolina
|
||
TI - From Memorization to Parameter Interference: How Overtraining Experts Harms Model Merging
|
||
T2 - arXiv
|
||
AN - arXiv:2506.14126
|
||
PY - 2025
|
||
PB - arXiv
|
||
DO - 10.48550/arXiv.2506.14126
|
||
UR - https://arxiv.org/abs/2506.14126
|
||
AB - Modern deep learning is increasingly characterized by the use of open-weight foundation models that can be fine-tuned on specialized datasets. This has led to a proliferation of expert models and adapters, often shared via platforms like HuggingFace and AdapterHub. Model merging has recently emerged as an effective way to leverage these existing resources, enabling the composition of capabilities from different model checkpoints. A natural pipeline has thus formed to harness the benefits of transfer learning and amortize sunk training costs: models are pre-trained on general data, fine-tuned on specific tasks, and then multiple checkpoints are merged to obtain a more capable model. A prevailing assumption is that improvements at one stage of this pipeline propagate downstream, leading to gains at subsequent steps. In this work, we challenge that assumption by examining how expert fine-tuning affects model merging. We show that long fine-tuning of experts that optimizes for their individual performance leads to degraded merging performance across vision and language modalities, multiple model scales, and both fully fine-tuned and LoRA-adapted models. We trace this degradation to the memorization of a small set of difficult examples that dominate late fine-tuning steps. This causes negative parameter interference and encodes knowledge that is forgotten during merging. Finally, we demonstrate that task-dependent aggressive early stopping strategies can significantly improve model merging performance.
|
||
N1 - Manuscript reference 76
|
||
ER -
|
||
TY - JOUR
|
||
AU - Livnat, Adi
|
||
AU - Papadimitriou, Christos
|
||
TI - Sex as an algorithm
|
||
T2 - Communications of the ACM
|
||
PY - 2016
|
||
VL - 59
|
||
IS - 11
|
||
PB - Association for Computing Machinery (ACM)
|
||
DO - 10.1145/2934662
|
||
UR - http://dx.doi.org/10.1145/2934662
|
||
AB - Looking at the mysteries of evolution from a computer science point of view yields some unexpected insights.
|
||
SP - 84
|
||
EP - 93
|
||
N1 - Manuscript reference 77
|
||
ER -
|
||
TY - JOUR
|
||
AU - Guo, Yanzhu
|
||
AU - Shang, Guokan
|
||
AU - Vazirgiannis, Michalis
|
||
AU - Clavel, Chloé
|
||
TI - The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Text
|
||
T2 - arXiv
|
||
AN - arXiv:2311.09807
|
||
PY - 2023
|
||
PB - arXiv
|
||
DO - 10.48550/arXiv.2311.09807
|
||
UR - https://arxiv.org/abs/2311.09807
|
||
AB - This study investigates the consequences of training language models on synthetic data generated by their predecessors, an increasingly prevalent practice given the prominence of powerful generative models. Diverging from the usual emphasis on performance metrics, we focus on the impact of this training methodology on linguistic diversity, especially when conducted recursively over time. To assess this, we adapt and develop a set of novel metrics targeting lexical, syntactic, and semantic diversity, applying them in recursive finetuning experiments across various natural language generation tasks in English. Our findings reveal a consistent decrease in the diversity of the model outputs through successive iterations, especially remarkable for tasks demanding high levels of creativity. This trend underscores the potential risks of training language models on synthetic text, particularly concerning the preservation of linguistic richness. Our study highlights the need for careful consideration of the long-term effects of such training approaches on the linguistic capabilities of language models.
|
||
N1 - Manuscript reference 78
|
||
ER -
|
||
TY - JOUR
|
||
AU - Padmakumar, Vishakh
|
||
AU - He, He
|
||
TI - Does Writing with Language Models Reduce Content Diversity?
|
||
T2 - arXiv
|
||
AN - arXiv:2309.05196
|
||
PY - 2023
|
||
PB - arXiv
|
||
DO - 10.48550/arXiv.2309.05196
|
||
UR - https://arxiv.org/abs/2309.05196
|
||
AB - Large language models (LLMs) have led to a surge in collaborative writing with model assistance. As different users incorporate suggestions from the same model, there is a risk of decreased diversity in the produced content, potentially limiting diverse perspectives in public discourse. In this work, we measure the impact of co-writing on diversity via a controlled experiment, where users write argumentative essays in three setups -- using a base LLM (GPT3), a feedback-tuned LLM (InstructGPT), and writing without model help. We develop a set of diversity metrics and find that writing with InstructGPT (but not the GPT3) results in a statistically significant reduction in diversity. Specifically, it increases the similarity between the writings of different authors and reduces the overall lexical and content diversity. We additionally find that this effect is mainly attributable to InstructGPT contributing less diverse text to co-written essays. In contrast, the user-contributed text remains unaffected by model collaboration. This suggests that the recent improvement in generation quality from adapting models to human feedback might come at the cost of more homogeneous and less diverse content.
|
||
N1 - Manuscript reference 79
|
||
ER -
|
||
TY - JOUR
|
||
AU - Doshi, Anil R.
|
||
AU - Hauser, Oliver P.
|
||
TI - Generative AI enhances individual creativity but reduces the collective diversity of novel content
|
||
T2 - Science Advances
|
||
PY - 2024
|
||
VL - 10
|
||
IS - 28
|
||
PB - American Association for the Advancement of Science (AAAS)
|
||
DO - 10.1126/sciadv.adn5290
|
||
UR - http://dx.doi.org/10.1126/sciadv.adn5290
|
||
AB - Creativity is core to being human. Generative artificial intelligence (AI)—including powerful large language models (LLMs)—holds promise for humans to be more creative by offering new ideas, or less creative by anchoring on generative AI ideas. We study the causal impact of generative AI ideas on the production of short stories in an online experiment where some writers obtained story ideas from an LLM. We find that access to generative AI ideas causes stories to be evaluated as more creative, better written, and more enjoyable, especially among less creative writers. However, generative AI–enabled stories are more similar to each other than stories by humans alone. These results point to an increase in individual creativity at the risk of losing collective novelty. This dynamic resembles a social dilemma: With generative AI, writers are individually better off, but collectively a narrower scope of novel content is produced. Our results have implications for researchers, policy-makers, and practitioners interested in bolstering creativity.
|
||
N1 - Manuscript reference 80
|
||
ER -
|