Towards Scalable Data Diversification for Language Model Pretraining via Leverage Score Sampling.

arXiv:2609.32484v1 Announce Type: new Abstract: Data selection for language model pretraining faces a fundamental tension between quality and diversity. While quality filtering is empirically effective, it often induces diversity collapse: by favoring texts si