Beyond Data Schedule: Fast LLM Pretraining on Data Mixtures by Loss-Reweighting.

arXiv:2610.32141v1 Announce Type: cross Abstract: Large language models often acquire specialized capabilities through successive pretraining stages, but these gains can come at the cost of forgetting earlier knowledge. In this work, we investigate whether suc