Fantastic Pretraining Optimizers and Where to Find Them.

arXiv:2509.02046v3 Announce Type: replace-cross Abstract: AdamW has long been the dominant optimizer in language model pretraining, despite numerous claims that alternative optimizers offer 1.4 to 2x speedup. We posit that two methodological shortcomings have