Demystifying Manifold Constraints in LLM Pre-training.

arXiv:2605.04418v2 Announce Type: replace-cross Abstract: The recent success of matrix optimizers (e.g., Muon) suggests that specific normalization of momentum, such as orthogonalization and row-wise normalization, benefits both the stability and acceleration