PowerStep: Memory-Efficient Adaptive Optimization via $\ell_p$-Norm Steepest Descent.

arXiv:2605.10335v2 Announce Type: replace-cross Abstract: Adaptive optimizers such as Adam are standard for training Transformers, but storing gradient first and second moments incurs substantial memory overhead. We introduce PowerStep, a memory-efficient opti