Gefen: Optimized Stochastic Optimizer.

arXiv:2606.13894v4 Announce Type: replace-cross Abstract: AdamW is a default optimizer for deep learning, but its moment states add two parameter-sized buffers to training memory, increasing the cost of large-scale pretraining. We propose Gefen, a memory-effic