Clean: Second-order LLM Training at Linear Memory Cost via Nystr{\"o}m Sketching.

arXiv:2610.04204v3 Announce Type: replace-cross Abstract: Training large language models (LLMs) entails a fundamental trade-off: memory-efficient optimizers such as Adam discard cross-parameter curvature, whereas full-curvature methods such as SOAP can acceler