NOSA: Native and Offloadable Sparse Attention.

arXiv:2510.13602v3 Announce Type: replace-cross Abstract: Decoding throughput improvements from larger inference batches are limited by GPU memory, which is largely consumed by the key-value (KV) cache. Prior training-free KV cache offloading alleviates this b