MOIRA: Mass-Oriented Indexing with Ragged Attention for Long-Context Decoding.

arXiv:2610.04313v1 Announce Type: new Abstract: Long-context decoding is limited by memory bandwidth, because every output token reads the KV cache of every layer. Sparse decoding reduces this cost by reading only part of the KV cache. We observe that the numb