Thinking Outside the Box: Retention and Transmission of Information in Sliding-Window KV Inference.

arXiv:2609.34049v1 Announce Type: new Abstract: Sliding-window KV inference refers to processing a sequence incrementally while retaining only a fixed-size cache of recent key and value states. It can be applied to pretrained causal transformers at inference t