KV-Kaizen: Learning Context-Adaptive Cache Compression Choices.

arXiv:2609.37988v2 Announce Type: replace Abstract: As the context size of text processed with an LLM grows, the size of KV caches can outstrip the memory allocated for the original model weights. This impacts LLM throughput negatively, since decoding is memor