MOSAIC: Multi-Objective Search for Allocating Inference Cache Across LLM Decoder Layers.

arXiv:2610.25431v1 Announce Type: cross Abstract: The Key-Value (KV) cache is the main memory bottleneck of long-context LLM inference. Token eviction methods such as SnapKV, H2O, AdaKV and L2Norm shrink it, but they give every transformer layer the same budge