Service-Induced Congestion in Memory-Constrained LLM Serving.

arXiv:2606.15555v3 Announce Type: replace-cross Abstract: In large language model (LLM) serving, each request accumulates persistent graphics processing unit (GPU) memory during service as its key-value cache grows with every generated token. Under high concur