Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving.

arXiv:2407.00079v5 Announce Type: replace-cross Abstract: Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI. It features a KVCache-centric disaggregated architecture that separates the prefill and decoding clusters. It al