Mind the Delay: Latency-aware Query Routing for Heterogeneous LLM pools.

arXiv:2607.18253v2 Announce Type: replace Abstract: Large language model (LLM) query routers typically assign each query to a suitable model that balances response accuracy and monetary cost. Subsequently, separate system-level load balancers based on policies