On the Fundamental Limits of Prefill-Based Decoding Length Prediction.

arXiv:2610.40541v1 Announce Type: new Abstract: Predicting how many tokens a large language model (LLM) will decode, using only prefill-phase hidden states, is a key primitive for inference scheduling. We show that current benchmarks conflate predictor quality