Decode-Branch Transformers: Decoupling the Primary Prefill Path from Additional Decode Computation.

arXiv:2608.12385v3 Announce Type: replace Abstract: As large language models serve ever more requests, cumulative inference cost is growing relative to the one-time cost of training. In typical serving, prompt prefill runs in parallel and is compute-bound, whe