Decoupling Token Roles in Autoregressive Pretraining.

arXiv:2609.33405v1 Announce Type: cross Abstract: Autoregressive pretraining increasingly draws on heterogeneous data, making it important to understand how a model learns from an individual token. The next-token prediction objective naturally identifies a tok