Achieving Tokenizer Flexibility in Language Models through Heuristic Adaptation and Supertoken Learning.

arXiv:2505.09738v2 Announce Type: replace-cross Abstract: Pretrained language models (LLMs) are often constrained by their fixed tokenization schemes, leading to inefficiencies and performance limitations, particularly for multilingual or specialized applicati