Tokka-Bench: Evaluating Tokenizers Across 100 Natural and 20 Programming Languages.

arXiv:2610.08794v1 Announce Type: cross Abstract: Large language models rely on subword tokenizers whose quality varies across languages, yet no standardized multi-metric framework exists for broad comparative evaluation. We introduce Tokka-Bench, an open-sour