The gap grows with language coverage. When researchers expanded the set to 200 languages, the token cost gap in one AI model grew by 17 times. The study argued the penalties reach beyond price: higher token counts mean higher latency and reduced effective context capacity, hitting hardest where compute and connectivity are already limited.

Rarity is not the only determinant; script plays a big role. Asturian, despite having few speakers and no Common Crawl data, benefits from the Latin-script advantage and representation from related Romance languages like Italian. Languages written in scripts tokenizers rarely saw, including Burmese, Tibetan, Tifinagh, Khmer, Lao and Ol Chiki, were consistently at the greatest disadvantage across nearly all models analysed.

Better training data alone will not close the gap, warned Dheemanth Reddy, CEO of Maya Research. Token prices are falling quickly, which should soften the direct cost impact over time, but fixing tokenization will require labs to deliberately design future tokenizers to handle these languages and scripts better. Pratyush Choudhury, co-founder of Activate AI, suggested language efficiency should be a product requirement, with major labs publishing a language-by-language scorecard of task costs and committing to measurable improvements for their worst-served languages.