Originally published on AI Tech Connect.
What the tokeniser tax costs you Every piece of cost-optimisation advice you have read — cache the prefix, route the easy requests to a smaller model, compress the context, batch the offline work — rests on an unstated assumption. It assumes that a token is a token: a fixed, neutral quantum of work, so that reducing the count reduces the cost proportionally and everyone measuring in tokens is measuring the same thing. For English that assumption is close enough to true to be harmless. For Devanagari, Tamil, Bengali and Malayalam it is not true at all, and the error is not a rounding error. It is a multiplier. The clearest measurement to date comes from a paper published on arXiv in July 2026 under the title The Tokenizer Tax: Quantifying and Explaining the Cross-Lingual Cost of Subword…
Top comments (0)