The Hidden Cost of Making AI Speak Indian Languages

Gnananvesh reddy — Sat, 19 Sep 2026 14:59:09 +0000

Why the same sentence can cost 3x more tokens in Hindi than in English — and what it actually takes to fix that.

A sentence that shouldn't cost more to say

Here's a simple test. Take the same short sentence, translate it into five languages, and see how "expensive" it is for an AI model to process.

Language	Characters	Data size (UTF-8 bytes)	Words	Tokens
English	18	18	4	5
Hindi	12	30	3	13
Telugu	17	45	3	20
Kannada	16	44	2	22
Tamil	30	84	3	34

Same sentence. Same meaning. But look at the last column — tokens, the small chunks of text an AI model actually reads (a token might be a full word, a word-fragment, or just a few characters). English needed about 1 token per word. Tamil needed almost 11.

That gap has nothing to do with Tamil being a more "complex" language to understand. It comes from a decision made long before you typed a prompt — how the model's tokenizer was built. And that decision has real costs: Indian-language conversations fill up the model's context window faster, cost more to run, and respond more slowly — for saying the exact same thing an English speaker could say more cheaply.

This piece covers three things: why that gap exists, what I found when I tested it myself, and what it actually takes to close it — including the technique real multilingual AI teams use to retrofit a tokenizer onto a model that's already been trained.

What a tokenizer actually does

Before an AI model "thinks" about your sentence, a tokenizer breaks it into tokens — its basic units of text, similar to Lego bricks. Everything downstream (the model's memory of the conversation, how much compute it burns, how fast it replies) is counted in tokens, not words or characters. So the table above isn't trivia — it's a direct readout of how efficiently each language got chunked.

The tokenizer's chunking isn't hand-designed. It's learned from a huge pile of training text, using an algorithm called byte-level BPE (Byte Pair Encoding). Here's the short version: start with raw bytes (the smallest possible unit of text), then repeatedly find whichever pair of adjacent bytes/characters shows up most often in the training text, and merge it into a single new token.

t h  →  th
th e →  the

Do this over and over and you end up with a vocabulary of tokens — each one earned by how often it appeared in training. "the," "-ing," "-tion" become single, cheap tokens in an English-heavy corpus, because they show up constantly.

Why the byte count is a red herring

English characters typically take 1 byte of storage; most Indic scripts (Devanagari, Telugu, Tamil, Kannada) take 3 bytes per character, because those scripts have far larger character sets. It's tempting to conclude "3 bytes → 3 tokens." That's wrong. A byte-level tokenizer can merge several bytes into one token — a 3-byte character doesn't automatically cost 3 tokens. Whether it does depends entirely on whether the tokenizer learned a good merge for that byte sequence, and that depends on how much of that language it actually saw during training.

Which is the real explanation for the first table: it's not about script complexity. It's about training data. If a tokenizer is trained on 90% English and 10% everything else, English patterns get thousands of chances to earn efficient one-token merges. Indic patterns get far fewer chances, so they stay chopped into small, expensive pieces — not because the algorithm is biased, but because frequency is the only thing BPE optimizes for.

Testing it myself: does rebalancing the training data actually help?

Rather than just claim this, I built three small tokenizers from scratch, changing only the language mix of the training data each time:

Tokenizer A — 90% English / 10% Indic (mirrors how many real-world tokenizers are built today, since most web text is English)
Tokenizer B — 50% English / 50% Indic
Tokenizer C — Balanced equally across English, Hindi, Telugu, Tamil, Kannada

(Caveat: this used a small, self-authored corpus to isolate one variable — not a production-scale dataset. Read the direction and size of the shifts, not the absolute numbers.)

Tokens per word, on held-out test sentences:

Tokenizer	English	Hindi	Telugu	Tamil	Kannada
A — 90% English	3.05	5.34	7.79	10.75	8.86
B — 50/50 split	3.08	4.32	6.36	8.47	7.34
C — Fully balanced	3.60	4.34	6.36	7.84	7.07

Going from A to B — giving Indic languages a fairer share of training data — cut Hindi's cost by ~19%, Telugu's by ~18%, Tamil's by ~21%, while English barely moved (3.05 → 3.08). Rebalancing has a very good cost-benefit ratio, at least at first.

But pushing further, to a fully balanced split (C), gave diminishing returns on the Indic side and actually cost English a bit (3.08 → 3.60). So "more balance is always better" doesn't hold — there's a sweet spot, not a straight line.

Does a bigger vocabulary fix it instead?

I tested that too, sweeping vocabulary size on the balanced corpus:

Vocabulary size	English	Hindi	Telugu	Tamil	Kannada
300	5.28	6.75	9.64	10.72	10.17
800	3.70	3.55	6.42	7.78	7.17
1,500	3.70	3.55	6.42	7.78	7.17
3,000	3.70	3.55	6.42	7.78	7.17

Going from 300 to 800 tokens helped every language substantially. Beyond that, nothing moved — my tiny training corpus simply ran out of frequent patterns worth turning into new tokens. At real production scale, you'd expect the same shape of curve for a related reason: each additional vocabulary slot ends up capturing rarer and rarer patterns, so returns shrink. Either way, the practical lesson holds: a bigger vocabulary isn't automatically better. It directly increases the size of the model's embedding table and output layer (more on that below), so it's a real cost, not a free upgrade.

So neither lever — rebalancing data or growing vocabulary — is a clean fix on its own. Which raises the harder, more practical question: what do you actually do when you already have a trained model with a bad, English-heavy tokenizer, and you want to fix it without starting over?

Why you can't just swap the tokenizer

The obvious idea — just replace the old tokenizer with a better multilingual one — doesn't work, because of how deeply the tokenizer is wired into the model.

During training, the model learns an embedding for every token: a vector of numbers representing that token's meaning, which the rest of the network operates on. Token ID 1234 might map to the embedding for "hello." If you swap in a new tokenizer that reassigns ID 1234 to "नमस्ते" instead, the model's embedding at that slot still represents "hello" — nothing tells the transformer that the meaning behind that ID changed. The result is broken, incoherent output, because the mapping between token IDs and everything the model learned about them has been silently invalidated.

How teams actually retrofit multilingual support: vocabulary expansion + frozen weights

The real technique — used across the industry, not any one company's secret — is vocabulary expansion, and it's done carefully enough to avoid the exact problem above. In broad strokes:

Keep the existing vocabulary untouched. Don't reassign any existing token IDs — that's what causes the semantic mismatch.
Add new tokens for the target languages. Train a new set of merges (often on a multilingual or language-specific corpus) and append these as new token IDs after the existing vocabulary, rather than replacing anything. The vocabulary might grow from, say, 32K tokens to 100K+.
Initialize new embeddings for the new tokens — often randomly, or by averaging nearby/related embeddings as a warmer starting point, since a purely random vector for a brand-new token gives the model nothing to work with initially.
Freeze the original weights, and train only the new embeddings first. This is the key move: the rest of the model — everything it already learned about English, reasoning, code, and so on — is locked (frozen) so it can't be disturbed. Only the newly added embedding rows (and often the output layer's new rows) are trainable. This forces the model to learn what the new tokens mean without the risk of forgetting what it already knew — a known failure mode called catastrophic forgetting.
Continue pretraining on multilingual text, still with most of the model frozen, until the new tokens' embeddings become meaningful enough that the frozen transformer layers can actually make use of them.
Gradually unfreeze more of the model (sometimes the top layers first, sometimes all of it) and continue training, now letting the whole network adapt to the expanded vocabulary rather than just the embedding layer.

The intuition: this is closer to adding new rooms onto a house than demolishing and rebuilding it. The model's existing knowledge stays intact and untouched throughout the early stages, while the new "rooms" — the added vocabulary — are furnished and learned separately before the whole house is allowed to rearrange itself around them.

This is also why vocabulary changes aren't just an input-side problem. The model has to both read the new tokens (input embeddings) and generate them in its replies (the output layer, sometimes called the LM head, which predicts a probability over every token in the vocabulary). Both sides need training, and depending on the model architecture, these two layers may or may not share the same weights.

(One honesty note: exact production pipelines — precise freezing schedules, exact vocabulary size chosen, how long each phase runs — are proprietary details specific to whichever team built a given model, and I'm deliberately not attributing an exact recipe to any specific company without a public source confirming it. The staged approach above — expand vocabulary, freeze and warm up new embeddings, then gradually unfreeze — is the general, well-documented pattern; the specific numbers any one team uses aren't public.)

What this doesn't prove

Worth being explicit about the limits of this argument: token efficiency is not the same as model quality. A tokenizer that produces fewer tokens per word tells you nothing about whether the model actually understands that language, reasons well in it, or translates it accurately. A language can be cheaply tokenized and still poorly modeled, or expensively tokenized and still well understood. Everything here is about the economics layer — cost, latency, context usage — not comprehension.

The takeaway

The full chain: different scripts get different byte representations → BPE learns merges based on frequency in the training data → underrepresented languages get fewer efficient merges → those languages need more tokens to say the same thing → that means less usable context, slower responses, and higher cost, for identical conversations.

Rebalancing training data and growing the vocabulary both help, but both plateau and both carry real costs. The technique that actually works at scale — vocabulary expansion with frozen weights, done carefully to avoid catastrophic forgetting — is more involved than either lever alone, which is exactly why it's the answer real teams reach for instead of just re-training a bigger, more balanced tokenizer from scratch.

For multilingual LLMs, the tokenizer isn't a preprocessing afterthought. It's part of the economics of language — and fixing it properly, after a model is already trained, takes real engineering, not just better data.

All experiment code and corpora for this piece are available on request — this was a small, from-scratch BPE run meant to isolate one variable at a time, not a benchmark of any production tokenizer.

Connect with me on LinkedIn

DEV Community: Gnananvesh reddy