AI APIs don't bill you per word. They bill you per token, and a token is whatever
the model's tokenizer decided a piece of text should be.
That sounds like a detail until you count. Here's one instruction, "Always answer in
JSON format.", on GPT-4o's tokenizer:
- English: 6 tokens
- Hindi: 9 tokens
- Punjabi: 18 tokens
Same meaning, three times the price. I've started calling this the language token
tax, and if you ship LLM features outside English, you're paying it on every call.
Tokens are not words
Modern tokenizers learn their vocabulary from training data. They see a huge amount of
English, so common English words become single tokens: answer, format, always.
Languages that show up less often in that data don't get that treatment. Their words
are split into smaller pieces, sometimes down to single characters or raw bytes. More
pieces means more tokens, and more tokens means a bigger bill and less room in the
context window.
I measured it in 55 languages
tokensaver, the open-source prompt-cost tool
I maintain, has a test that runs the same sentence in 55 languages. So I counted every
one of them with o200k_base, the tokenizer behind GPT-4o and OpenAI's other current
models:
| Language | Tokens | vs. English |
|---|---|---|
| English | 6 | 1.0× |
| Spanish | 7 | 1.2× |
| French, Russian, Persian, Chinese (Simplified) | 8 | 1.3× |
| German, Arabic, Hindi, Japanese | 9 | 1.5× |
| Korean, Swahili | 10 | 1.7× |
| Bengali | 11 | 1.8× |
| Tamil, Burmese | 14 | 2.3× |
| Telugu | 17 | 2.8× |
| Punjabi (Gurmukhi) | 18 | 3.0× |
36 of the 55 languages needed at least 1.5× as many tokens as English. Not one of
them came out cheaper.
You can check this yourself in a few lines:
import tiktoken
enc = tiktoken.get_encoding("o200k_base")
for lang, text in {
"English": "Always answer in JSON format.",
"Hindi": "हमेशा JSON प्रारूप में उत्तर दें।",
"Tamil": "எப்போதும் JSON வடிவத்தில் பதிலளிக்கவும்.",
}.items():
print(f"{lang:8} {len(enc.encode(text))} tokens")
It used to be much worse
The older cl100k_base tokenizer, used by GPT-4 and GPT-3.5 Turbo, shows how much has
changed. Same sentence:
| Language | cl100k_base (older) | o200k_base (current) |
|---|---|---|
| English | 6 | 6 |
| Hindi | 28 (4.7×) | 9 (1.5×) |
| Bengali | 36 (6.0×) | 11 (1.8×) |
| Tamil | 47 (7.8×) | 14 (2.3×) |
| Telugu | 71 (11.8×) | 17 (2.8×) |
Newer tokenizers have narrowed the gap a lot, but they haven't closed it. Two practical
takeaways:
- If you're still on an older model, the tax may be several times bigger than you think. Moving to a newer model can cut tokens for non-English prompts even before you touch the prompt.
- Every provider tokenizes differently. Claude and Gemini use their own tokenizers, and there's no offline way to count them exactly. Measure on the model you actually use.
A word of caution: this is one short sentence. Ratios move around with the text, so
treat these numbers as a direction, not a price list. Before you budget a product in
another language, count your own prompts in that language.
What you can't fix, and what you can
You can't change how a tokenizer splits Tamil. But a lot of what makes prompts expensive
has nothing to do with language:
- Pretty-printed JSON. Indentation and line breaks cost tokens in every script.
- Duplicated rules. A rule pasted twice is paid for twice, in any language.
- Extra whitespace. Trailing spaces and runs of blank lines.
These are exactly the fixes that work everywhere, so I made sure tokensaver applies them
in any language. I tested it on three sample prompts with that kind of waste: one each
in English, Persian and Chinese. Here are the results on GPT-4o with
tokensaver optimize --apply all:
| Prompt | Before | After | Saved |
|---|---|---|---|
| English | 486 | 386 | −20.6% |
| Persian | 350 | 259 | −26.0% |
| Chinese (Simplified) | 273 | 189 | −30.8% |
The non-English prompts saved more, in percentage terms. When every word costs more
tokens, removing a duplicated paragraph removes more tokens too.
Trying it on your own prompt takes two commands:
pip install prompt-tokensaver
tokensaver optimize prompt.txt
Every suggestion shows the tokens and dollars it saves on your model. Safe changes, like
minifying JSON, are applied automatically; tokensaver re-parses the JSON to make sure
the data didn't change. Anything that could change meaning, like removing a duplicate,
waits for you to review it. It runs fully offline, so your prompts never leave your
machine.
To see what the same prompt costs across models, cheapest first:
tokensaver compare prompt.txt
Filler is language-specific, so it needs native speakers
One kind of waste is tied to language: polite filler. "I would like you to please…" or
"Thanks in advance!" costs tokens and changes nothing about what the model does.
You can't write those rules from a dictionary. You need someone who knows which phrases
are truly empty, and which ones only look empty.
This week, a native Persian speaker contributed exactly that. tokensaver now removes
«لطفاً» ("please"), «از تو میخواهم که…» ("I want you to…") and «ممنون میشوم اگر…»
("I'd appreciate it if…"). It handles the different ways people actually type them, with
or without the invisible zero-width joiner in the middle of a word.
Just as interesting is what we left out: «اگر امکان دارد», "if possible". It looks
like filler, but "if possible, also write tests" is a real condition. Delete it and an
optional request becomes mandatory. That's the kind of call only a native speaker can
make with confidence.
Your language next?
The filler rule now understands English and Persian. If you speak another language, I'd
love your help: list 3–5 phrases that never change what the model should do, in
issue #5. No code required, though PRs
are very welcome.
TL;DR
- You pay per token, and most non-English text needs more tokens for the same meaning: 1.5× or more for 36 of 55 languages in my test, up to 3× for Punjabi.
- Newer tokenizers are much fairer than older ones, so check which one your model uses.
- You can't fix the tokenizer, but JSON minification, duplicate removal and whitespace cleanup work in any language. On my sample prompts they cut 26% in Persian and 31% in Chinese.
- Measure your own prompts:
github.com/behrouzbk/tokensaver (MIT,
offline,
pip install prompt-tokensaver).
If this was useful, a ⭐ on the repo helps other people find it.
Top comments (3)
Punjabi at 18 tokens against 6 for English makes the 3.0x gap hard to ignore, but the article's own warning matters too: one short sentence is not a price list. Budgeting needs counts from real prompts, not just this table.
The language gap lands on the text the user types. If the system prompt and tool schemas are in English, they are a fixed cost whatever language the user writes in. Size that part before changing anything else.
One operational fix that helps here is keeping system prompts, tool schemas, and output format rules in English while leaving only the user content and domain text in the target language. That avoids paying the 2x or 3x tax on your static instructions. It also protects prompt-cache hits, since translating the boilerplate fragments the cached prefix across every language variant.