The language token tax: why your non-English prompts cost more, and what to do about it

Behrooz — Thu, 29 Oct 2026 21:47:52 +0000

AI APIs don't bill you per word. They bill you per token, and a token is whatever
the model's tokenizer decided a piece of text should be.

That sounds like a detail until you count. Here's one instruction, "Always answer in
JSON format.", on GPT-4o's tokenizer:

English: 6 tokens
Hindi: 9 tokens
Punjabi: 18 tokens

Same meaning, three times the price. I've started calling this the language token
tax, and if you ship LLM features outside English, you're paying it on every call.

Tokens are not words

Modern tokenizers learn their vocabulary from training data. They see a huge amount of
English, so common English words become single tokens: answer, format, always.

Languages that show up less often in that data don't get that treatment. Their words
are split into smaller pieces, sometimes down to single characters or raw bytes. More
pieces means more tokens, and more tokens means a bigger bill and less room in the
context window.

I measured it in 55 languages

tokensaver, the open-source prompt-cost tool
I maintain, has a test that runs the same sentence in 55 languages. So I counted every
one of them with o200k_base, the tokenizer behind GPT-4o and OpenAI's other current
models:

Language	Tokens	vs. English
English	6	1.0×
Spanish	7	1.2×
French, Russian, Persian, Chinese (Simplified)	8	1.3×
German, Arabic, Hindi, Japanese	9	1.5×
Korean, Swahili	10	1.7×
Bengali	11	1.8×
Tamil, Burmese	14	2.3×
Telugu	17	2.8×
Punjabi (Gurmukhi)	18	3.0×

36 of the 55 languages needed at least 1.5× as many tokens as English. Not one of
them came out cheaper.

You can check this yourself in a few lines:

import tiktoken

enc = tiktoken.get_encoding("o200k_base")
for lang, text in {
    "English": "Always answer in JSON format.",
    "Hindi": "हमेशा JSON प्रारूप में उत्तर दें।",
    "Tamil": "எப்போதும் JSON வடிவத்தில் பதிலளிக்கவும்.",
}.items():
    print(f"{lang:8} {len(enc.encode(text))} tokens")

It used to be much worse

The older cl100k_base tokenizer, used by GPT-4 and GPT-3.5 Turbo, shows how much has
changed. Same sentence:

Language	cl100k_base (older)	o200k_base (current)
English	6	6
Hindi	28 (4.7×)	9 (1.5×)
Bengali	36 (6.0×)	11 (1.8×)
Tamil	47 (7.8×)	14 (2.3×)
Telugu	71 (11.8×)	17 (2.8×)

Newer tokenizers have narrowed the gap a lot, but they haven't closed it. Two practical
takeaways:

If you're still on an older model, the tax may be several times bigger than you think. Moving to a newer model can cut tokens for non-English prompts even before you touch the prompt.
Every provider tokenizes differently. Claude and Gemini use their own tokenizers, and there's no offline way to count them exactly. Measure on the model you actually use.

A word of caution: this is one short sentence. Ratios move around with the text, so
treat these numbers as a direction, not a price list. Before you budget a product in
another language, count your own prompts in that language.

What you can't fix, and what you can

You can't change how a tokenizer splits Tamil. But a lot of what makes prompts expensive
has nothing to do with language:

Pretty-printed JSON. Indentation and line breaks cost tokens in every script.
Duplicated rules. A rule pasted twice is paid for twice, in any language.
Extra whitespace. Trailing spaces and runs of blank lines.

These are exactly the fixes that work everywhere, so I made sure tokensaver applies them
in any language. I tested it on three sample prompts with that kind of waste: one each
in English, Persian and Chinese. Here are the results on GPT-4o with
tokensaver optimize --apply all:

Prompt	Before	After	Saved
English	486	386	−20.6%
Persian	350	259	−26.0%
Chinese (Simplified)	273	189	−30.8%

The non-English prompts saved more, in percentage terms. When every word costs more
tokens, removing a duplicated paragraph removes more tokens too.

Trying it on your own prompt takes two commands:

pip install prompt-tokensaver
tokensaver optimize prompt.txt

Every suggestion shows the tokens and dollars it saves on your model. Safe changes, like
minifying JSON, are applied automatically; tokensaver re-parses the JSON to make sure
the data didn't change. Anything that could change meaning, like removing a duplicate,
waits for you to review it. It runs fully offline, so your prompts never leave your
machine.

To see what the same prompt costs across models, cheapest first:

tokensaver compare prompt.txt

Filler is language-specific, so it needs native speakers

One kind of waste is tied to language: polite filler. "I would like you to please…" or
"Thanks in advance!" costs tokens and changes nothing about what the model does.

You can't write those rules from a dictionary. You need someone who knows which phrases
are truly empty, and which ones only look empty.

This week, a native Persian speaker contributed exactly that. tokensaver now removes
«لطفاً» ("please"), «از تو می‌خواهم که…» ("I want you to…") and «ممنون می‌شوم اگر…»
("I'd appreciate it if…"). It handles the different ways people actually type them, with
or without the invisible zero-width joiner in the middle of a word.

Just as interesting is what we left out: «اگر امکان دارد», "if possible". It looks
like filler, but "if possible, also write tests" is a real condition. Delete it and an
optional request becomes mandatory. That's the kind of call only a native speaker can
make with confidence.

Your language next?

The filler rule now understands English and Persian. If you speak another language, I'd
love your help: list 3–5 phrases that never change what the model should do, in
issue #5. No code required, though PRs
are very welcome.

TL;DR

You pay per token, and most non-English text needs more tokens for the same meaning: 1.5× or more for 36 of 55 languages in my test, up to 3× for Punjabi.
Newer tokenizers are much fairer than older ones, so check which one your model uses.
You can't fix the tokenizer, but JSON minification, duplicate removal and whitespace cleanup work in any language. On my sample prompts they cut 26% in Persian and 31% in Chinese.
Measure your own prompts: github.com/behrouzbk/tokensaver (MIT, offline, pip install prompt-tokensaver).

If this was useful, a ⭐ on the repo helps other people find it.

DEV Community: Behrooz