There is no single cheapest LLM for multilingual text: measured on the same passage, GPT-5.5 bills the fewest tokens for European languages, Hindi, and Korean, Kimi K2.5 is leanest on Chinese, and DeepSeek on Japanese. Claude Fable 5, Opus 4.8, and Sonnet 5 share one tokenizer (identical counts on every sample we sent) and are never the leanest: the same English paragraph bills 90 raw tokens on Claude against DeepSeek's 55, and the net premium runs from 1.3x on Japanese to 2.2x on Chinese. The token is the billing unit, so two things you rarely see on a pricing page decide your input cost: how densely a language packs meaning into characters, and how well each model's tokenizer compressed that script. They multiply, and the product is not what the per-character view suggests.
TL;DR
- Claude Fable 5, Opus 4.8, and Sonnet 5 share one tokenizer, never the leanest: 1.2-2.3x the cheapest count everywhere.
- The cheapest tokenizer flips by language: GPT-5.5 on European languages, Hindi, Korean; Kimi on Chinese; DeepSeek on Japanese.
- Per character CJK looks 3x worse, but per meaning Chinese sits near parity while Japanese and Korean run 1.5-2.4x.
- Cost is script density times tokenizer coverage; missing coverage multiplies it (GLM bills Hindi at 4.9x its English).
- Localizing rarely saves money; match model to language by tokens.
Counts were measured through the Synthorai gateway on 2026-07-08, always using each provider's own count, never a local tokenizer. Every count was identical across repeats.
The token is the billing unit, not the text
You are billed per token, but a token is not a character and not a word. Each model ships its own tokenizer with its own vocabulary, and the same sentence resolves to a different token count on each one. That count then multiplies the per-token price, so two things vary at once: how many tokens your text becomes, and what each token costs.
Most pricing pages only show the second number. This post measures the first. We sent three semantically aligned passages to seven models (claude-fable-5, claude-opus-4-8, claude-sonnet-5, deepseek-v4-flash, glm-5.2, gpt-5.5, kimi-k2.5) and read back what each one billed as input.
A casual narrative (a Saturday market story) exists in nine languages; a technical explanation (retry with exponential backoff) and a news brief (a city-budget vote) exist in English, Chinese, Japanese, Korean, German, and Hindi. A Python function and a JSON tool-call blob round out the set. The non-English versions are machine translations produced under a faithful, no-compression instruction and spot-checked by hand; translation verbosity is a real confound, and the register section below bounds it at roughly 20%.
Counting is always the provider's own: the Claude line via a real Messages call reading usage.input_tokens (the gateway does not currently proxy count_tokens), OpenAI-compatible models via a small call reading usage.prompt_tokens. A local tokenizer that disagrees with the invoice is the exact failure this avoids. One control matters: every request carries fixed framing (chat template, role markers) worth a handful of tokens, so we measure a two-character baseline sample and subtract it. Every ratio in this post is net of that envelope and compares the text, not the framing.
The same text, five tokenizers
Here is the raw input-token count for the narrative passage, per language, per tokenizer; the three Claude models share a column because they returned identical counts on every sample (more on that below). The other two passages reproduce the same pattern and are folded in further down. The character column is the length of that language's version; scripts pack meaning differently, so Chinese says the same thing in 77 characters that English needs 254 for.
| language | chars | fable-5 / opus-4-8 / sonnet-5 | deepseek-v4 | glm-5.2 | gpt-5.5 | kimi-k2.5 |
|---|---|---|---|---|---|---|
| en | 254 | 90 | 55 | 63 | 57 | 60 |
| zh | 77 | 96 | 50 | 58 | 69 | 50 |
| ja | 136 | 136 | 101 | 116 | 114 | 129 |
| ko | 143 | 160 | 104 | 123 | 93 | 129 |
| hi | 196 | 147 | 124 | 192 | 76 | 133 |
| de | 289 | 146 | 92 | 92 | 75 | 104 |
| fr | 259 | 111 | 76 | 79 | 66 | 93 |
| es | 253 | 112 | 75 | 79 | 66 | 91 |
| it | 272 | 127 | 84 | 91 | 78 | 100 |
Two facts jump out. The Claude column is one number for three models because Claude Fable 5, Opus 4.8, and Sonnet 5 returned identical counts on every sample, languages, code, and JSON alike: all three carry the tokenizer introduced with Opus 4.7, so a count on one is a count on all of them. And that column is the largest in every row except Hindi, where GLM's 192 is worse. Normalized on net counts so the leanest model in each language reads 1.00 (the envelope is subtracted first, so these ratios won't match a division of the raw cells above):
| language | fable-5 / opus-4-8 / sonnet-5 | deepseek-v4 | glm-5.2 | gpt-5.5 | kimi-k2.5 |
|---|---|---|---|---|---|
| en | 1.64 | 1.00 | 1.00 | 1.00 | 1.00 |
| zh | 2.20 | 1.12 | 1.12 | 1.55 | 1.00 |
| ja | 1.33 | 1.00 | 1.07 | 1.11 | 1.24 |
| ko | 1.77 | 1.15 | 1.28 | 1.00 | 1.38 |
| hi | 2.01 | 1.72 | 2.59 | 1.00 | 1.78 |
| de | 2.03 | 1.28 | 1.16 | 1.00 | 1.38 |
| fr | 1.75 | 1.20 | 1.12 | 1.00 | 1.41 |
| es | 1.76 | 1.19 | 1.12 | 1.00 | 1.37 |
| it | 1.68 | 1.11 | 1.10 | 1.00 | 1.27 |
The English row's four-way tie is not rounding: DeepSeek, GLM, GPT-5.5, and Kimi all land on exactly 50 net tokens for that passage. Claude runs 1.3x to 2.2x the leanest tokenizer on this passage, 1.2x to 2.3x across all three, a property of the vocabulary that applies to every call for the life of the model. The technical and news passages reproduce the ranking: summing both, Claude bills Chinese at 212 net tokens against Kimi's 114 (1.9x) and Hindi at 477 against GPT-5.5's 210 (2.3x). But there is no single winner underneath it. The leanest column moves as the language changes:
- GPT-5.5 is leanest on German, French, Spanish, Italian, Hindi, and Korean, and ties for leanest on English (the tie and fr/es/it hold on the narrative only). Its vocabulary is tuned toward Latin scripts and it holds up on Devanagari and Hangul.
- Kimi K2.5 is leanest on Chinese and competitive across CJK.
- DeepSeek-v4 is leanest on Japanese and close behind on Chinese.
- GLM 5.2 is middle-of-pack on most languages but posts the worst cells in the matrix on Hindi: 2.59x the leanest on the narrative (179 net tokens where GPT-5.5 needs 69), worse still on the formal passages, and the one column that exceeds even Claude.
The penalty is not limited to prose. On the Python function Claude runs 1.61x the leanest, and on the JSON tool-call 1.29x. The JSON gap is narrower because structured text is mostly punctuation and short ASCII keys that every tokenizer handles similarly. For a long-running agent that replays a large tool schema every turn, that per-turn tax compounds, which is where caching earns its keep. The prompt-caching series covers those mechanics.
The per-character trap: CJK looks worse than it bills
The tables above compared models. Hold the model fixed instead, and language still moves the count, but not the way the raw character view suggests. The most-quoted tokenizer number is tokens per character, and CJK dominates it: on Claude, Chinese runs about 114 net tokens per 100 characters, Korean 106, Japanese 94, against 32 for English. Read that column alone and CJK looks like a 3x tax, but it is the wrong column: you pay for meaning, not characters, and the aligned passage carries the same meaning in every language. Here are both views on Claude, for the narrative passage:
| language | chars | net tokens | tokens / 100 chars | tokens vs English |
|---|---|---|---|---|
| en | 254 | 82 | 32 | 1.00 |
| zh | 77 | 88 | 114 | 1.07 |
| ko | 143 | 152 | 106 | 1.85 |
| ja | 136 | 128 | 94 | 1.56 |
| hi | 196 | 139 | 71 | 1.70 |
| de | 289 | 138 | 48 | 1.68 |
| it | 272 | 119 | 44 | 1.45 |
| es | 253 | 104 | 41 | 1.27 |
| fr | 259 | 103 | 40 | 1.26 |
The two right-hand columns disagree, and Chinese is the sharpest case: the highest per-character density in the set, yet a per-meaning premium of only 1.07x English on this passage. 77 characters carry what English spends 254 on, so a steep per-character rate multiplies a tiny character count and nearly cancels out. Across all three passages the cancellation holds but is not magic: Chinese averages 1.17x Claude's own English, and 0.95x to 1.32x depending on the model, near parity rather than the 3x the per-character column threatens.
Japanese and Korean answer the obvious next question: same illusion, weaker cancellation. Both share the high per-character density, because Hangul and Japanese kana spell sounds out roughly one glyph per syllable rather than packing a whole word into each character the way Chinese hanzi do. So Korean needs 143 characters and Japanese 136 for the passage Chinese says in 77. More characters times a high per-character rate does not cancel, it compounds: on Claude, Korean averages 1.96x English per meaning across the three passages and Japanese 1.56x, both genuinely expensive, even though their per-character column looks a lot like Chinese's.
German is the mirror image of Chinese: a low per-character rate (48, near English) but the most characters of any language here (289, its compound words), which still totals 1.68x. Cost is the product of the two axes, and either one read alone misleads.
Why the numbers move: two factors, multiplied
The rule underneath every table above is one equation:
tokens for a passage = (characters to express the meaning) x (tokens per character)
The first factor is the writing system's density, a property of the language, not the model. It is a spectrum, not a Chinese exception. Logographic Chinese packs a morpheme into each character and sits at the dense extreme. Japanese kana and Korean Hangul spell sounds out, so they are less dense and need more characters. Devanagari and the Latin alphabets are less dense still. Meaning-per-character falls steadily from Chinese to English.
The second factor is how many tokens the model's vocabulary spends per character of that script, and it is entirely model-specific. A BPE tokenizer learns multi-character merges from its training corpus: scripts it saw often get compact tokens, scripts it saw rarely fall back toward character-by-character or even byte-level encoding, where one character can become two or three tokens. The same three languages, net tokens per character:
| tokens per character | Chinese | Hindi | English |
|---|---|---|---|
| Claude | 1.14 | 0.71 | 0.32 |
| DeepSeek | 0.58 | 0.61 | 0.20 |
| GPT-5.5 | 0.81 | 0.35 | 0.20 |
| GLM 5.2 | 0.58 | 0.91 | 0.20 |
| Kimi K2.5 | 0.52 | 0.63 | 0.20 |
The table explains three things. Chinese looks special in the totals because it is extreme on the first factor: even Claude's weak Chinese compression (1.14 tokens per character, still splitting some hanzi in two) can't make the total large when there are only 77 characters, and the China-trained models compress it well enough (0.52 to 0.58) to land near parity with their own English. Hindi's premium is the second factor, not density: GLM spends 0.91 tokens per Devanagari character, nearly one token per character because its vocabulary has almost no multi-character Devanagari merges, while GPT-5.5 spends 0.35 by covering whole syllable clusters, a coverage gap on the same script. And Claude is high everywhere because its per-character rate is high even on English (0.32 versus DeepSeek's 0.20), a model-level baseline that stacks on top of whatever the language does.
None of this is a quirk of our seven models. The research literature calls the phenomenon the token premium, and Petrov et al. (NeurIPS 2023) measured it across hundreds of language pairs, finding the same two root causes (character counts differ per meaning, and tokenizer coverage differs per script) and premiums up to 15x for low-resource languages, with the same consequences: higher cost, higher latency, and less usable context window, since a high-premium language fills the same context budget with less meaning. The gap also narrows as vendors invest: independent measurements put Chinese at +182% tokens versus English on GPT-3-era vocabularies and +24% on GPT-4o's, close to the +32% we measure on GPT-5.5 and the parity we measure on the China-trained models. Coverage is bought with vocabulary slots, and vendors keep buying.
Does localizing ever save money?
It is tempting to read the trap section as "Claude is flat across languages, so ignore localization" and "China models are cheap on Chinese, so localizing saves money." Both are wrong. Here is each language against that model's own English, averaged over the three passages, for the five languages that have all three:
| vs own English | zh | de | hi | ja | ko |
|---|---|---|---|---|---|
| Claude | 1.17 | 2.11 | 2.40 | 1.56 | 1.96 |
| DeepSeek | 1.00 | 1.94 | 3.11 | 1.85 | 1.99 |
| GLM 5.2 | 1.03 | 1.77 | 4.89 | 2.03 | 2.31 |
| GPT-5.5 | 1.32 | 1.53 | 1.70 | 2.09 | 1.72 |
| Kimi K2.5 | 0.95 | 2.20 | 3.15 | 2.18 | 2.41 |
Claude is not flat: Korean costs 1.96x its English, Hindi 2.40x. Chinese sitting near 1.17x is a one-language coincidence, not a property of the model. And the China models do not beat parity on Chinese so much as reach it: the best cell in the whole table is Kimi's 0.95x, five percent below its own English, and every other cell costs the same or more. On Hindi, Japanese, and Korean those same models carry a bigger penalty than Claude does, not a smaller one, because those scripts are further from their training focus. The pattern is not "vendor X is cheap"; it is that every model is leanest, relative to its own English, on the languages nearest its training data.
Register moves these numbers too. The casual narrative is the friendliest case; the technical and news passages raise nearly every multiplier, because terminology and loanwords are exactly what non-Latin vocabularies lack merges for. German goes from 1.68x (casual) to 2.29x (technical) on Claude, and GLM's Hindi reaches 5.98x its English on the news passage. A one-passage benchmark flatters whichever language got the friendliest translation, which is why the single narrative had Chinese at 0.80x on Kimi and three passages put it at 0.95x.
Relative-to-own-English is the wrong lens anyway. What you pay is absolute tokens, and there Claude is the most expensive in eight of the nine languages, with GLM's Hindi the one column that is worse. Chinese content that is "cheap relative to Claude's English" is still 88 net tokens on Claude against 40 on Kimi for the narrative. So the move is not to localize for savings but to match the model to the language: Chinese to Kimi or DeepSeek, Hindi and Korean to GPT-5.5, Japanese to DeepSeek. Claude is never the token-cost winner in any language, though it may still win on quality.
Counts are only half the bill
A token multiplier only matters next to the per-token price, and the two compound. Claude Fable 5 lists at $10 per million input tokens, Opus 4.8 at $5, and Sonnet 5 at $3 once its introductory pricing ends; on Chinese their shared tokenizer also counts the text at 2.2x the leanest model, so the count premium multiplies whatever rate gap the model already carries over the alternative you would route to. The opposite can happen too: a model can count leanly and still cost more per call because its rate is high. Neither number alone tells you the bill. We don't print the other vendors' rates here because they change faster than tokenizers do; the counts above are the durable half of the calculation.
The practical move is to stop comparing sticker prices and compare effective input cost: your real traffic mix, counted on each candidate model, multiplied by that model's input rate. On a Chinese-heavy or Korean-heavy product that reordering can flip which model is cheapest, and the gap is a durable 1.5x to 2x, not a rounding error. It is the same reason the number that matters when you cache is effective cost weighted by hit rate, not the headline rate, which the provider comparison works through. The version-to-version cut of this story, why Sonnet 5 counts 41% more than Sonnet 4.6 on the same English, is in the Sonnet 5 tokenizer post.
Bottom line
- Token cost is script density times tokenizer coverage. The language sets the first factor, the model sets the second, and either one read alone misleads.
- Claude Fable 5, Opus 4.8, and Sonnet 5 are 1.2x to 2.3x the leanest in every language, because their per-character rate is high even on English.
- The leanest model is language-specific: GPT-5.5 for European languages, Hindi and Korean, Kimi for Chinese, DeepSeek for Japanese. GLM is weakest on Hindi, at nearly one token per character.
- Formal and technical register raises the multiplier in nearly every language; benchmark in the register you actually ship.
- Don't localize for savings; match the model to the language on absolute tokens, then multiply by each model's rate to compare effective cost.
FAQ
Which LLM tokenizer is cheapest?
It depends on the language. Across seven models on the same aligned passages, GPT-5.5 was leanest on the European languages, Hindi and Korean (tied on English), Kimi K2.5 on Chinese, and DeepSeek-v4 on Japanese. The Claude family (Fable 5, Opus 4.8, Sonnet 5) was never the leanest, running 1.2x to 2.3x the cheapest count in every language and register.
Do Claude Fable 5, Opus 4.8, and Sonnet 5 use the same tokenizer?
Yes. All three produced identical token counts on every sample, in every language, code, and JSON. They carry the tokenizer introduced with Opus 4.7, so a count on one transfers to the others, and Fable 5's higher bill comes entirely from its per-token price.
Is Chinese more expensive than English on Claude?
Slightly: 1.17x per meaning, averaged over three passages (and roughly parity on the China-trained models). Per character it looks far worse (about 114 net tokens per 100 Chinese characters versus 32 for English), but Chinese conveys the same meaning in about a third of the characters, so the totals nearly cancel.
Do Japanese and Korean behave like Chinese?
Only halfway. They share Chinese's high per-character density, but Hangul and kana spell sounds out, so they need far more characters for the same passage (Japanese 136, Korean 143, versus Chinese's 77). The high per-character rate no longer cancels, so per meaning Japanese runs about 1.6x English on Claude and Korean about 2x, and 1.5x to 2.4x across the seven models.
How do I measure this for my own prompts?
Send a few real prompts, in the register you actually ship, to each candidate model and read the provider's own input-token count from the usage fields, rather than trusting a local tokenizer. One friendly passage can flatter a language by about 20%, so use several. Then multiply each count by that model's input price to get effective cost on your traffic.
Top comments (0)