One Ruler to Measure Them All: How Language Affects LLM Quality
Most discussions about LLM performance focus on the model architecture and prompting. But there's a hidden factor: the tokenizer. It determines how much of your text fits in the context window.
The Tokenizer Problem
Russian text consumes more tokens than English for the same information density. Some developers even switch to English prompts to save tokens and improve performance.
The Surprising Result
A recent arxiv study benchmarked multilingual long-context language models across different languages. The winner? Polish — 88% accuracy.
Russian placed 5th at 84% — ahead of English at 83.9%.
The gap widens on long-context tasks. More tokens = more opportunities for the model to lose coherence.
Important Caveat
The test used "weaker" models by 2026 standards:
- Gemini 1.5 Flash
- Qwen 2.5 72B
- Other mid-tier models
Top-tier models might show different patterns, but the tokenizer effect persists regardless of model quality.
Implications for Production
- Language choice matters for RAG. If your knowledge base is multilingual, retrieval quality varies by language.
- Long-context tasks favor compact languages. English is more token-efficient than Russian, but Polish outperformed both.
- Tokenizer-agnostic metrics are needed. BLEU and ROUGE don't capture tokenization bias.
What I'm Tracking
I'm monitoring whether newer models (Kimi k2.5, GLM-5, GPT-5.2 series) show the same pattern. Early signs suggest top-tier models compress better across languages, but the gap doesn't fully disappear.
More multilingual LLM analysis and production AI notes from inside a bank — follow my Telegram channel:
https://t.me/ai_tablet (Russian, technical)
More AI engineering notes, RAG benchmarks, and production insights from inside a bank — follow my Telegram channel:
🚀 https://t.me/ai_tablet (Russian, technical)
Top comments (0)