Most writing about large language models assumes English. The benchmarks are in English, the failure examples are in English, and the quiet implication is that whatever is true in English is true everywhere else, just a little worse. After spending a long stretch evaluating model output across English, Hindi, Tamil, and Marathi, I can say that assumption is wrong in ways that actually matter — and the differences taught me more about how these models work than any English-only evaluation did.
This is an opinion piece, not a benchmark. But the opinions come from sitting with thousands of real outputs in four languages, so here is what stuck.
The quality gap is not uniform, and it is not where you would guess
The naive mental model is a single dial: English is best, and every other language is some fixed amount worse. Reality is lumpier than that. A model can be genuinely strong at conversational Hindi and fall apart on Tamil technical vocabulary. It can produce fluent Marathi that is subtly wrong about facts, while producing slightly awkward English that is factually correct. The gap depends on the interaction of language and task, not on the language alone — and you only see that if you evaluate the same task across all four rather than testing each language on whatever happens to be convenient.
Fluency hides errors, and it hides them worse in some languages
The most dangerous output is the one that reads beautifully and is wrong. In English, reviewers have enough exposure to model quirks that a confident-but-wrong answer often still trips an alarm. In a language where fluent model output is rarer and more impressive, that same confident tone is far more likely to be taken at face value. I caught myself doing it: a well-formed Tamil paragraph feels authoritative in a way a clunky one does not, and that feeling has nothing to do with whether the content is correct. Evaluating across languages made me distrust fluency as a signal entirely, which improved my English evaluation too.
Script and tokenization problems are real and easy to miss
Some of the most concrete failures had nothing to do with meaning. A model would drop or mangle a diacritic, mix two scripts in one word, or transliterate when it should have translated. These are the kind of errors a monolingual English reviewer never encounters and never thinks to check for, and an automated exact-match check often waves them through because the answer is "close enough" at the character level while being clearly wrong to a native reader. If you are building evaluation tooling for multilingual output, normalization and script handling deserve as much care as the semantic checks — maybe more.
Code-switching is where models are most human and most fragile
Real speakers of these languages mix in English words constantly — technical terms, brand names, whole clauses. Sometimes the models handle this gracefully, which is genuinely impressive. Other times a single English word in a Hindi sentence throws off the grammar of everything around it, or the model over-corrects and translates a term that should have been left in English. This is exactly the kind of edge case that never shows up in a clean, single-language benchmark, and it is where the difference between "sounds fluent" and "is actually correct" is widest.
Why this matters beyond the four languages I worked in
The practical lesson is not about Hindi or Tamil or Marathi specifically. It is that evaluating a model in one language tells you how it performs in that language, and almost nothing reliable about the rest. If your product will be used in more than one language — and most will be — then single-language evaluation is not a smaller version of the real thing; it is measuring something different from what you actually ship. Building the habit of testing the same task across several languages, and of not trusting fluency as a proxy for correctness, changed how I read every model output, English included.
The models are getting better at all of this quickly. But "better on average" can still hide a specific language or a specific kind of task where the model quietly fails, and averages are exactly what single-language evaluation is built to produce. The only way I found to see the gaps was to look for them in more than one language on purpose.
Top comments (0)