When a model card reports multilingual performance, it is reporting on a set of languages chosen by whoever built the benchmark. That set is small, its membership is stable across benchmarks, and the reason both things are true is not a mystery.
The denominator
Ethnologue, the reference catalogue maintained by SIL International, lists on the order of seven thousand living languages — the exact count moves each edition as languages are reclassified, split, merged or recorded as no longer spoken (Ethnologue). The ISO 639-3 registry, which is the identifier scheme most datasets use, assigns codes to a substantially larger set including extinct and constructed languages.
Two complications are worth naming before doing any arithmetic. Many of those languages have no written tradition, so a text benchmark is not merely missing for them — it is not a meaningful object. And many “languages” in the list are dialect continua whose division into discrete entries is a judgement call. So “seven thousand” is the right order of magnitude and the wrong kind of precision.
Ethnologue’s living-language count changes with every annual edition and is behind a paywall for detail. Quote it as “roughly seven thousand” and cite the edition if you need a specific figure.
What the big benchmarks actually cover
The public multilingual benchmarks are countable, so count them. Each of these is the primary source for its own language list:
- FLORES-200 — 200 languages, machine translation, published by Meta AI with the NLLB work in 2022. The broadest widely used language coverage of any benchmark (NLLB).
- Belebele — reading comprehension across 122 language variants, Meta AI, 2023 (Belebele).
- XTREME — 40 languages across nine tasks, Google Research, 2020 (XTREME).
- XNLI — natural language inference in 15 languages, 2018, still one of the most reported cross-lingual numbers (XNLI).
- MMLU — the general-knowledge benchmark most model cards lead with, published in English only (MMLU). Translated variants exist and are separate datasets with their own language lists.
The gap, worked out
The arithmetic is trivial and the assumptions are the interesting part, so both are stated:
Assumptions
- Living languages (Ethnologue, order of magnitude) ~7,000
- Broadest text benchmark language count (FLORES-200) 200
- Cross-lingual reasoning benchmark (XTREME) 40
- Benchmark used for headline capability claims (MMLU) 1
Coverage
200 / 7,000 = 2.9% have any broad translation benchmark
40 / 7,000 = 0.6% have a multi-task cross-lingual benchmark
1 / 7,000 = 0.014% have the benchmark the headline number uses
Three caveats on those numbers, all of which cut in the same direction. The 7,000 denominator includes languages without writing systems, which inflates the apparent gap for text benchmarks. But the 200 numerator counts a language as covered if it has a translation test set, which is a very low bar for “supported”. And benchmark language lists overlap heavily — the 40 in XTREME are largely a subset of the 200 — so the union across all benchmarks is closer to the largest single list than to their sum.
The second gap: coverage is not depth
Counting languages hides the more consequential asymmetry. English is evaluated on reasoning, code, mathematics, factual recall, instruction following, long context, tool use, refusal behaviour and a dozen safety axes. A language in the FLORES-200 list is evaluated on whether a sentence can be translated into it.
Those are not the same claim, and treating them as one is how “the model supports 200 languages” ends up on a marketing page. A translation score tells you the model can map meaning into the language’s surface forms. It tells you nothing about whether the model reasons correctly when prompted in that language, whether its safety training transferred, or whether it hallucinates more — which, as the argument about low-resource hallucination sets out, it very likely does. For most of the world’s languages those questions have no published answer at all.
There is a third layer beneath even that, which is whether a translated benchmark measures what its English original measured. Take an English multiple-choice knowledge test and render it into forty languages, and you have built forty tests whose difficulty now depends on translation quality, whose subject matter is still American secondary and undergraduate curriculum, and whose distractors may no longer be plausible in the target culture. A model can score well on the translated version by being good at English knowledge and adequate at translation, which is not the capability the number is being read as. Benchmarks written natively by speakers avoid this and are far rarer, because writing one is many times the work of translating one.
And coverage in text says nothing about the other modalities. Speech recognition, text-to-speech and document OCR each have their own, shorter language lists, and a language can be present in a translation benchmark while having no evaluation data at all for the modality your product actually uses. If you are ingesting scanned documents or audio, the relevant coverage question is about that pipeline, not about the model’s language list.
Why it stays this way
Not indifference, mostly economics and a feedback loop. A benchmark needs items written or validated by fluent speakers who also understand the annotation task, which is a recruitment problem that gets sharply harder as speaker populations shrink and as the required literacy is in a language that is not the language of local higher education. A reasoning benchmark is harder still, because the items must be culturally sensible in the target language rather than translated American trivia — a translated benchmark tests translation as much as it tests reasoning.
Then the loop closes: labs optimise for what is measured, so covered languages improve, so covered languages look like the ones worth covering. Breaking it has required deliberate community effort — the Masakhane collective’s participatory work is the clearest example, and it produced datasets for African languages that no commercial roadmap was going to produce (Nekoto et al., 2020).
What to do instead of citing a leaderboard
If your language is not in a benchmark, the honest position is that you have no external evidence and must produce your own. That is less daunting than it sounds: a few hundred domain sentences with reference outputs, scored by a speaker, will tell you more about your actual workload than any published number would have. The procedure is in building for a language with no digital corpus. And if your language is in a benchmark, check which one: “present in FLORES-200” and “evaluated for reasoning” are separated by most of the work.
Producing your own numbers means running the same eval set against several models and re-running it whenever one is deprecated, which is mostly an integration chore rather than an evaluation one. Multigrid gives one API and one key across providers with per-request cost tracking, so the eval harness changes a model identifier rather than a client.
Top comments (0)