If a model claims to support your language, the natural next question is how anyone knows. For most of the world’s languages the answer is that nobody does, because there is no evaluation set to measure against — and that absence is measurable from the benchmarks’ own published language lists.
What each suite actually covers
These are the language counts the suites themselves publish, with the paper and year attached. Every number here comes from the source document rather than from a summary.
- FLORES-200 — released with Meta’s No Language Left Behind work in 2022, covering a little over 200 language varieties. This is the broadest of the standard sets, and it is a translation benchmark.
- Belebele — a reading-comprehension benchmark published by Meta in 2023, built over the FLORES passages, covering 122 language variants. It is the widest genuinely task-shaped multilingual set.
- XTREME — a cross-lingual transfer suite published in 2020, covering 40 languages across nine tasks.
- XNLI — published at EMNLP 2018, 15 languages of natural language inference. Still one of the most frequently reported multilingual numbers.
- MGSM — from the 2022 paper on multilingual chain-of-thought reasoning, which translated 250 grade-school maths problems into ten languages besides English.
These counts move upward as suites publish extensions and as translations of English benchmarks appear. Re-check the source papers rather than trusting this list after a year; the ordering of magnitude is what is durable, not the exact figures.
Deriving the gap
Set those against the denominator, which is the same one used on languages missing from training corpora: Ethnologue’s 26th edition (2023) counted 7,164 living languages. The broadest standard benchmark covers a little over 200 varieties. That is under three per cent of living languages with any standard evaluation set at all, and the derivation is just the division:
204 varieties in the broadest suite
/ 7,164 living languages (Ethnologue 26th ed., 2023)
= roughly 2.8%
Assumptions, stated:
- "varieties" and "languages" are not the same unit; FLORES counts
script variants separately, which inflates the numerator.
- Many of the 7,164 have no written form and so cannot appear in a
text benchmark at all, which deflates the denominator.
- Both adjustments push the true figure in the same direction as the
headline: coverage is a small single-digit percentage either way.
The honest statement is therefore not “exactly 2.8 per cent” but “a low single-digit percentage of living languages have a standard benchmark, on any reasonable accounting”. That is a claim the arithmetic supports and that no revision of either number will overturn.
Translation coverage is not task coverage
The subtler point, and the one that changes what you should conclude, is that the wide benchmarks measure a narrow thing. FLORES-200 measures translation quality. It does not tell you whether a model can follow an instruction in that language, reason in it, refuse appropriately in it, extract structured data from it, or hold a coherent multi-turn conversation in it.
The suites that do measure those things are dramatically narrower. Task benchmarks in the fifteen-to-forty language range, and reasoning benchmarks in the ten-language range, are the ceiling. So for a language in FLORES-200 but not in the task suites — which is the large majority of the 200 — the available evidence is that a model can translate it, and nothing more. A published claim of “support for 100+ languages” is very often a translation-shaped claim being read as a capability-shaped one.
This is also why benchmark absence and low-resource hallucination are the same story told twice. The absence of an evaluation set is not only a missing report card — it is a missing feedback signal during development. Nothing pushed the model toward correctness in that language, and nothing would have caught it if it drifted.
Why a benchmark does not simply get built
The obstacle is not compute and it is rarely goodwill. Building a benchmark requires fluent annotators who are also comfortable with the annotation task, a written standard everyone agrees on, and a licence permitting redistribution. Each of those fails independently.
- Annotator supply. A benchmark needs multiple fluent annotators per item for agreement scoring. For a language with few literate speakers in the professional annotation market, that pool may not exist at any price.
- Orthographic instability. Where spelling conventions are still settling, or where a language is written in two scripts, annotators disagree about the reference text itself, and low inter-annotator agreement is indistinguishable from a bad benchmark.
- Translation artefacts. The cheap route — translating an English benchmark — produces test items carrying English discourse structure and English cultural assumptions. A model can score well on translated MMLU-style items partly by recognising translationese, which is not the capability anyone wanted to measure.
- No institutional owner. Benchmarks need maintenance, hosting and a contamination policy. Without a lab or a university group that owns it, a released set decays into a broken download link within a few years.
Evaluating a language nobody has benchmarked
The practical answer is that you build a small one, and it is a smaller job than it sounds. A hundred items drawn from your own domain, written natively rather than translated, with expected outputs agreed by two fluent speakers, will tell you more about whether a model works for your product than any public leaderboard would. Fix the prompt, run the set across candidate models, and score against the agreed outputs.
Two design points make the difference between a useful set and a misleading one. Write the items in the language rather than translating them, so you measure the language rather than translationese. And keep the set private, since anything published becomes training data and its value decays. That is also why a private set can outlive a public leaderboard for your purposes — the contamination clock never starts.
Scoring is the part people abandon, so make it cheap. Exact-match items — extraction into a fixed schema, classification into a fixed label set, a numeric answer — score themselves and are worth over-weighting for that reason alone, even if they are not the whole of what your product does. For open-ended output, a fluent speaker reviewing fifty items takes an hour or two, and doing that once per candidate model is a smaller commitment than most teams assume. Using a model to judge output in a language where models are weak is the one shortcut to avoid: the judge inherits the same gaps as the model it is grading, so a low-resource language is exactly where automated judging is least trustworthy and most tempting.
Running the same hundred-item set across several candidate models is mostly a plumbing problem: different SDKs, different auth, different response shapes, one script per provider. A gateway that puts every model behind one API and one key — which is what Multigrid is — turns that into a loop over a list of model ids, which is worth knowing about mainly because it removes the excuse not to evaluate.
Top comments (0)