We Asked 330 Models a Question in Korean. Half of Them Answered in the Wrong Alphabet.

We graded 330 language models on Korean across seven axes. Before any of that, a four-line function
threw out a third of the answers.

That function turned out to be the most useful thing in the harness.

The check

def contamination(text: str) -> dict:
    total = len(text.strip())
    if not total:
        return {"hangul_ratio": 0.0, "hanja": 0, "kana": 0, "broken": True}
    h  = len(HANGUL.findall(text))
    hj = len(HANJA.findall(text))
    kn = len(KANA.findall(text))
    return {"hangul_ratio": round(h / total, 3), "hanja": hj, "kana": kn,
            # not Korean enough, or another script leaked in
            "broken": h / total < 0.25 or hj > 3 or kn > 0}

Three conditions, all countable, no model in the loop:

• less than 25% Hangul — the answer is not really in Korean
• more than 3 Hanja — Chinese characters bleeding in
• any kana at all — Japanese characters bleeding in

If broken, the answer is an F and we never pay a judge to read it.

What it caught

Answers auto-failed	766 of 2,304 — 33.2%
Models with at least one	171 of 330 — 51.8%
Models that failed all seven	54 — 16.4%

Fifty-four models did not produce one clean Korean answer out of seven attempts.

The failure is not evenly spread across tasks:

Task	Models auto-failed
Honorifics	142
Terminology	125
Korean institutions	113
Summarisation	105
Naturalness	99
Register	93
Format compliance	89

The hardest linguistic task is also where script discipline collapses first.

Why a mechanical gate and not the judge

An LLM judge will happily grade a Chinese-Korean hybrid on its register. It will produce a
plausible sentence about tone while the answer is not in the requested language at all. The grade
comes back looking like data.

So we kept a rule for this build of the harness: anything countable is counted, not judged.
Script mix is countable. Tone is not. The judge only sees answers that already passed the count.

This also cuts cost — 766 fewer judge calls — but that is a side effect. The point is that the two
failure modes are different and must not be averaged into one grade.

The harness refuses to start if it cannot tell good from bad

Before measuring anything, the script runs a fixed set of control answers with known expected
grades, and aborts if it does not score them perfectly:

rate = hits / len(CONTROLS)
print("채점기 정확도 %d/%d = %.0f%%" % (hits, len(CONTROLS), rate * 100))
#      ^ "judge accuracy" — verbatim from our source
if rate < 1.0:
    return False        # do not begin the real measurement

An A-expected control must come back A. An F-expected control must come back F. If the judge cannot
separate the two on cases we already know the answer to, the numbers it produces on unknown cases
are not worth having.

Two more things we do to the judge, both cheap:

• The judge never sees the model name. Self-preference is real and free to remove.
• The rubric says length is irrelevant, because judges reward long answers if you let them.

The grades, once the surviving answers are graded

330 models, 8 bands:

A+++   5      B+   55      D   29
A++    2      B    58      F  111
A+    18      C    52

7.6% reach any A band. 33.6% are F.

Per-axis A rates, and this is the part that surprised us:

Axis	A rate
Honorifics	8.5%
Knowledge of Korean institutions	9.4%
Terminology	31.2%
Format compliance	47.6%
Summary accuracy	49.2%
Freedom from translationese	53.1%
Register	53.3%

Two axes sit at roughly a tenth of the field while the rest cluster near half.

Vendor averages

Mean score, 0–3, for vendors with at least eight models measured:

Vendor	n	Mean
anthropic	14	2.29
mistralai	18	2.07
openai	51	1.96
qwen	51	1.48
google	27	1.48
deepseek	15	1.33
meta-llama	8	1.16
nvidia	10	0.93
minimax	8	0.70
z-ai	15	0.36

One prompt set, one judge, one run. Read it as a ranking of this measurement, not a verdict on the
vendors.

Recency and size predict nothing

Five models hold a perfect 3.00:

openai/gpt-5.4
openai/gpt-5.4-mini
openai/gpt-4o-2024-05-13
openai/gpt-3.5-turbo-16k      ← 2023
google/gemini-3.1-flash-image

A 2023 model sits with the 2026 flagships. And the 28 models that earn an A on honorifics include
gpt-4o-mini, qwen-2.5-72b-instruct, llama-3.1-70b-instruct — mid-size models, not the tops of
anyone's leaderboard.

If you are picking a model for a Korean-facing product, release date, parameter count and English
benchmark position are not proxies. They carry no signal on this axis.

Run it yourself

Open API, no key:

curl .../api/korean    # every model, every axis, with the judge's reason string
curl .../api/summary   # grade distribution, judge id, what the run cost

https://huggingface.co/spaces/ginigen-ai/open-router-leaderboard

426 models, 330 graded, refreshed daily. The judge id and the per-answer reason strings are in the
payload, so you can disagree with a specific grade rather than with the average.

The portable part

Whatever language you ship into, write the countable check first and let it fail answers before your
judge sees them. Ours is four lines and it disqualified a third of the corpus.

A judge asked to grade an answer that is in the wrong script will still return a grade. That grade
will look exactly like the real ones in your CSV.

---

Methodology: https://huggingface.co/blog/ginigen-ai/openrouter-leaderboard