Korean-Language AI Quality Leaderboard: 426 OpenRouter Models Benchmarked Across 7 Korean-Specific Dimensions

TL;DR: GeniGen AI has published a Korean-language quality leaderboard on Hugging Face, evaluating 330 of the 426 models available on OpenRouter across seven Korean-specific linguistic dimensions — from honorific accuracy to terminology consistency. Only 7.6% of models scored at Grade A or above, revealing a stark gap between English-benchmark performance and real-world Korean capability. VIDRAFT's Pocket-35B holds the #1 domestic download rank on the companion global download leaderboard.

What it is

GeniGen AI (지니젠AI), a Korean AI company, has released a publicly accessible Korean-language AI quality leaderboard hosted on Hugging Face. The dataset covers 426 models listed on OpenRouter — the multi-provider AI model routing platform that aggregates offerings from providers including Anthropic, Google, OpenAI, xAI, and DeepSeek under a single API.

Of those 426 models, 330 were fully evaluated across seven Korean-specific quality dimensions. The leaderboard is updated automatically at 08:00 KST every day and supports Korean, English, and Chinese display languages, as well as KRW, USD, and CNY pricing views.

The project is structured as a strategic collaboration between GeniGen AI and VIDRAFT (비드래프트): GeniGen AI owns the Korean-language quality evaluation track, while VIDRAFT maintains a companion global download leaderboard. The two tracks together give developers a dual-axis view — usage popularity and linguistic quality — for selecting models appropriate for Korean-language workloads.

How it works

The evaluation methodology is designed around transparency and reproducibility:

• Seven evaluation dimensions: Honorific accuracy (높임말), Korean institutional and regulatory knowledge, domain-specific terminology, instruction-following, summarization quality, avoidance of translation-artifact phrasing (번역투 배제), and stylistic consistency.
• Blind evaluation: To control for bias, the identity of both the evaluating system and the models being evaluated are concealed during the scoring process.
• Direct API measurement: Response latency is measured by calling each model's paid API endpoint directly, rather than relying on self-reported or estimated figures.
• No imputed data: Where measurement data is unavailable, fields are left blank rather than filled with estimates — a notable methodological choice given how common imputation is in leaderboard pipelines.
• Provider-level comparison: Because the same underlying model can be served by up to 107 different providers on OpenRouter, the leaderboard surfaces per-provider pricing, precision, and uptime alongside model-level scores.

The leaderboard data is also accessible via a public API for downstream consumption, enabling developers to build tooling or monitoring pipelines on top of the evaluation results.

Benchmarks & results

The headline findings from the September 2026 evaluation cycle:

• Grade A or above: Only 7.6% of evaluated models (25 out of 330) reached this tier.
• Grade F: A significant 33.6% (111 models) scored at the lowest grade.
• Perfect scorers: Five models achieved maximum scores across all seven dimensions — and notably, at least one of these was a model released as far back as 2023. This directly challenges the assumption that newer models or higher English-benchmark scores predict Korean-language quality.
• Common failure mode: Even models that produced grammatically natural-sounding Korean frequently made errors in honorific register and in domain-accurate details such as annual leave calculations or itemized administrative information.
• Provider pricing variance: For the same model, pricing across providers varied by a median factor of 1.87× and up to a maximum of 14.47×. Precision and uptime also differed meaningfully between providers serving the same underlying model.
• VIDRAFT Pocket-35B download rank: As of September 17, 2026, VIDRAFT's Pocket-35B model holds the #1 domestic download position, accounting for 38.1% share among the top 30 Korean models on the companion download leaderboard, with 825,835 cumulative downloads.

The evaluation context is also worth noting: OpenRouter as a platform reportedly surpassed 10 million users and 10 trillion tokens processed per day as of August 2026, making the distribution footprint of these models practically significant.

How to try it

The leaderboard is publicly available on Hugging Face, published by GeniGen AI. You can access it through the Hugging Face Datasets hub. The leaderboard data is also downloadable via a public API — exact endpoint documentation would be available on the Hugging Face dataset page itself.

No private access, waitlist, or special credentials are described in the source for viewing the leaderboard. If you want to explore VIDRAFT models referenced in the companion download leaderboard, searching Hugging Face for VIDRAFT is the appropriate starting point.

⚠️ The source article does not publish specific API endpoints, model card URLs, or shell commands. Refer directly to the Hugging Face dataset page for GeniGen AI to get current access details.

FAQ

Q: Why does English benchmark score not predict Korean-language quality?
A: The evaluation found that models scoring well on English-centric benchmarks frequently failed on Korean-specific dimensions — particularly honorific register, institutional knowledge (e.g., Korean HR and regulatory conventions), and avoidance of literal translation phrasing. A 2023-release model achieving a perfect score on this leaderboard illustrates that Korean quality is an orthogonal axis to general capability rankings.

Q: If I'm using OpenRouter for a Korean-language product, should I just pick the cheapest provider for a given model?
A: Not necessarily. The leaderboard shows that the same model can cost up to 14.47× more depending on which of the 107 OpenRouter providers you route through — but provider-level differences in precision and uptime are also measurable. The leaderboard surfaces all three variables (price, precision, uptime) per provider, so the optimal choice is workload-dependent rather than purely cost-driven.

---

Originally reported by AI타임스 (2026-09-17) — source article.