Who Leads Hugging Face's Official Benchmarks? Concentration, Gaps and Hidden Entries (Measured 2026-10-06)

QuantID — Mon, 05 Oct 2026 08:57:14 +0000

TL;DR

On 2026-10-06 we read every dataset tagged benchmark:official from the public Hugging Face Hub API, and every leaderboard attached to those datasets, twice: once in the default view and once with base_model=false. Every number below is a reading taken on that date, and about 40 lines of Python reproduce it.

48 official benchmark datasets, 44 with at least one leaderboard entry.
Default view: 1,024 entries from 95 organizations across 411 distinct models.
The top 5 organizations hold 40.3% of visible entries, the top 10 hold 59.0%.
Full population (with base_model=false): 1,479 entries, 161 namespaces. Gini 0.709, HHI 0.0369.
Median relative gap between #1 and #2: 3.69%. On 9 of 43 comparable boards the gap is under 1%.
455 entries (30.8%) never appear in the default view because the board hides models that declare a base_model. 34 of 44 boards are affected.

Many organizations take part, but entry counts are uneven. On about one board in five the top two are close enough that the order could flip on a rerun, and nearly a third of all submissions only show up if you change a query parameter.

Why measure benchmark leaderboards instead of ranking them?

QuantID works on measurement and diagnostics, so we treat a leaderboard as an instrument rather than a scoreboard. A leaderboard is a fixed test, a scoring rule and a published ordering. Like any instrument it has a resolution, a bias and a display layer that can hide part of the signal. Most writing about leaderboards asks who is first. We ask four questions about the instrument itself:

How many independent parties actually participate?
How unequal is that participation?
How far apart are the leaders, compared with plausible run-to-run noise?
How much of the data is filtered away before a human sees it?

Hugging Face lets model owners attach evaluation results to a model repository through .eval_results/*.yaml files. Datasets tagged as official benchmarks collect those files into a board. The whole system answers to a public API, so it can be audited like any other measurement pipeline.

What data did we read?

Three public calls, no authentication:

GET https://huggingface.co/api/datasets?filter=benchmark:official&limit=200
GET https://huggingface.co/api/datasets/{id}/leaderboard
GET https://huggingface.co/api/datasets/{id}/leaderboard?base_model=false

Each row carries rank, value, modelId, lower_is_better, a source link and author metadata. We define the organization of an entry as the namespace before the slash in modelId (so Qwen/Qwen3.8-Flash-Next belongs to Qwen). User namespaces count as organizations. We do not merge related accounts: the API does not say which accounts belong together, and guessing would inject our own bias into the measurement.

For each board, the larger of the two responses is the full population (in practice the base_model=false response). The default response is the visible population.

Quantity	Default view	Full population
Benchmark datasets	48	48
Boards with entries	44	44
Entries	1,024	1,479
Distinct models	411	647
Distinct organizations	95	161

New YAML files are merged daily, so the boards keep changing. Treat every number here as a snapshot, not a constant.

How concentrated is AI benchmark participation?

"Concentration" covers two separate questions: who submits, and who wins.

In the default view the five largest submitters are Qwen (136), zai-org (84), deepseek-ai (65), moonshotai (65) and nvidia (63). Together those five hold 40.3% of the 1,024 visible entries, and the top ten hold 59.0%. Entry volume mostly tracks how many model sizes a lab ships; it says little about quality.

Across the full population the inequality is easier to summarize with two numbers that seem to disagree:

Metric	Value
Gini (entries by org)	0.709
HHI	0.0369
Effective number of orgs (1/HHI)	about 27
Orgs appearing on exactly one board	86 of 161

In antitrust terms an HHI of 0.0369 is unconcentrated (below 0.15) because no single organization holds a large slice; the largest, Qwen, has 190 of 1,479 entries (12.8%). A Gini of 0.709 says the distribution is still very unequal. The long tail explains both readings: more than half of all organizations appear exactly once, while a few large labs submit many model sizes to many boards.

Reporting both indicators keeps "no monopoly" from being read as "equal participation". They are answers to different questions.

Who actually wins?

We find the #1 entry of each board by sorting on value: descending by default, ascending when lower_is_better is true. Across 44 boards, 24 distinct organizations hold at least one #1 position. The most frequent holders in this snapshot were FINAL-Bench (9 boards), zai-org (4), deepseek-ai (3) and moonshotai (3), then XiaomiMiMo, ornith-ai, tsinghua-sigs-robot-lab and tencent with 2 each. Leadership is spread across roughly two dozen teams, and the biggest submitters are mostly not the most frequent winners: Qwen has the most entries overall but holds the top spot on only one board in this reading.

How close are the top two models on a typical benchmark?

Concentration tells you who shows up. Competitiveness tells you whether the order at the top means anything. For each board with at least two entries we compute the relative gap between #1 and #2:

$$g = \frac{|v_1 - v_2|}{|v_1|} \times 100\%$$

A relative measure lets us compare boards scored in percent, in error rate or in arbitrary units. Measured over 43 comparable boards, the median gap is 3.69%, and the distribution is wide:

Why does a gap under 1% matter? Most of these benchmarks have a few hundred to a few thousand items. For a binomial accuracy $p$ on $n$ items the standard error is $\sqrt{p(1-p)/n}$. At $p = 0.8$ and $n = 1{,}000$ that is about 1.3 percentage points, or roughly 1.6% in relative terms. Self-reported runs also differ in sampling temperature, prompt template and harness. With all that, a lead under 1% usually cannot be told apart from a tie, so on about one board in five the current #1 could change on a rerun. At the other end, a handful of boards show leads above 20%; that usually means a genuinely new capability, a board with very few entries, or entries scored under different conditions. A large gap is a reason to open the source link, not a conclusion by itself. Our rule for any ranking: report the gap with an uncertainty estimate (2 SE or a bootstrap interval) and call it a tie when the interval contains zero.

What share of Hugging Face leaderboard entries are hidden by default?

The default /leaderboard response, and the default web view, leaves out entries from models whose card declares a base_model. That removes fine-tunes, merges, quantized variants and other derivative releases. Passing base_model=false brings them back.

$$\text{hidden share} = 1 - \frac{\sum_b |L_b^{default}|}{\sum_b |L_b^{full}|}$$

Measured: 455 of 1,479 entries (30.8%) are hidden in the default view, and 34 of 44 boards hide at least one entry.

The filter is a design choice, not a bug: it keeps the default view focused on original models. It still has measurable consequences. A fine-tuned model can outscore its base and stay invisible to anyone who does not change the query. Rankings quoted from the default page and rankings computed with the full population can disagree. Concentration statistics computed only on the default view understate the long tail, because small teams publish a large share of the derivative work. If you report a position on one of these boards, say which view it comes from.

How can I reproduce this measurement?

The script below needs only the standard library plus requests, and runs in under a minute.

import requests, collections, statistics as st
import concurrent.futures as cf

H = "https://huggingface.co/api"
ids = [d["id"] for d in requests.get(
    f"{H}/datasets", params={"filter": "benchmark:official", "limit": 200}, timeout=60).json()]

def fetch(i):
    def get(p):
        try:
            r = requests.get(f"{H}/datasets/{i}/leaderboard", params=p, timeout=60).json()
            return r if isinstance(r, list) else []
        except Exception:
            return []
    return i, get({}), get({"base_model": "false"})

rows = list(cf.ThreadPoolExecutor(8).map(fetch, ids))

def org(e): return e["modelId"].split("/")[0]
def gini(x):
    x = sorted(x); n = len(x); s = sum(x)
    return sum((2*i - n - 1) * v for i, v in enumerate(x, 1)) / (n * s)

entries, firsts = collections.Counter(), collections.Counter()
gaps, n_default, n_full = [], 0, 0
for bid, default, full in rows:
    full = full if len(full) >= len(default) else default
    n_default += len(default); n_full += len(full)
    for e in full:
        entries[org(e)] += 1
    if not full:
        continue
    lib = full[0].get("lower_is_better", False)
    s = sorted(full, key=lambda e: e["value"], reverse=not lib)
    firsts[org(s[0])] += 1
    if len(s) > 1 and s[0]["value"]:
        gaps.append(abs(s[0]["value"] - s[1]["value"]) / abs(s[0]["value"]) * 100)

v = sorted(entries.values(), reverse=True); N = sum(v)
print("orgs", len(entries), "entries", N, "gini", round(gini(v), 3))
print("top5", round(100*sum(v[:5])/N, 1), "top10", round(100*sum(v[:10])/N, 1))
print("median gap %", round(st.median(gaps), 2), "gap<1%", sum(g < 1 for g in gaps))
print("hidden %", round(100*(1 - n_default/n_full), 1))
print("top #1 holders", firsts.most_common(8))

Run it on another day and the numbers will differ. That is expected: the method stays fixed, and each run records the state of the boards on its own date.

What are the limits of this measurement?

An organization here is a namespace, not a legal entity. One company may publish under several namespaces, and one namespace may host several teams. We did not merge or split any.
Scores are mostly self-reported. Most rows carry verified: false and cite a model card. We measure the published board, not the truth behind each number.
Ties and sort direction. We trust the lower_is_better flag and treat multi-metric datasets exactly as the API returns them.
Snapshot effect. Entries arrive as pull requests to model repositories, so one large release can move the top-k share by several points in a day.
The gap metric ignores noise. A relative gap has no units and no per-board significance test; the binomial SE above is a rough guide.

Disclosure

QuantID is a technology alliance partner of VIDRAFT, which appears on several boards (including under the FINAL-Bench namespace); data shown here is computed identically for every organization.

FAQ

How many official benchmark leaderboards are on Hugging Face?
On 2026-10-06 the API returned 48 datasets tagged benchmark:official. Of those, 44 had at least one leaderboard entry.

Is AI benchmark leadership dominated by a few labs?
Participation is unequal: the full-population Gini is 0.709 and the top 10 organizations hold 59.0% of visible entries. Still, #1 positions are spread across 24 organizations, and no organization holds more than about 13% of all entries.

Why don't I see some models on a Hugging Face leaderboard?
The default view hides entries from models that declare a base_model in their card. Add ?base_model=false to the API call to see them. In this snapshot that filter hid 30.8% of all entries across 34 of 44 boards.

How close are the top two models on a typical board?
The median relative gap between #1 and #2 was 3.69%. On 9 of 43 boards it was under 1%, which is usually within run-to-run noise.

Why report both Gini and HHI?
HHI is driven by the biggest shares, so it stays low when no single organization is large. Gini measures inequality across the whole distribution, including the long tail of single-entry participants. Reporting both prevents "no monopoly" from being read as "equal participation".

Can I get these numbers live?
Yes. The script above reproduces them, and we maintain a live map of all official boards as a public Space.

DEV Community: QuantID