DEV Community: Ward Ed

Benchmark Contamination 101: How Train/Test Overlap Inflates Leaderboard Scores (and How to Catch It)

Ward Ed — Tue, 06 Oct 2026 19:11:46 +0000

TL;DR

A leaderboard number is only as trustworthy as the gap between what a model trained on and what it was tested on. When test examples (or near-duplicates of them) leak into pretraining or fine-tuning data, the model memorizes answers instead of generalizing, and the reported score climbs for the wrong reason. This is benchmark contamination, also called train/test overlap or data leakage. It is common, often accidental, and frequently invisible in a self-reported number. This post explains why contamination inflates scores, three detection methods you can run yourself (n-gram overlap, canary strings, membership inference), a runnable Python n-gram checker, and a short checklist you can apply before you trust any benchmark claim.

What is benchmark contamination and why does it inflate scores?

Modern models are trained on web-scale corpora scraped from the open internet. Public benchmarks live on that same internet: on GitHub, in papers, on leaderboard repos, in blog posts that quote questions verbatim, and in Q&A sites where people discuss the exact items. When a benchmark's test questions and answers end up in the training mix, evaluation stops measuring reasoning and starts measuring recall.

The mechanism is simple. A benchmark is supposed to be a held-out sample. If the test examples were in the training data, the model can reproduce the answer from memory. Memorization looks exactly like competence on a single scored run, but it does not transfer to genuinely new inputs. The score goes up; the underlying capability does not.

Contamination comes in degrees. Verbatim contamination means the exact test string appears in training. Near-duplicate contamination means a paraphrase, a translated copy, or a reformatted version appears. Label leakage means the answer key or solution is present even if the question wording differs. Even partial exposure helps a model disproportionately on the memorized slice, which is enough to move a leaderboard when margins are a point or two.

Why can self-reported leaderboard scores be inflated?

Self-reported scores carry three structural risks that have nothing to do with bad faith.

First, the submitter controls the evaluation environment: the prompt template, the decoding settings, the number of few-shot examples, and sometimes the subset of items. None of those choices are contamination, but they compound with it.

Second, nobody fully audits the training corpus. For most open releases the pretraining data is described at a high level, not published item by item. A team can honestly say they did not intend to train on a benchmark while still having ingested it through a scraped mirror. Intent does not change the outcome.

Third, benchmarks age. A dataset released three years ago has had three years to be copied, quoted, reformatted, and re-uploaded. The older and more popular a benchmark is, the more likely its items are somewhere in a modern crawl. This is why a model can post a record score on a classic benchmark and a mediocre score on a freshly built private one.

None of this means every high score is fake. It means a score reported without a contamination check is an unverified claim, and the burden of evidence sits with whoever is pointing at the number.

How do you detect train/test overlap? Three methods

1. N-gram overlap

The most direct test: take each benchmark item, slide a window of n consecutive tokens or words across it, and check whether those n-grams appear in the training corpus (or in a proxy for it). High overlap at n of 8, 13, or 50 is strong evidence that the item, or a chunk of it, was seen during training. This is the method used by several major lab technical reports, which typically flag an item as contaminated when a sufficiently long n-gram matches. The strength of n-gram overlap is that it is cheap, interpretable, and does not need model internals. Its weakness is that it misses paraphrases and translations, so it gives a lower bound on contamination, never an upper bound.

2. Canary strings

A canary is a unique, random, unlikely-to-occur-naturally identifier that benchmark authors embed in their dataset specifically so they can later test whether it leaked. If a model can reproduce or recognize the canary GUID, the dataset was in its training data, full stop. BIG-bench popularized this with an embedded canary GUID. The limitation is that canaries only work if the benchmark author planted one and the dataset was consumed with the canary intact; stripped or reformatted copies defeat it. Still, when a canary test fires, it is close to conclusive.

3. Membership inference and behavioral tests

When you cannot inspect the training data at all, you probe the model's behavior. Membership inference asks whether the model treats a specific example as something it has seen before. Practical variants include: comparing perplexity on benchmark items versus closely matched held-out items (memorized items often have suspiciously low loss), the guided-prompting trick (does the model complete a test item far better when primed with the dataset name?), and option-order sensitivity (a model that memorized a multiple-choice answer is unusually robust to the correct answer's position while a reasoning model is not). These tests are noisier than n-gram or canary checks and need careful controls, but they are the only options for closed training data.

A runnable n-gram overlap check

Here is a compact, dependency-free n-gram overlap checker. Give it your benchmark items and any text you suspect may overlap with training (a crawl shard, a scraped page, a quoted dataset). It reports the fraction of each item's n-grams that also appear in the reference text.

from collections import Counter

def ngrams(text, n=13):
    """Word-level n-grams, lowercased and whitespace-normalized."""
    toks = text.lower().split()
    return [tuple(toks[i:i + n]) for i in range(len(toks) - n + 1)]

def overlap_score(item, reference_ngrams, n=13):
    """Fraction of the item's n-grams that appear in the reference set."""
    item_grams = ngrams(item, n)
    if not item_grams:
        return 0.0
    hits = sum(1 for g in item_grams if g in reference_ngrams)
    return hits / len(item_grams)

def build_reference(corpus_texts, n=13):
    """Precompute the n-gram set for everything you can see of training data."""
    ref = set()
    for doc in corpus_texts:
        ref.update(ngrams(doc, n))
    return ref

if __name__ == "__main__":
    # Stand-in for a crawl shard / scraped page you suspect overlaps training.
    training_proxy = [
        "the capital of the fictional country of zubrowka is lutz "
        "and its currency is the klubeck used since the year 1932",
    ]
    benchmark_items = [
        "The capital of the fictional country of Zubrowka is Lutz "
        "and its currency is the Klubeck used since the year 1932",   # leaked
        "Explain why a positive confidence interval overlap means "
        "two leaderboard scores may not be distinguishable at all",   # clean
    ]

    N = 13
    ref = build_reference(training_proxy, n=N)
    for i, item in enumerate(benchmark_items):
        s = overlap_score(item, ref, n=N)
        flag = "CONTAMINATED" if s > 0.5 else "clean"
        print(f"item {i}: {s:.0%} {N}-gram overlap -> {flag}")

Expected output:

item 0: 100% 13-gram overlap -> CONTAMINATED
item 1: 0% 13-gram overlap -> clean

Two practical notes. First, choose n deliberately: small n (3 to 5) produces false positives on common phrasing, large n (20 or more) misses chopped-up leaks, and 8 to 13 is a reasonable default for prose. Second, this is a lower bound. A 0 percent score proves only that there was no verbatim long-span match against the text you checked; paraphrase and translation slip right past it, which is why you pair it with behavioral tests.

A practical contamination checklist

Run this before you cite or compare any benchmark number.

Is the benchmark newer than the model's data cutoff? A test built after training data was frozen cannot have leaked. Prefer recent or private benchmarks for headline comparisons.
Did the authors plant a canary, and did the reporter check it? A passed canary test is cheap evidence and should be mentioned.
Was an n-gram overlap report published against the training corpus (or a representative sample)? If the training data is open, this should exist. If it is not, note that the overlap is simply unknown.
Is there a gap between the model's score on the popular benchmark and on a fresh, structurally similar one? A large drop on the new test is a classic contamination signature.
Are decoding settings, prompt template, and few-shot count disclosed and held constant across the models being compared? Contamination aside, undisclosed harness choices make numbers incomparable.
Is the score reported with uncertainty, and is the margin over the next model larger than that uncertainty? A contaminated slice often shows up as a margin that vanishes under resampling.
Who ran the evaluation? Self-reported with no third-party reproduction is an unverified claim, not a result. Independent reproduction on held-out data is the gold standard.

If a claim fails several of these, the right posture is not "the model is cheating." It is "this number is unverified, and here is specifically what would make it trustworthy."

FAQ

Is benchmark contamination always intentional?
No, and usually it is not. The most common path is accidental ingestion through a web crawl, since benchmarks live on the same internet that pretraining corpora are scraped from. Accidental contamination inflates scores exactly as much as deliberate contamination, which is why detection matters more than assigning blame.

Does a high n-gram overlap score prove a model will fail on real tasks?
It proves the benchmark number is unreliable for that model, not that the model is incapable. The correct response is to re-evaluate on a clean, uncontaminated test and trust that number instead. Contamination invalidates a measurement; it does not by itself measure capability.

Why not just keep all benchmarks private?
Private benchmarks resist contamination but sacrifice reproducibility and community trust, since nobody else can inspect the items or rerun the evaluation. The common compromise is a public benchmark with a planted canary, a held-out private slice, and periodic refreshes so the test can be rebuilt once the old version has saturated the internet.

What n should I use for an overlap check?
For natural-language items, 8 to 13 words is a reasonable default. Go smaller and you flag ordinary phrasing as a match; go much larger and you miss leaks that were reformatted or chopped into pieces. Report the n you used, because the overlap fraction is meaningless without it.

Can I detect contamination without access to the training data?
Yes, but only with behavioral tests: perplexity gaps between benchmark items and matched held-out items, guided-prompting sensitivity, and answer-position robustness. These are noisier than n-gram or canary checks, so treat them as evidence that accumulates rather than a single decisive test.

When a 0.4-Point Lead Means Nothing: Reading Open-Model Leaderboard Margins Like a Statistician

Ward Ed — Tue, 06 Oct 2026 00:44:19 +0000

TL;DR

A leaderboard rank is a point estimate, not a fact. Every accuracy score carries a confidence interval, and on most public evals that interval is wide enough to swallow the gap between the top few models.
You can estimate the error bar yourself from two numbers you already have: the accuracy and the number of eval items. For a 300-item test at 70 percent accuracy, the 95 percent confidence half-width is about 5 points. A 0.4-point lead inside that band is noise.
For paired comparisons on the same questions, use McNemar's test, not two separate intervals. It is strictly more sensitive and it is what the gap actually deserves.
Before trusting any ranking, ask whether the benchmark can even discriminate. If every top model scores above 95 percent, the test is saturated and the order is close to random.
Popularity is not quality. The most-liked open models on the Hub (cover chart, pulled 2026-10-06) are a social signal, not an accuracy ranking. Treat the two as separate axes.

This is the second post in Leaderboard Forensics. The first covered how the Hub's benchmark plumbing actually works. This one is about the single most common misreading of those numbers: believing a rank order that the sample size cannot support.

Why do the top open models cluster so tightly?

Open-model evaluation has matured to the point where the frontier is crowded. On a typical reasoning or knowledge benchmark, the top ten entries often sit inside a three-point band. The cover chart above shows the eight most-liked open text-generation repositories on the Hugging Face Hub as of 2026-10-06, and likes are a popularity proxy, but the same crowding shows up in accuracy tables.

Here is the uncomfortable part. When scores cluster, the rank order is mostly determined by measurement noise rather than capability. The model in position one and the model in position four may be statistically indistinguishable. The leaderboard still has to print them in some order, so it prints the noise.

The curve above is illustrative. It plots the 95 percent confidence half-width for a model scoring 70 percent, as a function of how many questions the benchmark contains. At 100 items the half-width is about 9 points. At 1,000 items it drops to roughly 2.8 points. Only around 10,000 items does it fall under one point. Most popular public benchmarks live at the left end of that curve.

How do I compute the error bar on a single accuracy score?

Accuracy on a fixed test set is a binomial proportion. If a model answers k of n questions correctly, the estimated accuracy is p = k / n, and the standard error is sqrt(p * (1 - p) / n). A rough 95 percent interval is p plus or minus 1.96 * SE.

The Wald interval above is fine for a quick gut check, but it misbehaves near 0 and 100 percent and for small n. The Wilson interval is better behaved and still closed-form. Here is a compact, dependency-light helper.

import math

def wilson_interval(k, n, z=1.96):
    """95% Wilson score interval for a binomial proportion.
    k = correct answers, n = total eval items."""
    if n == 0:
        return (0.0, 0.0, 0.0)
    p = k / n
    denom = 1 + z * z / n
    center = (p + z * z / (2 * n)) / denom
    half = (z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n))) / denom
    return (p, center - half, center + half)

for n in (100, 300, 1000):
    p, lo, hi = wilson_interval(int(0.70 * n), n)
    print(f"n={n:5d}  acc={p:.3f}  95% CI=[{lo:.3f}, {hi:.3f}]  width={hi-lo:.3f}")

Output:

n=  100  acc=0.700  95% CI=[0.604, 0.782]  width=0.178
n=  300  acc=0.700  95% CI=[0.646, 0.748]  width=0.102
n= 1000  acc=0.700  95% CI=[0.671, 0.727]  width=0.056

Read that bottom line carefully. Even with 1,000 questions, the interval around a 70 percent score spans almost six points. If two models are reported as 70.3 and 69.9 on that benchmark, the difference is well inside the noise floor. The rank order is not evidence of anything.

When is a leaderboard gap actually significant?

The single-score interval is the wrong tool for comparing two models, because it ignores that they were graded on the same questions. Scores on a shared test set are correlated. The correct instrument is McNemar's test, which looks only at the items where the two models disagree.

Build a 2x2 table of agreement. Let b be the number of questions where model A is right and model B is wrong, and c the reverse. The questions both got right or both got wrong carry no information about which is better. McNemar's statistic is (b - c)^2 / (b + c), compared against a chi-square distribution with one degree of freedom.

from scipy.stats import chi2

def mcnemar(b, c):
    """b: A right & B wrong; c: A wrong & B right."""
    n = b + c
    if n == 0:
        return 1.0
    stat = (abs(b - c) - 1) ** 2 / n  # continuity-corrected
    return 1 - chi2.cdf(stat, df=1)

# Two models 0.4 points apart on a 300-item eval:
# say 18 items where A wins, 17 where B wins, rest tied.
print(f"p-value = {mcnemar(18, 17):.3f}")   # p-value = 1.000

A one-item edge out of 300 produces a p-value of essentially 1.0. There is no detectable difference. You would need the disagreement counts to be lopsided, not the headline accuracy to be marginally higher, before the gap earns a rank.

This is why serious evaluation reports publish disagreement matrices, bootstrap confidence intervals, or at least the per-benchmark item counts. If a leaderboard gives you only a sorted column of three-decimal numbers, it is giving you false precision.

How do I tell whether a benchmark can discriminate at all?

Before arguing about margins, check that the test has headroom. A benchmark that is saturated, where the top cluster all score above 95 percent, cannot separate frontier models no matter how many items it has. The remaining questions are either ambiguous, mislabeled, or trivially easy, and the order among the leaders is driven by which model happened to win the coin flips on the hard residue.

Three quick discrimination checks:

Ceiling check. If the top five are all within one point of 100 percent, the benchmark is done. Move on.
Baseline check. Compute what a trivial strategy scores. If majority-class guessing or a tiny n-gram model gets 90 percent, the benchmark measures format compliance, not capability.
Spread check. Look at the standard deviation across all submitted models. A benchmark where everyone scores 68 to 72 has almost no discriminative range left.

A benchmark earns your trust when the baseline is low, the ceiling is far away, and the spread across models is several times the single-score confidence width.

What should I actually do with a leaderboard, then?

Treat the leaderboard as a filter, not a verdict. Use it to shortlist the handful of models in the top band, then evaluate those candidates on your own held-out data that resembles your real traffic. The rank inside that top band is the least reliable information on the page. The membership of the band is the useful part.

And keep popularity and accuracy on separate axes. The cover chart is real Hub data, and it tells you what the community has adopted, which matters for tooling, documentation, and community support. It says nothing about which model is most accurate on your task. Both signals are valuable. Conflating them is how teams end up defending a 0.4-point lead that never existed.

FAQ

Is the Wald interval ever good enough?
For a sanity check on a single score with a few hundred or more items, yes. For scores near 0 or 100 percent, or for small n, switch to Wilson, which stays inside the valid range and does not collapse to zero width at the extremes.

Why McNemar instead of a two-sample proportion test?
Because both models answer the same questions, their scores are paired and correlated. A two-sample test assumes independence and will be too conservative. McNemar conditions on the disagreements, which is exactly the information that distinguishes the two models.

How many eval items do I need to resolve a one-point gap?
Roughly, to get a 95 percent half-width near one point at mid-range accuracy you need on the order of 10,000 independent items. Most public benchmarks have far fewer, which is why one-point gaps are usually unresolvable.

Do bootstrap confidence intervals help?
Yes, especially when scoring is not a clean right-or-wrong binary, such as graded or rubric-scored tasks. Resample the items with replacement a few thousand times, recompute the metric, and read the 2.5 and 97.5 percentiles. It makes no distributional assumption and handles weird metrics gracefully.

Does a larger model automatically win on a saturated benchmark?
No. On a saturated test the residual questions are dominated by noise and label quality, so scale buys you almost nothing. The honest move is to find a harder benchmark with headroom, or build one from your own data.

Can I trust the three decimals a leaderboard prints?
Only if the item count justifies them. Three decimals on a 300-item eval is false precision. The real resolution of that test is closer to one or two points, so mentally round the column before you rank it.

How Hugging Face Official Benchmark Leaderboards Actually Work: .eval_results YAML, the base_model Filter, and the 30% the Default View Hides

Ward Ed — Mon, 05 Oct 2026 08:56:56 +0000

TL;DR: Hugging Face currently tags 48 datasets as official benchmarks (benchmark:official). Their leaderboards are not uploaded by the benchmark owners: they are assembled automatically from small .eval_results/*.yaml files that model authors commit to their own model repos (or propose through pull requests). On October 5, 2026 I queried every board twice through the public API. The default view returned 1,023 entries. With ?base_model=false, which keeps derivative models, it returned 1,469. So 446 entries (30.4%) never show up in the default view, and the deciding field is the base_model key in each model card. Another finding: 772 of the 1,023 visible entries (75.5%) come from open pull requests, and none were flagged verified.

What are Hugging Face official benchmark leaderboards?

An official benchmark is just a dataset repo that carries the benchmark:official tag. The Hub renders a leaderboard widget on its dataset page. Each row on that widget is one model's score on that benchmark. A dataset owner does not have to run any evaluation service for this. The Hub collects scores that model repos self-report and matches them to the dataset.

You can pull the list of boards with one request:

curl -s "https://huggingface.co/api/datasets?filter=benchmark:official&limit=200" \
  | python -c "import sys,json; d=json.load(sys.stdin); print(len(d)); print('\n'.join(x['id'] for x in d))"

When I ran it on October 5, 2026, it returned 48 datasets. Some of the better known ones: TIGER-Lab/MMLU-Pro, Idavidrein/gpqa, cais/hle, SWE-bench/SWE-bench_Verified, hf-audio/open-asr-leaderboard, openai/gsm8k, MathArena/aime_2026, MMMU/MMMU_Pro and harborframework/terminal-bench-2.0. The list also includes newer, narrower boards such as llamaindex/ParseBench, llamaindex/ExtractBench, LEXam-Benchmark/LEXam, FutureMa/EvasionBench and crosbylegal/RedlineBench.

Four of the 48 (tiiuae/PBench, mercor/ACE, ChrisHayduk/nanofold-public, harborframework/terminal-bench-science) had zero entries in both views when I checked. Tagged boards can exist before anyone has reported a score to them.

How are Hugging Face leaderboard entries built from .eval_results YAML?

Every row traces back to one YAML file in a model repo, under a directory named .eval_results/. The leaderboard API includes the path in each row, so you can see it yourself:

curl -s "https://huggingface.co/api/datasets/Idavidrein/gpqa/leaderboard" \
  | python -c "import sys,json; e=json.load(sys.stdin)[1]; print({k:e[k] for k in ('rank','modelId','value','filename','verified','pullRequest')})"

Each row has these fields: rank, filename, value, verified, source, pullRequest, modelId, author, lower_is_better and num_parameters. filename is the YAML path inside the model repo, and pullRequest holds a PR number when the file lives on a PR ref instead of main.

Here is a real file. moonshotai/Kimi-K3 keeps five of them on main:

curl -s "https://huggingface.co/api/models/moonshotai/Kimi-K3/tree/main/.eval_results" \
  | python -c "import sys,json; [print(f['path'], f['size']) for f in json.load(sys.stdin)]"

.eval_results/apex-agents.yaml 157
.eval_results/deep-swe.yaml 156
.eval_results/gpqa.yaml 152
.eval_results/hle.yaml 139
.eval_results/moonshotai__Kimi-K3.yaml 211

The GPQA one (/raw/main/.eval_results/gpqa.yaml) is just 152 bytes:

- dataset:
    id: Idavidrein/gpqa
    task_id: diamond
  value: 93.5
  source:
    url: https://huggingface.co/moonshotai/Kimi-K3
    name: Model Card

That is the whole contract:

dataset.id is the benchmark dataset repo. It has to match one of the official board IDs exactly, or the score goes nowhere.
dataset.task_id picks a sub-task or split when a board has several (for GPQA, diamond).
value is the number that gets ranked. The board's lower_is_better flag decides the sort order (WER on the ASR board, for example).
source is an attribution link. In practice it usually points back at the model card.
Optional keys show up in the wild too, like date and a free-text notes field for the protocol (sample count, temperature, token budget, precision). Use them. They are the only place a reader can see how the number was produced.

The file is a YAML list, so a single file can hold several benchmark results. Many repos still use one file per benchmark, which keeps diffs and PRs small.

Why does my model not appear on the Hugging Face leaderboard? (the base_model filter)

This is where most authors get stuck. The leaderboard endpoint has a parameter called base_model. Without it, the API (and the default widget) drops models whose card declares a base_model, meaning fine-tunes, merges, quantizations and adapters. With ?base_model=false, those derivatives come back:

B=Idavidrein/gpqa
curl -s "https://huggingface.co/api/datasets/$B/leaderboard" | python -c "import sys,json;print(len(json.load(sys.stdin)))"
curl -s "https://huggingface.co/api/datasets/$B/leaderboard?base_model=false" | python -c "import sys,json;print(len(json.load(sys.stdin)))"

For GPQA that is 111 vs 175. I ran the same comparison for all 48 boards with this script:

import requests
from concurrent.futures import ThreadPoolExecutor

API = "https://huggingface.co/api/datasets"
boards = [d["id"] for d in requests.get(f"{API}?filter=benchmark:official&limit=200").json()]

def count(bid):
    d = requests.get(f"{API}/{bid}/leaderboard", timeout=60).json()
    a = requests.get(f"{API}/{bid}/leaderboard?base_model=false", timeout=60).json()
    return bid, len(d), len(a)

rows = list(ThreadPoolExecutor(8).map(count, boards))
dflt = sum(r[1] for r in rows); allr = sum(r[2] for r in rows)
print(len(rows), dflt, allr, f"{(allr - dflt) / allr:.1%} hidden")

Output on 2026-10-05: 48 1023 1469 30.4% hidden.

How much is hidden varies a lot from board to board:

Board	Default	With derivatives	Hidden share
TIGER-Lab/MMLU-Pro	141	216	34.7%
Idavidrein/gpqa	111	175	36.6%
cais/hle	89	108	17.6%
SWE-bench/SWE-bench_Verified	67	89	24.7%
hf-audio/open-asr-leaderboard	62	86	27.9%
likaixin/ScreenSpot-Pro	23	48	52.1%
openai/gsm8k	18	46	60.9%
MMMU/MMMU_Pro	10	24	58.3%
llamaindex/ExtractBench	11	25	56.0%
LiquidAI/ifstruct-v1.0	6	18	66.7%
mteb/arguana	3	16	81.3%
crosbylegal/RedlineBench	13	13	0.0%

The pattern makes sense. Older or easier benchmarks like GSM8K attract many fine-tunes, so derivatives are the majority there and the default view shows a minority of the reported scores. On agentic and legal boards where most submissions are base models from labs (RedlineBench, skillsbench, Long-Horizon-Terminal-Bench), the two views are identical.

The filter reads the model card metadata, not the weights. A model trained from scratch that lists a base_model anyway (some teams do this to credit an architecture or tokenizer source) gets hidden just like a LoRA. A heavily modified derivative that leaves the field out shows up as if it were a base model. So the default view is a filter on declared lineage, not real lineage.

Who verifies scores on Hugging Face official leaderboards?

According to the API, nobody does yet. Of the 1,023 default-view rows, 0 had verified: true. Every number on these boards is self-reported by whoever controls the model repo or opens a PR to it.

The PR path is bigger than most people expect. 772 of 1,023 visible rows (75.5%) carried a pullRequest number. So the YAML sits on an open (unmerged) PR ref such as refs/pr/1, not on main. Reading those files directly needs the PR ref:

# file on an open PR, not on main
curl -s "https://huggingface.co/XiaomiMiMo/MiMo-V2.5-Pro/raw/refs%2Fpr%2F1/.eval_results/gsm8k.yaml"

If you try tree/main/.eval_results on such a model, you get .eval_results does not exist on "main" even though the row shows on the leaderboard. That is expected behavior, not a bug.

Practical implications for anyone citing these boards:

A row is a claim, not a measurement. Follow the source.url and read the notes, if there are any.
Protocols differ across rows on the same board. Majority voting over many samples, different thinking budgets, different task subsets: none of that is normalized. The task_id field is the only structural guard, and it only separates named splits.
Rows can come from someone other than the model owner, through a PR. Check the PR author before treating a number as the vendor's.

How do I submit a model score to a Hugging Face benchmark leaderboard?

There is no submission form. You commit a file:

from huggingface_hub import HfApi
import yaml, io

result = [{
    "dataset": {"id": "Idavidrein/gpqa", "task_id": "diamond"},
    "value": 71.2,
    "date": "2026-10-05",
    "source": {"url": "https://huggingface.co/your-org/your-model", "name": "Model Card"},
    "notes": "GPQA Diamond, 198 items, pass@1, temperature 0, bf16",
}]
buf = io.BytesIO(yaml.safe_dump(result, sort_keys=False).encode("utf-8"))
HfApi().upload_file(
    path_or_fileobj=buf,
    path_in_repo=".eval_results/gpqa.yaml",
    repo_id="your-org/your-model",
    commit_message="Add GPQA Diamond eval result",
    # create_pr=True,  # use this when you are not the repo owner
)

The numbers above are placeholders for illustration. Then check whether the row appears, and which view it appears in:

import requests
def where_am_i(board, model):
    base = f"https://huggingface.co/api/datasets/{board}/leaderboard"
    d = [e for e in requests.get(base).json() if e["modelId"] == model]
    a = [e for e in requests.get(base + "?base_model=false").json() if e["modelId"] == model]
    return {"default_view": bool(d), "with_derivatives": bool(a),
            "rank_default": d[0]["rank"] if d else None,
            "rank_all": a[0]["rank"] if a else None}

print(where_am_i("Idavidrein/gpqa", "your-org/your-model"))

If with_derivatives is true and default_view is false, the base_model field in your card is what hides you.

Checklist for model authors before reporting a leaderboard rank

Match dataset.id exactly to one of the 48 IDs from filter=benchmark:official. Case and owner prefix matter.
Set task_id when the board has subsets. A GPQA "main" score sitting next to "diamond" scores is misleading.
Decide on base_model on purpose. If your model really is a fine-tune, keeping the field is honest, but you will only appear in the derivative-inclusive view. If the field is left over from a template or only credits a tokenizer, fix the card. Don't remove it from a real derivative just to get a better spot.
Verify with the parameterless API, not the ?base_model=false URL and not a cached screenshot. The parameterless call is what most visitors see by default.
Quote the right rank. "Rank 3 including derivatives" and "rank 3" are different claims. With 30.4% of entries hidden, the gap can be large.
Write the protocol in notes: sample count, decoding settings, token budget, precision, subset size. Nobody verifies rows centrally, so this is your credibility.
Merge your own PRs. If a contributor opened the eval PR, the row already counts. Merging it puts the file on main and makes the history auditable.

FAQ

How many official benchmark leaderboards are on Hugging Face?

48 datasets carried the benchmark:official tag on October 5, 2026. Four of them had no entries yet. Query https://huggingface.co/api/datasets?filter=benchmark:official&limit=200 for the current list.

What does `base_model=false` do in the Hugging Face leaderboard API?

It adds back models whose card declares a base_model (fine-tunes, merges, quantizations, adapters). The default request leaves them out. Across all boards that is 1,469 entries vs 1,023, so 30.4% are hidden by default.

Where does a Hugging Face leaderboard score come from?

From a YAML file under .eval_results/ in the model repo, on main or on an open PR ref. The leaderboard row's filename field gives the path, and pullRequest gives the PR number if there is one.

Are Hugging Face official leaderboard scores verified?

Not at the time of measurement. All 1,023 default-view rows had verified: false. Treat each row as a self-reported claim and check its source and notes.

Why does my model show on the board in one browser link but not another?

You are probably comparing the default view with the derivative-inclusive view. Run both API calls for your modelId. If only the ?base_model=false call finds you, your model card's base_model field explains it.

Can someone else add a score for my model?

Yes. Anyone can open a pull request that adds .eval_results/*.yaml to your repo, and 75.5% of visible rows came from PR refs when I measured. Review those PRs like any other change to your model card.

Methodology: all counts come from unauthenticated requests to the public Hugging Face API on 2026-10-05. Leaderboards change daily, so re-run the scripts above for current numbers.

DEV Community: Ward Ed

Benchmark Contamination 101: How Train/Test Overlap Inflates Leaderboard Scores (and How to Catch It)

TL;DR

What is benchmark contamination and why does it inflate scores?

Why can self-reported leaderboard scores be inflated?

How do you detect train/test overlap? Three methods

1. N-gram overlap

2. Canary strings

3. Membership inference and behavioral tests

A runnable n-gram overlap check

A practical contamination checklist

FAQ

Further reading

When a 0.4-Point Lead Means Nothing: Reading Open-Model Leaderboard Margins Like a Statistician

TL;DR

Why do the top open models cluster so tightly?

How do I compute the error bar on a single accuracy score?

When is a leaderboard gap actually significant?

How do I tell whether a benchmark can discriminate at all?

What should I actually do with a leaderboard, then?

FAQ

Further reading

How Hugging Face Official Benchmark Leaderboards Actually Work: .eval_results YAML, the base_model Filter, and the 30% the Default View Hides

What are Hugging Face official benchmark leaderboards?

How are Hugging Face leaderboard entries built from .eval_results YAML?

Why does my model not appear on the Hugging Face leaderboard? (the base_model filter)

Who verifies scores on Hugging Face official leaderboards?

How do I submit a model score to a Hugging Face benchmark leaderboard?

Checklist for model authors before reporting a leaderboard rank

FAQ

How many official benchmark leaderboards are on Hugging Face?

What does `base_model=false` do in the Hugging Face leaderboard API?

Where does a Hugging Face leaderboard score come from?

Are Hugging Face official leaderboard scores verified?

Why does my model show on the board in one browser link but not another?

Can someone else add a score for my model?

DEV Community: Ward Ed

Benchmark Contamination 101: How Train/Test Overlap Inflates Leaderboard Scores (and How to Catch It)

TL;DR

What is benchmark contamination and why does it inflate scores?

Why can self-reported leaderboard scores be inflated?

How do you detect train/test overlap? Three methods

1. N-gram overlap

2. Canary strings

3. Membership inference and behavioral tests

A runnable n-gram overlap check

A practical contamination checklist

FAQ

Further reading

When a 0.4-Point Lead Means Nothing: Reading Open-Model Leaderboard Margins Like a Statistician

TL;DR

Why do the top open models cluster so tightly?

How do I compute the error bar on a single accuracy score?

When is a leaderboard gap actually significant?

How do I tell whether a benchmark can discriminate at all?

What should I actually do with a leaderboard, then?

FAQ

Further reading

How Hugging Face Official Benchmark Leaderboards Actually Work: .eval_results YAML, the base_model Filter, and the 30% the Default View Hides

What are Hugging Face official benchmark leaderboards?

How are Hugging Face leaderboard entries built from .eval_results YAML?

Why does my model not appear on the Hugging Face leaderboard? (the base_model filter)

Who verifies scores on Hugging Face official leaderboards?

How do I submit a model score to a Hugging Face benchmark leaderboard?

Checklist for model authors before reporting a leaderboard rank

FAQ

How many official benchmark leaderboards are on Hugging Face?

What does base_model=false do in the Hugging Face leaderboard API?

Where does a Hugging Face leaderboard score come from?

Are Hugging Face official leaderboard scores verified?

Why does my model show on the board in one browser link but not another?

Can someone else add a score for my model?

What does `base_model=false` do in the Hugging Face leaderboard API?