DEV Community: Ward Ed

Reading a Hardware-Equivalence Claim Like an Evaluator: A Checklist Applied to an NPU vs GPU Pretraining Twin

Ward Ed — Mon, 26 Oct 2026 06:12:21 +0000

Disclosure: this account is run by members of the VIDRAFT team, and the work discussed below is VIDRAFT's own result. It is not an independent review.

TL;DR

"Matches the GPU" is easy to say and hard to show. Here are five checks an evaluator should run on any such claim, applied to VKORE-325M, pretrained on a Rebellions ATOM-Max NPU and on a B200 GPU. It passes four; the fifth (seed variance) is still open.

Check 1: Is there a true twin?

The reference must differ only in hardware. Here initialization, data, batch and schedule were identical.

Check 2: Is the evaluation data really unseen?

Read the sampler, not the split table. The first evaluation used windows from the end of the training file, but training drew random windows from the whole file, so those windows were partly seen. The corrected evaluation uses FineWeb-Edu shard 013, which was never in the training corpus (shards 000 and 001).

Check 3: Is the comparison paired?

Window-to-window variance dwarfs model differences. On identical windows the paired standard error is 0.0007, versus about 0.0049 for each model's mean.

Model	Loss, unseen (1,024 windows)
Rebellions ATOM-Max NPU	0.7993
B200 GPU	0.7994

Check 4: Can you rerun it?

Both checkpoints and evaluate_unseen.py are public in the model repo. An evaluator can reproduce the table without trusting the authors.

Check 5: Is variance across seeds reported?

Not yet: one run per hardware. A stronger claim would repeat each side with several seeds.

FAQ

What would falsify the claim? A paired difference well outside two standard errors on fresh data, or failure to reproduce with the published files.

Does the claim cover speed? No, only quality.

Sources

Technical report: https://huggingface.co/blog/FINAL-Bench/pretraining-a-325m-language-model-from-zero-on-an

What a Youden index of 77 actually means on the OWASP Benchmark

Ward Ed — Fri, 23 Oct 2026 09:23:10 +0000

Security tool vendors love a single headline number. The OWASP Benchmark gives one, the Youden index, and it is worth understanding before you compare anything.

What the Youden index is

The OWASP Benchmark runs a tool against about 2,700 test cases, each a known true or false vulnerability, and scores the result as a Youden index from 0 to 100. The index is sensitivity plus specificity minus one, rescaled. In plain terms, it rewards catching real issues and penalizes false alarms at the same time.

Why that matters

A tool can inflate raw precision by quietly reporting fewer findings, or inflate recall by flagging everything. Youden resists both, because dropping true positives hurts sensitivity and flagging noise hurts specificity. So a move from the low 50s to the high 70s is meaningful only if recall held while false positives fell.

How to compare tools fairly

Ask whether a score is self-measured or verified through the official scoring harness. Both are legitimate, but they are not the same claim.
Check that the test set and version are stated. The benchmark evolves.
Separate static-analysis-only numbers from results that include dynamic probing. They measure different things.
Treat mid-70s and above as the current commercial band, not a finish line.

The point is not any one product. It is that a number only means something when you know how it was measured.

GPQA Diamond Has 198 Questions: What the Hugging Face Leaderboard's 115 Rows Can and Cannot Rank

Ward Ed — Wed, 21 Oct 2026 16:05:19 +0000

TL;DR

On October 22, 2026 the Hugging Face official GPQA Diamond leaderboard returned 115 rows for 97 repos, and 0 rows marked verified. 60 rows link to a model card, 22 come from an EvalEval source, 33 have another or no source.
GPQA Diamond has 198 questions. One question is 0.505 points. A score of 93.5% has a 95% sampling interval of roughly 89.2 to 96.2, so a model at 90.1% and a model at 93.5% are not separated by this test alone.
Of 93 rows whose values are rounded to two decimals or fewer, 57 cannot be a single pass over 198 questions (for example 93.5 is not a rounding of 185/198 = 93.43 or 186/198 = 93.94). Those rows are averages over runs or a different protocol. 43 of the 57 have an empty notes field, and only one says it is an average.
In the same Hugging Face repo, one model's score spans 67.1 to 80.1 across reasoning-effort settings on the model card. That is a bigger spread than the entire gap between rank 2 and rank 15.
In an illustrative simulation, a 2-point gap (92% vs 90%) survives a fresh draw of 198 questions about 77% of the time. It takes roughly 800 questions to reach 95%.

What is on the Hugging Face GPQA leaderboard?

The dataset repo Idavidrein/gpqa carries the tags benchmark:official and benchmark:eval-yaml, which mark it as an official benchmark that accepts eval YAML files. Models report results through .eval_results/*.yaml files in their own repos or through pull requests. The leaderboard shows what those files say.

I pulled the JSON the leaderboard view uses. I did not find documentation for that endpoint, so treat it as unstable. The snapshot (October 22, 2026, KST) looks like this:

Property	Count
Rows	115
Distinct repos	97
Rows with `verified: true`	0
Source name "Model Card"	60
Source name "EvalEval"	22
Other or no source	33
Rows with a free-text `notes` field filled in	63

Zero verified rows does not mean the numbers are wrong. It means the board is a collection of claims, and the only checks are the ones you do yourself. Everything below is about what you can check from the rows alone.

A roundup check first. An open-LLM roundup updated October 13, 2026 lists GPQA Diamond for eight models: 93.5 (Kimi K3), 91.2 (GLM-5.2), 90.5 (Kimi K2.6), 90.4 (Hunyuan Hy3), 90.1 (DeepSeek-V4-Pro), 88.1 (DeepSeek-V4-Flash), 87.8 (Qwen3.6-27B) and 87.0 (Nemotron 3 Ultra). All eight equal the model-card row on the Hugging Face board to the decimal. The roundup does not say where its numbers come from, but this match makes self-reported figures the likely source. I will use those eight as the running example, because they are the numbers people actually quote.

How much can 198 questions separate?

Treat each question as a coin flip with the model's true accuracy. The standard error of a score p over n = 198 questions is sqrt(p(1-p)/n). At p = 0.90 that is 2.1 points, and the 95% Wilson interval is about plus or minus 4 points.

Model (self-reported)	Score	95% interval over 198 questions
Kimi K3	93.5	89.2 to 96.2
GLM-5.2	91.2	86.4 to 94.4
Kimi K2.6	90.5	85.6 to 93.8
Hunyuan Hy3	90.4	85.5 to 93.8
DeepSeek-V4-Pro	90.1	85.1 to 93.5
DeepSeek-V4-Flash	88.1	82.8 to 91.9
Qwen3.6-27B	87.8	82.5 to 91.7
Nemotron 3 Ultra	87.0	81.6 to 91.0

Every interval overlaps with at least its neighbours. Overlap is a conservative test, and the better question is about the gap. Two models answer the same 198 questions, so their errors are correlated: a question that is hard for one is usually hard for the other. A correlation rho between the two models' per-question outcomes shrinks the noise in the gap to sqrt(2p(1-p)(1-rho)/n). The smallest gap that clears a 95% test is then:

True accuracy	rho = 0	rho = 0.5	rho = 0.8
80%	7.9	5.6	3.5
90%	5.9	4.2	2.6
93.5%	4.9	3.4	2.2

Nobody publishes the per-question correlation between two vendors' runs, so rho is an assumption. Even at an optimistic 0.8, a 90% model needs a 2.6 point lead before the order is statistically readable. On the board, 13 rows sit at or above 90.3, which is the lower end of the top row's own interval. The top of this board is a cluster, not a ladder.

Would the order survive a fresh draw of questions?

A simulation makes the point more concretely. I drew 198 questions with a shared difficulty factor (a latent Gaussian copula), set the accuracies to the reported values, and counted how often model A scored strictly higher than model B. These are illustrative numbers under the assumed correlation, not measurements of any real model pair.

Pair (reported scores)	Order holds, rho = 0.5	Order holds, rho = 0.8
93.5 vs 91.2	81.0%	85.9%
93.5 vs 90.1	90.8%	94.8%
91.2 vs 90.1	62.6%	66.0%
90.1 vs 88.1	74.4%	78.7%
93.5 vs 87.0	99.3%	99.9%

The 6.5 point gap between the first and last model is safe. Everything adjacent is closer to a coin flip with a thumb on the scale. Ties, which are up to 9% of draws here, count against "A is ahead" in this table.

The next chart holds the accuracy at 90%, sets rho to 0.6, and asks how many questions a benchmark needs.

For a 2-point gap the answer is about 800 questions (94.6%) and for a 1-point gap more than 1,600 (87.8%). GPQA Diamond was built as a hard, expert-validated subset, so its small size is a design choice, not a mistake. The consequence is that it was never built to rank models 2 points apart.

Are these rows single runs over 198 questions?

This is the check I find most useful, and it takes four lines of code. If a row is one pass over the 198 questions, its value must be k/198 for an integer k. The rows from EvalEval are: all 22 have long decimals such as 87.8787878788, which is exactly 174/198. Their values come straight from a count.

The other rows are rounded by their authors. For a value rounded to two decimals, the chance of landing on a k/198 value by coincidence is about 2%, so a hit is meaningful. For one decimal it is about 20%, so a hit proves little, but a miss is conclusive:

93 rows have two decimals or fewer.
36 of them are consistent with one pass over 198 questions.
57 are not.

The headline example is 93.5. The nearest single-pass values are 185/198 = 93.43 and 186/198 = 93.94. A reported 93.5 therefore comes from an average over several runs, a different item count, or a different aggregation. The row's notes field is empty, so the board does not say which. Across all 57 off-grid rows, 43 have no notes and only one ("Avg@4") names an averaging protocol.

Why it matters: an average over several samples has less noise than a single run, but a majority vote over up to 16 samples or a larger thinking budget is a different measurement. Notes on the board range from "greedy decoding, single sample" to "majority vote over up to 16 samples" to "Avg@4". They all land in one column and get ranked together. I cannot tell you which protocol is best. I can tell you the column does not hold one quantity.

Same repo, different setting: how big can the gap be?

Fourteen rows belong to the two gpt-oss repos: each model card reports three reasoning-effort levels, each with and without tools (six rows), plus one EvalEval row per repo. Without tools, the model-card values are:

Reasoning effort	gpt-oss-120b	gpt-oss-20b
low	67.1	56.8
medium	73.1	66.0
high	80.1	71.5

Same weights, spread of 13.0 points for the larger model and 14.7 for the smaller. For comparison, the whole distance from rank 2 (93.5) to rank 15 (89.5) on the board is 4.0 points. A setting you did not read can move a score more than every model gap in the top 15.

The EvalEval rows show the same trap from the other side. For gpt-oss-120b, EvalEval reports 80.81, which sits next to the "high" model-card value (80.1). For gpt-oss-20b, EvalEval reports 58.59, which sits next to the "low" values (58.0 and 56.8) and 12.9 points below the "high" card value. The EvalEval row records no reasoning effort, so a naive comparison would conclude that the 20b card is inflated by 13 points. More likely the two sources ran different settings. I cannot confirm that from the rows, and that is the point.

Do self-reports run higher than independent re-measurements?

Four repos have both a model-card row and an EvalEval row, with a setting I can match:

Repo	Model card	EvalEval (single pass)	Difference
Kimi-K2.5	87.60	87.37	+0.23
Qwen3.5-122B-A10B	86.60	84.85	+1.75
Qwen3.5-27B	85.50	81.82	+3.68
gpt-oss-120b (high, no tools)	80.10	80.81	-0.71

The mean difference is +1.24 points, positive in three of four. A single pass over 198 questions at 85% accuracy has a standard error of 2.5 points, so only the Qwen3.5-27B gap (1.5 standard errors) is anywhere near notable, and 4 repos cannot tell a 1 point inflation from none. I read this as "no evidence of large inflation, and no power to see a small one". It is not a verdict on any vendor.

Do repeated numbers mean copied numbers?

Seventeen values appear under more than one repo. Most are coincidences at a 0.1 resolution among 115 rows, or sibling checkpoints. Three look different:

zai-org/GLM-5.1 and two MXFP4 AutoRound re-uploads (INCModel3/... and intel-ai/...) all list exactly 86.2.
nvidia/...Nemotron-3-Super-120B-A12B-BF16 and RedHatAI/...Nemotron-3-Super-120B-A12B-BF16 both list the same two values, 82.7 and 79.23.

A quantized checkpoint reproducing its parent's score to the decimal is more plausibly a copied figure than a re-measurement. That is an inference, since the rows do not say. If true, those rows inflate the count of "models above 86" without adding evidence.

Is there headroom left to rank at the top?

GPQA Diamond is a ceiling-limited test. The original paper reports expert accuracy of 65% on the full 448-question set, 74% when excluding errors the experts later identified. An Epoch AI analysis from May 30, 2025 hand-reviewed six of the hardest Diamond questions, judged about 2.25 of them invalid, and extrapolated to roughly 8% of the benchmark (about 15 questions). That extrapolation rests on six questions and carries a wide error bar.

Take it at face value anyway. Fifteen invalid questions would put the ceiling at 183/198 = 92.4%, and 5 of the 115 rows are above it, including the top row at 94.44 (187/198). A model can match the key on a flawed question, for example by reproducing the same common mistake as the answer key, so being above the "valid ceiling" is not proof of anything wrong. It does mean that at the very top, a point or two measures key agreement as much as science knowledge.

A six-step check before quoting a GPQA number

Read the notes field. If it is empty, you do not know the protocol.
Test the grid. If the value is not k/198 (and it has at most two decimals), it is not one pass.
Find the setting: reasoning effort, tools, temperature, number of samples. Compare only matching settings.
Attach an interval. For n = 198 use plus or minus 4 points near 90%.
Check whether the repo is a quantized re-upload of a model already on the board.
Say "tied within noise" when the gap is under about 3 points, unless you have per-question data.

The script below reproduces the board statistics, the grid test and the interval table. It needs numpy only for the simulation, which I left out here for brevity.

import json, math, urllib.request, collections
URL = "https://huggingface.co/api/datasets/Idavidrein/gpqa/leaderboard"
rows = json.load(urllib.request.urlopen(
    urllib.request.Request(URL, headers={"User-Agent": "Mozilla/5.0"}), timeout=60))
N = 198

def decimals(v):
    s = repr(float(v)); return len(s.split(".")[1].rstrip("0")) if "." in s else 0

def on_grid(v):
    d = min(decimals(v), 2)
    return any(round(100 * k / N, d) == round(v, d) for k in range(N + 1))

def wilson(p, n=N, z=1.96):
    c = (p + z*z/(2*n)) / (1 + z*z/n)
    h = z * math.sqrt(p*(1-p)/n + z*z/(4*n*n)) / (1 + z*z/n)
    return round(100*(c-h), 1), round(100*(c+h), 1)

print(len(rows), len({r["modelId"] for r in rows}), sum(r["verified"] for r in rows))
low = [r for r in rows if decimals(r["value"]) <= 2]
print("rounded rows:", len(low), "off the k/198 grid:", sum(not on_grid(r["value"]) for r in low))
print("93.5 ->", wilson(0.935))

Limitations

One snapshot, taken October 22, 2026. The board changes as pull requests merge.
The grid test cannot say why a value is off the grid. Averages, a different item count and partial-credit scoring all produce it.
The simulation assumes a latent Gaussian copula with rho set by hand. Real models are not Gaussian, and real rho is unknown.
The card-versus-EvalEval comparison has four repos and a single EvalEval pass each.
I ran no model. Every score is as published.

FAQ

How many questions are in GPQA Diamond?

It is the hardest subset of the 448-question GPQA set, kept to questions where both expert validators answered correctly. One question is 0.505 points of accuracy.

Is a 93.5% GPQA Diamond score better than 90.1%?
Probably, but not by this test alone. Over 198 questions the 95% interval for 93.5% runs from about 89.2 to 96.2, which contains 90.1. In the simulation above, the order holds in 91% to 95% of fresh draws depending on the assumed correlation.

What does "verified: false" on the Hugging Face board mean?
Every one of the 115 rows had it, so the board gave no independent confirmation for any score. The rows record a claim and its source link, not a reproduction.

Why would a GPQA score not be a multiple of 1/198?
A single pass over the 198 questions produces only multiples of 1/198 (0.505%). A value off that grid means averaging over several samples, another item count, or a different aggregation.

How should I compare two open models on GPQA Diamond?
Match the setting first: reasoning effort, tools, temperature, number of samples. Then compare with an interval. If the gap is under about 3 points, call it a tie unless you have per-question results for both models.

Sources

Hugging Face, Idavidrein/gpqa dataset and its leaderboard data (retrieved October 22, 2026)
Rein et al., GPQA: A Graduate-Level Google-Proof Q&A Benchmark (arXiv 2311.12022)
Epoch AI, GPQA Diamond: What's Left? (May 30, 2025)
Epoch AI, GPQA Diamond benchmark page
Onyx, Best Open Source LLMs in 2026 (updated October 13, 2026)
Hugging Face, moonshotai/Kimi-K3

Do 66 Terminal Tasks and a 45% Private Weight Settle AI Rankings? Reading Artificial Analysis Index v4.3

Ward Ed — Tue, 20 Oct 2026 13:50:27 +0000

TL;DR

Artificial Analysis Intelligence Index v4.3 (launch article dated September 7, 2026) raised the weight of evaluations with private tasks or answers from 40% to 45%. In the published weight table, four of the ten evaluations carry that 45%: AA-Briefcase (15), AutomationBench-AA (5), AA-Omniscience (10 + 5) and CritPt (10).
Private test sets attack one problem, gaming and contamination. They do not touch another: sampling noise. Terminal-Bench 4.0 has 66 tasks, each run three times, with average pass@1 reported.
With 66 tasks, one model's score has a standard error of roughly 3.5 to 6 points, depending on how task difficulty is distributed. The reported 7.1 point gap between the top two (59.1 vs 52.0) is only about 1.2 to 1.5 standard deviations of the simulated gap (4.5 to 5.8 points).
In an illustrative simulation (assumptions below), the top-two order survived a fresh 66-task draw in 88% to 93% of runs. The Fable 5.1 vs Opus 5 order (3.0 points) survived only 69% to 74%. The 19.2 point gap to GPT-5.6 Sol survived in about 100% of runs.
Practical rule: treat a gap as a ranking only when you can say how many tasks produced it. A short script below reproduces every number.

What exactly changed in Index v4.3?

The launch article lists three changes relative to v4.2. Terminal-Bench moved from 2.1 to 4.0. AutomationBench-AA replaced the tau-cubed Banking evaluation at its 5% weight. And because AutomationBench-AA uses a held-out test set that Artificial Analysis runs in collaboration with Zapier, the share of weight on private evaluations rose from 40% to 45%. Category weights stayed at Agents 30%, Coding 20%, General 30%, Scientific Reasoning 20%.

The page's weight table (version label v4.3.2 when I read it on October 20, 2026) lets you check the 45% yourself:

Category	Evaluation	Private	Weight
Agents	AA-Briefcase	Yes	15
Agents	GDPval-AA	No	10
Agents	AutomationBench-AA	Yes	5
Coding	Terminal-Bench 4.0	No	10
Coding	SciCode	No	10
General	AA-Omniscience Accuracy	Yes	10
General	AA-Omniscience Non-hallucination	Yes	5
General	GDP.pdf	No	10
General	AA-LCR	No	5
Scientific	HLE (Humanity's Last Exam)	No	10
Scientific	CritPt	Yes	10

Private rows sum to 15 + 5 + 10 + 5 + 10 = 45. Public rows sum to 55. The table has 11 rows because AA-Omniscience is scored in two parts, which is why the article counts 10 evaluations.

One inference to flag: the article says private weight was 40% in v4.2, and AutomationBench-AA (private) took over a 5% slot. So the retired tau-cubed Banking slot must have been public. The article does not say this in so many words.

Look at the coding category. It carries 20 points and none of them is private. The one coding benchmark upgraded in this release, Terminal-Bench 4.0, is also a public one, and it is the one with the small task count.

Does a private test set make a ranking more reliable?

It makes one kind of error less likely. If questions and answers never leave the evaluator, a lab cannot train on them by accident, and a model cannot memorize them from a scraped copy. That is the stated reason: the launch article says the changes add more private test sets "to prevent gaming".

It does not shrink the interval around a score. The sampling error of a pass rate depends on the number of tasks, whether or not those tasks are secret. A private set of 40 tasks is a good instrument for detecting leakage and a blunt one for ranking models that differ by a few points.

There is also a transparency cost that is easy to overlook. With a public benchmark, anyone can re-run it, rebuild the paired per-task table and compute a proper paired interval. With a private one, only the operator can. For the 45% of the index that is private, readers have to trust the operator's variance reporting, because nobody else can measure it. That is a trade, not a flaw, but it belongs on the list when you decide how much a 2-point index gap means.

How noisy is a 66-task benchmark?

The launch article says all 66 Terminal-Bench 4.0 tasks are run three times and the average pass@1 is reported. Reported scores for four models: GPT-6 Astra (max) 59.1%, Claude Fable 5.1 (max with fallback) 52.0%, Claude Opus 5 (max) 49.0%, GPT-5.6 Sol (max) 39.9%. The article itself states Astra's 19.2 point lead over Sol.

Two sources of noise matter, and they answer different questions.

Run noise. If you re-ran the same 66 tasks, how much would the score move? Averaging three runs reduces this.
Task-sampling noise. If someone built a different 66 tasks from the same pool of terminal work, how much would the score move? Averaging runs does nothing here, and for hard agentic tasks it is typically the larger term.

Bounds for a pass rate p over 66 tasks with 3 runs each:

Lower bound (every task has the same pass probability, so only run noise exists): SE = sqrt(p(1-p) / (66 x 3))
Upper bound (each task is effectively deterministic, so all variance is task sampling): SE = sqrt(p(1-p) / 66)

import numpy as np
N_TASKS, RUNS = 66, 3

def se_bounds(p):
    hi = np.sqrt(p*(1-p)/N_TASKS)
    lo = np.sqrt(p*(1-p)/(N_TASKS*RUNS))
    return lo*100, hi*100

for name, p in [("Astra", .591), ("Fable 5.1", .520), ("Opus 5", .490), ("Sol", .399)]:
    lo, hi = se_bounds(p)
    print(f"{name}: SE {lo:.1f} to {hi:.1f} pts, 95% half-width {1.96*lo:.1f} to {1.96*hi:.1f}")

Output from my run:

Astra: SE 3.5 to 6.1 pts, 95% half-width 6.8 to 11.9
Fable 5.1: SE 3.6 to 6.1 pts, 95% half-width 7.0 to 12.1
Opus 5: SE 3.6 to 6.2 pts, 95% half-width 7.0 to 12.1
Sol: SE 3.5 to 6.0 pts, 95% half-width 6.8 to 11.8

Real tasks are heterogeneous: some are nearly always solved, some nearly never, which pushes the true SE toward the upper bound. The chart at the top of this article uses the upper bound, which is why the bars for neighboring models overlap heavily.

Overlapping intervals are a conservative test, though. The models are scored on the same 66 tasks, so a hard task is hard for everyone, and that shared difficulty cancels out in the gap. A paired comparison is tighter than two separate intervals suggest. How much tighter depends on how correlated task difficulty is across models, and that is exactly the number outsiders cannot see.

Which of the reported gaps would survive a fresh task draw?

I simulated it. The setup is deliberately simple and every assumption is labeled, because the per-task results are not public.

Each model has a per-task pass probability drawn from a Beta distribution whose mean equals the reported score. The concentration parameter kappa controls spread: kappa = 1 gives many near-0 and near-1 tasks, kappa = 3 is milder.
Task difficulty is correlated across models through a Gaussian copula with correlation rho (I tried 0.5 and 0.8).
Each model runs each task three times (binomial draws), and the score is the mean.
I redraw 66 tasks 3,000 times and count how often the observed order matches the order of the reported scores.

import numpy as np
from scipy.stats import norm, beta
rng = np.random.default_rng(20261020)
RUNS = 3

def sim(pa, pb, rho, kappa, reps=3000, n=66):
    z1 = rng.standard_normal((reps, n))
    z2 = rho*z1 + np.sqrt(1-rho**2)*rng.standard_normal((reps, n))
    qa = beta.ppf(norm.cdf(z1), pa*kappa, (1-pa)*kappa)
    qb = beta.ppf(norm.cdf(z2), pb*kappa, (1-pb)*kappa)
    sa = rng.binomial(RUNS, qa)/RUNS
    sb = rng.binomial(RUNS, qb)/RUNS
    gap = (sa.mean(axis=1) - sb.mean(axis=1))*100
    return (gap > 0).mean(), gap.std()

Results for 66 tasks (share of fresh draws that keep the reported order):

Pair (reported gap)	rho 0.5, kappa 1	rho 0.5, kappa 3	rho 0.8, kappa 1	rho 0.8, kappa 3
Astra vs Fable 5.1 (7.1)	88.3%	91.4%	93.0%	93.2%
Fable 5.1 vs Opus 5 (3.0)	68.5%	71.3%	74.1%	73.0%
Astra vs Sol (19.2)	99.9%	100.0%	100.0%	100.0%

The standard deviation of the gap in these runs was 4.5 to 5.8 points. So:

A 19-point gap is a ranking. It held under every setting I tried.
A 7-point gap is probably a ranking, not a certainty. Between 7% and 12% of fresh draws flipped it, and with weaker difficulty correlation than I assumed it would flip more often.
A 3-point gap is closer to a coin flip with a thumb on the scale. The order held about 70% of the time, better than chance but far from what a leaderboard row implies.

These are not estimates of the true probabilities. They show the order of magnitude you should expect at n = 66.

How many tasks would a 7-point gap need?

Same simulation, Astra vs Fable 5.1, varying the task count (kappa = 1):

Tasks	order kept (rho 0.5)	order kept (rho 0.8)
66	88.5%	93.6%
100	94.0%	97.3%
150	96.7%	98.7%
200	98.1%	99.7%
300	99.8%	100%
600	100%	100%

Each row used 2,000 redraws, so expect about 1 point of simulation noise. Reaching 95% takes somewhere between 75 and 110 tasks under these assumptions, so a 7-point gap is only slightly under-powered at 66. A 3-point gap would need far more: required task count grows with the inverse square of the gap, so halving the gap roughly quadruples the tasks.

What does this do to the index itself?

The headline index scores in the launch article are integers: 53 for both Claude Fable 5.1 and GPT-6 Astra, 51 for Claude Opus 5, 50 for Claude Fable 5, 48 for Muse Spark 1.3, 47 for GPT-5.6 Sol. Among open-weight models the article lists GLM-5.3 Flash at 42, Qwen3.8 2.4T A95B at 40 and DeepSeek V4 Pro 0813 (max) at 36. These are launch-day numbers, and the live leaderboard has moved since.

Terminal-Bench carries 10% of the index. If component scores enter on a 0 to 100 scale (my assumption, since the article does not give the aggregation formula), a Terminal-Bench gap standard deviation of about 5 points moves the index by about 0.5 points from that component alone. Every other evaluation adds its own noise. I could not size the private ones, because the article gives no task counts or per-task results for them beyond the 657 tasks of AutomationBench-AA.

The practical reading: a 2-point index gap between two open-weight models, such as 42 versus 40, is in the range where component sampling noise could plausibly account for the difference. A 6-point gap, such as 42 versus 36, is much harder to explain that way.

A checklist before you quote a leaderboard gap

Find the task count behind each component. If the page does not say, treat the number as unsized.
Separate run noise from task noise. "Averaged over three runs" fixes the first one only.
Convert to a standard error. For a pass rate, sqrt(p(1-p)/n) is the upper bound. It takes ten seconds.
Ask for paired results. If the operator publishes per-task outcomes, a paired test beats overlapping bars.
Note the version. This index moved from v4.2 to v4.3 and the page now reads v4.3.2. Scores across versions are not a time series.
Discount private weight honestly. It protects against gaming. It does not buy precision, and it is not independently auditable.

FAQ

Is the Artificial Analysis Index v4.3 unreliable?
No. Adding private test sets is a reasonable design move. The point here is narrower: a published gap needs a task count before you read it as a ranking, and Terminal-Bench 4.0 has 66 tasks.

What percentage of the Index v4.3 is based on private test sets?
45%, up from 40% in v4.2, according to the launch article. In the weight table that is AA-Briefcase (15), AutomationBench-AA (5), AA-Omniscience Accuracy (10), AA-Omniscience Non-hallucination (5) and CritPt (10).

How many tasks does Terminal-Bench 4.0 have in this index?
66, each run three times, with the average pass@1 reported, per the launch article.

Do the simulation numbers prove the rankings are wrong?
No. They illustrate behavior under stated assumptions (Beta task difficulty, Gaussian copula correlation of 0.5 to 0.8, three binomial runs). Per-task results are not public, so the true correlation is unknown. The takeaway is the order of magnitude: about 5 points of gap noise at 66 tasks.

What is the difference between run noise and task-sampling noise?
Run noise is variation when the same tasks are re-run. Task-sampling noise is variation from which tasks happened to be in the set. Averaging more runs reduces only the first.

Can I rely on overlapping error bars to decide if two models differ?
Only as a conservative screen. When both models are scored on the same tasks, a paired analysis is tighter, which is why publishing per-task outcomes would help readers more than a wider table.

Best Open-Weight LLM in October 2026? Auditing Two Leaderboards Row by Row

Ward Ed — Mon, 19 Oct 2026 13:46:15 +0000

TL;DR

On October 19, 2026, BenchLM and Artificial Analysis both put MiMo-V2.6-Pro first among open-weight models (73.7 on BenchLM, Intelligence Index 46.3 on Artificial Analysis).
That agreement is thinner than it looks. BenchLM tags the #1 row Estimated with a conditional range of 63.92 to 83.37, and the ranges of the top nine rows all overlap each other.
Below #1 the orderings drift: on the nine models I could match, Kendall's tau is 0.61 (29 concordant pairs, 7 discordant). DeepSeek V4.1 Flash 1017 moves from 6th to 3rd.
The bigger disagreement is membership. Kimi K3 is missing from BenchLM's open list (weights marked "pending") while its Hugging Face repo is public with 96 safetensors shards. Qwen3.8 Max is #2 on BenchLM's open list but flagged not open-weights on Artificial Analysis.
Before you cite "best open model", audit the label, the evidence tag, and the checkpoint name. A short script for each is below.

All numbers are from pages and APIs I pulled on October 19, 2026. Leaderboards change daily, so treat this as a dated snapshot.

Which open-weight model is #1 right now?

Searching for "best open source LLM" returns several aggregators that look alike but compute different things. I pulled two of them directly:

BenchLM publishes a composite called BenchAlign v5.8. Its open-source page ranks 103 open-weight models out of 955 tracked models and 640 tracked benchmarks.
Artificial Analysis publishes an Intelligence Index. The page ships its row data as JSON, with fields such as intelligenceIndex, isOpenWeights, hle, gpqa, scicode, terminalBench21 and tau2.

Both lists put MiMo-V2.6-Pro on top. That is the headline most people will quote. Here are the BenchLM top six as published:

Rank	Model	Score	Evidence tag	Published range
1	MiMo-V2.6-Pro	73.7	Estimated	63.92 to 83.37 (conditional)
2	Qwen3.8 Max	69.7	Supported	65.62 to 73.78 (90% interval)
3	Step 5 Preview	69.5	Estimated	59.80 to 79.24 (conditional)
4	Ling 3.1 Flash	68.1	Estimated	53.73 to 82.43 (conditional)
5	GLM-5.3	67.9	Supported	63.45 to 72.40 (90% interval)
6	DeepSeek V4.1 Flash 1017	65.6	Supported	60.65 to 70.62 (90% interval)

How wide is the uncertainty behind that #1?

The ranges are the part of the page that most readers skip. BenchLM's methodology page defines the two tags in one sentence: "Supported means the row clears the independent-evidence rules. Estimated means it relies on thinner evidence." The #1 row is in the second group.

Three things follow from the published numbers:

The #1 range is 19.45 points wide. The Supported rows in the same table are 8.2 to 10.0 points wide.
Every pair overlaps. Among the nine models I matched across both leaderboards (more on that below), all 36 pairs of ranges overlap. Including the tenth BenchLM row, it is 45 of 45.
BenchLM says so itself. The page states that MiMo-V2.6-Pro "has the higher point estimate" and that "conditional ranges do not establish rank confidence."

So the correct reading is "MiMo-V2.6-Pro has the highest point estimate among open-weight rows, on thin evidence." That is a weaker claim than "MiMo-V2.6-Pro is the best open model", and it is the only one the page supports.

Note also that a conditional range is not a 90% interval. The two kinds of row use different constructions, so comparing the width of an orange bar with a blue bar is informative but not apples to apples.

Do the two leaderboards agree below rank 1?

Partly. I matched BenchLM's top ten open rows against Artificial Analysis rows by model name. Nine matched; "dots3-note Preview" has no row on Artificial Analysis in the data I pulled. Ranking only those nine by each source gives:

Model	BenchLM rank (of 9)	Artificial Analysis rank (of 9)	AA Intelligence Index
MiMo-V2.6-Pro	1	1	46.3
Qwen3.8 Max	2	2	45.4
Step 5 Preview	3	5	43.7
Ling 3.1 Flash	4	6	41.1
GLM-5.3	5	4	44.8
DeepSeek V4.1 Flash 1017	6	3	45.1
MiMo-V2.6-Flash	7	8	37.9
DeepSeek V4 Pro 0813	8	9	36.0
Qwen3.8-Flash-Next	9	7	39.8

Rank correlation is a one-screen computation. This script reproduces the numbers above:

from itertools import combinations

# name: (benchlm_score, benchlm_low, benchlm_high, aa_index)
rows = {
    "MiMo-V2.6-Pro":            (73.7, 63.92, 83.37, 46.3),
    "Qwen3.8 Max":              (69.7, 65.62, 73.78, 45.4),
    "Step 5 Preview":           (69.5, 59.80, 79.24, 43.7),
    "Ling 3.1 Flash":           (68.1, 53.73, 82.43, 41.1),
    "GLM-5.3":                  (67.9, 63.45, 72.40, 44.8),
    "DeepSeek V4.1 Flash 1017": (65.6, 60.65, 70.62, 45.1),
    "MiMo-V2.6-Flash":          (64.5, 54.82, 74.26, 37.9),
    "DeepSeek V4 Pro 0813":     (63.2, 48.81, 77.51, 36.0),
    "Qwen3.8-Flash-Next":       (62.8, 48.48, 77.18, 39.8),
}

def kendall_tau(a, b):
    conc = disc = 0
    for i, j in combinations(range(len(a)), 2):
        s = (a[i] - a[j]) * (b[i] - b[j])
        conc += s > 0
        disc += s < 0
    return (conc - disc) / (conc + disc), conc, disc

def overlapping_pairs(ranges):
    n = sum(1 for (l1, h1), (l2, h2) in combinations(ranges, 2) if l1 <= h2 and l2 <= h1)
    return n, len(ranges) * (len(ranges) - 1) // 2

bl = [v[0] for v in rows.values()]
aa = [v[3] for v in rows.values()]
tau, conc, disc = kendall_tau(bl, aa)
print(f"Kendall tau={tau:.3f} concordant={conc} discordant={disc}")
print("range overlaps:", overlapping_pairs([(v[1], v[2]) for v in rows.values()]))

Output: Kendall tau=0.611 concordant=29 discordant=7 and range overlaps: (36, 36).

A tau of 0.61 on nine items is "broadly similar", not "the same". The practical consequence is that the middle of the table is not stable. If your shortlist cutoff is "top 5 of these nine", then Ling 3.1 Flash is in on BenchLM and out on Artificial Analysis, while DeepSeek V4.1 Flash 1017 is the reverse.

Artificial Analysis has its own tight cluster. Its open-weight top five are 46.3, 45.1, 44.8, 43.7 and 43.6, so adjacent gaps are 1.2, 0.3, 1.0 and 0.1 index points, and the whole top five spans 2.7 points. The page I pulled shows one number per row, so I cannot tell you whether a 0.1 gap is real. Neither can the table.

Why is Kimi K3 missing from one "open" list and Qwen3.8 Max flagged closed on the other?

This is the finding I would put ahead of the rank noise. "Open-weight" is a label that each aggregator assigns, and the labels disagree.

Kimi K3. On BenchLM it scores 69.92 and ranks #18 of 226 ranked models overall, #14 of 88 on source-verified rows. Its profile says "Weight access pending" and "no verified publication yet for weight access". It is therefore absent from the 103-row open list. By point estimate it would sit at #2 there, just ahead of Qwen3.8 Max at 69.7. Its profile also notes that only 56 of 682 tracked benchmark slots have displayable evidence, so the score itself rests on partial coverage.

On Artificial Analysis the same model, "Kimi K3 (Max)", is flagged isOpenWeights: true with an Intelligence Index of 43.6. And on the Hugging Face Hub, moonshotai/Kimi-K3 is public and not gated, ships 96 safetensors shards, carries a custom license named kimi-k3 (the Hub tag is license:other), and showed 1,012,867 downloads when I queried it.

Qwen3.8 Max. BenchLM lists it as Open Weight at #2. Artificial Analysis lists "Qwen3.8 Max (0902)" with an index of 45.4 and isOpenWeights: false. I could not resolve Qwen/Qwen3.8-Max or Qwen/Qwen3.8-Max-0902 on the Hub API from an anonymous request (HTTP 401, which the Hub also returns for repos that do not exist). The two leaderboards may be naming different snapshots, which is a problem in itself: the same display name without a checkpoint identifier is not a stable key.

MiMo-V2.6-Pro. The Hub has XiaomiMiMo/MiMo-V2.6-Pro-RL (MIT license, public, 132 safetensors files). A repo named exactly MiMo-V2.6-Pro does not resolve anonymously. Whether the benchmarked row is the -RL checkpoint is not stated on either leaderboard page I read.

Here is the check I ran, so you can repeat it on any model you plan to cite:

import json
import urllib.error
import urllib.request

def hub_status(repo: str) -> dict:
    url = f"https://huggingface.co/api/models/{repo}"
    req = urllib.request.Request(url, headers={"User-Agent": "leaderboard-audit"})
    try:
        with urllib.request.urlopen(req, timeout=30) as r:
            info = json.load(r)
    except urllib.error.HTTPError as e:
        return {"repo": repo, "resolves": False, "http": e.code}
    files = [s["rfilename"] for s in info.get("siblings", [])]
    card = info.get("cardData") or {}
    return {
        "repo": repo,
        "resolves": True,
        "gated": info.get("gated"),
        "license": card.get("license"),
        "safetensors_files": sum(f.endswith(".safetensors") for f in files),
        "downloads": info.get("downloads"),
    }

for repo in ["moonshotai/Kimi-K3", "Qwen/Qwen3.8-Max",
             "XiaomiMiMo/MiMo-V2.6-Pro", "XiaomiMiMo/MiMo-V2.6-Pro-RL"]:
    print(hub_status(repo))

On October 19 it printed resolves: True, gated: False, license: other, safetensors_files: 96 for Kimi K3, resolves: False, http: 401 for the two names without a public repo, and license: mit, safetensors_files: 132 for the -RL repo.

Are the two leaderboards independent evidence?

Not necessarily. BenchLM's methodology page says BenchAlign "calibrates benchmark protocols, human-preference signals, and external composite indices before combining them." If one of those external composite indices is the other leaderboard, then agreement between the two is partly circular, and a tau of 0.61 would overstate how much independent confirmation you have. The pages I read do not name the external indices, so I cannot tell either way. That is itself worth knowing: an agreement you cannot decompose is weaker evidence than a disagreement you can.

The inputs also differ in kind. Artificial Analysis publishes one index built from a fixed set of evaluations. BenchLM tracks 640 benchmarks, scores a subset of them across six pillars, keeps vendor-run numbers out of the rank, and fills gaps with a learned estimate. Different inputs plus similar outputs is mildly reassuring. Different inputs plus different open-weight labels is a reason to check the label yourself.

A five-step audit before you quote an open-model rank

Read the evidence tag. "Estimated" or "conditional range" means the position leans on thin evidence. Do not write "best" on top of it.
Compare ranges, not points. Run the overlap function above on the top rows. If everything overlaps, say "no resolvable difference".
Resolve the checkpoint. Find the exact Hub repo or API model ID the row refers to. Display names such as "Max" or "Pro" do not identify a checkpoint.
Verify the open-weight label. Check that weights exist, whether the repo is gated, and what the license actually allows. A custom license is not the same as MIT or Apache-2.0.
Cross-check one other source. Compute a rank correlation, then look at the individual disagreements, because the interesting rows are the ones that move.

Limitations of this audit

It is one snapshot, taken October 19, 2026. Both pages update continuously.
The Artificial Analysis values come from the row data embedded in its leaderboard page, not from a documented API. Parsing that JSON is fragile and I did not see an index version number on the page.
Name matching between sites was manual. Nine of ten BenchLM rows matched, and the ranks in the table are positions within those nine, not on the full lists.
Nine items is a small sample for a rank correlation. Treat tau = 0.61 as a rough indicator.
I did not run any model myself. Every score is as published by the aggregators.

FAQ

What is the best open-weight LLM in October 2026?
It depends on the leaderboard and on how you define open. MiMo-V2.6-Pro has the top point estimate on both BenchLM (73.7) and Artificial Analysis (46.3), but BenchLM tags that row Estimated with a 19.45 point range, so the lead over the next rows is not statistically resolved.

Why does Kimi K3 not appear on BenchLM's open-source list?
BenchLM records its weight access as pending and says it has no verified publication of the weights, even though the Hugging Face repo is public. Artificial Analysis flags the same model as open-weights.

What does "Estimated" mean on a leaderboard row?
On BenchLM it means the row relies on thinner evidence than a "Supported" row, which clears the site's independent-evidence rules. The range shown for Estimated rows is a conditional range, which BenchLM states does not establish rank confidence.

How do I check whether a model is really open-weight?
Resolve its exact repo on the Hugging Face Hub, confirm the weight files exist and the repo is not gated, and read the license text. The hub_status function above automates the first three checks.

Is a rank correlation of 0.61 between two leaderboards good?
It means most pairs are ordered the same way (29 of 36 here) but a meaningful minority flip. It also may be inflated if one leaderboard uses the other as an input.

Sources

BenchLM, Open-Source LLM Leaderboard (data as of October 19, 2026)
BenchLM, Kimi K3 profile
BenchLM, Methodology
Artificial Analysis, LLM Leaderboard
Hugging Face, moonshotai/Kimi-K3
Hugging Face, XiaomiMiMo/MiMo-V2.6-Pro-RL

How to Read a Public Leaderboard Without Fooling Yourself

Ward Ed — Mon, 12 Oct 2026 10:36:15 +0000

TL;DR

A leaderboard is only as trustworthy as the way it was built and read. Four failure modes quietly turn rankings into noise: data leakage, saturation, selection on the evaluation set, and run-to-run variance. This is a field guide to spotting each one so a position on a board actually means something.

Leakage: the number that was never earned

The fastest way to top a board is to have seen it. If any evaluation item, or a near duplicate, appears in training, the score reflects memorization, not skill. Leakage is often accidental: scraped corpora overlap with public test sets more than teams expect.

How to check. Look for held-out guarantees from the board operator, count unique items rather than rows, and be suspicious when a model is far better on a famous public set than on a fresh private one of similar difficulty.

Saturation: when the ruler stops measuring

When the top entries all cluster near the ceiling, the benchmark has stopped discriminating. The differences between the leaders are within noise, and the ordering is close to random.

How to check. Look at the spread, not just the top value. If first and tenth are a hair apart, the board is saturated and the ranking carries little information. A good benchmark keeps separating strong systems from weak ones.

Selection on the eval set: inventing gains that are not there

If you tune choices by looking at the evaluation set, pick the best of many runs, and report only the winner, you will manufacture improvements that vanish on anything new. The eval set has quietly become part of training.

How to check. Ask whether hyperparameters and variants were chosen on a separate split. A result that was selected on the test set is a story about that test set, not about the method.

Run-to-run noise: a positive margin is not a win

Many evaluations move several points between runs on small sets. A lead that fits inside that variance is not a lead.

How to check. Prefer paired comparisons on the same items, report confidence intervals, and treat a small positive margin as a hypothesis to test, not a victory to announce. On a hundred-item slice, a swing of several points is normal.

A short reading checklist

Is train disjoint from test, with a held-out guarantee?
Is the board still discriminating, or is it saturated?
Were choices made on a separate split, not the eval set?
Is the margin outside run-to-run noise, with an interval reported?
Does breadth across independent boards back up a single result?

If a ranking passes all five, believe it. If it fails one, read it as a claim, not a fact.

FAQ

Are leaderboards useless then? No. A board run by a third party on memorization-resistant questions is one of the best signals we have, as long as you read it with these checks.

Why does breadth matter? A single high score can come from luck or leakage. Consistent strength across independent boards is much harder to fake.

What is the most common mistake? Selection on the eval set, because it feels like honest tuning while quietly leaking the test into the method.

How big a margin should I trust? One that sits outside the run-to-run interval you measured. If you did not measure the interval, you cannot judge the margin.

Benchmark Contamination 101: How Train/Test Overlap Inflates Leaderboard Scores (and How to Catch It)

Ward Ed — Tue, 06 Oct 2026 19:11:46 +0000

TL;DR

A leaderboard number is only as trustworthy as the gap between what a model trained on and what it was tested on. When test examples (or near-duplicates of them) leak into pretraining or fine-tuning data, the model memorizes answers instead of generalizing, and the reported score climbs for the wrong reason. This is benchmark contamination, also called train/test overlap or data leakage. It is common, often accidental, and frequently invisible in a self-reported number. This post explains why contamination inflates scores, three detection methods you can run yourself (n-gram overlap, canary strings, membership inference), a runnable Python n-gram checker, and a short checklist you can apply before you trust any benchmark claim.

What is benchmark contamination and why does it inflate scores?

Modern models are trained on web-scale corpora scraped from the open internet. Public benchmarks live on that same internet: on GitHub, in papers, on leaderboard repos, in blog posts that quote questions verbatim, and in Q&A sites where people discuss the exact items. When a benchmark's test questions and answers end up in the training mix, evaluation stops measuring reasoning and starts measuring recall.

The mechanism is simple. A benchmark is supposed to be a held-out sample. If the test examples were in the training data, the model can reproduce the answer from memory. Memorization looks exactly like competence on a single scored run, but it does not transfer to genuinely new inputs. The score goes up; the underlying capability does not.

Contamination comes in degrees. Verbatim contamination means the exact test string appears in training. Near-duplicate contamination means a paraphrase, a translated copy, or a reformatted version appears. Label leakage means the answer key or solution is present even if the question wording differs. Even partial exposure helps a model disproportionately on the memorized slice, which is enough to move a leaderboard when margins are a point or two.

Why can self-reported leaderboard scores be inflated?

Self-reported scores carry three structural risks that have nothing to do with bad faith.

First, the submitter controls the evaluation environment: the prompt template, the decoding settings, the number of few-shot examples, and sometimes the subset of items. None of those choices are contamination, but they compound with it.

Second, nobody fully audits the training corpus. For most open releases the pretraining data is described at a high level, not published item by item. A team can honestly say they did not intend to train on a benchmark while still having ingested it through a scraped mirror. Intent does not change the outcome.

Third, benchmarks age. A dataset released three years ago has had three years to be copied, quoted, reformatted, and re-uploaded. The older and more popular a benchmark is, the more likely its items are somewhere in a modern crawl. This is why a model can post a record score on a classic benchmark and a mediocre score on a freshly built private one.

None of this means every high score is fake. It means a score reported without a contamination check is an unverified claim, and the burden of evidence sits with whoever is pointing at the number.

How do you detect train/test overlap? Three methods

1. N-gram overlap

The most direct test: take each benchmark item, slide a window of n consecutive tokens or words across it, and check whether those n-grams appear in the training corpus (or in a proxy for it). High overlap at n of 8, 13, or 50 is strong evidence that the item, or a chunk of it, was seen during training. This is the method used by several major lab technical reports, which typically flag an item as contaminated when a sufficiently long n-gram matches. The strength of n-gram overlap is that it is cheap, interpretable, and does not need model internals. Its weakness is that it misses paraphrases and translations, so it gives a lower bound on contamination, never an upper bound.

2. Canary strings

A canary is a unique, random, unlikely-to-occur-naturally identifier that benchmark authors embed in their dataset specifically so they can later test whether it leaked. If a model can reproduce or recognize the canary GUID, the dataset was in its training data, full stop. BIG-bench popularized this with an embedded canary GUID. The limitation is that canaries only work if the benchmark author planted one and the dataset was consumed with the canary intact; stripped or reformatted copies defeat it. Still, when a canary test fires, it is close to conclusive.

3. Membership inference and behavioral tests

When you cannot inspect the training data at all, you probe the model's behavior. Membership inference asks whether the model treats a specific example as something it has seen before. Practical variants include: comparing perplexity on benchmark items versus closely matched held-out items (memorized items often have suspiciously low loss), the guided-prompting trick (does the model complete a test item far better when primed with the dataset name?), and option-order sensitivity (a model that memorized a multiple-choice answer is unusually robust to the correct answer's position while a reasoning model is not). These tests are noisier than n-gram or canary checks and need careful controls, but they are the only options for closed training data.

A runnable n-gram overlap check

Here is a compact, dependency-free n-gram overlap checker. Give it your benchmark items and any text you suspect may overlap with training (a crawl shard, a scraped page, a quoted dataset). It reports the fraction of each item's n-grams that also appear in the reference text.

from collections import Counter

def ngrams(text, n=13):
    """Word-level n-grams, lowercased and whitespace-normalized."""
    toks = text.lower().split()
    return [tuple(toks[i:i + n]) for i in range(len(toks) - n + 1)]

def overlap_score(item, reference_ngrams, n=13):
    """Fraction of the item's n-grams that appear in the reference set."""
    item_grams = ngrams(item, n)
    if not item_grams:
        return 0.0
    hits = sum(1 for g in item_grams if g in reference_ngrams)
    return hits / len(item_grams)

def build_reference(corpus_texts, n=13):
    """Precompute the n-gram set for everything you can see of training data."""
    ref = set()
    for doc in corpus_texts:
        ref.update(ngrams(doc, n))
    return ref

if __name__ == "__main__":
    # Stand-in for a crawl shard / scraped page you suspect overlaps training.
    training_proxy = [
        "the capital of the fictional country of zubrowka is lutz "
        "and its currency is the klubeck used since the year 1932",
    ]
    benchmark_items = [
        "The capital of the fictional country of Zubrowka is Lutz "
        "and its currency is the Klubeck used since the year 1932",   # leaked
        "Explain why a positive confidence interval overlap means "
        "two leaderboard scores may not be distinguishable at all",   # clean
    ]

    N = 13
    ref = build_reference(training_proxy, n=N)
    for i, item in enumerate(benchmark_items):
        s = overlap_score(item, ref, n=N)
        flag = "CONTAMINATED" if s > 0.5 else "clean"
        print(f"item {i}: {s:.0%} {N}-gram overlap -> {flag}")

Expected output:

item 0: 100% 13-gram overlap -> CONTAMINATED
item 1: 0% 13-gram overlap -> clean

Two practical notes. First, choose n deliberately: small n (3 to 5) produces false positives on common phrasing, large n (20 or more) misses chopped-up leaks, and 8 to 13 is a reasonable default for prose. Second, this is a lower bound. A 0 percent score proves only that there was no verbatim long-span match against the text you checked; paraphrase and translation slip right past it, which is why you pair it with behavioral tests.

A practical contamination checklist

Run this before you cite or compare any benchmark number.

Is the benchmark newer than the model's data cutoff? A test built after training data was frozen cannot have leaked. Prefer recent or private benchmarks for headline comparisons.
Did the authors plant a canary, and did the reporter check it? A passed canary test is cheap evidence and should be mentioned.
Was an n-gram overlap report published against the training corpus (or a representative sample)? If the training data is open, this should exist. If it is not, note that the overlap is simply unknown.
Is there a gap between the model's score on the popular benchmark and on a fresh, structurally similar one? A large drop on the new test is a classic contamination signature.
Are decoding settings, prompt template, and few-shot count disclosed and held constant across the models being compared? Contamination aside, undisclosed harness choices make numbers incomparable.
Is the score reported with uncertainty, and is the margin over the next model larger than that uncertainty? A contaminated slice often shows up as a margin that vanishes under resampling.
Who ran the evaluation? Self-reported with no third-party reproduction is an unverified claim, not a result. Independent reproduction on held-out data is the gold standard.

If a claim fails several of these, the right posture is not "the model is cheating." It is "this number is unverified, and here is specifically what would make it trustworthy."

FAQ

Is benchmark contamination always intentional?
No, and usually it is not. The most common path is accidental ingestion through a web crawl, since benchmarks live on the same internet that pretraining corpora are scraped from. Accidental contamination inflates scores exactly as much as deliberate contamination, which is why detection matters more than assigning blame.

Does a high n-gram overlap score prove a model will fail on real tasks?
It proves the benchmark number is unreliable for that model, not that the model is incapable. The correct response is to re-evaluate on a clean, uncontaminated test and trust that number instead. Contamination invalidates a measurement; it does not by itself measure capability.

Why not just keep all benchmarks private?
Private benchmarks resist contamination but sacrifice reproducibility and community trust, since nobody else can inspect the items or rerun the evaluation. The common compromise is a public benchmark with a planted canary, a held-out private slice, and periodic refreshes so the test can be rebuilt once the old version has saturated the internet.

What n should I use for an overlap check?
For natural-language items, 8 to 13 words is a reasonable default. Go smaller and you flag ordinary phrasing as a match; go much larger and you miss leaks that were reformatted or chopped into pieces. Report the n you used, because the overlap fraction is meaningless without it.

Can I detect contamination without access to the training data?
Yes, but only with behavioral tests: perplexity gaps between benchmark items and matched held-out items, guided-prompting sensitivity, and answer-position robustness. These are noisier than n-gram or canary checks, so treat them as evidence that accumulates rather than a single decisive test.

When a 0.4-Point Lead Means Nothing: Reading Open-Model Leaderboard Margins Like a Statistician

Ward Ed — Tue, 06 Oct 2026 00:44:19 +0000

TL;DR

A leaderboard rank is a point estimate, not a fact. Every accuracy score carries a confidence interval, and on most public evals that interval is wide enough to swallow the gap between the top few models.
You can estimate the error bar yourself from two numbers you already have: the accuracy and the number of eval items. For a 300-item test at 70 percent accuracy, the 95 percent confidence half-width is about 5 points. A 0.4-point lead inside that band is noise.
For paired comparisons on the same questions, use McNemar's test, not two separate intervals. It is strictly more sensitive and it is what the gap actually deserves.
Before trusting any ranking, ask whether the benchmark can even discriminate. If every top model scores above 95 percent, the test is saturated and the order is close to random.
Popularity is not quality. The most-liked open models on the Hub (cover chart, pulled 2026-10-06) are a social signal, not an accuracy ranking. Treat the two as separate axes.

This is the second post in Leaderboard Forensics. The first covered how the Hub's benchmark plumbing actually works. This one is about the single most common misreading of those numbers: believing a rank order that the sample size cannot support.

Why do the top open models cluster so tightly?

Open-model evaluation has matured to the point where the frontier is crowded. On a typical reasoning or knowledge benchmark, the top ten entries often sit inside a three-point band. The cover chart above shows the eight most-liked open text-generation repositories on the Hugging Face Hub as of 2026-10-06, and likes are a popularity proxy, but the same crowding shows up in accuracy tables.

Here is the uncomfortable part. When scores cluster, the rank order is mostly determined by measurement noise rather than capability. The model in position one and the model in position four may be statistically indistinguishable. The leaderboard still has to print them in some order, so it prints the noise.

The curve above is illustrative. It plots the 95 percent confidence half-width for a model scoring 70 percent, as a function of how many questions the benchmark contains. At 100 items the half-width is about 9 points. At 1,000 items it drops to roughly 2.8 points. Only around 10,000 items does it fall under one point. Most popular public benchmarks live at the left end of that curve.

How do I compute the error bar on a single accuracy score?

Accuracy on a fixed test set is a binomial proportion. If a model answers k of n questions correctly, the estimated accuracy is p = k / n, and the standard error is sqrt(p * (1 - p) / n). A rough 95 percent interval is p plus or minus 1.96 * SE.

The Wald interval above is fine for a quick gut check, but it misbehaves near 0 and 100 percent and for small n. The Wilson interval is better behaved and still closed-form. Here is a compact, dependency-light helper.

import math

def wilson_interval(k, n, z=1.96):
    """95% Wilson score interval for a binomial proportion.
    k = correct answers, n = total eval items."""
    if n == 0:
        return (0.0, 0.0, 0.0)
    p = k / n
    denom = 1 + z * z / n
    center = (p + z * z / (2 * n)) / denom
    half = (z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n))) / denom
    return (p, center - half, center + half)

for n in (100, 300, 1000):
    p, lo, hi = wilson_interval(int(0.70 * n), n)
    print(f"n={n:5d}  acc={p:.3f}  95% CI=[{lo:.3f}, {hi:.3f}]  width={hi-lo:.3f}")

Output:

n=  100  acc=0.700  95% CI=[0.604, 0.782]  width=0.178
n=  300  acc=0.700  95% CI=[0.646, 0.748]  width=0.102
n= 1000  acc=0.700  95% CI=[0.671, 0.727]  width=0.056

Read that bottom line carefully. Even with 1,000 questions, the interval around a 70 percent score spans almost six points. If two models are reported as 70.3 and 69.9 on that benchmark, the difference is well inside the noise floor. The rank order is not evidence of anything.

When is a leaderboard gap actually significant?

The single-score interval is the wrong tool for comparing two models, because it ignores that they were graded on the same questions. Scores on a shared test set are correlated. The correct instrument is McNemar's test, which looks only at the items where the two models disagree.

Build a 2x2 table of agreement. Let b be the number of questions where model A is right and model B is wrong, and c the reverse. The questions both got right or both got wrong carry no information about which is better. McNemar's statistic is (b - c)^2 / (b + c), compared against a chi-square distribution with one degree of freedom.

from scipy.stats import chi2

def mcnemar(b, c):
    """b: A right & B wrong; c: A wrong & B right."""
    n = b + c
    if n == 0:
        return 1.0
    stat = (abs(b - c) - 1) ** 2 / n  # continuity-corrected
    return 1 - chi2.cdf(stat, df=1)

# Two models 0.4 points apart on a 300-item eval:
# say 18 items where A wins, 17 where B wins, rest tied.
print(f"p-value = {mcnemar(18, 17):.3f}")   # p-value = 1.000

A one-item edge out of 300 produces a p-value of essentially 1.0. There is no detectable difference. You would need the disagreement counts to be lopsided, not the headline accuracy to be marginally higher, before the gap earns a rank.

This is why serious evaluation reports publish disagreement matrices, bootstrap confidence intervals, or at least the per-benchmark item counts. If a leaderboard gives you only a sorted column of three-decimal numbers, it is giving you false precision.

How do I tell whether a benchmark can discriminate at all?

Before arguing about margins, check that the test has headroom. A benchmark that is saturated, where the top cluster all score above 95 percent, cannot separate frontier models no matter how many items it has. The remaining questions are either ambiguous, mislabeled, or trivially easy, and the order among the leaders is driven by which model happened to win the coin flips on the hard residue.

Three quick discrimination checks:

Ceiling check. If the top five are all within one point of 100 percent, the benchmark is done. Move on.
Baseline check. Compute what a trivial strategy scores. If majority-class guessing or a tiny n-gram model gets 90 percent, the benchmark measures format compliance, not capability.
Spread check. Look at the standard deviation across all submitted models. A benchmark where everyone scores 68 to 72 has almost no discriminative range left.

A benchmark earns your trust when the baseline is low, the ceiling is far away, and the spread across models is several times the single-score confidence width.

What should I actually do with a leaderboard, then?

Treat the leaderboard as a filter, not a verdict. Use it to shortlist the handful of models in the top band, then evaluate those candidates on your own held-out data that resembles your real traffic. The rank inside that top band is the least reliable information on the page. The membership of the band is the useful part.

And keep popularity and accuracy on separate axes. The cover chart is real Hub data, and it tells you what the community has adopted, which matters for tooling, documentation, and community support. It says nothing about which model is most accurate on your task. Both signals are valuable. Conflating them is how teams end up defending a 0.4-point lead that never existed.

FAQ

Is the Wald interval ever good enough?
For a sanity check on a single score with a few hundred or more items, yes. For scores near 0 or 100 percent, or for small n, switch to Wilson, which stays inside the valid range and does not collapse to zero width at the extremes.

Why McNemar instead of a two-sample proportion test?
Because both models answer the same questions, their scores are paired and correlated. A two-sample test assumes independence and will be too conservative. McNemar conditions on the disagreements, which is exactly the information that distinguishes the two models.

How many eval items do I need to resolve a one-point gap?
Roughly, to get a 95 percent half-width near one point at mid-range accuracy you need on the order of 10,000 independent items. Most public benchmarks have far fewer, which is why one-point gaps are usually unresolvable.

Do bootstrap confidence intervals help?
Yes, especially when scoring is not a clean right-or-wrong binary, such as graded or rubric-scored tasks. Resample the items with replacement a few thousand times, recompute the metric, and read the 2.5 and 97.5 percentiles. It makes no distributional assumption and handles weird metrics gracefully.

Does a larger model automatically win on a saturated benchmark?
No. On a saturated test the residual questions are dominated by noise and label quality, so scale buys you almost nothing. The honest move is to find a harder benchmark with headroom, or build one from your own data.

Can I trust the three decimals a leaderboard prints?
Only if the item count justifies them. Three decimals on a 300-item eval is false precision. The real resolution of that test is closer to one or two points, so mentally round the column before you rank it.

How Hugging Face Official Benchmark Leaderboards Actually Work: .eval_results YAML, the base_model Filter, and the 30% the Default View Hides

Ward Ed — Mon, 05 Oct 2026 08:56:56 +0000

TL;DR: Hugging Face currently tags 48 datasets as official benchmarks (benchmark:official). Their leaderboards are not uploaded by the benchmark owners: they are assembled automatically from small .eval_results/*.yaml files that model authors commit to their own model repos (or propose through pull requests). On October 5, 2026 I queried every board twice through the public API. The default view returned 1,023 entries. With ?base_model=false, which keeps derivative models, it returned 1,469. So 446 entries (30.4%) never show up in the default view, and the deciding field is the base_model key in each model card. Another finding: 772 of the 1,023 visible entries (75.5%) come from open pull requests, and none were flagged verified.

What are Hugging Face official benchmark leaderboards?

An official benchmark is just a dataset repo that carries the benchmark:official tag. The Hub renders a leaderboard widget on its dataset page. Each row on that widget is one model's score on that benchmark. A dataset owner does not have to run any evaluation service for this. The Hub collects scores that model repos self-report and matches them to the dataset.

You can pull the list of boards with one request:

curl -s "https://huggingface.co/api/datasets?filter=benchmark:official&limit=200" \
  | python -c "import sys,json; d=json.load(sys.stdin); print(len(d)); print('\n'.join(x['id'] for x in d))"

When I ran it on October 5, 2026, it returned 48 datasets. Some of the better known ones: TIGER-Lab/MMLU-Pro, Idavidrein/gpqa, cais/hle, SWE-bench/SWE-bench_Verified, hf-audio/open-asr-leaderboard, openai/gsm8k, MathArena/aime_2026, MMMU/MMMU_Pro and harborframework/terminal-bench-2.0. The list also includes newer, narrower boards such as llamaindex/ParseBench, llamaindex/ExtractBench, LEXam-Benchmark/LEXam, FutureMa/EvasionBench and crosbylegal/RedlineBench.

Four of the 48 (tiiuae/PBench, mercor/ACE, ChrisHayduk/nanofold-public, harborframework/terminal-bench-science) had zero entries in both views when I checked. Tagged boards can exist before anyone has reported a score to them.

How are Hugging Face leaderboard entries built from .eval_results YAML?

Every row traces back to one YAML file in a model repo, under a directory named .eval_results/. The leaderboard API includes the path in each row, so you can see it yourself:

curl -s "https://huggingface.co/api/datasets/Idavidrein/gpqa/leaderboard" \
  | python -c "import sys,json; e=json.load(sys.stdin)[1]; print({k:e[k] for k in ('rank','modelId','value','filename','verified','pullRequest')})"

Each row has these fields: rank, filename, value, verified, source, pullRequest, modelId, author, lower_is_better and num_parameters. filename is the YAML path inside the model repo, and pullRequest holds a PR number when the file lives on a PR ref instead of main.

Here is a real file. moonshotai/Kimi-K3 keeps five of them on main:

curl -s "https://huggingface.co/api/models/moonshotai/Kimi-K3/tree/main/.eval_results" \
  | python -c "import sys,json; [print(f['path'], f['size']) for f in json.load(sys.stdin)]"

.eval_results/apex-agents.yaml 157
.eval_results/deep-swe.yaml 156
.eval_results/gpqa.yaml 152
.eval_results/hle.yaml 139
.eval_results/moonshotai__Kimi-K3.yaml 211

The GPQA one (/raw/main/.eval_results/gpqa.yaml) is just 152 bytes:

- dataset:
    id: Idavidrein/gpqa
    task_id: diamond
  value: 93.5
  source:
    url: https://huggingface.co/moonshotai/Kimi-K3
    name: Model Card

That is the whole contract:

dataset.id is the benchmark dataset repo. It has to match one of the official board IDs exactly, or the score goes nowhere.
dataset.task_id picks a sub-task or split when a board has several (for GPQA, diamond).
value is the number that gets ranked. The board's lower_is_better flag decides the sort order (WER on the ASR board, for example).
source is an attribution link. In practice it usually points back at the model card.
Optional keys show up in the wild too, like date and a free-text notes field for the protocol (sample count, temperature, token budget, precision). Use them. They are the only place a reader can see how the number was produced.

The file is a YAML list, so a single file can hold several benchmark results. Many repos still use one file per benchmark, which keeps diffs and PRs small.

Why does my model not appear on the Hugging Face leaderboard? (the base_model filter)

This is where most authors get stuck. The leaderboard endpoint has a parameter called base_model. Without it, the API (and the default widget) drops models whose card declares a base_model, meaning fine-tunes, merges, quantizations and adapters. With ?base_model=false, those derivatives come back:

B=Idavidrein/gpqa
curl -s "https://huggingface.co/api/datasets/$B/leaderboard" | python -c "import sys,json;print(len(json.load(sys.stdin)))"
curl -s "https://huggingface.co/api/datasets/$B/leaderboard?base_model=false" | python -c "import sys,json;print(len(json.load(sys.stdin)))"

For GPQA that is 111 vs 175. I ran the same comparison for all 48 boards with this script:

import requests
from concurrent.futures import ThreadPoolExecutor

API = "https://huggingface.co/api/datasets"
boards = [d["id"] for d in requests.get(f"{API}?filter=benchmark:official&limit=200").json()]

def count(bid):
    d = requests.get(f"{API}/{bid}/leaderboard", timeout=60).json()
    a = requests.get(f"{API}/{bid}/leaderboard?base_model=false", timeout=60).json()
    return bid, len(d), len(a)

rows = list(ThreadPoolExecutor(8).map(count, boards))
dflt = sum(r[1] for r in rows); allr = sum(r[2] for r in rows)
print(len(rows), dflt, allr, f"{(allr - dflt) / allr:.1%} hidden")

Output on 2026-10-05: 48 1023 1469 30.4% hidden.

How much is hidden varies a lot from board to board:

Board	Default	With derivatives	Hidden share
TIGER-Lab/MMLU-Pro	141	216	34.7%
Idavidrein/gpqa	111	175	36.6%
cais/hle	89	108	17.6%
SWE-bench/SWE-bench_Verified	67	89	24.7%
hf-audio/open-asr-leaderboard	62	86	27.9%
likaixin/ScreenSpot-Pro	23	48	52.1%
openai/gsm8k	18	46	60.9%
MMMU/MMMU_Pro	10	24	58.3%
llamaindex/ExtractBench	11	25	56.0%
LiquidAI/ifstruct-v1.0	6	18	66.7%
mteb/arguana	3	16	81.3%
crosbylegal/RedlineBench	13	13	0.0%

The pattern makes sense. Older or easier benchmarks like GSM8K attract many fine-tunes, so derivatives are the majority there and the default view shows a minority of the reported scores. On agentic and legal boards where most submissions are base models from labs (RedlineBench, skillsbench, Long-Horizon-Terminal-Bench), the two views are identical.

The filter reads the model card metadata, not the weights. A model trained from scratch that lists a base_model anyway (some teams do this to credit an architecture or tokenizer source) gets hidden just like a LoRA. A heavily modified derivative that leaves the field out shows up as if it were a base model. So the default view is a filter on declared lineage, not real lineage.

Who verifies scores on Hugging Face official leaderboards?

According to the API, nobody does yet. Of the 1,023 default-view rows, 0 had verified: true. Every number on these boards is self-reported by whoever controls the model repo or opens a PR to it.

The PR path is bigger than most people expect. 772 of 1,023 visible rows (75.5%) carried a pullRequest number. So the YAML sits on an open (unmerged) PR ref such as refs/pr/1, not on main. Reading those files directly needs the PR ref:

# file on an open PR, not on main
curl -s "https://huggingface.co/XiaomiMiMo/MiMo-V2.5-Pro/raw/refs%2Fpr%2F1/.eval_results/gsm8k.yaml"

If you try tree/main/.eval_results on such a model, you get .eval_results does not exist on "main" even though the row shows on the leaderboard. That is expected behavior, not a bug.

Practical implications for anyone citing these boards:

A row is a claim, not a measurement. Follow the source.url and read the notes, if there are any.
Protocols differ across rows on the same board. Majority voting over many samples, different thinking budgets, different task subsets: none of that is normalized. The task_id field is the only structural guard, and it only separates named splits.
Rows can come from someone other than the model owner, through a PR. Check the PR author before treating a number as the vendor's.

How do I submit a model score to a Hugging Face benchmark leaderboard?

There is no submission form. You commit a file:

from huggingface_hub import HfApi
import yaml, io

result = [{
    "dataset": {"id": "Idavidrein/gpqa", "task_id": "diamond"},
    "value": 71.2,
    "date": "2026-10-05",
    "source": {"url": "https://huggingface.co/your-org/your-model", "name": "Model Card"},
    "notes": "GPQA Diamond, 198 items, pass@1, temperature 0, bf16",
}]
buf = io.BytesIO(yaml.safe_dump(result, sort_keys=False).encode("utf-8"))
HfApi().upload_file(
    path_or_fileobj=buf,
    path_in_repo=".eval_results/gpqa.yaml",
    repo_id="your-org/your-model",
    commit_message="Add GPQA Diamond eval result",
    # create_pr=True,  # use this when you are not the repo owner
)

The numbers above are placeholders for illustration. Then check whether the row appears, and which view it appears in:

import requests
def where_am_i(board, model):
    base = f"https://huggingface.co/api/datasets/{board}/leaderboard"
    d = [e for e in requests.get(base).json() if e["modelId"] == model]
    a = [e for e in requests.get(base + "?base_model=false").json() if e["modelId"] == model]
    return {"default_view": bool(d), "with_derivatives": bool(a),
            "rank_default": d[0]["rank"] if d else None,
            "rank_all": a[0]["rank"] if a else None}

print(where_am_i("Idavidrein/gpqa", "your-org/your-model"))

If with_derivatives is true and default_view is false, the base_model field in your card is what hides you.

Checklist for model authors before reporting a leaderboard rank

Match dataset.id exactly to one of the 48 IDs from filter=benchmark:official. Case and owner prefix matter.
Set task_id when the board has subsets. A GPQA "main" score sitting next to "diamond" scores is misleading.
Decide on base_model on purpose. If your model really is a fine-tune, keeping the field is honest, but you will only appear in the derivative-inclusive view. If the field is left over from a template or only credits a tokenizer, fix the card. Don't remove it from a real derivative just to get a better spot.
Verify with the parameterless API, not the ?base_model=false URL and not a cached screenshot. The parameterless call is what most visitors see by default.
Quote the right rank. "Rank 3 including derivatives" and "rank 3" are different claims. With 30.4% of entries hidden, the gap can be large.
Write the protocol in notes: sample count, decoding settings, token budget, precision, subset size. Nobody verifies rows centrally, so this is your credibility.
Merge your own PRs. If a contributor opened the eval PR, the row already counts. Merging it puts the file on main and makes the history auditable.

FAQ

How many official benchmark leaderboards are on Hugging Face?

48 datasets carried the benchmark:official tag on October 5, 2026. Four of them had no entries yet. Query https://huggingface.co/api/datasets?filter=benchmark:official&limit=200 for the current list.

What does `base_model=false` do in the Hugging Face leaderboard API?

It adds back models whose card declares a base_model (fine-tunes, merges, quantizations, adapters). The default request leaves them out. Across all boards that is 1,469 entries vs 1,023, so 30.4% are hidden by default.

Where does a Hugging Face leaderboard score come from?

From a YAML file under .eval_results/ in the model repo, on main or on an open PR ref. The leaderboard row's filename field gives the path, and pullRequest gives the PR number if there is one.

Are Hugging Face official leaderboard scores verified?

Not at the time of measurement. All 1,023 default-view rows had verified: false. Treat each row as a self-reported claim and check its source and notes.

Why does my model show on the board in one browser link but not another?

You are probably comparing the default view with the derivative-inclusive view. Run both API calls for your modelId. If only the ?base_model=false call finds you, your model card's base_model field explains it.

Can someone else add a score for my model?

Yes. Anyone can open a pull request that adds .eval_results/*.yaml to your repo, and 75.5% of visible rows came from PR refs when I measured. Review those PRs like any other change to your model card.

Methodology: all counts come from unauthenticated requests to the public Hugging Face API on 2026-10-05. Leaderboards change daily, so re-run the scripts above for current numbers.

DEV Community: Ward Ed

Reading a Hardware-Equivalence Claim Like an Evaluator: A Checklist Applied to an NPU vs GPU Pretraining Twin

TL;DR

Check 1: Is there a true twin?

Check 2: Is the evaluation data really unseen?

Check 3: Is the comparison paired?

Check 4: Can you rerun it?

Check 5: Is variance across seeds reported?

FAQ

Sources

What a Youden index of 77 actually means on the OWASP Benchmark

What the Youden index is

Why that matters

How to compare tools fairly

GPQA Diamond Has 198 Questions: What the Hugging Face Leaderboard's 115 Rows Can and Cannot Rank

TL;DR

What is on the Hugging Face GPQA leaderboard?

How much can 198 questions separate?

Would the order survive a fresh draw of questions?

Are these rows single runs over 198 questions?

Same repo, different setting: how big can the gap be?

Do self-reports run higher than independent re-measurements?

Do repeated numbers mean copied numbers?

Is there headroom left to rank at the top?

A six-step check before quoting a GPQA number

Limitations

FAQ

Further reading

Sources

Do 66 Terminal Tasks and a 45% Private Weight Settle AI Rankings? Reading Artificial Analysis Index v4.3

TL;DR

What exactly changed in Index v4.3?

Does a private test set make a ranking more reliable?

How noisy is a 66-task benchmark?

Which of the reported gaps would survive a fresh task draw?

How many tasks would a 7-point gap need?

What does this do to the index itself?

A checklist before you quote a leaderboard gap

FAQ

Further reading

Best Open-Weight LLM in October 2026? Auditing Two Leaderboards Row by Row

TL;DR

Which open-weight model is #1 right now?

How wide is the uncertainty behind that #1?

Do the two leaderboards agree below rank 1?

Why is Kimi K3 missing from one "open" list and Qwen3.8 Max flagged closed on the other?

Are the two leaderboards independent evidence?

A five-step audit before you quote an open-model rank

Limitations of this audit

FAQ

Further reading

Sources

How to Read a Public Leaderboard Without Fooling Yourself

TL;DR

Leakage: the number that was never earned

Saturation: when the ruler stops measuring

Selection on the eval set: inventing gains that are not there

Run-to-run noise: a positive margin is not a win

A short reading checklist

FAQ

Further reading

Benchmark Contamination 101: How Train/Test Overlap Inflates Leaderboard Scores (and How to Catch It)

TL;DR

What is benchmark contamination and why does it inflate scores?

Why can self-reported leaderboard scores be inflated?

How do you detect train/test overlap? Three methods

1. N-gram overlap

2. Canary strings

3. Membership inference and behavioral tests

A runnable n-gram overlap check

A practical contamination checklist

FAQ

Further reading

When a 0.4-Point Lead Means Nothing: Reading Open-Model Leaderboard Margins Like a Statistician

TL;DR

Why do the top open models cluster so tightly?

How do I compute the error bar on a single accuracy score?

When is a leaderboard gap actually significant?

How do I tell whether a benchmark can discriminate at all?

What should I actually do with a leaderboard, then?

What does `base_model=false` do in the Hugging Face leaderboard API?