DEV Community: QuantID

How Do You Show Two Training Runs Are Equal? A Paired Test on Unseen Text, Using an NPU vs GPU Case

QuantID — Mon, 26 Oct 2026 05:54:22 +0000

Disclosure: the case study below is VIDRAFT's own result. VIDRAFT is a technology partner of QuantID. This post reviews the measurement design; it is not an independent verification.

TL;DR

To claim two training runs reached the same quality, you need two things: an evaluation set the models truly never saw, and a paired comparison on identical inputs. The case here is VIDRAFT's VKORE-325M, pretrained on a Rebellions ATOM-Max NPU and, as a reference, on a B200 GPU. On 1,024 unseen windows the losses are 0.7993 and 0.7994; the paired difference is -0.0001 with a standard error of 0.0007.

What makes a fair comparison between two runs?

Hold everything fixed except the variable you test. In this case the model, initialization, data, batch and learning-rate schedule were identical, and only the hardware differed.

Matching training curves are encouraging but not proof. Training loss is measured on data the model is fitting.

Is "held out" really held out?

This case contains a useful lesson. The first evaluation used 256 fixed windows from the end of the training file and was called "held out". But training sampled random windows from the whole file, about 0.72 passes over it, so those windows were partly seen during training. The comparison between the two runs was still fair, because both saw the same file, but the label was wrong.

The fix was to evaluate on a shard of the source dataset that was never part of the training corpus. The training file came from FineWeb-Edu sample/10BT shards 000 and 001; the new evaluation used shard 013.

Check: before calling a set held out, read the training sampler, not only the data split. Random-window samplers cover the whole file.

Why compare windows in pairs instead of averages?

Loss varies far more between text windows than between two good models. If you compare two averages, that window-to-window variation dominates the error bar.

Evaluating both models on identical windows and taking the per-window difference removes it. Here the per-model standard error is about 0.0049, while the paired standard error is 0.0007, about seven times tighter.

Quantity	Value
NPU loss, unseen (1,024 windows)	0.7993
GPU loss, unseen	0.7994
Paired mean difference (NPU minus GPU)	-0.0001
Paired standard error	0.0007
Windows where NPU is lower	50.1%

The honest reading is "no detectable difference at a resolution of about 0.0014 (two standard errors)". That is a stronger statement than "the averages look close", and a weaker one than "identical".

What else should a reader check?

Reproducibility. Both weights and the evaluation script are public, so the paired test can be rerun.
Scope. The result concerns a 325M base model on 5B tokens. It says nothing about larger scales.
What is not measured. Speed and cost per run are separate questions. The case also notes that the NPU alone does not enable training; it depends on the developer's own training stack.

FAQ

Why bits per byte? With a byte-level vocabulary, loss per byte divided by ln 2 gives bits per byte (1.1531 vs 1.1533 here), which is comparable across tokenizers.

Is 1,024 windows enough? For a paired test with this variance, yes: the standard error of the difference is 0.0007.

Could a single seed hide differences? Possibly. A stronger design repeats each run with several seeds; this case reports one run per hardware.

Who measured this? VIDRAFT measured its own models. This post reviews the design.

Hugging Face Official Benchmarks: Daily Update 2026-10-24

QuantID — Fri, 23 Oct 2026 15:01:23 +0000

This is the daily update of the Hugging Face Official Benchmarks map, which reads every Hugging Face official benchmark leaderboard straight from the public API.

As of 2026-10-24 (KST): 51 official benchmarks, 1,118 leaderboard entries.

What changed since the last snapshot

New official benchmarks: decision-index-leaderboard, decision-index-vision-leaderboard.
5 new leaderboard entries across all boards.

Organizations by number of #1 spots

Rank	Organization	#1 spots
1	FINAL-Bench	11
2	deepseek-ai	4
3	inclusionAI	4
4	zai-org	4
5	MiniMaxAI	2
6	XiaomiMiMo	2
7	moonshotai	2
8	ornith-ai	2
9	Edge0	1
10	KDLAI	1

Current #1 on every official benchmark

Benchmark	#1 model	Score	Entries
MMLU-Pro	FINAL-Bench/Darwin-180B-RSI	88.12	142
gpqa	FINAL-Bench/Darwin-180B-RSI	94.44	115
hle	deepseek-ai/DeepSeek-V4.1-Flash	63.9	98
SWE-bench_Verified	ornith-ai/Ornith-1.5-397B	86	68
open-asr-leaderboard	Edge0/ARK-ASR-3B	4.76	63
WildClawBench	Qwen/Qwen3.8-2.4T-A95B	708	51
SWE-bench_Pro	inclusionAI/Ling-3.1-flash	68.74	47
terminal-bench-2.0	mindlab-research/Macaron-V1-Venti	87.6	40
terminal-bench-2.1	deepseek-ai/DeepSeek-V4.1-Flash	90.6	37
ParseBench	KDLAI/KDL-Frontier-Parser-nano	76.36	34
SWE-bench_Multilingual	inclusionAI/Ling-3.1-flash	88.5	28
deep-swe	deepseek-ai/DeepSeek-V4.1-Flash-1017	77	27
aime_2026	FINAL-Bench/Darwin-180B-RSI	100	27
ScreenSpot-Pro	meta-models/Muse-Glimmer-30B	75.4	24
Claw-Eval	ornith-ai/Ornith-1.5-397B	81.4	23
olmOCR-bench	SoMarkAI/OXR-1.0	87.84	22
hmmt_feb_2026	FINAL-Bench/Darwin-180B-RSI	100	20
gsm8k	XiaomiMiMo/MiMo-V2.5-Pro	99.6	19
Toolathlon	XiaomiMiMo/MiMo-V2.6-Pro-RL	76.9	18
Real5-OmniDocBench	PaddlePaddle/PaddleOCR-VL-1.6	93.19	17
MDPBench	FINAL-Bench/Darwin-180B-RSI	83.65	16
RedlineBench	crosbylegal/gemini-4-argon	59.5	14
apex-agents	moonshotai/Kimi-K3	41	13
skillsbench	inclusionAI/Ling-3.1-flash	68.7	12
LEXam-hard	FINAL-Bench/Darwin-180B-RSI	45.72	12
ExtractBench	FINAL-Bench/Darwin-180B-RSI-R3	90.29	11
MMMU_Pro	FINAL-Bench/Darwin-180B-RSI	79.48	11
Vaani-Benchmark-V1.0	gnani/gnani	13	11
agents-last-exam	deepseek-ai/DeepSeek-V4.1-Flash-1017	35.4	10
yc-bench	MiniMaxAI/MiniMax-M3	2100000	10
LEXam	FINAL-Bench/Darwin-180B-RSI	68.94	10
Long-Horizon-Terminal-Bench	moonshotai/Kimi-K3	6	9
typed-decisions	FINAL-Bench/Darwin-27B-ZTC-v5	0.812	8
terminal-bench	inclusionAI/Ling-3.1-flash	46.03	8
WBench	robbyant/lingbot-world-fast	78.8	8
Video-MME-v2	MiniMaxAI/MiniMax-M3	85.4	7
ResearchClawBench	zai-org/GLM-5.2	20.709230769230768	7
ifstruct-v1.0	FINAL-Bench/Darwin-180B-RSI	98.95	6
EvasionBench	zai-org/GLM-4.7	82.91	5
arguana	google/embeddinggemma-300m	71.535	3
chi-bench	zai-org/GLM-5.1	18.7	3
APEX-v1-extended	zai-org/GLM-4.7	51.7	2
vlabench_primitive_ft_lerobot_video	nvidia/GR00T-N1-2B	39.7	1
BRIGHT	mteb/baseline-bm25s	0.11	1

How to read this

Most leaderboard entries are self-reported by model authors, and evaluation settings differ between submissions (sampling, voting, thinking budget). An entry is marked verified only where Hugging Face marks it so. Ties at rank 1 count for each tied organization.

Disclosure: QuantID is a technology partner of VIDRAFT, whose FINAL-Bench organization appears in these rankings. This post reports the public leaderboard data as published on Hugging Face; it is not an independent evaluation.

Hugging Face Official Benchmarks: Daily Update 2026-10-23

QuantID — Thu, 22 Oct 2026 15:00:37 +0000

This is the daily update of the Hugging Face Official Benchmarks map, which reads every Hugging Face official benchmark leaderboard straight from the public API.

As of 2026-10-23 (KST): 49 official benchmarks, 1,113 leaderboard entries.

What changed since the last snapshot

agents-last-exam: new #1 deepseek-ai/DeepSeek-V4.1-Flash-1017 (score 35.4), previously XiaomiMiMo/MiMo-V2.6-Pro-RL.
deep-swe: new #1 deepseek-ai/DeepSeek-V4.1-Flash-1017 (score 77), previously deepseek-ai/DeepSeek-V4.1-Flash.
5 new leaderboard entries across all boards.

Organizations by number of #1 spots

Rank	Organization	#1 spots
1	FINAL-Bench	11
2	deepseek-ai	4
3	inclusionAI	4
4	zai-org	4
5	MiniMaxAI	2
6	XiaomiMiMo	2
7	moonshotai	2
8	ornith-ai	2
9	Edge0	1
10	KDLAI	1

Current #1 on every official benchmark

Benchmark	#1 model	Score	Entries
MMLU-Pro	FINAL-Bench/Darwin-180B-RSI	88.12	142
gpqa	FINAL-Bench/Darwin-180B-RSI	94.44	115
hle	deepseek-ai/DeepSeek-V4.1-Flash	63.9	97
SWE-bench_Verified	ornith-ai/Ornith-1.5-397B	86	68
open-asr-leaderboard	Edge0/ARK-ASR-3B	4.76	63
WildClawBench	Qwen/Qwen3.8-2.4T-A95B	708	51
SWE-bench_Pro	inclusionAI/Ling-3.1-flash	68.74	47
terminal-bench-2.0	mindlab-research/Macaron-V1-Venti	87.6	40
terminal-bench-2.1	deepseek-ai/DeepSeek-V4.1-Flash	90.6	37
ParseBench	KDLAI/KDL-Frontier-Parser-nano	76.36	34
SWE-bench_Multilingual	inclusionAI/Ling-3.1-flash	88.5	28
aime_2026	FINAL-Bench/Darwin-180B-RSI	100	27
deep-swe	deepseek-ai/DeepSeek-V4.1-Flash-1017	77	26
ScreenSpot-Pro	meta-models/Muse-Glimmer-30B	75.4	24
Claw-Eval	ornith-ai/Ornith-1.5-397B	81.4	23
olmOCR-bench	SoMarkAI/OXR-1.0	87.84	22
hmmt_feb_2026	FINAL-Bench/Darwin-180B-RSI	100	20
gsm8k	XiaomiMiMo/MiMo-V2.5-Pro	99.6	19
Toolathlon	XiaomiMiMo/MiMo-V2.6-Pro-RL	76.9	18
Real5-OmniDocBench	PaddlePaddle/PaddleOCR-VL-1.6	93.19	17
MDPBench	FINAL-Bench/Darwin-180B-RSI	83.65	16
RedlineBench	crosbylegal/gemini-4-argon	59.5	14
apex-agents	moonshotai/Kimi-K3	41	13
skillsbench	inclusionAI/Ling-3.1-flash	68.7	12
LEXam-hard	FINAL-Bench/Darwin-180B-RSI	45.72	12
ExtractBench	FINAL-Bench/Darwin-180B-RSI-R3	90.29	11
MMMU_Pro	FINAL-Bench/Darwin-180B-RSI	79.48	11
LEXam	FINAL-Bench/Darwin-180B-RSI	68.94	10
yc-bench	MiniMaxAI/MiniMax-M3	2100000	10
Vaani-Benchmark-V1.0	gnani/gnani	13	10
agents-last-exam	deepseek-ai/DeepSeek-V4.1-Flash-1017	35.4	9
Long-Horizon-Terminal-Bench	moonshotai/Kimi-K3	6	9
typed-decisions	FINAL-Bench/Darwin-27B-ZTC-v5	0.812	8
WBench	robbyant/lingbot-world-fast	78.8	8
terminal-bench	inclusionAI/Ling-3.1-flash	46.03	7
Video-MME-v2	MiniMaxAI/MiniMax-M3	85.4	7
ResearchClawBench	zai-org/GLM-5.2	20.709230769230768	7
ifstruct-v1.0	FINAL-Bench/Darwin-180B-RSI	98.95	6
EvasionBench	zai-org/GLM-4.7	82.91	5
chi-bench	zai-org/GLM-5.1	18.7	3
arguana	google/embeddinggemma-300m	71.535	3
APEX-v1-extended	zai-org/GLM-4.7	51.7	2
vlabench_primitive_ft_lerobot_video	nvidia/GR00T-N1-2B	39.7	1
BRIGHT	mteb/baseline-bm25s	0.11	1

How to read this

Hugging Face Official Benchmarks: Daily Update 2026-10-20

QuantID — Mon, 19 Oct 2026 15:00:54 +0000

This is the daily update of the Hugging Face Official Benchmarks map, which reads every Hugging Face official benchmark leaderboard straight from the public API.

As of 2026-10-20 (KST): 49 official benchmarks, 1,107 leaderboard entries.

What changed since the last snapshot

olmOCR-bench: new #1 SoMarkAI/OXR-1.0 (score 87.84), previously infly/Infinity-Parser2-Pro.
5 new leaderboard entries across all boards.

Organizations by number of #1 spots

Rank	Organization	#1 spots
1	FINAL-Bench	11
2	inclusionAI	4
3	zai-org	4
4	XiaomiMiMo	3
5	deepseek-ai	3
6	MiniMaxAI	2
7	moonshotai	2
8	ornith-ai	2
9	Edge0	1
10	KDLAI	1

Current #1 on every official benchmark

Benchmark	#1 model	Score	Entries
MMLU-Pro	FINAL-Bench/Darwin-180B-RSI	88.12	142
gpqa	FINAL-Bench/Darwin-180B-RSI	94.44	115
hle	deepseek-ai/DeepSeek-V4.1-Flash	63.9	95
SWE-bench_Verified	ornith-ai/Ornith-1.5-397B	86	68
open-asr-leaderboard	Edge0/ARK-ASR-3B	4.76	63
WildClawBench	Qwen/Qwen3.8-2.4T-A95B	708	51
SWE-bench_Pro	inclusionAI/Ling-3.1-flash	68.74	47
terminal-bench-2.0	mindlab-research/Macaron-V1-Venti	87.6	40
terminal-bench-2.1	deepseek-ai/DeepSeek-V4.1-Flash	90.6	37
ParseBench	KDLAI/KDL-Frontier-Parser-nano	76.36	34
SWE-bench_Multilingual	inclusionAI/Ling-3.1-flash	88.5	28
aime_2026	FINAL-Bench/Darwin-180B-RSI	100	27
deep-swe	deepseek-ai/DeepSeek-V4.1-Flash	74.2	25
ScreenSpot-Pro	meta-models/Muse-Glimmer-30B	75.4	24
Claw-Eval	ornith-ai/Ornith-1.5-397B	81.4	23
olmOCR-bench	SoMarkAI/OXR-1.0	87.84	22
hmmt_feb_2026	FINAL-Bench/Darwin-180B-RSI	100	20
gsm8k	XiaomiMiMo/MiMo-V2.5-Pro	99.6	19
Toolathlon	XiaomiMiMo/MiMo-V2.6-Pro-RL	76.9	18
Real5-OmniDocBench	PaddlePaddle/PaddleOCR-VL-1.6	93.19	17
MDPBench	FINAL-Bench/Darwin-180B-RSI	83.65	15
RedlineBench	crosbylegal/gemini-4-argon	59.5	14
apex-agents	moonshotai/Kimi-K3	41	13
skillsbench	inclusionAI/Ling-3.1-flash	68.7	12
LEXam-hard	FINAL-Bench/Darwin-180B-RSI	45.72	12
ExtractBench	FINAL-Bench/Darwin-180B-RSI-R3	90.29	11
MMMU_Pro	FINAL-Bench/Darwin-180B-RSI	79.48	11
LEXam	FINAL-Bench/Darwin-180B-RSI	68.94	10
yc-bench	MiniMaxAI/MiniMax-M3	2100000	10
Vaani-Benchmark-V1.0	gnani/gnani	13	10
Long-Horizon-Terminal-Bench	moonshotai/Kimi-K3	6	9
typed-decisions	FINAL-Bench/Darwin-27B-ZTC-v5	0.812	8
agents-last-exam	XiaomiMiMo/MiMo-V2.6-Pro-RL	31.6	8
WBench	robbyant/lingbot-world-fast	78.8	8
ResearchClawBench	zai-org/GLM-5.2	20.709230769230768	7
Video-MME-v2	MiniMaxAI/MiniMax-M3	85.4	7
ifstruct-v1.0	FINAL-Bench/Darwin-180B-RSI	98.95	6
terminal-bench	inclusionAI/Ling-3.1-flash	46.03	6
EvasionBench	zai-org/GLM-4.7	82.91	5
chi-bench	zai-org/GLM-5.1	18.7	3
arguana	google/embeddinggemma-300m	71.535	3
APEX-v1-extended	zai-org/GLM-4.7	51.7	2
vlabench_primitive_ft_lerobot_video	nvidia/GR00T-N1-2B	39.7	1
BRIGHT	mteb/baseline-bm25s	0.11	1

How to read this

Hugging Face Official Benchmarks: Daily Update 2026-10-18

QuantID — Sun, 18 Oct 2026 03:18:26 +0000

This is the daily update of the Hugging Face Official Benchmarks map, which reads every Hugging Face official benchmark leaderboard straight from the public API.

As of 2026-10-18 (KST): 49 official benchmarks, 1,102 leaderboard entries.

This is the first snapshot in the series, so it records the baseline: who is ranked first on each board today. Later posts will report only what changed.

Organizations by number of #1 spots

Rank	Organization	#1 spots
1	FINAL-Bench	11
2	inclusionAI	4
3	zai-org	4
4	XiaomiMiMo	3
5	deepseek-ai	3
6	MiniMaxAI	2
7	moonshotai	2
8	ornith-ai	2
9	Edge0	1
10	KDLAI	1

Current #1 on every official benchmark

Benchmark	#1 model	Score	Entries
MMLU-Pro	FINAL-Bench/Darwin-180B-RSI	88.12	142
gpqa	FINAL-Bench/Darwin-180B-RSI	94.44	115
hle	deepseek-ai/DeepSeek-V4.1-Flash	63.9	95
SWE-bench_Verified	ornith-ai/Ornith-1.5-397B	86	68
open-asr-leaderboard	Edge0/ARK-ASR-3B	4.76	63
WildClawBench	Qwen/Qwen3.8-2.4T-A95B	708	51
SWE-bench_Pro	inclusionAI/Ling-3.1-flash	68.74	47
terminal-bench-2.0	mindlab-research/Macaron-V1-Venti	87.6	40
terminal-bench-2.1	deepseek-ai/DeepSeek-V4.1-Flash	90.6	37
ParseBench	KDLAI/KDL-Frontier-Parser-nano	76.36	34
SWE-bench_Multilingual	inclusionAI/Ling-3.1-flash	88.5	28
aime_2026	FINAL-Bench/Darwin-180B-RSI	100	27
deep-swe	deepseek-ai/DeepSeek-V4.1-Flash	74.2	25
ScreenSpot-Pro	meta-models/Muse-Glimmer-30B	75.4	23
Claw-Eval	ornith-ai/Ornith-1.5-397B	81.4	23
olmOCR-bench	infly/Infinity-Parser2-Pro	87.6	21
hmmt_feb_2026	FINAL-Bench/Darwin-180B-RSI	100	20
gsm8k	XiaomiMiMo/MiMo-V2.5-Pro	99.6	18
Toolathlon	XiaomiMiMo/MiMo-V2.6-Pro-RL	76.9	18
Real5-OmniDocBench	PaddlePaddle/PaddleOCR-VL-1.6	93.19	17
MDPBench	FINAL-Bench/Darwin-180B-RSI	83.65	15
RedlineBench	crosbylegal/gemini-4-argon	59.5	14
apex-agents	moonshotai/Kimi-K3	41	13
skillsbench	inclusionAI/Ling-3.1-flash	68.7	12
LEXam-hard	FINAL-Bench/Darwin-180B-RSI	45.72	12
ExtractBench	FINAL-Bench/Darwin-180B-RSI-R3	90.29	11
MMMU_Pro	FINAL-Bench/Darwin-180B-RSI	79.48	10
LEXam	FINAL-Bench/Darwin-180B-RSI	68.94	10
yc-bench	MiniMaxAI/MiniMax-M3	2100000	10
Vaani-Benchmark-V1.0	gnani/gnani	13	10
Long-Horizon-Terminal-Bench	moonshotai/Kimi-K3	6	9
agents-last-exam	XiaomiMiMo/MiMo-V2.6-Pro-RL	31.6	8
WBench	robbyant/lingbot-world-fast	78.8	8
typed-decisions	FINAL-Bench/Darwin-27B-ZTC-v5	0.812	7
ResearchClawBench	zai-org/GLM-5.2	20.709230769230768	7
Video-MME-v2	MiniMaxAI/MiniMax-M3	85.4	7
ifstruct-v1.0	FINAL-Bench/Darwin-180B-RSI	98.95	6
terminal-bench	inclusionAI/Ling-3.1-flash	46.03	6
EvasionBench	zai-org/GLM-4.7	82.91	5
chi-bench	zai-org/GLM-5.1	18.7	3
arguana	google/embeddinggemma-300m	71.535	3
APEX-v1-extended	zai-org/GLM-4.7	51.7	2
vlabench_primitive_ft_lerobot_video	nvidia/GR00T-N1-2B	39.7	1
BRIGHT	mteb/baseline-bm25s	0.11	1

How to read this

Measuring the Real Cost of an LLM Call: Tokens, Precision, and Honest Numbers

QuantID — Mon, 12 Oct 2026 10:34:35 +0000

TL;DR

The price per million tokens is the sticker, not the cost. The real cost of an LLM call depends on how many tokens you actually spend, the precision the provider serves at, how much the request waits, and whether the endpoint is even up. This post is a measurement checklist for putting a defensible number on an LLM call, written from the view of a team whose product is measurement.

The sticker is not the cost

Two providers can quote the same price per million tokens and cost you very different amounts. The gap hides in four places.

Token spend. Reasoning models and verbose system prompts change how many tokens a task actually burns. The unit price is fixed, the unit count is not.
Precision. A model served at a lower numeric precision is cheaper to run, and the provider may or may not tell you which precision you are getting. Precision is a cost and quality lever at the same time.
Latency. First-token latency and tokens per second decide whether a call fits a user-facing budget. Public price pages rarely publish these.
Availability. A cheap endpoint that returns a 503 under load has an effective cost of infinity for that request.

Precision deserves its own line

When you compare two deployments of the same model, the flags on the command line are not the whole story. The served weights can differ in precision even when the launch looks identical, and that changes both speed and output. If you are benchmarking, pin the precision you are actually measuring and state it. A fair comparison names the precision; an unfair one hides it.

A simple way to keep this honest is to treat precision as an explicit weight in your cost model rather than folding it silently into a single price number. State the assumption, then let readers adjust it.

Latency is a cost, not a vibe

First-token latency and tokens per second are measurable, and they belong in the cost story. The catch is that public APIs often do not expose them, so you have to call the endpoint yourself with the same prompt and record the numbers. One call is a sample, not a law. Report how many calls, and from where, because the answer moves with both.

A measurement checklist

Before you publish a cost number for an LLM call, confirm:

You counted actual tokens for a representative task, not just the unit price.
You named the serving precision you measured.
You measured first-token latency and throughput yourself, with the call count stated.
You checked availability under the load you care about, not at idle.
You separated numbers you measured from numbers you assumed, and labeled the assumptions.

If a comparison is missing any of these, it is a quote, not a measurement.

Why this matters now

As more products route across many models, the question is no longer which model is best in the abstract. It is which deployment gives the quality you need at a cost you can defend. That is a measurement problem, and measurement problems reward discipline over marketing.

Note on relationships: QuantID works with technology partners on model and serving research; where a partner is relevant we disclose it as a partner rather than an endorsement.

FAQ

Is price per million tokens useless? No. It is one input. The cost is price times actual token count, adjusted for precision, latency, and availability.

How do I measure precision if the provider does not say? Measure behavior and speed on fixed prompts, and state that the served precision is unverified when it is.

Why measure latency myself? Because most public APIs do not expose first-token latency, and it is decisive for user-facing budgets.

One call or many? Many. One call is a sample. State the count and the origin.

Eleven Number-One Records: Measuring a Model That Swept Math, Science, Law and Decisions

QuantID — Sun, 11 Oct 2026 02:50:52 +0000

TL;DR

A single self-improving model family holds eleven public number-one benchmark records at once, across mathematics, science, law, structured output, and decisions. This is a measurement note: the eleven, their scores, and why the spread matters more than any single win.

The eleven records

#	Benchmark	Field	Result
1	AIME 2026	Math	100% (perfect)
2	HMMT 2026	Math	100% (perfect)
3	GPQA Diamond	Science	94.44%
4	MMLU-Pro	Knowledge	88.12%
5	MMMU-Pro	Multimodal	79.48%
6	LEXam	Law	68.94%
7	LEXam-hard	Law	45.72%
8	ExtractBench	Extraction	90.29%
9	IFStruct	Structured output	98.95%
10	MDPBench	Decision process	83.65%
11	S1MB	Decision engine	Borda 89.58, Task Avg 66.46

Why the spread is the real signal

A single top score can come from tuning to one test. Eleven number-one records across unrelated fields cannot. Math, science, law, and typed decisions stress different capabilities, so holding the top of all of them at once is evidence of a general method rather than a lucky fit.

Two of the records are perfect scores, on AIME 2026 and HMMT 2026. When a benchmark saturates, the honest reading is that it no longer discriminates at the top, which is itself a measurement result and a reason to move evaluation toward harder, decision-oriented tests such as S1MB.

How these were produced

The family is self-improving: it runs a recursive loop tied to external verification rather than to its own opinion. For decisions, where the output is typed, scoring uses a zero-token method, which keeps the measurement deterministic and reproducible.

Check it yourself

S1MB number one model: https://github.com/final-bench/s1mb
ZTC decision method: https://github.com/final-bench/ztc
Models: https://huggingface.co/FINAL-Bench

FAQ

How many number-one records does the family hold?
Eleven public number-one benchmark records at once, across math, science, law, structured output, and decisions.

Which are perfect scores?
AIME 2026 and HMMT 2026, both at 100%.

Why does holding many unrelated benchmarks matter?
A single win can reflect tuning to one test. Leading many unrelated fields at once is evidence of a general method.

Are the results reproducible?
Decision results use a deterministic zero-token method, and the leaderboards and models are public.

Reading the S1MB result: why Borda aggregation makes a #1 harder to fake

QuantID — Fri, 09 Oct 2026 08:58:56 +0000

The System One Mosaic Benchmark (S1MB, by hotchpotch) compares 102 models across 137 specialized benchmarks in three task families: Noul (assess a condition), Choice (select an option), and Score (rate on a scale). This is a measurement note on how it scores and why its top position is informative.

Borda, not average

S1MB ranks by a Borda score. Instead of averaging raw task numbers, which lets one easy benchmark dominate the mean, Borda converts each of the 137 benchmarks into a rank and sums rank-based points. That rewards broad, consistent strength over a spike on a few favorable tasks. It is a deliberately conservative aggregation, and it makes a top position harder to manufacture: you cannot buy first place by overfitting a handful of suites.

The current top

Darwin-27B-ZTC-v2 leads with Borda 89.58, ahead of OpenJev-27B (87.50) and AutoJev-27B (87.07). The margin is widest on the Score family (60.21 vs 54.83), which is the ordinal-judgment task where most judges weaken. Leading where the task is hardest, rather than on an easy split, is the signal worth reading.

Why a deterministic judge is clean to measure

A zero-token judge emits a probability distribution in a single forward pass. No sampling means no output variance, so the reported numbers are exact values rather than sample estimates. There is no standard-deviation term to carry and no decoding hyperparameters (temperature, top-p, length) to confound cross-model comparison. The same input reproduces the same score.

On a related board, typed-decisions, the same approach is also #1 at accuracy 0.743 (zero-shot), with calibration KL 0.204 and Brier 0.097. Brier is a strictly proper scoring rule, so confidence cannot be gamed by inflating or deflating it.

The base_model Filter: Why 31% of Hugging Face Leaderboard Entries Are Hidden by Default

QuantID — Tue, 06 Oct 2026 19:13:14 +0000

TL;DR

On a Hugging Face official leaderboard, the default view silently drops any model whose card declares a base_model field. That one line of YAML, added so people can see what your model was built from, moves your entry out of sight. We measured the effect live on 2026-10-07 across every official benchmark board: 461 of 1,482 entries (31.1%) are hidden in the default view, and 35 of 44 populated boards hide at least one entry. The fix is a single API parameter, base_model=false, and this post gives you a reproducible script plus an author checklist so your own submission does not vanish.

This is a measurement note, not a complaint. The filter is a reasonable default. The problem is that most authors do not know it applies to them.

What is the base_model filter and why does it hide entries?

Model cards on the Hub carry YAML metadata. One common field is base_model, which records the parent a model was fine-tuned, merged, quantized, or adapted from:

base_model: Qwen/Qwen3-8B

That field is genuinely useful. It builds the model tree, powers "derived from" links, and tells readers the lineage at a glance. But Hugging Face leaderboards use it as a signal too. The default leaderboard view treats any card that declares a base_model as a derivative and leaves it out, so that the board shows mostly from-scratch base models rather than the long tail of fine-tunes, merges, and quantized variants.

The intent is clean boards. The side effect is that fine-tunes, LoRA merges, GGUF and AWQ quantizations, and distillations all disappear from the first screen most visitors ever look at, even when they rank at or near the top.

How large is the hidden share, measured today?

We pulled every dataset tagged benchmark:official from the public Hub API, then fetched each attached leaderboard twice: once with default parameters and once with base_model=false. The difference between the two is exactly the set of entries the filter removes.

Snapshot taken 2026-10-07:

Quantity	Value
Datasets tagged `benchmark:official`	48
Boards with at least one entry	44
Entries visible by default	1,021
Entries with `base_model=false`	1,482
Hidden by the filter	461
Hidden share	31.1%
Boards hiding at least one entry	35 of 44

Nearly one in three entries across the official boards sits outside the default view. This is not a few stray cards. It is a structural feature of how the ecosystem publishes derivative models, which is to say most of it.

Run the same script on another day and the exact counts will move. That is expected. The method is fixed, each run records the state of the boards on its own date, and the share has held near 30 to 31% across our recent snapshots.

How do I see the hidden entries?

The leaderboard API accepts a base_model query parameter. Setting it to false disables the derivative filter and returns the full population:

GET https://huggingface.co/api/datasets/{dataset_id}/leaderboard
GET https://huggingface.co/api/datasets/{dataset_id}/leaderboard?base_model=false

The first call is what the default web view shows. The second is everything. For every board, the larger response is the full population and the default response is the visible subset.

Can I reproduce the 31.1% number?

Yes. The script below needs only requests and finishes in under a minute. It fetches both views of every official board and computes the hidden share directly.

import requests
import concurrent.futures as cf

H = "https://huggingface.co/api"
ds = requests.get(f"{H}/datasets",
                  params={"filter": "benchmark:official", "limit": 200},
                  timeout=60).json()
ids = [d["id"] for d in ds]

def fetch(i):
    def get(params):
        try:
            r = requests.get(f"{H}/datasets/{i}/leaderboard",
                             params=params, timeout=60).json()
            return r if isinstance(r, list) else []
        except Exception:
            return []
    return i, get({}), get({"base_model": "false"})

rows = list(cf.ThreadPoolExecutor(8).map(fetch, ids))

n_default = n_full = boards = affected = 0
for bid, default, full in rows:
    full = full if len(full) >= len(default) else default
    n_default += len(default)
    n_full += len(full)
    if full:
        boards += 1
    if len(full) > len(default):
        affected += 1

print("datasets", len(ids))
print("boards with entries", boards)
print("visible", n_default, "full", n_full)
print("hidden", n_full - n_default,
      "hidden %", round(100 * (1 - n_default / n_full), 1))
print("boards affected", affected, "of", boards)

On 2026-10-07 this prints 48 datasets, 44 boards with entries, 1,021 visible, 1,482 full, 461 hidden (31.1%), and 35 of 44 boards affected.

Why this matters for authors

If you fine-tune, merge, quantize, or distill a model and submit it to an official board, you probably added base_model to the card without a second thought. It is good lineage hygiene, and many upload scripts add it automatically. But the moment it is present, your result leaves the default screen. A visitor who does not know to toggle the filter, which is most of them, never sees your score. A derivative that legitimately ranks first can be invisible on the board it tops.

This is a documented ecosystem behavior, not a bug, and it is easy to work around once you know it exists. The goal is not to game the board. It is to make an honest result visible.

Checklist: keep your entry from being hidden

Know the rule before you submit. Any base_model field in your card moves the entry out of the default view. Decide on purpose, do not discover it after the fact.
Check your card after upload. Open the raw card and confirm whether base_model is present. Many trainers and quantization tools inject it silently.
Verify both views. Fetch /leaderboard and /leaderboard?base_model=false for your board and confirm which list your entry is in.
If visibility matters, reconsider the field. If your model is a legitimate standalone result, you may choose to omit base_model. Never fabricate lineage, and never strip a field only to deceive. Honesty first, visibility second.
If you keep the field, say so. State in your card and README that the entry appears only with the derivative filter off, and link the base_model=false view so readers can find it.
Re-check on release day. Boards re-index. Confirm your entry shows where you expect on the day you announce it.

FAQ

Does declaring base_model lower my score?
No. It changes visibility, not value. The score is identical in both views. The entry is simply filtered out of the default list.

Is the filter a bug?
No. It is intended behavior that keeps boards focused on from-scratch models. The issue is that most authors do not realize it applies to their derivative, so a real result goes unseen.

How do I see every entry on a board?
Add ?base_model=false to the leaderboard API call, or use the filter toggle in the board's web UI. That returns the full population including fine-tunes, merges, and quantized variants.

Will the 31.1% change over time?
Yes. It is a live snapshot from 2026-10-07. The method stays fixed and each run records that day's state. Across recent snapshots the share has stayed near 30 to 31%.

I quantized a popular model. Is my entry hidden?
Almost certainly, if your card declares the source via base_model, which quantization tooling often adds. Quantized variants are exactly the derivative class this filter removes.

Should I remove base_model to climb the board?
Only if your model truly is a standalone result. Do not strip accurate lineage to deceive. If you keep the field, document the hidden status and link the unfiltered view instead.

Method notes

All numbers come from the public Hugging Face Hub API on 2026-10-07. We counted a board as affected when its base_model=false response contained more entries than its default response. The full population per board is the larger of the two responses. No authentication is required, and the script above is the complete method.

Disclosure: QuantID publishes measurement and diagnostics notes. VIDRAFT is a technology alliance partner.

Further reading on this account: "Who Leads Hugging Face's Official Benchmarks? Concentration, Gaps and Hidden Entries" and "How Is Quantum Computer Performance Actually Measured? Quantum Volume Explained."

How Is Quantum Computer Performance Actually Measured? Quantum Volume Explained

QuantID — Tue, 06 Oct 2026 13:12:59 +0000

TL;DR

Qubit count is the headline number, but it does not tell you how much real computation a quantum machine can finish before noise wins.
Quantum Volume (QV) is a single whole-system score that folds together qubit number, gate fidelity, connectivity, and measurement quality into one figure.
QV is measured, not claimed: you run random square circuits, compare the output against an ideal classical simulation, and check whether the heavy output probability clears a fixed statistical bar.
A higher QV means the machine can run deeper and wider circuits that still carry signal, which is what actually matters for algorithms.
Quantinuum has led this benchmark for years and recently reported a QV of 2^25 (about 33.5 million) on its 56-qubit System Model H2, a figure you can verify from its own disclosures and third-party coverage.

If you only remember one thing: a 56-qubit machine with high QV can out-compute a 100-qubit machine with low QV, because volume measures usable work, not inventory.

Why is qubit count a misleading headline?

Ask most people how to compare two quantum computers and they will reach for qubit count. It is concrete, it fits in a headline, and it goes up over time. The problem is that a qubit is only useful for as long as it holds its state and interacts cleanly with its neighbors.

Three things quietly erode that usefulness:

Gate error. Every one-qubit and two-qubit operation has a small probability of corrupting the state. Errors compound as circuits get deeper.
Connectivity. If qubit A cannot talk directly to qubit D, the compiler inserts extra swap operations to move information around, and each swap is more error exposure.
Readout and preparation error. Measuring and initializing qubits is itself imperfect.

A machine can have many qubits and still be unable to run a circuit of meaningful depth, because the error budget is spent long before the computation finishes. Counting qubits is like rating a warehouse by the number of shelves while ignoring whether anything stays on them. This is exactly the gap that a measurement-first benchmark is designed to close.

What is Quantum Volume in one sentence?

Quantum Volume is the size of the largest random square circuit that a quantum computer can run and still produce an output that is statistically closer to the ideal result than to noise.

"Square" is the key word. A square circuit uses the same number of qubits and layers of gates. If a machine can reliably run a circuit on m qubits with m layers of random two-qubit gates, then its Quantum Volume is 2 raised to the power of m. The exponent m is sometimes called the achievable circuit width and depth.

Because QV scales as a power of two, the numbers get large fast. A QV of 2^10 is 1,024. A QV of 2^20 is about one million. A QV of 2^25 is about 33.5 million. The jumps look dramatic, but each extra power of two represents one more qubit successfully folded into a reliable square circuit, which is genuinely hard to win.

How is the quantum volume formula actually computed?

Here is the worked logic, step by step, without hiding the statistics.

Step 1: Build random square circuits

For a candidate width m, you generate many random circuits on m qubits with m layers. Each layer pairs qubits at random and applies random two-qubit unitaries. These circuits have no structure to exploit, which is the point: they stress the whole system evenly.

Step 2: Simulate the ideal output

For each random circuit, you compute the ideal output distribution on a classical computer. For every possible output bitstring you get an ideal probability. You then sort these probabilities and find the median. Any bitstring whose ideal probability is above the median is called a heavy output.

Step 3: Run the circuit on real hardware and measure heavy outputs

You run the same circuit many times on the actual quantum machine and record how often it returns a heavy output. That fraction is the heavy output probability, or HOP.

Step 4: Apply the pass or fail test

For a perfect machine, random square circuits produce heavy outputs with probability near 0.85. For pure noise, the HOP falls toward 0.5. The accepted bar is that the average HOP across the circuit set must exceed 2/3, and it must clear that bar with two-sigma statistical confidence, meaning the result is not a lucky fluctuation.

Step 5: Find the largest passing width

You find the largest m for which the machine passes. The Quantum Volume is then:

QV = 2 ** m_max

# worked example
m_max = 25            # largest square-circuit width that passes HOP > 2/3 at 2-sigma
QV    = 2 ** m_max    # 33,554,432  (reported as 2^25)
print(QV)             # 33554432

A short illustration of the pass check itself:

import statistics

def passes_qv(hops, threshold=2/3):
    # hops: list of per-circuit heavy output probabilities
    mean = statistics.mean(hops)
    sigma = statistics.pstdev(hops) / (len(hops) ** 0.5)  # standard error of the mean
    lower_2sigma = mean - 2 * sigma
    return lower_2sigma > threshold, mean, lower_2sigma

ok, mean, bound = passes_qv([0.71, 0.69, 0.70, 0.72, 0.68])
print(ok, round(mean, 3), round(bound, 3))

The elegance is that a single scalar now reflects qubit count, two-qubit fidelity, connectivity overhead, and readout quality at the same time. You cannot fake it by adding idle qubits, because idle qubits that cannot join a reliable square circuit do not raise m.

Why does Quantum Volume matter more than qubit count?

QV answers the question an algorithm designer actually cares about: how deep and wide can I go before my results dissolve into noise? That is the operational ceiling.

Consider two machines:

Machine A: 100 qubits, modest fidelity, limited connectivity. It may only pass square circuits up to width 10, so QV is about 1,024.
Machine B: 56 qubits, high fidelity, all-to-all connectivity. It may pass square circuits up to width 25, so QV is about 33.5 million.

Machine B can run far larger reliable computations despite having fewer qubits. For near-term algorithms, where every extra layer of depth is precious, Machine B is the more capable device by a wide margin. This is why serious buyers and researchers read QV, error rates, and clock speed together rather than fixating on a single qubit tally.

What did Quantinuum actually report, and can you verify it?

Quantinuum has led the Quantum Volume benchmark for several years, roughly doubling its score on a regular cadence. According to the company's own disclosures and independent coverage, its 56-qubit trapped-ion System Model H2 reached a Quantum Volume of 2^25, which is 33,554,432. Reports describe this as a step up from an earlier 2^23 (about 8 million) figure the same year, and note that the H2 system uses all-to-all connectivity, which reduces the swap overhead that hurts fixed-layout hardware.

Two measurement notes keep this honest:

The score is a system-level claim validated by the HOP test above, not a marketing abstraction.
Independent outlets reported the same 2^25 figure, which is the kind of corroboration a measurement-first reader should look for before repeating a number.

Where a specific secondary detail could not be independently pinned down, treat it as approximate. The core figure to anchor on is QV equals 2^25 on a 56-qubit H2 system, which is well documented.

How should you read any quantum benchmark claim?

A short measurement checklist:

Is the metric whole-system or component-level? QV is whole-system. A single two-qubit fidelity number is component-level and tells you less on its own.
Was it validated against an ideal reference? QV compares to classical simulation. That is why QV itself stops being measurable once machines pass the size that classical computers can still simulate, which pushes the field toward newer benchmarks.
Is there a statistical bar? QV requires two-sigma confidence over the 2/3 threshold. Claims without error bars deserve caution.
Is it reproducible or corroborated? Prefer numbers that appear in primary disclosures and independent coverage.

FAQ

Is Quantum Volume the only benchmark that matters?

No. QV is an excellent single-number summary for near-term machines, but it saturates once circuits grow beyond classical simulation. The field now also tracks metrics like circuit layer operations per second (CLOPS) for speed, algorithmic qubit counts, and application-level benchmarks. Read them together.

Does a higher Quantum Volume always mean a better computer for my problem?

Usually it means more usable computational room, but the right machine depends on your workload. Some problems care more about speed or specific gate sets. QV is a strong general indicator, not a guarantee for every algorithm.

Why is Quantum Volume expressed as a power of two?

Because it reports the size of the state space of the largest reliable square circuit. A circuit on m qubits spans 2^m possible outcomes, so QV equals 2^m. Each extra reliable qubit doubles the score, which is why progress looks like a staircase of powers of two.

What is the heavy output probability threshold and why 2/3?

The 2/3 bar sits between the noise floor near 0.5 and the ideal value near 0.85. Passing it with two-sigma confidence shows the machine is meaningfully closer to ideal behavior than to random noise, which is the minimum evidence that the circuit carried real signal.

Can a vendor inflate Quantum Volume by adding qubits?

Not easily. Idle or low-fidelity qubits that cannot participate in a reliable square circuit do not increase the achievable width m, so they do not raise QV. That resistance to padding is a big part of why the benchmark earned trust.

Where can I find the raw data behind a QV claim?

Start with the vendor's technical disclosure, then look for independent reporting and any published circuit sets or HOP distributions. Corroboration across primary and secondary sources is the measurement-first standard.

Closing note

Quantum Volume is a reminder that in computing, as in science, the number that fits on a slide is rarely the number that governs real capability. Measure the whole system, hold it to a statistical bar, and prefer scores you can verify. That discipline is exactly how you separate a machine that looks impressive from one that can actually finish the job.

Disclosure: QuantID is a technology alliance partner of VIDRAFT, which was recently selected to use Quantinuum's platform. This article is written from a neutral, measurement-first perspective and relies on publicly reported figures.

Who Leads Hugging Face's Official Benchmarks? Concentration, Gaps and Hidden Entries (Measured 2026-10-06)

QuantID — Mon, 05 Oct 2026 08:57:14 +0000

TL;DR

On 2026-10-06 we read every dataset tagged benchmark:official from the public Hugging Face Hub API, and every leaderboard attached to those datasets, twice: once in the default view and once with base_model=false. Every number below is a reading taken on that date, and about 40 lines of Python reproduce it.

48 official benchmark datasets, 44 with at least one leaderboard entry.
Default view: 1,024 entries from 95 organizations across 411 distinct models.
The top 5 organizations hold 40.3% of visible entries, the top 10 hold 59.0%.
Full population (with base_model=false): 1,479 entries, 161 namespaces. Gini 0.709, HHI 0.0369.
Median relative gap between #1 and #2: 3.69%. On 9 of 43 comparable boards the gap is under 1%.
455 entries (30.8%) never appear in the default view because the board hides models that declare a base_model. 34 of 44 boards are affected.

Many organizations take part, but entry counts are uneven. On about one board in five the top two are close enough that the order could flip on a rerun, and nearly a third of all submissions only show up if you change a query parameter.

Why measure benchmark leaderboards instead of ranking them?

QuantID works on measurement and diagnostics, so we treat a leaderboard as an instrument rather than a scoreboard. A leaderboard is a fixed test, a scoring rule and a published ordering. Like any instrument it has a resolution, a bias and a display layer that can hide part of the signal. Most writing about leaderboards asks who is first. We ask four questions about the instrument itself:

How many independent parties actually participate?
How unequal is that participation?
How far apart are the leaders, compared with plausible run-to-run noise?
How much of the data is filtered away before a human sees it?

Hugging Face lets model owners attach evaluation results to a model repository through .eval_results/*.yaml files. Datasets tagged as official benchmarks collect those files into a board. The whole system answers to a public API, so it can be audited like any other measurement pipeline.

What data did we read?

Three public calls, no authentication:

GET https://huggingface.co/api/datasets?filter=benchmark:official&limit=200
GET https://huggingface.co/api/datasets/{id}/leaderboard
GET https://huggingface.co/api/datasets/{id}/leaderboard?base_model=false

Each row carries rank, value, modelId, lower_is_better, a source link and author metadata. We define the organization of an entry as the namespace before the slash in modelId (so Qwen/Qwen3.8-Flash-Next belongs to Qwen). User namespaces count as organizations. We do not merge related accounts: the API does not say which accounts belong together, and guessing would inject our own bias into the measurement.

For each board, the larger of the two responses is the full population (in practice the base_model=false response). The default response is the visible population.

Quantity	Default view	Full population
Benchmark datasets	48	48
Boards with entries	44	44
Entries	1,024	1,479
Distinct models	411	647
Distinct organizations	95	161

New YAML files are merged daily, so the boards keep changing. Treat every number here as a snapshot, not a constant.

How concentrated is AI benchmark participation?

"Concentration" covers two separate questions: who submits, and who wins.

In the default view the five largest submitters are Qwen (136), zai-org (84), deepseek-ai (65), moonshotai (65) and nvidia (63). Together those five hold 40.3% of the 1,024 visible entries, and the top ten hold 59.0%. Entry volume mostly tracks how many model sizes a lab ships; it says little about quality.

Across the full population the inequality is easier to summarize with two numbers that seem to disagree:

Metric	Value
Gini (entries by org)	0.709
HHI	0.0369
Effective number of orgs (1/HHI)	about 27
Orgs appearing on exactly one board	86 of 161

In antitrust terms an HHI of 0.0369 is unconcentrated (below 0.15) because no single organization holds a large slice; the largest, Qwen, has 190 of 1,479 entries (12.8%). A Gini of 0.709 says the distribution is still very unequal. The long tail explains both readings: more than half of all organizations appear exactly once, while a few large labs submit many model sizes to many boards.

Reporting both indicators keeps "no monopoly" from being read as "equal participation". They are answers to different questions.

Who actually wins?

We find the #1 entry of each board by sorting on value: descending by default, ascending when lower_is_better is true. Across 44 boards, 24 distinct organizations hold at least one #1 position. The most frequent holders in this snapshot were FINAL-Bench (9 boards), zai-org (4), deepseek-ai (3) and moonshotai (3), then XiaomiMiMo, ornith-ai, tsinghua-sigs-robot-lab and tencent with 2 each. Leadership is spread across roughly two dozen teams, and the biggest submitters are mostly not the most frequent winners: Qwen has the most entries overall but holds the top spot on only one board in this reading.

How close are the top two models on a typical benchmark?

Concentration tells you who shows up. Competitiveness tells you whether the order at the top means anything. For each board with at least two entries we compute the relative gap between #1 and #2:

$$g = \frac{|v_1 - v_2|}{|v_1|} \times 100\%$$

A relative measure lets us compare boards scored in percent, in error rate or in arbitrary units. Measured over 43 comparable boards, the median gap is 3.69%, and the distribution is wide:

Why does a gap under 1% matter? Most of these benchmarks have a few hundred to a few thousand items. For a binomial accuracy $p$ on $n$ items the standard error is $\sqrt{p(1-p)/n}$. At $p = 0.8$ and $n = 1{,}000$ that is about 1.3 percentage points, or roughly 1.6% in relative terms. Self-reported runs also differ in sampling temperature, prompt template and harness. With all that, a lead under 1% usually cannot be told apart from a tie, so on about one board in five the current #1 could change on a rerun. At the other end, a handful of boards show leads above 20%; that usually means a genuinely new capability, a board with very few entries, or entries scored under different conditions. A large gap is a reason to open the source link, not a conclusion by itself. Our rule for any ranking: report the gap with an uncertainty estimate (2 SE or a bootstrap interval) and call it a tie when the interval contains zero.

What share of Hugging Face leaderboard entries are hidden by default?

The default /leaderboard response, and the default web view, leaves out entries from models whose card declares a base_model. That removes fine-tunes, merges, quantized variants and other derivative releases. Passing base_model=false brings them back.

$$\text{hidden share} = 1 - \frac{\sum_b |L_b^{default}|}{\sum_b |L_b^{full}|}$$

Measured: 455 of 1,479 entries (30.8%) are hidden in the default view, and 34 of 44 boards hide at least one entry.

The filter is a design choice, not a bug: it keeps the default view focused on original models. It still has measurable consequences. A fine-tuned model can outscore its base and stay invisible to anyone who does not change the query. Rankings quoted from the default page and rankings computed with the full population can disagree. Concentration statistics computed only on the default view understate the long tail, because small teams publish a large share of the derivative work. If you report a position on one of these boards, say which view it comes from.

How can I reproduce this measurement?

The script below needs only the standard library plus requests, and runs in under a minute.

import requests, collections, statistics as st
import concurrent.futures as cf

H = "https://huggingface.co/api"
ids = [d["id"] for d in requests.get(
    f"{H}/datasets", params={"filter": "benchmark:official", "limit": 200}, timeout=60).json()]

def fetch(i):
    def get(p):
        try:
            r = requests.get(f"{H}/datasets/{i}/leaderboard", params=p, timeout=60).json()
            return r if isinstance(r, list) else []
        except Exception:
            return []
    return i, get({}), get({"base_model": "false"})

rows = list(cf.ThreadPoolExecutor(8).map(fetch, ids))

def org(e): return e["modelId"].split("/")[0]
def gini(x):
    x = sorted(x); n = len(x); s = sum(x)
    return sum((2*i - n - 1) * v for i, v in enumerate(x, 1)) / (n * s)

entries, firsts = collections.Counter(), collections.Counter()
gaps, n_default, n_full = [], 0, 0
for bid, default, full in rows:
    full = full if len(full) >= len(default) else default
    n_default += len(default); n_full += len(full)
    for e in full:
        entries[org(e)] += 1
    if not full:
        continue
    lib = full[0].get("lower_is_better", False)
    s = sorted(full, key=lambda e: e["value"], reverse=not lib)
    firsts[org(s[0])] += 1
    if len(s) > 1 and s[0]["value"]:
        gaps.append(abs(s[0]["value"] - s[1]["value"]) / abs(s[0]["value"]) * 100)

v = sorted(entries.values(), reverse=True); N = sum(v)
print("orgs", len(entries), "entries", N, "gini", round(gini(v), 3))
print("top5", round(100*sum(v[:5])/N, 1), "top10", round(100*sum(v[:10])/N, 1))
print("median gap %", round(st.median(gaps), 2), "gap<1%", sum(g < 1 for g in gaps))
print("hidden %", round(100*(1 - n_default/n_full), 1))
print("top #1 holders", firsts.most_common(8))

Run it on another day and the numbers will differ. That is expected: the method stays fixed, and each run records the state of the boards on its own date.

What are the limits of this measurement?

An organization here is a namespace, not a legal entity. One company may publish under several namespaces, and one namespace may host several teams. We did not merge or split any.
Scores are mostly self-reported. Most rows carry verified: false and cite a model card. We measure the published board, not the truth behind each number.
Ties and sort direction. We trust the lower_is_better flag and treat multi-metric datasets exactly as the API returns them.
Snapshot effect. Entries arrive as pull requests to model repositories, so one large release can move the top-k share by several points in a day.
The gap metric ignores noise. A relative gap has no units and no per-board significance test; the binomial SE above is a rough guide.

Disclosure

QuantID is a technology alliance partner of VIDRAFT, which appears on several boards (including under the FINAL-Bench namespace); data shown here is computed identically for every organization.

FAQ

How many official benchmark leaderboards are on Hugging Face?
On 2026-10-06 the API returned 48 datasets tagged benchmark:official. Of those, 44 had at least one leaderboard entry.

Is AI benchmark leadership dominated by a few labs?
Participation is unequal: the full-population Gini is 0.709 and the top 10 organizations hold 59.0% of visible entries. Still, #1 positions are spread across 24 organizations, and no organization holds more than about 13% of all entries.

Why don't I see some models on a Hugging Face leaderboard?
The default view hides entries from models that declare a base_model in their card. Add ?base_model=false to the API call to see them. In this snapshot that filter hid 30.8% of all entries across 34 of 44 boards.

How close are the top two models on a typical board?
The median relative gap between #1 and #2 was 3.69%. On 9 of 43 boards it was under 1%, which is usually within run-to-run noise.

Why report both Gini and HHI?
HHI is driven by the biggest shares, so it stays low when no single organization is large. Gini measures inequality across the whole distribution, including the long tail of single-entry participants. Reporting both prevents "no monopoly" from being read as "equal participation".

Can I get these numbers live?
Yes. The script above reproduces them, and we maintain a live map of all official boards as a public Space.

DEV Community: QuantID

How Do You Show Two Training Runs Are Equal? A Paired Test on Unseen Text, Using an NPU vs GPU Case

TL;DR

What makes a fair comparison between two runs?

Is "held out" really held out?

Why compare windows in pairs instead of averages?

What else should a reader check?

FAQ

Further reading

Hugging Face Official Benchmarks: Daily Update 2026-10-24

What changed since the last snapshot

Organizations by number of #1 spots

Current #1 on every official benchmark

How to read this

Hugging Face Official Benchmarks: Daily Update 2026-10-23

What changed since the last snapshot

Organizations by number of #1 spots

Current #1 on every official benchmark

How to read this

Hugging Face Official Benchmarks: Daily Update 2026-10-20

What changed since the last snapshot

Organizations by number of #1 spots

Current #1 on every official benchmark

How to read this

Hugging Face Official Benchmarks: Daily Update 2026-10-18

Organizations by number of #1 spots

Current #1 on every official benchmark

How to read this

Measuring the Real Cost of an LLM Call: Tokens, Precision, and Honest Numbers

TL;DR

The sticker is not the cost

Precision deserves its own line

Latency is a cost, not a vibe

A measurement checklist

Why this matters now

FAQ

Further reading

Eleven Number-One Records: Measuring a Model That Swept Math, Science, Law and Decisions

TL;DR

The eleven records

Why the spread is the real signal

How these were produced

Check it yourself

FAQ

Reading the S1MB result: why Borda aggregation makes a #1 harder to fake

Borda, not average

The current top

Why a deterministic judge is clean to measure

Links

The base_model Filter: Why 31% of Hugging Face Leaderboard Entries Are Hidden by Default

TL;DR

What is the base_model filter and why does it hide entries?

How large is the hidden share, measured today?

How do I see the hidden entries?

Can I reproduce the 31.1% number?

Why this matters for authors

Checklist: keep your entry from being hidden

FAQ

Method notes

How Is Quantum Computer Performance Actually Measured? Quantum Volume Explained

TL;DR

Why is qubit count a misleading headline?

What is Quantum Volume in one sentence?

How is the quantum volume formula actually computed?

Step 1: Build random square circuits

Step 2: Simulate the ideal output

Step 3: Run the circuit on real hardware and measure heavy outputs

Step 4: Apply the pass or fail test

Step 5: Find the largest passing width

Why does Quantum Volume matter more than qubit count?

What did Quantinuum actually report, and can you verify it?

How should you read any quantum benchmark claim?

FAQ

Is Quantum Volume the only benchmark that matters?

Does a higher Quantum Volume always mean a better computer for my problem?

Why is Quantum Volume expressed as a power of two?

What is the heavy output probability threshold and why 2/3?

Can a vendor inflate Quantum Volume by adding qubits?

Where can I find the raw data behind a QV claim?

Closing note