DEV Community: huggingface

Training FLUX.2 and Kolors LoRAs on a space dataset

Khadim Hussain — Tue, 13 Oct 2026 07:26:26 +0000

Originally published on khadim.tech.

In short: A space-themed image generation project: a dataset of 1,964 image-caption pairs from NASA, Wallhaven and LAION-Aesthetics, auto-captioned with JoyCaption, and LoRA adapters trained on FLUX.2-klein (9B) and Kolors. The dataset has 3,300+ downloads on Hugging Face.

The dataset

Kiffusion is 1,964 image-caption pairs of space and sci-fi imagery, collected from NASA, Wallhaven and LAION-Aesthetics. Every image was captioned automatically with JoyCaption, so the captions describe composition and lighting in the detail a diffusion model needs, instead of relying on short alt text.

The LoRAs

SpaceVision is the set of LoRA adapters trained on that dataset, using SimpleTuner and Diffusers:

FLUX.2-klein (9B), a 38 MB adapter
Kolors, a 46.6 MB adapter

A LoRA trains a small set of extra weights instead of the whole model, so the adapters stay small enough to download and swap in seconds.

What got used

Download counts from Hugging Face, all time, as of October 2026:

Artifact	Type	Downloads
kiffusion-space-scifi	Dataset	3,306
spacevision-flux2-lora	LoRA for FLUX.2-klein	204
spacevision-kolors-lora	LoRA for Kolors	66

The dataset is used far more than the adapters, which suggests that a well-captioned niche dataset is the more reusable artifact.

Testing the Hugging Face Inference API (Free Tier)

Phil IPS — Mon, 12 Oct 2026 05:24:26 +0000

Testing the Hugging Face Inference API (Free Tier)

Learn how to use Hugging Face's free inference API for AI model testing.

Prerequisites

Python 3.8+
requests library: pip install requests
Hugging Face account (free at https://huggingface.co)

Quick Start

import requests
import os

# Get your free token from Hugging Face settings
HF_TOKEN = os.getenv("HF_TOKEN")

# Use a free model (no auth required for some)
MODEL = "google/flan-t5-base"

def query_hf(prompt: str, model: str = MODEL) -> str:
    """Query Hugging Face Inference API."""
    url = f"https://api-inference.huggingface.co/models/{model}"
    headers = {"Authorization": f"Bearer {HF_TOKEN}"} if HF_TOKEN else {}

    response = requests.post(
        url,
        headers=headers,
        json={"inputs": prompt},
        timeout=30
    )

    if response.status_code == 200:
        return response.json()[0]["generated_text"]
    else:
        raise Exception(f"Error: {response.status_code} - {response.text}")

# Usage
if __name__ == "__main__":
    prompt = "Translate to French: Hello, how are you?"
    try:
        result = query_hf(prompt)
        print(f"Prompt: {prompt}")
        print(f"Result: {result}")
    except Exception as e:
        print(f"Error: {e}")

Testing Different Free Models

# Text generation
models = [
    "gpt2",
    "facebook/opt-350m",
    "google/flan-t5-base",
    "microsoft/DialoGPT-small"
]

for model in models:
    try:
        result = query_hf("The future of AI is", model)
        print(f"{model}: {result[:100]}...")
    except Exception as e:
        print(f"{model}: Error - {e}")

Error Handling & Rate Limits

import time

def robust_query(prompt: str, model: str = MODEL, retries: int = 3) -> str:
    """Query with exponential backoff for rate limits."""
    for attempt in range(retries):
        try:
            return query_hf(prompt, model)
        except Exception as e:
            if "429" in str(e) or "503" in str(e):
                wait = 2 ** attempt
                print(f"Rate limited. Waiting {wait}s...")
                time.sleep(wait)
            else:
                raise
    raise Exception("Max retries exceeded")

Conclusion

Hugging Face's free inference API provides access to thousands of models without any cost. Perfect for experimentation, prototyping, and learning AI integration.

Hugging Face Hits 100K Models: A New Era for Open AI

TechPulse — Sun, 11 Oct 2026 05:42:42 +0000

The 100,000 Model Milestone

The open-source artificial intelligence community has reached a historic threshold. Hugging Face, the leading platform for collaborative machine learning, officially announced this week that it has surpassed 100,000 active models on its Hub. This incredible surge in content, driven by the rapid adoption of large language models (LLMs) and multimodal architectures, marks a significant inflection point in the democratization of AI technology.

From Centralized to Distributed Innovation

For years, the AI landscape was dominated by a handful of tech giants who controlled access to state-of-the-art models. However, the last 18 months have seen a dramatic shift. Developers, researchers, and startups are no longer waiting for corporate releases to innovate. Instead, they are fine-tuning, quantizing, and adapting open-weights models like Llama 3, Mistral, and Stable Diffusion for specific niche applications.

"We are witnessing a shift from a centralized model ecosystem to a distributed one," said Clément Delangue, CEO and co-founder of Hugging Face. "When you have 100,000 models, you aren't just looking at general-purpose tools. You're seeing specialized engines for legal compliance, medical diagnosis, and creative coding that didn't exist two years ago. This is the true power of open source."

This decentralization lowers the barrier to entry for small businesses and independent developers. Previously, deploying a custom AI solution required significant capital for API costs or proprietary hardware. Now, a startup can download an open-weights model, fine-tune it on their own proprietary data, and deploy it on affordable cloud infrastructure, retaining full ownership of their intellectual property.

Industry Impact and Enterprise Adoption

The implications of this milestone are far-reaching. For enterprises, the abundance of open-source models provides a crucial hedge against vendor lock-in. Companies can now benchmark multiple open architectures against proprietary APIs, often finding that open-weights models offer competitive performance at a fraction of the cost.

Furthermore, the ecosystem effect is accelerating tool development. Frameworks like LangChain, LlamaIndex, and vLLM have matured rapidly to support this diverse library of models. This infrastructure maturity means that deploying a production-grade RAG (Retrieval-Augmented Generation) system is no longer a months-long R&D project but a matter of days for experienced engineering teams.

Security and compliance experts are also taking notice. With the ability to run models locally on-premises, sensitive data never leaves the company's firewall, addressing major concerns regarding privacy and data sovereignty that often hinder AI adoption in regulated industries like finance and healthcare.

What's Next: The Race for Efficiency

As the number of models continues to grow, the focus of the community is shifting from sheer scale to efficiency. The next frontier is not just building larger models, but making existing ones smaller and faster. Techniques such as quantization, pruning, and distillation are becoming standard practice, allowing high-performance inference on consumer-grade hardware.

Analysts predict that within the next 12 months, we will see a consolidation of the top-performing open models, with a clear hierarchy emerging. However, the long tail of specialized, domain-specific models will remain the primary driver of innovation. For startups, this means the competitive advantage is no longer having access to the best AI, but rather the ability to apply the right open model to a specific business problem with speed and precision.

The 100,000-model milestone is not just a number; it is a testament to the collaborative power of the open-source community. As we look toward 2025, the open AI ecosystem is poised to become the default infrastructure for the next generation of digital products, fundamentally reshaping how we build and deploy intelligent software.

Hugging Face Hits 1M Models: Open Source AI's Big Leap

TechPulse — Sun, 11 Oct 2026 04:42:57 +0000

A Landmark Moment for Collaborative AI

The open-source AI ecosystem just hit a historic high. Hugging Face, the leading collaborative platform for machine learning, announced today that its community has officially surpassed one million active model repositories. This milestone, reached in late 2023, marks a significant acceleration in the democratization of artificial intelligence, moving beyond just large language models to include a vast array of specialized, fine-tuned, and experimental architectures.

From Research to Reality

This surge isn't just about numbers; it represents a fundamental shift in how AI is developed. Previously, state-of-the-art models were largely siloed within big tech giants like OpenAI, Google, and Meta. Today, thanks to platforms like Hugging Face, independent developers, academic researchers, and small startups can access, fine-tune, and deploy these powerful tools transparently. The platform now hosts everything from text-to-image generators to code assistants, with a growing emphasis on efficiency and low-resource deployment.

"We have seen an explosion of creativity," said Clément Delangue, Co-founder and CEO of Hugging Face, in a recent interview. "The barrier to entry for building sophisticated AI applications has never been lower. This one-million-model milestone proves that the future of AI is not a single warehouse, but a diverse, open marketplace of ideas."

Why This Matters for Startups

For the startup sector, this development is a game-changer. Building a proprietary AI model from scratch is prohibitively expensive and time-consuming. Instead, startups can now leverage the 'Hugging Face Hub' to find pre-trained bases and fine-tune them on their specific data. This reduces development cycles from months to weeks, allowing agile teams to iterate faster and compete with larger corporations.

Moreover, the transparency of open source allows for better security auditing and bias detection, which is critical for companies looking to deploy AI in regulated industries like healthcare and finance. The ecosystem has also matured with better tooling for version control, collaboration, and deployment, making production-ready AI more accessible than ever before.

The Economic Implications

The open-source movement is creating a new digital economy. Developers are not only contributing code but also monetizing their expertise through consulting, specialized fine-tuning services, and hosting solutions. This has spawned a new class of 'AI-native' startups that build value layers on top of open models, focusing on specific verticals like legal tech, biotech, and creative design.

Industry analysts note that this trend is pressuring closed-source providers to offer more competitive pricing and better developer experiences. The race is no longer just about who has the biggest model, but who can build the most useful ecosystem around it.

What's Next

Looking ahead, Hugging Face is expected to focus on reducing the computational cost of running these models. With the rise of quantized models and smaller, more efficient architectures, the platform aims to make high-performance AI accessible on consumer-grade hardware. We can also expect tighter integrations with cloud providers and enterprise solutions to bridge the gap between open-source experimentation and enterprise-grade reliability. As the model count continues to climb, the challenge will shift from availability to curation, helping developers navigate an increasingly crowded landscape to find the right tool for their specific needs.

EmbeddingGemma 2 puts multimodal search on a phone

techaiwire — Wed, 07 Oct 2026 19:47:19 +0000

Google DeepMind released EmbeddingGemma 2 on October 6, 2026, an open model that turns text, code, images, video and audio into vectors in one shared space. It has 740 million parameters and ships under the Apache 2.0 license, according to Google's developer guide. The text-only version runs in about 191 MB of memory on a phone. That makes it practical to search a user's photos, voice notes and documents without sending them to a server.

What an embedding model does

An embedding model turns a piece of content into a list of numbers, called a vector. Similar content ends up with similar vectors. Search then becomes a matter of finding the nearest vectors to a query.

This is the engine behind RAG, or retrieval-augmented generation, where an app finds relevant documents before asking a language model to answer. EmbeddingGemma 2 puts every kind of media in the same space. A text query like "dog on a beach" can therefore find a photo, a video frame or an audio clip directly.

The edge guide describes it as a model that "maps text, images, video frames, and audio into a single, unified vector space."

How it is built

The model is made of stackable encoders that share one embedding space:

Part	Parameters
Text and code encoder	270M
Vision encoder	170M
Audio encoder	300M
Total	740M

Stackable means a text-only app can load just the 270M core, and add vision or audio only if it needs them.

The output is a 768-dimension vector. It can be cut down to 512, 256 or 128 dimensions using Matryoshka Representation Learning, a training method that packs the most important information into the front of each vector. Google says 128 dimensions cuts storage by 6x. "Most of the full quality of the original embedding on text and code is retained and about 95% on image, video, and speech retrieval," the developer guide says.

The context window is 8,192 tokens, four times the first version's. Unite.AI lists what fits in one call:

29 images, at 280 tokens each
58 video frames, at 140 tokens each
5.5 minutes of audio, at 25 tokens per second

Unite.AI adds that the model supports more than 100 languages and has a training data cutoff of January 2025.

Benchmarks and speed

Test	EmbeddingGemma 2	Source
MTEB Code	78.68 (EmbeddingGemma 1: 68.76)	Unite.AI, Crypto Briefing
MTEB multilingual	61.36	Unite.AI
MMEB v2 image retrieval	57.28	Unite.AI
MMEB v2 video retrieval	50.67	Unite.AI
MSEB audio	69.54	Unite.AI

The code score is the biggest jump, a gain of 9.92 points. Google's guide describes it as 14% higher than the first version.

On hardware, Google reports about 191 MB of RAM for text-only use and about 567 MB for full multimodal use on a Pixel 11 Pro, with quantization. The edge guide measured image embedding at 37.3 milliseconds on a MacBook M5 Pro GPU, or 26.9 images a second.

The same guide says the model can sort an input among 500 options in under 100 milliseconds, with no training needed. That puts it close to the new class of decision models such as Cloudflare's Clef.

Where to get it

The weights are on Hugging Face as google/embeddinggemma-2, and on Kaggle and the LiteRT Community. Crypto Briefing lists support in sentence-transformers, Hugging Face transformers, LiteRT and MediaPipe, MLX and Ollama. Google also ships pre-quantized LiteRT bundles as .litertlm files.

What this means for developers

Move search onto the device. At about 191 MB for text, the model fits in a mobile app. Private content can be indexed locally, which Unite.AI notes means personal files "don't have to be uploaded to a server just to become searchable."
Use truncation to cut storage. If your vector database is large, test 256 or 128 dimensions. Google reports a 6x saving at 128 with most text quality kept.
Re-embed if you change models. Vectors from EmbeddingGemma 1 and 2 are not interchangeable. Plan a full re-index rather than mixing them.
Try it for code search. A 9.92-point gain on MTEB Code makes it worth testing for repository search and code RAG.
Load only the encoders you need. A text-only feature does not need the 470M parameters of vision and audio encoders. Smaller loads mean faster start-up on phones.

Run it against your current embedding model on your own queries first. The public benchmarks show the direction, and your own data shows the real gain.

This article was first published on Tech AI Wire.

Also available in

Deutsch · 日本語 · Français · Español · Português

Related on Tech AI Wire

Cloudflare Clef: open-weight models that return decisions

Sources

EmbeddingGemma 2: The Developer Guide - Google Developers Blog
Bring multimodal semantic search to the edge with EmbeddingGemma 2 - Google Developers Blog
DeepMind Debuts EmbeddingGemma 2, Mapping Five Modalities Into One Space - Unite.AI
Google releases EmbeddingGemma 2, a 740 million parameter open-weight model under Apache 2.0 - Crypto Briefing

The base_model Filter: Why 31% of Hugging Face Leaderboard Entries Are Hidden by Default

QuantID — Tue, 06 Oct 2026 19:13:14 +0000

TL;DR

On a Hugging Face official leaderboard, the default view silently drops any model whose card declares a base_model field. That one line of YAML, added so people can see what your model was built from, moves your entry out of sight. We measured the effect live on 2026-10-07 across every official benchmark board: 461 of 1,482 entries (31.1%) are hidden in the default view, and 35 of 44 populated boards hide at least one entry. The fix is a single API parameter, base_model=false, and this post gives you a reproducible script plus an author checklist so your own submission does not vanish.

This is a measurement note, not a complaint. The filter is a reasonable default. The problem is that most authors do not know it applies to them.

What is the base_model filter and why does it hide entries?

Model cards on the Hub carry YAML metadata. One common field is base_model, which records the parent a model was fine-tuned, merged, quantized, or adapted from:

base_model: Qwen/Qwen3-8B

That field is genuinely useful. It builds the model tree, powers "derived from" links, and tells readers the lineage at a glance. But Hugging Face leaderboards use it as a signal too. The default leaderboard view treats any card that declares a base_model as a derivative and leaves it out, so that the board shows mostly from-scratch base models rather than the long tail of fine-tunes, merges, and quantized variants.

The intent is clean boards. The side effect is that fine-tunes, LoRA merges, GGUF and AWQ quantizations, and distillations all disappear from the first screen most visitors ever look at, even when they rank at or near the top.

How large is the hidden share, measured today?

We pulled every dataset tagged benchmark:official from the public Hub API, then fetched each attached leaderboard twice: once with default parameters and once with base_model=false. The difference between the two is exactly the set of entries the filter removes.

Snapshot taken 2026-10-07:

Quantity	Value
Datasets tagged `benchmark:official`	48
Boards with at least one entry	44
Entries visible by default	1,021
Entries with `base_model=false`	1,482
Hidden by the filter	461
Hidden share	31.1%
Boards hiding at least one entry	35 of 44

Nearly one in three entries across the official boards sits outside the default view. This is not a few stray cards. It is a structural feature of how the ecosystem publishes derivative models, which is to say most of it.

Run the same script on another day and the exact counts will move. That is expected. The method is fixed, each run records the state of the boards on its own date, and the share has held near 30 to 31% across our recent snapshots.

How do I see the hidden entries?

The leaderboard API accepts a base_model query parameter. Setting it to false disables the derivative filter and returns the full population:

GET https://huggingface.co/api/datasets/{dataset_id}/leaderboard
GET https://huggingface.co/api/datasets/{dataset_id}/leaderboard?base_model=false

The first call is what the default web view shows. The second is everything. For every board, the larger response is the full population and the default response is the visible subset.

Can I reproduce the 31.1% number?

Yes. The script below needs only requests and finishes in under a minute. It fetches both views of every official board and computes the hidden share directly.

import requests
import concurrent.futures as cf

H = "https://huggingface.co/api"
ds = requests.get(f"{H}/datasets",
                  params={"filter": "benchmark:official", "limit": 200},
                  timeout=60).json()
ids = [d["id"] for d in ds]

def fetch(i):
    def get(params):
        try:
            r = requests.get(f"{H}/datasets/{i}/leaderboard",
                             params=params, timeout=60).json()
            return r if isinstance(r, list) else []
        except Exception:
            return []
    return i, get({}), get({"base_model": "false"})

rows = list(cf.ThreadPoolExecutor(8).map(fetch, ids))

n_default = n_full = boards = affected = 0
for bid, default, full in rows:
    full = full if len(full) >= len(default) else default
    n_default += len(default)
    n_full += len(full)
    if full:
        boards += 1
    if len(full) > len(default):
        affected += 1

print("datasets", len(ids))
print("boards with entries", boards)
print("visible", n_default, "full", n_full)
print("hidden", n_full - n_default,
      "hidden %", round(100 * (1 - n_default / n_full), 1))
print("boards affected", affected, "of", boards)

On 2026-10-07 this prints 48 datasets, 44 boards with entries, 1,021 visible, 1,482 full, 461 hidden (31.1%), and 35 of 44 boards affected.

Why this matters for authors

If you fine-tune, merge, quantize, or distill a model and submit it to an official board, you probably added base_model to the card without a second thought. It is good lineage hygiene, and many upload scripts add it automatically. But the moment it is present, your result leaves the default screen. A visitor who does not know to toggle the filter, which is most of them, never sees your score. A derivative that legitimately ranks first can be invisible on the board it tops.

This is a documented ecosystem behavior, not a bug, and it is easy to work around once you know it exists. The goal is not to game the board. It is to make an honest result visible.

Checklist: keep your entry from being hidden

Know the rule before you submit. Any base_model field in your card moves the entry out of the default view. Decide on purpose, do not discover it after the fact.
Check your card after upload. Open the raw card and confirm whether base_model is present. Many trainers and quantization tools inject it silently.
Verify both views. Fetch /leaderboard and /leaderboard?base_model=false for your board and confirm which list your entry is in.
If visibility matters, reconsider the field. If your model is a legitimate standalone result, you may choose to omit base_model. Never fabricate lineage, and never strip a field only to deceive. Honesty first, visibility second.
If you keep the field, say so. State in your card and README that the entry appears only with the derivative filter off, and link the base_model=false view so readers can find it.
Re-check on release day. Boards re-index. Confirm your entry shows where you expect on the day you announce it.

FAQ

Does declaring base_model lower my score?
No. It changes visibility, not value. The score is identical in both views. The entry is simply filtered out of the default list.

Is the filter a bug?
No. It is intended behavior that keeps boards focused on from-scratch models. The issue is that most authors do not realize it applies to their derivative, so a real result goes unseen.

How do I see every entry on a board?
Add ?base_model=false to the leaderboard API call, or use the filter toggle in the board's web UI. That returns the full population including fine-tunes, merges, and quantized variants.

Will the 31.1% change over time?
Yes. It is a live snapshot from 2026-10-07. The method stays fixed and each run records that day's state. Across recent snapshots the share has stayed near 30 to 31%.

I quantized a popular model. Is my entry hidden?
Almost certainly, if your card declares the source via base_model, which quantization tooling often adds. Quantized variants are exactly the derivative class this filter removes.

Should I remove base_model to climb the board?
Only if your model truly is a standalone result. Do not strip accurate lineage to deceive. If you keep the field, document the hidden status and link the unfiltered view instead.

Method notes

All numbers come from the public Hugging Face Hub API on 2026-10-07. We counted a board as affected when its base_model=false response contained more entries than its default response. The full population per board is the larger of the two responses. No authentication is required, and the script above is the complete method.

Disclosure: QuantID publishes measurement and diagnostics notes. VIDRAFT is a technology alliance partner.

Further reading on this account: "Who Leads Hugging Face's Official Benchmarks? Concentration, Gaps and Hidden Entries" and "How Is Quantum Computer Performance Actually Measured? Quantum Volume Explained."

When a 0.4-Point Lead Means Nothing: Reading Open-Model Leaderboard Margins Like a Statistician

Ward Ed — Tue, 06 Oct 2026 00:44:19 +0000

TL;DR

A leaderboard rank is a point estimate, not a fact. Every accuracy score carries a confidence interval, and on most public evals that interval is wide enough to swallow the gap between the top few models.
You can estimate the error bar yourself from two numbers you already have: the accuracy and the number of eval items. For a 300-item test at 70 percent accuracy, the 95 percent confidence half-width is about 5 points. A 0.4-point lead inside that band is noise.
For paired comparisons on the same questions, use McNemar's test, not two separate intervals. It is strictly more sensitive and it is what the gap actually deserves.
Before trusting any ranking, ask whether the benchmark can even discriminate. If every top model scores above 95 percent, the test is saturated and the order is close to random.
Popularity is not quality. The most-liked open models on the Hub (cover chart, pulled 2026-10-06) are a social signal, not an accuracy ranking. Treat the two as separate axes.

This is the second post in Leaderboard Forensics. The first covered how the Hub's benchmark plumbing actually works. This one is about the single most common misreading of those numbers: believing a rank order that the sample size cannot support.

Why do the top open models cluster so tightly?

Open-model evaluation has matured to the point where the frontier is crowded. On a typical reasoning or knowledge benchmark, the top ten entries often sit inside a three-point band. The cover chart above shows the eight most-liked open text-generation repositories on the Hugging Face Hub as of 2026-10-06, and likes are a popularity proxy, but the same crowding shows up in accuracy tables.

Here is the uncomfortable part. When scores cluster, the rank order is mostly determined by measurement noise rather than capability. The model in position one and the model in position four may be statistically indistinguishable. The leaderboard still has to print them in some order, so it prints the noise.

The curve above is illustrative. It plots the 95 percent confidence half-width for a model scoring 70 percent, as a function of how many questions the benchmark contains. At 100 items the half-width is about 9 points. At 1,000 items it drops to roughly 2.8 points. Only around 10,000 items does it fall under one point. Most popular public benchmarks live at the left end of that curve.

How do I compute the error bar on a single accuracy score?

Accuracy on a fixed test set is a binomial proportion. If a model answers k of n questions correctly, the estimated accuracy is p = k / n, and the standard error is sqrt(p * (1 - p) / n). A rough 95 percent interval is p plus or minus 1.96 * SE.

The Wald interval above is fine for a quick gut check, but it misbehaves near 0 and 100 percent and for small n. The Wilson interval is better behaved and still closed-form. Here is a compact, dependency-light helper.

import math

def wilson_interval(k, n, z=1.96):
    """95% Wilson score interval for a binomial proportion.
    k = correct answers, n = total eval items."""
    if n == 0:
        return (0.0, 0.0, 0.0)
    p = k / n
    denom = 1 + z * z / n
    center = (p + z * z / (2 * n)) / denom
    half = (z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n))) / denom
    return (p, center - half, center + half)

for n in (100, 300, 1000):
    p, lo, hi = wilson_interval(int(0.70 * n), n)
    print(f"n={n:5d}  acc={p:.3f}  95% CI=[{lo:.3f}, {hi:.3f}]  width={hi-lo:.3f}")

Output:

n=  100  acc=0.700  95% CI=[0.604, 0.782]  width=0.178
n=  300  acc=0.700  95% CI=[0.646, 0.748]  width=0.102
n= 1000  acc=0.700  95% CI=[0.671, 0.727]  width=0.056

Read that bottom line carefully. Even with 1,000 questions, the interval around a 70 percent score spans almost six points. If two models are reported as 70.3 and 69.9 on that benchmark, the difference is well inside the noise floor. The rank order is not evidence of anything.

When is a leaderboard gap actually significant?

The single-score interval is the wrong tool for comparing two models, because it ignores that they were graded on the same questions. Scores on a shared test set are correlated. The correct instrument is McNemar's test, which looks only at the items where the two models disagree.

Build a 2x2 table of agreement. Let b be the number of questions where model A is right and model B is wrong, and c the reverse. The questions both got right or both got wrong carry no information about which is better. McNemar's statistic is (b - c)^2 / (b + c), compared against a chi-square distribution with one degree of freedom.

from scipy.stats import chi2

def mcnemar(b, c):
    """b: A right & B wrong; c: A wrong & B right."""
    n = b + c
    if n == 0:
        return 1.0
    stat = (abs(b - c) - 1) ** 2 / n  # continuity-corrected
    return 1 - chi2.cdf(stat, df=1)

# Two models 0.4 points apart on a 300-item eval:
# say 18 items where A wins, 17 where B wins, rest tied.
print(f"p-value = {mcnemar(18, 17):.3f}")   # p-value = 1.000

A one-item edge out of 300 produces a p-value of essentially 1.0. There is no detectable difference. You would need the disagreement counts to be lopsided, not the headline accuracy to be marginally higher, before the gap earns a rank.

This is why serious evaluation reports publish disagreement matrices, bootstrap confidence intervals, or at least the per-benchmark item counts. If a leaderboard gives you only a sorted column of three-decimal numbers, it is giving you false precision.

How do I tell whether a benchmark can discriminate at all?

Before arguing about margins, check that the test has headroom. A benchmark that is saturated, where the top cluster all score above 95 percent, cannot separate frontier models no matter how many items it has. The remaining questions are either ambiguous, mislabeled, or trivially easy, and the order among the leaders is driven by which model happened to win the coin flips on the hard residue.

Three quick discrimination checks:

Ceiling check. If the top five are all within one point of 100 percent, the benchmark is done. Move on.
Baseline check. Compute what a trivial strategy scores. If majority-class guessing or a tiny n-gram model gets 90 percent, the benchmark measures format compliance, not capability.
Spread check. Look at the standard deviation across all submitted models. A benchmark where everyone scores 68 to 72 has almost no discriminative range left.

A benchmark earns your trust when the baseline is low, the ceiling is far away, and the spread across models is several times the single-score confidence width.

What should I actually do with a leaderboard, then?

Treat the leaderboard as a filter, not a verdict. Use it to shortlist the handful of models in the top band, then evaluate those candidates on your own held-out data that resembles your real traffic. The rank inside that top band is the least reliable information on the page. The membership of the band is the useful part.

And keep popularity and accuracy on separate axes. The cover chart is real Hub data, and it tells you what the community has adopted, which matters for tooling, documentation, and community support. It says nothing about which model is most accurate on your task. Both signals are valuable. Conflating them is how teams end up defending a 0.4-point lead that never existed.

FAQ

Is the Wald interval ever good enough?
For a sanity check on a single score with a few hundred or more items, yes. For scores near 0 or 100 percent, or for small n, switch to Wilson, which stays inside the valid range and does not collapse to zero width at the extremes.

Why McNemar instead of a two-sample proportion test?
Because both models answer the same questions, their scores are paired and correlated. A two-sample test assumes independence and will be too conservative. McNemar conditions on the disagreements, which is exactly the information that distinguishes the two models.

How many eval items do I need to resolve a one-point gap?
Roughly, to get a 95 percent half-width near one point at mid-range accuracy you need on the order of 10,000 independent items. Most public benchmarks have far fewer, which is why one-point gaps are usually unresolvable.

Do bootstrap confidence intervals help?
Yes, especially when scoring is not a clean right-or-wrong binary, such as graded or rubric-scored tasks. Resample the items with replacement a few thousand times, recompute the metric, and read the 2.5 and 97.5 percentiles. It makes no distributional assumption and handles weird metrics gracefully.

Does a larger model automatically win on a saturated benchmark?
No. On a saturated test the residual questions are dominated by noise and label quality, so scale buys you almost nothing. The honest move is to find a harder benchmark with headroom, or build one from your own data.

Can I trust the three decimals a leaderboard prints?
Only if the item count justifies them. Three decimals on a 300-item eval is false precision. The real resolution of that test is closer to one or two points, so mentally round the column before you rank it.

Open 180B Model Leads 10 Official Hugging Face Leaderboards, and the Zero-Token Judge Behind It

AI OpenFree — Tue, 06 Oct 2026 00:42:51 +0000

TL;DR

Darwin-180B-RSI, an open-weight model built by VIDRAFT on top of Alibaba's Qwen3.8-Flash-Next, now holds the top rank on 10 official Hugging Face leaderboards. That is the most of any organization on a stage of 48 curated benchmarks and 95 competing orgs. The runner-up org holds 4. The newest two wins are enterprise-facing rather than exam-facing: ExtractBench (90.29) for pulling fields out of long PDFs, and IFStruct (98.95) for producing output that follows a strict schema. Separately, Gate Arcade, a demo that judges whether an agent action should run, was picked for Hugging Face Spaces of the Week. This post lists every board with numbers, explains the training idea in plain terms, and goes deep on ZTC, a judging method that reads a model's internal state and scores an action in 0.06 seconds with no generated tokens.

Which Hugging Face leaderboards does the model lead?

Ten official boards, spanning science, math, vision, law, enterprise tasks, and document parsing. All numbers below are the registered scores, each measured by the respective submitter using the board's official harness.

Board	Domain	Score	Field count
GPQA Diamond	Science and knowledge	94.44	111 models
MMLU-Pro	Science and knowledge	88.12	141 models
MMMU-Pro	Visual reasoning	79.48
AIME 2026	Math	100
HMMT 2026	Math	100
LEXam	Law	68.94
LEXam-hard	Law	45.72
ExtractBench	Enterprise, PDF extraction	90.29
IFStruct	Enterprise, structured output	98.95
MDPBench	Document parsing	83.65

The two most crowded boards, MMLU-Pro (141 models) and GPQA (111 models), are both held, so this is not a thin-field result. By org, the tally of official #1 boards is FINAL-Bench (the VIDRAFT research org) 10, Z.ai 4, Xiaomi 3, DeepSeek 3, Moonshot 3.

Why do the two newest wins matter more than another exam score?

Most leaderboard wins reward exam-style reasoning. The two newest wins reward the things a team actually hits first when it puts a model into a product.

IFStruct, built by Liquid AI, checks whether a model emits data in the exact requested shape (JSON or YAML). It fails a response if a single field is missing, mistyped, out of range, or invented. The model passed 1,979 of 2,000 items for 98.95, ahead of the prior leader Agents-A1 (93.25), gpt-oss-20b (91.95), and Nemotron-3-Nano-30B (86.80). In a pipeline where a program reads the model output and passes it to the next step, one wrong character stalls the job, so schema adherence is a gating requirement for adoption, not a nicety.

ExtractBench, built by LlamaIndex, measures pulling the right fields out of 370 PDFs ranging from a few pages to roughly 190 pages. The score of 90.29 came from Darwin-180B-RSI-R3, a further self-improved revision, which edged past its own base model, Qwen3.8-Flash-Next (89.88). The model was never specifically trained on legal text or document extraction, yet it leads both, which points to transfer rather than task-specific tuning.

How was the model trained?

The method is model-level recursive self-improvement (RSI). The model solves verifiable problems on its own, keeps only the self-generated solutions that check out as correct, and trains on that filtered set. Each round makes the model a little stronger, and because unverified solutions are discarded, errors are not reinforced into the weights. No human-written answers or worked solutions are needed for the loop. The spillover into law and document extraction, areas never taught directly, is the practical signal that careful reasoning habits learned on checkable problems carry over.

A short, deliberately honest caveat: leaderboard scores reflect the board's own harness and submission, and different boards normalize differently, so treat a cross-board ranking as a portfolio view, not a single global IQ number.

What is ZTC, and how does it judge an action without generating text?

As agents start sending mail, making payments, and running code on a person's behalf, catching a wrong action before it executes has become its own engineering problem. The common approach is to ask a second model, in words, whether an action is safe, and read back its written verdict. That second pass costs tokens and latency, and it ships data to an external API.

ZTC (Zero-Token Confidence) takes a different route. Instead of asking another model to write a judgment, it reads the internal state of the model that produced the action once, and computes the probability that the action is correct. The number of extra generated tokens is zero.

Measured on the Gate Arcade setup (closed-network deployment, snapshot of 2026-09-24):

Metric	External API judge (JEV)	ZTC
Judgment accuracy (AUC)	0.7350	0.7289
Time per judgment	0.591 s	0.0615 s
Judgments in 300 s	135	177

The accuracy gap (0.7350 vs 0.7289) is statistically a wash, while ZTC is about 10x faster per call. Over a 2,000-round head-to-head, ZTC won 73.7 percent of rounds on the combined speed-and-accuracy scoring.

Two properties follow from the design:

Data stays local. ZTC runs on the GPU next to the model, with no round trip to an external API, so it works on air-gapped networks, which suits defense, finance, and public-sector deployments.
The marginal cost is near zero. It reads the internals of a model that already produced the answer. Using this approach, a separate model raised the accuracy of judging its own answers from 0.76 to 0.88.

A minimal mental model of the call path looks like this:

# Conceptual sketch, not the production path.
logits, hidden = model.forward(prompt, return_internals=True)
action = decode(logits)              # the candidate action
confidence = ztc_head(hidden)        # one read, zero new tokens
if confidence < threshold:
    hold(action)                     # defer instead of executing
else:
    execute(action)

The judging space is getting crowded. Typesafe's JEV API set a de facto interface, the public Decision Index board now lists more than 70 models, Cloudflare shipped a JEV-compatible judge that crossed 1,100 Hugging Face likes within days, and research groups at Stanford and NVIDIA have published related models. ZTC's distinguishing bet is reading inside the model rather than asking it to speak.

FAQ

Are these official Hugging Face leaderboards or self-reported numbers?
They are official, curated boards. Each score is measured by the submitter with the board's official harness and then registered, which is the standard practice across these leaderboards.

Is the model open weight?
Yes. The base and the R3 revision are published under the FINAL-Bench org on Hugging Face, so the results are reproducible rather than API-gated.

Does ZTC replace a guardrail or policy model?
It complements them. ZTC is a fast, local confidence signal on whether an action is likely correct, so it is a cheap first gate that can defer uncertain actions before a heavier policy check runs.

How can ZTC be almost as accurate as an external judge while being 10x faster?
Because it skips generation entirely. An external judge has to produce tokens to express a verdict, while ZTC reads the hidden state the forward pass already computed, so it trades a small accuracy margin for a large latency win.

Why did a model win law and document extraction without domain training?
The RSI loop trains on verified reasoning over checkable problems, and those habits transfer. Leading LEXam and ExtractBench without targeted tuning is the observable evidence of that transfer.

What is the single most important takeaway?
The shift from exam scores to enterprise tasks. Leading ExtractBench and IFStruct means the model is measured on reading documents and emitting correct, schema-valid output, which is what production systems need.

허깅페이스 공인 1위 10개와 무토큰 판정 ZTC: 시험 잘 보는 AI에서 일 잘하는 AI로

Vidraft_lab — Tue, 06 Oct 2026 00:42:31 +0000

TL;DR

비드래프트가 허깅페이스 공인 리더보드에서 1위를 추가해 총 10개 부문 1위에 올랐다. 공인 벤치마크 48개, 95개 조직이 겨루는 무대에서 가장 많은 1위이고, 2위 지푸AI(4개)의 두 배가 넘는다.
새로 추가된 두 부문은 '시험'이 아니라 '실무'다. 형식 준수를 재는 IFStruct 98.95%, 문서 정보 추출을 재는 ExtractBench 90.29점으로 둘 다 1위.
AI 에이전트의 행동을 실행 전에 판정하는 ZTC(무토큰 확신도)는 외부 API 판정(JEV)과 같은 정확도(AUC 0.7289 vs 0.7350)에, 판정 1회 0.06초로 약 10배 빠르다.
Darwin-180B-RSI는 알리바바 Qwen3.8-Flash-Next를 '모델 수준 재귀적 자가개선(Model-level RSI)'으로 끌어올린 오픈 모델이다.

10개 부문 1위가 왜 중요한가?

허깅페이스 공인 리더보드는 각 제작사가 공식 평가 도구로 측정한 값을 등록하는 무대다. 비드래프트가 1위를 지키는 10개 부문은 '시험' 영역과 '실무' 영역을 모두 가로지른다.

영역	벤치마크	점수
과학·지식	GPQA Diamond	94.44
과학·지식	MMLU-Pro	88.12
수학	AIME 2026	100
수학	HMMT 2026	100
시각 추론	MMMU-Pro	79.48
법률	LEXam	68.94
법률	LEXam-hard	45.72
기업 실무	ExtractBench	90.29
기업 실무	IFStruct	98.95
문서 파싱	MDPBench	83.65

참가 모델이 가장 많은 MMLU-Pro(141개)와 GPQA(111개)에서도 1위를 지키고 있어, 경쟁이 가장 치열한 무대와 실무 영역을 함께 아우른다. 1위 보유 상위 조직은 비드래프트 10, 지푸AI 4, 샤오미 3, 딥시크 3, 문샷AI 3 순이다.

새로 추가된 1위는 왜 '실무'인가?

지금까지의 1위가 과학·수학·법률 같은 시험 영역이었다면, 이번 두 부문은 기업이 AI를 실제 업무에 투입할 때 가장 먼저 부딪히는 능력이다.

IFStruct 98.95%. 리퀴드AI(Liquid AI)가 만든 IFStruct는 AI가 요청받은 형식(JSON·YAML) 그대로 데이터를 만들어 내는지를 본다. 항목 개수, 필드 이름, 자료형, 허용값, 숫자 범위까지 조건을 하나라도 어기거나, 요청하지 않은 항목을 하나라도 지어내면 불합격이다. 비드래프트 모델은 2,000문항 가운데 1,979문항을 통과해 98.95%를 기록했다. 종전 1위 Agents-A1(93.25%), 오픈AI gpt-oss-20b(91.95%), 엔비디아 Nemotron-3-Nano(86.80%)를 큰 차이로 앞섰다.

AI의 출력을 프로그램이 곧바로 읽어 다음 단계로 넘기는 파이프라인에서는 형식이 한 글자만 틀려도 업무가 멈춘다. 아래처럼 스키마 검증을 바로 통과해야 쓸 수 있다.

import json, jsonschema

schema = {
    "type": "object",
    "properties": {"name": {"type": "string"}, "age": {"type": "integer"}},
    "required": ["name", "age"],
    "additionalProperties": False,  # 요청 안 한 항목을 지어내면 불합격
}

out = model.generate(prompt)          # 모델 출력
jsonschema.validate(json.loads(out), schema)  # 한 글자만 틀려도 예외

ExtractBench 90.29점. 라마인덱스(LlamaIndex)가 만든 ExtractBench는 수 쪽에서 190여 쪽에 이르는 PDF 370건에서 필요한 정보를 정확히 뽑아내는 능력을 잰다. 자가개선을 한 번 더 거친 Darwin-180B-RSI-R3가 90.29점으로 1위에 올랐다. 종전 1위는 이 모델의 기반인 알리바바 Qwen3.8-Flash-Next(89.88점)였다.

스스로 배우는 AI, RSI는 어떻게 동작하나?

Darwin-180B-RSI는 알리바바의 오픈 모델 Qwen3.8-Flash-Next를 기반으로, 비드래프트의 '모델 수준 재귀적 자가개선(Model-level RSI)' 기술로 성능을 끌어올렸다. 핵심은 세 단계다.

모델이 검증 가능한 문제를 스스로 푼다.
정답으로 확인된 자기 풀이만 골라낸다.
그 풀이만으로 다시 학습한다.

사람이 쓴 정답이나 풀이가 필요 없고, 검증되지 않은 풀이는 배우지 않아 오류가 강화되지 않는다. 법률이나 문서 추출을 따로 가르치지 않았는데도 이 영역에서 1위에 오른 것은, 검증 가능한 문제를 풀며 익힌 꼼꼼한 추론 습관이 다른 영역으로 옮겨 간 결과로 회사는 보고 있다.

ZTC는 어떻게 한 글자도 만들지 않고 판정하나?

AI 에이전트가 사람 대신 메일을 보내고 결제를 하고 코드를 실행하는 시대가 되면서, 틀린 행동을 실행 전에 걸러내는 '판정(decision)' 기술이 새 격전지가 됐다. 통상 방식은 다른 AI에게 "이 답이 맞느냐"를 물어 글로 된 답을 받는다. ZTC(Zero-Token Confidence, 무토큰 확신도)는 다르다. 답을 만든 모델의 내부 상태를 한 번 읽어 정답일 확률을 계산한다. 추가로 생성하는 글자가 0개다.

같은 정확도, 10배 빠른 속도. 판정 정확도(AUC)는 ZTC 0.7289, JEV 0.7350으로 통계적으로 같은 수준이다. 판정 1회 시간은 ZTC 0.06초, JEV 0.59초다. 같은 300초 동안 ZTC는 177건, JEV는 135건을 처리했고, 2,000판 대결에서 ZTC가 73.7%를 이겼다.
데이터가 밖으로 나가지 않는다. ZTC는 모델 옆 GPU에서 바로 돌아 외부 API를 왕복하지 않는다. 인터넷이 차단된 폐쇄망에서도 작동해 국방·금융·공공 현장에 적합하다.
추가 비용이 거의 없다. 이미 답을 만든 모델의 내부를 읽기만 한다. Darwin-397B-ZTC는 이 방식으로 자기 답의 정답 여부 판정 정확도를 0.76에서 0.88로 끌어올렸다.

지표	플레인	JEV(외부 API)	ZTC(비드래프트)
판정 정확도(AUC)	없음	0.7350	0.7289
판정 1회 시간	0초	0.591초	0.0615초
300초 처리 건수	해당 없음	135건	177건

같은 시기, AI 에이전트의 행동을 판정하는 비드래프트의 체험 앱 Gate Arcade는 허깅페이스 '이주의 스페이스(Spaces of the Week)'에 선정됐다. 약 150만 개 앱 가운데 매주 8개를 고르는 큐레이션으로, 비드래프트는 한 달 새 두 번째로 이름을 올렸다.

FAQ

Q1. 리더보드 점수는 누가 측정하나?
각 제작사가 공식 평가 도구로 측정해 등록한 값이다. 허깅페이스 공인 벤치마크는 측정 방법과 데이터가 공개돼 재현할 수 있다.

Q2. IFStruct에서 '불합격' 기준은 무엇인가?
항목 개수, 필드 이름, 자료형, 허용값, 숫자 범위 중 하나라도 어기거나, 요청하지 않은 항목을 하나라도 지어내면 불합격이다. 2,000문항 중 1,979문항 통과가 98.95%다.

Q3. ZTC와 JEV의 정확도 차이는 유의미한가?
AUC 0.7289 대 0.7350은 통계적으로 같은 수준이다. 대신 판정 1회 시간이 0.06초 대 0.59초로 약 10배 빠르다.

Q4. ZTC는 폐쇄망에서도 쓸 수 있나?
그렇다. 모델 옆 GPU에서 바로 돌아 외부 API 왕복이 없으므로 인터넷이 차단된 환경에서도 작동한다.

Q5. RSI는 학습을 거듭하면 오류가 쌓이지 않나?
정답으로 검증된 자기 풀이만 다시 학습하므로, 검증되지 않은 풀이는 반영되지 않아 오류가 강화되지 않는다.

Q6. 모델은 어디서 볼 수 있나?
huggingface.co/FINAL-Bench/Darwin-180B-RSI 와 huggingface.co/FINAL-Bench/Darwin-180B-RSI-R3 에 공개돼 있다.

허깅페이스 리더보드 1위 9개: 오픈소스 LLM Darwin-180B-RSI와 재귀적 자가개선(RSI) 개발자 가이드

Vidraft_lab — Mon, 05 Oct 2026 08:58:20 +0000

TL;DR

비드래프트(VIDRAFT)의 오픈소스 LLM Darwin-180B-RSI 계열이 허깅페이스 공인(official) 벤치마크 48개 중 9개에서 1위입니다. 95개 조직 가운데 가장 많고, 다음은 문샷AI·지푸AI(각 4개)입니다.
새로 추가된 1위는 실무형 두 개입니다. IFStruct 98.95%(구조화 출력, 2,000문항 중 1,979 통과)와 ExtractBench 90.29(PDF 370건 정보 추출, R3 버전)입니다.
비결은 모델 수준 재귀적 자가개선(Model-level RSI): 사람이 쓴 정답 없이 검증 가능한 문제를 스스로 풀고, 정답으로 확인된 자기 풀이만 골라 다시 학습합니다.
같은 팀의 ZTC는 답을 한 글자도 생성하지 않고 판정합니다. 외부 API 판정(JEV)과 정확도는 통계적으로 같고(AUC 0.7289 vs 0.7350), 판정 1회는 0.0615초 vs 0.591초로 약 10배 빠릅니다.
아래에 리더보드를 직접 조회하는 Python 코드, vLLM·Transformers 실행 코드, FAQ를 정리했습니다. 모든 리더보드 수치는 2026년 10월 5일 허깅페이스 API로 다시 확인했습니다.

허깅페이스 공인 리더보드 1위 9개는 정확히 무엇인가요?

허깅페이스는 일부 데이터셋을 "공인 벤치마크(benchmark:official)"로 지정하고, 각 데이터셋 페이지에 리더보드를 붙여 둡니다. 모델 제작자는 자기 모델 저장소에 .eval_results/*.yaml 파일로 점수를 등록하고, 허깅페이스가 이를 모아 순위를 매깁니다. 현재 공인 벤치마크는 48개, 참가 조직은 95개입니다.

2026년 10월 5일 API 기준 Darwin-180B-RSI 계열이 1위인 9개 보드와 바로 아래 모델은 다음과 같습니다.

영역	벤치마크 (데이터셋 ID)	Darwin 점수	2위 모델	2위 점수
과학	GPQA Diamond (`Idavidrein/gpqa`)	94.44	moonshotai/Kimi-K3	93.5
지식	MMLU-Pro (`TIGER-Lab/MMLU-Pro`)	88.12	MiniMaxAI/MiniMax-M2.1	88.0
수학	AIME 2026 (`MathArena/aime_2026`)	100	thinkingmachines/Inkling	97.1
수학	HMMT Feb 2026 (`MathArena/hmmt_feb_2026`)	100	moonshotai/Kimi-K2.6	92.7
시각 추론	MMMU-Pro (`MMMU/MMMU_Pro`)	79.48	moonshotai/Kimi-K2.6	79.4
법률	LEXam (`LEXam-Benchmark/LEXam`)	68.94	deepseek-ai/DeepSeek-R1	52.41
법률	LEXam-hard (`joelniklaus/LEXam-hard`)	45.72	thinkingmachines/Inkling	40.82
기업 실무	ExtractBench (`llamaindex/ExtractBench`)	90.29 (R3)	Qwen/Qwen3.8-Flash-Next	89.88
기업 실무	IFStruct (`LiquidAI/ifstruct-v1.0`)	98.95	InternScience/Agents-A1	93.25

참가 모델이 가장 많은 MMLU-Pro(약 141개)와 GPQA(약 111개)에서도 1위입니다. 경쟁이 가장 붐비는 보드와 실무형 보드를 함께 잡았다는 점이 개발자 입장에서 의미 있는 부분입니다.

정직하게 덧붙이면, 이 점수들은 모두 제작사가 공식 평가 도구로 측정해 등록한 자체 측정값입니다. 다수결 점수(예: AIME maj@16, GPQA 최대 16샘플 다수결)는 모델 카드에 그렇게 표기되어 있습니다. 숫자를 비교할 때 설정까지 같이 보시길 권합니다.

리더보드 순위를 코드로 직접 확인하려면 어떻게 하나요?

블로그 글의 숫자를 믿을 필요 없이, 허깅페이스 공개 API로 바로 확인할 수 있습니다. 토큰도 필요 없습니다.

import requests
from collections import Counter

API = "https://huggingface.co/api"

# 1) 공인 벤치마크 목록
boards = requests.get(f"{API}/datasets",
                      params={"filter": "benchmark:official", "limit": 200}).json()
print("공인 벤치마크 수:", len(boards))

# 2) 보드별 1위 집계
firsts = Counter()
for b in boards:
    rows = requests.get(f"{API}/datasets/{b['id']}/leaderboard").json()
    if not isinstance(rows, list) or not rows:
        continue
    top = rows[0]                      # rank 1
    firsts[top["modelId"].split("/")[0]] += 1
    if top["modelId"].startswith("FINAL-Bench/"):
        r2 = rows[1] if len(rows) > 1 else {}
        print(b["id"], top["value"], top["modelId"],
              "| 2위", r2.get("modelId"), r2.get("value"))

print(firsts.most_common(5))
# 2026-10-05 실행 결과: FINAL-Bench 9, moonshotai 4, zai-org 4, XiaomiMiMo 3, deepseek-ai 3

각 행에는 rank, value, modelId, source(모델 카드 링크), verified 같은 필드가 들어 있습니다. 사내 모델 선정 대시보드를 만들거나, 특정 벤치마크의 1위 변동을 주기적으로 감시하는 데 그대로 쓸 수 있습니다.

IFStruct 98.95%는 개발자에게 왜 중요한가요?

IFStruct는 미국 리퀴드AI(Liquid AI)가 만든 구조화 출력 벤치마크입니다. 실제 사용자가 요청하는 다양한 표현으로 JSON 또는 YAML 스키마를 지정하고, 모델이 그대로 지키는지 문항마다 합격/불합격으로 채점합니다. 기준이 꽤 엄격합니다.

항목 개수(또는 범위), 감싸는 키, JSON과 YAML 구분
코드 펜스를 요구하거나 금지하는 조건, "설명 없이"라는 요청
모든 필드의 자료형, 허용값(enum), 숫자 범위, 중첩 리스트 길이
스키마가 요구하지 않은 필드를 하나라도 지어내면 불합격

제약 디코딩(constrained decoding)을 쓰지 않으므로 모델 자체의 규율만 측정됩니다. Darwin-180B-RSI는 공식 하네스 기본값(temperature 0, 최대 16,000토큰, 생각 켬)으로 2,000문항 중 1,979문항을 통과했습니다. 한 문항은 하네스의 120초 읽기 시간 제한에 걸려 불합격 처리되었으니, 이 숫자는 보수적인 값입니다. 같은 모델에서 생각 모드를 끄면 89.7%로 내려갑니다. 마지막 몇 개의 제약 위반을 잡아내는 것이 바로 추론 단계라는 뜻입니다.

에이전트 파이프라인이나 백오피스 자동화에서 모델의 출력은 사람이 아니라 코드가 읽습니다. 자료형 하나, 지어낸 키 하나에 json.loads나 스키마 검증이 실패하고 파이프라인이 멈춥니다. 재시도 루프와 수리(repair) 로직을 줄일 수 있다는 점에서, 98.95%와 93.25%의 차이는 운영 비용의 차이입니다.

ExtractBench 90.29는 어떻게 나왔나요?

ExtractBench는 라마인덱스(LlamaIndex)가 만든 문서 정보 추출 벤치마크입니다. 몇 쪽짜리 서식부터 190여 쪽에 이르는 파일까지 PDF 370건에서 구조화된 필드를 뽑아야 합니다. 자가개선을 한 번 더 거친 Darwin-180B-RSI-R3가 90.29점으로 1위에 올랐고, 종전 1위는 이 모델의 기반인 알리바바 Qwen3.8-Flash-Next(89.88), 그다음은 Qwen3.8-27B(89.75)입니다. 이 실행은 공식 하네스로 생각 모드를 끈 상태였고, 제출 노트에 그렇게 명시했습니다.

기반 모델을 자기 계보의 다음 세대가 넘었다는 점이 RSI의 효과를 가장 직접적으로 보여 줍니다.

재귀적 자가개선(Model-level RSI)은 어떻게 작동하나요?

Darwin-180B-RSI는 알리바바의 오픈 모델 Qwen3.8-Flash-Next에서 출발합니다. 그 위에 비드래프트의 모델 수준 재귀적 자가개선(Model-level RSI)을 적용했습니다. 원리는 세 줄로 요약됩니다.

모델이 검증 가능한 문제를 스스로 풉니다.
풀이 중 정답으로 확인된 자기 풀이만 남깁니다.
그 풀이로 다시 학습하고, 이 과정을 반복합니다.

사람이 쓴 정답 풀이나 추론 흔적은 쓰지 않습니다. 검증되지 않은 풀이는 학습 데이터에 들어가지 않기 때문에, 모델이 자기 오류를 강화하는 일이 구조적으로 막힙니다. R3는 이 반복을 한 바퀴 더 돈 결과입니다.

흥미로운 점은 법률이나 문서 추출을 따로 가르치지 않았는데도 LEXam, LEXam-hard, ExtractBench에서 1위가 나왔다는 것입니다. 회사는 검증 가능한 문제를 풀며 익힌 꼼꼼한 단계별 추론 습관이 다른 영역으로 옮겨 간 결과로 보고 있습니다.

"Model-level"이라는 이름은 작업틀(하네스) 쪽 프롬프트나 도구를 고치는 방식과 구분하기 위한 것입니다. 바뀌는 것은 매뉴얼이 아니라 모델의 가중치, 즉 두뇌 자체입니다.

Darwin-180B-RSI를 직접 실행하려면 어떻게 하나요?

아래 코드는 모델 카드의 Quickstart를 그대로 옮긴 것입니다. 사양을 먼저 확인하세요.

구조: Mixture-of-Experts, 하이브리드 어텐션(선형 어텐션 36층 + 완전 어텐션 12층)
전문가 512개 중 토큰당 10개 활성 + 공유 전문가
컨텍스트 262,144토큰, 이미지+텍스트 입력, 텍스트 출력
bf16 기준 약 336GB

vLLM으로 서빙 (B200 8장 또는 동급)

vllm serve FINAL-Bench/Darwin-180B-RSI \
  --tensor-parallel-size 8 --enable-expert-parallel \
  --max-model-len 135168 --trust-remote-code

OpenAI 호환 API로 호출

from openai import OpenAI
c = OpenAI(base_url="http://localhost:8000/v1", api_key="-")
r = c.chat.completions.create(model="FINAL-Bench/Darwin-180B-RSI",
    messages=[{"role": "user", "content": "Explain why the sky is blue in two sentences."}],
    temperature=1.0, top_p=0.95, extra_body={"top_k": 20})
print(r.choices[0].message.content)

Transformers로 로드

from transformers import AutoProcessor, AutoModelForImageTextToText
model_id = "FINAL-Bench/Darwin-180B-RSI"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(model_id, torch_dtype="auto", device_map="auto")

실전 팁 두 가지입니다. 첫째, 이것은 생각(thinking) 모델입니다. 어려운 추론에는 32K에서 131K토큰의 생각 예산을 권장하며, 짧게 자르면 정확도가 떨어집니다. 둘째, 모델 카드의 평가 설정은 temperature 1.0, top_p 0.95, top_k 20입니다. 다만 IFStruct처럼 형식이 중요한 작업은 공식 하네스가 temperature 0을 썼다는 점도 참고하세요.

ZTC는 무엇이고 JEV와 무엇이 다른가요?

AI 에이전트가 메일을 보내고, 결제를 하고, 코드를 실행하는 시대에는 틀린 행동을 실행 전에 거르는 판정(decision) 기술이 중요해졌습니다. 타입세이프(Typesafe)의 판정 API JEV가 이 분야의 표준 규격을 만들었고, 이를 평가하는 Decision Index에는 70개가 넘는 모델이 올라 있습니다. 9월 30일에는 클라우드플레어가 JEV 호환 판정 모델 Clef를 공개했습니다.

비드래프트의 ZTC(Zero-Token Confidence, 무토큰 확신도)는 접근이 다릅니다. 다른 AI에게 "이 답이 맞나요?"를 물어 글로 된 답을 받는 대신, 답을 만든 모델의 내부 상태를 한 번 읽어 정답일 확률을 계산합니다. 추가로 생성하는 토큰이 0개입니다.

이를 직접 보여 주는 앱이 Gate Arcade이고, 9월 28일 주간 허깅페이스 "이주의 스페이스(Spaces of the Week)"에 선정되었습니다. 에이전트가 행동을 제안하면 관문이 실행 또는 보류를 정합니다. 맞는 행동 실행 +1, 틀린 행동 실행 -1, 보류 0점입니다. 2026년 9월 24일 기준, 폐쇄망 배치 실측값은 다음과 같습니다.

	판정 없음	JEV (외부 API)	ZTC (비드래프트)
판정 정확도 (AUC)	없음	0.7350	0.7289
판정 1회 시간	0초	0.591초	0.0615초
300초 처리 건수	해당 없음	135건	177건

정확도는 통계적으로 같은 수준이고 속도는 약 10배입니다. 2,000판 대결에서 ZTC가 73.7%를 이겼습니다. 판정을 위해 데이터를 밖으로 보낼 필요가 없어 인터넷이 막힌 폐쇄망에서도 돌아갑니다. 같은 방식으로 Darwin-397B-ZTC는 자기 답의 정답 여부 판정 정확도를 0.76에서 0.88로 끌어올렸습니다.

한계도 분명합니다. 모델 내부를 읽어야 하므로 API로만 쓰는 폐쇄형 모델에는 적용할 수 없고, 모델마다 판독기를 맞춰야 합니다. 그래서 최종 판정자라기보다 빠르고 싼 1차 관문으로 쓰는 것이 맞습니다. 참고로 Darwin-180B-RSI 저장소에 함께 들어 있는 ZTC 판독기는 모델 카드에 "초기 공개(early release)"로 표기되어 있으니, 높은 위험의 행동에는 카드 설명을 확인하고 쓰세요.

자주 묻는 질문 (FAQ)

Q1. 허깅페이스 리더보드 1위 9개는 어디서 확인하나요?
각 데이터셋 페이지(예: huggingface.co/datasets/LiquidAI/ifstruct-v1.0)의 리더보드 탭이나, 위의 Python 코드처럼 https://huggingface.co/api/datasets/{id}/leaderboard를 호출하면 됩니다. 48개 보드를 한 화면에 모은 스페이스도 있습니다.

Q2. 점수는 누가 측정했나요? 검증된 값인가요?
허깅페이스 공인 리더보드는 제작사가 공식 평가 도구로 측정해 .eval_results로 등록하는 구조입니다. Darwin의 점수도 자체 측정값이며, 샘플 수와 생각 예산 같은 설정을 모델 카드에 모두 공개했습니다.

Q3. 재귀적 자가개선(RSI)에 사람이 만든 정답이 정말 전혀 안 들어가나요?
네. 모델이 검증 가능한 문제를 풀고, 정답으로 확인된 자기 풀이만 남겨 다시 학습합니다. 사람이 쓴 풀이나 추론 흔적은 쓰지 않습니다.

Q4. 오픈소스 LLM으로 바로 써도 되나요? 하드웨어는 얼마나 필요한가요?
가중치는 허깅페이스에 공개되어 있습니다. bf16 기준 약 336GB이므로 모델 카드는 B200 8장(또는 동급)에서 vLLM 텐서 병렬 8, 전문가 병렬을 권장합니다.

Q5. 구조화 출력(JSON) 작업에 왜 이 모델인가요?
제약 디코딩 없이 IFStruct 98.95%를 기록했습니다. 2위(93.25%)보다 5.7%p 높아, 출력 수리와 재시도 로직을 줄이는 데 유리합니다.

Q6. ZTC를 우리 서비스에 붙이려면?
모델 내부 상태를 읽을 수 있는 오픈 웨이트 모델이어야 합니다. Gate Arcade에서 동작 방식을 먼저 체험해 보고, 모델 카드의 ztc/ 폴더 예제를 참고하세요.

링크

비드래프트: https://vidraft.net
모델: https://huggingface.co/FINAL-Bench/Darwin-180B-RSI
R3 (ExtractBench 1위): https://huggingface.co/FINAL-Bench/Darwin-180B-RSI-R3
Gate Arcade (이주의 스페이스): https://huggingface.co/spaces/FINAL-Bench/gate-tetris

시험을 잘 푸는 모델에서, 정해진 형식대로 일하는 모델로. 그리고 그 모델이 행동하기 전에 확신 없는 순간을 0.06초 만에 잡아내는 관문까지. 직접 돌려 보시고 결과를 댓글로 알려 주세요.

Who Leads Hugging Face's Official Benchmarks? Concentration, Gaps and Hidden Entries (Measured 2026-10-06)

QuantID — Mon, 05 Oct 2026 08:57:14 +0000

TL;DR

On 2026-10-06 we read every dataset tagged benchmark:official from the public Hugging Face Hub API, and every leaderboard attached to those datasets, twice: once in the default view and once with base_model=false. Every number below is a reading taken on that date, and about 40 lines of Python reproduce it.

48 official benchmark datasets, 44 with at least one leaderboard entry.
Default view: 1,024 entries from 95 organizations across 411 distinct models.
The top 5 organizations hold 40.3% of visible entries, the top 10 hold 59.0%.
Full population (with base_model=false): 1,479 entries, 161 namespaces. Gini 0.709, HHI 0.0369.
Median relative gap between #1 and #2: 3.69%. On 9 of 43 comparable boards the gap is under 1%.
455 entries (30.8%) never appear in the default view because the board hides models that declare a base_model. 34 of 44 boards are affected.

Many organizations take part, but entry counts are uneven. On about one board in five the top two are close enough that the order could flip on a rerun, and nearly a third of all submissions only show up if you change a query parameter.

Why measure benchmark leaderboards instead of ranking them?

QuantID works on measurement and diagnostics, so we treat a leaderboard as an instrument rather than a scoreboard. A leaderboard is a fixed test, a scoring rule and a published ordering. Like any instrument it has a resolution, a bias and a display layer that can hide part of the signal. Most writing about leaderboards asks who is first. We ask four questions about the instrument itself:

How many independent parties actually participate?
How unequal is that participation?
How far apart are the leaders, compared with plausible run-to-run noise?
How much of the data is filtered away before a human sees it?

Hugging Face lets model owners attach evaluation results to a model repository through .eval_results/*.yaml files. Datasets tagged as official benchmarks collect those files into a board. The whole system answers to a public API, so it can be audited like any other measurement pipeline.

What data did we read?

Three public calls, no authentication:

GET https://huggingface.co/api/datasets?filter=benchmark:official&limit=200
GET https://huggingface.co/api/datasets/{id}/leaderboard
GET https://huggingface.co/api/datasets/{id}/leaderboard?base_model=false

Each row carries rank, value, modelId, lower_is_better, a source link and author metadata. We define the organization of an entry as the namespace before the slash in modelId (so Qwen/Qwen3.8-Flash-Next belongs to Qwen). User namespaces count as organizations. We do not merge related accounts: the API does not say which accounts belong together, and guessing would inject our own bias into the measurement.

For each board, the larger of the two responses is the full population (in practice the base_model=false response). The default response is the visible population.

Quantity	Default view	Full population
Benchmark datasets	48	48
Boards with entries	44	44
Entries	1,024	1,479
Distinct models	411	647
Distinct organizations	95	161

New YAML files are merged daily, so the boards keep changing. Treat every number here as a snapshot, not a constant.

How concentrated is AI benchmark participation?

"Concentration" covers two separate questions: who submits, and who wins.

In the default view the five largest submitters are Qwen (136), zai-org (84), deepseek-ai (65), moonshotai (65) and nvidia (63). Together those five hold 40.3% of the 1,024 visible entries, and the top ten hold 59.0%. Entry volume mostly tracks how many model sizes a lab ships; it says little about quality.

Across the full population the inequality is easier to summarize with two numbers that seem to disagree:

Metric	Value
Gini (entries by org)	0.709
HHI	0.0369
Effective number of orgs (1/HHI)	about 27
Orgs appearing on exactly one board	86 of 161

In antitrust terms an HHI of 0.0369 is unconcentrated (below 0.15) because no single organization holds a large slice; the largest, Qwen, has 190 of 1,479 entries (12.8%). A Gini of 0.709 says the distribution is still very unequal. The long tail explains both readings: more than half of all organizations appear exactly once, while a few large labs submit many model sizes to many boards.

Reporting both indicators keeps "no monopoly" from being read as "equal participation". They are answers to different questions.

Who actually wins?

We find the #1 entry of each board by sorting on value: descending by default, ascending when lower_is_better is true. Across 44 boards, 24 distinct organizations hold at least one #1 position. The most frequent holders in this snapshot were FINAL-Bench (9 boards), zai-org (4), deepseek-ai (3) and moonshotai (3), then XiaomiMiMo, ornith-ai, tsinghua-sigs-robot-lab and tencent with 2 each. Leadership is spread across roughly two dozen teams, and the biggest submitters are mostly not the most frequent winners: Qwen has the most entries overall but holds the top spot on only one board in this reading.

How close are the top two models on a typical benchmark?

Concentration tells you who shows up. Competitiveness tells you whether the order at the top means anything. For each board with at least two entries we compute the relative gap between #1 and #2:

$$g = \frac{|v_1 - v_2|}{|v_1|} \times 100\%$$

A relative measure lets us compare boards scored in percent, in error rate or in arbitrary units. Measured over 43 comparable boards, the median gap is 3.69%, and the distribution is wide:

Why does a gap under 1% matter? Most of these benchmarks have a few hundred to a few thousand items. For a binomial accuracy $p$ on $n$ items the standard error is $\sqrt{p(1-p)/n}$. At $p = 0.8$ and $n = 1{,}000$ that is about 1.3 percentage points, or roughly 1.6% in relative terms. Self-reported runs also differ in sampling temperature, prompt template and harness. With all that, a lead under 1% usually cannot be told apart from a tie, so on about one board in five the current #1 could change on a rerun. At the other end, a handful of boards show leads above 20%; that usually means a genuinely new capability, a board with very few entries, or entries scored under different conditions. A large gap is a reason to open the source link, not a conclusion by itself. Our rule for any ranking: report the gap with an uncertainty estimate (2 SE or a bootstrap interval) and call it a tie when the interval contains zero.

What share of Hugging Face leaderboard entries are hidden by default?

The default /leaderboard response, and the default web view, leaves out entries from models whose card declares a base_model. That removes fine-tunes, merges, quantized variants and other derivative releases. Passing base_model=false brings them back.

$$\text{hidden share} = 1 - \frac{\sum_b |L_b^{default}|}{\sum_b |L_b^{full}|}$$

Measured: 455 of 1,479 entries (30.8%) are hidden in the default view, and 34 of 44 boards hide at least one entry.

The filter is a design choice, not a bug: it keeps the default view focused on original models. It still has measurable consequences. A fine-tuned model can outscore its base and stay invisible to anyone who does not change the query. Rankings quoted from the default page and rankings computed with the full population can disagree. Concentration statistics computed only on the default view understate the long tail, because small teams publish a large share of the derivative work. If you report a position on one of these boards, say which view it comes from.

How can I reproduce this measurement?

The script below needs only the standard library plus requests, and runs in under a minute.

import requests, collections, statistics as st
import concurrent.futures as cf

H = "https://huggingface.co/api"
ids = [d["id"] for d in requests.get(
    f"{H}/datasets", params={"filter": "benchmark:official", "limit": 200}, timeout=60).json()]

def fetch(i):
    def get(p):
        try:
            r = requests.get(f"{H}/datasets/{i}/leaderboard", params=p, timeout=60).json()
            return r if isinstance(r, list) else []
        except Exception:
            return []
    return i, get({}), get({"base_model": "false"})

rows = list(cf.ThreadPoolExecutor(8).map(fetch, ids))

def org(e): return e["modelId"].split("/")[0]
def gini(x):
    x = sorted(x); n = len(x); s = sum(x)
    return sum((2*i - n - 1) * v for i, v in enumerate(x, 1)) / (n * s)

entries, firsts = collections.Counter(), collections.Counter()
gaps, n_default, n_full = [], 0, 0
for bid, default, full in rows:
    full = full if len(full) >= len(default) else default
    n_default += len(default); n_full += len(full)
    for e in full:
        entries[org(e)] += 1
    if not full:
        continue
    lib = full[0].get("lower_is_better", False)
    s = sorted(full, key=lambda e: e["value"], reverse=not lib)
    firsts[org(s[0])] += 1
    if len(s) > 1 and s[0]["value"]:
        gaps.append(abs(s[0]["value"] - s[1]["value"]) / abs(s[0]["value"]) * 100)

v = sorted(entries.values(), reverse=True); N = sum(v)
print("orgs", len(entries), "entries", N, "gini", round(gini(v), 3))
print("top5", round(100*sum(v[:5])/N, 1), "top10", round(100*sum(v[:10])/N, 1))
print("median gap %", round(st.median(gaps), 2), "gap<1%", sum(g < 1 for g in gaps))
print("hidden %", round(100*(1 - n_default/n_full), 1))
print("top #1 holders", firsts.most_common(8))

Run it on another day and the numbers will differ. That is expected: the method stays fixed, and each run records the state of the boards on its own date.

What are the limits of this measurement?

An organization here is a namespace, not a legal entity. One company may publish under several namespaces, and one namespace may host several teams. We did not merge or split any.
Scores are mostly self-reported. Most rows carry verified: false and cite a model card. We measure the published board, not the truth behind each number.
Ties and sort direction. We trust the lower_is_better flag and treat multi-metric datasets exactly as the API returns them.
Snapshot effect. Entries arrive as pull requests to model repositories, so one large release can move the top-k share by several points in a day.
The gap metric ignores noise. A relative gap has no units and no per-board significance test; the binomial SE above is a rough guide.

Disclosure

QuantID is a technology alliance partner of VIDRAFT, which appears on several boards (including under the FINAL-Bench namespace); data shown here is computed identically for every organization.

FAQ

How many official benchmark leaderboards are on Hugging Face?
On 2026-10-06 the API returned 48 datasets tagged benchmark:official. Of those, 44 had at least one leaderboard entry.

Is AI benchmark leadership dominated by a few labs?
Participation is unequal: the full-population Gini is 0.709 and the top 10 organizations hold 59.0% of visible entries. Still, #1 positions are spread across 24 organizations, and no organization holds more than about 13% of all entries.

Why don't I see some models on a Hugging Face leaderboard?
The default view hides entries from models that declare a base_model in their card. Add ?base_model=false to the API call to see them. In this snapshot that filter hid 30.8% of all entries across 34 of 44 boards.

How close are the top two models on a typical board?
The median relative gap between #1 and #2 was 3.69%. On 9 of 43 boards it was under 1%, which is usually within run-to-run noise.

Why report both Gini and HHI?
HHI is driven by the biggest shares, so it stays low when no single organization is large. Gini measures inequality across the whole distribution, including the long tail of single-entry participants. Reporting both prevents "no monopoly" from being read as "equal participation".

Can I get these numbers live?
Yes. The script above reproduces them, and we maintain a live map of all official boards as a public Space.

How Hugging Face Official Benchmark Leaderboards Actually Work: .eval_results YAML, the base_model Filter, and the 30% the Default View Hides

Ward Ed — Mon, 05 Oct 2026 08:56:56 +0000

TL;DR: Hugging Face currently tags 48 datasets as official benchmarks (benchmark:official). Their leaderboards are not uploaded by the benchmark owners: they are assembled automatically from small .eval_results/*.yaml files that model authors commit to their own model repos (or propose through pull requests). On October 5, 2026 I queried every board twice through the public API. The default view returned 1,023 entries. With ?base_model=false, which keeps derivative models, it returned 1,469. So 446 entries (30.4%) never show up in the default view, and the deciding field is the base_model key in each model card. Another finding: 772 of the 1,023 visible entries (75.5%) come from open pull requests, and none were flagged verified.

What are Hugging Face official benchmark leaderboards?

An official benchmark is just a dataset repo that carries the benchmark:official tag. The Hub renders a leaderboard widget on its dataset page. Each row on that widget is one model's score on that benchmark. A dataset owner does not have to run any evaluation service for this. The Hub collects scores that model repos self-report and matches them to the dataset.

You can pull the list of boards with one request:

curl -s "https://huggingface.co/api/datasets?filter=benchmark:official&limit=200" \
  | python -c "import sys,json; d=json.load(sys.stdin); print(len(d)); print('\n'.join(x['id'] for x in d))"

When I ran it on October 5, 2026, it returned 48 datasets. Some of the better known ones: TIGER-Lab/MMLU-Pro, Idavidrein/gpqa, cais/hle, SWE-bench/SWE-bench_Verified, hf-audio/open-asr-leaderboard, openai/gsm8k, MathArena/aime_2026, MMMU/MMMU_Pro and harborframework/terminal-bench-2.0. The list also includes newer, narrower boards such as llamaindex/ParseBench, llamaindex/ExtractBench, LEXam-Benchmark/LEXam, FutureMa/EvasionBench and crosbylegal/RedlineBench.

Four of the 48 (tiiuae/PBench, mercor/ACE, ChrisHayduk/nanofold-public, harborframework/terminal-bench-science) had zero entries in both views when I checked. Tagged boards can exist before anyone has reported a score to them.

How are Hugging Face leaderboard entries built from .eval_results YAML?

Every row traces back to one YAML file in a model repo, under a directory named .eval_results/. The leaderboard API includes the path in each row, so you can see it yourself:

curl -s "https://huggingface.co/api/datasets/Idavidrein/gpqa/leaderboard" \
  | python -c "import sys,json; e=json.load(sys.stdin)[1]; print({k:e[k] for k in ('rank','modelId','value','filename','verified','pullRequest')})"

Each row has these fields: rank, filename, value, verified, source, pullRequest, modelId, author, lower_is_better and num_parameters. filename is the YAML path inside the model repo, and pullRequest holds a PR number when the file lives on a PR ref instead of main.

Here is a real file. moonshotai/Kimi-K3 keeps five of them on main:

curl -s "https://huggingface.co/api/models/moonshotai/Kimi-K3/tree/main/.eval_results" \
  | python -c "import sys,json; [print(f['path'], f['size']) for f in json.load(sys.stdin)]"

.eval_results/apex-agents.yaml 157
.eval_results/deep-swe.yaml 156
.eval_results/gpqa.yaml 152
.eval_results/hle.yaml 139
.eval_results/moonshotai__Kimi-K3.yaml 211

The GPQA one (/raw/main/.eval_results/gpqa.yaml) is just 152 bytes:

- dataset:
    id: Idavidrein/gpqa
    task_id: diamond
  value: 93.5
  source:
    url: https://huggingface.co/moonshotai/Kimi-K3
    name: Model Card

That is the whole contract:

dataset.id is the benchmark dataset repo. It has to match one of the official board IDs exactly, or the score goes nowhere.
dataset.task_id picks a sub-task or split when a board has several (for GPQA, diamond).
value is the number that gets ranked. The board's lower_is_better flag decides the sort order (WER on the ASR board, for example).
source is an attribution link. In practice it usually points back at the model card.
Optional keys show up in the wild too, like date and a free-text notes field for the protocol (sample count, temperature, token budget, precision). Use them. They are the only place a reader can see how the number was produced.

The file is a YAML list, so a single file can hold several benchmark results. Many repos still use one file per benchmark, which keeps diffs and PRs small.

Why does my model not appear on the Hugging Face leaderboard? (the base_model filter)

This is where most authors get stuck. The leaderboard endpoint has a parameter called base_model. Without it, the API (and the default widget) drops models whose card declares a base_model, meaning fine-tunes, merges, quantizations and adapters. With ?base_model=false, those derivatives come back:

B=Idavidrein/gpqa
curl -s "https://huggingface.co/api/datasets/$B/leaderboard" | python -c "import sys,json;print(len(json.load(sys.stdin)))"
curl -s "https://huggingface.co/api/datasets/$B/leaderboard?base_model=false" | python -c "import sys,json;print(len(json.load(sys.stdin)))"

For GPQA that is 111 vs 175. I ran the same comparison for all 48 boards with this script:

import requests
from concurrent.futures import ThreadPoolExecutor

API = "https://huggingface.co/api/datasets"
boards = [d["id"] for d in requests.get(f"{API}?filter=benchmark:official&limit=200").json()]

def count(bid):
    d = requests.get(f"{API}/{bid}/leaderboard", timeout=60).json()
    a = requests.get(f"{API}/{bid}/leaderboard?base_model=false", timeout=60).json()
    return bid, len(d), len(a)

rows = list(ThreadPoolExecutor(8).map(count, boards))
dflt = sum(r[1] for r in rows); allr = sum(r[2] for r in rows)
print(len(rows), dflt, allr, f"{(allr - dflt) / allr:.1%} hidden")

Output on 2026-10-05: 48 1023 1469 30.4% hidden.

How much is hidden varies a lot from board to board:

Board	Default	With derivatives	Hidden share
TIGER-Lab/MMLU-Pro	141	216	34.7%
Idavidrein/gpqa	111	175	36.6%
cais/hle	89	108	17.6%
SWE-bench/SWE-bench_Verified	67	89	24.7%
hf-audio/open-asr-leaderboard	62	86	27.9%
likaixin/ScreenSpot-Pro	23	48	52.1%
openai/gsm8k	18	46	60.9%
MMMU/MMMU_Pro	10	24	58.3%
llamaindex/ExtractBench	11	25	56.0%
LiquidAI/ifstruct-v1.0	6	18	66.7%
mteb/arguana	3	16	81.3%
crosbylegal/RedlineBench	13	13	0.0%

The pattern makes sense. Older or easier benchmarks like GSM8K attract many fine-tunes, so derivatives are the majority there and the default view shows a minority of the reported scores. On agentic and legal boards where most submissions are base models from labs (RedlineBench, skillsbench, Long-Horizon-Terminal-Bench), the two views are identical.

The filter reads the model card metadata, not the weights. A model trained from scratch that lists a base_model anyway (some teams do this to credit an architecture or tokenizer source) gets hidden just like a LoRA. A heavily modified derivative that leaves the field out shows up as if it were a base model. So the default view is a filter on declared lineage, not real lineage.

Who verifies scores on Hugging Face official leaderboards?

According to the API, nobody does yet. Of the 1,023 default-view rows, 0 had verified: true. Every number on these boards is self-reported by whoever controls the model repo or opens a PR to it.

The PR path is bigger than most people expect. 772 of 1,023 visible rows (75.5%) carried a pullRequest number. So the YAML sits on an open (unmerged) PR ref such as refs/pr/1, not on main. Reading those files directly needs the PR ref:

# file on an open PR, not on main
curl -s "https://huggingface.co/XiaomiMiMo/MiMo-V2.5-Pro/raw/refs%2Fpr%2F1/.eval_results/gsm8k.yaml"

If you try tree/main/.eval_results on such a model, you get .eval_results does not exist on "main" even though the row shows on the leaderboard. That is expected behavior, not a bug.

Practical implications for anyone citing these boards:

A row is a claim, not a measurement. Follow the source.url and read the notes, if there are any.
Protocols differ across rows on the same board. Majority voting over many samples, different thinking budgets, different task subsets: none of that is normalized. The task_id field is the only structural guard, and it only separates named splits.
Rows can come from someone other than the model owner, through a PR. Check the PR author before treating a number as the vendor's.

How do I submit a model score to a Hugging Face benchmark leaderboard?

There is no submission form. You commit a file:

from huggingface_hub import HfApi
import yaml, io

result = [{
    "dataset": {"id": "Idavidrein/gpqa", "task_id": "diamond"},
    "value": 71.2,
    "date": "2026-10-05",
    "source": {"url": "https://huggingface.co/your-org/your-model", "name": "Model Card"},
    "notes": "GPQA Diamond, 198 items, pass@1, temperature 0, bf16",
}]
buf = io.BytesIO(yaml.safe_dump(result, sort_keys=False).encode("utf-8"))
HfApi().upload_file(
    path_or_fileobj=buf,
    path_in_repo=".eval_results/gpqa.yaml",
    repo_id="your-org/your-model",
    commit_message="Add GPQA Diamond eval result",
    # create_pr=True,  # use this when you are not the repo owner
)

The numbers above are placeholders for illustration. Then check whether the row appears, and which view it appears in:

import requests
def where_am_i(board, model):
    base = f"https://huggingface.co/api/datasets/{board}/leaderboard"
    d = [e for e in requests.get(base).json() if e["modelId"] == model]
    a = [e for e in requests.get(base + "?base_model=false").json() if e["modelId"] == model]
    return {"default_view": bool(d), "with_derivatives": bool(a),
            "rank_default": d[0]["rank"] if d else None,
            "rank_all": a[0]["rank"] if a else None}

print(where_am_i("Idavidrein/gpqa", "your-org/your-model"))

If with_derivatives is true and default_view is false, the base_model field in your card is what hides you.

Checklist for model authors before reporting a leaderboard rank

Match dataset.id exactly to one of the 48 IDs from filter=benchmark:official. Case and owner prefix matter.
Set task_id when the board has subsets. A GPQA "main" score sitting next to "diamond" scores is misleading.
Decide on base_model on purpose. If your model really is a fine-tune, keeping the field is honest, but you will only appear in the derivative-inclusive view. If the field is left over from a template or only credits a tokenizer, fix the card. Don't remove it from a real derivative just to get a better spot.
Verify with the parameterless API, not the ?base_model=false URL and not a cached screenshot. The parameterless call is what most visitors see by default.
Quote the right rank. "Rank 3 including derivatives" and "rank 3" are different claims. With 30.4% of entries hidden, the gap can be large.
Write the protocol in notes: sample count, decoding settings, token budget, precision, subset size. Nobody verifies rows centrally, so this is your credibility.
Merge your own PRs. If a contributor opened the eval PR, the row already counts. Merging it puts the file on main and makes the history auditable.

FAQ

How many official benchmark leaderboards are on Hugging Face?

48 datasets carried the benchmark:official tag on October 5, 2026. Four of them had no entries yet. Query https://huggingface.co/api/datasets?filter=benchmark:official&limit=200 for the current list.

What does `base_model=false` do in the Hugging Face leaderboard API?

It adds back models whose card declares a base_model (fine-tunes, merges, quantizations, adapters). The default request leaves them out. Across all boards that is 1,469 entries vs 1,023, so 30.4% are hidden by default.

Where does a Hugging Face leaderboard score come from?

From a YAML file under .eval_results/ in the model repo, on main or on an open PR ref. The leaderboard row's filename field gives the path, and pullRequest gives the PR number if there is one.

Are Hugging Face official leaderboard scores verified?

Not at the time of measurement. All 1,023 default-view rows had verified: false. Treat each row as a self-reported claim and check its source and notes.

Why does my model show on the board in one browser link but not another?

You are probably comparing the default view with the derivative-inclusive view. Run both API calls for your modelId. If only the ?base_model=false call finds you, your model card's base_model field explains it.

Can someone else add a score for my model?

Yes. Anyone can open a pull request that adds .eval_results/*.yaml to your repo, and 75.5% of visible rows came from PR refs when I measured. Review those PRs like any other change to your model card.

Methodology: all counts come from unauthenticated requests to the public Hugging Face API on 2026-10-05. Leaderboards change daily, so re-run the scripts above for current numbers.

DEV Community: huggingface

Training FLUX.2 and Kolors LoRAs on a space dataset

The dataset

The LoRAs

What got used

Links

Testing the Hugging Face Inference API (Free Tier)

Testing the Hugging Face Inference API (Free Tier)

Prerequisites

Quick Start

Testing Different Free Models

Error Handling & Rate Limits

Conclusion

Hugging Face Hits 100K Models: A New Era for Open AI

The 100,000 Model Milestone

From Centralized to Distributed Innovation

Industry Impact and Enterprise Adoption

What's Next: The Race for Efficiency

Hugging Face Hits 1M Models: Open Source AI's Big Leap

A Landmark Moment for Collaborative AI

From Research to Reality

Why This Matters for Startups

The Economic Implications

What's Next

EmbeddingGemma 2 puts multimodal search on a phone

What an embedding model does

How it is built

Benchmarks and speed

Where to get it

What this means for developers

Also available in

Related on Tech AI Wire

Sources

The base_model Filter: Why 31% of Hugging Face Leaderboard Entries Are Hidden by Default

TL;DR

What is the base_model filter and why does it hide entries?

How large is the hidden share, measured today?

How do I see the hidden entries?

Can I reproduce the 31.1% number?

Why this matters for authors

Checklist: keep your entry from being hidden

FAQ

Method notes

When a 0.4-Point Lead Means Nothing: Reading Open-Model Leaderboard Margins Like a Statistician

TL;DR

Why do the top open models cluster so tightly?

How do I compute the error bar on a single accuracy score?

When is a leaderboard gap actually significant?

How do I tell whether a benchmark can discriminate at all?

What should I actually do with a leaderboard, then?

FAQ

Further reading

Open 180B Model Leads 10 Official Hugging Face Leaderboards, and the Zero-Token Judge Behind It

TL;DR

Which Hugging Face leaderboards does the model lead?

Why do the two newest wins matter more than another exam score?

How was the model trained?

What is ZTC, and how does it judge an action without generating text?

FAQ

Further reading

허깅페이스 공인 1위 10개와 무토큰 판정 ZTC: 시험 잘 보는 AI에서 일 잘하는 AI로

TL;DR

10개 부문 1위가 왜 중요한가?

새로 추가된 1위는 왜 '실무'인가?

스스로 배우는 AI, RSI는 어떻게 동작하나?

ZTC는 어떻게 한 글자도 만들지 않고 판정하나?

FAQ

Further reading

허깅페이스 리더보드 1위 9개: 오픈소스 LLM Darwin-180B-RSI와 재귀적 자가개선(RSI) 개발자 가이드

TL;DR

허깅페이스 공인 리더보드 1위 9개는 정확히 무엇인가요?

리더보드 순위를 코드로 직접 확인하려면 어떻게 하나요?

IFStruct 98.95%는 개발자에게 왜 중요한가요?

ExtractBench 90.29는 어떻게 나왔나요?

재귀적 자가개선(Model-level RSI)은 어떻게 작동하나요?

Darwin-180B-RSI를 직접 실행하려면 어떻게 하나요?

vLLM으로 서빙 (B200 8장 또는 동급)

OpenAI 호환 API로 호출

Transformers로 로드

ZTC는 무엇이고 JEV와 무엇이 다른가요?

What does `base_model=false` do in the Hugging Face leaderboard API?