SWE-Bench Pro Has Two Versions: What the Hugging Face Leaderboard's 47 Rows Mix Together

TL;DR

• On October 31, 2026 the Hugging Face SWE-bench Pro board lists 47 rows, 0 marked verified. Row 1 is 94.94, row 2 is 68.74, and the median row is 56.1. A 26.2 point cliff invites suspicion, but the first question is not "is it fake". It is "which benchmark is each row on".
• SWE-bench Pro now exists in two versions. V1 has 731 tasks. V2 (September 22, 2026) has 642: 89 tasks were dropped, and the authors' paper says 480 of the 731 were repaired or retired. The board's official task points at V2, but the top row reports 694/731 on V1.
• The top row is better documented than most: its evidence dataset holds a per-task result for all 731 ids. I checked it. The ids match V1 exactly, 694 are marked resolved, and the numbers in the README, summary file and results file agree.
• What the files cannot show: how many attempts each task got, whether the self-improvement rounds saw the same 731 tasks, whether the network was restricted, and whether grading was repeated in a clean container. The model card does not say. It also gives the baseline as 57.7 in one place and 57.0 in another.
• Two other rows copy a number from a different repo. One is a quantized checkpoint carrying its base model's score, byte for byte.

What does the Hugging Face SWE-bench Pro board contain?

The dataset repo ScaleAI/SWE-bench_Pro is tagged as an official benchmark with an eval.yaml. Models report scores through .eval_results/*.yaml files in their own repos, and the board lists what those files say. I pulled the JSON behind the leaderboard view. I found no documentation for that endpoint, so treat it as unstable. Snapshot of October 31, 2026 (KST):

Property	Value
Rows	47
Rows with verified: true	0
Top score / second score / median	94.94 / 68.74 / 56.1
Source is a model card	39
Source is a link to Scale's public leaderboard page	6
Source is a technical report	1
Source is an evidence dataset	1 (the top row)
Rows with a free-text notes field	9
Rows whose notes name an agent harness	6

The six rows that cite Scale's leaderboard are all older models (scores from 9.67 to 38.7). The six rows that name a harness name four different ones: Claude Code (three rows), LingCode, NexAU and SWE-Agent. That leaves 41 of 47 rows with no stated harness. In agentic coding the harness is part of the system, so a score without one is a score of "model plus unknown scaffold".

Two notes say the score was produced on the card's own "refined" version of the benchmark with problematic tasks corrected. Those rows are not on the same task set as their neighbours either.

What is the difference between SWE-bench Pro V1 and V2?

The dataset card says the default config is now V2 (2026-09-22): 642 tasks across 11 repositories (go 256, python 237, js 145, ts 4). The hard config is a 51-task subset. The original 731-task release is preserved unchanged as config v1. From the card:

• 89 tasks were dropped after review for invalid or unrepairable tests.
• 529 problem statements were rewritten so every graded assertion traces to a sentence in the text.
• 214 test patches and 38 gold patches were revised, and test lists were regenerated against repaired parsers.
• Every image was rebuilt from a sanitised repository bundle with no fixing commit, stray refs, stashes or hooks.

The V2 paper (arXiv 2610.24206, submitted October 16, 2026) says audits, adjudication and blind re-implementation "repaired or retired 480 of the 731 tasks", which leaves 251 untouched. The chart above splits the 731 into untouched (251), dropped (89) and repaired (391, which is 480 minus 89, my arithmetic). The paper also adds a hardened protocol: network access only to the model endpoint, and every patch re-graded in a clean container. It reports that with the network open, 32 of 642 runs of one frontier model (5.0%) contacted code hosts and 25 downloaded the task's own repository.

Scale's announcement also notes that the public repositories, their fixing commits and the benchmark itself have been on the open web since before the evaluated models were trained. That makes exposure a concern for V1 numbers in general. It is a statement about V1 as a measuring instrument, not about any one row.

The consequence for a board: a V1 score over 731 tasks and a V2 score over 642 tasks have different denominators and, after the refresh, different task quality. The board's task id (SWE_Bench_Pro) maps to the default config, which is V2. Nothing in the row format says which version a number came from.

Which version does the 94.94 row use?

The top row, PCL-RSI-lab/Seshat-agent, says "SWE-Bench Pro v1 (731), resolve rate 694/731=94.94%" in its notes. 694/731 is 94.938%, so the rounding is right. It is a V1 number on a V2 board. The 26.2 point gap to row 2 therefore mixes at least three things: model quality, harness, and benchmark version. For context, the V2 paper's abstract reports that frontier agents resolve up to 99.4% of the 642 V2 tasks under the hardened protocol, so a 94.94 on V1 is not out of range for a strong agent system by itself.

The row links to an evidence dataset, which is the right thing to do and rare on this board. I read it.

What can the evidence files confirm?

I downloaded the result files and the official V1 task list and compared them. The short script below reproduces the headline checks (it uses two public HF endpoints and the evidence dataset):

import json, urllib.request

def get(url):
    req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})
    return json.load(urllib.request.urlopen(req, timeout=120))

# 1. the board (undocumented JSON behind the leaderboard view; treat as unstable)
board = get("https://huggingface.co/api/datasets/ScaleAI/SWE-bench_Pro/leaderboard")
print(len(board), "rows,", sum(r["verified"] for r in board), "verified")

# 2. the task sets: 'default' is V2, 'v1' is the original release
def ids(config):
    out, off = set(), 0
    while True:
        d = get("https://datasets-server.huggingface.co/rows?dataset=ScaleAI/SWE-bench_Pro"
                f"&config={config}&split=test&offset={off}&length=100")
        out |= {r["row"]["instance_id"] for r in d["rows"]}
        off += 100
        if off >= d["num_rows_total"]:
            return out
v1, v2 = ids("v1"), ids("default")
print(len(v1), "v1 tasks,", len(v2), "V2 tasks,", len(v1 - v2), "dropped")

# 3. the evidence behind the top row
base = "https://huggingface.co/datasets/PCL-RSI-lab/SWE-Bench-Pro-Evaluation-Evidence/resolve/main/"
ev = get(base + "eval_results.json")
print(len(ev), "graded;", sum(ev.values()), "resolved; same ids as v1:", set(ev) == v1)
print("resolved on the 642 retained ids:", sum(ev[i] for i in ev if i in v2))

Output on October 31, 2026: 47 rows, 0 verified, 731 v1 tasks, 642 V2 tasks, 89 dropped, 731 graded; 694 resolved; same ids as v1: True, resolved on the 642 retained ids: 610.

Check	Result
Ids in the results file vs official V1 list	identical, 731 of 731
Resolved count vs README and summary file	694 in all three
Unresolved / empty patches	37 / 2
Submitted in two parts (no overlapping ids)	617 ids with 588 resolved (95.30%), 114 ids with 106 resolved (92.98%)
Resolved on the 642 ids that V2 kept	610 of 642 (95.02%)
Resolved on the 89 ids V2 dropped	84 of 89 (94.38%)
Resolved on the 51 ids in the hard config	41 of 51 (80.39%)
95% Wilson interval for 694/731	93.1 to 96.3

Per repository, the row resolves 100% of three repos (79 of 79, 56 of 56, 20 of 20 tasks) and 85.5% of the weakest (53 of 62).

Two cautions about reading this table. The 642 and 51 ids are V2 task names, but V2 rewrote instructions and tests for many of them, so these are the same ids and not the same tasks. And the 89 dropped ids show almost the same rate as the retained ones, which means that rate cannot tell you whether the tests on those 89 were passable for the right reasons.

I also compared the submitted patches with the reference patches. Of 731 patches, 729 are non-empty and none is byte-identical to the reference. For resolved tasks the median share of the reference patch's added lines that also appear in the submitted patch is 0.91, against 0.42 for the 35 non-empty unresolved ones. That is what correct fixes should look like. It says nothing about how the patches were produced. Separately, 111 of 729 patches touch a path that looks like a test file (by a simple name pattern), which is ordinary agent behaviour. Without a reference distribution from another system on the same tasks, none of these numbers can be read as evidence for or against anything. I report them so nobody has to recompute them.

What can the files not confirm?

The result files prove that 694 patches were marked resolved by some grader. They do not show:

1. Which grader and which environment. The V2 protocol re-grades every diff on a pristine image with the network locked to the model endpoint. A V1 run on the older images carries no such guarantee, and the card does not describe the setup.
2. How many attempts each task got. Pass@1, best of several, or a loop that retries until the grader is satisfied are different measurements.
3. Whether the evolution rounds saw the test tasks. The card describes a self-improving agent whose "successful traces feed back into the meta-knowledge base for the next round" and a curve that climbs "through successive self-improvement rounds" on SWE-Bench Pro. If the traces came from these same 731 tasks, the final number is a score on seen tasks. If they came from other tasks, it is a held-out score. The card does not say which. I searched its text for held-out, network, attempt, contamination, retry and sandbox, and none appears.
4. Internal consistency of the card. The baseline is "57.7%" in the key-results table and "57.0" in the figure table. The headline is 94.94 in one table and 95.1 in the other (the second is read off a figure and labelled as such).

None of this says the result is wrong. It says the row is a claim with a strong artifact trail and four open questions, and a reader who wants to rank it against other rows needs the answers.

How much can 731 tasks separate?

Even with a clean protocol, a task count limits resolution. At a resolve rate near 60%, one standard error over 731 tasks is 1.81 points. If two systems are scored on the same tasks, their outcomes are correlated, which shrinks the noise in the gap. The smallest gap that clears a 95% test is:

Tasks	rho = 0	rho = 0.5	rho = 0.8
731 (V1)	5.0	3.6	2.2
642 (V2)	5.4	3.8	2.4

Nobody publishes the per-task correlation between two vendors' runs, so rho is an assumption. The practical reading: on this board, twelve rows (ranks 4 to 15) sit inside a 7.1 point band, 66.1 down to 59.0. That is about the size of one unpaired detectable gap. Across different harnesses and task versions it is a cluster, not a ranking.

Why do two rows repeat a number from another repo?

Two rows cite a model card in a different repository:

• INCModel3/GLM-5.1-MXFP4-Mixed-CT-AutoRound shows 58.4 with source zai-org/GLM-5.1 and the note "high reasoning". Its YAML is identical to the base model's file. A 4-bit quantized checkpoint can score differently from its base, and nothing here says it was measured.
• MuVeraAI/Laguna-XS.2 shows 44.5 and cites poolside/Laguna-XS.2. That card's own file says 46.3. The copy and its cited source disagree by 1.8 points, and there is no note explaining why.

Other rows (the ornith-ai ones) cite a card under another organization name, which looks like a rename or mirror and is harmless. The point is that the board will list a row for any repo that carries a YAML file, and the only provenance field is a URL the repo owner typed.

A checklist for reading an agent-benchmark row

1. Which task version and how many tasks? V1 731, V2 642, a "refined" set, or the Hard 51 are different benchmarks.
2. Which harness and model, together? A bare model name with no scaffold is half a description.
3. Is there a per-task artifact? Per-task results and patches let you recompute the headline. A bare number does not.
4. Does the artifact trail answer attempts, network policy and held-out status? If not, write "unknown" next to the row.
5. Does the row measure the repo it sits in? Quantized and fine-tuned copies often inherit the base model's number.
6. Is the gap bigger than the noise? At 731 tasks, anything under about 4 points needs a paired test.

FAQ

What is the SWE-bench Pro V2 benchmark?
A revised public split of SWE-bench Pro with 642 tasks in 11 repositories, released September 22, 2026. It drops 89 invalid tasks, rewrites instructions, repairs tests, and ships images without the fixing commit. V1, with 731 tasks, is preserved as a separate config.

Can I compare a V1 score with a V2 score?
Not directly. The denominators differ (731 vs 642), and V2 changed instructions, tests and images for hundreds of tasks. The V2 authors also flag public exposure of the V1 tasks as a concern.

Does the Hugging Face leaderboard verify scores?
On this board, no. All 47 rows are marked verified: false. The board displays what each repo's .eval_results file declares, so treat each row as a claim with a source link.

Is the 94.94 result fake?
This analysis found no inconsistency in the evidence files, and it also cannot confirm the protocol. The files agree with each other and with the official V1 task list. The open questions are attempts, network policy and whether self-improvement rounds saw the test tasks.

What would make the row comparable to the others?
A run on the V2 default config under the locked protocol, with the harness, attempt count and training-task split stated.

What is the minimum gap that means something at 731 tasks?
About 5 points for independent runs near 60% accuracy, and roughly 2 to 4 points when both systems answer the same tasks and their outcomes are correlated. The correlation is an assumption until someone publishes it.

Further reading

• GPQA Diamond Has 198 Questions: What the Hugging Face Leaderboard's 115 Rows Can and Cannot Rank
• How Hugging Face Official Benchmark Leaderboards Actually Work

Sources

• SWE-bench Pro dataset card and configs: https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro
• SWE-Bench Pro V2 paper: https://arxiv.org/abs/2610.24206
• Scale announcement: https://labs.scale.com/blog/swe-bench-pro-v2
• Top row model card and evidence dataset: https://huggingface.co/PCL-RSI-lab/Seshat-agent and https://huggingface.co/datasets/PCL-RSI-lab/SWE-Bench-Pro-Evaluation-Evidence

All counts in this article are my own measurements from these sources on October 31, 2026 (KST); I have no affiliation with any model, lab or benchmark named here. The board JSON endpoint is undocumented and may change.