TL;DR
- On November 5, 2026 the Hugging Face SWE-bench Verified board lists 78 rows from 68 repos. 51 rows come from pull requests that are still open, 27 sit on a repo's main branch, and 0 are verified.
- Nine rows are above 79.2, the highest entry in the Verified table on swebench.com (data file last updated February 26, 2026). All nine are self-reports. The rank 1 row (86.0) names its harness only in the model card's footnote, and the board row has no note.
- Six repos take 16 of the 78 rows. One gpt-oss repo appears three times at three reasoning efforts, and the 20B model's rows span 23.3 points. A Nemotron model and its re-upload carry the same three harness scores, so six rows are three measurements.
- Three rows cite "SWE-Bench official evaluation" and match the swebench.com mini-SWE-agent runs of February 17, 2026 exactly. One of those repos, GLM-5, also has the maker's own 77.8 on the board, 5.0 points higher and 19 ranks above.
- One task is 0.2 points, and a 95% interval on a single 80% run is about 3.5 points either side. Ranks 6 to 9 (80.6 to 80.2) are 2 tasks apart.
Orange bars are rows whose YAML file exists only in an open pull request. Blue bars are rows already on the model repo's main branch. The axis starts at 74, so bar lengths exaggerate the gaps; read the numbers.
What does the SWE-bench Verified board on Hugging Face contain?
SWE-bench Verified is a human-validated subset of 500 issue and pull request pairs from popular Python repositories. Evaluation runs the repo's unit tests against the patch an agent produces. Hugging Face lists it as a benchmark dataset, so a model repo can publish a result in .eval_results/*.yaml and the dataset's board collects it.
I pulled the board on November 5, 2026 (KST) from the JSON behind the leaderboard view. It has 78 rows. The dataset's own eval.yaml defines two task ids, swe_bench_%_resolved and swe_bench_average_cost, and names the swe-bench evaluation framework. It does not name an agent, a scaffold, a number of attempts or a context length. That matters because for this benchmark the scaffold is part of the measurement. A model alone does not edit a repository; an agent loop does, and each team brings its own.
Of the 78 notes fields, 54 are empty. Nine name a harness or an official run. So most rows tell the reader nothing about how the number was produced, and the card behind each row is the only place to look.
Is anything on the board verified?
No row is. The verified field is false on all 78. I fetched the discussion behind every row that carries a pullRequest number.
| Source | Rows |
|---|---|
| Open PR by nielsr (Hugging Face staff account) | 23 |
| Open PR by SaylorTwift (Hugging Face staff account) | 28 |
| File on the repo's main branch | 27 |
| Verified | 0 |
All 51 pull requests are open. The oldest was opened on February 16, 2026, which is 262 days before my pull; the median age is 153 days, and 36 of the 51 are older than 90 days. The top row is one of them: Ornith-1.5-397B (86.0) comes from a pull request opened on August 19, 2026 by a Hugging Face account, not by the model's owner.
I am not criticizing the design. Opening a pull request with results taken from a model card is a reasonable way to bootstrap a board, and each file is labeled with its source. The cost is that the ranking blends results the owner has accepted with results someone else transcribed, and nothing in the rank column separates them. In the top ten, five rows are on main and five are open pull requests.
Why do nine rows beat the official table's best?
The swebench.com site publishes its own Verified table. The data file behind it has 180 entries, the newest dated February 26, 2026, and the highest value is 79.2, held by two agent submissions from December 2025 (Sonar Foundation Agent with Claude Opus 4.5, and live-SWE-agent with Claude Opus 4.5). Neither is marked as checked. The highest entry among the runs that use the site's bash-only mini-SWE-agent is 76.8 (Claude Opus 4.5, February 17, 2026).
On the Hugging Face board, 9 rows are above 79.2 and 18 are above 76.8. None of that proves anything is wrong. Newer models are stronger, and the site's table has not moved since February. But the two tables are not measuring the same thing. The site's mini-SWE-agent runs give every model the same minimal agent with one bash tool. The board rows come from whatever harness each team used, and the cards say so when you read them:
| Board rank | Repo | Score | What the card or note says about the setup |
|---|---|---|---|
| 1 | Ornith-1.5-397B | 86.0 | OpenHands harness, temperature 1.0, top_p 0.95, 256k context, git history removed, network disabled |
| 2 | Macaron-V1-Venti | 85.6 | Claude Code harness, retries up to 3 times only after an evaluation error (about 0.8%), reports the successful evaluated attempt |
| 7 | MiniMax-M3 | 80.5 | Claude Code scaffolding, internal infrastructure, 4 runs averaged (in the row's note) |
| 21 | MiniMax-M2.5 | 75.8 | swebench.com mini-SWE-agent run, February 17, 2026 |
| 43 | Qwen3-Coder-Next | 70.6 | SWE-Agent as harness (in the row's note) |
The anti-hacking measures in the rank 1 card are good practice. They also show that the number belongs to a specific environment. Among the top-ranked rows whose setup I could read, there are two different agent loops (OpenHands and Claude Code), and neither is the one the site uses.
How much does the harness move the same model's score?
The board itself contains the answer, because some repos have several rows.
NVIDIA's Nemotron-3-Super-120B card reports three SWE-bench Verified results for the same weights: 60.47 with the OpenHands harness, 59.20 with OpenCode and 53.73 with Codex. That is 6.74 points between harnesses, or about 34 tasks. The board shows them at ranks 60, 63 and 68. The RedHatAI repo with the same name carries the same three rows, with the source URL pointing at NVIDIA's card, at ranks 61, 64 and 69. Six rows, three measurements, two repos.
The gpt-oss repos use a different axis. OpenAI's model card gives low, medium and high reasoning effort, and the board lists all three. gpt-oss-120b has 47.9, 52.6 and 62.4 at ranks 75, 72 and 56. gpt-oss-20b has 37.4, 53.2 and 60.7 at ranks 77, 70 and 59. The spread is 14.5 points for the larger model and 23.3 for the smaller one. Nobody is misreporting here; the effort level is in the file name. But a reader who looks up gpt-oss-20b gets a different answer depending on which of its three rows they land on, and the best and worst rows are 18 ranks apart. openPangu-2.0-Flash has the same shape in smaller form: 63.1 thinking and 57.6 non-thinking, 5.5 points.
GLM-5 is the cleanest case. Its single YAML file contains two entries. One is 72.8, sourced to swebench.com with the note "high reasoning, official". The other is 77.8, sourced to the model card with the note "Z.ai reported number". The board shows 77.8 at rank 15 and 72.8 at rank 34, 19 ranks and 25 tasks apart.
The 72.8 matches the swebench.com data file exactly: mini-SWE-agent v2.0.0 with glm-5 at high effort, dated February 17, 2026. The same is true of MiniMax-M2.5 (75.8) and Kimi-K2.5 (70.8). So these three rows are the only ones on the board I could trace to a run under a common harness, and for GLM-5 that run scores 5.0 points under the maker's own figure. The maker's number may well come from a better scaffold, and I did not try to reproduce either one. The point is that the board puts both on the same scale.
Are all the rows distinct models?
Not always. Two more cases.
Ranks 2 and 3 are Macaron-V1-Venti (on main) and Macaron-V1-Coding-Venti (an open pull request by nielsr), both 85.6. I did not check whether the two repos hold the same weights. If they do, two of the top three ranks are one result. If they do not, the identical value is a coincidence worth a footnote.
Ranks 46 and 50 are poolside/Laguna-XS.2 at 69.9 and MuVeraAI/Laguna-XS.2 at 68.2. The second repo's YAML gives the poolside model card as its source. That card's table currently says 69.9, mean pass@1 over 4 runs, with a 262,144 token context. The MuVeraAI repo carries its own copy of the card, last modified April 28, 2026, with a 131,072 token context and 68.2 in the same table. Its downloads field reads 16, against 905,143 for the poolside repo. So the 68.2 looks like an earlier version of a card that was later updated, copied into another repo, and given its own rank. I can see the mismatch between the source link and the number; I cannot see why the copy exists.
Can 500 tasks separate these rows?
The benchmark has 500 tasks, so one task is 0.2 points. The arithmetic below is a sampling calculation for a single run on a fixed set. Prompt and scaffold variance is larger, as the Nemotron rows show.
| Quantity | Value |
|---|---|
| One task | 0.2 points |
| Rank 6 to rank 9 (80.6 to 80.2) | 0.4 points, 2 tasks |
| Rank 1 to rank 6 (86.0 to 80.6) | 5.4 points, 27 tasks |
| 95% half-width at 80% accuracy, n=500 (illustrative) | 3.51 points |
| 95% half-width at 86% accuracy, n=500 (illustrative) | 3.04 points |
| Nemotron-3-Super, Codex vs OpenHands | 6.74 points |
Ranks 8 and 9 are tied at 80.2, and rank 6 is 2 tasks above them. A paired comparison on the same 500 tasks has a narrower interval than the independent one I show, but it needs per-task results, which a board row does not carry. So the top 14 rows, from 86.0 to 78.0, contain only a few differences that I would call larger than noise, and none of them is established by a verified run.
What about the benchmark itself?
In February 2026 OpenAI said it would no longer treat SWE-bench Verified as a measure of frontier coding ability. Third-party summaries of that post, from Pebblous and Decrypt among others, report that of 138 hard tasks audited, 59.4% had flawed tests or problem statements, 35.5% of them from overly narrow tests, and that some tasks showed signs of leakage from training data. OpenAI's page returned HTTP 403 to my fetcher, so I am relaying the summaries and have not read the original; check it before you cite the percentages. At the time, OpenAI pointed to SWE-bench Pro as the alternative. I looked at the Pro board in an earlier post in this series, and it has its own mixing problem.
For a board reader, the implication is practical. A Verified score near 80 or 86 is a number on a benchmark whose main early adopter has questioned the grading. It can still rank models roughly, and the board is still useful as an index of claims. It is a weak basis for a procurement decision or a claim that one open model beats another by two points.
How do I pull the board and check this myself?
The script uses only the standard library. It counts rows by source, finds repos with several rows, and compares the board with the swebench.com data file.
import json, math, urllib.request, collections
def get(url):
req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})
return json.load(urllib.request.urlopen(req, timeout=120))
rows = get("https://huggingface.co/api/datasets/SWE-bench/SWE-bench_Verified/leaderboard")
repos = collections.Counter(r["modelId"] for r in rows)
multi = {k: v for k, v in repos.items() if v > 1}
authors = collections.Counter()
for r in rows:
pr = r.get("pullRequest")
if pr:
d = get(f"https://huggingface.co/api/models/{r['modelId']}/discussions/{pr}")
authors[(d["status"], d["author"]["name"])] += 1
print("rows:", len(rows), "| repos:", len(repos), "| verified:", sum(bool(r["verified"]) for r in rows))
print("open/merged PR rows by author:", dict(authors), "| main-branch rows:", sum(not r.get("pullRequest") for r in rows))
print("repos with 2+ rows:", len(multi), "carrying", sum(multi.values()), "rows")
print("empty notes:", sum(not (r.get("notes") or "").strip() for r in rows))
n = 500
print("one task = %.1f points" % (100 / n))
for acc in (0.70, 0.80, 0.86):
print("accuracy %.2f: 95%% half-width = %.2f points (illustrative, independent runs)" % (acc, 1.96 * math.sqrt(acc * (1 - acc) / n) * 100))
lb = json.load(urllib.request.urlopen(urllib.request.Request(
"https://raw.githubusercontent.com/swe-bench/swe-bench.github.io/master/data/leaderboards.json",
headers={"User-Agent": "Mozilla/5.0"}), timeout=120))
ver = next(l for l in lb["leaderboards"] if l["name"] == "Verified")["results"]
top = max(v["resolved"] for v in ver)
print("swebench.com Verified table: %d entries, max %.1f, newest %s" % (len(ver), top, max(v["date"] for v in ver)))
print("HF rows above that max:", sum(r["value"] > top for r in rows))
On November 5, 2026 it printed:
rows: 78 | repos: 68 | verified: 0
open/merged PR rows by author: {('open', 'SaylorTwift'): 28, ('open', 'nielsr'): 23} | main-branch rows: 27
repos with 2+ rows: 6 carrying 16 rows
empty notes: 54
one task = 0.2 points
accuracy 0.70: 95% half-width = 4.02 points (illustrative, independent runs)
accuracy 0.80: 95% half-width = 3.51 points (illustrative, independent runs)
accuracy 0.86: 95% half-width = 3.04 points (illustrative, independent runs)
swebench.com Verified table: 180 entries, max 79.2, newest 2026-02-26
HF rows above that max: 9
The board endpoint is undocumented, so field names may change. To check a row, open its pullRequest discussion, read the YAML, and then read the footnotes of the source card. Notes that name a harness are rare; the card usually has them.
Checklist for reading a Verified score
- Is the row verified, or an open pull request? Look at the badge, not the rank.
- Which harness and how many attempts? If the YAML note is empty, read the card footnotes.
- Does the same repo appear more than once? Pick the row whose setup you will actually run.
- Is a copy or re-upload carrying someone else's number? Follow the source URL and compare.
- Is the gap bigger than a few tasks and bigger than the harness spread? On this board, same-repo spreads run from 5 to 23 points.
- Does the benchmark still deserve the weight? Read what its own maintainers and its largest early adopter have said about the tests.
FAQ
How many models are on the Hugging Face SWE-bench Verified leaderboard?
On November 5, 2026 the board has 78 rows from 68 repos. Six repos have more than one row, carrying 16 rows between them.
Is the Hugging Face SWE-bench Verified leaderboard verified?
No row has verified set to true. 51 rows come from open pull requests, opened by two Hugging Face staff accounts, and 27 are on the model repos' main branches.
Why does the Hugging Face board show higher scores than swebench.com?
Nine rows are above the 79.2 maximum in the site's Verified data file, last updated February 26, 2026. The board rows are self-reported under each team's own harness, while the site's mini-SWE-agent runs use one minimal bash-only agent. Different harnesses move scores by several points, as the three Nemotron rows show.
Does the agent harness change a SWE-bench Verified score?
Yes. NVIDIA's Nemotron-3-Super-120B card gives 60.47 with OpenHands, 59.20 with OpenCode and 53.73 with Codex. GLM-5 has 77.8 from its maker and 72.8 from a swebench.com mini-SWE-agent run.
Should I still use SWE-bench Verified to compare coding models?
As a rough index, yes. As a deciding number, no. OpenAI said in February 2026 that it no longer treats it as a frontier measure, and the board mixes harnesses, efforts and copies. Run the model in your own agent on your own tasks.
Top comments (1)
순위의 한 행과 독립적인 한 번의 측정을 구분해 읽는 체크리스트가 유용하네요. 원본 결과의 주소·커밋·하네스·추론 설정을 묶은 측정 식별자를 두면, 다른 저장소에 복사된 같은 결과와 같은 모델의 다른 실험을 분리하기 좋을 것 같아요. 작은 점수 차이를 해석할 때도 행 순위만 보기보다 조건이 같은 결과와 과제별 성공 여부를 함께 볼 수 있도록 연결되면 더 도움이 되겠어요.