In October 2023, a team at Princeton published a benchmark called SWE-bench: 2,294 software engineering problems drawn from real GitHub issues and their corresponding pull requests across 12 popular Python repositories. The paper reported that the best model of the day, Claude 2, resolved 1.96% of them. Not nineteen percent. One point nine six.
In May 2025, Anthropic announced Claude Sonnet 4 with a score of 72.7% on SWE-bench Verified.
The tempting sentence writes itself: models got thirty-seven times better at software engineering in nineteen months. And the improvement is real. Anyone who has pointed a current coding agent at a bug knows the difference between now and 2023 is not statistical noise. But the sentence still does not parse, because the two numbers in it are not measurements of the same thing. Between 1.96 and 72.7, something happened that gets left out of every chart that plots them on one axis: the benchmark itself was repaired.
The instrument changed under the name
SWE-bench Verified, the variant nearly every headline score now runs on, is a subset of 500 tasks from the original test set, human-validated for quality. The validation exists because the original 2,294 were harvested from the wild, and wild tasks come with wild problems: issue descriptions that never specified what success looked like, tests that checked details no reasonable patch would produce, tasks that were effectively unsolvable as posed. Filtering those out was the responsible thing to do. A benchmark full of unanswerable questions measures patience, not engineering.
But notice what the repair quietly did. A model scoring 72.7% on the validated 500 is not answering the exam that produced the 1.96%. It is answering the subset of that exam which survived a quality review, which is to say the subset on which high scores are possible. Both numbers are reported under the name SWE-bench. The name is what travels; the referent is what changed. When a chart draws a line from 2023 to now, the line passes through a point where the ruler was swapped, and the chart does not mark it.
This is not an accusation. There is no villain in this story, and that is precisely what makes it worth telling. Benchmark criticism usually needs a bad actor: contaminated training data, teams gaming the leaderboard, a lab picking its best run from forty attempts. All real, all documented elsewhere. This failure needs none of them. Every party can act in good faith, the repair can be genuinely correct, and the number still stops licensing the conclusion people draw from it, because the conclusion compares a thing to a thing that no longer exists.
The population learned the test
The second drift is slower and has no announcement page. In 2023, no system had been built with SWE-bench in mind, which is a large part of why the best score was 1.96. The benchmark sampled a capability nobody had aimed at. That is what made it evidence: a sample tells you about the population when the population does not know it is being sampled.
By 2025, SWE-bench is the scoreboard of an industry. Agent scaffolds are designed around its shape: Python repositories, issues with linked pull requests, success defined by passing the repository's own tests. Anthropic's own announcement is admirably precise about configuration, distinguishing the 72.7% achieved with a bash tool and a file editor from the 80.2% achieved with parallel attempts and a scorer selecting the best candidate. That precision is the tell. When a score needs a paragraph of methodology to interpret, the score has become an engineering target, and improvement on a target is a different fact than improvement on a sample.
Improvement on the measured slice is still improvement. The unlicensed step is the inference from the slice to the field. "Resolves 72.7% of validated Python issues with test-verifiable fixes" and "can do software engineering" are claims of very different sizes, and the benchmark's name performs the promotion from the first to the second without anyone signing off on it.
Three questions before believing a benchmark number
The practical version of this essay fits on an index card. When a number arrives claiming a model got better, ask:
Which version? Original, Lite, Verified, Multimodal, a lab's internal fork? Scores across versions share a name and nothing else. If the version is not stated, the number is not yet information.
What was filtered out, and why? Every repair has a rationale, and the rationale tells you what the new number can no longer see. A set validated for well-specified tasks no longer measures performance on the under-specified ones, and under-specified is what most real work is.
Was the system built with this benchmark in the loop? Not as a gotcha about training contamination, but structurally: if the scaffold was tuned against this scoreboard, the score measures fit between the system and the instrument. Fit is worth knowing. It is just not the same fact as capability.
None of these questions accuses anyone of anything. They are the questions you would ask about a thermometer that had been recalibrated twice and taped to the radiator it was reporting on: not "who lied," but "what does this instrument now measure, and is that the thing I care about?"
The most honest number in the story
There is a case that 1.96% is the best measurement SWE-bench ever produced. It was taken before the repair, before the targeting, before the benchmark's name was worth anything, on an instrument nobody had optimized against. It measured exactly what it seemed to: state-of-the-art models of 2023, facing unfiltered work sampled from the world, mostly failed. Every number since has been produced by a better model taking a better-understood exam that was also becoming a nicer exam, and the shares of those three factors are not separable from the score alone.
A benchmark score is evidence about the world only while the benchmark is a sample of something. Repair it, and the sample changes. Target it, and it stops being a sample at all. The score keeps arriving either way, because a scoreboard's job is to produce numbers, and it will do that job long after the numbers have quietly changed their subject. The name is the last thing to drift. Read the version, read the filter, read the loop, and then decide what the number is allowed to mean.
Sources
Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, Karthik Narasimhan, "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?", arXiv:2310.06770 (submitted October 2023). Source of the 2,294-task count and the 1.96% figure, quoted from the abstract.
SWE-bench Verified dataset card, Hugging Face (princeton-nlp/SWE-bench_Verified): "a subset of 500 samples from the SWE-bench test set, which have been human-validated for quality."
Anthropic, "Introducing Claude 4" (May 2025): Claude Sonnet 4 at 72.7% on SWE-bench Verified with a bash tool and file editor; 80.2% in the high-compute configuration with parallel attempts and candidate selection.
Top comments (1)
The distinction between SWE-bench and SWE-bench Verified is crucial, as it highlights the issue of benchmark drift and the importance of understanding the context of the measurements. The fact that the original benchmark was harvested from the wild and later filtered for quality tasks means that the two scores, 1.96% and 72.7%, are not directly comparable. I've seen similar issues in other benchmarks, where changes to the evaluation methodology or dataset can significantly impact the results, and it's essential to consider these factors when interpreting the data. The author's observation that the population learned the test over time, with agent scaffolds being designed around the benchmark's shape, adds another layer of complexity to the analysis. How do you think researchers and developers can better account for these factors and provide more accurate interpretations of benchmark results?