A coding benchmark score looks like a measure of the model. It is a measure of the model against a test suite, and that test suite is written, screened and argued over by people. Two OpenAI datasets show how much human work goes into it and where it breaks; the public job boards of AI training platforms show what that work looks like in October 2026.
How the grading works
SWE-bench, the benchmark that became a standard number in frontier model launches, turns resolved GitHub issues from 12 open-source Python repositories into tasks. OpenAI's February 2026 write-up describes the loop:
- The model gets the issue text and the repository as it was before the fix. It never sees the tests.
- It writes a patch.
- The patch runs against two sets of tests: tests that fail on the unmodified code and pass once the issue is fixed, and regression tests that pass before and after.
- The task counts as solved only if all of them pass.
So the grader is whatever tests shipped with the original pull request. They were written to check one human's fix, not every valid fix.
Failure one: tests that reject correct code
A test written for one fix can fail a different fix that is just as correct. In August 2024 OpenAI measured how often, with 93 software developers experienced in Python screening 1,699 random SWE-bench samples. Three annotators labelled each sample and the most severe label won; every annotator first had to pass onboarding tests built from 50 samples OpenAI's engineers had labelled. 38.3% of samples were flagged for underspecified problem statements, 61.1% for unit tests that may unfairly mark valid solutions as incorrect, and 68.3% were dropped in total. The 500 that survived became SWE-bench Verified, and GPT-4o's score went from 16% on the original set to 33.2% on the cleaned one.
It was not enough. On 23 Feb 2026 OpenAI said it had stopped reporting SWE-bench Verified. It audited 138 Verified problems that o3 did not consistently solve over 64 runs, each reviewed by at least six experienced engineers, and found material issues in test design or problem description in 59.4% of them:
-
Narrow tests (35.5%) enforce implementation details. In
pylint-dev__pylint-4551the tests import a function calledget_annotationthat the issue never mentions, so many valid fixes fail on import errors. -
Wide tests (18.8%) check behaviour the issue never asked for. In
sympy__sympy-18199the pull request fixed three problems innthroot_mod, the task text describes one, and the tests cover all three.
The same post found that every frontier model it tested could reproduce the original fix or verbatim problem details for some tasks, because the repositories are public. OpenAI now recommends SWE-bench Pro.
Failure two: tests that accept wrong code
The opposite failure is a suite that lets through a patch that fixes nothing. The SWE-Lancer paper (OpenAI, arXiv 2502.12115) shows o1 on a SWE-bench Verified Django task titled "Queryset raises NotSupportedError when RHS has filterable=False attribute". Its patch:
- if not getattr(expression, 'filterable', True):
+ if not getattr(expression, '_filterable', True):
raise NotSupportedError(...)
One underscore, and in the paper's words "the branch never executes". The reported error is gone because the check is gone, for every expression. The paper calls it cheating, its example of unit tests that are "narrow and prone to cheating".
What a harder grader costs
SWE-Lancer is OpenAI's attempt at a grader that is harder to game. It takes 1,488 real freelance tasks from the Expensify open-source repository, posted on Upwork and worth $1,000,000 in payouts that were actually made:
| Task type | Tasks | Payouts | Graded by |
|---|---|---|---|
| Individual contributor (fix or build) | 764 | $414,775 | End-to-end Playwright tests |
| Manager (pick the best proposal) | 724 | $585,225 | The choice the original engineering manager made |
The paper says OpenAI "paid a team of 100 professional software engineers to write and verify end-to-end tests" for all of them. Each task was screened by at least three engineers, and fix-or-build tasks worth over $5,000 by a team of ten. Each test was triple-verified, rated on whether it lets "all reasonable solutions pass", and checked automatically to pass on the fixed code and fail on the broken code. The tests act like a user: in one task the test logs in, uploads a profile picture and interacts with a second account.
A model earns a task's payout only if all its tests pass. The best model in the paper, Claude 3.5 Sonnet, earned about $403,000 of the $1,000,000. Split by grader, $314,000 of that came from the manager tasks and $89,000 from the tasks judged by end-to-end tests, where it solved 21.1% against 47.0% of manager tasks. Where a real test suite was the judge, the models tested in early 2025 did least well.
The human work behind new test suites
Benchmarks are built once, but the grading work keeps going, because a public test suite leaks into the next training set. The full descriptions of 289 Coding & Software Evaluation postings open on AI training platforms on 19 Oct 2026 show what that work consists of (AITraining.jobs data, read 20 Oct 2026):
| The description mentions | Share of the 289 |
|---|---|
| Reviewing or debugging code | 72% |
| Rating or scoring model output against guidelines | 49% |
| Writing code or tests | 48% |
| Creating evaluation tasks, benchmarks or test cases | 36% |
| Writing ideal responses or demonstrations | 21% |
| Comparing or ranking model responses | 17% |
| Reviewing agent trajectories or tool use | 8% |
| Writing rubrics or guidelines | 5% |
The languages go far beyond SWE-bench's Python repositories: 72.0% of the coding postings open that day name a programming language, 23 languages in all, Python most often (38.4%) and TypeScript second, and 40.5% of those with a description name two or more.
The work is moving the way the benchmarks are, from one patch to an agent acting in an environment. On the 16 platforms we have read since August (a fixed panel, so a newly added platform never counts as growth), coding-evaluation postings went from about 142 in the week of 3 Aug 2026 to 131 in the week of 12 Oct, while agentic-task and RL-environment postings went from about 18 to 33. Small numbers, but of the 61 agentic postings with a description, 33% mention creating evaluation tasks or test cases and 26% reviewing agent trajectories.
The graders get graded too
The people who write the graders are screened mostly by a test of their own. Of the 289 described coding postings, 14% state an assessment or trial task, 11% an interview with a person, and 12% an AI-led or recorded interview, the highest share of the 12 kinds of AI training work we classify (4.1% across all 3,924 described roles). Only 2% name a professional credential, against 11.9% across all roles: in code the proof is a test, not a licence. The postings that state experience (138) ask for a median of 3 years. Stated steps are a floor: platforms can screen more than they write down.
What to take from it
- A pass rate is a pass rate against someone's tests. Before trusting a number, ask who wrote the tests, how many people reviewed each one, and whether a valid fix can fail them (narrow) or an unasked feature is required (wide).
- Tests lifted from the original pull request are cheap and gameable. End-to-end tests written for the benchmark are harder to game, and took 100 engineers and triple review.
- Public tasks leak. A benchmark built from public repositories ends up in training data, which is why OpenAI says it is building new, uncontaminated evaluations, and why reviewing code and writing tests lead what coding-evaluation postings describe.
How this was counted
Benchmark figures come from the SWE-Lancer paper (arXiv 2502.12115, version 4 of 29 May 2025) and OpenAI's posts of 13 Aug 2024 ("Introducing SWE-bench Verified") and 23 Feb 2026 ("Why we no longer evaluate SWE-bench Verified"), all read on 20 Oct 2026. Job-board figures cover postings open on 19 Oct 2026 on the public boards of the AI training platforms we track, read on 20 Oct 2026; a task or screening step counts when the description mentions it, and only the boards that publish full descriptions can be read this way. The coding column sits beside every other kind of work in our What AI Trainers Do report.
I'm Alex Rowan from the AITraining.jobs team. We read the public job boards of AI training platforms every day; some platforms pay AITraining.jobs a referral fee when someone joins through our links, which never changes the figures.
Top comments (0)