DEV Community: AITraining.jobs

How AI-written code gets graded: the tests decide, and the tests are often wrong

AITraining.jobs — Tue, 20 Oct 2026 08:52:30 +0000

A coding benchmark score looks like a measure of the model. It is a measure of the model against a test suite, and that test suite is written, screened and argued over by people. Two OpenAI datasets show how much human work goes into it and where it breaks; the public job boards of AI training platforms show what that work looks like in October 2026.

How the grading works

SWE-bench, the benchmark that became a standard number in frontier model launches, turns resolved GitHub issues from 12 open-source Python repositories into tasks. OpenAI's February 2026 write-up describes the loop:

The model gets the issue text and the repository as it was before the fix. It never sees the tests.
It writes a patch.
The patch runs against two sets of tests: tests that fail on the unmodified code and pass once the issue is fixed, and regression tests that pass before and after.
The task counts as solved only if all of them pass.

So the grader is whatever tests shipped with the original pull request. They were written to check one human's fix, not every valid fix.

Failure one: tests that reject correct code

A test written for one fix can fail a different fix that is just as correct. In August 2024 OpenAI measured how often, with 93 software developers experienced in Python screening 1,699 random SWE-bench samples. Three annotators labelled each sample and the most severe label won; every annotator first had to pass onboarding tests built from 50 samples OpenAI's engineers had labelled. 38.3% of samples were flagged for underspecified problem statements, 61.1% for unit tests that may unfairly mark valid solutions as incorrect, and 68.3% were dropped in total. The 500 that survived became SWE-bench Verified, and GPT-4o's score went from 16% on the original set to 33.2% on the cleaned one.

It was not enough. On 23 Feb 2026 OpenAI said it had stopped reporting SWE-bench Verified. It audited 138 Verified problems that o3 did not consistently solve over 64 runs, each reviewed by at least six experienced engineers, and found material issues in test design or problem description in 59.4% of them:

Narrow tests (35.5%) enforce implementation details. In pylint-dev__pylint-4551 the tests import a function called get_annotation that the issue never mentions, so many valid fixes fail on import errors.
Wide tests (18.8%) check behaviour the issue never asked for. In sympy__sympy-18199 the pull request fixed three problems in nthroot_mod, the task text describes one, and the tests cover all three.

The same post found that every frontier model it tested could reproduce the original fix or verbatim problem details for some tasks, because the repositories are public. OpenAI now recommends SWE-bench Pro.

Failure two: tests that accept wrong code

The opposite failure is a suite that lets through a patch that fixes nothing. The SWE-Lancer paper (OpenAI, arXiv 2502.12115) shows o1 on a SWE-bench Verified Django task titled "Queryset raises NotSupportedError when RHS has filterable=False attribute". Its patch:

- if not getattr(expression, 'filterable', True):
+ if not getattr(expression, '_filterable', True):
      raise NotSupportedError(...)

One underscore, and in the paper's words "the branch never executes". The reported error is gone because the check is gone, for every expression. The paper calls it cheating, its example of unit tests that are "narrow and prone to cheating".

What a harder grader costs

SWE-Lancer is OpenAI's attempt at a grader that is harder to game. It takes 1,488 real freelance tasks from the Expensify open-source repository, posted on Upwork and worth $1,000,000 in payouts that were actually made:

Task type	Tasks	Payouts	Graded by
Individual contributor (fix or build)	764	$414,775	End-to-end Playwright tests
Manager (pick the best proposal)	724	$585,225	The choice the original engineering manager made

The paper says OpenAI "paid a team of 100 professional software engineers to write and verify end-to-end tests" for all of them. Each task was screened by at least three engineers, and fix-or-build tasks worth over $5,000 by a team of ten. Each test was triple-verified, rated on whether it lets "all reasonable solutions pass", and checked automatically to pass on the fixed code and fail on the broken code. The tests act like a user: in one task the test logs in, uploads a profile picture and interacts with a second account.

A model earns a task's payout only if all its tests pass. The best model in the paper, Claude 3.5 Sonnet, earned about $403,000 of the $1,000,000. Split by grader, $314,000 of that came from the manager tasks and $89,000 from the tasks judged by end-to-end tests, where it solved 21.1% against 47.0% of manager tasks. Where a real test suite was the judge, the models tested in early 2025 did least well.

The human work behind new test suites

Benchmarks are built once, but the grading work keeps going, because a public test suite leaks into the next training set. The full descriptions of 289 Coding & Software Evaluation postings open on AI training platforms on 19 Oct 2026 show what that work consists of (AITraining.jobs data, read 20 Oct 2026):

The description mentions	Share of the 289
Reviewing or debugging code	72%
Rating or scoring model output against guidelines	49%
Writing code or tests	48%
Creating evaluation tasks, benchmarks or test cases	36%
Writing ideal responses or demonstrations	21%
Comparing or ranking model responses	17%
Reviewing agent trajectories or tool use	8%
Writing rubrics or guidelines	5%

The languages go far beyond SWE-bench's Python repositories: 72.0% of the coding postings open that day name a programming language, 23 languages in all, Python most often (38.4%) and TypeScript second, and 40.5% of those with a description name two or more.

The work is moving the way the benchmarks are, from one patch to an agent acting in an environment. On the 16 platforms we have read since August (a fixed panel, so a newly added platform never counts as growth), coding-evaluation postings went from about 142 in the week of 3 Aug 2026 to 131 in the week of 12 Oct, while agentic-task and RL-environment postings went from about 18 to 33. Small numbers, but of the 61 agentic postings with a description, 33% mention creating evaluation tasks or test cases and 26% reviewing agent trajectories.

The graders get graded too

The people who write the graders are screened mostly by a test of their own. Of the 289 described coding postings, 14% state an assessment or trial task, 11% an interview with a person, and 12% an AI-led or recorded interview, the highest share of the 12 kinds of AI training work we classify (4.1% across all 3,924 described roles). Only 2% name a professional credential, against 11.9% across all roles: in code the proof is a test, not a licence. The postings that state experience (138) ask for a median of 3 years. Stated steps are a floor: platforms can screen more than they write down.

What to take from it

A pass rate is a pass rate against someone's tests. Before trusting a number, ask who wrote the tests, how many people reviewed each one, and whether a valid fix can fail them (narrow) or an unasked feature is required (wide).
Tests lifted from the original pull request are cheap and gameable. End-to-end tests written for the benchmark are harder to game, and took 100 engineers and triple review.
Public tasks leak. A benchmark built from public repositories ends up in training data, which is why OpenAI says it is building new, uncontaminated evaluations, and why reviewing code and writing tests lead what coding-evaluation postings describe.

How this was counted

Benchmark figures come from the SWE-Lancer paper (arXiv 2502.12115, version 4 of 29 May 2025) and OpenAI's posts of 13 Aug 2024 ("Introducing SWE-bench Verified") and 23 Feb 2026 ("Why we no longer evaluate SWE-bench Verified"), all read on 20 Oct 2026. Job-board figures cover postings open on 19 Oct 2026 on the public boards of the AI training platforms we track, read on 20 Oct 2026; a task or screening step counts when the description mentions it, and only the boards that publish full descriptions can be read this way. The coding column sits beside every other kind of work in our What AI Trainers Do report.

I'm Alex Rowan from the AITraining.jobs team. We read the public job boards of AI training platforms every day; some platforms pay AITraining.jobs a referral fee when someone joins through our links, which never changes the figures.

Developers who grade AI-written code: what's your real hourly rate, and how steady is the work?

AITraining.jobs — Sun, 18 Oct 2026 06:44:48 +0000

I work on AITraining.jobs, which reads the public job boards of the platforms that hire people to evaluate AI-written code. On 18 Oct 2026 there were 285 open coding and software evaluation roles, advertising a median of $65/hr (typical range $60 to $120). Agent-task and RL-environment roles advertise less, a median of $50/hr, and turn over fastest: half of those listed between 3 Aug and 17 Oct came down within 8.3 days.

What the boards can't show:

what people actually earn per hour once unpaid assessments and idle weeks are counted
how many hours of tasks a week you really get, and for how long
whether senior engineers stay with it or use it to fill a gap between jobs

If you've done this work (Outlier, Mercor, micro1, Turing, DataAnnotation or anywhere else), I'd like to hear your numbers, even rough ones, and what surprised you.

Figures from our job-board data, read 18 Oct 2026.

Python is the most requested language in AI training work, and the lowest paid. The data says why

AITraining.jobs — Sat, 17 Oct 2026 19:16:49 +0000

AI labs pay developers by the hour to judge code that models write: review it, write reference solutions, build test cases and set up tasks for coding agents. The platforms that run this work post their openings on public job boards, and we read those boards every day. On 16 Oct 2026, 284 Coding & Software Evaluation roles were open across them, and 69.0% named at least one programming language. That is a fairly direct read of which languages this market is buying, and what it pays for each.

Which languages the roles ask for

Python is named in 100 of the 284 coding roles (35.2%), in the title or the description. TypeScript is second, with 67.
23 different languages are named in coding roles; 38.4% of the roles name one in the title.
Where a full description is published (238 of the roles), 38.7% name two or more languages. Many tasks mix them, for example a TypeScript repository with Python tooling.
31.0% name no language anywhere: 22 have a generic title and no published description, and 66 have a description that names none.

The tools named most often in the descriptions are Git or GitHub (45 of 238), REST or GraphQL APIs (42) and CI/CD (41). A sign that much of the work happens in real repositories, not single snippets.

What each language pays

Median advertised hourly rate of the open roles that name the language and state a rate, for languages with at least 8 such roles on 2 or more platforms (any kind of work, then coding roles only):

Language	Median	Middle half	Roles stating pay	Median in coding roles only
Lean (theorem prover)	$185	$135 to $185	9	n/a (math roles)
MATLAB	$76	$44 to $100	40	n/a
C++	$75	$65 to $85	33	$68
R	$75	$63 to $111	36	n/a
Rust	$75	$65 to $75	22	$75
C	$75	$33 to $82	8	n/a
Bash and shell	$65	$46 to $68	10	n/a
TypeScript	$63	$63 to $75	63	$63
Java	$63	$59 to $66	60	$63
JavaScript	$63	$60 to $70	51	$63
Go	$63	$63 to $75	48	$63
C#	$63	$63 to $63	35	$63
SQL	$60	$43 to $78	44	$68
Kotlin	$58	$36 to $69	10	$58
Python	$45	$40 to $75	337	$63

For comparison, the median across all Coding & Software Evaluation roles that state pay is $65/hr.

Why Python looks like the cheapest language

Python sits at the bottom of the table with a $45/hr median, but that number is a mix effect, not a verdict on Python developers. Python is named in 394 open roles, and only 100 of them are coding roles. Most of the rest are expert roles in finance, medicine, law, math and science that list Python as one tool among others, and they set the median. Across the board, 345 open roles outside Coding & Software Evaluation name a programming language, 202 of them in Law, Medicine & Finance and 83 in Math & STEM, and Python is the language they name most often.

Take only the coding roles and Python's median is $63/hr: exactly where Java, JavaScript, TypeScript, Go and C# sit. In this market, the mainstream languages are priced the same. The language is rarely what moves the rate.

What does move it

Formal proofs. Lean roles advertise a median of $185/hr (9 priced roles on 4 platforms), mostly in Math & STEM: writing machine-checked proofs, the kind of work math reasoning models are trained on. Coq (Rocq) and Agda show up too, in small numbers.
Systems languages. Rust ($75 in coding roles) and C++ ($68) sit above the $63 cluster.
The kind of work, more than the syntax. A senior reviewer who can judge an agent's multi-file change is paid differently from someone grading single functions, whatever the language.

Who hires for which languages

Share of each platform's open coding roles that name a language, 16 Oct 2026:

Platform	Open coding roles	Naming a language	Most named
SuperAnnotate (SME)	67	88.1%	Python
Alignerr	44	54.5% (titles only)	Python
Invisible	42	45.2%	Python
micro1	30	60.0%	Python
Anyone AI	29	93.1%	Python
Turing	23	82.6%	Python
Mercor	16	56.3%	Python

How this was counted

Every figure comes from the public job boards of the AI training platforms we track, read on 16 Oct 2026. Languages are read from role titles on every platform, and from the full description where the board publishes one. Names that are also ordinary words or letters (Go, C, R, Swift, Rust, Lean) count only in a programming context ("Golang", "C/C++", "Lean 4", a list beside another language). A median needs at least 8 roles stating an hourly rate. The figures are recomputed every day, so the live tables will have moved by the time you read this; the full tables and method are in our programming languages in AI training report.

I'm Alex from the AITraining.jobs team; we run the job board these figures come from. Some platforms pay AITraining.jobs a referral fee when someone joins through our links; that never changes the figures.

What AI labs pay developers to review code right now (data from 236 open roles)

AITraining.jobs — Sun, 04 Oct 2026 10:16:15 +0000

AI labs need code that is correct, readable and safe, and they need people who can tell the difference. So they pay developers by the hour to review model-written code, write reference solutions, grade answers and build realistic coding tasks. It is contract work, remote, and the rates are better than most people expect.

We track these roles every day from the hiring platforms' own job boards. Here is what the market looks like on October 4, 2026.

The numbers right now

236 open roles for coding and software evaluation, on 10 platforms today (29 platforms hire for this work at some point).
$55 to $75 an hour is the typical range: the middle half of the 200 hourly rates the postings state, from 9 platforms. The median is $65 an hour.
The top end is $150 to $350 an hour, for senior engineers, open-source maintainers and specialists.
36 of the 236 postings state no hourly rate at all, so ask before you invest hours in an assessment.

Where the open roles are today:

Platform	Open coding roles
OpenTrain AI	102
Invisible Technologies	42
micro1	38
Mercor	37
AfterQuery	8
Alignerr	3
Others	6

These counts move daily as projects open and close. The live list is on our coding and software evaluation page.

What the work actually is

Most of these roles are one of five kinds of work:

Reviewing model-written code. You get a prompt, the model's answer and sometimes a repository. You find the bugs, explain them and rate the answer. This is the bulk of the work.
Writing reference solutions and tests. You solve the task yourself so the model has a correct answer to learn from, often with unit tests that prove it.
Comparing answers. Two model answers, one prompt: which is better, and why? These ratings are the human feedback in RLHF.
Building tasks for coding agents. Realistic multi-step tasks in real repositories, with a way to check whether an agent solved them. This pays more because it takes judgment about what makes a task hard but fair.
Repository and open-source work. Some of the best-paid roles ask for maintainers of real GitHub projects, who review contributions the way they would review a pull request.

You rarely write production features. You are paid for judgment: spotting the subtle bug, knowing the idiomatic fix, and explaining it in a sentence a grader can check.

Which languages are in demand

Most postings ask for "software engineers" without naming a language. Among the titles that do name one, today's count is:

Language	Roles naming it
Python	16
Java	8
C#	8
JavaScript	7
TypeScript	6
Rust	6
SQL	5
C++	5
Kotlin	4
Go	3

Python leads, as you would expect, but Java, C#, JavaScript and Rust are well represented, and specialists in a less common language face less competition.

The best-paid roles today

A sample of the highest posted rates on October 4, 2026:

Software Engineering Specialist (micro1): $150 to $350 an hour
Senior Software Engineer (micro1): $245 to $280 an hour
Open Source GitHub Maintainer (micro1): $150 to $300 an hour
Expert Senior SWE (Mercor): $150 to $210 an hour
SWE Infrastructure Specialist, Java (Invisible Technologies): $150 an hour
Software Engineer Expert (Mercor): $100 to $150 an hour

Roles like these close fast. Treat this as what the top of the market pays, not as openings that will still be there next week.

How hiring works: expect an AI interview

The two platforms with the most high-paying coding roles both screen with an AI interviewer before a human looks at you.

micro1 interviews with an AI recruiter called Zara. It adapts its questions to your resume and the area you choose, asks follow-ups in real time, and is done in the browser on your own schedule. Reviewers put it at roughly 15 to 25 minutes, followed by a role-specific skill check. Reviews describe one to two weeks from application to certification, then days to weeks before you are matched to a project.

Mercor uses an AI video interview of about 20 minutes, in a desktop browser with your camera on, and some roles add a written assessment. You do each interview once and it is reused for similar roles. Mercor offers a free practice interview that does not count against you: take it first. Mercor says hearing back can take two to four weeks.

What gets people through, on both:

Go deep, not broad. Pick the one or two areas you can defend in detail. A profile that claims everything reads as nothing.
Use specifics. "Cut p95 latency 40% by replacing polling with WebSockets" beats "improved performance."
Say "I", not "we". The interviewer is trying to find out what you did.
Keep answers tight. Two or three minutes, structured, with a result at the end.
Make your resume true in every detail. The AI interviewer probes what you listed.

How to tell a real platform from a scam

This market attracts scams because "get paid to train AI" sounds easy. The rules that hold:

You never pay to apply, to train, or to "unlock" work. Real platforms pay you; they do not charge you.
Pay is per hour or per task, stated up front. If the rate only appears after an upfront fee or a "starter kit", walk away.
Expect identity verification, not bank logins. ID checks are normal on real platforms. Requests for your banking password or for payment in gift cards are not.
Work is project-based. Real platforms are honest that projects pause and end; anyone promising guaranteed full-time hours is selling something.

We check each platform's legitimacy signals on its own page, for example micro1 and Mercor.

Where to find these roles

Every open coding role, updated daily: coding and software evaluation jobs
All of one platform's roles with full descriptions: micro1 jobs and Mercor jobs
Pay across every kind of AI training work, measured daily: the AI training pay report

Disclosure: we run AITraining.jobs, a free job board for AI training work, and the figures above come from our daily read of the platforms' own job postings. Some apply links on our site are referral links that pay us if someone is hired. They never change what you are paid.