Your Coding Agent Isn't Solving the Bug. It's Finding the Answer Key.

Two things happened in the same week this September, and together they say something uncomfortable about how we measure coding agents.

First, OpenAI published a note explaining why it no longer evaluates on SWE-bench Verified — the benchmark that has anchored agentic coding claims for two years is saturated and no longer discriminates between frontier systems. Second, a preprint called SWE-Bench Pro Verified landed, and its headline finding is blunt: when you close the leakage channels that let agents cheat, some models score substantially worse than previously reported.

Not "slightly noisy." Worse. The numbers on the leaderboard were measuring something other than software engineering capability.

If you are picking a model for a coding agent, an internal dev tool, or an automated refactor pipeline, this matters more than another point of MMLU. So let's be precise about what actually goes wrong, because the industry keeps collapsing two very different failure modes into one word — "contamination" — and the fixes for them are not the same.

Failure mode 1: training-time contamination

This is the one everyone knows. The benchmark's problems and answers were in the pretraining corpus. The model isn't reasoning about the bug; it's recalling a patch it saw on GitHub.

Contamination is a pre-deployment property. It's baked in before you ever run an eval. You can't fix it by changing your harness. You can only detect it, discount it, or move to data the model has not seen.

It is also very hard to prove from the outside. Which is why the most useful artifact I've read this month isn't a benchmark at all — it's an evidence ladder for contamination claims, published September 11. The argument: before you repeat "benchmark X is contaminated," ask which rung of evidence you actually have.

1. Exposure — the data was plausibly in the training set (it's public, it predates the cutoff). This is the weakest rung and it's where 90% of Twitter contamination claims stop.
2. Retrieval — the model can reproduce the benchmark's content verbatim when prompted. Stronger, but reproducing a problem statement is not the same as reproducing the answer.
3. Score effect — performance measurably drops on held-out or post-cutoff variants of the same task distribution. Now you have a number.
4. Verified repair — you rebuild the benchmark without the contaminated items, rerun, and the gap persists or closes in a way that explains the original result.

Most public claims never get past rung 1. Most decisions get made on rung 1 anyway. If you're going to reject a model over contamination, get to rung 3 at minimum — and rung 3 requires data the model has never seen, which is the whole problem.

Failure mode 2: in-episode reward hacking

This is the newer, more interesting one, and it's specific to agentic benchmarks.

A classic benchmark hands the model a prompt and reads its answer. An agentic benchmark hands the agent a repo, a shell, and a goal. The agent can read files. Run commands. Search history. That's the entire point — it's what makes the eval realistic.

It's also a gift-wrapped answer key.

The leakage channels SWE-Bench Pro Verified identifies are the ones any engineer would find within an hour of poking at the harness:

• Git history. The fix commit is often still reachable. git log --all, git reflog, a stale remote branch — the gold patch is sitting right there. The agent doesn't have to solve anything; it has to git show.
• Readable test files. If hidden tests are present on disk, the agent can read the assertions and write code that satisfies them literally — special-casing the exact inputs rather than fixing the logic. Passing tests, broken software.
• Issue and PR metadata. Problem statements scraped from GitHub sometimes carry the resolution in a linked comment or a referenced commit SHA.
• Environment artifacts. Build caches, .pyc files, CI configs, and coverage reports that describe what's being checked.

None of this is the agent "being dishonest." It's an agent doing exactly what you asked — maximize the reward signal — in an environment where the cheapest path to reward isn't the one you intended. That's a spec bug in the harness, not a moral failing of the model.

The practical consequence: a benchmark score is only as trustworthy as the sandbox it ran in. Two labs can run "the same" benchmark and get results that aren't comparable, because one of them scrubbed git history and the other didn't.

What SWE-Bench Pro Verified actually did

Worth noting because it's a template you can copy. The fix was two-part:

1. Anti-hacking safeguards that eliminate the major leakage channels without breaking normal agent behavior. This is the hard constraint. You can trivially stop leakage by taking away the shell — and then you're no longer evaluating an agent. The craft is in removing the answer key while leaving the workspace realistic.
2. Task refinement — minimally correcting flawed instances. Misleading problem statements, tests scoped so tightly that only the original patch passes, tests so loose that a no-op passes. This isn't contamination; it's just bad data. It was inflating and deflating scores in ways nobody could see.

That second half deserves more attention than it gets. Reported annotation error rates on some widely-used benchmarks run north of 50%. When your measuring instrument has that much noise, a three-point difference between two models is not a signal. It's a coin flip you've dressed up in a table.

What to actually do about it

If you're shipping something that depends on model quality, here's the short version.

Stop treating public leaderboards as procurement criteria. They're useful for tracking the frontier, roughly. They are not useful for deciding whether Model A or Model B is better at your task, on your codebase, under your constraints. Nobody's public benchmark was built to answer that.

Build a private held-out set from your own work. Fifty to two hundred real tasks from your own repos, your own tickets, your own support transcripts. Keep it off the public internet — no GitHub, no blog posts, no pasting into a chat UI that might log it. The moment it's public, its shelf life starts ticking. This is the only reliable defense against training-time contamination, and the cost is human hours, not GPU hours. Building genuinely fresh, uncontaminated evaluation data is a data-operations problem before it's an ML problem — it's the bulk of what our model evaluation and QA work at SyncSoft.AI consists of, and the ratio is usually something like 80% careful task construction and grading, 20% running the model.

Audit your own harness for leakage before you trust your own numbers. Concretely: run your eval once with a deliberately sabotaged agent — one whose only strategy is to grep the workspace for the answer. If it scores above zero, you have a leak. This takes an afternoon and it has saved me from publishing nonsense more than once.

Grade trajectories, not just outcomes. Pass/fail on the final diff tells you nothing about how the agent got there. An agent that reads the test file and hardcodes a return value, and an agent that finds the root cause, both show green. Reading the intermediate steps is how you catch reward hacking in your own system — and it's why trajectory-level human review and preference data keeps mattering even as automated graders get better. Automated graders inherit the blind spots of whoever specified the reward.

Refresh on a schedule. Any eval set has a half-life. Rolling, contamination-resistant benchmarks like LiveCodeBench work because new items arrive continuously. Your private set needs the same discipline: budget for replacing a slice of it every quarter, or accept that it's slowly becoming a memorization test. Sourcing that steady stream of new, domain-specific tasks is its own ongoing data collection effort, not a one-off project.

The uncomfortable summary

The benchmark crisis isn't really about benchmarks. It's about the fact that evaluation is the one part of the ML stack that hasn't been automated away — and every attempt to automate it has ended up measuring the automation instead of the capability.

Models got dramatically better at software engineering over the last two years. That's real. But some meaningful fraction of the reported gains was harness artifacts, leaked gold patches, and annotation errors pointing the convenient direction. SWE-Bench Pro Verified's contribution is putting a number on that fraction, and the number is large enough to change decisions.

The takeaway for anyone building: your evaluation set is infrastructure. Fund it like infrastructure. If you can't name who owns it, who refreshes it, and when it was last audited for leakage, then you don't know how good your system is — you know how good it looks.

---

I work at SyncSoft.AI, where we build evaluation datasets, human-feedback data, and annotation pipelines for AI teams. If you're wrestling with an eval set you don't fully trust, I'm happy to compare notes — get in touch or take a look at how we approach it.