Our agent checked its work 26 extra times and scored the same. The runs that failed code review scored lower.
Most guides tell you to add a verification loop, so the agent reviews its own output before it ships. We run a public scorecard for coding-agent runs, and we went looking for what that checking is actually worth.
Here is the cleanest case we found. One model ran the same task twice, a week apart. The first time it stopped to check its own work 26 times, counting every verification action the run logged. The second time it checked zero times, and our scorecard printed "output shipped unchecked" on the result.
Both runs landed in the same place, 63.6 and 63.7 out of 100. Twenty-six extra self-checks came to nothing.
Six models did the same job twice
Five more models ran our largest task more than once, checking heavily on one run and barely at all on another. That gave us six chances to watch one model do the same job twice with only the checking changed.
| Model | checked a lot | barely checked | difference |
|---|---|---|---|
| Gemini Pro 3.1 preview | 27 checks, scored 64.6 | 5 checks, scored 56.1 | +8.5 |
| Deepseek Flash v4 | 36 checks, 81.4 and 24 checks, 67.1 | 2 checks, 72.4 and 0 checks, 69.2 | +3.4 |
| Kimi k2.6 | 26 checks, scored 63.6 | 0 checks, scored 63.7 | -0.1 |
| Glm 5.2 | 67 checks, scored 61.0 | 2 checks, scored 62.1 | -1.1 |
| Grok 4.5 | 22 checks, scored 64.1 | 7 checks, scored 67.1 | -3.0 |
| Gemini Flash 3 preview | 30 checks, 64.4 and 25 checks, 64.0 | 8 checks, scored 72.7 | -8.5 |
Heavy checking won twice and lost four times, averaging a tenth of a point in the wrong direction. The swings run from 8.5 points better to 8.5 points worse and they cancel each other out.
The Deepseek row shows the scale of the noise underneath all this. Both of its heavy runs are there, at 36 and 24 checks, scoring 81.4 and 67.1. That is one model doing one task at roughly one level of checking and landing fourteen points apart. Gemini Flash 3 preview did the same thing and moved 0.4 points.
Across every run we flag, the correlation is zero
Eighty-five of our 158 runs carry a flag in one direction or the other, and each prints its exact check count. Correlate that count against the final score across all 85 and you get r = +0.055.
| checks | runs | mean score |
|---|---|---|
| 0 | 9 | 63.0 |
| 1 to 4 | 15 | 65.3 |
| 5 to 14 | 19 | 66.0 |
| 15 to 29 | 15 | 68.5 |
| 30 to 69 | 11 | 70.5 |
| 70 or more | 16 | 66.8 |
The runs that checked the most sit below the runs that checked a moderate amount, so the line does not even rise all the way. The middle of that column is model tier showing through, which the next number settles.
That zero is sturdier than it looks. Bigger models check more, and bigger models also score higher, so the bias in that correlation runs toward checking. We stacked the deck in its favor and it still came out at nothing.
The model you pick is the one reliable lever
Twenty-seven models on our board have three or more runs behind them. The highest-averaging of those, Mimo v2.5, sits about 25 points above the lowest. That is twenty-five points out of a hundred, decided before the run starts. Narrow it to one Statement of Work so every model is doing the same job, and fifteen models still spread 22.8 points.
Checking offers nothing comparable, and the reason is not that its effect is small. Its effect has no dependable direction. Heavy checking won two of those six matchups and lost four, by as much as 8.5 points either way, so there is no effect size to bank on in the first place.
One of those two numbers you can plan around, and it is the one you set before the run starts. If you are deciding where to spend an afternoon of tuning, the model selector is where the predictable points are.
So what separates a high score from a low one?
Inside a single model, the answer turns out to have nothing to do with effort. We ranked all 41 findings that appear on at least fifteen runs by how far apart they separate the average score, and the top of the list and the bottom of the list turn out to be the same axis.
| goes with a HIGHER score | gap | goes with a LOWER score | gap |
|---|---|---|---|
| Cheap per line of code | +6.9 | Expensive per line of code | -5.6 |
| Cheap run | +6.3 | Expensive per request | -5.1 |
| Ran cheap per request | +5.2 | Ran expensive per request | -5.0 |
| Cheap per request | +5.0 | Expensive run | -4.4 |
| Ran token-lean | +4.3 | Many turns to completion | -4.1 |
Every entry at the top is a different way of saying the run did not waste anything. Every entry at the bottom is a different way of saying it did. Verifying heavily sits tenth at +2.4, well behind every cost finding above it.
The high scores go with the runs that wasted nothing. Waste here means tokens burned, requests spent and turns taken to reach the same finished work. Effort is cheap to add and it is not what the top of this board has in common.
What heavy checking comes with
Set the score aside and the heavy checkers look genuinely better behaved.
A run counts as a heavy checker when it verified at least 1.5 times as often as its peer group's median, and as barely checking when it managed 0.62 times that median or less.
| heavy checkers | barely checked | |
|---|---|---|
| Wasted context window (room paid for, unused) | 22% | 52% |
| High tool error rate | 16% | 40% |
| Requests failed | 35% | 48% |
| Slow wall-clock finish | 49% | 23% |
| Fast wall-clock finish | 0% | 27% |
| Many turns to completion | 24% | 8% |
They waste fewer context windows, hit fewer tool errors and drop fewer requests. That is a real difference, and if your agent is falling over on tool calls it is the argument for turning checking up.
It costs time on every row that measures time. Not one of the 37 heavy checkers earned a fast finish on any of the four tasks, against 13 of the 48 that barely checked. Heavy checking comes with cleaner execution and a slower clock, and the high scores on this board sit with the runs that did not spend the time.
The one review number that does move
Self-checking is what an agent does to itself. Review is what a second agent does to its work, and that number behaves completely differently.
Every Favur run puts each task in front of a code-review agent that returns one of two verdicts, PASS or REJECT WITH FIXES. Across the 69 runs whose logs carry those verdicts, that is 1,985 reviews and 296 rejections. This is what happens when they are sorted by how many times a run got sent back.
| times review rejected the work | runs | mean score |
|---|---|---|
| 0 | 30 | 71.4 |
| 1 to 2 | 19 | 68.8 |
| 3 to 5 | 6 | 64.4 |
| 6 to 9 | 3 | 63.2 |
| 10 or more | 11 | 63.8 |
Never rejected, a run averaged 71.4. Rejected three to five times, 64.4. That is a seven-point spread, and the correlation across all 69 runs is r = -0.307, against +0.055 for self-checking. Thirteen models have two runs in this set with different rejection counts, and the more-rejected run scored lower in eleven of them. This one survives the control that killed the other.
What review actually catches
The reviewer wrote 1,569 task reviews carrying 5,062 individual findings, each filed at one of four severities. Ninety-six percent of them are MEDIUM or LOW, and they are housekeeping. They flag a missing docstring, a duplicated helper, an unused import, a file drifting past the length limit. In a review that passes, under one percent of findings are CRITICAL or HIGH.
What blocks a task is a different kind of thing. Of the 200 blocking findings, nearly half are tests and specification compliance. The reviewer finds a test too weak to prove the thing it claims, an acceptance criterion that was never met, a function that does not match the architecture it was written against. Outright broken logic is 15 of the 200.
Thirteen of the 35 CRITICAL findings are the reviewer catching the developing agent claiming work it had not done. One review found a commit cited as producing artifacts that do not exist. Another found an "EXACT_MATCH" claim on two images that differ by 72,455 pixels. A third ran a recorded sweep of 431 passing tests and watched it fail. A fourth found a fix reported as reverted, still sitting in the working tree.
The expensive thing an agent does is reporting that it finished.
What the rejection count is really counting
A rejection sends the work back around the loop. The work goes back, the developing agent fixes it, the reviewer reads it again. Runs that were never rejected averaged 1.09 review rounds per task; runs rejected three to nine times averaged 1.58, and their LLM requests per task ran from 221 up to about 400.
So the rejection count carries most of the same information as the size of the run. It correlates +0.80 with LLM requests and +0.82 with orchestrator turns. Put that next to the ranking above, where every finding at the top of the board is a way of saying the run wasted nothing, and the mechanism falls out. The score is pricing the trip back around at least as much as it is docking points for the defect. Hold run size roughly constant and about half of that seven-point gap closes. Split the 69 runs at the median request count, and inside the smaller half the never-rejected runs average 71.7 against 68.8 for the rest; inside the larger half it is 69.9 against 65.5.
That is the same lesson as the first half of this piece, arriving from the other direction. A self-check count is a number the agent sets, and it tells you nothing. A rejection count is a number the work sets, and what it counts is how many times the job had to be done twice.
How to check this on your own harness
If you count self-checks, verification steps or test runs anywhere an agent gets judged, this takes an afternoon.
Run the same task twice on the same model, changing nothing. Record your check count and your quality metric both times. Then compare two differences. The first is between those two identical runs, and the second is between your careful runs and your careless ones. If the first is bigger, your counter is describing what the run ran into rather than how well it did.
Then do the same for your reviewer. Count the reviews that sent work back, not the reviews that ran, and count the rounds each task needed before it passed. If the rounds move with your quality metric, you have found the number worth watching, and it is measuring rework rather than care.
Then sort every counter you keep into two piles. The ones you set belong in configuration. The ones the work sets belong on a dashboard where somebody reads them. A check count sounds like something you set and behaves like something you set. A rejection count is the other kind.
Go watch the runs
Every run on the board has its own page, and the two this piece opens with are worth a minute each. Here is the one that checked 26 times and the one that shipped unchecked. The two Deepseek runs fourteen points apart are the 81.4 and the 67.1.
All 158 runs, every finding behind every number above, and the full model ranking are on the board. It is independent, self-funded and vendor-neutral, and answering which model to hand the job to is the whole reason it exists.
Favur, the harness these runs come from, is ours. The repositories its runs produce are open, so you can read the code it wrote.
Top comments (0)