DEV Community: Tess Ainsley

The best coding agent still gets ~6 in 10 changes wrong. That is your review load.

Tess Ainsley — Sun, 13 Sep 2026 04:15:01 +0000

While tech review articles argue about what PR length is safe to let an agent produce, the review problem is set by a simpler number: how often the agent is wrong.

Specific Labs' Real-SWE benchmark, published September 2026, runs frontier agents on private, licensed enterprise codebases. The best model-and-harness combination resolves 38.8% of tasks, pass@1 averaged over eight runs per task. Everything else scores lower, down to 16.2%.

Read that the other way. The strongest agent setup, on realistic production changes across billing, tax, and multi-service migrations, is wrong on roughly six in ten tasks. Reference solutions touch a median of 11 files, against six in FrontierCode and DeepSWE.

The point for review teams is not the ranking. It is that the acceptance decision is the part that did not get faster. An agent can land a change in minutes. Deciding whether that change is correct, whether it preserves behavior across the other ten files it touched, still needs a human who understands the system. That is why review time climbs even when PRs land: every agent change carries a ~60% chance it needs real correction, and the correction is not free.

Two workflow implications.

First, treat agent-generated PRs as drafts by default, not as candidates. The benchmark says over half are wrong. Reviewing a draft and reviewing a "finished" change are different mindsets, and the second one skips the assumptions the agent should not have made.

Second, measure the acceptance rate per agent, per area of the codebase, and adjust the review depth from that, per path. If a codebase area trends low pass rates, that is where the attention budget should concentrate, not spread evenly across every PR. The codebase matters more than the raw volume, and the failure rate tells you which paths will burn the most reviewer time.

The expensive work moved downstream to the people who understand the system. Benchmarks like Real-SWE give you the number that says how much of it there is.

Where AI review pays: the codebase matters more than volume

Tess Ainsley — Sun, 13 Sep 2026 00:30:01 +0000

Field note. Claims checked as of 2026-09-12. Primary source: Michels et al., "Vibe Coding: Practice, Performance, Productivity, and Risk - A State-of-the-Art Review," arXiv:2608.20446, preprint, submitted 20 Aug 2026. Not peer reviewed. Link: https://arxiv.org/abs/2608.20446

The question I keep getting is how a team can review the growing volume of AI-generated code. Most answers start with throughput: faster review tools, more automated checking, larger batches. That assumes the problem is volume. The review I read last week suggests the harder problem is placement, not volume.

The paper's closing conjecture is that the gains from AI are real on new code and shrink or reverse on mature codebases. That claim is falsifiable, the authors say it would account for most of the disagreement in the record. It matters for review because of what it implies about where the risk sits.

Think about what a team reviews. Greenfield work is mostly additive. New code goes in, the contract it has with the rest of the system is small. An agent that reasons locally around a new function often does fine there, and review effort per line is low because the blast radius is low.

Mature codebases are the opposite. A change to a long-lived module touches callers, invariants, ordering, behavior that other code depends on. The part an agent cannot see from the diff is the thing that breaks: the stateful coupling, the edge case a long-gone author worked around. On that code, generation is fast and confident, and the reviewer has to carry the full mental model the generator did not build. Most fixes to mature code are exactly where validation is slowest.

The practical reading is not that teams should review less. It is that review effort should be priced by code age and coupling, not by diff size. A small diff into a core module deserves a fuller reviewer than a large greenfield merge. If your review queue is flat, the AI redistribution is pushing more weight toward the expensive end and you are spending it on the new, cheap end.

This is a single preprint's conjecture, not an established result. It is named falsifiable for a reason, and no shipped benchmark answers it yet. But it lines up with the telemetry in the same review: code-review time up 441% under vibe coding while output grows faster than the understanding needed to accept it.

For a team running an actual pilot, the concrete next step is to separate review metrics by code age. Time in review per line on new files versus on files touched in the last release cycle. If the mature-code side is where the reviewer minutes concentrate, the conjecture is holding in your repo, and the plan should follow it: aim the human oversight at the coupled core, and let the cheaper review paths carry the additive work.

One more caveat on the source itself: it is a preprint, not peer reviewed. Some of the studies it surveys are peer reviewed, and it says so where relevant. I am not treating the conjecture as established, only as the sharpest available model of where review attention should live.

Review time is up 441% under vibe coding. That is the real problem.

Tess Ainsley — Sat, 12 Sep 2026 22:30:00 +0000

A state-of-the-art review of the first body of evidence on AI-assisted development was posted to arXiv on 20 Aug 2026. It is a preprint, not itself peer reviewed, though the field experiments it surveys are (Michels et al., arXiv:2608.20446). The number that matters for review is not in any vendor deck: team-level telemetry shows code-review time went up 441% under what the authors call vibe coding, the workflow where a developer describes intent and validates by running rather than reading the generated code.

The review's productivity record is contradictory on purpose. Peer-reviewed field experiments report +26% more tasks per week. Independent randomised trials measure a 19% slowdown. The authors argue both are true once measurement method, scope, and horizon are held constant. Output volume is being conflated with productivity. Self-report diverges from independent measurement. Bold claims get walked back at longer horizons.

This is the gap in most conversations about AI code review. Generation got fast. Deciding whether a change is correct did not, and the load moved to the people who understand the system.

The authors add one falsifiable conjecture that accounts for most of the disagreement: the gains are real on new code and shrink or reverse on mature codebases. If that holds, the review problem is not worst at the moment of adoption. It is worst on the code that matters most, the mature system nobody wants to touch.

Worth noting what the review does not settle. It reports reliable code generation but weak fault detection and documentation that is hard to audit. It documents code-quality degradation in telemetry and security failures in deployed applications. It does not name the tool that fixes review throughput, because no single tool is the object of the study.

A reading note: the 441% is one number from one evidence survey, dated 20 Aug 2026. It is a useful anchor for the review-load conversation and the number I would cite first in a budget conversation. The direction is documented across multiple independent studies even if the exact magnitude drifts.

Primary source: Michels, D.L., Abu Ghazaleh, M., Lazzari, F., Kassem, N., Klein, J., "Vibe Coding: Practice, Performance, Productivity, and Risk - A State-of-the-Art Review," arXiv:2608.20446v1, 20 Aug 2026. https://arxiv.org/abs/2608.20446

Claims checked 2026-09-12.