DEV Community

Pedro Nagatomo
Pedro Nagatomo

Posted on

What the Data Actually Says About AI Code Review

#webdev #programming #ai #softwareengineering

Most teams treat code review as a formality, a checkbox before merge. That's been true for years, and the research on what actually makes review effective has been sitting there, largely ignored, for about two decades.

Then AI-generated code started arriving faster than review capacity could keep up with it, and the gap between "reviewed" and "actually reviewed" got a lot more expensive.

This isn't a tool pitch. It's a look at what the research says about how review works, where AI changes the picture, and what a review process needs to account for now that a growing share of the code being reviewed wasn't typed by a human in the first place.

What the Older Research Already Established

Long before AI entered the picture, one of the largest empirical studies of code review — Cisco's internal analysis of roughly 2,500 reviews covering 3.2 million lines of code, later published by SmartBear — produced findings that still hold up.

Two results stand out:

  • Review size and duration matter more than most teams assume. A review of 200–400 lines of code conducted over 60–90 minutes tends to surface 70–90% of existing defects. Push past that window and effectiveness drops sharply, concentration on detail-heavy tasks degrades after roughly an hour.
  • Individual review beats group review, by a lot. Individual reviewers catch up to 85% of defects on their own. Group review sessions catch only around 4%. The same research found that keeping reviews under ~200 lines correlates with defect rates around 15 per hour — a rate that only holds at that smaller scope.

None of that has anything to do with AI. It's a description of how human attention behaves under a review task. And it's exactly the constraint that AI-generated code volume is now testing.

What's Changed: Volume Without Proportional Review

GitHub's 2025 Octoverse report gives a sense of scale. Over the twelve months tracked, developers merged an average of 43.2 million pull requests per month, a 23% year-over-year increase, alongside nearly a billion commits pushed across the year. Generative AI usage grew alongside it: more than 1.1 million public repositories now use an LLM SDK, up 178% year over year.

What the raw growth numbers don't show is what happened to review depth as volume increased. An independent analysis of the same Octoverse dataset found that while pull requests rose roughly 20%, comments on commits fell by about 27% over the same period — read as a sign that meaningful review isn't keeping pace with code volume.

The same analysis flagged a shift in the vulnerability landscape: Broken Access Control overtook injection flaws as the most common security issue on GitHub, up 172% year over year — attributed in part to AI-generated scaffolding that produces endpoints resembling correct code while omitting authentication checks.

Put next to the SmartBear findings, this points to a specific mismatch: the review methods proven to work best — individual, focused, bounded in scope — are exactly the ones under the most pressure as code volume scales past what human attention was ever measured to handle.

Where AI-Generated Code Actually Fails

Academic work here is still developing, but a few consistent patterns are showing up.

A study of security defects across the OpenStack and Qt open-source communities found that race conditions accounted for the largest share of security issues caught in review, at 39%, followed by crashes at 22.8% and resource leaks at 10.9%. These are categories that tend to surface only when a reviewer is reasoning about runtime behavior — not just reading syntax for correctness.

That distinction matters for AI-generated code specifically. A model can produce output that's locally coherent — correct-looking syntax, sensible variable names, plausible control flow — while still getting the runtime behavior wrong in exactly these categories.

This lines up with early field research on LLM-assisted review, which found developers using AI-assisted review reported concerns around false positives and insufficient contextual information, even while generally preferring the AI-led approach for large or unfamiliar pull requests.

The practical takeaway isn't that AI review tools are unreliable. It's that they change what kind of review still needs a human — and reviewers who don't adjust their focus accordingly will keep missing the categories the data says matter most.

Have you noticed AI-generated PRs failing in different ways than human-written ones on your team?

A Review Process That Matches the Data

Putting the research together, four things stand out as worth building into any review process — human or AI-assisted.

  1. Keep review scope bounded. The evidence for a 200–400 line window isn't a stylistic preference — it's tied directly to measured defect discovery rates. Splitting large pull requests into smaller ones isn't just tidiness; it's the difference between catching most defects and catching almost none.

  2. Review individually before reviewing as a group. The 85% vs. 4% gap is large enough that it should shape process design directly. Use group discussion for architectural decisions, not initial defect-finding.

  3. Reason about runtime behavior, not just syntax. Given where AI-generated defects concentrate — race conditions, unhandled null states, resource leaks — a review pass that only checks whether code "looks right" will systematically miss the categories most likely to be wrong.

  4. Track review depth as code volume grows, not just review completion. A pull request marked "reviewed" isn't the same as one that received the kind of attention the SmartBear data associates with real defect discovery. As AI-generated volume rises, that gap is exactly where quality erodes without necessarily showing up on a dashboard.

Practical Checklist

  • Split pull requests to stay near the 200–400 line range associated with high defect discovery
  • Review individually first; reserve group sessions for design discussion, not defect-finding
  • Explicitly check for runtime-behavior issues (race conditions, null handling, resource leaks), not just syntax correctness
  • Track comment depth and review time, not just review completion, as pull request volume grows
  • Apply extra scrutiny to authentication and access-control logic in AI-generated scaffolding

Common Mistakes

  • Treating a completed review as equivalent to a thorough one
  • Relying on large group review sessions as the primary defect-finding mechanism
  • Letting pull request size creep past a few hundred lines "because it's mostly AI-generated anyway"
  • Reviewing AI-generated code for syntax correctness while skipping runtime reasoning entirely
  • Not tracking review comment depth as a leading indicator when PR volume increases

Trade-offs

AI-assisted review tools clearly help with large or unfamiliar pull requests — that's where developers in the field research reported the most value. But the same developers flagged false positives and insufficient context as real costs. The practical implication: AI review works best as a first pass that surfaces candidates for human attention, not a replacement for it — especially for the runtime-behavior categories (race conditions, resource leaks, access control) that syntax-level checks tend to miss.

What's your team's current split between AI-assisted and human review — and where has that balance broken down?

FAQ

  • Is group code review really less effective than individual review? The SmartBear/Cisco research — one of the largest studies on this — found a substantial gap in favor of individual review for defect discovery specifically. Group sessions still have value for design discussion and knowledge sharing.
  • Does AI-generated code have more bugs than human-written code? The research is still developing, but security-defect data suggests AI-generated code concentrates errors in specific categories — like broken access control — that look correct on the surface but fail at runtime.
  • Does using an AI review tool replace the need for human review? Early field studies suggest developers value AI assistance most for large or unfamiliar pull requests, but they also report false-positive concerns — suggesting AI review works best as a first pass rather than a replacement.
  • Why does review size matter so much? Concentration on detail-oriented review work measurably drops after about an hour, and defect discovery rates drop sharply once a review exceeds a few hundred lines. This isn't process preference — it's a measured effect.
  • What's the single biggest blind spot in current review practices? Based on the Octoverse data, it's volume outpacing depth — pull request counts rising faster than the signals (like review comments) that indicate real scrutiny is happening.

Final Thoughts

The research on code review effectiveness has been stable for years: keep reviews small, review individually before reviewing as a group, and don't mistake a completed review for a thorough one.

What's changed is the volume of code moving through that process, and the categories of defects — race conditions, resource leaks, access-control gaps — that AI-generated code tends to introduce even when it reads as correct.

None of the fixes here require new tooling. They require treating review capacity as a constraint that scales with code volume, not a fixed cost you can outrun by merging faster.

Sources

Top comments (0)