We introduced 11 defects into a pull request and compared Codzee, Greptile, CodeAnt, and CodeRabbit. The results revealed some surprising blind spots.
A refactor can make code cleaner while making a system less reliable.
A function gets simplified. A few lines disappear. A developer fixes a bug and moves on.
Then, somewhere else in the application, something breaks.
This is one of the challenges of code review: the reviewer needs to understand not only what the code does now, but also what changed and whether the previous behaviour still works.
We wanted to see how well AI code review tools handle that challenge.
So we tested four tools against the same controlled pull request: Codzee, Greptile, CodeAnt, and CodeRabbit.
Here's what happened.
The setup
For this benchmark round, we evaluated a pull request that modified existing code rather than introducing entirely new files.
The change affected two existing files:
- 46 insertions
- 34 deletions
We planted 11 defects:
- Three fixes claimed in the pull request description that were still incomplete or incorrect.
- Eight regressions introduced during refactoring.
We also included two deliberately correct refactors to test for false positives.
Each tool reviewed the same commit. We collected the findings through the GitHub REST API and scored them against an answer key prepared before the branch was pushed.
We also examined deleted tests, severity classifications, inline comments, and whether each tool understood the previous implementation.
The goal wasn't to count comments. It was to identify meaningful defects.
The results
Here is the scorecard for this round.
Codzee: 9/11 defects detected (81.8%)
Greptile: 8/11 defects detected (72.7%)
CodeAnt: 6/11 defects detected (54.5%)
CodeRabbit: 5/11 defects detected (45.5%)
Codzee finished first, one defect ahead of Greptile.
But the score alone doesn't tell the whole story.
Codzee and Greptile each detected seven of the eight regressions. Codzee also detected two of the three claimed fixes that remained broken, compared with one for Greptile.
Let's look at the actual bugs.
1. The negative-input bug nobody caught
The pull request changed a loyalty-points redemption function.
The intention was to prevent customers from redeeming more points than their account balance. The new guard checked whether the requested points exceeded the available balance.
Seems reasonable, right?
Now consider this call:
redeem(acct, -100)
The new condition did not reject a negative redemption amount. Instead, the operation could credit points to the account.
All four tools missed this defect.
This is a useful reminder that reviewing the intended use case isn't enough. A condition can look correct while still failing for unexpected inputs.
When reviewing validation logic, ask:
- What happens with negative values?
- What happens at the exact boundary?
- Can an invalid input bypass the guard?
- Does the operation behave correctly for every supported input?
AI can help explore these cases, but the result shows why human judgement and tests remain essential.
2. The refactor that broke the tier lookup
Another regression involved a sorting operation that disrupted a loyalty-tier lookup.
The result was serious: every positive balance could be assigned to the bronze tier.
All four tools caught this defect.
Other regressions were less obvious.
One function changed from returning a list to returning a generator. Another converted an integer-based monetary value into a floating-point number. A negative-spend guard disappeared, and a function gained an unexpected side effect.
These aren't necessarily obvious by reading the final code alone.
The reviewer needs to compare the old implementation with the new one and understand which contracts must remain intact.
For example, changing a list into a generator can break callers that rely on len() or iterate over the result more than once.
The refactor might look elegant, but elegance doesn't guarantee compatibility.
3. A deleted test left a hole in coverage
The pull request also removed tests.
One deleted assertion was incorrect, so removing it was defensible. The problem was that no replacement test covered the intended tier-handling behaviour.
Codzee was the only tool to flag the deletion and explain the missing coverage.
That is more useful than simply saying, "A test was removed."
A good review should ask what the test protected and whether that behaviour is still covered.
Tests preserve knowledge about how software should behave. When a test disappears without its intent being preserved, the codebase can become more fragile—even if the refactored code looks cleaner.
4. The bug that escaped every reviewer
There was another important miss: a mutable module-level list was exposed as a default argument.
The benchmark identified this as a pattern covered by Ruff rule B006, yet none of the four tools flagged it.
This matters because static analysis can catch some problems that an AI reviewer overlooks.
The practical lesson isn't to choose between AI review and linting. It's to use complementary safeguards.
For Python projects, that means combining code review with automated tests, linting, type checking where appropriate, and a clear review process.
No single tool should be treated as a guarantee of correctness.
What we learned beyond the score
The benchmark revealed three broader patterns.
Review volume dropped by 39%.
Compared with the previous round, every tool produced fewer findings on the refactor. The research suggests that reviewing modifications to existing files can be more challenging than reviewing newly added code.
Reviewing the diff matters.
Several defects only became clear when the new implementation was compared with the previous behaviour. A reviewer needs to understand what changed, not merely inspect the final code.
Finding a defect isn't the end of the job.
A critical issue should be communicated clearly, and the GitHub check should not give developers a misleading impression of whether a pull request is safe to merge.
We found an issue in Codzee itself here: a critical finding could coexist with a successful check. That is an improvement we identified through the benchmark.
How did the tools perform across 11 rounds?
This experiment was part of a broader series covering scenarios such as authentication, payments, asynchronous behaviour, test quality, and Python/TypeScript consistency.
According to our team's running tally, Codzee and Greptile had each led or tied in six rounds. CodeAnt and CodeRabbit had each led or tied in one.
Codzee had won the last five rounds outright.
These figures are the benchmark team's reported tally, not a universal performance ranking. The scenarios varied across rounds, so the individual results and their context matter more than the leaderboard alone.
What should developers take away?
If you're evaluating AI code review tools, don't test them only on obvious bugs.
Include cases where:
- A claimed fix doesn't actually work.
- A refactor changes a return type or side effect.
- An input violates an assumption.
- A test disappears during cleanup.
- A regression becomes visible only when compared with the previous implementation.
Then inspect what each tool catches, what it misses, and whether its findings help you make a better engineering decision.
At Codzee, we're using these benchmark rounds to improve regression detection, test-file review, severity handling, and the way our tool communicates findings through GitHub.
We don't consider the work finished just because we led this round. The defects we missed are just as important to our next iteration.
If you're exploring AI code review for your team, take a look at Codzee.
The goal isn't to generate more review comments. It's to catch the problems that matter before they reach production.
Methodology: Round Eleven evaluated the head commit of GitHub pull request #12, with 46 insertions and 34 deletions across two existing files. Eleven planted defects were scored against an answer key prepared before the branch was pushed. Results reflect this controlled benchmark round and should not be interpreted as a universal comparison of the products.
Top comments (0)