We tested Codzee, Greptile, CodeRabbit, and CodeAnt against the same pull request. Here's what they caught, what they missed, and what developers can learn from the results.
AI code review tools are becoming a regular part of modern development workflows.
They promise faster reviews, fewer bugs, better security, and less manual effort.
But how well do they actually understand code?
To find out, our engineering team at Codzee conducted a comparative experiment involving four AI code review tools:
- Codzee
- Greptile
- CodeRabbit
- CodeAnt
Instead of comparing features, we gave all four tools the same pull request and evaluated their ability to identify real engineering defects.
The experiment focused on a major Axios upgrade, from version 0.21.1 to 1.7.7.
And the results revealed some interesting differences.
1. Setting up the experiment
The pull request included:
- 76 insertions and 3 deletions.
- One new file.
- One modified file.
- An updated
package.json. - Eleven scored defects.
- Three deliberately correct changes to test false positives.
We focused on four areas:
- Dependency compatibility.
- Security vulnerabilities.
- Request and retry logic.
- Custom helper functions.
All four tools reviewed the same changes. We evaluated their findings against a manually prepared defect scorecard using GitHub review activity.
2. The results
Here is the final scorecard.
| AI reviewer | Defects detected |
|---|---|
| Codzee | 10/11 |
| Greptile | 7/11 |
| CodeRabbit | 7/11 |
| CodeAnt | 5/11 |
Codzee identified ten of the eleven scored defects, finishing first in this round.
But detection scores alone don't explain why the results were different.
Let's examine some of the actual problems.
3. Bug #1: Authentication tokens exposed through error logging
Consider this scenario.
A request interceptor attaches an authentication token:
axios.interceptors.request.use((config) => {
config.headers.Authorization = `Bearer ${token}`;
return config;
});
Now imagine another interceptor logs the request configuration when an API request fails:
axios.interceptors.response.use(
response => response,
error => {
console.log(error.config);
return Promise.reject(error);
}
);
What happens?
The request configuration contains the Authorization header.
When the request fails, the configuration gets logged, potentially exposing the bearer token.
This is a security issue because application logs may be collected and accessed through external monitoring systems.
How did the tools perform?
| Tool | Result |
|---|---|
| Codzee | Detected; classified Critical |
| CodeRabbit | Detected; classified Medium |
| Greptile | Missed |
| CodeAnt | Missed |
Codzee and CodeRabbit both connected the two interceptors and identified the credential exposure.
The important part was understanding the relationship between the request interceptor and the error-handling interceptor.
4. Bug #2: An unused idempotency check in retry logic
Retry mechanisms are useful, but not every request should automatically be repeated.
Consider a payment API.
If an application retries a payment request without proper idempotency protection, the same operation could potentially execute more than once.
In our test, the implementation defined an isIdempotent guard but never actually used it in the retry mechanism.
Codzee identified this problem.
It also connected it to another configuration issue:
validateStatus: () => true
This configuration effectively treats all HTTP response statuses as successful from Axios's status-validation perspective.
As a result, the retry interceptor would only see network failures rather than HTTP status failures.
Codzee identified both issues and explained how they interacted.
CodeRabbit also identified both. Greptile and CodeAnt missed them.
5. Bug #3: Custom helper functions that don't behave as expected
Not every bug comes from complicated architecture.
Sometimes the problem is a small utility function that developers use throughout an application.
Our experiment included two such issues.
The isEmpty function
const isEmpty = value => !value?.length;
This implementation can incorrectly report objects, numbers, and Maps as empty because they don't have a length property.
For example:
isEmpty({ name: "Developer" }); // true
The object contains data, but the function reports it as empty.
The copyPayload function
This helper used a shallow spread:
const copyPayload = payload => ({ ...payload });
That creates an unexpected result when the input is an array:
copyPayload([1, 2, 3]);
// { 0: 1, 1: 2, 2: 3 }
The array becomes an object with numeric keys.
Codzee and Greptile identified both problems.
CodeRabbit missed both, while CodeAnt identified the array-handling issue.
6. Dependency upgrades: More than changing a version number
Major dependency upgrades deserve careful code review.
In our experiment, the Axios upgrade introduced several adaptation issues.
These included:
- Incorrect HTTP status validation.
- Using
response.json()instead of Axios'sresponse.data. - Incorrect timeout interpretation.
- Unguarded access to
error.response.data. - Retry logic that didn't properly use its idempotency guard.
Codzee identified five of the six issues in the Axios adaptation category.
Greptile identified three, CodeRabbit five, and CodeAnt three.
This demonstrates why dependency-related code review should consider actual library behaviour, not just whether the package version has changed.
7. What did the comparison reveal?
The experiment highlighted different strengths across the four tools.
| Area | Codzee | Greptile | CodeRabbit | CodeAnt |
|---|---|---|---|---|
| Total scored defects | 10 | 7 | 7 | 5 |
| Credential exposure | Yes | No | Yes | No |
| Retry-related defects | Yes | No | Yes | No |
| Both helper-function defects | Yes | Yes | No | No |
| Correct-change objections | 0 | 0 | 0 | 0 |
All four tools avoided objections to the three deliberately correct changes.
One additional observation: all four identified a real query-string encoding vulnerability in the custom serializer. This was an unplanted defect included in the scored results.
8. Key takeaways for developers
This experiment highlights a few important lessons:
Understand the entire execution flow. A security issue can emerge from multiple functions working together.
Don't blindly trust retry mechanisms. Consider idempotency before repeating operations.
Test utility functions with different data types. A function that works for strings may behave incorrectly with objects, arrays, or Maps.
Treat major dependency upgrades carefully. Review migration behaviour, API changes, and configuration differences.
Measure review quality, not comment quantity. More comments don't necessarily mean better code reviews.
Final results
This was Round 14 of our engineering team's comparative testing series.
Across fourteen rounds, Codzee led or tied for first in nine rounds and achieved eight consecutive wins in the tested sequence.
In this Axios upgrade experiment, Codzee identified 10 out of 11 scored defects, compared with seven for Greptile, seven for CodeRabbit, and five for CodeAnt.
The experiment demonstrates why evaluating AI code review tools through actual engineering problems can reveal much more than feature comparisons.
The real question isn't how many comments an AI reviewer generates.
It's how many meaningful problems it helps developers catch before those problems reach production.
Research methodology: All four tools reviewed the same Axios upgrade pull request. Eleven defects were scored using a manually prepared checklist and GitHub review activity. Three deliberately correct changes were included to evaluate false positives. Results represent this specific benchmark, not universal performance across all repositories.
Product: Codzee – AI Code Review

Top comments (0)