DEV Community

Codzee.io
Codzee.io

Posted on

We Put 4 AI Code Review Tools Against the Same Pull Request. Codzee Found 25 Out of 26 Bugs.

An engineering case study comparing Codzee, Greptile, CodeRabbit, and CodeAnt across 435 lines of code, 6 files, and 26 seeded defects.

Every developer knows that writing code is only one part of software development.

The real challenge often begins when a pull request is opened.

A function might work perfectly in isolation but introduce unexpected behavior elsewhere. A database migration might contain valid SQL statements but create a production risk. A security vulnerability might exist even when the code looks completely normal.

This is why AI code review tools are becoming an important part of modern engineering workflows.

But an important question remains:

How well can AI code review tools actually understand a real codebase and identify defects that developers might overlook?

We decided to test this through a hands-on experiment involving four AI code review tools:

  • Codzee
  • Greptile
  • CodeRabbit
  • CodeAnt

All four tools received the same pull request, the same code changes, and the same testing conditions.

The results placed Codzee at the top.

Let's break down what happened.

1. Setting Up the Experiment

We selected a challenging pull request containing 435 insertions across six files.

Instead of testing tools on simple coding examples, we introduced 26 deliberately seeded defects across different engineering scenarios.

The experiment covered:

Area What We Tested
Python logic Incorrect calculations and application behavior
Data exports Unsafe file handling and CSV vulnerabilities
Code security SQL injection, path traversal, and PII exposure
Revenue calculations VAT errors, refunds, and incorrect sorting
Database migrations Backfill ordering and destructive operations
Application reliability Runtime errors and operational risks

Each tool reviewed the same commit (e790b56) on pull request #16.

We prepared the answer key before pushing the branch, allowing us to evaluate findings against known defects rather than subjective impressions.

Our primary measurement was simple: how many of the 26 seeded defects could each tool correctly identify?

2. Benchmark Results: Codzee Leads the Comparison

Here are the results.

AI Code Review Tool Defects Detected Detection Rate
Codzee 25/26 96.2%
Greptile 19/26 73.1%
CodeRabbit 16/26 61.5%
CodeAnt 11/26 42.3%

Codzee achieved the highest detection score, identifying six more seeded defects than Greptile, nine more than CodeRabbit, and fourteen more than CodeAnt.

This was not just a difference in comment volume.

Codzee demonstrated stronger performance in several areas that require understanding the relationship between different parts of an application.

Let's examine some examples.

3. Where Codzee Outperformed the Competition

Case Study A: Detecting a Hidden CSV Security Vulnerability

One of the most interesting findings involved CSV formula injection.

The codebase contained a helper function named format_cell, which could have helped prevent unsafe CSV values.

However, the actual report-generation function never called this helper. Instead, it passed raw rows directly to csv.writer.

This created a vulnerability involving cells beginning with characters such as = or +.

Codzee identified both sides of the problem:

  • The security vulnerability in the export function.
  • The unused helper function that was effectively the missing fix.

This required connecting two separate pieces of code rather than simply identifying a suspicious line.

None of the other three tools identified this particular finding.

Result: Codzee delivered a unique finding in this experiment.

Case Study B: Understanding Database Migration Risks

Database migrations can be particularly dangerous because an incorrect sequence of operations can affect thousands of existing records.

In our experiment, the migration contained several problems, including:

  • Incorrect backfill ordering.
  • Irreversible column deletion.
  • Destructive operations on repeated execution.
  • Database-wide timeout configuration.
  • Unsafe permission assignments.

Codzee identified 7 out of 8 seeded migration defects.

The next-best result was CodeRabbit with 6 out of 8, followed by Greptile with 4 out of 8 and CodeAnt with 2 out of 8.

One particularly interesting finding involved a backfill using an inner join.

Codzee recognized that refunds without a matching invoice could retain a NULL booking ID and lose their only remaining relationship when the original column was dropped.

This is the kind of reasoning that matters in production engineering.

Case Study C: Checking Business Logic Against Documentation

Code reviews should not only check whether code executes correctly.

They should also verify whether the implementation follows its documented requirements.

In exports.py, the docstring explicitly prohibited exporting guest contact information, but the implementation included guest names and email addresses.

Codzee identified this contradiction.

It also detected issues involving incorrect export cleanup behavior, path traversal, and unclosed file handles.

On this file, Codzee detected all six seeded defects.

Greptile detected four, while CodeRabbit and CodeAnt detected one each.

This demonstrated a meaningful advantage in checking application behavior against its stated contract.

4. The Competitor Comparison: Where Each Tool Stood

Every tool demonstrated different capabilities, but the overall results were clear.

Codzee — 1st place (25/26)

Codzee delivered the strongest overall detection performance.

Its standout capabilities in this experiment included:

  • Highest total defect detection.
  • Best performance on the exports file.
  • Strongest performance on database migration defects.
  • Three unique findings among the seeded defects.
  • Four appropriately identified critical issues.

Greptile — 2nd place (19/26)

Greptile delivered the highest inline-comment count and achieved perfect detection on the revenue file.

However, it detected fewer migration and export defects than Codzee.

CodeRabbit — 3rd place (16/26)

CodeRabbit performed particularly well on the database migration, identifying six of eight seeded defects.

Its weaker area was the exports file, where it identified only one of six.

CodeAnt — 4th place (11/26)

CodeAnt produced 19 inline comments but detected only 11 seeded defects.

This illustrates why comment volume alone should not be treated as a reliable measurement of review quality.

The important distinction is between generating more comments and identifying more actual problems.

5. An Important Engineering Lesson: Detection Is Only Half the Experience

There was another interesting observation.

Codzee identified 33 total findings, but only 10 were delivered as inline comments.

The reason was a fixed inline-comment limit.

On a large pull request, this affected the developer experience. For example, Codzee detected five defects in occupancy.py, but none appeared as inline comments because the comment budget had already been reached.

Greptile, CodeRabbit, and CodeAnt presented all their findings inline in this round.

This is an important improvement opportunity for Codzee.

A powerful AI code review engine should not only detect problems but also make those problems accessible to developers where they work.

Improving how findings scale with pull request size is therefore a key next step.

6. Final Verdict: What Did We Learn?

The experiment demonstrated that AI code review quality cannot be measured by comment count alone.

We need to evaluate tools based on their ability to:

  1. Detect actual software defects.
  2. Understand application context.
  3. Identify code security vulnerabilities.
  4. Follow dependencies across multiple functions.
  5. Reason about operational database risks.
  6. Explain why a particular implementation is dangerous.

In this experiment, Codzee achieved 25 out of 26 seeded defects, outperforming Greptile, CodeRabbit, and CodeAnt on the overall detection score.

Its strongest results came from security analysis, multi-step reasoning, business logic validation, and operational SQL review.

There are still improvements to make, particularly around inline-comment delivery and whole-migration transaction analysis.

But the results provide a strong indication of what Codzee can achieve when reviewing complex pull requests.

Our takeaway: The future of AI code review isn't about generating more comments. It's about helping developers understand more problems before those problems reach production.

If you're building or maintaining software, that's the kind of engineering intelligence worth looking for.


About Codzee

Codzee is an AI code review tool designed to help engineering teams improve pull request reviews, identify potential code security vulnerabilities, and catch problems earlier in the development lifecycle.


Experiment methodology: All four tools reviewed the same commit, e790b56, on pull request #16. The benchmark contained 26 seeded defects across six files and 435 insertions. Findings were evaluated against an answer key prepared before the branch was pushed. Five test-file defects were excluded. Additional unplanted findings were discussed separately and were not included in the scored results. This is a controlled benchmark round, not a universal ranking of every tool or codebase.

Top comments (0)