We tested Codzee against Greptile, CodeRabbit, and CodeAnt. Here's what we learned from six rounds of AI code review benchmarks, including the bugs we found and the weaknesses we still need to fix.

We Thought the Code Was Fine. Then the AI Found Something Else.

Sometimes, the most important bugs are the ones nobody thinks to look for.

We've all seen it happen.

The code looks fine. The tests pass. The pull request gets approved. Then, later, someone discovers a bug that should have been caught during review.

That's one of the problems we want Codzee to help solve.

But as a team building an AI code review tool, we had to ask ourselves an uncomfortable question:

Is Codzee actually good at reviewing code, or does it just look good in a demo?

We decided to find out.

We tested Codzee against Greptile, CodeRabbit, and CodeAnt using the same pull requests, deliberately planted bugs, and a predefined answer key.

We ran six rounds, making each round harder than the last. Round six taught us the most — and it was the first round Codzee won.

Here's what happened.

The experiment: making the bugs harder to find

For Round 6, we created a pull request involving a guest billing system. It handled invoices, payments, taxes, and cancellations.

Instead of planting obvious mistakes in individual functions, we designed bugs that required understanding how different modules interacted.

For example, one module could calculate an invoice correctly while another ignored it and recalculated the payment amount. Nothing crashed, but the customer could be charged a different amount from the one shown on the invoice.

Here was our setup:

Test detail	Description
Pull request	PR #7 in a test repository
Code size	192 lines across two modules
Planted bugs	13
Cross-file bugs	6
Domain	Guest billing: invoices, payments, tax, cancellations
Tools tested	Codzee, Greptile, CodeRabbit, CodeAnt
Scoring method	Findings retrieved through the GitHub REST API and manually matched against a predefined answer key

We wrote down the answer key before the review began. We also committed to publishing the weaknesses we found in Codzee.

The first surprise: the bugs hidden between files

We started by examining the six defects that required understanding how two parts of the application worked together.

The bug	Codzee	Greptile	CodeRabbit	CodeAnt
Payment ignores the invoice and recomputes the charge	✓	✓	✗	✗
Tax is never collected	✓	✓	✓	✗
Balance reads a key the invoice never writes	✓	✓	✓	✓
Due date compared against a datetime	✓	✓	✓	✓
A timedelta passed where a day count is expected	✓	✓	✓	✓
Invoice numbers restart, causing payment records to collide	✓	✓	✓	✓
Total found	6/6	6/6	5/6	4/6

Four defects were caught by every tool. The two payment-related problems were more revealing: only Codzee and Greptile caught both.

The most serious involved code that built an invoice and then ignored it, recalculating the charge from another module. This could cause a guest to be billed an amount that didn't match their invoice.

Codzee marked this finding as Critical. Greptile also caught it, while CodeRabbit identified the tax issue but missed the related recomputation problem.

That was precisely the kind of reasoning gap we wanted the experiment to expose.

The Round 6 scoreboard

Next, we looked at the complete results, including bugs outside the original answer key.

Evaluation metric	Codzee	Greptile	CodeRabbit	CodeAnt
Bugs found (out of 13)	8	7	6	6
Cross-file bugs (out of 6)	6	6	5	4
Total findings	18	9	6	7
Genuine bugs we didn't plant	3	2	0	1
Findings on the test file	3	0	0	0
False positives	0	0	0	0

Source: Codzee's internal Round 6 benchmark. Results represent one pull request, not a universal ranking of the tools.

Codzee found eight of the thirteen planted bugs. But the most interesting result wasn't the score itself.

It found three additional genuine issues that weren't part of our original answer key.

One involved a declined payment potentially being recorded as successful because the gateway receipt wasn't checked. Another involved a retry that could potentially charge a customer twice because the idempotency key changed.

Greptile also found two additional issues, while CodeAnt found one.

These findings reminded us that a predefined answer key is useful, but incomplete. Real applications can contain problems nobody thought to plant.

The test file nobody reviewed

We had also included three tests that asserted incorrect behavior.

A passing test can create confidence in the code, but what if the test itself expects the wrong result?

Codzee was the only tool to flag all three problematic tests in this round:

• A due-date test should expect 15 June, not 1 July.
• An unpaid invoice should show 22,300 pence outstanding, not zero.
• An assertion simply repeated the implementation's calculation instead of independently checking the expected result.

This was a meaningful finding for us. Code review shouldn't stop at the implementation if the tests themselves are reinforcing incorrect behavior.

It was also the fourth round out of six in which Codzee was the only tool to review the test file.

How did Codzee perform across all six rounds?

One round can tell an interesting story, but repeated testing helps reveal patterns.

Tool	Round 1	Round 2	Round 3	Round 4	Round 5	Round 6
Greptile	5/6	10/12	3	6/6	9/9	7/13
Codzee	4/6	9/12	2	3/6	8/9	8/13
CodeRabbit	4/6	8/12	1	5/6	8/9	6/13
CodeAnt	1/6	8/12	3	4/6	6/9	6/13

The rounds used different numbers of planted bugs. The Round 3 figures are reproduced as reported in the original article.

Our honest reading of these results:

• Greptile was the most consistent across the series and found 13 unplanted bugs over six rounds.
• Codzee recorded its first win in Round 6 and found its first three unplanted issues in that round.
• CodeRabbit produced fewer findings and offered particularly clear fix guidance on the tax issue in Round 6.
• CodeAnt identified several defects, but struggled more with cross-file issues in this round.

These are observations from our tests, not permanent rankings of the products.

What we got wrong — and what we still need to fix

We promised ourselves that we would publish the bad news, too.

Three planted bugs defeated all four tools: a mutable default argument, a function-local import that obscured the recomputation path, and a rounding issue that could silently lose a penny on every invoice.

We also found weaknesses in Codzee itself.

Of its 18 findings, only 10 appeared inline on the code diff. The other eight were in the summary, making them harder to connect directly to the relevant code.

Our inline placement rates across the six rounds were 50%, 100%, 0%, 75%, 25%, and 56%. That inconsistency is something we need to address.

There was another problem: despite identifying a Critical payment defect, Codzee's merge-check conclusion remained neutral. A serious finding should be reflected clearly in the signal developers use when deciding whether to merge.

The good news is that repeated testing also helped us identify improvements in severity calibration. In Round 6, Codzee assigned the Critical rating to the payment defect and graded the problematic tests as Major.

But there is still work to do.

What we learned

This experiment reinforced a few lessons we think matter to anyone choosing an AI code review tool.

1. Context matters. Bugs can hide in the relationship between modules, even when individual functions look reasonable.

2. Unexpected findings matter. A tool should be evaluated on legitimate issues beyond the bugs deliberately planted for testing.

3. More comments don't necessarily mean better reviews. Greptile found six of six cross-file bugs with nine findings. Codzee produced twice as many findings and found one additional planted bug.

4. Tests deserve scrutiny. Three incorrect tests were present, and only Codzee flagged them in this round.

5. Placement is part of quality. A finding is more useful when developers can see exactly where the problem occurs.

6. AI review doesn't replace linters. All four tools missed defects that mechanical checks may be better equipped to catch.

We also didn't test everything. Security depth, CI/CD integration, customization, scale, and team adoption were outside the scope of this experiment.

So, should you trust Codzee?

We don't think trust should come from a marketing claim — including ours.

In Round 6, Codzee found eight of thirteen planted bugs, matched Greptile on all six cross-file defects, identified three additional genuine issues, and flagged three problematic tests.

Those results encouraged us. The missed bugs, inconsistent inline placement, and neutral merge signal gave us clear areas to improve.

Greptile remained a strong competitor, and every tool missed something.

We build Codzee, so we're not neutral observers. That's exactly why we believe the methodology, results, and limitations should be visible.

Our goal isn't to generate more comments. It's to help developers find meaningful problems before they reach production.

If you're evaluating AI code review tools, don't take our word for it. Test them on your own pull requests. Hide a bug between two files, include a test that asserts the wrong behavior, and see what each tool catches.

You can explore Codzee at codzee.io and read the

We'd love to hear from other developers: what's the kind of bug your AI reviewer consistently misses?