We Tested 4 AI Code Review Tools Against 18 Security and Logic Defects

Imagine opening a pull request that claims its tokens are secure, sessions expire correctly, and permissions fail safely.

The description sounds reassuring. The implementation is vulnerable.

Would your AI code review tool catch the difference?

We tested four AI code review tools against exactly this scenario. The results revealed more than which tool detected the most defects. They also exposed gaps in test analysis, review visibility, and GitHub merge protection.

The benchmark setup

For Round Ten of our AI code review benchmark, we compared:

• Codzee
• Greptile
• CodeRabbit
• CodeAnt

We created a deliberately vulnerable sandbox pull request containing 18 planted security and logic defects across three source files.

The pull request description made four security claims that the implementation contradicted. We also planted six defects in the test file and evaluated those separately.

All four tools reviewed the same commit. We collected findings from GitHub's REST API and scored them against an answer key prepared before the branch was pushed.

The results

| AI code review tool | Defects detected |

Codzee detected the most planted defects in this round. But the breakdown tells us more than the overall score.

The six test defects were scored separately: Codzee found all six, while the other three tools found none.

The security claims looked convincing. The code wasn't.

One claim in the pull request description said:

Tokens are signed and verified with HMAC-SHA256.

But the implementation used:

Buffer.from(payload).toString("base64")

Base64 is an encoding, not a cryptographic signature. Without proper signing and verification, an attacker could construct a token containing an administrator role.

Another claim said expired sessions were rejected. The implementation compared expiresAt with issuedAt instead of checking the current time. The expiration check therefore failed to enforce session expiry.

We also found a permission fallback that granted read and write access to unknown roles, even though the description promised a deny-by-default model.

These were not cosmetic code review comments. They were direct contradictions between the stated security requirements and the implementation.

A surprising result: the description didn't fool the tools

All four tools detected at least three of the four defects that the pull request description claimed were already handled.

Three competing tools explicitly connected their findings to the author's claims. Codzee detected all four defects but did not explicitly point out that the description itself was inaccurate.

That distinction matters during a real pull request review.

Compare these two findings:

• "The secret comparison is not timing-safe."
• "The secret comparison is not timing-safe, despite the security property claimed in the pull request."

The second gives the engineering team more context. It explains not only what is wrong, but why the implementation fails to meet its own stated requirements.

None of the four tools explicitly recommended correcting the misleading description before merging.

The defects that weren't mentioned

The largest difference appeared in the ten additional security defects that the author never mentioned.

Codzee detected all ten. Greptile detected seven, CodeRabbit five, and CodeAnt four.

The defects included:

• A missing ownership check that allowed access to another user's booking.
• A permission check that treated a role containing admin as an administrator role.
• A permission cache that was never invalidated after privilege changes.
• Session IDs generated using Math.random().
• A hardcoded fallback secret.
• Login error messages that revealed whether a username existed.

Codzee was the only tool to detect the final three issues.

This is an important dimension when evaluating AI code review tools: can they identify problems beyond the ones suggested by the pull request description?

Why test analysis matters

Codzee also detected all six planted test defects.

Two were especially concerning. One test expected a session to be created from an x-user-id header. Another expected the application to return a booking belonging to a different user.

These tests treated vulnerable behavior as correct.

A test suite can provide false confidence when it validates the wrong behavior. Reviewing application code without examining its tests can leave that problem undiscovered.

This result is promising, but it comes from six deliberately planted test defects in one sandbox. It should not be interpreted as proof that Codzee will find every test-related issue in production.

The result wasn't perfect, and that matters

Codzee missed one defect: logout() did not invalidate the token. A session that appeared to be logged out could remain valid.

The experiment also revealed problems in Codzee's own review workflow.

Of its 29 findings, only 10 appeared as inline comments. The remaining 19 were placed in a collapsed block, making some important findings harder to notice.

More seriously, Codzee reported ten Critical findings but returned a neutral GitHub check conclusion. Its legacy commit status simultaneously reported success.

This creates a gap between detecting a vulnerability and enforcing a security policy. If a team expects Critical findings to block merges, a neutral check does not meet that expectation.

We identified three priorities:

1. Make Critical findings enforceable through configurable GitHub check behavior.
2. Improve the visibility of findings, especially those that appear outside inline comments.
3. Include the pull request title and description in the review context so the tool can explicitly flag misleading security claims.

What this benchmark tells engineering teams

This experiment involved one deliberately constructed sandbox pull request, not thousands of production changes. It cannot establish that one tool will always outperform another.

It does, however, suggest that evaluating AI code review requires more than comparing a single score.

When assessing a tool, ask:

• Detection: Can it find security defects that the author never mentioned?
• Context: Does it compare the implementation with the stated requirements?
• Test quality: Can it identify tests that validate insecure behavior?
• Visibility: Are important findings easy for developers to discover?
• Enforcement: Can critical findings influence the merge decision?

These dimensions determine whether review findings can translate into better engineering decisions.

What we're improving at Codzee

Codzee scored 17/18 in this round, but the benchmark also showed where the product needs work.

Our priorities are to improve critical-finding enforcement, make review comments more visible, and use pull request descriptions more effectively.

We are sharing these gaps alongside the results because a useful engineering benchmark should reveal weaknesses, not just produce a leaderboard.

You can learn more about Codzee at https://www.codzee.io.

A question for developers and engineering teams: When choosing an AI code review tool, which matters most to you: finding more defects, explaining them with better context, or ensuring critical vulnerabilities cannot pass through the merge workflow?