Deleting code often feels safer than introducing new functionality. A developer removes unused helpers, cleans up old tests, and simplifies a module. The diff turns red, the test suite appears green, and the pull request looks ready to merge.
But what if the code was only unused because its tests were deleted in the same commit?
That question became the focus of our latest AI code review benchmark at Codzee. We evaluated Codzee alongside Greptile, CodeRabbit, and CodeAnt against a controlled pull request containing nine planted defects.
The results exposed a shared blind spot: all four tools missed the same five defects.
This is not simply a story about which tool ranked first. It is about why deletion-only pull requests can challenge AI reviewers—and what engineering teams should expect from repository-aware code review.
The experiment: 152 deleted lines and nine defects
Our developer and R&D team tested all four tools on GitHub pull request #14, commit 2dc904f.
The change contained:
- 152 deletions
- 2 insertions
- 8 files
- 9 planted defects
- 8 genuinely correct deletions used to test false-positive behaviour
The pull request was deliberately constructed to resemble a routine cleanup. Most of the removed code was genuinely unreferenced, but several deletions created production failures, removed important test coverage, or broke relationships between components.
We evaluated the tools against an answer key prepared before the branch was pushed. Findings were collected from GitHub inline review comments, pull-request comments, and relevant summary and merge-risk text.
The objective was to measure whether each tool identified the known defects without raising incorrect objections.
Round 13 results
Codzee ranked first in this round, detecting four of the nine planted defects. It also found four genuine pre-existing bugs outside the changed files.
Every tool avoided false positives on the eight verified correct deletions.
However, perfect precision did not mean the pull request had been reviewed safely. The tools were quiet rather than noisy, and five planted defects escaped every reviewer.
Three reasons deletion-only reviews can fail
1. The deleted-test problem
Three functions were removed on the grounds that nothing called them:
rates.deposit_duetransfers.round_to_poundpayments.amount_due
The problem was that their only callers were tests—and those tests were removed in the same commit.
After the deletion, the functions appeared unused. Before the deletion, they were not.
This creates a circular justification: remove the evidence of usage, then use the absence of that evidence to justify removing the implementation.
All four tools missed these three defects.
The lesson is straightforward. When reviewing deletions, reference analysis should not rely exclusively on the final repository state. Reviewers need to examine the base commit and determine whether the same change removes the only evidence that a symbol is used.
2. The cross-language contract problem
The pull request deleted points.can_redeem, while web/src/points.ts continued exporting canRedeem.
The server-side implementation disappeared, but the client-side capability remained.
All four tools missed this contract break.
The two files explicitly documented that they implemented the same set of rules. Yet the deletion-shaped change did not trigger the cross-language check needed to detect the inconsistency.
This illustrates why code review should consider relationships between components, not just the lines that changed.
3. The knock-on effect of removing a function
The pull request removed payments.refund_for_cancellation. That function was the only production caller of booking/cancellation.py.
As a result, the module and its 75-line test suite became unreachable.
None of the four tools identified this orphaned module.
A deletion can therefore create a new cleanup opportunity—or leave behind code that is no longer reachable. A useful reviewer should examine both sides of the change: what becomes broken and what becomes stranded.
What Codzee found beyond the changed files
One of the more interesting results was Codzee's detection of four genuine bugs in untouched code. Each was correctly labelled as predating the pull request.
A fare calculation that was 1,000 times too high
transfers.fare multiplied haversine distances in metres by a rate expressed per kilometre. A 25 km saloon journey was priced at approximately £45,000 instead of £49.50.
A vehicle dispatch radius of just 25 metres
transfers.nearby_vehicles passed SEARCH_RADIUS_KM, with a value of 25, into geo.within, which expected metres. Vehicle discovery was therefore restricted to a tiny radius.
An airport surcharge calculated in the wrong unit
The surcharge configuration was documented in pounds but added to a total expressed in pence. An £8 airport surcharge became 8p.
An unpaid invoice incorrectly reporting zero
The outstanding-balance calculation read amount, while invoice lines used amount_pence. The function returned zero, and a test incorrectly asserted that result was correct.
These were not defects planted for the round. They were real, pre-existing bugs that had survived earlier review rounds.
Finding issues outside the diff can be valuable, provided reviewers clearly distinguish newly introduced defects from existing problems.
The benchmark across eight rounds
Round 13 was part of a longer evaluation. Across rounds 6–13, Codzee led or tied in eight rounds, Greptile in six, CodeRabbit in one, and CodeAnt in one.
These scores represent our controlled benchmark, not universal detection rates or proof that one tool will outperform every alternative in every repository.
What should AI code reviewers do differently?
Our experiment suggests three practical improvements.
1. Resolve references against the base commit. When a symbol is deleted, check its callers before the change. If its only caller is also removed, flag the relationship.
2. Recompute reachability after deletions. Identify modules that lose all production importers and explain what the change leaves behind.
3. Treat deleted tests as a reason for caution. A passing test suite is weaker evidence when the pull request removes the tests that previously exercised the relevant behaviour.
Codzee also has work to do. It missed the broken _TIER_CACHE test that Greptile and CodeRabbit detected, failed to generalise the deleted-test pattern to all three circular-justification defects, and missed the orphaned module.
Those misses matter because a benchmark is useful only when it exposes the gaps of the tool being tested, too.
Final takeaway
Deleting code is not automatically a reduction in risk.
A small, clean-looking diff can remove a production dependency, erase the test that proves a function is needed, or break a contract in another language.
Our Round 13 experiment showed that all four tested AI code review tools struggled with the same five defects. Codzee ranked first in this round, but the broader lesson is about the limitations of reviewing code only in its final state.
The next step for AI code review is not simply generating more comments. It is understanding what a change removes, what it breaks, and what it leaves behind.
Methodology: All four tools reviewed commit 2dc904f in pull request #14. Nine planted defects were scored against a prewritten answer key. Eight additional deletions were verified as genuinely unreferenced to test false-positive behaviour. Relevant inline comments, pull-request comments, and summary or merge-risk statements were considered. One defect involving _TIER_CACHE was introduced accidentally rather than deliberately planted. Qodo did not participate because its trial had ended.
Top comments (0)