DEV Community: Codzee.io

AI Code Review Pricing... What “Unlimited” Really Means...

Codzee.io — Thu, 15 Oct 2026 10:23:24 +0000

AI code review tools can look affordable when you're comparing subscription prices. But what happens when your team starts reviewing hundreds of pull requests every month?

The answer depends on how each tool measures usage, what happens when you reach its limits, and whether the feedback is actually useful.

We examined the pricing models of several AI code review tools and analyzed 203 Greptile findings across 20 pull requests. Here's what the research revealed.

1. Every tool measures usage differently

AI code review tools don't all use the same billing model.

Codzee: Uses credits based on pull request size. Its Team plan adds 100 credits per developer to a shared team pool.
Greptile: Provides 50 credits per seat. Base reviews cost 1 credit, Plus reviews cost 3, and Apex reviews cost 10. Additional credits cost $1 each.
CodeRabbit: Publishes hourly review limits of 5, 8, or 10 reviews depending on the paid plan. Independent analyses also report adaptive throttling and per-file overage charges, but these details require confirmation from CodeRabbit.
Bito: Research identified a limit of 5,000 lines per seat per month, followed by a charge of $5 per additional 1,000 lines.

The important point is that a subscription price doesn't tell you the full cost. You need to understand what counts as usage and what happens when you exceed the allowance.

2. What happens with 250 pull requests a month?

Consider a modeled team of 10 developers reviewing 250 pull requests each month.

The supplied research produced these estimates:

Scenario	Estimated monthly cost
CodeRabbit Essentials, 5 files per PR	$553
CodeRabbit Essentials, 8 files per PR	$740
CodeRabbit Essentials, 15 files per PR	$1,178
Codzee Team, 250 medium-sized PRs	$240

These figures are based on the research's pricing assumptions, not actual customer invoices. The CodeRabbit estimates include reported overage charges that could not be confirmed through official rate-limit documentation.

For Codzee, the scenario assumes that 250 medium-sized pull requests consume 500 of the team's 1,000 pooled credits.

Why does the shared pool matter?

Imagine one developer handles most of the team's pull requests while others contribute less frequently. With pooled credits, available capacity can be used across the team instead of remaining tied to individual allowances.

That doesn't mean credits are unlimited. It means teams should evaluate how the allowance matches their actual workload.

3. We examined 203 Greptile findings

Pricing is only one part of evaluating an AI code reviewer. The quality of its findings matters just as much.

For this part of the research, we retrieved Greptile's inline comments through the GitHub API across 20 pull requests in the Johns223/John_Test_1 benchmark repository.

The 203 findings were distributed as follows:

Priority	Findings	Examples from the sample
P0	2	Authentication weaknesses involving forgeable session tokens and unauthorized privileged access
P1	154	SQL injection, unsafe deserialization, path traversal, and monetary calculation errors
P2	47	Test-quality issues
P3–P5	0 observed	Not present in the examined sample

Twenty-six findings also carried a separate Security badge.

One detail is worth noting: the P-labels represented priority or severity, not confidence. Greptile expressed confidence separately in its finding text.

These results describe one repository and 20 pull requests. They do not establish Greptile's overall detection accuracy, and they are not a direct comparison with Codzee. We cannot use this sample to claim that one tool catches more bugs than another.

4. What should developers evaluate?

Before choosing an AI code review tool, consider three practical questions:

How is usage measured? Pull request size, review depth, lines of code, and review frequency can produce very different costs.
What happens at the limit? Check whether reviews stop, become throttled, or incur additional charges.
Are the findings useful? A large number of comments doesn't automatically mean better code review. Engineers still need to verify issues and decide which ones matter.

Codzee's shared-credit approach may suit teams with uneven workloads. Whether it is the right choice depends on pull request size, review volume, pricing, and performance on your own repositories.

Final thoughts

The best AI code review tool isn't necessarily the one advertising unlimited reviews or the lowest starting price.

It's the one that fits your workflow, keeps usage understandable, and helps your team identify issues worth fixing.

If you're exploring AI-powered code review for GitHub, you can learn more about Codzee.

We Tested 4 AI Code Review Tools Against 18 Security and Logic Defects

Codzee.io — Mon, 12 Oct 2026 09:51:02 +0000

Imagine opening a pull request that claims its tokens are secure, sessions expire correctly, and permissions fail safely.

The description sounds reassuring. The implementation is vulnerable.

Would your AI code review tool catch the difference?

We tested four AI code review tools against exactly this scenario. The results revealed more than which tool detected the most defects. They also exposed gaps in test analysis, review visibility, and GitHub merge protection.

The benchmark setup

For Round Ten of our AI code review benchmark, we compared:

Codzee
Greptile
CodeRabbit
CodeAnt

We created a deliberately vulnerable sandbox pull request containing 18 planted security and logic defects across three source files.

The pull request description made four security claims that the implementation contradicted. We also planted six defects in the test file and evaluated those separately.

All four tools reviewed the same commit. We collected findings from GitHub's REST API and scored them against an answer key prepared before the branch was pushed.

The results

| AI code review tool | Defects detected |

Codzee detected the most planted defects in this round. But the breakdown tells us more than the overall score.

The six test defects were scored separately: Codzee found all six, while the other three tools found none.

The security claims looked convincing. The code wasn't.

One claim in the pull request description said:

Tokens are signed and verified with HMAC-SHA256.

But the implementation used:

Buffer.from(payload).toString("base64")

Base64 is an encoding, not a cryptographic signature. Without proper signing and verification, an attacker could construct a token containing an administrator role.

Another claim said expired sessions were rejected. The implementation compared expiresAt with issuedAt instead of checking the current time. The expiration check therefore failed to enforce session expiry.

We also found a permission fallback that granted read and write access to unknown roles, even though the description promised a deny-by-default model.

These were not cosmetic code review comments. They were direct contradictions between the stated security requirements and the implementation.

A surprising result: the description didn't fool the tools

All four tools detected at least three of the four defects that the pull request description claimed were already handled.

Three competing tools explicitly connected their findings to the author's claims. Codzee detected all four defects but did not explicitly point out that the description itself was inaccurate.

That distinction matters during a real pull request review.

Compare these two findings:

"The secret comparison is not timing-safe."
"The secret comparison is not timing-safe, despite the security property claimed in the pull request."

The second gives the engineering team more context. It explains not only what is wrong, but why the implementation fails to meet its own stated requirements.

None of the four tools explicitly recommended correcting the misleading description before merging.

The defects that weren't mentioned

The largest difference appeared in the ten additional security defects that the author never mentioned.

Codzee detected all ten. Greptile detected seven, CodeRabbit five, and CodeAnt four.

The defects included:

A missing ownership check that allowed access to another user's booking.
A permission check that treated a role containing admin as an administrator role.
A permission cache that was never invalidated after privilege changes.
Session IDs generated using Math.random().
A hardcoded fallback secret.
Login error messages that revealed whether a username existed.

Codzee was the only tool to detect the final three issues.

This is an important dimension when evaluating AI code review tools: can they identify problems beyond the ones suggested by the pull request description?

Why test analysis matters

Codzee also detected all six planted test defects.

Two were especially concerning. One test expected a session to be created from an x-user-id header. Another expected the application to return a booking belonging to a different user.

These tests treated vulnerable behavior as correct.

A test suite can provide false confidence when it validates the wrong behavior. Reviewing application code without examining its tests can leave that problem undiscovered.

This result is promising, but it comes from six deliberately planted test defects in one sandbox. It should not be interpreted as proof that Codzee will find every test-related issue in production.

The result wasn't perfect, and that matters

Codzee missed one defect: logout() did not invalidate the token. A session that appeared to be logged out could remain valid.

The experiment also revealed problems in Codzee's own review workflow.

Of its 29 findings, only 10 appeared as inline comments. The remaining 19 were placed in a collapsed block, making some important findings harder to notice.

More seriously, Codzee reported ten Critical findings but returned a neutral GitHub check conclusion. Its legacy commit status simultaneously reported success.

This creates a gap between detecting a vulnerability and enforcing a security policy. If a team expects Critical findings to block merges, a neutral check does not meet that expectation.

We identified three priorities:

Make Critical findings enforceable through configurable GitHub check behavior.
Improve the visibility of findings, especially those that appear outside inline comments.
Include the pull request title and description in the review context so the tool can explicitly flag misleading security claims.

What this benchmark tells engineering teams

This experiment involved one deliberately constructed sandbox pull request, not thousands of production changes. It cannot establish that one tool will always outperform another.

It does, however, suggest that evaluating AI code review requires more than comparing a single score.

When assessing a tool, ask:

Detection: Can it find security defects that the author never mentioned?
Context: Does it compare the implementation with the stated requirements?
Test quality: Can it identify tests that validate insecure behavior?
Visibility: Are important findings easy for developers to discover?
Enforcement: Can critical findings influence the merge decision?

These dimensions determine whether review findings can translate into better engineering decisions.

What we're improving at Codzee

Codzee scored 17/18 in this round, but the benchmark also showed where the product needs work.

Our priorities are to improve critical-finding enforcement, make review comments more visible, and use pull request descriptions more effectively.

We are sharing these gaps alongside the results because a useful engineering benchmark should reveal weaknesses, not just produce a leaderboard.

You can learn more about Codzee at https://www.codzee.io.

A question for developers and engineering teams: When choosing an AI code review tool, which matters most to you: finding more defects, explaining them with better context, or ensuring critical vulnerabilities cannot pass through the merge workflow?

We Thought the Code Was Fine. Then the AI Found Something Else.

Codzee.io — Fri, 09 Oct 2026 10:34:54 +0000

We tested Codzee against Greptile, CodeRabbit, and CodeAnt. Here's what we learned from six rounds of AI code review benchmarks, including the bugs we found and the weaknesses we still need to fix.

We Thought the Code Was Fine. Then the AI Found Something Else.

Sometimes, the most important bugs are the ones nobody thinks to look for.

We've all seen it happen.

The code looks fine. The tests pass. The pull request gets approved. Then, later, someone discovers a bug that should have been caught during review.

That's one of the problems we want Codzee to help solve.

But as a team building an AI code review tool, we had to ask ourselves an uncomfortable question:

Is Codzee actually good at reviewing code, or does it just look good in a demo?

We decided to find out.

We tested Codzee against Greptile, CodeRabbit, and CodeAnt using the same pull requests, deliberately planted bugs, and a predefined answer key.

We ran six rounds, making each round harder than the last. Round six taught us the most — and it was the first round Codzee won.

Here's what happened.

The experiment: making the bugs harder to find

For Round 6, we created a pull request involving a guest billing system. It handled invoices, payments, taxes, and cancellations.

Instead of planting obvious mistakes in individual functions, we designed bugs that required understanding how different modules interacted.

For example, one module could calculate an invoice correctly while another ignored it and recalculated the payment amount. Nothing crashed, but the customer could be charged a different amount from the one shown on the invoice.

Here was our setup:

Test detail	Description
Pull request	PR #7 in a test repository
Code size	192 lines across two modules
Planted bugs	13
Cross-file bugs	6
Domain	Guest billing: invoices, payments, tax, cancellations
Tools tested	Codzee, Greptile, CodeRabbit, CodeAnt
Scoring method	Findings retrieved through the GitHub REST API and manually matched against a predefined answer key

We wrote down the answer key before the review began. We also committed to publishing the weaknesses we found in Codzee.

The first surprise: the bugs hidden between files

We started by examining the six defects that required understanding how two parts of the application worked together.

The bug	Codzee	Greptile	CodeRabbit	CodeAnt
Payment ignores the invoice and recomputes the charge	✓	✓	✗	✗
Tax is never collected	✓	✓	✓	✗
Balance reads a key the invoice never writes	✓	✓	✓	✓
Due date compared against a datetime	✓	✓	✓	✓
A `timedelta` passed where a day count is expected	✓	✓	✓	✓
Invoice numbers restart, causing payment records to collide	✓	✓	✓	✓
Total found	6/6	6/6	5/6	4/6

Four defects were caught by every tool. The two payment-related problems were more revealing: only Codzee and Greptile caught both.

The most serious involved code that built an invoice and then ignored it, recalculating the charge from another module. This could cause a guest to be billed an amount that didn't match their invoice.

Codzee marked this finding as Critical. Greptile also caught it, while CodeRabbit identified the tax issue but missed the related recomputation problem.

That was precisely the kind of reasoning gap we wanted the experiment to expose.

The Round 6 scoreboard

Next, we looked at the complete results, including bugs outside the original answer key.

Evaluation metric	Codzee	Greptile	CodeRabbit	CodeAnt
Bugs found (out of 13)	8	7	6	6
Cross-file bugs (out of 6)	6	6	5	4
Total findings	18	9	6	7
Genuine bugs we didn't plant	3	2	0	1
Findings on the test file	3	0	0	0
False positives	0	0	0	0

Source: Codzee's internal Round 6 benchmark. Results represent one pull request, not a universal ranking of the tools.

Codzee found eight of the thirteen planted bugs. But the most interesting result wasn't the score itself.

It found three additional genuine issues that weren't part of our original answer key.

One involved a declined payment potentially being recorded as successful because the gateway receipt wasn't checked. Another involved a retry that could potentially charge a customer twice because the idempotency key changed.

Greptile also found two additional issues, while CodeAnt found one.

These findings reminded us that a predefined answer key is useful, but incomplete. Real applications can contain problems nobody thought to plant.

The test file nobody reviewed

We had also included three tests that asserted incorrect behavior.

A passing test can create confidence in the code, but what if the test itself expects the wrong result?

Codzee was the only tool to flag all three problematic tests in this round:

A due-date test should expect 15 June, not 1 July.
An unpaid invoice should show 22,300 pence outstanding, not zero.
An assertion simply repeated the implementation's calculation instead of independently checking the expected result.

This was a meaningful finding for us. Code review shouldn't stop at the implementation if the tests themselves are reinforcing incorrect behavior.

It was also the fourth round out of six in which Codzee was the only tool to review the test file.

How did Codzee perform across all six rounds?

One round can tell an interesting story, but repeated testing helps reveal patterns.

Tool	Round 1	Round 2	Round 3	Round 4	Round 5	Round 6
Greptile	5/6	10/12	3	6/6	9/9	7/13
Codzee	4/6	9/12	2	3/6	8/9	8/13
CodeRabbit	4/6	8/12	1	5/6	8/9	6/13
CodeAnt	1/6	8/12	3	4/6	6/9	6/13

The rounds used different numbers of planted bugs. The Round 3 figures are reproduced as reported in the original article.

Our honest reading of these results:

Greptile was the most consistent across the series and found 13 unplanted bugs over six rounds.
Codzee recorded its first win in Round 6 and found its first three unplanted issues in that round.
CodeRabbit produced fewer findings and offered particularly clear fix guidance on the tax issue in Round 6.
CodeAnt identified several defects, but struggled more with cross-file issues in this round.

These are observations from our tests, not permanent rankings of the products.

What we got wrong — and what we still need to fix

We promised ourselves that we would publish the bad news, too.

Three planted bugs defeated all four tools: a mutable default argument, a function-local import that obscured the recomputation path, and a rounding issue that could silently lose a penny on every invoice.

We also found weaknesses in Codzee itself.

Of its 18 findings, only 10 appeared inline on the code diff. The other eight were in the summary, making them harder to connect directly to the relevant code.

Our inline placement rates across the six rounds were 50%, 100%, 0%, 75%, 25%, and 56%. That inconsistency is something we need to address.

There was another problem: despite identifying a Critical payment defect, Codzee's merge-check conclusion remained neutral. A serious finding should be reflected clearly in the signal developers use when deciding whether to merge.

The good news is that repeated testing also helped us identify improvements in severity calibration. In Round 6, Codzee assigned the Critical rating to the payment defect and graded the problematic tests as Major.

But there is still work to do.

What we learned

This experiment reinforced a few lessons we think matter to anyone choosing an AI code review tool.

1. Context matters. Bugs can hide in the relationship between modules, even when individual functions look reasonable.

2. Unexpected findings matter. A tool should be evaluated on legitimate issues beyond the bugs deliberately planted for testing.

3. More comments don't necessarily mean better reviews. Greptile found six of six cross-file bugs with nine findings. Codzee produced twice as many findings and found one additional planted bug.

4. Tests deserve scrutiny. Three incorrect tests were present, and only Codzee flagged them in this round.

5. Placement is part of quality. A finding is more useful when developers can see exactly where the problem occurs.

6. AI review doesn't replace linters. All four tools missed defects that mechanical checks may be better equipped to catch.

We also didn't test everything. Security depth, CI/CD integration, customization, scale, and team adoption were outside the scope of this experiment.

So, should you trust Codzee?

We don't think trust should come from a marketing claim — including ours.

In Round 6, Codzee found eight of thirteen planted bugs, matched Greptile on all six cross-file defects, identified three additional genuine issues, and flagged three problematic tests.

Those results encouraged us. The missed bugs, inconsistent inline placement, and neutral merge signal gave us clear areas to improve.

Greptile remained a strong competitor, and every tool missed something.

We build Codzee, so we're not neutral observers. That's exactly why we believe the methodology, results, and limitations should be visible.

Our goal isn't to generate more comments. It's to help developers find meaningful problems before they reach production.

If you're evaluating AI code review tools, don't take our word for it. Test them on your own pull requests. Hide a bug between two files, include a test that asserts the wrong behavior, and see what each tool catches.

You can explore Codzee at codzee.io and read the

We'd love to hear from other developers: what's the kind of bug your AI reviewer consistently misses?

Our client Tested 4 AI Code Review Tools on the Same PR and Here’s What They Missed

Codzee.io — Wed, 07 Oct 2026 05:58:34 +0000

We wanted to answer a simple question:

How well do today’s leading AI code review tools actually reason about bugs that cross module boundaries?

So we gave Codzee, Greptile, CodeRabbit, and CodeAnt the exact same pull request.

The PR contained 10 planted defects across 172 lines and 2 modules, with unit-related bugs involving kilometres, metres, miles, pence, pounds, seconds, minutes, and more.

The result?

Three tools tied at the top. But Codzee found something nobody else did.
Press enter or click to view image in full size

On paper, it looks like a three-way tie.

But one finding changed the story.

The Bug Nobody Had Found Before
One of the planted defects involved banker’s rounding in a fare calculation.

The code used Python’s round() behavior in a situation where commercial rounding was expected.

For a fare of 1,250 pence, Python rounds it to:

1,200 pence — not 1,300.

Codzee caught it.

But it didn’t stop at saying “rounding may be incorrect.”

It reasoned through the behavior of Python’s standard library and explained the actual commercial consequence: half-pound fares can round differently depending on whether the pound value is even or odd.

Greptile, CodeRabbit, and CodeAnt all reviewed the same function.

None of them caught it.

And this wasn’t just a one-off.

This rounding defect had appeared in previous benchmark exercises. Across 16 benchmarked pull requests and roughly 500 findings, no tool had reported it.

Until Codzee.

That was the moment the benchmark became more interesting than a simple scorecard.

The Full Defect Breakdown
Press enter or click to view image in full size

The benchmark isn’t about claiming perfection. It’s about understanding where each tool succeeds, where it fails, and why.

What the Developer Running the Benchmark Noticed
From the developer’s perspective, another result stood out.

Codzee didn’t just review the implementation.

It also looked at the tests.

Two of its findings identified tests that were effectively validating incorrect behavior including a massively inflated fare and an unrealistic journey duration.

That matters because a bad test can make a bad implementation look correct.

A useful AI reviewer needs to question both.

And Then We Found an Even Bigger Lesson
The most interesting result wasn’t actually which tool won.

It was the 90% detection rate for unit-related defects.

These same types of defects had survived nine previous rounds across the tools.

This time, both modules explicitly stated their units in their opening docstrings.

That small change made a huge difference.

The lesson?
Give the reviewer context, and the reviewer can reason better.

For developers, documenting units is almost free.

For AI code review systems, it provides the context needed to connect values across functions and modules.

So, What Did We Learn?
Codzee
9/10 — tied for first and the only tool to catch the banker’s-rounding defect.

Greptile
9/10 — strong reasoning and consistently concrete findings.

CodeRabbit
9/10 — particularly strong at connecting implementation issues with incorrect tests.

CodeAnt
7/10 — good critical-severity detection, but missed the most severe fare calculation defect.

And Codzee?

Two consecutive strong rounds.

After winning Round 6 with 8/13, Codzee tied for first in Round 7 with 9/10.

The Takeaway
The benchmark started with a simple goal:

Find the bugs.

But the more interesting question became:

Can an AI code reviewer actually understand why the code is wrong?

The banker’s-rounding result suggests Codzee is getting closer to that level of reasoning.

It understood the code → calculation → consequence.

And that’s ultimately what we want from an AI code reviewer

Not more comments. Better understanding.

We Tested 4 AI Code Review Tools on the Same PR and The Results Were Surprising…

Codzee.io — Mon, 05 Oct 2026 10:53:21 +0000

A case study from six rounds of testing Codzee against Greptile, CodeRabbit and CodeAnt

It started with a question we couldn't answer

We build an AI code review tool called Codzee. At some point we realized we couldn't answer a simple question: is it actually good?

Our demos looked great, and so did every competitor's. Every landing page promises "context-aware" and "low-noise" review. None of that tells you what happens when a tool meets a real pull request with a bug it can't find by pattern matching.

So we stopped reading feature lists and ran an experiment. We built pull requests with bugs planted in them, gave the same PR to our tool and three competitors, and checked what each one caught.

We've now done this six times, and each round was harder than the last. This is the story of round six, the round that taught us the most, and the first one Codzee won.

The rules we set for ourselves

Because we were testing our own product, we didn't trust ourselves. We set two rules:

The answer key comes first. We wrote down every planted bug before the pull request was opened, so nobody could redefine what counts as a "catch" afterward.
Publish the bad news too. If our tool stumbled, it goes in the write-up.

Here's the setup for round six:


PR	#7 in a test repository
Size	192 lines across two modules
Planted bugs	13
Cross-file bugs	6
Domain	Guest billing: invoices, payments, tax, cancellations
Tools	Codzee, Greptile, CodeRabbit, CodeAnt
How we scored	Pulled every finding from the GitHub REST API and matched them to bugs by hand

Qodo was excluded from this benchmark for billing reasons.

The trap we built

Most AI review demos show a tool catching a bug in one function. That's easy mode, so we designed round six around something harder.

The two modules were tied together by a written contract. It said things like:

The invoice total is pre-tax.
The balance reads a key called amount_pence.
The due date is a date, not a datetime.
Invoice numbers are unique for life.

Six of our 13 bugs were cross-file defects: each module looks correct on its own, and the error exists only in how they fit together. You can't find them by reading either file. You have to read both against the contract.

The centerpiece was this one:

# Illustrative sketch, not copied from the PR
invoice = build_invoice(booking)     # built...
charge = rates.compute(booking)      # ...ignored, then recomputed
gateway.charge(guest, charge)        # guest sees one number, pays another

The payment path builds an invoice, throws it away, and recomputes the charge from another module. The guest is billed a figure that appears nowhere on their invoice, with tax and any extra lines dropped. Nothing crashes, and every function does its own job correctly.

The first thing we noticed: speed

Codzee finished in 57 seconds. CodeAnt took about two minutes. Greptile took 3 minutes 43 seconds. (We didn't record a time for CodeRabbit.)

We wouldn't read much into this, because the timings may not be directly comparable. But it was the first visible difference, before anyone looked at the findings.

The second thing we noticed: nobody was wrong

We pulled every finding from the API and matched them against the answer key. There were zero false positives across all four tools. Every finding anyone posted was real, so in this round the differences weren't about who was wrong. They were about who saw more, and where they put what they saw.

The seam

Here's how the four tools did on the six cross-file bugs:

The bug	Codzee	Greptile	CodeRabbit	CodeAnt
Payment ignores the invoice, recomputes the charge	✓	✓	✗	✗
Tax is never collected	✓	✓	✓	✗
Balance reads a key the invoice never writes	✓	✓	✓	✓
Due date compared against a datetime	✓	✓	✓	✓
A `timedelta` passed where a day count is expected	✓	✓	✓	✓
Invoice numbers restart, payment records collide	✓	✓	✓	✓
Total	6/6	6/6	5/6	4/6

Four of the six were caught by everyone. These were visible mismatches, such as a wrong key, a wrong type or a counter that resets. A tool sees a value in one place and the wrong shape of it in another.

The top two rows, the money bugs, were different. They split the field.

Only Codzee and Greptile found the recomputed charge, and in our assessment Codzee's description was the more complete of the two. It was also the one bug Codzee marked Critical, which felt right, since it's the only defect in the PR that silently bills a guest the wrong amount.

The most interesting result belonged to CodeRabbit. It commented on the tax problem in the same function but never noticed that the function was also ignoring the invoice it had just built. It saw the symptom and not the cause sitting a few lines away. We can't see inside the tool, so we can't say why. It was exactly the gap we'd designed the round to expose.

CodeRabbit did something else well, though. Its fix guidance on the tax bug was the clearest of the round: include tax in both the amount due and the gateway charge, with both call sites named.

The plot twist: the bugs we didn't plant

This is the part of the round that changed how we think about our own tool.

Our answer key is limited by our imagination. Real code has bugs nobody planted, so we also counted genuine findings outside the key.

For the first five rounds, this was the criticism we couldn't answer: Codzee had found zero of them. Greptile had found eleven.

In round six, Codzee found three. Two were serious, and no other tool reported either one:

The declined card that counted as paid. The payment path never checks the gateway receipt. A declined card is recorded as a full payment, so the invoice shows as settled and no money was taken.

The retry that charges twice. After a timeout, a retry builds a new invoice with a new number. That changes the idempotency key, and the guest gets charged twice.

Both require following control flow across steps, not reading a single line.

Greptile found two of its own, and both are about durability. An unsupported country code silently inherits the 20% GB rate, so an invoice looks plausible with a tax figure nobody chose. And payment records live only in an in-memory dictionary, so a restart wipes the history that balance and refund calculations rely on.

CodeAnt found one, in keeping with its style, which was precise and focused on a single function. Its findings included an unsynchronized read-and-increment on the invoice counter, unvalidated payment amounts, and a getter that hands back the live internal list so callers can edit the ledger.

The test file nobody opened

Three tests in the diff asserted the wrong behavior.

That's a nasty class of bug, because a passing test makes a defect look intentional. A reviewer who sees green reasonably assumes the behavior is correct.

Only Codzee opened the test file. This is now the fourth round out of six where it's been the only tool to do so. It posted one finding per bad test, each naming the correct expected value:

The due-date test should expect 15 June, not 1 July.
An unpaid invoice should show 22,300 pence outstanding, not zero.
Asserting that amount due equals the total just restates the implementation.

Greptile, CodeRabbit and CodeAnt had nothing to say about the test file.

The scoreboard

	Codzee	Greptile	CodeRabbit	CodeAnt
Bugs found (of 13)	8	7	6	6
Cross-file (of 6)	6	6	5	4
Total findings	18	9	6	7
Real bugs we didn't plant	3	2	0	1
Findings on test file	3	0	0	0
False positives	0	0	0	0

Codzee found the most bugs, had a perfect cross-file score and was the only tool with test-file findings. But the margin over Greptile was one defect on one PR. Greptile matched us on the hardest dimension with half as many findings.

What none of us caught

Three bugs defeated all four tools:

A mutable default argument, used in two functions
A function-local import that hid the recompute from view
A truncation that quietly loses a penny on every invoice

The first is a textbook lint finding. In round one, nobody caught banker's rounding or an unreachable return either. As far as we can tell, none of these tools runs a linter, though we haven't verified how each works internally.

We don't know why every tool missed these. Our hypothesis, and it is only a hypothesis, is that these tools put their effort into semantic reasoning and less into mechanical checks and subtle numeric behavior.

The practical lesson: an AI reviewer doesn't replace your linter.

The part where our own tool let us down

If we stopped here, this would be a flattering story, and we promised it wouldn't be.

Problem 1: almost half the findings weren't on the code. Of Codzee's 18 findings, only 10 were posted inline on the diff. The other 8 sat in the summary, including all three test findings, the ones that most need to sit next to the code they describe. Across six rounds, our inline rate has been 50%, 100%, 0%, 75%, 25%, 56%, with no trend. Every other tool anchored its findings inline.

Someone reading the pull request sees less of the win than the analysis shows.

Problem 2: eighteen findings and no verdict. Codzee flagged a Critical that bills guests incorrectly, and the check conclusion was still neutral. A neutral check can't block a merge. Greptile found fewer bugs and still reported a confidence score.

There was one thing we got right, though. In round five, Codzee had graded a worthless test suite as Minor, which was poor severity calibration. In round six it used three levels (1 Critical, 12 Major, 5 Minor), the Critical went to the right bug, and the bad tests were graded Major. Running the same experiment repeatedly let us confirm that a problem we found actually got fixed.

The six-round story

One round is an anecdote, so here's the full ramp:

Tool	R1	R2	R3	R4	R5	R6
Greptile	5/6	10/12	3	6/6	9/9	7/13
Codzee	4/6	9/12	2	3/6	8/9	8/13
CodeRabbit	4/6	8/12	1	5/6	8/9	6/13
CodeAnt	1/6	8/12	3	4/6	6/9	6/13

The honest reading:

Greptile is the most consistent tool in the series. It has never had a weak round and has found 13 unplanted bugs across six.
Codzee is rising. It got its first win, its first unplanted finds and a fixed severity problem, but it started behind.
CodeRabbit is steady and precise, and consistently the lowest-volume tool.
CodeAnt is strong wherever a defect is visible in one function and weaker wherever it requires comparing two things.

What we learned

1. The real test is reasoning distance. Every tool can read a function. The ones that stand out can read a function against a contract in another file.

2. The hardest bugs are absences. An ignored invoice and a missing receipt check are things the code didn't do, and they're harder to flag than a line that's visibly wrong.

3. More comments aren't a better review. Greptile got six of six cross-file from nine findings. We posted twice as many and found one more bug.

4. Read the tests. Three bad tests were quietly vouching for three bugs, and only one tool noticed.

5. The best findings are often unplanted. The declined card and the double charge weren't in our key, and they may be the two most serious findings in the PR.

6. Placement is part of quality. A finding that isn't next to the code is easy to miss.

Steal this checklist

You can use it even if you never touch Codzee.

Plant a cross-file bug. Have one module quietly ignore what another returns.
Plant a bad test. See whether the tool opens the test file.
Count findings, then count what you'd act on.
Check where comments land, inline or buried in a summary.
Check severity. Does "Critical" mean critical?
Check the merge signal. Can it block a merge or only comment?
Read the fix guidance. Does it name call sites and expected values?
Track unplanted finds.
Keep your linter.
Test what we didn't: security depth, CI/CD integration, customization, scale and team adoption were outside this experiment.

Limitations

This was an observational comparison, not a statistically significant benchmark.

It was one PR in this round: 192 lines, one domain, and probably one language.
We wrote the answer key. Codzee's team designed the bugs and has a stake in the result.
Matching findings to bugs was done by hand, which involves judgment.
Tool versions and configurations may differ across tools.
Time figures may not be directly comparable.
Tools change fast. These results describe the day of the run.

Nothing here shows that any tool is universally better.

What happens next

Round seven pushes reasoning distance further. The same quantity will be metres in one module and kilometres in another, the classic units-crossing-a-boundary bug. Every tool missed that class in an earlier benchmark.

We expect it to be hard, and we'll publish the results either way.

The takeaway

We started this series wanting to know whether our tool was any good. After six rounds, the answer is more useful than yes or no.

It's good at some things, such as reading across files, reviewing tests and finding bugs nobody planted. It's still weak at others, namely inline placement and merge verdicts. And it's competing with tools that are better in places: Greptile on consistency, CodeRabbit on precision, CodeAnt on single-function hazards.

If you're choosing an AI code review tool, don't trust a feature list, ours included. Hide a bug in the gap between two files, put one wrong test in the diff, and see who notices.

We build Codzee and ran this benchmark ourselves. The methodology and limitations are above. We'd rather you check our work than trust it.

What kinds of bugs do you find AI reviewers consistently miss? Tell us in the comments.

what if Dismissed an AI Code Review Comment. Then I Checked the Next PR.

Codzee.io — Mon, 21 Sep 2026 13:30:50 +0000

I dismissed an AI code review comment.

Nothing unusual there.

What was interesting was what happened in the next pull request.

A few PRs later, a similar pattern appeared in the code. I expected the same review comment.

It wasn't there.

That small behavior points to a bigger question for AI-assisted code review:

What should an AI reviewer do when an engineer disagrees with it?

If the answer is simply “nothing,” the reviewer stays stateless.

If the answer is “learn from the decision,” review behavior can become more aligned with the repository over time.

The annoying part of automated review

AI code review can catch things that are easy to miss.

But there is another failure mode: repetition.

A reviewer finds something that looks questionable according to a general coding guideline.

The developer knows the pattern is intentional.

The comment gets dismissed.

Then another PR comes along.

The same pattern appears.

The reviewer raises the same concern.

The developer dismisses it again.

And again.

At some point, the problem isn't that the reviewer cannot find issues.

The problem is that it hasn't learned which issues this repository doesn't need to discuss repeatedly.

That is where dismissal-learning becomes interesting.

PR #1: The first review

Imagine a repository where a particular implementation pattern is intentional.

An AI reviewer sees it and leaves a comment:

AI reviewer:

Consider changing X to Y for better consistency.

The comment is reasonable in isolation.

But the repository has a reason for using X.

So the engineer dismisses the comment:

Dismissed.

This pattern is intentional in this repository.

Normally, that interaction is over.

The comment disappears from the PR.

But the dismissal contains useful information.

A dismissal is engineering feedback

A dismissal doesn't necessarily mean:

“The reviewer is useless.”

It can mean:

“This recommendation doesn't apply to this codebase.”

That's an important distinction.

Repositories have local conventions that aren't always obvious from the code itself.

Some exist because of:

compatibility requirements
architectural decisions
performance constraints
legacy integrations
operational considerations
team conventions
deliberate trade-offs

A generic reviewer doesn't automatically know all of these.

The developer does.

So when a developer dismisses a review comment, that decision can be treated as feedback about the repository.

The interesting part is what happens next.

Turning the dismissal into repository memory

Instead of storing only the conversation, the system can extract the underlying pattern from the interaction.

Conceptually:

Review comment
      ↓
Developer dismissal
      ↓
Underlying pattern identified
      ↓
Repository memory
      ↓
Future reviews use that context

For example:

Repository memory:

"This pattern is accepted/preferred in this repository."

The important word here is repository.

This isn't a global rule saying that the pattern is always correct.

It is context associated with this particular codebase.

That's important because engineering decisions are rarely universal.

A pattern that is discouraged in one repository may be completely intentional in another.

PR #2: The same situation appears again

Now a different pull request arrives.

The code contains a similar pattern.

Without repository memory, the reviewer might produce:

AI reviewer:

Consider changing X to Y for better consistency.

Which leads to another dismissal.

With the previous feedback available as repository context, the reviewer can recognize that this type of comment has already been rejected for this repository.

So the result is:

AI reviewer:

[No comment]

That's the part I find interesting.

The system didn't need another configuration file.

The developer didn't need to write a permanent rule manually.

A previous engineering decision influenced a future review.

Before vs. after

Situation	Without dismissal-learning	With repository memory
First PR	Review comment appears	Review comment appears
Developer response	Dismisses comment	Dismisses comment
Feedback retained	Little or no context	Pattern becomes repository context
Similar later PR	Same comment may return	Reviewer can stay quiet
Developer experience	Repeated explanation	Less repeated feedback

This doesn't mean every similar comment should disappear.

Context still matters.

But it creates the possibility of a reviewer that becomes better aligned with the repository's actual engineering decisions.

This isn't “remembering a conversation”

This distinction is worth making.

When people hear “AI memory,” they often think about conversational memory:

“Remember what I told you yesterday.”

That's not the interesting part here.

The useful unit is the engineering decision.

The system needs to understand something closer to:

Pattern:
    X is used in this repository.

Developer decision:
    This usage is intentional.

Review implication:
    Don't repeatedly flag this pattern in the same context.

That's repository memory.

It turns feedback from an individual review interaction into context that can influence future reviews.

Why this feedback loop matters

Code review isn't just about detecting problems.

It's also about communicating what a team considers important.

Human reviewers naturally build this context over time.

After reviewing hundreds of pull requests, an experienced engineer knows:

which patterns are normal
which warnings are usually noise
which conventions are intentional
which architectural decisions shouldn't be revisited every PR

An AI reviewer that doesn't retain any of that context has to rediscover the same information repeatedly.

That creates review fatigue.

A feedback loop changes the model:

Find something
     ↓
Developer responds
     ↓
Learn from the response
     ↓
Apply the context later
     ↓
Produce more relevant feedback

The goal isn't fewer comments for the sake of fewer comments.

It's higher-signal comments.

There are caveats

Dismissal-learning shouldn't turn every dismissal into a permanent rule.

A developer can dismiss something because the comment is wrong.

They can also dismiss it because the context is different in that particular file or PR.

The architecture may change later.

A pattern that was acceptable six months ago might become a technical debt issue today.

So repository memory needs scope and context.

The right behavior isn't:

“This was dismissed once, never mention it again.”

It's closer to:

“Previous engineering feedback says this pattern is acceptable here. Consider that context before raising the same observation again.”

That's a much more useful mental model.

The interesting shift

AI code review is often discussed in terms of how much it can detect.

I think another question deserves equal attention:

How well can it learn what not to say?

A reviewer that finds everything but repeatedly generates low-value comments can still create friction.

A reviewer that incorporates developer feedback can gradually become more aligned with the repository.

That's what makes this behavior interesting.

One comment gets dismissed.

The underlying pattern becomes repository memory.

A similar situation appears later.

The reviewer stays quiet.

Not because the reviewer has learned everything.

Not because repeated comments magically disappear.

But because a real engineering decision from the repository has become part of the context for the next review.

That's what “gets quieter” actually means.

If you want to see this behavior rather than read about it, the most useful test is simple: try it on a repository with an established review convention and watch what happens across two similar PRs.

See the mechanism in action on your own repository.

How to Review AI-Generated Code Without Losing Your Mind

Codzee.io — Thu, 17 Sep 2026 12:53:01 +0000

AI can generate a working implementation in seconds.

That doesn't mean the implementation is correct.

For developers, this creates a slightly different code-review problem. You're no longer reviewing only code written from scratch by someone who understands the surrounding system. You're often reviewing code generated from a prompt, existing examples, partial context, or a combination of all three.

The result can compile, pass tests, and look perfectly reasonable while still being wrong.

The answer isn't to distrust every line of AI-generated code. It's to review it differently.

This article presents a practical approach for reviewing AI-generated changes without spending an hour reading every line with equal attention.

1. Why AI-Generated Code Is Different

AI-generated code has one particularly important characteristic:

It can be locally convincing while being globally incorrect.

Consider a simple API endpoint:

@app.get("/users/{user_id}")
def get_user(user_id: int, db: Session = Depends(get_db)):
    user = db.query(User).filter(User.id == user_id).first()

    if not user:
        raise HTTPException(status_code=404, detail="User not found")

    return user

At first glance, this looks fine.

But several questions immediately matter:

Is the endpoint authenticated?
Can any authenticated user retrieve another user's data?
Is User safely serializable?
Does returning the ORM object expose fields that shouldn't leave the API?
Is there an authorization requirement that isn't represented in this function?

The code itself doesn't contain enough information to answer those questions.

That's common with AI-generated code.

The model optimizes for producing a plausible implementation from the available context. It doesn't automatically understand your organization's security model, undocumented business rules, production traffic patterns, or architectural constraints.

So code review needs to evaluate two things:

What does this code do?

and

What assumptions does this code make?

The second question is often where the important bugs are.

2. What Should Reviewers Check First?

Don't start by reading every line.

Start with the change itself.

Before looking at implementation details, answer:

What problem is this PR solving?

If the PR description says:

"Add caching to improve API performance."

you should immediately ask:

What endpoint?
What data is cached?
How long?
What invalidates it?
Is stale data acceptable?
Is the cache shared between users?

Then look at the diff.

A useful first pass is:

Understand the intended behavior.
Identify the files changed.
Look for changes to security boundaries.
Look for database and external-service interactions.
Look for new state or caching.
Check error-handling paths.
Only then inspect implementation details.

This prevents a common review failure: spending ten minutes debating naming while missing a broken authorization check.

3. Logic and Edge Cases

AI-generated code frequently handles the happy path well.

The interesting bugs tend to live elsewhere.

Suppose an AI-generated function processes payments:

function calculateRefund(order, requestedAmount) {
    if (requestedAmount <= order.total) {
        return requestedAmount;
    }

    return order.total;
}

Looks harmless.

But what happens with:

calculateRefund(order, -100);

The function returns -100.

What about floating-point currency?

0.1 + 0.2 !== 0.3

What about an already-refunded order?

What about concurrent refund requests?

What about an order whose total has changed after partial refunds?

The important question isn't:

"Does this function work?"

It's:

"Under what conditions does this function stop working?"

For AI-generated code, explicitly test:

Empty input
Null/undefined values
Zero values
Negative values
Very large values
Duplicate requests
Concurrent requests
Missing records
Deleted records
Partial failures
Timeouts
Retries
Invalid state transitions

Consider this common pattern:

if not user:
    create_user(email)

That looks reasonable until two requests arrive simultaneously.

Both requests can observe that the user doesn't exist.

Both attempt to create it.

Now you've discovered a race condition.

The database should usually enforce the invariant as well:

CREATE UNIQUE INDEX users_email_unique
ON users(email);

Then the application needs to handle the constraint violation appropriately.

Don't assume the generated code has considered concurrency simply because the sequential logic looks correct.

4. Security

Security deserves an earlier pass than ordinary correctness.

AI-generated code can accidentally introduce vulnerabilities because security depends heavily on context.

SQL injection

Bad:

query = f"SELECT * FROM users WHERE email = '{email}'"
db.execute(query)

Better:

query = text("SELECT * FROM users WHERE email = :email")
db.execute(query, {"email": email})

But parameterized queries aren't the end of the review.

Ask:

Is authorization enforced?
Is sensitive information logged?
Are secrets exposed?
Is user-controlled input trusted?
Are uploaded files validated?
Are redirects controlled?
Are permissions checked server-side?
Are internal errors exposed to clients?

For example:

app.get("/admin/users", authenticate, async (req, res) => {
    const users = await getUsers();
    res.json(users);
});

Authentication exists.

Authorization might not.

A reviewer should ask whether req.user actually has permission to access the endpoint.

Logging is another common problem

An AI-generated debugging statement might look innocent:

logger.info("Payment request: %s", request.json)

If the request contains card information, authentication tokens, addresses, or other sensitive fields, you've just created a data-exposure problem.

Review security based on data flow, not just individual functions.

5. Tests

Passing tests don't prove that generated code is correct.

They prove that the tested scenarios currently behave as expected.

AI-generated tests can have another problem: they may simply encode the implementation rather than validate the requirement.

For example:

def test_discount():
    result = calculate_discount(100, 10)
    assert result == 90

Useful, but incomplete.

What about:

calculate_discount(100, 0)
calculate_discount(100, 100)
calculate_discount(100, -10)
calculate_discount(100, 110)
calculate_discount(0, 10)

More importantly, ask whether the tests represent actual business requirements.

A strong review checks:

Coverage of behavior

Not just line coverage.

You want coverage of:

Happy paths
Failure paths
Boundary conditions
Authorization
State transitions
Retries
Timeouts
Concurrency where relevant

Test independence

Watch for tests that depend on execution order or shared mutable state.

Mock realism

AI-generated tests often mock everything:

mock_db.return_value = fake_user
mock_api.return_value = {"status": "ok"}

The test may pass while the real integration is broken.

Mocks should isolate behavior intentionally, not hide the behavior you're supposed to verify.

6. Architecture

This is where human review becomes particularly important.

AI is very good at producing an implementation that fits the immediate request.

It isn't necessarily good at knowing whether that implementation belongs in your architecture.

Imagine a service that adds:

const result = await database.query(...);
const response = await externalApi.call(...);
await cache.set(...);
await sendEmail(...);

All inside one HTTP controller.

Each operation may work.

The architectural problem is that one request handler now owns:

persistence
external integration
caching
notifications
business logic

The PR might technically work while making the system harder to evolve.

Ask:

Does this belong in this layer?
Are responsibilities separated?
Is existing architecture being bypassed?
Does this introduce a new abstraction unnecessarily?
Does this duplicate existing functionality?
Does this create a new dependency?
Does the data flow still match the system's boundaries?

AI-generated code often creates new helpers or abstractions when an existing one already exists.

Search the repository before approving new infrastructure.

The best implementation may already be somewhere in the codebase.

7. Maintainability

Readable code isn't necessarily maintainable code.

Look for unnecessary complexity.

For example:

const shouldProcess =
    user &&
    user.status &&
    user.status !== "disabled" &&
    (!user.deletedAt || user.deletedAt === null) &&
    permissions &&
    permissions.includes("write");

The code works, but the condition is difficult to reason about.

Could the domain logic be clearer?

function canWrite(user, permissions) {
    if (!user || user.status === "disabled") {
        return false;
    }

    if (user.deletedAt) {
        return false;
    }

    return permissions?.includes("write") ?? false;
}

During review, consider the next developer.

Will they understand why this code exists?

Can they modify it safely?

Does it introduce duplicated logic?

Does the naming describe the domain?

Is there enough context around non-obvious decisions?

AI can produce syntactically clean code very quickly. That doesn't automatically make the codebase easier to maintain.

8. Common AI-Review Mistakes

There are several ways reviewers can make AI-generated PRs harder to review.

Mistake 1: Reviewing every line equally

Not every line has the same risk.

Focus attention on:

Security boundaries
Database mutations
Authentication/authorization
Concurrency
External APIs
State changes
Error handling

Mistake 2: Trusting green tests

Tests are evidence.

They're not proof.

Ask what isn't tested.

Mistake 3: Focusing on style before behavior

Don't spend ten comments discussing variable naming while missing incorrect business logic.

Automate formatting and linting wherever possible.

Mistake 4: Assuming generated code understands your system

It doesn't have your entire organizational context.

Review assumptions explicitly.

Mistake 5: Ignoring PR size

A 1,500-line AI-generated PR is difficult to review regardless of how clean the code looks.

Breaking changes into smaller PRs makes correctness easier to establish.

Mistake 6: Asking AI to review AI without verification

An AI reviewer can be useful for finding patterns humans might miss.

But its findings still require validation.

Don't replace human judgment with another generated judgment.

9. A Practical AI-Code Review Checklist

Before approving an AI-assisted PR, run through this checklist.

Context

[ ] Do I understand what problem this PR solves?
[ ] Is the scope clear?
[ ] Is the PR reasonably sized?

Logic

[ ] Does the implementation match the requirement?
[ ] What assumptions does the code make?
[ ] What happens with empty, invalid, or extreme input?
[ ] Are race conditions possible?
[ ] Are retries and duplicate requests safe?

Security

[ ] Is authentication correct?
[ ] Is authorization enforced?
[ ] Is user input handled safely?
[ ] Are secrets protected?
[ ] Could sensitive information reach logs?
[ ] Are errors exposing internal information?

Data

[ ] Are database queries safe?
[ ] Are transactions required?
[ ] Are constraints enforced at the database level?
[ ] Could concurrent requests corrupt state?
[ ] Is caching introducing stale or cross-user data?

Tests

[ ] Are important behaviors tested?
[ ] Are edge cases covered?
[ ] Are failure paths tested?
[ ] Are tests testing requirements rather than implementation details?
[ ] Are mocks hiding important integration behavior?

Architecture

[ ] Does the change fit existing patterns?
[ ] Is there already code that solves this problem?
[ ] Are responsibilities in the right layer?
[ ] Does the change introduce unnecessary dependencies?

Maintainability

[ ] Is the code understandable?
[ ] Is complexity justified?
[ ] Are names meaningful?
[ ] Is duplication introduced?
[ ] Will another developer understand the reasoning six months from now?

The Goal Isn't to Review More. It's to Review Better.

AI changes the economics of writing code.

When producing another function, test, refactor, or API endpoint becomes dramatically cheaper, teams can generate changes faster than humans can carefully inspect them.

That creates a new bottleneck:

engineering attention.

The answer isn't to manually inspect every generated line forever.

It's to make review more deliberate.

Start with the highest-risk behavior.

Question assumptions.

Test boundaries.

Trace data.

Inspect security boundaries.

Understand architectural consequences.

And keep changes small enough that another engineer can actually understand them.

AI can make code generation cheap.

Human attention is still expensive.

Code review should spend that attention where it has the highest chance of preventing something from reaching production.

A Label Isn't a Code Review

Codzee.io — Tue, 15 Sep 2026 10:53:47 +0000

Consider two comments on the same piece of code.

The first:

“Consider refactoring this function for readability.”

The second:

“This returns null when no active subscription exists, but the caller immediately dereferences .plan. That turns a valid ‘no subscription’ state into an exception. Handle the empty case before accessing plan.”

The first comment might be correct.

The function might genuinely be difficult to read.

But it leaves an important question unanswered:

What is the engineer supposed to do with that information?

The second comment contains considerably more information. It identifies a behavior, connects it to another part of the system, describes the failure mode, and suggests a direction.

That seems straightforward.

But it raises a more interesting question:

What information does a code review comment actually need to contain to be useful?

Code review isn't just defect detection

It is tempting to think about code review as a simple pipeline:

code
  ↓
reviewer
  ↓
problem detected
  ↓
comment
  ↓
code changed

But modern code review is doing several things simultaneously.

It can catch defects.

It can question design decisions.

It can communicate project conventions.

It can transfer knowledge between engineers.

It can document decisions.

It can expose assumptions that aren't visible in the code itself.

Research on modern code review has found that reviewers and authors value activities beyond defect detection, including knowledge sharing and increasing awareness of the codebase.

A Microsoft Research study of peer review also found that review is a form of knowledge exchange in which design rationale can be surfaced during the discussion.

So a review comment isn't merely a defect marker.

It is a small communication channel between two people who may have very different amounts of context.

That changes what “good” means.

Flagging isn't explaining

Consider:

“Potential race condition.”

There may indeed be a race condition.

But the comment assumes the author will reconstruct the reviewer's reasoning.

Where is the race?

Under what interleaving?

Does it produce incorrect data?

Duplicate work?

A crash?

Is it harmless?

Now compare:

“Two requests can both observe that the key is absent before either calls set(). That means fetchValue() can execute twice and the second result can overwrite the first. If fetchValue() has side effects, the current check-then-set sequence isn't sufficient; use an atomic get-or-create operation.”

The second comment isn't merely more verbose.

It exposes the reasoning.

That distinction matters.

A label such as:

[bug]
[performance]
[security]
[style]

answers:

What category did the reviewer assign this observation?

It doesn't necessarily answer:

Why should I change the code?

A label is metadata.

An explanation contains a model of the problem.

What information should a review comment contain?

I don't think there is a universal template.

But useful comments often contain some combination of:

WHAT
What behavior or property is concerning?

WHY
Why does it matter?

CONTEXT
What assumption, constraint, or system behavior makes it important?

CONSEQUENCE
What can happen if it remains unchanged?

NEXT
What should the author investigate or change?

A comment doesn't need all five.

For an obvious typo:

“recieve → receive.”

is probably sufficient.

For a concurrency bug:

“Two requests can both pass this check…”

may need considerably more context.

The amount of explanation should depend on the amount of ambiguity.

Correctness is not the same as usefulness

This is an important distinction.

Suppose a reviewer writes:

“This function is too complex.”

And suppose the function really is too complex.

The statement can be correct while the review remains unhelpful.

The author still has to determine:

Which complexity?
Why is it a problem?
What boundary should change?
Is the reviewer concerned about readability?
Testing?
Runtime behavior?
Future modification?
Something else?

Compare:

“This function now validates input, persists the record, and publishes an event. These operations have different failure semantics, so changing the event behavior now requires modifying persistence logic. Consider separating event publication from the persistence operation.”

The second comment doesn't necessarily prescribe the correct refactoring.

It does something more fundamental:

It makes the reviewer's concern inspectable.

The author can agree.

The author can disagree.

The author can explain why the coupling is intentional.

That is a healthy property of a review comment.

Why “why” matters

Code frequently contains decisions that are not obvious from the code itself.

Imagine:

if user.IsDeleted() {
    return ErrNotFound
}

A reviewer might ask:

“Why return not found instead of forbidden?”

That question could be perfectly reasonable.

But suppose the actual reason is security-sensitive API behavior:

“The public endpoint intentionally returns 404 for deleted users so callers can't distinguish between an existing deleted account and an account that never existed. This is consistent with the behavior of the other account lookup endpoints.”

Now the comment isn't just about an error code.

It transfers context.

This is why established review guidelines emphasize explaining why rather than simply describing what the code does. GitLab's review guidance explicitly recommends explaining rationale and making comments actionable.

Google's code-review guidance similarly asks whether comments are clear and useful and distinguishes substantive concerns from optional style feedback.

The “why” matters because code often cannot contain the entire reasoning behind itself.

But there is a danger in over-explaining

Here is the obvious counterargument:

Could this framework cause reviewers to over-explain everything?

Absolutely.

Imagine receiving this review comment:

“This variable is named i, which could potentially reduce semantic clarity because variable names are an important part of communicating intent. Consider using index instead, since…”

Nobody needs a paragraph for that.

The problem is not lack of context.

The problem is that the context is already obvious.

A useful principle is therefore not:

“More explanation is better.”

It is:

“Enough explanation to remove meaningful ambiguity.”

For a typo, that's almost nothing.

For a distributed-system failure mode, it might be several sentences.

Comment length should follow uncertainty.

Signal versus noise

This becomes especially important when review comments are cheap to generate.

A human reviewer has a natural constraint: attention.

An automated reviewer can potentially produce many observations.

That creates a dangerous feedback loop:

more detection
     ↓
more comments
     ↓
more things to inspect
     ↓
less attention per comment
     ↓
more review fatigue

The problem isn't hypothetical in the broader code-review literature.

Researchers have studied reviewer participation, usefulness of comments, review size, reviewer experience, and other factors that affect modern code-review effectiveness.

For example, Bosu, Greiler, and Bird analyzed approximately 1.5 million review comments from five Microsoft projects to investigate characteristics associated with useful review feedback. Their work found relationships between usefulness and factors including reviewer experience and change characteristics.

The implication isn't that automated review is bad.

It's that review bandwidth is finite.

If a system produces ten additional comments, the relevant question isn't whether all ten are technically defensible.

It's whether the engineer's attention was spent better because they received them.

The preference problem

There is another source of noise that has nothing to do with AI.

Human reviewers do it constantly.

Consider:

“I'd use a map here.”

That might be a good suggestion.

But why?

Maybe the map makes lookup complexity clearer.

Maybe it avoids repeated scans.

Maybe it is simply the reviewer's preferred style.

Those are different claims.

Compare:

“I'd use a map here.”

with:

“We're doing a linear scan for every lookup, so the cost grows with the number of entries. Since this collection is accessed repeatedly, an indexed structure would avoid repeated scans.”

Now the author can evaluate the reasoning.

This distinction between preference and engineering necessity is important.

Google's review guidance essentially argues that when multiple approaches are consistent with sound engineering principles, reviewers should accept the author's choice rather than treating their own preferred implementation as mandatory.

A review culture that treats every preference as a defect creates noise even when every reviewer is acting in good faith.

Five examples

1. Nullability

Weak

“Handle null.”

Stronger

“getSubscription() returns null when the user has no subscription, but this caller immediately reads .plan. That turns a valid empty state into a runtime exception. Handle the empty case before dereferencing the result.”

The second comment exposes the contract and failure mode.

2. Database queries

Weak

“N+1.”

Stronger

“getCustomer() runs a database query inside this loop. With 500 orders, this can result in hundreds of additional queries. Load the required customers before the loop and reuse them here.”

The label tells an experienced engineer what to investigate.

The explanation tells them why this particular code has the problem.

3. Error semantics

Weak

“Don't swallow the error.”

Stronger

“Returning nil here makes a failed request indistinguishable from a valid empty response. The caller treats nil as success, so a network failure could be interpreted as ‘no data.’ Preserve the error so the caller can decide whether to retry or surface it.”

The concern isn't merely “errors are important.”

It's that information is being destroyed.

4. Authorization

Weak

“Security issue.”

Stronger

“Authentication establishes that the user has a valid session, but it doesn't establish that they can modify this document. Since the document isn't checked against the user's project, this path could allow cross-project modification. Verify authorization before the update.”

The reviewer is identifying the missing security boundary.

5. Maintainability

Weak

“This should be refactored.”

Stronger

“This method now handles validation, persistence, retries, and notification. Those operations have different failure modes and are already changing independently. Separating the retry/notification path would keep future notification changes from touching persistence logic.”

The second comment explains why the proposed structural change matters.

What if the reviewer doesn't know the author's intent?

This is where the framework gets uncomfortable.

Suppose a reviewer sees:

if response.status_code == 404:
    return []

They might write:

“This is wrong. A 404 should be an error.”

But perhaps the API deliberately uses 404 to represent “no matching resources.”

The reviewer has made an assumption about intent.

A more useful comment might be:

“Is 404 expected to mean ‘no matching resources’ for this endpoint? If it represents a missing resource instead, returning an empty list here would hide an API error.”

That's a question rather than a declaration.

And sometimes that is the better review comment.

Good review doesn't require the reviewer to pretend they know things they don't know.

Uncertainty should be communicated as uncertainty.

There is a major difference between:

“This is wrong.”

and:

“I may be missing context, but if X is possible, doesn't this create Y?”

The second invites the author to supply missing information.

Review comments are conversations, not verdicts

This is easy to forget when reviews are represented as annotations attached to lines.

The interface makes a comment look like a judgment.

But a review is usually a conversation.

The reviewer proposes an interpretation.

The author provides context.

The two converge on a decision.

That means a useful review comment doesn't necessarily need to prove that the reviewer is right.

It needs to make the disagreement productive.

For example:

“I think this could race if two workers execute this block concurrently. Is there a lock around this operation elsewhere that I'm missing?”

That might be a better comment than a confidently incorrect:

“This has a race condition. Fix it.”

The first comment contains uncertainty.

The second hides it.

What about AI-generated review comments?

AI makes this problem more interesting.

An automated system can examine code at a scale that humans can't.

It can notice patterns.

It can compare implementations.

It can identify suspicious data flows.

It can explain some classes of issues.

But there is a fundamental limitation:

The code is not the entire system.

Intent may live in:

product requirements,
architecture documents,
incident history,
database constraints,
undocumented operational assumptions,
conversations between engineers,
downstream consumers,
or simply someone's knowledge of why the system works the way it does.

An AI looking only at a pull request may not have access to that context.

So consider an automated comment:

“This validation is redundant.”

Maybe it is.

Or maybe the validation exists because an external system violates the contract once every few thousand requests.

Without broader context, the model may correctly understand the local code and incorrectly understand the system.

That's an important distinction.

Local correctness is not necessarily system correctness.

Can AI generate specific comments without understanding the whole system?

Sometimes.

For straightforward cases, specificity may not require deep system-level intent.

For example:

“This value is checked for None here, but dereferenced unconditionally three lines later.”

The evidence is local.

The model doesn't need to understand the entire architecture.

But consider:

“This retry count is too low.”

That might require knowing:

upstream service behavior,
timeout characteristics,
SLA requirements,
idempotency guarantees,
traffic patterns,
and production failure modes.

The more a comment depends on external intent, the less safe it is to infer that intent from code alone.

This suggests a useful boundary for automated review:

The system should distinguish observations supported directly by code from conclusions that depend on missing context.

That could mean expressing uncertainty.

It could mean asking a question.

It could mean requesting additional repository context.

It could mean not commenting at all.

Silence is sometimes better than a plausible but unsupported explanation.

Explanation isn't evidence

There's another subtle failure mode.

A detailed comment can sound authoritative simply because it is detailed.

Consider an AI-generated comment:

“This implementation creates a race condition because the cache is not synchronized across worker threads, which could cause stale reads and inconsistent state…”

It sounds convincing.

But does the system actually know that the workers share the same cache?

Does it know the operation is concurrent?

Does it know stale reads matter?

A long explanation can create false confidence.

So review quality cannot simply become:

“More detailed comments are better.”

The real requirement is:

The explanation should be proportional to the evidence.

A concise, well-supported observation can be more trustworthy than a sophisticated explanation built on assumptions.

The unit of review isn't always the line

Another limitation of line-level comments is that many engineering problems aren't located on one line.

Consider an authorization bug.

The problem might be:

controller
    ↓
service
    ↓
repository
    ↓
database

The individual lines can all look reasonable.

The vulnerability emerges from how the components interact.

Likewise with:

distributed systems,
caching,
transactions,
retries,
consistency,
API contracts,
concurrency,
lifecycle management.

Sometimes the most useful review comment needs to refer to a relationship, not a line.

For example:

“The transaction ends before the event is published, so a successful response can be returned even if event publication fails. Is that intentional? If the event is part of the operation's contract, the transaction/outbox boundary probably needs to move.”

That's difficult to express as a simple label.

It requires a model of the interaction between components.

So what makes a review comment good?

I don't think the answer is “make every comment more detailed.”

Nor is it “always explain the why.”

There are cases where the why is obvious.

Nor is it “never use labels.”

Labels are useful.

The more interesting answer is:

A good review comment contains the minimum information necessary for another engineer to correctly evaluate the concern.

Sometimes that's:

“Typo: recieve → receive.”

Sometimes it's:

“This returns null when X happens, but Y dereferences it immediately, turning a valid state into an exception.”

Sometimes it's a question:

“Is this 404 intentional? If not, are we hiding an API failure by returning an empty list?”

Sometimes the right review is no comment at all.

That last case matters.

A reviewer should not be rewarded simply for finding something to say.

A small framework: WHAT → WHY → NEXT

If I had to give reviewers a practical heuristic, it would be:

WHAT?

What exactly are you observing?

“This can return null.”

WHY?

Why does that matter?

“The caller dereferences it immediately.”

NEXT?

What should the author investigate or change?

“Handle the empty case before accessing the property.”

Together:

“This can return null, but the caller immediately dereferences it, which can turn a valid empty state into a runtime exception. Handle the empty case before accessing the property.”

But the framework should be treated as a diagnostic tool, not a writing template.

If the issue is obvious, don't force an explanation.

If the reviewer is uncertain, don't manufacture certainty.

If the concern depends on system context the reviewer doesn't have, ask.

If the implementation is simply a matter of preference, say so.

The deeper problem

There is a tendency to think of software engineering communication as secondary to the “real” engineering work.

The code is the artifact.

The review comment is just discussion around it.

I'm not convinced that's true.

A codebase contains decisions.

Reviews contain the reasoning behind some of those decisions.

When that reasoning is missing, future engineers have to reconstruct it.

When it is vague, authors have to interpret it.

When it is excessively noisy, people stop paying attention.

And when it is confidently wrong, the review system can actively make engineering decisions worse.

That applies equally to human and automated reviewers.

The hard problem isn't merely detecting something suspicious.

It's knowing what information another engineer needs to evaluate the observation.

Open questions

I don't think we've solved this.

A few questions seem particularly interesting.

How much context is enough?

Can review systems estimate when a comment needs one sentence versus five?

Can usefulness be measured without reducing it to “did the author change the code”?

A correct review may result in no change because the author provides missing context.

How should automated reviewers represent uncertainty?

Should a system say:

“This is a bug.”

or:

“If X is possible, this may produce Y. Is X guaranteed not to happen?”

How much repository context should an AI reviewer consume before making a claim?

More context can improve reasoning, but it can also increase cost and introduce irrelevant information.

Can automated systems distinguish preference from correctness reliably?

Or will they reproduce the same subjective review behavior humans already struggle with?

What happens to review quality when the cost of generating comments approaches zero?

If comments become effectively free, attention becomes the scarce resource.

And perhaps the most basic question:

Should the objective of code review be to find more problems, or to help engineers make better decisions?

Those are related goals.

They aren't necessarily the same.

A label tells us that someone thinks there is a problem.

A useful review comment gives us enough reasoning to decide whether they are right.

That's why a label isn't a code review.

Why AI Code Review Needs to Learn From Developers

Codzee.io — Tue, 08 Sep 2026 11:30:32 +0000

After 15+ years of building software, reviewing pull requests, debugging production systems, and working with engineering teams, I've become increasingly convinced that we're asking the wrong question about AI code review.

The obvious question is:

How accurately can AI detect problems in code?

It's an important question.

But there's another question that becomes more important once a team uses the system for months:

How well does the system understand what this particular team considers worth reviewing?

Those are very different problems.

A reviewer can be technically accurate and still be frustrating to use.

It can identify real complexity, real duplication, real security risks, and real maintainability concerns—and still lose the trust of the engineers receiving its comments.

Why?

Because software engineering isn't just a collection of rules.

It's context, conventions, trade-offs, history, architecture, and judgment.

And much of that knowledge isn't written anywhere.

It's learned through interaction.

That is why I believe the next generation of AI code review needs to learn from developers.

Static Analysis Was Never the Whole Story

We've had static analysis for decades.

Linters are incredibly useful.

They catch formatting problems, suspicious constructs, unused variables, dangerous patterns, complexity issues, and countless other classes of problems.

Static analysis is deterministic and predictable.

That's exactly why engineers like it.

You can define a rule:

if condition X is true
    report finding Y

You can then configure how that finding behaves.

Warning
Error
Info
Ignore

You can add thresholds.

You can exclude directories.

You can configure allowlists and denylists.

You can override rules for specific repositories.

This works remarkably well for problems that can be expressed clearly.

But there's a fundamental limitation:

The configuration describes what we think matters. It doesn't necessarily describe what the team actually considers useful.

Those aren't always the same thing.

Configuration Has a Ceiling

Consider a team working on a large backend system.

They configure a complexity threshold:

complexity > 10 → warning

Seems reasonable.

Then the system starts flagging a particular class of functions.

The team repeatedly dismisses them.

Why?

Maybe those functions are generated around a complex business domain.

Maybe they're deliberately structured that way because splitting them would make the code harder to understand.

Maybe the threshold is appropriate for most of the repository but not this subsystem.

Maybe the team has accepted the complexity because the alternative introduces a worse architectural trade-off.

You can solve this with another configuration rule.

Add an exception.

Then another exception.

Then another.

Eventually, your configuration starts looking like this:

flag X
unless repository Y
unless directory Z
unless pattern A
unless service B
unless annotation C

At some point, we're essentially trying to encode years of engineering judgment into a configuration file.

That's not necessarily the best abstraction.

Engineering Teams Are Dynamic Systems

The bigger problem is that repositories don't stay still.

Neither do teams.

An organization might move from a monolith to services.

A team might adopt a new framework.

A legacy subsystem might become effectively frozen.

A new architectural standard might be introduced.

Performance requirements might change.

Security requirements might become stricter.

The team might deliberately relax a particular style rule because it's slowing development without providing meaningful value.

Configuration tends to represent a snapshot.

Engineering practice is a moving target.

That's why I think AI code review needs something beyond configuration.

It needs a feedback loop.

The Most Interesting Signal Might Be a Dismissal

Here's a simple example.

Suppose an AI review system repeatedly reports:

This function is too complex.

The developer dismisses the comment.

The same pattern appears in another pull request.

The developer dismisses it again.

Then again.

What should happen?

A conventional system records three dismissed findings.

End of story.

But I think we're throwing away valuable information.

Repeated dismissal could mean several things.

1. The rule is too broad

Perhaps the complexity threshold isn't appropriate for this codebase.

2. The system lacks context

The function may look complex locally but be perfectly reasonable given the surrounding architecture.

3. The team has an intentional exception

The engineers may understand the trade-off and have consciously decided to accept it.

4. The feedback is technically correct but practically irrelevant

Something can be objectively improvable without being worth interrupting a developer about.

5. The team may actually be wrong

This possibility is important too.

Learning from developers shouldn't mean blindly accepting every dismissal.

Sometimes an engineer dismisses a legitimate security issue.

Sometimes a team has normalized technical debt.

Sometimes people ignore warnings because fixing them is inconvenient.

So the answer isn't:

"Developer dismissed it, therefore never report it again."

The better question is:

"What does this interaction tell us, and how confident should we be when adapting?"

That's a much more interesting engineering problem.

A Review Comment Is an Interaction

We usually model code review as something like:

Code → Analysis → Finding

But real code review is closer to:

Code
  ↓
Analysis
  ↓
Finding
  ↓
Developer judgment
  ↓
Accept / dismiss / modify / ignore
  ↓
Future behavior

That final step is where a lot of today's systems stop paying attention.

But it may be the most valuable part.

Every interaction provides information.

A developer accepts a finding.

That's a signal.

A developer dismisses it.

That's a signal.

A developer repeatedly modifies recommendations in the same way.

That's a signal.

An entire team consistently ignores a category of feedback.

That's a much stronger signal.

The challenge is turning those signals into useful adaptation without creating unpredictable behavior.

The Repository Shouldn't Be Treated as an Isolated Installation

One thing that bothers me about the current mental model is the idea that every repository starts from zero.

Install the tool.

Configure it.

Run it.

Done.

But an established codebase already contains enormous amounts of engineering knowledge.

You can find it in:

Existing implementation patterns
Pull request discussions
Architectural decisions
Test strategies
Naming conventions
Error-handling approaches
Accepted technical debt
Framework-specific patterns
Exceptions to general standards

Some of this can be explicitly documented.

Much of it isn't.

An AI system operating inside that repository should have the opportunity to understand those patterns.

Not simply analyze the current pull request in isolation.

False Positives Are More Expensive Than They Look

False positives aren't just an accuracy problem.

They're a trust problem.

Imagine a developer receives ten review comments.

Two are genuinely useful.

Eight aren't.

After a while, the developer stops carefully evaluating each comment.

They start scanning.

Eventually they start dismissing automatically.

That's when the system has failed even if its underlying detection model is technically sophisticated.

The cost isn't just the time spent reading irrelevant comments.

The larger cost is signal degradation.

Once developers stop believing that a comment is likely to matter, even good findings become harder to trust.

This is why reducing noise isn't merely a UX improvement.

It is part of the intelligence of the system.

But Adaptation Needs Guardrails

There's an obvious danger here.

If we allow AI systems to learn from developer behavior without constraints, they could learn the wrong lessons.

Imagine a team consistently dismisses security warnings because fixing them is inconvenient.

Should the system become quieter?

Probably not.

Some categories should have stronger guarantees.

Some findings should remain visible regardless of preference.

Some adaptations should require explicit confirmation.

Some changes should be explainable.

For example:

"This finding has been repeatedly dismissed for this repository. Should future findings of this type be deprioritized?"

That's very different from silently changing behavior.

The goal shouldn't be autonomous customization at any cost.

The goal should be safe adaptation.

What Would an Adaptive Review System Look Like?

I don't think this requires some magical autonomous agent.

The underlying architecture could be relatively straightforward.

Imagine maintaining a feedback model for findings.

Something like:

Finding
├── category
├── location
├── repository context
├── confidence
├── severity
├── historical interactions
└── developer/team feedback

Then review decisions become part of the system's future reasoning.

For example:

Finding A
→ accepted 27 times
→ rarely dismissed
→ high relevance

Finding B
→ dismissed 18 times
→ accepted 1 time
→ low relevance in this repository

Finding C
→ dismissed frequently by one team
→ accepted frequently by another
→ highly context dependent

Now the system isn't simply asking:

"Is this technically a problem?"

It can also ask:

"How relevant has this type of finding historically been in this environment?"

That is a fundamentally different signal.

And it can become increasingly repository-specific without requiring engineers to manually encode every preference.

The Goal Isn't Fewer Findings

This distinction matters.

Adaptive code review shouldn't simply become a system that says less.

The objective is higher-value output.

There will be situations where the correct behavior is to become more aggressive.

Suppose a team repeatedly accepts findings related to a particular security pattern.

The system now has evidence that this category matters.

That should increase confidence that similar findings are worth surfacing.

Adaptation should work in both directions.

Developer behavior
       ↓
Observed preference
       ↓
Updated relevance
       ↓
Future review behavior
       ↓
More developer feedback
       ↓
Updated preference

That's a feedback loop.

And feedback loops are how systems become better aligned with their environment.

The Metric We Should Care About

This leads to a question I think the industry should spend more time discussing.

We measure code-review systems by things like:

Detection accuracy
Precision
Recall
False-positive rate
Number of findings
Vulnerability coverage

These metrics are useful.

But they're incomplete.

I'd also want to know:

What percentage of findings does a team consider useful?

And perhaps more importantly:

Does that percentage improve over time?

Imagine two systems.

System A finds 100 issues in the first month and continues finding roughly 100 every month.

System B finds 70 issues initially, learns from developer interactions, and eventually surfaces 30 highly relevant findings.

Which one is better?

The answer depends on what those findings contain.

But for an engineering team, I would strongly prefer the system that earns the right to speak less—and gets better at knowing when to speak.

This Is Bigger Than AI Code Review

There's a broader principle here.

Good developer tools don't merely perform tasks.

They fit into human workflows.

Git became useful because developers could build mental models around it.

IDEs became useful because developers shaped them around their workflows.

Linters became useful because teams could define what mattered.

The next generation of AI tools has another opportunity:

They can learn from the workflow itself.

That's different from traditional configuration.

Configuration says:

"Here is what we told the system."

Behavioral feedback says:

"Here is what the team actually does."

The combination could be much more powerful.

Signal Isn't a Feature. It's a Relationship.

After 15+ years in software engineering, I don't think developer trust comes from a feature checklist.

It comes from repeated evidence.

You make a judgment.

The system observes it.

It adapts.

You make another judgment.

It adapts again.

Over time, the system becomes less generic and more aligned with the environment in which it operates.

That's what makes the idea of adaptive AI code review compelling to me.

The next generation won't compete only on who can detect the most bugs.

They'll compete on who can understand which bugs, patterns, and risks are actually worth surfacing to a particular engineering team.

Because detection gets you into the workflow.

Relevance keeps you there.

And relevance comes from understanding context.

That is why I believe:

Signal isn't a feature. It's a relationship.

The best AI code-review systems will be the ones that earn that relationship carefully—through feedback, context, transparency, and repeated interaction.

Not during installation.

Over time.

And ultimately, the real measure of an AI reviewer won't be how much it can say.

It will be how quickly—and how safely—it earns the team's trust.
:::

How can AI code review tools be evaluated reliably for real-world pull requests?

Codzee.io — Tue, 01 Sep 2026 14:40:12 +0000

I am evaluating AI-assisted code review systems and have found that published benchmarks use substantially different methodologies. Some replay historical bugs, some evaluate complete pull requests, some inject synthetic defects into real code, and others use human or LLM-based judgments of review comments.

I am trying to design a fair evaluation for systems that review real-world pull requests.

The main issues I am running into are:

Dataset construction: Should the test set consist primarily of historical production bugs, clean pull requests, synthetic defects, or a combination?
Ground truth: How should a benchmark define the complete set of valid findings when the original human review may have missed defects?
Precision and recall: Should findings be matched individually, or should a PR count as correct if the system identifies at least one valid issue?
False positives: How should speculative, stylistic, duplicate, or technically correct but non-actionable comments affect the score?
Severity: Should critical security or correctness defects receive more weight than low-severity maintainability issues?
Human review: What level of expert annotation or adjudication is necessary to establish reliable ground truth?
Repository context: Should systems receive the full repository, issue description, tests, commit history, and search capabilities, or should all systems be restricted to the diff?
Languages and projects: How should results be normalized across programming languages, repositories, domains, and different patch sizes?
Reproducibility: What artifacts and experimental details should be published so another team can independently reproduce the evaluation?

For example, a system that produces 20 comments and correctly identifies 10 defects is quite different from one that produces 5 comments and correctly identifies 4, even if their aggregate scores happen to be similar. Likewise, a benchmark based only on human review comments may penalize a system for finding a legitimate defect that the original reviewers missed.

I am looking for a methodology that measures useful defect detection on real pull requests, rather than simply similarity to historical review comments.

What benchmark design and scoring procedure would provide the most statistically and methodologically defensible comparison of AI code review systems, including dataset construction, ground-truth validation, finding-level precision/recall, severity weighting, repository context, human adjudication, and reproducibility?

Your CODEOWNERS File Might Be Correct And Still Send Your PR to the Wrong Person

Codzee.io — Mon, 31 Aug 2026 11:15:03 +0000

You open a PR. GitHub looks at the changed paths, checks them against CODEOWNERS, and auto-assigns a reviewer. This happens so often you probably don't think about it anymore. It's infrastructure — it just works.

Except sometimes it doesn't, and it fails in a way that never throws an error.

Here's a realistic example:

# CODEOWNERS
/payments/  @engineer-a

Engineer A hasn't touched payments/ in eight months — they moved to a different team in Q1. Engineer B has made the last 18 commits in that directory and has reviewed most of the recent PRs that touched it. Engineer B isn't in the file.

Your PR touches payments/refunds.go. It gets routed to Engineer A.

Nothing is misconfigured. The glob pattern matches. The username resolves to a real, active employee. GitHub does exactly what it's supposed to do. The routing logic is just missing context it was never designed to have.

This is the gap this article is about: CODEOWNERS tells you who owns a file. It doesn't tell you who should review a specific change. Those turn out to be different questions, and the difference gets bigger as your team and codebase grow.

1. What CODEOWNERS Gets Right

Before picking it apart, it's worth being clear that CODEOWNERS solves a real problem well, and nothing below is an argument for ripping it out.

Explicit ownership. Anyone can look at one file and see who's accountable for a given path. No tribal knowledge required.
Required reviews. You can enforce that certain paths always get eyes from the right team before merge — critical for things like auth, billing, or infra-as-code.
Clear accountability. When something in /payments/ breaks, there's no ambiguity about who's supposed to be paying attention to that code.
Simple, predictable behavior. It's a text file with glob patterns. No ML model, no black box, no surprises. You can git blame it.

For a small team, or for stable, slow-moving parts of a codebase, this is close to sufficient. The problem shows up specifically as team size and code churn increase — which is exactly when the cost of getting routing wrong also goes up.

2. Where Static Routing Breaks

CODEOWNERS answers one question — who owns this path — and organizations often treat that as a proxy for a different question: who should review this change. Here's where those diverge:

Signal	Static CODEOWNERS	Dynamic routing
File ownership	Yes	Yes
Recent contribution	No	Yes
Current expertise	Limited	Yes
Review history	No	Yes
Reviewer load	No	Yes
Expertise decay	No	Yes

Walking through the "No" column:

Recent contribution — CODEOWNERS has no time dimension. An entry from two years ago carries the same weight as one from yesterday.
Current expertise — "Limited" because ownership is often a reasonable proxy for expertise at the moment the file was written, but it's frozen after that. It doesn't update when someone's knowledge of an area grows or fades.
Review history — the file doesn't know who's actually been reviewing changes in an area, only who's nominally responsible.
Reviewer load — https://www.codzee.io/will happily assign a fifth simultaneous PR to someone already buried in review requests. It has no concept of "currently busy."
Expertise decay — there's no mechanism for an owner's relevance to degrade over time as they stop working in an area, and no mechanism for someone else's relevance to rise as they start.

None of this is a criticism of CODEOWNERS as a tool — it's just outside its scope. It's a static ownership file, not a reviewer-relevance model. The mistake is using it as if it were the second thing.

3. Model Expertise as Something That Changes

The core idea: expertise isn't binary, and it isn't permanent. "Alice owns payments" is a snapshot. What you actually want to know is closer to "how relevant is Alice's expertise, right now, to this specific diff."

Useful inputs for that:

Last contribution date — when did this person last touch this area?
Number of recent relevant changes — not lifetime commits, recent ones. Someone with 3 commits last month is a stronger current signal than someone with 200 commits two years ago.
Similarity to the current PR — has this person touched the same files, the same functions, the same data model recently?
Review activity — have they been reviewing PRs in this area, even if they haven't authored changes there?
Historical ownership — were they ever formally responsible for this, even if the file has since changed?

You can express this as a rough conceptual model — not a formula anyone should treat as an industry standard, just a way to make the tradeoff explicit:

reviewer_relevance = ownership_weight
                    + expertise_weight
                    + recency_weight
                    - load_penalty

The actual weights are going to be organization-specific, and honestly team-specific within an org. A security-critical path might want ownership weighted heavily no matter what. A fast-moving product surface might want recency and review history to dominate. The point of writing it this way isn't to hand you a formula — it's to make explicit that "who reviews this" is a function of multiple decaying/growing signals, not a single static lookup.

4. Work Through a Sample PR

Say you've got a repo with three top-level domains:

/payments/
/identity/
/notifications/

A PR comes in touching payments/refunds.go. CODEOWNERS says payments/ belongs to Engineer A. Here's how three candidate reviewers actually stack up:

Reviewer	Ownership	Recent work	Similar reviews	Current load
A	High	Low	Medium	High
B	None	High	High	Medium
C	Medium	Medium	Low	Low

Engineer A is the listed owner, but hasn't done recent work in this directory, and is already carrying a heavy review load.
Engineer B isn't in CODEOWNERS at all, but has been actively working in this area recently and has reviewed similar changes — they've probably seen the exact class of bug this PR could introduce.
Engineer C has moderate ownership and moderate recent activity, low similarity to this specific change, but has bandwidth right now.

A purely static system routes to A every time, because A is the only name it knows how to check against. But on the actual signals that predict a useful review — recent, hands-on context and demonstrated familiarity with similar changes — B is the stronger candidate, and C is a reasonable fallback if B is unavailable.

This doesn't mean A should be removed from the review entirely — for something in a sensitive path, keeping the formal owner in the loop for accountability might still make sense. It means A shouldn't be the only signal the system checks.

5. A Practical Routing Framework

Distilling this into something you could actually reason about (or build) for your own repo:

Ownership — who's formally accountable, per CODEOWNERS or equivalent?
Recency — who's touched this code lately?
Expertise — who has a track record of relevant, substantive contributions here?
Review history — who's reviewed similar changes before, and reviewed them well (not just approved fast)?
Load — who has the bandwidth to actually do this review in a reasonable timeframe?

None of these should unilaterally win. Ownership keeps governance intact — you don't want a recency model quietly routing security-sensitive changes away from the team accountable for them. Load is a modifier, not a primary filter — someone who's slightly busy but the clear best fit is often still the right call over someone who's fully free but has no context. The five signals are meant to be combined, with the weighting depending on what kind of change you're looking at (a one-line config tweak needs way less of this than a cross-cutting refactor).

6. The Routing-Decay Loop

Here's the pattern that causes CODEOWNERS files to quietly go stale, laid out as a loop:

People change teams / roles
        ↓
Code changes (new services, refactors, deprecations)
        ↓
Expertise shifts (some people gain context, others lose it)
        ↓
Ownership file stays the same (nobody's job to update it proactively)
        ↓
Routing quality declines

Nothing in that loop is anomalous — it's just normal organizational and codebase churn. The failure isn't that people move around or that code changes. The failure is that CODEOWNERS has no way to hear about any of it. It's a file, not a feed.

The fix isn't "update the file more often" — that's a losing battle against how fast real orgs change. It's closer to: continuously derive dynamic signals from data you already have (commit history, PR review history, recent activity) and layer them on top of the static file, rather than trying to keep the static file itself perfectly current. Contribution and review data refresh automatically as people work — no one has to remember to edit a YAML-adjacent file when Bob becomes the de facto payments expert. The signal is already sitting in your git history and PR review logs; it's just not being used for routing.

7. What Teams Can Measure Today

You don't need new tooling to start getting visibility into how well your current routing is actually working. Most of this is derivable from data already in GitHub:

Reviewer reassignment rate — how often does the auto-assigned reviewer get swapped out before the PR merges? High rate = your default routing isn't matching reality.
Time to first meaningful review — not first comment, first substantive review. If the assigned reviewer routinely takes days to engage, they may be overloaded or not the right fit.
Review load concentration — what percentage of review volume in a given area lands on one person? If it's consistently one name, you likely have a bottleneck, whether or not that person is technically the best expert.
PRs routed to inactive contributors — a rough version of this is: for each CODEOWNERS entry, check git log --since="6 months ago" -- <path> and see if the listed owner shows up. If not, that entry is a candidate for drift.
Review handoffs — PRs where the review gets passed from one person to another before merge are a decent proxy for "the first routing decision missed."
Recent contribution coverage — what share of merged PRs in an area were reviewed by someone who'd actually committed to that area in the last N weeks?

None of these require a new system to start tracking. A few are a git log or a GitHub API query away. The value isn't in any single metric — it's in noticing, over a few months, whether your routing is drifting in a direction nobody's paying attention to.

8. Practical Takeaway

None of this is an argument for dropping CODEOWNERS. It's still the right mechanism for encoding accountability — who's on the hook for a given part of the codebase, and which teams need guaranteed visibility into changes that touch their area.

What it's not sufficient for, on its own, is figuring out who should actually review a given PR today.

Keep CODEOWNERS for accountability. Add dynamic signals for relevance. Ownership answers "who's responsible." Recency, expertise, review history, and load together answer "who's actually the right person to look at this, right now." A routing setup that only asks the first question will keep sending PRs to people who used to be the right answer. A routing setup that ignores the first question entirely gives up the governance guarantees CODEOWNERS exists to provide. You want both.

How are you handling reviewer routing today — static CODEOWNERS, team rotation, tribal knowledge ("just tag whoever's around"), or something more dynamic? Curious what's actually working at different team sizes — drop your setup in the comments.

Our AI code reviewer problem wasn't that it was wrong. It was that it wouldn't shut up

Codzee.io — Thu, 27 Aug 2026 13:16:37 +0000

I've been thinking about a problem with AI code review that I don't see discussed enough: a reviewer can be technically correct and still make the whole review experience worse.

Here's the PR that made this click for me. Small change, maybe 200 lines, nothing exotic. New validation path, a couple helper functions. The kind of thing that used to get a comment or two and an approve.

The AI reviewer came back with something like a dozen comments. A naming suggestion. A "this null check might be redundant" note. A theoretical race condition that would only show up under conditions the code was never going to hit in prod. A "you could extract this into a function" suggestion on a function that was already about ten lines long.

And here's the annoying part most of these weren't wrong. The variable name genuinely could've been clearer. The null check probably was redundant. Individually, I could defend almost every comment on that list.

But there was one comment in there about an actual edge case in the validation logic that genuinely mattered. And it was sitting in the middle of eleven other comments that didn't. The dev opening the PR skimmed most of it, dismissed a few without really reading them, and nearly skipped the important one too — because by comment seven or eight, he'd already stopped expecting any of it to be worth his time.

Nobody wanted "more thorough." They wanted to know what actually mattered.

That's the distinction I think gets missed constantly: "is this technically an issue" and "is this worth interrupting someone for" are completely different questions, and most tooling in this space only ever answers the first one.

The assumption baked into a lot of AI review tools is basically: if finding more issues is good, finding even more must be better. But review was never just an issue-detection problem, it's a signal-to-noise problem, and honestly kind of a trust problem too. Once a reviewer starts treating a PR like a checklist to maximize, it stops mattering how accurate any individual comment is — the aggregate experience is what people remember, and the aggregate experience was "annoying."

That frustration eventually became one of the reasons I started working on Codzee. Not going to turn this into a pitch — the short version is the goal isn't "find everything," it's figuring out what actually deserves a developer's attention. Still early, still figuring a lot of it out.

Genuinely curious how other people think about this though:

How many AI review comments do you actually want to see on a normal PR? Is there a number where it tips from "helpful" to "annoying" for you?
Do you find yourself tuning out AI-generated comments after a tool has burned you with noise a few times?
Where's your personal line between a useful warning and something that's just noise dressed up as a finding?
Would you rather a reviewer miss a low-confidence issue entirely, or flag it anyway and let you decide?
What actually makes you trust an automated reviewer, versus just tolerate it?

Not trying to manufacture a debate here, I'm actually unsure about some of these myself — like the miss-vs-flag question, I go back and forth on what the right default even is. Curious what this sub thinks.