DEV Community: Codzee.io

what if Dismissed an AI Code Review Comment. Then I Checked the Next PR.

Codzee.io — Mon, 21 Sep 2026 13:30:50 +0000

I dismissed an AI code review comment.

Nothing unusual there.

What was interesting was what happened in the next pull request.

A few PRs later, a similar pattern appeared in the code. I expected the same review comment.

It wasn't there.

That small behavior points to a bigger question for AI-assisted code review:

What should an AI reviewer do when an engineer disagrees with it?

If the answer is simply “nothing,” the reviewer stays stateless.

If the answer is “learn from the decision,” review behavior can become more aligned with the repository over time.

The annoying part of automated review

AI code review can catch things that are easy to miss.

But there is another failure mode: repetition.

A reviewer finds something that looks questionable according to a general coding guideline.

The developer knows the pattern is intentional.

The comment gets dismissed.

Then another PR comes along.

The same pattern appears.

The reviewer raises the same concern.

The developer dismisses it again.

And again.

At some point, the problem isn't that the reviewer cannot find issues.

The problem is that it hasn't learned which issues this repository doesn't need to discuss repeatedly.

That is where dismissal-learning becomes interesting.

PR #1: The first review

Imagine a repository where a particular implementation pattern is intentional.

An AI reviewer sees it and leaves a comment:

AI reviewer:

Consider changing X to Y for better consistency.

The comment is reasonable in isolation.

But the repository has a reason for using X.

So the engineer dismisses the comment:

Dismissed.

This pattern is intentional in this repository.

Normally, that interaction is over.

The comment disappears from the PR.

But the dismissal contains useful information.

A dismissal is engineering feedback

A dismissal doesn't necessarily mean:

“The reviewer is useless.”

It can mean:

“This recommendation doesn't apply to this codebase.”

That's an important distinction.

Repositories have local conventions that aren't always obvious from the code itself.

Some exist because of:

compatibility requirements
architectural decisions
performance constraints
legacy integrations
operational considerations
team conventions
deliberate trade-offs

A generic reviewer doesn't automatically know all of these.

The developer does.

So when a developer dismisses a review comment, that decision can be treated as feedback about the repository.

The interesting part is what happens next.

Turning the dismissal into repository memory

Instead of storing only the conversation, the system can extract the underlying pattern from the interaction.

Conceptually:

Review comment
      ↓
Developer dismissal
      ↓
Underlying pattern identified
      ↓
Repository memory
      ↓
Future reviews use that context

For example:

Repository memory:

"This pattern is accepted/preferred in this repository."

The important word here is repository.

This isn't a global rule saying that the pattern is always correct.

It is context associated with this particular codebase.

That's important because engineering decisions are rarely universal.

A pattern that is discouraged in one repository may be completely intentional in another.

PR #2: The same situation appears again

Now a different pull request arrives.

The code contains a similar pattern.

Without repository memory, the reviewer might produce:

AI reviewer:

Consider changing X to Y for better consistency.

Which leads to another dismissal.

With the previous feedback available as repository context, the reviewer can recognize that this type of comment has already been rejected for this repository.

So the result is:

AI reviewer:

[No comment]

That's the part I find interesting.

The system didn't need another configuration file.

The developer didn't need to write a permanent rule manually.

A previous engineering decision influenced a future review.

Before vs. after

Situation	Without dismissal-learning	With repository memory
First PR	Review comment appears	Review comment appears
Developer response	Dismisses comment	Dismisses comment
Feedback retained	Little or no context	Pattern becomes repository context
Similar later PR	Same comment may return	Reviewer can stay quiet
Developer experience	Repeated explanation	Less repeated feedback

This doesn't mean every similar comment should disappear.

Context still matters.

But it creates the possibility of a reviewer that becomes better aligned with the repository's actual engineering decisions.

This isn't “remembering a conversation”

This distinction is worth making.

When people hear “AI memory,” they often think about conversational memory:

“Remember what I told you yesterday.”

That's not the interesting part here.

The useful unit is the engineering decision.

The system needs to understand something closer to:

Pattern:
    X is used in this repository.

Developer decision:
    This usage is intentional.

Review implication:
    Don't repeatedly flag this pattern in the same context.

That's repository memory.

It turns feedback from an individual review interaction into context that can influence future reviews.

Why this feedback loop matters

Code review isn't just about detecting problems.

It's also about communicating what a team considers important.

Human reviewers naturally build this context over time.

After reviewing hundreds of pull requests, an experienced engineer knows:

which patterns are normal
which warnings are usually noise
which conventions are intentional
which architectural decisions shouldn't be revisited every PR

An AI reviewer that doesn't retain any of that context has to rediscover the same information repeatedly.

That creates review fatigue.

A feedback loop changes the model:

Find something
     ↓
Developer responds
     ↓
Learn from the response
     ↓
Apply the context later
     ↓
Produce more relevant feedback

The goal isn't fewer comments for the sake of fewer comments.

It's higher-signal comments.

There are caveats

Dismissal-learning shouldn't turn every dismissal into a permanent rule.

A developer can dismiss something because the comment is wrong.

They can also dismiss it because the context is different in that particular file or PR.

The architecture may change later.

A pattern that was acceptable six months ago might become a technical debt issue today.

So repository memory needs scope and context.

The right behavior isn't:

“This was dismissed once, never mention it again.”

It's closer to:

“Previous engineering feedback says this pattern is acceptable here. Consider that context before raising the same observation again.”

That's a much more useful mental model.

The interesting shift

AI code review is often discussed in terms of how much it can detect.

I think another question deserves equal attention:

How well can it learn what not to say?

A reviewer that finds everything but repeatedly generates low-value comments can still create friction.

A reviewer that incorporates developer feedback can gradually become more aligned with the repository.

That's what makes this behavior interesting.

One comment gets dismissed.

The underlying pattern becomes repository memory.

A similar situation appears later.

The reviewer stays quiet.

Not because the reviewer has learned everything.

Not because repeated comments magically disappear.

But because a real engineering decision from the repository has become part of the context for the next review.

That's what “gets quieter” actually means.

If you want to see this behavior rather than read about it, the most useful test is simple: try it on a repository with an established review convention and watch what happens across two similar PRs.

See the mechanism in action on your own repository.

How to Review AI-Generated Code Without Losing Your Mind

Codzee.io — Thu, 17 Sep 2026 12:53:01 +0000

AI can generate a working implementation in seconds.

That doesn't mean the implementation is correct.

For developers, this creates a slightly different code-review problem. You're no longer reviewing only code written from scratch by someone who understands the surrounding system. You're often reviewing code generated from a prompt, existing examples, partial context, or a combination of all three.

The result can compile, pass tests, and look perfectly reasonable while still being wrong.

The answer isn't to distrust every line of AI-generated code. It's to review it differently.

This article presents a practical approach for reviewing AI-generated changes without spending an hour reading every line with equal attention.

1. Why AI-Generated Code Is Different

AI-generated code has one particularly important characteristic:

It can be locally convincing while being globally incorrect.

Consider a simple API endpoint:

@app.get("/users/{user_id}")
def get_user(user_id: int, db: Session = Depends(get_db)):
    user = db.query(User).filter(User.id == user_id).first()

    if not user:
        raise HTTPException(status_code=404, detail="User not found")

    return user

At first glance, this looks fine.

But several questions immediately matter:

Is the endpoint authenticated?
Can any authenticated user retrieve another user's data?
Is User safely serializable?
Does returning the ORM object expose fields that shouldn't leave the API?
Is there an authorization requirement that isn't represented in this function?

The code itself doesn't contain enough information to answer those questions.

That's common with AI-generated code.

The model optimizes for producing a plausible implementation from the available context. It doesn't automatically understand your organization's security model, undocumented business rules, production traffic patterns, or architectural constraints.

So code review needs to evaluate two things:

What does this code do?

and

What assumptions does this code make?

The second question is often where the important bugs are.

2. What Should Reviewers Check First?

Don't start by reading every line.

Start with the change itself.

Before looking at implementation details, answer:

What problem is this PR solving?

If the PR description says:

"Add caching to improve API performance."

you should immediately ask:

What endpoint?
What data is cached?
How long?
What invalidates it?
Is stale data acceptable?
Is the cache shared between users?

Then look at the diff.

A useful first pass is:

Understand the intended behavior.
Identify the files changed.
Look for changes to security boundaries.
Look for database and external-service interactions.
Look for new state or caching.
Check error-handling paths.
Only then inspect implementation details.

This prevents a common review failure: spending ten minutes debating naming while missing a broken authorization check.

3. Logic and Edge Cases

AI-generated code frequently handles the happy path well.

The interesting bugs tend to live elsewhere.

Suppose an AI-generated function processes payments:

function calculateRefund(order, requestedAmount) {
    if (requestedAmount <= order.total) {
        return requestedAmount;
    }

    return order.total;
}

Looks harmless.

But what happens with:

calculateRefund(order, -100);

The function returns -100.

What about floating-point currency?

0.1 + 0.2 !== 0.3

What about an already-refunded order?

What about concurrent refund requests?

What about an order whose total has changed after partial refunds?

The important question isn't:

"Does this function work?"

It's:

"Under what conditions does this function stop working?"

For AI-generated code, explicitly test:

Empty input
Null/undefined values
Zero values
Negative values
Very large values
Duplicate requests
Concurrent requests
Missing records
Deleted records
Partial failures
Timeouts
Retries
Invalid state transitions

Consider this common pattern:

if not user:
    create_user(email)

That looks reasonable until two requests arrive simultaneously.

Both requests can observe that the user doesn't exist.

Both attempt to create it.

Now you've discovered a race condition.

The database should usually enforce the invariant as well:

CREATE UNIQUE INDEX users_email_unique
ON users(email);

Then the application needs to handle the constraint violation appropriately.

Don't assume the generated code has considered concurrency simply because the sequential logic looks correct.

4. Security

Security deserves an earlier pass than ordinary correctness.

AI-generated code can accidentally introduce vulnerabilities because security depends heavily on context.

SQL injection

Bad:

query = f"SELECT * FROM users WHERE email = '{email}'"
db.execute(query)

Better:

query = text("SELECT * FROM users WHERE email = :email")
db.execute(query, {"email": email})

But parameterized queries aren't the end of the review.

Ask:

Is authorization enforced?
Is sensitive information logged?
Are secrets exposed?
Is user-controlled input trusted?
Are uploaded files validated?
Are redirects controlled?
Are permissions checked server-side?
Are internal errors exposed to clients?

For example:

app.get("/admin/users", authenticate, async (req, res) => {
    const users = await getUsers();
    res.json(users);
});

Authentication exists.

Authorization might not.

A reviewer should ask whether req.user actually has permission to access the endpoint.

Logging is another common problem

An AI-generated debugging statement might look innocent:

logger.info("Payment request: %s", request.json)

If the request contains card information, authentication tokens, addresses, or other sensitive fields, you've just created a data-exposure problem.

Review security based on data flow, not just individual functions.

5. Tests

Passing tests don't prove that generated code is correct.

They prove that the tested scenarios currently behave as expected.

AI-generated tests can have another problem: they may simply encode the implementation rather than validate the requirement.

For example:

def test_discount():
    result = calculate_discount(100, 10)
    assert result == 90

Useful, but incomplete.

What about:

calculate_discount(100, 0)
calculate_discount(100, 100)
calculate_discount(100, -10)
calculate_discount(100, 110)
calculate_discount(0, 10)

More importantly, ask whether the tests represent actual business requirements.

A strong review checks:

Coverage of behavior

Not just line coverage.

You want coverage of:

Happy paths
Failure paths
Boundary conditions
Authorization
State transitions
Retries
Timeouts
Concurrency where relevant

Test independence

Watch for tests that depend on execution order or shared mutable state.

Mock realism

AI-generated tests often mock everything:

mock_db.return_value = fake_user
mock_api.return_value = {"status": "ok"}

The test may pass while the real integration is broken.

Mocks should isolate behavior intentionally, not hide the behavior you're supposed to verify.

6. Architecture

This is where human review becomes particularly important.

AI is very good at producing an implementation that fits the immediate request.

It isn't necessarily good at knowing whether that implementation belongs in your architecture.

Imagine a service that adds:

const result = await database.query(...);
const response = await externalApi.call(...);
await cache.set(...);
await sendEmail(...);

All inside one HTTP controller.

Each operation may work.

The architectural problem is that one request handler now owns:

persistence
external integration
caching
notifications
business logic

The PR might technically work while making the system harder to evolve.

Ask:

Does this belong in this layer?
Are responsibilities separated?
Is existing architecture being bypassed?
Does this introduce a new abstraction unnecessarily?
Does this duplicate existing functionality?
Does this create a new dependency?
Does the data flow still match the system's boundaries?

AI-generated code often creates new helpers or abstractions when an existing one already exists.

Search the repository before approving new infrastructure.

The best implementation may already be somewhere in the codebase.

7. Maintainability

Readable code isn't necessarily maintainable code.

Look for unnecessary complexity.

For example:

const shouldProcess =
    user &&
    user.status &&
    user.status !== "disabled" &&
    (!user.deletedAt || user.deletedAt === null) &&
    permissions &&
    permissions.includes("write");

The code works, but the condition is difficult to reason about.

Could the domain logic be clearer?

function canWrite(user, permissions) {
    if (!user || user.status === "disabled") {
        return false;
    }

    if (user.deletedAt) {
        return false;
    }

    return permissions?.includes("write") ?? false;
}

During review, consider the next developer.

Will they understand why this code exists?

Can they modify it safely?

Does it introduce duplicated logic?

Does the naming describe the domain?

Is there enough context around non-obvious decisions?

AI can produce syntactically clean code very quickly. That doesn't automatically make the codebase easier to maintain.

8. Common AI-Review Mistakes

There are several ways reviewers can make AI-generated PRs harder to review.

Mistake 1: Reviewing every line equally

Not every line has the same risk.

Focus attention on:

Security boundaries
Database mutations
Authentication/authorization
Concurrency
External APIs
State changes
Error handling

Mistake 2: Trusting green tests

Tests are evidence.

They're not proof.

Ask what isn't tested.

Mistake 3: Focusing on style before behavior

Don't spend ten comments discussing variable naming while missing incorrect business logic.

Automate formatting and linting wherever possible.

Mistake 4: Assuming generated code understands your system

It doesn't have your entire organizational context.

Review assumptions explicitly.

Mistake 5: Ignoring PR size

A 1,500-line AI-generated PR is difficult to review regardless of how clean the code looks.

Breaking changes into smaller PRs makes correctness easier to establish.

Mistake 6: Asking AI to review AI without verification

An AI reviewer can be useful for finding patterns humans might miss.

But its findings still require validation.

Don't replace human judgment with another generated judgment.

9. A Practical AI-Code Review Checklist

Before approving an AI-assisted PR, run through this checklist.

Context

[ ] Do I understand what problem this PR solves?
[ ] Is the scope clear?
[ ] Is the PR reasonably sized?

Logic

[ ] Does the implementation match the requirement?
[ ] What assumptions does the code make?
[ ] What happens with empty, invalid, or extreme input?
[ ] Are race conditions possible?
[ ] Are retries and duplicate requests safe?

Security

[ ] Is authentication correct?
[ ] Is authorization enforced?
[ ] Is user input handled safely?
[ ] Are secrets protected?
[ ] Could sensitive information reach logs?
[ ] Are errors exposing internal information?

Data

[ ] Are database queries safe?
[ ] Are transactions required?
[ ] Are constraints enforced at the database level?
[ ] Could concurrent requests corrupt state?
[ ] Is caching introducing stale or cross-user data?

Tests

[ ] Are important behaviors tested?
[ ] Are edge cases covered?
[ ] Are failure paths tested?
[ ] Are tests testing requirements rather than implementation details?
[ ] Are mocks hiding important integration behavior?

Architecture

[ ] Does the change fit existing patterns?
[ ] Is there already code that solves this problem?
[ ] Are responsibilities in the right layer?
[ ] Does the change introduce unnecessary dependencies?

Maintainability

[ ] Is the code understandable?
[ ] Is complexity justified?
[ ] Are names meaningful?
[ ] Is duplication introduced?
[ ] Will another developer understand the reasoning six months from now?

The Goal Isn't to Review More. It's to Review Better.

AI changes the economics of writing code.

When producing another function, test, refactor, or API endpoint becomes dramatically cheaper, teams can generate changes faster than humans can carefully inspect them.

That creates a new bottleneck:

engineering attention.

The answer isn't to manually inspect every generated line forever.

It's to make review more deliberate.

Start with the highest-risk behavior.

Question assumptions.

Test boundaries.

Trace data.

Inspect security boundaries.

Understand architectural consequences.

And keep changes small enough that another engineer can actually understand them.

AI can make code generation cheap.

Human attention is still expensive.

Code review should spend that attention where it has the highest chance of preventing something from reaching production.

A Label Isn't a Code Review

Codzee.io — Tue, 15 Sep 2026 10:53:47 +0000

Consider two comments on the same piece of code.

The first:

“Consider refactoring this function for readability.”

The second:

“This returns null when no active subscription exists, but the caller immediately dereferences .plan. That turns a valid ‘no subscription’ state into an exception. Handle the empty case before accessing plan.”

The first comment might be correct.

The function might genuinely be difficult to read.

But it leaves an important question unanswered:

What is the engineer supposed to do with that information?

The second comment contains considerably more information. It identifies a behavior, connects it to another part of the system, describes the failure mode, and suggests a direction.

That seems straightforward.

But it raises a more interesting question:

What information does a code review comment actually need to contain to be useful?

Code review isn't just defect detection

It is tempting to think about code review as a simple pipeline:

code
  ↓
reviewer
  ↓
problem detected
  ↓
comment
  ↓
code changed

But modern code review is doing several things simultaneously.

It can catch defects.

It can question design decisions.

It can communicate project conventions.

It can transfer knowledge between engineers.

It can document decisions.

It can expose assumptions that aren't visible in the code itself.

Research on modern code review has found that reviewers and authors value activities beyond defect detection, including knowledge sharing and increasing awareness of the codebase.

A Microsoft Research study of peer review also found that review is a form of knowledge exchange in which design rationale can be surfaced during the discussion.

So a review comment isn't merely a defect marker.

It is a small communication channel between two people who may have very different amounts of context.

That changes what “good” means.

Flagging isn't explaining

Consider:

“Potential race condition.”

There may indeed be a race condition.

But the comment assumes the author will reconstruct the reviewer's reasoning.

Where is the race?

Under what interleaving?

Does it produce incorrect data?

Duplicate work?

A crash?

Is it harmless?

Now compare:

“Two requests can both observe that the key is absent before either calls set(). That means fetchValue() can execute twice and the second result can overwrite the first. If fetchValue() has side effects, the current check-then-set sequence isn't sufficient; use an atomic get-or-create operation.”

The second comment isn't merely more verbose.

It exposes the reasoning.

That distinction matters.

A label such as:

[bug]
[performance]
[security]
[style]

answers:

What category did the reviewer assign this observation?

It doesn't necessarily answer:

Why should I change the code?

A label is metadata.

An explanation contains a model of the problem.

What information should a review comment contain?

I don't think there is a universal template.

But useful comments often contain some combination of:

WHAT
What behavior or property is concerning?

WHY
Why does it matter?

CONTEXT
What assumption, constraint, or system behavior makes it important?

CONSEQUENCE
What can happen if it remains unchanged?

NEXT
What should the author investigate or change?

A comment doesn't need all five.

For an obvious typo:

“recieve → receive.”

is probably sufficient.

For a concurrency bug:

“Two requests can both pass this check…”

may need considerably more context.

The amount of explanation should depend on the amount of ambiguity.

Correctness is not the same as usefulness

This is an important distinction.

Suppose a reviewer writes:

“This function is too complex.”

And suppose the function really is too complex.

The statement can be correct while the review remains unhelpful.

The author still has to determine:

Which complexity?
Why is it a problem?
What boundary should change?
Is the reviewer concerned about readability?
Testing?
Runtime behavior?
Future modification?
Something else?

Compare:

“This function now validates input, persists the record, and publishes an event. These operations have different failure semantics, so changing the event behavior now requires modifying persistence logic. Consider separating event publication from the persistence operation.”

The second comment doesn't necessarily prescribe the correct refactoring.

It does something more fundamental:

It makes the reviewer's concern inspectable.

The author can agree.

The author can disagree.

The author can explain why the coupling is intentional.

That is a healthy property of a review comment.

Why “why” matters

Code frequently contains decisions that are not obvious from the code itself.

Imagine:

if user.IsDeleted() {
    return ErrNotFound
}

A reviewer might ask:

“Why return not found instead of forbidden?”

That question could be perfectly reasonable.

But suppose the actual reason is security-sensitive API behavior:

“The public endpoint intentionally returns 404 for deleted users so callers can't distinguish between an existing deleted account and an account that never existed. This is consistent with the behavior of the other account lookup endpoints.”

Now the comment isn't just about an error code.

It transfers context.

This is why established review guidelines emphasize explaining why rather than simply describing what the code does. GitLab's review guidance explicitly recommends explaining rationale and making comments actionable.

Google's code-review guidance similarly asks whether comments are clear and useful and distinguishes substantive concerns from optional style feedback.

The “why” matters because code often cannot contain the entire reasoning behind itself.

But there is a danger in over-explaining

Here is the obvious counterargument:

Could this framework cause reviewers to over-explain everything?

Absolutely.

Imagine receiving this review comment:

“This variable is named i, which could potentially reduce semantic clarity because variable names are an important part of communicating intent. Consider using index instead, since…”

Nobody needs a paragraph for that.

The problem is not lack of context.

The problem is that the context is already obvious.

A useful principle is therefore not:

“More explanation is better.”

It is:

“Enough explanation to remove meaningful ambiguity.”

For a typo, that's almost nothing.

For a distributed-system failure mode, it might be several sentences.

Comment length should follow uncertainty.

Signal versus noise

This becomes especially important when review comments are cheap to generate.

A human reviewer has a natural constraint: attention.

An automated reviewer can potentially produce many observations.

That creates a dangerous feedback loop:

more detection
     ↓
more comments
     ↓
more things to inspect
     ↓
less attention per comment
     ↓
more review fatigue

The problem isn't hypothetical in the broader code-review literature.

Researchers have studied reviewer participation, usefulness of comments, review size, reviewer experience, and other factors that affect modern code-review effectiveness.

For example, Bosu, Greiler, and Bird analyzed approximately 1.5 million review comments from five Microsoft projects to investigate characteristics associated with useful review feedback. Their work found relationships between usefulness and factors including reviewer experience and change characteristics.

The implication isn't that automated review is bad.

It's that review bandwidth is finite.

If a system produces ten additional comments, the relevant question isn't whether all ten are technically defensible.

It's whether the engineer's attention was spent better because they received them.

The preference problem

There is another source of noise that has nothing to do with AI.

Human reviewers do it constantly.

Consider:

“I'd use a map here.”

That might be a good suggestion.

But why?

Maybe the map makes lookup complexity clearer.

Maybe it avoids repeated scans.

Maybe it is simply the reviewer's preferred style.

Those are different claims.

Compare:

“I'd use a map here.”

with:

“We're doing a linear scan for every lookup, so the cost grows with the number of entries. Since this collection is accessed repeatedly, an indexed structure would avoid repeated scans.”

Now the author can evaluate the reasoning.

This distinction between preference and engineering necessity is important.

Google's review guidance essentially argues that when multiple approaches are consistent with sound engineering principles, reviewers should accept the author's choice rather than treating their own preferred implementation as mandatory.

A review culture that treats every preference as a defect creates noise even when every reviewer is acting in good faith.

Five examples

1. Nullability

Weak

“Handle null.”

Stronger

“getSubscription() returns null when the user has no subscription, but this caller immediately reads .plan. That turns a valid empty state into a runtime exception. Handle the empty case before dereferencing the result.”

The second comment exposes the contract and failure mode.

2. Database queries

Weak

“N+1.”

Stronger

“getCustomer() runs a database query inside this loop. With 500 orders, this can result in hundreds of additional queries. Load the required customers before the loop and reuse them here.”

The label tells an experienced engineer what to investigate.

The explanation tells them why this particular code has the problem.

3. Error semantics

Weak

“Don't swallow the error.”

Stronger

“Returning nil here makes a failed request indistinguishable from a valid empty response. The caller treats nil as success, so a network failure could be interpreted as ‘no data.’ Preserve the error so the caller can decide whether to retry or surface it.”

The concern isn't merely “errors are important.”

It's that information is being destroyed.

4. Authorization

Weak

“Security issue.”

Stronger

“Authentication establishes that the user has a valid session, but it doesn't establish that they can modify this document. Since the document isn't checked against the user's project, this path could allow cross-project modification. Verify authorization before the update.”

The reviewer is identifying the missing security boundary.

5. Maintainability

Weak

“This should be refactored.”

Stronger

“This method now handles validation, persistence, retries, and notification. Those operations have different failure modes and are already changing independently. Separating the retry/notification path would keep future notification changes from touching persistence logic.”

The second comment explains why the proposed structural change matters.

What if the reviewer doesn't know the author's intent?

This is where the framework gets uncomfortable.

Suppose a reviewer sees:

if response.status_code == 404:
    return []

They might write:

“This is wrong. A 404 should be an error.”

But perhaps the API deliberately uses 404 to represent “no matching resources.”

The reviewer has made an assumption about intent.

A more useful comment might be:

“Is 404 expected to mean ‘no matching resources’ for this endpoint? If it represents a missing resource instead, returning an empty list here would hide an API error.”

That's a question rather than a declaration.

And sometimes that is the better review comment.

Good review doesn't require the reviewer to pretend they know things they don't know.

Uncertainty should be communicated as uncertainty.

There is a major difference between:

“This is wrong.”

and:

“I may be missing context, but if X is possible, doesn't this create Y?”

The second invites the author to supply missing information.

Review comments are conversations, not verdicts

This is easy to forget when reviews are represented as annotations attached to lines.

The interface makes a comment look like a judgment.

But a review is usually a conversation.

The reviewer proposes an interpretation.

The author provides context.

The two converge on a decision.

That means a useful review comment doesn't necessarily need to prove that the reviewer is right.

It needs to make the disagreement productive.

For example:

“I think this could race if two workers execute this block concurrently. Is there a lock around this operation elsewhere that I'm missing?”

That might be a better comment than a confidently incorrect:

“This has a race condition. Fix it.”

The first comment contains uncertainty.

The second hides it.

What about AI-generated review comments?

AI makes this problem more interesting.

An automated system can examine code at a scale that humans can't.

It can notice patterns.

It can compare implementations.

It can identify suspicious data flows.

It can explain some classes of issues.

But there is a fundamental limitation:

The code is not the entire system.

Intent may live in:

product requirements,
architecture documents,
incident history,
database constraints,
undocumented operational assumptions,
conversations between engineers,
downstream consumers,
or simply someone's knowledge of why the system works the way it does.

An AI looking only at a pull request may not have access to that context.

So consider an automated comment:

“This validation is redundant.”

Maybe it is.

Or maybe the validation exists because an external system violates the contract once every few thousand requests.

Without broader context, the model may correctly understand the local code and incorrectly understand the system.

That's an important distinction.

Local correctness is not necessarily system correctness.

Can AI generate specific comments without understanding the whole system?

Sometimes.

For straightforward cases, specificity may not require deep system-level intent.

For example:

“This value is checked for None here, but dereferenced unconditionally three lines later.”

The evidence is local.

The model doesn't need to understand the entire architecture.

But consider:

“This retry count is too low.”

That might require knowing:

upstream service behavior,
timeout characteristics,
SLA requirements,
idempotency guarantees,
traffic patterns,
and production failure modes.

The more a comment depends on external intent, the less safe it is to infer that intent from code alone.

This suggests a useful boundary for automated review:

The system should distinguish observations supported directly by code from conclusions that depend on missing context.

That could mean expressing uncertainty.

It could mean asking a question.

It could mean requesting additional repository context.

It could mean not commenting at all.

Silence is sometimes better than a plausible but unsupported explanation.

Explanation isn't evidence

There's another subtle failure mode.

A detailed comment can sound authoritative simply because it is detailed.

Consider an AI-generated comment:

“This implementation creates a race condition because the cache is not synchronized across worker threads, which could cause stale reads and inconsistent state…”

It sounds convincing.

But does the system actually know that the workers share the same cache?

Does it know the operation is concurrent?

Does it know stale reads matter?

A long explanation can create false confidence.

So review quality cannot simply become:

“More detailed comments are better.”

The real requirement is:

The explanation should be proportional to the evidence.

A concise, well-supported observation can be more trustworthy than a sophisticated explanation built on assumptions.

The unit of review isn't always the line

Another limitation of line-level comments is that many engineering problems aren't located on one line.

Consider an authorization bug.

The problem might be:

controller
    ↓
service
    ↓
repository
    ↓
database

The individual lines can all look reasonable.

The vulnerability emerges from how the components interact.

Likewise with:

distributed systems,
caching,
transactions,
retries,
consistency,
API contracts,
concurrency,
lifecycle management.

Sometimes the most useful review comment needs to refer to a relationship, not a line.

For example:

“The transaction ends before the event is published, so a successful response can be returned even if event publication fails. Is that intentional? If the event is part of the operation's contract, the transaction/outbox boundary probably needs to move.”

That's difficult to express as a simple label.

It requires a model of the interaction between components.

So what makes a review comment good?

I don't think the answer is “make every comment more detailed.”

Nor is it “always explain the why.”

There are cases where the why is obvious.

Nor is it “never use labels.”

Labels are useful.

The more interesting answer is:

A good review comment contains the minimum information necessary for another engineer to correctly evaluate the concern.

Sometimes that's:

“Typo: recieve → receive.”

Sometimes it's:

“This returns null when X happens, but Y dereferences it immediately, turning a valid state into an exception.”

Sometimes it's a question:

“Is this 404 intentional? If not, are we hiding an API failure by returning an empty list?”

Sometimes the right review is no comment at all.

That last case matters.

A reviewer should not be rewarded simply for finding something to say.

A small framework: WHAT → WHY → NEXT

If I had to give reviewers a practical heuristic, it would be:

WHAT?

What exactly are you observing?

“This can return null.”

WHY?

Why does that matter?

“The caller dereferences it immediately.”

NEXT?

What should the author investigate or change?

“Handle the empty case before accessing the property.”

Together:

“This can return null, but the caller immediately dereferences it, which can turn a valid empty state into a runtime exception. Handle the empty case before accessing the property.”

But the framework should be treated as a diagnostic tool, not a writing template.

If the issue is obvious, don't force an explanation.

If the reviewer is uncertain, don't manufacture certainty.

If the concern depends on system context the reviewer doesn't have, ask.

If the implementation is simply a matter of preference, say so.

The deeper problem

There is a tendency to think of software engineering communication as secondary to the “real” engineering work.

The code is the artifact.

The review comment is just discussion around it.

I'm not convinced that's true.

A codebase contains decisions.

Reviews contain the reasoning behind some of those decisions.

When that reasoning is missing, future engineers have to reconstruct it.

When it is vague, authors have to interpret it.

When it is excessively noisy, people stop paying attention.

And when it is confidently wrong, the review system can actively make engineering decisions worse.

That applies equally to human and automated reviewers.

The hard problem isn't merely detecting something suspicious.

It's knowing what information another engineer needs to evaluate the observation.

Open questions

I don't think we've solved this.

A few questions seem particularly interesting.

How much context is enough?

Can review systems estimate when a comment needs one sentence versus five?

Can usefulness be measured without reducing it to “did the author change the code”?

A correct review may result in no change because the author provides missing context.

How should automated reviewers represent uncertainty?

Should a system say:

“This is a bug.”

or:

“If X is possible, this may produce Y. Is X guaranteed not to happen?”

How much repository context should an AI reviewer consume before making a claim?

More context can improve reasoning, but it can also increase cost and introduce irrelevant information.

Can automated systems distinguish preference from correctness reliably?

Or will they reproduce the same subjective review behavior humans already struggle with?

What happens to review quality when the cost of generating comments approaches zero?

If comments become effectively free, attention becomes the scarce resource.

And perhaps the most basic question:

Should the objective of code review be to find more problems, or to help engineers make better decisions?

Those are related goals.

They aren't necessarily the same.

A label tells us that someone thinks there is a problem.

A useful review comment gives us enough reasoning to decide whether they are right.

That's why a label isn't a code review.

Why AI Code Review Needs to Learn From Developers

Codzee.io — Tue, 08 Sep 2026 11:30:32 +0000

After 15+ years of building software, reviewing pull requests, debugging production systems, and working with engineering teams, I've become increasingly convinced that we're asking the wrong question about AI code review.

The obvious question is:

How accurately can AI detect problems in code?

It's an important question.

But there's another question that becomes more important once a team uses the system for months:

How well does the system understand what this particular team considers worth reviewing?

Those are very different problems.

A reviewer can be technically accurate and still be frustrating to use.

It can identify real complexity, real duplication, real security risks, and real maintainability concerns—and still lose the trust of the engineers receiving its comments.

Why?

Because software engineering isn't just a collection of rules.

It's context, conventions, trade-offs, history, architecture, and judgment.

And much of that knowledge isn't written anywhere.

It's learned through interaction.

That is why I believe the next generation of AI code review needs to learn from developers.

Static Analysis Was Never the Whole Story

We've had static analysis for decades.

Linters are incredibly useful.

They catch formatting problems, suspicious constructs, unused variables, dangerous patterns, complexity issues, and countless other classes of problems.

Static analysis is deterministic and predictable.

That's exactly why engineers like it.

You can define a rule:

if condition X is true
    report finding Y

You can then configure how that finding behaves.

Warning
Error
Info
Ignore

You can add thresholds.

You can exclude directories.

You can configure allowlists and denylists.

You can override rules for specific repositories.

This works remarkably well for problems that can be expressed clearly.

But there's a fundamental limitation:

The configuration describes what we think matters. It doesn't necessarily describe what the team actually considers useful.

Those aren't always the same thing.

Configuration Has a Ceiling

Consider a team working on a large backend system.

They configure a complexity threshold:

complexity > 10 → warning

Seems reasonable.

Then the system starts flagging a particular class of functions.

The team repeatedly dismisses them.

Why?

Maybe those functions are generated around a complex business domain.

Maybe they're deliberately structured that way because splitting them would make the code harder to understand.

Maybe the threshold is appropriate for most of the repository but not this subsystem.

Maybe the team has accepted the complexity because the alternative introduces a worse architectural trade-off.

You can solve this with another configuration rule.

Add an exception.

Then another exception.

Then another.

Eventually, your configuration starts looking like this:

flag X
unless repository Y
unless directory Z
unless pattern A
unless service B
unless annotation C

At some point, we're essentially trying to encode years of engineering judgment into a configuration file.

That's not necessarily the best abstraction.

Engineering Teams Are Dynamic Systems

The bigger problem is that repositories don't stay still.

Neither do teams.

An organization might move from a monolith to services.

A team might adopt a new framework.

A legacy subsystem might become effectively frozen.

A new architectural standard might be introduced.

Performance requirements might change.

Security requirements might become stricter.

The team might deliberately relax a particular style rule because it's slowing development without providing meaningful value.

Configuration tends to represent a snapshot.

Engineering practice is a moving target.

That's why I think AI code review needs something beyond configuration.

It needs a feedback loop.

The Most Interesting Signal Might Be a Dismissal

Here's a simple example.

Suppose an AI review system repeatedly reports:

This function is too complex.

The developer dismisses the comment.

The same pattern appears in another pull request.

The developer dismisses it again.

Then again.

What should happen?

A conventional system records three dismissed findings.

End of story.

But I think we're throwing away valuable information.

Repeated dismissal could mean several things.

1. The rule is too broad

Perhaps the complexity threshold isn't appropriate for this codebase.

2. The system lacks context

The function may look complex locally but be perfectly reasonable given the surrounding architecture.

3. The team has an intentional exception

The engineers may understand the trade-off and have consciously decided to accept it.

4. The feedback is technically correct but practically irrelevant

Something can be objectively improvable without being worth interrupting a developer about.

5. The team may actually be wrong

This possibility is important too.

Learning from developers shouldn't mean blindly accepting every dismissal.

Sometimes an engineer dismisses a legitimate security issue.

Sometimes a team has normalized technical debt.

Sometimes people ignore warnings because fixing them is inconvenient.

So the answer isn't:

"Developer dismissed it, therefore never report it again."

The better question is:

"What does this interaction tell us, and how confident should we be when adapting?"

That's a much more interesting engineering problem.

A Review Comment Is an Interaction

We usually model code review as something like:

Code → Analysis → Finding

But real code review is closer to:

Code
  ↓
Analysis
  ↓
Finding
  ↓
Developer judgment
  ↓
Accept / dismiss / modify / ignore
  ↓
Future behavior

That final step is where a lot of today's systems stop paying attention.

But it may be the most valuable part.

Every interaction provides information.

A developer accepts a finding.

That's a signal.

A developer dismisses it.

That's a signal.

A developer repeatedly modifies recommendations in the same way.

That's a signal.

An entire team consistently ignores a category of feedback.

That's a much stronger signal.

The challenge is turning those signals into useful adaptation without creating unpredictable behavior.

The Repository Shouldn't Be Treated as an Isolated Installation

One thing that bothers me about the current mental model is the idea that every repository starts from zero.

Install the tool.

Configure it.

Run it.

Done.

But an established codebase already contains enormous amounts of engineering knowledge.

You can find it in:

Existing implementation patterns
Pull request discussions
Architectural decisions
Test strategies
Naming conventions
Error-handling approaches
Accepted technical debt
Framework-specific patterns
Exceptions to general standards

Some of this can be explicitly documented.

Much of it isn't.

An AI system operating inside that repository should have the opportunity to understand those patterns.

Not simply analyze the current pull request in isolation.

False Positives Are More Expensive Than They Look

False positives aren't just an accuracy problem.

They're a trust problem.

Imagine a developer receives ten review comments.

Two are genuinely useful.

Eight aren't.

After a while, the developer stops carefully evaluating each comment.

They start scanning.

Eventually they start dismissing automatically.

That's when the system has failed even if its underlying detection model is technically sophisticated.

The cost isn't just the time spent reading irrelevant comments.

The larger cost is signal degradation.

Once developers stop believing that a comment is likely to matter, even good findings become harder to trust.

This is why reducing noise isn't merely a UX improvement.

It is part of the intelligence of the system.

But Adaptation Needs Guardrails

There's an obvious danger here.

If we allow AI systems to learn from developer behavior without constraints, they could learn the wrong lessons.

Imagine a team consistently dismisses security warnings because fixing them is inconvenient.

Should the system become quieter?

Probably not.

Some categories should have stronger guarantees.

Some findings should remain visible regardless of preference.

Some adaptations should require explicit confirmation.

Some changes should be explainable.

For example:

"This finding has been repeatedly dismissed for this repository. Should future findings of this type be deprioritized?"

That's very different from silently changing behavior.

The goal shouldn't be autonomous customization at any cost.

The goal should be safe adaptation.

What Would an Adaptive Review System Look Like?

I don't think this requires some magical autonomous agent.

The underlying architecture could be relatively straightforward.

Imagine maintaining a feedback model for findings.

Something like:

Finding
├── category
├── location
├── repository context
├── confidence
├── severity
├── historical interactions
└── developer/team feedback

Then review decisions become part of the system's future reasoning.

For example:

Finding A
→ accepted 27 times
→ rarely dismissed
→ high relevance

Finding B
→ dismissed 18 times
→ accepted 1 time
→ low relevance in this repository

Finding C
→ dismissed frequently by one team
→ accepted frequently by another
→ highly context dependent

Now the system isn't simply asking:

"Is this technically a problem?"

It can also ask:

"How relevant has this type of finding historically been in this environment?"

That is a fundamentally different signal.

And it can become increasingly repository-specific without requiring engineers to manually encode every preference.

The Goal Isn't Fewer Findings

This distinction matters.

Adaptive code review shouldn't simply become a system that says less.

The objective is higher-value output.

There will be situations where the correct behavior is to become more aggressive.

Suppose a team repeatedly accepts findings related to a particular security pattern.

The system now has evidence that this category matters.

That should increase confidence that similar findings are worth surfacing.

Adaptation should work in both directions.

Developer behavior
       ↓
Observed preference
       ↓
Updated relevance
       ↓
Future review behavior
       ↓
More developer feedback
       ↓
Updated preference

That's a feedback loop.

And feedback loops are how systems become better aligned with their environment.

The Metric We Should Care About

This leads to a question I think the industry should spend more time discussing.

We measure code-review systems by things like:

Detection accuracy
Precision
Recall
False-positive rate
Number of findings
Vulnerability coverage

These metrics are useful.

But they're incomplete.

I'd also want to know:

What percentage of findings does a team consider useful?

And perhaps more importantly:

Does that percentage improve over time?

Imagine two systems.

System A finds 100 issues in the first month and continues finding roughly 100 every month.

System B finds 70 issues initially, learns from developer interactions, and eventually surfaces 30 highly relevant findings.

Which one is better?

The answer depends on what those findings contain.

But for an engineering team, I would strongly prefer the system that earns the right to speak less—and gets better at knowing when to speak.

This Is Bigger Than AI Code Review

There's a broader principle here.

Good developer tools don't merely perform tasks.

They fit into human workflows.

Git became useful because developers could build mental models around it.

IDEs became useful because developers shaped them around their workflows.

Linters became useful because teams could define what mattered.

The next generation of AI tools has another opportunity:

They can learn from the workflow itself.

That's different from traditional configuration.

Configuration says:

"Here is what we told the system."

Behavioral feedback says:

"Here is what the team actually does."

The combination could be much more powerful.

Signal Isn't a Feature. It's a Relationship.

After 15+ years in software engineering, I don't think developer trust comes from a feature checklist.

It comes from repeated evidence.

You make a judgment.

The system observes it.

It adapts.

You make another judgment.

It adapts again.

Over time, the system becomes less generic and more aligned with the environment in which it operates.

That's what makes the idea of adaptive AI code review compelling to me.

The next generation won't compete only on who can detect the most bugs.

They'll compete on who can understand which bugs, patterns, and risks are actually worth surfacing to a particular engineering team.

Because detection gets you into the workflow.

Relevance keeps you there.

And relevance comes from understanding context.

That is why I believe:

Signal isn't a feature. It's a relationship.

The best AI code-review systems will be the ones that earn that relationship carefully—through feedback, context, transparency, and repeated interaction.

Not during installation.

Over time.

And ultimately, the real measure of an AI reviewer won't be how much it can say.

It will be how quickly—and how safely—it earns the team's trust.
:::

How can AI code review tools be evaluated reliably for real-world pull requests?

Codzee.io — Tue, 01 Sep 2026 14:40:12 +0000

I am evaluating AI-assisted code review systems and have found that published benchmarks use substantially different methodologies. Some replay historical bugs, some evaluate complete pull requests, some inject synthetic defects into real code, and others use human or LLM-based judgments of review comments.

I am trying to design a fair evaluation for systems that review real-world pull requests.

The main issues I am running into are:

Dataset construction: Should the test set consist primarily of historical production bugs, clean pull requests, synthetic defects, or a combination?
Ground truth: How should a benchmark define the complete set of valid findings when the original human review may have missed defects?
Precision and recall: Should findings be matched individually, or should a PR count as correct if the system identifies at least one valid issue?
False positives: How should speculative, stylistic, duplicate, or technically correct but non-actionable comments affect the score?
Severity: Should critical security or correctness defects receive more weight than low-severity maintainability issues?
Human review: What level of expert annotation or adjudication is necessary to establish reliable ground truth?
Repository context: Should systems receive the full repository, issue description, tests, commit history, and search capabilities, or should all systems be restricted to the diff?
Languages and projects: How should results be normalized across programming languages, repositories, domains, and different patch sizes?
Reproducibility: What artifacts and experimental details should be published so another team can independently reproduce the evaluation?

For example, a system that produces 20 comments and correctly identifies 10 defects is quite different from one that produces 5 comments and correctly identifies 4, even if their aggregate scores happen to be similar. Likewise, a benchmark based only on human review comments may penalize a system for finding a legitimate defect that the original reviewers missed.

I am looking for a methodology that measures useful defect detection on real pull requests, rather than simply similarity to historical review comments.

What benchmark design and scoring procedure would provide the most statistically and methodologically defensible comparison of AI code review systems, including dataset construction, ground-truth validation, finding-level precision/recall, severity weighting, repository context, human adjudication, and reproducibility?

Your CODEOWNERS File Might Be Correct And Still Send Your PR to the Wrong Person

Codzee.io — Mon, 31 Aug 2026 11:15:03 +0000

You open a PR. GitHub looks at the changed paths, checks them against CODEOWNERS, and auto-assigns a reviewer. This happens so often you probably don't think about it anymore. It's infrastructure — it just works.

Except sometimes it doesn't, and it fails in a way that never throws an error.

Here's a realistic example:

# CODEOWNERS
/payments/  @engineer-a

Engineer A hasn't touched payments/ in eight months — they moved to a different team in Q1. Engineer B has made the last 18 commits in that directory and has reviewed most of the recent PRs that touched it. Engineer B isn't in the file.

Your PR touches payments/refunds.go. It gets routed to Engineer A.

Nothing is misconfigured. The glob pattern matches. The username resolves to a real, active employee. GitHub does exactly what it's supposed to do. The routing logic is just missing context it was never designed to have.

This is the gap this article is about: CODEOWNERS tells you who owns a file. It doesn't tell you who should review a specific change. Those turn out to be different questions, and the difference gets bigger as your team and codebase grow.

1. What CODEOWNERS Gets Right

Before picking it apart, it's worth being clear that CODEOWNERS solves a real problem well, and nothing below is an argument for ripping it out.

Explicit ownership. Anyone can look at one file and see who's accountable for a given path. No tribal knowledge required.
Required reviews. You can enforce that certain paths always get eyes from the right team before merge — critical for things like auth, billing, or infra-as-code.
Clear accountability. When something in /payments/ breaks, there's no ambiguity about who's supposed to be paying attention to that code.
Simple, predictable behavior. It's a text file with glob patterns. No ML model, no black box, no surprises. You can git blame it.

For a small team, or for stable, slow-moving parts of a codebase, this is close to sufficient. The problem shows up specifically as team size and code churn increase — which is exactly when the cost of getting routing wrong also goes up.

2. Where Static Routing Breaks

CODEOWNERS answers one question — who owns this path — and organizations often treat that as a proxy for a different question: who should review this change. Here's where those diverge:

Signal	Static CODEOWNERS	Dynamic routing
File ownership	Yes	Yes
Recent contribution	No	Yes
Current expertise	Limited	Yes
Review history	No	Yes
Reviewer load	No	Yes
Expertise decay	No	Yes

Walking through the "No" column:

Recent contribution — CODEOWNERS has no time dimension. An entry from two years ago carries the same weight as one from yesterday.
Current expertise — "Limited" because ownership is often a reasonable proxy for expertise at the moment the file was written, but it's frozen after that. It doesn't update when someone's knowledge of an area grows or fades.
Review history — the file doesn't know who's actually been reviewing changes in an area, only who's nominally responsible.
Reviewer load — https://www.codzee.io/will happily assign a fifth simultaneous PR to someone already buried in review requests. It has no concept of "currently busy."
Expertise decay — there's no mechanism for an owner's relevance to degrade over time as they stop working in an area, and no mechanism for someone else's relevance to rise as they start.

None of this is a criticism of CODEOWNERS as a tool — it's just outside its scope. It's a static ownership file, not a reviewer-relevance model. The mistake is using it as if it were the second thing.

3. Model Expertise as Something That Changes

The core idea: expertise isn't binary, and it isn't permanent. "Alice owns payments" is a snapshot. What you actually want to know is closer to "how relevant is Alice's expertise, right now, to this specific diff."

Useful inputs for that:

Last contribution date — when did this person last touch this area?
Number of recent relevant changes — not lifetime commits, recent ones. Someone with 3 commits last month is a stronger current signal than someone with 200 commits two years ago.
Similarity to the current PR — has this person touched the same files, the same functions, the same data model recently?
Review activity — have they been reviewing PRs in this area, even if they haven't authored changes there?
Historical ownership — were they ever formally responsible for this, even if the file has since changed?

You can express this as a rough conceptual model — not a formula anyone should treat as an industry standard, just a way to make the tradeoff explicit:

reviewer_relevance = ownership_weight
                    + expertise_weight
                    + recency_weight
                    - load_penalty

The actual weights are going to be organization-specific, and honestly team-specific within an org. A security-critical path might want ownership weighted heavily no matter what. A fast-moving product surface might want recency and review history to dominate. The point of writing it this way isn't to hand you a formula — it's to make explicit that "who reviews this" is a function of multiple decaying/growing signals, not a single static lookup.

4. Work Through a Sample PR

Say you've got a repo with three top-level domains:

/payments/
/identity/
/notifications/

A PR comes in touching payments/refunds.go. CODEOWNERS says payments/ belongs to Engineer A. Here's how three candidate reviewers actually stack up:

Reviewer	Ownership	Recent work	Similar reviews	Current load
A	High	Low	Medium	High
B	None	High	High	Medium
C	Medium	Medium	Low	Low

Engineer A is the listed owner, but hasn't done recent work in this directory, and is already carrying a heavy review load.
Engineer B isn't in CODEOWNERS at all, but has been actively working in this area recently and has reviewed similar changes — they've probably seen the exact class of bug this PR could introduce.
Engineer C has moderate ownership and moderate recent activity, low similarity to this specific change, but has bandwidth right now.

A purely static system routes to A every time, because A is the only name it knows how to check against. But on the actual signals that predict a useful review — recent, hands-on context and demonstrated familiarity with similar changes — B is the stronger candidate, and C is a reasonable fallback if B is unavailable.

This doesn't mean A should be removed from the review entirely — for something in a sensitive path, keeping the formal owner in the loop for accountability might still make sense. It means A shouldn't be the only signal the system checks.

5. A Practical Routing Framework

Distilling this into something you could actually reason about (or build) for your own repo:

Ownership — who's formally accountable, per CODEOWNERS or equivalent?
Recency — who's touched this code lately?
Expertise — who has a track record of relevant, substantive contributions here?
Review history — who's reviewed similar changes before, and reviewed them well (not just approved fast)?
Load — who has the bandwidth to actually do this review in a reasonable timeframe?

None of these should unilaterally win. Ownership keeps governance intact — you don't want a recency model quietly routing security-sensitive changes away from the team accountable for them. Load is a modifier, not a primary filter — someone who's slightly busy but the clear best fit is often still the right call over someone who's fully free but has no context. The five signals are meant to be combined, with the weighting depending on what kind of change you're looking at (a one-line config tweak needs way less of this than a cross-cutting refactor).

6. The Routing-Decay Loop

Here's the pattern that causes CODEOWNERS files to quietly go stale, laid out as a loop:

People change teams / roles
        ↓
Code changes (new services, refactors, deprecations)
        ↓
Expertise shifts (some people gain context, others lose it)
        ↓
Ownership file stays the same (nobody's job to update it proactively)
        ↓
Routing quality declines

Nothing in that loop is anomalous — it's just normal organizational and codebase churn. The failure isn't that people move around or that code changes. The failure is that CODEOWNERS has no way to hear about any of it. It's a file, not a feed.

The fix isn't "update the file more often" — that's a losing battle against how fast real orgs change. It's closer to: continuously derive dynamic signals from data you already have (commit history, PR review history, recent activity) and layer them on top of the static file, rather than trying to keep the static file itself perfectly current. Contribution and review data refresh automatically as people work — no one has to remember to edit a YAML-adjacent file when Bob becomes the de facto payments expert. The signal is already sitting in your git history and PR review logs; it's just not being used for routing.

7. What Teams Can Measure Today

You don't need new tooling to start getting visibility into how well your current routing is actually working. Most of this is derivable from data already in GitHub:

Reviewer reassignment rate — how often does the auto-assigned reviewer get swapped out before the PR merges? High rate = your default routing isn't matching reality.
Time to first meaningful review — not first comment, first substantive review. If the assigned reviewer routinely takes days to engage, they may be overloaded or not the right fit.
Review load concentration — what percentage of review volume in a given area lands on one person? If it's consistently one name, you likely have a bottleneck, whether or not that person is technically the best expert.
PRs routed to inactive contributors — a rough version of this is: for each CODEOWNERS entry, check git log --since="6 months ago" -- <path> and see if the listed owner shows up. If not, that entry is a candidate for drift.
Review handoffs — PRs where the review gets passed from one person to another before merge are a decent proxy for "the first routing decision missed."
Recent contribution coverage — what share of merged PRs in an area were reviewed by someone who'd actually committed to that area in the last N weeks?

None of these require a new system to start tracking. A few are a git log or a GitHub API query away. The value isn't in any single metric — it's in noticing, over a few months, whether your routing is drifting in a direction nobody's paying attention to.

8. Practical Takeaway

None of this is an argument for dropping CODEOWNERS. It's still the right mechanism for encoding accountability — who's on the hook for a given part of the codebase, and which teams need guaranteed visibility into changes that touch their area.

What it's not sufficient for, on its own, is figuring out who should actually review a given PR today.

Keep CODEOWNERS for accountability. Add dynamic signals for relevance. Ownership answers "who's responsible." Recency, expertise, review history, and load together answer "who's actually the right person to look at this, right now." A routing setup that only asks the first question will keep sending PRs to people who used to be the right answer. A routing setup that ignores the first question entirely gives up the governance guarantees CODEOWNERS exists to provide. You want both.

How are you handling reviewer routing today — static CODEOWNERS, team rotation, tribal knowledge ("just tag whoever's around"), or something more dynamic? Curious what's actually working at different team sizes — drop your setup in the comments.

Our AI code reviewer problem wasn't that it was wrong. It was that it wouldn't shut up

Codzee.io — Thu, 27 Aug 2026 13:16:37 +0000

I've been thinking about a problem with AI code review that I don't see discussed enough: a reviewer can be technically correct and still make the whole review experience worse.

Here's the PR that made this click for me. Small change, maybe 200 lines, nothing exotic. New validation path, a couple helper functions. The kind of thing that used to get a comment or two and an approve.

The AI reviewer came back with something like a dozen comments. A naming suggestion. A "this null check might be redundant" note. A theoretical race condition that would only show up under conditions the code was never going to hit in prod. A "you could extract this into a function" suggestion on a function that was already about ten lines long.

And here's the annoying part most of these weren't wrong. The variable name genuinely could've been clearer. The null check probably was redundant. Individually, I could defend almost every comment on that list.

But there was one comment in there about an actual edge case in the validation logic that genuinely mattered. And it was sitting in the middle of eleven other comments that didn't. The dev opening the PR skimmed most of it, dismissed a few without really reading them, and nearly skipped the important one too — because by comment seven or eight, he'd already stopped expecting any of it to be worth his time.

Nobody wanted "more thorough." They wanted to know what actually mattered.

That's the distinction I think gets missed constantly: "is this technically an issue" and "is this worth interrupting someone for" are completely different questions, and most tooling in this space only ever answers the first one.

The assumption baked into a lot of AI review tools is basically: if finding more issues is good, finding even more must be better. But review was never just an issue-detection problem, it's a signal-to-noise problem, and honestly kind of a trust problem too. Once a reviewer starts treating a PR like a checklist to maximize, it stops mattering how accurate any individual comment is — the aggregate experience is what people remember, and the aggregate experience was "annoying."

That frustration eventually became one of the reasons I started working on Codzee. Not going to turn this into a pitch — the short version is the goal isn't "find everything," it's figuring out what actually deserves a developer's attention. Still early, still figuring a lot of it out.

Genuinely curious how other people think about this though:

How many AI review comments do you actually want to see on a normal PR? Is there a number where it tips from "helpful" to "annoying" for you?
Do you find yourself tuning out AI-generated comments after a tool has burned you with noise a few times?
Where's your personal line between a useful warning and something that's just noise dressed up as a finding?
Would you rather a reviewer miss a low-confidence issue entirely, or flag it anyway and let you decide?
What actually makes you trust an automated reviewer, versus just tolerate it?

Not trying to manufacture a debate here, I'm actually unsure about some of these myself — like the miss-vs-flag question, I go back and forth on what the right default even is. Curious what this sub thinks.

AI code review: Are more comments actually

Codzee.io — Tue, 25 Aug 2026 12:23:34 +0000

we’ve been thinking about something that seems easy to overlook with AI-assisted code review.

Imagine a PR gets 24 AI-generated comments.

That sounds thorough—but how many of those comments would you actually want a developer to spend time on?

There’s a big difference between:

“This could cause a production bug.”
“This might create a security issue.”
“This code could be more readable.”
“You could rename this variable.”
“Consider refactoring this small section.”

All of these may technically be valid observations, but they don't have the same value.

For developers, the real challenge isn't just finding issues anymore.

It's prioritizing them.

Too many low-value comments can create review fatigue, hide important findings, and slow down the person who actually needs to make the decision.

I'm curious how other developers handle this:

When using AI for code review, what do you prefer?

AI should report everything and let developers filter it.
AI should only report high-confidence, high-impact issues.
AI should report everything but rank comments by severity.
Something else?

We've been exploring this problem at Codzee and believe that AI code review should optimize for signal, not comment count.

Would love to hear how this works in your team and what you've found useful—or annoying—about AI code review.
www.codzee.io

How I'd Benchmark an AI Code Reviewer Before Putting It on 100 Engineers' PRs

Codzee.io — Mon, 24 Aug 2026 12:55:11 +0000

AI code-review tools are easy to demo.

Give one a pull request, wait a few seconds, and it produces a handful of comments that look surprisingly useful. Maybe it catches a missing authorization check. Maybe it spots a race condition. Maybe it complains about a test that doesn't cover an edge case.

That's a good demo.

It is not a benchmark.

If you're considering putting an AI reviewer on the pull requests of 100 engineers, the question isn't "How good does this look?" It's:

How often does this tool find problems that matter in our code, how often does it waste developers' time, and does it improve the review process enough to justify the cost?

I'd answer that question with an evaluation built around your own repositories, languages, review practices, and historical bugs.

Here's how.

1. Build a representative PR dataset

Start with a dataset of pull requests that looks like your actual engineering workload.

Don't build it entirely from toy examples. A benchmark containing 50 artificially constructed security bugs will tell you something about the model, but not necessarily much about what happens when it encounters your production code.

I'd aim for a few categories:

Historical production bug fixes
PRs that introduced bugs but were later fixed
Normal feature PRs with no known bugs
Refactoring PRs
Dependency updates
Performance-related changes
Security-sensitive changes
Database/schema changes
API changes
Test-only changes

The important part is preserving the context available during a real review: the PR diff, repository state, relevant files, tests, configuration, and whatever other context your normal review process exposes.

You should also keep a hidden ground-truth record for each PR.

For example:

PR-1842
Repository: payments-api
Language: Go
Known issue: authorization bypass
Severity: Critical
Introduced in: PR
Fixed by: PR-1911
Expected finding: Yes

For clean PRs, record that too:

PR-2017
Repository: customer-service
Language: Kotlin
Known issue: None
Expected finding: No

That gives you both positive and negative examples.

The negative examples are essential. An AI reviewer that finds something suspicious in every PR can look impressive until developers have to deal with the resulting noise.

2. Include known real-world bugs

Historical bugs are some of the most valuable test cases you have.

Look through incident reports, postmortems, security fixes, bug tickets, and reverted changes. Find bugs that were actually introduced into your codebase.

Then construct benchmark cases that reproduce the review situation.

For example, suppose six months ago a PR introduced:

if user.is_admin:
    return export_all_accounts()

when the intended condition was:

if user.is_admin and user.can_export_accounts:
    return export_all_accounts()

The benchmark should test whether the reviewer catches the authorization problem from the PR context—not whether it can identify a deliberately planted TODO: SECURITY BUG comment.

This distinction matters.

A useful benchmark measures performance on problems engineers genuinely encounter.

3. Include different bug severities

Not every finding has the same value.

A reviewer that catches one critical security issue but misses 20 minor problems may still be extremely valuable. Conversely, a tool that produces hundreds of "potential issue" comments while missing serious defects can become a productivity problem.

Classify your known issues.

One simple taxonomy is:

Severity	Example
Critical	Data loss, privilege escalation, severe security vulnerability
High	Production outage, major correctness issue
Medium	Significant edge case or reliability problem
Low	Minor correctness issue, maintainability problem

You can adapt these categories to your existing incident or vulnerability classification.

Then measure recall separately by severity.

For example:

Critical: 4/5 found
High:     11/18 found
Medium:   17/31 found
Low:      9/22 found

That tells you much more than a single "82% accuracy" number.

4. Test multiple repositories and languages

If your organization has six repositories and four languages, don't benchmark the tool on one favorite service written in the language it handles best.

Split the dataset across representative environments.

For example:

payments-api       Go
web-frontend       TypeScript
mobile-backend     Kotlin
data-platform      Python
infrastructure     Terraform

You don't necessarily need equal numbers of PRs in every repository. You do need enough samples to identify obvious differences.

The question isn't only "Does the tool work?"

It's:

"Does it work consistently enough across the environments where we're planning to deploy it?"

A reviewer that performs brilliantly on TypeScript but poorly on Terraform may still be useful. You just need to know that before making it mandatory across the organization.

5. Give every reviewer the same context

Benchmark fairness is surprisingly easy to get wrong.

If you're comparing two AI reviewers, make the input as equivalent as possible.

Give each one the same:

PR diff
Base revision
Repository contents
Relevant configuration
Tests
Documentation
Existing review context
Tool permissions, where applicable

Don't manually explain a tricky bug to one system because "it needed a little help."

Likewise, don't give one tool access to information that the other cannot see unless that difference is part of the product you're evaluating.

Context also needs to be documented.

For each benchmark run, record exactly what the reviewer was allowed to inspect.

Otherwise, six months later, you'll have two benchmark results that look comparable but weren't generated under comparable conditions.

6. Measure bugs correctly identified

The first metric I'd track is bug recall.

At its simplest:

Bug recall = known bugs correctly identified / known bugs

Suppose your dataset contains 100 known bugs and the tool identifies 72 of them.

That's 72% recall.

But there's a catch: what counts as "identified"?

A comment saying:

"This code might have an issue."

isn't necessarily equivalent to correctly identifying the actual defect.

Define acceptance criteria before running the benchmark.

For example, a finding might count as correct only if it:

Points to the relevant code.
Describes the actual failure mode.
Explains why the change causes the problem.
Provides enough information for an engineer to validate it.

This prevents optimistic scoring.

7. Measure false positives

Now run the other half of the experiment.

Give the reviewer PRs with no known bugs and see what it reports.

A useful metric is false-positive rate, but I also like tracking the simpler number:

False findings per PR

If a tool produces:

Known bugs found:      72
False findings:       180
PRs reviewed:          100

then its 72% recall doesn't look nearly as exciting.

You can also calculate precision:

Precision = correct findings / all findings

If the reviewer produces 100 findings and only 40 are genuinely actionable, precision is 40%.

For developers, this metric can matter enormously.

A reviewer that catches everything but complains about everything isn't necessarily useful.

8. Measure duplicate and noisy findings

False positives aren't the only source of noise.

AI reviewers can report the same underlying problem multiple times.

For example, one missing validation might produce:

"Input isn't validated."
"Potential invalid state."
"Unexpected value may reach database."
"Consider adding defensive checks."

Four comments, one issue.

I'd explicitly measure unique actionable findings.

You can group findings that describe the same underlying defect and count them once.

Also track findings that are technically true but don't deserve review attention.

For example:

"This function could be refactored to reduce complexity."

That may be reasonable advice, but if your benchmark is evaluating bug detection, it shouldn't count as a valuable bug finding.

A useful classification is:

Correct + actionable
Correct + low value
Duplicate
False positive
Not a bug

This makes the output much easier to analyze.

9. Measure severity accuracy

Finding the bug isn't enough.

The reviewer should also understand its impact.

Suppose your ground truth says:

Authorization bypass → Critical

and the AI says:

Potential code quality issue → Low

Technically, it noticed something. Operationally, it failed an important part of the review.

Compare predicted severity with your ground truth:

                  Actual
Predicted       Critical  High  Medium  Low
Critical            7      2      0      0
High                3     14      4      1
Medium              0      5     19      7
Low                 0      1      8     11

You don't need sophisticated statistics to start. Even a confusion matrix like this will show whether the tool systematically underestimates serious defects.

10. Measure whether developers actually accept findings

Eventually, the benchmark has to leave the spreadsheet.

Ask engineers to review the AI findings without knowing which tool produced them.

For every finding, capture something like:

Valid issue?       Yes / No
Actionable?        Yes / No
Would fix it?      Yes / No
Would mention it?  Yes / No

If you're running a live pilot, you can also measure what happens to findings in actual PRs:

Accepted
Fixed
Dismissed
Ignored
Marked duplicate

This is arguably one of the strongest signals you can get.

A finding with 95% technical accuracy but a 10% acceptance rate may not be particularly useful.

A finding that engineers consistently validate and act on is much more valuable.

11. Measure review latency

Don't forget the economics of the workflow.

Measure how long it takes from:

PR opened
    ↓
AI review starts
    ↓
AI findings available

Then compare that with the existing review process.

Latency matters because an AI reviewer that takes 45 minutes to analyze every PR may be technically impressive but operationally awkward.

You should also test different PR sizes.

For example:

Small:   <200 changed lines
Medium:  200–1,000
Large:   >1,000

A tool might be excellent on small PRs and degrade badly on large ones.

That's something you want to discover in a benchmark, not after deployment.

12. Measure developer time saved

This is where the benchmark becomes a business case.

Estimate the human effort associated with the findings.

For example, during a pilot you might measure:

PRs reviewed:                  200
Valid AI findings:              86
Findings accepted by authors:   61
Estimated review effort saved:  24 hours
Additional triage time:          7 hours
Net time saved:                 17 hours

Be conservative.

If an AI comment catches something a human reviewer would definitely have found anyway, don't automatically count the entire human review as "saved."

Instead, ask:

"How much work did this finding actually remove from the engineering process?"

You can estimate time through short developer surveys, sampled review sessions, or controlled experiments.

The goal isn't to manufacture a perfect number. It's to get a defensible estimate.

13. Repeat the evaluation over time

One benchmark run isn't enough.

AI systems change. Your repositories change. Prompting changes. Models change. Vendors ship new detection logic.

Run the benchmark periodically.

I'd keep a fixed golden dataset containing a representative sample of your most important historical cases, then add new cases over time.

For example:

Baseline
├── 100 historical bugs
├── 100 clean PRs
└── 50 security-sensitive PRs

Quarter 2
├── +25 new production bugs
└── +25 new clean PRs

Quarter 3
├── +20 new production bugs
└── +20 new clean PRs

Now you can track whether performance is actually improving.

You can also detect regressions.

If version 4.2 of your AI reviewer goes from 78% to 84% recall but doubles false findings, that's not simply "better."

It's a trade-off.

A simple scoring framework

You don't need a complicated machine-learning benchmark.

Here's a framework I'd start with:

Metric	Weight
Bug recall	30%
Precision / false positives	20%
Severity accuracy	10%
Developer acceptance	15%
Duplicate/noise rate	10%
Review latency	5%
Developer time saved	10%

Score each category from 0–100, then calculate the weighted total.

For example:

Recall                  82 × 0.30 = 24.6
Precision               76 × 0.20 = 15.2
Severity accuracy       80 × 0.10 =  8.0
Developer acceptance    71 × 0.15 = 10.7
Noise                   85 × 0.10 =  8.5
Latency                 90 × 0.05 =  4.5
Time saved              68 × 0.10 =  6.8

Total                               78.3 / 100

The exact weights are less important than agreeing on them before you see the results.

And I'd add one more rule: define minimum thresholds for critical metrics.

For example:

Overall score       ≥ 75
Critical bug recall ≥ 90%
Precision            ≥ 60%
P95 review latency   ≤ 10 minutes

That prevents a tool from compensating for catastrophic performance in one area with excellent performance in another.

What about vendor benchmarks?

Vendor benchmarks are useful.

They can tell you how a tool performs against standardized datasets, and they're worth looking at when you're building your shortlist.

But they answer a different question.

A vendor benchmark asks:

"How well does our system perform on this benchmark?"

Your benchmark asks:

"How well does this system work for our engineers, in our codebases, under our review process?"

That's the number I'd use when deciding whether to put the tool on 100 engineers' PRs.

For example, if you're evaluating a tool such as Codzee, treat its published benchmark results as one input into your research—not as the final deployment decision.

The same methodology should apply to every vendor.

The benchmark I'd actually run

If I were setting this up for an engineering organization tomorrow, I'd make the process roughly:

1. Select 300–500 representative PRs
              ↓
2. Label known bugs and expected severity
              ↓
3. Include clean PRs as negative examples
              ↓
4. Run each AI reviewer with equivalent context
              ↓
5. Normalize and deduplicate findings
              ↓
6. Blind-score correctness and severity
              ↓
7. Run a developer acceptance study
              ↓
8. Measure latency and engineering effort
              ↓
9. Calculate weighted score
              ↓
10. Repeat periodically

Most importantly, keep the benchmark independent from the vendor.

You choose the PRs.

You define what constitutes a correct finding.

You decide how severity is classified.

You determine which findings are useful.

And you keep the dataset private enough that it can't simply become another benchmark for vendors to optimize against.

That's how you turn an AI code-review demo into an engineering evaluation.

The goal isn't to find the AI reviewer with the highest benchmark score.

It's to find the one that makes your code-review process better without creating a new category of developer noise.

What would you add to this benchmark?

I Put an AI Reviewer on Every PR. Here's What Happened After the 100th Review.

Codzee.io — Thu, 20 Aug 2026 12:03:16 +0000

At first, adding an AI code reviewer to every pull request feels like cheating.

Every PR gets another pair of eyes.

Every diff gets analyzed.

Potential bugs are pointed out before a human reviewer even opens the PR.

The first few reviews are impressive.

Then something strange happens.

You stop reading every comment.

Not because the reviewer is always wrong.

Because you learn that most comments aren't worth stopping for.

That's the part of AI code review I think we don't talk about enough.

The reviewer's first job isn't finding bugs

It's earning the developer's attention.

Imagine this PR:

AI review

✓ Possible null handling issue
✓ Consider extracting this function
✓ This variable could be renamed
✓ Potential race condition
✓ Missing test case
✓ This comment could be clearer
✓ Consider using...
✓ Consider using...
✓ Consider using...

The problem isn't that every comment is bad.

Some might be useful.

The problem is that the developer now has to perform another review:

“Which of these comments should I care about?”

That's review fatigue.

And it can happen surprisingly quickly.

There are four kinds of AI review noise

I've found it useful to think about review noise in four categories.

1. Wrong

The reviewer identifies something that isn't actually a problem.

Example:

This condition can never be true.

Except it can.

The reviewer simply misunderstood how the value reaches the function.

This is the easiest type of noise to recognize.

2. Technically correct

This one is more dangerous.

Example:

This function is 45 lines long.
Consider splitting it into smaller functions.

Fine.

Maybe that's a good idea.

But if the PR is fixing a production incident and the function is intentionally kept together for transactional behavior, this isn't the problem the reviewer should be talking about.

The comment isn't wrong.

It just isn't useful right now.

3. Missing context

This happens constantly.

The reviewer sees:

if (!user) {
    return;
}

and says:

“Potentially unsafe null handling.”

But the API intentionally permits anonymous users.

The reviewer doesn't understand the business rule.

The developer does.

Now the developer has to explain the business rule to the AI reviewer.

That's backwards.

4. Repetition

This is the fastest way to turn useful tooling into background noise.

The same type of comment appears over and over.

Eventually developers recognize the pattern.

They stop reading.

And once they stop reading, the good comments get ignored too.

The worst metric for AI code review

I don't care how many comments an AI reviewer generates.

Seriously.

Imagine two systems.

System A

100 comments.

8 accepted.

System B

15 comments.

9 accepted.

Which one would you rather have?

Obviously B.

But comment volume is seductive because it's easy to put on a dashboard.

PRs reviewed: 2,381

Issues detected: 14,921

That sounds impressive.

It doesn't tell you whether developers found those issues useful.

I'd rather see:

Findings: 1,200

Accepted: 780
Dismissed: 310
Duplicates: 70

Now we're measuring something interesting.

Try a review-noise audit

If your team already uses AI code review, here's a simple experiment.

Take the last 100 AI-generated review comments.

Put each one into one of these buckets:

A — Fixed
B — Discussed / investigated
C — Dismissed
D — Duplicate
E — Not actionable

Then calculate the percentages.

For example:

Fixed:              31%
Investigated:       18%
Dismissed:           27%
Duplicate:           12%
Not actionable:      12%

Now you have a much more useful picture of your reviewer.

The interesting number isn't “31% fixed.”

It's everything else.

Because every dismissed or low-value comment still consumed some amount of developer attention.

Then look at severity

This is another place where AI reviewers can lose trust.

Suppose a normal PR gets:

5 critical issues

You investigate them.

Three are style suggestions.

One is a minor maintainability concern.

One is actually important.

After a few PRs like that, what does “critical” mean to you?

Probably not much.

Severity needs a contract.

For example:

Critical

Could realistically cause:

security compromise
data loss
major outage
broken core functionality

High

Meaningful production risk.

Medium

Worth fixing but not urgent.

Low

Minor concern.

And then there's another category:

Don't comment

If the reviewer isn't confident enough, it should stay quiet.

The uncomfortable lesson

Developers don't need an AI reviewer that notices everything.

They need one that knows what not to mention.

That's a very different optimization target.

A human reviewer naturally filters.

They might see 20 questionable things and comment on 2.

Why?

Because they understand that every comment costs the author something.

Time.

Attention.

Discussion.

Potential context switching.

AI systems don't naturally feel that cost.

They need to be designed around it.

AI writing code makes this more important

There's another reason this problem is getting bigger.

AI coding agents can produce much larger changes much faster.

A developer might ask:

“Add OAuth login.”

The agent might modify:

auth/
middleware/
routes/
database/
frontend/
tests/
configuration/
documentation/

Now someone has to understand that entire change.

The bottleneck has moved.

We used to ask:

“How can we write code faster?”

Now we increasingly need to ask:

“How can we verify more code without increasing cognitive load at the same rate?”

That is a much harder engineering problem.

So what should an AI reviewer actually do?

My ideal reviewer has five rules.

Rule 1: Don't repeat deterministic tools

If ESLint already catches it, don't produce another paragraph explaining it.

If the compiler catches it, don't create an AI warning.

AI should focus on things that require reasoning.

Rule 2: Understand the surrounding code

A diff without context is often not enough.

The reviewer should understand relevant:

functions
tests
interfaces
configuration
repository conventions
PR intent

Rule 3: Prioritize risk

A potential authentication vulnerability should not appear underneath 12 variable-naming suggestions.

Rank findings.

Rule 4: Explain why the finding matters

Don't say:

“This could cause a problem.”

Say what could happen.

For example:

“This query now executes once per customer because it sits inside the loop. At 10,000 customers, this turns the operation into an N+1 query pattern.”

That's actionable.

Rule 5: Be willing to say nothing

This may be the most important rule.

A review that returns:

“No high-confidence issues found.”

can be a successful review.

Silence is better than noise.

Where Codzee fits

This is the problem space where Codzee is interesting.

The useful question isn't:

“Can we put AI on every pull request?”

That's relatively easy now.

The harder question is:

“Can we put AI on every pull request without making developers hate the review process?”

That requires better prioritization, useful context, and a focus on actionable findings rather than maximizing comment volume.

That's the direction AI-assisted code review needs to move.

A simple mental model

Think about AI review like a security guard.

A bad security guard stops everyone.

“Bag check.”

“Bag check.”

“Bag check.”

“Bag check.”

Eventually everyone stops paying attention.

A good security guard doesn't stop everyone.

They identify the situations that actually deserve attention.

AI code review should work the same way.

Don't maximize interruptions.

Maximize useful interruptions.

The real success metric

After a few months of using an AI reviewer, ask your team one question:

“When this reviewer leaves a comment, do you usually read it?”

If everyone says yes, you've probably built trust.

If people say:

“I usually just dismiss them.”

You have a problem.

And it's not necessarily model accuracy.

It's relevance.

The reviewer has become background noise.

That's what I mean by earned irrelevance.

The AI didn't necessarily fail because it couldn't find bugs.

It failed because developers stopped believing that its comments were worth their attention.

And in code review, attention is the resource that matters most.

What about your team?

If you're already using an AI code reviewer, try the 100-comment audit.

Take your last 100 comments.

Classify them.

Then see how many were actually useful.

I'd be genuinely interested in what teams are finding.

Are AI reviewers reducing your review workload, or have they just created another thing you have to review?

Disclosure: This article discusses Codzee as an example of AI-assisted code-review tooling. The broader argument is about the design of AI review workflows, not a claim that any single tool solves the problem.

The AI Code Review Checklist I Actually Use Before Merging a PR

Codzee.io — Wed, 19 Aug 2026 12:48:48 +0000

I've been using AI coding tools heavily for a while now, and the thing that changed my workflow the most wasn't the code generation itself, it was building a review habit that actually catches the failure modes specific to generated code. AI output tends to look more finished than it is. The variable names are good, the structure is clean, the comments are confident. That polish is exactly what makes it easy to under-review.

This is the checklist I actually run through, not the one I aspire to run through. Eight categories, each with a quick explanation of why it matters and a small example of the kind of thing that slips past a fast skim.

1. Correctness: does it do what was asked, not just something plausible

AI models are very good at producing code that satisfies the letter of a prompt while missing the actual intent. If you ask for "a function that returns the most recent order for a user," you need to check what "most recent" means in the data. Sorted by creation date? Updated date? What if there are two orders with the same timestamp?

def get_most_recent_order(orders):
    return sorted(orders, key=lambda o: o.created_at)[-1]

This works until two orders share a timestamp, and then the result is whichever one Python's sort happens to place last, which is not "most recent" in any meaningful sense. It's arbitrary. Nobody asked for arbitrary.

Why it matters: correctness bugs from AI output are rarely syntax errors, they're semantic gaps between what was asked and what got built. You have to read the code against the requirement, not just against itself.

2. Edge cases: the inputs nobody put in the prompt

If your prompt didn't mention empty lists, null values, negative numbers, or duplicate entries, the model probably didn't think about them either. It optimizes for the happy path described in the request.

function calculateAverage(scores) {
  const total = scores.reduce((sum, s) => sum + s, 0);
  return total / scores.length;
}

Pass an empty array and you get NaN silently propagating into whatever calls this next. No error, no crash, just a quietly wrong number working its way downstream.

Why it matters: edge cases are where AI-generated code fails most often, because the model is pattern-matching against the common case in its training data, not reasoning about your specific data guarantees.

3. Security: don't assume the model thought about it unless you told it to

If you didn't explicitly ask for input sanitization, parameterized queries, or auth checks, don't assume they're there. They often aren't, because the prompt didn't ask for them.

query = f"SELECT * FROM users WHERE email = '{user_email}'"
cursor.execute(query)

This is straightforward SQL injection, and it's also one of the most common patterns I see generated when someone asks for "a function that looks up a user by email" without mentioning security at all.

Why it matters: generated code defaults to the simplest version of a solution, and the simplest version is very often the insecure version. Treat every generated data access or input-handling function as unreviewed for security until you've explicitly checked it.

4. Performance: check the complexity, not just the output

Generated code frequently reaches for the most obvious algorithm rather than the most efficient one, especially for anything involving lookups or nested iteration.

def find_duplicates(items):
    duplicates = []
    for i in range(len(items)):
        for j in range(i + 1, len(items)):
            if items[i] == items[j] and items[i] not in duplicates:
                duplicates.append(items[i])
    return duplicates

This is O(n²) for a problem that's a few lines away from O(n) with a set or a counter. It'll pass every test on a small list and quietly become a bottleneck the moment someone runs it against production-sized data.

Why it matters: small inputs during testing hide complexity problems that only show up at scale, and AI output doesn't come with a complexity analysis attached. You have to do that part yourself.

5. Maintainability: will the next person understand this without you

Generated code is often correct in isolation but disconnected from the conventions of the rest of your codebase. Different error handling style, different naming pattern, a slightly different way of structuring similar logic that already exists three files over.

# existing pattern in the codebase
class UserNotFoundError(Exception):
    pass

# what got generated for a new function
def get_user(user_id):
    user = db.query(user_id)
    if not user:
        return None  # inconsistent with the rest of the codebase's error pattern
    return user

Why it matters: consistency is a maintainability feature. Code that solves the problem correctly but ignores established patterns adds friction for every future reader, even if it never causes a bug.

6. Tests: check what's actually being asserted, not just that tests exist

Generated tests can create a false sense of safety. It's common to get tests that run without error but don't assert anything meaningful.

def test_process_payment():
    result = process_payment(order)
    assert result is not None

This test passes whether the payment succeeded, failed silently, or returned an empty error object. It confirms the function returned something, not that it did the right thing.

Why it matters: a test suite with weak assertions is worse than no test suite in some ways, because it creates confidence that isn't backed by actual verification. Read the assertions, not just the test count.

7. Dependencies: verify the package actually exists and does what's claimed

Models occasionally reference packages, methods, or APIs that don't exist, or that existed in an older version and have since changed. This is less common than it used to be but still worth a direct check, especially for less mainstream libraries.

import pandas as pd
df.iteritems()  # removed in pandas 2.0, replaced by df.items()

Code like this can look completely reasonable to someone who hasn't kept up with a specific library's recent changes, and it'll fail immediately in CI or, worse, only in a slightly older pinned environment.

Why it matters: a broken import or a deprecated method is an easy fix once caught, but it's an annoying one to debug blind if it ships and fails somewhere downstream instead of at review time.

8. Production behavior: what happens when a dependency it relies on goes down

This is the category that's easiest to skip because it doesn't show up in normal testing at all. Does the code handle timeouts? Partial failures? What does logging look like when something goes wrong at 3am and someone's trying to debug it without you in the room?

def send_notification(user_id, message):
    response = requests.post(NOTIFICATION_API_URL, json={"user_id": user_id, "message": message})
    return response.json()

No timeout, no retry, no handling of a non-200 response, and no logging if it fails. When the notification service has a bad five minutes, this function will hang or throw an unhandled exception with zero context about what actually happened.

Why it matters: this is the category of bug that doesn't show up until it's already in production, because local development rarely simulates a flaky dependency. It's worth explicitly asking "what does this look like when the thing it depends on is unavailable" for anything that talks to an external service.

The pattern across all eight

Every one of these examples looks fine at a glance. That's not a coincidence, it's the actual challenge with reviewing generated code. The syntax is clean, the naming is sensible, the code reads as intentional. The gaps aren't in how the code looks, they're in what it doesn't account for, and you only catch that by asking specific questions of it rather than scanning for things that look wrong.

Save this, print it, stick it next to your monitor, whatever works. The categories don't change much project to project, only the specifics do.

What would you add to this list? Curious what's bitten people that isn't covered here.

Before You Merge AI-Generated Code, Ask These 12 Questions

Codzee.io — Fri, 14 Aug 2026 12:21:07 +0000

I've merged plenty of AI-generated code that was genuinely fine. I've also caught myself almost merging code that looked fine and wasn't, because it read like something a competent person wrote and my brain filled in the rest.

Over the last year I've settled into a rough set of questions I run through before approving anything I didn't write line by line myself, generated or not. Here they are, in the order I actually ask them.

1. What problem is this code actually solving?

It's easy to review whether code works and skip whether it solves the right thing. AI tends to answer the literal prompt, not the intent behind it.

def get_active_users():
    return db.query("SELECT * FROM users WHERE active = true")

If "active" was supposed to mean "logged in within 30 days" and not a boolean flag that's rarely updated, this passes every test and still solves the wrong problem.

Reviewer tip: Read the original ticket or request before reading the diff. Check the code against the intent, not just the literal ask.

2. Do I actually understand the implementation?

Not "does it look reasonable," actually understand it, line by line, well enough to explain it to someone else.

Reviewer tip: Try to explain the function out loud in one sentence per major step. If you get stuck anywhere, that's the part you haven't actually reviewed yet, just skimmed.

3. What assumptions is it making?

Every implementation bakes in assumptions about the shape of the data, the order things happen in, or what "normal" looks like.

function getLatestOrder(orders) {
  return orders[orders.length - 1];
}

This assumes orders is sorted chronologically and never empty. Neither assumption is stated anywhere.

Reviewer tip: Ask "what does this assume about its inputs that isn't checked anywhere?" Write the answer down, literally, in the PR comment if it matters.

4. What happens with bad input?

Bad input isn't an edge case, it's a certainty over a long enough timeline.

def parse_age(value):
    return int(value)

Pass it "25" and it works. Pass it "twenty-five", None, or -5 and you get a crash or a nonsensical value with no complaint.

Reviewer tip: Pick three inputs that would never appear in a demo but could plausibly appear in production: empty, wrong type, absurdly large. Trace through what actually happens.

5. What happens when an external service fails?

Generated code frequently assumes the network, database, or third-party API always responds successfully.

async function getExchangeRate(currency) {
  const response = await fetch(`https://api.example.com/rates/${currency}`);
  const data = await response.json();
  return data.rate;
}

No timeout, no handling for a non-200 response, no fallback. If that API is slow or down, this fails in whatever way fetch and .json() happen to fail, which may not be a clear error at all.

Reviewer tip: For every external call, ask "what does the caller see if this times out or returns an error status?" If the answer is "an unhandled exception," that's worth a comment.

6. Are permissions actually enforced, not just checked?

There's a difference between "there's an auth check" and "it's the right auth check."

if (!req.user) return res.status(401).send('Unauthorized');
const doc = await db.documents.findById(req.params.id);
res.json(doc);

This confirms someone is logged in. It never confirms they own or have access to this document.

Reviewer tip: For any endpoint touching a specific resource, ask "does this check the resource belongs to the requester, or just that the requester is logged in?"

7. Is sensitive data exposed anywhere?

Look at what actually goes into logs, error responses, and API payloads, not just what the happy path returns.

catch (err) {
  res.status(500).json({ error: err.message, stack: err.stack });
}

Fine in local development. In production this can leak file paths, query fragments, or internal structure to whoever triggers the error.

Reviewer tip: Grep the diff for console.log, print, and catch blocks. Check what they actually expose.

8. Is this more complex than the problem needs?

Generated code sometimes over-engineers a simple problem with extra configuration, unnecessary abstraction layers, or generic solutions to specific problems.

Reviewer tip: Ask "could this be half the length and still be correct?" If yes, that's worth pushing back on, complexity has an ongoing cost even when it's not technically wrong.

9. Are the tests actually meaningful?

Generated tests often confirm the code does what it does, not that it does what it should.

def test_apply_discount():
    assert apply_discount(100, 10) == 90

This confirms the arithmetic. It says nothing about a discount over 100%, a negative price, or invalid input.

Reviewer tip: For each test, ask "what wrong implementation would still pass this?" If you can think of one easily, the test isn't pinning down enough.

10. Does it fit existing conventions?

Locally correct code can still be a long-term problem if it introduces a new pattern the codebase doesn't already use, a different error-handling style, a new HTTP client, a different logging approach.

Reviewer tip: Before approving, check one comparable file elsewhere in the codebase. If the patterns don't match, ask whether that's intentional.

11. What happens under unusual load or concurrency?

Code that's correct for one request at a time can break under concurrent access, especially anything involving shared state or caching.

counter = {}

def increment(key):
    counter[key] = counter.get(key, 0) + 1

Fine single-threaded. Under concurrent requests this can lose increments, since read-then-write isn't atomic.

Reviewer tip: For anything touching shared state, ask "what happens if this runs twice at the exact same moment?"

12. Will the next developer understand this without me?

The final check. If someone opens this file in eight months with zero memory of this PR, can they figure out what it does and why from the code and comments alone?

Reviewer tip: Read the diff as a stranger would, not as someone who already knows what it's supposed to do. If it doesn't hold up, add a comment now while the reasoning is still fresh.

Copy-paste checklist for your PR template

## AI-Generated Code Review Checklist

- [ ] Solves the actual requirement, not just a literal reading of it
- [ ] I understand every part of this well enough to explain it
- [ ] Assumptions are identified and actually hold true
- [ ] Handles empty, invalid, and unexpected input
- [ ] Handles external service failures and timeouts
- [ ] Permissions checked against the specific resource, not just login state
- [ ] No sensitive data leaked via logs, errors, or responses
- [ ] No unnecessary complexity for the problem size
- [ ] Tests check behavior, not just current output
- [ ] Matches existing codebase conventions
- [ ] Considered behavior under concurrent access or load
- [ ] Understandable to a developer with no context on this PR

None of this is exotic review practice. It's the same discipline good engineers apply to any code they didn't personally trace through, generated or not. The only thing that's changed is how often that situation comes up, and how easy it is to skip these questions when the first draft already looks like someone competent wrote it.