DEV Community: Muhannad Salkini

How to review pull requests written by coding agents

Muhannad Salkini — Fri, 16 Oct 2026 17:23:41 +0000

A pull request from a coding agent usually looks better than one from a tired human. The description is complete, the code is formatted, the tests pass, and every changed line has a reason. That polish is exactly what makes it hard to review. The usual signals that something is off (a messy diff, a vague description, a missing test) aren't there.

The mistakes agents make are different from the ones people make. They rarely forget a semicolon. They often solve the problem in a way your team decided against, or change something that other code relies on without noticing the other code exists. Both of those are invisible if you only read the diff.

Here is the checklist we use.

1. Read the description last

Agents write confident summaries of what they intended to do. Read the diff first and form your own view of what changed, then compare it with the description. Gaps between the two are where the bugs are: "refactored the retry logic" that also changed the timeout default, "added a field" that also made an existing one optional.

2. Ask who depends on what changed

For every function, type or exported value the PR changes, find the callers. Not the ones in the diff, all of them.

An agent asked to "add a required orgId parameter to loadSettings" will update the function and the callers it found. Whether it found all of them depends on how it searched. Your editor's "find references" or grep takes ten seconds and is the highest-value thing you can do on an agent PR.

Be suspicious when a tool says nothing calls a function. In our own measurements, a code graph reported zero callers, labelled "exact", for 17 of 26 functions that had real callers. "No results" from a tool means "unknown", not "safe".

3. Check it against what the team already decided

Every codebase has rules that aren't enforced by the compiler: all database access goes through the repository layer, prompts live in files and not inline strings, model calls go through one shared client, no new dependencies without discussion. People learn these from review comments. Agents don't, unless they're written down somewhere the agent reads.

When you review, ask: does this follow how we do things here, or just a way that works? Agent PRs drift toward the most common pattern on the internet, not the one your team picked.

4. Look for scope creep

Agents like to tidy. A PR to fix one bug touches three unrelated files because the agent "noticed" something. Each tidy-up is a small, unrequested change that nobody asked to review. Ask for them to be split out, or reverted.

5. Check the tests test the change

Agents are good at making tests pass, and occasionally that means changing the test. Look at test diffs as carefully as code diffs: a loosened assertion, a new skip, a mock that now returns whatever the code expects. A green check on a PR that edited its own tests means less than it seems.

6. Check error paths, not just the happy path

Agent code tends to handle the case it was asked about. Look at what happens on the failure branches: swallowed errors, a catch that logs and continues, a fallback that silently returns an empty list. These pass every test that only exercises the success case.

Move the checks earlier

Most of this checklist is the reviewer doing work the agent could have done before opening the PR. You can move a lot of it to write time:

Write your decisions down where the agent reads them. CLAUDE.md, .cursor/rules or AGENTS.md, with the reason for each rule, not just the rule. Agents follow rules they can see.
Make "find all callers" part of the task. Tell the agent to list every caller of anything whose signature it changes, and to update or explain each one. It works noticeably better than hoping it does it on its own.
Keep PRs small. One concern per PR. Agents will happily do five things at once if you let them, and five things are much harder to review than one.

Automate the two hard parts

Steps 2 and 3 are the ones that don't scale by hand. Finding every caller of every changed symbol, and remembering every decision the team has made, is a lot to ask of a reviewer on their tenth agent PR of the day.

That is what we built Tently for. It is an AI code reviewer for GitHub that checks each pull request against the code that depends on it and against your team's decisions, which it learns from merged pull requests, review threads and docs. Findings land on the PR with evidence: the affected callers, or the decision and where it came from. It runs next to whatever reviewer you already use.

The same decisions are available while the agent writes, through an open-source MCP server for Claude Code, Cursor and Codex (tently-team/tently-mcp), so fewer problems reach the PR at all.

With or without a tool, the checklist holds:

Read the diff before the description.
Find every caller of what changed.
Check it against your team's decisions.
Push back on scope creep.
Review test changes as carefully as code.
Read the error paths.

What diff-only AI code review misses: regressions outside the diff and decision drift

Muhannad Salkini — Mon, 12 Oct 2026 07:21:15 +0000

Most AI code reviewers work the same way. They read the pull request diff, maybe a few files around it, and comment on what they see: a missing null check, an unhandled promise, a SQL string built by hand. They are good at this now, and if your team uses one, keep it.

But a whole class of bugs is invisible to that approach. The diff looks fine. The problem is somewhere the reviewer never looks.

They come in two kinds.

1. Regressions outside the diff

A function gets a new required parameter. The pull request updates the function and the two callers the author remembered. There are three more callers in other packages. They are not part of the diff, so the reviewer never reads them, the type checker doesn't help if one of them goes through an any-typed adapter, and tests pass if nobody tests that path.

The diff is correct. The change is not.

This gets worse with coding agents. An agent changing a function rarely goes looking for every caller unless you tell it to, and the pull request it opens is usually tidy and confident. A reviewer that reads only that pull request has nothing to object to.

To catch this, the reviewer needs one question answered before it reads the diff: who depends on the code that changed?

2. Decision drift

The second kind is quieter. Say your team decided, months ago and on purpose, that every LLM call goes through one shared client, so cost tracking and retries live in one place. The decision is in a merged pull request, a review thread and a line in CLAUDE.md.

A new pull request calls the provider SDK directly from a new file. It works. It has tests. It is also the start of the thing you decided not to do, and nothing in the diff says so.

A generic reviewer can't flag this, because the rule isn't a generic rule. It's yours. It lives in your history, not in the code being changed.

To catch this, the reviewer needs a second question answered: which of our decisions apply to the files that changed?

Answering "who depends on this?" is harder than it looks

The obvious tool for the first question is a code graph: index the repository, then ask for the callers of each changed symbol. We built our regression check on one, and then we measured it, because a missed caller is a missed regression.

We labelled ground truth with the TypeScript LanguageService (every real call site of a target function, attributed to the function that contains it) and compared a code graph's answers against it.

On our own TypeScript monorepo, with an off-the-shelf graph tool:

57.4% recall: it found 27 of 47 real callers.
17 of 26 functions came back with zero callers, even though every one of them had real ones.
Some misses were same-file calls, not just cross-package ones.
Every empty result was reported as exact, with risk LOW.

That last point is the one that matters. A tool that says "I don't know" is fine. A tool that says "nothing calls this, safe to change" when three things call it is how regressions get approved.

We ran the same kind of study on Vite with a different engine. It named the exact caller 67.9% of the time, but every one of the 185 call sites fell inside some caller it reported. The engine was naming the wrong enclosing function, not missing the call. That is a much better failure, because a reviewer reading the reported callers still finds the call.

What we took from this:

An empty caller list is not evidence of safety. Treat it as "unknown", search the code anyway, and read likely consumers before deciding a change has no impact.
Measure your graph against a real oracle, on a real repository. A small clean fixture will read 100% and tell you nothing. The misses only show up at scale.
Pass the reviewer the uncertainty, not just the answer. Our impact results carry an explicit "approximate" label and a list of reasons they may be incomplete (dynamic dispatch, re-exports, reflective calls), and the reviewing model sees those.

Answering "which decisions apply?" needs a decision store

Decisions are scattered: merged pull requests, review threads where someone said "please don't do this, we use X", ADRs, READMEs, CLAUDE.md. To check a pull request against them you need them in one place, each with a scope (the paths it covers) and a link to where it came from.

We mine candidate decisions from that history automatically, but a mined decision is a guess. So they start as proposals, and someone on the team approves, edits or rejects each one. A finding that cites an approved decision can block a merge. One that cites an unconfirmed proposal is labelled as such and weighted lower.

The link back to the source matters more than we expected. "This breaks a decision" is easy to dismiss. "This breaks the decision from that pull request six months ago, here is the thread" is not.

Keeping the noise down

Both checks are easy to make noisy. A reviewer that flags every caller of every changed function is useless within a week. What worked for us:

Two passes. A cheap model sorts candidate findings, then a stronger model has to confirm each one against the actual code before anything is posted.
Evidence on every finding. The affected callers by name and file, or the decision and its source. No evidence, no comment.
Feedback that sticks. A finding the team dismisses is not posted again on the next push.

Run it next to your current reviewer, not instead of it

Diff-level review and context-level review catch different bugs. We run both. The diff reviewer catches the null check. The context check catches the caller in the other package and the decision from six months ago.

We built this into Tently, an AI code reviewer for GitHub pull requests that checks every change against the code that depends on it and against your team's decisions. The same decision store is also available while the agent writes code, through an open-source MCP server (tently-team/tently-mcp), so Claude Code, Cursor or Codex can look up the rules before the pull request exists.

Whatever you use, the two questions are worth asking of any pull request, human or agent-written:

Who depends on the code that changed?
Which of our decisions apply to the files that changed?

If your reviewer can't answer them, someone on your team is answering them by memory.