DEV Community: Tess Ainsley

AI code review on GitHub: what the platform decides for you

Tess Ainsley — Fri, 23 Oct 2026 00:30:09 +0000

Search for how to add an AI reviewer to GitHub and you land on a pile of tool READMEs, a Reddit thread asking whether any of them are worth it, and GitHub's own Copilot documentation. Each one describes the product. None of them describe the layer underneath, which is where the behavior of the reviewer actually gets decided.

A reviewer installed on GitHub can read certain files, write certain things, and carries a credential while it does. Those are set by the installation and the workflow, before the model sees a single line of the diff. Get them wrong and the reviewer still posts comments and still looks healthy, which means the failure does not arrive as an error. It arrives as context that quietly was not there.

Two ways a reviewer attaches to GitHub, owned by different parties

The first shape is a hosted app installed on the repository. It receives a scoped token from GitHub, runs on the vendor's infrastructure, and reads the diff through the API. You control neither the runner nor the working tree the model sees. GitHub Copilot code review, CodeRabbit, Qodo, and Greptile all work this way to varying degrees.

The second shape is a workflow inside your own repository. AI Review ships as a pip package and a Docker image, runs in your pipeline, and posts inline comments, summaries, and now replies on existing review threads through a token you supply. Its README states that it never proxies or inspects your requests, and that it talks to OpenAI, Claude, Gemini, Ollama, Bedrock, OpenRouter, or Azure OpenAI depending on your config. claude-code-action runs Claude Code inside a job in your repository with your permissions block. PR-Agent and Alibaba's OpenCodeReview sit in the same family. Kodus spans both shapes: a hosted app, plus a self-hosted deployment and a CLI that runs the review wherever you point it.

The difference between the two shapes lands in who owns the failure. With a hosted app, the API access the vendor negotiates is the whole story, and their scoping choices become yours to accept. With a workflow, the runner, the checkout, and the token are repository configuration, and so is every way they can be configured wrong.

The trigger event sets what the job is allowed to hold

GitHub's secure use reference for Actions is direct about the starting position: the GITHUB_TOKEN should default to read-only for repository contents, and individual jobs should raise permissions only as far as they need. A pull request from a fork already runs with a read-only token and no secrets.

An AI reviewer needs a secret. Either the model provider key or an API token for the review service. That pushes teams toward pull_request_target, and it works, because that event runs with the base repository's GITHUB_TOKEN and access to repository and organization secrets. It also means the job holds credentials while a pull request from anyone can influence what the job does. GitHub's separate guide on securely using pull_request_target explains the trade and names the specific way teams break it: checking out the untrusted pull request head into the workspace root before the trusted step runs.

Anthropic documents the same pattern for its own action and recommends a two-checkout arrangement. The base ref goes at the workspace root, since the action expects a git repository there. The head ref goes into a subdirectory that gets passed to Claude through --add-dir, so the model can read the proposed changes without them occupying the paths your trusted configuration resolves against.

If you want the review to post inline comments, the job needs pull-requests: write. If you want it to push fix commits or resolve threads, it needs more, and each additional scope is something a prompt injection in the diff can use. Writing the permissions block is writing the review policy. A reviewer that cannot push produces a bad comment when it is wrong. A reviewer that can push produces a bad commit.

Which files the reviewer reads, and where they came from

This is the question I would want answered before installing anything, and it is the one most vendor pages skip: when an agentic reviewer runs against a pull request, which files in its working tree came from the base branch?

Anthropic answers it for its own tool in the claude-code-action security documentation. Before Claude starts, the action restores a fixed list of paths from the pull request's base branch: .claude/, .mcp.json, .claude.json, .gitmodules, .ripgreprc, CLAUDE.md, CLAUDE.local.md, and .husky/. Paths on that list that do not exist on the base branch are deleted, and the pull-request versions are kept aside under .claude-pr/ for reference. Everything else in the working tree stays at the pull request head, including package.json, lockfiles, the Makefile, node_modules/, and formatter or linter configuration.

The split is deliberate and it is also incomplete by design, which the doc says outright. If a hook in the base branch's .claude/settings.json runs a package manager script, a make target, or a repo-relative script, that command resolves through files the pull request supplies. The recommendation is to call the tool directly with a pinned version and pass configuration on the command line, something like bunx prettier@3.5.3 --no-config --write . rather than bun run format.

A reviewer whose rules come from files in the repository has the same exposure. If your AI reviewer reads AGENTS.md or .cursorrules from the pull request head, the diff can rewrite the rules that are being applied to it. Base-branch rule loading is the property worth checking, and it is the reason I keep pointing at where rules come from rather than how many a tool claims to support. Kodus documents a rules file detection path that imports .cursorrules, .cursor/rules/**/*.mdc, .github/copilot-instructions.md, and AGENTS.md, together with inheritance from global to repository to directory, which is the shape that lets you decide what is centrally controlled and what a pull request can touch. If the reviewer reads sibling repositories for cross-repo context, that set of repositories is configuration too, and it is the same context problem that shows up with monorepos.

What to check before you install anything

The criteria that separate these options are not model quality. They are where the job runs, who holds the credential, where the rules are read from, and what a wrong review is capable of doing.

	Hosted app	Action in your CI	Self-hosted platform
Where the review runs	Vendor infrastructure	Your runner, under the event you choose	Your infrastructure
Who holds the provider credential	Vendor	Your repository secrets	Your deployment
Where rules are read from	Vendor config, usually plus repo files	Files in the checkout, at whichever ref you set	Your deployment, with rule inheritance across scopes
What a bad review can do	Depends on the scopes the app was granted	Bounded by the job's permissions	Bounded by the token you issue
Failure mode when misconfigured	Missing context you cannot inspect	Silent gaps in the checkout	Silent gaps in the config
Examples	Copilot code review, CodeRabbit, Qodo, Greptile	AI Review, claude-code-action, PR-Agent, OpenCodeReview	Kodus self-hosted, OpenCodeReview on your own runner

The table is not a ranking. It is a list of the four things to ask a vendor before you connect a repository, and the honest answer is that the hosted column is often unknown from the outside. That is not a reason to avoid hosted reviewers. It is a reason to know which column you are in before you assume the reviewer sees what you think it sees.

There is also a cost side to the choice, and it mostly lands on the self-hosted and CI columns, since those are the ones where runners, setup steps, and caches become yours to pay for. I looked at what self-hosting an AI reviewer actually costs in more detail, and the short version is that the setup cost recurs on every push rather than once per installation.

Where the gate sits, and who decides

The last piece is the one that gets decided in the repository settings rather than in the tool, which is what the reviewer is allowed to do about a failing review.

Most reviewers post findings and leave the merge decision to a human, and that is the safe default because it keeps a wrong finding in the comment thread where it costs a minute. The alternative is letting the reviewer block a merge, which only makes sense if you have decided what a block means and who can clear it. Kodus explains its own position in the review policy documentation, including what blocks a pull request, what does not, and who decides. That is the level of detail to ask for from any reviewer you plan to put in front of a merge queue, because the general problem with leaning on review policy to catch AI-authored code is that a policy built around guessing which code was machine-written tends to fail quietly when the guess is wrong.

GitHub's own documentation for Copilot code review is worth reading on this point; the default review action is a comment, not an approval, so the gate stays where it was unless you move it on purpose.

A starting checklist

Read the security page before the features page. If the reviewer runs as an action, find out which event triggers it and what that event grants the job, then check whether the checkout puts any untrusted ref at the workspace root. Confirm the permissions: block is read-only except for the one scope the reviewer needs to comment. If the reviewer reads rule files or configuration from the repository, establish whether those come from the base branch or the pull request head, and treat anything from the head as input the diff controls. Then decide, deliberately, whether the reviewer comments or blocks.

Four answers, obtainable before installation, and they tell you more about how a reviewer will behave on your repository than any benchmark score does.

FAQ

Does an AI code reviewer on GitHub need pull_request_target?

Only if it must read secrets. Pull requests from forks run with a read-only GITHUB_TOKEN and no secrets, so a reviewer that calls a model provider needs a trigger that has credentials. pull_request_target grants the base repository's token and secrets, which is why GitHub documents checking out the untrusted head ref into the workspace root as the specific mistake to avoid. Check out the base ref at the workspace root and pass the head ref through a subdirectory instead.

Can a pull request change the rules my AI reviewer applies?

It can, if the reviewer reads its rule files from the pull request head. Agentic reviewers restore a fixed list of configuration paths from the base branch for exactly this reason, and everything outside that list stays at the head. If your coding standards live in AGENTS.md or .cursorrules in the repository, confirm which ref the reviewer loads them from.

Should an AI reviewer be allowed to block a merge?

That depends on whether you have defined what a block means and who can clear it. The safer default is comment-only, because a wrong finding costs a minute in a thread rather than a stuck pull request. Reviewers that do block usually document what blocks a merge and what does not, and that documentation is the thing to read before putting one in front of a merge queue.

Originally published at agentwrotethis.dev.

Which AI reviewer approvals satisfy your merge requirement

Tess Ainsley — Wed, 21 Oct 2026 00:30:06 +0000

Whether an AI reviewer's approval clears a merge requirement is a repository setting, and on most platforms it starts off. GitHub's Copilot code review will post a review the moment it is enabled, but a separate toggle decides whether that review can submit an approving review, and a second decides whether the approval counts toward the required approvals on the branch. CodeRabbit's request-changes workflow ships disabled and only exists on paid tiers. Kodus documents auto-approve as optional and recommends leaving blocking behavior off until your thresholds and CI are settled.

The consequence is that a team can run an AI reviewer for months, watch it comment on every pull request, and never notice its approval was never in the merge path. The reverse is worse. Someone enables the toggle, a bot approval clears the gate on a change nobody read, and the first time anyone notices is an incident.

What the approval toggle actually controls

GitHub splits this across three settings on the repository, under Settings, then Copilot, then Code review. Allow Copilot to approve pull requests turns on approving reviews. Allow Copilot approvals to count toward merge requirements is the one that lets those approvals satisfy branch protection. The third is a file path list, one glob per line, up to 15 entries, which narrows where an approval counts: it counts only on pull requests where every changed file matches one of the globs.

That glob list is the most useful of the three and the one most teams leave blank. If the bot is allowed to approve and its approvals count, an empty list means a bot approval carries the same weight on a payment handler as it does on a changelog. Filling it in keeps the approval in the merge path for the changes where it is worth something and out of it everywhere else. GitHub labels the whole arrangement as public preview, which is worth knowing if your merge gate is load-bearing.

Kodus takes the opposite default and says so in its review policy documentation: suggestions are non-blocking by default, request-changes and auto-approve are optional behaviors you turn on, and the page recommends enabling blocking only where the team has agreed thresholds and strong CI in place. That is a defensible starting position because it keeps the failure mode to a noisy thread rather than a stuck pull request.

The effort dial is a separate control from the approval

Copilot code review has a review effort level with three values. Lite is described as a standard review, Balanced goes deeper on complex logic, security-sensitive code, and cross-service changes, and Max is listed as coming soon. The built-in default is Balanced.

Effort changes what the reviewer examines and what it costs. GitHub's own estimate is $0.05 to $1 of AI credits for a Lite review and $0.25 to $5 for a Balanced one, excluding Actions minutes, with consumption rising alongside pull request size and repository custom instructions. Those are estimates from the vendor, not measurements, so treat the ranges as an order of magnitude.

Effort and approval are independent. Raising the effort level changes what the reviewer finds. It does not change whether the reviewer's approval can clear the gate, and it does not change which paths that approval counts on. A team that wants a stronger review and a narrower gate sets two different controls, and teams regularly conflate them.

There is a second layer here that shows up on a bad CI day. Copilot's agentic capabilities, meaning full project context gathering and passing suggestions to the cloud agent, run on GitHub Actions. If Actions is unavailable or the workflow fails, reviews still post, just without the agentic extras. The same repository can produce a shallower review because a runner was unavailable, and nothing in the comment tells you which mode you got. That is the coverage question, and it is why coverage per change is the number to ask a vendor for rather than a comment count.

When an approval stays pending

CodeRabbit's request changes workflow is the most explicit of the tools about this. When enabled, CodeRabbit requests changes if a review posts actionable inline comments, and approves once the approval requirements are met. Approval stays pending when the latest review is missing, in progress, failed, or rate-limited; when the current pull request HEAD is not among the commits it reviewed; when a required review thread is unresolved; when a pre-merge check is failing; or when HEAD changes while CodeRabbit is verifying the requirements.

Two of those are worth reading twice. A rate-limited run can still resolve threads it considers addressed, but the approval stays pending because the latest commit did not complete a review. And if every changed file is excluded by path filters, CodeRabbit skips the review content but still evaluates the remaining approval requirements and can approve. An empty review is not the same as a blocked one. Path filters can produce an approval on a change the reviewer never read.

CodeRabbit also ships two override commands, @coderabbitai resolve and @coderabbitai approve, which can submit approval without requiring the latest commit to have completed a review and without pre-merge checks passing. By default the pull request author can run them on their own pull request, and reviews.allow_author_approval: false turns that off. So with the workflow on, an author-facing command can approve around the checks the workflow exists to enforce. Whether that is acceptable is a policy decision, but it should be a made one, in the same spirit as a review policy that cannot lean on detecting AI code.

Who the review is billed to

Copilot attributes an automatic review's AI credits to the pull request author by default, and a manually requested review to whoever requested it. For pull requests authored by the Copilot cloud agent, usage is attributed first to the human co-author, and if that person cannot be billed, to the organization. For pull requests authored by other bots, or when a bot requests the review, it is billed directly to the organization.

The attribution is about cost, and it still shapes who enables what. An organization that pays for every bot-authored review has a reason to keep automatic review off on agent-heavy repositories, and that choice removes the reviewer from those pull requests entirely. The trigger event and the token scopes sit in the same layer, which is the part the platform decides for you before a model sees the diff.

The tools side by side

Tool	Approval default	What can block or clear a merge	Limits worth knowing
GitHub Copilot code review	Approval off by default, counting toward merge requirements off by default	Approving reviews, then a separate toggle to count them as required approvals	File path globs, up to 15, gate which pull requests count; public preview
CodeRabbit	Request-changes workflow disabled by default, Essentials tier and above	Requests changes on actionable inline comments, approves when requirements are met	Approval stays pending on a rate limit or a stale HEAD; author-facing resolve and approve commands
Kodus	Non-blocking suggestions by default	Request-changes and auto-approve are optional, enabled per repository	Docs recommend blocking only with agreed thresholds and strong CI in place
Graphite AI reviews	Reviews run without manual configuration	Not documented on the pages I read	unknown

Graphite's AI reviews page describes automatic review of every pull request and points at acceptance rates and rule effectiveness in a dashboard, but the pages I read do not document approval or merge-gate behavior. That is an unknown, not an absence.

Questions to settle before a bot approval counts

Four questions cover most of the risk.

Does the approval count as a required approval, and on which paths? If the answer is all paths, the glob list is your first piece of work.

Can the approval survive a stale HEAD or a rate limit? If not, the bot will be the reason a pull request sits, and you should know that before reviewers start blaming each other.

Who can override it, and can the author of the pull request override their own? An override command is a feature. An override command that the author runs on their own change is a policy decision.

What does the reviewer do when it reviewed nothing? Path filters and failed agentic runs both produce a posted review with less behind it than the comment suggests.

None of these are reasons to keep a bot out of the merge gate. They are the settings that decide whether the gate means what the team thinks it means. The reviewer's opinion and its authority are separate things, and only one of them is set on the branch.

What your review queue's ranker can and cannot see

Tess Ainsley — Mon, 19 Oct 2026 02:30:01 +0000

A team with more open pull requests than review hours does not need another reviewer first. It needs something that decides which pull request to look at, and on at least one platform that decision is now a documented product feature with stated limits. CodeRabbit's Triage builds a ranked queue across repositories, and its documentation spends nearly as much space on what the ranking cannot see as on what it can. That part is worth reading before you put a ranker in front of a backlog that agents keep filling.

This matters because the volume problem agents create shows up as a ranking problem before it shows up as a review-time problem. When volume rises and review time plateaus, the plateau is a disengagement signal rather than a throughput win, which is what the Salesforce review data showed. A ranker is the tool teams reach for at that point, so it is worth knowing exactly what evidence it ranks on.

Where the mechanical state stops being the signal

Triage separates two questions per pull request. Priority is how much acting on it is worth compared to everything else in the queue. Next action is what specifically has to happen for it to move. Its prioritization documentation keeps those apart on purpose, and the examples make the split concrete: failing checks, merge conflicts, draft status, a branch behind its base, unresolved review threads, and a stale approval all shape the next action and can make a pull request un-mergeable, while none of them make it urgent.

A trivial dependency bump with red checks should not outrank a customer-facing fix that is simply waiting on a reviewer. The state of a pull request tells you what to do about it. It does not tell you what the change is worth.

That is the whole argument for a queue over a repository list once volume climbs. Sorting by last-updated or by author puts a decision in front of you that has nothing to do with consequence, and reviewers respond by reading the top of the list.

The ranker publishes what it cannot match

Administrators describe how their organization wants pull requests ranked, in plain language, and CodeRabbit turns that into an ordered list of rules assigning P0 through P3. The matchable facts are broad: the repository, its owner, the target branch, draft state, change type, issue severity, risk, security risk, blast radius, review effort, check status, merge conflict status, workflow state, next action, involved people with their roles, words in the title, and how many other pull requests are blocked behind this one.

Then the exclusions. Rules cannot match labels, file paths, CODEOWNERS, team membership, assignees, or the code itself. When part of a description asks for something rules cannot express, CodeRabbit reports that part as unsupported and does not apply it, rather than approximating it.

The CODEOWNERS exclusion is the one to underline. CODEOWNERS is GitHub's own answer to who owns which path, and the queue's ranking layer cannot read it. The two systems answer different questions. The ownership file decides who gets asked to review, and the ranker decides what is worth reviewing now, and nothing forces them to agree. If your mental model of priority is whatever touches the paths your senior people own, that is not expressible as a ranking rule, and the tool says so instead of guessing.

The file itself is also less solid than people assume. GitHub's CODEOWNERS reference documents that a line with invalid syntax is skipped, that a file over 3 MB is not loaded at all, that docs/* does not match nested paths like docs/build-app/troubleshooting.md, and that gitignore-style ! negation and [ ] character ranges do not work. If your priority model depends on an ownership map, check the map first.

What it takes for the ranker to call something urgent

Automatic P0 requires external evidence. A pull request reaches P0 on its own only when it carries a validated linked issue at High or Urgent priority in Jira or Linear. Without that, the base score is capped at P1 no matter how severe the internal signals look. An administrator rule can assign P0 without a linked issue, and a person can set P0 by hand.

The base score comes from three signals: the validated severity of issues the reviewer found, the priority of a linked external issue, and how many other open pull requests are blocked behind this one. That third signal is the interesting one. Downstream dependency is something a ranker can compute and a human rarely does, and it is exactly the quantity that grows when agents open changes faster than anyone merges them.

Triage does not show a numeric score. Levels are the interface, and the number behind them is deliberately undocumented because it changes as the ranking model improves. Two pull requests in the same provider state routinely land on different levels, which is the intent. The trade is that you cannot audit a rank the way you can audit a rule.

Where the queue stops and the merge starts

The ranker also carries actions, and the merge action has a documented limit. Triage merges only the commit you were shown, and if someone pushed since, the merge is refused. On GitHub there is no merge method to choose, because Triage uses the first method the repository allows in the order merge commit, squash, rebase. When the branch requires a merge queue, the control becomes Add to merge queue and GitHub controls the ordering, checks, and method, and a queued pull request is not yet merged.

Manual actions run as you, using your own Git provider authorization, so an action you could not perform in the provider is refused in the queue too. Automated actions use the organization's integration and need administrator authorization. Opening the queue at all requires an active Review seat, and the feature is Team plan and above, generally available on GitHub Cloud with GitLab in beta and Bitbucket unsupported, and not available on self-hosted CodeRabbit deployments.

If what you actually care about is whether a bot approval clears branch protection, that is set in the repository, and the approval toggles and path globs are a separate decision from anything the queue does.

The other layer: what happened to the suggestions

Ranking decides what to read. It says nothing about whether anyone acted on it. Kodus measures that separately in its Cockpit, on a tab called Kodus Review, which asks whether the team is acting on what the reviewer said. The headline metric is implementation rate, implemented suggestions divided by sent suggestions, and only delivered suggestions count, meaning drafts the tool filtered out before commenting are never in the denominator. Suggestions are attributed to the week the pull request closed, because implementation status is only final once it merges.

Two details there are easy to reuse in any review measurement. Negative vote rate shows raw counts instead of a percentage below ten reactions, and charts built on very few suggestions are faded and marked, because a 0% or 100% from a handful of suggestions is not a signal. The severity chart also carries an All / Kodus only toggle, because a custom rule carries the severity you set on the rule rather than a risk assessment by the tool, and mixing the two distorts the read.

The metric I have not seen elsewhere is critical suggestions left unimplemented on pull requests that already merged. That is a risk list rather than a quality score, and it is the closest thing to a measure of what the review process let through. It also pairs with the coverage question, since how much of a change a reviewer actually reads determines what a suggestion count can mean in the first place.

Reading the numbers the platform hands you

If you pull review and adoption numbers from GitHub's own Copilot usage metrics, the caveats are documented and specific. Most metrics come from client-side IDE telemetry, and end users need telemetry enabled in their IDE for the richest data. Server-side telemetry catches active users that client telemetry misses, and those users count toward active totals, but their per-feature and lines-of-code breakdowns stay empty. The data excludes Copilot Chat on GitHub.com and GitHub Mobile, and license and seat information lives in a separate API.

Attribution follows organization membership rather than where the action happened, so one person's usage can appear in several organization dashboards while counting once at the enterprise level. GitHub states that organization-level analytics are not designed to be compared directly with enterprise totals. The Copilot impact dashboard groups users into adoption cohorts and connects adoption to pull request output, which is the nearest the platform gets to tying a review tool to delivery.

None of that makes the numbers wrong. It makes them numbers with a stated scope, and a review metric quoted without its scope is how a planning meeting turns into an argument.

Three measurement layers, side by side

Each of these answers a different question, and none of them substitutes for the others.

Layer	Question it answers	What it can read	What it declares out of scope
CodeRabbit Triage	Which pull request is worth acting on now	Repository, branch, draft state, change type, severity, risk, blast radius, review effort, checks, conflicts, workflow state, involved people, title words, downstream dependents	Labels, file paths, CODEOWNERS, team membership, assignees, the code itself; the numeric score is undocumented
Kodus Cockpit (Kodus Review tab)	Whether the team acts on what the reviewer said	Delivered suggestions, their implementation status at PR close, reactions, per-rule trigger counts, review-run outcomes	Filtered-out drafts never count; rates are suppressed below ten reactions and thin samples are marked
GitHub Copilot usage metrics	How Copilot is adopted and how that tracks pull request output	IDE, CLI and agent app telemetry, server-side active-user signals, repository-level PR activity, adoption cohorts	Copilot Chat on GitHub.com and GitHub Mobile, license and seat data, org-level figures compared to enterprise totals

The scoping differences are the point of the table. A Triage rule can see blast radius but not a file path. A Kodus implementation rate sees what happened to a suggestion but not which path it touched. GitHub's metrics see adoption and pull request activity but not the reviews themselves. If you want one number that covers all three, you are going to build it from two exports and a join, and you will spend the first week reconciling definitions.

What to check before you trust a queue

The cheapest test I know takes an afternoon and no integration work. Take the ten pull requests your team would rank as most important today, in order, by hand. Then write the rule you would need to reproduce that ranking, in whatever language the tool accepts. If the rule reaches for file paths, labels, CODEOWNERS, or team membership, the ranker cannot express it, and the queue you get will be ordered by something else.

Do the same for the number you plan to report upward. Ask what the denominator is, which population it counts, and which cases the documentation says are excluded. A ranker that tells you what it cannot see is easier to trust than one that does not mention the question, and the ones that document their blind spots have just handed you the audit you were going to need anyway.

The context question sits underneath all of this. A queue that ranks across repositories still hands the reader one change at a time, which is why cross-repository context is a separate problem from ranking order. Ranking decides when you look. It does not decide whether what you are looking at makes sense on its own.

Which of your rule files the reviewer actually reads

Tess Ainsley — Fri, 16 Oct 2026 00:15:06 +0000

Every AI review tool claims it reads your AGENTS.md or CLAUDE.md. What decides whether your standards actually land is which rule wins when two apply to the same file, and whether the tool can show you which one it used for a specific pull request.

A reviewer that follows your team's standards is making a claim about precedence and about what the tool will show you. It says very little about how well the model writes code. Almost every AI review tool now picks up AGENTS.md, CLAUDE.md, .cursorrules or a Copilot instructions file, so the yes/no version of the question resolves to yes for all of them. The version that decides whether your standards land comes down to which files it reads, which rule wins when two apply to the same file, and whether you can prove for one specific pull request that the rule you care about was in scope.

Detection is the easy part

CodeRabbit's code guidelines page lists the patterns it detects by default: AGENTS.md, .cursorrules, .github/copilot-instructions.md, .github/instructions/*.instructions.md, CLAUDE.md, GEMINI.md, .cursor/rules/*, .windsurfrules, .clinerules/*, .rules/*, and AGENT.md. Kodus's rules file detection page covers a similar set and adds .aider.conf.yml, .opencode.json, and docs/coding-standards/**/*. GitHub Copilot supports three kinds of custom instructions, documented on the repository custom instructions page: a repository-wide copilot-instructions.md in .github, path-specific NAME.instructions.md files under .github/instructions, and AGENTS.md files stored anywhere in the repository.

Three vendors, three lists that overlap almost completely. If detection were the whole problem, any of them would do, and the comparison would be about model quality. It is not, because none of those lists tell you which file wins when your repo has a root CLAUDE.md, a services/billing/CLAUDE.md, and a shared standards repository that also gets imported. That is where teams lose enforcement without noticing.

Which rule wins when two apply

Kodus resolves rules through an inheritance hierarchy: global, then repository, then directory. The effective set at any level is the rules defined there, plus the rules inherited from the parent level, minus anything explicitly excluded at that level. There are two ways to stop a rule flowing down. You can mark it non-inheritable when you create it, in which case it applies only where it was created. Or you can exclude a parent rule at a child level, and this is the part worth reading twice: the exclusion does not cascade. Exclude a global rule at the repository level and a configured directory inside that same repository still inherits it, unless you exclude it there too. Global rule sources let you point Kodus at a standards repository and import its rule files as global rules that apply to every connected repo, and the source repo itself is treated as configuration rather than as a codebase, so Kodus skips review on pull requests opened in it.

CodeRabbit resolves configuration differently, and the default matters. Configuration inheritance is disabled by default. You have to set inheritance: true, after which CodeRabbit merges up the chain and stops at the first level where inheritance is false or unset. Merge behavior depends on the data type. Objects deep merge, scalars take the child value, and arrays append unique parent items after the child's, deduplicated by path, label, name, id or key. The vendor's own worked example shows the consequence: a central path_instructions entry for src/** is dropped because the repository already defines an entry with the same path. No error, no warning. The central instruction is simply not in the merged config.

Copilot's rule is positional. The nearest AGENTS.md in the directory tree takes precedence, and when a path-specific instruction file matches the file being reviewed, both it and the repository-wide file are used. The docs also note a prerequisite that is easy to miss: for Copilot code review, your personal setting for using custom instructions has to be enabled, which it is by default. Turn it off and the review silently runs without your rules.

The failures that look like success

The dropped array entry above is the cleanest example of a rule that exists, is detected, and never applies. There are several more of the same shape.

A path glob that does not match. Kodus's own troubleshooting line is blunt: a rule only runs on files matching its path, so a rule can show as active in the UI and never appear in the evaluation trace for the files you expected. Nothing about the UI tells you the difference.

Case sensitivity. CodeRabbit's guideline patterns are case-sensitive, so a file named claude.md is not matched by **/CLAUDE.md. A team that renames a file, or that has one contributor on a case-insensitive filesystem, ends up with guidelines that quietly stop applying.

Guidelines reviewed instead of used. Adding CLAUDE.md to path_instructions tells CodeRabbit to review that file as changed code, not to apply it as a guideline. The docs call this a common mistake, and the symptom is a reviewer that comments on your standards document instead of enforcing it.

LLM conversion of a free-form file. Kodus imports .kody/rules/** and rules/**/*.md verbatim, preserving your wording, identifiers and Bad/Good examples. Files that do not follow that template fall back to the LLM importer, which merges several guidelines from one file into a single rule. Want two independently manageable rules? They need to be two files. There is a related trap: if you hand-edit a rule that was created from an IDE rule file, the next sync of that file overwrites your edit.

Markers that are live controls. Adding @kody-ignore to a rule file makes Kodus skip it on the next sync and remove any rules previously created from it, and removing the marker brings them back. That is the intended behavior, and it means a comment in a markdown file is a control plane for what gets enforced.

Cross-repo guidelines that are not pinned. CodeRabbit can read guideline files from another repository in the same organization, and it reads them from that repository's default branch at review time. Freshness is the benefit. The cost is reproducibility: a comment from last month may not reproduce today because the guideline text behind it has moved.

Rule files that drift. Cloudflare's CI-native reviewer includes a plugin that verifies the repository's AGENTS.md is up to date, which is a reasonable response to stale rule files. They also built the whole system themselves, because off-the-shelf tools did not offer enough flexibility for an organization their size. Stack Overflow's piece on building shared coding guidelines for AI makes the adjacent point that guidelines for agents need to be more explicit and demonstrative than guidelines for people, because an agent does not absorb the tacit conventions a new hire picks up by reading the codebase.

How to prove a rule actually fired

This is the part most vendor pages do not answer, and it is the part that turns the claim into something checkable.

Kodus on a self-hosted deployment writes grep-able markers into the API logs. [kody-rules-sync] summarizes each sync run, including which files were imported, which were skipped with a reason, and which were removed. [kody-rules-eval] is a per-file evaluation trace that names the rule uuid and title selected into the prompt for each reviewed file. Two docker logs greps give you an answer for a specific pull request rather than for the feature in general.

CodeRabbit exposes the resolved configuration rather than the prompt. Running @coderabbitai configuration on a pull request prints the fully resolved YAML annotated with source comments, so each setting shows which level in the merge chain supplied it. That is a direct way to catch the dropped src/** entry.

For GitHub Copilot, the documentation describes how the three kinds of instructions are assembled, and I did not find a documented way to inspect the assembled instructions for a particular review. That capability is unknown from the sources I read, which is different from saying it does not exist.

The check that works on every tool is a canary rule. Write one rule scoped to a single glob that forbids one obvious thing, open a pull request that violates it inside a file matching that glob, and confirm the finding appears. Then move the violation to a file outside the glob and confirm it does not. That is ten minutes of work and it exercises detection, scoping, precedence and firing in one pass. Repeat it for the language you care most about, because rule evaluation is not uniform across file types, and repeat it after any change to the rule file layout.

The criteria that decide this

Tool	Rule files detected	How scope is decided	How precedence resolves	Can you see which rule fired
Kodus	IDE rule files plus `.kody/rules/`, `rules//.md`, `docs/coding-standards/*`	Nested files scoped to their directory; per-rule path globs; directory-level settings	Global to repository to directory inheritance; non-inheritable rules and child exclusions that do not cascade	Yes on self-hosted, via `[kody-rules-sync]` and `[kody-rules-eval]` log markers
CodeRabbit	`AGENTS.md`, `.cursorrules`, Copilot instruction files, `CLAUDE.md`, `GEMINI.md`, `.cursor/rules/`, `.windsurfrules`, `.clinerules/`, `.rules/*` (case-sensitive)	Directory-scoped by default, with explicit `applyTo` mapping and cross-repo `repo:path` sources	Inheritance off by default; when on, chain stops at first false or unset level, and arrays dedupe by path so a parent entry can be dropped	Yes, `@coderabbitai configuration` prints the resolved config with source annotations
GitHub Copilot	`copilot-instructions.md`, `NAME.instructions.md`, `AGENTS.md`, `CLAUDE.md`, `GEMINI.md`	Nearest `AGENTS.md` in the directory tree wins; path-specific files apply on path match	Positional for `AGENTS.md`; repository-wide and path-specific both apply when the path matches	Unknown from the documentation I read

A fourth option is what Cloudflare did: build a coordinator around an open-source agent and treat your internal standards as one more plugin with its own compliance check. That gives you full control over precedence and over what the trace looks like, and it costs an internal tool to own. Their post reports running it across tens of thousands of merge requests, which is the kind of volume that justifies the build.

What to do before you trust the claim

Look at how many rule files your repository actually contains, including the nested ones. Pick the rule you would most hate to see silently skipped and find its path glob, then check that glob against the files it is supposed to cover. If two files define guidance for the same path, determine which one wins on your tool and delete or scope the loser. Find the command or log marker that shows the resolved rule set, and run it on a real pull request rather than trusting the settings page. Then add the canary rule and keep it, because the thing that breaks rule enforcement later is a file layout change nobody connected to the reviewer.

A rule file is an artifact, and it deserves the same treatment as the code it governs: version it, scope it, and check that it fires. Teams that get real value out of "the reviewer follows our standards" can point at the file, the glob, and the log line. Teams that cannot are running a policy that exists in a repository and not in their review. If your rules live in a central standards repo, the multi-repo context question is the next one to settle, and if you are still deciding where the reviewer runs at all, the operational cost of self-hosting it changes which of these verification paths you get. A policy that leans on flagging AI code rather than on scoped rules has the same blind spot, one layer up.

Claims checked 2026-10-04.

The artifact worth reviewing when agents write the code

Tess Ainsley — Wed, 14 Oct 2026 00:15:05 +0000

GitHub's Security Lab reported 24 Android vulnerabilities found by an AI security agent, and the part worth copying is what ended up version-controlled. Not the exploit code. The set of YAML taskflows that guided the model through the audit, checked into a public repository, runnable by anyone with a Copilot license. The agent did the finding. The taskflows are the thing a team can read, diff, and argue about.

That is a small shift with a large consequence for review. When a person writes a patch, the diff is a decent summary of the judgment that went into it, because the person held the reasoning and the code is the residue. When an agent writes the patch, the reasoning lives somewhere else entirely: a prompt, a rule file, a model version, a sequence of tool calls. Reviewing only the residue means reading the output of a process you never see.

The diff stops being the whole artifact

Ian Bicking's post on what a serious AI product would look like puts the failure precisely. Checking for mistakes, he writes, is pushed off into code review, which encourages the author to hand the work to a reviewer without ever looking. He wants a two-column worksheet where every claim has a checkbox you only tick after you have verified it, and he notes that the vendors' own warning labels ("AI can make mistakes, double-check responses") are fine print designed to move responsibility onto the user.

His complaint is about products. The same structure shows up in review. If the process that produced a change is invisible, then the reviewer becomes the only place where checking happens, and the volume of changes is exactly what makes that unworkable. The fix is not to ask reviewers to work harder. It is to make the process itself a reviewable object, so that some of the checking happens where it can be repeated instead of where it cannot.

What a reviewable workflow actually looks like

The GitHub Security Lab agent is a concrete example. The taskflows are YAML files with names like gather_mobile_entry_point_info.yaml and classify_application_local.yaml. One splits entry points into mobile and non-mobile so the agent understands the right attack surface. Another enumerates vulnerability classes and asks the model to consider them against each entry point, so an intent-based entry point gets checked against known intent-based bug classes rather than whatever the model feels like noticing.

The team also ran both a strict prompt and a broad prompt across multiple runs, because the model is non-deterministic. Output lands in a SQLite table with a has_vulnerability column you filter with a checkmark. There is a real cost attached, and the post is upfront about it: a GitHub Copilot license, premium model requests, many tool calls, and one to two hours on a medium-sized repository.

Every one of those attributes is reviewable before you run anything. You can read the vulnerability class list and ask whether it covers your stack. You can see the run is repeated. You can see where the findings land and what they claim. Compare that with a code review comment that says "this looks suspicious" and moves on.

Cloudflare's CI-native reviewer takes a different shape to the same end. They built it on OpenCode and put several reviewers behind a deduplication coordinator, so a finding that two reviewers agree on surfaces once. There is also a check on whether the repo's AGENTS.md file is current. The coordinator is the reviewable part. Without it, the same issue appears four times and the reviewer learns to skim.

Rules are part of the workflow

The rules a reviewer applies live in the same category. Stack Overflow's post on building shared coding guidelines for AI is written mostly from the generation side, how to get agents to write to your standards, and it contains the line that matters for review: code review will be most engineers' first look at the code, since they did not write it. Vish Abrams, chief architect at Heroku, makes the point that seasoned engineers assume things like DRY as common knowledge, and that assumption does not transfer to an agent.

A rule that only exists in a config file the reviewer never reads is decoration. What matters is whether the reviewer ingests the files your team already keeps, AGENTS.md, CLAUDE.md, .cursorrules, Copilot instruction files, and scopes them so a rule for the payments service does not fire on the CLI tool. CodeRabbit reads repo config, Copilot reads instruction files, and Kodus ingests repo rule files as scoped rules rather than as loose context. Those are three shapes of the same idea, and the useful test is whether you can point at the file and the line that produced a comment.

A test you can run before you trust the workflow

Take any agent-produced change your team merged last week and try to answer these without opening the diff.

Where is the prompt or taskflow, and is it in the repository or in someone's chat history? Which model version ran, and would a rerun use the same one? Which rules were in scope for this file, and where do those rules live? What did the workflow check that a test suite would not catch? If the workflow found nothing, is that because there was nothing, or because the run was cut short?

If the answers are all "somewhere in a chat window," the review process is carrying work it cannot carry at volume. If the answers are files, you have something a second engineer can review before anyone merges anything, which is the point.

What this does to review time

None of this makes review faster in the sense of reading fewer lines. It moves a specific category of checking out of the reviewer's head and into an artifact that can be read once and rerun every time. That is the difference between reviewing a change and reviewing a process, and only the second one scales with the number of changes an agent can produce.

There is a cost, and it is worth naming. Versioning prompts and taskflows adds config files to maintain, and they drift, which is why Cloudflare checks the AGENTS.md currency at review time instead of trusting it. A workflow nobody updates is worse than no workflow, because reviewers will assume it still reflects how the team works.

The practical thing to do this week is boring. Pick the agent workflow your team uses most, find its prompt or config, and put it in the repository next to the code it governs. Then review it the way you review anything else, with the diff open beside it. Teams that already wrote a review policy around flagging AI code have this half done, because a policy is a workflow with a name. Teams that have not should notice that their reviewer is currently doing a job the workflow file should be doing, and that the plateau in review time on the largest pull requests is what that looks like when it stops working.

This piece first ran on agentwrotethis.dev, where I keep the source links for every claim.

How much of code review an AI reviewer actually carries

Tess Ainsley — Mon, 12 Oct 2026 22:30:01 +0000

Teams trying to handle a rising volume of agent-written changes usually ask how many findings an AI reviewer produces. The number that decides whether the workflow holds is smaller and duller: what share of the review that reviewer actually carries. That share is measurable, and the one firm-level measurement published so far puts it in the minority.

The measurement comes from a working paper by Fiona Chen and James Stratton at Harvard, Artificial Intelligence in the Firm: Bottlenecks in Software Production, in its August 2026 version. It draws on anonymised engineering-event metadata from 718 firms and roughly 300 million events between January 2021 and March 2026, supplied through Jellyfish. The design compares firms that adopted agent tooling at different points in time, so the estimates are observational. It is a working paper, and it has not been peer reviewed.

Ars Technica's report on the paper carries the top-line figures. After agent adoption, lines of code per worker rose about 30%, commits 20%, and pull requests 23%. Resolved issues and epics tracked in tools like Jira did not move in a statistically detectable way, with the 95% upper bound on the issue gain around 12% of baseline. That upper bound matters. The study cannot claim output fell, and it cannot detect a firm-level improvement either.

Downstream of the pull request, the elapsed time from submission to merge rose 49% on average, given in the paper as 3.45 days against a 7.03-day baseline. The share of pull requests with changes requested nearly doubled, and comments per pull request rose 35%.

Then the number that should change how you read a vendor page. By March 2026, nearly 80% of the firms in the sample had used some form of AI code review. AI agents accounted for 23.3% of all review comments, and 10.8% of pull requests received at least one AI-generated comment. A close reading of the paper flags one framing slip in the coverage: the 10.8% is the share of pull requests that received an AI comment, not the share authored by an agent.

The tooling is close to universal and produces about a quarter of the review text on the changes it touches. That ratio is the coverage figure, and it is the one to ask for.

Coverage is a configuration value, not a feature list

Coverage comes from four settings, and none of them appear on a pricing page.

Which events start a pass. A reviewer that runs when a pull request opens behaves differently from one that also reviews every new push, and differently again from one triggered by a repository or organisation ruleset that a pull request cannot override.

Which changes it skips. Skip conditions are where coverage quietly disappears. Kodus skips when there are no new commits since the last run, when a push contains only merge commits, when every changed file matches an ignore pattern, when the pull request exceeds the configured file limit, and when the user who opened it has no licence. It also pauses follow-up reviews during rapid consecutive pushes, three within fifteen minutes by default.

How much it posts. A severity threshold decides which findings reach the thread. Kodus defaults to medium, with an option to apply the same filter to violations of its own rules. Deduplication decides how many near-identical comments survive.

Whether the verdict has consequences. This one changes behaviour on both sides of the review. CodeRabbit's request-changes workflow is disabled by default, and when enabled its approval waits on the latest commit having completed a review, all its threads being resolved, and its pre-merge checks passing. Authors can clear that with @coderabbitai resolve or @coderabbitai approve unless reviews.allow_author_approval is set to false. GitHub Copilot code review's configuration page puts approval in public preview, with separate toggles for whether Copilot can approve and whether those approvals count toward merge requirements, plus file globs to limit which pull requests count. Kodus keeps request-changes and auto-approve off by default and documents the cases where turning them on makes sense, such as security-sensitive paths or repositories with stable rules and reliable CI.

Here is how those settings compare across the reviewers that come up in this conversation.

Reviewer	Starts a pass on	Skips when	Comment volume control	Verdict
Kodus	pull request opened, new commit per cadence, or `@kody start-review`	no new commits, merge-only commits, ignored paths, file limit, unlicensed user	severity threshold at medium by default, per-category toggles, dedupe before posting	request changes and auto-approve are separate toggles, off by default
CodeRabbit	pull request opened, new commits	path filters can exclude every changed file	inline comments plus pre-merge checks	request-changes workflow off by default, approval gated on a reviewed commit and resolved threads
GitHub Copilot code review	user setting, repository ruleset, or organisation ruleset	not stated on the configuration page	review effort level, with Lite and Balanced available and Max marked coming soon	approvals in public preview, can be set to count toward merge requirements
Cloudflare's internal reviewer	merge request opened	not published	a coordinator deduplicates findings from up to seven specialist reviewers into one comment	blocks merges on serious problems or security issues, per Cloudflare's own post

Two entries in that table are not products you can install. Cloudflare's system, described in their post on orchestrating AI code review, is built on OpenCode and runs inside their CI across tens of thousands of merge requests. The coordinator is the part worth copying, because it is what stops a seven-reviewer setup from posting the same finding seven times, and a reviewer who sees the same comment repeatedly stops reading it. The same idea applies to the rules a reviewer loads. If a rule file is read from the pull request head, the change can rewrite the standards being applied to it. Which files a reviewer reads is worth answering before installation, and Kodus documents how it imports and scopes rule files found in the repository, including a @kody-ignore marker for files that should stay in the repo without becoming review rules.

Queue wait is where the elapsed time goes

The 49% figure from the Harvard paper is calendar time between submission and merge. It is not reviewer minutes, and the paper does not report reviewer minutes at all. That gap matters, because the two clocks can move in opposite directions.

LinearB's 2026 engineering benchmarks, built from 8.1 million pull requests across 4,800 teams, reports that AI-generated pull requests wait 4.6 times longer before someone picks them up, and are then reviewed twice as fast once picked up. The same report puts acceptance for AI-generated pull requests at 32.7%, against 84.4% for manual ones.

Read together, those numbers describe a queue problem. A change that sits for hours before its first reviewer touches it shows up as a long merge time even when the review itself takes minutes. If your team is measuring one number for the whole path, an improvement to review quality will be invisible, and the thing that helps, getting to the diff sooner, will not be the thing you invest in. Splitting pickup time from review time is the cheapest useful instrumentation change available.

There is a second reason to watch the split. When review time on the largest changes flattens while volume climbs, the workflow is not getting faster. In Salesforce's account of its own review surge, that plateau meant reviewers had stopped meaningfully engaging with the changes in front of them.

When the queue stops working, teams close it

The QGIS project merged QEP428, which changes its policy to a strict not-allowed position on AI contributions. The discussion in that pull request is unusually concrete about why. A maintainer writes that AI use from external contributors puts massive and unbearable pressure on maintainers, and that the AI stream is higher than what human review can take seriously, leading either to a longer list of unreviewed pull requests or to code that was never fully reviewed. Another contributor reports roughly 50 pull requests tagged AI Assisted, of which about 40 were closed without merging. The mailing list thread behind the proposal includes a moratorium proposal and a separate suggestion that new contributors record a screencast of their change working before it is considered.

You can disagree with the policy and still use it as a measurement. QGIS ran the experiment, watched what it did to its review queue, and closed the intake. A team that would rather keep the intake open needs coverage numbers and queue numbers to know whether its version of the same trade is working, and most teams do not have them. Making the workflow itself reviewable is one way to get part of that back, since the artifacts worth reviewing when agents write the code are the prompt, the rule set, the model version, and the tool calls rather than the diff alone.

The metric that survives

Chen and Stratton measure what firms produced. Another paper measures what survived the trip to users, and it points the same direction. NBER Working Paper 35275, Writing Code vs. Shipping Code by Mert Demirer, Leon Musolff and Liyuan Yang, follows more than 500,000 GitHub developers with their AI usage telemetry. Commits rise 30% with autocomplete, 180% with interactive coding agents, and 240% with autonomous agents. Those gains attenuate as you move down the production chain: 80% for the number of projects and 30% for actual releases. The authors estimate an elasticity of substitution of 0.23 between AI effort and human effort, which they read as strong complementarity. The human step in the chain does not get cheaper just because generation does.

Neither paper measures whether the merged lines were still there a month later. That is unknown at firm scale, and it is the number I would want next. Until someone publishes it, the practical version is local. For a two-week window, log four things per change: time to first review comment, comments split by whether a human or a tool wrote them, the change-request rate, and what happened to the merged lines thirty days on, still present or already reverted. Those four columns will tell you more about your review workflow than any published benchmark, and they will tell you whether an AI reviewer is carrying enough of the load to justify its place in the path.

Coverage below half the review is not a failure. It is a description of where the work still sits, and it should change how you staff the queue.

Figures as of October 2026.

Where AI review saves time: the write-back, not the analysis

Tess Ainsley — Fri, 09 Oct 2026 00:15:03 +0000

A tool that "supports Azure DevOps" may only be able to read your pull request. What cuts review time is the write-back: posting an inline comment where the problem is, replying and resolving a thread, applying a committable suggestion, submitting a review, marking the pull request ready, merging it. Those are separate capabilities from the analysis, and on some hosts they are missing entirely.

That distinction matters for anyone trying to cut review time, because the analysis quality is the part every vendor page shows you and the write-back surface is the part you have to go digging for in the docs. The digging is worth it, because write-back is what the reviewer actually feels.

Reading support and writing support are different claims

Take a concrete example. CodeRabbit's Change Stack provider support page documents a per-provider capability matrix, and the Azure DevOps column is almost entirely empty on the writing side. Artifacts open read-only. No inline review comments, no file-level comments, no draft reviews, no submitting a review, no replying or resolving threads, no committable suggestions, no merge, no mark-ready-for-review, and no Change Stack chat. Azure artifacts also miss the live freshness check the other providers get, so an artifact cannot tell you the head has moved past the snapshot you are reading.

Read that page carefully, because it is about Change Stack specifically, which is a review workspace sitting on top of CodeRabbit's PR review feature. The claim is narrower than the headline suggests. CodeRabbit can still comment on Azure DevOps pull requests through its PR review feature. The workspace layer, where a reviewer would normally resolve and apply, is read-only on Azure DevOps, and that is where the difference shows up. "We support your provider" and "we can act on your pull request" are two different sentences, and only the second one moves your cycle time.

PR-Agent, the open-source reviewer that Qodo donated to the community, shows the same thing from the other direction. Its Azure DevOps installation page is explicit about a host limitation that has nothing to do with AI: Azure Repos Git does not honor YAML pr: triggers, so you wire the pipeline up through Build Validation as a required branch policy instead. The pipeline path also cannot start a run from a pull request comment, which the docs list as an open gap rather than a setting you missed. The webhook path does support a comment trigger, but only for API v2.0, and only after you register the webhook and give PR-Agent a stable identity on the project. On GitHub you type /review in a comment and the agent runs with nothing else to set up. PR-Agent covers GitHub, GitLab, Bitbucket, Azure DevOps and Gitea, so the same tool gives you a different amount of interactivity depending on where your code lives.

What the write-back surface looks like per tool

The useful way to compare reviewers is to build the list of actions first, then check which ones each tool can perform on your host. This is the list I use: post an inline comment, reply and resolve a thread, apply a committable suggestion, submit a review with a verdict, mark ready for review, merge or enter the merge queue, and run locally before push.

Tool	Hosts listed	Azure Repos write-back	Self-host / local	Notable constraint on the docs
CodeRabbit	GitHub, GitLab, Bitbucket, Azure DevOps	Change Stack artifacts read-only: no comments, suggestions, review submission, or merge	Not advertised as self-hosted	Draft reviews on GitLab and Bitbucket are held by the vendor and expire after 24 hours; Bitbucket needs authentication even for public repos
PR-Agent (open source)	GitHub, GitLab, Bitbucket, Azure DevOps, Gitea	Pipeline and webhook paths work; no PR-comment trigger on the pipeline path, Build Validation policy required	CLI, Docker, self-hosted	`/help_docs` disabled since v0.36.1 pending a credential-exposure fix
GitHub Copilot code review	GitHub and GitHub Enterprise	Not applicable	Runs in GitHub's infrastructure	Hosted review feature, so it is not a candidate if your repos are not on GitHub
Kodus	GitHub, GitLab, Bitbucket, Azure Repos, Forgejo	Works in pull requests on Azure Repos; self-hosted runners supported	Self-hosted or hosted, CLI, CI/CD	Community tier caps Kody Rules at 10; SSO, RBAC and audit logs are Enterprise only

The Kodus row comes from its repository README, which lists pull request support across those five hosts and self-hosted runners, and from its pricing docs, which put Kody Rules at up to 10 on Community and hold SSO plus audit logs for Enterprise. Qodo also publishes an enterprise Azure DevOps page describing review embedded in the ADO pull request workflow, and while it argues the case well, it does not itemize the write-back actions per provider, so the specifics stay unknown from that page. GitHub's own Copilot code review documentation confirms that feature is scoped to GitHub and GitHub Enterprise.

Why this shows up as review time rather than as review quality

A reviewer's minute goes into reading and deciding. The bookkeeping that follows is a separate cost, and it is the one that scales with the number of pull requests rather than with their size. If a tool can apply a suggestion in place, the author never re-opens an editor, never re-pushes, never waits for the reviewer to come back. If it cannot, every finding becomes a message that a human has to transcribe into code, which is why tools with identical finding quality can produce different cycle times on different hosts.

This is also where the plateau in review time comes from, the one Salesforce reported when code volume rose 30% and review time on the largest pull requests flattened. Teams that measure time-in-review per pull request often find the number stops improving after the first few weeks, even as the tool keeps producing findings. The reading got faster and the bookkeeping did not, so the floor is set by the writing side.

There is a cost dimension too. Self-hosting changes which write-back paths are even available to you, because a locally hosted runner can act through the provider API in ways a hosted app cannot, and the tradeoff is operational. That is a separate decision from the one this article is about, and I covered the operational side in what self-hosting AI code review actually costs.

How to audit this on your own backlog

Do not start from the vendor list. Start from your last fifty merged pull requests and count, per pull request, how many review comments required a code change and how many were resolved without one. The first number is your write-back demand. If it is small, write-back gaps will not hurt you and analysis quality is the only thing worth comparing. If it is large, the capability matrix above is the more important document on any vendor page.

Then run the check by hand once, on your actual host. Open a pull request, ask the tool to review it, and try to do each of these from inside the provider: apply the suggested change, resolve the resulting thread, answer the agent in the same thread, and mark the pull request ready. Whatever refuses is your real write-back surface. It takes twenty minutes and it beats reading any comparison page, including this one.

Two smaller signals are worth watching as you go. One is whether the tool's reply loop survives without a comment trigger; if your host cannot start a run from a pull request comment, then every re-review is a pipeline run, and each one has a cost and a queue position. The other is whether the agent knows the difference between a fresh diff and a snapshot. CodeRabbit's docs say Azure artifacts are static and cannot compare against the head. A reviewer working from a stale artifact is not being helped, and the failure is silent.

What to ask a vendor

Ask which write-back actions are supported on your provider, by name, and ask for the docs page rather than the answer. Ask whether the tool can run as you or only as itself, because on some hosts the actions run under the tool's own identity and that changes who can resolve a thread. Ask what happens to a draft review if the session drops, and ask how a re-review gets triggered once the author pushes a fix. If the answer is a matrix with a mostly empty column for your host, that is not a defect in the product. It is a fact about what the review will feel like, and it is better to learn it from the docs than from a sprint that did not get shorter.

Claims checked as of 2026-10-05.

Four config gates decide if an agent's PR reaches a reviewer

Tess Ainsley — Tue, 06 Oct 2026 00:00:11 +0000

An agent opens a pull request with a clean diff, and whether a human ever reads it depends on configuration most teams wrote before agents existed. Four gates sit between the change and a reviewer: who gets requested, whether a reviewer ran at all, what that reviewer was allowed to post, and whether any policy requires a human approval. Each gate can exclude the change without anyone noticing, and the same two fields drive all four, the author of the commits and the branch the change targets.

That is worth checking before you buy anything, because a reviewer that never runs looks exactly like a reviewer that found nothing.

The reviewer of record is chosen from the base branch, not the diff

GitHub reads CODEOWNERS from the base branch of the pull request, not from the branch the agent pushed. If the agent opened the PR against main but the ownership file lives on a release branch, no code owners are requested. The same rule applies to forks: the PR uses the upstream base branch's file.

Two more constraints in GitHub's CODEOWNERS documentation matter for agent workflows. Code owners are not automatically requested on draft pull requests, and they only get requested once the PR is marked ready. When the owner is a team, the team has to be visible and hold write permission on the repository, which is a common gap when ownership sits with a team that was renamed or moved.

Then branch protection decides whether the review is decorative. Requiring an approving review, or requiring approval from a code owner, is a separate setting from having owners at all. Two details in GitHub's protected branch documentation are easy to miss: restrictions do not apply to people with admin permissions unless you turn that on, and only a single branch protection rule applies to a branch at a time, so overlapping rules resolve by priority rather than by combining.

For an agent-authored change, the practical question is whether the approval that satisfies branch protection can come from the agent's own account or from a service identity with write access. If it can, the gate is open.

The exclusion lists were built to filter bot noise

Every reviewer ships skip rules, and those rules were written when the main problem was dependency bots opening PRs nobody wanted to read.

CodeRabbit's automatic review controls are the clearest example. The documented GitHub entries for ignore_usernames are dependabot[bot], renovate[bot], and github-actions[bot]. Matching is exact and case sensitive, with no wildcards. Username skipping also takes precedence over every other control, so a PR from a listed author is skipped regardless of labels or any other setting. Skips are silent: ignore_title_keywords drops the PR with no review posted, drafts are skipped by default, and base_branches defaults to the default branch only, so a PR targeting develop gets no automatic review until you add the pattern.

Kodus documents the same class of settings in its review configuration reference. Default ignored paths include yarn.lock, package-lock.json, package.json, .env, and **/*.json. Draft PRs are reviewed by default, which is the opposite of CodeRabbit's default. An empty baseBranches means every base branch is reviewed. Review cadence defaults to pausing after three pushes inside a fifteen minute window, and Kody's skip reasons include a configured file limit and a PR where every changed file matched an ignore pattern.

Two Kodus behaviors are worth flagging for anyone who treats a config file as the source of truth. kodus-config.yml is ignored until kodusConfigFileOverridesWebPreferences is turned on, so committing the file alone does nothing, and unknown keys in that file are dropped without an error or warning. A typo reads as a setting that had no effect.

PR-Agent runs its review when the command is invoked or when the workflow triggers. The GitHub Action in the PR-Agent README fires on pull_request types opened and synchronize, and the CLI or comment path (/review) is how the tool is reached otherwise. There is no automatic pass on a PR nobody points it at. The same README notes that /help_docs is disabled pending a fix for a credential-exposure issue, which is a reminder that the trigger path is also the credential path.

Put together, the failure mode is specific. If your coding agent commits under a service account, an app identity, or a bot name, any exclusion list written to silence dependency bots will silence it too, and the skip is silent. You find out by looking for a review that does not exist.

The same author field can be used to require a human instead

Anthropic's agent approval check reads the same author field in the opposite direction. It requires a set number of human approvals, two by default, on any PR containing commits authored by a configured agent identity, and it hooks into branch protection so the requirement is enforced at merge rather than suggested in a comment.

The contrast is the useful part. The identity of the author is available in both cases. One policy uses it to skip the change, the other uses it to demand a human signature on it. Teams usually end up with the first policy by accident, because the bot exclusions were already there, and the second only when someone writes it deliberately.

What the reviewer is allowed to post decides whether it can hold a merge

A reviewer that comments and a reviewer that blocks are different products, and most tools ship the commenting version.

Kodus posts non-blocking comments by default. Its review policy page states that requesting changes and auto-approving are both opt-in, that request-changes is not available on GitLab, and that blocking behavior should only be enabled when the team agrees on thresholds and has strong CI.

CodeRabbit's request changes workflow is disabled by default. When enabled, approval stays pending until the latest commit has been reviewed, required threads are resolved, and pre-merge checks pass, and CodeRabbit re-verifies that the PR head has not changed immediately before approving. Two escape hatches matter: explicit @coderabbitai approve and @coderabbitai resolve commands override the workflow, and a rate-limited run leaves approval pending because the latest commit never completed a review.

That last point is where review tooling and review policy interact. If your merge requirement is satisfied by a tool's approval, then the tool's rate limit, skip rules, and configuration defaults become merge requirements.

Reviewer	Default on a PR from an agent identity	Silent skips to audit	Blocks a merge by default
Kodus	Reviews, including drafts	Ignore paths (includes `*/.json`), ignored title keywords, base branch narrowing, cadence pause, file limit	No. Request changes and auto-approve are opt-in, and request changes is unavailable on GitLab
CodeRabbit	Reviews, drafts skipped	`ignore_usernames` (takes precedence over everything), `ignore_title_keywords`, base branches limited to the default branch, pause after five reviewed commits	No. Request changes workflow is off by default and has explicit approve and resolve overrides
PR-Agent	Runs only when invoked by comment or workflow trigger	Nothing automatic to skip, because nothing runs unprompted	No. It posts review output

The defaults differ in ways that change coverage. Kodus reviews drafts and every base branch unless narrowed. CodeRabbit skips drafts and reviews only the default branch unless you add patterns. Neither blocks a merge without an explicit opt-in.

A check you can run on one agent PR this week

Take a merged PR that an agent authored and walk the four gates in order. Find the review request list and confirm a human or team actually appears, which tells you whether CODEOWNERS resolved from the base branch. Look for evidence that a reviewer ran, a review, an inline comment, a status reaction, and treat silence as a skip rather than a clean pass. Open the reviewer's config on the base branch and look for the author name or title pattern that would have matched this PR. Finally, check whether branch protection requires a human approval that the agent's own account cannot provide.

Any of the four can be wrong on its own, and the symptoms are identical from the outside. A change that never reached a reviewer, a reviewer that ran and found nothing, and a reviewer that found something nobody was required to read all show up as a merged PR.

The wider shift this belongs to is that review effort has moved toward deciding whether a change is acceptable, which is what the artifact worth reviewing when agents write the code covers from the process side. Routing is the part that comes before any of it. A team can buy the best reviewer available and still have agent output land unread, because the config that decides who reads what was tuned for a different kind of author. Salesforce's numbers on review time plateauing while code volume rose are the downstream version of this, and GitHub's own eight step checklist for reviewing AI-generated code assumes a human is already in the loop.

Checking the four gates costs about ten minutes on one PR, and it tells you whether the rest of your review investment is connected to anything.

Your pull request review time is mostly waiting and rework

Tess Ainsley — Wed, 30 Sep 2026 00:15:11 +0000

Time to merge is the number teams quote when they say AI review sped things up. It bundles at least three separate costs, and reading the diff is only one of them. When the number drops after adding a tool, the drop usually comes from the other two.

Two sets of measurements make the split concrete. CodeRabbit's guide to reviewing AI-generated diffs cites LinearB's 2026 benchmark report: AI-assisted pull requests are 2.6 times larger than unassisted ones, take 4.6 times longer to receive a first review, and have a 30-day acceptance rate of 32.7 percent against 84.4 percent for manual PRs. CodeRabbit is careful to label those as observed associations rather than causal effects, and the association framing is the honest way to read them. The acceptance gap is the part worth sitting with. If roughly two of every three AI-assisted changes are not accepted within 30 days, the cycle spans several passes rather than one review.

The second set is narrower and better instrumented. Obada Kraishan's study of five autonomous coding agents covers 37,623 provenance-labeled PRs across 2,807 repositories, with the pipeline code released for replication. It is a preprint, not peer reviewed. Review effort concentrates unevenly: Copilot PRs drew the most human reviews and change requests, and Claude Code PRs waited a median of 12.6 hours for a first human review. Post-merge outcomes differ by vendor too, with Codex PRs reverted about half as often as human PRs, 6.1 percent against 11.5 percent, while Devin PRs were reverted more often at 14.5 percent.

That 12.6-hour median is the number to keep. Reviewers do not read diffs for twelve hours. The wait is queue time: an agent opens a PR, and nothing happens until a human picks it up. A tool that reads faster than a human shortens the few minutes at the end of that window and leaves the twelve hours at the front of it untouched. Teams that buy review speed by buying reading speed often watch the merge metric barely move, and this is why. The 441 percent increase in code-review time that shows up in the vibe coding literature sits on top of a queue that was already the slow part.

Wait, read, and rework are different numbers

Split the cycle at four timestamps: when the PR opens, when the first review lands, when the first change request lands, and when it is approved and merged. Wait time is the gap between the first pair. Reading time is between the second and third. Rework is everything after a change request, including the author's fix, the re-review, and any further round trip.

Instrument those four per PR, split by provenance and by repository, and the tool decision falls out of the data instead of the demo. If wait dominates, the lever is the trigger. If rework dominates, the lever is whether the tool produces one precise change request or a stream of comments the author answers in four passes. If reading time dominates and your diffs are large, reading speed is finally the right thing to buy.

The rework share is where vendor differences show up most clearly. Copilot PRs drew the most reviews and change requests in the Kraishan dataset, which means more round trips per change, not slower readers. Round trips are also the cost that a review tool can multiply. Ten findings in one pass is one loop. The same ten findings spread across four review cycles is four loops, and each loop carries a context reload, a re-review, and a chance that the author argues with a comment that already stopped being true.

Trigger timing is a review feature

When a review runs matters as much as what it says. A review that fires on PR open puts its feedback into the same window as the wait, so the author and the assigned reviewer see the same first pass. A review that fires after CI, or only when a human asks for it, adds its latency on top of the queue instead of overlapping it. Ask any vendor for the trigger list: open, ready-for-review, CI green, manual comment command. Where the tool sits in that list determines whether it can touch the biggest number in the cycle.

Rules cut round trips only when they fire on the right files

Stack Overflow's post on coding guidelines for AI and people contains the line that ties standards to cycle time. Code review will be most engineers' first look at code they did not write. Heroku chief architect Vish Abrams makes the related point there that principles seasoned engineers assume, like DRY, are not common knowledge to an agent. Rules are supposed to move the standards check earlier so a human reviewer does not spend the first pass on naming and layout, which is exactly the work that generates change requests when it is missed.

A rule only reduces that work if it fires on the right files. A payments rule that fires on the CLI tool produces comments the author has to triage, and triage is a round trip. Kodus imports the rule files teams already keep, including AGENTS.md, CLAUDE.md, .cursorrules, Copilot instruction files, Windsurf rules, and docs/coding-standards, scopes each rule to a path glob, and discovers nested files so a services/billing/CLAUDE.md applies to services/billing/** without extra setup. On self-hosted deployments it writes a per-file trace to the API log under the marker [kody-rules-eval], listing the rule ids selected into the prompt for each reviewed file. The same page documents a limit worth knowing before you plan around it: an unlicensed Community Edition instance evaluates at most 10 rules per review, oldest first, so a rule past the tenth may not fire at all. A rule that silently does not run is one fewer thing checked before a human, and one more thing that turns up later as a comment.

The trace matters beyond debugging. It is the difference between a tool that states your standards load correctly and one where you can point at the rule id and the file it evaluated. Everything else in this section assumes that mechanism works.

What each tool documents about the rework half

The criteria below are the ones that affect round trips: which rule files a tool ingests, whether rules are scoped, and whether you can see what ran on a given file. Unknown means the source I read did not say.

Tool	Rule files ingested	Path scoping	Per-file trace
Kodus	AGENTS.md, CLAUDE.md, .cursorrules, Copilot instructions, Windsurf rules, docs/coding-standards	Glob per rule, nested files discovered and scoped	`[kody-rules-eval]` log marker per reviewed file on self-hosted
CodeRabbit	Repo configuration and custom pre-merge checks, per its review guide	Unknown	Unknown
GitHub Copilot	.github/copilot-instructions.md and nested instruction files, per the patterns Kodus documents as shared conventions	Unknown	Unknown

For large multi-repo organizations there is a second question that sits under the same heading, which is whether the reviewer can see the dependents of a changed file at all. A change that breaks a caller in another repository is a rework loop nobody can avoid by reading the diff harder, and cross-repo context is the capability that decides it. The same logic that pushes effort toward coupled code applies here, since the codebase matters more than the volume when you decide where review attention goes.

Instrument before you buy

Run the four timestamps for a month before you evaluate anything. Report wait, reading, and rework separately, split by provenance and by repository, and you will know which of the three is eating the cycle. Then ask each vendor which of the three its product moves: the trigger window, the number of change requests per merged PR, or the time a human spends reading. A tool that can answer for the first two is reducing cycle time. A tool that can only answer for the third is reducing the smallest number on most teams, and its benchmark will still look impressive, because reading speed is measurable while a queue nobody instrumented stays invisible.

Claims checked 2026-09-29.

What the top AI code review comparison pages in 2026 leave out

Tess Ainsley — Tue, 29 Sep 2026 00:15:01 +0000

Search for a 2026 comparison of AI code review tools and most of what comes back compares coding assistants instead. SWE-bench scores, model pricing, IDE integration, which agent resolves more GitHub issues. Useful for picking something that writes code. Almost nothing about what decides whether a review tool can actually gate a merge.

That gap matters more now that agents author a large share of the diff. CodeRabbit's guide to reviewing AI-generated diffs cites LinearB's 2026 benchmark data: AI-assisted pull requests are 2.6 times larger than unassisted ones, take 4.6 times longer to get a first review, and have a 30-day acceptance rate of 32.7 percent against 84.4 percent for manual PRs. LinearB labels those as associations, not causal effects, and the acceptance gap alone is worth sitting with. Whatever the cause, the review step is where those changes are accepted or rejected, and a comparison that never tests the accepting step is answering a different question.

What the ranking pages actually compare

The top results for the query are honest about what they measure. Paperclipped's Cursor vs Claude Code vs Copilot vs Devin breakdown leads with SWE-bench Verified numbers (Claude Code at 80.8 percent on Opus 4.6, Cursor around 63 to 65 depending on model, Copilot around 58, Devin near 67 on its own metric) and then notes something more interesting: three different tools running the same Opus model landed 17 problems apart across 731 SWE-bench issues in February 2026 testing, which says the scaffolding around the model moves the result as much as the model does. It also cites SWE-CI, where 75 percent of agents broke previously working code during continuous integration even when their initial patch passed tests.

Tembo's roundup of Copilot alternatives covers 15 tools, and Tembo sells review infrastructure, so the list ends near its own product. bugstack's AI bug-fixing comparison has the most useful taxonomy of the batch, splitting bug-fixing tools into copilots, AI code review, and autonomous repair, then comparing them on whether a human has to start the work. Bugstack also sells in one of those categories.

None of that is dishonest. It is just generation-side measurement applied to a review-side decision. The one thing a buyer needs to know about a reviewer, whether it can stop a bad change, is not in any of them.

The criteria that decide a review tool

Four things separate review tools, and they are testable from the vendor's own documentation.

Who reviews, relative to who wrote. CodeRabbit's explanation of what code review is makes the point plainly: when the authoring agent reviews its own output, that is a closed loop with no outside check, and independence gets more valuable as more code comes from an agent. Any comparison of review tools should state which agent produced the diff and which one comments on it. Nearly none do.

What the review does to the merge gate. A comment and a blocked merge are different products. GitHub Copilot's code review posts a Comment review by default rather than an approval or a change request, so it advises without gating. Kodus documents the same shape and makes it explicit: suggestions are non-blocking by default, and marking a review as changes requested or auto-approving are opt-in switches a team turns on when it has the CI to back them. This is worth verifying per tool because the default is what most teams ship with.

Whether the tool records which rule ran on which file. A standards rule that lives in config and never fires on a file is decoration. Kodus imports the rule files teams already keep, including AGENTS.md, CLAUDE.md, .cursorrules, and Copilot instruction files, scopes each to a path glob, and on self-hosted deployments writes a per-file evaluation trace you can grep with [kody-rules-eval] to see which rule ids were selected into the prompt for each reviewed file. That trace is the difference between a tool that claims to follow your standards and one you can audit. GitHub's own guide to reviewing AI-generated code puts the standards check on a human reviewer, which leaves the same audit question open.

Deployment path and rule limits. CodeRabbit's pricing page lists self-hosting and custom RBAC under Enterprise, with managed plans from $24 to $72 per developer per month billed annually, and the old Pro and Pro Plus names replaced by Essentials and Team at unchanged monthly prices. Kodus publishes its source and supports self-hosting, with one documented caveat worth reading before you plan around it: an unlicensed Community Edition instance evaluates at most 10 rules per review, oldest first, and licensed instances have no such cap. If a rule past the tenth silently does not fire, that limit is why. I did not find an equivalent published limit for Greptile or Qodo, so those stay unknown until someone reads their docs.

A comparison by review mechanism

Verified from each vendor's own pages where noted; unknown means the source I read did not say.

Tool	Reviewer independent of authoring agent	Merge-gate default	Rule ingestion and trace	Deployment
CodeRabbit	States independence as a requirement; does not tie reviews to the authoring agent	Pre-merge checks on Team tier and above	Custom pre-merge checks on Team and above; trace behavior unknown	Self-host on Enterprise
Kodus	Review runs as a separate agent from the PR	Non-blocking comments; changes-requested and auto-approve are opt-in	Imports AGENTS.md, CLAUDE.md, .cursorrules, Copilot instructions; per-file rule trace in self-hosted logs	Open source, self-host; 10-rule cap unlicensed
GitHub Copilot	Reviews the PR, not the writing agent	Comment	Repo instructions via Copilot instruction files; trace unknown	GitHub and Azure Repos
Greptile	Marketed as independent review with custom context	Unknown	Learning and custom context documented; rule-file import unknown	Unknown
Qodo	Positions review as separate from generation	Unknown	Governance framing across the SDLC; rule trace unknown	Unknown

The unknown cells are the point. A comparison page that fills them with confidence it did not verify is doing the reader a disservice, and the reader can close them in an afternoon by reading one docs page per tool and running a single agent PR through each.

Why this decides the buyer question

The question teams actually ask is how to review the growing volume of AI-generated code, and the ranking pages answer a neighboring question about producing it. The measured evidence says volume and scrutiny rose together, which is roughly what my earlier look at the 441 percent review-time increase points at. A tool that only adds comments scales the queue with the diff. A tool that runs a pre-human gate, where the reviewer is separate from the authoring agent and you can see which of your rules it applied, moves work out of the queue instead.

That is also the cheapest thing to verify about any candidate, including the ones I could not fully check here. Read the review policy page. Read the rules-import page. Run one agent-authored PR and look for the evidence of which rules fired. If the tool cannot show you that, the standards claim is marketing, and what self-hosting actually costs is a separate bill you can price before you commit.

The self-hosted AI review decision is really a data residency decision

Tess Ainsley — Mon, 28 Sep 2026 18:00:02 +0000

The most common question about self-hosting an AI code reviewer is which tool is cheapest. The test that actually answers it puts the number somewhere else. Augment Code ran ten open-source AI code review tools against a 450K-file Python, TypeScript, Java and Go monorepo over 40+ hours, published 2026-01-16 and updated 2026-08-17, and the finding that reframes the decision is that the license is the free part. Everything around it costs money.

The published cost is not the license

Augment estimates a self-hosted stack at $4,100 to $9,100 a month at any team size, combining published GPU rates with 0.25 to 0.5 FTE of maintenance at the US Bureau of Labor Statistics mean developer wage. The comparison point in the same test is $24 to $30 per developer per month for a commercial per-seat reviewer. On that math, self-hosting is not the cheap option. It is the privacy and data-residency option, and the shape of the cost changes from a subscription line item to a capital and staffing one.

That range comes with a method caveat worth stating. The hourly components are itemized and the assembly is public, but it is an estimate built from rates, not a bill from a real deployment. The bottom of the range assumes modest GPU use. A team running large models on every pull request, or needing high throughput across many repos, should plan on the high end.

Install quality filters the list before features do

Ten tools went in. Three held up. The rest lacked maintenance, broke during configuration, gated the controls that matter behind a commercial license, or reviewed files in isolation. For anyone choosing off a self-hosted query, the install path is the first real signal, because a tool that will not stand up in a 40-hour test is not a tool you will keep.

SonarQube Community Build was the strongest on detection, with near-zero false positives across 21 languages, though its analysis runs on the main branch only. Semgrep came second on custom rules, and caught framework-specific patterns the generic tools missed.

On local inference, the results separate cleanly. Tabby self-hosted as documented and runs local models through Ollama, with review staying secondary to its code-completion focus. PR-Agent also supports local inference, but a configuration issue, #2098, caused silent fallback to hosted models during testing, which defeats the purpose of a local stack if nobody catches it. Kodus and Hexmos LiveReview run local models on your own infrastructure as well.

Two smaller entries show why maintenance status matters. anc95/ChatGPT-CodeReview is a reasonable free experiment. villesau/ai-codereviewer, the more familiar name, has shipped nothing since December 2023, and Augment notes it defaults to a model snapshot that OpenAI retires in October 2026. A self-hosted tool that depends on a retired model snapshot is not self-hosted in any useful sense.

CodeQL is worth naming for a different reason. If a team already pays for GitHub Code Security at $30 per active committer per month, it is the strongest fit and it caught data-flow vulnerabilities across multiple function calls in testing. If not, it is not the cheap entry.

"Open source" and "auditor-ready" are two questions

This is where the label and the checklist come apart. Most of these tools ship an open-source core and hold the enterprise controls behind a commercial key.

SonarQube Community Build gates audit logging at Enterprise Edition. Kodus puts SSO, RBAC and audit logs behind a commercial license key. Tabby documents no audit trail at all and places single sign-on on its Enterprise plan. So the honest answer to "is it open source" is yes for the code, and not by default for the controls a security review will ask about. If the reason you are self-hosting is compliance, check the control matrix before the license, because the license being free tells you nothing about whether the deployment will pass.

The requirement open source genuinely satisfies better than any hosted option is data residency. That is the axis the choice should start from, and it is why the published cost of a self-hosted stack matters less than it looks. You are not buying a cheaper product. You are buying a different placement of the model and the data, and paying for it in infrastructure and maintenance hours.

Where all ten stop

None of the tools in the test detected cross-service breaking changes across the four languages. Every one of them operates at file level. That is a real ceiling, and it sits exactly where agent-generated code creates risk. A file-scoped reviewer can read a diff and miss the caller, the invariant, or the ordering assumption the change breaks. Teams hit the same gap on any review that spans more than one repository, and no amount of local GPU capacity changes it.

How the options compare on the criteria that decide it

Tool	Self-host path	Enterprise controls	Notable result in the test
SonarQube Community Build	Yes	Audit logs gated at Enterprise Edition; main-branch analysis only	Strongest overall, near-zero false positives over 21 languages
Semgrep	Yes	Data not published for the tested build	Second on custom rules, caught framework-specific patterns
Tabby	Yes, worked as documented	No audit trail documented; SSO on Enterprise	Local inference via Ollama; review is secondary to completion
PR-Agent	Yes, local inference supported	Data not published	Config issue #2098 caused silent fallback to hosted models
Kodus	Yes, local models on your infrastructure	SSO, RBAC and audit logs behind a commercial license key	Included in the tested open-source set
Hexmos LiveReview	Yes, local models on your infrastructure	Data not published	Included in the tested set
CodeQL	Yes	Tied to GitHub Code Security	Caught data-flow vulnerabilities across function calls; $30 per active committer per month
villesau/ai-codereviewer	Yes	Not documented	No commits since December 2023; depends on a model snapshot retiring October 2026

What to do with this

Decide the model placement first and the tool second. If code cannot leave your network, the shortlist is SonarQube Community Build, Semgrep, Tabby, PR-Agent, Kodus, LiveReview and CodeQL, and the tiebreakers are install evidence, which controls you actually need, and whether the repo is still active. If code can leave, the hosted per-seat figure is lower and the whole exercise is a price check rather than a residency decision.

Then run your own numbers with your own GPU sizing and your own maintenance estimate, because the published per-seat figures for hosted tools are stable while the monthly cost of self-hosting is the one teams most often leave out. The license is the only free part.

Read the original test for the parts I left out, including the per-tool configuration detail and the local-inference setup notes, at Augment Code's 450K-file monorepo test.

Azure DevOps repos quietly disqualify half the AI review tools

Tess Ainsley — Mon, 28 Sep 2026 17:00:02 +0000

If your repositories live in Azure Repos, most AI code review comparisons are close to useless, because on Azure DevOps the thing that decides whether a tool works is the integration path, and not the model behind it. A reviewer can be strong on GitHub and still be a fight to turn on in Azure Repos: different auth, a different permission model, a different billing route, and in at least one case a feature that is still in limited preview with no service level agreement attached.

I read the primary docs for the tools that keep coming up for Azure DevOps, and compared the parts that actually decide whether you can switch them on: the connection method, whether the bot posts as itself or under a person's name, the write scopes it needs, how automatic review triggers, and where the cost lands. The vendor listicles rank themselves first and test none of this.

The setup details decide the answer

On GitHub, auth is one OAuth click and the bot is a known identity. Azure DevOps splits the identity model. You either hand the tool a personal access token, which makes reviews show up under a human's name, or you register a Microsoft Entra application and add its service principal to the organization as a user. The second path gives you a clean bot identity and a real audit trail. The first path is faster and leaves you with a review that looks like a colleague wrote it.

The permission model has a wrinkle that catches people. A tool that posts comments and updates pull requests is doing write operations, and Azure DevOps requires an identity read scope on top of the resource-specific write permissions so the credential can resolve who it is acting as before it writes anything. In CodeRabbit's Azure DevOps documentation this is called out directly: write operations need Identity: Read (vso.identity) in addition to the code and work item permissions. Miss it and the setup looks complete until the bot fails to comment.

Automatic review also works differently here. GitHub Copilot's Azure Repos integration does not fire on every pull request by default. You configure a branch policy per target branch to get unpaid, unrequested reviews. The tool's review effort level is a project setting with an optional repository override, and higher effort means more tokens and more cost, billed through Azure Cost Management because the feature needs an Azure subscription linked to the organization.

For anyone on Azure DevOps Server instead of the cloud service, there is a network step that appears in several of these setups: you allowlist the vendor's IP so the hosted bot can reach a self-hosted instance. Kodus documents its connection to Azure DevOps through a token, and its docs list 52.55.217.197 as the IP to allow for Azure DevOps Server, self-hosted GitLab, and Bitbucket Data Center.

What the primary documentation says, tool by tool

GitHub Copilot code review for Azure Repos is the option with the most caveats. The Microsoft Learn page marks it as limited preview, meaning preview features have no SLA and limited support, and functionality can change without notice. Turning it on takes three scopes: a Project Collection Administrator enables it at the organization, a Project Administrator can enable it at the project, and a repository admin enables it per repo when overrides are allowed. TFVC is not supported, only Git. Users may need to opt in through Preview features unless an administrator enables it for everyone.

Microsoft's own internal reviewer is the strongest measured case in the set. In an Engineering@Microsoft post from July 2025, the internal AI review assistant had scaled to cover over 90 percent of pull requests across the company, more than 600,000 PRs a month, and 5,000 onboarded repositories saw a 10 to 20 percent median improvement in PR completion time. Two design choices in that writeup are worth copying regardless of which tool you pick: the assistant never commits changes itself, the author clicks to apply a suggestion, and review behavior is configurable through repository-specific guidelines and custom prompts. Those numbers are the company's own reporting, not an independent benchmark.

CodeRabbit's Azure DevOps guide is the most explicit about the admin work. A Microsoft Entra service principal is the recommended method for new organizations and new Enterprise SSO workspaces, while existing PAT-based setups keep the PAT flow. Setup needs a CodeRabbit organization administrator plus Azure DevOps Project Administrator or Project Collection Administrator rights. The docs recommend adding the service principal to each project's Contributors group, and note it also needs permission to manage service hooks because that is how the webhooks get installed. Client secrets expire, and reviews stop until you rotate them.

Kodus takes a different default. The quickstart connects Azure DevOps with a token, and the platform is open source with a self-hosted deployment path, which matters if the repo cannot leave your network. Its review policy ships non-blocking comments by default, with request-changes and auto-approve as opt-in behaviors, so the merge gate stays a human decision until your team chooses otherwise. On the standards side, rules file detection imports the files teams already keep, including AGENTS.md, CLAUDE.md, .cursorrules, and .github/copilot-instructions.md, and self-hosted instances log a per-file evaluation trace under [kody-rules-eval] so you can see which rules actually ran on which file. That trace is the part most reviewers skip.

The comparison, on the criteria Azure DevOps cares about

Tool	Connection for Azure Repos	Posting identity	Extra permission noted in docs	Automatic review	Cost path
GitHub Copilot code review	Enabled at org, project, repo scopes	Copilot bot	Org/project/repo admin to enable	Branch policy per branch	Azure subscription via Cost Management
CodeRabbit	Entra service principal, or PAT	Configurable service account or PAT identity	Identity: Read plus resource write, service hooks	Project and repo defaults	Vendor plan
Kodus	Token, self-hosted or cloud	Kody bot	Repo and PR write for the token	Configurable cadence, non-blocking by default	Community/Teams/Enterprise, open source self-host
SonarQube	Extension plus server config	Analysis bot	Depends on server setup	Quality gate on the branch	Server license

Where a cell is not answered by the vendor's own docs, I have written the field narrowly rather than guess, because this is the kind of table that goes stale the moment a preview graduates or an auth default changes.

Why the lists you find do not help

The query for the best AI code review tools for Azure DevOps returns pages from CodeAnt, Panto, and similar sites, and in each one the vendor ranking first is the vendor hosting the page. None of them walk through the permission model I just described, and none of them mention that Copilot's Azure Repos support is in limited preview. That is not a knock on their products, it is a note that a list written to rank itself tells you nothing about the integration you have to live with.

The details that decide a rollout are the boring ones. Which identity posts the review, so your audit log is honest. Which scopes the credential needs, so setup does not half-work. Where the money goes, so a finance review does not surprise you a month in. Whether the tool records which rules it ran on which file, so a rule that exists only in config does not read as a rule that was enforced. That last one is the same problem I wrote about in a code reviewer that silently skips your rules, and it is worse on a platform with a stricter permission model, because there are more places for the chain to quietly break.

If you are running a pilot, the concrete next step is to request the write scopes before you request a demo. Ask which identity the bot uses, whether the review can block a merge or only comment, and whether the tool exposes a per-file record of which rules it evaluated. If the answer to the last one is a dashboard number with no per-file detail, you have found the limit of what you can verify after the fact. The self-hosting question also folds into this, and I went through what self-hosting AI code review actually costs separately, since on Azure DevOps the network and identity setup is where the hidden hours land.