Ask Google which AI code review tool to use on GitHub and the answer is a self-ranking list. The query "best AI code review tools for GitHub" currently surfaces a Reddit thread whose top comment is one person's five-tool list, then a chain of vendor pages. In its roundup, DeepSource writes that CodeRabbit is "the most installed AI code review app on GitHub, with over 2 million repositories connected." CodeAnt headlines its comparison with "300K-PR" in the title. Monterail reports it tested GitHub Copilot, Cursor BugBot, and CodeRabbit hands-on.
None of those counts are independently verifiable, and the ranking logic is circular. A vendor that sells a GitHub review app writes the list, puts itself first, and cites its own install number as the reason. The answer an engineering team needs is not one more of those lists. It is a way to test a candidate tool against the one entity that publishes checkable facts: GitHub.
What GitHub actually publishes
GitHub documents the two things that matter for any review tool, and both are checkable on your own screen.
First, the app permission model. Per the GitHub Apps overview in GitHub's documentation, apps extend GitHub using fine-grained permissions, and the person installing controls which repositories the app can access. Short-lived tokens limit what could be done if an app's credentials leak. That is the audit surface for review tooling: you can see, in the installation flow and in the app's declared permissions, exactly what the tool can read and write on your repos.
That is the single most useful fact about a code review tool, and no listicle leads with it. Before asking "is this tool any good at finding bugs," ask what it is allowed to do to your codebase. A tool that asks for broad write access to every repository is making a different security bet than one that reads a pull request and posts a comment. The fine-grained permission model is where that difference shows up, and it is published, not claimed.
Second, GitHub's own native review features. GitHub's code review page frames the current state plainly: code arrives "whether written by you, a teammate, or an AI agent," and the platform's answer is context plus gates. Stacked pull requests break a large change into reviewable pieces. A merge queue checks each PR against the latest base branch before merging. Rulesets protect branches. This matters because an AI review tool does not replace GitHub's review workflow: it plugs into it as another signal. A tool that works against stacked PRs, respects rulesets, and integrates with the merge queue is doing something different from one that only reads a single diff.
Distinguish the automated pass from the human decision
The one evaluation worth treating as more than marketing comes from a peer-reviewed study, not a vendor. In Communications of the ACM, researchers described an efficiency-throughput gap: individual developer efficiency gains do not translate to organizational throughput. The practical reading for review: an AI review tool that finds more issues per PR does not automatically shorten time to merge or reduce the human bottleneck, because those are different systems.
That is the reason to be suspicious of any tool advertisement that promises to cut review time by a percentage, or that reports "repos connected" as a quality signal. Repos connected is a reach number. It says nothing about whether the review is accurate, whether it respects your permission boundaries, or whether a human still has to read the diff. GitHub's Copilot code review documentation describes it as "an automated pass alongside your team's judgment." That phrasing, "alongside," is the correct mental model for every tool in this category. The pass is automated. The judgment is human. The tool's job is to make the human's pass cheaper and more accurate, and the only verifiable facts about that job are structural.
A short verification checklist
When you evaluate a GitHub integration, the number to check first is not precision or bug count, because those come from the vendor's own dataset. It is access and placement. Walk these in order.
Does the tool read the pull request, or the surrounding context? The diff alone is not review context. A tool that reads the issue thread, the linked PR description, and the base branch has more to work with than one that scores a raw diff. GitHub's permission model lets you see whether the app requests that breadth. Context is the lever that changes what an AI review can do, as issue-aware review benchmarks from ACM's Foundations of Software Engineering conference show.
Which repositories does it touch, and with what write scope? Fine-grained app permissions make this explicit. Prefer the narrowest grant that still lets the tool post feedback.
Where does the code run? This is a model-placement decision. If the tool sends source to a third-party model outside GitHub, that is a data boundary that has nothing to do with the feature list. Self-hosting shifts that boundary, which is why the cost profile changes with it.
Does it respect GitHub's workflow features? Stacked PRs, rulesets, and the merge queue are not optional extras for teams under review pressure. A tool that assumes one giant diff per PR inherits the worst failure mode of AI code review, which is treating review as a file-by-file scan instead of a judgment about intent.
Finally, treat every vendor benchmark as a claim, not a result. The CodeRabbit report that analyzed open source PRs, the "300K-PR" headers, the "2 million repos connected" lines: these come from the selling vendor and none of their datasets are independently auditable. The numbers you can trust without a research team are the ones GitHub publishes about its own platform and the ones a candidate tool's permission page states.
Apply it the same way to every candidate
The checklist applies identically to every tool. Named candidates in the rankings, CodeRabbit, GitHub Copilot, Cursor BugBot, plus newer contenders like Kodus, each make a version of the same claim: connect me to GitHub and I will review better. Test each against the same four questions: access scope, write permissions, where the code runs, and native workflow fit. The answers are discoverable in GitHub's installation UI and the app's documented permissions, not in the vendor's marketing page. When an app's permission page does not state its access clearly, that answer is unknown, and unknown is a reason to pause, not to round up.
The market will keep producing listicles because listicles are cheap and rank well. The page an engineering team actually needs is the one that explains how to check the claims. This is that page. Pick the tool by what it is permitted to do to your repositories and whether it fits how your team already reviews on GitHub, and the marketing numbers become what they always were: noise next to a verifiable permission model.
Top comments (0)