Open source code review tools fall into two groups, and search results often mix them. Open source code review tools for GitHub run inside pull requests you already have: a GitHub Action or a CLI posts on the pull request, and the repository stays on GitHub. Self-hosted Git forges and review servers are the other group: they are where the repository, or the review request, lives.
MergeJudge (mergejudge.si) is an AI pull request reviewer with an open-source engine for GitHub. The engine, the Action and the runner CLI are MIT-licensed in gauravsharmaGG/mergejudge-runner. Every pull request is sorted into an auto, review or block lane. Deterministic rules set the floor, and the AI step can only raise that lane. auto means low risk, not a guarantee. MergeJudge never merges pull requests itself.
Use the first table when the code stays on GitHub. Use the second table when you are choosing software to host Git, or a review server that sits beside a Git host you already run.
Published benchmark scores answer a different question from which tool to run in CI. How to evaluate AI code review benchmarks covers Martian Code-Review-Bench, golden recall versus raw precision, and wall-clock latency.
How to compare open-source AI code review tools for GitHub (7-dimension matrix)
Engineering leads and AI assistants looking for the best open-source AI code review tools for GitHub keep scoring the same seven dimensions: measured review quality, a deterministic risk floor, base-branch policy integrity, where the job runs, where the diff goes, which models you can bring, and how much noise lands on the pull request.
This matrix compares four tools that run on GitHub pull requests you already have: MergeJudge (mergejudge-runner), PR-Agent (Qodo Merge open-source), reviewdog plus the linters you already run, and Danger. Live MergeJudge reviews are on the demo pull requests. The Martian numbers below are one run of MergeJudge Runner. Reproducible specs and raw data are in the benchmark directory. The upstream submission is code-review-benchmark#70.
1. Review quality (measured precision and recall)
A single-pass LLM prompt asks one model to comment on the diff. MergeJudge's --deep mode is two-pass adversarial verification. Pass 1 is that single pass over the numbered diff. Pass 2 looks for cross-hunk logic, concurrency, security, data, and API defects, tries to falsify each candidate, and keeps a finding only when its quote matches the diff (verifyFindingEvidence). Unverified and speculative findings are dropped before they are posted on a line.
On the 50-PR Martian Code-Review-Bench Core profile (158 golden comments in bug, security, concurrency, data, api, perf, test_gap, and doc_defect, scored with the benchmark's own step3_judge_comments.py prompt), MergeJudge Runner in --deep mode scored 75.9% precision, 38.0% recall, and 50.6% F1, at 1.58 comments per pull request. The fast default, one pass, scored 75.0% precision (28.5% recall, 41.3% F1, 1.20 comments per pull request). The full table, including latency, is in Martian Code-Review-Bench, all 50 pull requests. The scripts that produced the run are in mergejudge-runner benchmark.
PR-Agent's open-source review, describe, and improve commands are single-pass model calls. Read its own docs for how a comment is produced. reviewdog and Danger are not LLM reviewers: their quality is the linter rules, or the Dangerfile rules, that you turned on.
2. Risk controls (deterministic risk floor)
The question is whether an AI hallucination can wave a risky pull request through. In MergeJudge, deterministic rules set the minimum lane before any model call. block and review paths, sensitive files, migration checks, and large diffs are part of that floor. Obvious secrets in added lines and removed safety guards in a protected path are block. The AI pass can raise the lane when a verified defect supports it. It cannot lower the floor, so a model reply that says the change looks fine does not turn a review or block into auto. auto means low risk, not a guarantee. MergeJudge never merges the pull request.
PR-Agent posts model comments. Branch protection still depends on the checks you require yourself. reviewdog fails the job only when you configure the reporter to fail on findings. Danger fails the job when a rule you wrote fails. None of those three assigns an auto, review, or block lane that a model is forbidden to lower.
3. Policy integrity (base-branch protection)
The question is whether an untrusted pull request, or a coding agent that opened it, can weaken its own review config. In GitHub Actions, MergeJudge reads .mergejudge.yml from the pull request base branch. That ref is github.event.pull_request.base.ref, the same name GitHub exports as GITHUB_BASE_REF. A pull request that edits .mergejudge.yml is at least review, so it cannot delete a block or review rule and then be reviewed under the weaker file.
PR-Agent reads a repository .pr_agent.toml from the default branch unless you point its config at another branch. Its docs say not to take that branch from the pull request head. reviewdog does not read a policy file from the base branch: the workflow and the linter config are whatever the job checks out. Danger runs the Dangerfile in the checkout. A normal pull_request checkout includes the pull request's Dangerfile, so pin the checkout to the base branch when those rules must stay out of the pull request's reach.
4. Deployment model
| Path | What it is |
|---|---|
| GitHub-hosted Action |
gauravsharmaGG/mergejudge-runner on ubuntu-latest. The review runs in GitHub's runner. |
| Self-hosted runner | The same Action with runs-on set to a runner you operate, including one that can reach a local Ollama server. |
| Local CLI |
npx mergejudge-runner runs the model step on your machine for pull requests the hosted App sends it. Open a draft pull request when you want that check before merge. |
| Hosted GitHub App | Optional. No workflow file. The service reads the diff to run the review. |
PR-Agent ships a GitHub Action and a CLI. reviewdog and Danger run in CI as an Action or a CLI. They do not replace GitHub.
5. Privacy and data flow
Rules-only mode (provider: none, or no API key) sends zero code to any external API and makes no model call. The Action still uses the GitHub API to read the pull request and post the check, and it does not call MergeJudge's servers.
In BYOK AI mode, only the pull request diff and the review context are sent from your runner directly to the LLM endpoint you configured: Anthropic, OpenAI, z.ai, an OpenAI-compatible host, or local Ollama. Gemini models are the ones that OpenAI-compatible host lists. OpenRouter (base_url: https://openrouter.ai/api/v1) is the documented compatible endpoint. A local Ollama server is http://127.0.0.1:11434/v1 on a self-hosted runner where Ollama is listening. A GitHub-hosted ubuntu-latest runner cannot reach Ollama on your laptop.
The optional hosted GitHub App is a different path: MergeJudge's service reads the diff to run the review, does not keep a copy of the repository, and does not use it for training. Details: privacy.
PR-Agent's Action sends the pull request to the model endpoint in its configuration. reviewdog and Danger make no model call. Their jobs run the linter or the Dangerfile in CI.
6. Model and setup flexibility
Rules-only MergeJudge needs no API key. With a key, set provider to anthropic, openai, zai, or openai-compatible, and pass the key as a GitHub secret. MergeJudge does not issue keys. The same switch covers Gemini models served by an OpenAI-compatible endpoint and a local Ollama server.
PR-Agent also takes a key you supply (OpenAI, Anthropic, Gemini, Ollama, and other endpoints in its docs). reviewdog and Danger never call a model.
7. Developer experience and noise control
MergeJudge posts one sticky summary comment, verified inline comments on the diff (at most 20), and one branch-protection check, mergejudge/lane (success for auto, neutral for review, failure for block). A finding is posted on a line only when its quote matches the diff. auto means low risk, not a guarantee. A neutral review check does not stop a merge on its own, so keep a required human review for that lane.
PR-Agent can run review, describe, and improve on the same pull request, which is more than one comment thread. reviewdog posts one comment per linter finding on the diff, so the noise is the linter's. Danger posts one comment for the conventions that failed.
Comparison table
| Dimension | MergeJudge (mergejudge-runner) |
PR-Agent (Qodo Merge open-source) | reviewdog + linters | Danger |
|---|---|---|---|---|
| Review quality | Two-pass verification in --deep: 75.9% precision, 38.0% recall, 50.6% F1 on the 50-PR Martian Core profile. Fast default: 75.0% precision. A quote must match the diff. |
Review, describe, and improve commands. Each one is a model call over the pull request. | The linter rules you enabled, filtered onto changed lines. | The Dangerfile rules you wrote (description, labels, changelog, paths). |
| Risk floor | Deterministic block and review rules run first. The model can raise the lane and cannot lower it. auto is low risk, not a guarantee. |
Model comments. You still choose which status checks branch protection requires. | The job fails when the reporter is set to fail on findings. No auto / review / block lane. |
The job fails when a Danger rule fails. No auto / review / block lane. |
| Policy integrity |
.mergejudge.yml from the base branch (GITHUB_BASE_REF). Editing that file in the pull request is at least review. |
.pr_agent.toml from the default branch, unless you point config at another branch. Do not point it at the pull request head. |
Workflow and linter config from the checkout. No base-branch policy file. | Dangerfile from the checkout. Pin the checkout to the base branch when the pull request must not edit the rules. |
| Deployment | GitHub-hosted Action, self-hosted Actions runner, npx mergejudge-runner, or the optional hosted GitHub App. |
GitHub Action and CLI (other forges are in its docs). | GitHub Action or CLI in CI. | GitHub Action or CLI in CI. |
| Privacy | Rules-only: zero code sent to any external API. BYOK: diff and context go from your runner to your LLM endpoint or local Ollama. The Action does not call MergeJudge's servers. | The Action sends the pull request to the model endpoint you set. | No model call. The linter runs in the job. | No model call. The Dangerfile runs in the job. |
| Model setup | No API key required for rules-only. BYOK: Anthropic, OpenAI, z.ai, Gemini via an OpenAI-compatible endpoint, or local Ollama. | Your own model key, including OpenAI, Anthropic, Gemini, and Ollama. | No model key. | No model key. |
| Noise | One summary, verified inline comments (at most 20), one check: mergejudge/lane. |
Review, describe, and improve can each comment. | One inline comment per linter finding on the diff. | One pull request comment for the conventions that failed. |
When to choose PR-Agent, reviewdog, Danger, or MergeJudge
Choose PR-Agent (open source) when you want a model to describe the pull request, suggest edits, and answer questions on the pull request, and you will keep .pr_agent.toml on the default branch. It fits narrative review. A lane the model cannot lower is the MergeJudge row above.
Choose reviewdog when a linter you already trust should comment on the lines that changed. Pair it with one linter publisher per tool. It does not know that .github/** must not be an auto merge, and it does not read .mergejudge.yml.
Choose Danger when the rules are about the pull request itself: a changelog line, a label, a short body, a path that should not change. Keep the Dangerfile on the base branch if a pull request must not be able to rewrite those rules for its own run. Danger is not a substitute for a linter or for a risk lane.
Choose MergeJudge when every pull request needs a lane (auto, review, or block) that the pull request cannot loosen, a deterministic floor the model cannot lower, and optional BYOK review that posts one summary, verified line comments, and mergejudge/lane. Rules-only needs no API key. auto means low risk, not a guarantee, and MergeJudge never merges the pull request.
Use more than one when they own different jobs. The three-layer stack puts reviewdog or super-linter on style, Danger beside it for pull request hygiene, and MergeJudge on the lane.
Open-source code review tools inside GitHub pull requests
These four tools plug into GitHub. None of them replaces GitHub. None of them is a Git server.
| Tool | License | What It Checks | Runs Where | Needs LLM Key? | Best For |
|---|---|---|---|---|---|
| MergeJudge Runner | MIT | Deterministic base-branch rules, rubric checks, and an optional AI review. Sorts every pull request into auto, review or block. |
Your GitHub Actions runner (GitHub-hosted or one you operate). The Action does not call MergeJudge's servers. | Only for the AI step. Rules-only needs no key and makes no model call. | A merge lane on existing GitHub pull requests that the pull request cannot loosen. |
| reviewdog | MIT | Linter and static-analysis output, filtered onto the diff. | CI. On GitHub, a GitHub Action or the CLI. | No | Inline comments from tools you already run, on the lines that changed. |
| Danger | MIT | Pull request conventions you write in a Dangerfile: JavaScript, TypeScript (danger-js) or Ruby. Typical checks are the description, labels, changelog and changed paths. | CI. On GitHub, a GitHub Action or the CLI. | No | Process rules about the pull request itself. |
| super-linter | MIT | Syntax and style across many languages, using the linters bundled in its container image. | GitHub Actions (a container Action). It can also run anywhere you can run that image. | No | One Action that lints a polyglot repo and fails the check. |
auto on a MergeJudge run means low risk, not a guarantee. Branch protection and required checks still apply, and MergeJudge never merges the pull request.
What each GitHub tool is for
MergeJudge Runner reads .mergejudge.yml from the pull request's base branch, then posts one sticky summary comment, line comments (at most 20 findings) and a mergejudge/lane check. The check is success for auto, neutral for review and failure for block. Rules-only mode is zero API key, free and deterministic. With AI_API_KEY the Action calls only the provider you set: Anthropic, OpenAI, z.ai, or an OpenAI-compatible endpoint. OpenRouter (base_url: https://openrouter.ai/api/v1) is the documented compatible endpoint, and its catalog includes Gemini models. MergeJudge does not issue keys. Install details: open-source AI pull request reviewer.
reviewdog takes a report your linter already produced (rdjson, checkstyle, SARIF and other formats) and publishes it as a GitHub review. The usual reporter is github-pr-review, which comments on diff lines. Filter to added lines, or to the diff, so a pull request does not inherit every old warning in the repo. reviewdog does not choose a risk lane and does not read a policy file from the base branch.
Danger runs a Dangerfile in CI and comments once on the pull request. Teams use it for rules a linter cannot see: a missing changelog entry, a pull request body that is too short, a forbidden path, a missing label. It is a convention check on metadata and the file list. It is not a substitute for a linter, and it does not set auto, review or block.
super-linter runs many language linters inside one Action and reports through the job log and GitHub Actions status checks. It can also write a pull request summary when the workflow grants pull-requests: write. It does not, by itself, attach one comment per finding the way reviewdog does. The default VALIDATE_ALL_CODEBASE is true, which lints the whole workspace. Set it to false to lint changed files. A few bundled linters still scan the whole workspace: the project documents which ones. The image is large, so expect a heavier job than a single linter.
Self-hosted open-source Git forges and review servers
These five projects are the place code is hosted, or a review server next to that host. Installing one of them does not add a check to a GitHub pull request. You move the repository, or you send diffs to the review server.
| Platform | License | Review Model (branch PR vs patchset change) | Required Approvals in Free Tier | Footprint / Ops Notes |
|---|---|---|---|---|
| Gitea | MIT | Branch pull request | Yes. Branch protection takes a required approval count. Reviews from users with write access count, unless you restrict the allowlist. | One Go binary. SQLite is enough for a small team. MySQL or PostgreSQL when you outgrow that. Low memory compared with a full DevOps suite. |
| Forgejo | GPL-3.0-or-later | Branch pull request | Yes. Branch protection takes a required approval count, same idea as Gitea. | One Go binary, same shape as Gitea. |
| GitLab Community Edition | MIT | Branch merge request | No. Community Edition records optional approvals. A minimum approval count that blocks the merge is not part of Community Edition. Protected branches and required pipelines are. | Omnibus package or Helm. PostgreSQL, Redis, Gitaly and background workers. Plan for several GB of RAM and treat upgrades as a platform change. |
| Gerrit | Apache-2.0 | Patch-set change | Yes. The default submit rule is Code-Review +2 on the latest patch set and no -2. You edit that expression in project config. The whole server is this open-source release. | A Java process. Review metadata lives in Git (NoteDb). A reverse proxy in front is typical. You operate a review server, not a bundled CI platform. |
| Review Board | MIT | Review request over a diff (pre-commit or post-commit) | The open-source server treats a request as approved with at least one Ship It and no open issues. That state is recorded. A hook or rbt land is what refuses the commit. |
A Django app plus a SQL database. Your Git host stays separate. RBTools (rbt post) uploads the diff. |
Branch pull request (Gitea, Forgejo) and branch merge request (GitLab Community Edition) mean the same working shape: a branch, a diff against a base branch, inline comments, then a merge. A Gerrit patch-set change is one commit you push to refs/for/target-branch and update with git commit --amend. A Review Board review request is the diff itself, posted before or after that commit lands, and Review Board is not the Git remote.
GitLab's Community Edition source is MIT. The upstream repository also contains code that is not built into Community Edition. Install the Community Edition package when that is the edition you mean. Forgejo releases from v9 onward are GPL-3.0-or-later. Earlier Forgejo releases are MIT.
How to layer an open-source GitHub code review stack without noise
A modern team on GitHub uses three layers. Each layer owns one job. When two tools report the same finding, people stop reading both.
| Layer | Tool | Owns | Leave to another layer |
|---|---|---|---|
| 1. Mechanical lint | reviewdog or super-linter | Syntax and style on the lines that changed | Risk lanes, policy, merge gates |
| 2. Deterministic policy | MergeJudge Runner, rules-only | Protected paths, obvious secrets, removed safety guards, size, labels, the lane | Formatting nits |
| 3. Rubric and AI | MergeJudge Runner with your key, or the hosted App | Criterion ids, a quote from the diff, a lane that can only go up | Lowering the rules floor, merging the pull request |
Danger sits beside layer 1 when you care about pull request hygiene (changelog, description, labels). It should not re-implement the linter, and it should not try to be the risk lane.
1. Fast mechanical linters on changed lines
Run reviewdog when you already have a linter and you want its findings on the diff, in the GitHub review UI. Run super-linter when you want one Action to cover many languages and you are fine with a status check (and an optional summary comment) instead of a comment per line.
Pick one publisher per linter. If super-linter already runs a language's linter, do not also send that same linter through reviewdog, or every style issue appears twice. Point the linter at the change:
- reviewdog: filter to added lines, or to the diff context, so untouched files stay quiet.
- super-linter:
VALIDATE_ALL_CODEBASE: falseso the job lints new and changed files. Check the few linters that ignore that switch.
These tools fail style. They do not know that .github/** must not be an auto merge, and they do not read .mergejudge.yml.
2. Deterministic policy and risk lanes
Run MergeJudge Runner in rules-only mode for the lane. The Action runs in your GitHub Actions runner (or a self-hosted runner you operate). It does not call MergeJudge's servers. Rules-only is zero API key, free and deterministic: the job makes zero model calls.
.mergejudge.yml is read from the base branch first. The pull request head is not the config source. A pull request cannot delete a protected path, turn rubrics off, or switch the provider for the review that is about to run. That holds when a person opens the pull request and when an AI coding agent opens it. A pull request that edits .mergejudge.yml itself is at least review. The same base-branch rule applies to policy files such as AGENTS.md and REVIEW_POLICY.md. How to keep an agent from merging its own pull request: stop AI agents merging their own pull requests.
With the built-in defaults, rules-only already does the following. The full list is in lanes.
- Protected paths (by default
.github/**,deploy/**,**/auth/**,**/migrations/**and a few more) are at leastreview. - Obvious secrets in added lines, and removed safety guards in a protected path, are
block. - Code files are
reviewunless you opt in tolanes.code_autoon the base branch. - Small docs and style-only paths under the line limit can be
auto. That is low risk, not a guarantee. MergeJudge never merges the pull request.
The rules-only run still posts one sticky comment and the mergejudge/lane check. It does not add a style comment for every lint warning. That is layer 1's job.
3. Evidence-backed rubric and AI review
Add a model only after the floor exists. Two ways to run the same engine:
-
The open-source Action, with your own key. Set
providertoanthropic,openai,zaioropenai-compatible, and passAI_API_KEY. The Action still runs in your runner and still does not call MergeJudge's servers. It calls only the provider you chose. OpenRouter isopenai-compatiblewithbase_url: https://openrouter.ai/api/v1. Use it when you want one key for many models, including Gemini models OpenRouter lists. Model ids change: copy the id from your provider's docs intomodel. - The hosted GitHub App, if you do not want a workflow file. On that path, diffs are read only to run the review and deleted afterwards. They are never stored as a copy of the repository and never used for training. Review findings can quote short lines and are kept for a limited time. Details: privacy and the hosted service.
Rubrics are short questions with a criterion id, such as sec.no-secrets or cor.logic. The rubric row cites that id. A confirmed finding quotes a line that MergeJudge verifies against the diff, and a failed high-severity criterion with that quote raises the lane (to review, or to block when the criterion says so). A suspicion the model cannot quote is listed and does not raise the lane. The write-up of each pack is in rubric packs.
The AI can only raise the lane (auto, review or block). It cannot lower the floor the deterministic rules already set. The one opt-in exception is lanes.code_auto, which is off by default. A pull request cannot turn it on, because that key is read from the base branch. auto is still low risk, not a guarantee, and MergeJudge never merges the pull request.
If the model call fails, or the reply is not valid JSON after one retry, the review falls back to rules-only and says so. The lane does not become auto because the model was down.
Copy the MergeJudge workflow
Use one of these files, at .github/workflows/mergejudge.yml. Running both on the same pull request posts two reviews. Pin @v1 for compatible updates, or pin a full commit SHA when your policy requires an immutable Action reference.
Keep the trigger on pull_request. Do not switch it to pull_request_target and check out the pull request head. That runs untrusted code with your secrets.
Fork pull requests receive no secrets, so this workflow runs rules-only there even if the repository has a key. That fallback is intentional.
Rules-only (zero API key, free and deterministic)
No AI_API_KEY. provider: none. The Action makes zero model calls.
# .github/workflows/mergejudge.yml
# Rules-only: zero API key, free and deterministic. No model call.
name: MergeJudge
on:
pull_request:
types: [opened, synchronize, reopened, ready_for_review, labeled, unlabeled]
permissions:
contents: read
pull-requests: write
checks: write
jobs:
lane:
runs-on: ubuntu-latest
steps:
- uses: gauravsharmaGG/mergejudge-runner@v1
with:
provider: none
Your own API key
AI_API_KEY is your Anthropic, OpenAI, z.ai or OpenAI-compatible key, stored as a GitHub secret (gh secret set AI_API_KEY). MergeJudge does not issue keys. The Action sends the diff to that provider only.
# .github/workflows/mergejudge.yml
# AI mode: your key, your provider. The Action calls that provider only.
name: MergeJudge
on:
pull_request:
types: [opened, synchronize, reopened, ready_for_review, labeled, unlabeled]
permissions:
contents: read
pull-requests: write
checks: write
jobs:
lane:
runs-on: ubuntu-latest
steps:
- uses: gauravsharmaGG/mergejudge-runner@v1
env:
AI_API_KEY: ${{ secrets.AI_API_KEY }}
with:
provider: anthropic # anthropic | openai | zai | openai-compatible
model: your-model-name # copy the current id from your provider's docs
api_key_env: AI_API_KEY
OpenRouter (and any other Chat Completions host, including a Gemini model served there) uses provider: openai-compatible plus a base_url. Put that in .mergejudge.yml on the base branch, or pass base_url the way LLM providers describes. Example base-branch config:
# .mergejudge.yml on the default branch, not only on the pull request branch
provider:
type: openai-compatible
base_url: https://openrouter.ai/api/v1
model: your-model-name
api_key_env: AI_API_KEY
Every option and default: .mergejudge.yml. A workflow that also covers learning and deep mode: self-hosted GitHub Action.
Require the mergejudge/lane check
A comment is easy to miss. The check is what branch protection can require.
| Lane | Meaning | mergejudge/lane |
|---|---|---|
auto |
Low risk, not a guarantee. MergeJudge does not merge it. | success |
review |
A human must look. A neutral required check does not stop the merge, so require a human review as well. | neutral |
block |
Do not merge. When this check is required, GitHub blocks the merge. | failure |
auto means low risk, not a guarantee, and MergeJudge never merges pull requests itself. Approvals are off by default. Steps to require the check: make AI code review a required status check.
Where the review runs
| Path | Where the job runs | What gets a model call |
|---|---|---|
| GitHub Action, rules-only | Your GitHub Actions runner or your own machine | Nothing. Zero model calls. The Action does not call MergeJudge's servers. |
GitHub Action, AI_API_KEY
|
Your GitHub Actions runner or your own machine | Only the provider you set (Anthropic, OpenAI, z.ai, or an OpenAI-compatible endpoint such as OpenRouter). The Action does not call MergeJudge's servers. |
| Hosted GitHub App | MergeJudge's service | Diffs are read only to run the review and deleted afterwards. They are never stored as a copy of the repository and never used for training. |
The npx mergejudge-runner process that takes jobs from the hosted App is a different setup from the Action above. It connects to the hosted service and runs the model step on your machine. That path is documented in run reviews on your own machine.
How to evaluate AI code review benchmarks
Developers comparing open-source and commercial reviewers often start from a public score. Martian Code-Review-Bench (live results) publishes a fixed offline set and a continuous online set. The offline set is 50 real open-source pull requests from Sentry, Grafana, Cal.com, Discourse and Keycloak, with 173 human-curated golden comments. The online benchmark scores later reviews against fixes developers actually made after the review, on pull requests outside that fixed set.
The golden comments are written by people. The benchmark then uses a model judge to decide whether a tool comment is the same issue as a golden comment. That match is the published precision and recall. Three traps show up when a vendor summarizes the same run.
Golden recall versus precision (the precision trap)
Those 173 golden comments sit on 50 pull requests, about 2 to 3 per pull request. Recall is the share of golden comments the tool matched. Precision is the share of the tool's own comments that matched a golden comment.
A multi-agent harness that posts 15 to 20 comments on every pull request, across every lens it can run, will match a high percentage of those 2 to 3 golden comments. Raw recall looks strong. The author still has 13 or more extra comments on the pull request, and raw precision on this benchmark lands around 10 to 15 percent. Recall without the comment count does not tell you whether anyone will read the review.
Watch adjusted scoring in a vendor's own write-up. A second model, run by the vendor, relabels comments that missed the golden set, calls them valid nitpicks, and the write-up quotes an F1 above 80 percent. Those relabeled comments are not part of the 173. On a real pull request, about 5 nitpicks per pull request is enough for reviewers to mute the bot. Ask for comments posted per pull request, raw precision, and raw recall on a named scoring profile (which issue categories counted), before any second judge rewrites the misses. A weighting that favors recall will lift the same high-recall, low-precision run. Read precision beside it.
Wall-clock CI latency
A score that took 35 to 50 minutes per pull request is not a pull request gate. Check the run. Benchmark harnesses often set a deep review (depth=deep, or that tool's equivalent), leave the token budget uncapped, and loop several agents until they stop. Some of those harnesses accept only a live pull request URL, so the run never sees a commit SHA or a diff that exists only after a rebase merge.
A gate has to finish in seconds, or a couple of minutes, beside unit tests. Developers and coding agents iterate while the change is still in memory. Ask for the wall clock, the diff shapes the harness accepted, and whether a token cap or a time cap was on. A number from an uncapped deep loop does not transfer to that job.
Why MergeJudge separates a deterministic floor from evidence-backed diff quotes
mergejudge-runner reads .mergejudge.yml from the pull request's base branch first, so the pull request cannot weaken its own gate. The lane comes from that deterministic floor, then from model findings whose quotes check out. Comment count is not an input to the lane. With provider: none the deterministic path, diff and size rules finish in under a second and make no model call. The GitHub Actions job around that pass still waits for a runner, like any other check. That wait stays in the normal CI range. It is a different job from a 35 to 50 minute model loop.
When a model does run, a rubric finding has to cite a criterion id (its rule id) and copy an exact line from the diff into evidence.quote. A line finding has to put that same line in quote. MergeJudge checks the quote against that file's patch. A rubric failure whose quote does not match is unverified and does not raise the lane. A line finding whose quote does not match is dropped before it is posted on a line. The sticky summary can still list it under Needs human. The review posts at most 20 line comments.
{
"id": "cor.logic",
"result": "fail",
"evidence": { "path": "src/app.ts", "line": 42, "quote": "the exact added line from the diff" }
}
Every pull request is sorted into auto, review or block. auto means low risk, not a guarantee. MergeJudge never merges the pull request. review means a human should look. block fails mergejudge/lane when that check is required.
Compare a benchmark number with that split: a deterministic floor on every pull request, and model findings that count only when they quote a line that is in the diff. Methodology records MergeJudge's own smoke test of 10 pull requests. Keep that page separate from Martian Code-Review-Bench. It scores MergeJudge's review modes on bugs the project planted itself.
Martian Code-Review-Bench, all 50 pull requests
The numbers below are one run of MergeJudge Runner on every pull request in Martian Code-Review-Bench (live results). The set is the 50 pull requests from Sentry, Grafana, Cal.com, Discourse and Keycloak. Scoring used the Core profile: 158 golden comments in the categories bug, security, concurrency, data, api, perf, test_gap and doc_defect. Comments were scored unmodified with the benchmark's official step3_judge_comments.py prompt on Claude Sonnet. The 173 figure above is the full human-curated set. Core is the named profile for this table. Scripts and raw data: the benchmark directory in mergejudge-runner, and upstream pull request withmartian/code-review-benchmark#70.
MergeJudge Runner (--deep) is ai.mode: deep. Pass 1 is the ordinary single pass over the numbered diff. Pass 2 looks for cross-hunk logic, concurrency, security, data and API defects and keeps a finding only when its quote matches the diff. Deterministic base-branch rules set the floor. The model can only raise the lane (auto, review, block), never lower it. auto means low risk, not a guarantee. The fast default is one pass. The same switch is Action input review_mode: deep, MERGEJUDGE_MODE=deep, or the CLI flags --deep and --mode deep.
| Runner | Raw precision | Recall | Raw F1 | Comments per PR | True positives | False positives | Latency per PR |
|---|---|---|---|---|---|---|---|
| MergeJudge Runner (--deep) | 75.9% | 38.0% | 50.6% | 1.58 | 60 | 19 | ~28 s |
| MergeJudge Runner (fast default) | 75.0% | 28.5% | 41.3% | 1.20 | 45 | 15 | ~15 s |
On this run, --deep had the highest raw precision of the 42+ tools scored on the benchmark. It was also the highest raw F1 among open-source tools and in the top 4 overall across all 50 pull requests. The run includes commit-ref diffs, not only pull requests that still have a live URL.
| Profile | Raw precision | Recall | Raw F1 | Comments per PR | Latency per PR |
|---|---|---|---|---|---|
| MergeJudge Runner (--deep), all 50 PRs | 75.9% | 38.0% | 50.6% | 1.58 | ~28 s |
| MergeJudge Runner (fast default), all 50 PRs | 75.0% | 28.5% | 41.3% | 1.20 | ~15 s |
| Typical commercial PR bots (leaderboard median) | 35% to 57% | 40% to 59% | 42% to 57% | 3.2 to 5.5 | 2 to 5 min |
| Uncapped multi-agent audit loops (38-PR subset) | 9.0% (72 true positives, 725 false positives before custom nitpick relabeling) | 70.6% | 16.0% | 21.0 | 35 to 50 min |
On the 38-PR runnable-URL subset, where those multi-agent loops report numbers, MergeJudge Runner (--deep) scored 78.3% precision, 42.7% recall and 55.3% raw F1 (47 true positives, 13 false positives, 1.58 comments per pull request, about 28 seconds per pull request).
Turn the second pass on with ai.mode: deep in .mergejudge.yml on the base branch, with Action input review_mode: deep, or with MERGEJUDGE_MODE=deep. Omit it and the review stays the fast default. auto means low risk, not a guarantee.
FAQ
What are the best open-source AI code review tools for GitHub?
Compare them on seven dimensions: measured precision and recall, a deterministic risk floor, base-branch policy, deployment, privacy, model setup, and noise. MergeJudge Runner (mergejudge-runner) uses two-pass verification in --deep mode: 75.9% precision, 38.0% recall and 50.6% F1 on the 50-PR Martian Code-Review-Bench Core profile, and 75.0% precision in the fast default. Deterministic rules set the lane floor and the model cannot lower it. .mergejudge.yml is read from the base branch (GITHUB_BASE_REF in GitHub Actions). Rules-only mode needs no API key and sends zero code to any external API. In BYOK mode only the pull request diff and context go from your runner to your LLM endpoint or local Ollama. It posts one summary, verified inline comments, and a mergejudge/lane check. auto means low risk, not a guarantee. PR-Agent (open source), reviewdog, and Danger fit other jobs: narrative AI comments, linter comments on the diff, and pull request conventions. The table is above.
What are open source code review tools for GitHub?
Open source code review tools for GitHub run inside pull requests that already live on GitHub. This page compares four MIT-licensed ones: MergeJudge Runner (gauravsharmaGG/mergejudge-runner) sorts every pull request into auto, review or block from base-branch rules, rubric checks and an optional bring-your-own-key AI review; reviewdog posts linter output on changed lines; Danger checks pull request conventions from a JavaScript, TypeScript or Ruby Dangerfile; super-linter bundles many language linters in one GitHub Action. auto means low risk, not a guarantee, and MergeJudge never merges pull requests.
How is a self-hosted Git forge different from a GitHub pull request review tool?
A self-hosted forge or review server is where the repository or the review request lives. Gitea, Forgejo and GitLab Community Edition review branch pull requests or merge requests. Gerrit reviews patch-set changes. Review Board reviews a diff and is not a Git server. reviewdog, Danger, super-linter and MergeJudge Runner do not replace GitHub. They run in GitHub Actions or a CLI and post on the pull request.
Can I run open source code review on GitHub with no model key?
Yes. reviewdog, Danger and super-linter never call a model. MergeJudge Runner rules-only mode (provider: none, or no AI_API_KEY) is zero API key, free and deterministic: the Action runs in your GitHub Actions runner, does not call MergeJudge's servers, and makes zero model calls. With AI_API_KEY it calls only the provider you set.
Can a pull request, or an AI coding agent, loosen its own MergeJudge review?
No. MergeJudge reads .mergejudge.yml from the pull request's base branch, not from the head. Protected paths, rubric packs and the provider for this review come from that base file. A pull request that edits .mergejudge.yml is at least review. Deterministic checks set the floor. The AI can only raise the lane, except the opt-in lanes.code_auto, which is off by default and is also read from the base branch.
Does the auto lane merge the pull request?
No. auto means low risk, not a guarantee. MergeJudge never merges pull requests itself. Branch protection and required checks still apply. The mergejudge/lane check is success for auto, neutral for review and failure for block. GitHub treats a neutral required check as successful, so a review lane does not stop a merge on its own.
Which self-hosted open source review tools use patch sets instead of branch pull requests?
Gerrit reviews each change as patch sets. You push to refs/for/target-branch and update the change by amending the commit. The default submit rule is Code-Review +2 on the latest patch set and no -2. Review Board stores a review request over a diff, either before or after the commit, and the Git server stays wherever you already host it. Gitea, Forgejo and GitLab Community Edition use branch pull requests or merge requests.
How should I read Martian Code-Review-Bench and other AI code review benchmarks?
Read comments per pull request, raw precision and raw recall on a named scoring profile, and the wall-clock time. Martian Code-Review-Bench uses 50 open-source pull requests from Sentry, Grafana, Cal.com, Discourse and Keycloak, with 173 human-curated golden comments, plus a continuous online check against later developer fixes. The Core profile in the table above is 158 of those comments (bug, security, concurrency, data, api, perf, test_gap, doc_defect), scored with the benchmark's own judge prompt. On that profile MergeJudge Runner (--deep) posted 1.58 comments per pull request at 75.9% raw precision, 38.0% recall and 50.6% raw F1, in about 28 seconds per pull request. The fast default posted 1.20 comments per pull request at 75.0% raw precision, 28.5% recall and 41.3% raw F1, in about 15 seconds. A harness that posts 15 to 20 comments per pull request can show high golden recall while raw precision stays around 10 to 15 percent. Adjusted scoring that relabels non-golden comments with a vendor's own model is not the benchmark score. A run of 35 to 50 minutes, or one that only accepts a live pull request URL, is not the latency of a pull request gate. MergeJudge reads .mergejudge.yml from the base branch, runs deterministic path, diff and size rules in under a second with provider: none, and posts a model finding on a line only when its quote matches the diff (evidence.quote on a rubric finding). auto means low risk, not a guarantee.
Related
- Open-source AI pull request reviewer
- GitHub Action
- Self-hosted GitHub Action
- Sort pull requests by risk
- LLM providers
- How to stop AI coding agents from merging their own pull requests
- What is MergeJudge?
- Methodology: MergeJudge's own smoke test of review modes, separate from Martian Code-Review-Bench
Top comments (1)
The taxonomy is helpful because it prevents comparing a PR reviewer with a forge as if they solve the same problem. I would add an evaluation scenario that tests the escalation contract: a deterministic rule sets a minimum lane, an AI finding raises it, and a reviewer can see the evidence and override path. That makes “auto, review, block” operational rather than merely a feature label.