Self-Hosted AI Code Review: The 2026 Buyer's Guide

Dharmesh Gurnani — Wed, 14 Oct 2026 09:16:02 +0000

Cross-posted from the CodeOtter blog. The canonical, always-up-to-date version of this article lives at codeotter.io.

This post is the field guide for engineering leads deciding between **self-hosted AI code review* and the $24–$72/dev/mo SaaS incumbents in 2026.*

The question every platform lead is asking in 2026

Every engineering team we talk to has the same conversation in 2026: "Do we keep paying $24–$72/dev/mo for a hosted AI reviewer, or do we self-host?" The honest answer is that the question is mostly about five architectural questions, not vendor preference. Once you answer the five, the vendor falls out.

This guide walks through the five questions, ports the 2026 numbers, and shows the matrix an engineering lead should run before signing a renewal or starting a self-host pilot. TL;DR at the top; math below.

The five questions that decide the purchase

Every AI PR reviewer in 2026 — hosted or self-hosted — answers (or fails to answer) these five questions. They are not vendor-specific. They are architectural.

Where does your code live during a review? (privacy / data routing)
Are the scores calibrated, or are they what an LLM felt like scoring? (calibration / Score)
What does it cost at 50 engineers over three years? (TCO)
Can it run air-gapped, with no outbound network calls? (offline / air-gap)
Does it enforce AGENTS.md / CLAUDE.md at the merge gate? (team rules)

If a vendor cannot answer one of these in writing — with an endpoint, a gate name, or a hosting diagram — they do not have the architecture; they have a vibe. Skip them.

Question 1 — Where does your code live?

The default answer for every hosted AI reviewer in 2026 is: the diff leaves your GitHub tenant and lands on the vendor's cloud. CodeRabbit routes to AWS, Greptile to their managed SaaS, GitHub Copilot Code Review to Microsoft's tenant, Qodo Merge to theirs. They all require telemetry, all require a DPA, and none can credibly say "your diff never touched our disk" without a SOC 2 Type II report you cannot audit.

For a startup selling B2B SaaS, this is a tradeoff most teams sign. For a regulated platform team — fintech under PCI, healthtech under HIPAA, gov under FedRAMP, defense under ITAR — it is a hard block. The ICP-1 buyer (privacy-first platform teams, 25–250 engineers) is the one whose legal/compliance ticket says "we cannot send proprietary code to a third party." There is no SLA that closes that ticket.

The architectural answer is self-hosted with zero telemetry: the review runs on a Docker container or a Compose stack on your hardware, the model is downloaded once at admin time into a local pb_data volume, and the only outbound calls during a review are to your GitHub webhook. You can prove this with tcpdump on the review host. CodeOtter's offline mode post walks through the architecture.

If your code can leave your VPC, you do not need self-hosting. If it cannot, you already do.

Question 2 — Are the scores calibrated?

Every hosted AI reviewer in 2026 produces numeric scores by asking a chat model to "give this PR a score between 0 and 100." The well-known failure mode is the 75–85 clustering problem: ask GPT-5 or Claude or Gemini to score ten PRs in a row and the answers cluster in a tight band around 80, regardless of whether the PR is a typo, a config flip, or a 600-line refactor of the auth path. You can prove this yourself with a 15-line script and your last ten PRs.

The architectural answer is to separate the scoring engine from the narrative engine. CodeOtter ships a dual-engine architecture for exactly this reason: a deterministic System 1 rubric engine (POST /v1/systemone) owns every 0–100 gauge — Quality, Blast Radius, Risk, Tests, Readability, PR Hygiene — and every one of the 9 binary pre-merge gates (Title, Description, Security, Complexity, Tests, Docs, Scope, Guidelines, Issue Requirements). The chat model never gets the token for the scores. It writes the executive summary, the file walkthrough, and the inline suggestion blocks. Either engine can also operate standalone; the dashboard lets you re-run System 1 alone for a known-good AGENTS.md flip.

What this means in a procurement conversation: a Quality: 42 from CodeOtter is reproducible. You can defend it in a postmortem because the rubric says why. A "72/100 — looks risky" from a chat-LLM-only reviewer is whatever the model felt like scoring on the day. For a regulated team building an audit trail, only one of those two is a real signal.

Question 3 — What does it cost at 50 engineers over three years?

The headline seat price is one of four cost inputs. A defensible TCO adds per-PR review overages, per-author overages (Greptile charges per-author, not per-seat — consultants and OSS maintainers eat overages silently), infrastructure to host the alternative, and engineering hours to wire each tool into CI, branch protection, and AGENTS.md enforcement. The full table is in the companion 3-year TCO post (CodeOtter pricing table comparison); the version below is rendered below.

Tool	Per-seat	3-yr TCO (50 eng)	Cost shape
CodeOtter (self-host)	$0	~$0–$2,400	Self-hosted infra (Hetzner AX162 or 2× M2 Pro)
CodeRabbit Pro	$24/mo	$43,200	Per-seat SaaS, telemetry required
CodeRabbit Pro+ / Team	$48/mo	$86,400	Per-seat SaaS, adds org-wide policy
CodeRabbit Advanced	$72/mo	$129,600	Per-seat SaaS, adds compliance reporting
Greptile Pro	$30/seat/mo	$54,000+	Per-author + $1/credit overage past 50
GitHub Copilot Business	$19/mo	$34,200	Per-seat SaaS, 200 PR-reviews/mo cap
GitHub Copilot Enterprise	$39/mo	$70,200	Per-seat SaaS, unlimited PR reviews
Qodo PR-Agent OSS	$0	~$3,000–$8,000	Self-hosted OSS, you operate it (no System 1)
Qodo Merge (managed)	Custom	varies	Sales-quoted

The gap is roughly 54x between the cheapest self-hosted option and the most expensive SaaS. Even on a beefy self-hosted box (2× AMD EPYC, 128 GB RAM, RTX 4090 at ~$300/mo), the 3-year run-rate for CodeOtter is ~$10,800 — still 12x cheaper than the cheapest SaaS. The detailed battlecards against each — including the lines the headline price leaves out — are at the bottom of this post.

Question 4 — Can it run air-gapped, with zero outbound calls?

For the air-gapped / regulated infra operator (defense, gov, critical infrastructure, on-prem GPU boxes with no internet egress at runtime), the answer is binary: yes or no, with proof. Procurement rule: a reviewer is air-gapped iff it can run a full PR review with the host's network cable unplugged and produce a scorecard on the PR thread.

The architectural answer in 2026 is to download models once at admin time and run them locally. CodeOtter ships five open-weight checkpoints out of the box (how they run on local hardware): Laya Typed-Decisions and Kev 0.8B ride a ggmlc sidecar; Qwen2.5-Coder 1.5B / 7B ride a llama.cpp llama-server; Microsoft CodeReviewer rides a Python sidecar. The Admin → Local Models flow downloads each model with Hugging Face URLs, verifies shards with modelReady, detects Vulkan (Linux/Windows) or Metal (macOS arm64) automatically, and shuts down idle runtimes after 15 minutes. There is no auto-update, no telemetry, and no SaaS fallback. The models.json registry is yours to edit.

If you want the best narrative review, point System 2 at Claude or OpenAI via BYOK; the whole point of System 1 is that the numbers do not depend on the LLM. You can run System 1 offline and System 2 online, separately.

Question 5 — Does it enforce `AGENTS.md` at the merge gate?

Most teams already have a team-rules doc. It is usually called AGENTS.md, CLAUDE.md, or .github/copilot-instructions.md. It is aspirational — nothing automated reads it at the merge gate. A reviewer that ignores the doc is a reviewer that has not earned its $24–$72 per dev per month.

The architectural answer is zero-config auto-detection at the repo root, with a typed Guidelines merge gate that fails closed on violations. CodeOtter reads AGENTS.md and CLAUDE.md at the repo root on first review, injects the rules into every subsequent prompt, and emits a binary Guidelines gate on the scorecard. A finding the team marks as "noise" can be dismissed with 1 click; the dismissal persists as a team learning into repoSettings[repo].learnings and the finding never re-appears on the next review. The deep-dive on the merge-gate semantics is at AGENTS.md as a merge gate; the live flow is: dismiss a finding → it persists → re-review never re-flags it.

A SaaS reviewer that requires you to maintain a .coderabbit.yaml or a .pr_agent.toml to mirror what is already in AGENTS.md is a reviewer that has not read the room.

The architectural matrix

Once you have answered the five questions, the matrix falls out. The rows below are the architectural claims that hold in 2026 — they are not benchmark theater, they are the shape of each product.

Capability	CodeOtter	Hosted AI reviewer (CodeRabbit / Greptile / Copilot CR)	Qodo / PR-Agent OSS
Self-hosted (Docker / Compose)	✅ First-class	❌ SaaS only	⚠️ You operate Python deps + secrets
100% Offline (GGUF + sidecars)	✅ `llama.cpp` + `ggmlc` + Metal/Vulkan	❌	❌
Deterministic System 1 scoring	✅ 6 calibrated 0–100 gauges	❌ LLM-only numbers (75–85 cluster)	❌ LLM-only numbers
9 typed binary pre-merge gates	✅	⚠️ Heuristic status check only	⚠️
`AGENTS.md` / `CLAUDE.md` auto-detection	✅ Auto-detect, zero config	⚠️ `.coderabbit.yaml` only	⚠️ `.pr_agent.toml` only
1-click "Dismiss & remember rule"	✅ Persisted as team learning	❌	❌
Incremental delta review (`prevSha → headSha`)	✅ Resolved-finding tracking	⚠️ Re-runs the full PR	⚠️
Inline committable `suggestion` blocks	✅ `POST /api/review-suggestions`	✅	✅
In-place PR comment upsert	✅ `<!-- codeotter:scores -->`	❌ Duplicates	❌
License	AGPL-3.0 (free self-host, modify, redistribute)	Proprietary	Apache 2.0 (community PR-Agent)
Per-seat cost (50-eng team, 3-yr)	$0–$2,400	$34,200–$129,600	$3,000–$8,000 (you operate it)

The single sentence to internalize: if your code can leave your VPC, you do not need CodeOtter. If it cannot, you already do.

Honest corners — when self-hosted is the wrong call

A credible buyer's guide names its own weak spots. Self-hosted is the wrong call if any of these is true:

You want the slickest SaaS dashboard and do not care where the code lives. CodeRabbit's UX leads; CodeOtter is functional, not glossy. Keep CodeRabbit.
You rely on a full-repo graph Q&A workflow daily. Greptile's repo-graph Q&A is genuinely good. Plan the transition if it is part of your daily loop.
You have no platform bandwidth. CodeOtter's $0 headline is not free without 4 hours/month of upgrades and patches; a 5-person startup with no ops budget should stay on a hosted reviewer.
You want the best narrative, not reproducible numbers. A 7B Qwen won't match GPT-5 on nuanced critique. Point System 2 at GPT-5 or Claude via BYOK; System 1 numbers don't depend on the LLM.

CodeOtter is free to self-host and customize; make the call that fits your team.

The 8 vendor questions to ask before signing

A vendor should answer every one of these in a single document, with endpoints and diagrams, before you sign a 3-year deal. A vendor that hides behind "contact sales" for any of them is a vendor you cannot size without spending the engineering hours to size them.

What is the per-seat, per-author, and per-PR breakdown? Show me the formula.
What happens if I cross the PR-review cap? Degraded dashboard, or billed for overages?
What is the telemetry default? Can I turn it off, or do I need a DPA?
Where does my code route during a review? Does it touch a third-party cloud?
How do I enforce AGENTS.md / CLAUDE.md at the merge gate? Auto-detection supported?
How do I persist team learnings across PRs? Is there 1-click "Dismiss & remember rule"?
What is the audit story for compliance? Can I export the review graph?
What is the cold-start latency on the first review of the day? What model warms up — and is it local or hosted?

If the vendor's answer to 1–4 is "call sales," pass. If the answer to 5–8 names an endpoint or a file path, you are talking to a vendor worth a 12-month pilot.

Decision matrix by ICP

The 2026 buyer falls into one of four shapes:

ICP-1 — Privacy-first platform team (regulated, 25–250 eng). Self-host or do not buy. CodeOtter on a Hetzner box answers Q1, Q2, Q4 with proof. See vs CodeRabbit and vs Greptile.
ICP-2 — Cost-conscious indie / OSS maintainer (1–10 eng). Self-host on a MacBook or a $20/mo VPS; the 1.5B Qwen runs offline on M-series silicon. See vs CodeRabbit for the per-seat math.
ICP-3 — Senior IC building their own tools (50–500 eng). Self-host and wire AGENTS.md as a merge gate; the 1-click team-learning flow is what makes the doc a real gate, not a wiki page. See vs PR-Agent.
ICP-4 — Air-gapped / regulated infra operator. Self-host with no outbound calls; models download once at admin time and live in pb_data. Verify with tcpdump on the review host. See the offline mode post and the local models deep-dive.

What to read next

The five questions above are the matrix; the deep-dives below are the architecture and battlecards.

TCO math — the companion 3-year TCO post for a 50-engineer team covers the four real cost inputs and the line items the headline price hides (publishing alongside this guide in W1).
Dual-engine architecture — Why We Built a Dual-Engine Architecture (System One + LLM). The calibration problem in full and the System 1 / System 2 split behind the AGPL-3.0 release.
CodeRabbit battlecard — CodeOtter vs CodeRabbit (2026). The 72× 3-year TCO gap, the AGENTS.md enforcement delta, and why CodeRabbit's .coderabbit.yaml is a config layer not a gate.
Greptile battlecard — CodeOtter vs Greptile (2026). The repo-graph Q&A Greptile wins on, the 150×–240× TCO gap, and why per-author billing is a foot-gun at the boundary.
PR-Agent battlecard — CodeOtter vs PR-Agent (Qodo) (2026). The Apache 2.0 vs AGPL-3.0 license posture, the 14×–60× TCO gap, and the multi-agent Python CLI workflow library PR-Agent ships.
AGENTS.md as a merge gate — How CodeOtter Turns AGENTS.md From a Wiki Page Into a Merge Gate. The 1-click "Dismiss & remember rule" flow and the Guidelines gate that fails closed.
Offline mode — Running 100% Offline AI Code Reviews with Local GGUF, laya & CodeReviewer Sidecars. The 5-minute cold-start, the three sidecar runtimes, the modelReady shard-verify lifecycle.
Local models — How CodeOtter Runs Five Open-Weight Models Locally. The models.json registry, GPU detection, and the "no auto-download" admin contract.

The five questions — privacy, calibration, TCO, offline, team rules — are the matrix. The deep-dives are the architecture. Run the eight vendor questions, port the TCO math to your team's PR volume, and the procurement conversation gets shorter by an hour. Self-hosted is the right call when the code cannot leave the VPC, the numbers have to be reproducible, and the per-seat tax no longer earns its line item.

CodeOtter ships under the GNU AGPL-3.0 — free to self-host, $0/seat, zero telemetry, single Docker image + PocketBase + Node 20. The CLI is bin/codeotter.mjs (v0.4.0, zero runtime dependencies). The dual-engine System 1 + System 2 pipeline is the same in the platform dashboard, in CI, and in the MCP server.

DEV Community: Dharmesh Gurnani