Commercial AI code review tools are good at catching the problems that every codebase shares, null pointer risks, resource leaks, common vulnerability patterns, style violations. They’re trained on public code and they review your code against those public patterns.
But your codebase isn’t public. Your codebase has conventions that only your team uses. Architecture patterns that evolved over years. Domain-specific anti-patterns that cause production incidents in your specific context. Security constraints that your specific compliance requirements impose. Integration patterns with your specific third-party services that have known failure modes nobody outside your team has encountered.
Commercial tools don’t know any of this. They can’t. They weren’t trained on your code and they don’t have your team’s institutional knowledge.
We learned this gap the practical way. A SaaS company we work with deployed a well-known commercial review tool. It caught genuine issues, about 60 percent of the bug categories they cared about. It missed every domain-specific pattern that had caused their last five production incidents. The missed patterns weren’t exotic. They were things like “this database query pattern causes deadlocks in our specific transaction isolation configuration” and “this API call sequence creates a race condition with our payment provider’s webhook timing.”
The commercial tool couldn’t know these things. A custom AI code review agent trained on your codebase, your incidents, and your team’s institutional knowledge can.
This article walks through how to build one, the architecture, the training data preparation, the review pipeline, the CI/CD integration, and the code that makes it work. This isn’t the $2,000/month SaaS alternative. This is the review agent that catches what commercial tools structurally miss.
The architecture: why a custom agent outperforms a generic model
A commercial code review tool runs a general-purpose model against your diff and returns findings based on patterns learned from millions of public repositories. The model is excellent at common patterns and blind to your specific ones.
A custom review agent inverts this. It starts with a foundation model’s general code understanding and layers your specific knowledge on top, your conventions, your incident history, your architectural decisions, your security constraints. The result is a reviewer that catches everything a commercial tool catches plus the domain-specific patterns that only your team knows about.
The architecture has four components:
PR opened
│
▼
┌──────────────────────┐
│ Context Assembler │ Gathers diff + surrounding files +
│ │ project conventions + relevant
│ │ incident history
└──────────┬───────────┘
│
▼
┌──────────────────────┐
│ Review Engine │ Foundation model + project-specific
│ │ rules + learned patterns
└──────────┬───────────┘
│
▼
┌──────────────────────┐
│ Confidence Filter │ Scores each finding, suppresses
│ │ low-confidence noise
└──────────┬───────────┘
│
▼
┌──────────────────────┐
│ Comment Publisher │ Posts findings to the PR with
│ │ explanations and fix suggestions
└──────────────────────┘
Each component has specific engineering requirements. Let’s build them.
Component 1: Context assembler
The biggest mistake in custom code review agents is sending just the diff to the model. A diff without context is a conversation without history, the model can see what changed but not why it matters.
The context assembler gathers everything the review needs:
import subprocess
from pathlib import Path
class ContextAssembler:
"""Builds the full review context for a PR."""
def __init__(self, repo_path: str, conventions_path: str,
incidents_path: str):
self.repo = Path(repo_path)
self.conventions = self._load_conventions(conventions_path)
self.incidents = self._load_incident_patterns(incidents_path)
def assemble(self, base_ref: str, head_ref: str) -> ReviewContext:
# Get the diff
diff = self._get_diff(base_ref, head_ref)
changed_files = self._parse_changed_files(diff)
# Get surrounding context for each changed file
file_contexts = {}
for filepath in changed_files:
file_contexts[filepath] = {
"full_content": self._read_file(filepath),
"imports": self._get_imports(filepath),
"importers": self._get_files_importing(filepath),
"recent_incidents": self._get_related_incidents(filepath),
"git_blame_hotspots": self._get_churn_hotspots(filepath),
}
# Get applicable conventions
applicable_rules = self._match_conventions(changed_files)
return ReviewContext(
diff=diff,
changed_files=changed_files,
file_contexts=file_contexts,
conventions=applicable_rules,
incident_patterns=self._get_relevant_patterns(changed_files),
)
def _get_related_incidents(self, filepath: str) -> list[dict]:
"""Find past incidents related to this file or module."""
related = []
module = self._get_module(filepath)
for incident in self.incidents:
if (filepath in incident.get("affected_files", []) or
module in incident.get("affected_modules", [])):
related.append({
"incident_id": incident["id"],
"summary": incident["summary"],
"root_cause_pattern": incident["pattern"],
"prevention_rule": incident["prevention"],
})
return related
def _get_churn_hotspots(self, filepath: str) -> list[dict]:
"""Identify frequently-changed sections - likely complexity."""
result = subprocess.run(
["git", "log", "--follow", "--format=%H", "-20", "--", filepath],
capture_output=True, text=True, cwd=self.repo
)
commit_count = len(result.stdout.strip().split('\n'))
return {"file": filepath, "commits_last_20": commit_count}
The _get_related_incidents method is where the custom agent’s advantage lives. Commercial tools don’t have your incident history. They don’t know that the last three production outages involved a specific query pattern in a specific module. Your custom agent does, and it flags changes to that module with extra scrutiny.
The _get_churn_hotspots method identifies files that change frequently, a proxy for complexity and risk. A change to a file with 200 commits in the last year deserves more careful review than a change to a file with 3 commits.
Component 2: Review engine
The review engine combines a foundation model’s code understanding with your project-specific rules. The key design decision: separate deterministic rule checking from AI-assisted review.
import anthropic
import json
class ReviewEngine:
"""Two-tier review: deterministic rules + AI interpretation."""
def __init__(self, rules_path: str, model: str = "claude-sonnet-4-6"):
self.client = anthropic.Anthropic()
self.model = model
self.rules = self._load_rules(rules_path)
async def review(self, context: ReviewContext) -> list[ReviewFinding]:
findings = []
# Tier 1: Deterministic rule checks - fast, consistent
rule_findings = self._check_rules(context)
findings.extend(rule_findings)
# Tier 2: AI-assisted review - contextual, interpretive
ai_findings = await self._ai_review(context)
findings.extend(ai_findings)
return findings
def _check_rules(self, context: ReviewContext) -> list[ReviewFinding]:
"""Apply project-specific rules deterministically."""
findings = []
for rule in self.rules:
violations = rule.check(context.diff, context.file_contexts)
for violation in violations:
findings.append(ReviewFinding(
file=violation.file,
line=violation.line,
severity="warning",
category="convention",
message=violation.message,
rule_id=rule.id,
source="deterministic",
confidence=1.0, # Rules are binary
))
return findings
async def _ai_review(self, context: ReviewContext) -> list[ReviewFinding]:
"""Use LLM for the review that rules can't handle."""
prompt = self._build_review_prompt(context)
response = self.client.messages.create(
model=self.model,
max_tokens=4096,
temperature=0.1, # Near-deterministic for review consistency
messages=[{"role": "user", "content": prompt}]
)
return self._parse_findings(response.content[0].text)
def _build_review_prompt(self, context: ReviewContext) -> str:
"""The prompt is where institutional knowledge becomes review capability."""
incident_context = ""
for filepath, ctx in context.file_contexts.items():
if ctx["recent_incidents"]:
incident_context += f"\n\nCAUTION for {filepath}:\n"
for incident in ctx["recent_incidents"]:
incident_context += (
f"- Past incident: {incident['summary']}\n"
f" Root cause pattern: {incident['root_cause_pattern']}\n"
f" Prevention rule: {incident['prevention_rule']}\n"
)
return f"""Review this code diff against the project's specific conventions
and known risk patterns. Focus on issues that matter for this specific codebase,
not generic code quality observations.
PROJECT CONVENTIONS:
{json.dumps(context.conventions, indent=2)}
KNOWN RISK PATTERNS FROM PAST INCIDENTS:
{incident_context}
FILES THAT IMPORT FROM CHANGED FILES:
{self._format_importers(context)}
DIFF TO REVIEW:
{context.diff}
For each finding, provide:
- file: the filename
- line: approximate line number
- severity: critical / warning / info
- category: security / logic / performance / convention / incident_pattern
- message: clear explanation of the issue
- suggestion: how to fix it
IMPORTANT RULES:
- Flag ONLY genuine issues. No style-only comments unless violating
a specific convention listed above.
- If a change touches code related to a past incident pattern,
flag it as severity:critical with category:incident_pattern.
- If you're not confident an issue is real, don't flag it.
False positives erode developer trust faster than missed bugs.
Return findings as a JSON array. Return an empty array if no issues found."""
The prompt is the agent’s personality. Two design decisions in this prompt matter significantly.
The incident patterns section injects institutional memory into every review. When a developer modifies code related to a past outage, the model has the context to assess whether the change risks reintroducing the same pattern. This is the capability that commercial tools cannot replicate because they don’t have your incident data.
The “if you’re not confident, don’t flag it” instruction is the anti-noise directive. False positives destroy developer trust in AI review faster than anything else. A team that sees three irrelevant comments in a row stops reading the AI’s comments entirely. Precision matters more than recall for adoption, it’s better to catch 70 percent of issues with high precision than 90 percent with high noise.
Component 3: Confidence filter
The confidence filter sits between the review engine and the comment publisher, suppressing findings that are likely to be noise.
class ConfidenceFilter:
"""Suppress low-confidence findings to maintain developer trust."""
def __init__(self, thresholds: dict):
self.thresholds = thresholds
def filter(self, findings: list[ReviewFinding]) -> list[ReviewFinding]:
filtered = []
for finding in findings:
threshold = self.thresholds.get(
finding.category,
self.thresholds.get("default", 0.7)
)
if finding.source == "deterministic":
filtered.append(finding) # Rules always pass through
elif finding.confidence >= threshold:
filtered.append(finding)
else:
# Log suppressed findings for threshold calibration
self._log_suppressed(finding)
return filtered
def _log_suppressed(self, finding: ReviewFinding):
"""Track what we suppress so we can tune thresholds."""
# Periodically review suppressed findings to determine:
# - Are we suppressing too many real issues? Lower threshold.
# - Are we still letting through too much noise? Raise threshold.
pass
# Category-specific thresholds
REVIEW_THRESHOLDS = {
"security": 0.6, # Lower threshold - security findings are worth noise
"incident_pattern": 0.5, # Lowest - past incident patterns are high-value
"logic": 0.75, # Higher - logic findings have high false positive rate
"performance": 0.8, # Highest - perf suggestions are often subjective
"convention": 1.0, # Only deterministic rule matches (always 1.0)
"default": 0.7,
}
The category-specific thresholds reflect a reality about AI code review: different finding types have different false positive profiles. Security and incident-pattern findings are worth tolerating some noise because the cost of missing them is high. Performance suggestions have a high subjective component and generate the most developer complaints, so the threshold is highest.
Component 4: CI/CD integration
The review agent runs as a CI pipeline stage triggered on every PR.
# .github/workflows/ai-review.yml
name: AI Code Review
on:
pull_request:
types: [opened, synchronize]
jobs:
review:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0 # Full history for context assembly
- name: Run AI review agent
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
python -m review_agent \
--base ${{ github.event.pull_request.base.sha }} \
--head ${{ github.event.pull_request.head.sha }} \
--pr-number ${{ github.event.pull_request.number }} \
--conventions .review/conventions.yaml \
--incidents .review/incident_patterns.yaml \
--thresholds .review/confidence_thresholds.yaml
The fetch-depth: 0 is critical. The context assembler needs the full git history for blame analysis, churn hotspot detection, and incident correlation. A shallow clone breaks all of these.
The configuration files live in the repo alongside the code:
.review/
├── conventions.yaml # Team coding conventions
├── incident_patterns.yaml # Past incidents and their patterns
├── confidence_thresholds.yaml # Per-category noise thresholds
└── ignored_patterns.yaml # Known false positives to suppress
This is one of the most important design decisions: the review configuration lives in the repo, versioned alongside the code. When the team adds a new convention, updates an incident pattern, or adjusts a threshold, it’s a PR that goes through the same review process as any other change. The review agent’s knowledge evolves with the codebase.
The training data that matters
The custom agent’s advantage comes from project-specific training data that commercial tools don’t have. Three data sources make the difference:
The incident database captures past production incidents with their root cause patterns and the code patterns that should have been caught in review. Each entry includes the affected files, the code pattern that caused the issue, and the review rule that would have prevented it.
# .review/incident_patterns.yaml
incidents:
- id: INC-2024-047
date: "2024-11-15"
summary: "Payment double-processing during concurrent webhook delivery"
affected_modules: ["payments", "webhooks"]
affected_files: ["src/payments/processor.py", "src/webhooks/handler.py"]
pattern: "Non-idempotent payment processing in webhook handler"
prevention: |
Any payment state mutation in a webhook handler must use
idempotency key verification before processing. Check for
SELECT ... FOR UPDATE or Redis SETNX before any balance modification.
severity: critical
- id: INC-2025-012
date: "2025-03-22"
summary: "Database deadlock during concurrent order processing"
affected_modules: ["orders"]
affected_files: ["src/orders/fulfillment.py"]
pattern: "Acquiring locks on orders and inventory in inconsistent order"
prevention: |
All transactions involving both orders and inventory tables must
acquire locks in the order: inventory first, then orders.
Reverse ordering causes deadlocks under concurrent load.
severity: critical
The conventions file captures the team’s coding standards - not generic style guidelines but the project-specific patterns that evolved from experience.
The ignored patterns file captures known false positives - patterns the model flags that the team has reviewed and determined are acceptable. This file prevents the same false positive from appearing on every PR.
The metrics that determine success
Track four metrics from day one:
class ReviewMetrics:
"""Track what matters for review agent adoption."""
def track(self, pr_id: str, findings: list, developer_actions: dict):
metrics = {
"precision": self._calculate_precision(findings, developer_actions),
"incident_pattern_catch_rate": self._incident_catches(findings),
"time_to_first_review": self._review_latency(pr_id),
"developer_sentiment": developer_actions.get("thumbs_up_ratio", None),
}
return metrics
def _calculate_precision(self, findings, actions) -> float:
"""What % of findings led to code changes?"""
actionable = sum(1 for f in findings
if actions.get(f.id) in ["fixed", "acknowledged"])
return actionable / len(findings) if findings else 1.0
Precision, the percentage of findings that developers actually act on, is the North Star metric. If precision drops below 60 percent, developers stop reading the comments. If it’s above 80 percent, the agent has earned trust.
Incident pattern catch rate tells you whether the custom agent’s primary advantage over commercial tools is delivering. If it’s not catching changes related to past incidents, the incident database needs updating.
The data residency question
For teams with proprietary code that cannot be sent to external APIs, the same architecture works with on-premise AI code review deployment, running open-weight models locally instead of calling cloud APIs. The architecture is identical. The inference layer changes from an API call to a local vLLM server. The review quality on code-specific tasks with models like DeepSeek Coder V3 or Llama 3.3 70B is strong enough for production use, though not quite at the level of frontier cloud models.
The decision between cloud and on-premise is a data residency decision, not a capability decision. If your code can leave the network, the cloud API is simpler and produces higher-quality reviews. If it can’t, the local deployment works, at the cost of GPU infrastructure and slightly lower review quality on the most nuanced findings.
What this costs to build and run
Development time: two to three weeks for the core pipeline if your team has experience with LLM integration. Add one week for the incident database population and conventions codification. Add one week for threshold calibration, running the agent against recent PRs and tuning until precision is above 75 percent.
API costs: at 150 to 200 PRs per month with an average diff size of 300 lines, the Claude API costs run roughly $400 to $700 monthly. Smaller models reduce this but also reduce the quality of nuanced findings.
Maintenance: the conventions file, incident database, and threshold configuration need ongoing attention. Budget two to three hours per week for a team member to review suppressed findings, add new incident patterns, and adjust thresholds. This maintenance is what keeps the agent’s advantage over commercial tools, without it, the project-specific knowledge stagnates and the agent degrades toward generic review.
The total cost over three years, development, API, maintenance, is typically lower than a commercial review tool subscription for teams above 15 engineers, with the added benefit of catching the domain-specific patterns that commercial tools miss entirely.
For teams that want the custom review capability without the build investment, AI agent builders like Dextra Labs design and deploy custom review agents calibrated to your specific codebase, conventions, and incident history, including the ongoing maintenance that keeps the agent’s knowledge current.
The honest assessment
A custom review agent catches things commercial tools miss. It does not catch everything. The 70 percent of issues it flags with high precision are the 70 percent that matter most for your specific codebase. The remaining 30 percent still needs human reviewers, the architectural judgment, the business logic correctness, the design decisions that require understanding intent rather than analysing code.
The custom agent makes your human reviewers better by handling the mechanical and project-specific pattern checking, so human review starts at architecture and logic rather than at conventions and known anti-patterns. That’s the same value proposition as commercial tools, extended to the patterns that only your team knows about.
Build the agent. Feed it your institutional knowledge. Let it catch the things that commercial tools can’t see. And keep your human reviewers focused on the judgment that no tool, custom or commercial, can replicate.
Published by Dextra Labs - AI Consulting and Enterprise Agent Development
Top comments (0)