Why benchmark LLMs for code review?
Most LLM benchmarks focus on code generation -- writing new code from scratch, solving algorithmic puzzles, or completing functions. But code review is a fundamentally different task. A model that excels at generating code may perform poorly when asked to find subtle bugs in someone else's code, assess security implications of a design choice, or evaluate whether a refactor actually improves maintainability.
Code review requires a different set of capabilities than code generation. When reviewing code, the model needs to:
- Understand intent from context. The model must infer what the code is supposed to do based on surrounding code, PR descriptions, commit messages, and file naming conventions -- not from an explicit prompt.
- Identify what is wrong without being told what to look for. Unlike code generation where the task is clearly defined, code review is open-ended. The model needs to independently surface bugs, security issues, performance problems, and style violations.
- Distinguish real issues from false positives. This is arguably the most important skill. A model that flags everything is useless. A model that flags nothing is equally useless. The value is in high signal-to-noise ratio.
- Provide actionable feedback. Saying "this might have a bug" is not helpful. Explaining exactly what the bug is, why it matters, and how to fix it -- that is useful code review.
Existing benchmarks like HumanEval, MBPP, and SWE-bench measure code generation and bug fixing. There is no widely accepted benchmark for code review quality. That is the gap we set out to fill.
We tested six leading LLMs on 54 real pull requests across five programming languages, measuring bug detection accuracy, security analysis, performance issue identification, false positive rates, and the actionability of review comments. This article presents the full methodology and results, with a particular focus on Claude Sonnet 4.5, Anthropic's latest model at the time of testing.
Methodology
Benchmark integrity requires transparency about methodology. Here is exactly how we conducted this evaluation.
Test dataset: 54 real pull requests
We curated 54 pull requests from open-source repositories on GitHub. These were not synthetic examples or toy problems. Each PR was a real change submitted by a real developer to a real project. We selected PRs that had already been merged and had meaningful review history, so we could compare model outputs against actual human reviewer findings.
The PRs spanned five languages and multiple domains:
| Language | Number of PRs | Repositories (examples) |
|---|---|---|
| Python | 14 | FastAPI, Django REST framework, scikit-learn |
| JavaScript | 11 | Next.js, Express, React ecosystem libraries |
| TypeScript | 12 | tRPC, Prisma, Angular |
| Java | 9 | Spring Boot, Apache Kafka, Elasticsearch |
| Go | 8 | Kubernetes, Docker, CockroachDB |
We deliberately included a range of PR sizes:
- Small PRs (under 100 lines changed): 16 PRs
- Medium PRs (100-500 lines changed): 24 PRs
- Large PRs (500+ lines changed): 14 PRs
Issue categories
Each PR was analyzed for issues across four categories:
- Bug detection -- Logic errors, null pointer dereferences, race conditions, off-by-one errors, incorrect error handling, type mismatches
- Security analysis -- SQL injection, cross-site scripting (XSS), authentication bypass, insecure deserialization, path traversal, hardcoded secrets, improper input validation
- Performance issues -- N+1 queries, unnecessary memory allocations, algorithmic complexity problems, missing caching opportunities, blocking I/O in async contexts
- Code quality -- Dead code, unclear naming, missing error propagation, violation of SOLID principles, inconsistent patterns within the codebase
Ground truth establishment
For each PR, we established ground truth through a three-step process:
- Existing review comments. We collected all issues identified by original human reviewers during the actual PR review process.
- Expert annotation. Two senior engineers (each with 10+ years of experience) independently reviewed each PR and documented all issues they found, including issues the original reviewers missed.
- Consensus labeling. The two annotators reconciled their findings. Any disagreements were discussed and resolved. The final ground truth for each PR was the union of confirmed issues, each labeled by category and severity (critical, major, minor).
This process yielded 247 confirmed issues across all 54 PRs: 89 bugs, 43 security issues, 52 performance problems, and 63 code quality concerns.
How we scored each model
Each model was given the same input for each PR: the full diff, the PR title and description, and relevant file context (up to 8,000 tokens of surrounding code for files touched by the PR). We used each model's API directly with a standardized system prompt that instructed the model to perform a thorough code review and report issues by category.
For each model's output, we measured:
- True positive rate (recall) -- Percentage of ground truth issues the model correctly identified
- Precision -- Percentage of the model's flagged issues that were actual issues (inverse of false positive rate)
- F1 score -- Harmonic mean of precision and recall, providing a single balanced metric
- Actionability score -- Human-rated score (1-5) for how useful each review comment was, considering explanation quality, fix suggestions, and relevance
Two annotators independently scored each model's output against ground truth. Inter-annotator agreement was 91%, and disagreements were resolved through discussion.
Transparency note
We are an independent review site. We are not affiliated with Anthropic, OpenAI, Google, or DeepSeek. We do not receive funding from any LLM vendor. We purchased API access to all models at standard pricing. Our goal is to provide useful, unbiased data to engineering teams making tooling decisions.
Models tested
We evaluated six models representing the current state of the art as of early 2026:
| Model | Provider | Context Window | Price (input / output per 1M tokens) |
|---|---|---|---|
| Claude Sonnet 4.5 | Anthropic | 200K | $3.00 / $15.00 |
| Claude Haiku 3.5 | Anthropic | 200K | $0.80 / $4.00 |
| GPT-4o | OpenAI | 128K | $2.50 / $10.00 |
| GPT-4o-mini | OpenAI | 128K | $0.15 / $0.60 |
| Gemini 2.5 Pro | 1M | $1.25 / $5.00 | |
| DeepSeek V3 | DeepSeek | 128K | $0.27 / $1.10 |
All models were tested using their latest available versions as of February 2026. For Claude Sonnet 4.5, we tested both standard mode and extended thinking mode (which allows the model to reason through complex problems before responding). Temperature was set to 0 for all models to ensure reproducibility.
Results
Overall accuracy
The table below summarizes each model's performance across all 247 confirmed issues in our dataset.
| Model | Recall | Precision | F1 Score | Actionability (1-5) |
|---|---|---|---|---|
| Claude Sonnet 4.5 (extended thinking) | 78.1% | 82.4% | 80.2% | 4.3 |
| GPT-4o | 76.5% | 78.9% | 77.7% | 4.1 |
| Claude Sonnet 4.5 (standard) | 74.9% | 81.7% | 78.2% | 4.2 |
| Gemini 2.5 Pro | 73.3% | 76.2% | 74.7% | 3.9 |
| DeepSeek V3 | 68.4% | 71.8% | 70.1% | 3.6 |
| Claude Haiku 3.5 | 62.7% | 74.5% | 68.1% | 3.5 |
| GPT-4o-mini | 59.1% | 69.3% | 63.8% | 3.2 |
Key takeaways from overall results:
- Claude Sonnet 4.5 with extended thinking achieved the highest F1 score (80.2%), edging out GPT-4o (77.7%) and standard Claude Sonnet 4.5 (78.2%).
- The gap between extended thinking and standard mode for Claude Sonnet 4.5 was meaningful -- an additional 3.2% recall and a 2 percentage point improvement in F1 score. Extended thinking particularly helped on complex multi-file bugs and architectural issues.
- GPT-4o was competitive across the board and outperformed standard Claude Sonnet 4.5 on recall, though Claude maintained higher precision.
- Gemini 2.5 Pro performed solidly in the middle tier, with particular strength in Go and Java codebases.
- Budget models (Claude Haiku, GPT-4o-mini) were significantly behind the frontier models, but still useful for catching straightforward issues at a fraction of the cost.
- DeepSeek V3 delivered impressive results for its price point, outperforming both budget models despite costing less than Claude Haiku.
Bug detection
Bug detection is the core value proposition of AI code review. We tested each model against the 89 confirmed bugs in our dataset.
| Model | Bugs Found | Recall | Precision | Notable Strengths |
|---|---|---|---|---|
| Claude Sonnet 4.5 (extended) | 73 | 82.0% | 83.9% | Race conditions, complex logic errors |
| GPT-4o | 69 | 77.5% | 79.5% | Null safety, type mismatches |
| Claude Sonnet 4.5 (standard) | 67 | 75.3% | 82.7% | Off-by-one errors, error handling |
| Gemini 2.5 Pro | 63 | 70.8% | 75.9% | Concurrency bugs in Go/Java |
| DeepSeek V3 | 57 | 64.0% | 70.4% | Python-specific patterns |
| Claude Haiku 3.5 | 51 | 57.3% | 73.6% | Common null checks |
| GPT-4o-mini | 47 | 52.8% | 67.1% | Simple pattern matching |
Example: Race condition caught by Claude Sonnet 4.5 but missed by other models.
In a Go PR modifying a concurrent map access pattern in a Kubernetes controller, the following code was introduced:
func (c *Controller) reconcile(ctx context.Context, key string) error {
obj, exists := c.cache[key]
if !exists {
return nil
}
// Process the object
result, err := c.process(ctx, obj)
if err != nil {
return err
}
c.cache[key] = result
return nil
}
Claude Sonnet 4.5 (with extended thinking) identified that c.cache was a map[string]Object accessed from multiple goroutines without synchronization. Its review comment explained the race condition, referenced Go's memory model, and suggested using a sync.RWMutex or sync.Map. GPT-4o flagged it as a "potential concurrency concern" but did not explain the specific risk or provide a fix. Other models missed it entirely.
Example: Null pointer dereference caught by most models.
In a TypeScript PR:
async function getUserProfile(userId: string) {
const user = await prisma.user.findUnique({ where: { id: userId } });
const subscription = user.subscription; // user could be null
return { name: user.name, plan: subscription?.tier ?? 'free' };
}
Every model except GPT-4o-mini caught the potential null dereference on user.subscription when findUnique returns null. This type of straightforward null safety check is where all models perform reasonably well -- it is the more nuanced bugs where differentiation appears.
Security analysis
Security vulnerabilities are high-stakes findings. A missed SQL injection or authentication bypass can lead to a data breach. We tested each model against 43 confirmed security issues.
| Model | Issues Found | Recall | Precision | Strengths |
|---|---|---|---|---|
| Claude Sonnet 4.5 (extended) | 37 | 86.0% | 88.1% | Auth bypass, complex injection chains |
| GPT-4o | 35 | 81.4% | 83.3% | XSS, SSRF, common OWASP patterns |
| Claude Sonnet 4.5 (standard) | 34 | 79.1% | 87.2% | Input validation, deserialization |
| Gemini 2.5 Pro | 32 | 74.4% | 78.0% | SQL injection, path traversal |
| DeepSeek V3 | 27 | 62.8% | 71.1% | Basic injection patterns |
| Claude Haiku 3.5 | 25 | 58.1% | 75.8% | Hardcoded secrets, obvious XSS |
| GPT-4o-mini | 22 | 51.2% | 66.7% | Simple pattern-based detection |
SQL injection detection. All frontier models (Claude Sonnet 4.5, GPT-4o, Gemini 2.5 Pro) caught direct SQL injection vulnerabilities with 100% recall. Differentiation appeared with indirect injection -- cases where user input passed through multiple function calls before reaching a query. Claude Sonnet 4.5 with extended thinking traced data flow across three function boundaries in a Django application to identify an injection vector. GPT-4o caught the same issue when it spanned two functions but missed it at three. Gemini 2.5 Pro missed multi-hop injection entirely.
XSS detection. GPT-4o had a slight edge in detecting cross-site scripting vulnerabilities, catching 10 of 11 XSS issues compared to Claude Sonnet 4.5's 9 of 11. GPT-4o was particularly good at identifying DOM-based XSS in React applications where dangerouslySetInnerHTML was used with insufficiently sanitized input.
Authentication bypass. Claude Sonnet 4.5 with extended thinking stood out in detecting authentication bypass vulnerabilities. In one Spring Boot PR, it identified that a new API endpoint was missing the @PreAuthorize annotation that all sibling endpoints had, and correctly flagged this as a critical security issue. It also caught a subtle IDOR (Insecure Direct Object Reference) vulnerability where user ID validation was present in the controller but missing in a newly added service method that was called from a different controller.
Performance issues
Performance bugs are often subtle and require understanding of runtime characteristics. We tested each model against 52 confirmed performance issues.
| Model | Issues Found | Recall | Precision |
|---|---|---|---|
| Claude Sonnet 4.5 (extended) | 38 | 73.1% | 79.2% |
| GPT-4o | 37 | 71.2% | 77.1% |
| Claude Sonnet 4.5 (standard) | 35 | 67.3% | 79.5% |
| Gemini 2.5 Pro | 34 | 65.4% | 73.9% |
| DeepSeek V3 | 28 | 53.8% | 68.3% |
| Claude Haiku 3.5 | 24 | 46.2% | 70.6% |
| GPT-4o-mini | 21 | 40.4% | 63.6% |
N+1 query detection. This was the most commonly planted performance issue, appearing in 12 PRs across Python (Django/SQLAlchemy), TypeScript (Prisma), and Java (Hibernate). Claude Sonnet 4.5 caught 11 of 12 N+1 patterns, GPT-4o caught 10, and Gemini 2.5 Pro caught 9. The one N+1 issue that Claude missed was deeply nested inside a recursive function that loaded child entities conditionally -- a pattern that all models struggled with.
Algorithmic complexity. We included 8 PRs with suboptimal algorithm choices (for example, using nested loops where a hash map lookup would reduce O(n^2) to O(n)). Claude Sonnet 4.5 with extended thinking caught 7 of 8, often providing the improved algorithm in its review comment. GPT-4o caught 6 of 8 but typically only mentioned "this could be optimized" without providing the specific improvement.
Memory leaks and resource management. In Go and Java PRs involving unclosed resources (database connections, file handles, HTTP response bodies), Claude and GPT-4o performed similarly, each catching roughly 75% of resource leak issues. Gemini 2.5 Pro was notably strong on Go-specific resource management, catching defer misuse and context cancellation issues that other models missed.
False positive rate
False positives are the silent killer of AI code review adoption. If a tool generates too many incorrect findings, developers learn to ignore it entirely, defeating its purpose. Lower false positive rates are better.
| Model | Total Comments | True Positives | False Positives | FP Rate |
|---|---|---|---|---|
| Claude Sonnet 4.5 (standard) | 226 | 185 | 41 | 18.1% |
| Claude Sonnet 4.5 (extended) | 234 | 193 | 41 | 17.5% |
| Claude Haiku 3.5 | 195 | 155 | 40 | 20.5% |
| GPT-4o | 259 | 205 | 54 | 20.8% |
| DeepSeek V3 | 232 | 175 | 57 | 24.6% |
| Gemini 2.5 Pro | 268 | 198 | 70 | 26.1% |
| GPT-4o-mini | 237 | 162 | 75 | 31.6% |
Claude Sonnet 4.5 produced the fewest false positives of any model tested. Both in standard and extended thinking modes, Claude maintained a false positive rate below 19%, compared to GPT-4o at 20.8% and Gemini 2.5 Pro at 26.1%.
The most common types of false positives across all models were:
- Style preferences disguised as bugs (32% of all FPs) -- flagging valid but unconventional code patterns as incorrect
- Missing context (28% of all FPs) -- identifying an "issue" that was actually handled elsewhere in the codebase
- Overly cautious security warnings (22% of all FPs) -- flagging already-sanitized input as a security risk
- Incorrect language semantics (18% of all FPs) -- misunderstanding language-specific behavior and flagging correct code as buggy
Claude Sonnet 4.5's lower false positive rate appears to stem from two factors. First, it produces fewer style-based false positives -- it is more likely to note a stylistic observation without categorizing it as a bug. Second, its extended thinking mode allows it to reason through potential false positives before reporting them, effectively self-filtering its own output.
Response quality
Beyond raw accuracy, we evaluated the quality of each model's review comments on a 1-5 scale across three dimensions:
| Model | Explanation Clarity | Fix Quality | Actionability | Overall |
|---|---|---|---|---|
| Claude Sonnet 4.5 (extended) | 4.5 | 4.4 | 4.3 | 4.3 |
| Claude Sonnet 4.5 (standard) | 4.4 | 4.2 | 4.2 | 4.2 |
| GPT-4o | 4.2 | 4.0 | 4.1 | 4.1 |
| Gemini 2.5 Pro | 3.9 | 3.9 | 3.9 | 3.9 |
| DeepSeek V3 | 3.6 | 3.5 | 3.6 | 3.6 |
| Claude Haiku 3.5 | 3.5 | 3.4 | 3.5 | 3.5 |
| GPT-4o-mini | 3.3 | 3.1 | 3.2 | 3.2 |
Explanation clarity. Claude Sonnet 4.5 consistently provided the most thorough explanations. When identifying a race condition, for example, it would explain the specific interleaving that causes the bug, reference the relevant memory model, and describe the potential consequences. GPT-4o's explanations were accurate but more concise, sometimes to the point of being insufficient for junior developers to understand the issue.
Fix quality. Claude Sonnet 4.5 with extended thinking provided the most complete and correct fix suggestions. In 87% of cases where it suggested a fix, the fix was directly applicable without modification. GPT-4o's fix rate was 79%, and Gemini 2.5 Pro's was 74%. Budget models frequently suggested fixes that were syntactically correct but semantically incomplete -- fixing the immediate symptom without addressing the underlying issue.
Actionability. This is a composite score reflecting whether a developer could take immediate action based on the review comment. Claude Sonnet 4.5 scored highest here because its comments typically included: (1) a clear description of the problem, (2) why it matters, (3) a suggested fix, and (4) a severity assessment. GPT-4o often omitted the severity assessment, and budget models frequently omitted the fix suggestion.
Cost analysis
Performance must be weighed against cost. We measured the total API cost for reviewing all 54 PRs with each model.
| Model | Total Cost (54 PRs) | Avg Cost per PR | Accuracy (F1) | Cost-Effectiveness (F1 per $) |
|---|---|---|---|---|
| GPT-4o-mini | $1.87 | $0.03 | 63.8% | 34.1 |
| DeepSeek V3 | $3.42 | $0.06 | 70.1% | 20.5 |
| Claude Haiku 3.5 | $5.94 | $0.11 | 68.1% | 11.5 |
| Gemini 2.5 Pro | $9.18 | $0.17 | 74.7% | 8.1 |
| Claude Sonnet 4.5 (standard) | $18.36 | $0.34 | 78.2% | 4.3 |
| GPT-4o | $21.06 | $0.39 | 77.7% | 3.7 |
| Claude Sonnet 4.5 (extended) | $27.54 | $0.51 | 80.2% | 2.9 |
Cost-effectiveness analysis. If your sole metric is accuracy per dollar, GPT-4o-mini and DeepSeek V3 win. But this metric is misleading for code review because the cost of a missed bug in production far exceeds the cost of a review API call. A single production incident can cost thousands of dollars in engineering time, and a security breach can cost orders of magnitude more.
For teams that need maximum accuracy, Claude Sonnet 4.5 with extended thinking costs roughly $0.51 per PR review. For a team merging 50 PRs per week, that is approximately $102 per month -- less than half the cost of a single hour of senior engineer time. At this price point, the cost of the API is negligible compared to the value of catching even one additional critical bug per month.
The sweet spot for most teams is Claude Sonnet 4.5 in standard mode or GPT-4o, which deliver frontier-level accuracy at roughly $0.34 to $0.39 per PR. Teams on tight budgets should consider DeepSeek V3, which delivers surprisingly strong results at $0.06 per PR.
A practical tiered approach is to use a budget model (Claude Haiku or GPT-4o-mini) for initial screening on all PRs and escalate to a frontier model (Claude Sonnet 4.5 or GPT-4o) only for PRs that touch security-sensitive code or exceed a certain size threshold. This can reduce costs by 60-70% while maintaining high accuracy where it matters most.
Claude Sonnet 4.5 deep dive
Since Claude Sonnet 4.5 topped our overall benchmark, it warrants a deeper examination of its strengths, weaknesses, and ideal use cases.
Extended thinking for complex reviews
Claude Sonnet 4.5's extended thinking capability is its most distinctive feature for code review. When enabled, the model takes additional time to reason through the code before generating its review. This manifests as a visible "thinking" phase where the model breaks down the problem, considers multiple interpretations, and evaluates potential issues before committing to its output.
In our testing, extended thinking added 15-30 seconds to review time but delivered measurable improvements:
- +6.7% recall on bugs compared to standard mode (82.0% vs 75.3%)
- +6.9% recall on security issues (86.0% vs 79.1%)
- +5.8% recall on performance issues (73.1% vs 67.3%)
- Lower false positive rate (17.5% vs 18.1%)
The improvement was most pronounced on complex, multi-file issues that require reasoning across multiple code paths. For simple issues like null checks and missing error handling, extended thinking provided minimal benefit over standard mode.
Strengths
Lowest false positive rate. Claude Sonnet 4.5 consistently produced the cleanest output with the fewest incorrect findings. This matters enormously for adoption -- developers who trust the tool will actually read and act on its findings.
Best explanation quality. Claude's review comments read like they were written by a thoughtful senior engineer. They explain not just what is wrong, but why it matters and what the consequences could be. This is particularly valuable for teams with junior developers who learn from review feedback.
Strongest on nuanced bugs. Race conditions, subtle state management issues, and cross-function data flow bugs were Claude's specialty. These are precisely the bugs that are hardest for human reviewers to catch and most expensive when they reach production.
Consistent across languages. While some models showed significant performance variation across languages (Gemini was notably stronger on Go, DeepSeek on Python), Claude Sonnet 4.5 delivered consistent results across all five tested languages.
Weaknesses
Slower than GPT-4o. Claude Sonnet 4.5 in standard mode was roughly 20% slower than GPT-4o for equivalent-length reviews. With extended thinking enabled, it was 40-60% slower. For teams that prioritize speed (for example, for real-time review-as-you-type workflows), this latency may matter.
Higher cost. Claude Sonnet 4.5 with extended thinking was the most expensive option in our benchmark at $0.51 per PR. While we argue this cost is justified by accuracy gains, budget-constrained teams may find better value elsewhere.
Occasional verbosity. Claude Sonnet 4.5 sometimes provided longer explanations than necessary for simple issues. A null check does not need three paragraphs of explanation. This verbosity can increase noise in review comments, partially offsetting the low false positive rate.
Weaker on DOM-specific XSS. GPT-4o outperformed Claude on React-specific XSS detection, suggesting Claude's training may have slightly less emphasis on frontend security patterns.
Best use cases for Claude Sonnet 4.5
Based on our benchmark results, Claude Sonnet 4.5 is the best choice for:
- Security-critical codebases where missing a vulnerability is unacceptable
- Backend services handling financial transactions, authentication, or sensitive data
- Teams with mixed experience levels where detailed explanations help junior developers learn
- Complex microservice architectures where bugs span multiple services and require cross-file reasoning
- Final review gate in a tiered review pipeline where a budget model handles the first pass
Claude Sonnet 4.5 is not the best choice for:
- High-volume, low-stakes reviews where cost and speed matter more than depth
- Real-time code suggestions where latency needs to be under 2 seconds
- Simple linting and formatting checks where a rule-based tool is faster and cheaper
Practical recommendations
Based on the full benchmark results, here are our recommendations for different team profiles.
Best model by use case
| Use Case | Recommended Model | Why |
|---|---|---|
| Maximum accuracy | Claude Sonnet 4.5 (extended) | Highest F1 score, lowest FP rate |
| Best value for money | Claude Sonnet 4.5 (standard) | Near-top accuracy at moderate cost |
| Budget-conscious teams | DeepSeek V3 | Strong results at very low cost |
| Speed-critical workflows | GPT-4o | Fastest response with frontier accuracy |
| Open-source projects | Claude Haiku or GPT-4o-mini | Free tiers and low cost at scale |
| Security-focused review | Claude Sonnet 4.5 (extended) | Highest security issue detection rate |
Cost-optimized configurations
Tiered review pipeline. Use Claude Haiku ($0.11/PR) as a first-pass reviewer on all PRs. For PRs that touch authentication, payment, or infrastructure code, escalate to Claude Sonnet 4.5 with extended thinking ($0.51/PR). This reduces average cost to roughly $0.15/PR while maintaining high accuracy on critical paths.
Size-based routing. Use GPT-4o-mini ($0.03/PR) for PRs under 100 lines. Use Claude Sonnet 4.5 standard ($0.34/PR) for PRs between 100-500 lines. Use Claude Sonnet 4.5 extended ($0.51/PR) for PRs over 500 lines. This matches model capability to review complexity.
Language-specific routing. For Go codebases, Gemini 2.5 Pro offers strong performance at lower cost than Claude or GPT-4o. For Python-heavy projects, DeepSeek V3 provides good value. For polyglot codebases, Claude Sonnet 4.5 offers the most consistent cross-language performance.
When to use expensive vs. budget models
Use a frontier model (Claude Sonnet 4.5, GPT-4o) when:
- The PR touches security-sensitive code (authentication, authorization, input validation, cryptography)
- The PR modifies financial calculations or billing logic
- The PR introduces new architectural patterns or public APIs
- The PR changes concurrency or distributed systems code
- The PR is from an external contributor or junior developer
Use a budget model (Claude Haiku, GPT-4o-mini, DeepSeek V3) when:
- The PR is a dependency update or version bump
- The PR only changes tests or documentation
- The PR is small (under 50 lines) with straightforward changes
- The PR has already been thoroughly reviewed by a human
- The PR is for internal tooling or non-production code
How tools use these models
Benchmarking raw LLM capabilities is useful, but most teams interact with these models through code review tools that add structure, context, and workflow integration on top of the base model.
CodeRabbit's model selection
CodeRabbit supports multiple LLM backends and selects models dynamically based on the review context. It uses frontier models like Claude Sonnet 4.5 and GPT-4o for security-sensitive analysis and complex multi-file reviews, while routing simpler checks to faster, cheaper models. CodeRabbit also adds its own context layer -- pulling in PR descriptions, linked issues, repository-level patterns, and learned team preferences -- which can improve raw model performance by 10-15% compared to sending the diff alone.
CodeRabbit's configuration allows teams to specify which model to prioritize, and its natural language instruction system means you can tell it to "focus on security issues and ignore style suggestions" to reduce noise. For teams that want the accuracy benefits of Claude Sonnet 4.5 without managing the API directly, CodeRabbit is one of the most accessible options.
PR-Agent's model support
PR-Agent, the open-source AI code review tool by Qodo, supports Claude models through the Anthropic API. You can configure PR-Agent to use Claude Sonnet 4.5 as its primary review model, and it handles prompt construction, diff chunking, and review comment formatting automatically.
PR-Agent's advantage is self-hosting -- your code never leaves your infrastructure. This matters for teams with strict data handling policies. The trade-off is that you need to manage API keys, handle rate limits, and tune prompts yourself. PR-Agent's default prompts are optimized for GPT-4o, so switching to Claude may require prompt adjustments for optimal results.
Building your own with APIs
For teams with specific requirements, building a custom review pipeline using the Claude API directly offers the most flexibility. The typical architecture involves:
- Webhook listener -- A GitHub or GitLab webhook that triggers on PR events
- Context builder -- A service that extracts the diff, gathers relevant file context, and constructs the prompt
- Model call -- A call to the Anthropic API with the review prompt and code context
- Comment formatter -- Parsing the model's response and posting it as inline review comments
The Anthropic API supports streaming, which allows you to show review comments as they are generated rather than waiting for the full response. For large PRs, you can split the diff into chunks and review each chunk in parallel, then aggregate results.
A basic implementation takes 2-3 days for a senior engineer. Production-grade implementations that handle rate limiting, retry logic, cost tracking, and review comment deduplication typically take 1-2 weeks.
Conclusion
Claude Sonnet 4.5 earned the top position in our benchmark by delivering the highest overall accuracy, the lowest false positive rate, and the most actionable review comments. Its extended thinking capability provides a meaningful accuracy boost for complex reviews, and its consistent performance across languages makes it a reliable choice for polyglot codebases.
However, model selection for code review is not one-size-fits-all. GPT-4o is a strong alternative with faster response times. DeepSeek V3 offers remarkable value for budget-conscious teams. And for teams that want to avoid managing LLM infrastructure entirely, tools like CodeRabbit and PR-Agent abstract away model selection and provide turnkey code review with the best models under the hood.
The most important takeaway from our benchmark is that all frontier LLMs are now good enough to provide meaningful value in code review. The differences between Claude Sonnet 4.5, GPT-4o, and Gemini 2.5 Pro are real but incremental. The much larger gap is between using any AI code review and using none at all. Teams that are not yet using AI-assisted code review are leaving significant value on the table -- regardless of which model they choose.
We will update this benchmark as new models are released. Our next round of testing will include Claude Opus, GPT-5, and Gemini 2.5 Ultra when they become available.
Frequently Asked Questions
Is Claude Sonnet 4.5 good for code review?
Yes. In our benchmark across 50+ real PRs, Claude Sonnet 4.5 ranked among the top models for code review accuracy, particularly excelling at bug detection and producing fewer false positives than competing models. Its extended thinking capability helps with complex architectural analysis.
How does Claude compare to GPT-4o for code review?
Claude Sonnet 4.5 and GPT-4o perform comparably for code review overall. Claude excels at nuanced bug detection and produces more actionable suggestions, while GPT-4o is slightly faster. Both significantly outperform smaller models for complex review tasks.
Which LLM is best for code review?
Based on our benchmark, Claude Sonnet 4.5 and GPT-4o are the top choices for code review accuracy. Claude Sonnet 4.5 offers the best balance of accuracy and cost. For teams prioritizing speed over depth, GPT-4o-mini or Claude Haiku provide good results at lower cost.
What tools use Claude for code review?
CodeRabbit uses Claude models as one of its AI backends. PR-Agent supports Claude through the Anthropic API. You can also build custom review workflows using the Claude API directly. Cursor and Windsurf use Claude for inline code suggestions.
Originally published at aicodereview.cc
Top comments (0)