DEV Community

TOMOAKI ishihara
TOMOAKI ishihara

Posted on

AI-Powered Code Reviews with OpenRouter - Complete PR-Agent Setup Guide

#github #ai #codereview #openrouter

👋 Introduction

Code reviews take time, and large teams often struggle with PR . PR-Agent (now Qodo Merge) uses AI to automatically review pull requests, generate descriptions, and suggest improvements.

This guide shows you how to set up PR-Agent with OpenRouter, giving you cost-effective access to multiple AI models (GPT-4o, Claude, Gemini, etc.) through a unified API.

📌 Why OpenRouter?

Challenges with Direct OpenAI API

  • Single provider dependency (OpenAI models only)
  • Complex management of multiple provider APIs
  • Difficult to compare pricing across different model

OpenRouter Benefits

  • Unified Multi-Model API - Access GPT-4o, Claude, Gemini, Llama, etc. through one interface
  • Flexible Model Selection - Choose the right model for each task, optimizing cost-effectiveness
  • Early Access to Latest Models - Quick access to new models like GPT-5
  • Transparent Pricing - Clear pricing for each model (5.5% fee on credit purchases)

Prerequisites

  • GitHub repository (private or public)
  • OpenRouter account (free to create)
  • GitHub Actions permissions

Step 1: OpenRouter Setup

1.1 Account Creation and API Key

  1. Visit OpenRouter and create an account
  2. Purchase Credits (Important)
    • Go to the "Credits" page in your dashboard
    • Minimum purchase is $5 (5.5% fee applies to credit purchases)
    • Auto-recharge option available for convenience
    • ⚠️ Free Tier: New accounts receive a small free credit allowance, but additional purchases are needed for production use(FAQ: What free tier options exist?)
  3. In the dashboard, go to "API Keys" and generate a new key
  4. Copy and securely store the key

1.2 Available Models

OpenRouter provides access to models like:

openai/gpt-5           # Latest model (2025 release)
openai/gpt-4o          # Balanced performance
anthropic/claude-3-opus-20240229  # High-quality analysis
google/gemini-pro      # Google's model
meta-llama/llama-2-70b-chat  # Open source
Enter fullscreen mode Exit fullscreen mode

Check pricing at OpenRouter Models.

Step 2: GitHub Actions Workflow Setup

2.1 Configure Secrets

In your GitHub repository:

  1. Go to Settings → Secrets and variables → Actions
  2. Click "New repository secret"
  3. Add the following secret: 
Name: OPENROUTER_API_KEY
Value: or-xxxxxxxx (your OpenRouter API key)
Enter fullscreen mode Exit fullscreen mode

2.2 Create Workflow File

Create .github/workflows/pr_agent.yml

name: Code Review(OpenRouter)

on:
  pull_request:
  pull_request_review_comment:
    types: [created]

# Concurrency control
concurrency:
  group: ${{ github.repository }}-${{ github.event.number || github.head_ref || github.sha }}-${{ github.workflow }}-${{ github.event_name == 'pull_request_review_comment' && 'pr_comment' || 'pr' }}
  cancel-in-progress: ${{ github.event_name != 'pull_request_review_comment' }}

jobs:
  pr_agent_job:
    if: |
      github.event.sender.type != 'Bot' &&
      github.event_name == 'pull_request' &&
      github.event.pull_request.changed_files < 50
    runs-on: ubuntu-latest
    permissions:
      issues: write
      pull-requests: write
      contents: write
      id-token: write
    name: Run pr agent on every pull request
    steps:
      - name: PR Agent action step with OpenRouter
        id: pragent
        uses: qodo-ai/pr-agent@main
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          OPENROUTER.KEY: ${{ secrets.OPENROUTER_API_KEY }} # OpenRouter API認証
          github_action_config.auto_review: "true" # enable\disable auto review
          github_action_config.auto_describe: "true" # enable\disable auto describe
          github_action_config.auto_improve: "true" # enable\disable auto improve
Enter fullscreen mode Exit fullscreen mode

Step 3: Configuration File

Create .pr_agent.toml in your repository root.

[config]
model = "openai/gpt-5" # I specified gpt-5, but an error occurred. The correct setting is "openai/gpt-5".
fallback_models = ["openai/gpt-4o", "anthropic/claude-opus-4.1"]
ai_timeout = 300
custom_model_max_tokens = 200000 # Critical! Errors without this
response_language = "en-US"
ignore_pr_title = ["^\\[Auto\\]", "^Auto"]
ignore_pr_labels = ['invalid']

[ignore]
glob = ['dist/**']

[github_action_config]
# set as env var in .github/workflows/pr-agent.yaml
#auto_review = true
#auto_improve = true
#auto_describe = true

[pr_reviewer]
extra_instructions = """\
(1) Act as a highly experienced software engineer
(2) Provide thorough review of code, documents, and articles
(3) Suggest concrete code snippets for improvement

(4) **Never** comment on indentation, whitespace, blank lines, or other purely stylistic issues unless they change program semantics.

(5) **Priority Review Areas - Check systematically:**
- **Security**: Plaintext passwords, SQL injection, input validation, exception leaks
- **Error Handling**: Bare except clauses, missing try-catch, silent error suppression  
- **Resource Management**: Missing context managers (with statements), unclosed connections/files
- **Type Safety**: Missing type hints, incorrect type usage, unjustified Any types
- **Performance**: Inefficient algorithms (O(n²) or worse), unnecessary loops, memory leaks
- **Code Quality**: Magic numbers, unclear variable names, unused imports/variables
- **API Design**: Missing input validation, no error responses, required field checks
- **Architecture**: Single responsibility violations, tight coupling, global state usage

(6) Focus on concrete, actionable issues with specific code examples and fix recommendations.
"""
num_code_suggestions = 5
inline_code_comments = true
ask_and_reflect = true

[pr_description]
extra_instructions = "Generate clear and comprehensive descriptions"
generate_ai_title = true
add_original_user_description = false 
publish_description_as_comment = true

[pr_code_suggestions]
extra_instructions = "Provide actionable code suggestions with examples"
commitable_code_suggestions = true
demand_code_suggestions_self_review=true
Enter fullscreen mode Exit fullscreen mode

Step 4: Testing the Setup

4.1 Create Test PR

Create a PR with intentionally problematic code.

# Missing type hints
def fizz_buzz(n):
    for i in range(1, n):
        if i % 15 == 0:
            print("FizzBuzz")
        elif i % 3 == 0:
            print("Fizz") 
        elif i % 5 == 0:
            print("Buzz")
        else:
            printo(i)  # Intentional bug

# Security issue
def add_user(name, password):
    users = []
    user = {"name": name, "password": password}  # Plaintext password
    users.append(user)
Enter fullscreen mode Exit fullscreen mode

4.2 Expected Output

PR-Agent will provide.

Auto-generated PR Description:

  • Summary of changes
  • File-by-file change details
  • Impact analysis

Code Review Comments:

  • Fix for printo(i) → print(i)
  • Type hint suggestions
  • Security warning about plaintext passwords
  • Performance improvement suggestions

Step 5: Advanced Configuration

5.1 Cost Optimization

Use different models for different tasks.

[config]
model = "openai/gpt-4o-mini"  # For lightweight tasks
model_turbo = "openai/gpt-5"  # For complex analysis
fallback_models = ["anthropic/claude-3.7-sonnet", "google/gemini-2.5-pro"]
Enter fullscreen mode Exit fullscreen mode

5.2 Team-Specific Rules 

[pr_reviewer]
extra_instructions = """\
(1) Company-specific checks:
- Use structured logging for all log outputs
- External API calls must have timeout settings
- Database access must use transactions
- Never hardcode secrets (API keys, passwords, etc.)
"""
Enter fullscreen mode Exit fullscreen mode

5.3 File Exclusions 

[ignore]
glob = ['dist/**', '*.min.js', 'node_modules/**', '__pycache__/**']

[config]
ignore_pr_title = ["^\\[Auto\\]", "^WIP:", "^Draft:"]
ignore_pr_labels = ['work-in-progress', 'do-not-review']
Enter fullscreen mode Exit fullscreen mode

Troubleshooting

Common Issues and Solutions

1. Model not recognized

Error: Model openai/gpt-5 is not defined in MAX_TOKENS
Solution: Add custom_model_max_tokens = 200000 to config
Enter fullscreen mode Exit fullscreen mode

This error occurs when using a model that PR-Agent doesn't recognize. The models supported by PR-Agent are listed in this file. If your chosen model isn't in this list, you need to set the custom_model_max_tokens configuration.

2. Inconsistent language responses

[config]
response_language = "en-US"  # Be specific

[pr_reviewer]
extra_instructions = "Always respond in English and explain technical terms clearly"
Enter fullscreen mode Exit fullscreen mode

3. GitHub Actions permission errors

permissions:
  issues: write
  pull-requests: write
  contents: write  # Add this line
  id-token: write
Enter fullscreen mode Exit fullscreen mode

4. Timeouts on large PRs

[config]
ai_timeout = 600  # Extend to 10 minutes
large_patch_policy = "clip"  # Split large PRs
Enter fullscreen mode Exit fullscreen mode

Cost Management Best Practices

1. Appropriate Model Selection

Task Recommended Model Pricing (approx.)
Simple reviews gpt-4o-mini $0.15/1M tokens
Detailed analysis gpt-4o $2.5/1M tokens
Highest quality openai/gpt-5 $1.25/1M tokens

2. PR Size Limits 

if: github.event.pull_request.changed_files < 50  # Limit file count
Enter fullscreen mode Exit fullscreen mode

3. Control Automatic Execution 

[config]
ignore_pr_labels = ['draft', 'wip', 'skip-ai-review']
ignore_pr_title = ["^\\[WIP\\]", "^Draft:", "^Auto"]
Enter fullscreen mode Exit fullscreen mode

Impact Measurement

Before/After Comparison Metrics

Quantitative Metrics:

  • Reduced code review time
  • Increased bug detection
  • Earlier security issue identification
  • Faster PR merge times

Qualitative Metrics:

  • Reduced reviewer burden
  • Improved code quality
  • Learning effect (developer skill improvement)

Real-World Results Example 

Before: Average review time 2 hours/PR
After: Average review time 45 minutes/PR (62% reduction)

Security issue detection:
- Before: 2 issues/month average
- After: 8 issues/month average (4x increase)
Enter fullscreen mode Exit fullscreen mode

📝 Conclusion

Setting up PR-Agent with OpenRouter provides these benefits.

✅ Cost Performance - Use multiple models through OpenRouter for cost optimization

✅ Quality Improvement - Consistent review standards reduce oversights

✅ Efficiency - Significant reduction in review time

✅ Learning Effect - Developers learn best practices from AI feedback

✅ Flexibility - Customizable for team-specific requirements

We recommend starting with a small project for testing and gradually expanding. The configuration file allows extensive customization to match your team's needs.

📚 References

Hope this guide helps improve your code review process! Feel free to share questions or suggestions in the comments.

Top comments (0)