DEV Community

Paul Robertson
Paul Robertson

Posted on

AI-Powered Code Review: Automate Pull Request Analysis with GitHub Actions

#ai #devops #github #automation

This article contains affiliate links. I may earn a commission at no extra cost to you.

title: "AI-Powered Code Review: Automate Pull Request Analysis with GitHub Actions"
published: true
description: "Learn how to set up automated AI code reviews using GitHub Actions and OpenAI API to improve code quality and catch issues before they reach production."
tags: ai, github, automation, codereview, devops

cover_image: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ai-code-review-banner.png

Code reviews are essential for maintaining quality, but they're also time-consuming and sometimes inconsistent. What if you could have an AI assistant that never gets tired, catches common issues instantly, and provides detailed feedback on every pull request?

In this tutorial, we'll build an automated AI code review system using GitHub Actions and the OpenAI API. By the end, you'll have a workflow that analyzes pull requests, identifies potential issues, and comments directly on your PRs with actionable feedback.

Why AI Code Reviews Matter

Traditional code reviews face several challenges:

  • Time constraints: Reviewers often rush through large PRs
  • Inconsistency: Different reviewers focus on different aspects
  • Fatigue: Human reviewers miss obvious issues when tired
  • Knowledge gaps: Not every reviewer is an expert in every domain

AI code review doesn't replace human reviewers—it augments them by catching common issues, enforcing coding standards, and highlighting potential security vulnerabilities before human review.

Setting Up the GitHub Action Workflow

First, let's create the basic GitHub Actions workflow. Create .github/workflows/ai-code-review.yml in your repository:

name: AI Code Review

on:
  pull_request:
    types: [opened, synchronize]

jobs:
  ai-review:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write

    steps:
      - name: Checkout code
        uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Get changed files
        id: changed-files
        uses: tj-actions/changed-files@v40
        with:
          files: |
            **/*.js
            **/*.ts
            **/*.py
            **/*.java
            **/*.go
            **/*.rb

      - name: AI Code Review
        if: steps.changed-files.outputs.any_changed == 'true'
        uses: ./.github/actions/ai-review
        with:
          openai-api-key: ${{ secrets.OPENAI_API_KEY }}
          github-token: ${{ secrets.GITHUB_TOKEN }}
          changed-files: ${{ steps.changed-files.outputs.all_changed_files }}
Enter fullscreen mode Exit fullscreen mode

Creating the AI Review Action

Now let's build the custom action that handles the AI analysis. Create .github/actions/ai-review/action.yml:

name: 'AI Code Review'
description: 'Analyze code changes using AI'
inputs:
  openai-api-key:
    description: 'OpenAI API Key'
    required: true
  github-token:
    description: 'GitHub Token'
    required: true
  changed-files:
    description: 'List of changed files'
    required: true

runs:
  using: 'node20'
  main: 'index.js'
Enter fullscreen mode Exit fullscreen mode

Create the main logic in .github/actions/ai-review/index.js:

const core = require('@actions/core');
const github = require('@actions/github');
const OpenAI = require('openai');
const fs = require('fs');

async function run() {
  try {
    const openaiApiKey = core.getInput('openai-api-key');
    const githubToken = core.getInput('github-token');
    const changedFiles = core.getInput('changed-files').split(' ');

    const openai = new OpenAI({ apiKey: openaiApiKey });
    const octokit = github.getOctokit(githubToken);

    const context = github.context;
    const { owner, repo } = context.repo;
    const pullNumber = context.payload.pull_request.number;

    // Process each changed file
    for (const file of changedFiles) {
      if (file.trim()) {
        await reviewFile(openai, octokit, owner, repo, pullNumber, file.trim());
      }
    }
  } catch (error) {
    core.setFailed(error.message);
  }
}

async function reviewFile(openai, octokit, owner, repo, pullNumber, filePath) {
  try {
    // Read file content
    const fileContent = fs.readFileSync(filePath, 'utf8');

    // Get file extension for language-specific prompts
    const fileExtension = filePath.split('.').pop();

    // Generate AI review
    const review = await generateReview(openai, fileContent, filePath, fileExtension);

    if (review.issues.length > 0) {
      // Post review comments
      await postReviewComments(octokit, owner, repo, pullNumber, filePath, review);
    }
  } catch (error) {
    console.error(`Error reviewing file ${filePath}:`, error);
  }
}

run();
Enter fullscreen mode Exit fullscreen mode

Creating Language-Specific Prompts

Different programming languages have different best practices and common pitfalls. Let's create targeted prompts:

function getLanguagePrompt(fileExtension) {
  const prompts = {
    'js': `
      Analyze this JavaScript code for:
      - Potential security vulnerabilities (XSS, injection attacks)
      - Performance issues (unnecessary loops, memory leaks)
      - ES6+ best practices
      - Error handling
      - Code clarity and maintainability
    `,
    'py': `
      Analyze this Python code for:
      - PEP 8 compliance
      - Security issues (SQL injection, unsafe eval)
      - Performance bottlenecks
      - Proper exception handling
      - Type hints usage
    `,
    'ts': `
      Analyze this TypeScript code for:
      - Type safety issues
      - Proper interface usage
      - Generic type constraints
      - Async/await best practices
      - Import/export organization
    `,
    'java': `
      Analyze this Java code for:
      - Memory management issues
      - Thread safety concerns
      - Exception handling patterns
      - SOLID principles adherence
      - Resource cleanup (try-with-resources)
    `
  };

  return prompts[fileExtension] || prompts['js'];
}

async function generateReview(openai, fileContent, filePath, fileExtension) {
  const prompt = `
    ${getLanguagePrompt(fileExtension)}

    File: ${filePath}

    Code:
    \`\`\`
    ${fileContent}
    \`\`\`

    Provide feedback in JSON format:
    {
      "issues": [
        {
          "line": number,
          "severity": "high|medium|low",
          "type": "security|performance|style|logic",
          "message": "Clear description of the issue",
          "suggestion": "Specific code improvement suggestion"
        }
      ],
      "summary": "Overall assessment of the code quality"
    }

    Only report actual issues. If the code is good, return empty issues array.
  `;

  const response = await openai.chat.completions.create({
    model: 'gpt-4o-mini', // Cost-effective model
    messages: [{ role: 'user', content: prompt }],
    temperature: 0.1,
    max_tokens: 1000
  });

  try {
    return JSON.parse(response.choices[0].message.content);
  } catch (error) {
    console.error('Failed to parse AI response:', error);
    return { issues: [], summary: 'Failed to analyze code' };
  }
}
Enter fullscreen mode Exit fullscreen mode

Implementing Smart Commenting

Now let's add the logic to post meaningful comments on pull requests:

async function postReviewComments(octokit, owner, repo, pullNumber, filePath, review) {
  // Get PR diff to find line numbers
  const { data: pullRequest } = await octokit.rest.pulls.get({
    owner,
    repo,
    pull_number: pullNumber
  });

  // Post individual line comments for specific issues
  for (const issue of review.issues) {
    if (issue.line && issue.severity !== 'low') {
      await octokit.rest.pulls.createReviewComment({
        owner,
        repo,
        pull_number: pullNumber,
        commit_id: pullRequest.head.sha,
        path: filePath,
        line: issue.line,
        body: formatComment(issue)
      });
    }
  }

  // Post summary comment if there are multiple issues
  if (review.issues.length > 3) {
    await octokit.rest.issues.createComment({
      owner,
      repo,
      issue_number: pullNumber,
      body: formatSummaryComment(filePath, review)
    });
  }
}

function formatComment(issue) {
  const emoji = {
    'high': '🚨',
    'medium': '⚠️',
    'low': '💡'
  };

  return `
${emoji[issue.severity]} **${issue.type.toUpperCase()}**: ${issue.message}

**Suggestion:**
\`\`\`
${issue.suggestion}
\`\`\`

*Generated by AI Code Review*
  `;
}

function formatSummaryComment(filePath, review) {
  const highIssues = review.issues.filter(i => i.severity === 'high').length;
  const mediumIssues = review.issues.filter(i => i.severity === 'medium').length;

  return `
## 🤖 AI Code Review Summary for \`${filePath}\`

${review.summary}

**Issues Found:**
- 🚨 High Priority: ${highIssues}
- ⚠️ Medium Priority: ${mediumIssues}

*This is an automated review. Please verify suggestions before implementing.*
  `;
}
Enter fullscreen mode Exit fullscreen mode

Cost Optimization Strategies

AI API calls can get expensive quickly. Here are strategies to keep costs manageable:

1. Smart File Filtering 

function shouldReviewFile(filePath, fileSize) {
  // Skip large files (>50KB)
  if (fileSize > 50000) return false;

  // Skip generated files
  const skipPatterns = [
    /node_modules/,
    /\.min\./,
    /dist\//,
    /build\//,
    /coverage\//,
    /\.generated\./
  ];

  return !skipPatterns.some(pattern => pattern.test(filePath));
}
Enter fullscreen mode Exit fullscreen mode

2. Diff-Only Analysis 

async function getFileDiff(octokit, owner, repo, pullNumber, filePath) {
  const { data: comparison } = await octokit.rest.repos.compareCommits({
    owner,
    repo,
    base: pullRequest.base.sha,
    head: pullRequest.head.sha
  });

  const file = comparison.files.find(f => f.filename === filePath);
  return file ? file.patch : null;
}
Enter fullscreen mode Exit fullscreen mode

3. Rate Limiting and Batching 

const delay = ms => new Promise(resolve => setTimeout(resolve, ms));

async function reviewFilesWithRateLimit(files, batchSize = 3) {
  for (let i = 0; i < files.length; i += batchSize) {
    const batch = files.slice(i, i + batchSize);
    await Promise.all(batch.map(file => reviewFile(file)));

    // Wait between batches to avoid rate limits
    if (i + batchSize < files.length) {
      await delay(2000);
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

Setting Up Secrets

Add these secrets to your GitHub repository:

  1. Go to Settings → Secrets and variables → Actions
  2. Add OPENAI_API_KEY with your OpenAI API key
  3. GITHUB_TOKEN is automatically available

Testing and Fine-Tuning

Start with a small repository and monitor the results:

  1. Review accuracy: Are the suggestions helpful?
  2. False positives: Is the AI flagging correct code?
  3. Cost tracking: Monitor your OpenAI usage
  4. Performance: How long do reviews take?

Adjust prompts based on your team's coding standards and the types of issues you want to catch.

Conclusion

AI-powered code reviews can significantly improve your development workflow by catching issues early and maintaining consistent standards. The system we've built provides:

  • Automated analysis of every pull request
  • Language-specific feedback
  • Cost-effective operation through smart filtering
  • Actionable comments that help developers improve

Remember, AI code review is a tool to enhance human review, not replace it. Use it to catch the obvious issues so your human reviewers can focus on architecture, business logic, and complex design decisions.

Start small, iterate based on feedback, and gradually expand the system as your team becomes comfortable with AI-assisted reviews. The time investment upfront will pay dividends in improved code quality and faster review cycles.

Have you implemented AI code reviews in your workflow? Share your experiences and optimizations in the comments below!

Tools mentioned:

Top comments (1)

Collapse
 
with_geun profile image
Eddie •

This is a great walkthrough — especially the idea of using AI to reduce review fatigue rather than replace humans.

I’ve noticed a similar pattern on the analysis side: AI can generate feedback quickly, but the bigger challenge is keeping the reasoning structured and traceable over time.

Curious if you’ve thought about adding a “review memory” layer — something that tracks recurring issues or patterns across PRs?

Feels like that could turn AI review from reactive feedback into a learning loop.