cuongnp

Posted on Sep 23

How to Build an AI-Powered Code Review Bot with GitHub Action, Slack, and LLMs

#devops #ai #productivity #openai

Introduction

Recently I usually see many teams struggle with code reviews due to time constraints and inconsistent feedback. So, I decided to create an AI-powered code review bot that automates the process, providing quick and consistent feedback on pull requests. In this post, I'll walk you through building this bot using GitHub Actions, Slack, and LLMs.

What You’ll Learn

How to trigger AI code reviews on every pull request
How to send review results to Slack with interactive buttons
How to connect GitHub, Slack, and your backend for seamless automation
How to use LLMs (like OpenAI) for code intelligence

How It Works

Figure: System Architecture Overview

Tech Stack

GitHub Actions for automation
NextJs for example codebase
Node.js/Express backend for Slack interactivity
Slack App for notifications and actions (You should create a trial workspace, and 2 accounts for testing)
Cloudflare Tunnel a free service that creates a temporary, secure, and public-facing URL for your localhost-based services
OpenAI (or any LLM) for code review intelligence (OpenAI, Gemini, etc)

Prerequisites

GitHub account
Slack workspace
Node.js (v18+)
GitHub personal access token (repo, pull_requests scopes)
Slack bot token (chat:write, commands, interactivity)
OpenAI API key (or other LLM provider)
(Optional) ngrok or Cloudflare Tunnel for local testing

Project folder

Here’s the project structure:

code-review-llm/
├── .github/
│   └── workflows/
│       └── code-review.yml         # GitHub Actions workflow for AI code review
├── src/
│   └── services/
│       ├── reviewRunner.js         # Main script to run LLM code review
│       ├── llmService.js           # LLM API integration
│       ├── reviewService.js        # Review logic
│       └── ...                     # Other utilities/services
├── public/
│   └── vite.svg                    # Static assets
├── App.jsx, main.jsx, App.css      # Frontend entry and styles
├── package.json, README.md         # Project metadata and docs
└── slack-backend/
    ├── index.js                    # Express backend for Slack interactivity
    ├── package.json                # Backend dependencies
    ├── .env                        # Backend environment variables (not committed)
    └── README.md                   # Backend setup instructions

Step-by-Step Guide

1. Set Up the GitHub Action

Triggers on pull requests, runs AI review, posts to Slack.
Add OPENAI_API_KEY and SLACK_BOT_TOKEN as repository secrets.

Figure: GitHub Action Setting

name: AI Code Review
on:
  pull_request:
    types: [opened, synchronize, reopened]
jobs:
  code-review:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write
      actions: read
      issues: read
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
        with:
          fetch-depth: 0
          token: ${{ secrets.GITHUB_TOKEN }}
      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '18'
          cache: 'npm'
      - name: Install dependencies
        run: npm ci
      - name: Get PR diff
        id: diff
        run: |
          git fetch origin ${{ github.event.pull_request.base.ref }}
          DIFF=$(git diff origin/${{ github.event.pull_request.base.ref }}...HEAD --unified=3)
          echo "diff<<EOF" >> $GITHUB_OUTPUT
          echo "$DIFF" >> $GITHUB_OUTPUT
          echo "EOF" >> $GITHUB_OUTPUT
      - name: Run AI Code Review
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          PR_NUMBER: ${{ github.event.pull_request.number }}
          REPO_OWNER: ${{ github.repository_owner }}
          REPO_NAME: ${{ github.event.repository.name }}
          PR_DIFF: ${{ steps.diff.outputs.diff }}
        run: node src/services/reviewRunner.js
      - name: Send review summary log to Slack
        env:
          SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}
        run: |
          SUMMARY=$(cat review-summary.txt 2>/dev/null || echo "No summary file found.")
          curl -X POST https://slack.com/api/chat.postMessage \
            -H "Authorization: Bearer $SLACK_BOT_TOKEN" \
            -H "Content-type: application/json" \
            --data "{\"channel\": \"test-bot\", \"text\": \"$SUMMARY\"}"
      - name: Send LLM review to Slack with buttons
        env:
          SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}
        run: |
          curl -X POST https://slack.com/api/chat.postMessage \
            -H "Authorization: Bearer $SLACK_BOT_TOKEN" \
            -H "Content-type: application/json" \
            --data '{
              "channel": "test-bot",
              "text": "Code review for PR #${{ github.event.pull_request.number }}: ${{ github.event.pull_request.title }}",
              "blocks": [
                {
                  "type": "section",
                  "text": {
                    "type": "mrkdwn",
                    "text": "*Code Review Request*\n*PR #${{ github.event.pull_request.number }}:* ${{ github.event.pull_request.title }}\n\n*LLM Review:*\nReview output here\n\nPlease review and take action below:"
                  }
                },
                {
                  "type": "actions",
                  "elements": [
                    {
                      "type": "button",
                      "text": { "type": "plain_text", "text": "✅ Approve" },
                      "value": "approve_${{ github.event.pull_request.number }}",
                      "action_id": "approve_action",
                      "style": "primary"
                    },
                    {
                      "type": "button",
                      "text": { "type": "plain_text", "text": "💬 Request Changes" },
                      "value": "comment_${{ github.event.pull_request.number }}",
                      "action_id": "comment_action",
                      "style": "danger"
                    }
                  ]
                }
              ]
            }'

2. Backend for Slack Interactivity

Handles button clicks and updates GitHub PRs.
Use a dedicated GitHub bot account for API token.

app.post('/slack/interactive', async (req, res) => {
  const payload = JSON.parse(req.body.payload);
  const { actions, user, response_url } = payload;
  const action = actions[0];
  const prNumber = action.value.replace(/^(approve|comment)_/, '');
  if (action.action_id === 'approve_action') {
    // Approve PR via GitHub API
});

Use Cloudflare Tunnel or ngrok for local testing.

npm install -g cloudflared
cloudflared tunnel --url http://localhost:3000

Configure Slack app with the tunnel URL for interactivity.

Figure: Slack Request URL

4. Create a new branch and push a commit

Open a pull request to see the bot in action and we got the result like below:

Figure: AI code review summary

We will see the bot posts a summary to Slack with buttons to approve or request changes.

Figure: Slack bot message

Clicking "Approve" or "Request Changes" will update the PR accordingly.

Figure: Approve or Request Changes

Next Steps

Extend Slack actions (e.g., custom comments, assign reviewers)
Integrate with other chat tools (Discord, Teams)

Repo

GitHub: code-review-llm

Conclusion

You’ve built an AI-powered code review bot that saves time and boosts productivity. Try customizing it for your team and share your results!

This post is originally published on TechCodex - If you found this guide helpful, please share it with your network and leave a comment below! Happy coding! 🚀

DEV Community