Build your own AI code review agent in CI

#ai #automation #githubactions #cicd

If you do PR reviews all day, you already know the pain:

reviewers miss stuff when they’re tired
the same comments repeat forever (naming, tests, edge cases)
“quick PR” turns into a 45-minute context rebuild

Now imagine this instead:

Every pull request gets an instant, structured code review (correctness, security, performance, tests) posted automatically by CI using your choice of model (OpenAI / Anthropic / OpenRouter / local Ollama) — without paying yet another “AI code review” subscription.

That’s the workflow.

This post shows a simple way to build a “coding agent” that reviews PR diffs and produces an actionable review. I’ll use Jazz as the runner because it’s designed for workflows + automation.

The 80/20 in one minute

You want a pipeline that does:

1) CI checks out your repo
2) an agent reads the PR diff
3) it outputs a review in Markdown
4) CI posts that review as a PR comment (or logs it / saves it as an artifact)

The most important part is not the model.

It’s the review rubric (the prompt/workflow) that forces useful structure:

separate high-risk issues from nitpicks
demand concrete fixes and test suggestions
require “what I looked at” and “what I’m unsure about”

Why DIY can be cheaper than subscriptions

Chat subscriptions are awesome for interactive use, but CI code review is a different pricing pattern:

it only runs when PRs happen
you can use pay-as-you-go APIs
you can route cheaper/faster models for small diffs
or run local models (Ollama) where the marginal cost is near zero

With a DIY pipeline you control the knobs model choice, max tokens, when it runs, and what counts as “worth reviewing”.

What you’ll build

By the end, you’ll have:

a GitHub Action that runs on every PR
a code review workflow that outputs a structured Markdown review

Step 1: run the agent in GitHub Actions

Create this file in your repo:

Filename: .github/workflows/ai-code-review.yml

name: AI Code Review
on:
  pull_request:

jobs:
  review:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write

    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Install Jazz
        run: npm install -g jazz-ai

      - name: Run code review workflow
        run: jazz --output raw workflow run code-review --auto-approve
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

Notes:

--output raw is nice in CI (easy to capture/redirect).
--auto-approve makes it fully unattended.
permissions are intentionally minimal.

If you don’t want OpenAI, swap the environment variables for your provider (Anthropic / OpenRouter / etc.).

Step 2: define what “good code review” means (the rubric)

This is where most “AI code review” attempts fail: they generate vibes, not review.

A good rubric forces:

severity (what would actually break prod)
confidence (what’s a guess)
next actions (exact fixes / tests)

Create a workflow file (this is the prompt your agent will run):

Filename: workflows/code-review/WORKFLOW.md

Here’s a template you can start with:

---
name: code-review
description: Review PR diff and produce a structured report
autoApprove: read-only
---

Review the current PR diff.

Output GitHub-flavored Markdown with:

1) Summary (2–4 bullets)
2) High-risk issues (correctness + security)
3) Performance / complexity concerns
4) API / UX footguns
5) Test gaps + concrete test suggestions
6) Nitpicks (style/readability)

Rules:
- Be specific: reference files/functions.
- Prefer minimal diffs / smallest safe fix.
- If you’re unsure, say so and propose how to verify.
- No generic advice ("add tests") — propose exact test cases.

Step 3: post the review as a PR comment

The simplest reliable approach:

1) generate a markdown file
2) post it using gh

In CI, you can write the review output to:

Filename: review.md

- name: Generate review markdown
  run: jazz --output raw workflow run code-review --auto-approve > review.md
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

- name: Comment on PR
  run: gh pr comment "$PR_NUMBER" --body-file review.md
  env:
    GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
    PR_NUMBER: ${{ github.event.pull_request.number }}

Inline annotations are possible later, but they’re not required to get value immediately.

Safety: keep it read-only in CI

If you only take one thing from this post:

Don’t let your CI agent mutate the repo.

Keep auto-approve at read-only for review jobs.

Even if your tool can run shell commands or commit changes, you’ll get most of the value without giving it that power.

Practical tips to keep reviews useful (not noisy)

Force it to rank issues (High/Medium/Low). If everything is “important”, nothing is.
Add a “false positive budget”: if it’s noisy for a week, devs will ignore it forever.
Route by diff size: cheap model for small PRs, stronger model for large refactors.
Require it to list: files reviewed, assumptions, and what it didn’t check.