Create a Custom AI PR Reviewer from scratch with GitHub Actions

Thierry RAKOTOMANANA (Thierry) — Thu, 16 Jul 2026 10:37:56 +0000

Every time I submit a Pull Request (PR) to an open-source project, I watch the CI pipeline kick off. More and more often, an AI assistant like CodeRabbit, Copilot, or Claude pops up a few minutes later to review my changes, categorize bugs, or suggest updates.

It got me wondering: How hard is it to build one of these from scratch?

As it turns out, you don't need a heavy, expensive third-party platform. You can build a fully customized, line-by-line AI code reviewer directly inside GitHub Actions using a short Python script and the Gemini API.

Here is exactly how it works, what the code looks like, and a breakdown of how each piece functions.

What We Are Building

The goal is to create a lightweight, dependency-free automation loop that acts like a senior engineer on your team.

Trigger on PR Events: Step 1.
The workflow triggers whenever a PR is opened, updated, or reopened.
Extract the Git Diff: Step 2.
We isolate the exact line changes and trim them to fit comfortably within LLM context constraints.
Send to Gemini with Structured Output: Step 3.
An inline Python script passes the diff to Gemini, using a strict JSON schema to guarantee we get back structured comments tied to specific files and lines.
Post Inline Comments: Step 4.
A GitHub script reads the JSON payload and attaches native code-review comments directly onto the modified lines in the PR.

The Complete Workflow Code

Create a file named .github/workflows/ai-review.yml in your repository and paste the following configuration:

name: AI PR Review

on:
  pull_request:
    types: [opened, synchronize, reopened]

permissions:
  contents: read
  pull-requests: write

jobs:
  ai-review:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Get PR diff
        env:
          BASE_SHA: ${{ github.event.pull_request.base.sha }}
          HEAD_SHA: ${{ github.event.pull_request.head.sha }}
        run: |
          git diff "$BASE_SHA...$HEAD_SHA" > diff.txt
          head -c 800000 diff.txt > diff_trimmed.txt

      - name: Ask Gemini for a review
        env:
          GEMINI_API_KEY: ${{ secrets.GEMINI_API_KEY }}
        run: |
          python3 - <<'PYEOF'
          import json, os, re, sys, time
          import urllib.error
          import urllib.request

          MAX_DIFF_BYTES = 800_000

          def write_outputs(status, summary, comments):
              open("status.txt", "w").write(status)
              open("review.md", "w").write(summary)
              open("comments.json", "w").write(json.dumps(comments))

          original_bytes = os.path.getsize("diff.txt")
          truncated = original_bytes > MAX_DIFF_BYTES

          with open("diff_trimmed.txt", errors="replace") as f:
              diff = f.read()

          if not diff.strip():
              write_outputs("clean", "No changes to review.", [])
              sys.exit(0)

          def parse_commentable_lines(diff_text):
              valid = {}
              current_file, new_line = None, None
              for line in diff_text.splitlines():
                  if line.startswith("+++ "):
                      raw = line[4:].strip()
                      current_file = None if raw == "/dev/null" else raw[2:]  # strip "b/"
                      if current_file:
                          valid.setdefault(current_file, set())
                      new_line = None
                      continue
                  m = re.match(r"^@@ -\d+(?:,\d+)? \+(\d+)(?:,\d+)? @@", line)
                  if m:
                      new_line = int(m.group(1))
                      continue
                  if current_file is None or new_line is None:
                      continue
                  if line.startswith("+"):
                      valid[current_file].add(new_line)  # only new/added lines are offered
                      new_line += 1
                  elif line.startswith(" "):
                      new_line += 1
              return valid

          commentable = parse_commentable_lines(diff)

          prompt_text = """You are a senior software engineer reviewing a pull request diff.

          Focus strictly on:
          1. Architecture & Logic: bugs, security issues, missing error handling, race conditions, lifecycle conflicts.
          2. UI & Responsiveness
          3. UX & Information Density.
          4. Maintainability: structural and style problems.

          Only flag lines that are actually part of the diff (added or modified lines). Reference the NEW file's line numbers.
          Zero emojis. Be brief, direct, pragmatic.
          If there is nothing significant, return status "clean" with an empty comments array.
          """ + diff

          response_schema = {
              "type": "OBJECT",
              "properties": {
                  "status": {"type": "STRING", "enum": ["clean", "issues_found"]},
                  "summary": {"type": "STRING"},
                  "comments": {
                      "type": "ARRAY",
                      "items": {
                          "type": "OBJECT",
                          "properties": {
                              "file": {"type": "STRING"},
                              "line": {"type": "INTEGER"},
                              "severity": {"type": "STRING", "enum": ["high", "medium", "low"]},
                              "comment": {"type": "STRING"},
                          },
                          "required": ["file", "line", "comment"],
                      },
                  },
              },
              "required": ["status", "summary", "comments"],
          }

          req = urllib.request.Request(
              "https://generativelanguage.googleapis.com/v1beta/models/gemini-3.5-flash:generateContent",
              data=json.dumps({
                  "contents": [{"parts": [{"text": prompt_text}]}],
                  "generationConfig": {
                      "responseMimeType": "application/json",
                      "responseSchema": response_schema,
                  },
              }).encode(),
              headers={
                  "Content-Type": "application/json",
                  "x-goog-api-key": os.environ["GEMINI_API_KEY"],
              },
          )

          def call_with_retries(request, max_attempts=3, base_delay=2):
              last_err = None
              for attempt in range(1, max_attempts + 1):
                  try:
                      with urllib.request.urlopen(request, timeout=180) as resp:
                          return json.loads(resp.read())
                  except urllib.error.HTTPError as e:
                      last_err = e
                      if e.code in (429, 500, 502, 503, 504) and attempt < max_attempts:
                          time.sleep(base_delay * attempt)
                          continue
                      raise
                  except urllib.error.URLError as e:
                      last_err = e
                      if attempt < max_attempts:
                          time.sleep(base_delay * attempt)
                          continue
                      raise
              raise last_err

          try:
              payload = call_with_retries(req)

              candidates = payload.get("candidates") or []
              if not candidates:
                  reason = payload.get("promptFeedback", {}).get("blockReason", "unknown")
                  raise ValueError(f"no candidates returned (blockReason={reason})")
              if "content" not in candidates[0] or "parts" not in candidates[0]["content"]:
                  raise ValueError("response missing content/parts")

              raw_text = candidates[0]["content"]["parts"][0].get("text", "").strip()
              raw_text = re.sub(r"^```
{% endraw %}
json\s*|\s*
{% raw %}
```$", "", raw_text)  # in case it wraps anyway
              parsed = json.loads(raw_text)

              status = parsed.get("status", "issues_found")
              summary = parsed.get("summary", "").strip() or "Review complete."
              raw_comments = parsed.get("comments", [])

              safe_comments = []
              for c in raw_comments:
                  f, ln = c.get("file"), c.get("line")
                  if f in commentable and ln in commentable[f]:
                      severity = c.get("severity", "medium").upper()
                      safe_comments.append({
                          "path": f,
                          "line": ln,
                          "side": "RIGHT",
                          "body": f"**[{severity}]** {c.get('comment', '').strip()}",
                      })

              if truncated:
                  summary = (
                      f"Note: diff exceeded {MAX_DIFF_BYTES // 1000}KB and was truncated — "
                      f"this review may not cover the full PR.\n\n" + summary
                  )

              write_outputs(status, summary, safe_comments)

          except urllib.error.HTTPError as e:
              detail = e.read().decode(errors="replace")[:500]
              print(f"::warning::Gemini API HTTP {e.code}: {detail}", file=sys.stderr)
              write_outputs("skipped", f"AI review skipped: Gemini API returned HTTP {e.code}.", [])

          except urllib.error.URLError as e:
              print(f"::warning::Gemini API network error: {e}", file=sys.stderr)
              write_outputs("skipped", f"AI review skipped: network error contacting Gemini API ({e.reason}).", [])

          except (KeyError, IndexError, ValueError, json.JSONDecodeError) as e:
              print(f"::warning::Gemini API unexpected response: {e}", file=sys.stderr)
              write_outputs("skipped", f"AI review skipped: unexpected response from Gemini API ({e}).", [])
          PYEOF

      - name: Submit PR Review
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');

            const status = fs.readFileSync('status.txt', 'utf8').trim();
            const summary = fs.readFileSync('review.md', 'utf8');
            let comments = [];
            try {
              comments = JSON.parse(fs.readFileSync('comments.json', 'utf8'));
            } catch (e) {
              comments = [];
            }

            const reviewEvent = (status === 'issues_found' && comments.length > 0)
              ? 'REQUEST_CHANGES'
              : 'COMMENT';

            const base = {
              owner: context.repo.owner,
              repo: context.repo.repo,
              pull_number: context.issue.number,
              event: reviewEvent,
            };

            try {
              await github.rest.pulls.createReview({ ...base, body: summary, comments });
            } catch (e) {
              core.warning(`Inline comments rejected (${e.message}); falling back to summary only.`);
              await github.rest.pulls.createReview({
                ...base,
                body: summary + '\n\n_Some inline comments could not be posted._',
              });
            }

Piece-by-Piece Breakdown

Let’s look at exactly how each component functions to pull this off seamlessly.

1. Permissions and Setup

At the top of the workflow, we define the event triggers and permissions:

pull_request types ensure this runs when code changes or a new PR goes live.
We explicitly give the job pull-requests: write permissions. Without this, the runner won't be allowed to leave comments on the PR thread.
fetch-depth: 0 ensures the full git history is pulled down so we can accurately compare branches.

2. Extracting and Trimming the Diff

git diff "$BASE_SHA...$HEAD_SHA" > diff.txt
head -c 800000 diff.txt > diff_trimmed.txt

Instead of relying on heavy ecosystem dependencies, we pull standard git diff text. Huge pull requests can quickly exceed large language model limits or API limits, so head -c 800000 safely caps the payload size at around 800KB to prevent structural drops.

3. The Python Script (Validating Lines and Calling Gemini)

The core logic runs in a dependency-free Python step using standard libraries (urllib). It handles three main problems:

Line Target Matching (parse_commentable_lines): LLMs sometimes hallucinate and try to comment on lines that weren't even changed. This function parses the unified diff headers (@@ -line,v +line,v @@) to build a map of lines that actually exist in the new incoming code version.
Injecting Context Rules: The prompt_text provides structural guardrails. You can adapt this to fit your specific stack's technical requirements (e.g., advising on UI lifecycles, CSS frameworks, or structural preferences).
Enforcing Structured JSON: We define a response_schema directly inside the API request options. This forces the AI model to return raw data exactly matching an object containing a status string, markdown summary, and an array of inline comments.

The script ends by verifying that the AI's feedback strictly lines up with changed lines and writes the output payload into local text files.

4. Submitting the Feedback to GitHub

const reviewEvent = (status === 'issues_found' && comments.length > 0) ? 'REQUEST_CHANGES' : 'COMMENT';
...
await github.rest.pulls.createReview({ ...base, body: summary, comments });

The final step uses the official actions/github-script action to process those text files. If the AI flags real logic errors and populates valid comments, the script marks the PR review state as REQUEST_CHANGES. If everything looks clear, it defaults to a standard tracking COMMENT.

If a specific line index mismatch occurs, a try/catch block acts as a fallback to post the core high-level summary review anyway so you don't lose the AI insight.

Make it Your Own

The beauty of this setup is how easy it is to alter. By adjusting the prompt_text section in the Python block, you can easily instruct the LLM to write in a specific language, focus tightly on security vulnerabilities, or ignore everything except your performance benchmarks. Drop the GEMINI_API_KEY into your GitHub repository secrets, and you have an automated, self-hosted code guardrail running completely under your control!

Github link

DEV Community: Thierry RAKOTOMANANA (Thierry)