Every time I submit a Pull Request (PR) to an open-source project, I watch the CI pipeline kick off. More and more often, an AI assistant like CodeRabbit, Copilot, or Claude pops up a few minutes later to review my changes, categorize bugs, or suggest updates.
It got me wondering: How hard is it to build one of these from scratch?
As it turns out, you don't need a heavy, expensive third-party platform. You can build a fully customized, line-by-line AI code reviewer directly inside GitHub Actions using a short Python script and the Gemini API.
Here is exactly how it works, what the code looks like, and a breakdown of how each piece functions.
What We Are Building
The goal is to create a lightweight, dependency-free automation loop that acts like a senior engineer on your team.
Trigger on PR Events: Step 1.
The workflow triggers whenever a PR is opened, updated, or reopened.Extract the Git Diff: Step 2.
We isolate the exact line changes and trim them to fit comfortably within LLM context constraints.Send to Gemini with Structured Output: Step 3.
An inline Python script passes the diff to Gemini, using a strict JSON schema to guarantee we get back structured comments tied to specific files and lines.Post Inline Comments: Step 4.
A GitHub script reads the JSON payload and attaches native code-review comments directly onto the modified lines in the PR.
The Complete Workflow Code
Create a file named .github/workflows/ai-review.yml in your repository and paste the following configuration:
name: AI PR Review
on:
pull_request:
types: [opened, synchronize, reopened]
permissions:
contents: read
pull-requests: write
jobs:
ai-review:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Get PR diff
env:
BASE_SHA: ${{ github.event.pull_request.base.sha }}
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
run: |
git diff "$BASE_SHA...$HEAD_SHA" > diff.txt
head -c 800000 diff.txt > diff_trimmed.txt
- name: Ask Gemini for a review
env:
GEMINI_API_KEY: ${{ secrets.GEMINI_API_KEY }}
run: |
python3 - <<'PYEOF'
import json, os, re, sys, time
import urllib.error
import urllib.request
MAX_DIFF_BYTES = 800_000
def write_outputs(status, summary, comments):
open("status.txt", "w").write(status)
open("review.md", "w").write(summary)
open("comments.json", "w").write(json.dumps(comments))
original_bytes = os.path.getsize("diff.txt")
truncated = original_bytes > MAX_DIFF_BYTES
with open("diff_trimmed.txt", errors="replace") as f:
diff = f.read()
if not diff.strip():
write_outputs("clean", "No changes to review.", [])
sys.exit(0)
def parse_commentable_lines(diff_text):
valid = {}
current_file, new_line = None, None
for line in diff_text.splitlines():
if line.startswith("+++ "):
raw = line[4:].strip()
current_file = None if raw == "/dev/null" else raw[2:] # strip "b/"
if current_file:
valid.setdefault(current_file, set())
new_line = None
continue
m = re.match(r"^@@ -\d+(?:,\d+)? \+(\d+)(?:,\d+)? @@", line)
if m:
new_line = int(m.group(1))
continue
if current_file is None or new_line is None:
continue
if line.startswith("+"):
valid[current_file].add(new_line) # only new/added lines are offered
new_line += 1
elif line.startswith(" "):
new_line += 1
return valid
commentable = parse_commentable_lines(diff)
prompt_text = """You are a senior software engineer reviewing a pull request diff.
Focus strictly on:
1. Architecture & Logic: bugs, security issues, missing error handling, race conditions, lifecycle conflicts.
2. UI & Responsiveness
3. UX & Information Density.
4. Maintainability: structural and style problems.
Only flag lines that are actually part of the diff (added or modified lines). Reference the NEW file's line numbers.
Zero emojis. Be brief, direct, pragmatic.
If there is nothing significant, return status "clean" with an empty comments array.
""" + diff
response_schema = {
"type": "OBJECT",
"properties": {
"status": {"type": "STRING", "enum": ["clean", "issues_found"]},
"summary": {"type": "STRING"},
"comments": {
"type": "ARRAY",
"items": {
"type": "OBJECT",
"properties": {
"file": {"type": "STRING"},
"line": {"type": "INTEGER"},
"severity": {"type": "STRING", "enum": ["high", "medium", "low"]},
"comment": {"type": "STRING"},
},
"required": ["file", "line", "comment"],
},
},
},
"required": ["status", "summary", "comments"],
}
req = urllib.request.Request(
"https://generativelanguage.googleapis.com/v1beta/models/gemini-3.5-flash:generateContent",
data=json.dumps({
"contents": [{"parts": [{"text": prompt_text}]}],
"generationConfig": {
"responseMimeType": "application/json",
"responseSchema": response_schema,
},
}).encode(),
headers={
"Content-Type": "application/json",
"x-goog-api-key": os.environ["GEMINI_API_KEY"],
},
)
def call_with_retries(request, max_attempts=3, base_delay=2):
last_err = None
for attempt in range(1, max_attempts + 1):
try:
with urllib.request.urlopen(request, timeout=180) as resp:
return json.loads(resp.read())
except urllib.error.HTTPError as e:
last_err = e
if e.code in (429, 500, 502, 503, 504) and attempt < max_attempts:
time.sleep(base_delay * attempt)
continue
raise
except urllib.error.URLError as e:
last_err = e
if attempt < max_attempts:
time.sleep(base_delay * attempt)
continue
raise
raise last_err
try:
payload = call_with_retries(req)
candidates = payload.get("candidates") or []
if not candidates:
reason = payload.get("promptFeedback", {}).get("blockReason", "unknown")
raise ValueError(f"no candidates returned (blockReason={reason})")
if "content" not in candidates[0] or "parts" not in candidates[0]["content"]:
raise ValueError("response missing content/parts")
raw_text = candidates[0]["content"]["parts"][0].get("text", "").strip()
raw_text = re.sub(r"^```
{% endraw %}
json\s*|\s*
{% raw %}
```$", "", raw_text) # in case it wraps anyway
parsed = json.loads(raw_text)
status = parsed.get("status", "issues_found")
summary = parsed.get("summary", "").strip() or "Review complete."
raw_comments = parsed.get("comments", [])
safe_comments = []
for c in raw_comments:
f, ln = c.get("file"), c.get("line")
if f in commentable and ln in commentable[f]:
severity = c.get("severity", "medium").upper()
safe_comments.append({
"path": f,
"line": ln,
"side": "RIGHT",
"body": f"**[{severity}]** {c.get('comment', '').strip()}",
})
if truncated:
summary = (
f"Note: diff exceeded {MAX_DIFF_BYTES // 1000}KB and was truncated — "
f"this review may not cover the full PR.\n\n" + summary
)
write_outputs(status, summary, safe_comments)
except urllib.error.HTTPError as e:
detail = e.read().decode(errors="replace")[:500]
print(f"::warning::Gemini API HTTP {e.code}: {detail}", file=sys.stderr)
write_outputs("skipped", f"AI review skipped: Gemini API returned HTTP {e.code}.", [])
except urllib.error.URLError as e:
print(f"::warning::Gemini API network error: {e}", file=sys.stderr)
write_outputs("skipped", f"AI review skipped: network error contacting Gemini API ({e.reason}).", [])
except (KeyError, IndexError, ValueError, json.JSONDecodeError) as e:
print(f"::warning::Gemini API unexpected response: {e}", file=sys.stderr)
write_outputs("skipped", f"AI review skipped: unexpected response from Gemini API ({e}).", [])
PYEOF
- name: Submit PR Review
uses: actions/github-script@v7
with:
script: |
const fs = require('fs');
const status = fs.readFileSync('status.txt', 'utf8').trim();
const summary = fs.readFileSync('review.md', 'utf8');
let comments = [];
try {
comments = JSON.parse(fs.readFileSync('comments.json', 'utf8'));
} catch (e) {
comments = [];
}
const reviewEvent = (status === 'issues_found' && comments.length > 0)
? 'REQUEST_CHANGES'
: 'COMMENT';
const base = {
owner: context.repo.owner,
repo: context.repo.repo,
pull_number: context.issue.number,
event: reviewEvent,
};
try {
await github.rest.pulls.createReview({ ...base, body: summary, comments });
} catch (e) {
core.warning(`Inline comments rejected (${e.message}); falling back to summary only.`);
await github.rest.pulls.createReview({
...base,
body: summary + '\n\n_Some inline comments could not be posted._',
});
}
Piece-by-Piece Breakdown
Let’s look at exactly how each component functions to pull this off seamlessly.
1. Permissions and Setup
At the top of the workflow, we define the event triggers and permissions:
-
pull_requesttypes ensure this runs when code changes or a new PR goes live. - We explicitly give the job
pull-requests: writepermissions. Without this, the runner won't be allowed to leave comments on the PR thread. -
fetch-depth: 0ensures the full git history is pulled down so we can accurately compare branches.
2. Extracting and Trimming the Diff
git diff "$BASE_SHA...$HEAD_SHA" > diff.txt
head -c 800000 diff.txt > diff_trimmed.txt
Instead of relying on heavy ecosystem dependencies, we pull standard git diff text. Huge pull requests can quickly exceed large language model limits or API limits, so head -c 800000 safely caps the payload size at around 800KB to prevent structural drops.
3. The Python Script (Validating Lines and Calling Gemini)
The core logic runs in a dependency-free Python step using standard libraries (urllib). It handles three main problems:
-
Line Target Matching (
parse_commentable_lines): LLMs sometimes hallucinate and try to comment on lines that weren't even changed. This function parses the unified diff headers (@@ -line,v +line,v @@) to build a map of lines that actually exist in the new incoming code version. -
Injecting Context Rules: The
prompt_textprovides structural guardrails. You can adapt this to fit your specific stack's technical requirements (e.g., advising on UI lifecycles, CSS frameworks, or structural preferences). -
Enforcing Structured JSON: We define a
response_schemadirectly inside the API request options. This forces theAI modelto return raw data exactly matching an object containing a status string, markdown summary, and an array of inline comments.
The script ends by verifying that the AI's feedback strictly lines up with changed lines and writes the output payload into local text files.
4. Submitting the Feedback to GitHub
const reviewEvent = (status === 'issues_found' && comments.length > 0) ? 'REQUEST_CHANGES' : 'COMMENT';
...
await github.rest.pulls.createReview({ ...base, body: summary, comments });
The final step uses the official actions/github-script action to process those text files. If the AI flags real logic errors and populates valid comments, the script marks the PR review state as REQUEST_CHANGES. If everything looks clear, it defaults to a standard tracking COMMENT.
If a specific line index mismatch occurs, a try/catch block acts as a fallback to post the core high-level summary review anyway so you don't lose the AI insight.
Make it Your Own
The beauty of this setup is how easy it is to alter. By adjusting the prompt_text section in the Python block, you can easily instruct the LLM to write in a specific language, focus tightly on security vulnerabilities, or ignore everything except your performance benchmarks. Drop the GEMINI_API_KEY into your GitHub repository secrets, and you have an automated, self-hosted code guardrail running completely under your control!
Top comments (0)