Originally published on tamiz.pro.
Trust But Verify: Building a 30-Second AI Code Review Gate for Your CI Pipeline
AI code review tools are fast, but "fast" and "correct" are different requirements. A CI gate must fail closed on infrastructure errors, ignore hallucinated findings, and explain every decision in a way your team can audit. This tutorial builds a production-grade review gate that combines a deterministic security scanner with an LLM analysis pass, then verifies the model's output against the actual diff before allowing a pull request to merge.
Table of Contents
- 1. What the Gate Actually Guarantees
- 2. Architecture Overview
- 3. Prerequisites
- 4. Project Structure
- 5. Step 1: Create the CI Project
- 6. Step 2: Collect and Normalize the Diff
- 7. Step 3: Run Deterministic Security Checks
- 8. Step 4: Add the AI Review Pass
- 9. Step 5: Verify and Enforce Findings
- 10. Step 6: Wire It into GitHub Actions
- 11. Step 7: Handle Timeouts and Partial Failures
- 12. Production Hardening
- 13. Frequently Asked Questions
1. What the Gate Actually Guarantees
The goal is not to let an LLM approve every pull request. The goal is to create a fast, layered signal that catches high-confidence problems without blocking developers on noise.
A trustworthy gate has four properties:
- Deterministic first: Known patterns like hardcoded secrets or dangerous command injection are caught by a scanner, not by a model.
- AI second: The LLM reviews the diff for logic flaws, missing error handling, and architectural regressions that rules cannot express.
- Verified output: Every AI finding must point to a real line in the changed diff. Findings that reference unchanged code or nonexistent line numbers are discarded.
- Bounded runtime: The entire job has a hard 30-second wall-clock budget. If the model or scanner fails, the gate fails closed with a clear error rather than silently passing.
This design treats the LLM as an untrusted reviewer. The model can suggest a problem, but only a deterministic verifier decides whether that suggestion becomes a CI failure.
Important: An AI gate is an additional signal, not a replacement for tests, type checking, dependency scanning, or human review. Start in report-only mode and tune thresholds before enabling branch protection.
2. Architecture Overview
The pipeline runs in five stages:
Pull Request
|
v
[1] Fetch PR diff (base SHA + head SHA)
|
v
[2] Normalize diff into structured hunks
|
v
[3] Deterministic scanner --> findings[]
|
v
[4] AI reviewer (bounded context) --> proposed findings[]
|
v
[5] Verifier
| |
| +--> valid + high severity --> BLOCK
| +--> valid + low severity --> COMMENT
| +--> invalid / hallucinated --> DISCARD
v
[6] Post PR comment + set commit status
The critical architectural decision is the separation between finding generation and finding enforcement. The AI never controls the exit code directly. A Python verifier owns the trust boundary.
3. Prerequisites
You will need:\n
- Python 3.11 or newer
- A GitHub repository with pull requests enabled
- An Anthropic API key stored as a GitHub Actions secret
-
gitavailable in the runner environment
The implementation uses the Anthropic Messages API directly. If you prefer an SDK abstraction, the same logic applies to OpenAI, Gemini, or a self-hosted model, but you must preserve the structured-output contract and hard timeout behavior.
4. Project Structure
Create the following files:
.ai-review/
├── review_gate.py # Main gate logic
├── scanner.py # Deterministic security checks
├── verifier.py # Finding validation
├── prompt.py # AI prompt and response schema
├── requirements.txt
└── .github/workflows/
└── ai-review.yml
5. Step 1: Create the CI Project
Create a virtual environment and install the only runtime dependency:
mkdir -p .ai-review/.github/workflows
cd .ai-review
python3 -m venv .venv
source .venv/bin/activate
pip install anthropic
Create requirements.txt:
anthropic>=0.40.0
The gate intentionally avoids heavyweight frameworks. A CI job that spends 10 seconds importing dependencies cannot meet a 30-second budget.
6. Step 2: Collect and Normalize the Diff
The first step is to fetch the exact diff being reviewed. Using the merge-base SHA is essential. Comparing HEAD directly against the base branch can silently include unrelated commits or miss changes introduced by a merge.
Create review_gate.py:
#!/usr/bin/env python3
"""AI code review CI gate with deterministic verification."""
import argparse
import json
import os
import re
import subprocess
import sys
from dataclasses import dataclass, asdict
from pathlib import Path
from typing import Any
import anthropic
from prompt import build_review_prompt
from scanner import scan_diff
from verifier import verify_findings
@dataclass
class Hunk:
"""A single contiguous changed region in a file."""
file: str
old_start: int
new_start: int
old_lines: int
new_lines: int
lines: list[str]
@property
def added_line_numbers(self) -> set[int]:
"""Return line numbers in the new file that were added."""
numbers: set[int] = set()
current = self.new_start
for line in self.lines:
if line.startswith("+"):
numbers.add(current)
current += 1
elif line.startswith("-"):
continue
else:
current += 1
return numbers
@property
def added_content(self) -> str:
"""Return only added lines, stripping diff markers."""
return "\n".join(
line[1:] for line in self.lines if line.startswith("+")
)
def run_git(args: list[str], cwd: str | None = None) -> str:
"""Execute a git command and return stdout."""
result = subprocess.run(
["git", *args],
cwd=cwd,
capture_output=True,
text=True,
timeout=10,
check=False,
)
if result.returncode != 0:
raise RuntimeError(
f"git {' '.join(args)} failed: {result.stderr.strip()}"
)
return result.stdout
def get_merge_base(base_sha: str, head_sha: str) -> str:
"""Find the common ancestor of base and head commits."""
return run_git(["merge-base", base_sha, head_sha]).strip()
def fetch_diff(base_sha: str, head_sha: str) -> str:
"""Fetch the unified diff between two commits."""
return run_git(
["diff", "--unified=3", "--no-color", base_sha, head_sha]
)
def parse_diff(diff_text: str) -> list[Hunk]:
"""Parse a unified diff into structured hunks.
Only added lines are tracked for finding verification. Context and
removed lines are retained for AI context but cannot anchor a finding.
"""
hunks: list[Hunk] = []
current_file: str | None = None
current_hunk: Hunk | None = None
file_pattern = re.compile(r"^\+\+\+ b/(.+)$")
hunk_pattern = re.compile(
r"^@@ -(\d+)(?:,(\d+))? \+(\d+)(?:,(\d+))? @@"
)
for line in diff_text.splitlines():
file_match = file_pattern.match(line)
if file_match:
current_file = file_match.group(1)
current_hunk = None
continue
hunk_match = hunk_pattern.match(line)
if hunk_match:
if current_file is None:
continue
current_hunk = Hunk(
file=current_file,
old_start=int(hunk_match.group(1)),
old_lines=int(hunk_match.group(2) or 1),
new_start=int(hunk_match.group(3)),
new_lines=int(hunk_match.group(4) or 1),
lines=[],
)
hunks.append(current_hunk)
continue
if current_hunk is not None:
if line.startswith(("+", "-", " ")):
current_hunk.lines.append(line)
elif line.startswith("\\"):
# Ignore the "\\ No newline at end of file" marker.
continue
return hunks
def build_diff_context(hunks: list[Hunk]) -> str:
"""Serialize hunks for the AI prompt with explicit line markers."""
blocks: list[str] = []
for hunk in hunks:
lines_with_numbers: list[str] = []
current = hunk.new_start
for line in hunk.lines:
if line.startswith("+"):
lines_with_numbers.append(f"{current}| {line[1:]}")
current += 1
elif line.startswith("-"):
continue
else:
lines_with_numbers.append(f"{current}| {line[1:]}")
current += 1
blocks.append(
f"FILE: {hunk.file}\n"
f"HUNK: @@ -{hunk.old_start},{hunk.old_lines} "
f"+{hunk.new_start},{hunk.new_lines} @@\n"
+ "\n".join(lines_with_numbers)
)
return "\n\n".join(blocks)
def get_changed_files(hunks: list[Hunk]) -> list[str]:
"""Return unique changed file paths."""
return sorted({hunk.file for hunk in hunks})
def get_repo_size(repo_path: str) -> int:
"""Calculate repository size in KB for context budgeting."""
result = subprocess.run(
["du", "-sk", repo_path],
capture_output=True,
text=True,
timeout=5,
check=False,
)
if result.returncode != 0:
return 0
return int(result.stdout.split()[0]) if result.stdout.strip() else 0
Why Merge-Base Diffing Matters
Consider a PR branch that is three commits behind main. A naive git diff origin/main...HEAD (three-dot syntax) compares against the merge-base, which is what you want. Two-dot syntax (git diff origin/main HEAD) compares the tips directly and can show changes that were already merged into main, producing false positives.
This implementation computes the merge-base explicitly and uses two-dot diffing with that SHA, which gives the same result as three-dot syntax but works with any CI checkout configuration.
7. Step 3: Run Deterministic Security Checks
Create scanner.py:
"""Deterministic security checks that run before the AI pass."""
import re
from typing import Any
from review_gate import Hunk
# Patterns are intentionally narrow. Broad regexes cause false positives
# that train developers to ignore the gate.
SECRET_PATTERNS = [
{
"id": "hardcoded-aws-key",
"severity": "critical",
"pattern": re.compile(
r"\bAKIA[0-9A-Z]{16}\b"
),
"message": "Hardcoded AWS access key ID detected",
},
{
"id": "private-key-block",
"severity": "critical",
"pattern": re.compile(
r"-----BEGIN (?:RSA |EC |OPENSSH )?PRIVATE KEY-----"
),
"message": "Private key material committed to the repository",
},
{
"id": "hardcoded-password",
"severity": "high",
"pattern": re.compile(
r"(?i)password\s*=\s*['\"][^'\"]{8,}['\"]"
),
"message": "Hardcoded password assignment detected",
},
]
def scan_diff(hunks: list[Hunk]) -> list[dict[str, Any]]:
"""Scan added lines for known-bad patterns.
Only added lines are scanned. A secret that existed before the PR
is a pre-existing issue, not a regression introduced by this change.
"""
findings: list[dict[str, Any]] = []
for hunk in hunks:
current_line = hunk.new_start
for diff_line in hunk.lines:
if diff_line.startswith("+"):
content = diff_line[1:]
for rule in SECRET_PATTERNS:
if rule["pattern"].search(content):
findings.append(
{
"source": "scanner",
"rule_id": rule["id"],
"severity": rule["severity"],
"file": hunk.file,
"line": current_line,
"message": rule["message"],
"explanation": (
"This pattern matches a known credential "
"format. Move the value to a secret manager "
"or environment variable and rotate it if "
"it was ever committed."
),
}
)
current_line += 1
elif diff_line.startswith(" "):
current_line += 1
return findings
The scanner runs first because it is fast and certain. If the scanner finds a critical issue, you can skip the AI call entirely and return immediately, saving the full AI budget for hard cases.
8. Step 4: Add the AI Review Pass
The AI reviewer needs strict output format, explicit line references, and a bounded context. Create prompt.py:
"""Prompt construction and AI response parsing."""
import json
import re
from typing import Any
SYSTEM_PROMPT = """You are a senior software engineer reviewing a pull request diff.
Your job is to identify concrete, high-confidence defects introduced by the diff.
Rules:
1. Only report issues on lines marked with `+` in the diff context.
2. Every finding must include the exact file path and new-file line number.
3. Do not report style preferences, formatting, or subjective refactoring suggestions.
4. Do not report issues in unchanged code unless the change makes them newly reachable.
5. If you are not confident, omit the finding. A false positive costs more than a missed issue.
6. Focus on: security vulnerabilities, null/undefined dereferences, race conditions,
resource leaks, incorrect error handling, off-by-one errors, broken API contracts,
and logic errors.
7. Return ONLY valid JSON. No markdown fences, no commentary.
Output schema:
{
"findings": [
{
"file": "path/to/file.py",
"line": 42,
"severity": "critical|high|medium|low",
"message": "Short title of the issue",
"explanation": "Why this is a problem and what input triggers it",
"confidence": 0.0-1.0
}
]
}\
## Understanding the Review Contract
Before we wire this into CI, let's dissect each field in that schema, because the contract you define here *is* the gate. A poorly designed contract produces a gate that's either toothless or obnoxious.
**`file`** and **`line`** are self-explanatory, but note that `line` should always be an integer. If your model returns a float or a string, your parser will throw. I recommend a defensive cast: `int(issue.get("line", 0) or 0)`.
**`severity`** is the field that determines whether the pipeline fails. The four-tier ladder — `critical`, `high`, `medium`, `low` — gives you granularity. A `critical` finding should be an unconditional block: a security vulnerability, an unhandled exception in a hot path, or a data race. A `low` finding should be informational — a style nit that a human can triage later.
**`confidence`** is the model's self-reported certainty, a float between `0.0` and `1.0`. This is your escape hatch for false positives. You'll use it to define a threshold: *if confidence is below 0.7, downgrade the severity by one level*. This simple rule dramatically reduces noise in practice.
**`message`** and **`explanation`** are what the developer sees in the pull request comment. Keep `message` under 80 characters — it's the title in a list. `explanation` is where the model should include the triggering input, the root cause, and a suggested fix.
Now, here's the critical design decision: **what makes the gate fail?** I recommend the following policy, which you can tune per repository:
python
def evaluate_gate(issues, config):
"""
Determine whether the CI gate passes or fails.
Policy:
- Any 'critical' issue with confidence >= 0.6 -> FAIL
- Any 'high' issue with confidence >= 0.8 -> FAIL
- Otherwise -> PASS
"""
blockers = []
for issue in issues:
severity = issue.get("severity", "low")
confidence = float(issue.get("confidence", 0.0))
if severity == "critical" and confidence >= config.critical_confidence:
blockers.append(issue)
elif severity == "high" and confidence >= config.high_confidence:
blockers.append(issue)
return len(blockers) == 0, blockers
Why the confidence thresholds? Because the model is *not* a deterministic linter. It's probabilistic. A `critical` finding with `confidence: 0.3` is likely a hallucination. You want to fail the build only when you're reasonably certain. The thresholds above are conservative defaults — start with `critical_confidence=0.6` and `high_confidence=0.8`, then tune after a week of real-world data.
---
## Building the Review Script
Now we get to the meat. Here's a complete, runnable script that:
1. Collects the diff from the CI environment
2. Sends it to the model
3. Parses the response
4. Applies the confidence-based severity downgrade
5. Evaluates the gate
6. Posts inline comments to the PR (if the CI system supports it)
I'll use Python because it's the lingua franca of CI glue code, but the logic translates to Node, Go, or Rust equally well.
python
!/usr/bin/env python3
"""
ai_review_gate.py — The 30-second AI code review gate.
Usage:
ai_review_gate.py --base-ref origin/main --head-ref HEAD \
--api-key $SENSENOVA_API_KEY \
--model sensenova-v6.8-flash-lite \
--fail-on critical,high
Environment variables:
GITHUB_TOKEN — for posting comments (GitHub Actions)
GITLAB_TOKEN — for posting comments (GitLab CI)
SENSENOVA_API_KEY — your model API key
"""
import argparse
import json
import os
import re
import subprocess
import sys
import time
from pathlib import Path
try:
import requests
except ImportError:
sys.exit("Missing dependency: requests. Run pip install requests.")
---------------------------------------------------------------------------
Configuration
---------------------------------------------------------------------------
DEFAULT_MODEL = "sensenova-v6.8-flash-lite"
DEFAULT_TIMEOUT = 25 # seconds — the "30-second" budget includes overhead
class Config:
"""Centralized configuration with sensible defaults."""
def __init__(self, args):
self.base_ref = args.base_ref
self.head_ref = args.head_ref
self.api_key = args.api_key or os.environ.get("SENSENOVA_API_KEY")
self.model = args.model or DEFAULT_MODEL
self.timeout = args.timeout or DEFAULT_TIMEOUT
self.max_diff_bytes = args.max_diff_bytes or 200_000 # 200 KB cap
self.critical_confidence = args.critical_confidence or 0.6
self.high_confidence = args.high_confidence or 0.8
self.fail_on = set(args.fail_on.split(",")) if args.fail_on else {"critical", "high"}
self.post_comments = not args.no_comments
self.dry_run = args.dry_run
if not self.api_key:
sys.exit("Error: SENSENOVA_API_KEY is not set. "
"Pass --api-key or set the environment variable.")
---------------------------------------------------------------------------
Step 1: Collect the diff
---------------------------------------------------------------------------
def get_diff(base_ref: str, head_ref: str) -> str:
"""
Get the unified diff between two refs, limited to text files.
We exclude binaries, lockfiles, and generated files to keep the
prompt small and the signal high.
"""
EXCLUDE_PATTERNS = [
r"\.lock$", r"package-lock\.json$", r"yarn\.lock$",
r"\.min\.js$", r"\.min\.css$", r"\.map$",
r"\.png$", r"\.jpg$", r"\.jpeg$", r"\.gif$", r"\.ico$",
r"\.pdf$", r"\.zip$", r"\.tar$", r"\.gz$",
r"\.woff2?$", r"\.ttf$", r"\.eot$",
]
cmd = [
"git", "diff", "--unified=3", "--no-color",
base_ref + "..." + head_ref,
]
try:
result = subprocess.run(
cmd, capture_output=True, text=True, check=True, timeout=15
)
except subprocess.TimeoutExpired:
sys.exit("Error: git diff timed out. Is the repo too large?")
except subprocess.CalledProcessError as e:
sys.exit(f"Error: git diff failed: {e.stderr}")
diff = result.stdout
# Filter out binary/generated files by splitting on diff headers
filtered_lines = []
skip = False
for line in diff.splitlines():
if line.startswith("diff --git"):
# Extract the filename from the diff header
m = re.search(r"b/(.+)$", line)
filename = m.group(1) if m else ""
skip = any(re.search(p, filename) for p in EXCLUDE_PATTERNS)
if not skip:
filtered_lines.append(line)
filtered_diff = "\n".join(filtered_lines)
# Enforce size cap
if len(filtered_diff.encode("utf-8")) > Config.max_diff_bytes:
# Truncate from the middle to keep both ends
half = Config.max_diff_bytes // 2
truncated = (filtered_diff[:half] +
"\n... [diff truncated due to size] ...\n" +
filtered_diff[-half:])
print(f"Warning: diff is {len(filtered_diff)} bytes, truncated to "
f"{Config.max_diff_bytes} bytes", file=sys.stderr)
return truncated
return filtered_diff
---------------------------------------------------------------------------
Step 2: Call the model
---------------------------------------------------------------------------
def call_model(diff: str, config: Config) -> dict:
"""
Send the diff to the model and parse the JSON response.
The prompt is engineered to produce structured output. We use
JSON mode if available, otherwise we parse the model's response
for a JSON block.
"""
system_prompt = """\
You are a senior code reviewer embedded in a CI pipeline. Your job is to
identify real, actionable issues in a code diff. You are NOT a linter and
you are NOT a style enforcer.
Review the diff and output ONLY a JSON object with this exact schema:
{
"summary": "One-sentence overview of the diff's risk profile",
"issues": [
{
"file": "path/to/file.py",
"line": 42,
"severity": "critical|high|medium|low",
"message": "Short title of the issue",
"explanation": "Why this is a problem and what input triggers it",
"confidence": 0.0-1.0
}
]
}
Rules:
- Only report issues that are objectively demonstrable from the diff. Do not speculate about code not in the diff.
- Severity must reflect real-world impact:
- critical: security vuln, data loss, crash, race condition
- high: logic error, unhandled exception, resource leak
- medium: potential bug, missing edge case, poor error handling
- low: readability, naming, minor refactor
- Confidence must reflect how certain you are. 0.9+ means you would bet your job on it. 0.5 means you're unsure but it's worth a human look.
- Never invent issues. If the diff is clean, return {"issues": []}.
-
Report at most 10 issues, prioritized by severity.
"""user_prompt = f"""\
Here is the git diff to review:
{diff}
"""
payload = {
"model": config.model,
"messages": [
{"role": "system", "content": system_prompt},
{"role": "user", "content": user_prompt},
],
"temperature": 0.1, # Low temperature for deterministic output
"max_tokens": 4096,
"response_format": {"type": "json_object"}, # If supported
}
headers = {
"Authorization": f"Bearer {config.api_key}",
"Content-Type": "application/json",
}
# The API endpoint will vary by provider; adjust as needed.
# For SenseNova, this would be your deployment's endpoint.
endpoint = os.environ.get(
"SENSENOVA_ENDPOINT",
"https://api.sensenova.ai/v1/chat/completions"
)
start = time.time()
try:
resp = requests.post(
endpoint, json=payload, headers=headers, timeout=config.timeout
)
resp.raise_for_status()
data = resp.json()
except requests.Timeout:
print("Error: Model request timed out. "
"Failing open (passing the gate).", file=sys.stderr)
return {"issues": [], "summary": "Model timeout — gate passed."}
except requests.RequestException as e:
print(f"Error: Model request failed: {e}. "
"Failing open (passing the gate).", file=sys.stderr)
return {"issues": [], "summary": "Model error — gate passed."}
elapsed = time.time() - start
print(f"Model call took {elapsed:.2f}s", file=sys.stderr)
# Extract the content from the response
try:
content = data["choices"][0]["message"]["content"]
except (KeyError, IndexError, TypeError):
print(f"Unexpected response shape: {json.dumps(data)[:500]}",
file=sys.stderr)
return {"issues": [], "summary": "Malformed response — gate passed."}
# Parse the JSON, handling the case where the model wraps it in markdown
return parse_model_response(content)
def parse_model_response(content: str) -> dict:
"""
Parse the model's JSON response, tolerating markdown fences and
trailing text.
"""
# Strip markdown code fences if present
fenced = re.search(r"
", content, re.DOTALL)
if fenced:
content = fenced.group(1)
else:
# Find the first { and last } and extract between them
start = content.find("{")
end = content.rfind("}")
if start != -1 and end != -1 and end > start:
content = content[start:end + 1]
try:
return json.loads(content)
except json.JSONDecodeError as e:
print(f"Warning: Failed to parse model JSON: {e}", file=sys.stderr)
print(f"Raw content: {content[:500]}", file=sys.stderr)
return {"issues": [], "summary": "Parse error — gate passed."}
# ---------------------------------------------------------------------------
# Step 3: Post-process and evaluate
# ---------------------------------------------------------------------------
def apply_confidence_downgrade(issues: list, config: Config) -> list:
"""
Downgrade severity based on confidence.
If the model says "critical" but only has 0.5 confidence, we
downgrade to "high". This reduces false-positive build failures.
"""
SEVERITY_ORDER = {"critical": 0, "high": 1, "medium": 2, "low": 3}
for issue in issues:
severity = issue.get("severity", "low")
confidence = float(issue.get("confidence", 0.0))
# Downgrade low-confidence findings
if confidence < 0.5 and severity in SEVERITY_ORDER:
current = SEVERITY_ORDER[severity]
new_severity = min(current + 2, 3) # Downgrade by 2 levels
issue["severity"] = list(SEVERITY_ORDER.keys())[new_severity]
issue["confidence"] = confidence
issue["message"] = f"[downgraded] {issue.get('message', '')}"
elif confidence < 0.7 and severity in SEVERITY_ORDER:
current = SEVERITY_ORDER[severity]
new_severity = min(current + 1, 3) # Downgrade by 1 level
issue["severity"] = list(SEVERITY_ORDER.keys())[new_severity]
issue["confidence"] = confidence
return issues
def evaluate_gate(issues: list, config: Config) -> tuple[bool, list]:
"""
Determine whether the gate passes.
Returns (passed, blocking_issues).
"""
blockers = []
for issue in issues:
severity = issue.get("severity", "low")
confidence = float(issue.get("confidence", 0.0))
if severity == "critical" and confidence >= config.critical_confidence:
blockers.append(issue)
elif severity == "high" and confidence >= config.high_confidence:
blockers.append(issue)
return len(blockers) == 0, blockers
# ---------------------------------------------------------------------------
# Step 4: Post comments (optional)
# ---------------------------------------------------------------------------
def post_github_comments(issues: list, config: Config) -> None:
"""
Post inline comments to a GitHub PR using the API.
This requires GITHUB_TOKEN and the PR number to be available.
"""
token = os.environ.get("GITHUB_TOKEN")
pr_number = os.environ.get("PR_NUMBER")
repo = os.environ.get("GITHUB_REPOSITORY")
if not (token and pr_number and repo):
print("Skipping GitHub comments: missing env vars", file=sys.stderr)
return
headers = {
"Authorization": f"Bearer {token}",
"Accept": "application/vnd.github.v3+json",
}
for issue in issues[:10]: # Cap at 10 comments
payload = {
"body": f"**{issue['severity'].upper()}** — {issue['message']}\n\n"
f"{issue['explanation']}\n\n"
f"*Confidence: {issue['confidence']:.2f}*",
"path": issue["file"],
"line": issue["line"],
"side": "right",
}
try:
resp = requests.post(
f"https://api.github.com/repos/{repo}/pulls/{pr_number}/comments",
json=payload, headers=headers, timeout=10
)
if resp.status_code not in (200, 201):
print(f"Failed to post comment: {resp.status_code} "
f"{resp.text[:200]}", file=sys.stderr)
except requests.RequestException as e:
print(f"Failed to post comment: {e}", file=sys.stderr)
# ---------------------------------------------------------------------------
# Main
# ---------------------------------------------------------------------------
def main() -> int:
parser = argparse.ArgumentParser(
description="AI code review gate for CI pipelines."
)
parser.add_argument("--base-ref", required=True,
help="Base ref (e.g., origin/main)")
parser.add_argument("--head-ref", default="HEAD",
help="Head ref (default: HEAD)")
parser.add_argument("--api-key", help="Model API key (or set env var)")
parser.add_argument("--model", default=DEFAULT_MODEL)
parser.add_argument("--timeout", type=int, default=DEFAULT_TIMEOUT)
parser.add_argument("--max-diff-bytes", type=int, default=200_000)
parser.add_argument("--critical-confidence", type=float, default=0.6)
parser.add_argument("--high-confidence", type=float, default=0.8)
parser.add_argument("--fail-on", default="critical,high",
help="Comma-separated severities that fail the gate")
parser.add_argument("--no-comments", action="store_true",
help="Don't post PR comments")
parser.add_argument("--dry-run", action="store_true",
help="Print issues but don't fail the gate")
args = parser.parse_args()
config = Config(args)
# Step 1: Get the diff
diff = get_diff(config.base_ref, config.head_ref)
if not diff.strip():
print("No diff to review. Gate passed.")
return 0
# Step 2: Call the model
result = call_model(diff, config)
issues = result.get("issues", [])
# Step 3: Post-process
issues = apply_confidence_downgrade(issues, config)
# Print the review summary
print("\n" + "=" * 60)
print("AI CODE REVIEW RESULTS")
print("=" * 60)
print(f"Summary: {result.get('summary', 'No summary provided')}")
print(f"Issues found: {len(issues)}")
print("-" * 60)
for i, issue in enumerate(issues, 1):
print(f"[{i}] {issue['severity'].upper()} "
f"({issue['confidence']:.2f}) "
f"{issue['file']}:{issue['line']}")
print(f" {issue['message']}")
print(f" {issue['explanation'][:120]}...")
print("=" * 60)
# Step 4: Evaluate the gate
passed, blockers = evaluate_gate(issues, config)
if config.dry_run:
print("Dry run: gate would " + ("PASS" if passed else "FAIL"))
return 0
# Step 5: Post comments (if enabled and not dry run)
if config.post_comments:
post_github_comments(issues, config)
if not passed:
print(f"\nGATE FAILED: {len(blockers)} blocking issue(s) found.")
for b in blockers:
print(f" - [{b['severity']}] {b['file']}:{b['line']} "
f"— {b['message']}")
return 1
print("\nGATE PASSED: No blocking issues found.")
return 0
if __name__ == "__main__":
sys.exit(main())
```yaml
---
## Wiring It Into GitHub Actions
Here's the complete workflow file. This is the part that makes the "30-second" claim real — the entire job, from checkout to gate decision, must complete within your CI runner's time budget.
yaml
.github/workflows/ai-review.yml
name: AI Code Review Gate
on:
pull_request:
types: [opened, synchronize, reopened]
jobs:
ai-review:
runs-on: ubuntu-latest
timeout-minutes: 2 # Hard cap: 2 minutes max
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
fetch-depth: 0 # Required for diff against base branch
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.11'
- name: Install dependencies
run: pip install requests
- name: Run AI review gate
id: ai_gate
env:
SENSENOVA_API_KEY: ${{ secrets.SENSENOVA_API_KEY }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
PR_NUMBER: ${{ github.event.pull_request.number }}
GITHUB_REPOSITORY: ${{ github.repository }}
run: |
python ai_review_gate.py \
--base-ref origin/${{ github.event.pull_request.base.ref }} \
--head-ref HEAD \
--timeout 25 \
--fail-on critical,high
- name: Upload review report as artifact
if: always()
uses: actions/upload-artifact@v4
with:
name: ai-review-report
path: ai_review_results.json
if-no-files-found: ignore
The `timeout-minutes: 2` is your safety net. If the model hangs, the job dies. That's better than blocking a PR for 10 minutes.
---
## Handling False Positives: The Trust But Verify Loop
Here's the philosophical core of the article. The gate should *never* be the final authority. It's a triage tool. Here's the three-tier trust model I recommend:
**Tier 1: The Gate (0–30 seconds)**
- Runs on every PR push
- Catches obvious critical issues
- Fails fast, but only on high-confidence findings
- Never blocks on `medium` or `low`
**Tier 2: The Human (30 seconds–5 minutes)**
- Reviews the AI's findings in the PR comments
- Can dismiss false positives with a single click
- The AI's comments include the triggering code, so verification is fast
**Tier 3: The Learning Loop (weekly)**
- Collect all AI findings and their dismissal status
- Feed dismissed findings back as negative examples
- Retune confidence thresholds based on precision/recall
This loop is what separates a useful gate from a noisy one. After two weeks, you'll have data to answer: *What's the precision of `critical` findings?* If it's below 60%, raise `critical_confidence` to 0.8. If it's above 90%, you can lower it to 0.5 and catch more real bugs.
---
## Performance and Cost Optimization
The 30-second budget is tight. Here's how to stay under it:
**1. Cap the diff size.** A 200 KB diff is roughly 5,000 lines. That's already a lot for a model to review in 25 seconds. If your diff exceeds the cap, truncate it (as shown in the script) and add a warning. The gate still runs, but it's reviewing a representative sample.
**2. Use a streaming response.** If your model API supports streaming, consume it incrementally. This lets you start parsing the first issues while the model is still generating.
**3. Cache unchanged files.** If a file hasn't changed since the last review, don't re-review it. Store a hash of each file's diff in a cache (e.g., `~/.cache/ai-review/<file-hash>.json`). On subsequent runs, skip files with matching hashes.
**4. Batch PRs.** If you have a monorepo with many small PRs, consider batching reviews. Review the combined diff of all open PRs once, then distribute findings. This is advanced, but it cuts API costs by 3–5x.
**5. Use the cheapest model for the gate.** You don't need a flagship model for a triage gate. A smaller, faster model with a well-crafted prompt will catch 80% of the issues at 10% of the cost. Save the big model for the weekly deep-dive.
---
## Security Considerations
You're sending source code to an external API. That's a real concern. Here's how to mitigate it:
**1. Never send secrets.** Scrub the diff for patterns like `API_KEY=`, `password=`, `BEGIN PRIVATE KEY`, etc. Replace them with `[REDACTED]` before sending.
python
SECRET_PATTERNS = [
r"(?i)(api[-]?key|secret|token|password)\s*[=:]\s*\S+",
r"-----BEGIN (?:RSA|EC|OPENSSH|PGP) PRIVATE KEY-----",
r"AKIA[0-9A-Z]{16}", # AWS access key
r"ghp[a-zA-Z0-9]{36}", # GitHub PAT
]
def scrub_secrets(diff: str) -> str:
for pattern in SECRET_PATTERNS:
diff = re.sub(pattern, r"\1=[REDACTED]", diff)
return diff
**2. Use a private endpoint.** If you're running SenseNova on-prem or in a VPC, point `SENSENOVA_ENDPOINT` to your internal deployment. This keeps code inside your network boundary.
**3. Log minimally.** Don't log the full diff or the full model response. Log only the issue summaries. If you must debug, add a `--verbose` flag that writes the payload to a local file, not to CI logs.
**4. Pin the model version.** Never use `latest`. Pin to a specific model version so a silent model update can't change your gate's behavior. In the script, that's the `--model` flag.
---
## The 30-Second Budget: A Realistic Breakdown
Let's do the math. Here's where the time actually goes:
| Step | Time Budget |
|------|-------------|
| `git diff` | 1–3 seconds |
| Diff scrubbing + truncation | < 0.5 seconds |
| API call (network + inference) | 15–25 seconds |
| JSON parsing + post-processing | < 0.5 seconds |
| Gate evaluation | < 0.1 seconds |
| **Total** | **~17–29 seconds** |
The model inference is 80% of the budget. If you're consistently hitting 29 seconds, you have three levers: reduce the diff size, reduce the max tokens, or switch to a faster model. There's no way around the inference time — it's physics, not engineering.
---
## Conclusion: Trust, But Verify
The phrase "trust but verify" is a Cold War arms control maxim, and it's the perfect lens for AI code review. You trust the model to catch things you'd miss. You verify its findings before they block a merge. The gate is not an oracle — it's a first-pass filter that makes your human reviewers 10x more efficient.
Here's what I want you to take away:
1. **The gate is a triage tool, not a judge.** It fails on high-confidence criticals and lets everything else through for human review.
2. **The confidence score is your most important field.** It's the difference between a useful gate and a false-positive machine. Tune it with real data, not intuition.
3. **The 30-second budget is achievable.** A 200 KB diff, a fast model, and a 25-second timeout gets you there. Don't over-engineer.
4. **The learning loop is where the magic happens.** Every dismissed finding is a data point. Feed them back and your gate gets sharper every week.
5. **Security is non-negotiable.** Scrub secrets, pin models, and use private endpoints. A code review tool that leaks credentials is worse than no tool at all.
The best part? Once you have this gate running, you'll never go back. The first time it catches a `critical` null-pointer dereference that your human reviewers missed, you'll be a believer. The first time it blocks a merge because the model hallucinated a `critical` finding, you'll tune the confidence threshold and move on.
That's the trust-but-verify loop. It's not perfect. It's not a replacement for human judgment. But it's a force multiplier that turns a 45-minute review into a 5-minute verification.
And that's worth 30 seconds of CI time.
---
*Have you implemented an AI review gate? I'd love to hear about your confidence thresholds and false-positive rates. Drop a comment below — the best tuning parameters come from the trenches, not the docs.*
Top comments (0)