My last post here was about a pre-push gate that fails loud instead of often. Since then I've been running an experiment on top of it: adding an AI review pass to the same gate, so a second set of (artificial) eyes looks at my diff before any human does.
The constraint I set for myself: it had to cost nothing to run, it had to work on plain git diff output, and it had to be a gate — meaning it produces a decision, not a wall of commentary I ignore.
Here's the workflow and the script. Everything below runs locally against a diff; the only network call is the model inference.
The problem with AI review as usually practiced
Most AI-assisted review I've seen fails in one of two ways:
- It's a chat, not a gate. You paste code into a chat window, get prose back, and nothing forces a decision. The signal-to-noise ratio decays fast.
- It reviews files, not diffs. Reviewing a whole file with a model means it flags pre-existing issues you didn't touch, and you learn to ignore it.
The fix for both: review only the diff, require structured output, and make the script exit non-zero above a severity threshold. Same philosophy as a linter with --max-warnings.
Why free models are good enough for this
A diff-review gate is a high-volume, low-stakes-per-call workload. You run it on every push, most pushes are fine, and a false positive costs you thirty seconds of reading. That's exactly the profile where a free model tier makes sense: you'd never justify a paid API call for "check whether my 40-line diff does something dumb," but you'll happily run it if it costs nothing.
I'm using MonkeyCode for this, which offers free model access and a free server option, so the whole gate runs without a billing account or a GPU on my machine. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The script below is deliberately written against a generic OpenAI-compatible chat endpoint, though — if you have another provider with an OpenAI-style API, change the base URL and it works the same.
One thing I want to be honest about up front: I haven't benchmarked the free tier's latency or rate limits under heavy use, and I don't know what its long-term availability terms are. For a pre-push hook that makes one call per push, none of that has mattered to me so far. If you're gating CI for a team of fifty, you'd want to verify those limits first.
The gate script
Save as ai_review_gate.py, run it from your pre-push hook or manually with python ai_review_gate.py origin/main.
#!/usr/bin/env python3
"""AI review gate for Python diffs.
Reviews only the current diff (not whole files), requires structured
JSON findings, and exits 1 if any finding meets the severity threshold.
"""
import json
import os
import subprocess
import sys
import urllib.request
BASE_URL = os.environ.get("AI_REVIEW_BASE_URL", "https://YOUR-ENDPOINT/v1")
API_KEY = os.environ.get("AI_REVIEW_API_KEY", "")
MODEL = os.environ.get("AI_REVIEW_MODEL", "default") # whatever free model your endpoint exposes
BLOCK_AT = {"high", "critical"} # severities that fail the gate
MAX_DIFF_CHARS = 24_000
PROMPT = """You are reviewing a git diff of a Python project.
Report ONLY issues introduced or worsened by this diff. Do not comment
on pre-existing code the diff merely touches.
For each issue, return a JSON array element:
{"file": str, "line": int|null, "severity": "low"|"medium"|"high"|"critical",
"kind": "bug"|"security"|"correctness"|"regression-risk",
"summary": str (one sentence), "suggestion": str (one sentence)}
Return [] if the diff looks fine. Return ONLY the JSON array, no prose.
DIFF:
"""
def get_diff(base: str) -> str:
diff = subprocess.run(
["git", "diff", f"{base}...HEAD", "--", "*.py"],
capture_output=True, text=True, check=True,
).stdout
if len(diff) > MAX_DIFF_CHARS:
# Truncate rather than skip silently — a truncated review is
# still a review; a silent pass is not.
diff = diff[:MAX_DIFF_CHARS] + "\n# [diff truncated for review]\n"
return diff
def call_model(diff: str) -> list[dict]:
body = json.dumps({
"model": MODEL,
"messages": [{"role": "user", "content": PROMPT + diff}],
"temperature": 0,
}).encode()
req = urllib.request.Request(
f"{BASE_URL}/chat/completions",
data=body,
headers={"Content-Type": "application/json",
"Authorization": f"Bearer {API_KEY}"},
)
with urllib.request.urlopen(req, timeout=120) as resp:
payload = json.load(resp)
text = payload["choices"][0]["message"]["content"].strip()
# Models sometimes wrap JSON in fences despite instructions.
if text.startswith("```
"):
text = text.split("\n", 1)[1].rsplit("
```", 1)[0]
try:
findings = json.loads(text)
except json.JSONDecodeError:
# Unparseable output = inconclusive, not a pass.
print("AI review returned unparseable output; treating as inconclusive.")
sys.exit(2)
if not isinstance(findings, list):
sys.exit(2)
return findings
def main() -> None:
base = sys.argv[1] if len(sys.argv) > 1 else "origin/main"
diff = get_diff(base)
if not diff.strip():
sys.exit(0) # nothing to review
findings = call_model(diff)
blocking = [f for f in findings if f.get("severity") in BLOCK_AT]
for f in findings:
loc = f"{f.get('file', '?')}:{f.get('line') or '?'}"
print(f"[{f.get('severity', '?'):>8}] {loc} ({f.get('kind', '?')})")
print(f" {f.get('summary', '')}")
if f.get("suggestion"):
print(f" -> {f['suggestion']}")
if blocking:
print(f"\nGate FAILED: {len(blocking)} high/critical finding(s).")
sys.exit(1)
print(f"\nGate passed ({len(findings)} non-blocking finding(s)).")
if __name__ == "__main__":
main()
Three design decisions worth calling out, because they're the difference between a gate and a toy:
1. Unparseable output exits 2, not 0. If the model returns prose instead of JSON, the gate is inconclusive, and my hook treats exit 2 as "warn me but let me decide." Exiting 0 there would train me to trust a gate that sometimes isn't running at all.
2. Truncation is loud. Diffs over ~24k characters get truncated with a visible marker, because silent truncation means the model reviewed a different diff than the one I'm pushing. For genuinely huge diffs, the honest answer is to split the push — which the gate nagging you about is arguably a feature.
3. Severity threshold, not zero-findings. Low and medium findings print but don't block. If every style nit failed the gate, I'd disable it within a week. The gate exists to catch the "wait, that's a mutable default argument" class of mistake.
Wiring it into pre-push
# .git/hooks/pre-push (or via pre-commit framework)
#!/bin/sh
python ai_review_gate.py origin/main
code=$?
if [ $code -eq 1 ]; then
echo "AI review gate failed. Fix findings or push with --no-verify (and own it)."
exit 1
fi
# exit 2 (inconclusive) warns but allows the push
exit 0
Note the --no-verify escape hatch stays available. A gate with no override is a gate people uninstall; the point is to make skipping it a visible, deliberate act, not to make it impossible.
What I feed it and what I don't
| Scenario | Verdict |
|---|---|
| Solo side projects, every push | Runs unconditionally; cost is zero, false positives are cheap |
| Work repo with human review anyway | Still useful — catches things before a colleague has to |
| Diff touching auth/crypto/payments paths | Gate runs, but I treat a "pass" as meaningless; human review required regardless |
| Diff > ~500 lines changed | Split the change; both the model and your reviewer will miss things |
| Generated code (migrations, lockfiles, protobuf) | Excluded via the *.py pathspec and a per-repo ignore list |
Limitations, honestly
- It has no project context. The model sees a diff, not your architecture. It will flag things that are fine given your conventions, and miss things that are wrong given your invariants. It catches local mistakes, not design errors.
-
Severity calibration drifts between models. I tuned
BLOCK_ATagainst the free model I was using; if you switch models, re-check what it calls "high" for a week before trusting the threshold. -
Free tiers change. Free access and free servers are availability claims as of when I set this up, not a contract. The script points at an OpenAI-compatible endpoint precisely so that if the free option disappears or its limits stop fitting my usage, I swap
AI_REVIEW_BASE_URLand keep the gate. - Non-determinism is real even at temperature 0 across provider-side changes. The gate is a heuristic net, not a proof.
Who should skip this
If your team already has fast, thorough human review with short queues, this adds little — the human pass subsumes it. If your diffs are routinely enormous, fix that first; no review method, human or machine, survives a 2,000-line diff. And if you're in a regulated environment where code can't leave your perimeter, a hosted free endpoint is the wrong tool entirely — run a local model or don't do this.
Where this leaves the gate
My pre-push stack is now: lint and type checks (fast, deterministic), then this AI pass (slower, probabilistic), then a human for anything that matters. The AI layer catches a different class of mistake than the linters — "this logic inverts the flag" rather than "this line is too long" — and at zero marginal cost per push, the trade is easy.
If you try it, the thing I'd most like to hear is where the severity threshold lands for you. My guess is "high" is right for solo work and wrong for teams, but I only have my own repos to judge from. If you want to run the same setup, MonkeyCode's free tier is what I pointed the script at; the README for whatever endpoint you choose will have the base URL and model identifier to drop into the env vars.
Top comments (0)