Last month I wrote about adding an explanation gate to pull requests so that AI-generated code never lands without a human-readable rationale. This post is the natural sequel: once you accept that every AI-assisted PR needs review, the next problem is review economics. Running every diff through the strongest available model is wasteful; running everything through the weakest one is reckless. The answer I keep coming back to is a boring, old-fashioned idea — triage — applied to model selection.
New, cheaper, surprisingly capable models keep appearing, and pricing changes faster than my side projects ship. So instead of betting on any specific model, I built a small routing layer that treats "which model reviews this diff?" as a decision I can test, log, and change. Below is the workflow, a runnable artifact, and the honest list of cases where it fails.
The idea in one paragraph
Score each diff for risk signals (does it touch auth? migrations? crypto? concurrency? huge line counts?). Low-risk diffs go to a cheap-or-free model tier. High-risk diffs escalate to the strongest model you have access to — but only those. You get most of the safety of "review everything with the best model" at a fraction of the cost and latency, and you keep a log that lets you audit whether the routing itself is working.
A reproducible artifact: route_diff.py
This is a self-contained Python script (stdlib only, so you can run it anywhere — including on a free server you control). It doesn't call any model API itself; it produces the routing decision plus a review prompt, so you can wire it to whatever provider you like.
#!/usr/bin/env python3
"""route_diff.py — decide which model tier should review a diff.
Usage:
git diff main...HEAD | python3 route_diff.py
Exit codes:
0 -> routed to 'fast' tier (cheap/free model is fine)
2 -> escalated to 'strong' tier (use your best model)
The script prints a JSON decision to stdout so CI can log it.
This is a heuristic gate, not a security scanner. Tune the weights
for your own codebase.
"""
import json
import re
import sys
# Path patterns that should always escalate. Edit for your repo.
HIGH_RISK_PATHS = [
r"(^|/)auth/",
r"(^|/)(migrations|schema)/",
r"(^|/)crypto",
r"payment|billing|invoice",
r"\.env", r"secret", r"credential",
r"dockerfile", r"\.github/workflows",
]
# Content signals inside the diff body.
HIGH_RISK_CONTENT = [
r"\beval\(", r"\bexec\(",
r"subprocess|os\.system|shell=True",
r"SELECT .*;|DROP TABLE|DELETE FROM", # raw SQL
r"async def|threading|multiprocessing", # concurrency
r"jwt|bcrypt|argon2|hmac",
r"TODO|FIXME|HACK", # AI left homework
]
WEIGHTS = {
"path": 5, # per risky file path hit
"content": 3, # per risky content hit
"size": 1, # per 100 changed lines
}
ESCALATE_AT = 5 # tune with your own logs (see below)
def score(diff_text: str) -> dict:
files = re.findall(r"^diff --git a/(\S+)", diff_text, re.M)
added = len(re.findall(r"^\+(?!\+\+)", diff_text, re.M))
removed = len(re.findall(r"^-(?!--)", diff_text, re.M))
path_hits = sorted({
f for f in files
for pat in HIGH_RISK_PATHS if re.search(pat, f, re.I)
})
content_hits = sorted({
pat for pat in HIGH_RISK_CONTENT
if re.search(pat, diff_text, re.I)
})
total = (
WEIGHTS["path"] * len(path_hits)
+ WEIGHTS["content"] * len(content_hits)
+ WEIGHTS["size"] * ((added + removed) // 100)
)
return {
"tier": "strong" if total >= ESCALATE_AT else "fast",
"score": total,
"changed_lines": added + removed,
"risky_paths": path_hits,
"risky_signals": content_hits,
}
def build_prompt(decision: dict, diff_text: str) -> str:
focus = (
"This diff touched high-risk areas: "
+ ", ".join(decision["risky_paths"] or decision["risky_signals"])
+ ". Review for correctness, injection, and auth bypass first."
if decision["tier"] == "strong"
else "Routine review: logic errors, edge cases, naming."
)
return f"{focus}\n\n```
{% endraw %}
diff\n{diff_text}\n
{% raw %}
```"
if __name__ == "__main__":
diff = sys.stdin.read()
d = score(diff)
print(json.dumps(d, indent=2))
with open("review_prompt.md", "w") as fh:
fh.write(build_prompt(d, diff))
sys.exit(2 if d["tier"] == "strong" else 0)
Test it against your own history before trusting it:
# Did the router escalate the PR where you later found a bug?
git show <bugfix-commit>^..<bugfix-commit> | python3 route_diff.py
echo $? # you *want* 2 for commits that fixed real incidents
That last step matters. A routing gate you never calibrate is just superstition with a config file.
Where the free tiers actually fit
Two operator-supplied facts make this cheaper to run than it used to be: MonkeyCode currently offers free model access and a free server option, which means both the routing script and the fast-tier review calls can run without a paid plan.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
In my setup, the free tier handles the "fast" lane: the router script runs in CI on the free server, and routine diffs (renames, tests, docs, UI tweaks) get reviewed by a free-tier model. Escalated diffs — the ~10–20% that trip the risk signals — go to whichever strongest model I have credentials for that month. I deliberately don't hardcode model names in the script or the pipeline, because the cheap/strong landscape reshuffles constantly; the router's job is to keep that swap a one-line config change instead of a workflow rewrite. If you're evaluating tooling for this pattern, the MonkeyCode free tier is a low-commitment place to prototype the fast lane before you wire in anything paid.
Two things I did not assume, and you shouldn't either:
- Permanence and quotas. "Free" describes today's offer, not a guarantee. The router logs every decision to JSON so that if a free tier disappears or tightens, I can see exactly how much volume I need to move and what it would cost.
- Capability. I never assumed the free-tier model is "good enough" — I measured it (next section).
Calibration: is the fast lane actually safe?
The workflow only works if you validate the cheap tier instead of trusting it. My method:
- Take 30–50 historical PRs where a human reviewer (or a production incident) found a real problem.
- Run each through the fast-tier model with the router's generated prompt.
- Record a hit if the model flags the same class of issue.
- If the fast tier misses more than ~15–20% of routine-lane issues, tighten
ESCALATE_ATor promote more path patterns toHIGH_RISK_PATHS— don't just hope.
This is a small-sample, self-audit method, not a benchmark. Your numbers will be specific to your codebase and your model, which is exactly why the log file matters more than any vendor claim.
Limitations, and who should skip this
- It is a heuristic, not a security boundary. A determined bad diff can dodge every regex. Keep human review for anything that escalates, and treat the fast lane as "assisted review," not "no review."
-
Regex-based risk signals rot. Framework conventions change; a quarterly five-minute review of
HIGH_RISK_PATHSis part of the deal. - Skip this entirely if your repo is small enough that reviewing everything with your best model costs less than an hour of your time per month, if you're in a regulated environment where every change requires documented human review anyway, or if your diffs are dominated by generated/lockfile noise — the size signal will mislead you.
- Free tiers are a starting line, not architecture. Design so that losing the free tier is a config change, not an outage.
The takeaway
The durable skill isn't picking the right model this month — it's building the routing, logging, and calibration loop that lets you swap models as the market moves. Triage the diffs, spend strong tokens only where the risk is, and keep receipts so you can prove the cheap lane is earning its place.
If you try this, the most useful thing you can share isn't your threshold number — it's what your calibration set taught you about your own codebase's risk profile. That's the part no model can tell you.
Top comments (0)