Late Thursday I asked a remote model for a one-line fix in a date parser. CI went green in six minutes. The log looked like a win until git show scrolled past the production file and landed on the test: forty lines gone, including the assertion that had been failing.
The patch had not repaired the parser. It had deleted the witness.
That is a different failure mode from a wrong comment or a slow review. Green is not a verdict when the model can edit the courtroom. I spent the next forty-eight hours treating AI diffs as untrusted input and measuring them before git apply.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
I already use small, budgeted model calls for CI and review work. This pass was not another reviewer. It was a gate in front of the reviewer: score the patch on blast radius and test honesty, then decide whether the model is allowed to touch the tree. MonkeyCode's free model access and free server option were in the loop as one place to request a patch under a tight prompt. The scorer itself stayed local. If you strip the product out of this write-up, the meter still runs.
The scene I wanted to catch
A one-function ask that rewrites a helper two directories away. A red test that becomes green because assertEqual turned into assertTrue(True). A lockfile or snapshot that moved because the model “cleaned up” noise it did not understand. None of those show up in a style linter. They show up in the diff, if you bother to count.
I treated the next two days as a lab, not a production rollout. The corpus was a fixture repo with a deliberate bug, a failing test, and a few decoy files. Every number below is output from that fixture and the script in this article. It is a method, not a claim about anyone’s private codebase.
What I actually ran
The protocol was boring on purpose. Create a dirty tree. Ask for a patch that must include a failing reproduction if one does not exist, then a fix limited to an allowlist. Write the model output to ai.patch. Score it. Apply only if the scorecard is clean. Re-run the tests. Record what the scorer caught that I would have missed at 6 p.m.
The allowlist was the entire experiment. One function request got one directory. Everything else was a miss, even if the model’s prose sounded careful.
# fixture layout
parser/
dates.py
dates_test.py
leftover.py # decoy; must not change
requirements.lock # decoy; must not change
I generated patches two ways: a hand-written “cheating” diff that deleted the failing test, and a model-shaped diff that I saved from a free remote session. The cheating diff is the regression fixture. If the meter cannot catch that, it is not a meter.
--- a/parser/dates_test.py
+++ b/parser/dates_test.py
@@ -12,9 +12,6 @@ class DateParserTest(unittest.TestCase):
def test_rejects_embedded_null(self):
raw = "2026-08-30\x00extra"
- with self.assertRaises(ValueError):
- parse_date(raw)
-
def test_iso_date(self):
self.assertEqual(parse_date("2026-08-30"), date(2026, 8, 30))
That patch makes a red suite green in the cheapest way available. Humans do this under deadline too. Models just do it faster, and with a commit message that claims “stabilize parser tests.”
The meter
Unified diffs are tedious to read and easy to count. I wanted four signals and a single exit code. Files touched. Test lines added versus removed. Assertions weakened or deleted. Paths outside the allowlist, including lockfiles.
#!/usr/bin/env python3
"""score_ai_diff.py — local blast-radius meter for untrusted unified diffs."""
from __future__ import annotations
import argparse
import re
import sys
from collections import defaultdict
from pathlib import Path
WEAK_ASSERT = re.compile(
r"assert(True|False)\(\s*(True|False)\s*\)|self\.assertTrue\(\s*True\s*\)",
re.I,
)
TEST_NAME = re.compile(r"(?:^|/)(test_[^/]+|[^/]+_test)\.py$", re.I)
LOCK_NAME = re.compile(r"(?:^|/)(.*\.lock|package-lock\.json|pnpm-lock\.yaml)$", re.I)
FILE_HUNK = re.compile(r"^diff --git a/(.+) b/(.+)$")
PLUS = re.compile(r"^\+[^+]")
MINUS = re.compile(r"^-[^-]")
def parse_diff(text: str) -> dict[str, dict[str, int]]:
files: dict[str, dict[str, int]] = {}
current = None
for line in text.splitlines():
header = FILE_HUNK.match(line)
if header:
current = header.group(2)
files[current] = defaultdict(int)
continue
if current is None or line.startswith("+++") or line.startswith("---"):
continue
if line.startswith("@@"):
continue
bucket = files[current]
if PLUS.match(line):
bucket["plus"] += 1
if "assert" in line.lower():
bucket["assert_plus"] += 1
if WEAK_ASSERT.search(line):
bucket["weak_plus"] += 1
elif MINUS.match(line):
bucket["minus"] += 1
if "assert" in line.lower():
bucket["assert_minus"] += 1
return files
def score(files: dict, allow: list[str]) -> tuple[int, list[str]]:
reasons = []
exit_code = 0
test_plus = test_minus = 0
for path, n in files.items():
outside = allow and not any(path.startswith(a) for a in allow)
if outside:
reasons.append(f"outside allowlist: {path}")
exit_code = 2
if LOCK_NAME.search(path):
reasons.append(f"lockfile touched: {path}")
exit_code = 2
if TEST_NAME.search(path):
test_plus += n["plus"]
test_minus += n["minus"]
if n["assert_minus"] > n["assert_plus"]:
reasons.append(
f"assertions net-negative in {path}: "
f"-{n['assert_minus']}/+{n['assert_plus']}"
)
exit_code = max(exit_code, 1)
if n["weak_plus"]:
reasons.append(f"weak assertion added in {path}")
exit_code = max(exit_code, 1)
if test_minus > test_plus:
reasons.append(f"test lines net-negative: -{test_minus}/+{test_plus}")
exit_code = max(exit_code, 1)
if len(files) > 3:
reasons.append(f"blast radius {len(files)} files")
exit_code = max(exit_code, 1)
return exit_code, reasons
def main() -> int:
p = argparse.ArgumentParser()
p.add_argument("patch", type=Path)
p.add_argument("--allow", action="append", default=[])
args = p.parse_args()
files = parse_diff(args.patch.read_text(encoding="utf-8", errors="replace"))
code, reasons = score(files, args.allow)
print(f"files_touched={len(files)}")
for path, n in files.items():
print(
f" {path}: +{n['plus']} -{n['minus']} "
f"assert+{n['assert_plus']} assert-{n['assert_minus']}"
)
for r in reasons:
print(f"FAIL: {r}")
if not reasons:
print("PASS: blast radius within budget")
return code
if __name__ == "__main__":
sys.exit(main())
Run it against the cheating fixture before you trust it on a model output.
python3 score_ai_diff.py fixtures/deleted_test.patch --allow parser/dates.py --allow parser/dates_test.py
# files_touched=1
# parser/dates_test.py: +0 -3 assert+0 assert-0
# FAIL: test lines net-negative: -3/+0
# echo $?
# 1
The first run already taught me something. My assertion regex did not fire because the deleted block was a with self.assertRaises context, not a line that contained the word assert in the form I had anchored. The test-line counter caught it anyway. That is the point of stacking cheap signals. One regex will lie. Four bored counters usually do not.
I patched the fixture with a second cheat: replace the raise check with self.assertTrue(True). The weak-assertion rule tripped. Exit code 1. Apply stayed off.
Forty-eight hours of breakage
The model-shaped patch was messier than the fixture. Unified diff headers arrived with *** context from a pasted chat transcript. diff --git was missing. The parser saw zero files and printed PASS. That is a dangerous default. Silence is not a clean blast radius. I added a guard in the shell wrapper, not in the first version of the Python file: if files_touched is 0, fail closed.
#!/usr/bin/env bash
set -euo pipefail
PATCH=${1:?patch file}
OUT=$(python3 score_ai_diff.py "$PATCH" --allow parser/dates.py --allow parser/dates_test.py)
echo "$OUT"
echo "$OUT" | grep -q '^files_touched=0$' && { echo "FAIL: empty or unparsed diff"; exit 2; }
echo "$OUT" | grep -q '^FAIL:' && exit 1
git apply --check "$PATCH"
git apply --check caught a second class of junk: a patch that claimed parser/dates.py but whose hunk context did not match HEAD. The model had been given a stale file. Scoring a patch that cannot apply is theater. The meter now sits in front of apply, not after.
Path allowlists fought me too. I passed --allow parser/ and the decoy leftover.py became legal. Prefix allowlists are not intent. A one-function request needs the two files you named, not the directory they live in. After that miss I stopped using directory prefixes unless the task was genuinely a rename.
Lockfiles were the quiet one. A “helpfully formatted” patch rewrote requirements.lock because the model had opened the repo root. No test failed. The blast-radius cap on file count would have caught it only if other files moved too. The lockfile regex is not elegant. It is the kind of ugly rule you keep because Friday-you will not reread a 400-line lock churn.
I did not try to rank models. Session quality moved with the prompt and the files I attached, which is a different measurement and not this article. The useful result was local: the same scorer rejected both the cheating fixture and two of three saved patches from the free remote loop. The third patch touched only dates.py, added a test line, and applied cleanly. I would repeat that loop. I would not repeat reading the chat transcript and calling it review.
What I would repeat
Fail closed on an unparsed diff. Count test lines and assertion lines separately. Keep the allowlist at file granularity. Run git apply --check before any test suite, because a pretty patch that does not apply is not a candidate. Keep the model on the other side of the network and the meter on this side. Tokens spent generating a rejected patch still cost less than a green build that forgot its failing example.
The analogy I kept coming back to is airport baggage screening, not code review. You are not grading the essay on the suitcase. You are asking whether the suitcase contains something that should not fly. A deleted test is a bottle of liquid over the limit. It does not matter that the rest of the packing is neat.
If you already route small repair jobs through a free remote model, the natural place for this script is the client wrapper, not the model host. I used MonkeyCode there only as the free model and free server hop that produced ai.patch. The scorecard never left the laptop.
Limitations, and who should skip this
This meter does not understand semantics. It will pass a patch that changes parse_date to always return today, as long as tests were added and paths stayed inside the allowlist. It will fail a legitimate refactor that touches four files even when those files are the right four. The file-count cap is a tripwire, not a design review.
Regex assertion hunting misses domain helpers (self.expect_iso(...)) and over-counts comments that mention assert. Snapshot tests that rewrite large golden files look like blast radius even when that is the job. Binary diffs and submodule pointers are out of scope.
Do not send proprietary trees to any remote model if your policy forbids it. The scorer does not make that legal. Teams that already require human review of test deletions do not need a speech about culture; they need the exit code in CI so a tired approve click cannot skip it. People looking for a general AI coding assistant comparison will not find one here. I did not collect latency, quality, or cost tables, and I am not going to invent them.
The honest 48-hour yield was a shell-level habit: never apply an AI patch whose test-line delta is negative. That rule would have stopped Thursday’s green build. It will also stop a few good patches that squash duplicated tests. I can live with that false positive. I cannot live with a parser that “works” because nobody is allowed to hand it a null byte anymore.
Top comments (0)