Your AI Reviewer Needs a Reviewer: A Free Pipeline for Stress-Testing Suggested Diffs

Last month I applied an AI-suggested fix to a retry loop without checking it properly. The change looked clean: fewer lines, clearer naming, and my test suite stayed green. What I didn't notice was that the model had moved a sleep outside a conditional, so every successful request now paid a delay that used to apply only to retries. The tests passed because no test ever timed that path.

That experience changed how I treat AI code-review suggestions. The suggestion itself is a hypothesis, not an answer, and hypotheses deserve experiments. This post describes the experiment setup I now run on every non-trivial suggestion: an isolated copy of the repo, a behavioral probe that refuses to accept "zero tests ran" as success, a static scan, and an adversarial second model pass. The whole thing runs on free model access and a free hosted server, so there is no per-call cost excuse for skipping the second opinion.

What actually goes wrong with suggested diffs

Obvious garbage is easy to reject. The suggestions that hurt you share three traits:

• They are locally sensible but violate an invariant that lives outside the snippet the model was shown.
• They alter edge-case behavior — short-circuit order, regex reach, error paths — in ways the stated reasoning never mentions.
• They occasionally weaken a validation or sanitization step, reopening a class of bug someone fixed years ago.

A verification pipeline therefore has to produce three kinds of evidence: the changed code still behaves (tests), it doesn't introduce known-bad patterns (static analysis), and its stated reasoning survives hostile scrutiny (a second model asked to find divergence, not to agree).

Step one: quarantine the suggestion

The model's edit never touches my real working tree. I apply it in a throwaway git worktree so a bad suggestion is rm -rf away from disappearing:

#!/usr/bin/env bash
# quarantine.sh — isolate an AI-suggested patch for verification
set -euo pipefail

PATCH_FILE="$1"
SCRATCH="$(mktemp -d)/suggestion-check"

git worktree add "$SCRATCH" HEAD
git -C "$SCRATCH" apply "$PATCH_FILE"
echo "$SCRATCH"   # hand the path to the next stage

This costs nothing and changes the psychology of the review: the suggestion is a specimen under glass, not a half-merged edit I'm emotionally invested in keeping.

Step two: a behavioral probe that can't lie about zero tests

The classic self-deception in this kind of automation is a test selector that matches nothing. Green output, zero signal. So the probe counts matched tests first and treats zero as a hard failure:

#!/usr/bin/env bash
# probe.sh <worktree> <test-selector>
set -uo pipefail
cd "$1"

MATCHED=$(pytest --collect-only -q -k "$2" 2>/dev/null | grep -c '::' || true)

if [ "$MATCHED" -eq 0 ]; then
  echo "REJECT: selector matched no tests — the suite has nothing to say about this change."
  exit 2
fi

pytest -x -q -k "$2" || { echo "REJECT: behavioral probe failed ($MATCHED tests ran)."; exit 1; }
echo "OK: $MATCHED targeted tests passed."

The REJECT on zero matches is the highest-value line in this entire article. In my own usage, that guard has caught more bad merges than the static scanner — usually by revealing that the "passing suite" I trusted never covered the changed module at all.

Step three: static scan, new findings only

A pattern-based scanner (Semgrep with its default registry rules, or your language's equivalent) runs against just the changed files. It will not catch logic bugs, but it reliably catches the "simplified input validation" category of regression. Treat any finding on the diff as a human-review gate, not an auto-reject — false positives exist, but the review takes two minutes.

Step four: hostile cross-examination

Here is the part people skip because a second model call costs money on a metered plan. Instead of asking a model "is this change correct?" — which invites a confident re-derivation of the same reasoning — I ask it to attack the change:

You are a skeptical examiner reviewing a proposed patch.

CONTEXT BEFORE THE CHANGE:
{original}

PATCH:
{diff}

AUTHOR'S STATED REASONING:
{rationale}

Rules:
- Do not summarize or praise the patch.
- Name up to three concrete inputs or system states whose behavior this
  patch changes without the reasoning mentioning it.
- Classify each as SAFE, UNSAFE, or UNDETERMINABLE from the shown context.
- If the patch touches parsing, authentication, concurrency, or error
  handling, state the one invariant a human must verify by hand.
- If there is genuinely nothing, output exactly: NO DIVERGENCE FOUND.
  Fabricating concerns is a failure.

Any OpenAI-compatible client can send this. The gold is in the UNDETERMINABLE bucket: it is a precise, generated checklist of what you personally still owe the review. A response of "looks fine" carries almost no information; a list of three unverifiable claims carries a lot.

Making the second opinion free

The cross-examination stage used to be where cost crept in — one extra model call per suggestion, multiplied across every PR. Two ways to bring the marginal cost to zero:

1. Hosted free tier. MonkeyCode currently provides free model access together with a free server option, which means the Stage 4 call can hit a hosted endpoint instead of a billed API key, so the adversarial pass can run on every suggestion rather than only the intimidating ones. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Read "free" as a statement about current availability, not a permanent contract — free tiers change their quotas, latency, and model lineups, so confirm the present terms before wiring this into anything your team depends on.
2. Your own hardware. An OpenAI-compatible local server (Ollama, llama.cpp) drops into the same client by changing one base URL. A mid-size quantized model is weaker than a frontier model, but the cross-examination prompt above is a structured checklist task, and mid-size models handle those better than their benchmark rankings suggest.

The pipeline is deliberately indifferent to which backend answers. The prompt, the evidence categories, and the decision rules stay identical.

Reading the results

Outcome combination	Interpretation	My move
Probe passes, scan clean, examiner reports NO DIVERGENCE	Safe for the shown context	Apply after my own skim of the diff
Probe passes, examiner returns UNDETERMINABLE items	Model perceives risk it cannot resolve	Hand-check exactly those invariants
Probe rejects: zero tests matched	No behavioral evidence exists at all	Write a characterization test before anything merges
Scan flags the diff	Possible security-relevant regression	Blocked until a human clears it
Examiner contradicts the author's rationale	The reasoning itself is unstable	Reject — unstable reasoning is a worse sign than a wrong answer

Where this breaks down

• Independence is required. Cross-examining with the same model that authored the suggestion preserves its blind spots. Use a different model, ideally a different provider.
• Scanners have a known ceiling. They match patterns, not novel exploitable logic.
• Free offerings move. Quotas, availability, and latency on any free tier can shift without warning; keep the local fallback exercised so it works the day you need it.
• Scope matters. This pipeline assumes a review-sized diff. A 40-file agent-generated refactor needs characterization tests and staged rollout, not a checklist.

Who should skip this: teams with enforced coverage gates and mature mutation testing already own most of this value. And if your code cannot leave your perimeter for compliance reasons, the hosted free tier is simply not an option — run the examiner locally or not at all.

Closing thought

The point is not suspicion; it is making trust something each suggestion earns individually rather than something the tool enjoys by default. A quarantined worktree, a probe that fails loudly on zero tests, a pattern scan, and one hostile second opinion cover most of the risk — and with free model access plus a free server, the second opinion costs nothing but thirty seconds. If you adapt this pipeline, the thing I'd be curious to hear is which stage fires most often for you — my money stays on the zero-tests guard.