Reproduce the AI Reviewer's Claim Before You Fix It

It's 9:47 AM. You hand a pull request to your AI reviewer, and it flags a CRITICAL vulnerability on line 19 with a reference to CVE-2024-0xxx. Confident. Decisive. Dead wrong.

A colleague checks the CVE: it belongs to a completely different code path. The line number is off by eleven. The proposed fix would have removed a nil check that was doing real work.

The review looks authoritative. It reads like someone audited your code. Nobody audited the auditor.

That's the trade nobody counts when they delegate review work to a free model.

When the reviewer is the unauthenticated input

Let me be concrete about what changed. A few weeks ago, I started using MonkeyCode's free-model tier and free server option for smaller security chores.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Free quota is a great reason to try a platform. It is not a reason to trust its output. The moment your development workflow consumes model-generated security findings, the model becomes an input to your review process. And inputs have trust boundaries.

The problem is rarely "the model is 100% wrong." The problem is "the model is 30% wrong, and the wrong 30% looks identical to the right 70%." One plausible-sounding CVE citation can burn more engineering time than a real vulnerability would.

So before you fix anything, build a tiny gate that asks one question: can the claim be reproduced inside a pinned environment?

Trust boundary: where the claim becomes a change

[pinned repo] -> [AI reviewer (free tier)] -> claim
      ^                                     |
      +---------- gate.sh ------------------+
                        |
                   pass / fail / no-info
                        |
                   human decides fix

The boundary you actually need is between the claim and the fix. Everything before that boundary — prompt, model, free server — is somebody else's chaos. Your job starts at the claim.

Step 1: Pin the reviewer's world

A security finding that can't be reproduced isn't a finding. It's a hypothesis with UI.

Before you run any AI review, pin the inputs:

export REVIEW_REV=$(git rev-parse HEAD)
export REVIEW_IMAGE=your-app:test-fixture
docker build -t "$REVIEW_IMAGE" .

Then run the review against that exact revision. Never evaluate an unpinned review. If the repo moved, the claim is meaningless.

Step 2: Force structured claims

Free-form review output is a nightmare to verify. Nudge the reviewer toward a schema instead. Ask for each finding to include an id, file, line, claim, and an executable evidence block.

Example fixture (review.json):

[
  {
    "id": "AI-1",
    "file": "requirements.txt",
    "line": 5,
    "claim": "Unpinned Flask dependency enables a known dependency-confusion chain.",
    "evidence": [
      { "type": "exec", "cmd": "grep -qE '^Flask==[0-9]+\\.[0-9]+\\.[0-9]+$' requirements.txt", "expect": "match" }
    ]
  },
  {
    "id": "AI-2",
    "file": "app/routes.py",
    "line": 42,
    "claim": "Path traversal via unvalidated filename in download endpoint.",
    "evidence": [
      { "type": "exec", "cmd": "grep -nE 'os\\.path\\.join|send_file' app/routes.py | wc -l", "expect": "no-match" }
    ]
  }
]

No magic. Each claim carries its own reproduction plan. If a model can't produce evidence, that's a signal, not a bug.

Step 3: Run the gate

Here's the runner I use. It's a template, not a tested product — keep that in mind before you wire it into CI.

#!/usr/bin/env bash
set -euo pipefail
REVIEW_JSON="${1:-review.json}"

pass=0
fail=0
skip=0

while read -r finding; do
  id=$(jq -r '.id' <<< "$finding")
  claim=$(jq -r '.claim' <<< "$finding")
  count=$(jq '.evidence | length' <<< "$finding")
  result="skip"

  for ((i=0; i<count; i++)); do
    cmd=$(jq -r ".evidence[$i].cmd" <<< "$finding")
    expect=$(jq -r ".evidence[$i].expect" <<< "$finding")

    if [[ "$expect" == "match" ]]; then
      if eval "$cmd" >/dev/null 2>&1; then result="pass"; else result="fail"; break; fi
    else
      if eval "$cmd" >/dev/null 2>&1; then result="fail"; else result="pass"; fi
    fi
  done

  case "$result" in
    pass) pass=$((pass+1)); echo "[PASS]     $id : $claim" ;;
    fail) fail=$((fail+1)); echo "[FAIL]     $id : $claim" ;;
    *)    skip=$((skip+1)); echo "[NO-INFO]  $id : $claim" ;;
  esac
done < <(jq -c '.[]' "$REVIEW_JSON")

echo "---"
echo "Verified: $pass | Contradicted: $fail | Unverifiable: $skip"

Two warnings before you run it:

1. eval on evidence commands is dangerous. For a real CI gate, restrict it to an allowlist of commands, or parse the evidence into a safe runner. Don't copy this verbatim.
2. no-match evidence means "the absence of a pattern" — absence isn't proof of a vulnerability. It's a triage flag, not a verdict.

What the scoreboard means

Gate result	Meaning	Your action
PASS	The failure reproduced in a pinned environment	Fix it, then keep the test as a regression gate
FAIL	Evidence contradicts the claim	Treat as unconfirmed; do not refactor on top of it
NO-INFO	No reproducible path was provided	Manual review; no fix gets merged without a repro

That last row is the one teams skip. "It's a free model, let's just fix what it found." No. If there's no repro, there's no vulnerability — there's a paragraph.

Who should not use this approach

This gate is for development-time triage. It is not a compliance artifact.

If you're in a regulated environment, or the asset under review is a legal or customer-data boundary, don't route findings through a free tier at all — the review itself becomes evidence, and you can't audit a server you don't operate. Also, my review.json examples above are unexecuted templates. I applied this workflow to toy fixtures, not a production codebase, so run your own drill before trusting the pattern.

The gate also fails silently for whole classes of bugs: logic errors with no observable command output, race conditions, business-rule violations. Absence of a repro doesn't mean absence of a bug.

The boundary question

Free model, free server, free tokens — none of that changes the cost of a false finding. The gate is cheap. The unpinned, unverified fix is expensive.

Now ask yourself: which of these invariants belongs in your CI, and which ones should your prompt design enforce before the claim ever exists? And when the next zero-cost review arrives with a confident CVE citation, will your reviewer be able to say "show me the repro" — or just "trust me"?