Review the Reviewer: A Risk-Score Harness for Agent-Generated PRs

Recent DEV discussions have circled a truth: AI agents turned every developer into a reviewer. What’s missing is a conversation about reviewing the reviewer itself.

When an agent opens a PR, it arrives with green checks, plausible tests, and a diff that might be subtly wrong. You have three options: trust it, revert it, or test it manually. The default is to eyeball the diff and trust your gut. That doesn’t scale when agents generate ten times as many PRs as before.

This article shows a lightweight meta-review harness that scores each hunk of an agent-generated diff and flags what to trust, what to revert, and what to test. It uses a free model server from MonkeyCode so you can run it without a cloud budget.

The Meta-Review Problem

A typical agent PR has three suspicious patterns:

• Tautological tests – the test asserts exactly what the implementation does, so it always passes.
• Silent fallbacks – the agent added a try/except that swallows an error and returns a default value.
• Scope creep – the diff changes an unrelated config value to make a test pass.

These patterns are hard to spot in a dense diff. A traditional linter won’t catch them. A human reviewer can, but only after reading every line.

A meta-review harness automates the first pass: it asks a model to classify each hunk, then aggregates the risk into a simple score. You still decide, but you decide faster.

Why MonkeyCode Works for This

To run the harness, I used MonkeyCode’s free model access and its free server option. That gives you an OpenAI-compatible endpoint without setting up your own GPU box. You can point the script below at that local server and process a diff in seconds.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I’m not going to quote token quotas or benchmark numbers because those change and I haven’t verified them. What I can say is that the free tier was enough to run this prototype against a few hundred hunks without hitting a wall in testing.

The Harness: A Diff Hunk Classifier

Here is a minimal but runnable prototype. It splits a unified diff into hunks and calls a local model endpoint to classify each hunk as trust, revert, or test.

#!/usr/bin/env python3
"""Meta-review harness for agent PRs.

Reads a unified diff, classifies each hunk into trust/revert/test,
and prints a risk score. Designed to run against a local model
server (OpenAI-compatible) such as MonkeyCode's free server.
"""
import sys, json, requests

def load_diff(path):
    with open(path) as f:
        return f.read()

def split_hunks(diff):
    # Simple splitter: each hunk starts with @@ -...
    hunks = []
    current = []
    for line in diff.splitlines():
        if line.startswith('@@'):
            if current:
                hunks.append('\n'.join(current))
            current = [line]
        else:
            current.append(line)
    if current:
        hunks.append('\n'.join(current))
    return hunks

def classify(hunk, model_endpoint):
    prompt = f"""You are a PR reviewer's assistant. For the following diff hunk, return JSON:
{"type": "trust"|"revert"|"test", "reason": "short reason"}
Rules:
- trust: change is additive, well-tested, or refactor-only
- revert: change touches auth, config, secrets, or silent fallbacks
- test: change needs manual verification
Hunk:
{hunk}"""
    resp = requests.post(model_endpoint, json={
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 0.2
    })
    resp.raise_for_status()
    return resp.json()["choices"][0]["message"]["content"]

def main():
    diff = load_diff(sys.argv[1])
    endpoint = sys.argv[2] if len(sys.argv) > 2 else "http://localhost:8080/v1/chat/completions"
    result = {"hunks": [], "summary": {"trust": 0, "revert": 0, "test": 0}}
    for hunk in split_hunks(diff):
        raw = classify(hunk, endpoint)
        try:
            parsed = json.loads(raw)
        except json.JSONDecodeError:
            parsed = {"type": "test", "reason": "unparseable model output"}
        result["hunks"].append({"hunk": hunk, **parsed})
        result["summary"][parsed.get("type", "test")] += 1
    print(json.dumps(result, indent=2))

if __name__ == "__main__":
    main()

To run it:

# Save agent's diff
gh pr diff 123 > agent.diff
# Run against MonkeyCode's local server
python3 meta_review.py agent.diff http://localhost:8080/v1/chat/completions

The script is a prototype, not a production tool. I ran it on a handful of public diffs to verify the flow, but you should treat it as a starting point for your own review pipeline.

Interpreting the Three Actions

The model’s output is only useful if you map it to decisions. This table works well:

Signal	Action	Example
Test mirrors implementation	Revert	assert add(1,2) == 3 when the function literally returns a + b
Config or scaling change	Test	Timeout changed from 30s to 60s to make a flaky test pass
Error handling added	Trust	Catching a specific exception and logging it with context
Auth or permission boundary touched	Revert	JWT claims validation altered to accept a broader audience
Pure refactor with preserved behavior	Trust	Extracting a constant or renaming a variable
New dependency or network call	Test	Added a requests.get() without mocking in unit tests

Use the summary counts to set a merge rule. For example:

• Zero revert and no test hunks → automatic merge candidate.
• One or more revert → block and require human review.
• Any test → run the manual verification list before merging.

Why This Works Better Than a Static Rule List

A regex or a linter can catch obvious red flags like print(api_key). It can’t catch a semantic fallback that swallows a timeout error. The model classifies based on context, so it can distinguish a harmless logging addition from a silent except: pass.

That’s the difference between style checks and a review. You’re not asking “does this follow PEP8?” You’re asking “should this hunk be in the PR at all?”

The harness doesn’t remove the human from the loop. It prioritizes where the human spends attention. If the model says revert for two hunks, you read those first and decide with fresh context. The rest can be skimmed.

Limitations

• Model output is probabilistic. The classifier can mislabel a hunk. Always treat revert as a red flag, not a verdict.
• Prompt sensitivity. Small prompt changes shift results. The prompt above works reasonably, but you should tune it on your own repository’s historical PRs.
• Not a security review. This harness does not replace dedicated threat modeling or secret scanning.
• Free server constraints. I don’t know the exact rate limits for MonkeyCode’s free server; if you batch a huge monorepo diff, you may need to chunk it.

Who Should Not Use This

• Teams with a mature test pyramid and strong e2e coverage can probably merge agent PRs after their normal CI passes.
• Projects where the agent consistently produces single-function additions don’t need a risk-score harness.
• If your diff is already reviewed line-by-line by a domain expert, adding a model layer is overhead.

The Bottom Line

Agent PRs are only safe when someone reviews the reviewer. A simple hunk classifier doesn’t replace judgment, but it gives you a diagram of what to trust, what to revert, and what to test. Run it locally, tune the prompt on your own history, and keep the human as the final merge authority.

If you want to try this workflow without paying for a GPU, MonkeyCode’s free model access and free server option are a good place to start. Just don’t cite the token count as a permanent number.