PR Descriptions Lie: A Diff-Truth Checker Built on Free AI Models

AI-generated pull request descriptions are dangerously fluent. They say Refactors auth middleware to use scoped tokens while the diff quietly renames two variables and bumps a dependency. The writer never sees the code; the reviewer rarely checks the narrative against the diff line by line. That gap is where review standards decay.

You can close it without paying for enterprise tooling. Using MonkeyCode's free model access for extraction and its free server for orchestration, I built a small diff-truth checker that flags claims in PR descriptions that the actual code change does not support. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Check the current MonkeyCode README for exact quota details before you rely on them in a busy repo.

What the Checker Does

The tool follows three steps on every pull request:

1. Ask a free model to list explicit behavioral claims from the PR description (e.g., adds retry logic, renames endpoint).
2. Parse the diff with lightweight heuristics and an AST where possible.
3. Compare the two lists and emit a mismatch report.

The goal is not to judge whether the PR is good. It is to give reviewers a focused list of statements that need extra verification.

Step 1: Extract Claims with Free Models

You send the description plus the diff summary to MonkeyCode's free model endpoint. A strict prompt keeps output parseable:

Given the pull request description below, extract every factual claim about what the code does.
Output JSON array with fields: "claim", and "subject" (file, function, or module).
Ignore tone, motivation, or roadmap statements.
Only output JSON.

A tiny Python client runs that prompt:

import json, requests

def extract_claims(description: str, diff_summary: str) -> list:
    prompt = f"""Given the PR description and diff summary, extract factual claims.\n\nDescription: {description}\n\nDiff summary: {diff_summary}"""
    # MonkeyCode free model endpoint; adapt to current SDK docs
    resp = requests.post("https://api.monkeycode.example/v1/chat", json={
        "model": "free-model",
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 0
    }, timeout=30)
    data = resp.json()
    return json.loads(data["choices"][0]["message"]["content"])

This step is intentionally cheap. If the description is one paragraph, you burn only a few hundred tokens.

Step 2: Verify Claims Against the Diff

Verification does not need another model call. A small parser can handle the most common patterns. For example, to check a claim like adds retry logic:

import re

def verify_claim(claim: str, full_diff: str) -> bool:
    if "retry" in claim.lower():
        return bool(re.search(r"retry|backoff|attempt", full_diff, re.I))
    if "rename" in claim.lower():
        return bool(re.search(r"renamed?\s+\w+", full_diff, re.I))
    if "remov" in claim.lower():
        return bool(re.search(r"^-\s*\w+", full_diff, re.M))
    return True  # unknown patterns pass to human review

You can extend this table with your team's common verbs. The point is deterministic, non-LLM checks: no hallucinations, same result every run.

Step 3: Run It on the Free Server

MonkeyCode's free server gives you a place to run this validator as a scheduled job or a webhook. A minimal FastAPI app accepts PR payloads:

from fastapi import FastAPI, Request
app = FastAPI()

@app.post("/check")
async def check_pr(request: Request):
    payload = await request.json()
    desc = payload["description"]
    diff = payload["diff"]
    claims = extract_claims(desc, diff)
    mismatches = [c for c in claims if not verify_claim(c["claim"], diff)]
    return {"mismatches": mismatches}

Deploy that to the free server, point your GitHub webhook at it, and you get a report on every PR without burning your laptop battery.

Decision Table: What to Automate vs. What to Escalate

Claim type	Can the checker validate?	Reviewer action
adds a function named X	Yes, via regex/AST	Fine if found
fixes memory leak in cache	Hard, needs profiling	Manually inspect
renames endpoint /users	Yes, diff search	Fine if found
improves performance by 20%	Impossible from diff	Ask for benchmark data
updates dependency	Yes, check package files	Fine if found

This table keeps the tool honest: it automates only what code diff can prove.

Limitations You Should Own

• Free model output may not always be valid JSON. Add a retry or fallback parser.
• The verification step is pattern-based, so it misses claims phrased in unusual ways. That is fine; the goal is catching the obvious lies, not all possible ones.
• Free server resources are not infinite. For a high-volume monorepo, you may need to batch check or move to self-hosted after verifying value.

Who Should Not Use This Approach

If your compliance team treats AI-generated analysis as evidence, stop here. This checker is a reviewer aid, not an audit tool. Also, if your repository is confidential and policies block external API calls, do not send full diffs to any cloud model. Run the extraction with a local model instead; the verification step stays the same.

Try It, Then Decide

The point of this exercise is not to replace human review. It is to make the first pass boring so the second pass can be sharp. With MonkeyCode's free models doing the extraction and a free server hosting the endpoint, you can run this experiment for exactly zero dollars and see whether it catches one misleading PR in your next sprint.

Review with your eyes, not with vibes. Give the robot the part that involves line counting, and keep the judgment for yourself.

Comments

[Luis Cruz]

The implementation of the diff-truth checker is a clever use of free AI models, particularly in how you extract claims and validate them against the actual code changes. This approach effectively minimizes human error during PR reviews, which can often lead to missed details. One potential improvement could be expanding the verification logic to include more nuanced checks based on the context of claims, as some alterations might be more complex than simple keyword matches. If you're considering enhancing this tool or need additional hands on the next phase of development, I’d be glad to explore a paid collaboration. How do you envision iterating on this project to incorporate more complex verification scenarios?