I Built an AI to Fact-Check My Other AI. For Two Weeks It Approved Everything — Including a Draft I Deliberately Sabotaged.

After my agent emailed 40 customers about work that wasn't done, I built a second AI to check the first one's output before anything ships. The idea was simple: nothing leaves my Raspberry Pi without passing a review step.

For the first week, the reviewer approved everything. 100% pass rate. I felt like a genius — until I fed it a deliberately broken draft as a test. It said "looks good, ship it."

My safety net was a rubber stamp. This is the post-mortem of how I built it wrong, and what actually made output validation work.

The setup: why a validator at all

I run a small one-person operation where AI agents do real work: drafting customer replies, generating reports, preparing content. The failure that started all this: an agent confidently reported a 3-hour task as complete, and my pipeline happily sent "done!" emails to customers. The task had crashed 40 minutes in.

The root problem wasn't the model. It was that nothing in my system distinguished "the agent says it's done" from "it's actually done." Those are two very different claims, and I was treating the first as proof of the second.

So I added a gate: before any customer-facing output leaves the box, a second LLM call reviews it against the original task and the evidence the agent produced. If the review fails, the output is held and I get a message on my phone.

Conceptually it's the oldest idea in software — code review, but for prompts. In practice, my first version was worse than no review at all, because it gave me false confidence.

Failure #1: "Review this" is not an instruction

My original validator prompt was essentially:

You are a helpful reviewer. Check the following draft response for
quality and correctness. Reply APPROVE or REJECT with a reason.

TASK: {task}
DRAFT: {draft}

Clean, reasonable, useless. The model had no criteria, no context about what "correct" meant, and a strong prior toward being agreeable. Given a fluent, confident-sounding draft, it approved. Every time. LLMs are people-pleasers by default — if you ask "is this good?" they will find reasons to say yes.

What fixed it was making the validator adversarial and specific. Instead of "check for quality," I gave it a rubric with concrete, checkable claims:

You are auditing a draft before it is sent to a real customer.
Assume the draft is WRONG until the evidence proves otherwise.
Your job is to find the reason to reject it.

Check each item. For each, quote the exact line from the EVIDENCE
that supports it. If you cannot quote supporting evidence, the item FAILS.

1. Does the draft claim any task was completed? Is that completion
   visible in the evidence (logs, file listings, API responses)?
2. Does the draft contain any number, date, or price? Does each one
   appear verbatim in the evidence?
3. Does the draft promise any future action or deadline? Is that
   authorized in the task instructions?
4. Does the draft mention any product, feature, or policy NOT present
   in the task instructions?

EVIDENCE:
{logs, tool outputs, file diffs}

TASK INSTRUCTIONS:
{task}

DRAFT:
{draft}

Output JSON: {"verdict": "APPROVE"|"REJECT", "failed_checks": [...], "quotes": [...]}

Three changes matter here:

1. "Assume it's wrong until proven otherwise" flips the model's agreeable default. It's now looking for rejection reasons, which is the posture you want from a gate.
2. "Quote the exact line from the evidence" is the killer feature. A model can hand-wave an approval; it cannot fabricate a verbatim quote from a log it was given (well — it can, which is failure #3 below). Forcing quotes turns vague review into grounded verification.
3. Structured JSON output means my pipeline can act on the verdict programmatically, and failed_checks tells me which rule fired when I get a rejection alert at 2 AM.

After this change, the validator caught the exact class of bug that started this: drafts claiming completion with no completion visible in the evidence.

Failure #2: the validator hallucinated quotes

About a week in, I got an APPROVE on a draft containing a delivery date that appeared nowhere in the evidence. When I inspected the validator's output, it had "quoted" a log line supporting the date. The log line didn't exist. The model invented evidence to justify its verdict.

This is the part nobody warns you about with LLM-as-judge setups: your judge can commit the same sins as your defendant.

The fix was dumb and mechanical, which is why it worked: I stopped trusting the quotes. My pipeline now takes every quote the validator returns and does a literal substring check against the evidence blob. If a quote isn't found verbatim, the verdict is automatically downgraded to REJECT and flagged as "validator hallucination."

Roughly 1 in 20 approvals failed this substring check in the first month. Every single one was a catch I would have shipped otherwise. A ten-line Python function did more for my output quality than any prompt tweak.

Lesson: when an AI checks an AI, verify the checker with code, not with another AI. Turtles don't go all the way down — they bottom out at if quote in evidence:.

Failure #3: I validated things that shouldn't have been validated

Emboldened, I put the validator gate on everything: internal reports, log summaries, my own daily digest. Two problems showed up fast.

Cost and latency. Every output now required a second, long-context LLM call. My nightly report job went from 90 seconds to 6 minutes, and my token spend roughly doubled for outputs only I would ever read.

Alert fatigue. The validator, tuned to be adversarial, rejected internal drafts for "unverifiable claims" that didn't matter — a summary saying "traffic seems up this week" got rejected for lacking a quoted metric. Within two weeks I was ignoring its alerts, which is the exact state the gate was supposed to prevent. A safety mechanism you ignore is worse than none, because it also has a maintenance cost.

The rule I landed on: validate at the boundary, not everywhere. The gate only runs on output that crosses a trust boundary — anything a customer sees, anything that spends money, anything that sends a message. Internal artifacts get a cheap heuristic check (schema validation, length sanity, forbidden-string scan) instead of a full LLM review. Rejections on boundary outputs page me; everything else goes in a log I read weekly.

What the system looks like now

The full pipeline on the Pi:

1. Agent does the task and must write an evidence file — raw tool outputs, logs, diffs. "I completed it" is not evidence; artifacts are.
2. Boundary check: is this output customer-facing, money-moving, or message-sending? If no → cheap checks, ship internally.
3. If yes → adversarial validator with the rubric above, JSON verdict.
4. Substring-verify every quote against the evidence blob. Hallucinated quote → auto-REJECT.
5. REJECT → hold the output, push an alert with failed_checks to my phone. APPROVE → send, and archive the draft + evidence + verdict together (this audit trail has saved me twice when a customer disputed what we told them).

Current numbers: the gate reviews ~60 outputs a week, rejects 4–6, and of those rejections about half are real catches (the rest are over-strict rubric hits I've been tuning out). The real catches include one draft quoting a retired pricing tier and one claiming a refund was processed when the API call had silently failed. Either would have cost me a customer and an afternoon of cleanup. The gate has paid for itself many times over in token costs alone.

The honest summary

• An LLM validator with a vague prompt is theater. It will approve everything and make you feel safer while doing it.
• Adversarial framing + forced evidence quotes turns review into verification.
• Verify the verifier with plain code. Substring checks beat trust.
• Gate the boundaries, not everything. A noisy safety system gets ignored, then removed.
• The deepest fix wasn't the validator at all — it was making the working agent produce evidence as a first-class artifact. Once completion claims had to come with receipts, half the validation problem disappeared before the judge ever saw it.

If you're running agents that touch customers or money, I'd urge you to test your safety nets the way I should have on day one: feed them something deliberately broken and watch what they do. A gate you've never seen fail is a gate you can't trust.

The validator rubric above is one of the patterns I keep refining, and I write up the specific prompts — validation, evidence collection, escalation, the whole boundary-gating setup — in The Agent Prompt Vault — $3, lifetime updates. Steal the ones that fit your workflow.

Comments

[Reid Marlow]

The requirement to cite exact log strings in the validation step matches what we ended up with. When the reviewer has to ground its verdict on verbatim strings from the tool receipts, the agreeable default drops off immediately. The second failure mode that showed up for us was validator hallucination on missing evidence, where the model would accept an action because the wording felt familiar even without the log line. Enforcing a hard rejection when the quote field returns empty was the only piece that made our pipeline reliable.