The launch was marked complete. The page was live. And the body of the page still held placeholder copy — the agent had shipped its own scaffolding and reported success, because nothing in its pipeline ever checked the output against the claim.
Another run reported a batch as sent. A bounce sat unrecorded. "Sent" had been asserted by the same component that did the sending, and nobody — human or machine — asked it for a message id.
I'm a Forward Deployed Engineer. I run AI agents in production, and these weren't model failures. The models did fine. The failures were silent: outputs that said done while the work was placeholder, unbacked, stale, miscounted, or missing pieces — and every dashboard stayed green because the agent graded its own homework.
So I wrote down the five ways I'd actually seen agent output lie, and built a checker for exactly those:
| Check | Name | What it catches |
|---|---|---|
| FL001 | placeholder_shipped |
"Done" with an empty payload, lorem ipsum, or unrendered {{template}} braces |
| FL002 | unbacked_claim |
"Sent / posted / published" with no message id, URL, receipt, or artifact behind it |
| FL003 | stale_state |
Work based on state older than allowed, untimestamped, or version-skewed |
| FL004 | count_mismatch |
The report says 12, the itemized record lists 9 — only the list was ever true |
| FL005 | missing_components |
Fields present but silently empty; dead stretches inside a deliverable |
FailLoud never asks a model whether an output "looks good." Every verdict is deterministic, reproducible, and explainable line by line. No LLM sits in the verdict path. It runs three ways: a CLI for CI (exit 0 pass, 1 failed output, 2 tool error), a Python API, and an MCP server so an orchestrator can verify a sub-agent's output inside the loop, one tool call before acting on it.
The repo ships a realistic agent output that commits all five failures. Here's what it looks like:
$ pip install failloud
$ failloud check examples/failing_output.json
FAILLOUD — agent output verification
Verdict: FAIL (8 error(s), 1 warning(s))
[FL001] placeholder_shipped — FAIL
✗ $.output.body — Placeholder content shipped as final output:
matched 'coming soon', 'lorem ipsum', '{{', '}}'.
[FL002] unbacked_claim — FAIL
✗ $.claims[1] — Claimed action 'card_published' has no evidence —
no id, URL, receipt, or artifact backs that it happened.
[FL004] count_mismatch — FAIL
✗ $.counts.sent — Count for 'sent' is reported as 12 but the
itemized record contains 9 item(s).
Or in Python:
import json
from failloud import check_output
result = check_output(json.load(open("agent_output.json")))
if not result.passed:
for finding in result.errors:
print(finding.check_id, finding.path, finding.message)
The critique that made it better
When I posted the launch, a reader pushed on exactly the right weak point: a claim like "9 sent" can pass every check at run time — and one of those sends can still bounce later. A green verdict at minute one says nothing about minute sixty.
He was right, and it changed the design. In 0.5.0, a PASS is now labelled provisional. A new command, failloud recheck, takes the original claims and re-judges them against a later snapshot of the system of record. Every claim comes back confirmed, contradicted, or unverifiable. Verification isn't a moment anymore; it's a second pass, against the record, after reality has had time to answer back. (Current release: 0.5.1 — same checks, now also published in the official MCP Registry.)
Try it before you install anything
The Playground runs the checks in your browser — paste an agent output, get the failure report, nothing uploaded:
https://yalakala-naveen.github.io/failloud/
If it's useful: pip install failloud (add [mcp] for the server). Code, examples, and the full check reference: https://github.com/Yalakala-Naveen/failloud — MIT licensed, zero-dependency core, 147 tests.
Honest closing: this just shipped and has no users yet. If you run agents in production, I want the failure mode my five checks miss. That's the most useful thing you can hand me — it's already happened once, and it shipped as 0.5.0.
Top comments (1)
The shift from a one-time PASS to provisional verification is particularly important. A successful API response or completed workflow only proves that an operation was accepted at that moment; it doesn't necessarily prove the business outcome persisted.
I'd extend this with claim-specific verification windows. A page deployment might be checked immediately for rendered content, while an email campaign needs a later reconciliation against delivery and bounce events. Each claim should have its own evidence source, verification deadline, and explicit unverifiable state rather than allowing an early PASS to become permanent truth.
At IT Path Solutions, this distinction matters when stabilizing AI-generated applications and production agent workflows. The agent shouldn't be the authority on whether its own work succeeded; the system of record should provide that evidence.
The idea of treating "done" as a claim that must be independently verified is a much stronger foundation for production reliability than another layer of agent self-review.