DEV Community

Yalakala-Naveen
Yalakala-Naveen

Posted on

My agent said 'done'. Nothing had happened.

The run completed. The status said done. The summary was confident, specific, and completely untethered from reality: it reported a batch of emails as sent while one of them sat bounced, asserted by the same component that did the sending, checked by nobody.

Nothing crashed. That's the point of this article. Agent failures that throw exceptions are the easy kind — they page you. The failures that cost me real debugging days in production are the ones where every signal says success, because the agent graded its own homework and the output document was never asked to prove a single claim in it.

I'm a Forward Deployed Engineer, and after the third or fourth time this shape bit me, I started writing the shapes down. There are six. I call them the silent failures, and I eventually built the checker for them — FailLoud, my own open-source project, so disclosure up front: FailLoud is a deterministic verification harness for LLM agent outputs — it catches the silent failures agents ship as "done." No LLM in the verdict path. CLI for CI (exit 0 pass / 1 failed output / 2 tool error), Python API, and an MCP server that verifies a sub-agent's output inside the loop, before you act on it.

Here is the taxonomy. If you run agents, you have met at least three of these.

FL001 — placeholder_shipped

The output is declared done while the payload is empty or still holds draft scaffolding: lorem ipsum, unrendered {{template}} braces, "coming soon." A page once went live — launch marked complete — with placeholder copy still in the body. Every existence check passed. The file was there. It just contained nothing.

The tell: the deliverable exists and is hollow.

FL002 — unbacked_claim

The output claims an action — sent, posted, published, updated — and nowhere in the document is there evidence it happened: no message id, no URL, no receipt, no artifact. This is the bounced-email failure. "Sent" was a sentence the agent wrote about itself.

The tell: a claim with no receipt. In my experience this is the most common silent failure and the most expensive, because downstream steps build on it immediately.

FL003 — stale_state

The work is flawless and the world it describes is gone. The run planned from a state snapshot older than allowed — in one output I kept, the state timestamp was 77.1 hours old against a 24-hour limit — or the state carried no freshness timestamp at all, or the state version the run was handed (9) disagrees with the version it carried (7). Nobody checks the snapshot; everybody checks the plan.

The tell: correct work, dead context.

FL004 — count_mismatch

The summary says 12. The itemized record in the same document contains 9. Reports count actions attempted; audiences read them as actions completed; and nobody ever reconciles the number against the list, because both are sitting in the same JSON and the JSON "looks fine." Only the list was ever true.

The tell: the count and the items live in the same document and disagree.

FL005 — missing_components

Every required field is present — and one of them is empty. Or a segmented deliverable (a video, a multi-part report) contains a zero-length segment, or a silent gap where content should be. A delivered video once passed every check we had while one segment inside it was literally silent. Presence is not completeness.

The tell: the slot is occupied and contains nothing.

FL006 — unsupported_answer

The RAG failure, and the newest of the six — it shipped in 0.6.0 because practitioners kept posting the same complaint: the system retrieves, it cites, and the answer is still wrong. FL006 checks the parts of grounding that can be checked deterministically:

  • every citation must resolve to a source that was actually retrieved — a citation pointing at nothing is fabricated, however confident it looks
  • every cited source must carry text — an empty source backs nothing
  • every number in the answer must appear in the sources (1,200, 1200 and 1200.0 are the same number)

On the example in the repo:

$ failloud check examples/rag_failing.json --only FL006
Verdict: FAIL (4 error(s), 0 warning(s))
Enter fullscreen mode Exit fullscreen mode

Two fabricated citations, two invented figures. What FL006 deliberately does not claim: semantic entailment. It will not judge whether the prose follows from a source. No model sits in the verdict path, and the moment one does, you're back to grading homework with the class's most confident student.

The demo that makes it real

The repo ships a realistic agent output that commits the first five failures at once, and its corrected twin:

$ failloud check examples/failing_output.json
FAILLOUD — agent output verification
Verdict: FAIL (8 error(s), 1 warning(s))
$ failloud check examples/passing_output.json
Verdict: PASS (0 error(s), 0 warning(s))
Enter fullscreen mode Exit fullscreen mode

Every finding carries a JSON path and the numbers behind it — $.counts.sent — reported as 12 but the itemized record contains 9 item(s). "Something seems off" is not a finding. A path and a number is.

You can run the whole thing in your browser first — paste an agent output, get the failure report, nothing uploaded: the FailLoud Playground.

The hole a reader found, and the second pass

When I first wrote about this, a reviewer pointed at the real hole: an output can be consistently wrong. "9 sent," nine items listed, an id on every claim — and one of those messages still bounces an hour later. At completion time, every internal check says PASS. The world disagrees afterwards.

They were right, so a FailLoud PASS is now labelled provisional: it means "this document does not contradict itself," not "this happened." The second pass, failloud recheck, re-judges the original claims against a later snapshot from the system of record — a provider delivery export, a database extract:

$ failloud recheck examples/second_pass_original.json examples/second_pass_record.json
FAILLOUD — second-pass verification
First pass: PASS (provisional)
Second pass: CONTRADICTED

  ✗ email_sent → reader09@acme-digest.com — evidence 'msg_8f09' appears in a
    failure context (bounced / failed / rejected) in the system of record.
  ✗ delivered — reported as 9 but the record shows 8.
Enter fullscreen mode Exit fullscreen mode

Every claim comes back confirmed, contradicted, or unverifiable — and a record that never mentions the evidence has not confirmed it. Silence is not upgraded into a pass.

What this doesn't do

FailLoud never asks a model whether an output "looks good." It produces no quality scores, no faithfulness percentages, no dashboards. If you need scored judgements over test sets, DeepEval and Ragas are built for that and I'll say so in their hearing. FailLoud answers a narrower, more embarrassing question: did the thing the agent said happened, actually happen — and does its own output prove it?

It's a young project — first release was November 12, 2026, 162 tests, zero-dependency core, MIT, and I can't point you at a user base yet. What I can point you at is the taxonomy, because it came from production and not from a brainstorm. If your agents have shipped a seventh silent failure, I want its shape: the checks in this tool exist because a real agent failed that way first.

Next in this series: putting the gate in CI so a bad agent output fails the build like a failing test — and then the MCP version, where the verifier sits inside the loop.

Top comments (1)

Collapse
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

FL002 is the one that stings most — "sent" as a sentence the agent wrote about itself, with no message id anywhere. The count-mismatch one (FL004) is almost worse though, because the summary and the itemized list sit in the same document and nobody reconciles them. Do you run FailLoud as a gate before downstream steps consume the output, or as a post-hoc audit?