DEV Community: Aditya Mishra

A silent failure costs you $2,500–8,000 and 6–15 unbillable hours. Here's how to stop finding out three weeks late.

Aditya Mishra — Sun, 04 Oct 2026 09:27:17 +0000

Those numbers aren't mine. They're from a consultant in this community who's cleaned up after them. He also said the part that stings: that diagnosis time is unbillable. You eat it.

Three threads here this month, same shape:

28 consecutive failures running three days before anyone looked. A workflow scoring every lead as zero for six weeks. An agent telling a customer "I can see you were charged" after both lookups came back 403 — run finished COMPLETED.

None of them were red. That's the whole problem. Your error workflow fires on exceptions. These throw nothing.

What you're actually paying for right now

Time to notice: 48 hours to 3 weeks. Usually a client tells you.
Time to diagnose: 6–15 hours digging through executions, unbillable.
Cost of the incident itself: $2,500–8,000.

Per incident. Four a year and that's $10,000–32,000 plus 60 hours you can't charge for.

What I built

Matrix takes what the run claimed — "email sent", "record updated" — and reads the authoritative system to check whether it actually happened. Not the execution log, which the run wrote itself. The actual mailbox.

Three verdicts: contradicted, confirmed, inconclusive. Evidence attached to each.

Two of the three checks need no access to anything:

The claim was made and no tool was ever called — the absence is the evidence.
A lookup returned 403 and the run then stated what that lookup would have shown. The call completed; the result was a refusal. Most wrappers collapse those.

The third needs a connected mailbox: the tool was called, returned 200, the record isn't there.

Why inconclusive matters

If it can't prove the trace was complete, or a tool has a name it doesn't recognise, it refuses to judge rather than calling your working workflow a liar. A monitor that resolves its own uncertainty optimistically is worth nothing on the day it matters.

What it doesn't do

Gmail is the only external adapter. It can't catch a correct call with a wrong argument — right recipient format, wrong recipient, real send, Gmail agrees. And a false claim in text that was never sent has no outcome to read back against.

Setup

One line pasted into Cursor or Claude Code and it wires itself. Or four calls by hand, TypeScript or Python.

matrixverify.dev — free, 20+ users, and I'd rather it got broken than ignored.

Happy to run it against one of your executions and tell you what it finds, including if it finds nothing.

Your execution says success. Your database says nothing changed. Which one do you believe?

Aditya Mishra — Tue, 29 Sep 2026 12:24:35 +0000

Three threads here this month have circled the same gap: a green execution over work that never happened. The answers are all correct and all the same shape — RETURNING id, an If node, throw. Wire it per step, per workflow.

I got tired of wiring it per step, so I built the version that runs once.

What it does

Takes what the run claimed — "email sent", "order updated" — and goes and reads the authoritative system. Not the execution record, which the run wrote itself. The actual mailbox.

Three verdicts, never two:

contradicted — the claim and the world disagree
confirmed — every property in the claim was checked and holds
inconclusive — something couldn't be established, so no judgement

That third one is the whole design. If the trace can't be shown complete, or a tool has a name it doesn't recognise, or the mailbox can't be identified — it refuses to judge rather than calling a working workflow a liar. A monitor that resolves its own uncertainty optimistically is worth nothing on the day it matters.

Two of the three checks need no access to anything

The claim was made and no tool was ever called — the absence is the evidence, no external read needed.

A lookup returned 403 and the run then stated what that lookup would have shown. The call completed; the result was a refusal. Those are different things, and most wrappers collapse them.

Only the third needs a connected mailbox: the tool was called, returned 200, and the record isn't there.

A real one it found

Someone published a production trace at me. Two order lookups came back 403 — the agent got nothing. The final reply still said "I can see you were charged $29 twice on 3 October." Run finished COMPLETED. Nothing in the execution list looked wrong.

That trace produced nothing the first time I ran it. Four separate causes. Fixing them took two days and three of the four were things I'd assumed worked.

What it doesn't do

Setup

One line pasted into Cursor or Claude Code and it wires itself. Or four calls by hand, TypeScript or Python.

matrixverify.dev — free, zero users, and I'd rather it got broken than ignored. Happy to run it against one of your executions and tell you what it finds, including if it finds nothing.

When a silent failure hit you, what did it actually cost?

Aditya Mishra — Sat, 26 Sep 2026 04:31:54 +0000

Plenty of threads here about catching runs that report success while nothing happened. I've read most of them. What I can't find anywhere is the number.

So: when it happened to you —

How long between it happening and someone noticing?
What did it cost — refund, lost client, hours of digging, something else?
Did you end up building something for it, or decide it wasn't worth it?

Asking because someone in another thread said diagnosing this is billable work for a consultancy, so a tool that shortens diagnosis shortens the invoice. If that's true for most people here, that changes what's worth building.

Real numbers more useful than patterns. Even "it cost nothing, we caught it in an hour" is an answer.

Matrix – Check whether your AI agent actually did what it claimed

Aditya Mishra — Fri, 25 Sep 2026 04:45:55 +0000

Agents report success for actions that never happened — no error, clean trace, and every observability tool reads it as a success, because they're all reading the agent's own account of itself.

This doesn't read the trace differently. It queries the authoritative system instead — the actual Gmail mailbox — and returns confirmed / contradicted / inconclusive with the evidence attached: the account queried, the search window, and what was found in it.

Two failure classes, different evidence. If there's no tool call in the trace at all, the absence is the evidence and nothing external is needed. If the call happened and returned cleanly but nothing landed, only the mailbox can tell you.

The parts I'd most want criticised:

Inconclusive is a first-class verdict. If the trace can't be shown complete, or the tool span has a name I don't recognise, or the mailbox can't be established — it refuses to judge rather than calling a working agent a liar. A false accusation costs more than a missed detection. I may have that balance wrong.

It cannot catch a correct call with a wrong argument. Asked to mail one person, confidently mails another — the send is real, Gmail confirms it, verdict is confirmed, correctly. The instruction is recorded next to the arguments so a human can see it. No verdict catches it.

Gmail only so far. LangChain via a callback handler, or the SDK by hand. No n8n, no CrewAI. Zero users — this is day one.

matrixverify.dev

Matrix – Check whether your AI agent actually did what it claimed

Aditya Mishra — Tue, 15 Sep 2026 15:26:59 +0000

Agents report success for actions that never happened — no error, clean trace, and every observability tool reads it as a success, because they're all reading the agent's own account of itself.

The parts I'd most want criticised:

Gmail only so far. LangChain via a callback handler, or the SDK by hand. No n8n, no CrewAI. Zero users — this is day one.

I Built a Tool That Verifies What AI Agents Actually Did

Aditya Mishra — Sat, 12 Sep 2026 06:39:47 +0000

AI agents can report success for actions that never happened.
The trace looks clean, but observability tools often rely on the agent's own account of what happened.
So I built Matrix — a tool that checks the real system instead of trusting the agent.
For example, if an agent says it sent an email, Matrix checks the actual Gmail mailbox and returns:
CONFIRMED — it happened
CONTRADICTED — it didn't
INCONCLUSIVE — there isn't enough evidence
The verdict comes with the evidence behind it.
Gmail only for now. Zero users. Day one.
I'm building this because AI agents need a way to prove what they actually did — not just report that they did it.

Try Matrix:
https://matrix-snowy-beta.vercel.app⁠�

I'd love feedback from anyone building AI agents: How are you currently verifying agent actions?