Agents report success for actions that never happened — no error, clean trace, and every observability tool reads it as a success, because they're all reading the agent's own account of itself.
This doesn't read the trace differently. It queries the authoritative system instead — the actual Gmail mailbox — and returns confirmed / contradicted / inconclusive with the evidence attached: the account queried, the search window, and what was found in it.
Two failure classes, different evidence. If there's no tool call in the trace at all, the absence is the evidence and nothing external is needed. If the call happened and returned cleanly but nothing landed, only the mailbox can tell you.
The parts I'd most want criticised:
Inconclusive is a first-class verdict. If the trace can't be shown complete, or the tool span has a name I don't recognise, or the mailbox can't be established — it refuses to judge rather than calling a working agent a liar. A false accusation costs more than a missed detection. I may have that balance wrong.
It cannot catch a correct call with a wrong argument. Asked to mail one person, confidently mails another — the send is real, Gmail confirms it, verdict is confirmed, correctly. The instruction is recorded next to the arguments so a human can see it. No verdict catches it.
Gmail only so far. LangChain via a callback handler, or the SDK by hand. No n8n, no CrewAI. Zero users — this is day one.
matrixverify.dev
Top comments (2)
Keeping inconclusive as a first-class verdict is the right call. The quickest way an observability tool gets turned off is crying hallucination whenever an upstream token expired or a telemetry span got dropped. Treating missing evidence as unknown rather than failure preserves trust in the alerts that actually fire.
The semantic argument mismatch is the nastier edge case. When an agent emails the wrong recipient or overwrites the wrong file, the external API will happily return a 200 with valid metadata. In our pipelines, the only check that caught those without running another judge model was extracting the explicit target entities from the initial user prompt and asserting that the tool arguments had non-empty overlap with that set before checking external confirmation.
The overlap check is the part I hadn't got to, and it's cheaper than I assumed — I'd been treating the wrong-argument case as needing a judge model, which is why it's still on my limits page.
I already capture the instruction alongside the tool arguments, and I extract concrete identifiers out of prose for a different check. So the pieces are there; what's missing is asserting the overlap between them before the external read.
One thing I'd want to get right: an empty target set in the prompt. "Reply to this ticket" names nobody, and the recipient legitimately comes from the ticket rather than the instruction. Zero overlap there isn't a mismatch, it's an instruction that never specified one. Did you end up distinguishing those, or did you only run the check where the prompt named a target explicitly?