Agents report success for actions that never happened — no error, clean trace, and every observability tool reads it as a success, because they're all reading the agent's own account of itself.
This doesn't read the trace differently. It queries the authoritative system instead — the actual Gmail mailbox — and returns confirmed / contradicted / inconclusive with the evidence attached: the account queried, the search window, and what was found in it.
Two failure classes, different evidence. If there's no tool call in the trace at all, the absence is the evidence and nothing external is needed. If the call happened and returned cleanly but nothing landed, only the mailbox can tell you.
The parts I'd most want criticised:
Inconclusive is a first-class verdict. If the trace can't be shown complete, or the tool span has a name I don't recognise, or the mailbox can't be established — it refuses to judge rather than calling a working agent a liar. A false accusation costs more than a missed detection. I may have that balance wrong.
It cannot catch a correct call with a wrong argument. Asked to mail one person, confidently mails another — the send is real, Gmail confirms it, verdict is confirmed, correctly. The instruction is recorded next to the arguments so a human can see it. No verdict catches it.
Gmail only so far. LangChain via a callback handler, or the SDK by hand. No n8n, no CrewAI. Zero users — this is day one.
Top comments (2)
@aditya_mishra_2417, confirmed, contradicted, and inconclusive is the right vocabulary for postcondition verification. The hard case is the one you name: a real side effect with the wrong recipient. I’d capture the expected recipient, time window, and content fingerprint as a typed claim before the call, then compare that claim with authoritative Gmail evidence; existence alone proves too little. In agent-inspect I’ve been separating execution traces from outcome evidence along the same boundary. Would Matrix model a retry after an inconclusive result as a new attempt under the same claim ID so it cannot accidentally double-send?
Matrix doesn't retry — it only observes, so it never issues a send and can't double-send itself. But the risk you're naming is real for whoever acts on the verdict, and right now I don't help them avoid it.
What exists today: verifications are append-only, so a re-verification of the same claim writes a new row with an incremented sequence rather than replacing the old one. The history is intact, so you can see an inconclusive followed by a later contradicted. What's missing is exactly your point — those are attempts against one claim, but nothing downstream is told "this claim already has an attempt in flight, don't act again."
On the typed claim before the call: that's the version I keep circling. I extract the claim from what the agent said afterwards, which means I'm parsing prose written by the thing I'm checking. A claim declared before the call — recipient, window, content fingerprint — would be evidence rather than narration. The cost is it has to be threaded through the agent, and most people can't rebuild theirs.
How are you separating traces from outcome evidence in agent-inspect? That boundary is the part I'd most want to compare notes on.