Three threads here this month have circled the same gap: a green execution over work that never happened. The answers are all correct and all the same shape — RETURNING id, an If node, throw. Wire it per step, per workflow.
I got tired of wiring it per step, so I built the version that runs once.
What it does
Takes what the run claimed — "email sent", "order updated" — and goes and reads the authoritative system. Not the execution record, which the run wrote itself. The actual mailbox.
Three verdicts, never two:
contradicted — the claim and the world disagree
confirmed — every property in the claim was checked and holds
inconclusive — something couldn't be established, so no judgement
That third one is the whole design. If the trace can't be shown complete, or a tool has a name it doesn't recognise, or the mailbox can't be identified — it refuses to judge rather than calling a working workflow a liar. A monitor that resolves its own uncertainty optimistically is worth nothing on the day it matters.
Two of the three checks need no access to anything
The claim was made and no tool was ever called — the absence is the evidence, no external read needed.
A lookup returned 403 and the run then stated what that lookup would have shown. The call completed; the result was a refusal. Those are different things, and most wrappers collapse them.
Only the third needs a connected mailbox: the tool was called, returned 200, and the record isn't there.
A real one it found
Someone published a production trace at me. Two order lookups came back 403 — the agent got nothing. The final reply still said "I can see you were charged $29 twice on 3 October." Run finished COMPLETED. Nothing in the execution list looked wrong.
That trace produced nothing the first time I ran it. Four separate causes. Fixing them took two days and three of the four were things I'd assumed worked.
What it doesn't do
Gmail is the only external adapter. It can't catch a correct call with a wrong argument — right recipient format, wrong recipient, real send, Gmail agrees. And a false claim in text that was never sent has no outcome to read back against.
Setup
One line pasted into Cursor or Claude Code and it wires itself. Or four calls by hand, TypeScript or Python.
matrixverify.dev — free, zero users, and I'd rather it got broken than ignored. Happy to run it against one of your executions and tell you what it finds, including if it finds nothing.
Top comments (0)