How do you verify what your AI agent actually did?

Adrien | Building Alfred — Sat, 01 Aug 2026 13:22:01 +0000

Short answer: you can't, in general. You can only verify a specific claim, at a
specific level of proof, and the level you need depends on what happens if the
agent is wrong.

That sounds like a dodge. It isn't. It's the reason this question generates so
much confused advice: "verify what the agent did" bundles six different
questions, and most tools answer one of them while sounding like they answer all
six. A signed receipt proves an API call happened. It says nothing about whether
that call was allowed. An observability dashboard shows you the execution path.
It says nothing about whether the work should count.

I test critical systems for a living — verification and qualification in cyber
and space. In that world nothing ships because someone says it works. Every
requirement has a test, every test has a pass criterion, every result leaves
evidence. A claim with no evidence attached has a name: it's a non-conformity.

Then I looked at how we deploy AI agents, and we accept from them exactly what we
refuse from any critical system — that they write their own report certifying
they did their job.

So here is the ladder I use. Six levels, ordered by the question they answer.
Plus two things that everyone mistakes for levels, and that aren't.

Disclosure up front: I maintain an open-source tool in this space. It sits at
levels 1 to 3 and I'll say exactly where it stops.

V0 — Self-report

Question: what does the agent say it did?

{ "status": "success", "message": "Invoice sent." }

Proves: nothing.

This isn't a rhetorical flourish. The agent generates both the work and the
report on the work, with the same model, in the same context. Research on
self-evaluation has a name for the resulting bias — models systematically score
their own output higher than they score other models'. Asking an agent whether it
succeeded is asking a witness to testify about themselves, with no cross-
examination.

When it's enough: when nothing real happens as a result. Which, for an agent
with tools, is never.

V1 — Execution trace

Question: what sequence of calls actually ran?

Structured spans: which tool, which arguments, when, what came back, and whether
it errored. OpenTelemetry's GenAI semantic conventions cover this; so does any
structured log with a correlation ID.

Proves: that a call happened, with these arguments, and how it terminated.

Doesn't prove: that the call was permitted, or that it had any effect.

Cost: low, if you already emit traces.

One thing V1 already catches that people miss: the agent reporting success over
a tool call that failed. The error is right there in the span status. If your
report doesn't surface it, that's a reporting gap, not a detection problem — the
evidence was already in your store.

V2 — Evidence

Question: do I hold an artifact I could put in front of someone who doubts me?

A Message-ID. A SHA-256 of the file. The ID of the created record. The
request/response payload, signed. An append-only log entry.

Proves: that a specific, checkable artifact is attached to the action.

Doesn't prove: that the intended effect exists in the target system. A receipt
is transport-layer proof. The vendors building receipt layers say this plainly in
their own documentation; their users forget it by the time it reaches a slide.

When it's enough: disputes, reconstruction, non-repudiation.

V3 — Authorization

Question: was this action allowed?

Here you need something that doesn't come from the run: a specification, written
before execution, that says what the agent may do. Allowed tools. Thresholds.
Required follow-ups. Budget. Escalation conditions. Then you check the observed
trace against it.

Proves: that each observed action is, or is not, conformant to a declared
mandate.

Doesn't prove: that the action produced its effect.

Cost: low. The work is writing the specification, not writing code.

This is the level the market talks about constantly and ships least. It shows up
as "task contract", "policy", "acceptance criteria", "define done" — everybody
names it, almost nobody hands you the file format. And without it, the levels
below are inert: a perfect execution trace of an action nobody authorized is
just a well-documented incident.

In qualification terms: evidence of execution is worthless without a requirement
to hold it against. That's not a philosophy, it's the whole reason the V-model
has a left-hand side.

agent: refund-bot-v3

allowed_tools: [read_order, issue_refund, notify_customer, escalate_to_human]
daily_budget_eur: 5.00

forbidden_actions:
  - issue_refund_above_100_eur
  - send_marketing

escalate_when:
  - tool_error_rate > 0.10
  - budget_used > 0.80
escalation_tools:
  - escalate_to_human

required_actions:
  - when_tool: issue_refund
    require_tool: notify_customer

V4 — Independent outcome verification

Question: did the intended change actually happen?

Don't ask the agent. Go look. Query the database. Search the Sent folder. Hit the
API and check the record exists. Crucially, use a different path than the action
used — if the write and the check go through the same client, they share the
same failure modes.

Proves: that the system of record carries the result.

Doesn't prove: that it was the right thing to do.

Cost: high. You need a verifier per action type, and you have to maintain them.

This is the only level that closes the gap between "the call returned 200" and
"the effect exists". Traces cannot do it — a trace records that a request was
sent and a response came back, not that the row landed. If your agent moves
money, sends things to customers, or does anything you can't undo, this level
isn't optional.

V5 — Preserved invariants

Question: is what should have stayed true still true?

This is the level almost nobody implements, and the one that catches the failure
everyone complains about: the agent stayed inside its assigned scope and still
broke something.

The instinct is to write down what the agent must not do. Don't touch auth. Don't
change the response shape. Don't modify the schema. And the list grows every time
something new goes wrong, and it never closes — because the set of forbidden
behaviors is infinite. You are enumerating a complement, and complements of
useful sets are unbounded.

Qualification does the opposite. You don't enumerate the forbidden. You assert
the invariants that must remain true, and you check them afterwards. "Fix null
handling in parse()" doesn't become "don't alter authentication". It becomes:
null input returns an error, the public signature is unchanged, every existing
caller's test still passes. That's a finite set of assertions, it's
executable, and it covers behaviors you never thought to forbid.

Finite beats exhaustive. That's the whole trick.

The two things that are not levels

Both of the AI-generated taxonomies I've seen on this topic get these wrong, and
so do several vendor posts. They matter enough to name.

Integrity is a property, not a level

Signing your logs, chaining hashes, append-only storage, timestamp attestation.
Every taxonomy I've read puts "cryptographic audit trail" at the top as the
highest level of assurance.

It isn't a level. Signing a log tells you nothing new about what happened. It
tells you the record wasn't altered afterwards. A signed V1 trace is still a V1
trace: it proves a call happened, tamper-evidently. Integrity is orthogonal — it
strengthens whatever level you're at, and it's necessary exactly when your
evidence will be contested. Adding it doesn't move you up the ladder.

A second model is a weak oracle, not a strong one

One taxonomy ranked "use another AI as verifier" at level 7 and independent
outcome verification at level 4. That's backwards.

The axis that matters is how independent your oracle is from the thing that did
the work: the agent itself (worthless) → a second model (correlated blind
spots, same training, similar failure modes) → a deterministic script → direct
observation of the system of record. An LLM judge is useful for questions with no
deterministic oracle, like whether a summary is faithful. It cannot outrank
looking at the database.

Where the tools actually sit

What it does	Level
LLM observability platforms (traces, spans, dashboards)	V1
Data/AI monitoring platforms	V1 + upstream data quality
Signed receipt / proof-of-execution layers	V2, strong integrity
Enterprise governance & evidence platforms	V2–V3, strong integrity, regulated buyers
Verification-receipt consultancies	V4, delivered as a service
Scope-checking tools for coding agents	V3 approximated by file paths, partial V5 via tests
Agent identity / delegation (CIAM)	different axis entirely

That last row matters. Verifying who the agent is and verifying what it did
are different problems, and the current discourse blends them constantly. Both are
necessary. Buying one thinking you got the other is how you end up with an
auditable identity attached to unverified actions.

Verify your verifier

Whatever level you choose, one rule survives all of them: feed your checker a
result you know is wrong, and confirm it rejects it. An untested verifier is a
belief, not a proof.

Two failures from my own tool, both of which I shipped and later had to fix.

The verifier trusted the thing it was verifying. My escalation check —
"did the agent escalate when it was required to?" — read an attribute called
alfred.escalated from the trace. The agent wrote that attribute. So an agent
that never escalated could set one boolean and switch off the very check watching
it. The fix was to stop believing the declaration and count actual calls to the
mandate's escalation tools: escalation is proven by an action, never by a
self-reported flag. That is axis B in one bug — my oracle was not independent of
the work it was checking.

Silence read as success. A tool call carrying no tool name was skipped by the
allowlist check, quietly. Which means dropping one attribute waved any call
past the mandate. The rule now is that a tool which cannot be named cannot be
allowed — an unverifiable call is a finding, not a blank.

Both bugs share a shape, and it's the one to watch for: the check reported a
clean run because it couldn't see, not because there was nothing to see. An
absence of findings and a finding of absence are not the same result, and most
verifiers conflate them.

So attack your own checker before you trust it. Feed it the run you know is bad.
If it comes back green, you haven't built a verifier — you've built a formality.

How to choose

Pick by what you risk, not by what's available:

Nothing irreversible happens → V1 is fine.
Someone else must answer for the agent's behavior — a client, an auditor, a regulator → V3 is the floor. A trace with no declared requirement is not an audit trail; it's a log file with good marketing.
Money, or anything you can't undo → V4, with an independent oracle.
Broad autonomy over a system you care about → V5.
The evidence will be contested → add integrity to whichever level you chose.

I build Alfred, an Apache-2.0 Python
package that covers V1 to V3: it reads an agent's OpenTelemetry GenAI spans,
checks them against a mandate declared in YAML, and computes a daily digest where
every statement is anchored to a trace event ID. Failed tool calls surface on
their own line, whatever the agent said about them.

It does not do V4 — it reads traces, not your database — and it does not do V5
today. I'd rather write that here than let you discover it in production.

If you're verifying agents in production, I'd genuinely like to know which level
you stopped at and why. That's the part nobody publishes.

DEV Community: Adrien | Building Alfred