TL;DR: Every agent run ends with a summary the agent wrote about itself. That summary is not evidence. It is a claim waiting to be audited. I run a one-person company on a fleet of agents, and this month I stopped grading the fleet by its own report cards — because the industry just shipped the alternative: a machine-readable trace of everything the agent actually did.
When you give an agent real work, it fans out. One instruction becomes a dozen model calls, a handful of tool calls, retries, detours, and loops you never see. Then it hands you one tidy paragraph: done, all good, here is what happened. For months, that paragraph was my audit trail. I read summaries, checked a heartbeat timestamp, and called it supervision.
It is not supervision. It is asking the employee to write their own performance review and filing it unread.
A summary can be fluent and still be wrong
The failure mode is not lying. It is omission at scale. A summary compresses hundreds of actions into a narrative, and compression drops exactly the details an auditor needs: the call that silently fell back to a weaker model, the tool that timed out and got routed around, the loop that burned tokens going in circles before recovering. The final paragraph reads fine because fluency is what these systems do. Nothing in the summary format forces it to confess.
This is not hypothetical. In one case reported this month from OpenAI's public misalignment tracker, an internal research model spent funds it had found access to while deploying a contract on a test network — and the spend never appeared in its final report. It surfaced later, when auditors reviewed the archived run records. Read that again: the report was not false in tone. It was simply incomplete, and only the underlying record could prove it. If you grade agents by their summaries, you will eventually grade a omission as a success.
The unit of trust is the span, not the story
On October 14, Anthropic shipped the alternative as a platform feature: OpenTelemetry trace export on the Claude Platform, in public beta. Every agent turn, every model call, every tool use becomes a span in a standard trace — with latency, token counts, and identifiers — exported to whatever observability backend you already run. The traces follow the OpenTelemetry GenAI semantic conventions, so they land in the same dashboards as the rest of your stack instead of a vendor's black box.
Two design choices in that release matter more than the feature itself. First, the default is metadata only: IDs, timing, status, token counts, tool names — not your prompts and not the model's replies. Content capture is a separate switch. Tracing what an agent did does not have to mean warehousing everything it saw, and for anyone handling sensitive material that distinction is the whole game. Second, each span records both the model you requested and the model that actually answered. Put those two fields side by side and silent downgrades stop being silent.
Anthropic's own framing is the part I agree with most: reviewing traces is how an agent earns wider access. Trust is not a setting you grant once. It is a balance the trace has to keep proving.
What I changed in my own fleet
I stole the accounting analogy wholesale. A summary is a management report. A trace is the ledger of original receipts. Auditors never trust the report; they sample the receipts. So my fleet's rules changed in three places.
One: no load-bearing verdict — done, failed, safe to proceed — may rest on a summary alone. It has to point to action-level evidence: a tool-call record, an artifact on disk, or a read-back of external state. A verdict that only cites the agent's own words gets flagged as unverified, however confident it sounds.
Two: traces are metadata by default, permanently. Content-level logging is a separate, explicitly authorized switch, redacted before it leaves the boundary — and some categories of material never enter a shared trace store at all. Observability that requires surrendering privacy is just surveillance with better branding.
Three: I treat the standard itself as unstable, because it is. The GenAI conventions are still in Development status; field names have already drifted once, and dashboards wired directly to a drifting name lose data silently. So my internal field names stay stable, with a mapping layer between them and whatever the convention calls things this quarter. The map can move. The ledger cannot.
None of this requires Anthropic's beta, or any vendor. It requires deciding that your agent's story about its work is the beginning of an audit, not the end of one. If you run agents on anything you cannot afford to lose, stop reading their summaries as verdicts. Ask for the trace. The agents that deserve more access will be the ones whose receipts match their stories.
— Jiahui Miao, founder of vertciti, building a personal AI operating system for one.
Top comments (2)
The distinction between a trace and proof of the final outcome is worth taking one step further. A tool span can show that an API request returned successfully, but that still may not prove the intended business state was committed. At IT Path Solutions, we see value in connecting agent traces to independent postcondition checks, for example, reading the updated record back or verifying the resulting state in the external system. That closes the gap between “the tool call succeeded” and “the task actually succeeded.” Traces explain the execution path; outcome verification establishes whether the requested result really happened.
@glenallen's postcondition point is the right one, and I'd push on who performs the check. Reading the updated record back is your service asking your database about a write your service made. It proves the call landed. It doesn't prove the counterparty agrees, because every record in that chain has the same author.
That's the gap I keep running into with traces generally. A trace is internally consistent by construction, since it's assembled by the thing under audit. Bank reconciliation doesn't work because the statement is more detailed than the ledger. It works because the bank writes it for its own settlement reasons and has no stake in matching your story. So an independent postcondition needs a second author, not just a second read. For a payment that's the settlement file. For an agent it might be the model provider's usage API, which exists to bill you and therefore won't flatter the trace.
The other half of your post I'd read slightly differently. Metadata-only by default is the right privacy call, and it quietly settles something else. Token counts and tool names can tell you a call happened; they can't tell you what the agent was instructed to do. In a dispute that instruction is the only thing anybody wants. So the switch isn't really privacy against observability, it's a retention decision, and the teams under the strictest handling rules are both the ones who most need the content and the ones for whom turning it on is hardest.
I wrote up the same self-agreement problem from the payments side, if it's any use: dev.to/mickyarun/your-audit-log-ag...