Your agent issued a €1,180 payment. Your traces can't prove it.

Quentin Natoly — Fri, 18 Sep 2026 10:56:42 +0000

A claims-processing agent read a file, checked a policy, and issued a €1,180 transfer. The function ran. The record exists. The money moved.

I had the official OpenTelemetry GenAI auto-instrumentation switched on the whole time, with content capture enabled and the latest semantic conventions opted in. Then I went looking for the span that attested to the payment.

There isn't one. Not on Anthropic, not on OpenAI. Four spans per run, none of them execute_tool.

This isn't a bug report. Everything behaved exactly as specified. The problem is that what the specification produces and what an auditor needs are two different things, and nothing in the docs tells you where the gap is.

Observability and forensics are not the same problem

An observability trace tells you what happened if you trust it. A forensic record has to hold up against someone who disputes it — a regulator, a customer, an insurer, an internal audit committee.

Observability	Forensics
Sampled	Exhaustive on consequential actions
Mutable, deletable	Tamper-evident
Short retention	Retention matched to liability
User identity	Distinct agent identity

Most teams running agents in production have the first and believe they have the second. I wanted to know exactly how wide that gap was, so I built a scenario small enough to reason about completely and measured it.

What the traces actually contain

The tool call is in the trace. It just isn't where the spec normalises it. Inside gen_ai.output.messages, each assistant turn carries:

{
  "arguments": { "amount": 1180, "beneficiary": "client_44190" },
  "name": "issue_payment",
  "id": "toolu_018RmF8kpxzy8vambkdmyMzz",
  "type": "tool_call"
}

Both providers emit the same shape. So the information isn't lost — but read what it is:

Provable: what the model decided to do, and when it decided it.
Not provable: that it happened, when, or what the system returned.

"The model requested a transfer" and "a transfer was made" are different claims. Only the first one is in your traces. Every dispute you will ever have lives in that gap.

There's a second-order problem that I find worse. An investigator following the specification looks for gen_ai.tool.call.arguments, the normalised location, and finds nothing. The data is recoverable — but only if you already know to open a nested JSON blob that the conventions never point you to. A trace that hides its own evidence from someone reading the spec correctly is not much of a record.

This is not what the spec asks for

I expected to find that the conventions simply delegated tool execution to application developers. That's not what they say. On the execute_tool span:

GenAI instrumentations that can instrument tool execution calls SHOULD do so, unless another instrumentation can reliably cover all supported tool types.

Neither package does, on either provider. The same note adds that application developers are encouraged to instrument tool calls that automatic instrumentation doesn't cover — and today that second sentence is carrying all the weight. The burden the spec places on instrumentations has landed entirely on you.

Worth noting too: gen_ai.tool.call.arguments and gen_ai.tool.call.result are opt_in. Without explicit activation you know an action occurred, not which one. Not the amount, not the recipient.

One provider loses the agent's mandate

gen_ai.tool.definitions — the list of tools exposed to the model — appears on OpenAI spans and not on Anthropic ones. Same tool specification sent to both.

This one isn't cosmetic. The first question after an incident is whether the action was within the agent's mandate. Without the tool definitions captured at call time, you cannot reconstruct what the agent was allowed to do at the moment it acted. If issue_payment was added to the agent the day before the incident, the trace will never say so.

The fix is structural, and it's yours to write

Adding execute_tool spans with a propagated tool_call_id, under an enclosing invoke_agent span, takes executions proven from 0 to 2 and normalised forensic coverage from 4/11 to 8/11. Roughly two dozen lines.

But the attributes aren't what makes it work — the tree is. The tool_call_id on the execution span matches the id the model emitted, so the chain from decision to action holds. Provider and model resolve by walking up to the parent span.

Which produces an operational consequence I didn't anticipate and now think is the most practically important finding here: sampling is unsafe on consequential actions. The attributes are spread across the tree. Drop a parent span and the child action becomes uninterpretable even though its own span is perfectly intact.

Where I got it wrong

While verifying every claim against the conventions YAML, I found that my own benchmark was wrong.

I had listed gen_ai.agent.version as something the conventions don't provide, and injected a custom attribute to fill the gap. It does provide it — conditionally_required on invoke_agent. My scenario had the value sitting in a variable and simply never set it in the right place.

That single fix moved coverage by a full point. A gap I had attributed to the specification was a gap in my own instrumentation.

I'm including this because it is the lesson. Classifying why an attribute is missing — not sent, sent but not emitted, or genuinely absent from the spec — is the entire discipline, and it's easy to get wrong in the direction that flatters your argument. Read model/*.yaml at a pinned commit, not the generated docs.

What no amount of instrumentation fixes

Three attributes stay absent even after correct instrumentation. Two are real gaps:

gen_ai.agent.id exists, but the spec scopes it to hosted agent resources — a Bedrock ARN, a GCP Agent Registry identifier — and explicitly discourages recording in-memory instance ids. For a self-hosted agent, which covers most enterprise deployments, there's no appropriate attribute. The agent's actions stay indistinguishable from those of the human whose credentials it runs under.

gen_ai.conversation.id exists, but the spec says instrumentations should not invent one when no natural identifier is available. No UUID, no trace id, no hash. So it's frequently absent, and nothing ties a sequence of actions to a business object.

The third, gen_ai.system_instructions, is absent because my scenario sends no separate system prompt — an artefact of the setup, not a gap.

Limitations

Two providers, one scenario, one agent shape. Nothing here generalises to multi-agent systems, MCP servers, streaming, or frameworks like LangChain.

Everything in gen_ai.* is still marked Development in the conventions. These numbers are pinned to specific versions and will drift — that's expected, and tracking the drift is part of the problem.

And none of this addresses integrity. The spans are freely mutable and deletable. Hash chaining and signing are a separate problem that I have not solved.

Reproduce it

Everything is in the repository, including the four span dumps the numbers come from. Both tools are pure functions of those dumps, so you can re-derive every figure offline with no API key:

python compare.py anthropic.json openai.json

https://github.com/Quentin-NA/agent-trace-forensics

Pinned to semantic-conventions-genai b5d8440 (2026-09-08), instrumentation packages 1.1b0/1.1b1, models claude-sonnet-4-6 and gpt-4o-mini, run of 2026-09-16.

If you're running agents that touch money, records, or anything a regulator cares about, the question isn't whether you have traces. It's whether a span exists that proves the action happened — and whether it's still interpretable once the parent has been sampled away.

I'd be glad to hear from anyone who has measured this differently, or on other providers.

DEV Community: Quentin Natoly