A claims-processing agent read a file, checked a policy, and issued a €1,180 transfer. The function ran. The record exists. The money moved.
I had the official OpenTelemetry GenAI auto-instrumentation switched on the whole time, with content capture enabled and the latest semantic conventions opted in. Then I went looking for the span that attested to the payment.
There isn't one. Not on Anthropic, not on OpenAI. Four spans per run, none of them execute_tool.
This isn't a bug report. Everything behaved exactly as specified. The problem is that what the specification produces and what an auditor needs are two different things, and nothing in the docs tells you where the gap is.
Observability and forensics are not the same problem
An observability trace tells you what happened if you trust it. A forensic record has to hold up against someone who disputes it — a regulator, a customer, an insurer, an internal audit committee.
| Observability | Forensics |
|---|---|
| Sampled | Exhaustive on consequential actions |
| Mutable, deletable | Tamper-evident |
| Short retention | Retention matched to liability |
| User identity | Distinct agent identity |
Most teams running agents in production have the first and believe they have the second. I wanted to know exactly how wide that gap was, so I built a scenario small enough to reason about completely and measured it.
What the traces actually contain
The tool call is in the trace. It just isn't where the spec normalises it. Inside gen_ai.output.messages, each assistant turn carries:
{
"arguments": { "amount": 1180, "beneficiary": "client_44190" },
"name": "issue_payment",
"id": "toolu_018RmF8kpxzy8vambkdmyMzz",
"type": "tool_call"
}
Both providers emit the same shape. So the information isn't lost — but read what it is:
- Provable: what the model decided to do, and when it decided it.
- Not provable: that it happened, when, or what the system returned.
"The model requested a transfer" and "a transfer was made" are different claims. Only the first one is in your traces. Every dispute you will ever have lives in that gap.
There's a second-order problem that I find worse. An investigator following the specification looks for gen_ai.tool.call.arguments, the normalised location, and finds nothing. The data is recoverable — but only if you already know to open a nested JSON blob that the conventions never point you to. A trace that hides its own evidence from someone reading the spec correctly is not much of a record.
This is not what the spec asks for
I expected to find that the conventions simply delegated tool execution to application developers. That's not what they say. On the execute_tool span:
GenAI instrumentations that can instrument tool execution calls SHOULD do so, unless another instrumentation can reliably cover all supported tool types.
Neither package does, on either provider. The same note adds that application developers are encouraged to instrument tool calls that automatic instrumentation doesn't cover — and today that second sentence is carrying all the weight. The burden the spec places on instrumentations has landed entirely on you.
Worth noting too: gen_ai.tool.call.arguments and gen_ai.tool.call.result are opt_in. Without explicit activation you know an action occurred, not which one. Not the amount, not the recipient.
One provider loses the agent's mandate
gen_ai.tool.definitions — the list of tools exposed to the model — appears on OpenAI spans and not on Anthropic ones. Same tool specification sent to both.
This one isn't cosmetic. The first question after an incident is whether the action was within the agent's mandate. Without the tool definitions captured at call time, you cannot reconstruct what the agent was allowed to do at the moment it acted. If issue_payment was added to the agent the day before the incident, the trace will never say so.
The fix is structural, and it's yours to write
Adding execute_tool spans with a propagated tool_call_id, under an enclosing invoke_agent span, takes executions proven from 0 to 2 and normalised forensic coverage from 4/11 to 8/11. Roughly two dozen lines.
But the attributes aren't what makes it work — the tree is. The tool_call_id on the execution span matches the id the model emitted, so the chain from decision to action holds. Provider and model resolve by walking up to the parent span.
Which produces an operational consequence I didn't anticipate and now think is the most practically important finding here: sampling is unsafe on consequential actions. The attributes are spread across the tree. Drop a parent span and the child action becomes uninterpretable even though its own span is perfectly intact.
Where I got it wrong
While verifying every claim against the conventions YAML, I found that my own benchmark was wrong.
I had listed gen_ai.agent.version as something the conventions don't provide, and injected a custom attribute to fill the gap. It does provide it — conditionally_required on invoke_agent. My scenario had the value sitting in a variable and simply never set it in the right place.
That single fix moved coverage by a full point. A gap I had attributed to the specification was a gap in my own instrumentation.
I'm including this because it is the lesson. Classifying why an attribute is missing — not sent, sent but not emitted, or genuinely absent from the spec — is the entire discipline, and it's easy to get wrong in the direction that flatters your argument. Read model/*.yaml at a pinned commit, not the generated docs.
What no amount of instrumentation fixes
Three attributes stay absent even after correct instrumentation. Two are real gaps:
gen_ai.agent.id exists, but the spec scopes it to hosted agent resources — a Bedrock ARN, a GCP Agent Registry identifier — and explicitly discourages recording in-memory instance ids. For a self-hosted agent, which covers most enterprise deployments, there's no appropriate attribute. The agent's actions stay indistinguishable from those of the human whose credentials it runs under.
gen_ai.conversation.id exists, but the spec says instrumentations should not invent one when no natural identifier is available. No UUID, no trace id, no hash. So it's frequently absent, and nothing ties a sequence of actions to a business object.
The third, gen_ai.system_instructions, is absent because my scenario sends no separate system prompt — an artefact of the setup, not a gap.
Limitations
Two providers, one scenario, one agent shape. Nothing here generalises to multi-agent systems, MCP servers, streaming, or frameworks like LangChain.
Everything in gen_ai.* is still marked Development in the conventions. These numbers are pinned to specific versions and will drift — that's expected, and tracking the drift is part of the problem.
And none of this addresses integrity. The spans are freely mutable and deletable. Hash chaining and signing are a separate problem that I have not solved.
Reproduce it
Everything is in the repository, including the four span dumps the numbers come from. Both tools are pure functions of those dumps, so you can re-derive every figure offline with no API key:
python compare.py anthropic.json openai.json
https://github.com/Quentin-NA/agent-trace-forensics
Pinned to semantic-conventions-genai b5d8440 (2026-09-08), instrumentation packages 1.1b0/1.1b1, models claude-sonnet-4-6 and gpt-4o-mini, run of 2026-09-16.
If you're running agents that touch money, records, or anything a regulator cares about, the question isn't whether you have traces. It's whether a span exists that proves the action happened — and whether it's still interpretable once the parent has been sampled away.
I'd be glad to hear from anyone who has measured this differently, or on other providers.
Top comments (2)
Adding your own execute_tool span does not close the gap between "the model requested a payment" and "a payment happened". The process that performed the action is the same process that writes that span, inside the same trust boundary. It stays self-report. To a reader who disputes it, you have now recorded the decision twice.
What actually raises it is the counterparty. If the receiving side or the payment rail writes the same tool_call_id into its own record, reconciliation becomes a join across records with two separate writers, and it stops being one writer's account of itself.
I would also sharpen the sampling claim. The real property you lack is that the record cannot be excerpted. Interpretation is scattered across ancestors, so an action span that is perfectly intact on its own still means nothing when shown alone. The fix is to denormalise onto consequential action spans only: the agent identifier, the authority set it was acting under, and the tool_call_id, copied down so the leaf stands by itself. You pay bytes on the small set of spans that can ever be disputed.
On the mandate question, storing the whole tool list at every call is expensive and it drifts. Record one digest of the tool set per action, and keep the expansion once per distinct digest. Then "was issue_payment added the day before" is a digest comparison first, with the expansion there only when they differ.
The limit is real. If the counterparty writes nothing back, self-report is the ceiling, exactly as you observed.
Did the receiving side of that EUR 1,180 leave anything behind that you could join against?
You're right, and it's the sharper version of what I wrote. The execution span is written by the process that performed the action, inside the same trust boundary. It removes the "did the tool even run" ambiguity for anyone already inside that boundary, and nothing more. Self-report with better structure is still self-report.
To answer your question directly: no. issue_payment in the benchmark is a Python function appending to an in-memory list. There is no counterparty and nothing to join against. That's a real limit of the setup, and it's the boundary your comment makes visible. I was measuring instrumentation coverage and reading it as evidential strength.
The denormalisation framing is better than mine. "The record cannot be excerpted" is the actual property, and copying the agent identifier, authority set and tool_call_id down onto consequential leaves is a cheaper fix than not sampling at all. Same for the tool-set digest: one hash per action with the expansion stored once per distinct digest answers "what was it allowed to do" without paying for the list every call.
One follow-up on the counterparty side, since that's where I have no data. Who writes it in practice, and into what? A payment rail doesn't record a tool_call_id on its own. Someone has to decide to carry it through as a reference field, end to end, and keep it queryable. Is that something you've seen done, or is it currently a design requirement without an implementation? Whether the identifier survives the hop determines whether the join is available after the fact, or only if you planned for it beforehand.