Most teams running AI agents in production already trace them. If that tracing is built on OpenTelemetry, there is now a shared vocabulary for it: the GenAI semantic conventions, which define spans for model calls, agent invocations and tool executions. This piece walks through what those conventions record about an agent, what they leave out on purpose, and why a trace that answers "what went wrong?" is a different thing from a record that answers "show me what your agent did" when the person asking is a customer or an auditor.
Where the conventions live now
The GenAI conventions started inside OpenTelemetry's main semantic-conventions repository. They have since moved to a dedicated repository, and the old pages now say only that the conventions "have moved". Every document in the set is marked Status: Development. The new repository has not cut a release yet, and its pending changelog includes breaking changes, such as removing a required attribute from the invoke_agent internal span. If you build dashboards or alerts on these attribute names, pin the version your instrumentation emits and expect to migrate.
What an agent trace contains
The agent spans document defines spans for creating an agent, invoking one, running a workflow, planning, and executing a tool. Tool execution has refinements for specific cases: a tool call that hands work to another agent, loading an agent skill, and a "command execution span" that "describes a tool call that executes one or more commands". Separate documents cover MCP and Agent2Agent calls.
The command execution span is the one most relevant to accountability, because it covers the moment an agent touches a real system. Its required attributes are gen_ai.operation.name and gen_ai.tool.name. Conditionally required ones include the agent's name, the tool call ID, process.executable.name and process.exit.code. The span name should take the form execute_tool {gen_ai.tool.name} {process.executable.name} when the executable is known. An illustrative span for an agent that deletes something through a shell tool, with default settings, looks like this:
execute_tool bash rm
gen_ai.operation.name = execute_tool
gen_ai.tool.name = bash
gen_ai.agent.name = release-bot
gen_ai.tool.call.id = call_7f3a...
process.executable.name = rm
process.exit.code = 0
That is useful. You can see which agent ran which binary, whether it succeeded, how long it took and where it sat in the agent's run. What you cannot see is what rm was told to delete.
Content is off by default, on purpose
The arguments and result of a tool call are carried by gen_ai.tool.call.arguments and gen_ai.tool.call.result. Both are Opt-In, and both carry the warning "This attribute may contain sensitive information." The model spans document sets the general rule: "Model instructions, user messages, and model outputs are considered sensitive and are often large in size", and "OpenTelemetry instrumentations SHOULD NOT capture them by default, but SHOULD provide an option for users to opt in."
That option, where it is an environment variable, must be called OTEL_INSTRUMENTATION_GENAI_CAPTURE_CONTENT. Its values are NO_CONTENT (the default), SPAN_ONLY, EVENT_ONLY and SPAN_AND_EVENT, and the setting governs tool-call arguments and results as well as prompts and completions. The convention describes three patterns: record nothing (the default), record content on the spans, which it suggests for cases such as "pre-production environments", or upload content to external storage and put a reference on the span, which it recommends for production environments where volume or sensitive data is a concern. For that last pattern it is explicit that it "does not prescribe the URI scheme, object layout, serialization format, retention policy, or access mechanism."
This is the right default for telemetry. Prompts and tool results routinely contain personal data, and an observability backend is usually readable by far more people than should see it. But it means a default agent trace records that a destructive action happened, without recording what the action was.
Observability is allowed to forget
Traces are also sampled. OpenTelemetry's sampling guide describes head sampling, where a decision "is made based on the trace ID and the desired percentage of traces to sample", giving its example as "5% of all traces". Its own list of reasons not to sample includes being "bound by circumstances such as regulation that prohibit dropping data", where unsampled data cannot be routed to cheaper storage instead. The span format itself, in the OTLP trace definition, has counters for attributes, events and links that were dropped.
None of that is a flaw. Observability exists to answer aggregate and diagnostic questions cheaply, and losing a fraction of the data is an acceptable trade. One detail in the GenAI conventions is worth knowing here: if an external content upload hook is configured, instrumentations "SHOULD invoke it regardless of the span sampling decision". So the content can survive even when the span that refers to it was dropped. Check what your own pipeline does before relying on either.
Nothing in a span is signed
The OTLP Span message holds IDs, a name, start and end times, attributes, events, links and a status. It has no signature field, and the OTLP specification deals with delivering and acknowledging telemetry, not with proving its integrity. The start time is, in the proto's own words, "the time kept by the local machine where the span execution starts".
The W3C Trace Context recommendation, which OpenTelemetry's default propagator settings use to link spans across services, is frank about abuse. Its security section warns that when a service with a public API naively continues incoming traces, an attacker could "forge trace-id collisions that make monitoring data unusable".
Put together, a trace is a set of statements that the instrumented process made about itself, timed by its own clock, passed through a collector and stored in a backend the operator controls. Anything that can run code in that process, or write to that backend, can add, change or drop spans. For debugging, that is fine: you trust your own telemetry because you have nothing to gain from lying to yourself. When the question comes from outside, after an agent has deleted a customer's data or been steered by injected instructions, the trace is the operator vouching for the operator.
A quick test for your tracing set-up: pick one destructive tool call your agents made last week. Can you show the exact arguments it ran with? Do you know that its trace was sampled? Could anyone with backend access have edited the span since? If someone outside your organisation asked, what would you hand them, and how would they check it without trusting you?
Traces and signed records side by side
A signed execution record answers a different question. The Traceseal receipt specification, for example, defines a JSON document in which "an operator attests that a specific piece of code ran inside a specific sandbox and produced a specific result", signed with ed25519 and checkable by a third party "without access to the operator's machine, the original code, or any private keys". Compared point by point:
- Purpose. A trace is for finding out what went wrong. A receipt is for showing someone else what ran.
- Coverage. Traces are often sampled. A signed record only helps if one exists for the run in question, so our advice is not to sample them for consequential actions.
- Content. Traces carry plaintext content only if you opt in. A receipt carries hashes of inputs and outputs, so the record can be shared while the content stays private; if a verifier later needs the content, the operator supplies it and the verifier recomputes the hash.
- Integrity. A span is unsigned. A receipt is signed, so any change after signing is detectable.
- Time. A span's time is the local clock. A receipt's time is also only the signer's claim. Fixing when a record existed needs something outside the operator, such as a trusted timestamp or a witnessed log covering it.
To be exact about the limits: a receipt is not proof that the operator told the truth. The specification says so: it "proves that the operator's key signed the attestation. It does NOT prove the operator is honest." It also records what the operator claims the sandbox was, not that the sandbox enforced it, and hashes of short or guessable values can be reversed by guessing. What a receipt adds is that the claim is fixed, attributable and checkable by someone else. Who holds the signing key matters as much as the signature; if the agent can reach the key, it can sign its own account of events, which is why the sandbox and the signer should be separate.
Using both
There is no need to choose. Keep OpenTelemetry for what it is good at: latency, cost, error rates and debugging individual runs. For the actions that would matter in a dispute (deleting data, sending messages, moving money, changing production), produce a signed record outside the agent's process as well. Then link the two. The GenAI conventions do not define an attribute for an external evidence record, so add one under your own namespace, for example the receipt's hash on the execute_tool span. An investigator can then start from the trace, which is easy to search, and end at the receipt, which can be checked.
For what to capture on each tool call in the first place, our piece on MCP tool-call logging goes into the fields, and the audit trail piece covers retention and who should hold the record.
What to take from this
- The OpenTelemetry GenAI conventions are a good, still-moving baseline. They now have agent, tool, command-execution, MCP and A2A spans. Pin your version.
- Defaults record that a tool ran, not what it did. Arguments and results are opt-in, for sound privacy reasons. Decide deliberately where that content goes.
- Traces are sampled, unsigned and timed by the local clock. That suits debugging and does not make them evidence.
- Link traces to signed records for consequential actions. Searchable on one side, verifiable on the other.
The Traceseal receipt format, verifier and log are open: the spec, the verifier on PyPI, and the public log.
Originally published at traceseal.io.
Top comments (0)