Anatomy of an OpenTelemetry span: fields and DQL

On a Tuesday evening, a shop’s cart page goes blank for some customers. Dashboards are green, no alert has fired, yet the ticket’s trace shows one call that took four seconds. To work out what happened you need the anatomy of an OpenTelemetry span in your head: every field answers one precise question, and a missing answer is already a clue. This guide follows that investigation field by field, then replays it with DQL in Dynatrace SaaS.

The scope is SaaS only; the DQL and Grail documentation I rely on does not describe Dynatrace Managed. And to be straight about what is verified: I ran no query. The span below is made up for the occasion; the queries come from the documentation or are labelled « not executed ».

Anatomy of an OpenTelemetry span: the ID card

On the wire (OTLP, the trace.proto file), a span is just a record with a handful of numbered fields. The trace API specification sums it up: a name, an immutable context, an optional parent, a kind, two timestamps, attributes, links, events and a status. This is the mapping I keep nearby.

OTLP field	The question it answers
trace_id (16 bytes, required)	Which story does this span belong to?
span_id (8 bytes, required)	Which one, within that story?
parent_span_id	Who called it? Empty for the root.
name (required)	What was it doing?
kind	Which side of the exchange was it on?
start_time_unix_nano, end_time_unix_nano	When, in nanoseconds since the Unix epoch?
attributes	Under what conditions (route, method, host)?
events, links	What happened along the way? Who was it related to?
status (code, message)	How did it end?

Two details from the specification save you from surprises: the IDs must contain at least one non-zero byte, and the span also carries trace_state (the W3C tracestate header) plus three counters of discarded items (dropped_attributes_count, dropped_events_count, dropped_links_count). A non-zero counter means the SDK trimmed its own story.

Our suspect: the cart span

Here is the span I imagined for the ticket, in OTLP JSON. That encoding has its own rules: hexadecimal IDs, lowerCamelCase names, enums as integers, 64-bit integers as strings. The values are fictional.

Quick read: the name GET /cart/{id} is a route, not the real URL, which keeps the number of distinct names low. kind: 2 is SPAN_KIND_SERVER (the proto reserves 0 for « unspecified »), and the status code: 2 is STATUS_CODE_ERROR. Everything is in order here. The point of the investigation is what happens when it is not.

First trap: which side are we on?

The kind tells you who is waiting for whom. Five values:

• SERVER: handles a request that came from outside while a caller waits;
• CLIENT: sends a request and waits for the answer itself;
• PRODUCER: launches or schedules work, and often finishes before that work begins;
• CONSUMER: runs work left by a producer that does not wait for the outcome;
• INTERNAL: the default, a step inside the process.

In my scenario the cart calls a stock service, so I expect a CLIENT span on the cart side and a SERVER span on the stock side. If both are INTERNAL, instrumentation is partial and dependency maps lose their arrows. In Dynatrace, the documentation writes span.kind in lowercase (server, client, consumer, producer).

Second trap: attributes set too late

Attributes tell the conditions of the operation. The specification advises supplying them at span creation, because a sampler can only judge what exists at that instant; and once the span’s end is fixed, nothing moves. For a server-side HTTP span, the semantic conventions expect http.request.method, url.path and url.scheme, plus http.route and http.response.status_code when known. On the client side: http.request.method, server.address, server.port and url.full.

To know which names are stable or in transition, see the OpenTelemetry semantic conventions; for what Dynatrace adds, OpenTelemetry attributes in Dynatrace. One more useful point: Dynatrace keeps arrays and key-value lists and makes them queryable in DQL. The ingestion page mentions a common limit of 128 attributes, events and links per span, which Dynatrace does not enforce.

Third trap: the error that is not an error

Back to the blank cart. The span shows an exception, yet the dashboard stays green. The likely culprit is the status: it is Unset by default, and becomes Ok or Error only if someone decides so; the description message only makes sense with Error. The specification tells instrumentation libraries not to set Ok without explicit configuration. For HTTP, a 4xx leaves the SERVER span unset, while a 5xx calls for Error.

Events and exceptions

An event is a timestamped note inside the span. RecordException creates one named exception, with exception.type, exception.message and, on request, the stack trace. A caught-and-swallowed exception therefore leaves a trace in events without ever touching the status: exactly our cart. Note that the conventions page for exceptions on spans is marked deprecated in favour of logs; check what your SDK does.

Links

A link points to the context of another span, in the same trace or elsewhere, and carries its own attributes. It stands in when parenthood does not fit: a batch job consuming twenty messages cannot have twenty parents.

Replaying the investigation with DQL

In Dynatrace SaaS, fetch spans exposes this model: trace.id, span.id, span.parent_id (bytes shown as hexadecimal), span.kind, start_time, end_time and duration. The status is span.status_code, present only for error or ok, with span.status_message. Events and links are arrays: span.events, span.links.

Query 1 re-reads our suspect’s trace (not executed); query 2 is the one from the Advanced Tracing Analytics documentation, counting exceptions by type.

1. Load the trace with query 1 in a notebook, using a real ID and a time frame that covers the span’s time.
2. Find the root: it is the row whose span.parent_id is empty; the others point to their parent’s span.id.
3. Look for real failures with filter span.status_code == "error" (not executed, based on the field definition), then compare with the number of spans that carry an exception but no status.
4. Open the events with query 2: expand span.events, then fieldsFlatten, make their attributes filterable.

If the commands are new to you, start with filter, summarize and timeseries.

How to check it works

Send a test span: query 1 should return one row per span, with the right span.parent_id and duration. The OTLP ingest token needs the scope named on the « OTLP API » page (openpipeline:traces:ingest when I read it), and that API only takes HTTP with Protobuf, neither gRPC nor JSON. Other posts on this blog mention openTelemetryTrace.ingest: confirm the current scope before minting the token. If nothing shows up, follow OpenTelemetry traces missing in Dynatrace.

What derails the investigation

• Clock outside the window: per the ingestion documentation, the span’s end must be no more than 60 minutes old and no more than 10 minutes ahead.
• Two possible parents: OneAgent and OpenTelemetry on the same trace can create duplicates. The documentation provides span.alternate_parent_id, an experimental field; details in duplicate spans.
• Variable names: a customer ID inside the span name blows up the number of distinct values.

My instrumentation habits

• Name by a stable route or operation.
• Pass the required attributes at span creation.
• Put Error on real failures, caught exceptions included, and leave the rest Unset.
• Keep personal data out of attributes and exception messages.

Frequently asked questions

What is the difference between a span and a trace?

The span is one operation. The trace is the set of spans sharing one trace_id, tied together by their parents.

Should I use an event or an attribute?

An attribute qualifies the whole span; an event dates one precise moment, like an exception or a retry.

Where do I read a span’s status in Dynatrace?

In span.status_code (and span.status_message). The field does not exist when the status is Unset, which explains many invisible errors.

Conclusion

Facing a suspect span, ask the same questions in the same order: which story, who called it, which side, under what conditions, what happened, and how it ended. On Dynatrace SaaS, fetch spans answers each one. Take a real trace, read it back with query 1 and compare with what your SDK emits. Reference: the OpenTelemetry trace API specification.

span.observer is an independent community blog, not affiliated with Dynatrace LLC. Trademarks belong to their owners. Tools change fast: check fields and queries in the official documentation.

---

Originally published on span.observer. French version: https://span.observer/fr/anatomie-d-un-span-opentelemetry/.