A user triggers an invoice, the API answers in 80 ms, and a worker generates it from a queue forty seconds later. In your tracing tool you see two unrelated stories. Async context propagation did not happen: nobody wrote the trace context into the message, so the worker started a new trace.
This tutorial is for SRE and platform teams that run message queues (RabbitMQ, SQS, Redis, a jobs table) or background tasks. It explains what must travel with the message, when to choose a span link rather than a parent, and how to check the result. Kafka is covered in Kafka context propagation in Dynatrace; here the method applies to any transport. OTLP ingestion applies to Dynatrace SaaS; the DQL queries assume Grail, so a SaaS environment.
Async context propagation: the principle
Over HTTP, the client writes a traceparent header and the server reads it. According to the OpenTelemetry documentation, the context is « injected into the carrier » on the sender side and « extracted » on the receiver side. For a protocol without a dedicated metadata field, you choose where to store the context, and the receiver must read it from the same place.
For a queue, the carrier is whatever the message offers: message properties or headers, SQS message attributes, a field in the JSON body, a column in the jobs table. The principle is the same: the producer injects, the consumer extracts.
OpenTelemetry’s default format is W3C Trace Context. OTEL_PROPAGATORS defaults to tracecontext,baggage in the Python SDK (source: opentelemetry.propagate documentation). Both sides must use the same format.
What the traceparent header contains
In version 00 the header is one 55-character line: the version, a 32-character hexadecimal trace ID, a 16-character parent span ID, then the flags. Only the sampled flag is defined in version 00; it is the last bit. An all-zero ID is invalid (W3C specification).
The second panel summarises the messaging semantic conventions: a create operation gives a PRODUCER span, process a CONSUMER span, receive and settle CLIENT spans.
Step 1 — Inject the context when sending
Create a publish span, then inject the current context into a dictionary. That dictionary travels with the message. The code below is deliberately generic: queue.put stands for your client’s call.
Critical lines: propagate.inject(carrier) writes traceparent (and tracestate if present) from the current context, that is, from the PRODUCER span opened just before. Per the OpenTelemetry conventions, a producer « SHOULD » attach a creation context to each message; without it, consumer traces cannot be correlated with producer traces.
For Celery, SQS or another client, first check whether an OpenTelemetry instrumentation exists for your library: it does this work without code. Otherwise the manual version above still applies.
Step 2 — Extract the context in the worker
The worker reads the dictionary, calls propagate.extract and gets a Context. Two choices follow: make it the parent of the processing span, or create a span link to it.
The OpenTelemetry messaging conventions present links as the default correlation mechanism. They give three reasons: links are the only structure guaranteed across all messaging models, the only option for a batch (a span has just one parent), and the only one that works when processing already runs inside an ambient context such as an HTTP server span. Using the creation context as parent only suits a single message.
Critical lines: Link(span_context) takes the SpanContext of the span to link to (signature Link(context, attributes=None)); start_as_current_span accepts the kind, links and context parameters. To get a parent rather than a link for a single message, pass context=ctx instead of links=.
Step 3 — Handle batches, retries and scheduled jobs
- Batches: one processing span can cover several messages. Per the conventions, it then links to the creation context of each message it handles. Dynatrace documents a typical limit of 128 links per span (a common OpenTelemetry SDK limit, not enforced by Dynatrace, per its trace ingestion page): beyond that, split the batch.
- Prefetched messages: a receive span must not be created for messages merely held in a cache; it appears when the message is handed to application code.
- Retries: a replayed message keeps its original context. The retry trace therefore attaches to the initial request, sometimes hours later. That helps diagnosis, but the time gap lengthens the trace.
- Scheduled jobs (cron): there is no upstream context. Create a root span; that is normal.
- Intermediate hops: a router, bridge or connector that copies the message must copy the context too. Test every hop.
Step 4 — Sampling and ingestion
The sampled flag travels with the context. With a parent-following sampler, the worker honours the decision taken at the API; if sampling differs on each side, the trace is truncated. For Collector-side sampling, see tail sampling with OpenTelemetry and Dynatrace.
Export to Dynatrace follows the usual OTLP path, described in the OpenTelemetry Collector for Dynatrace. If nothing arrives, start with the missing traces diagnosis. If OneAgent and OpenTelemetry instrument the same client, beware of duplicate spans.
How to check that it works
In Dynatrace SaaS, open a Notebook. This query finds traces that cross at least two services for your queue:
fetch spans, from: now() - 1h
| filter messaging.system == "rabbitmq"
| summarize services = collectDistinct(service.name), spans = count(), by: { trace.id }
| filter arraySize(services) >= 2
| limit 20
When processing is joined by link rather than parent, the worker trace stays separate: look instead for spans that carry links, with filter isNotNull(span.links) (the span.links field appears in the Dynatrace semantic dictionary). Field names and the messaging.system value depend on your instrumentation: check them first with fetch spans | limit 10. If nothing comes back although both services emit spans, the context is lost between them. Confirm this on your own environment: span.observer has not run the query.
Best practices
- One propagation format everywhere:
tracecontext. - Name spans « publish » and « process » for readable search.
- Measure the delay between publish and processing: it is a natural SLI for an asynchronous pipeline, more telling than HTTP latency.
- Keep the context out of the business data of the message: put it in a dedicated metadata field, for example
trace_headers. - Avoid personal data in
baggage: it travels with every message.
Frequently asked questions
Span link or parent: which should a worker use?
For a batch, the link is the only option. For a single message, the parent gives one continuous trace, but the OpenTelemetry conventions advise against it by default when processing already runs inside another span. When in doubt, use the link.
Does the context survive a delay of several hours?
The context is just a text string stored with the message: it survives as long as the message does. The trace itself depends on retention and ingestion limits. For example, Dynatrace states that a span must not end more than 60 minutes in the past at ingestion.
Does the broker need changing?
No. The broker carries the message as is. Only producer and consumer inject and extract the context.
Conclusion
Inject on send, extract on receipt, link whenever a batch is possible: three steps keep a request and its deferred processing connected. The on-call team can then follow an invoice from the API to the worker without manual correlation, and the processing-delay SLI becomes computable. For the big picture, see OpenTelemetry end to end. Official sources: OpenTelemetry messaging conventions and the W3C Trace Context specification.
span.observer is an independent community blog, not affiliated with Dynatrace LLC. Dynatrace is a trademark of Dynatrace LLC. Features change quickly: check their availability in your environment and in the official documentation.
Originally published on span.observer. French version: https://span.observer/fr/propagation-contexte-asynchrone-jobs/.
Top comments (0)