DEV Community: Muutassim Mukhtar

[Boost]

Muutassim Mukhtar — Mon, 18 Aug 2025 00:44:22 +0000

Muutassim Mukhtar

Aug 18 '25

Lifting the Hood on Trace Propagation in OpenTelemetry

#opentelemetry #distributedtraces #go #systemdesign

Comments

6 min read

Lifting the Hood on Trace Propagation in OpenTelemetry

Muutassim Mukhtar — Mon, 18 Aug 2025 00:40:17 +0000

OpenTelemetry is really changing the game for observability, especially by offering the first widely adopted, vendor neutral telemetry libraries. Tracing was the first signal the project tackled, and its now generally available in most major programming languages.

Personally, Im a big fan of tracing. I think its a much cleaner and more effective way to pass data to your observability provider. they can take that data and turn it into something truly useful, whether thats a Gantt style view of spans or metrics that drive alerts.

One of the most powerful features is distributed tracing. the ability to connect spans across different services into a single trace. When one service makes an RPC call to another, the trace context can follow along, giving you a complete, end-to-end view of requests across your entire platform.

In the screenshot above for example,the trace has spans from 3 different services: frontend, productservice, and statsservice.

What is trace propagation?

To connect traces across different services, you need to pass some context along with each request. This process is known as trace propagation, and its what we will be diving into in this article.

W3C Trace Context?

Unless you are working with a legacy system that already uses a different tracing format, you should stick with the W3C Trace Context Recommendation. Its the recommended standard for propagating trace context over HTTP. While its designed with HTTP in mind, many of its concepts can also be applied to other communication channels, like Kafka messages, for example.

Trace Context specifies two HTTP headers that will be used to pass context around, traceparent and tracestate

traceparent

The traceparent HTTP header includes the root of context propagation. It consists in a comma-separated suite of fields that include:

The version of Trace Context being used. Only one version, 00 exists in 2023
Then, for version 00:
The current Trace ID, as a 16-byte array representing the ID of the entire trace.
The current Span ID (called parent-id in the spec), an 8-byte array representing the ID of the parent request.
Flags, an 8-byte hex-encoded field which controls tracing flags such as sampling.

tracestate

The tracestate HTTP header is meant to include proprietary data used to pass specific information across traces.

Its value is a comma-separated list of key/values, where each pair is separated by an equal sign. Obviously, the trace state shouldn’t include any sensitive data.

For example, with requests coming from public API endpoints which can be called either by internal services, or by external customers, both could be passing a traceparent header. However, external ones would generate orphan spans, as the parent one is stored within the customers service, not ours.

So we add a tracestate value indicating the request comes from an internal service, and we only propagate context if that value is present

A context is passed

With both these fields being passed, any tracing library should have enough information to provide distributed tracing.

A request could pass the following headers:
traceparent: 00-d4cda95b652f4a1592b449d5929fda1b-6e0c63257de34c92-01 tracestate: myservice=true

The traceparent header indicates a trace ID (d4cda95b652f4a1592b449d5929fda1b), a span ID (6e0c63257de34c92), and sets a flag indicating the parent span was sampled (so its likely we want to sample this one too).

The tracestate header provides a specific key/value that we can use to make appropriate decisions, such as whether we want to keep the context or not.

How OpenTelemetry implements propagation

The OpenTelemetry specification defines a Propagators interface to allow any implementation to establish its own propagation convention, such as W3C TraceContext.

A propagator must implement two methods:

Inject – to insert the current span context into a carrier object (such as an HTTP headers map).
Extract– to retrieve the span context from a carrier object.

For example, the following shows how this is implemented in Go:

Each instrumentation library making or receiving external calls then has the responsibility to call inject/extract to write/read the span context and have it passed around.

Extract and Inject examples

For example, the following is Rack extracting the context from the propagator to generate a new span in Ruby:

And the following is NodeJS adding the context into HTTP headers when making a request with the http package:

The full propagation flow

To put in other words, the diagram above shows what each service is expected to perform to enable propagation. The library emitting an HTTP call is expected to call inject, which will add the proper HTTP headers to the request. The library receiving HTTP requests is expected to call extract to retrieve the proper span context from the request’s HTTP headers.

Note that each language implementation of OpenTelemetry provides multiple contrib packages that allow easy instrumentation of common frameworks and libraries. Those packages will handle propagation for you. Unless you write your own framework or HTTP library, you should not need to call inject or extract yourself. All you need to do is configure the global propagation mechanism (see below).

Non-HTTP propagation

Not all services communicate through HTTP. For example, you could have one service emitting a Kafka message, and another one reading it.

The OpenTelemetry propagation API is purposefully generic, as all it does is read a hash and return a span context, or read a span context and inject data into a hash. So you could replace a hash of HTTP headers with anything you want.

The Python Kafka instrumentation calls inject on the message’s headers when producing a message:

And calls extract on those same headers when reading a message:

Any language or library that uses the same convention can benefit from distributed tracing within kafka messages, or any other communication mechanism.

Setting a propagator

Great! We now know how propagation for traces works within Otel. But how do we set it up?
Each OpenTelemetry library is expected to provide methods for setting and retrieving a global propagator.

For example, the Rust implementation provides global::set_text_map_propagator and global::get_text_map_propagator that will allow configuring and retrieving the global propagator.

As you may have seen in the above specification link, the default propagator will be a no-op:

The OpenTelemetry API MUST use no-op propagators unless explicitly configured otherwise

You should therefore always ensure your propagator of choice is properly set globally, and each library that needs to call inject or extract will then be able to retrieve it.

Each OpenTelemetry implementation will implement several propagators natively, including TraceContext, which you can use directly within your service. For example, the java one can be found as io.opentelemetry.api.trace.propagation.W3CTraceContextPropagator

Moving On

Thanks for following along with this deep dive into context propagation in Otel. Hopefully, you now have a clearer understanding of how distributed tracing works within the library. You should be equipped to implement context propagation in any library instrumentation that makes or receives calls from external services.

With distributed tracing properly set up across your platform, you will be able to see the full journey of every request, making it much easier to identify bottlenecks, trace issues, and debug problems effectively

How to Configure a Network Block Device on a Debian-Based System

Muutassim Mukhtar — Thu, 17 Oct 2024 13:52:54 +0000

In this article, we will configure a Network Block Device on a Debian system. The communication will occur via a loopback device, with both the client and server running on the same system.

As you may know, Network Block Devices (NBD) allow us to access remote storage devices that are not physically present on the local machine. With NBD, we can access and utilize these remote storage devices on the local machine in the following three ways:

SWAP
File System
RAW

Let's get right into it. For the NBD export, export, an empty file will be used.The /opt directory is to be used for the exported device and configuration files. But Production environments may prefer different locations.

-Install the NBD client and server packages.

$ mkdir ~/src
$ cd ~/src
$ git clone https://github.com/NetworkBlockDevice/nbd.git

The packages docbook-utils and autoconf-archive may not be installed on your system, install them now.

$ sudo apt install nbd-client nbd-server

Create an nbd-server.conf file in the /opt directory. A minimal, no-comments file is sufficient. (A more interesting configuration file can be found in the SOLUTIONS directory

$ sudo vi /opt/nbd-server.conf

/opt/nbd-server.conf

Create two files for exporting and set the ownership to your user

$ sudo dd if=/dev/zero of=/opt/dsk1 status=progress bs=100M count=5

$ sudo dd if=/dev/zero of=/opt/dsk2 status=progress bs=100M count=5

$ sudo chmod 777 /opt

$ sudo chown muutassim.muutassim /opt/*

Verify the nbd respond to a query

$ sudo nbd-server -C /opt/nbd-server.conf
$ sudo nbd-client -l 127.0.0.1 -p 10042

Install the nbd kernel module, and verify the block devices are present

$ sudo modprobe -i nbd
$ ls /dev/nbd*

Connect the server supplied image, using the ip address, port to the local block device

sudo nbd-client -N export_block_device 127.0.0.1 10042 /dev/nbd0

Wipe out the NBD

$ sudo dd if=/dev/zero of=/dev/nbd0 bs=1M count=5

Create GPT label oon the NBD:

sudo sh -c "echo 'label: gpt' | sfdisk /dev/nbd0"

Add a partition to the NBD:

$ sudo sh -c "echo';'| sfdisk /dev/nbd0"

Add an ext3 filesystem to the network block device and test:

$ sudo mkfs.ext3 -L nbd-foo /dev/nbd0

$ sudo mount LABEL=nbd-foo /mnt

$ sudo touch /mnt/file1

$ ls -l /mnt