DEV Community

Cover image for Design a distributed tracing system: the system design interview walkthrough
Gowtham Potureddi
Gowtham Potureddi

Posted on

Design a distributed tracing system: the system design interview walkthrough

Design a distributed tracing system: the system design interview walkthrough

The request to design a distributed tracing system sounds like it is already solved — stamp a request id on every log line and grep for it later. For a handful of services that is nearly enough. But the interview asks you to follow a single request as it fans out across dozens of microservices, reconstruct the exact tree of calls and where the milliseconds went, do it for a million requests a second each spawning twenty spans, add almost no latency to the request path, and still keep the one slow or failing trace that actually matters — and that is a throughput, sampling and storage problem, not a correlation id. This is one of the harder system design interview prompts because the simple brief hides a firehose of spans, a context that has to survive every network hop, and a distributed tracing system design that lives or dies on what it is willing to throw away.

The round turns on three decisions. First, the data model and how context travels — a trace is a tree of spans, and the only thing holding that tree together is a trace id and parent pointers propagated across every service boundary, including async ones. Second, sampling: twenty million spans a second is too expensive to keep, so the system decides what to store — cheaply at the front door, or expensively after it has seen the whole trace. Third, storage and reassembly: spans arrive out of order from many services and must be collected by trace id and rebuilt into one timeline, cheaply enough to retain for days. This walkthrough follows the path a strong candidate draws on the whiteboard — requirements, estimate, API and data model, high-level design, then the deep dives — and at each step it names the follow-up the interviewer uses to tell a memorised answer from an engineer who understands the tradeoff.

Title card for a system design interview walkthrough on designing a distributed tracing system, showing a single request fanning out across services that each emit a span, the spans flowing through a collector into a buffer and store, and reassembling into a trace tree, on a dark citrus-and-lavender background.

How the distributed tracing round is actually run

A distributed tracing round runs about 45 minutes on a shared canvas — you draw the boxes and arrows while you talk, and the interviewer interrupts. It is a favourite for an infrastructure loop because the one-line problem opens onto deep water fast: the moment you say "give every request a trace id," the question becomes how does that id survive twenty hops, how do you not store twenty million spans a second, and how do you rebuild one request's tree out of spans that arrive late and out of order — and every answer after that (propagation, sampling, assembly, storage) compounds. Get the span model, the sampling strategy and the storage path clean and you have shown the whole observability pipeline in one problem.

The interviewer rarely lets "tag everything with a trace id and store it" stand. "A million requests a second times twenty spans is twenty million spans a second — you cannot keep all of it. What do you store, and how do you make sure you still catch the one trace that errored?" That question — surviving the span flood without sampling away the exact trace someone needs at 3am — is the one a copied answer can never rehearse, and the one this round is really testing.

The rounds here are modelled on how the loop runs, not scraped from any candidate's recall. What matters for prep is the shape: a blank canvas, a clock, and a follow-up after every decision you make.

The five-step framework for any HLD round

Every high-level design answer fits the same spine. Say it out loud at the start so the interviewer knows you have a plan, then walk it:

  1. Clarify requirements — functional and non-functional, and what you are explicitly not building.
  2. Estimate — requests per second, spans per request, bytes per span and read vs write mix, to justify every later choice.
  3. High-level design — the boxes and arrows: instrumented app, collector, queue, processing, span store, trace index, query.
  4. Deep dive — the two or three decisions that actually decide this system.
  5. Follow-ups — bottlenecks, failure modes, and what you would change at ten times the scale.

What the debrief measures

The score is not "right or wrong design." A good HLD debrief reads on four dimensions, and every move below maps to one: problem solving (did you reach for sampling the moment the span volume blew up, and pick a policy that keeps errors), communication (did you say why a trace is a tree and context must propagate, in a sentence), technical execution (W3C trace context propagation, collector plus Kafka buffer, head vs tail sampling, store-and-reassemble by trace id), and time management (did you spend your 45 minutes on propagation, sampling and storage, not on the exact wire format of a span). Reading a walkthrough teaches the design; it does not teach you to deliver it on the clock while someone pushes back.

Step 1 — Requirements: what a distributed tracing system must do

Spend the first few minutes here. Jumping to boxes before you have scoped the problem is the fastest way to design the wrong system.

Type Requirement
Functional Assign a trace id to each incoming request and propagate it across every service it touches; record a span per unit of work with timing and parent/child links; collect spans from all services; reassemble them into one trace by trace id; let an engineer look up a trace and search traces by service, operation, latency and error.
Functional (optional) Build a service dependency graph from the spans; correlate traces with logs and metrics; attach the trace id to log lines — call these out, then scope most of them for later.
Non-functional Extremely high, bursty span write throughput; near-zero overhead on the traced request path (instrumentation must never block or slow the app); the pipeline is best-effort — losing some spans is acceptable; fast trace lookup by id and fast recent search; cost-efficient retention for days; horizontally scalable and fault-tolerant.

The sentence that earns points: "This is a write-heavy telemetry pipeline that must add no latency to the traced request and cannot afford to store every span — so the real design is context propagation, a sampling policy that keeps the interesting traces, and a store you can reassemble a trace out of." Say that and you have framed the entire design before drawing a box.

Step 2 — Capacity estimation (back of the envelope)

Estimation here is about proving one thing: the hard axis is a flood of spans you mostly cannot afford to keep, which is why sampling is not optional and the write path dominates.

Quantity Assumption Result
Request rate ~1M requests/sec at peak the top of the funnel
Spans per request ~15–20 services/hops touched ~15–20M spans/sec at peak
Span size ~500 bytes–1KB (ids, timestamps, tags) ~10GB/sec of raw span data → sampling is mandatory
After sampling keep ~1% (plus all errors/slow traces) ~150–200k spans/sec stored → still write-heavy, now affordable
Reads trace lookups by id + searches by service/latency point lookups + recent search, dwarfed by writes

Two conclusions fall out, and you should say both out loud. First, you cannot store everything — twenty million spans a second is roughly ten gigabytes a second of raw data, so a sampling decision sits in the path and the only question is where it is made and what it keeps. Second, the write path is the system — spans must be exported off the request thread, buffered, and drained asynchronously so tracing never slows the app it is observing, which is why a collector and a durable queue sit in the middle. You have now pointed the interview at sampling and ingestion — the real deep dives — before anyone asked.

Step 3 — API and data model

A distributed tracing system's "interface" is a thin span-ingestion contract plus a query API, and underneath it the span — the one data structure the whole system is about.

Stage call Operation
span.start(trace_id, parent_id, name) → span the instrumented service opens a span for a unit of work, inheriting the trace id and parent from the incoming context
inject(ctx, carrier) / extract(carrier) write the trace context into an outgoing request's headers; read it from an incoming request to continue the same trace
collector.export(batch) the SDK ships a batch of finished spans off the request path to a local collector/agent, which forwards them upstream
getTrace(trace_id) → spans[] fetch every span for a trace id and rebuild the tree
searchTraces(filter) → traces[] find traces by service, operation, latency bucket, error flag or tag over a time range

The data model is the span, plus the trace it rolls up into and the context that carries it between services:

Entity Notes
span trace_id, span_id, parent_span_id, service, operation, start_ts, duration, status (ok/error), plus a bag of tags/attributes and timed events. A unit of work.
trace all spans sharing a trace_id, assembled into a tree by parent_span_id — one request's end-to-end timeline.
trace_context the propagated trace_id + current span_id + a sampled flag, carried over the wire (the W3C traceparent header) so every hop joins the same trace.
service_graph an aggregate derived from spans: which service calls which, with latency and error rates — built offline from the span stream.

Step 4 — High-level design: the tracing pipeline

Now the boxes. The whole system is one pipeline that moves spans from instrumented apps to a place you can look them up and search them, with a collector and a buffer in the middle so the apps never block and bursts never hit the slow parts directly:

  • Instrumented app (SDK): a tracing library (OpenTelemetry and the like) creates spans, reads the incoming trace context, and exports finished spans in batches off the request thread — so the request never waits on tracing.
  • Collector / agent: a local agent or gateway collector receives spans, does the first sampling decision if head-based, buffers, and forwards upstream — it decouples the app's SDK from the backend.
  • Ingestion queue (Kafka): a durable, partitioned queue absorbs the span burst and decouples collectors from processing; partitioning by trace id keeps all of one trace's spans together for downstream sampling and assembly.
  • Processing / sampling workers: consume spans, apply the sampling policy (and, for tail sampling, buffer a trace's spans until it is complete before deciding), then write the kept spans to storage and index them.
  • Span store + trace index + query service: spans land in a store keyed by trace id (for assembly) and indexed by service/operation/latency/error/time (for search); a query service serves single-trace lookups and trace search, and a background job rolls spans up into the service dependency graph.

Say explicitly that spans are exported asynchronously and buffered, never written synchronously on the request path — the app hands spans to a local collector and moves on, and everything slow (sampling, storage, indexing) happens downstream, which is exactly why tracing can run in production without adding latency and why a backend slowdown only grows the queue instead of hurting live traffic. Naming the async export and the buffer unprompted is a signal the interviewer is listening for.

A distributed tracing architecture diagram showing instrumented applications with tracing SDKs exporting batched spans to a collector, the collector forwarding into a Kafka ingestion queue partitioned by trace id, sampling and processing workers that apply the sampling policy and write kept spans to a span store and trace index, and a query service plus a service-graph builder reading from storage.

Deep dive: the span model and context propagation

This is the first decision that actually decides the round, because it is the one that makes a distributed trace possible at all. A trace is a tree: one root span for the request, child spans for each downstream call, each span carrying its own span_id and a parent_span_id that points at the caller. Reassembling that tree later only works if every service stamped the right ids — and that is entirely a propagation problem.

How the context travels. When a request enters the system, the first service either reads an existing trace context or mints a new trace_id. On every outbound call it injects the context — the trace_id, its own span_id as the new parent, and a sampled flag — into the carrier: an HTTP header for synchronous calls (the W3C traceparent standard, e.g. traceparent: 00-4bf9...4736-00f067aa0ba902b7-01), or message metadata for a queue. The next service extracts it, opens a child span with parent_span_id set to the caller's span, and repeats. Miss the propagation on one hop — an async job, a thread pool, a third-party client that drops headers — and the trace breaks into two disconnected trees. That instrumentation-gap problem is the one to name.

The clock-skew trap. Spans come from different hosts whose wall clocks disagree by milliseconds, so you cannot order spans by comparing timestamps across machines. Causality comes from the parent/child structure, not the clock: a child span is within its parent by construction. You render durations from each span's own start_ts and duration (measured on one host, so self-consistent) and lay children under parents by the tree, not by sorting global timestamps. Saying "I order by the span tree, not by cross-host clocks" is the senior tell here.

"A trace spans twenty services on twenty hosts with slightly different clocks — how do you order the spans and rebuild the request's timeline?" You do not sort by wall-clock timestamp across hosts; you rebuild the tree from parent_span_id and show each span's own measured duration. Causality is the parent/child structure; the clock only measures a single span locally. Name the tree-based assembly and the clock-skew trap.

A span-model and context-propagation diagram showing a request entering an API gateway and fanning out to payment, inventory and database services, each hop carrying a W3C traceparent header with the shared trace id and a new parent span id, the spans assembling into a trace tree by parent pointers, and a note that ordering comes from the tree not from cross-host clocks.

Deep dive: sampling at scale — head vs tail

This is the decision that decides the round. Twenty million spans a second is too expensive to keep, so the system stores a fraction — and how it picks that fraction is the whole ballgame, because the traces worth keeping (errors, slow outliers) are rare and uniform random sampling throws most of them away.

Head-based sampling. Decide at the start, on the root span, before you know how the request turns out — usually a deterministic function of the trace id (keep 1 in N), with the decision encoded in the sampled flag and propagated so every service on the trace makes the same choice and you never get half a trace. It is cheap and stateless, adds no buffering, and is how Google's Dapper ran production at roughly 1-in-1,024. Its flaw is that it is blind to outcome: the request that errored or took two seconds is sampled away at the same rate as everything else, so your rare, interesting traces are exactly the ones you lose.

Tail-based sampling. Decide at the end, after the whole trace has finished, so you can keep because it is interesting — any trace with an error, any trace over a latency threshold, a small random slice of the boring rest. The catch is mechanical: to judge a complete trace you must first buffer all of its spans, which means every span of one trace has to reach the same sampler (route by trace id) and the sampler must hold spans in memory for a wait window until the trace looks done. That costs memory and routing, and a trace that runs longer than your window gets a sampling decision on an incomplete picture. It is strictly more useful and strictly more expensive and complex.

The senior answer is usually both: a modest head sample to cap raw volume cheaply, then tail sampling on what survives to guarantee you keep every error and slow trace. Name the tradeoff explicitly — head is cheap and outcome-blind, tail is outcome-aware and needs trace-affinity routing plus a buffer — and note the subtle cost: heavy sampling skews your latency percentiles, because the slow requests are the ones most likely sampled out, so you often compute RED-style metrics before sampling.

"You can only afford to keep one percent of traces — how do you make sure the error traces and the slow ones survive?" Head sampling alone can't: it decides on the root before the outcome is known. Add tail sampling — buffer each trace's spans (routed by trace id so they land on one sampler), wait for the trace to complete, then keep it if it errored or breached a latency threshold, plus a random slice of the rest. Name head vs tail and why tail needs trace-affinity routing and a buffer.

A head-versus-tail sampling comparison diagram, two cards. Left card head-based sampling: decide on the root span from the trace id, propagate a sampled flag so the whole trace agrees, cheap and stateless but blind to errors. Right card tail-based sampling: route all of a trace's spans to one sampler by trace id, buffer until the trace completes, then keep it if it errored or was slow plus a random sample, outcome-aware but needs memory and routing.

System Design
HLD · Infrastructure · 45 min
Design a distributed tracing system out loud, with follow-ups

Practise on Zynter →

Deep dive: storage, trace assembly and queries

This is the decision most candidates under-think. Once spans are sampled and buffered, two very different access patterns fall on the store, and a single design for both is wrong: you look up one trace by id and rebuild its whole tree, and you search across many traces by service, latency or error. The store has to serve both without scanning everything.

Writing and assembling. Spans stream in out of order and from many services, so the store is keyed so that all spans of a trace_id sit together — a point lookup gathers them, and assembly rebuilds the tree by walking parent_span_id. Because spans of one trace can arrive late (a slow async hop finishes after the rest), assembly waits a bounded window and renders a partial trace if a straggler never comes — a complete-trace guarantee is best-effort, and saying so is the point. A columnar or wide-column store (ClickHouse, Cassandra and the like) suits the high-write, append-only span stream.

Searching and the dependency graph. Separately, you index spans by service, operation, latency bucket, status and time so "show me the slow checkout traces in the last hour" hits an index, not a full scan — time-partitioned so a dated query touches one partition. A background job aggregates the span stream into the service dependency graph (who calls whom, with error and latency rates), which is a rollup, not a per-trace query. And like logs, traces tier by age: recent traces stay in a fast store, older ones roll to cheap object storage on a retention TTL, because almost every lookup is for something that happened minutes to days ago.

"How do you store spans so that both 'open this one trace' and 'find all slow traces today' are fast — and what happens when one span of a trace shows up late?" Key the store by trace id so a lookup gathers a trace's spans and you rebuild the tree by parent pointers; separately index by service/latency/time for search; aggregate a service graph offline. Late spans are handled with a bounded assembly window and a best-effort partial trace. Name the two access patterns and the late-span window.

System Design
HLD · Infrastructure · 45 min
Defend sampling and trace storage out loud

Practise on Zynter →

The follow-ups that decide the distributed tracing round

Scan the probes from each deep dive and they rhyme. Master these five and you are ready for a version of the question that is not drawn exactly like this one.

  • "Twenty million spans a second — what do you keep, and how do you still catch the error trace?" → A cheap head sample to cap volume, then tail sampling that buffers each trace and keeps it if it errored or was slow — outcome-aware, routed by trace id. (problem solving)
  • "Clocks differ across hosts — how do you order the spans?" → Rebuild the tree from parent_span_id and show each span's own measured duration; causality is the parent/child structure, not cross-host wall clocks. (technical execution)
  • "A hop drops the trace context — what happens?" → The trace splits into disconnected trees; propagation (inject/extract the traceparent on every hop, including async) is the whole game, and instrumentation gaps are the main failure mode. (communication)
  • "Does tracing slow down the traced service?" → No — spans are created in-process but exported asynchronously in batches off the request thread, and sampling caps the work; tracing is best-effort and must never block the app. (problem solving)
  • "How is this different from a logging system?" → Same ingest-and-correlate backbone, but tracing adds the span tree, context propagation across services, and tail sampling; a trace id stamped on log lines bridges the two. (technical execution)

Each maps to a debrief dimension. Reading this list teaches you the answers; it does not teach you to deliver them while someone interrupts — that only comes from saying them out loud.

How to prepare: a one-week plan

You do not need fifty systems. You need this one cold and the framework automatic, so the next "design X" is just new nouns in the same five steps.

  • Days 1–2: Learn the five-step framework until it is reflex. Do the tracing system end to end on paper — requirements, estimate, span/trace/context model, the app → collector → Kafka → processing → store/index → query pipeline.
  • Days 3–4: Drill the span model and propagation: trace as a tree, parent_span_id, injecting/extracting the W3C context on every hop, the clock-skew trap, and instrumentation gaps. Say each tradeoff in one sentence.
  • Days 5–6: Own sampling and storage — head vs tail, trace-affinity routing and the tail buffer, keeping errors and slow traces, store keyed by trace id for assembly vs indexed for search, late-span windows and retention tiers. Answer each of the five follow-ups out loud.
  • Day 7: A full timed 45-minute round, narrating the whole way, with someone (or something) interrupting. This is the rep that counts.

This post is the deep walkthrough behind one row of the broader Top 25 system design interview questions — once the tracing pipeline is automatic, the same spine carries the rest of the list.

Frequently asked questions

How do I design a distributed tracing system step by step?

Follow the five-step HLD framework. Clarify requirements (assign and propagate a trace id, record spans with parent/child links, collect and reassemble them, look up and search traces — at a million requests a second, near-zero overhead, best-effort); estimate to show the hard axis is a span flood you mostly cannot afford to keep, so sampling is mandatory and the write path dominates; define the span-ingestion and query contract plus the span/trace/trace_context model; draw the pipeline (instrumented app → collector → Kafka → sampling/processing workers → span store + trace index → query + service graph); then deep-dive the decisions that matter — the span model and context propagation, head vs tail sampling, and storage with trace assembly and queries.

What is the difference between head-based and tail-based sampling?

Head-based sampling decides whether to keep a trace at the very start, on the root span, usually as a deterministic function of the trace id, and propagates that decision so every service agrees and you never store half a trace. It is cheap and stateless but blind to how the request turned out, so it sheds error and slow traces at the same rate as everything else. Tail-based sampling decides after the trace completes, so it can keep a trace because it errored or breached a latency threshold — but to do that it must route all of a trace's spans to one sampler and buffer them in memory until the trace looks done, which costs memory and routing and struggles with traces longer than the wait window. The common production answer is both: a cheap head sample to cap raw volume, then tail sampling to guarantee the interesting traces survive.

How does trace context propagation work across services?

Each service, on every outbound call, injects the trace context — the shared trace_id, its own span_id as the new parent, and a sampled flag — into the carrier: an HTTP header for synchronous calls (the W3C traceparent standard) or message metadata for a queue. The receiving service extracts that context, opens a child span with parent_span_id pointing at the caller, and repeats on its own outbound calls. This is what stitches spans from many independent services into one trace. If a hop fails to propagate the context — a thread pool, an async job, a client that strips headers — the trace breaks into disconnected pieces, which is why instrumentation coverage is the main real-world failure mode.

How do you reassemble a trace from spans that arrive out of order?

Spans are written to a store keyed by trace_id, so a single point lookup gathers every span for one trace regardless of which service sent it or when it arrived. Assembly then rebuilds the tree by walking parent_span_id from the root, and lays each span's own locally measured duration under its parent — ordering comes from the parent/child structure, not from comparing timestamps across hosts whose clocks disagree. Because an async hop can finish after the rest of the trace, assembly waits a bounded window for stragglers and renders a best-effort partial trace if one never arrives; a guaranteed-complete trace is not something the system promises.

How is a distributed tracing system different from a logging system?

They share the same ingest-and-correlate backbone — agents or SDKs ship telemetry off the host into a durable queue, workers process it, and it lands in a store you can search — and a trace id stamped on every log line is exactly the bridge between them. The differences are the data shape and the volume control: logs are mostly independent lines with free-form text, controlled by log levels and retention, while traces are structured spans that assemble into a tree with timing and causality, carried between services by context propagation, and controlled by sampling rather than retention alone. Tracing tells you where in the request the time or the error went; the correlated logs tell you what happened inside that hop.

How do I practise a system design interview out loud?

Reading a walkthrough is not the same as defending a design while someone interrupts. On Zynter, the HLD track runs live voice rounds modelled on how real system design loops run — you talk through a shared design on the canvas, and a voice interviewer pushes with the follow-ups above one at a time, then gives you a debrief on problem solving, communication, technical execution, and time management. The score is a practice signal, not a hiring decision.

Practise it on Zynter

You can draw every span. The round is won on sampling and how you rebuild a trace.

Pick the Distributed Tracing round, draw the instrument-to-query pipeline on the canvas, and defend it out loud to a voice interviewer that keeps asking "what do you keep out of twenty million spans a second?" and "how do you order spans when the clocks disagree?" — then read the debrief. Zynter.ai is on-demand mock interviews modelled on real company loops, with 150+ rounds across 20 companies including the HLD system design track (Sep 2026).

Practise the Distributed Tracing round →
Browse all interview tracks

Top comments (0)