Prometheus exemplars are the shortest path we know from "the slow tail on this route moved at 14:10" to one specific request. The histogram sample carries a bounded reference, usually a trace ID, and the dashboard turns the point on the graph into a link. It removes the step where an on-call engineer guesses labels and scrolls a trace search by timestamp.
What surprises teams is how many independent systems have to agree before that click resolves.
Emitting a trace ID is one hop. The latency observation has to be recorded while the request's span context is still current and valid, and that is exactly where wrapper metrics, asynchronous callbacks, thread hops and queue consumers lose it. Automatic instrumentation covering a service is not evidence that it covers those paths. The exposition format and the scraper then have to preserve the exemplar, and so does remote write, and so does the long-term store: exemplar support and retention can differ from sample support, so forwarding is worth testing on its own rather than assumed from a working metric. The dashboard has to map the ID to the correct trace data source and tenant, carrying a time range with margin for clock skew. And the trace backend has to still hold the trace.
That last one is the failure we would look for first. Metrics commonly live for months and traces for days, so a link on a two-week-old graph is predictably dead, and nothing about the graph says so.
The subtler version is a sampling mismatch. If the metric SDK records an exemplar from a trace context that tail sampling later drops, the reference was written for something that no longer exists. The reservoir policy and the trace sampling policy have to be coordinated, or two systems make independent decisions about which request mattered and then disagree.
So the rule we work to is that correlation is a path, and each hop gets tested. Measure link resolution as a rate and split the failures by cause: no active context, reservoir selection, exposition loss, storage loss, trace sampling, expiry, tenant mapping, dashboard configuration. Without that split, "exemplars sometimes don't work" turns into raising trace retention when the fault was in metric ingestion.
One thing worth saying to whoever carries the pager: an exemplar is one representative observation from a bucket. It is not the worst request in that bucket and it is not proof of cause.
The full path is written up on our own site: choosing bucket boundaries around the alert threshold, keeping trace IDs out of ordinary metric labels, what each reservoir policy costs, and the failure-injection drill (collector delay, trace-backend rejection, remote-write retry, clock skew) that tells you which hop is actually broken.
Prometheus Exemplars: Move from a Bad Metric to the Exact Trace
Written by the engineering team at Edilec.
Top comments (0)