Your p99 latency alert just fired, the Grafana graph shows a spike at 14:07, and the on-call question never changes: which request was slow? Prometheus exemplars answer exactly that. They pin the ID of a real trace to a metric point, so you jump from the graph to the OpenTelemetry trace in one click.
The facts come from the Prometheus documentation, the OpenMetrics specification and the Grafana documentation, read on 12 November 2026. I did not build a full stack for this article: the screenshots render example files built from those pages, and anything I did not verify is flagged.
If spans are new to you, start with the anatomy of an OpenTelemetry span. The two signals complement each other: the metric tells you when and how much, the trace tells you which one.
The problem: an average points at no request
A latency metric folds thousands of requests into a few counters. That is what keeps it cheap. But when p99 climbs, you need one concrete request, and the metric has already forgotten it.
Without a direct link, you dig: note the time of the spike, open the trace tool, filter by service and duration, then hope sampling did not drop the right trace. Every minute spent there adds to your MTTR.
The operational decision is therefore simple: are you willing to store a handful of trace references next to your series? That is what an exemplar is, and it has a precise memory cost, quantified below.
Prometheus exemplars: what the specification allows
An exemplar is a reference to data outside the metric, typically a trace ID. The Prometheus feature flags page ties it to OpenMetrics: that format lets a scrape target attach exemplars to certain metrics.
The OpenMetrics specification narrows the scope. It allows an exemplar on histogram bucket values and on a counter total. It says nothing about gauges or summaries. On a text line, the exemplar follows the value after a hash sign, with its own labels and observed value.
One constraint to remember: per the specification, the combined length of an exemplar’s label names and values must not exceed 128 UTF-8 code points. A trace ID fits easily. Do not slip in other labels « for extra context ».
Enabling exemplar-storage without a memory surprise
By default, Prometheus does not keep the exemplars it scrapes. You must start the server with --enable-feature=exemplar-storage. According to the documentation, exemplars then go into a fixed-size in-memory circular buffer shared by all series.
- Start Prometheus with the
exemplar-storagefeature flag. - Size the buffer in the
storagethenexemplarsblock of the configuration:max_exemplarsdefaults to 100,000 on the configuration page and can be resized at runtime. - Check that your targets really expose exemplars in their scrape format.
The documentation gives an order of magnitude: an exemplar holding only a trace_id weighs roughly 100 bytes. At 100,000 entries that is around 10 MB, which is my arithmetic from that figure, not a measurement. Because the buffer is circular, the oldest exemplars disappear first. On a busy fleet the window may be only a few minutes: check it on your own system.
Also, per the same page, exemplars are written to the WAL too, for local persistence and for the WAL’s lifetime. I did not read how remote write handles them: if you ship to remote storage, read that section, like the one in Grafana Mimir for Prometheus metrics storage.
Querying exemplars through the API
Before configuring Grafana, check that Prometheus actually received them. The HTTP API offers /api/v1/query_exemplars, as GET or POST, marked experimental in the documentation. It takes a PromQL query and a start / end window.
The response lists, per series, its seriesLabels and an exemplars array. Each exemplar carries its labels (for example trace_id), a value and a timestamp in Unix seconds. If that array is empty, go no further: the problem is upstream, in exposition or in the feature flag.
From the graph to the OTel trace in Grafana
In Grafana, an exemplar shows as a star on the graph. Hovering reveals the trace ID, and a button opens the trace in the tracing data source, Tempo in the documentation’s example. Exemplar support is stated for the Prometheus data source only.
The link is declared in the Prometheus data source. The Label name field gives the key of the label carrying the ID, and an internal link points to the tracing source. As provisioning, it looks like this.
If you have no trace backend yet, our article on Grafana Tempo with kube-prometheus-stack covers an install. The real watch point lies elsewhere: the ID must be the same on both sides. It is when the application reads the trace_id of the active OpenTelemetry span at the moment it records the duration.
The sampling trap
This is the case that catches people most often. Your exemplar points at a trace that tail sampling discarded: the click lands on a « not found » page. The application records the exemplar, while the decision to keep the trace is made further down the pipeline.
Two remedies, to weigh against your budget. Always keep slow and failed traces, as shown in the Collector tail sampling rules. Or record an exemplar only when the trace is flagged as sampled: I did not verify that your client library offers this, so check its documentation.
Good practices
- Start with the latency histogram of one critical service, not the whole estate.
- Keep
trace_idas the only exemplar label: the limit is 128 characters. - Check with
query_exemplarsbefore blaming Grafana. - Keep slow traces in tail sampling, or the star leads nowhere.
- Write it in the runbook: « p99 spike, hover the stars, open the slowest trace ».
Frequently asked questions
Do exemplars work with any metric?
No. The OpenMetrics specification provides them for histogram buckets and counter totals. It does not describe them for gauges.
How much memory do exemplars cost?
The documentation gives about 100 bytes for an exemplar reduced to a trace_id, in a fixed-size buffer set by max_exemplars (100,000 by default). Measure on your own system before changing it.
What if I use Dynatrace?
This article only covers Prometheus, Grafana and OpenMetrics. For linking metrics to traces in Dynatrace, see docs.dynatrace.com and keep SaaS and Managed apart: I mix no feature from it here. To compare approaches, see OpenTelemetry and Prometheus.
Conclusion
Prometheus exemplars turn a latency spike into a list of candidate traces for a few megabytes of memory. Enable the feature flag, check with the API, declare the link in Grafana, then protect your slow traces from sampling. Sources: Prometheus feature flags, OpenMetrics specification, exemplars in Grafana.
span.observer is an independent community blog, not affiliated with Dynatrace LLC, Grafana Labs or the Prometheus Authors. Trademarks belong to their owners. Tools change fast: check commands and options against the official documentation.
Originally published on span.observer. French version: https://span.observer/fr/exemplars-prometheus-pic-latence-trace/.



Top comments (0)