📈 One Path Variable Can Bankrupt Your Prometheus

A path variable in your metrics tag will quietly bankrupt your Prometheus backend.

It passes code review. It compiles. It works, for a while. Then one day the dashboards start timing out, and it takes the team days to trace the slowdown back to a single line of instrumentation that looked completely normal.

The line that looks fine

Here is the kind of code that ships this problem. A Timer around an order lookup, tagged with the orderId so you can slice latency per request.

@GetMapping("/api/orders/{orderId}")
public Order getOrder(@PathVariable String orderId) {
    return Timer.builder("order.lookup")
            .tag("orderId", orderId)   // <-- the problem
            .register(meterRegistry)
            .record(() -> orderService.findById(orderId));
}

Nothing about this fails a review. It reads as "measure how long an order lookup takes, and let me break it down by order." Reasonable intent. The metric even works when you test it locally with three orders.

What actually happens

Prometheus is a time series database. The identity of a time series is its metric name plus the full set of label key-value pairs. Change one label value and you do not add a point to an existing series. You create a brand new series.

So order.lookup{orderId="1001"} and order.lookup{orderId="1002"} are two completely separate series, each with its own storage, its own index entry, its own memory footprint.

Now run that in production. Every distinct orderId that flows through the endpoint mints a new series. A million orders means a million series from this one metric. Add a userId tag somewhere else and the counts multiply. This is cardinality explosion, and /actuator/prometheus will happily expose all of it.

The failure is gradual, which is what makes it nasty:

1. Storage grows far faster than your request volume would suggest.
2. The Prometheus head block balloons, memory pressure climbs, scrape durations creep up.
3. Queries that touch the metric get slow, then time out.
4. Someone files "Prometheus is slow" or "the orders dashboard times out."

Notice what is missing from that chain: nobody says "the orderId tag is the problem." The symptom is three steps removed from the cause. I have watched a team spend the better part of a week bisecting scrape configs and bumping memory limits before someone finally ran a cardinality check and found one metric responsible for millions of series.

You can catch it directly once you suspect it:

# top metrics by series count
topk(10, count by (__name__)({__name__=~".+"}))

But you have to suspect it first, and the whole point is that nothing pointed you there.

The fix is not a new tool

The instinct is to reach for a sampling library or a relabeling rule to drop the bad label. You do not need either. Spring already hands you the right value.

For every request, Spring resolves the matched route pattern, the same mechanism its built-in HTTP server metrics (http.server.requests) use to keep their uri tag bounded. The best-matching pattern is available on the request as an attribute.

@GetMapping("/api/orders/{orderId}")
public Order getOrder(@PathVariable String orderId, HttpServletRequest request) {
    String pattern = (String) request.getAttribute(
            HandlerMapping.BEST_MATCHING_PATTERN_ATTRIBUTE); // "/api/orders/{orderId}"

    return Timer.builder("order.lookup")
            .tag("route", pattern)     // bounded to the number of routes
            .register(meterRegistry)
            .record(() -> orderService.findById(orderId));
}

Now the tag value is /api/orders/{orderId}, the template, not the resolved id. Every request through this endpoint lands on the same series. Cardinality for this metric is bounded to the number of routes you have, which is a small, fixed number that does not grow with traffic.

If all you wanted was per-endpoint latency, you may not need a custom timer at all. http.server.requests already gives you templated URI, method, and status out of the box. Reach for a custom metric only when you need a dimension the built-in one does not expose, and when you do, tag it with something bounded.

The honest trade-off

Templating the URI collapses per-entity granularity in the metric itself. That is a real loss, not a footnote.

Once the tag is /api/orders/{orderId}, the metric can no longer answer "how slow was the lookup for order 1002 specifically." It can only tell you about the route as a whole: p50, p99, error rate across all orders.

If you genuinely need to investigate one entity, that is now a logs or traces question, not a metrics question. Attach the orderId to a span or a structured log field, where high cardinality is expected and the backend is built for it. Metrics are for bounded, aggregatable dimensions. Traces and logs are for the long tail of individual cases.

That split is the right one. Metrics answer "is the system healthy and how is this route trending." Traces answer "what happened to this one request." Putting a unique id in a metric tag is asking metrics to do the job of tracing, and the bill for that mistake is paid by your storage backend, quietly, until it is not quiet anymore.

The rule

Any tag value that is not drawn from a small, known set does not belong on a metric. User ids, order ids, request ids, email addresses, raw paths: all of them are cardinality bombs. Route patterns, status codes, HTTP methods, enum-like states: all fine.

The tell is simple. Before you add a tag, ask how many distinct values it can take over the life of the service. If the answer scales with your traffic, you have found the bug before it finds you.

---

Has your Prometheus setup ever fallen over from a label nobody flagged, and how long did it take to trace it back to the metric? Curious how other teams caught it, and what guardrails you put in place afterward.