Our metrics bill went up by a factor of six in one quarter, and the cause was a single line in a middleware: someone had added the request path as a label on an HTTP counter. Not the route template. The path. Every unique URL, including ones containing order IDs, session tokens and, for a while, a search query string, became its own time series.
Cardinality is the one observability failure mode that punishes you slowly and then all at once. The change looked harmless in review and worked perfectly in staging, where a handful of test users produced maybe forty distinct paths. In production it produced two point three million series in three weeks. Queries that used to return in under a second started timing out. The Prometheus instance OOMed during an incident, which is a special kind of unlucky, because the thing you use to understand outages became part of one.
What made it hard to spot was that nothing was obviously wrong. Dashboards rendered. Alerts fired. The degradation was in query latency and retention, which nobody watches until they need them. We only found the cause by querying the metrics system about itself and sorting series counts by metric name, at which point the answer was a single row taller than everything else combined.
The fixes were mechanical. Label values must come from a bounded set known at code-review time: route templates rather than paths, status classes rather than exact codes where the exact code doesn't drive a decision, and never a user identifier, ever. We put a limit on the scrape config so a runaway target gets dropped instead of taking the server with it, and added an alert on series growth per job so the next one shows up in a day rather than a quarter.
The habit change matters more than the config. Before adding a label, ask how many distinct values it can take over a year, and be honest that the answer for anything derived from user input is "unbounded." High-cardinality data belongs in logs or traces, which are built for it. Metrics are for aggregates.
A dimension you cannot bound is not a dimension. It's a leak with a schema.
– Sergey Shinder
Top comments (0)