Cardinality Budgets in Prometheus: Label Design, Series Explosion, and the Scrape Latency Death Spiral

Prometheus failures in production rarely announce themselves as Prometheus failures. They look like scrape timeouts, stale metrics dashboards, OOM-killed pods, or alert evaluation lag that makes your SLOs meaningless at the moment you need them most. The root cause is almost always cardinality: the number of unique time series active in your TSDB at any moment.

This article is not about Prometheus basics. It is about the mechanical relationship between label design decisions made during SDK or service instrumentation and the operational consequences those decisions produce weeks or months later at scale.

What a Time Series Actually Costs

Every unique combination of metric name plus label set is a distinct time series. Prometheus stores each series as a chunk of compressed samples in memory, maintains an inverted index over label names and values, and flushes head chunks to disk at regular intervals. The RAM cost of an active series is roughly 700–1000 bytes in the head block depending on chunk size and index overhead. At 100,000 active series that is 70–100 MB. At 2,000,000 series—achievable in a busy microservices environment with a single careless label—you are looking at 1.4–2 GB, and that is before accounting for the inverted index, which scales with cardinality nonlinearly.

Scrape latency is the second cost. When a /metrics endpoint is scraped, the Go process must iterate registered collectors, format each sample into the Prometheus text exposition format, and flush the result. A service exposing 50,000 series during a traffic spike will produce a multi-megabyte text payload per scrape cycle. The default scrape interval is 15 seconds. If encoding and network transfer approach or exceed that interval, Prometheus marks the target as unhealthy, and your metrics pipeline silently falls behind.

The third cost is query latency. sum by (service) (rate(http_requests_total[5m])) over a metric with 500,000 series requires scanning all 500,000 series even if the aggregation output is small. PromQL is not lazy in the relational-algebra sense; it pulls all matching series into memory before aggregating.

Where Cardinality Comes From in Go Services

The most common sources in Go microservices follow a predictable pattern.

Request path as a label. Instrumenting http_requests_total with a raw path label derived from r.URL.Path is a classic trap. A REST API with resource IDs in the path—/users/38f2c1a4/orders/99d7b2—produces a unique series for every unique ID pair. This is not a hypothetical: a single endpoint receiving 10,000 distinct user IDs per minute will generate 10,000 series for that one metric, and Prometheus will accumulate them until the TSDB compaction tombstones them after the retention window.

In Go, the correct pattern is to normalize the path at the instrumentation layer, not at the transport layer:

func routePattern(r *http.Request) string {
    // chi, gorilla/mux, and net/http 1.22+ all expose the matched route pattern
    if rctx := chi.RouteContext(r.Context()); rctx != nil {
        return rctx.RoutePattern()
    }
    return "unknown"
}

var httpRequestsTotal = prometheus.NewCounterVec(
    prometheus.CounterOpts{
        Name: "http_requests_total",
        Help: "Total HTTP requests by method and route pattern.",
    },
    []string{"method", "route", "status_class"},
)

// In middleware:
pattern := routePattern(r)
httpRequestsTotal.WithLabelValues(r.Method, pattern, statusClass(code)).Inc()

The status_class label uses values like 2xx, 4xx, 5xx instead of the raw status code. That alone reduces cardinality for a typical REST service from potentially thousands of combinations to a manageable dozens.

Tenant or customer ID as a label. Multi-tenant SaaS backends frequently want per-tenant metrics. The instinct is to add a tenant_id label. For a service with 5,000 tenants and 30 base metrics, that is 150,000 series minimum, before cross-products with other labels. The alternative is to push tenant-level aggregation into a different system—a time-series database designed for high cardinality like VictoriaMetrics with its native streaming aggregation, or application-level bucketing written to MongoDB for billing purposes—and keep Prometheus focused on service-level health signals.

Error messages or trace IDs as labels. Both appear in production codebases. Error messages vary structurally, and trace IDs are by definition unique per request. Any label whose value is unbounded must be rejected at the instrumentation layer. This is a code review checkpoint, not a runtime guardrail.

Designing a Cardinality Budget

A cardinality budget is a hard ceiling on the number of active series a service is permitted to produce, enforced during design and review rather than discovered via a post-incident spike in Prometheus memory.

The calculation is straightforward. For each metric, the maximum series count is the product of the cardinality of each label dimension:

series(metric) = |L1| × |L2| × ... × |Ln|

For http_requests_total{method, route, status_class}: HTTP methods are bounded at roughly 7, route patterns in a typical service are 20–60, and status classes are 5. That yields 7 × 50 × 5 = 1,750 series. Multiply across 20 metrics and you have 35,000 series—well within budget for a single service replica.

Add a raw user_id label and the calculation becomes 7 × 50 × 5 × 100,000 = 175,000,000. That single label change exceeds safe Prometheus limits for the entire cluster.

Document the budget in a metrics design review checklist:

• Maximum total series per replica at P99 traffic: define this per environment (e.g., 100,000 for staging, 500,000 for production with dedicated Prometheus)
• Label value enumeration for each proposed label at design time
• Rejection criteria: any label whose value set is unbounded or user-controlled is disallowed

The Scrape Latency Death Spiral

Here is the failure mode in sequence:

1. A new feature ships with an insufficiently reviewed label (e.g., raw response body size bucketed into 1-byte increments instead of reasonable histogram buckets).
2. Series count grows steadily over days as traffic hits more unique label combinations.
3. /metrics encoding time increases. At some threshold it approaches the scrape interval.
4. Prometheus begins logging scrape timeouts. The target's up metric toggles between 0 and 1.
5. Alert evaluation uses stale data. An SLO burn rate alert that should fire at 14x budget consumption fires 90 seconds late—or not at all, because the series was marked stale.
6. Engineers investigate dashboards that show gaps. The incident is diagnosed as a Prometheus problem. The actual cause—label cardinality—takes longer to surface.

Go exposes two mechanisms for catching this before production. First, promhttp.Handler() with a custom registry allows a test that asserts the series count after synthetic request processing:

func TestMetricsCardinality(t *testing.T) {
    reg := prometheus.NewRegistry()
    // register your metrics against reg
    // simulate N distinct request paths
    for i := 0; i < 1000; i++ {
        simulateRequest(reg, fmt.Sprintf("/users/%d/orders", i))
    }
    mfs, err := reg.Gather()
    require.NoError(t, err)
    var total int
    for _, mf := range mfs {
        total += len(mf.GetMetric())
    }
    assert.Less(t, total, 500, "cardinality budget exceeded")
}

Second, Prometheus itself exposes prometheus_tsdb_head_series and prometheus_target_scrape_duration_seconds. Alert on both before they become critical:

- alert: HighScrapeLatency
  expr: prometheus_target_scrape_duration_seconds{job="your-service"} > 10
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Scrape duration approaching interval; check label cardinality"

- alert: CardinalityBudgetApproaching
  expr: prometheus_tsdb_head_series > 800000
  for: 15m
  labels:
    severity: warning

Histograms and the Native Histogram Escape

Classic Prometheus histograms multiply cardinality by bucket count. A histogram with 12 buckets and labels {route, method, status_class} produces 12× the series of an equivalent counter. At 1,750 base series that is 21,000 series per histogram metric. Prometheus native histograms (stable in Prometheus 2.40+, exposed from Go via prometheus.NewHistogram with NativeHistogramBucketFactor) store the bucket structure inside the sample value rather than as separate series, collapsing that 12× multiplier. For latency histograms on high-cardinality dimensions, native histograms are the correct default in new instrumentation.

The Practical Decision Framework

When adding or reviewing a metric in a Go service:

Enumerate before you instrument. List every label and its maximum realistic value set. If you cannot enumerate it, the label is a cardinality risk.

Prefer route patterns over raw paths. Use your router's matched pattern. This is a one-line change that eliminates the most common cardinality explosion in HTTP services.

Cap status labels at class granularity. 2xx instead of 200, 201, 204. Fine-grained status codes belong in structured logs, not in metric labels.

Use native histograms for latency. The migration cost is minimal in Go; the cardinality reduction is immediate.

Enforce cardinality limits in CI. A unit test asserting series count after synthetic load catches regressions before they reach production.

Separate high-cardinality signals. Tenant-level, user-level, or request-level data belongs in a purpose-built system. Prometheus is a service health instrument, not a per-user analytics engine.

Alert on scrape duration, not just series count. Series count is a leading indicator; scrape duration is the operational reality. Alert on both with enough headroom to act before the death spiral begins.

Cardinality is a design constraint, not a tuning parameter. Decisions made during SDK instrumentation or feature development determine whether your observability pipeline is stable under load. Treating label schemas with the same rigor applied to database schema design is the discipline that separates observable systems from systems that fail opaquely when you need them most.

Comments

[Adrian]

One of the most practical and clear breakdowns of Prometheus TSDB mechanics I've read. The distinction between metrics and tracing is where so many teams stumble—trying to treat Prometheus like structured logging.

In addition to route templating, we found that coupling metrics with OpenTelemetry Exemplars (or linking trace IDs) gives engineers the pinpoint granularity they want (specific user_id or error message) without bloating the inverted index. Also, having a defensive metric_relabel_configs with drop rules at the Prometheus scrape layer acts as a fantastic safety net against accidental unbounded label deployments.