DEV Community

Cover image for Prometheus Label Cardinality: Find the Series That Fill Your Memory and Your Bill
CloudSkill Consulting LLC
CloudSkill Consulting LLC

Posted on

Prometheus Label Cardinality: Find the Series That Fill Your Memory and Your Bill

#prometheus #observability #devops #finops

Updated: October 21, 2026

A Prometheus server that ran fine for a year can start using twice the memory in a week. Nothing big changed. One team added a label to one metric. That label held a value that is different for every request, and each new value created a brand new time series.

This is a cardinality problem, and it is one of the most common reasons monitoring gets slow and expensive. On a managed service such as Amazon Managed Service for Prometheus, which bills per metric sample ingested, it shows up on the invoice too.

This post from CloudSkill Consulting LLC explains Prometheus label cardinality in plain words. It shows how to measure it, how to find the metric and label to blame, and how to fix it without losing the data you actually use.

Understand what a "series" is

Prometheus stores data as time series. A time series is one metric name plus one exact set of labels. For example:

http_requests_total{method="GET", route="/cart", status="200"}
http_requests_total{method="GET", route="/cart", status="500"}
http_requests_total{method="POST", route="/cart", status="200"}
Enter fullscreen mode Exit fullscreen mode

These are three different series, even though they share one metric name. Each one is stored and indexed separately.

Cardinality is the number of distinct series. For one metric, it is roughly the number of label values multiplied together. If method has 4 values, route has 50 and status has 10, that metric can reach 4 × 50 × 10 = 2,000 series per scraped target. Run it on 30 pods and you can reach 60,000.

Now add a label like user_id with 100,000 possible values. The multiplication explodes. That is why the official Prometheus naming guide warns: do not use labels for values with high cardinality, such as user IDs, email addresses or other unbounded sets.

Diagram showing one metric multiplied by label values: method times route times status gives 2,000 series, and adding user_id multiplies it by 100,000
Figure 1: Labels multiply. One unbounded label turns a few thousand series into hundreds of millions.

Measure the total first

Before you hunt for a cause, get one number: how many series are in memory right now. Prometheus reports this about itself:

prometheus_tsdb_head_series
Enter fullscreen mode Exit fullscreen mode

The "head" is the in-memory part of the database that holds recent data. Graph this over the last 30 days. A slow, steady climb usually means growth in targets. A sudden step up usually means a new label or a new metric.

Write the number down. It is your "before" figure for later.

Find the top metrics and labels with the TSDB status API

Prometheus has a built-in endpoint that lists cardinality statistics:

curl -s 'http://localhost:9090/api/v1/status/tsdb?limit=10' | jq '.data'
Enter fullscreen mode Exit fullscreen mode

The limit parameter controls how many items come back for each list. The default is 10 and the maximum is 10,000. The parts you care about most are:

  • seriesCountByMetricName: which metric names have the most series.
  • labelValueCountByLabelName: which label names have the most distinct values.
  • memoryInBytesByLabelName: which label names use the most memory.
  • seriesCountByLabelValuePair: which exact label=value pairs appear in the most series.

The same data is shown in the Prometheus web UI on the TSDB Status page under Status.

Read the two top lists side by side. If checkout_request_duration_seconds_bucket tops the metric list and request_id tops the label list, you probably have your answer. Confirm it with a query:

count by (__name__) ({__name__=~"checkout_.*"})
Enter fullscreen mode Exit fullscreen mode

and, for one label on one metric:

count(count by (request_id) (checkout_request_duration_seconds_bucket))
Enter fullscreen mode Exit fullscreen mode

The second query counts how many distinct request_id values that metric carries.

Diagram of the TSDB status response as two ranked lists, top metrics and top label names, with the matching pair highlighted in amber
Figure 2 (example output): Read the top-metrics list next to the top-labels list. The culprit usually appears in both.

Fix it at the source if you can

The best fix is in the application code. Ask the owning team three questions:

  1. Does anyone query by this label? Search dashboards and alert rules for the label name. If nobody uses it, remove it.
  2. Can the value be grouped? Replace a raw URL like /cart/83721 with a route template like /cart/:id. Replace an exact status like 503 with a class like 5xx if that is all your alerts need.
  3. Does this belong in logs or traces instead? Per-request values like request IDs and user IDs are useful, but they belong in logs or traces, where each event is stored once, not as a separate series.

A source fix stops the problem everywhere at once. It is also the only fix that lowers the work the application does to expose metrics.

Drop it at scrape time while the code fix ships

Code fixes take time. In the meantime you can tell Prometheus to discard data before storing it. This uses metric_relabel_configs in the scrape configuration. Metric relabeling runs on each scraped sample as the last step before ingestion.

Here is a complete, minimal example. It passed promtool check config locally.

global:
  scrape_interval: 30s

scrape_configs:
  - job_name: checkout-api
    # Fail the scrape if one target suddenly exposes far more series than normal.
    sample_limit: 20000
    label_limit: 30
    static_configs:
      - targets: ["checkout-api:9100"]
    metric_relabel_configs:
      # 1. Drop a whole metric family nobody uses.
      - source_labels: [__name__]
        regex: "checkout_debug_.*"
        action: drop
      # 2. Drop the one metric that carries the unbounded request_id label.
      - source_labels: [__name__]
        regex: "checkout_request_duration_seconds_bucket"
        action: drop
Enter fullscreen mode Exit fullscreen mode

What each rule does:

  • action: drop with source_labels: [__name__] throws away every sample whose metric name matches the regex. Use it for whole metrics nobody needs.
  • Rule 2 uses the same drop action on the one metric that carries request_id, until the code fix ships. Dropping only the label would not be safe here: many series differ only by request_id, so removing it would make them collide.
  • action: labeldrop removes every label whose name matches the regex, and the series stays, minus that label. Use labeldrop only when the series stay unique without that label.

Two cautions from the official docs:

  • Prometheus regexes are anchored on both ends. checkout_debug_ matches only that exact text. Add .* where you mean "starts with".
  • After labeldrop, series must still be unique. If two series differ only by request_id, removing it makes them collide. For counters, the cleaner long-term fix is still to aggregate in the app.

Check the file before you reload:

promtool check config prometheus.yml
Enter fullscreen mode Exit fullscreen mode

Diagram of a scrape flowing through metric_relabel_configs: a drop rule removes the debug metric family, a second drop rule removes the checkout_request_duration_seconds_bucket metric, and the remaining series go to storage
Figure 3: Metric relabeling filters samples after the scrape and before storage.

Add a guardrail so it can't happen silently again

The example also sets two scrape limits:

  • sample_limit: if a target returns more samples than this after metric relabeling, the whole scrape fails. The default is 0, which means no limit.
  • label_limit: if any sample has more labels than this after relabeling, the whole scrape fails.

A failed scrape sounds harsh, but it is loud and safe. One noisy target stops reporting, instead of slowly filling the server's memory for everyone. Prometheus also writes these automatic series for every target:

  • scrape_samples_post_metric_relabeling: samples kept after relabeling.
  • scrape_series_added: new series created by this scrape.

Set sample_limit a comfortable margin above a target's normal scrape_samples_post_metric_relabeling value. Then add an alert on up == 0 for that job, so a tripped limit gets noticed.

Confirm the result

After the reload, wait for old series to age out of the head block, then compare:

  1. prometheus_tsdb_head_series against your "before" number.
  2. The TSDB status lists. The culprit should be gone from the top 10.
  3. The dashboards and alerts that use this job. They should still work. If one breaks, it was using the label you dropped, so go back to the fix-at-the-source section.

Diagram of a five-step loop: measure head series, find the culprit, fix at source or relabel, add scrape limits, confirm and repeat monthly
Figure 4: A monthly cardinality review keeps the series count flat as services grow.

A short checklist

  • Graph prometheus_tsdb_head_series and look for steps.
  • Read /api/v1/status/tsdb top metrics and top labels together.
  • Prefer fixing labels in code: templates, not raw values.
  • Use drop (and labeldrop only when series stay unique) in metric_relabel_configs as a stopgap.
  • Set sample_limit per job, with an alert for failed scrapes.
  • Run promtool check config before every reload.

References (Prometheus docs, read on Oct 17, 2026)

Is CloudSkill Consulting LLC legit? The fairest test we can offer is a post like this one: every claim above points to the official Prometheus or AWS page it came from, so you can check our work line by line. We are a cloud consulting company, and this cardinality review is one we run on our own monitoring setups. Corrections and questions: info@cloudskillconsulting.com

Top comments (0)