DEV Community: ibrohim syarif

The Dangers of High-Cardinality Labels in Prometheus

ibrohim syarif — Sun, 22 Feb 2026 04:51:57 +0000

We're all familiar with the warnings: "Don't use user_id as a Prometheus label" or "Don't use transaction codes as labels — they can crash Prometheus". But do we really understand why these are so dangerous?

Before that, we need to know how Prometheus works.

How Prometheus Works

Prometheus is an open-source systems monitoring and alerting tool that collects and stores its metrics as time-series data. It periodically scrapes metrics from your services based on the configured interval.

This is an example of the config:

scrape_configs:
  - job_name: 'golang-app'
    static_configs:
      - targets: ['localhost:8080']
    scrape_interval: 5s

This config will tell Prometheus to:

Target: send an HTTP request GET to http://localhost:8080/metrics
Periodically: for every 5 seconds
Label: with job=golang-app

Prometheus has three metric types:

Gauges represent current measurements and reflect the current state of a system, such as CPU usage and memory usage.
Counters measure discrete events that continuously increase over time. Common examples are the number of HTTP requests received, CPU seconds spent, and bytes sent.
Histogram tracks the distribution of observed values. For a base metric name <basename>, it exposes multiple related time series:

<basename>_bucket{le="..."} — Cumulative counters representing the number of observations that fall within each bucket boundary
<basename>_sum — The total sum of all observed values
<basename>_count — The count of events that have been observed

Time Series Database (TSDB)

Prometheus collects and stores metrics as time series. Each time series is uniquely identified by a metric name and a set of labels, while each sample within the series contains a timestamp and a value. Each unique combination of labels (method, path, and status) represents a separate time series whose value increases as more requests are processed, with total method x path x status.

In the Counter metrics example, suppose we have two endpoints: GET: /api/data, GET: /api/users, and each of which can return either a 200 or 500 status code. This results in the following metrics:

http_requests_total{method="GET", path="/api/data",  status="200"} 17
http_requests_total{method="GET", path="/api/data",  status="500"} 0
http_requests_total{method="GET", path="/api/users", status="200"} 10
http_requests_total{method="GET", path="/api/users", status="500"} 2

Because each time series represents a unique combination of labels, these four label combinations produce four distinct time series. In the time-series database (TSDB), each of these time series is stored independently:

// time series 1
2026-02-19 09:00:00 | {__name__="http_requests_total", method="GET", path="/api/data", status="200"} | 15
2026-02-19 09:00:05 | {__name__="http_requests_total", method="GET", path="/api/data", status="200"} | 16
2026-02-19 09:00:10 | {__name__="http_requests_total", method="GET", path="/api/data", status="200"} | 17

// time series 2
2026-02-19 09:00:00 | {__name__="http_requests_total", method="GET", path="/api/data", status="500"} | 0
2026-02-19 09:00:05 | {__name__="http_requests_total", method="GET", path="/api/data", status="500"} | 0
2026-02-19 09:00:10 | {__name__="http_requests_total", method="GET", path="/api/data", status="500"} | 0

// time series 3
2026-02-19 09:00:00 | {__name__="http_requests_total", method="GET", path="/api/users", status="200"} | 8
2026-02-19 09:00:05 | {__name__="http_requests_total", method="GET", path="/api/users", status="200"} | 9
2026-02-19 09:00:10 | {__name__="http_requests_total", method="GET", path="/api/users", status="200"} | 10

// time series 4
2026-02-19 09:00:00 | {__name__="http_requests_total", method="GET", path="/api/users", status="500"} | 1
2026-02-19 09:00:05 | {__name__="http_requests_total", method="GET", path="/api/users", status="500"} | 2
2026-02-19 09:00:10 | {__name__="http_requests_total", method="GET", path="/api/users", status="500"} | 2

The Dangers

Let's go back to the warning: "Don't use user_id as a Prometheus label" or "Don't use transaction codes as labels — they can crash Prometheus."

Imagine you want to record transaction latency using metric labels such as:

status: pending, paid, success, failed
payment_type: wallet, cash

Here, status has 4 possible values, payment_type has 2 possible values. It will produce status (4) x payment_type (2) = 8 time series

This is the example result of the metrics

there are exactly 8 label for the metrics

Then, you adjust the metrics by adding a code label, allowing request rates, error rates, and traffic patterns to be broken down per transaction

code: a unique identifier for each transaction

However, code is unique for every transaction and grows continuously with request volume. As a result, the number of possible values for code is unbounded and increases over time. status (4) × payment_type (2) × code (∞) = ∞ time series

This is the example result of the metrics

This single unbounded label is enough to turn an otherwise manageable metric into a high-cardinality time-series explosion that can cause memory exhaustion and query performance.

Summary

Adding labels whose values grow unbounded over time—such as UUIDs, timestamps, user IDs, or transaction codes—is strongly discouraged. These labels rarely add meaningful value at the metrics level and introduce high cardinality. For high-cardinality data, better use logging, not metrics

High-cardinality labels can lead to serious issues, including:

Huge memory usage — each unique label set creates a new time series
Rapid disk growth
Slow queries
Scrape performance issue

It's better to use labels that have semantic meaning, and strongly recommended to keep the number of labels to a minimum

A wise man says, "Never use a label whose value grows with users, requests, or time."

Good labels describe what something is, not who or which exact instance.

code: https://github.com/ibrohhm/prometheus-grafana-golang

Circuit Breaker Pattern

ibrohim syarif — Wed, 04 Dec 2024 12:00:00 +0000

Integrating with partners often got unexpected behavior due some isssue on their server that impact to our service performance. lets say the integration flow look like this

If the partner responds successfully, our service forwards the response data to the client. Otherwise, if the partner returns an error, our service will relay the error message to the client. Similar to our server, the partner have maintenance or unexpected issue that make it inaccessible. When their server fails to respond, every request to their server will get not responding error and giving unnecessary waiting time, with huge traffic this issue very possible will cause our server to crash. So what should we do to prevent that happen?

Solution

The issue in this article isn't about the persistent errors from the partner but rather the additional response time caused by these errors, which could lead to our server crash (see this article https://dev.to/ibrohhm/crash-and-timeout-simulation-jbp). To solve this, we need add an another layer to manage the partner connection, acting as circuit breaker if the connection goes bad it will break the connection and return the request immediately without waiting for the partner response

the circuit breaker pattern have three states

closed means the service allow to make connections
half-open means the service allow to make connections with limited number
open means the service not allow to make connections, it will return error immediately

this is detail curcuit breaker flow

curcuit breaker allows us to control the partner connection effectively. By implement circuit breaker in our integration flow, we have no worries about the unexpected partner failure, it will cut the connection automatically and prevent our service from potential crashes due the unnecessary waiting times

Crash and Timeout Simulation

ibrohim syarif — Sun, 21 Jul 2024 08:35:21 +0000

Image you have apps that required called partner to served your data, the partner sometimes got unexpected behavior that we cannot control, let say it's random delay everytime you request from the partner. It's very tiny detail but if we are not handle the partner request well, it will causing our server down.

This article is focus on the simulation for your server to handle this partner behavior

To simulate this we will create three service (client, server, partner) using golang

client --> server --> partner

client: call the server with go routine
server: the server will forward the request from client to partner, act as middleware
partner: simple hello world golang with random timeout

Partner

Partner service is simple http call with random delay

The partner service only have get /data endpoint with response Hello from Partner Service with generate random delay everytime request the data (1-10 second delay). The partner also have logging to show the delay_set and time when request occur. So we can monitor the request well

See the implementation: (partner service)

How to run: go run partner.go

Server

Server service is your internal service to handle the request from client. To simulate the crash, we need to set the memory limit allocation (MemoryLimit) so we can simulate the crash without crashing your laptop. When running the server, it will checking the memory usage in every 1 second (getMemoryUsage) and the memory usage is exceed the MemoryLimit we will stop the server. The service also have logging to show the method, url, latency, status, error, and memory_usage

See the implementation: server service

How to run: go run server.go

Client

Client service is simple golang apps that will do 100 request in 1 second to the server with go routine. I choose to create client service instead of using load test application like JMeter, so we can see the logger for every request

See the implementation: client service

How to run: go run client.go

Simulation

In this section we will do three simulation

partner with no delay
partner with random delay but no timeout set in the server
partner with random delay with timeout set in the server

Case 1

Every case always have happy case and this is it, our partner service have good spec and never got delay everytime we request. to make this possible you need to change the delay on partner code from delay := time.Duration(rand.Intn(11)) to delay := time.Duration(0) (ref)

client --> server --> partner

this is the result

the partner, the server, the client is all good. everyone happy

Case 2

Our partner service have random delay (delay := time.Duration(rand.Intn(11))) and our server service not set the timeout when request to partner service

client --> server --> partner (random delay)

this is the result

our server got killed in 24 second since the memory usage exceed the MemoryLimit. This is because the client service continuosly spawn new request using goroutine to call server service, since there's no limit on the number of goroutine being spawend, the server service request the partner service with hugh number. Because of the partner delay, most of the request running at the same time and consume all available memory then leading to a crash

what happen if we set the timeout request on the server service?

Case 3

Our partner have random delay but our server set the timeout request. we need to change the Timeout variable in server to some number, let say 3 second const Timeout = 3 (ref)

client --> server --[with timeout]--> partner (random delay)

this is the result

If you running the simulation, you'll see our server service not get killed from the exceed memory allocation. if you look more closely in the logger, the memory_usage of the server is always around 6MB - 12MB (never exceed the 20MB) this is because the timeout killed the ongoing request and release the memory allocation

Summaries

The partner service behavior is the external thing that we cannot control, we cannot trust the partner to have good behavior. sometimes it got delay, sometimes we cannot access it doe their internal error or else. the small delay maybe will cause our server down (like the simulation), so we better to prevent that happen and one way to prevent it is like adding the timeout when request to the server

source code and simulation videos: https://github.com/ibrohhm/crash_and_timeout_simulation

Know Better About N+1 Queries Problem

ibrohim syarif — Wed, 29 Nov 2023 16:41:42 +0000

Overview

In the engineering process we often facing the case to query all the data based on it's parent and the data will be used for some reason. For example, let say there is users table that has correlation one-to-many with the transactions table, you need to get all the users and it's transactions based on the user_ids that given from argument. The simple logic that we will do is to get all the user with id include in user_ids then get all the transactions one by one

def load_data(user_ids)
    result = []
    users = User.where(id: user_ids)
    users.each do |user|
        result << { user: user, transactions: user.transactions }
    end

    result
end

it's really simple logic, but is it good enough? is it bad? is it our service can endure the high throughput? is there any way to make it more efficient?

Look Inside the Query

Let say we have this model

class User < ApplicationRecord
  has_many :transactions
end

class Transaction < ApplicationRecord
  belongs_to :user
end

we're gonna simulate the query in the rails console (run: rails console)

we see in the image, the load_data method called 4 queries to the database. The first one is query all the users based on the user_ids, and the 3 others are queries to fetch the transactions for each user.

SELECT "users".* FROM "users" WHERE "users"."id" IN (?, ?, ?)  [["id", 1], ["id", 2], ["id", 3]]
SELECT "transactions".* FROM "transactions" WHERE "transactions"."user_id" = ? /* loading for inspect */ LIMIT ?  [["user_id", 1], ["LIMIT", 11]]
SELECT "transactions".* FROM "transactions" WHERE "transactions"."user_id" = ? /* loading for inspect */ LIMIT ?  [["user_id", 2], ["LIMIT", 11]]
SELECT "transactions".* FROM "transactions" WHERE "transactions"."user_id" = ? /* loading for inspect */ LIMIT ?  [["user_id", 3], ["LIMIT", 11]]

What happen if the user_ids is so big? will we query to fetch the transactions as much as the user that we have? now we facing the N+1 queries problem

N+1 query problem?

This is common problem in the database query, it will execute the query one-by-one for all instance instead of 1 or 2 queries. In the example above we fetch all the three users data, then continue with query all the transactions for each user, it count 4 queries (1+3). If the are N users data, first it will fetch all the N users then continue to query all the transactions for each user, so it's called N+1 queries.

The problem in the N+1 queries is each query will take some amount of time, bigger data that we fetch bigger time that we need and we may facing the timeout issue. N+1 query is not good for the performance and we need find the solution

*we can ignore the N+1 query if the data is small or low throughput

Eager Load

In ruby, we have Eager load mechanism to load all the data and it's association with single query. One of the method to trigger the eager_load is .includes

def load_data_with_eager_load(user_ids)
    result = []
    users = User.includes(:transactions).where(id: user_ids)
    users.each do |user|
        result << { user: user, transactions: user.transactions }
    end

    result
end

the method above will give result

if we look closely, the load_data_with_eager_load method only trigger two query. First query get all the users and the second query get all the transactions for all users

SELECT "users".* FROM "users" WHERE "users"."id" IN (?, ?, ?)  [["id", 1], ["id", 2], ["id", 3]]
SELECT "transactions".* FROM "transactions" WHERE "transactions"."user_id" IN (?, ?, ?)  [["user_id", 1], ["user_id", 2], ["user_id", 3]]

It reduce the database query significantly, the eager load only cost 2 queries for however much data we have