Your Metrics Are Lying After They Hit the Cardinality Limit

Your dashboard says request traffic is normal.

Your error-rate alert is quiet.

But users are reporting failures.

This is not always a bad query or a delayed exporter. Your metrics may have exceeded their cardinality limit, causing OpenTelemetry to preserve the total while silently removing the attributes your dashboards depend on.

That is a dangerous failure mode because the data still looks believable.

The Problem Is Not Request Volume

Metric cardinality is the number of unique attribute combinations recorded for a metric.

Consider this request counter:

requestCounter.add(1, {
  "http.route": "/checkout",
  "http.request.method": "POST",
  success: false
});

OpenTelemetry maintains aggregation state for each unique combination of attributes.

These are four different combinations:

{ route="/checkout", method="POST", success="true"  }
{ route="/checkout", method="POST", success="false" }
{ route="/cart",     method="GET",  success="true"  }
{ route="/cart",     method="GET",  success="false" }

Recording one million requests with the same combination does not create one million metric series.

Recording one request for each of one million user IDs potentially does.

The dangerous part is therefore not necessarily request volume. It is how many different attribute combinations those requests produce.

The Common Mistake: Treating Metrics Like Logs

A developer wants more context during an incident, so they add a few useful-looking attributes:

requestCounter.add(1, {
  route: req.originalUrl,
  user_id: req.user.id,
  tenant_id: req.user.tenantId,
  request_id: req.id,
  success: response.statusCode < 500
});

This looks helpful. Every measurement contains enough information to identify the exact request.

It is also a cardinality explosion.

Values such as these are usually unbounded:

• user IDs
• request IDs
• session IDs
• raw URL paths
• search queries
• email addresses
• exception messages
• dynamically generated resource names

They can be appropriate in logs or traces, where individual events matter.

Metrics are designed for aggregation. Their dimensions should normally come from small, predictable sets.

If an attribute helps you find one specific request, it probably belongs in a trace or log, not in a metric.

Raw Paths Are a Frequent Source of Trouble

These URLs may look like one endpoint:

/users/10482/orders/9001
/users/25118/orders/9002
/users/78341/orders/9003

But as raw metric attribute values, they are three separate combinations.

The bounded route template is:

/users/{userId}/orders/{orderId}

That difference matters.

OpenTelemetry distinguishes a raw URL path from a route representing the matched application endpoint. A route template groups requests that execute the same code path, while a raw path can contain identifiers supplied by users.

A safer measurement looks like this:

requestCounter.add(1, {
  "http.route": "/users/{userId}/orders/{orderId}",
  "http.request.method": "GET",
  success: true
});

The identifiers can remain on the corresponding span:

span.setAttributes({
  "user.id": userId,
  "order.id": orderId
});

Now metrics answer aggregate questions, while traces preserve request-level context.

What Happens When the Limit Is Reached?

OpenTelemetry metrics SDKs use a cardinality limit to prevent unlimited aggregation state from consuming process memory.

The default aggregation cardinality limit defined by OpenTelemetry is 2,000 combinations per metric stream, although SDK support and configuration details can vary.

When the limit is reached, additional combinations are aggregated into an overflow data point:

{ otel.metric.overflow=true }

The measurement value is not simply discarded.

The original measurement attributes are.

Suppose this measurement arrives after the stream reaches its limit:

{
  route="/checkout",
  tenant_id="tenant-48291",
  success="false"
} 1

It may be incorporated into:

{ otel.metric.overflow=true } 1

Notice what disappeared:

• route
• tenant_id
• success

The unbounded tenant_id caused the explosion, but OpenTelemetry does not preserve the other attributes on the overflow point.

That includes the low-cardinality success boolean.

Your Total Can Be Correct While Your Alert Is Wrong

Imagine the service processed 100,000 requests:

98,000 successful requests
 2,000 failed requests

The total query can still return:

100,000

But if 1,400 failed requests were folded into overflow without their original attributes, a filtered query might only see:

success="false" -> 600

Your total request count remains correct.

Your calculated error rate becomes:

600 / 100000 = 0.6%

The actual error rate is:

2000 / 100000 = 2%

If the alert threshold is 1%, the alert does not fire.

This can affect any query that filters or groups by measurement attributes:

• error-rate alerts
• route-level latency dashboards
• per-tenant SLOs
• status-code breakdowns
• regional traffic analysis
• autoscaling signals
• capacity reports

This is what makes overflow so deceptive. The monitoring system continues returning numbers. Only some dimensions have become incomplete.

Cardinality Multiplies Faster Than It Looks

Assume a metric contains:

• 120 routes
• 5 HTTP methods
• 6 status-code groups
• 2 success values
• 4 deployment regions

The theoretical maximum is:

120 × 5 × 6 × 2 × 4 = 28,800 combinations

Not every combination will necessarily occur, but this calculation exposes the risk.

Now add 10,000 tenants:

28,800 × 10,000 = 288,000,000 combinations

Adding “just one label” can change the metric from bounded to effectively unbounded.

Histograms deserve additional attention because their exported representation can produce multiple time series per attribute combination, depending on the aggregation and backend.

Why local testing rarely reveals this problem

Development traffic usually exercises:

• a small number of routes
• one or two users
• a few tenants
• short process lifetimes
• limited parallel activity

Production introduces the full value space. A metric may behave correctly in tests and staging, then overflow only after enough unique combinations have accumulated.

With cumulative temporality, a process can retain combinations across collection cycles. A long-running production instance may therefore hit a limit that a frequently restarted staging instance never reaches.

Detect Overflow Before Debugging the Dashboard

Overflow points use this attribute:

otel.metric.overflow=true

Prometheus-compatible systems often normalize dots to underscores, exposing it as:

otel_metric_overflow="true"

A PromQL query for recent overflow can look like this:

count by (__name__, job) (
  last_over_time(
    {otel_metric_overflow="true"}[5m]
  )
) > 0

Adjust resource labels such as job, service_name, cluster, or namespace to match your environment.

For an OTLP-native backend, use its query language to:

1. filter on otel.metric.overflow = true
2. group by metric name
3. group by service name
4. identify the first deployment or time window where overflow appeared

Do not treat an empty result as absolute proof that cardinality is safe. Confirm that your SDK implements the relevant limit and that your exporter and backend retain the overflow attribute.

Do Not Fix It by Blindly Raising the Limit

Increasing the limit can stop overflow temporarily.

It can also allow substantially more aggregation state to accumulate in application memory and push more expensive series into your observability backend.

A higher limit is appropriate when:

• every attribute is intentional
• the dimensions are bounded
• the expected number of combinations is known
• the process has sufficient memory
• the backend can handle the resulting series
• overflow is continuously monitored

It is not the right first response to a request ID, raw path, or exception message appearing in a metric.

Fix the attribute model first.

Build a Cardinality Budget

Before shipping a metric, list every attribute and estimate its value space.

For example:

http.route             80 known values
http.request.method     5 known values
status_class             5 known values
region                   3 known values

A pessimistic upper bound is:

80 × 5 × 5 × 3 = 6,000 combinations

This does not mean all 6,000 combinations will exist simultaneously. It gives you a useful boundary to challenge before deployment.

Ask these questions for every attribute:

1. Is its value space bounded?
2. Can a user create new values?
3. Does it contain an identifier?
4. Can it include raw input?
5. Do we actually group or filter by it?
6. Would the value be more useful on a span or log?
7. What happens if one process sees every possible value?

If you cannot estimate the value space, treat the attribute as high-cardinality until proven otherwise.

Use Views to Remove Dangerous Attributes

OpenTelemetry Views can customize metric streams, including which attributes are retained.

The exact API differs between SDK languages, but the intended transformation is simple:

Input attributes:
  http.route
  http.request.method
  http.response.status_code
  user_id
  request_id

Retained metric attributes:
  http.route
  http.request.method
  http.response.status_code

The identifiers remain available in traces or logs.

The metric keeps only dimensions that are useful for aggregation.

This separation gives you both:

• affordable, reliable aggregate monitoring
• detailed context when investigating individual requests

Prefer Categories Over Raw Values

Sometimes the problematic attribute contains useful information but has too many possible values.

Replace the raw value with a bounded category.

Instead of recording the full exception message:

errorCounter.add(1, {
  error_message: error.message
});

Classify the failure:

errorCounter.add(1, {
  error_type: classifyError(error)
});

Where the classifier returns a controlled set:

function classifyError(error) {
  if (error instanceof TimeoutError) {
    return "timeout";
  }

  if (error instanceof ValidationError) {
    return "validation";
  }

  if (error instanceof DependencyError) {
    return "dependency";
  }

  return "unknown";
}

The complete error message belongs in a log or span event:

span.recordException(error);

The metric answers:

Are timeout failures increasing?

The trace answers:

What happened to this request?

Trying to make one telemetry signal answer both questions is what creates the problem.

Add Overflow to Your Observability Tests

Instrumentation deserves tests just like business logic.

A useful test should generate many unique values and verify that the exported metric does not retain an unbounded attribute.

Conceptually:

for (let index = 0; index < 10_000; index += 1) {
  requestCounter.add(1, {
    route: `/users/${index}`,
    request_id: `request-${index}`,
    success: true
  });
}

The test should fail if route contains raw identifiers or if request_id reaches the exported metric stream.

A corrected version records a normalized route:

for (let index = 0; index < 10_000; index += 1) {
  requestCounter.add(1, {
    "http.route": "/users/{userId}",
    success: true
  });
}

You can also add CI checks that inspect instrumentation changes for suspicious attribute names:

user_id
request_id
session_id
email
query
url.path
error.message

This is not a perfect detector, but it forces deliberate review before an unbounded dimension reaches production.

Production Checklist

Before deploying a new or modified metric:

• [ ] List every metric attribute.
• [ ] Estimate the number of possible values for each attribute.
• [ ] Calculate a pessimistic combination count.
• [ ] Replace raw paths with route templates.
• [ ] Keep request, user, and session IDs out of metrics.
• [ ] Replace raw error messages with bounded error categories.
• [ ] Move request-level context to traces or logs.
• [ ] Verify the SDK's cardinality-limit behavior.
• [ ] Query for otel.metric.overflow=true.
• [ ] Alert on overflow for metrics used by SLOs or paging.
• [ ] Review cumulative versus delta temporality.
• [ ] Raise the limit only after validating the attribute model.
• [ ] Load-test instrumentation with realistic value diversity.
• [ ] Recheck cardinality after framework or instrumentation upgrades.

Before changing the limit: Review the OpenTelemetry cardinality-limit documentation and verify that the metric attributes are intentionally bounded.

The Takeaway

A cardinality limit protects your application from unbounded metric state.

It does not guarantee that every dashboard, SLO, or alert remains accurate after overflow.

Once measurements are folded into an overflow point:

• totals can remain correct
• attributes can disappear
• filtered queries can undercount
• error alerts can stay quiet
• dashboards can continue looking believable

The safest approach is not to maximize the limit. It is to design metric dimensions deliberately:

• use bounded categories
• normalize routes
• keep identifiers in traces and logs
• define a cardinality budget
• monitor overflow as a production failure

Your monitoring should never fail silently while continuing to display convincing numbers.

Which high-cardinality attribute has caused the hardest observability problem in one of your production systems?

---

Sources

• Metric cardinality limits in OpenTelemetry: a practical guide
• OpenTelemetry Metrics: Cardinality Limits
• OpenTelemetry OBI Metrics Cardinality
• OpenTelemetry Attribute Requirement Levels

---

Connect with Me

If you found this article helpful, let's connect!

• 💻 GitHub: johnnylemonny

𝗝𝗼𝗵𝗻Follow

Freelance fullstack developer creating fast, modern, and refined applications. I focus on simplicity, performance, and real value for users. I also write tech-related articles here.

Comments

[nyaomaru]

Great article, John! 😸

Observability is essential in production, especially in systems where auditability and traceability really matter.

But the harder question is "whether the telemetry itself is actually trustworthy".

If cardinality overflow silently removes attributes, the dashboard can still look convincing while making incident investigation much harder.

I really liked the idea of designing a cardinality budget, keeping only bounded attributes in metrics, and moving request-level detail into traces or logs instead. That feels much safer than simply increasing the limit.

This helped me understand OpenTelemetry much better. Nice write-up! 🚀

[𝗝𝗼𝗵𝗻]

Thank you! I’m really glad the cardinality budget idea helped clarify the problem. That separation is key: metrics should stay bounded and reliable, while traces and logs carry the request-level context needed for deeper investigation. Simply raising the limit may only postpone the same failure in a less visible way. Thanks for the thoughtful comment!