Oleksandr Kuryzhev

Posted on Sep 25 Originally published at kuryzhev.cloud

An unbounded Loki label turned a logging bill into shock

#devops

Originally published on kuryzhev.cloud

An unbounded Loki label is one of those things that looks harmless in a Helm values file and then quietly reshapes an entire billing cycle. A single dynamic label gets attached to log lines. It might be a request ID, a session token, or a raw user agent string. Over the following weeks, the number of active streams grows by orders of magnitude. The invoice can arrive before anyone notices the dashboard was already struggling to load.

This is a documented failure mode, not an edge case. Grafana's own Loki documentation warns explicitly about label cardinality. The warning is easy to miss, though, when a team is moving fast and just wants logs searchable by trace_id or pod_ip. The fix is straightforward once the cause is understood. The hard part is noticing before the bill does.

Symptoms

The pattern is commonly reported in writeups about Loki cost and performance problems, and it tends to show up roughly in this order:

Ingester memory usage climbs steadily over days or weeks, with no corresponding traffic increase.
Queries that used to return quickly start timing out or taking far longer than before.
loki_ingester_memory_streams keeps growing in Prometheus, well out of proportion to traffic. Meanwhile, rate(loki_ingester_streams_created_total[...]) stays persistently high.
Object storage (S3, GCS, or equivalent) usage and request counts grow disproportionately to actual log volume. Many small, poorly compressed chunks and a larger index replace fewer, well-packed chunks.
The monthly cloud bill for the logging stack shows a line item that used to be a rounding error and is now a budget conversation.

Individually, each symptom could point elsewhere: a noisy neighbor pod, a storage misconfiguration, or a Grafana dashboard bug. Together, and especially when the streams metric climbs without a matching traffic increase, they point at cardinality.

Watch out for: a slow query is often blamed on Grafana or the underlying object store first. The real bottleneck is often the ingester trying to hold too many distinct streams in memory at once.

Root cause

Loki indexes log lines by label set, not by content. Every unique combination of label key-value pairs creates a new stream. Each stream carries its own chunks, its own index entries, and its own memory footprint on the ingester.

A label with low cardinality creates a bounded, predictable number of streams. Examples are namespace, app, and environment. A label with high or unbounded cardinality creates a new stream for every unique value. Examples are user_id, request_id, ip, or a raw error message used as a label instead of staying in the log line. Idle streams are eventually flushed from ingester memory. However, as long as new values keep arriving, new streams keep being created, and every one of them leaves chunks and index entries behind in storage.

The typical failure path looks like this:

A developer wants to filter logs by a specific request during debugging.
A request ID gets promoted from log content into a label via the Alloy or Promtail pipeline config.
It works well in a staging environment with a handful of requests per minute.
In an environment handling thousands of requests per second, it creates new streams at roughly the request rate, each with its own chunk and index overhead.

Loki's architecture makes stream count, far more than raw log volume, a primary driver of ingester memory and query performance. Grafana's guidance is to keep labels low-cardinality and to avoid putting unbounded values in labels at all. See the official label best practices documentation for current recommendations. Practical limits depend on Loki version, index type (TSDB vs. the older boltdb-shipper), and cluster sizing.

Cost compounds as well. Hosted offerings such as Grafana Cloud Logs primarily meter ingested volume. Self-hosted clusters pay in ingester memory, compute, and object storage requests. In both cases unbounded labels do more than slow queries. They inflate label and index overhead and force more infrastructure to be provisioned just to keep ingestion healthy.

Fix #1: Find and quantify the offending label

Before changing any config, quantify which label is responsible. Loki exposes cardinality information through its HTTP API, through logcli, and through ingester metrics.

# Count distinct values of one suspect label over the last hour.
# Add the X-Scope-OrgID header when multi-tenancy is enabled.
# (date -d is GNU date syntax; adjust on macOS/BSD)
curl -s -G "http://loki.example.internal:3100/loki/api/v1/label/request_id/values" \
  -H "X-Scope-OrgID: ${TENANT_ID}" \
  --data-urlencode "start=$(date -d '-1 hour' +%s)" \
  | jq '.data | length'

# Or let logcli rank every label on matching streams by distinct values
logcli --addr="http://loki.example.internal:3100" --org-id="${TENANT_ID}" \
  series '{namespace="payments"}' --analyze-labels --since=1h

A single label with a distinct-value count far larger than every other label is the cardinality source. Cross-reference with the ingester's own view of stream churn:

# Per-second rate of new streams created, averaged over 1h, split by pod
sum by (pod) (rate(loki_ingester_streams_created_total[1h]))

The --analyze-labels output, or the raw /loki/api/v1/series endpoint, shows the actual label combinations in use. This usually makes the offending label obvious, since one label will dwarf every other in distinct value count.

Fix #2: Move high-cardinality fields out of labels

A request ID, session token, or raw IP belongs with the log line, not in the label set. Loki supports structured metadata for exactly this case: key-value pairs attached to individual log lines that can be filtered in queries but do not create new streams. The feature was introduced as experimental in Loki 2.9 and is enabled by default in Loki 3.x. It requires the TSDB index with schema v13.

Grafana Alloy is the recommended collector. Promtail is deprecated and in long-term support only, although it has an equivalent structured_metadata stage. In Alloy, the pipeline looks like this:

// Alloy: keep level as a label, demote request_id to structured metadata
loki.process "demote_request_id" {
  forward_to = [loki.write.default.receiver]

  stage.json {
    expressions = {
      request_id = "request_id",
      level      = "level",
    }
  }

  // low cardinality, safe as a stream label
  stage.labels {
    values = {
      level = "",
    }
  }

  // queryable, but not part of the stream's label set
  stage.structured_metadata {
    values = {
      request_id = "",
    }
  }
}

This preserves the ability to filter by request ID in LogQL, for example {app="api"} | request_id="abc123", without creating a new stream per request. Make sure no other pipeline stage still promotes request_id to a label. See Grafana's structured metadata documentation for version-specific details.

Fix #3: Set per-tenant limits so it can't happen silently again

Even with clean pipeline config today, nothing stops a future deploy from reintroducing a bad label. Loki's limits_config can cap active streams per tenant, which turns a silent cost explosion into a loud, immediate rejection.

# loki config: limits_config block
limits_config:
  max_global_streams_per_user: 10000  # cluster-wide active stream ceiling per tenant
  max_streams_per_user: 0             # per-ingester limit; 0 disables it in favor of the global limit
  per_stream_rate_limit: 3MB          # throttle runaway single-stream writers
  per_stream_rate_limit_burst: 15MB
  reject_old_samples: true
  reject_old_samples_max_age: 168h

The value 10000 is a placeholder. Size it from the stream counts you measured in Fix #1, with headroom for normal growth.

With these limits set, a misconfigured label promotion produces visible 429 "streams limit exceeded" errors in the client pushing logs. Without them, the pipeline looks smooth and the damage only shows up on the next invoice. Be aware that rejected lines are dropped unless the client retries and eventually succeeds, so alert on these errors rather than letting them pile up. That trade-off, noisy failure over silent cost growth, is a sensible default for most teams. Full field descriptions are in the Loki configuration reference.

Prevention

Cardinality problems are cheap to prevent and expensive to unwind after months of accumulated streams and chunks in object storage. A few habits close most of the gap:

Review every new label added to a logging pipeline in code review, the same way a schema migration would be reviewed. Ask explicitly: what's the maximum number of distinct values this can take?
Alert on the growth rate of loki_ingester_memory_streams, not just its absolute value. A slow, steady climb is often the first visible sign, well before query latency degrades.
Set max_global_streams_per_user deliberately in every environment, including staging, so a bad pattern is caught before it reaches production traffic volumes.
Track active streams and ingested-bytes trends in a dashboard reviewed monthly, tied to actual billing data where the metering model allows it.

None of this requires exotic tooling. It requires treating labels as a schema decision with real cost implications, not a free-text convenience field. Teams building observability stacks from scratch often find it faster to bake these limits into the initial Helm values or Terraform module than to retrofit them after the first painful invoice. More patterns like this are covered on kuryzhev.cloud. The billing surprise is avoidable, but it has to be designed against before the first unbounded label ships, not after.

DEV Community