Loki deployments often become expensive in the label model before they run out of storage. Turning every user_id, request_id, pod name, or file path into an indexed label does more than make a field searchable: it creates more log streams, encourages smaller chunks, and expands the index. Cost control is therefore not a compression setting to add later. It is a schema decision that belongs before ingestion starts.
This guide does not offer a magic config.yaml to paste into every deployment. It shows how to classify fields, where to place high-cardinality metadata, how to enable retention safely, and how to verify that a change reduced operational cost instead of merely moving it.
Where Loki costs originate
Rather than indexing every word in each log line as a conventional full-text engine would, Loki organizes log streams by label sets. Entries with the same label names and values belong to the same stream. Changing one value creates a different stream. A field such as environment="prod", with a small and stable value set, fits this model. A request_id that changes for every request does not.
High cardinality creates cost in several places. First, ingesters must track more active streams. Second, sparse streams can close chunks before those chunks become usefully large, increasing the number of small objects. Third, the index and query planner must deal with a wider stream set. The impact is not limited to disk usage; memory, object-store requests, and query latency can all increase.
The useful question is not simply, "Do people search by this field?" Ask instead: "Is this field stable and low-cardinality enough to narrow stream selection effectively?" Frequent searches for a high-cardinality field do not automatically make it a good indexed label.
Indexed label, structured metadata, or log body?
The following policy is a practical starting point. Validate the final decision against your real traffic and query patterns.
| Field example | Recommended location | Reason |
|---|---|---|
environment, cluster, namespace
|
Indexed label | Small, relatively stable value set |
service_name, app
|
Indexed label | Usually the first selector in a query |
level |
Measured choice: label or parsed field | Low value count, but not always needed on every stream |
pod, container_id, process_id
|
Structured metadata | Operationally useful but highly cardinal |
trace_id, request_id, user_id
|
Structured metadata or log body | Can change per request and explode stream count |
| Message, stack trace, free text | Log body | Evaluate through filtering and parsing after stream selection |
Current Grafana Loki documentation recommends moving fields such as pod names and service.instance.id from index labels to structured metadata. Structured metadata attaches a field to an entry without placing it in the indexed label set or embedding it in the message. The feature requires schema v13 or newer and therefore chunk format V4. OTLP ingestion also relies on structured metadata, so do not disable it before checking your schema and ingestion path.
A query makes the separation concrete:
{service_name="checkout", environment="prod"}
|= "timeout"
| json
| trace_id="0242ac120002"
The low-cardinality labels on the first line reduce the stream set. The trace_id filter is evaluated over the selected entries. You can still find a single request without creating a separate indexed stream for every request identifier.
Treat the label budget as a contract
Do not let application teams add indexed labels as an unrestricted list. Every proposed label should have an owner, an expected number of unique values, a lifetime, and at least one example query. "It is small today" is not enough; document how the value set grows with tenants, customers, processes, or pods.
A useful admission review asks four questions:
- Does the value set grow over hours, days, or months?
- Can
service_name, environment, and cluster already select the required streams? - Can the same information live in structured metadata or the JSON body?
- Which measurements will prove that the change improved stream and chunk behavior?
Loki's limits_config provides stream and query controls, but copying a universal number from a blog post is unsafe. Measure current p95 and p99 behavior, add headroom for expected growth, and introduce limits per tenant in stages. A limit does not repair a poor schema. It limits how much of the cluster that schema can consume.
Design storage schema and retention together
For new installations, the current Loki documentation recommends store: tsdb with schema: v13. Object-storage cost is not only about how many days of logs you retain; schema choices also affect how index and chunk objects are organized. When changing the schema of an existing system, preserve the old period and add a new period_config entry with an appropriate start date. Do not reinterpret existing data as if it had been written under the new schema.
The following is deliberately a partial configuration. It shows the relationships that matter for retention, not a complete production deployment:
schema_config:
configs:
- from: 2026-08-01
store: tsdb
object_store: s3
schema: v13
index:
prefix: index_
period: 24h
limits_config:
retention_period: 720h # 30 days; derive this from business and legal needs
compactor:
working_directory: /var/loki/compactor
retention_enabled: true
delete_request_store: s3
The bucket name, authentication method, and storage_config are intentionally omitted. Complete them from the current documentation for your Loki version and S3-compatible service. For a schema transition, choose the from date as part of the rollout plan. For a new installation with no existing data, use a valid date in the past as described by the schema documentation.
With TSDB or BoltDB Shipper, the Compactor applies retention. Setting retention_period without enabling retention on the Compactor does not produce the expected deletion behavior. The documentation also requires a 24-hour index period and a configured delete_request_store when retention is enabled. If you add an object-store lifecycle rule, keep it longer than Loki's retention period; deleting chunks first can leave index references to missing data.
Rollback boundary: retention deletion is irreversible. Validate it in a test tenant first, check object-store versioning or backup policy, and monitor the Compactor's deletion work. Reverting the configuration can stop future deletion. It cannot reconstruct chunks that have already been removed.
Bound query cost as well
A sound label schema does not make every query safe. Wide time ranges, weak selectors, expensive regular expressions, and excessive parallelism can cause large object-store reads. Start dashboard queries with low-cardinality selectors, restrict the time range to the operational need, and apply JSON or regex parsing only after stream selection.
If a dashboard or automation can starve other users inside a shared tenant, evaluate query fairness. Loki's query scheduler can use the X-Loki-Actor-Path header to place actors into separate subqueues within a tenant. Generate that header in a controlled Grafana data source or authentication proxy instead of trusting arbitrary end-user input. Confirm the scheduler topology and supported configuration in the documentation for the version you operate before enabling it.
Safe rollout: measure, constrain, then move
Apply a label redesign as a controlled migration rather than a cluster-wide edit:
- Calculate the unique-value trend for each label over the last 24 hours and seven days.
- Identify the fastest-growing fields, especially request identities and ephemeral infrastructure identifiers.
- Move those fields from indexed labels to structured metadata or the log body in the collector configuration.
- Release the change to one service or tenant as a canary.
- Compare active streams, chunk creation, ingestion rejections, object-store requests, and query latency with the previous period.
- Update saved queries and dashboards for the new field location.
- Roll back the collector change if expected searches break; keep retention changes in a separate rollout.
Grafana's current documentation recommends Grafana Alloy as the primary way to send logs to Loki, and Alloy can perform label and structured-metadata transformations in the ingestion pipeline. For older Promtail-based installations, check the current support timeline and map pipeline stages to Alloy components in a separate migration plan instead of combining collector migration with label and retention changes.
Conclusion
The main control on Loki cost is not the compression level; it is the label schema that determines stream count. Keep stable, low-cardinality fields that genuinely reduce the search space as indexed labels. Move request, user, and ephemeral infrastructure identifiers to structured metadata or the log body. Treat TSDB v13, Compactor retention, and query limits as parts of the same design. Most importantly, do not call a rollout successful merely because ingestion still works. Require active-stream behavior, chunk efficiency, object-store traffic, and query latency to improve together.
Top comments (0)