Originally published on kuryzhev.cloud
Your Loki bill isn't growing because of log volume — it's growing because someone added a trace_id label back in 2024 and nobody ever removed it. We spent two sprints on Loki retention tuning last quarter, and the actual fix had almost nothing to do with the retention_period setting we started with. Here's what we changed, in the order it actually moved the needle.
Stop treating retention_period as one global dial
A single global retention setting either wastes storage on low-value debug logs or violates compliance requirements for audit trails — you can't win with one number. Loki supports per-tenant overrides in runtime_config, and they reload without restarting ingesters, so there's no excuse for lumping everything together.
We had payments audit logs and frontend debug logs under the same 30-day policy for over a year. Splitting them into separate tenant overrides cut storage for the noisy tenant by 70% while actually extending retention for the compliance-sensitive one.
Cardinality, not volume, is what bankrupts your index
Everyone tunes retention_period first. Almost nobody checks cardinality first, and that's backwards. Every unique combination of label values creates a new stream, and every stream gets its own chunk — so a label like request_id or pod IP doesn't just add data, it multiplies your index size.
Before touching any retention setting, run a cardinality audit:
# find labels driving the most unique streams
logcli series --analyze-labels --since=24h '{}'
Gotcha: if this command returns thousands of unique values for a single label, retention tuning won't save you — you're solving the wrong problem.
Let the compactor do the retention work, then tune it
Setting retention_period without enabling the compactor is a silent no-op — Loki will happily accept the config and delete nothing. We learned this after "shortening" retention for a tenant and watching storage stay flat for two weeks.
compactor:
retention_enabled: true
retention_delete_delay: 2h
delete_request_store: aws # required for the Delete API
compaction_interval: 10m
The delete delay is intentional — it gives you a window to cancel a deletion request — but it also means "expired" data still occupies (and costs) storage until the compactor actually runs.
Chunk size and age settings decide your object storage bill
At high ingestion volume, object storage cost is often driven by request count, not raw bytes. Small chunks mean more frequent PUT/GET calls, and S3-style pricing punishes that hard. Bumping chunk_target_size toward 1.5–4MB reduced our PUT volume by roughly a third.
ingester:
chunk_target_size: 2097152 # ~2MB, fewer object storage PUTs
max_chunk_age: 1h # balance flush frequency vs memory
The tradeoff: larger max_chunk_age means fewer flushes but more memory pressure on ingesters. Watch ingester OOMs after this change — we saw a couple during a traffic spike before we adjusted memory limits.
Use structured metadata instead of new labels
Loki 2.9+ and 3.x support structured metadata, which lets you attach high-cardinality fields like trace_id or request_id to log lines without putting them in the label index. This is the single change that had the biggest cardinality impact for us.
Migration isn't free — pipelines pushing everything through label relabeling in Promtail or Grafana Alloy need a rewrite to route those fields into structured metadata instead. It's worth the afternoon it takes; check the Grafana Loki structured metadata docs for the exact pipeline stage syntax.
Guardrail queries against your own retention window
Long retention is pointless — and expensive — if queries against old data time out or hammer cold storage every time someone opens a dashboard. We found a Grafana panel silently scanning 90 days of cold S3-backed chunks on every 30-second auto-refresh. That one panel was responsible for a noticeable chunk of our egress cost.
overrides:
team-payments:
retention_period: 2160h # 90 days, compliance requirement
ingestion_rate_mb: 20
per_stream_rate_limit: 5MB
team-web:
retention_period: 168h # 7 days, high-volume/low-value logs
max_query_length: 72h # block accidental 90-day scans
Pair max_query_length and split_queries_by_interval with per-tenant ingestion limits — otherwise one noisy team forces retention and index scaling decisions on everyone else sharing the cluster. See the official Loki retention documentation for the full list of tunables.
Remember logs are data too — secure them like it
Retention tuning conversations tend to skip security entirely, which is a mistake once you're storing PII or audit trails in log lines. Restrict IAM and bucket policies on your chunk and index storage separately from application data buckets — don't inherit broad defaults just because it's "only logs."
Gotcha: retention_period is not the same as guaranteed erasure. Compaction lag and backup snapshots can keep PII around long after it should be gone. For right-to-erasure requests, use the Delete API directly instead of waiting on a scheduled retention cycle.
Before you change a single retention setting, run through this checklist — it's saved us from at least three wasted afternoons:
Decision checklist — before you touch retention_period:
[ ] Have you measured stream cardinality? (logcli series --analyze-labels)
[ ] Are trace/request IDs in labels instead of structured metadata? -> fix first
[ ] Is compactor.retention_enabled actually true? (silent no-op otherwise)
[ ] Do different log sources need different retention (audit vs debug)?
[ ] Is query_timeout set to match your longest allowed retention window?
[ ] Do storage bucket IAM policies match your data sensitivity, not just app defaults?
[ ] Have you priced object storage requests, not just GB stored?
Loki retention tuning isn't one setting — it's cardinality control, compactor configuration, chunk sizing, and query guardrails working together. Get the cardinality and compactor pieces right first, and the retention_period number itself becomes almost trivial to set. For more log pipeline debugging, check the monitoring category on kuryzhev.cloud.
Top comments (0)