Fixing Loki Regex Pipeline Stage Failures on Nginx Logs

Originally published on kuryzhev.cloud

---

Your Loki dashboard shows the raw nginx access log fine — but every label query returns zero results, and nobody notices until the ingesters start OOMing. We hit this exact scenario on a client's edge cluster: {job="nginx"} |= "500" in Grafana Explore returned every 500 error you'd expect, but {job="nginx", status="500"} came back empty. That gap — raw text works, labels don't — is almost always a broken loki regex pipeline stage, and it's one of the most common Promtail/Alloy misconfigurations we see in the field.

Symptoms

The failure mode is deceptively quiet. Here's what we actually saw before we understood what was happening:

• LogQL queries using extracted labels like {job="nginx", status="500"} return zero results, even though {job="nginx"} |= "500" shows plenty of matching raw lines.
• Grafana Explore renders the full raw nginx line correctly, but the "Detected fields" panel is empty — no remote_addr, no status, nothing.
• Promtail logs spam level=warn ... msg="pipeline stage failed to parse", or — worse — there's no error at all and lines get ingested unparsed, silently.
• Some time later, Loki ingesters spike in memory and you start seeing too many streams or per-stream rate limit exceeded errors, usually right after someone "fixed" the regex by promoting every field to a label.

That last one is the sneaky part. The regex fix often looks successful in isolation — fields finally extract — but it introduces a second, worse problem: cardinality explosion. We'll get to that in Fix #3.

Root cause

Almost every time, the root cause is the same: someone copy-pasted a regex from a tutorial that assumed nginx's default combined log format —

$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"

— but the server actually runs a custom log_format main with a different field order, extra fields, or different quoting. The regex compiles fine, Promtail starts fine, and nothing in the config-load step complains. It just doesn't match.

Two Go/RE2-specific gotchas make this worse. First, Promtail and Grafana Alloy use Go's RE2 engine, which has no backreferences and no lookahead/lookbehind — patterns that work on regex101 in PCRE mode will silently misbehave here. Greedy .* inside a quoted field will happily swallow the next quoted field too. Second, named capture groups are mandatory: (?P<remote_addr>\S+) produces a usable field, but a plain (\S+) group is discarded entirely by the regex stage — no warning, it just vanishes.

The last piece: nobody validated the pattern against a real log line before deploying. YAML escaping of literal [, ], and " inside the regex string is a classic trap — a malformed escape doesn't throw a config error, it just breaks the match at runtime, exactly the "silent failure" symptom above.

Fix #1: Align regex with the actual nginx log_format

Stop guessing. Pull the exact directive straight from /etc/nginx/nginx.conf:

log_format main '$remote_addr - $remote_user [$time_local] '
                 '"$request" $status $body_bytes_sent '
                 '"$http_referer" "$http_user_agent"';

Now build the regex field-by-field against that, not against a doc example. Before touching the running pipeline, validate offline using Promtail's dry-run mode against a real sample line:

promtail -config.file=/etc/promtail/config.yml -dry-run -inspect \
    < /var/log/nginx/access.log | head -1

If you don't have Promtail handy locally, paste the pattern into regex101 in "Golang" mode — it's the closest thing to RE2 semantics you'll get outside of the actual binary. The most common breakage we see: unescaped [ around $time_local, and literal " around $request / $http_user_agent that need proper escaping inside the YAML string. Get this wrong and the stage just silently no-ops — check the official Loki regex stage docs for the exact syntax expectations.

Fix #2: Fix pipeline stage ordering and timestamp parsing

Getting the regex right isn't enough if the stages downstream are in the wrong order. Stage order matters a lot: regex → timestamp → labels → drop. Put labels before regex and you'll extract nothing, because there's no captured data yet for it to reference — we've seen this exact ordering mistake ship to production twice on the same team.

Next, wire up the timestamp stage so Loki uses the real request time from $time_local, not Promtail's scrape time:

- timestamp:
    source: time_local
    format: "02/Jan/2006:15:04:05 -0700"

That Go reference layout string is not optional boilerplate — it's how Go's time package interprets your custom format. Get it wrong and every log lands with a slightly-off timestamp, which quietly breaks time-range correlation during incident review.

Finally: be disciplined about what gets promoted to labels. Only genuinely low-cardinality fields — method, a bucketed status_class — belong there. Path, IP, and user agent should stay as extracted fields only. This single decision is what separates a healthy Loki deployment from one that OOMs its ingesters, which is exactly what Fix #3 addresses.

Fix #3: Contain cardinality with structured_metadata and template

This is the fix that actually saves your ingesters. Move high-cardinality fields — remote_addr, request_uri, user_agent — into the structured_metadata stage (available since Loki 2.9, stable as of 3.0) instead of labels. They stay fully queryable via | label_format without becoming part of the index.

Use a template stage to bucket status into status_class before labeling — you get 2xx/3xx/4xx/5xx dashboard grouping without paying per-status-code cardinality:

# /etc/promtail/config.yml
server:
  http_listen_port: 9080

positions:
  filename: /var/lib/promtail/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: nginx
    static_configs:
      - targets: [localhost]
        labels:
          job: nginx
          __path__: /var/log/nginx/access.log

    pipeline_stages:
      # Stage 1: match the ACTUAL log_format, not a tutorial default.
      # nginx.conf: log_format main '$remote_addr - $remote_user [$time_local] '
      #                              '"$request" $status $body_bytes_sent '
      #                              '"$http_referer" "$http_user_agent"';
      - regex:
          expression: '^(?P<remote_addr>\S+) - (?P<remote_user>\S+) \[(?P<time_local>[^\]]+)\] "(?P<method>\S+) (?P<request_uri>\S+) (?P<protocol>\S+)" (?P<status>\d{3}) (?P<body_bytes_sent>\d+) "(?P<http_referer>[^"]*)" "(?P<http_user_agent>[^"]*)"'

      # Stage 2: parse the real request timestamp, not scrape time
      - timestamp:
          source: time_local
          format: "02/Jan/2006:15:04:05 -0700"

      # Stage 3: derive a coarse, low-cardinality bucket for dashboards
      - template:
          source: status_class
          template: '{{ Substr .status 0 1 }}xx'

      # Stage 4: promote ONLY low-cardinality fields to indexed labels
      - labels:
          method:
          status_class:

      # Stage 5: everything else stays as structured metadata (Loki >=2.9)
      # queryable, not indexed -> avoids stream explosion
      - structured_metadata:
          remote_addr:
          request_uri:
          http_user_agent:
          status:

      # Stage 6: redact sensitive query params before they leave the box
      - replace:
          expression: '(token|auth)=[^&\s]+'
          replace: '$1=REDACTED'

Verify the fix locally before you restart the service, and confirm stream count is sane afterward:

# Verifying the fix locally before restarting promtail
$ promtail -config.file=/etc/promtail/config.yml -dry-run -inspect \
    < /var/log/nginx/access.log | head -1

# Expected extracted fields (successful match):
remote_addr=10.0.4.12
remote_user=-
time_local=14/May/2024:09:12:03 +0000
method=GET
request_uri=/api/v1/orders?token=REDACTED
protocol=HTTP/1.1
status=500
status_class=5xx
body_bytes_sent=214
http_user_agent=curl/8.4.0

# If the regex is broken you'll instead see (silent, no crash):
level=warn ts=... msg="pipeline stage failed to parse" stage=regex

# Confirm cardinality didn't explode after deploy:
$ logcli series '{job="nginx"}' --stats
Total series: 12   # was 48,000+ before moving IP/URI out of `labels`

That drop from 48,000+ series to 12 is not an exaggeration — one unique IP + path combination used to mean one new stream, one new chunk set, growing index size, and slower queries. Each of those streams also costs real money in object storage (S3/GCS chunks), so this isn't just a performance fix, it's a cost fix. If you're managing Loki cost at scale, check out our broader notes on observability and cost tuning on kuryzhev.cloud for related patterns.

Prevention

We stopped trusting "it parses in Explore" as proof the pipeline is healthy. Here's what actually prevents recurrence:

• Add a comment block directly above log_format in nginx.conf listing the exact Promtail regex that depends on it. Treat the two configs as coupled — because they are, whether anyone admits it or not.
• Add a CI or pre-commit check that runs promtail -dry-run against a sample log fixture whenever promtail-config.yml changes. This catches breakage before it ships, not after someone notices empty dashboards.
• Alert on promtail_files_active_total dropping unexpectedly (file discovery issues) and on Loki's discarded_samples_total{reason="stream_limit"} to catch cardinality regressions the moment they start, not after ingesters fall over.
• Redact sensitive fields — Authorization headers, session cookies, tokens in query strings — with a replace stage before anything reaches structured metadata. Loki has no field-level encryption; access logs leak secrets by accident more often than teams admit, and once it's in a chunk in object storage, it's there.

One more note if you're on Grafana Alloy instead of classic Promtail: the concepts carry over one-for-one, but the syntax is River, not YAML — stage.regex { expression = "..." } instead of the YAML block above. Same RE2 engine, same gotchas, different config shape. Check the Grafana Alloy documentation before porting an existing pipeline. Getting the loki regex pipeline stage right the first time — matched to your real log_format, ordered correctly, and cardinality-aware — saves you from debugging this twice.

Related

• More log pipeline and Grafana/Loki troubleshooting patterns
• Kubernetes observability and namespace-level log isolation tips
• Secrets redaction and hardening patterns for logging pipelines