DEV Community: Oleksandr Kuryzhev

Why an LLM reviewer misses risky Terraform plan changes

Oleksandr Kuryzhev — Sat, 26 Sep 2026 07:02:00 +0000

Originally published on kuryzhev.cloud

Consider an illustrative scenario. A pull request adds one line to a security group resource, and the pipeline runs terraform plan. An LLM reviewer summarizes the diff as "minor tagging update," and a human approves it almost immediately. The actual plan output shows a forced replacement of the primary RDS instance, because a subnet group reference changed upstream. Nobody read past the AI summary. This is the failure mode teams walk into when they bolt an LLM reviewer onto Terraform plan changes and treat it as a stamp of trust, instead of as a component with its own failure surface.

Failure scenario

A platform team introduces an LLM reviewer that consumes the JSON output of terraform plan -out=tfplan && terraform show -json tfplan and posts a comment on the pull request. The comment reads well: bullet points, a risk rating and a plain-English summary. For a while it works fine on additive changes such as new S3 buckets, new IAM policies and extra tags.

Then a module update bumps a provider version, and the default value of an attribute that forces replacement changes. The human-readable plan shows -/+ resource "aws_db_instance" with a # forces replacement annotation. In the JSON, the same change appears only as an actions array of ["delete", "create"] plus replace_paths and action_reason fields. The LLM's summary mentions "instance configuration updated" without flagging replacement. The human reviewer approves, having learned to trust the AI summary after weeks of accurate output. The apply destroys and recreates a production database.

A second, quieter variant: the LLM reviewer is given a token budget that truncates large plans. Imagine a plan several thousand lines long for a multi-account VPC change, cut off partway through, with the destructive change sitting past the cut. The model reviews what it was given, honestly, and says nothing about what it never saw. Nothing in the pipeline output indicates truncation happened, so the missing risk is invisible to everyone downstream.

Why it happens

An LLM reviewer for Terraform plan changes is usually built to summarize text, not to reason reliably over structured infrastructure state. Terraform's JSON plan format encodes replacement, deletion and sensitive-value changes in specific fields:

actions, where replacement is ["delete","create"] or ["create","delete"], never a literal "replace";
replace_paths;
action_reason;
before_sensitive and after_sensitive.

A general-purpose prompt that just says "review this Terraform plan for risk" will not consistently surface these fields. The model's fluency creates false confidence. A clear, well-structured sentence about a risky change is only as good as its accuracy, and it reads exactly like an accurate one.

Token limits compound this. Large plans, especially from monorepos or wide blast-radius modules, can exceed context windows or budget-limited API calls. Truncation is silent unless the pipeline explicitly checks for it. Frontier LLMs and smaller local open-weight models both have this ceiling: the model doesn't know what it wasn't shown.

Watch out for teams treating "the LLM approved it" or "the LLM flagged it green" as equivalent to a policy check. An LLM reviewer is a heuristic summarizer, not a gate with guaranteed recall. It can miss a pattern it wasn't prompted to look for, especially a newly introduced one like a provider default change.

Prompt drift is the other quiet cause. As teams tweak the prompt to reduce false positives, such as alert fatigue on tagging changes, they can loosen the language enough that genuine risk categories get deprioritized. Replacement, deletion and IAM privilege escalation can slip down the list without anyone noticing the regression until a bad apply happens.

The fix (with code)

Treat the LLM as a second opinion layered on top of deterministic checks, never as the sole gate. Start with structured, non-AI detection of destructive actions directly from the JSON plan. Using jq against Terraform's documented JSON plan format is enough. Because replacements are always encoded as a delete paired with a create, matching on delete catches both deletions and replacements.


# Generate a plan and machine-readable JSON output
terraform plan -out=tfplan
terraform show -json tfplan > plan.json

# Count resources that will be deleted or replaced.
# Replacement appears as ["delete","create"] or ["create","delete"].
# "// []" avoids a jq error (and a fail-open gate) when there are no changes.
destructive=$(jq '
  [(.resource_changes // [])[]
    | select(.change.actions | index("delete"))]
  | length
' plan.json) || { echo "PLAN_PARSE_FAILED"; exit 1; }

if [ "$destructive" -gt 0 ]; then
  echo "DESTRUCTIVE_CHANGE_DETECTED: $destructive resource(s)"
  exit 1
fi

This deterministic gate runs before the LLM ever sees the plan. A replacement or deletion therefore cannot slip through on the strength of a friendly summary, and a plan that fails to parse blocks the pipeline instead of passing it. Only after this check passes, or fails and receives an explicit human override, should the LLM reviewer generate its explanation for reviewers.

Feed the LLM a scoped extract of the plan rather than the raw JSON, and require it to return structured output you can validate programmatically instead of free text.


import json

def load_plan(path="plan.json"):
    with open(path) as f:
        return json.load(f)

def summarize_risk(plan):
    risky = []
    for c in plan.get("resource_changes", []):
        change = c["change"]
        actions = change["actions"]
        # Replacement is encoded as delete+create, so "delete" covers it.
        # "update" is included so in-place changes are also reviewed.
        if "delete" in actions or "update" in actions:
            risky.append({
                "address": c["address"],
                "actions": actions,
                "replacement": "delete" in actions and "create" in actions,
                "replace_paths": change.get("replace_paths", []),
                "action_reason": c.get("action_reason"),
                "touches_sensitive": bool(
                    change.get("before_sensitive") or change.get("after_sensitive")
                ),
                "provider": c.get("provider_name"),
            })
    # This structured payload, not raw text, goes to the model.
    # Required response schema: {verdict, cited_addresses[], reasons[], confidence}.
    return risky

if __name__ == "__main__":
    plan = load_plan()
    risky_changes = summarize_risk(plan)
    if risky_changes:
        print(json.dumps(risky_changes, indent=2))

Route the structured risk list, not the entire plan, to a frontier LLM or to an internal service backed by Amazon Bedrock or the OpenAI API. If the list is still too large for the model's context, split it by resource address into multiple requests rather than truncating it. Require a schema-constrained response containing a verdict, cited resource addresses, reasons and a confidence score. Reject any response that doesn't parse against the schema, and treat a malformed response as "review failed," not "review passed." Log every plan/response pair for audit, since auditors or postmortems may later need to know exactly what the model saw.

Prevention checklist

Before relying on an LLM reviewer for Terraform plan changes in a merge gate, verify the following:


[ ] Deterministic check runs first: any delete action (including the
    delete half of a replacement) blocks merge without human override,
    independent of LLM output.
[ ] Plan JSON size is measured before sending to the model; oversized
    plans are chunked by resource address, never silently truncated.
[ ] LLM output is schema-validated (verdict + cited resources), and a
    malformed or missing response is treated as "block," not "pass."
[ ] Sensitive value changes (before_sensitive/after_sensitive fields)
    are surfaced explicitly, not summarized away as "no visible change."
[ ] The exact prompt version is pinned and versioned in the repo, so a
    prompt edit is reviewable like any other pipeline change.
[ ] A regular sample of approved plans is manually re-reviewed against
    the LLM's verdict to catch silent accuracy drift.
[ ] IAM/policy-widening changes have a separate, stricter rule set,
    since privilege escalation risk isn't always visible in plan diffs.
[ ] Reviewers are trained that the LLM summary is advisory; the plan
    output linked in the PR is the source of truth, always.

None of this replaces standard Terraform hygiene such as remote state locking, mandatory terraform validate and module version pinning. It sits on top of it. For teams building this kind of guarded pipeline from scratch, the broader CI/CD and infrastructure automation patterns covered on kuryzhev.cloud are a reasonable starting point before layering AI review on.

Verify plan JSON field names against the HashiCorp documentation for your Terraform version, and check the format_version field in the output. The format is versioned, and newer releases may add fields that a reviewer tuned to an older schema will ignore or misread.

An unbounded Loki label turned a logging bill into shock

Oleksandr Kuryzhev — Fri, 25 Sep 2026 07:02:27 +0000

Originally published on kuryzhev.cloud

An unbounded Loki label is one of those things that looks harmless in a Helm values file and then quietly reshapes an entire billing cycle. A single dynamic label gets attached to log lines. It might be a request ID, a session token, or a raw user agent string. Over the following weeks, the number of active streams grows by orders of magnitude. The invoice can arrive before anyone notices the dashboard was already struggling to load.

This is a documented failure mode, not an edge case. Grafana's own Loki documentation warns explicitly about label cardinality. The warning is easy to miss, though, when a team is moving fast and just wants logs searchable by trace_id or pod_ip. The fix is straightforward once the cause is understood. The hard part is noticing before the bill does.

Symptoms

The pattern is commonly reported in writeups about Loki cost and performance problems, and it tends to show up roughly in this order:

Ingester memory usage climbs steadily over days or weeks, with no corresponding traffic increase.
Queries that used to return quickly start timing out or taking far longer than before.
loki_ingester_memory_streams keeps growing in Prometheus, well out of proportion to traffic. Meanwhile, rate(loki_ingester_streams_created_total[...]) stays persistently high.
Object storage (S3, GCS, or equivalent) usage and request counts grow disproportionately to actual log volume. Many small, poorly compressed chunks and a larger index replace fewer, well-packed chunks.
The monthly cloud bill for the logging stack shows a line item that used to be a rounding error and is now a budget conversation.

Individually, each symptom could point elsewhere: a noisy neighbor pod, a storage misconfiguration, or a Grafana dashboard bug. Together, and especially when the streams metric climbs without a matching traffic increase, they point at cardinality.

Watch out for: a slow query is often blamed on Grafana or the underlying object store first. The real bottleneck is often the ingester trying to hold too many distinct streams in memory at once.

Root cause

Loki indexes log lines by label set, not by content. Every unique combination of label key-value pairs creates a new stream. Each stream carries its own chunks, its own index entries, and its own memory footprint on the ingester.

A label with low cardinality creates a bounded, predictable number of streams. Examples are namespace, app, and environment. A label with high or unbounded cardinality creates a new stream for every unique value. Examples are user_id, request_id, ip, or a raw error message used as a label instead of staying in the log line. Idle streams are eventually flushed from ingester memory. However, as long as new values keep arriving, new streams keep being created, and every one of them leaves chunks and index entries behind in storage.

The typical failure path looks like this:

A developer wants to filter logs by a specific request during debugging.
A request ID gets promoted from log content into a label via the Alloy or Promtail pipeline config.
It works well in a staging environment with a handful of requests per minute.
In an environment handling thousands of requests per second, it creates new streams at roughly the request rate, each with its own chunk and index overhead.

Loki's architecture makes stream count, far more than raw log volume, a primary driver of ingester memory and query performance. Grafana's guidance is to keep labels low-cardinality and to avoid putting unbounded values in labels at all. See the official label best practices documentation for current recommendations. Practical limits depend on Loki version, index type (TSDB vs. the older boltdb-shipper), and cluster sizing.

Cost compounds as well. Hosted offerings such as Grafana Cloud Logs primarily meter ingested volume. Self-hosted clusters pay in ingester memory, compute, and object storage requests. In both cases unbounded labels do more than slow queries. They inflate label and index overhead and force more infrastructure to be provisioned just to keep ingestion healthy.

Fix #1: Find and quantify the offending label

Before changing any config, quantify which label is responsible. Loki exposes cardinality information through its HTTP API, through logcli, and through ingester metrics.

# Count distinct values of one suspect label over the last hour.
# Add the X-Scope-OrgID header when multi-tenancy is enabled.
# (date -d is GNU date syntax; adjust on macOS/BSD)
curl -s -G "http://loki.example.internal:3100/loki/api/v1/label/request_id/values" \
  -H "X-Scope-OrgID: ${TENANT_ID}" \
  --data-urlencode "start=$(date -d '-1 hour' +%s)" \
  | jq '.data | length'

# Or let logcli rank every label on matching streams by distinct values
logcli --addr="http://loki.example.internal:3100" --org-id="${TENANT_ID}" \
  series '{namespace="payments"}' --analyze-labels --since=1h

A single label with a distinct-value count far larger than every other label is the cardinality source. Cross-reference with the ingester's own view of stream churn:

# Per-second rate of new streams created, averaged over 1h, split by pod
sum by (pod) (rate(loki_ingester_streams_created_total[1h]))

The --analyze-labels output, or the raw /loki/api/v1/series endpoint, shows the actual label combinations in use. This usually makes the offending label obvious, since one label will dwarf every other in distinct value count.

Fix #2: Move high-cardinality fields out of labels

A request ID, session token, or raw IP belongs with the log line, not in the label set. Loki supports structured metadata for exactly this case: key-value pairs attached to individual log lines that can be filtered in queries but do not create new streams. The feature was introduced as experimental in Loki 2.9 and is enabled by default in Loki 3.x. It requires the TSDB index with schema v13.

Grafana Alloy is the recommended collector. Promtail is deprecated and in long-term support only, although it has an equivalent structured_metadata stage. In Alloy, the pipeline looks like this:

// Alloy: keep level as a label, demote request_id to structured metadata
loki.process "demote_request_id" {
  forward_to = [loki.write.default.receiver]

  stage.json {
    expressions = {
      request_id = "request_id",
      level      = "level",
    }
  }

  // low cardinality, safe as a stream label
  stage.labels {
    values = {
      level = "",
    }
  }

  // queryable, but not part of the stream's label set
  stage.structured_metadata {
    values = {
      request_id = "",
    }
  }
}

This preserves the ability to filter by request ID in LogQL, for example {app="api"} | request_id="abc123", without creating a new stream per request. Make sure no other pipeline stage still promotes request_id to a label. See Grafana's structured metadata documentation for version-specific details.

Fix #3: Set per-tenant limits so it can't happen silently again

Even with clean pipeline config today, nothing stops a future deploy from reintroducing a bad label. Loki's limits_config can cap active streams per tenant, which turns a silent cost explosion into a loud, immediate rejection.

# loki config: limits_config block
limits_config:
  max_global_streams_per_user: 10000  # cluster-wide active stream ceiling per tenant
  max_streams_per_user: 0             # per-ingester limit; 0 disables it in favor of the global limit
  per_stream_rate_limit: 3MB          # throttle runaway single-stream writers
  per_stream_rate_limit_burst: 15MB
  reject_old_samples: true
  reject_old_samples_max_age: 168h

The value 10000 is a placeholder. Size it from the stream counts you measured in Fix #1, with headroom for normal growth.

With these limits set, a misconfigured label promotion produces visible 429 "streams limit exceeded" errors in the client pushing logs. Without them, the pipeline looks smooth and the damage only shows up on the next invoice. Be aware that rejected lines are dropped unless the client retries and eventually succeeds, so alert on these errors rather than letting them pile up. That trade-off, noisy failure over silent cost growth, is a sensible default for most teams. Full field descriptions are in the Loki configuration reference.

Prevention

Cardinality problems are cheap to prevent and expensive to unwind after months of accumulated streams and chunks in object storage. A few habits close most of the gap:

Review every new label added to a logging pipeline in code review, the same way a schema migration would be reviewed. Ask explicitly: what's the maximum number of distinct values this can take?
Alert on the growth rate of loki_ingester_memory_streams, not just its absolute value. A slow, steady climb is often the first visible sign, well before query latency degrades.
Set max_global_streams_per_user deliberately in every environment, including staging, so a bad pattern is caught before it reaches production traffic volumes.
Track active streams and ingested-bytes trends in a dashboard reviewed monthly, tied to actual billing data where the metering model allows it.

None of this requires exotic tooling. It requires treating labels as a schema decision with real cost implications, not a free-text convenience field. Teams building observability stacks from scratch often find it faster to bake these limits into the initial Helm values or Terraform module than to retrofit them after the first painful invoice. More patterns like this are covered on kuryzhev.cloud. The billing surprise is avoidable, but it has to be designed against before the first unbounded label ships, not after.

When Cloudflare Rate Limit Locks Out Your Own Players

Oleksandr Kuryzhev — Thu, 24 Sep 2026 07:02:43 +0000

Originally published on kuryzhev.cloud

The scenario

A Cloudflare rate limit rule is often deployed with good intentions: stopping connection floods against a game server's matchmaking API. It can end up blocking the very players it was meant to protect. This is a well-known failure mode, not a rare edge case.

Rate limiting rules that count requests per IP address behave badly the moment real players share an IP. That happens constantly behind mobile carrier NAT, university networks, and office proxies. From Cloudflare's edge, a launch night spike in concurrent players can look very similar to a credential-stuffing attack or a DDoS probe against the login endpoint.

The typical failure unfolds like this:

A Rate Limiting Rule is set to something like "block if more than 20 requests per 10 seconds from one IP."
That threshold was tuned against a staging environment with a handful of testers.
On launch night, several hundred players on the same carrier-grade NAT pool hit the same public IP.
The counter trips, and Cloudflare returns 429 or a challenge.
An entire region's worth of legitimate players gets treated as one abusive client.

Support tickets can pile up faster than the on-call engineer can read Cloudflare's analytics dashboard.

The fix is not "turn off rate limiting." Game servers are a common DDoS target, and an unprotected matchmaking endpoint is a real risk. The fix is a rate limiting configuration that distinguishes abusive traffic from legitimate shared-IP traffic, using the signals Cloudflare actually exposes for that purpose. Official reference: Cloudflare Rate Limiting Rules documentation.

Prerequisites

Before touching a rate limiting rule on a production zone, confirm the following:

A plan that supports the counting you need. Rate Limiting Rules exist on all plans, but the options differ:
- Free and Pro count by IP only.
- Business adds the "IP with NAT support" characteristic.
- Counting by request header, cookie, or query value requires Enterprise with Advanced Rate Limiting.
Check the plan limits table in the documentation, since periods, rule counts, and available fields also vary by plan.
Permissions. You need API or dashboard access with permission to edit WAF and Rate Limiting rules for the zone. For the API, use a scoped API token.
Visibility into real traffic. Use Cloudflare Logpush or the dashboard's Security Events view, so rule impact can be verified against real traffic, not guesses.
A list of legitimate high-volume clients. This includes game server backends, health check services, and CI pipelines hitting the API. Where available, also include known proxy egress ranges used in the target region.
A safe rollout path. Use a staging zone or a narrowly scoped canary rule. On Enterprise, you can deploy a rule with the Log action before switching to Block.

Watch out for one common gotcha: Cloudflare counts by the client IP as seen at its edge, which is whatever connected to Cloudflare.

A load balancer behind Cloudflare, at your origin, does not change that.
A proxy, VPN, or another CDN sitting in front of Cloudflare does. Cloudflare then sees that proxy's egress IP, which turns every player behind it into a single counted entity.

Confirm what IP Cloudflare actually sees before writing any threshold.

Step 1: Identify the actual counting key

A rate limiting rule has two parts:

A matching expression, which decides which requests the rule applies to.
A list of characteristics, which defines the counting key.

Characteristics can include the IP, the IP with NAT support (Business and Enterprise), and, on Enterprise with Advanced Rate Limiting, request headers, cookies, and other fields. An optional counting expression can further control which requests increment the counter.

For a game server API where many legitimate players share carrier NAT, counting purely by IP is the root cause of the lockout. A better key combines IP with something session-specific, such as a stable client-generated session header.


Rate limiting rule (dashboard equivalent):
Expression: (http.request.uri.path eq "/api/matchmaking/join")
Characteristics: cf.colo.id, ip.src, http.request.headers["x-player-session"]
  # combine, not IP alone; header names are lowercase; header counting needs Enterprise ARL
Period: 10 seconds
Requests: 15
Action: Block, mitigation timeout 60 seconds

In the dashboard, the data center characteristic (cf.colo.id) is applied implicitly. Via the API, it must be listed explicitly, because counters are maintained per Cloudflare data center.

Header-based keys are client-controlled. An attacker script hammering the endpoint with rotating session headers can evade a header-keyed rule, so a second, looser IP-only rule should act as a backstop (shown in Step 2). The benefit is that fifty legitimate players behind the same NAT gateway, each with a distinct session token, no longer collapse into a single counted client.

If header counting is not available on your plan, Business plans can use "IP with NAT support." It is designed to separate distinct clients behind a shared IP.

Step 2: Separate authenticated and unauthenticated traffic

Matchmaking and lobby endpoints usually split into two kinds of traffic, and they need different tolerances:

Pre-auth traffic (login, token refresh) is the actual attack surface for credential stuffing and deserves a tight IP-based limit.
Post-auth traffic (join queue, submit match result) carries a session token and can afford a looser limit keyed on that token.

The call below writes all three rules to the zone's rate limiting phase entrypoint.


# PUT replaces ALL rules in the zone's http_ratelimit entrypoint ruleset.
# Fetch existing rules first with GET if you have any.
curl -X PUT \
  "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/rulesets/phases/http_ratelimit/entrypoint" \
  -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" \
  -H "Content-Type: application/json" \
  --data '{
  "rules": [
    {
      "description": "pre-auth login guard",
      "expression": "http.request.uri.path eq \"/api/auth/login\"",
      "action": "block",
      "ratelimit": {
        "characteristics": ["cf.colo.id", "ip.src"],
        "period": 60,
        "requests_per_period": 10,
        "mitigation_timeout": 60
      }
    },
    {
      "description": "post-auth matchmaking",
      "expression": "http.request.uri.path eq \"/api/matchmaking/join\" and len(http.request.headers[\"authorization\"]) gt 0",
      "action": "block",
      "ratelimit": {
        "characteristics": ["cf.colo.id", "ip.src", "http.request.headers[\"x-player-session\"]"],
        "period": 10,
        "requests_per_period": 20,
        "mitigation_timeout": 60
      }
    },
    {
      "description": "matchmaking IP-only backstop",
      "expression": "http.request.uri.path eq \"/api/matchmaking/join\"",
      "action": "block",
      "ratelimit": {
        "characteristics": ["cf.colo.id", "ip.src"],
        "period": 10,
        "requests_per_period": 300,
        "mitigation_timeout": 60
      }
    }
  ]
}'

Two caveats apply to this configuration:

Header presence is not token validation. The authorization check only confirms the header is present, not that the token is valid. Token validation still has to happen at the origin, or with API Shield JWT validation where available.
The thresholds are placeholders. Tune them against your own traffic.

The pre-auth rule uses Block with a short timeout rather than a Managed Challenge, and the reason is specific to game clients. Challenges require a browser environment to solve. A native game client or launcher calling a JSON API cannot complete one, so for those clients a challenge behaves exactly like a block, just less transparently.

Managed Challenge is a reasonable choice only when the login flow runs in a real browser or embedded webview. In that case, consider Turnstile in the login page as well.

Step 3: Deploy in Log mode first, then promote

On Enterprise, every new or modified rate limiting rule should ship with the Log action before it ever blocks anything. Log mode records what would have happened without enforcing it, which is the most reliable way to know how many real players a threshold would catch.

On other plans, the Log action is not available. Instead:

Start with conservative thresholds.
Scope the rule narrowly at first.
Watch Security Events closely immediately after enabling it.

Review the logged events for at least one full peak-traffic window before promoting the rule to Block. For a game server, that usually means a weekend evening, not a Tuesday morning.

Watch out for a second gotcha: Log mode data can look clean during a quiet weekday and still produce a wave of false positives under the exact conditions the rule was built to survive, such as a launch, a patch day, or a regional event.

Step 4: Add an allowlist for known infrastructure

Game server backends often call their own APIs from a small, known set of IPs, for health checks, cross-region state sync, or admin tooling. These should bypass rate limiting through an explicit Skip rule rather than being tuned around indirectly.

Skip rules are custom rules in the http_request_firewall_custom phase, which always runs before the http_ratelimit phase.


Custom rule with Skip action (http_request_firewall_custom phase):
Expression: ip.src in {203.0.113.10 203.0.113.11 198.51.100.20}
Action: Skip -> All rate limiting rules
  (optionally also: remaining custom rules; skipping WAF Managed Rules is a separate security tradeoff)

Ordering matters within each phase, but not in the way it might seem:

Relative to rate limiting: the custom rules phase always runs first, so a Skip rule there can bypass the rate limiting phase regardless of how the rate limiting rules are ordered.
Within the custom rules phase: rules are evaluated in list order. The Skip rule must sit above any custom rule it is meant to bypass, such as a block rule.

Only skip WAF Managed Rules for these IPs if you trust them fully, since a compromised internal host would then bypass managed protections too. Phase ordering is documented in the Ruleset Engine reference.

Verify and test

Verify the configuration change with something more concrete than "it looks fine in the dashboard." Start by confirming the counting key with a controlled test: send authenticated requests from the same IP using two distinct session headers, and confirm they count separately.


# Send 25 requests with one session header; with a limit of 15/10s,
# expect 429 responses after roughly 15 requests
for i in $(seq 1 25); do
  curl -s -o /dev/null -w "%{http_code}\n" \
    -H "Authorization: Bearer PLACEHOLDER_TEST_TOKEN" \
    -H "X-Player-Session: session-A" \
    https://api.example-game.com/api/matchmaking/join
done
# Immediately repeat with X-Player-Session: session-B from the same source IP.
# If the rule is keyed correctly, session-B starts with a fresh counter.

Then cross-check and extend the testing:

Cross-check Security Events. Filter by rule ID to confirm which requests were counted or blocked, and why. Counting is approximate and per data center, so expect small deviations from exact thresholds.
Verify the allowlist. Health-check traffic from the known infrastructure IPs should never appear in the rate limit's triggered events. If it does, check the following:
- That the Skip rule's IP set matches the real egress IPs.
- That the Skip rule is enabled.
- That it lives in the custom rules phase and skips rate limiting rules.
Load-test the pre-auth login path during a simulated peak window. Don't rely on steady-state traffic, because credential-stuffing patterns and legitimate login bursts both spike sharply and look similar in aggregate.

A Cloudflare rate limit is a blunt instrument by design, and the honest fix is rarely a single magic threshold. It is a set of rules that:

Separate authenticated from anonymous traffic.
Key on something more specific than a shared IP, where your plan allows it.
Roll out cautiously before enforcement.
Carve out known infrastructure explicitly rather than hoping it slips through.

For more infrastructure guides, see kuryzhev.cloud. It is worth treating rate limiting as a living configuration tied to launch calendars and patch schedules, not a rule written once during initial setup and forgotten until the next locked-out player ticket arrives.

SLO burn-rate alerts explained through a traffic spike

Oleksandr Kuryzhev — Tue, 22 Sep 2026 07:02:23 +0000

Originally published on kuryzhev.cloud

SLO burn-rate alerts are supposed to tell an on-call engineer one thing clearly: how fast the error budget is being consumed. A typical failure scenario looks like this: a marketing campaign or a bot crawler sends a sudden burst of traffic, request volume triples for ten minutes, and within minutes a page fires claiming the service will exhaust its monthly error budget in a couple of hours at the current burn rate. The dashboard looks alarming. The actual error rate, when checked manually, is well within normal bounds. The alert was arguably correct about the math and not very useful about the reality.

This is a common misunderstanding about SLO burn-rate alerts: they measure a rate of consumption, not an absolute failure count, and ratios can move sharply when volume changes even if underlying reliability has not meaningfully degraded. Understanding what burn-rate alerts actually compute — and where the traffic-spike blind spot comes from — is the difference between trusting the page and quietly routing it to a muted channel.

Failure scenario

Consider a service with a 99.9% availability SLO over a 30-day window. That target allows 0.1% of requests to fail; if traffic were perfectly uniform, it corresponds to roughly 43 minutes of full unavailability across the month. Most implementations express the budget in failed requests rather than wall-clock minutes, because request-based SLIs are what the alerting rules actually query.

A burn-rate alert compares the ratio of failed requests to total requests over a short window (say 5 minutes) and a longer window (say 1 hour), then compares that ratio against a multiple of the error budget. A burn rate of 1 means the budget is being consumed exactly as fast as the window allows; a burn rate of 14.4 means a 30-day budget would be gone in roughly two hours if the rate held.

During a traffic spike, the denominator grows quickly while the numerator can grow for reasons unrelated to a code regression. Suppose baseline traffic is 1,000 requests per minute with 1 error — a 0.1% error rate, exactly at the target, so the baseline burn rate is 1x. If traffic climbs to 5,000 requests per minute and errors climb to 72 because a connection pool sized for baseline load starts rejecting, the ratio is 1.44%: a 14.4x burn rate against the 0.1% budget, which is enough to trip a paging threshold on its own. Note what the ratio does and does not tell you here: errors grew 72x in absolute terms while the ratio grew only 14.4x, because the denominator grew alongside them. A rule keyed only to the ratio cannot distinguish that shape from a sustained regression at steady traffic, and may page on behavior the service returns from as soon as autoscaling catches up.

Watch out for alerts configured with a single short window (like 5 minutes). They react fastest to spikes and are the most likely to fire on transient traffic shape changes rather than genuine reliability degradation.

Why it happens

Burn-rate alerting, as described in the Google SRE workbook's multiwindow, multi-burn-rate approach, is designed to balance two competing needs: catching fast, severe outages quickly, and avoiding pages for noise. Implementations commonly use a short window (5m–1h) for fast detection and a long window (1h–6h) for confirmation, requiring both to breach a threshold before paging.

The traffic-spike problem appears when the short window is sensitive enough to catch a real two-minute outage, while the underlying infrastructure — connection pools, downstream rate limits, autoscaling lag — genuinely does produce more errors under sudden load. In that case the alert is not wrong: the error ratio did increase, and if it held for the full budget period it would exhaust the SLO. The weakness is the implicit assumption that current conditions are representative of the next hour, which often fails for a spike that resolves once autoscaling catches up or campaign traffic tapers off.

A second, quieter cause is how the query is constructed. If the burn-rate expression recalculates from raw counters at every evaluation cycle with no confirmation window, a five-minute burst dominates the short-window ratio while the rolling hour it is compared against has barely moved. Verify with the query itself rather than assuming: run the short-window and long-window expressions against Prometheus for the spike interval using promtool query instant, or the HTTP /api/v1/query_range endpoint for a time series, and compare the short-window ratio against the long-window ratio over the same period. If the short window jumped far beyond what the hour confirms, the alert shape is the problem rather than the service.

One more detail worth checking: the selector must match how failures are actually recorded. A rule that matches only status=~"5.." will miss client-side timeouts that never produced a response status, so the SLI definition and the alert expression need to agree on what counts as a failed request.

The fix

The standard fix is not to remove burn-rate alerting but to implement it with multiple windows and multiple burn-rate thresholds, as documented in the Google SRE workbook. A fast-burn alert (high threshold, short window) should require confirmation from a longer window before paging; a single-window alert should not page on its own.

Here is a Prometheus recording rule and alert pair implementing a multi-window burn-rate check for a 99.9% SLO:

groups:
  - name: slo-burn-rate
    rules:
      # Single source of truth for the SLO error budget (1 - 0.999)
      - record: slo:error_budget:ratio
        expr: vector(0.001)

      # Fast window: catches severe short outages
      - record: slo:requests_errors:ratio_rate5m
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m]))
          /
          sum(rate(http_requests_total[5m]))

      # Slow window: confirms the trend isn't a transient spike
      - record: slo:requests_errors:ratio_rate1h
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[1h]))
          /
          sum(rate(http_requests_total[1h]))

  - name: slo-burn-rate-alerts
    rules:
      - alert: ErrorBudgetFastBurn
        # A 14.4x burn rate exhausts a 30-day budget in roughly
        # two hours if sustained. Page only when BOTH windows
        # agree, not just the noisy 5m one.
        expr: |
          slo:requests_errors:ratio_rate5m
            > on() group_left() (14.4 * slo:error_budget:ratio)
          and
          slo:requests_errors:ratio_rate1h
            > on() group_left() (14.4 * slo:error_budget:ratio)
        for: 2m
        labels:
          severity: page
        annotations:
          summary: "Fast error budget burn confirmed over 5m and 1h windows"

Requiring agreement across windows filters out the common case where a spike produces a sharp five-minute ratio jump that the one-hour window does not confirm. For teams building this in Grafana, the Grafana Alerting documentation covers multi-condition alert rules; depending on the Grafana version, two queries can be combined with a boolean expression in the rule editor rather than maintaining two separate alert definitions.

A complementary fix is gating on absolute error volume alongside the ratio, since a true incident usually produces both a ratio increase and a meaningful rise in raw error count:

sum(increase(http_requests_total{status=~"5.."}[5m])) > 50
and
slo:requests_errors:ratio_rate5m
  > on() group_left() (14.4 * slo:error_budget:ratio)

Pick the absolute threshold from the service's own baseline rather than copying 50; it is a floor that says "too few failures to wake anyone," and the right value depends on traffic volume.

Watch out for hardcoding the numeric SLO target in every alert expression. When the target changes — say from 99.9% to 99.95% — every burn-rate rule needs updating in sync, and it is easy to miss one. Recording the budget once as its own rule (as above), or templating the rule files, keeps the threshold in one place; see the Alertmanager routing notes on DevOps_DayS for how the resulting labels affect routing.

Prevention checklist

A few structural choices prevent most traffic-spike false pages before they happen:

- Use multi-window, multi-burn-rate alerts (5m+1h, 30m+6h) instead of a single window
- Require BOTH windows to breach threshold before paging, not either
- Alert on absolute error count as a secondary gate, not ratio alone
- Set separate burn-rate thresholds for page vs. ticket severity
  (e.g. 14.4x for page, 6x for a lower-urgency ticket)
- Keep the SLI selector aligned with the SLO definition (timeouts included)
- Review burn-rate alert history after every known traffic spike
  and note whether it paged unnecessarily
- Keep the SLO target as a single source of truth (recording rule or template),
  not copy-pasted across every alert expression
- Document the expected behavior during known high-traffic events
  (product launches, sales, scheduled batch jobs) in the runbook

None of this eliminates every noisy page. Capacity problems that only appear under load — a connection pool sized for baseline traffic, a downstream API with its own rate limit — are real reliability risks and may well deserve a page, even if they only reveal themselves during a spike. The goal of tuning SLO burn-rate alerts is not silence; it is making sure the page reflects a sustained trend the error budget actually cares about, rather than a five-minute ratio artifact. The burn-rate math and window-selection tables live in the Google SRE workbook chapter on alerting on SLOs, and the Prometheus alerting best practices guide covers the more general question of what belongs in a page at all — both are worth reading before finalizing thresholds for a specific SLO.

IRSA explained: how EKS pods borrow AWS permissions

Oleksandr Kuryzhev — Mon, 21 Sep 2026 07:01:49 +0000

Originally published on kuryzhev.cloud

A pod in an EKS cluster needs to read from S3, and someone on the team pastes an AWS access key into a Kubernetes secret because it's the fastest way to unblock a deploy. This is the moment IAM roles for service accounts, better known as IRSA, exists to prevent. Instead of shipping long-lived credentials into a cluster, IRSA lets a pod exchange a short-lived token for temporary AWS credentials tied to a specific IAM role. No keys stored anywhere, no rotation cron job, no secret sitting in etcd waiting to leak.

The mechanism is not magic, and understanding the moving parts is what separates teams that configure IRSA correctly from teams that burn time debugging AccessDenied errors that appear to make no sense. This explainer walks through what IRSA actually does under the hood, the failure modes that get reported most often, the setup that avoids them, and where the pattern gets interesting once you're running many workloads across multiple accounts.

What this actually does

EKS clusters expose an OIDC (OpenID Connect) issuer URL, which is effectively an identity provider that AWS IAM can trust. When you enable IRSA, you register that OIDC provider with IAM, then create an IAM role with a trust policy scoped to a specific Kubernetes namespace and service account name. The pod's service account gets annotated with the role's ARN, and the EKS pod identity webhook injects a projected service account token volume and environment variables into any pod using that service account.

At runtime, the AWS SDK inside the container reads AWS_WEB_IDENTITY_TOKEN_FILE and AWS_ROLE_ARN, calls STS's AssumeRoleWithWebIdentity, and gets back temporary credentials. Their lifetime depends on the role's configured maximum session duration and on what the SDK requests. The SDK handles refresh automatically as long as it supports the web identity credential provider — current major versions do, including AWS SDK for JavaScript v3, boto3, and AWS SDK for Go v2. Very old SDK majors may not, so check the SDK version if credentials never appear.

The important detail: the token is a JWT signed by the cluster's OIDC issuer, and STS validates that signature against the keys the issuer publishes at its JWKS endpoint. The thumbprint you supply when registering the provider pins the TLS certificate chain used to reach that endpoint. There is no AWS-side database mapping pods to roles. Trust lives entirely in the IAM role's trust policy conditions, which check the token's subject claim (namespace and service account) and audience claim. Get those conditions wrong and you either lock everyone out or, worse, open the role to more service accounts than intended. See the official IRSA documentation for the exact trust policy shape AWS expects.

How people use it wrong

A commonly reported failure is copying a trust policy from one project to another without updating the namespace or service account name in the condition. The role assumes fine in the environment it was written for, then fails in the next one because the condition still references staging-app. The error returned to the pod is a generic AccessDenied from STS, which gives almost no hint about which part of the trust condition didn't match.

Watch out for using StringEquals when the intent was to allow multiple service accounts with a wildcard. IAM condition operators compare exact strings unless you explicitly use StringLike. A typical failure pattern: someone adds a second service account to a namespace, assumes it inherits the same role because it's in the same namespace, and is surprised when it can't assume anything.

Another frequent mistake is granting IAM permissions far broader than the workload needs, on the theory that IRSA itself is "secure enough" so the attached policy doesn't need much scrutiny. IRSA controls who can assume the role, not what the role can do once assumed. A role trusted only by one exact service account can still have an AdministratorAccess policy attached, which defeats most of the point.

Watch out for pods that don't restart after a service account annotation changes. The webhook injects the token volume and environment variables at pod creation time, so updating the IAM role ARN on an existing service account does nothing until pods are recreated — a rolling restart or deployment rollout is required, and this is easy to forget mid-incident.

The correct approach

Start with least privilege on the IAM policy side, scoped to specific resource ARNs rather than *. Then build the trust policy to match exactly one namespace and service account per role wherever practical — this keeps blast radius contained and makes audits straightforward.

A minimal trust policy for a service account named s3-reader in namespace data-pipeline looks like this. Verify the OIDC provider ARN and issuer URL against your cluster with aws eks describe-cluster --name <cluster> --query "cluster.identity.oidc.issuer" before pasting.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::123456789012:oidc-provider/oidc.eks.eu-central-1.amazonaws.com/id/EXAMPLE"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.eu-central-1.amazonaws.com/id/EXAMPLE:sub": "system:serviceaccount:data-pipeline:s3-reader",
          "oidc.eks.eu-central-1.amazonaws.com/id/EXAMPLE:aud": "sts.amazonaws.com"
        }
      }
    }
  ]
}

The service account itself only needs the role ARN annotation. This is the piece that actually wires a pod to the role at admission time.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: s3-reader
  namespace: data-pipeline
  annotations:
    # this ARN must point at a role whose trust policy matches this namespace and name
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/s3-reader-role

Always include the aud condition, not just sub. The audience claim is what ties the token to STS as its intended recipient; checking it prevents a token minted for a different audience from satisfying the condition. Cross-check your Terraform or CloudFormation modules against the current Kubernetes service account documentation, since projected token behavior has changed across cluster versions.

Advanced patterns

Multi-account setups are where IRSA gets genuinely interesting. A cluster in account A can assume a role in account B if account B has the cluster's OIDC issuer registered as an identity provider and the role's trust policy references that provider with the same sub and aud conditions. No role chaining through account A's IAM is required. This is useful for centralized logging or shared data-lake accounts where the EKS cluster lives in a workload account but needs to write to a platform account's S3 bucket.

EKS Pod Identity is now generally available and is AWS's recommended default for new EKS clusters. It removes the OIDC trust policy work in favor of an EKS-managed association between a service account and a role, configured through the EKS API rather than IAM trust conditions. IRSA remains fully supported and is still the more portable pattern if you run workloads outside EKS or on Kubernetes distributions that have no Pod Identity agent.

For platform teams managing many roles, a naming convention like irsa-<namespace>-<serviceaccount> combined with a Terraform module that generates both the role and the trust policy from the same two input variables prevents the copy-paste namespace mismatch described earlier. Pairing this with OPA/Gatekeeper or Kyverno policies that reject service account annotations pointing at roles outside an approved naming pattern catches drift before it reaches production. If you're standardizing this across a platform, the DevOps_DayS archive covers related Kubernetes hardening patterns worth pairing with an IRSA rollout.

Performance notes

IRSA's overhead is mostly invisible day-to-day but not zero. The projected token has a bounded lifetime set by the pod identity webhook, and the kubelet rotates it before expiry. The SDK caches STS credentials until close to expiry, so steady-state API calls don't pay an extra round trip per request. The cost shows up at pod startup: the first AWS SDK call in a freshly started container blocks on an AssumeRoleWithWebIdentity call, adding one network round trip before any actual AWS API work begins.

For latency-sensitive cold-start workloads — short-lived Jobs, for example — that STS round trip can matter. One practical option is to initialize the credential provider early in application startup so the exchange overlaps with other initialization work; the broader mitigation is keeping pods long-running where possible rather than spinning up a fresh pod per unit of work.

At scale, watch STS throttling. STS applies regional request limits, and a cluster where many pods restart simultaneously during a rolling node replacement can generate a burst of AssumeRoleWithWebIdentity calls. Whether this becomes a problem depends heavily on cluster size, workload churn, and what else in the account calls STS. During large node group upgrades it's worth checking CloudTrail for throttling errors rather than assuming AccessDenied always means a policy misconfiguration. Verify with CloudTrail event history filtered on the STS API before concluding it's an IAM roles for service accounts trust issue versus a rate limit.

BGP routing basics for engineers tired of traffic drops

Oleksandr Kuryzhev — Sun, 20 Sep 2026 07:01:49 +0000

Originally published on kuryzhev.cloud

A deploy goes out, nothing in the application changed, and yet a slice of users in one region starts seeing timeouts for several minutes before everything quietly recovers. Nobody touched the load balancer. Nobody touched DNS. What actually happened lives one layer down, in BGP routing basics that most application engineers never had to learn until the day a route flap took down their weekend. BGP is not exotic anymore — it runs inside cloud interconnects, between regions, and at the edge of every CDN — so understanding the failure modes is now a normal part of the job, not a networking specialty.

These are independent, practical points. Skip around, verify claims in your own environment, and treat anything with real financial risk as a candidate for a lab test before it hits production.

BGP does not know your traffic is dropping

BGP only cares about reachability, not performance. A path can be fully "up" from BGP's perspective while packet loss on that path is severe. Withdrawal happens when the session goes down, when hold timers expire, when an upstream withdraws the prefix, or when policy changes — not because latency or loss crossed a threshold. This is the single most common source of confusion when engineers assume routing will "route around" a bad path the way an application load balancer would.

If you need loss-aware failover, that has to be built with BFD (Bidirectional Forwarding Detection) or active health checks layered on top of BGP, not BGP alone. Watch out for teams that assume multi-homing alone gives them automatic quality-based failover — it gives redundancy, not intelligence.

Route flapping is usually a symptom, not the disease

When a prefix appears and disappears repeatedly (flapping), the instinct is to blame BGP. In practice the underlying cause is frequently something else: a flaky physical link, an interface resetting, a misconfigured timer, or a route reflector under memory pressure. BGP is often just the messenger reporting instability that already existed at a lower layer.

Route flap dampening exists to suppress the noise, but it has a real cost — accumulated penalties can delay re-advertisement of a prefix after it stabilizes, and how long depends entirely on the configured half-life, reuse, and suppress values. Check dampening state with a command like this on a Cisco IOS-style device:

show ip bgp dampening dampened-paths
# columns show the penalty and the reuse time remaining
# a high penalty on a prefix that "should" be stable is worth investigating

show ip bgp dampening parameters
# confirm the half-life, reuse, suppress and max-suppress values in effect

If dampening is masking a real outage, that can be worse than no dampening at all — it hides the symptom from monitoring for exactly the window when responders need visibility. Note that dampening is disabled by default in many implementations, so confirm whether it is configured before blaming it.

AS_PATH length is not a reliability signal

BGP's best-path selection prefers the shortest AS_PATH, but "shortest" has nothing to do with "healthiest." AS_PATH is also only one step in the decision process — weight, local preference, locally originated routes, and AS_PATH all come before MED and IGP metric, so AS_PATH is decisive only when the earlier tie-breakers are equal. When they are, a three-hop path through a congested transit provider can beat a four-hop path through a well-provisioned one.

For example, a peering session with a regional ISP can look attractive on paper (short AS_PATH) while carrying more loss than a longer path through a better-provisioned backbone. Verify actual path performance with traceroute and packet-loss sampling, not just show ip bgp output, before trusting AS_PATH length as a proxy for quality.

Prefix hijacks are still mostly a config-hygiene problem

Route leaks and hijacks — where a network accidentally or maliciously announces prefixes it doesn't own — remain one of the more consequential BGP failure modes, and they are still largely reducible with basic filtering. RPKI (Resource Public Key Infrastructure) lets a network cryptographically validate that a route announcement comes from an authorized origin AS. Deployment varies considerably by region and operator and is not universal, so treat origin validation as a strong mitigation rather than a guarantee — it does not, on its own, validate the rest of the AS_PATH.

On a router configured with an RPKI validator session, the validation state can be inspected like this:

show ip bgp rpki servers
# confirm the router actually has an established session to a validator

show ip bgp rpki table
# lists prefixes with their registered origin AS and max-length

show ip bgp 203.0.113.0/24
# per-prefix output shows the validation state:
#   valid   - origin AS matches a ROA
#   invalid - origin AS or prefix length conflicts with a ROA
#   not found - no ROA covers this prefix; policy decides what happens

Watch out for the "not found" state being silently treated the same as "valid" in permissive configurations — that is the gap leaks exploit, and it is a policy decision you have to make explicitly. The validation model itself is specified in RFC 6811 (BGP Prefix Origin Validation) and the broader architecture in RFC 6480; read those before rolling anything out, because the difference between marking invalids and dropping them has real blast radius.

Cloud interconnects hide BGP but don't remove it

Anyone running AWS Direct Connect, Azure ExpressRoute, or Google Cloud Interconnect is running BGP whether they realize it or not — those services establish real BGP sessions with the provider's edge routers. The abstraction is convenient until a session drops and traffic that should have failed over silently doesn't, because a static route or a mismatched BGP community kept the dead path preferred.

For AWS specifically, Direct Connect virtual interfaces expose BGP session state and advertised route counts that are worth checking after any change to on-premises routers; the design considerations are documented in the AWS Direct Connect resiliency guide. A plausible failure here is a BGP session staying "established" while the underlying circuit is degraded — session state alone is not proof the path is healthy, which is precisely why BFD exists on these links.

Communities are how you signal behavior across AS boundaries

BGP communities are often introduced as metadata for filtering or documentation, but they are also a primary lever for requesting routing behavior across AS boundaries — things like "don't advertise this to peers," "prefer this path regionally," or "de-prioritize during maintenance." A missing or incorrect community tag during planned maintenance is a commonly documented cause of unexpected traffic shifts, and it is easy to miss because the configuration itself looks syntactically fine.

The important detail: a community is only a signal. It changes nothing unless the receiving AS has a matching policy, and it is not even transmitted unless the session is configured to send it.

! tag outbound announcements with a community the upstream acts on
ip community-list standard UPSTREAM-DEPREF permit 64512:100
!
route-map SET-COMMUNITY permit 10
 set community 64512:100 additive
!
router bgp 64512
 neighbor 203.0.113.1 remote-as 64513
 neighbor 203.0.113.1 send-community
 neighbor 203.0.113.1 route-map SET-COMMUNITY out

Without send-community, the attribute is stripped and the policy silently does nothing. Standard communities also do not carry well-known meaning across providers beyond the reserved ones (NO_EXPORT, NO_ADVERTISE, NO_EXPORT_SUBCONFED), so the numeric values must come from the upstream's published policy. If a provider documents a "graceful maintenance" community, confirm it is actually applied and visible on the receiving side before assuming a maintenance window will be traffic-safe.

Convergence time is the number that actually matters

The metric that predicts user-visible impact during a failure is convergence time — how long it takes every affected router to agree on the new best path after a change. The default timers in the BGP-4 specification (60-second keepalive, 180-second hold time) were designed for stability over speed, though vendor defaults vary and should be confirmed per platform. If a failure is detected by link-down and fast external fallover, teardown is immediate; the painful case is a failure that keeps the interface up — a degraded circuit, a broken path beyond the peer, a black-holing middlebox — where nothing happens until the hold timer expires.

Tuning timers down helps but introduces its own risk: overly aggressive timers cause false-positive session resets under normal jitter. A commonly recommended pattern is BFD for sub-second failure detection paired with less aggressive BGP timers, which separates fast detection from protocol stability, but the right values are vendor- and topology-dependent. The BGP-4 specification (RFC 4271) is the canonical reference for timer semantics when vendor documentation is ambiguous.

None of this replaces reading your specific vendor's or cloud provider's documentation for exact syntax and defaults — behavior varies enough between implementations that assumptions from one platform can quietly break another. For more networking and infrastructure breakdowns written for engineers who'd rather understand the failure than memorize the command, check the rest of the writing at kuryzhev.cloud.

WordPress performance profiling with php-fpm status and slow logs

Oleksandr Kuryzhev — Sat, 19 Sep 2026 07:01:46 +0000

Originally published on kuryzhev.cloud

A WordPress site that loads fine on a warm cache but crawls under real traffic is a recurring pattern in inherited infrastructure. Before reaching for a bigger instance or a caching plugin, WordPress performance profiling with php-fpm's status page and slow log gives you actual evidence instead of guesses. It is tempting to jump straight to "add more workers" or "enable object cache," which sometimes helps and sometimes just delays the same problem at a slightly higher traffic level.

Why this checklist

php-fpm sits between nginx (or Apache) and the PHP code that WordPress actually runs. It manages a pool of worker processes, and the questions that matter most — is the site CPU-bound, database-bound, or waiting on external APIs — tend to show up in its status page and slow log earlier than in higher-level dashboards. Skipping this data source means you're profiling blind.

The complication is that php-fpm's status page is not exposed by default in most distribution packages, and the slow log requires both a threshold and a log path before it records anything. Neither is dangerous to enable, but both are frequently forgotten until an incident forces someone to look for them under pressure. That's the wrong time to learn the syntax.

This checklist assumes a fairly standard setup: nginx or Apache as the front end, php-fpm on a currently supported PHP 8.x branch, and WordPress running as the application. Directive names have been stable across recent 8.x releases, but verify them against your installed version rather than assuming. It does not assume a specific hosting stack, since the same principles apply whether php-fpm runs in a container, a VM, or bare metal.

The goal is not to memorize commands. It's to build a repeatable habit: enable the status endpoint, enable the slow log, correlate both against real request patterns, and only then decide whether the fix is code, configuration, or capacity. For infrastructure-level context on how this fits into a broader monitoring strategy, see the DevOps_DayS knowledge base.

The checklist

Confirm the pool manager mode. Check whether pm is set to dynamic, static, or ondemand in your pool config. Dynamic is the common packaged default; static gives more predictable memory usage under load and is easier to reason about while profiling.
Enable the status page. Add pm.status_path to the pool configuration and expose it through the web server, restricted to internal addresses only.
Enable the slow log with a realistic threshold. Set request_slowlog_timeout to something like 2s as a starting point, not 30s, or you'll miss most of what's worth seeing. It has no effect unless slowlog is also set.
Set a slow log path that log rotation actually covers. A slow log with no rotation policy fills disks quietly over weeks.
Cross-check pool size against actual concurrency. Use the status page's active processes and max children reached counters, not just CPU graphs.
Watch for "max children reached" events specifically. This is a strong signal that php-fpm is undersized for current traffic rather than that PHP itself is slow.
Correlate slow log entries with the database, not just PHP. WordPress slow requests frequently trace back to unindexed queries or plugin-driven wp_options autoload bloat.
Check request duration distribution, not just averages. A healthy-looking median can hide a long tail that only appears under concurrent load, which is why percentiles matter here.
Verify opcache is enabled and not being invalidated constantly. Frequent invalidation from file changes (common on staging or auto-deploy setups) undercuts much of the benefit.
Rule out external HTTP calls inside request handling. Slow log backtraces often point to blocking calls to third-party APIs, ad networks, or license-check servers inside plugin code.
Confirm the status page format matches what your monitoring tool expects. php-fpm can emit plain text, JSON, XML, and HTML; pick JSON for anything feeding Prometheus or a similar system.
Document the baseline before changing anything. Without a documented "before" state, you can't honestly claim the fix worked.

Here's a minimal pool configuration enabling both the status page and the slow log, with inline notes on the non-obvious parts.

[www]
; expose the status endpoint internally; the web server must proxy this path
pm.status_path = /status

; ping endpoint for liveness checks separate from application health
; ping.response defaults to "pong"; nginx must proxy this path too
ping.path = /ping
ping.response = pong

; slowlog path is required for request_slowlog_timeout to take effect
slowlog = /var/log/php-fpm/www-slow.log
request_slowlog_timeout = 2s

; static avoids spawning workers mid-spike; max_children is the fixed pool size
pm = static
pm.max_children = 12

To pull the status page in a script-friendly format, request JSON output through the web server that proxies the endpoint. Quote the URL so the shell does not interpret ? and &:

curl -s "http://127.0.0.1/status?full&json" \
  -H "Host: example.com" \
  | jq '{active: .["active processes"],
         maxed: .["max children reached"],
         queue: .["listen queue"]}'

# "max children reached" > 0 over any recent window means the pool is undersized
# a nonzero, growing "listen queue" means requests wait before PHP even starts

Commonly missed items

The status page's full query parameter, which lists per-process detail including the currently executing script and its request duration, gets overlooked constantly. Without it, you see aggregate numbers but not which specific request is stuck right now — the difference between confirming a problem and diagnosing it.

It is also easy to enable the slow log once during an incident, find the offending plugin, then forget to revisit the threshold afterward. A permanent slow log with too aggressive a threshold generates noise that trains people to ignore it — the same failure mode as alert fatigue in any monitoring pipeline.

Watch out for: restarting php-fpm to apply pool changes without checking whether a graceful reload is available. A hard restart can drop in-flight requests. On Debian and Ubuntu the unit is version-qualified, so use something like systemctl reload php8.3-fpm; on RHEL-family systems it is typically systemctl reload php-fpm. Confirm the unit name with systemctl list-units 'php*fpm*' and check your distribution's documentation, since reload semantics differ between packages and versions.

Another frequently missed item is whether opcache.validate_timestamps is appropriate for the environment. Enabled, it makes PHP check file modification times, throttled by opcache.revalidate_freq; the cost depends on your filesystem, revalidation frequency, and PHP version, so treat it as something to measure rather than assume. Disabled in a frequently-deployed staging environment, it will serve stale bytecode silently until you reload the pool.

Watch out for: confusing php-fpm's slow log with PHP's own error log. They serve different purposes — the slow log writes a PHP backtrace of the script that exceeded the timeout, while the error log captures fatal errors and warnings — and conflating the two during an investigation wastes time chasing the wrong signal.

Finally, the status page's listen queue value is routinely ignored even though it's one of the clearest indicators of saturation: a nonzero and growing listen queue means requests are waiting for a free worker before PHP even starts executing.

Automation ideas

Scraping the JSON status endpoint on a schedule and feeding it into Prometheus via a lightweight exporter turns this checklist into an ongoing dashboard instead of a one-time investigation. Several community exporters for php-fpm exist; whichever you choose, verify it exposes max children reached and listen queue as first-class metrics, since those are the two values most worth alerting on.

A cron-based slow log summarizer, run hourly, can group entries by the triggering plugin or theme function and post a daily digest. That turns raw log noise into a short, reviewable list without requiring a full log aggregation platform for smaller sites.

Here's a small Prometheus alerting rule for the listen queue metric, assuming an exporter that surfaces it as phpfpm_listen_queue. Adjust the metric and label names to match your exporter:

groups:
  - name: php-fpm
    rules:
      - alert: PhpFpmListenQueueGrowing
        expr: phpfpm_listen_queue > 0
        # avoid firing on single-request blips
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "php-fpm listen queue nonzero for 2+ minutes"

For teams running php-fpm inside containers, wiring the status endpoint into a readiness probe is worth considering carefully — a probe that queries /status can surface pool exhaustion before it becomes a full outage, though it adds a dependency the probe logic must handle gracefully if the status page itself is misconfigured or unreachable. Consult the official php-fpm configuration documentation for the current list of directives, since defaults and available options have shifted across PHP 8.x releases.

None of this replaces application-level profiling tools, but it establishes the baseline layer almost everything else depends on. WordPress performance profiling that starts at the pool level tends to produce clearer answers than one that starts at the plugin level, because a caching layer built on top of an undersized php-fpm pool mostly moves the saturation point rather than removing it.

Cloudflare cache bypass mistakes on dynamic WordPress paths

Oleksandr Kuryzhev — Fri, 18 Sep 2026 11:00:51 +0000

Originally published on kuryzhev.cloud

A WordPress site behind Cloudflare starts serving the wrong cart contents to different visitors, or the admin dashboard flashes a cached homepage instead of the login form. Both symptoms can trace back to the same root cause: a Cloudflare cache bypass rule that doesn't actually cover every dynamic path it needs to. This is a commonly reported misconfiguration on WordPress installs sitting behind Cloudflare's proxy, and it often surfaces after someone widens caching to raise the hit ratio and then starts trusting the cache more than the rules justify.

WordPress generates a mix of static and dynamic content from the same domain — product pages that can be cached for hours next to cart, checkout, and account pages that must never be cached. Cloudflare's default cache behavior, combined with Cache Rules, does not know this distinction unless it's told explicitly. Cloudflare's documented default is to cache based on a list of file extensions and a few related heuristics, which is not enough for a CMS where nearly every page resolves through index.php.

Context

Cloudflare sits in front of the origin as a reverse proxy. By default it caches static assets — images, CSS, JS — matched by extension, while HTML responses are not cached by default because HTML is not in that default extension list. That default is the only thing standing between a dynamic WordPress page and the shared edge cache, and any rule that overrides it takes on the job of distinguishing personalized responses from public ones. WordPress complicates this because plugins like WooCommerce, membership systems, and custom REST endpoints all serve dynamic HTML from paths that look, to a naive rule, indistinguishable from static pages.

The operational goal is usually stated simply: cache everything cacheable, bypass everything session-specific. In practice this requires enumerating every dynamic path pattern a WordPress install can produce, and that list grows as plugins are added. A cache rule written during initial setup rarely gets revisited when a new plugin introduces its own dynamic endpoint, and that gap is a plausible origin for each of the failure modes below. Cloudflare's Cache Rules documentation (developers.cloudflare.com/cache/how-to/cache-rules) is the authoritative reference for expression syntax and precedence, and it's worth reading before assuming a rule behaves the way its name suggests. Note also that legacy Page Rules are deprecated for new configuration; Cache Rules are the current mechanism, and mixing both makes precedence harder to reason about.

Common failure 1: bypass rules that only match by URL path, not by cookie

A typical setup writes a Cache Rule matching http.request.uri.path contains "/cart" or similar and calls it done. This catches the obvious checkout flow but misses the broader problem: WooCommerce and most session-aware plugins set cookies — for example woocommerce_cart_hash or wp_woocommerce_session_ — that indicate a personalized response regardless of URL. If the homepage or a product page renders differently based on such a cookie, showing "3 items in cart" in a header, and the cache rule doesn't account for it, a cached response can serve one visitor's cart state to another.

The fix is to bypass cache when specific cookies are present, not just when specific paths are requested. A Cache Rule expression combining both conditions is more resilient:

(http.request.uri.path contains "/cart") or
(http.request.uri.path contains "/checkout") or
(http.request.uri.path contains "/my-account") or
(http.cookie contains "woocommerce_cart_hash") or
(http.cookie contains "wp_woocommerce_session_") or
(http.cookie contains "wordpress_logged_in_")

Set that rule's cache eligibility to Bypass cache. The exact cookie names depend on plugin versions and any prefix customization, so confirm them in browser devtools against the actual site rather than copying a list.

Watch out for: the number of Cache Rules per zone and the maximum expression length are plan-dependent. Verify current limits against the Cache Rules limits documentation before assuming an expression this long will save on a given plan.

Common failure 2: caching REST API and admin-ajax responses meant to be dynamic

A second recurring pattern involves overly broad "cache everything" rules applied at the zone level, often introduced to fix a low cache hit ratio. These rules can sweep up /wp-json/ and /wp-admin/admin-ajax.php, both of which WordPress uses constantly for dynamic operations — form submissions, live search, cart updates, nonce validation. Caching admin-ajax responses can cause a stale nonce to be served repeatedly, which may manifest as forms failing with a security-check error that has nothing to do with the plugin logic itself.

The safer pattern is to exclude these paths explicitly with a dedicated bypass rule:

(http.request.uri.path contains "/wp-json/") or
(http.request.uri.path contains "/wp-admin/") or
(http.request.uri.path eq "/wp-cron.php") or
(http.request.uri.path eq "/wp-admin/admin-ajax.php")

The action for that rule is cache eligibility Bypass cache. The explicit admin-ajax.php condition is redundant while the /wp-admin/ condition is present, but it is worth keeping if someone later narrows the admin path match.

Watch out for: rule order matters, and not in the direction people usually assume. Cloudflare evaluates all matching Cache Rules in list order, and for a given setting the last matching rule wins. So a broad "cache everything" rule placed below a bypass rule will override the bypass for overlapping requests. Put the narrow bypass rules after the broad rule, or scope the broad rule so it cannot match dynamic paths at all. Confirm evaluation order and the resulting effective settings in the dashboard's rule list rather than inferring them from rule names.

Common failure 3: relying on page-level exclusions instead of edge cache TTL discipline

A third failure mode is subtler and shows up over time rather than immediately. A site starts with careful path-based bypass rules, but as traffic grows, someone adds a blanket cache-everything rule with a long Edge Cache TTL to reduce origin load, intending it to apply only to static assets. Without respecting the Cache-Control headers WordPress and plugins already send, an aggressive fixed TTL can override origin intent for paths nobody meant to include.

This is where respecting existing headers matters. Many WordPress caching plugins (W3 Total Cache, WP Rocket, LiteSpeed Cache) send Cache-Control headers distinguishing dynamic from static responses. A Cloudflare rule that overrides those headers with a flat Edge TTL defeats that origin-level logic.

# Origin-side nginx snippet for a dynamic WordPress endpoint.
# Equivalent headers can be emitted from PHP instead, but not via
# nginx directives placed in functions.php.
location = /wp-admin/admin-ajax.php {
    add_header Cache-Control "no-store, no-cache, must-revalidate" always;
    include fastcgi_params;
    fastcgi_pass unix:/run/php/php-fpm.sock;
}

The Cloudflare-side counterpart is a Cache Rule that sets cache eligibility to Eligible for cache and sets Edge TTL to "Use cache-control header if present, use default Cloudflare caching behavior if not", rather than a fixed override value. Check the default cache behavior documentation for which headers Cloudflare honors and when a fixed TTL still wins regardless of origin headers — this detail has shifted across Cloudflare product iterations, so re-check current docs rather than assuming legacy Page Rules behavior applies to Cache Rules.

Safer operating pattern

A more durable Cloudflare cache bypass strategy for WordPress treats dynamic path exclusion as a maintained list, not a one-time setup task. Every new plugin that introduces a checkout flow, a membership gate, or a custom REST endpoint is a candidate for a new bypass condition, and the review is cheaper at plugin install time than after a symptom appears.

Checklist before enabling broader Cloudflare caching on WordPress:
1. List all cookie names set by session-aware plugins (cart, auth, membership)
2. Confirm /wp-json/, /wp-admin/, /wp-cron.php are explicitly bypassed
3. Verify rule order and effective settings: for a given setting the last
   matching Cache Rule wins, so scope or order the broad rule accordingly
4. Check Edge TTL setting: respect origin cache-control vs fixed override
5. Test with two separate browser sessions (private + normal) and compare
   responses that should be personalized
6. Inspect cf-cache-status on dynamic URLs; BYPASS or DYNAMIC is expected,
   HIT on a personalized page is a defect
7. Re-audit the rule list after any plugin that touches checkout, auth,
   or account pages is added or updated

Testing with two genuinely separate sessions — not just two tabs — is a practical way to catch a leaking cache before a monitoring alert or a customer report does. It is also worth pairing this with cache analytics in the Cloudflare dashboard to confirm hit ratios move in the expected direction after each rule change, rather than assuming a rule works because it saved without error.

None of this replaces reading the current Cache Rules documentation for the account's specific plan tier, since expression limits, evaluation order behavior, and header precedence have changed across Cloudflare's product history and may change again. For broader infrastructure hardening patterns that apply the same "verify before trusting the default" discipline, see the write-ups on kuryzhev.cloud.

Alertmanager Routing Fixes to Cut Prometheus Alert Fatigue

Oleksandr Kuryzhev — Thu, 17 Sep 2026 10:18:50 +0000

Originally published on kuryzhev.cloud

A pager goes off at 3 a.m. with dozens of notifications for the same disk-pressure event on one node, spread across three teams who all have the node exporter alert wired to their phones. By the time someone actually reads one of them, the disk has already recovered and the incident channel is full of "ack, resolved" messages nobody trusts anymore. This is alert fatigue in Prometheus Alertmanager routing, and it is a common reason on-call engineers start muting entire Slack channels instead of fixing root causes.

The frustrating part is that Alertmanager was built specifically to solve this problem. Grouping, inhibition, and silences exist precisely to stop duplicate noise. When teams still drown in alerts, it is commonly a routing configuration problem, not a Prometheus rule problem.

Failure scenario

Picture a mid-sized platform team running Prometheus with fifteen exporters across sixty nodes. Someone adds a new rules.yml file for disk space, memory pressure, and pod restarts, all firing independently. The route block in alertmanager.yml still uses the default configuration copied from a tutorial two years ago: a single receiver, no group_by customization beyond the default, and no inhibition rules.

During a real node failure, kubelet reports NotReady, the node exporter fires DiskPressure and MemoryPressure, and every pod scheduled on that node starts restarting. Each condition maps to a separate alert rule. With grouping left at the tool's default of grouping by all labels, near-identical alerts rarely merge, so Alertmanager ends up sending a batch of individual notifications instead of one grouped message covering the failing node.

The on-call engineer receives a wall of pings, most of which are symptoms of the same root cause. Over weeks, this repeats for every flapping node, every deployment rollout, and every cert renewal. Engineers start snoozing the pager app notification sound entirely, which is the exact failure mode alerting is supposed to prevent — a real incident gets treated the same as background noise.

Why it happens

Alert fatigue in Prometheus Alertmanager routing commonly traces back to a handful of root causes, and they compound each other.

First, grouping is too granular or missing entirely. If group_by is left unset, Alertmanager groups by all labels, which effectively treats every distinct label combination as its own notification. Even when group_by is set explicitly, including alertname means alerts with different names never merge into one notification regardless of what other labels they share — DiskPressure and MemoryPressure on the same node stay separate. Consolidating correlated symptoms into fewer notifications requires dropping alertname from group_by and grouping by scope labels like cluster and instance instead, accepting that alerts of different types on the same node will then land in one group.

Second, inhibition rules are absent. Inhibition tells Alertmanager to suppress lower-severity alerts when a related higher-severity alert is already firing. Without it, a NodeDown alert and every downstream symptom alert (pod restarts, service unavailable, high latency) all fire in parallel instead of NodeDown suppressing the rest. Alertmanager cannot synthesize a root-cause summary message on its own — the realistic outcomes are a single batched notification containing many related alerts, or fewer notifications overall through inhibition.

Third, severity labels are inconsistent or unused. If every alert rule uses severity: warning because nobody agreed on a taxonomy, the routing tree cannot distinguish "wake someone up" from "check this during business hours." Watch out for teams that add severity labels late and forget to backfill existing rules — half the alerts route correctly and half don't, which is worse than having no severity labels at all because it looks fixed when it isn't.

A fourth, quieter cause: repeat_interval left at its default. If it is shorter than the actual mean-time-to-resolve for a given alert class, the same unresolved incident re-notifies every hour and gets mentally filed as spam. Note that a child route's repeat_interval can be overridden independently, but it still inherits group_interval from its parent unless that is also set explicitly.

The fix (with code)

The fix has two parts: tighten grouping and timing, then add inhibition so related alerts collapse into fewer notifications. The example below is a partial config — a working file also needs global and fully defined receivers for every name referenced in route.

route:
  receiver: default-slack
  group_by: ['cluster', 'namespace', 'instance']  # correlate by scope, not by rule name
  group_wait: 30s        # wait to batch near-simultaneous firings
  group_interval: 5m     # minimum gap between updates to an existing group
  repeat_interval: 4h    # avoid re-paging for the same unresolved issue every hour
  routes:
    - matchers:
        - severity="critical"
      receiver: pagerduty-oncall
      group_wait: 10s
      repeat_interval: 1h   # overrides parent; still inherits group_interval: 5m
      continue: false       # default behavior for a matched route; listed for clarity
    - matchers:
        - severity="warning"
      receiver: slack-warnings
      repeat_interval: 12h
    - matchers:
        - alertname="Watchdog"   # always-firing heartbeat rule
      receiver: null-receiver    # confirms the pipeline is alive without paging anyone

receivers:
  - name: default-slack
    slack_configs:
      - api_url: "https://hooks.slack.com/services/PLACEHOLDER"
        channel: "#alerts"
  - name: pagerduty-oncall
    pagerduty_configs:
      - service_key: "PLACEHOLDER_KEY"
  - name: slack-warnings
    slack_configs:
      - api_url: "https://hooks.slack.com/services/PLACEHOLDER"
        channel: "#alerts-warnings"
  - name: null-receiver   # no notifier configs: matched alerts are discarded

Grouping by scope instead of by alertname means DiskPressure and MemoryPressure notifications for the same node land in one message body rather than two, but different alert types on unrelated nodes stay in separate groups because cluster/namespace/instance differ.

Next, add inhibition so a node-level failure suppresses the symptom alerts it causes. This is the piece most teams skip, and label overlap between source and target matters more than it looks:

inhibit_rules:
  - source_matchers:
      - alertname="NodeDown"
    target_matchers:
      - severity="warning"
    equal: ['node']   # node exporter alerts share 'node', not 'instance', with symptom alerts — verify against your label schema

  - source_matchers:
      - alertname="KubeAPIDown"
    target_matchers:
      - alertname=~"KubePodCrashLooping|KubeDeploymentReplicasMismatch"
    equal: ['cluster']  # confirm these exact rule names exist in your kube-prometheus-stack ruleset

Verify with amtool config routes test --config.file=alertmanager.yml severity=critical against representative label sets before rolling this into production, and use amtool check-config alertmanager.yml to catch syntax and reference errors before reload — an undefined receiver name or a matcher targeting a label that doesn't exist in your rules will fail validation, not fail silently. Both source_matchers/target_matchers and the unified matchers list require Alertmanager 0.22 or later; earlier versions only understand source_match/target_match and match/match_re, which still work in current releases but are discouraged in favor of the newer syntax. The official Alertmanager configuration reference documents every matcher and timing field for the version in use.

Prevention checklist

Fixing one bad routing tree solves today's fatigue; a checklist keeps it from creeping back in over the next quarter of new alert rules.

Standardize a severity taxonomy before writing new rules. Two or three tiers (critical, warning, info) is enough; document what each means for response time.
Group by scope, and decide deliberately whether alertname belongs in group_by. Keeping it in keeps notifications per-rule; dropping it merges different alert types sharing a scope into one message.
Add inhibition rules whenever a new "root cause" alert is introduced. Check that equal lists labels actually shared between source and target alerts — a mismatch means the rule silently suppresses nothing.
Set repeat_interval per severity tier, not globally. Critical alerts can repeat hourly; warnings should not. Remember child routes inherit unset fields like group_interval from the parent.
Run a quarterly noise audit. Pull notification counts per alertname from your paging tool and question anything firing repeatedly within a short window — that is usually a threshold problem, not a routing problem, but it belongs on the same review.
Test config changes with amtool before reload, always passing --config.file and sample labels, and keep a staging Alertmanager instance if the routing tree is complex enough to warrant it.
Watch out for silences that outlive their incident. A silence created during a maintenance window and never removed quietly disables alerting for that scope indefinitely — audit active silences alongside the noise audit.

None of this requires new tooling, only a deliberate pass over configuration that most teams write once and never revisit. For more on pairing this with dashboard-side alert visibility, see the Prometheus and Grafana setup notes on kuryzhev.cloud.

Terraform Drift Detection in CI: Building a Remediation Pipeline

Oleksandr Kuryzhev — Mon, 14 Sep 2026 07:08:31 +0000

Originally published on kuryzhev.cloud

The scenario

Someone opened the AWS console during an incident and added an inbound rule to a security group to unblock a debugging session. The incident closes, the ticket gets marked resolved, and nobody touches Terraform. Three weeks later a scheduled terraform apply runs, sees that the security group no longer matches the `.tf` config, and either reverts the fix silently or fails with a diff nobody expected. This is terraform drift detection's core problem: state stops reflecting reality, and the gap is invisible until something breaks.

Drift comes from more than console fixes. Another team's automation might tag resources directly through the AWS SDK. Auto-scaling groups rewrite instance counts. Cloud providers backfill default attributes — a default security group rule, a provider-assigned ARN suffix, KMS key rotation metadata — that Terraform never wrote but will happily "fix" on the next apply. Each of these is a small divergence between the state file and the actual infrastructure, and none of them show up until a plan or an incident forces the question.

The cost of unmanaged drift is twofold. First, the next apply can revert an intentional emergency change because Terraform has no way to know it was intentional — it only knows the config doesn't match. Second, plans become noisy and untrustworthy: engineers start ignoring diffs because "there's always something," which is exactly the habit that lets a real, dangerous change slip through unnoticed. The goal here is a scheduled CI job that runs a plan, classifies what it finds, opens a PR or issue for human review, and — only for narrowly scoped, low-risk cases — remediates automatically.

Prerequisites

A few things need to be true before wiring drift detection into CI, or the scheduled job will produce false positives or, worse, false confidence.

Remote state with locking is non-negotiable. Terraform Cloud/HCP, GCS with its native locking, or an S3 backend using native locking (use_lockfile = true, available from Terraform 1.10+) all work. Local state gives no locking and no shared visibility across a team, so it's a bad fit for scheduled runs that might overlap a human-triggered plan or apply. Locking exists specifically to stop a collision from corrupting anything — a lock collision just fails the operation with an error, and terraform plan never writes to the state file in the first place; only apply does. See the Terraform S3 backend documentation for the locking configuration; the older dynamodb_table argument is deprecated as of Terraform 1.11 in favor of native S3 locking.

Pin the Terraform CLI version with required_version and commit .terraform.lock.hcl. If the scheduled CI run resolves a newer provider version than the one used for the last manual plan, the diff will include provider-attribute changes that have nothing to do with real infrastructure drift — a classic false positive that erodes trust in the whole pipeline.

Finally, the CI platform needs scheduled triggers — GitHub Actions schedule:, GitLab CI pipeline schedules, or equivalent — plus credentials for opening PRs or issues, and cloud credentials scoped to read/plan-only for the detection run itself. Even a plan-only role needs write access to the lock: dynamodb:PutItem/dynamodb:DeleteItem on the lock table for the legacy backend, or s3:PutObject/s3:DeleteObject on the lockfile object for native S3 locking. A role with only s3:GetObject and read-only DynamoDB permissions will fail on every single run with a lock-acquisition error, not a clean read-only plan. Apply-capable credentials should be a separate role, gated behind approval, introduced later in the pipeline.

Step 1 — Run scheduled drift detection

The detection job is terraform plan -refresh-only -detailed-exitcode, not a plain plan. A plain plan's exit code 2 fires for both real drift and any unapplied change to the `.tf` config — two very different situations, and lumping them together defeats the point of a drift report. The -refresh-only flag isolates the first case: it refreshes state against real infrastructure and reports only what changed outside Terraform, without folding in pending config edits.

Documented Terraform CLI behavior for -detailed-exitcode returns 0 for no changes, 1 for an error, and 2 when changes are found. That exit code is the branching point for everything downstream — but the workflow has to capture it carefully. GitHub Actions runs step scripts with bash -e by default, and exit code 2 will abort the script before a naive echo "exitcode=$?" on the next line ever runs, leaving every if: condition downstream silently false.

# .github/workflows/drift-detection.yml
name: Terraform Drift Detection

on:
  schedule:
    - cron: "0 6 * * *"   # daily at 06:00 UTC, tune per workspace criticality
  workflow_dispatch: {}     # allow manual trigger for verification

permissions:
  id-token: write        # required for OIDC role assumption
  contents: write        # required to push the remediation branch
  pull-requests: write   # needed to open the remediation PR

jobs:
  detect-drift:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5

      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets.DRIFT_READONLY_ROLE_ARN }}
          aws-region: us-east-1

      - uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: "1.13.1"   # pin to match lockfile expectations

      - name: Init
        run: terraform init -input=false

      # -refresh-only isolates real drift from unapplied config changes;
      # -detailed-exitcode: 0=no drift, 1=error, 2=drift found
      - name: Plan and capture exit code
        id: plan
        run: |
          set +e
          terraform plan -refresh-only -detailed-exitcode -out=drift.tfplan
          code=$?
          echo "exitcode=$code" >> "$GITHUB_OUTPUT"
          exit 0

      - name: Fail loudly on a real plan error
        if: steps.plan.outputs.exitcode == '1'
        run: |
          echo "terraform plan failed with exit code 1 — this is not drift, it's a broken plan"
          exit 1

      - name: Export plan as JSON for parsing
        if: steps.plan.outputs.exitcode == '2'
        run: terraform show -json drift.tfplan > drift.json

      - name: Persist the plan for later review and apply
        if: steps.plan.outputs.exitcode == '2'
        uses: actions/upload-artifact@v4
        with:
          name: drift-plan-${{ github.run_id }}
          path: |
            drift.tfplan
            drift.json
          retention-days: 14

      - name: Open remediation PR if drift detected
        if: steps.plan.outputs.exitcode == '2'
        run: ./scripts/open-drift-pr.sh drift.json drift.tfplan

Run this with a read-only or plan-only IAM role assumed via OIDC through aws-actions/configure-aws-credentials, not the same admin credentials used for apply. A common mistake in early drift-detection setups is reusing broad, apply-capable credentials for the scheduled job — if that role leaks, or the workflow gets misconfigured, a "detection" run could mutate production. terraform show -json against the saved plan gives structured, parseable output instead of scraping plan text with regex, and uploading the plan as a build artifact means the exact plan that was inspected is still around days later, instead of existing only inside one ephemeral runner that's already gone.

Step 2 — Classify and report drift

A raw plan diff dumped into a Slack channel is not actionable — it gets scrolled past. Running terraform show -json against a refresh-only plan produces a resource_drift array specifically for changes found during the refresh; the resource_changes array in the same output mixes those in with changes driven by edits to the `.tf` config, which is not what this pipeline is trying to surface. Read resource_drift, and each entry's change.actions field (create, update, delete, or delete, create for a replacement) is enough to build a structured report without parsing plan text.

Not every non-empty refresh-only plan is drift that needs fixing. Provider-computed or eventually-consistent attributes — certain AWS timestamps, KMS key rotation metadata, some load balancer attributes — generate diffs that are noise, not signal. Filtering these out, or scoping them with ignore_changes, keeps the report trustworthy. If every scheduled run reports "drift" on the same cosmetic field, the team will start ignoring the report entirely, which defeats the purpose.

The report itself should separate cosmetic changes (tags, descriptions) from structural ones (security group rules, IAM policies, instance types, anything touching data stores or networking). That severity tag is what determines whether the change goes through auto-remediation or requires a human to look at it.

# Example classified drift report attached to the remediation PR

Drift summary — workspace: prod-network (2026-01-14T06:00Z)

  aws_security_group.app[0]        UPDATE   severity: structural
    ingress: [+] 0.0.0.0/0:22 (added outside Terraform)

  aws_instance.web[2]              UPDATE   severity: cosmetic
    tags.CostCenter: "eng" -> "eng-platform"

  aws_s3_bucket.logs                CREATE   severity: structural
    (resource deleted outside Terraform — plan wants to recreate it)

Decision:
  - structural changes -> require manual review + approval before apply
  - cosmetic-only changes -> eligible for auto-merge/auto-apply path
  - CREATE drift on a "missing" resource -> confirm intent before applying (recreation risk)

Posting this as a PR comment or a filed issue, rather than raw plan output to chat, gives owners a durable, auditable trail tied to version control — chat messages scroll away, PRs don't.

Step 3 — Remediate via a gated pipeline

The fix for drift should go through the same review path as any other infrastructure change, not another console click. There are two distinct outcomes here, and they need different plan types — collapsing them into one "apply the drift plan" step is how remediation pipelines quietly do the wrong thing.

If the drift should be reverted, applying the refresh-only plan from Step 1 won't do it: terraform apply against a refresh-only plan only updates the state file to match what's actually running, it does not touch real infrastructure. See the refresh-only mode documentation for the exact behavior. Reverting requires a standard terraform plan -out=revert.tfplan run against the now-refreshed state, which computes the actual create/update/destroy actions needed to bring infrastructure back in line with the `.tf` config. Upload that plan with actions/upload-artifact, the same way the detection job does, so the apply job triggered from the remediation PR downloads and applies the exact plan a human reviewed, instead of re-running terraform plan at apply time and reopening a race against whatever else might change the account in the meantime.

If the drift should be kept, don't apply anything — update the `.tf` config to match reality (Step 4 covers this) and let a follow-up plan confirm the diff is gone.

For narrowly scoped, pre-approved categories — tag-only diffs are the usual example — an auto-merge or auto-apply path for the revert plan can be allowed, but only with a strict resource-type allowlist. Anything touching IAM, networking, or a data store should require a human, regardless of how small the diff looks. Gate the apply job behind a protected environment or required reviewers in GitHub, or the equivalent manual gate in whatever CI platform is running this.

Step 4 — Handle unmanageable or intentional drift

Not all drift should be reverted. If the manual security group fix from the incident should stay, the correct move is updating the `.tf` config to match reality and confirming terraform plan comes back clean — not applying over the fix.

Watch out for resources that will always show noise from provider-managed defaults. The `lifecycle { ignore_changes = [...] }` block suppresses drift reporting for the listed attributes, but scope it to specific fields rather than `all`. Ignoring everything on a resource hides real future drift along with the noise, and that resource effectively drops out of drift detection entirely.

Watch out for drift caused by a resource deleted outside Terraform — the next plan shows a `create` action, and reapplying it recreates something that may have been intentionally decommissioned. Run terraform state list and terraform state show <address> to confirm whether the resource is still tracked before deciding whether to recreate, import, or remove it from state. Any `terraform state rm` or import action changes the state file and should go through the same PR review as code, for auditability.

Verify and test

Before trusting the pipeline, induce drift deliberately in a sandbox environment — change a security group rule via CLI or console — and confirm the scheduled job flags it, exits with code 2, and posts the report with the expected resource address and severity tag.

Check the remediation PR's apply step against the plan summary counts (add/change/destroy) captured when the revert plan was generated, to confirm it changes exactly the drifted resource with no unrelated diffs. A revert plan that touches more resources than expected usually points to state that shifted between plan and apply, or to a workspace mix-up in multi-workspace setups — run detection per workspace rather than as one aggregate plan, since an aggregate plan can mask which specific environment actually drifted.

Finally, confirm the read-only detection role genuinely cannot apply. Attempt an intentional apply from the detection credentials and expect an access-denied error. Catching a privilege-scoping mistake in a test run is far cheaper than discovering it because a scheduled job silently mutated production.

Drift detection only pays off if remediation flows through the same reviewed, versioned pipeline as every other infrastructure change — otherwise a team just trades console drift for a new kind of pipeline drift, where auto-applies happen with no one watching. Start with detection-and-report-only for a few weeks, let the team see what kinds of drift actually occur in the environment, and only enable an auto-apply path for the narrow, low-blast-radius categories that show up repeatedly and safely. More CI and infrastructure patterns like this one are covered under DevOps_DayS on kuryzhev.cloud.

nginx-proxy acme-companion: why certs never get issued

Oleksandr Kuryzhev — Sun, 13 Sep 2026 07:05:37 +0000

Originally published on kuryzhev.cloud

Every container reports healthy, docker ps shows no restarts, and the browser still throws a certificate warning. This is the recurring shape of nginx-proxy acme-companion troubleshooting: nothing has crashed, and no certificate has issued.

Symptoms

The browser shows "Not Secure" or ERR_CERT_AUTHORITY_INVALID, while the acme-companion container itself sits there running with no exit code. A running container is not evidence of a functioning ACME client — it only proves the process hasn't died.

docker logs <acme-companion> usually shows one of the real acme.sh/acme-companion failure strings: "Verify error", "Invalid status", or a line like "Creating/renewal ... certificate for '<domain>' failed". The domain resolves fine in a browser and plain HTTP works, but the ACME validation path specifically doesn't.

Port 443 is often still serving nginx-proxy's default self-signed certificate. That default is expected right after first startup — it's only a symptom once it's still there hours later. Worth checking separately: if /etc/nginx/certs/default.crt is missing entirely (verify with docker exec <nginx-proxy> ls -la /etc/nginx/certs/default.crt), port 443 refuses the connection outright instead of serving an untrusted cert. That's a different failure signature pointing at a broken nginx-proxy startup, not a stalled acme-companion.

Together — running containers, real ACME error strings in the logs, and a stuck self-signed cert — these point at a broken link in the challenge or volume chain, not a Let's Encrypt outage.

Root cause

nginx-proxy and acme-companion aren't one service; they're three moving parts coordinating over shared state — nginx-proxy (docker-gen plus nginx), acme-companion (the ACME client), and the Docker socket that ties them together via container labels like VIRTUAL_HOST and LETSENCRYPT_HOST.

The HTTP-01 flow itself is easy to get backwards: acme-companion writes the challenge token into the shared html volume, and nginx-proxy — not acme-companion — serves it on port 80. acme-companion has no listener of its own; it never receives an inbound HTTP request. If port 80 is blocked, redirected, or intercepted upstream, nginx-proxy never gets asked for the file, and the failure happens on nginx-proxy's side of the handshake even though the error surfaces in acme-companion's logs.

Three failure modes show up repeatedly in nginx-proxy's and acme-companion's own issue trackers: the challenge path never reaching nginx-proxy at all, containers that mount volumes under the same name but not actually shared, and env vars like LETSENCRYPT_EMAIL or LETSENCRYPT_HOST missing or mismatched against VIRTUAL_HOST.

A fourth cause is easy to miss: acme-companion tries to autodetect which container is nginx-proxy over the Docker socket, and that autodetection isn't guaranteed, especially with custom container names or more than one nginx-proxy on the host. Set the com.github.nginx-proxy.nginx-proxy=true label on the nginx-proxy container, or the NGINX_PROXY_CONTAINER env var on acme-companion pointing at its container name, and stop relying on the guess.

Watch out: env vars cannot be changed on a running container — you have to recreate it, and that recreation is itself a Docker event docker-gen already watches for. If config still doesn't re-render after a recreate, the real problem is usually a dead docker-gen watcher or a stale socket mount, not a timing issue with variables. Restarting nginx-proxy resets the watcher.

Fix #1 — Verify the ACME HTTP-01 path is actually reachable

Before touching any container config, rule out network and DNS. Confirm the domain's A/AAAA record points at the host's public IP — Let's Encrypt validates from the internet, not from inside a VPC or local network.

Check whether anything upstream blocks port 80: a Cloudflare orange-cloud proxy, a cloud load balancer, or a security group rule. HTTP-01 needs plain HTTP on port 80, even on a site that forces HTTPS everywhere else. Gotcha: an aggressive "redirect HTTP to HTTPS" rule at the edge — not inside nginx-proxy — silently swallows every challenge request before it reaches the container.

curl -I http://yourdomain/.well-known/acme-challenge/anything
# expect: HTTP/1.1 404, with a "Server: nginx" header

# a 404 alone can come from the proxied app instead of nginx-proxy;
# confirm by dropping a real file into the shared html volume and curling it directly
docker exec <nginx-proxy> sh -c 'mkdir -p /usr/share/nginx/html/.well-known/acme-challenge && echo ok > /usr/share/nginx/html/.well-known/acme-challenge/probe'
curl http://yourdomain/.well-known/acme-challenge/probe
# expect: "ok"

If the probe file comes back as "ok", nginx-proxy is serving the challenge path correctly and the html volume is genuinely shared. A timeout, or a response from something that isn't nginx, means the network layer is the problem — fix that before touching any compose file.

Fix #2 — Fix shared volumes and required env vars

Containers that look configured but don't actually share state come up repeatedly in nginx-proxy acme-companion troubleshooting threads. Confirm all three containers mount the same named volumes: certs, html, and vhost (mounted at /etc/nginx/vhost.d — the volume name and the mount path aren't the same string, which trips up anyone grepping compose files for "vhost.d"). acme-companion additionally mounts acme plus the Docker socket, read-only.

Set VIRTUAL_HOST and LETSENCRYPT_HOST identically on the app container. For multiple domains, use comma-separated values, but every value in LETSENCRYPT_HOST must also appear in VIRTUAL_HOST — a domain listed only under LETSENCRYPT_HOST is a recurring copy-paste mistake. Also set LETSENCRYPT_EMAIL (or DEFAULT_EMAIL globally on acme-companion). A missing email doesn't fail issuance outright, but it's the only way to recover the ACME account or hear from Let's Encrypt about a required action — Let's Encrypt discontinued expiration notification emails in June 2025, so don't rely on this field for renewal alerting; use an independent expiry check instead.

services:
  nginx-proxy:
    image: nginxproxy/nginx-proxy:1.7
    restart: unless-stopped
    ports:
      - "80:80"     # required for ACME HTTP-01 challenge
      - "443:443"
    labels:
      - "com.github.nginx-proxy.nginx-proxy=true"
    volumes:
      - certs:/etc/nginx/certs:ro       # shared: read-only here, RW on acme-companion
      - vhost:/etc/nginx/vhost.d
      - html:/usr/share/nginx/html
      - /var/run/docker.sock:/tmp/docker.sock:ro

  acme-companion:
    image: nginxproxy/acme-companion:2.4
    restart: unless-stopped
    depends_on:
      - nginx-proxy
    volumes:
      - certs:/etc/nginx/certs:rw       # must be RW here
      - vhost:/etc/nginx/vhost.d
      - html:/usr/share/nginx/html
      - acme:/etc/acme.sh
      - /var/run/docker.sock:/var/run/docker.sock:ro
    environment:
      DEFAULT_EMAIL: ops@example.com
      NGINX_PROXY_CONTAINER: nginx-proxy   # skip autodetection instead of the label above

  app:
    image: your-app:latest
    restart: unless-stopped
    environment:
      VIRTUAL_HOST: example.com,www.example.com
      LETSENCRYPT_HOST: example.com,www.example.com
      LETSENCRYPT_EMAIL: ops@example.com

volumes:
  certs:
  vhost:
  html:
  acme:

Gotcha: older tutorial snippets sometimes define certs as visually identical but separately declared volumes per service. It breaks the handshake with no obvious error — container names differ, volume names look right, but the data never actually crosses between services.

Fix #3 — Diagnose CA/rate-limit issues without burning production quota

Sometimes the challenge passes and no certificate appears anyway. That's usually a CA-side rejection, not a container bug.

# Diagnostic sequence — run in this order

# 1. Is acme-companion seeing your container's env vars?
docker logs <acme-companion-container> --tail 100 | grep -i example.com

# 2. Confirm volumes are actually shared, not per-container copies
docker inspect <nginx-proxy-container> --format '{{range .Mounts}}{{.Name}} -> {{.Destination}}{{"\n"}}{{end}}'
docker inspect <acme-companion-container> --format '{{range .Mounts}}{{.Name}} -> {{.Destination}}{{"\n"}}{{end}}'
# both should list the SAME volume names for certs/vhost/html

# 3. Check for a rate-limit rejection specifically
docker logs <acme-companion-container> 2>&1 | grep -iE "rate limit|too many certificates|ratelimited"

# 4. Confirm the served cert matches the domain and isn't the default
openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null \
  | openssl x509 -noout -issuer -dates

Historical figures on the Let's Encrypt rate-limit page put failed validations at roughly 5 per account/hostname per hour and 50 issued certificates per registered domain per week — but Let's Encrypt revised its rate-limit model in 2025 around rolling New Order windows, so treat any specific number here as dated and check the live page before planning around it. That weekly limit is a rolling window, not a flat lockout: hitting it mid-incident doesn't force a fixed week-long wait, it blocks new issuance until older certificates in the window age out, which can still be slow but isn't absolute.

Switch acme-companion to Let's Encrypt's staging directory before debugging repeatedly against production: set ACME_CA_URI to https://acme-staging-v02.api.letsencrypt.org/directory. Removing that override alone doesn't finish the job — acme-companion caches the staging cert/key under the certs volume and the staging ACME account under the acme volume. Delete the staging cert files for that domain from certs and the account directory under acme, then restart acme-companion so it registers a fresh production account and reissues from scratch. Skip that cleanup and the untrusted staging cert just keeps getting served, with no reissue triggered.

If logs show a successful challenge but nginx never reloads with the new cert, check docker-gen's own logs next. A syntax error in a custom nginx snippet under vhost.d/<domain> can block the template render entirely, even after a valid certificate has already landed in the certs volume.

Prevention

Add a monitoring check for certificate expiry instead of trusting acme-companion's internal renewal loop as the only signal. A Prometheus blackbox exporter probe, or a cron job running openssl x509 -checkend, catches a silent renewal failure weeks before it becomes an outage — especially now that Let's Encrypt no longer sends expiry emails.

Pin nginxproxy/nginx-proxy and nginxproxy/acme-companion to explicit tags, as in the compose example above, instead of latest. An untested major-version bump landing exactly when a cert is due for renewal is a bad time to discover a breaking change.

The older jrcs/letsencrypt-nginx-proxy-companion repository, still referenced in some tutorials, has been archived by its maintainers — check the repository's own deprecation notice on GitHub before using it for anything new.

Document the volume and env-var contract directly in the compose file with comments, so the next edit doesn't silently break the certs/html/vhost chain. Restrict the Docker socket mount on acme-companion to read-only, and never expose that socket to less-trusted app containers; consult the Docker security documentation for the tradeoffs of socket-mounting in multi-container setups.

For new deployments, weigh whether Traefik or Caddy — both with built-in ACME support and no companion-container choreography — sidesteps this exact class of failure. nginx-proxy's split-container design is legacy-compatible and well understood, but it has more moving parts and shared state to misconfigure than a single binary managing its own certificate lifecycle. If the current stack already works and the volume contract is documented, there's no urgent reason to migrate. If nginx-proxy acme-companion troubleshooting keeps recurring across routine compose edits, that recurrence is the signal worth acting on. For related Docker networking and TLS patterns, see the DevOps_DayS archive.

Nginx TLS Hardening: Manual Directives vs Delegated Edge Policy

Oleksandr Kuryzhev — Sat, 12 Sep 2026 07:05:42 +0000

Originally published on kuryzhev.cloud

Nginx TLS hardening usually gets attention at the worst possible moment: a pentest report flags missing security headers, or a compliance auditor asks why SSLv3 is still negotiable on a box nobody remembers deploying. The fix people reach for first is a header. The actual fix is deciding, once, who owns TLS and header policy — nginx itself, or a layer in front of it.

When this choice matters

The decision point shows up in a few recognizable situations: preparing for an SSL Labs–style scan ahead of a compliance audit, standing up an nginx reverse proxy in front of a growing list of backend services, or closing out a pentest finding about missing Content-Security-Policy or Strict-Transport-Security headers.

None of these are one-time fixes. Cipher recommendations and header guidance change periodically rather than on any fixed schedule — the Mozilla SSL Configuration Generator and NIST's TLS guidance both get revised as weak algorithms are deprecated and new attack classes get published. A config that scored an "A" two years ago can quietly degrade as TLS 1.0/1.1 deprecation guidance tightens or as CAs change what they support. Nobody re-scans a working proxy until something breaks it.

This comparison is about where hardening policy lives — hand-written in nginx directives, or delegated to a shared generator/edge layer — not a line-by-line cipher suite recommendation. Cipher lists specifically should come from a maintained generator, not a static list copied from an old blog post, since TLS 1.3 ignores legacy ssl_ciphers syntax entirely and stale strings give a false sense of hardening.

Option A: Manual, hand-tuned nginx directives

This is the default approach for most teams: TLS settings and security headers written directly into server {} blocks, reviewed in pull requests, and deployed the same way as any other nginx change.

Pros: full control per virtual host, no external dependency or third-party trust boundary, works fine air-gapped or on-prem, and the entire policy is reviewable in git history — useful when an auditor asks "who approved this change and when."

Cons: drift is the real cost. Across ten or twenty server blocks, it's easy to forget the always flag on add_header, or to let one vhost fall behind when TLS guidance changes. Upkeep becomes a recurring chore rather than a one-time task.

The gotcha that catches people repeatedly: add_header directives do not merge across nested blocks. If a location block defines its own add_header, it silently discards every header set in the parent server block — nginx does not combine the lists. A team can set five hardened headers at the server level, add one location /api block with a single add_header for CORS, and lose the other four without any warning. The documented behavior is in the nginx headers module reference; verify with curl -I against the actual endpoint, not just the config file.

Option B: Delegated/automated hardening

The alternative is centralizing policy at a CDN, WAF, or API gateway layer, or generating a shared nginx snippet from a single source of truth and including it everywhere via infrastructure-as-code.

Pros: one policy, applied consistently across many services. Updating a cipher suite or rotating a header value happens in one place instead of N places. It can also offload TLS termination cost and CPU overhead from the application tier.

Cons: less per-service granularity — a service with unusual requirements has to fight the shared policy or get an exception. There's also real risk of duplicate or conflicting headers between the edge and nginx itself, and vendor-specific quirks in how headers get injected, rewritten, or stripped in transit.

Two gotchas matter here. First, when both the edge layer and nginx set the same header, the two mechanisms don't fail the same way. Duplicate or conflicting X-Frame-Options values are generally treated as invalid by the browser and the header gets dropped outright — you lose the protection silently, with no error anywhere. Content-Security-Policy behaves differently: duplicate CSP headers are combined, and the browser enforces the most restrictive intersection of the two policies, which can produce a working-but-unintended result that's hard to trace back to its source. Either way, this only shows up when you inspect the live response with curl or browser dev tools, not when you read the config. Second, "hardened at the edge" quietly becomes an assumption that the internal hop is fine too. If the CDN terminates TLS with a strong profile but the connection from reverse proxy to backend runs on an outdated internal certificate or a legacy cipher set, the public-facing scan looks great while the actual attack surface between proxy and app server stays wide open.

Decision matrix

Use these axes to decide instead of defaulting to whichever approach the team already knows:

Control granularity — manual wins if individual services genuinely need different policies (e.g., one legacy internal API can't yet drop an old cipher).
Maintenance overhead — delegated wins once you're updating the same directive in more than a handful of places for the same reason.
Audit reproducibility — IaC-applied shared config (Terraform or Ansible pushing the identical snippet everywhere) produces evidence auditors can trust faster than hand-edited per-host files, which is directly relevant for PCI-DSS or SOC 2 evidence collection.
Latency/cost impact — edge termination can reduce load on app-tier nginx, but adds a network hop and a vendor dependency to reason about during incidents.
Team size and service count — this is a heuristic, not a derived number. A single service or a small team can run manual directives safely if paired with automated scanning. Once a shared reverse proxy sits in front of several backend services — five is a reasonable trigger point for many teams, but the real signal is how often you're copy-pasting the same directive across vhosts, not the count itself — centralize via a shared include file or an edge policy instead of duplicating configuration by hand.

Neither option removes the need for testing. Both fail the same way if nobody scans the live endpoint after a change.

Evidence-based recommendation

Treat hardening as code regardless of which option you pick. A hand-written directive set and a generated edge policy are both fine as long as they live in version control, get applied through CI/CD, and aren't edited directly on a running host. The example below is Option A, but centralized into a shared snippet to avoid the per-vhost drift that makes manual configs risky at scale.

# security-headers.conf — shared snippet included by every server{} block
# This file must NOT contain server{} or listen directives — add_header
# is invalid in nginx's main context and the config test will refuse to load it.

add_header X-Content-Type-Options "nosniff" always;
add_header Referrer-Policy "strict-origin-when-cross-origin" always;

# X-Frame-Options is superseded by the CSP frame-ancestors directive below.
# Keep it only for legacy browsers that don't support frame-ancestors —
# it is not the primary clickjacking control anymore.
add_header X-Frame-Options "DENY" always;

# Ramp HSTS gradually: start short and without includeSubDomains. A 5-minute
# max-age combined with includeSubDomains fails PCI-DSS/SOC 2 scan checks and
# SSL Labs' HSTS grading outright. Raise max-age over weeks, and add
# includeSubDomains only once every subdomain is confirmed HTTPS-only —
# HSTS preload submission is effectively irreversible for months.
add_header Strict-Transport-Security "max-age=300" always;

# CSP: keep in Report-Only in staging until it stops breaking things,
# then enforce on a known date. CSP mitigates XSS but doesn't close the gap
# on its own — treat it as a defense-in-depth layer alongside output
# encoding and input sanitization, not a replacement for either.
add_header Content-Security-Policy "default-src 'self'; object-src 'none'; frame-ancestors 'none'" always;

# backend.conf — one upstream and one server block per hardened virtual host

upstream backend_upstream {
    server 10.0.1.10:8443;
    server 10.0.1.11:8443;
}

server {
    listen 443 ssl;
    server_name example.internal;

    ssl_certificate     /etc/nginx/ssl/fullchain.pem;
    ssl_certificate_key /etc/nginx/ssl/privkey.pem;

    # Modern-only protocol set — TLS 1.0/1.1 should not be offered in a 2026 baseline
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers off;   # irrelevant for TLS1.3; harmless for 1.2 fallback

    # Pull the current string from https://ssl-config.mozilla.org/ — nginx's
    # built-in default (HIGH:!aNULL:!MD5) still allows non-PFS and CBC-SHA1
    # suites on TLS 1.2 and will fail a strict scan without this line.
    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384;

    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 1d;
    ssl_session_tickets off;         # avoid stale ticket key reuse across restarts/instances

    # OCSP stapling still helps on certs from CAs that publish OCSP responses,
    # but Let's Encrypt shut down its OCSP responder in 2025, and CA/Browser
    # Forum policy is moving away from OCSP fleet-wide. Confirm your issuer
    # still supports it before treating this as a required control.
    ssl_stapling on;
    ssl_stapling_verify on;
    ssl_trusted_certificate /etc/nginx/ssl/chain.pem;
    resolver 1.1.1.1 8.8.8.8 valid=300s ipv6=off;
    resolver_timeout 5s;

    include /etc/nginx/snippets/security-headers.conf;

    location / {
        proxy_pass https://backend_upstream;   # hardened hop, not plain http

        # proxy_ssl_verify alone has no CA store to validate against — pair
        # it with proxy_ssl_trusted_certificate and proxy_ssl_server_name on,
        # or the handshake to the backend fails.
        proxy_ssl_verify on;
        proxy_ssl_trusted_certificate /etc/nginx/ssl/internal-ca.pem;
        proxy_ssl_server_name on;
    }
}

Whichever architecture is chosen, validate every deploy against staging with an automated scanner instead of eyeballing the config file. Config review catches typos; it does not catch a location block silently discarding parent headers, or a stapling responder that's unreachable from the current network.

# Validation loop — run in CI against staging before merging nginx changes

# 1. Confirm headers are actually served, not stripped by a location override
curl -sI https://staging.example.com | grep -Ei \
  'strict-transport-security|content-security-policy|x-frame-options|x-content-type-options'

# 2. Grade the TLS handshake and cipher/protocol support
testssl.sh --protocols -E https://staging.example.com

# Expected outcome checklist:
# [ ] TLSv1.0 / TLSv1.1   -> NOT offered
# [ ] TLSv1.3             -> offered, preferred
# [ ] HSTS                -> present, max-age matches current rollout stage
# [ ] CSP                 -> enforced (not Report-Only) past the rollout deadline
# [ ] OCSP stapling       -> conditional check: "stapling supported" only applies
#                             if the issuing CA still runs an OCSP responder;
#                             a Let's Encrypt cert issued post-2025 will show
#                             "not supported" and that's expected, not a failure
# [ ] Duplicate headers   -> none (check edge layer output separately if one is in front)

Three concrete recommendations follow from this: first, run testssl.sh or sslyze against staging on every nginx change, treating the TLS and header config as testable code rather than something reviewed by eye. Second, apply the low-risk headers — X-Content-Type-Options: nosniff and Referrer-Policy: strict-origin-when-cross-origin — everywhere regardless of which architecture you pick; the risk is low but not zero, since a stricter Referrer-Policy value can break referrer-dependent analytics or OAuth-adjacent redirect flows that expect the full referrer, so check those paths before flipping the value fleet-wide. Third, and most often skipped: verify the internal hop from proxy to backend is hardened too, because an edge layer with a perfect SSL Labs grade says nothing about the connection behind it.

For teams running this across a Kubernetes ingress fleet rather than standalone nginx hosts, the same centralization argument applies at the ingress-controller layer — see the related coverage on rolling out pod security standards on kuryzhev.cloud for how policy-as-code decisions play out at that layer.

Nginx TLS hardening is not a header you paste once and forget. It's an architectural decision about where policy lives, how it's tested, and who's responsible when the next TLS deprecation notice lands. Pick manual directives for small, tightly controlled fleets paired with CI-driven scanning; pick a delegated edge policy once per-host drift becomes the bigger risk than losing per-service granularity. Either way, the scanner output — not the config file — is the source of truth.

DEV Community: Oleksandr Kuryzhev

Why an LLM reviewer misses risky Terraform plan changes

Failure scenario

Why it happens

The fix (with code)

Prevention checklist

Related

An unbounded Loki label turned a logging bill into shock

Symptoms

Root cause

Fix #1: Find and quantify the offending label

Fix #2: Move high-cardinality fields out of labels

Fix #3: Set per-tenant limits so it can't happen silently again

Prevention

Related

When Cloudflare Rate Limit Locks Out Your Own Players

The scenario

Prerequisites

Step 1: Identify the actual counting key

Step 2: Separate authenticated and unauthenticated traffic

Step 3: Deploy in Log mode first, then promote

Step 4: Add an allowlist for known infrastructure

Verify and test

Related

SLO burn-rate alerts explained through a traffic spike

Failure scenario

Why it happens

The fix

Prevention checklist

Related

IRSA explained: how EKS pods borrow AWS permissions

What this actually does

How people use it wrong

The correct approach

Advanced patterns

Performance notes

Related

BGP routing basics for engineers tired of traffic drops

BGP does not know your traffic is dropping

Route flapping is usually a symptom, not the disease

AS_PATH length is not a reliability signal

Prefix hijacks are still mostly a config-hygiene problem

Cloud interconnects hide BGP but don't remove it

Communities are how you signal behavior across AS boundaries

Convergence time is the number that actually matters

Related

WordPress performance profiling with php-fpm status and slow logs

Why this checklist

The checklist

Commonly missed items

Automation ideas

Related

Cloudflare cache bypass mistakes on dynamic WordPress paths

Context

Common failure 1: bypass rules that only match by URL path, not by cookie

Common failure 2: caching REST API and admin-ajax responses meant to be dynamic

Common failure 3: relying on page-level exclusions instead of edge cache TTL discipline

Safer operating pattern

Related

Alertmanager Routing Fixes to Cut Prometheus Alert Fatigue

Failure scenario

Why it happens

The fix (with code)

Prevention checklist

Related

Terraform Drift Detection in CI: Building a Remediation Pipeline

The scenario

Prerequisites

Step 1 — Run scheduled drift detection

Step 2 — Classify and report drift

Step 3 — Remediate via a gated pipeline

Step 4 — Handle unmanageable or intentional drift

Verify and test

Related

nginx-proxy acme-companion: why certs never get issued

Symptoms

Root cause

Fix #1 — Verify the ACME HTTP-01 path is actually reachable

Fix #2 — Fix shared volumes and required env vars

Fix #3 — Diagnose CA/rate-limit issues without burning production quota