DEV Community: Oleksandr Kuryzhev

Kubernetes NetworkPolicy Checklist: Locking Down Namespace Isolation

Oleksandr Kuryzhev — Sun, 16 Aug 2026 07:01:49 +0000

Originally published on kuryzhev.cloud

Why this checklist

Last audit we ran, the pentest report had one line that made the whole platform team wince: "lateral movement possible between all namespaces." Our cluster had Calico installed, NetworkPolicy support was there, dashboards were green — and yet a compromised pod in a low-trust namespace could reach the billing service directly on port 5432. Nobody had actually written a kubernetes networkpolicy for namespace isolation. The CNI supported it. We just never used it.

Here's the part people miss: without any NetworkPolicy object applied, Kubernetes networking is default-allow. Every pod can reach every other pod, on any port, across every namespace, full stop. That's the platform's factory setting, and it stays that way until you deploy something that says otherwise.

NetworkPolicy is also namespace-scoped and purely additive. You can't "half isolate" a namespace by dropping in a few allow rules — if there's no deny-all baseline, your allow rules add nothing, because everything was already open. This is mistake #1 I see constantly: teams write "allow app-a to talk to app-b" and feel like they've done isolation work. They haven't. Without a deny-all first, that rule is decorative.

What actually forces teams to write this baseline in practice is one of three triggers: a multi-tenant cluster where different teams or customers share nodes, a compliance push (PCI-DSS, SOC2) where an auditor asks "how do you enforce network segmentation between workloads," or exactly what happened to us — a pentest finding lateral movement. If you're reading this before any of those hit you, you're ahead of most teams.

The checklist (numbered)

This is the order we now apply to every namespace before calling it "isolated." Skipping steps, or doing them out of order, is how you end up with policies that look right in kubectl get networkpolicy and do nothing in practice.

Default-deny-all ingress. Empty podSelector: {}, policyTypes: [Ingress], no rules. This blocks all inbound traffic to every pod in the namespace unless explicitly allowed. Verify: kubectl describe networkpolicy default-deny-all -n <ns> and confirm ingress is listed.
Default-deny-all egress. Same pattern with Egress. This is the one people forget — an empty policy without explicit policyTypes: [Egress] only blocks ingress in some Kubernetes versions, egress stays wide open. Explicit is mandatory here, not optional.
Explicit allow for DNS egress. CoreDNS lives in kube-system on 53/UDP and 53/TCP. Skip this and every pod breaks silently right after step 2 lands. Verify: kubectl exec <pod> -- nslookup kubernetes.default.
Explicit allow for intra-namespace pod-to-pod traffic, if your app actually needs it (microservices talking within the same namespace). Don't add this reflexively — plenty of namespaces don't need it.
Explicit allow for ingress controller / mesh sidecar traffic. Nginx-ingress or Traefik usually sits in its own namespace; without a rule allowing traffic from there, external requests get silently dropped after your baseline lands.
Allow monitoring scrape traffic. Prometheus pulls metrics, but "pull" still requires an ingress rule on the target side. Miss this and your dashboards go dark with zero error in the app logs.
Allow egress to specific external endpoints instead of 0.0.0.0/0. Scope to actual CIDRs — package registries, external APIs — not "port 443 to anywhere," which defeats the entire point of egress isolation.
Verify the CNI actually enforces policy. kube-proxy alone enforces nothing. Confirm with a live traffic test, not by reading YAML.

The reference bundle below covers items 1–4 and 6 for one namespace — apply per-namespace or template it via GitOps.

# baseline-networkpolicy.yaml
# Apply this bundle to EVERY namespace as the isolation baseline.
# kubectl apply -n <namespace> -f baseline-networkpolicy.yaml

---
# 1. Default deny all ingress and egress traffic
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: my-app-ns
spec:
  podSelector: {}          # applies to all pods in the namespace
  policyTypes:
    - Ingress
    - Egress
  # no ingress/egress rules defined = deny everything by default

---
# 2. Allow DNS egress — required after default-deny-egress, or pods can't resolve names
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-egress
  namespace: my-app-ns
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

---
# 3. Allow ingress from the shared ingress-controller namespace only
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-ingress-controller
  namespace: my-app-ns
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: my-app   # match your ACTUAL pod labels, not assumed ones
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: ingress-nginx
      ports:
        - protocol: TCP
          port: 8080

---
# 4. Allow monitoring namespace to scrape metrics endpoint
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-prometheus-scrape
  namespace: my-app-ns
spec:
  podSelector: {}
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: monitoring
      ports:
        - protocol: TCP
          port: 9090

Commonly missed items

Gotcha #1: DNS breaks after you enable egress deny. The symptom is almost always the same — dial tcp: lookup <service>: i/o timeout, and every app in the namespace starts failing at once. Nine times out of ten it's the missing DNS egress rule. We got bitten by this rolling out the baseline to a staging namespace at 4pm on a Friday — three services went red simultaneously and it took twenty minutes to connect the dots back to the policy we'd just applied.

Namespace labels not being set is the second big one. namespaceSelector rules depend on the automatically-added label kubernetes.io/metadata.name=<ns>, which has existed since Kubernetes 1.21. Older, manually-created namespaces sometimes had this label stripped by an overzealous label-cleanup script or a Helm chart that overwrites labels on upgrade. The policy applies fine, shows up in kubectl get networkpolicy, and matches absolutely nothing. It looks present. It does nothing.

Gotcha #2: hostNetwork pods and DaemonSets bypass NetworkPolicy entirely. kube-proxy, CNI agents, log shippers with hostNetwork: true — none of that traffic is subject to your policies. If your threat model includes "what if someone gets a shell on a node," your NetworkPolicy baseline gives you zero protection there. That's a host-level hardening problem, not a NetworkPolicy problem — don't let the checklist give you false confidence about it.

Mistake #2, which is sneaky: policies written against app: myapp when the Helm chart actually renders app.kubernetes.io/name: myapp. The selector never matches, the policy is a silent no-op, and it survives code review because nobody diffs pod labels against policy selectors. Always run kubectl get pods -n <ns> --show-labels before writing the selector, not after.

Last one: assuming "we have a CNI" means you're covered. Plain Flannel does not enforce NetworkPolicy at all — you need Calico, Cilium, or Weave Net layered on top. Check the official NetworkPolicy docs for the enforcement caveat; it's easy to skim past.

Automation ideas

Writing YAML by hand for every namespace doesn't scale, and it drifts the moment someone runs a quick kubectl edit to unblock a demo. Here's what's worked for us.

Lint in CI before merge. Cilium ships cilium policy validate, and for plain Kubernetes NetworkPolicy YAML, a simple OPA/Conftest rule catches the biggest offender: policies missing Egress from policyTypes. We run this as a required check on every PR touching networkpolicy/.

Bake the baseline into namespace creation. Instead of trusting humans to remember steps 1–3, use an admission webhook — Kyverno or OPA Gatekeeper — that auto-injects the default-deny and DNS-allow policies the moment a namespace is created. We moved to this after the third time someone spun up a namespace via kubectl create ns and forgot the whole checklist existed. If you're managing namespaces via GitOps already, check our DevOps notes on ArgoCD sync patterns for how we template the namespace bootstrap bundle alongside app manifests.

Below is the verification script we run in CI and on-demand after applying the baseline — it's saved us more debugging time than any dashboard.

# Quick verification script — run after applying the baseline
# to confirm enforcement actually works (not just "applied").

NS="my-app-ns"
POD=$(kubectl get pod -n "$NS" -l app.kubernetes.io/name=my-app -o jsonpath='{.items[0].metadata.name}')

echo "== Checking DNS resolution (should succeed) =="
kubectl exec -n "$NS" "$POD" -- nslookup kubernetes.default.svc.cluster.local

echo "== Checking cross-namespace access WITHOUT allow rule (should FAIL/timeout) =="
kubectl exec -n "$NS" "$POD" -- curl -m 2 -sv http://some-service.other-ns.svc.cluster.local:80 \
  || echo "Blocked as expected ✅"

echo "== Checking allowed ingress from ingress-nginx namespace =="
kubectl run tmp-curl --rm -i --tty --restart=Never \
  -n ingress-nginx --image=curlimages/curl -- \
  curl -m 2 -sv http://my-app.my-app-ns.svc.cluster.local:8080 \
  || echo "Unexpected block ❌ check namespaceSelector labels"

# Sanity check: does the namespace even have the required label?
kubectl get ns my-app-ns --show-labels | grep kubernetes.io/metadata.name \
  || echo "WARNING: namespace missing metadata.name label — namespaceSelector rules won't match"

# Expected output snippet on success:
# Server:    10.96.0.10
# Address:   10.96.0.10:53
# Name:   kubernetes.default.svc.cluster.local
# Address: 10.96.0.1

For drift detection, we run a scheduled job that diffs live cluster NetworkPolicies against the git-tracked baseline and alerts on manual edits — anyone editing a policy live via kubectl gets flagged within the hour. And a CI assertion job checks every namespace has both Ingress and Egress in policyTypes, failing the pipeline otherwise. It's a small amount of automation for a large amount of "we didn't actually notice namespace isolation had quietly regressed."

Worth watching: AdminNetworkPolicy (KEP-2091) is moving toward GA and adds cluster-level baseline enforcement that doesn't depend on every namespace remembering its own policy set. Check the Kubernetes docs on AdminNetworkPolicy if you're planning next year's roadmap — it's the direction this checklist is eventually heading.

None of this replaces reviewing your actual kubernetes networkpolicy namespace isolation posture manually at least once a quarter. Automation catches drift; it doesn't catch a threat model that's changed since you wrote the rules.

S3 Lifecycle Policy Mistakes That Quietly Inflate Your AWS Bill

Oleksandr Kuryzhev — Sat, 15 Aug 2026 07:01:54 +0000

Originally published on kuryzhev.cloud

Your versioned S3 bucket is quietly billing you for millions of noncurrent objects that no S3 lifecycle policy was ever written to catch. We found this out the hard way during a routine Cost Explorer review — a "small" logging bucket was sitting at 40TB of noncurrent versions nobody knew existed. If you're running compliance workloads, backups, or high-volume logging on S3, these are the rules I now treat as non-negotiable.

Split lifecycle rules by prefix and tag, never one rule per bucket

A single broad rule with no prefix filter is a blast radius waiting to happen. I've seen a rule meant for logs/ accidentally match logs-backup/ because someone typed a prefix without the trailing slash, and it deleted three weeks of audit data before anyone noticed. Filters can combine one prefix with a tag set using an and block, so use that instead of loose wildcards whenever object count matters.

Map rule IDs 1:1 to a naming convention — logs/, backups/daily/, compliance/retain/ — so anyone reading the Terraform state knows exactly what a rule targets without cross-referencing the console. Also know your ceiling: AWS allows up to 1,000 lifecycle rules per bucket. If you're running a multi-tenant bucket and approaching that limit, it's a sign you should split into separate buckets, not cram more and conditions into one rule.

Use Object Lock in Compliance mode for anything with a retention mandate

Object Lock and a plain S3 lifecycle policy solve different problems — expiration schedules storage cost, Object Lock enforces legal retention that nobody, including root, can override. Watch out: Object Lock can only be enabled at bucket creation time. There's no retrofit; if you forgot it on a bucket that's already holding compliance data, you're migrating objects to a new bucket, not flipping a setting.

Governance mode lets privileged users shorten or remove a retention period — fine for internal policy, useless for an audit. Compliance mode is the only one that's actually audit-safe, since no principal can undo it before the retention date. It also requires versioning to be enabled, and it interacts directly with expiration rules: a locked version blocks deletion even if an expiration rule fires against it, so a lifecycle rule silently no-ops on protected objects rather than erroring.

Transition logs aggressively, expire even more aggressively

Logs are high-volume and low-value per object — treat them differently from backups from day one. A tiering path like STANDARD → STANDARD_IA at 30 days → GLACIER_IR at 90 days → expire at 400 days works well for most CloudTrail and ALB access log volumes, but tune the numbers to your actual query patterns, not a copy-pasted default.

Gotcha: STANDARD_IA and Glacier/Glacier IR carry minimum billable storage durations — 30 days for IA, 90 days for Glacier — so transitioning objects that get deleted or re-uploaded before that window closes triggers a prorated early-deletion charge. For logs with a lifespan under 30 days, plain expiration is often cheaper than any transition at all. Model the cost before applying a tiering rule blindly, and give log types their own prefixes so a policy tuned for CloudTrail doesn't accidentally reshape retention for application logs with a different mandate.

# terraform: s3-lifecycle-governance.tf
# Requires: aws provider ~> 5.0, versioning + Object Lock pre-enabled on bucket

resource "aws_s3_bucket_versioning" "compliance" {
  bucket = aws_s3_bucket.compliance.id
  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_lifecycle_configuration" "compliance" {
  bucket = aws_s3_bucket.compliance.id

  # Rule 1: application logs — aggressive tiering + expiration
  rule {
    id     = "logs-tiering"
    status = "Enabled"

    filter {
      prefix = "logs/"
    }

    transition {
      days          = 30
      storage_class = "STANDARD_IA"
    }

    transition {
      days          = 90
      storage_class = "GLACIER_IR"
    }

    expiration {
      days = 400
    }

    # Orphaned multipart uploads from log shippers — always include this
    abort_incomplete_multipart_upload {
      days_after_initiation = 7
    }
  }

  # Rule 2: backups — longer retention, slower tiering
  rule {
    id     = "backups-retention"
    status = "Enabled"

    filter {
      and {
        prefix = "backups/"
        tags = {
          "retention" = "long"
        }
      }
    }

    transition {
      days          = 60
      storage_class = "GLACIER"
    }

    expiration {
      days = 1825 # 5 years, adjust to compliance mandate
    }
  }

  # Rule 3: noncurrent versions — closes the "silent growth" gap
  rule {
    id     = "noncurrent-cleanup"
    status = "Enabled"

    filter {
      prefix = "" # applies bucket-wide
    }

    noncurrent_version_transition {
      noncurrent_days = 30
      storage_class   = "GLACIER_IR"
    }

    noncurrent_version_expiration {
      noncurrent_days = 180
    }
  }
}

Watch noncurrent version growth — it's the silent cost and compliance trap

Every PUT against a versioned bucket creates a new version, and without an explicit noncurrent_version_expiration block, that storage grows unbounded and completely invisible in the console UI. This is exactly the mistake that caused our 40TB surprise — versioning had been enabled for durability years earlier, and nobody ever added the matching cleanup rule. It only surfaced when Cost Explorer flagged an unexplained S3 spend jump.

Add noncurrent_version_transition and noncurrent_version_expiration as their own block, separate from your current-version rules — see Rule 3 in the Terraform above. One more subtlety worth knowing: locked versions are correctly skipped by expiration, but unlocked noncurrent versions in the same bucket still get deleted on schedule, which can create gaps in your retained history if you assumed Object Lock covered everything.

Always add an abort-incomplete-multipart-upload rule

Orphaned multipart uploads are pure waste, and they're invisible unless you go looking. A backup tool or CI job that gets interrupted mid-upload leaves parts sitting in S3, billed at full STANDARD rate, indefinitely, with no expiration ever applied by default.

Set abort_incomplete_multipart_upload { days_after_initiation = 7 } on every single bucket — no exceptions, no "we'll add it later." Verify it's actually catching things periodically, since lifecycle evaluation isn't instant; it runs roughly once a day, so expect up to 24-48 hours of lag between a rule change and it taking effect.

# Validate what's actually configured before assuming it's correct
aws s3api get-bucket-lifecycle-configuration \
  --bucket compliance-prod-logs \
  --query 'Rules[].{ID:ID,Status:Status,Filter:Filter,Expiration:Expiration}' \
  --output table

# Check for orphaned multipart uploads lifecycle hasn't caught yet
aws s3api list-multipart-uploads --bucket compliance-prod-logs \
  --query 'Uploads[].{Key:Key,Initiated:Initiated}' --output table

# Example failure when a rule has a filter but no action — silently no-ops
# {
#   "Error": {
#     "Code": "InvalidArgument",
#     "Message": "Found rule without expiry or transition"
#   }
# }

# Audit noncurrent version bytes manually (or use S3 Storage Lens for this at scale)
aws s3api list-object-versions --bucket compliance-prod-logs \
  --query 'Versions[?IsLatest==`false`].[Key,Size]' --output text | \
  awk '{sum+=$2} END {print "Noncurrent bytes:", sum}'

Lifecycle rules are not access control — pair them with deny policies

An S3 lifecycle policy expires objects on a schedule; it does nothing to stop someone from deleting them manually five minutes earlier. For compliance buckets, add an explicit bucket policy Deny on s3:DeleteObject and s3:DeleteObjectVersion for every principal outside a documented break-glass role, and back that up with SCP-level guardrails at the OU level so a bucket policy edit can't quietly undo the protection.

Security note worth flagging: lifecycle-driven deletions don't appear in CloudTrail as a delete action by a principal — they show up as "Lifecycle Expiration" events instead. If your audit tooling only watches for DeleteObject API calls, expirations will slip through unnoticed. Plan for that by enabling S3 server access logs or an EventBridge rule for object expiration events if you need a complete audit trail. We cover the Terraform side of locking down state and access patterns like this in more depth in our Terraform security notes.

Validate before you ship — dry-run and inventory-audit every rule change

There's no native dry-run flag for S3 lifecycle configuration, so test scope changes on a cloned prefix in a non-production bucket first. Run aws s3api get-bucket-lifecycle-configuration before and after a change and diff the output — it's tedious, but it's the only way to confirm the JSON you shipped is the JSON that's actually active, since Terraform state and reality drift more often than people admit.

Gotcha that catches almost everyone eventually: lifecycle transitions and expirations evaluate against an object's creation date, not its last-modified date. Restoring an object from Glacier or re-uploading it resets "last accessed" but not creation date, which means it can transition again on a schedule you didn't expect. Enable S3 Inventory reports to track age and storage-class distribution over time, and if you're managing this across many accounts, S3 Storage Lens's free tier will surface lifecycle rule counts and noncurrent version bytes per bucket without writing a single custom script.

None of this is exotic — it's the same handful of edge cases repeating across every account we've audited. Get the S3 lifecycle policy basics right per bucket (prefix scoping, noncurrent cleanup, multipart abort, Object Lock where it's actually required), and the rest is just tuning tiering windows to match how your data is actually used. For the full lifecycle configuration reference, AWS's own S3 Object Lifecycle Management docs and the Object Lock guide are worth bookmarking — they're more precise on edge cases than most blog posts, including this one.

Ansible Zero-Downtime Deployment: Draining the Load Balancer Right

Oleksandr Kuryzhev — Fri, 14 Aug 2026 07:01:55 +0000

Originally published on kuryzhev.cloud

The scenario

We had a Flask API running on 8 app servers behind HAProxy, deployed with a straightforward Ansible playbook. It worked fine for a year — until traffic grew and someone finally noticed the pattern: every deploy, without exception, produced a 20-40 second window of 502s. Nobody had flagged it because deploys were scheduled at 3am specifically to dodge the pain, which tells you everything about how "acceptable" the workaround had become.

The root cause was embarrassingly simple. The playbook used serial: 100%, meaning Ansible hit every host at once, restarted every service at once, and HAProxy's health checks failed simultaneously across the entire backend pool. There was no ansible zero-downtime deployment strategy in place — just a full-fleet restart dressed up as automation. Users hit dead backends for the exact duration it took systemd to bring the process back up and pass its first health check.

The fix isn't exotic. It's serial plus max_fail_percentage plus explicit load balancer deregistration, wired together so that a node stops receiving traffic before it gets touched, and only rejoins the pool once it's actually ready — not just "port open." That last distinction matters more than people expect, and I'll get to why.

Prerequisites

Before touching the playbook, make sure the environment actually supports this pattern:

Ansible core 2.15+ — serial and max_fail_percentage are play-level keywords, not task-level, and older versions have inconsistent batch-rounding behavior.
An inventory group (webservers) with SSH access and become privileges configured.
A load balancer that exposes some form of programmatic control. We used HAProxy's runtime socket (/var/run/haproxy/admin.sock), but if you're on AWS, the amazon.aws.elb_target module (collection amazon.aws >= 5.0.0) does the equivalent against an ALB target group.
An app-level /healthz endpoint that returns 200 only when the app is actually warmed up — database pool initialized, cache connected — not just "process is alive."

If any of these are missing, fix that first. Building rolling deploy logic on top of a health check that lies is worse than not having one, because it gives you false confidence right before it breaks in production.

Step 1 — Structure the playbook with serial and batching

The skeleton starts at the play level, before any load balancer logic gets involved. serial: "25%" tells Ansible to work through the inventory in batches, and it rounds up — on an 8-host inventory that's batches of 2, but on a 7-host inventory, 25% rounds to batches of 2 as well since Ansible ceils the fraction rather than truncating it. Don't assume the math lines up neatly; check it against your actual host count.

max_fail_percentage: 0 combined with any_errors_fatal: true gives the strictest possible behavior: if a single host in a batch fails, the entire run stops immediately, including batches that haven't started yet. That's intentional. A partial bad rollout across half your fleet is a much worse night than a deploy that stops early.

order: sorted matters more than it looks — default inventory order can shift between runs if you're using a dynamic inventory script, which makes batch membership non-deterministic. That's a debugging nightmare when you're trying to reproduce a failure that only happened on "whichever three hosts landed in batch two."

Step 2 — Deregister from the load balancer before touching the host

This is the step naive rolling deploys skip, and it's the number one mistake I see. Restarting the service before pulling the node out of rotation means HAProxy is still routing live requests to a process that's mid-restart. You don't get a clean 502 — you get dropped connections mid-request, which is worse for anything transactional.

The fix is a pre_tasks block that disables the backend server first, then pauses long enough for in-flight connections to drain before anything else happens.

pre_tasks:
  - name: Deregister node from HAProxy backend
    ansible.builtin.shell: |
      echo "disable server backend_app/{{ inventory_hostname }}" | \
      socat stdio /var/run/haproxy/admin.sock
    changed_when: true

  - name: Wait for in-flight connections to drain
    ansible.builtin.pause:
      seconds: "{{ drain_wait }}"

The drain wait should roughly match your load balancer's connection draining timeout. AWS ALB defaults to a 300-second deregistration delay, which is absurd for a rolling deploy — most teams tune it down to 15-30 seconds. Whatever value you land on, keep the Ansible pause in sync with it, or you'll drain traffic on paper while the LB config still thinks it needs five minutes.

Step 3 — Deploy, restart, and health-check before re-registering

Once the node is out of rotation, deploy the artifact, restart the service, and — critically — wait on a real readiness check before doing anything else. This is where the second big mistake shows up: checking wait_for: port=8080 and calling it done. A port being open tells you the process bound to it. It tells you nothing about whether the DB connection pool finished initializing or the JIT warmed up. I've watched a "successful" deploy re-register a node into the pool and immediately start throwing 500s for the next ten seconds, because the health check was checking the wrong thing.

tasks:
  - name: Deploy latest release artifact
    ansible.builtin.unarchive:
      src: "/opt/releases/app-{{ app_version }}.tar.gz"
      dest: "/opt/app/current"
      remote_src: false

  - name: Restart app service
    ansible.builtin.systemd:
      name: myapp
      state: restarted
      daemon_reload: true

  - name: Wait for app to report healthy
    ansible.builtin.uri:
      url: "{{ healthcheck_url }}"
      status_code: 200
      return_content: true
    register: health
    until: >
      health.status == 200 and
      (health.content | from_json).status == "ok"
    retries: 10
    delay: 3

The /healthz response should include the dependencies that actually matter — something like {"status":"ok","db":"connected","cache":"connected"} — not just process liveness. If you're running DB migrations as part of the deploy, keep them in a separate play with run_once: true, not inline in this one. If the first host in the first batch runs migrations and then fails before the app deploy step completes, you're left with a schema change applied but no app deployed to match it — a nasty state to debug at 3am.

Step 4 — Re-register and move to next batch

Only after the health check passes does the node go back into rotation. Re-enable it on HAProxy, add a short soak period, then let Ansible move to the next batch.

post_tasks:
  - name: Re-register node with HAProxy backend
    ansible.builtin.shell: |
      echo "enable server backend_app/{{ inventory_hostname }}" | \
      socat stdio /var/run/haproxy/admin.sock
    changed_when: true

  - name: Soak period before next batch
    ansible.builtin.pause:
      seconds: 10

That soak period isn't decorative. It gives your monitoring a window to catch a regression on this one node before the next batch also goes down for deployment — cheap insurance against compounding a mistake across the whole fleet. If you need finer-grained concurrency control within a batch — say, only letting one host at a time run a heavy cache-warm task even though serial allows three — throttle: N on that specific task overrides the batch concurrency without changing your overall rollout shape.

Also worth flagging on the security side: whatever credentials Ansible uses to talk to the LB — HAProxy socket permissions or an AWS IAM role — should be scoped to target group registration only, not full ELB admin. Store them via Ansible Vault or an IAM role, never plaintext in inventory. It's an easy thing to overlook once the deploy pipeline is "working."

Verify and test

Don't take zero-downtime on faith — prove it. Run a continuous load test against the LB VIP for the duration of the deploy and check for non-2xx responses:

hey -z 90s -c 10 http://lb.internal/ | tee load_result.txt
grep -E "\[[45][0-9]{2}\]" load_result.txt && echo "FOUND ERRORS" || echo "CLEAN RUN"

In parallel, watch the HAProxy stats socket to confirm nodes actually transition to MAINT and back to UP rather than dropping out silently:

watch -n1 'echo "show stat" | socat stdio /var/run/haproxy/admin.sock | \
  awk -F"," "{print \$2, \$18}"'

# Expected during a healthy rolling deploy:
# node1  UP
# node2  MAINT   <- briefly, while deregistered
# node3  UP
# node2  UP      <- back after health check passes

Then SSH into the host being deployed and confirm it shut down gracefully instead of getting killed mid-request:

ssh node2 "journalctl -u myapp --since '2 min ago' | grep -i 'sigterm|shutting down'"

That last check ties back to your systemd unit — set TimeoutStopSec=30 and make sure the app actually handles SIGTERM by finishing in-flight requests, or you'll get hard kills even after LB deregistration if the app takes too long to drain on its own. Finally, before you ever point this at production, run ansible-playbook site.yml --limit webservers --check against staging, then a real run with -vv so you can see the exact HTTP/socket calls hitting the LB before they touch anything that matters. See the DevOps_DayS archive for more on load balancer health check patterns if you're running this against something other than HAProxy.

Closing

Zero-downtime isn't a flag you flip — it's a chain of three things that all have to hold at once: batched rollout via serial, explicit load balancer state changes wrapping the actual deploy, and a health check that means "ready" instead of just "alive." Drop any one of those and you silently reintroduce the exact outage window you thought you'd fixed, and it usually won't show up until someone's staring at 502 logs wondering why the "zero-downtime deploy" isn't. Once this pattern is solid on a single playbook, the natural next step is wiring it into CI — GitHub Actions or GitLab pipelines with a manual approval gate between batches — so a human can eyeball metrics before the rollout continues past the first few hosts. For more on the underlying HAProxy and load balancer mechanics, the HAProxy documentation and the Ansible strategies guide are worth keeping bookmarked.

Cloudflare Origin Hardening Checklist: Firewall, Bots, Strict SSL

Oleksandr Kuryzhev — Thu, 13 Aug 2026 07:02:06 +0000

Originally published on kuryzhev.cloud

Why this checklist

A client of mine once showed me their Cloudflare dashboard during what they thought was a DDoS attack. Traffic graphs were flat. Zero anomalies. Meanwhile their origin server's CPU was pegged at 100% and the app was falling over every few minutes. The attacker hadn't gone through Cloudflare at all — they'd found the origin IP and were hitting it directly, completely bypassing the WAF, the rate limiting, the bot protection, everything. That's the moment I started treating cloudflare origin hardening as a separate discipline from "just turning Cloudflare on."

Here's the uncomfortable truth: proxying your DNS through Cloudflare (the orange cloud icon) does not protect your origin server by itself. It protects requests that go through Cloudflare's edge. If someone finds your real IP — via old A records in DNS history tools like SecurityTrails, via certificate transparency logs on crt.sh, via a misconfigured mail server SPF record, or just by brute-forcing common subdomains before you enabled the proxy — they can talk directly to your server. No WAF, no rate limits, no bot challenge. Just raw TCP straight to your box.

This checklist closes three specific gaps: network-level access control (who can even reach the origin's IP), request-level filtering (bots, malicious payloads, brute force), and TLS trust between the edge and the origin (so the "encrypted" connection Cloudflare shows you is actually meaningful). If you only do one of these three, you haven't done cloudflare origin hardening — you've done a fraction of it.

The checklist (numbered)

Work through these in order. Each item is independently verifiable — don't move to the next until you've confirmed the current one actually works, not just that you clicked the toggle.

Restrict the origin firewall to Cloudflare's published IP ranges. Pull the current lists from cloudflare.com/ips-v4 and ips-v6, and lock down your security group / iptables to allow ports 80 and 443 only from those ranges.
Cover IPv6, not just IPv4. If your origin has an AAAA record and Cloudflare is proxying it, the IPv6 range needs the same restriction. This gets forgotten constantly.
Restrict management ports separately. SSH (22), database ports, admin panels — none of these should sit behind "Cloudflare-only" rules since Cloudflare doesn't proxy them. Lock these to your VPN or bastion IP, full stop.
Verify with a direct curl test. Run curl -H "Host: yourdomain.com" https://ORIGIN_IP/ from a machine outside your office/VPN. It should time out or get refused. If it returns a 200, your firewall rules aren't scoped correctly.
Enable Authenticated Origin Pulls. This makes your origin only accept requests carrying a client certificate signed by Cloudflare. Import Cloudflare's origin-pull CA cert to /etc/nginx/certs/cloudflare-origin-pull-ca.pem and set ssl_verify_client on; in your nginx server block.
Set SSL/TLS mode to Full (strict). "Full" accepts any certificate, including self-signed junk — it validates encryption, not identity. Only "Full (strict)" checks the cert chain against a trusted CA.
Issue a Cloudflare Origin CA certificate (15-year validity, free, via the dashboard under SSL/TLS > Origin Server or the /certificates API) and install it on the origin. This cert is only trusted between Cloudflare and your server — don't try to use it for anything public-facing.
Double check "Always Use HTTPS" isn't paired with Flexible mode. Flexible + Always HTTPS is a classic redirect-loop combo (ERR_TOO_MANY_REDIRECTS) if your origin doesn't actually serve HTTPS.
Turn on Bot Fight Mode (free tier, coarse) or Super Bot Fight Mode (Pro/Business, JS-based detection) depending on budget. Enterprise gets ML-based Bot Management if you need it.
Add WAF managed rules and set them to "Log" first, not "Block." Review false positives for at least a week before flipping the switch.
Add rate limiting on sensitive endpoints — login, password reset, API auth — separate from general WAF rules. Budget for this; Free/Pro plans cap you around 10 rate limit rules as of 2024.
Explicitly verify WAF coverage for WebSocket and gRPC traffic. Some managed rulesets don't inspect Upgrade: websocket requests by default, leaving a quiet gap.

Here's a Terraform module that codifies most of the above so it isn't a one-time dashboard exercise:


# main.tf — Cloudflare origin hardening baseline via Terraform
# Provider version pinned to avoid v4->v5 breaking changes in zone_settings resources
terraform {
  required_providers {
    cloudflare = {
      source  = "cloudflare/cloudflare"
      version = "~> 4.30"
    }
  }
}

variable "zone_id" {
  type = string
}

# 1. Force strict SSL between edge and origin — never use "flexible" in prod
resource "cloudflare_zone_settings_override" "strict_ssl" {
  zone_id = var.zone_id
  settings {
    ssl                      = "strict"
    always_use_https         = "on"
    min_tls_version          = "1.2"
    tls_1_3                  = "on"
    automatic_https_rewrites = "on"
  }
}

# 2. Enable Authenticated Origin Pulls at the zone level
resource "cloudflare_authenticated_origin_pulls" "origin_pull" {
  zone_id = var.zone_id
  enabled = true
}

# 3. Enable Super Bot Fight Mode (requires Pro plan or higher)
resource "cloudflare_bot_management" "bots" {
  zone_id                = var.zone_id
  fight_mode              = true
  using_latest_model      = true
  suppress_session_score  = false
}

# 4. Rate limit login endpoint — 20 req/min per IP, then challenge
resource "cloudflare_rate_limit" "login_protect" {
  zone_id   = var.zone_id
  threshold = 20
  period    = 60
  match {
    request {
      url_pattern = "example.com/login*"
      schemes     = ["HTTPS"]
      methods     = ["POST"]
    }
  }
  action {
    mode    = "challenge"
    timeout = 600
  }
  disabled = false
}

# 5. Firewall rule: block anything failing the authenticated origin pull check
resource "cloudflare_ruleset" "origin_only" {
  zone_id = var.zone_id
  name    = "reject-non-cf-origin-checks"
  kind    = "zone"
  phase   = "http_request_firewall_custom"

  rules {
    action      = "block"
    expression  = "(not cf.tls_client_auth.cert_verified)"
    description = "Block requests failing authenticated origin pull check"
    enabled     = true
  }
}

Commonly missed items

Even teams that "complete" the checklist above leave doors open. I've audited enough setups to see the same four gaps repeatedly.

Stale IP ranges. Cloudflare updates its published IP list occasionally, without much announcement. If your firewall rules were hardcoded from a Terraform apply six months ago, you're either blocking legitimate Cloudflare edge nodes (causing intermittent 5xx errors that look random) or leaving gaps that an attacker could theoretically slot into. Watch out for this — it's silent until it isn't.

Port 80 left wide open. This is the mistake I see most. Teams lock 443 down to Cloudflare IPs and completely forget port 80. Result: plaintext HTTP requests reach the origin directly, bypassing the redirect-to-HTTPS logic that only exists at the edge. Your "hardened" origin is still accepting raw connections.

IPv6 origin address exposure. Same story as above but easier to miss because most people mentally model their infrastructure as IPv4-only. If your host has an AAAA record and IPv6 firewall rules weren't updated alongside IPv4, you've got an open backdoor nobody's watching.

Cert expiry with no monitoring. Cloudflare Origin CA certs last 15 years, so that's usually fine. But teams that use a public cert (Let's Encrypt, 90-day validity) on "Full (strict)" mode and never wire up renewal specifically for the origin-facing cert will eventually see error 526 — Invalid SSL Certificate. It happens quietly. No alert, no dashboard warning, just a slow bleed of failed requests until someone notices traffic dropped. I stopped using Let's Encrypt on origins behind Cloudflare after this bit a client twice — Origin CA certs with a 15-year runway remove the whole class of problem.

Also worth knowing: error 525 means the SSL handshake between edge and origin failed outright (usually a cipher mismatch or missing intermediate cert), while 526 specifically means the cert itself is invalid or expired under strict mode. Different root causes, same symptom of "site is down and Cloudflare's error page doesn't tell you why."

Automation ideas

A checklist you run once during setup rots. Cloudflare origin hardening needs to be enforced continuously, not remembered.

Codify the settings above in Terraform (see the module earlier) so SSL mode, WAF rules, and Authenticated Origin Pulls live in version control instead of dashboard clicks nobody documented. Check the Cloudflare Terraform provider docs for the current resource names — v4 renamed a few things and pinning your provider version avoids surprise breaking changes on the next terraform init.

For the IP range rot problem, run a scheduled job — GitHub Actions cron or a plain crontab entry — that pulls the current lists from ips-v4/ips-v6 and diffs them against what's in your security group. Alert on any change instead of silently applying it; you want a human to glance at what changed before it goes live.

And build this into CI as a smoke test after every deploy — fail the pipeline if the origin is directly reachable:


#!/usr/bin/env bash
# verify_origin_lockdown.sh — CI smoke test for origin hardening
# Run in pipeline after deploy; fails build if origin is directly reachable

ORIGIN_IP="203.0.113.45"     # replace with real origin IP, ideally from Terraform output
DOMAIN="example.com"

echo "== Test 1: direct IP access should be refused =="
STATUS=$(curl -o /dev/null -s -w "%{http_code}" --max-time 5 \
  -H "Host: ${DOMAIN}" "https://${ORIGIN_IP}/" -k || echo "TIMEOUT")

if [[ "$STATUS" == "TIMEOUT" || "$STATUS" == "000" ]]; then
  echo "PASS: origin unreachable directly ($STATUS)"
else
  echo "FAIL: origin responded directly with HTTP $STATUS — firewall misconfigured"
  exit 1
fi

echo "== Test 2: SSL mode check via edge =="
curl -sI "https://${DOMAIN}/" | grep -i "strict-transport-security" \
  && echo "PASS: HSTS present" \
  || echo "WARN: HSTS header missing — check Always Use HTTPS + HSTS settings"

echo "== Test 3: confirm cert issuer is Cloudflare-trusted chain (origin side) =="
openssl s_client -connect "${ORIGIN_IP}:443" -servername "${DOMAIN}" /dev/null \
  | openssl x509 -noout -issuer -dates

# Expected: issuer should reference "CloudFlare Origin SSL Certificate Authority"
# and dates should show >30 days remaining before "notAfter"

This turns cloudflare origin hardening from a one-time audit into something your pipeline actively guards. The nginx side of Authenticated Origin Pulls is documented in the nginx ssl module docs if you need to fine-tune ssl_verify_client beyond the basic "on" setting. For more on layering rate limiting and WAF decisions correctly, I wrote about picking between nginx-level limits and a full API gateway on kuryzhev.cloud — worth a read if you're deciding where that logic should live.

The core lesson: a green Cloudflare dashboard means nothing if your origin's real IP is one DNS history lookup away from being public knowledge. Treat the origin as its own attack surface, not just a backend behind a proxy.

Nginx Rate Limiting vs API Gateway: Picking the Right Layer

Oleksandr Kuryzhev — Wed, 12 Aug 2026 07:02:03 +0000

Originally published on kuryzhev.cloud

Your nginx.conf can block 90% of API abuse for free — so why do teams rush to pay for Kong or Cloudflare Enterprise before they even need it? I've watched this happen at three different companies now: a scraper burst hits /search, credential-stuffing hammers /login, or a "trusted" third-party integrator decides your fair-use policy is more of a suggestion. Someone panics, opens a ticket for "add rate limiting," and two sprints later there's a Redis cluster and a Kong Enterprise line item nobody asked for. Nginx rate limiting was sitting there the whole time, already installed, already free.

When you face this choice

The trigger is almost always the same shape: traffic that looks legitimate at the edge but isn't sustainable at the backend. A scraper hammering your product catalog at 200 req/s from a rotating pool of IPs. A credential-stuffing run against /login that's technically valid HTTP but is clearly not a human. Or the more mundane case — a partner integration you signed off on eighteen months ago that's now sending 5x the agreed volume because their product grew and nobody told you.

At that point you're at a fork. Option one: solve it in nginx.conf, close to the socket, cheap and static. Option two: push the logic to a gateway or edge layer that's dynamic, distributed, and — let's be honest — usually costs money or ops overhead you didn't budget for.

The thing that actually forces the decision isn't philosophy, it's topology. If you're running a single nginx instance in front of your API, native rate limiting just works — it has one process, one shared memory zone, one source of truth. The moment you scale to multiple nginx or LB nodes without shared state, that single-node assumption breaks, and you need to decide whether to bolt on synchronization or move the whole problem to a layer built for distributed counting. That's the real question behind "nginx rate limiting vs gateway," and it's worth answering honestly before you write a single limit_req_zone line.

Option A: Native Nginx rate limiting (limit_req / limit_conn)

Nginx has shipped ngx_http_limit_req_module since version 1.1.8 — there's nothing to install on a modern 1.24/1.25 box, it's compiled in by default. That alone makes it the obvious starting point. The pros are hard to argue with: sub-millisecond overhead, request rejection happens at L7 before anything touches your app servers, and the whole config lives in one file you can read top to bottom.

Performance-wise, zone lookups are O(1) against a shared memory hash table, so even at 10k+ req/s the CPU cost is negligible. The real cost — and this bit me once — is logging. If you don't set limit_req_log_level warn;, every single rejection writes a line to error.log, and under a sustained attack that log file grows fast enough to fill a disk.

Now the cons, and they're real. Rate limiting is per-instance — the memory zone lives in one nginx process, so if you run multiple nginx or LB nodes, each one counts independently and your effective global limit multiplies by node count. Limits are IP-based by default, which quietly breaks behind NAT, corporate proxies, or a CDN that masks the real client IP — everyone ends up hashed to the load balancer's address unless you configure real_ip_header correctly. And there's no built-in per-API-key or per-JWT-claim limiting; you're writing map/geo hacks to fake it, which gets ugly fast once you have more than a couple of tiers.

Option B: Gateway/edge-based rate limiting (Kong, Cloudflare, or OpenResty+Redis)

This is where you go when "per IP" stops being granular enough. A Redis-backed layer — whether that's Kong's rate-limiting plugin with policy=redis, or hand-rolled OpenResty with lua-resty-limit-traffic — gives you true distributed counting across N nodes. Per-API-key, per-JWT, per-tenant limits work out of the box with Kong plugins or Cloudflare rulesets, which is exactly what you need once you're billing customers by usage tier.

Cloudflare specifically has the added benefit of blocking abusive traffic before it reaches your infrastructure at all — that saves bandwidth and compute, not just app-layer noise. But it's not free: Cloudflare's Rate Limiting Rules are billed per rule/request volume starting at the Pro plan; the free tier only gives you basic firewall rules, not granular rate limiting. Kong OSS's local-counter default doesn't help you across nodes either — you need the Redis policy, and that policy has noticeably less config flexibility than what Kong Enterprise offers.

The operational cons stack up too. Every request now takes a Redis round-trip — roughly 1-2ms with lua-resty-limit-traffic — which is fine until Redis has a bad night and now your rate limiter is a single point of failure for your entire API. You're also running and monitoring another service, and if you're going the Kong Enterprise route, licensing cost is a real line item, not a rounding error.

Here's the native config I run as a baseline on most APIs before reaching for anything heavier. It covers per-IP request limiting, connection capping for slow-POST protection, and a whitelist for internal/partner traffic:


# /etc/nginx/conf.d/api-ratelimit.conf
# Option A: native nginx rate limiting — single-node setup

# Define shared memory zone: 10m ≈ 160k tracked client IPs
# rate=10r/s is the sustained limit per IP
limit_req_zone $binary_remote_addr zone=api_limit:10m rate=10r/s;

# Separate zone for concurrent connection capping (slow-POST protection)
limit_conn_zone $binary_remote_addr zone=conn_limit:10m;

# Whitelist internal monitoring/partner IPs — bypasses limiting entirely
geo $limit_whitelist {
    default 0;
    10.0.0.0/8 1;      # internal network
    203.0.113.5/32 1;  # trusted partner
}
map $limit_whitelist $limit_key {
    0 $binary_remote_addr;
    1 "";  # empty key = not tracked/limited
}
limit_req_zone $limit_key zone=api_limit_wl:1m rate=10r/s;

server {
    listen 443 ssl;
    server_name api.example.com;

    # If sitting behind a CDN/LB, must trust the real client IP header
    set_real_ip_from 10.0.0.0/8;
    real_ip_header X-Forwarded-For;
    real_ip_recursive on;

    location /v1/ {
        # burst allows short spikes, nodelay serves them immediately
        # instead of queuing (avoids added latency for legit bursts)
        limit_req zone=api_limit burst=20 nodelay;
        limit_conn conn_limit 5;

        # return 429 instead of nginx's default 503
        limit_req_status 429;
        limit_conn_status 429;

        # avoid flooding error.log on every rejection
        limit_req_log_level warn;

        add_header Retry-After 1 always;
        proxy_pass http://backend_upstream;
    }
}

Two gotchas here that I've seen take down or under-protect production more than once. First: nginx's default rejection status is 503 Service Unavailable, not 429. If you don't explicitly set limit_req_status 429;, client retry logic that expects a proper 429 with Retry-After will misbehave — some clients treat 503 as "server is down" and back off way longer than needed, others retry immediately and make things worse. Second: if you're sitting behind Cloudflare or an ALB and you rate limit on $binary_remote_addr without configuring real_ip_header and set_real_ip_from, every single request appears to come from the load balancer's IP. You'll rate limit your entire user base as if it were one client — I've seen this take a service to its knees within minutes of deploy.

Always run nginx -t before reload. A missing m or k suffix on a zone size fails silently in some nginx versions and defaults to a tiny zone — you won't notice until an attack with high IP cardinality evicts old entries via LRU and lets abuse through under sustained load. Test with a quick loop:


# Quick test: hammer the endpoint and observe 429s kick in after burst

for i in $(seq 1 30); do
  curl -s -o /dev/null -w "%{http_code}\n" https://api.example.com/v1/ping
done

# Expected output (rate=10r/s, burst=20, nodelay):
# 200
# 200
# ... (first ~20 pass due to burst allowance)
# 429
# 429
# 429
# ...

# Check nginx error log for the rejection reason:
tail -f /var/log/nginx/error.log | grep "limiting requests"
# 2024/06/01 12:03:41 [warn] 1234#0: *5678 limiting requests,
# excess: 10.400 by zone "api_limit", client: 203.0.113.9,
# server: api.example.com, request: "GET /v1/ping HTTP/1.1"

One more thing worth knowing: IP-based limiting alone is trivially bypassed by botnets rotating across thousands of residential proxy IPs. If you're facing that kind of adversary, pair rate limiting with request fingerprinting — User-Agent plus TLS JA3 — or a CAPTCHA challenge. Rate limiting alone won't stop a determined attacker with a large IP pool; it just raises the cost of the attack.

Decision matrix

Here's how I map real situations to a choice, based on topology, granularity needs, budget, and team comfort with running Redis/Lua in production:

Situation	Recommendation
Single nginx box fronting an internal or low-traffic public API	Option A — native nginx
Startup MVP, no paying multi-tenant customers yet	Option A now, revisit at scale
Need per-API-key or per-tenant quotas for billing	Option B — Kong/OpenResty+Redis
Multi-region API with paying tenants and SLA tiers	Option B, with edge (Cloudflare) in front
High-volume public API, no budget for edge licensing	Option A + fail2ban, tuned aggressively
Team has zero Redis/Lua experience and tight deadline	Option A — don't add ops debt for a launch

Most real setups I've built end up as a hybrid: native nginx handles coarse IP/connection limits at the edge of the config, and a gateway or app-layer service handles business-logic quotas — the "you get 10,000 API calls a month on the Starter plan" kind of enforcement that genuinely needs a database or Redis behind it. That split is honest about which layer is good at what.

My pick

I'll say it plainly: start with native nginx limit_req/limit_conn for anything under roughly 50 req/s baseline traffic and a single-node or simple LB setup. It's free, it's already running, and it handles 90% of the abuse patterns you'll actually see — scraper bursts, login brute-forcing, slow-POST DoS attempts. I stopped recommending Kong or Cloudflare Enterprise rate limiting as a first move after watching a client burn two weeks and a chunk of their infra budget solving a problem that limit_req_zone and a whitelist rule would've fixed in an afternoon.

My concrete setup on new projects: native nginx rate limiting as described above, paired with fail2ban parsing the error log — a filter watching for "limiting requests" that auto-bans repeat offenders at the firewall (iptables/nftables) level so they don't even reach nginx on their next attempt. That combination is free, low-latency, and shockingly effective. I only reach for Redis-backed OpenResty or a real gateway once there's an actual business requirement for per-API-key quotas across multiple nodes — not before. Nginx rate limiting isn't a stopgap you outgrow immediately; for most APIs, it's the right permanent answer. If you're building out the surrounding infra, our DevOps notes and setups cover a lot of the adjacent pieces — logging, load balancer configs, and the fail2ban side of this exact stack. For the module internals, the official ngx_http_limit_req_module docs and Cloudflare's rate limiting rules docs are worth reading before you commit either direction.

Docker BuildKit Cache Setup That Actually Speeds Up CI

Oleksandr Kuryzhev — Tue, 11 Aug 2026 07:01:43 +0000

Originally published on kuryzhev.cloud

Last month a client asked me why their "cached" Docker builds still took nine minutes on every single pull request. They had --cache-from in their GitHub Actions workflow, a green checkmark, and a nagging suspicion something was off. Turned out their BuildKit cache had never actually hit once in three months — it was pulling a stale :latest tag as cache source and silently falling back to a full rebuild every time. This is the single most common failure mode I see with Docker BuildKit cache CI setups, and it's almost always invisible until someone actually times the build.

What BuildKit Cache Actually Does Under the Hood

The legacy Docker build cache (pre-BuildKit) was dead simple: each instruction in the Dockerfile produced a layer, and that layer was reused if the instruction and its parent layer hadn't changed. It's content-addressable, but tightly coupled to instruction order. Change line 3, and everything below it invalidates — no exceptions.

BuildKit changes the model. Cache is keyed by a digest computed from the actual inputs to a step: the base image digest, the build context checksum for anything copied in, and the resolved command. That's why two builds with identical Dockerfiles but different base image digests will miss cache even though the text is byte-identical — the input hash changed, not the instruction.

There are three cache backends that matter in CI. Inline cache (BUILDKIT_INLINE_CACHE=1) embeds cache metadata directly into the pushed image — this is deprecated since Buildx v0.10 in favor of registry cache. Registry cache (--cache-to type=registry) pushes cache blobs to a separate manifest in your registry, independent of the final image tag. And local/GHA cache (type=local, type=gha) stores cache on disk or in GitHub's Actions cache service, capped at 10GB per repo.

The key thing to internalize: a cache "hit" requires the build context checksum, base image digest, AND the instruction to all match what's in the cache manifest. Any COPY or ADD touching a changed file invalidates that layer and every layer after it. This is exactly why layer ordering matters so much, which is where most teams get it wrong. Docker's own BuildKit cache documentation covers the backend types in more depth if you want the full matrix.

How People Use It Wrong

The first mistake I ran into with that client: pulling --cache-from myimage:latest as the cache source. The problem is :latest drifts — it's whatever the most recent successful build pushed, which might be from a completely different branch with a different dependency tree. BuildKit computes the cache key against what's actually in that manifest, and if it doesn't match, you get a silent miss with zero error message. No warning, just a slow build that looks "normal."

The second mistake is structural: COPY . . before npm ci or pip install. I've seen this in maybe 70% of Dockerfiles I've audited. Every commit — even a one-line README change — invalidates the entire dependency install layer, because the build context checksum for that COPY includes every file in the repo. You end up reinstalling node_modules or your virtualenv on every single push, even when the lockfile hasn't moved.

The third mistake is relying purely on the CI runner's local disk. GitHub-hosted runners are ephemeral — each job gets a fresh VM. If you never export cache to a registry or the GHA backend, that "warm cache" from your last build simply doesn't exist anymore. I stopped trusting local-only caching on hosted runners after watching a team's "optimized" pipeline run cold for six months without anyone noticing, because nothing ever raised an error — the pipeline was just always slow. A related mistake: using --cache-from without --cache-to. You pull old cache but never write updates back, so the cache goes stale after the first real change and never refreshes.

The Correct Approach

The fix has two parts: Dockerfile structure and cache backend configuration. Structurally, order instructions from least volatile to most volatile — copy lockfiles first, install dependencies, then copy source last.

FROM node:20-slim@sha256:abcd1234...  # pinned digest avoids silent cache invalidation

WORKDIR /app

# 1. Copy only lockfiles first — this layer stays cached until deps actually change
COPY package.json package-lock.json ./

# 2. Use cache mount for npm store — persists package cache across builds
RUN --mount=type=cache,target=/root/.npm,sharing=locked \
    npm ci --prefer-offline

# 3. Source code copied last — only invalidates this layer + below on every commit
COPY . .

RUN npm run build

# Expected CI log on cache hit:
# => CACHED [2/5] COPY package.json package-lock.json ./
# => CACHED [3/5] RUN npm ci --prefer-offline
# => [4/5] COPY . .                                   0.4s
# => [5/5] RUN npm run build                          22.1s

Pinning the base image by digest matters more than people think — node:20-slim can get re-tagged upstream without any change on your end, which silently busts your cache even though nothing in your repo changed.

For the backend, registry cache is the most portable option since it works regardless of which runner picks up the job:

# .github/workflows/docker-build.yml
# CI pipeline demonstrating correct BuildKit cache setup with registry cache backend
name: docker-build

on:
  push:
    branches: [main]
  pull_request:

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
        with:
          driver-opts: image=moby/buildkit:v0.13.2   # pin buildkit version explicitly

      - name: Log in to registry
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Build and push with registry cache
        uses: docker/build-push-action@v5
        with:
          context: .
          push: true
          tags: ghcr.io/org/app:${{ github.sha }}
          # cache-from pulls previous layers; cache-to writes updated ones
          cache-from: type=registry,ref=ghcr.io/org/app:buildcache
          cache-to: type=registry,ref=ghcr.io/org/app:buildcache,mode=max
          build-args: |
            BUILDKIT_INLINE_CACHE=0   # not needed, we use registry cache explicitly

      - name: Prune stale cache tags (weekly)
        if: github.event_name == 'schedule'
        run: |
          # crude example: delete cache manifests older than 14 days via crane
          crane ls ghcr.io/org/app | grep buildcache | while read tag; do
            echo "would prune $tag if older than 14d"
          done

Gotcha: if you see ERROR: failed to solve: failed to load cache key, don't panic — it usually just means the cache ref doesn't exist yet (first run ever) or your registry auth expired mid-pull. It's not a corruption issue, just resolve auth or let the first build populate the ref.

Advanced Patterns

Once you've got the basics working, there are a few patterns worth adopting for teams running multiple services or monorepos. First, mode=max vs mode=min: max exports every intermediate layer from every build stage, which is essential if you have multi-stage Dockerfiles with shared base stages across microservices — otherwise those intermediate stages never get cached for reuse elsewhere. mode=min only exports the final stage, which is cheaper but useless if other targets depend on the same intermediate layer.

Second, in monorepos, shard your cache keys by a hash of the Dockerfile plus the relevant lockfile, not just by repo name. Otherwise service A's cache pollutes service B's cache ref and you get cross-project false hits or unnecessary invalidation storms.

Third, if you're building several targets in one CI job, docker buildx bake with an HCL file lets you define cache-from/cache-to once and apply it across a matrix of targets, instead of copy-pasting the same flags into five separate build steps. It also avoids redundant registry pulls when multiple targets share a base stage. I wrote more about structuring shared build configs like this in our DevOps_DayS archive if you want to see it applied to a full CI matrix.

Watch out: multi-arch builds (--platform linux/amd64,linux/arm64) need separate cache refs per platform, or your arm64 build won't be able to reuse amd64 cache layers at all — I've seen teams assume one cache ref covers both architectures and wonder why arm64 builds are always cold.

Performance Notes

Numbers matter more than theory here. A cold CI runner with zero cache backend typically takes 8-12 minutes for a standard Node or Python app build. With registry cache configured correctly and dependency layers ordered properly, warm builds routinely drop to 30-90 seconds. That's the real payoff — but it's not free.

Registry cache push/pull adds latency per layer, usually 5-30 seconds depending on registry throughput and image size. It's a net win only if you're building frequently enough that the amortized savings outweigh that overhead — for a repo pushed a few times a day, it's an easy yes.

Storage cost is the tradeoff nobody budgets for. mode=max exports every intermediate layer, which can balloon registry storage to 2-3x your actual image size. On ECR, GCR, or ACR, that's billed per GB-month, and unbounded caches across dozens of feature branches will quietly grow your bill. Set a lifecycle rule to expire *:buildcache tags older than 14 days — it's a five-minute fix that prevents a very avoidable line item.

One more thing worth flagging as a security note, not just performance: cache blobs pushed to a shared or public registry can leak build secrets or leftover file contents if you're not using --mount=type=secret. Cache layers aren't scrubbed of env vars or files copied in before a later cleanup RUN — if a secret touched the filesystem at any point, it can persist in the cache manifest even after you delete it in a later layer. I've had to explain this one twice to teams who assumed a final RUN rm -rf was enough. Check the Docker build secrets docs before you bake anything sensitive into a cached stage.

RDS Backup Restore Testing: 7 Checks Before You Trust It

Oleksandr Kuryzhev — Mon, 10 Aug 2026 07:01:42 +0000

Originally published on kuryzhev.cloud

Your RDS backups show "available" every night in the console. Green checkmark, no alarms, everyone's happy. But have you actually restored one of those snapshots and queried the data inside it? I ask because we didn't — for almost a year — until a corrupted binlog turned our "reliable" backup strategy into a three-hour incident call. RDS backup restore testing isn't optional infrastructure hygiene, it's the only way to know your recovery plan actually works.

Automate the Restore, Don't Trust the "Available" Status

An RDS snapshot marked "available" only confirms the backup process finished without error — it says nothing about whether the data inside is usable. I've seen teams treat a green CloudWatch backup alarm as proof of recoverability, then discover during a real incident that the snapshot restores into an instance with half the expected rows. The fix is boring but effective: schedule a recurring EventBridge rule that triggers a Lambda to run restore-db-instance-from-db-snapshot weekly or monthly, automatically, without a human remembering to do it.

This isn't about paranoia — it's about closing the gap between "backup completed" and "backup is recoverable." Those are two very different claims, and RDS only ever promises you the first one.

Restore Into an Isolated Network, Not Your Production VPC

Restoring a snapshot into the same VPC as production is asking for CIDR and security-group conflicts, and in the worst case, test traffic touching real systems. We keep a dedicated "restore-testing" subnet group with zero routes to prod app servers, and tag every restored instance with Purpose=restore-test plus an expiry timestamp so cleanup automation can find it later.

Watch out: if you forget to attach a security group during the restore call, the instance comes up "available" but is completely unreachable — your validation script will just hang or fail silently with a connection timeout, and it's easy to mistake that for a data problem instead of a networking one.

Verify Point-in-Time Recovery, Not Just Snapshot Age

Snapshots are simple — PITR is not. Point-in-time recovery depends on continuous transaction log shipping, and that pipeline fails silently far more often than a nightly snapshot job does. Before you assume PITR can restore to "now," check LatestRestorableTime:

$ aws rds describe-db-instances \
    --db-instance-identifier prod-orders-db \
    --query 'DBInstances[0].[LatestRestorableTime,EarliestRestorableTime]' \
    --output table

Output looks like this:

-----------------------------------------------
|            DescribeDBInstances              |
+-----------------------------+---------------+
|  2024-05-14T09:42:17.000Z  | 2024-04-09T00:00:00.000Z |
+-----------------------------+---------------+

# Gotcha: if LatestRestorableTime is more than 5-10 minutes behind current
# time, binlog retention or transaction log backup may be misconfigured —
# investigate before trusting PITR for RTO planning.

For MySQL/MariaDB, PITR quietly depends on binlog retention configured via mysql.rds_set_configuration('binlog retention hours', 168). If that's not set, your recoverable window shrinks without any alarm firing. And don't confuse Aurora's "backtrack" feature with PITR — backtrack tops out at 72 hours and only works on Aurora MySQL, it's not a substitute for transaction-log-based recovery.

Validate Data Integrity After Restore, Not Just Connectivity

A restored instance that accepts connections isn't the same as a restored instance with intact data. I've watched a "successful" restore return a working psql prompt against a table that was silently truncated three days before the incident — connectivity checks alone would've called that a pass. Run row-count comparisons and checksum queries against known baselines: CHECKSUM TABLE for MySQL, pg_checksums --check for Postgres (note: Postgres checksums have to be enabled at initdb time, you can't bolt them on retroactively).

Here's a script we run after every scheduled restore test — it restores, waits, checks row counts against a known minimum, then tears itself down:

#!/usr/bin/env bash
# restore-test.sh - automate an RDS snapshot restore + basic validation
set -euo pipefail

SNAPSHOT_ID="prod-db-snapshot-2024-05-01"
TEST_INSTANCE_ID="restore-test-$(date +%s)"
SUBNET_GROUP="restore-testing-subnet-group"
SECURITY_GROUP="sg-0123456789abcdef0"

echo "Starting restore of $SNAPSHOT_ID into $TEST_INSTANCE_ID..."

aws rds restore-db-instance-from-db-snapshot \
  --db-instance-identifier "$TEST_INSTANCE_ID" \
  --db-snapshot-identifier "$SNAPSHOT_ID" \
  --db-subnet-group-name "$SUBNET_GROUP" \
  --vpc-security-group-ids "$SECURITY_GROUP" \
  --no-publicly-accessible \
  --tags Key=Purpose,Value=restore-test Key=ExpiryDate,Value="$(date -d '+1 day' +%F)"

echo "Waiting for instance to become available..."
aws rds wait db-instance-available --db-instance-identifier "$TEST_INSTANCE_ID"

ENDPOINT=$(aws rds describe-db-instances \
  --db-instance-identifier "$TEST_INSTANCE_ID" \
  --query 'DBInstances[0].Endpoint.Address' --output text)

echo "Instance available at $ENDPOINT. Running validation checks..."

# Basic connectivity + row count check (Postgres example)
ROW_COUNT=$(PGPASSWORD="$DB_PASSWORD" psql -h "$ENDPOINT" -U app_readonly -d appdb -t -c \
  "SELECT count(*) FROM orders;")

EXPECTED_MIN_ROWS=100000

if [ "$ROW_COUNT" -lt "$EXPECTED_MIN_ROWS" ]; then
  echo "FAIL: row count $ROW_COUNT below expected minimum $EXPECTED_MIN_ROWS"
  exit 1
fi

echo "PASS: row count check ($ROW_COUNT rows)"

# Cleanup - delete test instance after validation
aws rds delete-db-instance \
  --db-instance-identifier "$TEST_INSTANCE_ID" \
  --skip-final-snapshot

echo "Restore test complete. Test instance deleted."

Wire this into your CI runner or a Lambda triggered by EventBridge, and you get an automated proof-of-recoverability report instead of a green dashboard icon that means nothing.

Test Cross-Region and Cross-Account Snapshot Copies for DR

If your disaster recovery plan only tests same-region restores, it isn't really tested — a regional outage is exactly the scenario your DR plan exists for, and that's the one path most teams never exercise. Cross-region snapshot copies require re-encryption with a KMS key that exists in the destination region, and this trips people up constantly because the source key ARN simply doesn't resolve there.

aws rds copy-db-snapshot \
  --source-region us-east-1 \
  --source-db-snapshot-identifier arn:aws:rds:us-east-1:111122223333:snapshot:prod-orders-2024-05-01 \
  --target-db-snapshot-identifier prod-orders-dr-copy \
  --kms-key-id arn:aws:kms:us-west-2:111122223333:key/abcd-1234-efgh-5678

Confirm the KMS key ARN is region-local before automating this, and log copy duration — a 500GB+ snapshot can take 30-60 minutes, which directly eats into your RTO budget if you haven't accounted for it. See the AWS RDS snapshot copy docs for the full parameter list and region-specific caveats.

Watch Snapshot Storage Costs During Test Cycles

Automated RDS backups cap out at 35 days retention and expire on their own. Manual snapshots don't — they sit there billed per GB-month indefinitely until someone deletes them, and if you're running weekly restore tests without cleanup, that bill creeps up quietly for months before anyone notices. Storage cost tracks the actual used size within the snapshot, not the allocated instance storage, but frequent test cycles still add up fast on multi-hundred-GB databases.

Automate deletion of test-restore instances and snapshots with a Lambda that scans for the Purpose=restore-test tag and an expired ExpiryDate, then deletes both the instance and any snapshots it spawned. I stopped relying on manual cleanup after finding four forgotten restore-test instances running for two months — that's real money for zero value.

Lock Down IAM and Encryption for Restore Operations

Restore permissions are more powerful than most teams realize, and they're frequently over-provisioned at the account level instead of scoped per environment. Restrict rds:RestoreDBInstanceFromDBSnapshot and rds:CopyDBSnapshot to specific roles using IAM condition keys — anyone with broad RDS access can otherwise spin up a full copy of production data in a subnet you never intended.

Two gotchas worth flagging explicitly. First, KMS key policies must grant kms:CreateGrant to the restore role or cross-account/cross-region restores fail with KMSKeyNotAccessibleFault — a confusing error that has nothing to do with the RDS permissions themselves. Second, restoring always creates a brand-new instance (there's no in-place restore in RDS), and storage class or public accessibility settings can silently default differently than the source — always pass --no-publicly-accessible explicitly rather than trusting the default. Check the RDS restore documentation before scripting this into your pipeline, and pin your tooling — AWS CLI v2.15+ and Terraform's aws provider ~> 5.40 both have relevant fixes for snapshot copy support.

None of this is exotic. It's just the difference between a backup strategy that looks good on a dashboard and one that survives an actual incident. If your restore automation lives inside a broader Terraform or Ansible pipeline, it's worth reviewing how the rest of your infrastructure automation handles secrets and state before you wire restore-testing into CI.

EventBridge Retry Policy vs SQS DLQ: The Right Call in Production

Oleksandr Kuryzhev — Sun, 09 Aug 2026 07:01:58 +0000

Originally published on kuryzhev.cloud

Your EventBridge target has been silently dropping failed events into a DLQ with zero write permissions — for three weeks — and nobody noticed until a customer asked where their order went. We pulled up the CloudWatch metrics, saw FailedInvocations climbing, and then discovered the DLQ had exactly zero messages in it. Not because nothing failed. Because the resource policy on the queue never granted events.amazonaws.com permission to write to it. That's the moment I stopped treating EventBridge retry policy and DLQ setup as a checkbox and started treating it as an actual design decision.

When you face this choice

This comes up the moment you wire an EventBridge rule to a target that isn't 100% reliable — a Lambda that occasionally times out, a Step Functions execution that throttles under load, or an API destination hitting a flaky third-party endpoint. You need a resilience story before this ships to production, and you have two real paths: lean entirely on EventBridge's built-in per-target RetryPolicy and DeadLetterConfig, or put an SQS queue in front of your actual consumer and own the retry/backoff/redrive logic yourself.

Here's the part that trips people up: this decision is made per target, not per bus and not per rule. I've seen teams configure a beautiful DLQ on their first target, ship a second target on the same rule three sprints later, and assume the DLQ config "just applies." It doesn't. Every target gets its own DeadLetterConfig, and if you forget it, that target fails open — into nothing, with no alert, no trace, no evidence anything went wrong until someone downstream notices missing data.

So before you write a single line of Terraform, decide: is native retry + DLQ enough for this target, or do you need the control that only a buffer queue gives you? Don't default to whichever pattern you copy-pasted from the last project. They solve different problems.

Option A: Native EventBridge RetryPolicy + DLQ (pros/cons)

This is the zero-infrastructure option. You set maximum_retry_attempts and maximum_event_age_in_seconds on the target, point DeadLetterConfig.Arn at an SQS queue, and EventBridge handles retries with its own exponential backoff. No Lambda glue, no extra queue to monitor for the happy path, and it works the same whether your target is Lambda, Step Functions, or an API destination.

The catch: those two knobs are the only knobs. You don't control the shape of the backoff curve — only the ceiling. Default MaximumRetryAttempts is 185, and default MaximumEventAgeInSeconds is 86400 (24 hours). Most teams never touch these defaults and don't realize what that means in practice: EventBridge will keep retrying a failing target for a full day, and every one of those retries is a billed invocation downstream. During an outage, a Lambda hammered by 185 retry attempts across dozens of events can spike your invocation cost 100x almost overnight.

Watch out: setting MaximumRetryAttempts: 0 alone does not give you immediate DLQ routing. You still need to explicitly set MaximumEventAgeInSeconds, or the event stays "alive" for the default 24 hours even with zero retry attempts configured — a confusing half-state that looks like a bug but is documented behavior.

Debugging is the other weak point. The message that lands in your DLQ is the raw event, not an error trace. You get no built-in "why did this fail" — you're cross-referencing CloudWatch Logs and metrics like TargetErrorCount and DeadLetterInvocations manually, event by event.

Option B: SQS buffer queue as the real target (pros/cons)

Here you don't target the Lambda or Step Function directly — you target an SQS queue, and your actual consumer reads from that queue. This buys you real control: batching behavior, visibility timeout tuning, and backoff shaped by your Lambda's SQS trigger config, not EventBridge's fixed curve. Redrive is native and supported — set RedrivePolicy.maxReceiveCount and AWS will actually let you redrive messages back out of the DLQ using aws sqs start-message-move-task, GA since late 2023.

It also decouples delivery from processing. EventBridge-to-SQS delivery is near-instant and reliable, so any failure is isolated to the consumer side — easier to reason about, easier to load-test independently of the bus.

The cost is real, though. You're now provisioning, monitoring, and paying for an extra moving part. You own idempotency — SQS is at-least-once delivery, and your own retry logic on top of that can double-process events if your handler isn't idempotent. There's also a subtle latency hit on the happy path since every event now makes an extra hop.

Gotcha: don't assume SQS-as-target gives you ordering. Standard queues don't guarantee it. Switching to FIFO fixes ordering but caps you at 300 msg/s without batching — a real bottleneck if your bus expects high fan-out volume. I've watched a team switch to FIFO "for safety" and immediately create a backlog during a traffic spike that the standard queue would have absorbed fine.

Decision matrix

Use this to skip the re-reading and go straight to a decision.

Row                          Native RetryPolicy+DLQ        SQS Buffer Queue
Retry control granularity    Low — ceiling only            High — full backoff/visibility control
Ease of setup                High — one-liner target block Medium — extra queue + policy + trigger
Cost overhead                Low, but retry storms spike   Moderate, predictable
                              downstream invocation cost
Observability quality        Poor — raw event, no error    Good — CloudWatch + queue depth +
                              context in DLQ                 redrive task status
Replay/redrive capability     Manual only, no native tool   Native via start-message-move-task
Latency impact                None                          Small added hop, usually milliseconds
Best-fit target               Lambda w/ idempotent handler  Step Functions, batch consumers,
                                                              high-throughput fan-out
Team size / maturity          Small teams, low on-call      Needs someone who can own redrive
                              overhead — good default        runbooks and queue monitoring

That last row matters more than people admit. I've seen small teams adopt the SQS buffer pattern because it's "more correct," then quietly regret it six months later when nobody remembers how the redrive script works and the DLQ has 40,000 stale messages nobody wants to touch.

My pick

I default to native RetryPolicy + DLQ for Lambda targets with idempotent handlers and non-critical volume. It's good enough for roughly 80% of the pipelines I've built, and I stopped over-engineering this after watching a team spend two sprints building custom SQS redrive tooling for a target that failed maybe twice a month. Not every event pipeline needs a queue in front of it.

I switch to the SQS buffer pattern only when I need batch processing, something close to ordering guarantees, or a redrive workflow that an on-call engineer can actually run at 2am without writing custom Lambda glue first. If your target is Step Functions, remember EventBridge's RetryPolicy only covers its own delivery attempt — internal state machine failures need their own Retry/Catch blocks regardless of which pattern you pick.

My non-negotiable stance: never ship an EventBridge target to production without some DLQ configured, and never trust that it's working without checking the SQS resource policy explicitly. The number of teams that discover silently-dropped events during an incident postmortem — not before — is the entire reason I write posts like this. We've covered similar production gotchas in our DevOps notes on kuryzhev.cloud if you want more of these before-it-bites-you writeups.

Here's the Terraform for the native approach, DLQ policy included — this is the one-liner people forget to pair with an actual resource policy:


# EventBridge rule targeting Lambda, with native RetryPolicy + DeadLetterConfig (Option A)

resource "aws_sqs_queue" "eventbridge_dlq" {
  name                      = "orders-eventbridge-dlq"
  message_retention_seconds = 1209600  # 14 days — max retention, buys time to investigate
  kms_master_key_id         = "alias/aws/sqs"  # encrypt at rest, security baseline
}

# Explicit resource policy — without this, DLQ writes fail silently
resource "aws_sqs_queue_policy" "dlq_policy" {
  queue_url = aws_sqs_queue.eventbridge_dlq.id
  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [{
      Sid       = "AllowEventBridgeSend",
      Effect    = "Allow",
      Principal = { Service = "events.amazonaws.com" },
      Action    = "sqs:SendMessage",
      Resource  = aws_sqs_queue.eventbridge_dlq.arn,
      Condition = {
        ArnEquals = { "aws:SourceArn" = aws_cloudwatch_event_rule.orders_created.arn }
      }
    }]
  })
}

resource "aws_cloudwatch_event_rule" "orders_created" {
  name          = "orders-created"
  event_pattern = jsonencode({ source = ["orders.service"] })
}

resource "aws_cloudwatch_event_target" "process_order" {
  rule      = aws_cloudwatch_event_rule.orders_created.name
  arn       = aws_lambda_function.process_order.arn
  target_id = "process-order-lambda"

  retry_policy {
    maximum_retry_attempts       = 3          # cap retries — don't let defaults hammer for 24h
    maximum_event_age_in_seconds = 3600       # give up after 1h, not the 86400s default
  }

  dead_letter_config {
    arn = aws_sqs_queue.eventbridge_dlq.arn   # this is per-TARGET, not per-rule
  }
}

And when messages do end up in the DLQ, here's how to inspect and redrive them without writing a custom Lambda — this uses the native SQS message-move task:


# Redriving a stuck DLQ back to the consumer queue (Option B pattern),
# using the native SQS message-move task — no custom Lambda needed.

# 1. Inspect what's sitting in the DLQ before redriving blindly
aws sqs get-queue-attributes \
  --queue-url "$DLQ_URL" \
  --attribute-names ApproximateNumberOfMessages

# Example output:
# {
#   "Attributes": {
#     "ApproximateNumberOfMessages": "47"
#   }
# }

# 2. Start the move task — requires awscli v2 >= 2.13.0 / botocore >= 1.31.0
aws sqs start-message-move-task \
  --source-arn "$DLQ_ARN" \
  --destination-arn "$MAIN_QUEUE_ARN" \
  --max-number-of-messages-per-second 5   # throttle to avoid re-triggering the original failure spike

# 3. Track progress of the redrive
aws sqs list-message-move-tasks --source-arn "$DLQ_ARN"

# Common gotcha: if the original failure was a bad payload (not transient),
# redriving just replays the same crash — check CloudWatch Logs error signature
# on a sample message BEFORE redriving 47 messages back into production.

Set CloudWatch alarms on InvocationsSentToDlq > 0 no matter which option you pick — EventBridge will never alert you on its own, and the whole point of the DLQ is to catch failures you'd otherwise never see. For the exact semantics of retry policies and dead-letter configs, the AWS EventBridge DLQ documentation and the SQS redrive policy docs are worth bookmarking before you ship either pattern.

GitHub Actions Reusable Workflow Environment Protection Checklist

Oleksandr Kuryzhev — Sat, 08 Aug 2026 07:01:58 +0000

Originally published on kuryzhev.cloud

Why this checklist

Last quarter one of our platform teams shipped a shiny .github/workflows/reusable-deploy.yml from a central repo. Every service team switched their pipelines over to call it via workflow_call within a week. It looked great — one workflow, consistent deploy logic, less duplicated YAML across forty repos. Then someone noticed a "protected" production deploy had gone out with no approvals at all, no wait timer, nothing. The environment protection rules reusable workflows are supposed to enforce simply weren't there.

The root cause is something GitHub doesn't explain clearly anywhere obvious: protection rules apply to the job that references an environment, evaluated against the caller repo's settings — not the repo that hosts the reusable workflow. If the consumer repo never created an environment named exactly production, GitHub just auto-creates one on first run, with zero reviewers, zero branch policy, zero wait timer. It doesn't error. It doesn't warn. It deploys.

This causes two failure classes we've seen repeatedly. First, secrets over-exposure — teams reach for secrets: inherit because it's convenient, and suddenly the reusable workflow has access to every secret the caller repo owns, including ones it never needed. Second, silent unprotected deploys — an environment name typo (Production vs production) creates a brand new, completely open environment instead of failing the run. We built this checklist after auditing 40+ repos and finding both problems live in prod.

The checklist (numbered)

Run through this before you trust any reusable workflow + environment combo touching production.

Environment exists in the calling repo, not just the source repo. Check Settings → Environments in the consumer repo itself. If it's missing, GitHub will silently create an unprotected one on the first run.
Environment name is case-exact. environment: Production in the workflow input vs production in repo settings creates two different environments. Diff them character by character if you're paranoid — we now are.
Permissions declared explicitly per job. The top-level permissions: block from the caller does not flow into the reusable workflow's jobs. Declare contents: read, id-token: write, etc. inside the reusable workflow itself, especially for OIDC-based cloud auth.
No secrets: inherit. Replace it with an explicit mapping like secrets: { DEPLOY_TOKEN: ${{ secrets.PROD_DEPLOY_TOKEN }} }. Inherit passes everything, used or not — a bigger blast radius than most teams realize until an incident review forces them to enumerate it.
Required reviewers list is populated and sane. Max is 6 users/teams per environment — if you're at the cap, that's usually a sign the team is too big for a single gate.
Wait timer value is intentional. Valid range is 0–43200 minutes (30 days). We've found wait timers left at defaults nobody remembers setting.
Deployment branch policy isn't "all branches." For prod, it should be "protected branches only" or a tight glob like release/*.
Outputs are declared under workflow_call.outputs. Referencing an undeclared output returns an empty string with no error — a silent failure that's brutal to debug at 2am.
Concurrency group is set explicitly inside the reusable workflow. It is not inherited from the caller. Skip this and you can get two parallel deploys hitting the same environment.
Nesting depth ≤4, total reusable calls ≤20 per run. Exceed either and you get "The workflow is not valid... too many levels of nested workflows" — usually discovered mid-incident, not in review.
Self-hosted runner labels passed as inputs. runs-on can't be overridden directly by the caller; interpolate it via ${{ inputs.runner_label }}.

Here's a working example that gets most of this right — pin the version, don't call @main:


# .github/workflows/reusable-deploy.yml (lives in platform repo, called by consumers)
on:
  workflow_call:
    inputs:
      env_name:
        required: true
        type: string
      runner_label:
        required: false
        type: string
        default: ubuntu-latest
    secrets:
      DEPLOY_TOKEN:          # must be explicitly declared to receive it via non-inherit calls
        required: true
    outputs:
      deployment_url:
        description: "URL of the deployed environment"
        value: ${{ jobs.deploy.outputs.url }}

permissions:
  contents: read             # explicit — do NOT rely on caller's permissions being inherited

concurrency:
  group: deploy-${{ inputs.env_name }}   # prevents parallel deploys to same environment
  cancel-in-progress: false

jobs:
  deploy:
    runs-on: ${{ inputs.runner_label }}
    environment: ${{ inputs.env_name }}   # protection rules evaluated against CALLER repo's env
    outputs:
      url: ${{ steps.set_url.outputs.url }}
    steps:
      - uses: actions/checkout@v4

      - name: Deploy
        env:
          DEPLOY_TOKEN: ${{ secrets.DEPLOY_TOKEN }}
        run: |
          echo "Deploying to ${{ inputs.env_name }} on ${{ inputs.runner_label }}"
          # actual deploy command here

      - id: set_url
        run: echo "url=https://${{ inputs.env_name }}.example.com" >> "$GITHUB_OUTPUT"

---
# .github/workflows/ci.yml (consumer repo — this is where the environment must exist)
on: [push]

jobs:
  call-deploy:
    uses: my-org/platform-workflows/.github/workflows/reusable-deploy.yml@v1.4.0  # pin, don't use @main
    with:
      env_name: production        # must exactly match an environment created in THIS repo's settings
      runner_label: ubuntu-latest
    secrets:
      DEPLOY_TOKEN: ${{ secrets.PROD_DEPLOY_TOKEN }}   # explicit, not `inherit`

Commonly missed items

These pass a five-minute review but bite you during an actual incident.

Watch out for: fork-originated pull requests. If your workflow triggers on pull_request from a fork, GitHub withholds environment secrets entirely — regardless of approvals — even if the job references a protected environment. We spent an hour once debugging a "missing secret" error that turned out to be expected fork behavior. If you need this to work, switch to pull_request_target or split the deploy step into a separate workflow_run trigger. See the GitHub Actions docs on reusable workflows for the exact secret-passing rules.

Watch out for: cross-org or cross-repo reusable workflow calls. Teams often check the environments configured in the repo hosting the reusable workflow (the platform repo) and assume that's what's enforced. It isn't. GitHub evaluates protection against the caller repo's environment of the same name. I've seen teams audit the wrong repo entirely and walk away confident nothing was wrong.

Approval fatigue is the third trap, and it's a human one, not a config one. When workflow A calls workflow B, and B references an environment, the approval UI at two levels of nesting doesn't clearly surface which environment is actually being targeted. Reviewers click "approve" without registering it's prod. Fix this by naming jobs explicitly — name: Deploy to ${{ inputs.env_name }} — so the environment shows up in the run summary, not just buried in YAML.

One thing worth knowing: jobs paused waiting on approval don't burn billable Actions minutes. So there's no cost excuse for skipping gates — it's effectively free insurance against exactly this class of incident.

Automation ideas

Manual review doesn't scale past a handful of repos, so we automated most of this checklist.

First, codify environments with Terraform instead of clicking through Settings → Environments. The github_repository_environment and github_repository_environment_deployment_policy resources make reviewers, wait timers, and branch policy diffable in a PR — no more "who changed the reviewer list and when."

Second, run a scheduled audit across the org. This catches drift between what Terraform says should exist and what's actually configured (someone always clicks around in the UI eventually):


# audit-environments.sh — checks all repos in an org for weak/missing protection rules
ORG="my-org"

for repo in $(gh repo list "$ORG" --limit 200 --json name -q '.[].name'); do
  envs=$(gh api "repos/$ORG/$repo/environments" --jq '.environments[].name' 2>/dev/null)

  for env in $envs; do
    reviewers=$(gh api "repos/$ORG/$repo/environments/$env" \
      --jq '.protection_rules[] | select(.type=="required_reviewers") | .reviewers | length')

    if [ -z "$reviewers" ] || [ "$reviewers" -eq 0 ]; then
      echo "⚠️  $repo/$env has NO required reviewers configured"
    fi
  done
done

# Example output:
# ⚠️  billing-service/production has NO required reviewers configured
# ⚠️  legacy-cron/prod has NO required reviewers configured

Third, gate this at PR time, not after merge. A custom actionlint plugin or a small Python script can fail CI if a reusable workflow call uses secrets: inherit or a called workflow omits an explicit permissions: block. We wired this into our pre-merge checks alongside the broader CI/CD checklist we use for release gates, and it's caught at least three risky PRs before they hit main.

None of this is exotic tooling. It's Terraform, a bash loop, and a linter rule — the same pattern I'd apply to any environment protection rules reusable workflows setup where "we'll just review it manually" is the thing that eventually fails at 2am.

Jenkins Shared Library Structure: vars/ vs src/ for Pipelines

Oleksandr Kuryzhev — Fri, 07 Aug 2026 07:02:09 +0000

Originally published on kuryzhev.cloud

When you face this choice

The Jenkins shared library structure question usually shows up the moment your third team copies a Jenkinsfile from your first team. You've got five repos now, each with a nearly identical stage('Build Docker') block, and someone just fixed a registry login bug in repo #2 without touching repos #1, #3, #4, or #5. Nobody notices until a deploy fails three weeks later with a stale credential error nobody can reproduce locally.

That's the trigger. Once you've got 3+ pipelines duplicating the same logic, you extract it into a shared library — and then you hit a fork in the road that every team using the Pipeline: Shared Groovy Libraries plugin eventually hits: do you put your logic in vars/*.groovy as Global Variables, or do you push it into src/**/*.groovy as real Groovy classes and keep vars/ as a thin entry point?

This isn't a cosmetic decision. I've seen teams treat it like a style preference and regret it within a year. The structure you pick determines whether you can unit test your pipeline logic, how much blast radius a bad commit has across every consuming Jenkinsfile, and whether onboarding a fifth or tenth team is a five-minute conversation or a week of confused Slack threads. I've built both, maintained both in production, and I have a strong opinion — which I'll get to.

Option A — vars/-only convention (Global Variables)

The vars/-only approach means every reusable step — buildDockerImage(), notifySlack(), deployToK8s() — is a Groovy script sitting directly in vars/ with a call() method. It's the path of least resistance, and honestly, it's how most shared libraries start.

Pros: zero boilerplate. You write vars/deployApp.groovy, define call(Map config), and it's immediately usable as deployApp(env: 'staging') in any Jenkinsfile that imports the library. There's a clean 1:1 mapping between the file name and the pipeline step name, which makes the codebase readable to anyone who already knows Jenkinsfile syntax — no Java/Groovy OOP background required. For prototyping a new shared step, this is the fastest option by far. I've written a working vars/ script in under ten minutes more times than I can count.

Cons: there's no real object-oriented design here. No inheritance, no interfaces, no dependency injection. If two scripts need to share state, you end up hacking it through env variables or global maps passed around like hot potatoes. Worse — unit testing a vars/ script means mocking the entire pipeline DSL (sh, withCredentials, input, all of it), which gets painful fast once the logic branches more than a few times.

Failure mode at scale: I watched a vars/deployApp.groovy grow to 400+ lines because every new requirement — canary logic, rollback, Slack notifications, feature flags — got bolted onto the same call() method. It became the exact monolith the team extracted the library to avoid. Merge conflicts came back with a vengeance, just one directory level up from where they started.

Option B — src/-driven class structure with thin vars/ wrappers

The alternative treats the shared library like an actual codebase: real classes in src/org/pkgname/, with vars/ reduced to a handful of lines that instantiate and delegate. Think DockerBuilder, SlackNotifier, K8sDeployer — each a standalone class with constructor-injected steps context.

Pros: this is where testability actually becomes possible. With JenkinsPipelineUnit 1.19+, or Spock if your team prefers it, you can unit test a DockerBuilder class in complete isolation — no live Jenkins master, no waiting for a build queue. Constructor injection means you're not fighting global state; each class gets exactly the context it needs. Code review also gets dramatically better: a diff to a 40-line class method is something a reviewer can actually reason about, versus a diff buried in a 400-line script.

Cons: the learning curve is real for teams that don't have a Java/Groovy background. You'll also run into CPS transformation gotchas — Jenkins Pipeline runs Groovy through a continuation-passing-style transform for durability across restarts, and classes that don't play nice with it throw java.io.NotSerializableException the moment a build pauses at an input step and Jenkins needs to serialize the whole call stack. The fix is usually @NonCPS annotations or marking fields transient, but it's a debugging session most teams don't expect on day one. There are also just more files to navigate — a one-line config tweak might mean touching three files instead of one.

Where it pays off: once you're past 10+ consuming pipelines across multiple teams, the compile-time-ish safety of real classes and the ability to actually write tests before merging changes outweighs the extra navigation cost. I've seen this prevent at least two org-wide outages that a vars/-only structure would have shipped straight to every pipeline simultaneously.

Decision matrix

Here's how I actually score this when a team asks me to make the call. Weight these against your real numbers, not aspirational ones.

Criteria	vars/-only wins when...	src/-driven wins when...
Team size	1-2 teams, single owner	3+ teams sharing the library
CI job count	< 5 consuming Jenkinsfiles	> 10 consuming Jenkinsfiles
Need for unit tests	Low — changes are simple, low risk	High — compliance, audit trail required
Groovy/OOP proficiency	Team knows Jenkinsfile syntax only	Team comfortable with classes, packages
Versioning strategy	Loose, moving fast, prototype phase	Strict semver tags, changelog discipline

Most teams I've worked with don't land on a pure extreme — and that's fine. The realistic middle ground is what most shared libraries look like after 12 months in production: a handful of trivial one-liner steps stay in vars/ (things like notifySlack() that genuinely don't need a class), while anything with branching logic, external dependencies, or multi-step state gets pushed into src/. Pure vars/-only libraries rarely survive past the 400-line monolith stage. Pure src/-only libraries — with zero thin wrappers — are rare because someone always needs a quick throwaway step and doesn't want to write a class for it.

My pick

I'll say it plainly: thin vars/ wrappers (10-20 lines, no logic beyond instantiation and delegation) backed by real src/ classes is the only structure I recommend for anything beyond a two-week prototype. It gives you the readable entry point everyone expects from a Jenkinsfile, without letting logic sprawl into an untestable monolith. Below is the layout and code I actually use.

// --- Directory structure (comment-only reference) ---
// ci-shared-lib/
// ├── vars/
// │   └── buildDockerImage.groovy      <- thin entrypoint, calls into src/
// ├── src/
// │   └── com/kuryzhev/ci/
// │       └── DockerBuilder.groovy     <- actual logic, unit-testable
// └── resources/
//     └── com/kuryzhev/ci/templates/Dockerfile.tpl

// ===== vars/buildDockerImage.groovy =====
def call(Map config = [:]) {
    // 'this' is the pipeline script context — pass it explicitly to the class
    def builder = new com.kuryzhev.ci.DockerBuilder(this, config)
    builder.build()
    builder.push()
}

// ===== src/com/kuryzhev/ci/DockerBuilder.groovy =====
package com.kuryzhev.ci

class DockerBuilder implements Serializable {
    def steps          // injected pipeline context (sh, echo, etc.)
    String imageName
    String registry
    String tag

    DockerBuilder(steps, Map config) {
        this.steps = steps
        this.imageName = config.imageName ?: error('imageName is required')
        this.registry  = config.registry  ?: 'registry.kuryzhev.cloud'
        this.tag       = config.tag       ?: steps.env.BUILD_NUMBER
    }

    void build() {
        // libraryResource pulls the templated Dockerfile from resources/
        def dockerfile = steps.libraryResource('com/kuryzhev/ci/templates/Dockerfile.tpl')
        steps.writeFile file: 'Dockerfile.generated', text: dockerfile
        steps.sh "docker build -t ${registry}/${imageName}:${tag} -f Dockerfile.generated ."
    }

    void push() {
        // credentials() binding — never hardcode registry creds in resources/
        steps.withCredentials([steps.usernamePassword(
            credentialsId: 'docker-registry-creds',
            usernameVariable: 'DOCKER_USER',
            passwordVariable: 'DOCKER_PASS'
        )]) {
            steps.sh "echo \$DOCKER_PASS | docker login ${registry} -u \$DOCKER_USER --password-stdin"
            steps.sh "docker push ${registry}/${imageName}:${tag}"
        }
    }
}

// ===== Consuming Jenkinsfile =====
@Library('ci-shared-lib@v2.3.1') _   // pinned version, NOT @main

pipeline {
    agent any
    stages {
        stage('Build & Push') {
            steps {
                buildDockerImage(imageName: 'checkout-service', tag: env.GIT_COMMIT.take(7))
            }
        }
    }
}

Two non-negotiables come with this pick. First: pin the library version with git tags — @Library('ci-shared-lib@v2.3.1') _ — and never reference @main or @master in a production Jenkinsfile. I've watched a single bad commit to a shared library silently break every consuming pipeline org-wide within minutes, simply because nobody had pinned a version. This one habit alone prevents most "why did prod break at 2am" incidents. Second: mandate JenkinsPipelineUnit tests before merge. Without tests, a shared library isn't infrastructure — it's untested code with an organization-wide blast radius.

Here's the test that would have caught a regression in DockerBuilder before it ever reached a consuming pipeline:

// ===== test/groovy/com/kuryzhev/ci/DockerBuilderTest.groovy =====
// Run with JenkinsPipelineUnit 1.19 — no live Jenkins master needed
import com.lesfurets.jenkins.unit.BasePipelineTest
import com.kuryzhev.ci.DockerBuilder
import org.junit.Test
import static org.junit.Assert.assertTrue

class DockerBuilderTest extends BasePipelineTest {

    @Test
    void 'build generates correct docker build command'() {
        def config = [imageName: 'checkout-service', tag: 'abc1234']
        def builder = new DockerBuilder(binding.getVariable('steps'), config)

        builder.build()

        // helper.callStack captures every mocked step invocation
        def shCalls = helper.callStack.findAll { it.methodName == 'sh' }
        assertTrue(shCalls.any { it.args[0].toString().contains('checkout-service:abc1234') })
    }

    @Test(expected = Exception)
    void 'missing imageName throws error'() {
        new DockerBuilder(binding.getVariable('steps'), [:])
    }
}

// Expected console output on `mvn test` / `gradle test`:
//
// DockerBuilderTest > build generates correct docker build command PASSED
// DockerBuilderTest > missing imageName throws error PASSED
//
// BUILD SUCCESSFUL in 2s

Watch out for: if you skip the constructor injection pattern and let a src/ class call sh or env directly without passing steps in, you'll hit groovy.lang.MissingPropertyException: No such property: steps the first time it's actually invoked inside a pipeline — it works fine in isolation until it doesn't. And if a field on your class holds something non-serializable across an input or long-running sh step, expect NotSerializableException the moment Jenkins tries to persist pipeline state mid-build. Mark it transient or move the logic into a @NonCPS method.

One more thing worth flagging on the security side: don't dump credentials into resources/ files bundled with the library repo. That's a fast way to leak secrets into git history permanently. Use Jenkins' credentials() binding every time, as shown in the push() method above. And when the sandbox rejects an unapproved Groovy method call, resist the urge to blanket-approve everything in Manage Jenkins → In-process Script Approval — audit each approval individually, especially anything touching @Grab or external classloaders.

For teams sitting at more than 50 builds a day, also be deliberate about how the library loads. Loading a heavyweight @Library implicitly at the top of every Jenkinsfile means a fresh checkout and Groovy compile on every single build — that adds up. Reserve implicit loading for libraries genuinely used everywhere, and lazy-load the rest with the library 'name' step inside the specific stage that needs it. We've covered similar CI/CD pipeline hardening patterns over on kuryzhev.cloud's CI/CD category if you want more war stories from this side of the pipeline.

Get the shared library structure right once, pin your versions, write the tests, and this stops being a recurring 2am problem. Get it wrong, and every team you onboard multiplies the risk instead of the value. Full reference on the plugin mechanics is in the official Jenkins shared libraries documentation — worth reading end to end before you commit to a structure.

Helm Rollback Strategy: Safe Values Promotion in Production

Oleksandr Kuryzhev — Thu, 06 Aug 2026 07:01:43 +0000

Originally published on kuryzhev.cloud

One bad helm upgrade and your production values file is history — literally — unless you have a helm rollback strategy that survives concurrent deploys, stateful side effects, and plain human error. I've watched a team roll back to what they *thought* was the last good revision, only to land on a "superseded" release with a broken feature flag still baked in. Here's what we changed after that incident, and what we now enforce before every promotion.

Pin exact chart versions and snapshot values before every promotion

The biggest source of rollback confusion is not knowing what was actually deployed. values.yaml in your chart repo is not the source of truth — the live cluster state can drift from it after hotfixes, manual --set overrides, or an emergency patch someone forgot to commit. Before every helm upgrade, run helm get values <release> -o yaml and store it alongside your CI artifacts, tagged with the git commit SHA and the release revision number. If you can't answer "what values were live five minutes ago" in under 30 seconds, your rollback plan is theoretical, not real.

Roll back by revision number, never by "previous" assumptions

Always run helm history <release> first and confirm the exact revision status before touching anything. helm rollback <release> 0 means "go to previous revision" — and that's dangerous the moment two deploys happen close together, because "previous" might be a failed or superseded revision, not the last known-good one. I stopped trusting revision 0 after a rollback landed us on a revision that had already been marked superseded due to a race between two pipeline runs.

# Wrong: assumes "previous" is safe
helm rollback payments-api 0 -n prod

# Right: explicit revision confirmed via helm history first
helm history payments-api -n prod --max 5
helm rollback payments-api 41 -n prod --wait --timeout 5m

Watch out: without --wait --timeout 5m, Helm — and your CI/CD pipeline — will report success the instant the rollback command is accepted, not when pods are actually healthy. That's how you get a "successful" deploy alert five minutes before the real outage.

Increase --history-max deliberately, but know the storage tradeoff

Helm's default --history-max=10 is fine for low-churn services, but if you've had a bad deploy streak, ten revisions can disappear fast. Each revision is stored as a Kubernetes Secret in the release namespace (sh.helm.release.v1.<name>.v<rev>), so bumping history-max isn't free — on a release doing 50+ deploys a day, that's real etcd bloat and slower helm list/helm history calls. We settled on 15–20 for critical services and purge history entirely on decommissioned releases with helm uninstall --keep-history=false.

Diff before you rollback — don't trust memory

Install the helm-diff plugin (v3.9+) and run helm diff rollback <release> <revision> before you commit to anything. Combine it with helm rollback --dry-run --debug to surface hook ordering or immutable field issues before they hit the cluster. Skipping this because "it's just a rollback" is the mistake I see most often — values promoted since that revision (secrets, replica counts, resource limits) can silently regress and nobody notices until the pods start OOMKilling.

Rollback doesn't undo everything — watch stateful side effects

This is the misconception that bites teams hardest: Helm rollback only reverts the Kubernetes objects Helm tracks. It does not rerun pre-upgrade hooks, does not undo a database migration, and does not restore PVC data. If your Job has helm.sh/hook: pre-upgrade with hook-delete-policy: before-hook-creation, a rollback can leave that Job orphaned in the namespace — audit it manually, don't assume Helm cleaned up after itself. Treating a Helm rollback as equivalent to a full git revert for the entire system is how migrations end up permanently mismatched with the application code they were supposed to support.

Promote values through environments, not through manual edits

Use layered values files — values-base.yaml, values-staging.yaml, values-prod.yaml — merged in a fixed -f order, and never inline --set flags in production pipelines. Promotion should mean copying a validated, already-tested values file from staging to prod through a CI step with a visible diff, not someone re-typing numbers from memory on a Friday afternoon.

# values-base.yaml — shared defaults, no secrets
replicaCount: 2
image:
  repository: registry.internal/payments-api
  tag: "1.8.2"          # pinned explicitly, promoted via CI, never "latest"

resources:
  requests:
    cpu: 250m
    memory: 256Mi

---
# values-prod.yaml — env-specific overrides
replicaCount: 6
resources:
  requests:
    cpu: 500m
    memory: 512Mi
secretsRef:
  # Real values live in a SOPS-encrypted file, referenced here, never inlined
  name: payments-api-secrets-sops

Keep secrets out of these files entirely. Use SOPS or the External Secrets Operator pattern so a rollback never resurrects an old plaintext credential sitting in Secret history from three revisions ago — that's a real security consideration, not a hypothetical one.

Test the rollback path itself, not just the deploy path

Run rollback drills in a staging namespace on a schedule — monthly is enough — to confirm helm rollback actually restores service without someone SSH-ing in to patch things manually. Lock CI/CD concurrency per release with a mutex or a concurrency: group in GitHub Actions so a rollback and a fresh deploy can never race against each other. The worst time to discover your rollback path fails on an immutable field error (looking at you, Deployment selector changes) is during a live incident at 2am.

#!/usr/bin/env bash
# rollback-with-snapshot.sh
# Safe Helm rollback workflow: snapshot current values, diff, confirm, rollback.

set -euo pipefail

RELEASE="payments-api"
NAMESPACE="prod"
SNAPSHOT_DIR="./values-snapshots"

mkdir -p "$SNAPSHOT_DIR"

echo "==> Fetching release history"
helm history "$RELEASE" -n "$NAMESPACE" -o json > "${SNAPSHOT_DIR}/history-$(date +%s).json"

# List the last 5 revisions for operator review
helm history "$RELEASE" -n "$NAMESPACE" --max 5

read -rp "Enter target revision number to rollback to: " TARGET_REV

echo "==> Snapshotting CURRENT live values before touching anything"
helm get values "$RELEASE" -n "$NAMESPACE" -o yaml \
  > "${SNAPSHOT_DIR}/current-before-rollback-$(date +%Y%m%d%H%M).yaml"

echo "==> Snapshotting TARGET revision values for comparison"
helm get values "$RELEASE" -n "$NAMESPACE" --revision "$TARGET_REV" -o yaml \
  > "${SNAPSHOT_DIR}/target-rev-${TARGET_REV}.yaml"

echo "==> Diffing current vs target (requires helm-diff plugin)"
helm diff rollback "$RELEASE" "$TARGET_REV" -n "$NAMESPACE" || true

read -rp "Proceed with rollback to revision ${TARGET_REV}? (yes/no) " CONFIRM
if [[ "$CONFIRM" != "yes" ]]; then
  echo "Aborted."
  exit 1
fi

echo "==> Executing rollback"
helm rollback "$RELEASE" "$TARGET_REV" \
  -n "$NAMESPACE" \
  --wait \
  --timeout 5m \
  --history-max 15

echo "==> Post-rollback verification"
helm status "$RELEASE" -n "$NAMESPACE"
kubectl get pods -n "$NAMESPACE" -l app.kubernetes.io/instance="$RELEASE"

echo "Done. Snapshots saved in ${SNAPSHOT_DIR}/"

If you're running ArgoCD or Flux, this whole helm rollback strategy shifts: prefer git revert plus a sync over a manual helm rollback, so you don't end up with git and cluster state disagreeing about what "current" means. We cover more of that gitops-vs-manual tradeoff in our DevOps_DayS archive if you want the longer version. Whatever pattern you pick, the goal is the same: a rollback should be boring, tested, and never the first time you've actually run the command in anger.

EKS IRSA Setup: Fix S3 AccessDenied From Node IAM Roles

Oleksandr Kuryzhev — Wed, 05 Aug 2026 07:01:44 +0000

Originally published on kuryzhev.cloud

The problem we hit

It was 3:40am when the pager went off. A backup CronJob that had been running fine for months suddenly started throwing AccessDenied on s3:PutObject. Nothing had changed in the application code, no deploy had gone out, and the IAM policy attached to the role looked exactly like it always had. The only thing that had happened was a "routine" node group rotation earlier that evening — new nodes, same AMI family, same everything, or so we thought.

The on-call engineer's first instinct was to check the IAM role tied to the service account. Policy looked correct: s3:PutObject on the right bucket ARN. But the pod kept failing. Digging into CloudTrail, the actual caller wasn't the intended IRSA role at all — it was arn:aws:sts::<acct>:assumed-role/eksctl-nodegroup-role/i-0abcd.... The pod was quietly falling back to the EC2 instance profile attached to the node, not the role we thought we'd scoped down months ago.

That's when the real scope of the problem became clear. The node's IAM role was shared across twelve other unrelated pods on the same node group — logging sidecars, a metrics exporter, a couple of internal APIs. All of them had been running with broad, node-level permissions the entire time, and nobody noticed because everything "worked." The backup job breaking was almost a lucky accident — it forced us to look at something that had been a ticking time bomb for a long time. Overprivileged access doesn't announce itself; it just sits there until an incident makes you look.

Why it happens

Without a proper EKS IRSA setup, pods don't get their own AWS identity — they inherit whatever the EC2 instance profile on the node grants. That's the default behavior of the AWS SDK credential chain: if no explicit credentials are found, it walks up to instance metadata and grabs the node's role. It's broad, it's shared across every pod scheduled on that node, and it's nearly impossible to trace a specific API call back to a specific workload in CloudTrail unless you know exactly what to look for.

IRSA (IAM Roles for Service Accounts) fixes this by binding a Kubernetes ServiceAccount to a specific IAM role via OIDC federation. Here's the chain: the EKS cluster has an OIDC identity provider registered in IAM. A ServiceAccount gets annotated with eks.amazonaws.com/role-arn. When a pod using that ServiceAccount starts, a mutating webhook injects a projected token volume and two environment variables — AWS_WEB_IDENTITY_TOKEN_FILE and AWS_ROLE_ARN. The AWS SDK reads that token, calls sts:AssumeRoleWithWebIdentity, and gets short-lived credentials scoped to exactly that role.

In our incident, two things had gone wrong. First, the cluster had been recreated via Terraform months earlier and the OIDC provider registration hadn't been re-associated — a classic drift issue that nobody caught because most workloads still "worked" using node-role fallback. Second, and this is the gotcha that really got us: the trust policy's Condition block had a subtly wrong sub value from an earlier copy-paste — the namespace didn't match exactly. When the AssumeRoleWithWebIdentity call failed silently, several SDKs just fell back to the default credentials chain instead of erroring loudly. That's the dangerous part — it doesn't crash, it just quietly uses the wrong identity.

The fix (with code)

The fix has three parts: confirm the OIDC provider is actually registered, write a trust policy scoped to an exact namespace and service account (no wildcards, ever), and attach a least-privilege policy instead of reaching for AmazonS3FullAccess as a "temporary" patch. We ran this on EKS 1.27 with eksctl 1.147.0 and aws-cli 2.15.x — worth noting that anything older than aws-cli 2.9 doesn't play nicely with IRSA debugging defaults.

#!/usr/bin/env bash
# irsa-setup.sh — reproduce the fix from the incident: scoped IAM role for a single ServiceAccount
set -euo pipefail

CLUSTER_NAME="prod-eks-01"
NAMESPACE="backup-jobs"
SERVICE_ACCOUNT="s3-backup-sa"
BUCKET_ARN="arn:aws:s3:::acme-backup-bucket"
ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)

# 1. Confirm OIDC provider exists (this was the actual root cause in our incident —
#    the provider was missing after a cluster recreate via Terraform)
OIDC_ISSUER=$(aws eks describe-cluster --name "$CLUSTER_NAME" \
  --query "cluster.identity.oidc.issuer" --output text)
OIDC_ID=$(echo "$OIDC_ISSUER" | sed 's|https://oidc.eks.*/id/||')

if ! aws iam list-open-id-connect-providers | grep -q "$OIDC_ID"; then
  echo "OIDC provider missing — associating now"
  eksctl utils associate-iam-oidc-provider --cluster "$CLUSTER_NAME" --approve
fi

# 2. Write a trust policy scoped to exact namespace:serviceaccount (no wildcards)
cat > trust-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": { "Federated": "arn:aws:iam::${ACCOUNT_ID}:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/${OIDC_ID}" },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": {
        "oidc.eks.us-east-1.amazonaws.com/id/${OIDC_ID}:sub": "system:serviceaccount:${NAMESPACE}:${SERVICE_ACCOUNT}",
        "oidc.eks.us-east-1.amazonaws.com/id/${OIDC_ID}:aud": "sts.amazonaws.com"
      }
    }
  }]
}
EOF

# 3. Create the role — least privilege, no AmazonS3FullAccess "temporary fix"
aws iam create-role \
  --role-name irsa-s3-backup \
  --assume-role-policy-document file://trust-policy.json

cat > s3-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["s3:PutObject", "s3:GetObject"],
    "Resource": "${BUCKET_ARN}/*"
  }]
}
EOF

aws iam put-role-policy \
  --role-name irsa-s3-backup \
  --policy-name s3-backup-put \
  --policy-document file://s3-policy.json

echo "Role ARN: arn:aws:iam::${ACCOUNT_ID}:role/irsa-s3-backup"

Once the role exists, the ServiceAccount needs the annotation applied before the pod is created — this bit us too. The mutating webhook only injects the token env vars at admission time, so annotating an existing ServiceAccount and doing a rolling update won't help until the pods are fully recreated.

# service-account.yaml — annotate BEFORE the pod is created; webhook injects env vars at admission time
apiVersion: v1
kind: ServiceAccount
metadata:
  name: s3-backup-sa
  namespace: backup-jobs
  annotations:
    eks.amazonaws.com/role-arn: "arn:aws:iam::123456789012:role/irsa-s3-backup"
---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: s3-backup-job
  namespace: backup-jobs
spec:
  schedule: "0 3 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: s3-backup-sa  # must match trust policy "sub" exactly
          containers:
          - name: backup
            image: amazon/aws-cli:2.15.0
            command: ["aws", "s3", "cp", "/data/dump.tar.gz", "s3://acme-backup-bucket/"]
          restartPolicy: OnFailure

# Verification after apply:
# kubectl exec -it <pod> -n backup-jobs -- aws sts get-caller-identity
# Expected output:
# {
#   "UserId": "AROAEXAMPLE:botocore-session-1234567890",
#   "Account": "123456789012",
#   "Arn": "arn:aws:sts::123456789012:assumed-role/irsa-s3-backup/botocore-session-1234567890"
# }
# If Arn instead shows "assumed-role/eksctl-nodegroup-role/..." — annotation wasn't
# applied before pod creation, or SA name/namespace mismatch in trust policy.

Also worth checking manually: the token file lives at /var/run/secrets/eks.amazonaws.com/serviceaccount/token inside the pod. If that file is missing, the webhook never fired — usually because the annotation went on too late. I stopped trusting "it should be fine, we applied the annotation" the moment I saw this happen twice in the same week across two different teams.

Prevention checklist

This class of incident is entirely preventable with a few disciplined habits baked into how you manage IAM for EKS workloads.

One IAM role per workload, never shared. Reusing a role across unrelated pods is exactly how a backup job's misconfiguration turns into a fleet-wide privilege audit. Run aws iam list-roles | grep irsa periodically and map each role back to a single namespace/service account.
Scope trust policies exactly — no wildcards. The Condition block must use StringEquals with the precise sub value: system:serviceaccount:<namespace>:<sa-name>. A typo or wrong namespace doesn't throw a loud error — it silently falls back to node credentials in some SDKs, which is worse than an outright failure.
Detect OIDC provider drift automatically. After any cluster upgrade or recreation, verify the provider is still registered with aws eks describe-cluster --name <cluster> --query "cluster.identity.oidc.issuer" against your IAM provider list. Wire this into a CI job or a Terraform plan check so it never depends on someone remembering.
Never leave "temporary" broad policies in place. If you attach AmazonS3FullAccess to unblock an incident at 4am, put a ticket and a revert date on it immediately. It will not get revisited otherwise.
Segregate roles by environment. Don't reuse the same trust policy pattern across dev, staging, and prod namespaces — that's how privilege leaks across environments during a "quick copy-paste."
Use a tested module instead of hand-rolled JSON. The terraform-aws-modules IRSA module generates trust policies correctly and removes an entire class of typo bugs.

Getting the EKS IRSA setup right isn't a one-time task — it's an ongoing discipline. We now run a quarterly audit that cross-references every ServiceAccount annotation against its IAM role and trust policy, and we treat any mismatch as a P2. For more patterns on locking down cloud IAM without breaking workloads, check out the DevOps_DayS archive — there's a related piece on Terraform-managed Vault secrets that pairs well with this.

For the official reference, AWS's own IAM roles for service accounts documentation is worth bookmarking — it's the source of truth for webhook behavior across EKS versions.

DEV Community: Oleksandr Kuryzhev

Kubernetes NetworkPolicy Checklist: Locking Down Namespace Isolation

Why this checklist

The checklist (numbered)

Commonly missed items

Automation ideas

Related

S3 Lifecycle Policy Mistakes That Quietly Inflate Your AWS Bill

Split lifecycle rules by prefix and tag, never one rule per bucket

Use Object Lock in Compliance mode for anything with a retention mandate

Transition logs aggressively, expire even more aggressively

Watch noncurrent version growth — it's the silent cost and compliance trap

Always add an abort-incomplete-multipart-upload rule

Lifecycle rules are not access control — pair them with deny policies

Validate before you ship — dry-run and inventory-audit every rule change

Related

Ansible Zero-Downtime Deployment: Draining the Load Balancer Right

The scenario

Prerequisites

Step 1 — Structure the playbook with serial and batching

Step 2 — Deregister from the load balancer before touching the host

Step 3 — Deploy, restart, and health-check before re-registering

Step 4 — Re-register and move to next batch

Verify and test

Closing

Related

Cloudflare Origin Hardening Checklist: Firewall, Bots, Strict SSL

Why this checklist

The checklist (numbered)

Commonly missed items

Automation ideas

Related

Nginx Rate Limiting vs API Gateway: Picking the Right Layer

When you face this choice

Option A: Native Nginx rate limiting (limit_req / limit_conn)

Option B: Gateway/edge-based rate limiting (Kong, Cloudflare, or OpenResty+Redis)

Decision matrix

My pick

Related

Docker BuildKit Cache Setup That Actually Speeds Up CI

What BuildKit Cache Actually Does Under the Hood

How People Use It Wrong

The Correct Approach

Advanced Patterns

Performance Notes

Related

RDS Backup Restore Testing: 7 Checks Before You Trust It

Automate the Restore, Don't Trust the "Available" Status

Restore Into an Isolated Network, Not Your Production VPC

Verify Point-in-Time Recovery, Not Just Snapshot Age

Validate Data Integrity After Restore, Not Just Connectivity

Test Cross-Region and Cross-Account Snapshot Copies for DR

Watch Snapshot Storage Costs During Test Cycles

Lock Down IAM and Encryption for Restore Operations

Related

EventBridge Retry Policy vs SQS DLQ: The Right Call in Production

When you face this choice

Option A: Native EventBridge RetryPolicy + DLQ (pros/cons)

Option B: SQS buffer queue as the real target (pros/cons)

Decision matrix

My pick

Related

GitHub Actions Reusable Workflow Environment Protection Checklist

Why this checklist

The checklist (numbered)

Commonly missed items

Automation ideas

Related

Jenkins Shared Library Structure: vars/ vs src/ for Pipelines

When you face this choice

Option A — vars/-only convention (Global Variables)

Option B — src/-driven class structure with thin vars/ wrappers

Decision matrix

My pick

Related

Helm Rollback Strategy: Safe Values Promotion in Production

Pin exact chart versions and snapshot values before every promotion

Roll back by revision number, never by "previous" assumptions

Increase --history-max deliberately, but know the storage tradeoff

Diff before you rollback — don't trust memory