DEV Community: Oleksandr Kuryzhev

Kubernetes DRA Extended Resources: GPUs Without Device Plugins

Oleksandr Kuryzhev — Sat, 07 Nov 2026 08:01:34 +0000

Originally published on kuryzhev.cloud

A platform team has 200 training manifests that all say nvidia.com/gpu: 1. They also want per-model GPU selection, MIG slices and shared devices, which the device plugin model cannot express. Rewriting every manifest to use ResourceClaims would be a large, slow and risky project. DRA extended resources exist to remove that trade-off.

Kubernetes v1.37 graduated DRA extended resources to GA, as covered in the v1.37 release announcement. It matters now because the migration path is finally stable: you can move GPU nodes from a device plugin to a Dynamic Resource Allocation (DRA) driver without touching workload YAML.

Short answer: install a DRA driver for your GPUs, create a DeviceClass that declares the extended resource name it should satisfy, and keep requesting nvidia.com/gpu in resources.limits. Roll it out node pool by node pool. It applies if you run Kubernetes 1.37+ and want DRA features, but skip it if a device plugin already does everything you need.

What DRA extended resources actually do

Classic extended resources work like this. A device plugin on each node tells the kubelet "I have 8 of nvidia.com/gpu", and the count shows up in node allocatable. The scheduler treats it as an opaque integer. It cannot tell an H100 from a T4, and it cannot see topology or partitions.

DRA replaces that integer with structured data. A driver publishes ResourceSlice objects describing real devices and their attributes. The scheduler allocates specific devices through ResourceClaim objects, using CEL selectors in a DeviceClass or claim.

The extended resources bridge connects the two worlds. A DeviceClass can name an extended resource, for example nvidia.com/gpu. When a pod asks for that name in its limits, the scheduler creates a claim on the pod's behalf, allocates a device from the matching class, and records the result in the pod status. Per the KEP and docs, the pod author sees no difference, while the platform team gets DRA's allocation model underneath.

The details (field names, status fields, driver support) are documented in the DRA documentation. Confirm them against your exact patch version with kubectl explain deviceclass.spec.

How people use it wrong

Running both stacks on the same GPUs. A typical failure is installing the DRA driver while the device plugin DaemonSet is still running. Both then advertise the same physical devices through different mechanisms, and the system can over-promise. Pods land on a node, one path says the GPU is free, and the container fails at start or two pods fight over a device. The documented guidance is that a device is managed by one mechanism at a time.

Rewriting everything to ResourceClaims on day one. Teams read "DRA" and convert all manifests to resourceClaims plus ResourceClaimTemplate. That is only needed for workloads that need selection or sharing. The bridge exists so most pods can stay unchanged.

Treating it as a drop-in for any cluster. Mixed-version node pools, older kubelets, or a driver that does not support the extended resource mapping will produce pods stuck in Pending. Check your driver's release notes, not just the Kubernetes version.

Forgetting the accounting side. Quotas, cost dashboards and alerts often key off node allocatable nvidia.com/gpu. Once a pool moves to DRA, that number may no longer be the source of truth. Verify how your quota and metrics tooling treat DRA-allocated devices before you cut over.

The correct approach

Migrate one labeled node pool at a time. Follow these steps:

Cordon and drain the pool.
Remove the device plugin from it. For example, add node affinity to the device plugin DaemonSet that excludes the pool's label.
Install the vendor DRA driver.
Create the DeviceClass.
Uncordon the nodes.

The NVIDIA driver is distributed as the nvidia-dra-driver-gpu Helm chart. Follow its README for current values, because chart options change between releases.

This DeviceClass maps the legacy resource name onto devices from the driver. The driver name gpu.nvidia.com is NVIDIA's; substitute your vendor's. The CEL expression is the part to check against your driver's published attributes.

apiVersion: resource.k8s.io/v1
kind: DeviceClass
metadata:
  name: gpu.example.com-default
spec:
  selectors:
  - cel:
      expression: device.driver == "gpu.nvidia.com"   # match only this driver's devices
  extendedResourceName: nvidia.com/gpu                # pods requesting this name get a DRA-backed claim

🧪 Lab — try this in your browser

Kubernetes Manifest Linter — Lint multi-document Kubernetes YAML for production readiness and security, with line:col diagnostics, severity grouping, a 0-100 score and fix hints.

The workload stays exactly as it was. Nothing in it mentions DRA:

apiVersion: v1
kind: Pod
metadata:
  name: gpu-smoke-test
spec:
  restartPolicy: Never
  containers:
  - name: cuda
    image: nvcr.io/nvidia/cuda:<tag>   # pick a CUDA base image matching your driver
    command: ["nvidia-smi", "-L"]
    resources:
      limits:
        nvidia.com/gpu: 1               # unchanged legacy request

To verify, check three things:

The driver is publishing devices. kubectl get resourceslices should list one or more slices per GPU node.
A claim was generated for the pod. kubectl get resourceclaims should show a claim in the pod's namespace in an allocated state.
The pod status records the claim. Run kubectl get pod gpu-smoke-test -o yaml and look for the extended resource claim status section.

Then kubectl logs gpu-smoke-test should print one GPU line from nvidia-smi -L.

Watch out: if the pod stays Pending, run kubectl describe pod and read the scheduler events first. The usual causes are a missing extendedResourceName on the DeviceClass or a CEL expression that matches nothing.

Advanced patterns

Once a pool runs on DRA, you can mix styles. Most pods keep the legacy request. The few that need a specific GPU model, a MIG profile or a shared device use an explicit ResourceClaimTemplate with a CEL selector on device attributes. Run kubectl get resourceslices -o yaml to see the attribute names your driver really publishes before writing selectors.

You can also define several DeviceClasses for different tiers, for example one that excludes older cards. Teams then get access through namespaces and RBAC on claim templates. That moves policy from node labels and taints into declarative objects the scheduler understands.

Multi-pod jobs are another fit. v1.37 also ships Workload/PodGroup gang scheduling in beta, so the pods of a distributed training job can be scheduled together instead of a partial set starting and holding devices. Because it is beta, check the release notes for its feature gates, API shape and how it interacts with DRA allocation before relying on it.

Watch out: the bridge only covers the simple "N devices of a kind" case. If you need a pod to combine a legacy request and rich selection on the same device, test that exact combination first. The docs list the supported interactions.

When not to bother: a small fleet of identical GPUs with whole-device allocation and no sharing needs gets little from DRA. You would add a driver, new API objects to debug and a new failure surface in exchange for features you do not use. The device plugin is still a fine choice there.

Performance notes

The costs are different from the device plugin's, not necessarily bigger. The scheduler now runs a DynamicResources plugin that evaluates CEL selectors against ResourceSlices, and the control plane stores slices and claims. Large GPU fleets with many short-lived pods generate a claim object per pod, so watch API server object counts and etcd size.

I won't quote numbers, because they depend on cluster size and driver. Measure instead:

In Prometheus, compare scheduler_pending_pods before and after migrating a pool.
Compare the scheduler plugin duration histogram scheduler_plugin_execution_duration_seconds, filtered to the DynamicResources plugin.
Track pod startup time from scheduled to running, since the kubelet must call the driver to prepare devices before the container starts. Your driver documents its own metrics for that step.

For rollback, keep the old device plugin manifest ready and migrate pools one at a time. Reverting a pool then means draining it, removing the driver and restoring the plugin. For related platform patterns, browse more posts on kuryzhev.cloud. The authoritative reference for allocation semantics, status fields and supported drivers is the Kubernetes DRA documentation.

CRA vulnerability reporting: a 24h and 72h runbook for small teams

Oleksandr Kuryzhev — Fri, 06 Nov 2026 08:02:12 +0000

Originally published on kuryzhev.cloud

The scenario: a Friday-evening exploit report

It is 18:40 on a Friday. A customer emails your two-person team a proof-of-concept for your firmware update daemon, with a link to a forum thread where someone is already using it. Before September 2026 that was a bad evening. Now it also starts a legal clock, because CRA vulnerability reporting has been mandatory since 11 Sep 2026. For actively exploited vulnerabilities, manufacturers of products with digital elements must send an early warning within 24 hours and a fuller notification within 72 hours. The Commission's overview is at digital-strategy.ec.europa.eu/en/policies/cra-reporting.

Large companies have legal and compliance staff for this. Small teams have whoever is on call. The deadlines count in hours, weekends included, so the runbook has to exist before the report arrives.

Short answer:

Write down who decides that a vulnerability is "actively exploited."
Record the moment you became aware.
Open a tracked issue that computes the 24h and 72h deadlines.
Prepare the early-warning and notification content as templates, so submitting is a copy-and-paste job.

This applies if you place products with digital elements on the EU market. Verify scope with counsel, because this article is engineering guidance, not legal advice.

Prerequisites

You need a few things in place before the runbook is worth anything:

A scope decision. List which of your products are "products with digital elements" under the CRA. Pure hosted SaaS is generally handled differently, and open-source stewards have lighter obligations. Check the regulation text (Regulation (EU) 2024/2847) and the Commission page above.
Access to the EU single reporting platform. The CRA routes reports through a single reporting platform established by ENISA. Submissions reach the CSIRT designated as coordinator and ENISA. Check the Commission page for current onboarding steps, and get your account created before an incident, not during one.
Two named people, a decider and a submitter, each with a deputy. One person cannot cover a 72-hour window.
An SBOM per release, and the gh CLI authenticated with issue-write rights (cli.github.com/manual).
Tooling: GNU date (Linux, or gdate on macOS), plus jq, syft and grype for the evidence step.

Cost to be aware of: this adds on-call duty and process weight. If you ship nothing to the EU market, skip it.

Step 1: Make intake findable and define "aware"

The clock runs from when you become aware of active exploitation, so the first job is a reachable intake channel:

Publish a SECURITY.md.
Publish a /.well-known/security.txt.
Enable GitHub's private vulnerability reporting (see GitHub security advisories docs).

A minimal security.txt served at https://example.com/.well-known/security.txt:

Contact: mailto:security@example.com
Expires: 2027-06-30T00:00:00.000Z
Policy: https://example.com/security-policy
Preferred-Languages: en
Canonical: https://example.com/.well-known/security.txt

Then put a definition in docs/cra-runbook.md. For example: "Awareness time is the timestamp when the decider records that credible evidence of exploitation exists (a vendor report, telemetry, a public exploit used in the wild), not when the first email arrived."

Whether that definition holds up legally is a question for your counsel. What matters operationally is that it is written down and timestamped, so nobody argues about it at 3 a.m.

Watch out for: an unmonitored security@ mailbox. If a report sits unread over a weekend, ask counsel whether that delay could count against you. Do not assume the clock waits for you to read it. Route the mailbox to the same pager as production alerts.

Step 2: Start the clock with one command

The cheapest control is a script that stamps the awareness time and creates a tracked issue with both deadlines. The labels must exist first:

gh label create cra-clock
gh label create sev-exploited

The script uses epoch arithmetic rather than date-string parsing, so the offsets are always exactly 24 and 72 hours.

#!/usr/bin/env bash
# cra-clock.sh <product> <cve-or-internal-id>
set -euo pipefail
PRODUCT="${1:?product}"; VULN="${2:?cve or internal id}"

FMT='+%Y-%m-%dT%H:%M:%SZ'
AWARE_EPOCH=$(date -u +%s)
AWARE=$(date -u -d "@${AWARE_EPOCH}" "$FMT")
T24=$(date -u -d "@$((AWARE_EPOCH + 86400))" "$FMT")    # early warning due
T72=$(date -u -d "@$((AWARE_EPOCH + 259200))" "$FMT")   # notification due

gh issue create \
  --title "CRA clock: ${VULN} (${PRODUCT})" \
  --label cra-clock --label sev-exploited \
  --assignee "@me" \
  --body "$(cat <<EOF
**Aware at:** ${AWARE}
**Early warning due (24h):** ${T24}
**Notification due (72h):** ${T72}

- [ ] Decider confirmed evidence of exploitation
- [ ] Member states where product is available listed
- [ ] Early warning submitted (record reference ID)
- [ ] Notification drafted: product, nature of vuln/exploit, mitigations
- [ ] Notification submitted (record reference ID)
- [ ] Corrective measure released; final report due within 14 days of that (confirm in Art. 14)
EOF
)"

An alias worth keeping in your shell profile: alias craclock='~/ops/cra-clock.sh'. Starting the clock takes seconds. GitHub records the issue's creation time, and any later edits to the body appear in the issue's edit history.

Watch out for: macOS ships BSD date, which does not support -d. Use gdate from coreutils, or the script will fail at the worst moment.

Step 3: Gather evidence and pre-write the reports

The early warning is meant to be fast and thin. For an actively exploited vulnerability, the regulation asks it to indicate, where applicable, the member states in which you know the product has been made available. (The "suspected malicious act" indication applies to severe-incident early warnings.)

The 72h notification adds:

general product information
the general nature of the vulnerability and exploit
the corrective or mitigating measures taken
the measures users can take

Keep one Markdown template for each report in the repo, and fill in the blanks rather than composing prose under pressure.

For evidence, generate an SBOM of the artifact you actually ship and cross-check it against CISA's Known Exploited Vulnerabilities catalog. Grype often reports GitHub advisory (GHSA) IDs, so the filter also reads the related CVE IDs:

syft dir:./release -o cyclonedx-json=sbom.cdx.json            # SBOM of the shipped artifact
grype sbom:./sbom.cdx.json -o json > findings.json            # match components to vulns

curl -fsSL https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json \
  | jq -r '.vulnerabilities[].cveID' | LC_ALL=C sort -u > kev.txt

jq -r '.matches[] | .vulnerability.id, (.relatedVulnerabilities[]?.id)' findings.json \
  | grep '^CVE-' | LC_ALL=C sort -u \
  | LC_ALL=C comm -12 - kev.txt                                 # CVEs in your SBOM AND in KEV

Each output line is a CVE in your shipped components that is known to be exploited somewhere. Treat that as a strong trigger for the decider, not as the legal definition. KEV is one signal; your own telemetry and customer reports count too, and absence from KEV proves nothing.

Verify the flags against the current Syft and Grype documentation, since both tools change output formats between releases.

Step 4: Assign roles and hand-offs across the 72 hours

Write a simple timeline into the runbook so tired people are not improvising:

Hour 0: run craclock as soon as a credible report arrives. The decider then confirms or rejects the exploitation evidence and notes the reason in the issue. Starting at triage is deliberately conservative: it can only make your deadlines earlier than the legal ones. If the evidence is rejected, close the issue with the reason.
Hours 0-8: the submitter lists affected member states and files the early warning. Aim to finish well inside 24 hours, since the platform may be slow or your login may have expired.
Hours 8-60: engineering builds a mitigation (config change, feature flag, patch). Draft the notification in parallel, not after.
Hours 60-72: the submitter files the notification and links the reference IDs back in the issue. Do not wait for the full fix; the notification describes measures taken so far.

After a corrective or mitigating measure is available, a final report is due. Article 14 sets this at no later than 14 days for actively exploited vulnerabilities; confirm against the regulation text and put a dated reminder in the issue.

Keep users informed too. The CRA expects you to tell impacted users about the vulnerability and what they can do about it.

Verify and test

Run a drill before you need one:

Clock script. From a clone of a test repo, run ./cra-clock.sh demo-product TEST-0001. Confirm the issue appears with the two timestamps exactly 24 and 72 hours after "Aware at." If the script errors, the cause is almost always a non-GNU date.
Evidence pipeline. Build a throwaway fixture that contains a component at a version affected by a KEV-listed CVE, for example log4j-core 2.14.1 (CVE-2021-44228). Confirm the comm -12 line prints that CVE. An empty result against a deliberately vulnerable fixture means the pipeline is broken, not that you are safe.
Tabletop exercise. Time the elapsed time from craclock to a filled-in early-warning draft; the target is hours, not a day. Log in to the reporting platform during the drill, but do not submit real reports for fake incidents.

Repeat the drill after staff changes, and calendar the security.txt expiry date.

Closing

CRA vulnerability reporting rewards boring preparation: a monitored inbox, a written definition of awareness, a one-command clock script, and templates that turn a Friday-night scramble into a checklist.

None of this replaces legal review of your scope and obligations. A small team should confirm both with the Commission's guidance and the regulation text. Set it up this week, rehearse it once, and put the platform login in your password manager. For more practical security and CI/CD runbooks, browse kuryzhev.cloud.

PostgreSQL 18 Asynchronous I/O: Settings Worth Testing

Oleksandr Kuryzhev — Thu, 05 Nov 2026 08:01:50 +0000

Originally published on kuryzhev.cloud

A reporting replica gets upgraded to the new major version. Someone reads a headline about "async I/O making scans faster," and a ticket appears: "turn on whatever gives us the speedup." Nobody can say which knob that is, or whether the workload even does the kind of reads that benefit. That is the normal state of PostgreSQL 18 asynchronous I/O adoption: a real feature with vague expectations.

The PostgreSQL 18 release notes introduce an asynchronous I/O subsystem. PostgreSQL 18 is the current stable line (18.6 at the time of writing). The feature matters now because it is active after the upgrade with its default settings, so you are already running it whether you tuned it or not. The useful question is which few settings deserve a controlled test on your data.

Short answer: keep the default io_method = worker unless two conditions hold:

You are on Linux with a kernel and PostgreSQL build that support io_uring.
Your queries are read-heavy with cold or large working sets.

If both hold, A/B test worker against io_uring, then io_workers, then effective_io_concurrency, one change at a time. If your hot data fits in shared_buffers and the OS cache, expect little or nothing.

What PostgreSQL 18 asynchronous I/O actually does

Before 18, a backend asked the kernel for a block and waited. Read-ahead and posix_fadvise hints helped in places, but reads were still synchronous from the backend's point of view. With the new subsystem, a backend can submit several read requests and continue working while they are in flight.

Two things decide how that happens. The first is the io_method setting, which picks the mechanism:

worker (the default) hands reads to dedicated I/O worker processes.
io_uring uses the Linux kernel's ring interface directly from the backend.
sync keeps the old blocking behavior.

The second is the set of code paths that use it. As documented for 18, the main beneficiaries are sequential scans, bitmap heap scans and vacuum. Writes and WAL are not part of this change, so write-heavy OLTP gains little. Verify the exact scope in the release notes.

Related defaults also moved: effective_io_concurrency and maintenance_io_concurrency were raised in 18 (check with SHOW on your build). In practice, an upgraded server behaves differently before you touch anything. That is why a before/after comparison on identical queries matters.

How people use it wrong

The most common error is copying a tuning snippet from a blog post and restarting production with several changed values. Then nobody knows which value helped, hurt, or did nothing. One plausible failure mode is a latency regression caused by oversized worker counts competing with query backends for CPU.

The second error is benchmarking on a warm cache. If the table is already in shared_buffers or the page cache, there is almost no read I/O to overlap, and every method looks identical. The reverse also happens: someone drops caches once, sees a gain, and treats it as steady-state behavior for a workload that is mostly cached.

Third, people assume io_uring is simply "the fast one." It depends on Linux, a PostgreSQL build with liburing support (--with-liburing), and a kernel that allows it. Watch out for container runtimes: default seccomp profiles and some hardened hosts restrict the io_uring syscalls. A setting that works on a VM can therefore fail inside a container. Check your runtime's documentation rather than assuming.

Finally, people expect gains on index-driven point lookups, write-heavy ingestion or WAL-bound workloads. The subsystem in 18 does not target those paths, so no setting will fix them.

The correct approach

Treat it as an experiment with one variable at a time. First, record your baseline: the current values and a representative set of slow read queries. Run the first block below on the target version to capture the starting point.

-- Baseline: current AIO-related settings (values shown are whatever your build uses)
SELECT name, setting, unit, context
FROM pg_settings
WHERE name IN ('io_method','io_workers','io_max_concurrency',
               'io_combine_limit','effective_io_concurrency',
               'maintenance_io_concurrency','track_io_timing');

-- Confirm I/O worker processes exist when io_method = worker
SELECT pid, backend_type
FROM pg_stat_activity
WHERE backend_type = 'io worker';

Next, enable timing so the numbers mean something, and write a candidate config. io_method and io_max_concurrency need a restart, while io_workers can be changed with a reload (confirm with the context column above). Put changes in a drop-in file so they are easy to revert.

# /etc/postgresql/18/main/conf.d/aio-test.conf  (path varies by distro)
track_io_timing = on          # adds I/O time to EXPLAIN and pg_stat_io
io_method = io_uring          # candidate; baseline run used the default 'worker'
# io_workers = 6              # only relevant for io_method = worker; test separately
# effective_io_concurrency = 32   # change in a later, separate round

Run the same queries with EXPLAIN (ANALYZE, BUFFERS, SETTINGS) against a cold cache on a non-production host. For a cold start, restart PostgreSQL and clear the OS page cache (sync; echo 3 > /proc/sys/vm/drop_caches as root, test machine only). Repeat each case several times and compare medians, not the best run.

To verify the change took effect, SHOW io_method; must return the new value after restart, and the server log must show a clean start. If io_uring cannot be used, the server refuses to start rather than silently falling back. This happens either because the build lacks liburing, which makes the value invalid, or because the kernel or seccomp blocks the syscalls. You find out immediately, which is another reason to test this on a non-production host first.

Advanced patterns

Once the method is chosen, tune concurrency against your storage. Low-latency local NVMe often reaches its useful limit with less concurrency than network storage, but measure rather than assume. On network-attached volumes such as cloud block storage, latency dominates, and more outstanding requests usually pay off up to the volume's IOPS or throughput cap. That cap is the real ceiling, so check it before concluding that PostgreSQL is the bottleneck.

For worker mode, size io_workers against CPU headroom. Each worker is a process, and on a busy host the workers compete with query backends. Increase the count in small steps and watch whether scan time improves or the CPU run queue grows instead.

Maintenance work is a separate test. Vacuum reads are governed by maintenance_io_concurrency, so vacuuming a bloated table on a staging copy is a clean way to measure it without touching query latency. Pair the test with whatever bloat monitoring you already run; more PostgreSQL operations material is on kuryzhev.cloud.

For live observation, sample the in-flight view while a test query runs. pg_aios requires superuser or membership in pg_read_all_stats.

-- Run repeatedly (e.g. with \watch 1) during a scan; empty output means nothing is in flight
SELECT * FROM pg_aios;

-- Read volume and time per backend type, since stats reset
SELECT backend_type, object, context, reads, read_bytes, read_time
FROM pg_stat_io
WHERE reads > 0
ORDER BY read_time DESC NULLS LAST;

If you export pg_stat_io to Prometheus and graph it in Grafana, record the delta of read_time and reads per test window. Average time per read is a better signal than raw totals. Column sets differ between major versions (18 added byte columns such as read_bytes), so verify exporter queries against the PostgreSQL 18 documentation.

Performance notes

Expect uneven results. Large sequential scans over data not in cache are the likeliest winners. Queries that are CPU-bound after the read, such as heavy aggregation or expression evaluation, stay limited by the CPU; faster I/O only exposes that. Highly selective index lookups mostly will not move.

Gotcha: changing io_combine_limit alters the size of individual read requests, which interacts with storage block sizes and OS read-ahead. Leave it at the default unless a test shows a specific reason. Never edit developer-only options such as debug_io_direct on a real system.

The costs are modest but real:

io_uring adds a kernel and build dependency and may complicate container hardening.
Extra workers consume CPU and memory.
Higher concurrency can let one noisy analytical query take a larger share of shared storage bandwidth, which hurts neighbors on a shared volume.

If your replica serves mixed traffic, test with the real mix running.

When not to bother: small databases that fit in RAM, write-dominated systems, or any server where the upgrade is already stable and the queries meet their SLOs. Defaults in 18 are chosen to be safe, and "no change" is a valid outcome of the experiment.

A simple decision rule to keep:

If cold-cache EXPLAIN shows significant I/O time, change one setting.
Rerun the identical query set.
Keep the change only if the median improves without raising CPU or p99 latency elsewhere.
Write the result next to the config file so the next engineer does not repeat the test.

What changed in Argo CD 3.5 for ops teams upgrading from 3.x

Oleksandr Kuryzhev — Wed, 04 Nov 2026 08:01:49 +0000

Originally published on kuryzhev.cloud

The scenario

A platform team runs Argo CD 3.2 or 3.3 in a shared cluster, with a few hundred Applications, and the ticket says "get to Argo CD 3.5 before the next audit". Argo CD 3.5 was released in August 2026 (see the official releases page), and the cycle has since collected patch releases up to 3.5.6. That matters now because the upstream support policy covers only the newest minor lines, and every month you wait widens the gap you have to cross.

Short answer: do not treat this as a version bump in a manifest. Back up the Argo CD state, read the upgrade notes for every minor between your version and 3.5, rehearse on a non-production instance with a pinned 3.5.x manifest, then promote and watch sync metrics. This applies if you are on any 3.x minor; if you are still on 2.x, this is a different, bigger migration.

One caveat up front: this article does not reproduce the 3.5 changelog line by line. The authoritative list of behavior changes lives in the upgrade pages of the Argo CD documentation (operator manual, "Upgrading" section) and the release notes. What follows is the process that turns that list into a safe rollout.

Prerequisites

Before touching anything, make sure you have the following in place:

Sufficient Kubernetes permissions: admin rights in the namespace running Argo CD (assumed argocd below), plus cluster-scoped permission to apply CRDs and cluster roles.
The argocd CLI at a version matching the server you are targeting. A CLI that is several minors behind can behave oddly against a newer API server.
A non-production Argo CD with a representative slice of Applications, ApplicationSets and your real argocd-cm / argocd-rbac-cm settings. A vanilla install tells you little.
Your install method written down: raw upstream manifests (standard or HA), Kustomize overlay, or the community Helm chart. Each pins versions differently, and a chart version is not the same number as the app version.
A Kubernetes compatibility check. If your clusters are moving to Kubernetes 1.37, confirm the tested-versions information in the Argo CD docs for your target release.

Watch out for exposure through ingress-nginx. That controller was retired in March 2026 and gets no further security fixes. If argocd-server sits behind it, plan the move to Gateway API or another maintained controller as a separate change. Do not do it in the same window as the Argo CD upgrade, because mixing the two makes any login or gRPC failure ambiguous.

Step 1: Snapshot the current state

You need two things to compare against later: a restorable backup of Argo CD's own configuration and a baseline of what every Application looks like today. The block below captures both.

# Record the running versions (client and server)
argocd version

# Backup Applications, AppProjects, settings and secrets handled by Argo CD
argocd admin export -n argocd > argocd-backup-$(date +%F).yaml

# Baseline: name, sync status, health for every app
argocd app list -o json \
  | jq -r '.[] | [.metadata.name, .status.sync.status, .status.health.status] | @tsv' \
  | sort > baseline-apps.tsv

wc -l baseline-apps.tsv   # keep this count, you will compare it later

Store the export somewhere encrypted, because it can contain repository credentials and cluster secrets. Also note any Applications that are already OutOfSync or Degraded. A typical failure during upgrades is blaming the new version for drift that existed beforehand.

Step 2: Read the notes for every minor and diff your config

Go to the operator manual's upgrading section and read each page between your current minor and 3.5, in order. For example, read 3.3 to 3.4, then 3.4 to 3.5. Upstream documents upgrades one minor at a time, and skipped notes are where defaults changed without you noticing.

While reading, sort each item into three buckets: breaking changes that need a config edit, changed defaults that alter runtime behavior, and deprecations with a removal date. Then check your own settings against them. Look specifically at argocd-cm (resource customizations, tracking method, exclusions), argocd-cmd-params-cm (component flags), argocd-rbac-cm (policies) and any config management plugins.

Watch out for resource tracking and ignore-difference settings. Changes in how Argo CD tracks or diffs resources are exactly the kind of thing that can flip hundreds of Applications to OutOfSync after an otherwise clean upgrade. If a note touches either, test it on the staging instance first with argocd app diff on several real Applications.

Also review your pinned dependencies. The Helm and Kustomize versions bundled in the repo-server image can change between minors, which can alter rendered output for the same chart. Run argocd admin settings --help to see the validation helpers your CLI version provides, and use them on your edited ConfigMaps.

Step 3: Rehearse on staging with a pinned manifest

Pin an exact patch version, never a moving tag like stable. The documented way to handle the large Argo CD CRDs without annotation-size errors is to apply the upstream install manifest with server-side apply. Use the manifest that matches your current topology: manifests/install.yaml for a standard install, manifests/ha/install.yaml for HA. Applying the non-HA manifest over an HA install replaces the HA components.

# Staging context first
kubectl config use-context staging

# Pin the patch release; v3.5.6 is the latest 3.5.x at time of writing
# (HA installs: use manifests/ha/install.yaml instead)
kubectl apply -n argocd --server-side --force-conflicts \
  -f https://raw.githubusercontent.com/argoproj/argo-cd/v3.5.6/manifests/install.yaml

# Wait for control plane components
kubectl -n argocd rollout status deploy/argocd-server
kubectl -n argocd rollout status deploy/argocd-repo-server
kubectl -n argocd rollout status statefulset/argocd-application-controller

If you use a Kustomize overlay or Helm chart, change the version there and render the diff first. For Kustomize, use kustomize build . | kubectl diff --server-side -f -. For Helm, use helm diff upgrade <release> <chart> --version <chart-version>, which requires the helm-diff plugin.

Gotcha: --force-conflicts makes the apply take ownership of every field the upstream manifest sets, even where another field manager changed it. Any hand edits to those fields will be reset unless they live in your overlay; this covers container args, images and env made with kubectl edit or kubectl patch. Fields the manifest does not set are left alone. Review the diff for that before applying.

Step 4: Promote to production and keep a way back

Repeat the same command against production during a quiet window, and avoid doing it while a big release is mid-sync. The application controller restarts and re-reconciles everything, so expect a burst of repo-server and API-server load; for large installs this is the real cost of the upgrade.

Your rollback plan is concrete. First, re-apply the previous pinned manifest. If state looks damaged, restore from the export with argocd admin import -n argocd - < argocd-backup-YYYY-MM-DD.yaml. Be aware that CRD schema changes are not always cleanly reversible, so the rehearsal on staging is what actually de-risks this, not the rollback.

When not to do this: do not upgrade in the middle of an incident, a freeze, or while you cannot get a human to watch the first reconciliation. If you manage many clusters from one Argo CD, also consider upgrading a secondary instance first. The extra instance costs money and operational overhead, but limits blast radius.

Verify and test

Start with versions and component health, then compare against the baseline you took in step 1.

argocd version                      # server should report v3.5.6

argocd app list -o json \
  | jq -r '.[] | [.metadata.name, .status.sync.status, .status.health.status] | @tsv' \
  | sort > after-apps.tsv

diff baseline-apps.tsv after-apps.tsv && echo "no status drift"

An empty diff means every Application kept its sync and health status. Any lines that differ are your investigation list. Run argocd app diff <name> on each to see whether the cause is rendering, tracking or a genuine change. Then test the full path with one throwaway Application: sync it, change a value in Git, and confirm that the change is detected and applied.

For ongoing confidence, watch the Prometheus metrics Argo CD exposes:

argocd_app_info, grouped by sync_status and health_status, should return to its pre-upgrade shape within a few reconcile cycles.
argocd_app_sync_total, filtered by its phase label, should show failed or errored syncs increasing only when you expect them.
Pod restarts and repo-server logs are worth checking too. Repeated restarts after the first hour usually point to resource limits that need adjusting.

Closing

An Argo CD 3.5 upgrade from 3.x is mostly a reading and rehearsal exercise:

Snapshot with argocd admin export.
Work through each minor's upgrade notes.
Pin v3.5.6 or later, using the manifest that matches your install topology.
Apply with server-side apply on staging.
Compare application status before and after.

Keep the official documentation open while you do it, because defaults and flags are version-specific and anything in this article should be checked against it. For more runbooks on GitOps and platform tooling, browse kuryzhev.cloud.

OpenTofu or Terraform: What to Pick for Existing Code

Oleksandr Kuryzhev — Tue, 03 Nov 2026 08:01:59 +0000

Originally published on kuryzhev.cloud

The question of OpenTofu or Terraform tends to arrive on a quiet Tuesday. A repo with 200 resources, a remote state in S3 and a CI pipeline pinned to terraform works fine, until someone asks whether the next upgrade should be to the other tool. Nothing is broken, so the real question is whether switching buys anything that justifies the risk.

The timing is worth noting. Both projects shipped new minor cycles recently: OpenTofu 1.13 (latest patch 1.13.5, cycle released 2026-09-30, release notes) and Terraform 1.17 (latest patch 1.17.1, cycle released 2026-10-21, release notes). Anyone planning a quarterly upgrade has two fresh targets. The two projects follow independent roadmaps, so check each changelog for features that exist in only one of them.

Short answer: if your code is plain AWS/GCP/Azure/Kubernetes resources and you have no contract with HCP Terraform, you can usually stay on Terraform 1.17 or move to OpenTofu 1.13 at low risk, so choose based on licensing and features. If you depend on HCP Terraform, Stacks or other HashiCorp-hosted services, stay on Terraform. If you want client-side state encryption or an open-governance licence, trial OpenTofu with the dry-run below before committing.

When the OpenTofu or Terraform choice actually matters

For a small repo with a single backend, the choice is mostly cosmetic. It starts to matter in four situations:

Licensing review. Terraform releases after 1.5.x use the Business Source License; OpenTofu is MPL-2.0 under the Linux Foundation. Legal teams sometimes care, particularly if you ship tooling to customers. Read the current licence text yourself rather than relying on a blog summary.
State sensitivity. State holds secrets in plain text unless the backend encrypts it. OpenTofu has built-in state and plan encryption. Terraform relies on backend encryption and on its ephemeral and write-only mechanisms to keep values out of state. Check the Terraform docs for what your version supports.
Platform coupling. If your org runs workspaces, policies and run tasks in HCP Terraform or Terraform Enterprise, the tool and the platform are one decision.
Feature divergence. Since the 2023 fork the two projects have added different features and, in some cases, different syntax for similar ideas. Config that uses a tool-specific feature will not run on the other.

Neither tool has a blanket advantage. Compare the changelogs for your specific version jump, not for the headline releases.

Option A: OpenTofu 1.13

Pros

Open governance and an MPL-2.0 licence, which simplifies procurement and redistribution questions.
Native state and plan encryption with pluggable key providers (including AWS KMS), configured in the terraform block. This matters when state lands in shared buckets.
Features that landed after the fork, such as for_each on providers and the -exclude flag, are documented on opentofu.org. Confirm what is in 1.13 against its changelog.
Same HCL, same provider protocol, same workflow. The binary is tofu and the commands match, so the CLI is familiar.

Cons

Migration becomes effectively one-way once you encrypt state or adopt OpenTofu-only syntax. Terraform cannot read encrypted state, so going back means decrypting with OpenTofu or reverting the config first.
Provider and module sourcing goes through the OpenTofu registry. Your .terraform.lock.hcl entries will change, and any tooling hard-coded to registry.terraform.io needs a check.
Third-party integrations (scanners, IDE plugins, CI templates) vary in how well they support tofu. Test yours rather than assuming.
Commercial support comes from vendors rather than the original authors, which may or may not matter to your org.

Option B: Terraform 1.17

Pros

Zero migration. Your pipeline, lock files, runbooks and team habits stay as they are, and upgrades inside the 1.x line follow the documented upgrade guides.
First-party integration with HCP Terraform and Terraform Enterprise, including features that exist only there (Stacks, hosted policy and drift tooling). HashiCorp is now part of IBM, so vendor support contracts are a single conversation for many enterprises.
Most vendor docs and provider examples still assume terraform, so troubleshooting is easier to search.

Cons

The BSL licence restricts offering a competing product. Most internal use is fine, but verify the terms against your situation, especially if you resell infrastructure automation.
No built-in client-side state encryption. You depend on backend-side encryption (for example S3 with KMS) and on keeping secrets out of state in the first place.
Roadmap and licence decisions sit with one vendor. That is a trade-off, not a defect, but it is a dependency you should name.

Decision matrix

Situation	Lean toward	Why
HCP Terraform / Enterprise in daily use	Terraform 1.17	Platform features are tied to the vendor CLI
Self-managed CI with S3/GCS backend	Either	Low switching cost; decide on licence and features
Need encrypted state independent of the backend	OpenTofu 1.13	Built-in encryption block
Legal requires an OSI-approved licence	OpenTofu 1.13	MPL-2.0
Heavy use of vendor-supported Terraform tooling	Terraform 1.17	Fewer integration surprises
Config written for a much older Terraform (1.5.x or earlier)	Upgrade first, decide later	Big version jumps are the risk, not the tool
Small team, no pain today	Stay put	Switching has a cost and no immediate payoff

Watch out: do not combine a tool switch with a major provider upgrade in one change. When a plan shows unexpected diffs, you want exactly one variable to blame.

Evidence-based recommendation

The evidence for an urgent move in either direction is thin. Both tools plan and apply the same core HCL against the same providers. The differences that matter are licence, state handling, and platform lock-in, and you can evaluate those without touching production.

So the practical approach is a read-only trial. The following runs OpenTofu against a copy of your current setup. It takes a state backup first, then produces a plan and never applies. Run it from a branch.

# 1. Back up current state with the tool you use today
terraform state pull > ../backup-$(date +%F).tfstate

# 2. Check versions; OpenTofu evaluates required_version against its own
#    version number, so review any terraform { required_version } constraint
terraform version
tofu version

# 3. Re-init with OpenTofu (rewrites .terraform.lock.hcl; review the diff)
rm -rf .terraform
tofu init
tofu providers lock -platform=linux_amd64 -platform=darwin_arm64

# 4. Read-only comparison: -lock=false avoids blocking teammates; never apply here
tofu plan -lock=false -detailed-exitcode
echo "exit code: $?"

To verify the result, check the exit code. 0 means no changes, which is what you want from an unchanged codebase. 2 means a diff, so read it before you do anything else. 1 is an error, typically a provider source or backend setting. If the plan is clean, run the same steps in CI on a non-production workspace.

Watch out: run tofu apply only on a workspace you can afford to roll back. A newer tool can write state that the older one refuses to read, and the backup from step 1 is your way out.

If the trial is clean and you want OpenTofu's encryption, add it deliberately as a separate change. Existing state is unencrypted, so the first configuration needs an unencrypted fallback to read it. This example uses AWS KMS; the key alias is a placeholder for your own.

# encryption.tf (OpenTofu only; Terraform will reject this block)
terraform {
  encryption {
    key_provider "aws_kms" "main" {
      kms_key_id = "alias/tofu-state"   # your KMS key alias
      region     = "eu-west-1"
      key_spec   = "AES_256"
    }

    method "aes_gcm" "default" {
      keys = key_provider.aws_kms.main
    }

    # Temporary: lets OpenTofu read the existing unencrypted state
    method "unencrypted" "migrate" {}

    state {
      method = method.aes_gcm.default
      fallback {
        method = method.unencrypted.migrate
      }
    }
  }
}

After the next tofu apply writes encrypted state, remove the unencrypted method and the fallback block, and add enforced = true inside state so OpenTofu refuses to write unencrypted state. Verify by reading the raw object from the backend rather than through tofu, which decrypts transparently, for example aws s3 cp s3://<your-state-bucket>/<your-state-key> - | head -c 200. Encrypted state is a JSON envelope with encryption metadata instead of readable resource attributes. Details, plan encryption and rollback options are in the OpenTofu documentation, so follow those rather than improvising.

The recommendation, with conditions:

Stay on Terraform 1.17 if you use HCP Terraform or Enterprise, or if nothing in the matrix hurts today.
Move to OpenTofu 1.13 if licence terms or state encryption are real requirements, and the dry-run above shows a clean plan.
Do neither yet if you are several minor versions behind. Upgrade within your current tool first, then compare.

For related infrastructure patterns, browse more guides on kuryzhev.cloud. Whichever way you go, pin the binary version in CI so the choice stays a decision and never becomes drift.

Jenkins Agent Hardening: Lock Down Build Execution

Oleksandr Kuryzhev — Mon, 02 Nov 2026 08:02:17 +0000

Originally published on kuryzhev.cloud

The scenario

A pull request from a contributor changes a Jenkinsfile, and the build runs on a long-lived agent that also builds your release branch. Nothing in the pipeline looks malicious. However, the agent has a Docker socket mounted, a cloud credentials file in the home directory, and a workspace left over from the last job. Jenkins agent hardening exists to make that situation boring. A build can run arbitrary code, so the machine running it should have nothing worth stealing and no way to reach beyond its own job.

The Jenkins project documents the controller/agent trust model in its controller isolation guide. The short version is that anyone who can change a pipeline can execute code on an agent, so agents are a trust boundary. This guide applies that to four concrete areas:

keeping builds off the controller
shrinking agent privileges
confining permanent agents
scoping secrets

The examples assume Kubernetes-based ephemeral agents plus one class of permanent Linux agents. Adjust them to your environment, and check the Jenkins and plugin documentation for version-specific behavior.

Prerequisites

A current Jenkins LTS controller reachable over HTTPS. Agent-to-controller access rules have been always-on since Jenkins 2.326, so there is no legacy toggle to disable.
Administrator access, ideally with Configuration as Code (JCasC) so the settings below are reviewable in Git.
The Kubernetes plugin and a cluster where you can create pods in a dedicated namespace (for the ephemeral agents).
A Linux host with systemd and Java installed (for the permanent agents).
A test pipeline you can run safely. Do not validate hardening against your production release job.

Also decide up front which jobs are trusted. Pull-request builds from forks and release builds should never share an agent label. Hardening one agent type well is of little use if a single agent type is shared by everything.

Step 1: Keep builds off the controller and shrink the connection surface

The built-in node runs with the controller's filesystem access, including secrets and job configuration. Set its executors to zero so no build can land there.

Then disable the fixed inbound TCP agent port and use WebSocket connections, which travel over your existing HTTPS endpoint.

# jenkins.yaml (Configuration as Code)
jenkins:
  numExecutors: 0        # built-in node runs no builds
  slaveAgentPort: -1     # disables the separate inbound TCP agent port
  mode: EXCLUSIVE        # only jobs with a matching label run on the built-in node

Inbound agents then connect with the -webSocket option. The @ prefix on -secret reads the secret from a file. That keeps it off the command line, where it would show up in the process list. Confirm the exact flags for your agent version with java -jar agent.jar -help.

# Run as the unprivileged agent user, not root
java -jar agent.jar \
  -url https://jenkins.example.com/ \
  -name build-linux-01 \
  -secret @/etc/jenkins-agent/secret \
  -workDir /var/lib/jenkins-agent \
  -webSocket

Watch out for: turning off the TCP port while agents are still configured to use it. Existing inbound agents will fail to reconnect. Migrate them to WebSocket first, or schedule the change in a maintenance window. This includes Kubernetes plugin pod agents: enable the WebSocket option on the Kubernetes cloud configuration before you disable the port.

Step 2: Make agents ephemeral and unprivileged

A fresh pod per build removes the biggest persistence risk: leftover workspaces, caches, and tampered tools. The pod spec should:

run as a non-root user
drop Linux capabilities
avoid mounting the host's container socket

The Kubernetes documentation on security contexts covers each field.

apiVersion: v1
kind: Pod
spec:
  automountServiceAccountToken: false   # builds rarely need Kubernetes API access
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    seccompProfile:
      type: RuntimeDefault
  containers:
    - name: jnlp                         # default agent container name in the Kubernetes plugin
      image: jenkins/inbound-agent:latest-jdk21   # pin a version or digest in real use
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        capabilities:
          drop: ["ALL"]
      volumeMounts:
        - name: work
          mountPath: /home/jenkins/agent
        - name: tmp
          mountPath: /tmp
  volumes:
    - name: work
      emptyDir: {}
    - name: tmp
      emptyDir: {}

Watch out for: readOnlyRootFilesystem breaks tools that write to the home directory or a cache path. Add an emptyDir for each path that needs writes rather than relaxing the setting.

If builds need to produce container images, use a rootless builder instead of a privileged Docker-in-Docker sidecar. BuildKit in rootless mode and Buildah both work. Avoid Kaniko for new setups, because its original upstream repository has been archived.

A pipeline can define its own pod template. A hardened default therefore does not stop a malicious Jenkinsfile from requesting a privileged pod. Enforce the limits at the cluster level by labeling the agent namespace with pod-security.kubernetes.io/enforce: restricted. Pod Security Admission then rejects pods that violate the Restricted standard. The spec above complies with it.

Pair the pod spec with a Kubernetes NetworkPolicy. Limit egress to the controller, DNS, your artifact registry, and the package mirrors you actually use. NetworkPolicy is only enforced if your CNI plugin supports it. Default-open egress gives a compromised build an easy path to exfiltrate data.

Step 3: Confine permanent agents with systemd

Some workloads, such as hardware tests or macOS and Windows builds, cannot be ephemeral pods. For Linux agents of this kind, run the agent as a dedicated user under a systemd unit that restricts what the process can touch. The directives below are documented in systemd.exec(5). Verify that your distribution's systemd version supports each one. Note that systemd only treats lines beginning with # as comments, so comments cannot follow a value.

# /etc/systemd/system/jenkins-agent.service
[Unit]
Description=Jenkins inbound agent
Wants=network-online.target
After=network-online.target

[Service]
User=jenkins-agent
Group=jenkins-agent
ExecStart=/usr/bin/java -jar /opt/jenkins-agent/agent.jar \
  -url https://jenkins.example.com/ -name build-linux-01 \
  -secret @/etc/jenkins-agent/secret -workDir /var/lib/jenkins-agent -webSocket
# Block privilege escalation via setuid binaries
NoNewPrivileges=true
# Filesystem read-only except ReadWritePaths
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
ReadWritePaths=/var/lib/jenkins-agent
# Empty set: no capabilities
CapabilityBoundingSet=
Restart=on-failure

[Install]
WantedBy=multi-user.target

Make the secret file readable only by the agent user (chmod 600, owned by jenkins-agent). Also clean the workspace between builds, because a permanent agent keeps whatever the last job left behind. Use the cleanWs() step from the Workspace Cleanup plugin, or a scheduled wipe.

Step 4: Scope credentials and label trust levels

Even a hardened agent is only as safe as the secrets it receives. Store credentials in the Jenkins credentials store. Scope them to the folder that needs them rather than globally, and bind them only around the steps that use them. Use separate labels so untrusted builds never share an agent with deploy jobs.

pipeline {
  agent { label 'untrusted-ephemeral' }   // PR builds: no deploy credentials exist here
  stages {
    stage('Test') {
      steps { sh './run-tests.sh' }
    }
    stage('Publish') {
      when {
        beforeAgent true                  // evaluate before allocating the trusted agent
        branch 'main'
      }
      agent { label 'trusted-release' }   // separate pool for release work
      steps {
        withCredentials([string(credentialsId: 'registry-token', variable: 'TOKEN')]) {
          sh 'echo "$TOKEN" | docker login registry.example.com -u ci --password-stdin'
        }
      }
    }
  }
}

Labels in a Jenkinsfile are not a control on their own, because a pull request can edit them. Enforce the separation on the controller:

Configure your branch source so fork pull requests run the Jenkinsfile from the target branch. The GitHub Branch Source plugin does this through its fork trust setting.
Restrict which jobs may use the trusted pool, for example with the Job Restrictions plugin or a cloud only the release folder can use.

Watch out for: credential masking only hides exact values in the console log. It does not stop a malicious step from transforming a secret and sending it elsewhere. That is why the trusted pool must be unreachable from untrusted pull requests. Prefer short-lived tokens over static keys, such as OIDC federation to your cloud provider where supported.

Verify and test Jenkins agent hardening

Run checks as the build would, using a throwaway pipeline. Each test should fail in the way you expect. A pass on a forbidden action means the control is missing.

# 1. Identity and privileges inside an ephemeral agent
kubectl -n jenkins-agents exec POD_NAME -c jnlp -- id                       # expect uid 1000, not 0
kubectl -n jenkins-agents exec POD_NAME -c jnlp -- touch /home/jenkins/probe  # expect "Read-only file system"

# 2. Container socket and API access
kubectl -n jenkins-agents exec POD_NAME -c jnlp -- ls /var/run/docker.sock   # expect not found
kubectl -n jenkins-agents exec POD_NAME -c jnlp -- ls /var/run/secrets/kubernetes.io/serviceaccount   # expect not found

# 3. Controller TCP agent port closed (run from a network peer)
nc -zv jenkins.example.com 50000    # use your previous port; expect failure

# 4. Permanent agent sandbox rating
systemd-analyze security jenkins-agent.service

Then complete these checks:

In the Jenkins UI, confirm the built-in node shows zero executors and that a job with no matching label does not start on it.
Run a pull-request build that attempts to read a deploy credential ID, and confirm it cannot resolve.
Try to apply a privileged pod in the agent namespace, and confirm Pod Security Admission rejects it.
Check that outbound requests to a host outside your NetworkPolicy allowlist time out.

Closing

Jenkins agent hardening is mostly about removing four assumptions:

that the controller can build
that agents live forever
that containers may be root
that every job deserves every secret

Start with executors set to zero and ephemeral unprivileged pods, which are a practical first step for most setups. Then add systemd confinement and credential scoping for the agents that must persist. Keep the configuration in JCasC and Git so changes are reviewed. Revisit the Jenkins security documentation when you upgrade, since defaults and plugin behavior change between releases. For more infrastructure guides, browse kuryzhev.cloud.

Kubernetes NetworkPolicy Baseline: 3 Mistakes That Break Isolation

Oleksandr Kuryzhev — Sun, 01 Nov 2026 08:01:53 +0000

Originally published on kuryzhev.cloud

A team applies a default-deny policy to a staging namespace on Friday afternoon, and within seconds every pod starts logging "temporary failure in name resolution". Or the opposite happens: the policy applies cleanly, nothing breaks, and nothing is actually blocked either. A NetworkPolicy baseline sounds like a simple YAML exercise, but the documented behavior has enough sharp edges that the same few mistakes keep appearing. This retrospective walks through those mistakes as patterns, not as a report of any specific outage.

Context: what a NetworkPolicy baseline is supposed to do

By default, Kubernetes pods accept traffic from any other pod in the cluster and can reach any destination. Namespaces are an organizational boundary, not a network one. A NetworkPolicy baseline changes that default: each namespace starts from "deny everything", and teams add narrow allow rules on top.

Three properties of the networking.k8s.io/v1 API shape everything that follows:

Policies are allow-only and additive. There is no deny rule; the union of all policies selecting a pod defines what is permitted.
A pod is only isolated for a direction (ingress or egress) once at least one policy selects it for that direction.
The API object is only a declaration. Enforcement is the job of the CNI plugin or network provider.

A minimal baseline usually has two parts: a default-deny policy for both directions, and an explicit allowance for cluster DNS. The example below shows the shape. Verify the DNS labels against your own cluster, since they depend on the distribution.

# Baseline for one namespace: deny all ingress and egress for every pod
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: team-a
spec:
  podSelector: {}          # empty selector = every pod in the namespace
  policyTypes:
    - Ingress
    - Egress
---
# Allow DNS lookups to CoreDNS in kube-system
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-egress
  namespace: team-a
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system   # set automatically by the control plane (v1.21+)
          podSelector:
            matchLabels:
              k8s-app: kube-dns                          # CoreDNS label in most distributions
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53                                       # needed for large responses and truncation fallback

Common failure 1: default-deny without a DNS and platform exception

The most typical failure is applying the deny policy first and discovering afterward that egress includes DNS. With policyTypes: [Egress] and no egress rules, pods cannot resolve service names. The symptom looks like an application bug: connection errors, readiness probes that depend on external calls failing, startup crashes on config fetches.

The fix is the DNS allowance shown above, but there are variations that catch people out. Watch out for NodeLocal DNSCache: when it is deployed, pods may send queries to a node-local link-local address rather than the CoreDNS service IP. The node-local-dns pods typically run with hostNetwork, so a namespace-and-pod selector may not match them, and an ipBlock rule for the link-local address may be needed. Whether that works also depends on the CNI. Check your DNS configuration (/etc/resolv.conf inside a pod) and consult your CNI documentation before assuming the CoreDNS rule is sufficient.

Other platform dependencies tend to surface the same way. Operators and controllers need egress to the Kubernetes API server. Metrics scrapers need ingress from a monitoring namespace. Service meshes need to reach their control planes. API server traffic is a common trap because the control plane usually sits outside the pod network. Pod and namespace selectors do not match it, so it typically needs an ipBlock rule. Whether that rule should target the kubernetes Service IP or the actual endpoint IPs depends on how your CNI orders address translation and policy evaluation. A deny baseline needs a short, reviewed list of these platform exceptions, ideally delivered by the same mechanism that creates the namespace, so they never lag behind.

Common failure 2: policies that exist but are never enforced

The Kubernetes API server accepts a NetworkPolicy object regardless of whether anything enforces it. The documentation is explicit that creating the resource without a supporting network plugin has no effect. A cluster using a CNI without policy support, or a managed cluster where the policy feature has not been enabled, will happily store the baseline while traffic flows freely.

This is the dangerous failure because it produces a green dashboard and a false sense of isolation. Reviewing the YAML alone cannot detect it; a traffic test is what confirms enforcement. Watch out for managed platforms where enforcement is an opt-in add-on or a separate component, and where the behavior may differ by version. Check your provider's documentation rather than assuming.

A simple connectivity probe turns the baseline into something testable. For example, run a throwaway pod in the isolated namespace and try to reach a service in another namespace. Because the baseline denies egress from team-a, the expected result is a timeout, not a response. Verify with a command like the one below, adjusting the target to your environment.

# Expect a timeout (exit code non-zero) if the egress baseline is enforced
kubectl -n team-a run np-test --rm -it --restart=Never \
  --image=busybox:stable -- \
  wget -qO- -T 3 http://web.team-b.svc.cluster.local   # -T 3 = 3 second timeout

Note that a DNS failure also exits non-zero. Busybox reports it as a "bad address" error rather than a timeout, so read the output, not just the exit code. Test against a pod IP as well, so you know which layer is blocking. This probe exercises team-a egress. To verify ingress isolation for a namespace, run the probe from a namespace that does not deny egress and target a pod in the protected namespace.

Common failure 3: selector semantics that quietly widen access

The third pattern is subtle: YAML indentation changes the meaning of a rule. Inside a from or to list, a namespaceSelector and podSelector in the same list item are combined with AND. Placed as separate items, they are combined with OR. Take a rule intended as "pods labeled app.kubernetes.io/name=prometheus in the monitoring namespace". Split into two items, it becomes "every pod in the monitoring namespace, plus every pod with that label in the policy's own namespace".

# Intended: only Prometheus pods from the monitoring namespace
ingress:
  - from:
      - namespaceSelector:          # one list item, two selectors = AND
          matchLabels:
            kubernetes.io/metadata.name: monitoring
        podSelector:
          matchLabels:
            app.kubernetes.io/name: prometheus
# Risky variant: a second dash before podSelector makes it OR

Related traps include relying on custom namespace labels that anyone with namespace edit rights can change, and using ipBlock for in-cluster traffic, where pod IPs may be rewritten or ephemeral. Watch out for hostNetwork pods and node-originated traffic too. The documentation notes that behavior for hostNetwork pods is implementation-dependent, and that a pod cannot block access from its own node. That is why kubelet health probes generally keep working, but verify rather than assume. For namespace matching, prefer the kubernetes.io/metadata.name label. The control plane sets it to the namespace name, and it cannot be changed to a different value.

Safer operating pattern

The pattern that avoids all three failures is boring and incremental. Prove enforcement first, ship the deny and DNS policies together, then add allow rules per application with review. Treat the policies as code, applied by the same pipeline that provisions namespaces, so no namespace exists in an unprotected state.

A practical checklist, in order:

1. Confirm the CNI enforces NetworkPolicy (read provider docs, then run a deny test)
2. Inventory platform dependencies: DNS, API server, metrics, mesh, ingress controller
3. Apply default-deny + DNS allow together, never deny alone
4. Add per-app allow policies using kubernetes.io/metadata.name and stable pod labels
5. Review every from/to block for AND vs OR (selectors in one item vs separate items)
6. Run automated allow and deny connectivity tests in CI or post-deploy
7. Use your CNI's flow logs or policy-verdict logs to find denied traffic before tightening
8. Re-test after CNI, Kubernetes, or node image upgrades

For clusters with many namespaces, also look at the cluster-scoped AdminNetworkPolicy and BaselineAdminNetworkPolicy APIs from SIG Network's Network Policy API subproject. They aim to let platform teams set a baseline that namespace owners cannot override. However, they are distributed as CRDs rather than built into Kubernetes, are still pre-GA, and work only with CNIs that implement them. Check the project's current status and your CNI's support before depending on them.

Finally, remember what a baseline does not cover: it is layer 3/4 filtering, not authentication or encryption. It pairs well with RBAC, Pod Security Admission, and mutual TLS, not instead of them. The authoritative reference for semantics and current behavior is the Kubernetes NetworkPolicy documentation. More practical Kubernetes and security notes are available at kuryzhev.cloud.

AWS Cost Anomaly Alerts: Route to Owners with Tag Context

Oleksandr Kuryzhev — Sat, 31 Oct 2026 08:02:19 +0000

Originally published on kuryzhev.cloud

The scenario

A shared finance mailbox gets an email titled "Cost anomaly detected," and nobody knows whose problem it is. This is the usual failure with default AWS cost anomaly alerts: detection works, but the message goes to a person who cannot act on it. By the time someone finds the owning team, the spend may have run for another day.

The goal here is narrower than "more alerts." Each anomaly should reach the team that owns the spend, with enough context to start investigating immediately. That context includes the service, account, usage type, dollar impact, and a link to the details. AWS Cost Anomaly Detection already does the statistical work. What you build is the routing layer around it.

The design has four parts:

Monitors scoped by service and by a cost allocation tag.
An SNS topic that Cost Anomaly Detection can publish to.
A subscription with a sensible impact threshold.
A small Lambda that adds ownership context and forwards the alert to the right channel.

The documented SNS payload carries the following fields:

the anomaly ID and the monitor ARN
impact figures
root causes: service, region, linked account, and usage type

The payload does not carry the tags of individual resources. Ownership therefore has to come from which monitor fired or from your own mapping. Verify the current payload shape in the Cost Anomaly Detection documentation before relying on specific fields.

Prerequisites

Cost Anomaly Detection depends on Cost Explorer, so Cost Explorer must be enabled in the account. If you use AWS Organizations, create monitors in the management account to cover linked accounts. Cost Explorer and anomaly APIs are served from us-east-1, so set that region for the CLI calls.

Your tagging has to be ready too. Pick one tag key that identifies ownership, for example team, and activate it as a cost allocation tag in the Billing console.

Watch out for the activation delay: activation can take up to 24 hours to appear in cost data. By default, the tag only applies going forward. You can request a backfill of historical data for activated tags, so check the current backfill limits in the Billing documentation. A tag monitor created before there is tagged cost history will have little to evaluate.

You also need permissions to create the pieces:

ce:CreateAnomalyMonitor and ce:CreateAnomalySubscription
sns:CreateTopic and sns:SetTopicAttributes
Lambda and IAM permissions for the router function

Have a chat webhook URL ready and a short list mapping each owner to a channel. The example below posts a {"text": ...} body, which matches Slack incoming webhooks. Other tools expect their own payload format. For Microsoft Teams, legacy Office 365 connectors have been retired in favor of Workflows.

If you prefer a managed chat integration, Amazon Q Developer in chat applications (formerly AWS Chatbot) can subscribe to an SNS topic directly. Check current docs for how it renders Cost Anomaly messages. A custom Lambda is what lets you add your own ownership fields.

Step 1: Create monitors that match how you assign ownership

A monitor defines what is watched. Start with a monitor on the AWS service dimension, which catches surprises nobody tagged. Accounts are typically limited to one AWS services monitor, and many already have a default one. Run aws ce get-anomaly-monitors first and reuse an existing monitor if present.

Then add one cost allocation tag monitor per team value. A separate monitor per team means the alert's monitorArn tells you exactly whose tag the spend fell under.

# Service-level monitor: catches spikes in any AWS service (skip if one exists)
aws ce create-anomaly-monitor --region us-east-1 --anomaly-monitor '{
  "MonitorName": "aws-services",
  "MonitorType": "DIMENSIONAL",
  "MonitorDimension": "SERVICE"
}'

# One tag monitor per team, so monitorArn in the alert identifies the owner
for team in payments data platform; do
  aws ce create-anomaly-monitor --region us-east-1 \
    --query MonitorArn --output text --anomaly-monitor "{
    \"MonitorName\": \"team-${team}\",
    \"MonitorType\": \"CUSTOM\",
    \"MonitorSpecification\": {
      \"Tags\": {\"Key\": \"team\", \"Values\": [\"${team}\"], \"MatchOptions\": [\"EQUALS\"]}
    }
  }"
done

Each call returns a MonitorArn. Save them all, because they become keys in the owner map. Monitor counts are subject to service quotas, so verify the current limits if you have many teams. The service monitor will also fire for untagged spend, which makes it a useful catch-all owned by your platform or FinOps group.

Step 2: Create an SNS topic Cost Anomaly Detection can publish to

Email subscribers can receive daily or weekly summaries, but SNS subscribers only receive individual (immediate) alerts. That is what you want for routing.

The topic needs a resource policy that allows the costalerts.amazonaws.com service principal to publish. Without it, the subscription is created but alerts never arrive. If the topic uses SSE-KMS encryption, the key policy must also allow that principal to use the key.

aws sns create-topic --name cost-anomaly-alerts --region us-east-1

# policy.json: allow the Cost Anomaly Detection service to publish
cat > policy.json <<'EOF'
{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "AllowCostAnomalyPublish",
    "Effect": "Allow",
    "Principal": {"Service": "costalerts.amazonaws.com"},
    "Action": "SNS:Publish",
    "Resource": "arn:aws:sns:us-east-1:111122223333:cost-anomaly-alerts",
    "Condition": {"StringEquals": {"aws:SourceAccount": "111122223333"}}
  }]
}
EOF

aws sns set-topic-attributes --region us-east-1 \
  --topic-arn arn:aws:sns:us-east-1:111122223333:cost-anomaly-alerts \
  --attribute-name Policy --attribute-value file://policy.json

Replace the account ID and ARN with your own. The aws:SourceAccount condition is a good-practice guard against cross-account misuse. Confirm the recommended policy in the current docs.

Step 3: Attach a subscription with a meaningful threshold

A subscription links monitors to subscribers and sets the alert threshold. Set it too low and every small fluctuation pages someone, and people then mute the channel. The 75 USD absolute total impact below is only an illustrative starting value. Tune it once you see your own anomaly history.

aws ce create-anomaly-subscription --region us-east-1 --anomaly-subscription '{
  "SubscriptionName": "route-to-lambda",
  "MonitorArnList": ["<SERVICE_MONITOR_ARN>", "<TEAM_PAYMENTS_MONITOR_ARN>",
                     "<TEAM_DATA_MONITOR_ARN>", "<TEAM_PLATFORM_MONITOR_ARN>"],
  "Subscribers": [{
    "Type": "SNS",
    "Address": "arn:aws:sns:us-east-1:111122223333:cost-anomaly-alerts"
  }],
  "Frequency": "IMMEDIATE",
  "ThresholdExpression": {
    "Dimensions": {
      "Key": "ANOMALY_TOTAL_IMPACT_ABSOLUTE",
      "MatchOptions": ["GREATER_THAN_OR_EQUAL"],
      "Values": ["75"]
    }
  }
}'

Older examples use a plain Threshold field, which is deprecated. ThresholdExpression is the current form and also supports percentage-based impact. The two fields cannot be combined, so check the API reference for your CLI version.

Step 4: Enrich and route with a small Lambda

Subscribe a Lambda function to the SNS topic. It reads the anomaly, looks up an owner from a mapping you maintain, and posts a readable message.

The mapping is passed as an environment variable. Its keys are either monitor ARNs (for the per-team tag monitors) or linked account IDs (for spend caught by the service monitor). In a larger setup, keep it in SSM Parameter Store or DynamoDB instead.

import json, os, urllib.request

# Keys: monitor ARNs or linked account IDs, e.g.
# {"arn:aws:ce::111122223333:anomalymonitor/EXAMPLE": {"team": "payments", "webhook": "https://..."},
#  "444455556666": {"team": "data", "webhook": "https://..."}}
OWNERS = json.loads(os.environ["OWNER_MAP"])
DEFAULT = {"team": "finops", "webhook": os.environ["DEFAULT_WEBHOOK"]}

def handler(event, context):
    for record in event["Records"]:
        a = json.loads(record["Sns"]["Message"])
        causes = a.get("rootCauses") or [{}]
        top = causes[0]
        owner = (OWNERS.get(a.get("monitorArn"))
                 or OWNERS.get(top.get("linkedAccount"))
                 or DEFAULT)  # fall back to FinOps
        text = (
            f"Cost anomaly for {owner['team']}\n"
            f"Impact: ${a['impact']['totalImpact']:.2f} "
            f"(expected ${a['impact']['totalExpectedSpend']:.2f})\n"
            f"Service: {top.get('service')} | Region: {top.get('region')} | "
            f"Usage type: {top.get('usageType')}\n"
            f"Details: {a.get('anomalyDetailsLink')}"
        )
        req = urllib.request.Request(
            owner["webhook"],
            data=json.dumps({"text": text}).encode(),
            headers={"Content-Type": "application/json"},
        )
        # Let errors raise: Lambda retries failed async invocations
        urllib.request.urlopen(req, timeout=5)

Allow SNS to invoke the function with aws lambda add-permission, then subscribe the function to the topic. SNS invokes Lambda asynchronously, so a failed webhook call is retried by Lambda's async retry mechanism, not by SNS. Configure an on-failure destination or dead-letter queue so alerts that exhaust their retries are not lost.

Gotcha: the payload's root causes may already include a linked account name. If you resolve extra account details or account tags through AWS Organizations, note that only the management account (or a delegated administrator) can call those APIs. A function in a member account will get access denied unless you provide a cross-account role.

Verify and test

You cannot force a real anomaly on demand, so test in two layers.

First, publish a synthetic message shaped like the documented payload to your topic. This confirms the Lambda parses it, finds an owner, and reaches the webhook. Trim the sample to the fields your code reads, and use a real monitor ARN from your map to exercise that lookup path.

aws sns publish --region us-east-1 \
  --topic-arn arn:aws:sns:us-east-1:111122223333:cost-anomaly-alerts \
  --message '{"anomalyDetailsLink":"https://console.aws.amazon.com/cost-management/home#/anomaly-detection",
    "monitorArn":"<TEAM_PAYMENTS_MONITOR_ARN>",
    "impact":{"totalImpact":240.5,"totalExpectedSpend":60.0},
    "rootCauses":[{"service":"Amazon EC2","region":"eu-west-1",
      "linkedAccount":"111122223333","usageType":"BoxUsage:m5.4xlarge"}]}'

# Check the function logs for parse errors or webhook failures
aws logs tail /aws/lambda/cost-anomaly-router --since 10m --region us-east-1

Second, confirm the real path:

Run aws ce get-anomalies with a --date-interval to see whether the monitors have produced history.
Check the Cost Anomaly Detection console for subscription status.
Watch for the first genuine alert.

If the topic receives nothing while the console shows anomalies, recheck the SNS topic policy (and the KMS key policy, if the topic is encrypted). That is a frequent cause and the first thing to rule out. A related failure is a tag monitor that never fires. This happens when the tag was activated recently or when the values do not match exactly, including case.

Also confirm the fallback works by testing an account ID and monitor ARN that are missing from the owner map. The alert should land in the FinOps channel, not disappear.

Where this leaves you

Routing is only as good as the ownership data behind it. Treat the owner map and the tag policy as living configuration, and update the map whenever you add a team monitor or a new account. Revisit the threshold after a month of real alerts. Keep the catch-all channel staffed so unowned spend gets triaged instead of ignored.

For more AWS operational patterns, browse the rest of kuryzhev.cloud. Re-check payload fields and API limits against the official documentation, since both can change between releases.

WireGuard Key Rotation Checklist for Multi-Peer Setups

Oleksandr Kuryzhev — Fri, 30 Oct 2026 08:01:44 +0000

Originally published on kuryzhev.cloud

WireGuard key rotation usually gets planned the day after a laptop disappears from a taxi, when nobody can say which of the 40 peers on the hub belongs to it. The tunnel itself is simple. The operational picture around it (who holds which key, which address belongs to whom, what breaks if a key changes) is the part that rarely gets written down.

This checklist is for hub-and-spoke or partial-mesh deployments with enough peers that rotating by hand and from memory becomes risky. It assumes Linux hosts using wg and wg-quick, though the logic applies to other implementations.

Why this WireGuard key rotation checklist exists

WireGuard negotiates fresh session keys every couple of minutes, but the static keypair identifying each peer never changes on its own. There is no certificate expiry, no revocation list, and no built-in rotation. A peer's public key is its identity, so rotation is an operational process you have to design yourself.

Two properties make multi-peer rotation tricky. First, a given AllowedIPs range maps to exactly one peer on an interface, so you cannot keep the old and new key of the same peer active with the same address. Second, the hub's key is configured on every spoke, so rotating it touches the whole fleet at once.

A typical failure is a rotation that succeeds on the hub and is never finished on the spoke. The tunnel silently stops, and the only symptom is a handshake timestamp that never advances. The list below is meant to prevent that: plan, stage, verify, and keep a way back. Confirm details against the official WireGuard documentation and the wg(8) man page for your tool version.

The checklist

Build an inventory first. For every peer, record public key, assigned AllowedIPs, owner, creation date, and whether a preshared key is used. A spreadsheet or a Git-tracked file works. Without this, you cannot tell which keys are stale.
Set a rotation trigger. Decide on a schedule (for example, every 6 to 12 months) plus event triggers: device loss, staff departure, suspected disclosure. Write both down.
Generate keys on the peer itself. The private key should never travel. Only the public key (and, if used, a preshared key delivered over a separate secure channel) moves between machines.
Back up the current config. Copy the hub's config and the output of wg show before touching anything. This is your rollback.
Pick a cutover pattern per peer. Because one address maps to one key, either swap atomically on the hub (remove old, add new in one command) or give the new key a temporary address and move traffic over.
Rotate the spoke, then the hub entry. Apply the new private key (and preshared key, if used) locally, then apply the swap on the hub immediately after. Keep the gap between the two steps to seconds. If you manage the spoke over the tunnel itself, have out-of-band access ready, because the tunnel drops until both sides match.
Verify the handshake. Generate some traffic, since handshakes only happen when packets flow. Then check wg show wg0 latest-handshakes for the peer and test real application traffic, not just ping to the tunnel address.
Persist the change. Runtime changes made with wg set are lost on restart unless the config file is updated too.
Remove the old public key everywhere. Check the hub, any secondary hubs, and any spokes that peer directly with it.
Update the inventory and archive the old private key securely, or destroy it. Record the rotation date.

The next block shows steps 3, 5 and 6 for a single spoke. The first part runs on the peer, the second on the hub.

# On the peer: umask keeps new files unreadable by other users
umask 077
wg genkey | tee /etc/wireguard/wg0.key.new | wg pubkey > /etc/wireguard/wg0.pub.new
wg genpsk > /etc/wireguard/wg0.psk.new   # optional per-peer preshared key

# On the peer, at cutover: switch to the new private key and set the
# preshared key for the hub peer (omit the second line if no PSK is used)
wg set wg0 private-key /etc/wireguard/wg0.key.new \
  peer HUB_PUBLIC_KEY preshared-key /etc/wireguard/wg0.psk.new

# On the hub: one command removes the old peer and adds the new one,
# so the AllowedIPs move without a window where neither key is configured.
# peer-b.psk must contain the same value as the peer's wg0.psk.new,
# copied over a separate secure channel.
wg set wg0 \
  peer OLD_PUBLIC_KEY remove \
  peer NEW_PUBLIC_KEY preshared-key /etc/wireguard/peer-b.psk allowed-ips 10.8.0.12/32

Treat the hub's own key as a separate project. One workable pattern is to bring up a second interface with the new key on a different ListenPort and its own tunnel subnet. Overlapping AllowedIPs on two interfaces produce conflicting routes. Then migrate spokes in waves, updating their Endpoint port and addresses, and retire the old interface when the last handshake moves over.

Commonly missed items

Watch out for a stale Endpoint or PersistentKeepalive assumption. A peer behind NAT that was reachable because of keepalives from the old config may go quiet if the new config omits them. The hub cannot initiate to it until the spoke sends a packet first.

Watch out for SaveConfig = true. With it enabled, wg-quick rewrites the config file from runtime state on shutdown. A hand-edited file can be overwritten by the old in-memory state. Pick one source of truth, and verify the behavior in the wg-quick man page for your version.

Other items that tend to slip through:

Preshared keys. They are symmetric and per peer pair. Rotating the keypair but leaving a years-old preshared key in place defeats part of the purpose.
Private key file permissions. Config files containing PrivateKey should be mode 600, owned by root. Backups and config-management copies often end up world-readable.
Keys in shell history, tickets, and chat. Public keys are fine to share. A private key pasted into a ticket must be treated as compromised.
Firewall rules and DNS entries tied to the old key or address. If you used a temporary address during cutover, remove it afterward.
Monitoring that keys off the public key. Dashboards and alerts labeled by old key will show a "missing" peer and a new unnamed one.
Offline peers. A laptop that is off for a month will return with the old key. Decide in advance whether it gets re-enrolled or stays blocked.

Automation ideas

Rotation by hand is manageable for a handful of peers and becomes error-prone as the peer count grows. The goal of automation is not to hide the process but to make the inventory, the swap, and the verification repeatable. Start with the pieces that cause the most mistakes.

Staleness reporting. A scheduled job can list peers with no recent handshake, which catches both dead devices and half-finished rotations. For example:

# Report peers with no handshake in the last 10 minutes (timestamp 0 = never)
now=$(date +%s)
wg show wg0 latest-handshakes | while read -r key ts; do
  if [ "$ts" -eq 0 ] || [ $((now - ts)) -gt 600 ]; then
    echo "stale: $key"   # feed this into your alerting or ticketing
  fi
done

Note that an idle but healthy peer without keepalives can also show an old handshake, so tune the threshold to your traffic pattern.

Config generation. Keep peer definitions as data (YAML or similar) in Git and render wg0.conf from it with Ansible, Terraform, or a small script. Reviews of the rendered diff catch duplicate AllowedIPs before they reach the hub. Apply changes with wg syncconf wg0 <(wg-quick strip wg0) so existing sessions are not torn down; verify this against your installed version.

Secret handling. Store private keys in a secrets manager or generate them on the host and never export them. A timer-driven job on each spoke can generate a new keypair, submit only the public key to an enrollment API, and wait for confirmation before switching.

Staged rollout. Rotate a canary peer first, check the handshake, then proceed in batches with automatic halt on failed verification. For more infrastructure operations material, see kuryzhev.cloud.

Whatever you automate, keep the manual rollback path documented and tested. The backup from step 4 is only useful if someone has restored it before.

Ansible Rolling Deployment: 6 Tips for Zero-Downtime Releases

Oleksandr Kuryzhev — Thu, 29 Oct 2026 08:01:43 +0000

Originally published on kuryzhev.cloud

A single ansible-playbook site.yml that restarts every web node at once turns a routine release into a short outage. An Ansible rolling deployment avoids that by updating hosts in small batches. Each host is taken out of the load balancer first and put back only after it proves healthy. The tips below are independent, so use whichever fits your setup.

The examples assume an inventory group web, an HAProxy host lb1, and a /healthz endpoint on each app node. Swap in your own load balancer module and health check as needed.

1. Start with one host, then widen the batches

Use a list for serial so the first batch acts as a canary.

serial accepts integers, percentages, or a list of both. A list such as [1, "25%", "100%"] updates one host, then a quarter of the fleet, then everything that remains. If the canary breaks, the blast radius is one node. Percentages are calculated against the hosts in the play, so they scale as the fleet grows.

- name: Rolling application release
  hosts: web
  serial: [1, "25%", "100%"]      # canary, then a quarter, then the rest
  max_fail_percentage: 0          # any failure in a batch stops the rollout
  tasks:
    - name: Deploy release
      ansible.builtin.include_role:
        name: app_release

Watch out for: max_fail_percentage is evaluated per batch, not across the whole play. With a one-host canary and a 30% threshold, a single failure is already 100% of that batch. Check how the threshold behaves at your smallest batch size.

2. Drain each node before touching it

Remove the host from the load balancer in pre_tasks and delegate the call to the balancer.

Restarting a service that is still receiving traffic drops in-flight requests, even when the batches are small. Disable the backend server first and wait for connections to finish. The delegate_to keyword runs the task on the balancer while still using the current app host's variables.

  pre_tasks:
    - name: Drain node from HAProxy
      community.general.haproxy:
        backend: app_backend
        host: "{{ inventory_hostname }}"   # must match the server name in haproxy.cfg
        state: disabled
        drain: true                        # let existing sessions finish
        wait: true
        wait_retries: 30
        wait_interval: 2
      delegate_to: lb1

The module talks to HAProxy through its runtime stats socket. That socket must be configured with level admin in haproxy.cfg, and its path set with the socket option if it differs from the default /var/run/haproxy.sock.

Check the module options with ansible-doc community.general.haproxy, since parameters can differ between collection versions. On AWS, the equivalent is deregistering the target from the target group and waiting for the deregistration delay to pass.

3. Gate re-entry on a real health check

A service that is "started" is not a service that is ready.

For Type=simple or Type=forking units, systemd reports the start as successful as soon as the process is launched. It does not wait for the application to be ready to serve. Only Type=notify units that send an explicit readiness signal change that. Even then, the app may still be warming caches or waiting for its database pool.

Poll the health endpoint with until, and re-enable the node in post_tasks only after it returns 200.

  post_tasks:
    - name: Wait for application health
      ansible.builtin.uri:
        url: "http://{{ inventory_hostname }}:8080/healthz"
        status_code: 200
      register: health
      until: health.status == 200
      retries: 20
      delay: 3                 # up to ~60 seconds before the host is marked failed

    - name: Return node to rotation
      community.general.haproxy:
        backend: app_backend
        host: "{{ inventory_hostname }}"
        state: enabled
      delegate_to: lb1

Make the health endpoint check real dependencies, not just return a static "ok". Otherwise this gate only proves the web server process is running.

4. Flush handlers before the health check

Restart handlers run late by default, so force them to run inside the batch.

Handlers run at the end of the current section (pre_tasks, tasks, post_tasks), not right after the task that notifies them. If your restart handler is notified in tasks, it fires after tasks finishes, which is before post_tasks. That ordering is usually fine. It breaks as soon as someone moves the health check into tasks. Make the order explicit instead of relying on it.

    - name: Render app config
      ansible.builtin.template:
        src: app.conf.j2
        dest: /etc/app/app.conf
      notify: Restart app

    - name: Apply pending restarts now
      ansible.builtin.meta: flush_handlers   # restart happens here, not later

Watch out for: if a task fails before the handler runs, the notified handler is skipped for that host. Set force_handlers: true (or pass --force-handlers) to change that. A node with changed files but no restart is a typical source of "the playbook failed but the files changed" confusion.

5. Decide what a failed host looks like

Know whether you want a broken node left drained or put back.

When a host fails, Ansible drops it from the rest of the play. In an Ansible rolling deployment, that means it stays disabled in the load balancer, because the re-enable task never ran. That is usually the safe outcome, since a bad node does not get traffic. Still, make it deliberate and visible, because a silently drained node reduces capacity until someone notices.

Use block/rescue if you want a rollback step, such as reinstalling the previous artifact. Be aware that a host whose rescue succeeds is not counted as failed. It continues into post_tasks, does not count toward max_fail_percentage, and will be returned to rotation if the health check passes. If you want a rolled-back node to stay drained, or the rollout to stop, end the rescue with ansible.builtin.fail.

Use any_errors_fatal: true if one failure should abort the whole rollout. The current task finishes on the batch, then the play ends for all hosts, and no further batches run.

Pair either choice with an alert or a final summary task. That way a reduced pool is reported, not discovered later.

Also confirm that the remaining healthy nodes can carry the load. With serial: "50%", half the fleet is out of rotation during every batch. The percentage should reflect your headroom, not just your patience.

6. Keep migrations out of the serial play

Run schema changes once, in a separate play, and keep them backward compatible.

The documented behavior of run_once combined with serial is that the task runs once per batch, not once per play. A migration marked run_once: true inside a serial play will execute again for every batch. Put it in its own play ahead of the rolling one.

- name: Database migration (once)
  hosts: web[0]                 # a single host runs the migration
  tasks:
    - name: Run additive migrations
      ansible.builtin.command: /opt/app/bin/migrate --additive-only   # placeholder for your migration tool
      changed_when: true        # the command gives no change signal; verify with your tool

- name: Rolling application release
  hosts: web
  serial: [1, "25%", "100%"]
  # ... pre_tasks, tasks, post_tasks from the tips above

During a rollout, old and new code run side by side against the same database. Use the expand/contract approach: add columns and tables first, deploy the code, and drop old structures in a later release. Without that, even perfect batching causes errors on the nodes still running the previous version.

For serial, max_fail_percentage, and run_once, see the official Ansible documentation on playbook strategies and rolling batches. For delegate_to, see the official Ansible documentation on controlling where tasks run. More automation and CI/CD notes are on kuryzhev.cloud.

Cloudflare Origin Hardening: Authenticated Pulls vs IP Allowlist

Oleksandr Kuryzhev — Wed, 28 Oct 2026 08:02:12 +0000

Originally published on kuryzhev.cloud

A team turns on managed WAF rules and bot challenges, watches the attack traffic drop in the dashboard, and then finds the same junk still hitting the application logs. The cause is usually mundane: the origin server still answers on a public IP, so anyone who learns the address can skip the edge. Cloudflare origin hardening exists to close that gap, and there are two main ways to do it: restrict by network address, or require proof of identity from the connecting client.

Both approaches pair with the same two supporting pieces: Full (strict) SSL between Cloudflare and the origin, and edge rules for bot checks. This article compares the approaches by class so the choice does not depend on a specific plan tier or product revision. Verify plan-specific features in the Cloudflare SSL modes documentation before you commit.

When this choice matters

Edge protections only work on traffic that passes through the edge. A WAF custom rule, a managed challenge, or a rate limit does nothing against a request sent straight to the origin IP. Origin addresses leak in several ways:

Old DNS records.
Certificate transparency logs that reveal hostnames, some of which resolve to unproxied IPs.
Mail headers.
Forgotten subdomains that are not proxied.
Internet-wide scan data.

The choice matters most in these situations:

You rely on Cloudflare bot checks or rate limiting to protect login, checkout, or API endpoints.
The origin has an expensive code path, such as search or report generation, that a direct flood could exhaust.
You recently moved a site behind Cloudflare, so the old origin IP may already be known.
Compliance or customer contracts require that traffic is inspected before it reaches the application.

It matters less for a static site served from object storage, where there is no origin to attack. It also matters less if the origin already sits in a private network with no public listener. Rotating the origin IP after the move is cheap and worth doing, but on its own it is obscurity, not a control.

Option A: network-layer lockdown with Full (strict) SSL

The first class of approach is to make the origin firewall accept port 443 only from Cloudflare's published address ranges. Pair it with Full (strict) mode, which makes Cloudflare validate the origin certificate. The certificate must be unexpired, match the hostname, and be issued by a public CA or by Cloudflare Origin CA. The ranges are listed on the Cloudflare IP addresses page.

The following script rebuilds host firewall rules from the published lists. It uses ufw as an example; adapt it to nftables or a cloud security group. Note that it resets all existing ufw rules, so add back any other services the host needs before running it.

#!/usr/bin/env bash
set -euo pipefail

ADMIN_CIDR="192.0.2.0/24"   # placeholder: your management network for SSH

tmp=$(mktemp)
trap 'rm -f "$tmp"' EXIT
# The published files may lack a trailing newline, so add one between them
{ curl -fsSL https://www.cloudflare.com/ips-v4; echo; \
  curl -fsSL https://www.cloudflare.com/ips-v6; echo; } > "$tmp"

# Refuse to continue if the list is empty or contains anything but CIDRs
grep -Eq '^[0-9a-fA-F:.]+/[0-9]+$' "$tmp" || { echo "empty IP list"; exit 1; }
if grep -Evq '^([0-9a-fA-F:.]+/[0-9]+)?$' "$tmp"; then
  echo "unexpected content in IP list"; exit 1
fi

# reset deletes ALL existing rules (including stale ranges) and disables ufw
ufw --force reset
ufw default deny incoming
ufw default allow outgoing
ufw allow from "$ADMIN_CIDR" to any port 22 proto tcp   # keep SSH access
while read -r cidr; do
  [ -z "$cidr" ] && continue
  ufw allow from "$cidr" to any port 443 proto tcp   # HTTPS only from Cloudflare
done < "$tmp"
ufw --force enable   # requires IPV6=yes in /etc/default/ufw for v6 rules

Pros

Works with any web server, load balancer, or cloud firewall, with no TLS changes beyond Full (strict).
Drops scanner traffic before it reaches the TLS handshake, which keeps logs quiet.
Easy to audit: the rule set is a list of CIDRs.

Cons

It proves the packet came from Cloudflare, not from your zone. Any Cloudflare customer can proxy to your origin IP from those same ranges, bypassing your WAF rules.
The ranges change occasionally, so the list needs scheduled refreshes. Watch out for a stale list after a range is added; it can cause intermittent 521 or 522 errors that look like an application fault.
Forgetting IPv6 leaves a second door open on dual-stack hosts.

Option B: identity-based origin hardening with Authenticated Origin Pulls

The second class asks Cloudflare to present a client certificate when it connects, and the origin rejects any connection without one. This is mutual TLS, documented as Authenticated Origin Pulls. Because the check is cryptographic, it does not depend on source addresses staying stable.

Here is an nginx server block that enforces it. The CA file comes from the Cloudflare documentation, or from your own CA if you use a custom certificate. Enable Authenticated Origin Pulls in Cloudflare first. If the origin starts requiring a client certificate before Cloudflare sends one, every request fails.

server {
    listen 443 ssl;
    server_name app.example.com;

    ssl_certificate     /etc/ssl/origin/app.pem;   # Origin CA or public CA cert
    ssl_certificate_key /etc/ssl/origin/app.key;

    # CA that signs the client cert Cloudflare presents
    ssl_client_certificate /etc/ssl/origin/origin-pull-ca.pem;
    ssl_verify_client on;   # nginx answers 400 if no valid client cert is sent

    location / {
        proxy_pass http://127.0.0.1:8080;
    }
}

Apply the same verification to every server block that listens on 443, including the default_server. A request with an unknown Host or SNI value lands on the default block. If that block skips verification, it bypasses the check.

Pros

A direct request to the origin is rejected with HTTP 400 before it reaches the application, even if the attacker knows the IP.
No list of ranges to maintain.
With a zone-level or per-hostname custom certificate, other Cloudflare tenants cannot route through to you.

Cons

Requires TLS configuration on every origin, and not every load balancer or managed platform supports client certificate verification. Check yours.
Watch out for the shared default certificate. The documented default uses a certificate common to all Cloudflare customers, which gives much weaker isolation than a zone-specific one.
Certificate expiry is a new failure mode. A lapsed client cert on the Cloudflare side or a lapsed CA file at the origin takes the site down.

A related variant is Cloudflare Tunnel, where a connector on the origin makes outbound connections and no inbound port is open. It removes the public listener entirely, at the cost of running and monitoring the connector. See the Tunnel documentation for current setup.

Decision matrix

Bot checks do not change the matrix, since both options protect the path that the edge rules depend on. Those rules live at the edge: Bot Fight Mode or its higher-tier equivalents, plus WAF custom rules that issue a managed challenge on sensitive paths. Which bot features you get depends on plan, so verify the current tiers in the Cloudflare docs.

Situation	Allowlist + strict	Authenticated pulls + strict	Tunnel
Legacy origin, no TLS control	Fits	Often impossible	Fits if a connector can run
Concern: other Cloudflare tenants	Does not address	Addresses (custom cert)	Addresses
Operational burden	Range refresh job	Certificate lifecycle	Connector uptime
Public inbound port needed	Yes	Yes	No
Rollout effort	Lowest	Higher; needs staged testing	Moderate; new component to run

Use this checklist before changing anything in production:

[ ] Origin cert valid, hostname matches, not expiring within 30 days
[ ] Test Full (strict) on a staging hostname first (a bad cert returns 526)
[ ] Firewall rules cover IPv4 AND IPv6, and SSH/admin access is preserved
[ ] Health checks and monitoring probes allowed or routed via Cloudflare
[ ] Authenticated Origin Pulls enabled in Cloudflare BEFORE origin enforces it
[ ] WAF custom rule (managed challenge) staged: Log action if your plan has it,
    otherwise scoped to a test hostname first
[ ] Direct-to-origin request from outside Cloudflare now fails
[ ] Alert on cert expiry and on 52x error rate

Evidence-based recommendation for Cloudflare origin hardening

The documented behavior points to a layered default:

Full (strict) SSL always.
A firewall restriction to Cloudflare ranges as the baseline.
Authenticated Origin Pulls with a zone-specific certificate wherever the origin can verify client certificates.

The allowlist is cheap and blocks opportunistic scanning from non-Cloudflare addresses. Mutual TLS closes the remaining hole that an allowlist cannot: another tenant on the same ranges.

Choose the allowlist alone when the origin is a legacy system you cannot reconfigure, the threat is commodity scanning, and you accept the cross-tenant gap. Choose mutual TLS when you run your own reverse proxy such as nginx and the application is a likely target. Choose Tunnel when you can run a connector and want no inbound exposure at all.

Whichever you pick, confirm it works. From a host outside Cloudflare, request the origin IP directly with the correct Host and SNI, for example curl -v --resolve app.example.com:443:203.0.113.10 https://app.example.com/. With a firewall restriction, expect a timeout or refused connection. With Authenticated Origin Pulls in nginx, expect an HTTP 400 complaining about a missing client certificate.

Stage the bot and WAF rules before enforcing them, because challenges can break API clients and webhooks that cannot solve them. Use the Log action where your plan offers it, or scope rules to a test hostname first. For more infrastructure comparisons, browse kuryzhev.cloud. Exact UI labels and plan limits change, so treat the Cloudflare documentation as the source of truth.

Nginx Rate Limiting Checklist for Public APIs

Oleksandr Kuryzhev — Tue, 27 Oct 2026 08:02:21 +0000

Originally published on kuryzhev.cloud

A public API goes live behind Nginx. A week later, one client starts retrying a failing call in a tight loop while a scraper walks every ID. Nginx rate limiting was added "later" as a single limit_req line, and the upstream still falls over because the line keys on the wrong address or never applies to the location that matters. This checklist covers the settings that decide whether limiting protects the API or only looks like it does.

Why this checklist matters for Nginx rate limiting

Rate limiting in Nginx is a few directives, so it is easy to believe it is finished once the config reloads. The failure modes are quiet. A zone keyed on the wrong variable throttles a whole CDN instead of one client, and an empty key disables limiting for that request. A limit set at the server level can vanish when a location adds its own limit_req.

Nginx implements limiting with a leaky-bucket model. limit_req controls request rate, and limit_conn controls concurrent connections. Both are cheap and run before the request reaches your application. Neither knows who your users are, whether a token is valid, or how expensive an endpoint is. Treat them as a first layer, not a complete abuse strategy.

The list below runs roughly from foundational settings to refinements. Behavior described here follows the official limit_req module documentation. Some options depend on the Nginx version: delay= requires 1.15.7, and limit_req_dry_run requires 1.17.1. Verify them against the docs for your build. More operations-focused write-ups are on the kuryzhev.cloud home page.

The checklist

Decide the identity first. Per-IP is the default fallback. Per-API-key or per-account is better for authenticated traffic. Many clients share one NAT address, so IP-only limits punish offices and mobile carriers.
Restore the real client IP. Behind a load balancer or CDN, $remote_addr is the proxy. Configure set_real_ip_from and real_ip_header from the realip module, and trust only addresses you control.
Use $binary_remote_addr as the key. It is more compact than $remote_addr, which matters for zone sizing. According to the documentation, a stored state takes 64 bytes on 32-bit platforms and 128 bytes on 64-bit platforms. One megabyte therefore holds about 16,000 or about 8,000 states. Longer string keys, such as API keys, can use more.
Size the zone for peak distinct clients, not average. When the zone is full, Nginx removes the least recently used state. If it still cannot create a new state, it rejects the request. Plan for attacks that rotate source addresses.
Set rates per endpoint class. Login, token issuance, search, and bulk export deserve separate zones with different limits from cheap reads.
Choose burst deliberately. burst sets a queue size. Without nodelay or delay=, queued requests are delayed to match the rate, which ties up client connections.
Return 429, not 503. Set limit_req_status 429 and limit_conn_status 429. The default is 503, which monitoring and client retry logic read as a server fault.
Send Retry-After. Well-behaved clients back off when told how long. Use a named error location so the header and a JSON body are consistent.
Add limit_conn for expensive streams. Downloads, long polls, and uploads cost connections, not request count.
Tighten timeouts and sizes. Review client_header_timeout, client_body_timeout, send_timeout, keepalive_timeout, and client_max_body_size so slow clients cannot hold sockets indefinitely.
Protect the upstream. Cap proxy_read_timeout and limit proxy_next_upstream_tries, so a rejected or slow call is not retried into a storm.
Test in dry-run first. limit_req_dry_run on (1.17.1 and later) logs what would be rejected without rejecting it.

This is the http-level setup. It keys on the API key when present and falls back to the IP. It also adds a strict per-IP zone for authentication endpoints and exempts trusted internal ranges from every zone.

# http { } context
# Internal monitors and health checks are exempt from all zones (empty key).
geo $is_trusted {
    default      0;
    10.0.0.0/8   1;
}

# Fall back to client IP when no API key header is sent.
map $http_x_api_key $api_client_raw {
    default $http_x_api_key;
    ""      $binary_remote_addr;
}
map $is_trusted $api_client {
    0 $api_client_raw;
    1 "";
}
map $is_trusted $ip_key {
    0 $binary_remote_addr;
    1 "";
}

limit_req_zone  $api_client zone=api_key:20m rate=20r/s;
limit_req_zone  $ip_key     zone=api_ip:20m  rate=50r/s;
limit_req_zone  $ip_key     zone=auth_ip:10m rate=10r/m;  # per minute for fractional rates
limit_conn_zone $ip_key     zone=conn_ip:10m;

limit_req_status  429;
limit_conn_status 429;
limit_req_log_level warn;

upstream api_backend {
    server 127.0.0.1:8080;   # placeholder backend
}

And the server block that applies the zones and answers rejections with a useful body:

server {
    listen 443 ssl;
    server_name api.example.com;
    ssl_certificate     /etc/nginx/tls/example.crt;   # placeholder paths
    ssl_certificate_key /etc/nginx/tls/example.key;

    error_page 429 = @rate_limited;

    location /v1/ {
        limit_req  zone=api_key burst=40 nodelay;  # absorb short spikes, no queue delay
        limit_req  zone=api_ip  burst=100 nodelay; # exceeding either zone rejects the request
        limit_conn conn_ip 20;
        proxy_pass http://api_backend;
    }

    location = /v1/auth/token {
        limit_req zone=auth_ip burst=5 nodelay;
        proxy_pass http://api_backend;
    }

    location @rate_limited {
        default_type application/json;
        add_header Retry-After 5 always;           # 'always' is needed on error responses
        return 429 '{"error":"rate_limited","retry_after":5}';
    }
}

Commonly missed items

Watch out for empty keys. The documentation states that requests with an empty key value are not accounted. That is useful for allow-lists, as above. It becomes a hole when a variable you expected to be populated is not. A key built from a header that clients can omit, or from a variable that is only set after authentication, silently disables the limit.

Watch out for unauthenticated key headers. In the example, a client can send a fresh random X-Api-Key on every request and get a fresh bucket each time. That is why the per-IP zone sits alongside it. Only key on identity that Nginx has actually validated, for example through auth_request or a signed token check. Otherwise, keep the IP zone as the backstop.

Inheritance surprises. limit_req directives are inherited from the parent level only if the current level defines none of its own. Adding one limit_req in a location can drop the server-level limits for that location. Run nginx -T and read the effective config per location.

Per-instance state. Zones live in shared memory on one host. With several Nginx nodes behind a balancer, each node enforces its own limit. A single client can therefore get up to roughly the limit times the node count, depending on how the balancer spreads its requests. If you need a true global quota, that belongs in an API gateway or a shared store.

IPv6. Attackers with a /64 can rotate through enormous numbers of addresses, and per-address limits then do little. Aggregating the key by prefix needs a regex map on the address, so check how your traffic looks before relying on IP keys.

Retries from your own clients. SDKs that retry on 429 without jitter, or without honoring Retry-After, turn a limit into a synchronized wave. Document the header in your API reference.

Automation ideas

Keep limits in version control and treat them as code. Generate the limit_req_zone lines from a small YAML file of endpoint classes and rates, and render them with your configuration tool. Run nginx -t in CI before any deploy. A failing syntax check is cheap to catch there and expensive on a production reload.

Roll out new limits in stages. Enable limit_req_dry_run on where your version supports it (1.17.1 and later). Watch the log for what would have been rejected, then switch dry-run off.

Add a post-deploy smoke test that confirms the limit actually fires. A parallel burst against a non-production endpoint should return some 429 responses that carry a Retry-After header. Run it from an address outside the trusted range, or the exemption will hide the limit.

#!/usr/bin/env bash
# Fire a parallel burst; require 429s that carry Retry-After.
# %header{} in -w needs curl 7.84.0 or later.
URL="https://staging-api.example.com/v1/ping"
results=$(seq 1 200 | xargs -P 20 -I{} \
  curl -s -o /dev/null -w '%{http_code} %header{retry-after}\n' "$URL")
echo "$results" | sort | uniq -c
echo "$results" | grep -q '^429' || { echo "FAIL: no 429 observed"; exit 1; }
echo "$results" | grep -Eq '^429 [0-9]+' || { echo "FAIL: 429 without Retry-After"; exit 1; }

For monitoring, use a JSON log_format that includes $status, $limit_req_status (1.17.6 and later), the client key, and the upstream response time. Export the 429 rate per endpoint to your metrics system and alert on sudden changes in either direction. A spike means abuse or a misbehaving client. A drop to zero after a deploy may mean the limit was lost to inheritance.

Finally, review the numbers periodically. Limits chosen at launch rarely match real usage, and the useful signal is which legitimate clients touch them. Adjust zones from the logs, not from guesses.