DEV Community: Infraforge

How a do-not-disrupt annotation broke Karpenter consolidation

Muhammad Hassaan Javed — Tue, 18 Aug 2026 11:36:38 +0000

Karpenter launched 2,800 nodes over one weekend and consolidated only 180 of them, and our EC2 on-demand bill jumped 2.2x. The cause was one line added to a shared internal Helm chart on Friday afternoon: karpenter.sh/do-not-disrupt: true, applied to every pod template the chart rendered. Renovate auto-merged the bump across 34 tenant repos over the weekend, and by Monday about 1,450 pods carried the annotation. Karpenter's consolidation loop disqualifies any node holding at least one annotated pod, so roughly 85% of nodes became unconsolidatable while new pods kept arriving. Here is how we spotted it and rolled it back without triggering a fleet-wide restart of every workload the chart owned.

Problem signals:

Karpenter's launched instance events run 5-10x baseline while disrupting via consolidation events sit at half of normal
EC2 on-demand line items on the two largest instance sizes double while spot line items stay flat
Node count climbs steadily through a quiet weekend with no matching increase in pod count or RPS
kubectl finds karpenter.sh/do-not-disrupt: true on a majority of pods even though only a handful of workloads legitimately need it
On-demand share of total node hours drifts from ~30% to >60% even though the NodePool weights spot at 100

The metric that redirected us away from HPAs

The launch rate was 7x normal; the consolidation rate was half

Our first read was wrong. The Monday FinOps digest fired at 08:15 UTC showing EC2 on-demand up 2.2x for the trailing 72 hours, and the platform lead's instinct was 'someone shipped an HPA that went sideways over the weekend.' We checked requests per second on the top 20 services; nothing moved more than 8% week over week. Total pod count was up about 40 pods against a fleet of 2,400. Not an HPA problem. Something was provisioning nodes without a matching demand signal, and the extra capacity was mostly on-demand even though the NodePool weighted spot at 100.

The evidence that pointed us at the right layer came from Karpenter's own controller logs. We asked it to summarise its own actions over the incident window:

kubectl -n karpenter logs deployment/karpenter --since=72h \
  | grep -E 'launched instance|disrupting via consolidation' \
  | awk '{print $6}' | sort | uniq -c

 2812 launched
  178 disrupting

# Baseline weekend, same window a fortnight prior:
  412 launched
  347 disrupting

Karpenter was doing both halves of its job. It was just doing one seven times too often and the other half as often as it should.

The provisioning path was healthy. There were no failed provisioning or unschedulable events. Karpenter was launching nodes about seven times as often as a normal weekend and consolidating them at half the usual rate, so capacity was arriving faster than it was leaving. Over 60 hours that gap accumulated into 98 extra nodes, most of them on-demand, sitting quietly and costing money.

One line in a shared chart, 34 tenants downstream

The chart shipped Friday at 15:04 UTC and Renovate did the rest

Once the question was 'why isn't consolidation firing', the answer came out fast. Karpenter's consolidation logic evaluates candidate nodes by checking whether every pod on the node can be safely rescheduled. Any node holding a pod with karpenter.sh/do-not-disrupt: true is immediately disqualified from consolidation until that pod is gone. The annotation exists for legitimate reasons: Kafka Streams state, long-running batches with expensive warmup, workloads that lose data on SIGTERM. In our cluster, four workloads owned it and were the only ones that should have had it.

The reality was different:

kubectl get pods --all-namespaces -o json \
  | jq -r '[.items[] | select(.metadata.annotations["karpenter.sh/do-not-disrupt"] == "true")] | length'

1450

kubectl get pods --all-namespaces -o json \
  | jq -r '[.items[] | select(.metadata.annotations["karpenter.sh/do-not-disrupt"] == "true") | .metadata.namespace] | unique | length'

38

1,450 pods across 38 namespaces carried the annotation. With pods distributed by the default scheduler, that pinned roughly 85% of nodes.

Our first suspicion was a mutating admission webhook. kubectl get mutatingwebhookconfigurations returned nothing new. The annotation was in the pod specs at the source. We picked one pod, walked back to its ReplicaSet, saw the annotation in the pod template, and ran git blame on the tenant's Deployment manifest. The blame line pointed at an internal chart bump: internal-charts/base-deployment moved from 2.7.4 to 2.7.5 on Friday at 19:47 UTC. The diff was three lines:

+ annotations:
+   karpenter.sh/do-not-disrupt: "true"

Commit message: 'add do-not-disrupt to prevent midday restarts during batch runs.'

The chart owner had one tenant whose Kafka Streams StatefulSet was losing ~90 seconds of state on every consolidation event. The narrow fix was to annotate that one StatefulSet, which required coordinating with the tenant. The wide fix was to put the annotation in the shared chart, which required coordinating with no one. They picked the wide fix. Our Renovate config had trust minor bumps from internal charts on auto-merge. Thirty-four tenant repos consumed base-deployment. Over the weekend, thirty-four PRs opened, thirty-four PRs merged, thirty-four ArgoCD syncs rolled deployments, and every rolled pod inherited the annotation.

A single opinionated chart plus a permissive auto-merge is the shape of the whole incident. Nothing in the loop was malicious. Every step was doing what it was configured to do.

The right recovery is on the pods, not the templates

The patch we almost ran would have rolled 1,450 pods

By 09:00 UTC Monday we had the diagnosis and a bad plan. The obvious move was to patch the Deployment template in each of the 34 tenants, removing the annotation at the source:

kubectl patch deployment <name> -n <ns> \
  -p '{"spec":{"template":{"metadata":{"annotations":{"karpenter.sh/do-not-disrupt":null}}}}}'

The command that looked surgical and was not.

We almost ran the loop. Then one of us asked the question that matters: does changing a template annotation trigger a rollout? It does. The Deployment controller's ComputeHash runs DeepHashObject over the entire PodTemplateSpec, and metadata.annotations is part of the template. Change any pod-template annotation and the template hash changes, which creates a new ReplicaSet, which rolls every pod. The cleanest proof of this is that kubectl rollout restart triggers a rollout precisely by stamping kubectl.kubernetes.io/restartedAt into spec.template.metadata.annotations. If template annotations were excluded from the hash, rollout restart could not work.

If we had run the patch loop across 34 tenants at 09:00 UTC on a Monday, we would have rolled about 1,450 pods in flight, cascaded PDB blocks into deploy pipelines, and (worst) triggered even more Karpenter provisioning as the rollouts churned. The recovery would have looked identical to a second, larger incident.

The correct move was to strip the annotation from the running pods, not from the template. A ReplicaSet reconciles pod count and pod ownership, not annotation drift on pods that already exist. So mutating a live pod's annotations is safe and does not trigger replacement. Karpenter re-checks consolidation eligibility on its next loop (roughly every 30 seconds) and starts disrupting the newly-eligible nodes on its own.

Here is what we ran, iterating one namespace at a time so we could abort if anything looked off:

for ns in $(kubectl get ns -o jsonpath='{.items[*].metadata.name}'); do
  case "$ns" in kube-*|karpenter) continue ;; esac
  kubectl annotate pods -n "$ns" --all \
    karpenter.sh/do-not-disrupt- --overwrite 2>/dev/null || true
done

The trailing dash on the annotation key removes it. Total wall time across ~2,440 pods was 47 seconds.

Verification was two commands, one for the pod count and one for Karpenter's response:

kubectl get pods --all-namespaces -o json \
  | jq -r '[.items[] | select(.metadata.annotations["karpenter.sh/do-not-disrupt"] == "true")] | length'

24

kubectl -n karpenter logs deployment/karpenter --tail=200 \
  | grep 'disrupting via consolidation' | tail -5

...disrupting via consolidation, 3 candidates
...disrupting via consolidation, 5 candidates
...disrupting via consolidation, 2 candidates
...disrupting via consolidation, 4 candidates
...disrupting via consolidation, 3 candidates

Down to 24 annotated pods (the four legitimate workloads plus a 20-pod fraud-scoring batch six hours into its run) and Karpenter firing consolidation events within four minutes.

Over the next 90 minutes node count dropped from 198 to 141 and plateaued. The remaining 24 pods were pinning about 22 nodes: 20 fraud-batch pods spread across 18 nodes, and 4 legitimate pods on 4 more. Consolidation could not touch those. We coordinated a restart of the fraud batch with its owning team. It checkpointed cleanly and restarted onto 5 nodes, tight-packed by Karpenter's provisioning-time bin packing, which freed 13 nodes and took us to 128. Over the next 90 minutes natural pod churn shifted another 20 pods off nodes that had held only transient workloads, and Karpenter consolidated those too. Node count landed at 108 by 15:00 UTC, back inside baseline range.

Then we rolled out the fixed chart. base-deployment 2.7.6 removed the blanket annotation and gated it behind an explicit values.yaml opt-in (karpenter.doNotDisrupt: false default, true for the four legitimate tenants). This rollout DID replace pods, because it changed the template hash, and we scheduled it deliberately. Renovate opened PRs but no longer auto-merged them. The platform team merged them in batches of five with a ten-minute soak between batches so any interaction with PDBs or startup probes surfaced before the next wave. Six hours end to end, no SLO impact.

The changes that landed in the two weeks after

Three controls that would have caught this on Saturday

The postmortem produced three durable controls. Each one closes a specific step in the causal chain above.

The first is a Kyverno ClusterPolicy that only permits the annotation on pods that explicitly opt in via a label. It ran in Audit mode for two weeks, we watched the drift metric go to zero, then we flipped it to Enforce:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-karpenter-do-not-disrupt
spec:
  validationFailureAction: Enforce
  rules:
  - name: require-opt-in-label
    match:
      any:
      - resources:
          kinds: [Pod]
    preconditions:
      all:
      - key: "{{ request.object.metadata.annotations.\"karpenter.sh/do-not-disrupt\" || '' }}"
        operator: Equals
        value: "true"
    validate:
      message: "karpenter.sh/do-not-disrupt requires label karpenter.platform/opt-in=true"
      pattern:
        metadata:
          labels:
            karpenter.platform/opt-in: "true"

A future chart change that adds the annotation without the matching label is rejected at admission and never lands.

The second is an SLO alert on Karpenter's own consolidation metric. karpenter_disruption_actions_performed_total (the metric renamed from karpenter_deprovisioning_actions_performed_total around v0.32; the semantics are the same) was already in our Prometheus scrape config; nobody had put it on a dashboard. We added a Grafana panel showing consolidation actions per hour filtered to action="consolidation", and paged if the rate stayed below 20% of the 7-day rolling median for more than two hours during business hours. Backtesting against the incident, that alert would have fired around 04:00 UTC Saturday, roughly 8 hours in, instead of 60 hours in.

The third is a check in the shared chart's CI pipeline. Any minor bump that touches a scheduling-related annotation key (karpenter.sh/*, karpenter.k8s.aws/*, scheduler.alpha.kubernetes.io/*, node.kubernetes.io/*, cluster-autoscaler.kubernetes.io/*) sets a platform-review-required flag on the PR, and Renovate is configured to leave those PRs open rather than auto-merge them. Non-scheduling changes still flow through the previous auto-merge path unchanged. The tradeoff we now pay is about one platform-team review per quarter on this chart; the incident that check would have caught cost us roughly $3,500 in on-demand overage across the 60-hour window.

We considered tightening the NodePool's disruption.budgets and did not. The root cause was not consolidation being too eager; consolidation was fine when it was allowed to fire. Tightening budgets would slow legitimate consolidation without preventing another annotation-blast. We put a comment in the NodePool YAML pointing to the postmortem so a future engineer does not try to 'fix' the budgets in response to reading it.

FAQ: Karpenter consolidation and the do-not-disrupt annotation

Common questions we get about this pattern

Does karpenter.sh/do-not-disrupt pin just the pod or the whole node? It effectively pins the node. Consolidation eligibility is evaluated per-node, and any node holding at least one annotated pod is disqualified until that pod is gone. One annotated pod on a 32-vCPU node prevents the node from ever consolidating.

Would tightening PodDisruptionBudgets have prevented this? No. PDBs govern voluntary disruption of pods that Karpenter has already decided to touch. The annotation blocks node candidacy upstream of that check, so the eviction call PDBs guard never happens. PDBs sit downstream of the gate that closed here.

Can we detect this via CloudWatch or Cost Explorer alone? Not fast enough. Cost signals arrive at daily granularity at the earliest, and our FinOps digest ran weekly. The signal you want is Karpenter's own consolidation-actions rate from /metrics, which moves within minutes of the problem.

Does this pattern apply to Karpenter v1? The mechanism is the same. The annotation moved under different API groups across versions, and v1 uses karpenter.sh/do-not-disrupt on pods with identical semantics. The Deployment controller's PodTemplateSpec hash behavior is a Kubernetes property, not a Karpenter property, and it does not change.

What if we want every pod in a Deployment to be do-not-disrupt by design? Put the annotation in the template and accept that changing it triggers a rollout. Use maxSurge and PDBs to control that rollout the way you would for any other template change. The mistake in our incident was not that a template had the annotation; it was that 34 templates got it without the owning teams knowing.

Where Karpenter regressions get stuck

If your on-demand line jumped and no one shipped anything

The awkward thing about this class of Karpenter regression is that every piece looks correct in isolation. The annotation was a real feature intended for real workloads. The chart bump was a legitimate response to a real production pain. The Renovate auto-merge policy was tuned to reduce toil on well-behaved chart bumps. The NodePool was configured the way the docs recommend. It took the combination, plus a weekend, plus a weekly cost digest, to produce a doubled node count and a mid-four-figure overage. The diagnostic move that mattered was comparing Karpenter's launch rate against its consolidation rate. The recovery move that mattered was knowing which kubectl mutations trigger a rollout and which do not.

We have written more on this shape of failure in the Kubernetes and CI/CD stabilization pillar, and the specific pattern of a shared chart change cascading through GitOps sits in the ArgoCD and GitOps recovery cluster. If your Karpenter cluster is growing and you cannot see why, book an infrastructure review and we will pull the consolidation-actions metric together, walk your controller logs the same way we walked ours, and get you a diagnosis inside a single working day.

Originally published at https://infraforge.agency/insights/karpenter-consolidation-broken-by-do-not-disrupt-annotation/.

If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.

How we recovered a k3s cluster after its client certs expired

Muhammad Hassaan Javed — Thu, 13 Aug 2026 14:54:52 +0000

119 ephemeral preview namespaces should have been reaped between 2am and 6am. None were. The teardown cron had been failing on x509: certificate has expired or not yet valid for six hours before anyone looked at the log. The k3s server was 366 days old, nobody owned certificate rotation, and the first fix we tried made zero difference because we misread which cert was expired. This is how we got kubectl working again, in what order we restarted the control plane, and the two things we changed so the next cluster does not do this to us.

Problem signals:

kubectl get nodes returns 'Unable to connect to the server: x509: certificate has expired or not yet valid'
A CI or lifecycle script that used to work is now failing on every kubectl call, and the cluster has been up more than 11 months
systemctl status k3s shows the service is active but the k3s.service journal has TLS handshake errors from the agent
The kubeconfig at /etc/rancher/k3s/k3s.yaml decodes to a client cert whose notAfter is in the past
Ephemeral or preview environments are stuck: nothing is being created, nothing is being deleted, and the queue is growing

What we walked into

119 namespaces that should have been gone by 6am

The cluster ran preview environments for a mid-size SaaS product. Every merged PR got a namespace, every closed PR got its namespace reaped by a cron that shelled out to kubectl. Simple, effective, and completely dependent on kubectl being able to talk to the API server. On the morning of the incident the reap job had been erroring every minute for about six hours before the platform on-call noticed the namespace count on the Grafana panel had gone flat instead of sawtoothing.

The error in the cron log was one line, repeated:

Unable to connect to the server: x509: certificate has expired or not yet valid: current time 2025-04-12T06:14:22Z is after 2025-04-11T18:02:17Z

The API server had been unreachable for twelve hours and twelve minutes by the time we opened the log.

This is a single-server k3s install with one agent node, running on two small EC2 instances. It had been up for 366 days, one day longer than its own certificates were valid for. Nobody had ever rotated a certificate on it. Nobody had written down that k3s issues its own internal client certs with a 12 month lifetime and quietly renews them on restart if they are within 90 days of expiry. Ours were nowhere near that window when the server last restarted, because the last restart was the kernel patch cycle that brought this cluster up in the first place, twelve months earlier. So they aged out on schedule and the cluster locked itself out.

The first thing we tried, and why it did not work

The kubeconfig regen that fixed nothing

The instinct on the first responder was correct-looking and wrong. They assumed the kubeconfig on the CI runner had drifted, copied a fresh /etc/rancher/k3s/k3s.yaml from the server, rewrote the server URL, and reran the reap. Same x509 error. They tried it a second time with KUBECONFIG pointed explicitly at the new file. Same error.

The reason it did not work is the piece that catches people out with k3s specifically. The k3s.yaml on disk is not a pointer to cert files; it is a self-contained kubeconfig with client-certificate-data and client-key-data base64-embedded directly in it. Regenerating the file by copying it from the server just copies the same expired bytes to a new path. The cert material lives at /var/lib/rancher/k3s/server/tls/client-admin.crt, and that is what had actually expired. Until that file was rotated, every kubeconfig on every machine, freshly copied or not, was carrying the same dead cert.

This is the point where we stopped guessing and asked openssl what it saw.

How we confirmed the actual expiry before touching anything

openssl said notAfter=Apr 11

Before running any rotation command we wanted to see the cert dates ourselves. Two commands, run on the k3s server:

$ sudo openssl x509 -noout -dates -in /var/lib/rancher/k3s/server/tls/client-admin.crt
notBefore=Apr 11 18:02:17 2024 GMT
notAfter=Apr 11 18:02:17 2025 GMT

$ for f in /var/lib/rancher/k3s/server/tls/client-*.crt /var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt; do
>   echo "$f"
>   sudo openssl x509 -noout -enddate -in "$f"
> done

The client-admin cert expired at 18:02 the previous evening. Every other client cert on the server had the same notAfter within a few seconds.

The serving cert for the API server itself was also expired. That mattered for the restart order later, because a k3s server with an expired serving cert will start, but agents cannot re-establish TLS to it until the rotation writes new material and the server picks it up. If we had restarted the agent first, or in parallel, we would have watched it fail to rejoin and then chased a second ghost.

Related reading: the same failure mode shows up during cluster migrations when a snapshot from an old server gets restored past the cert lifetime. We wrote about that pattern in the migration recovery notes.

The recovery sequence that actually worked

Rotate, restart server, restart agent, in that order

k3s ships a rotation subcommand that regenerates the internal certs on disk. It does not restart the service, and it does not touch k3s.yaml; the restart in step 3 is what rewrites that file. The second trap is everything downstream of the server, which we get to below. The full sequence, run as root on the server, was:

# 1. Stop the server so nothing is holding the old TLS material.
sudo systemctl stop k3s

# 2. Rotate all internal certs. On k3s v1.28+ this rewrites everything
# under /var/lib/rancher/k3s/server/tls including client-admin.crt.
sudo k3s certificate rotate

# 3. Bring the server back up. It will pick up the new certs on start.
sudo systemctl start k3s

# 4. Wait until the API is responsive with the local (root-owned) kubeconfig.
sudo KUBECONFIG=/etc/rancher/k3s/k3s.yaml k3s kubectl get nodes

# 5. On the agent node, restart so it re-handshakes with the new server cert.
sudo systemctl restart k3s-agent

Stop first, rotate, start, verify, then touch the agent. The k3s docs are explicit that the server has to be stopped before rotating, and we have separately seen a rotate against a live server leave the API server holding the old serving cert in memory until a restart anyway.

Step 4 worked from root on the server. On the CI runner, kubectl was still failing. This is where the k3s.yaml quirk from earlier comes back, with the sign flipped. k3s rewrites /etc/rancher/k3s/k3s.yaml on every start, so by the time step 4 ran, the server's own kubeconfig already carried the new embedded material; that is exactly why step 4 worked. Nothing rewrites the copies. Every kubeconfig we had ever scp'd to a runner or a laptop still held the dead bytes, and no amount of restarting the server was going to reach them. We had to export the fresh one and re-seed each consumer by hand:

# On the server, export the kubeconfig k3s rewrote when it started.
# config view --raw only prints what is already on disk; the restart
# in step 3 is what put the fresh cert material there.
sudo k3s kubectl config view --raw > /tmp/k3s-fresh.yaml

# Verify the embedded client cert is not the dead one.
grep client-certificate-data /tmp/k3s-fresh.yaml \
  | awk '{print $2}' | base64 -d \
  | openssl x509 -noout -dates
# notAfter=Apr 12 06:41:07 2026 GMT   <-- future date, good.

# Distribute to the CI runner's kubeconfig path. In our case:
scp /tmp/k3s-fresh.yaml ci-runner:/etc/lifecycle/kube/config
ssh ci-runner 'sudo chown lifecycle:lifecycle /etc/lifecycle/kube/config && sudo chmod 600 /etc/lifecycle/kube/config'

The server's own kubeconfig is correct after the restart. The copies are not, and nothing on the cluster knows they exist. Every consumer holding one has to be re-seeded by hand.

Elapsed time from opening the log to a successful kubectl get nodes on the CI runner was 47 minutes. Most of that was the first-fix detour. The rotation itself took about 90 seconds; the server restart took under 20.

The left branch is where we spent 30 wasted minutes. The right branch is what the runbook now says to do first.

FAQ

Questions the team asked when we posted the internal postmortem

Can I just add insecure-skip-tls-verify to the kubeconfig and move on?

No. The API server is presenting an expired serving cert, but it is also validating incoming client certs. Once the client-admin cert is past notAfter, the server rejects the request during TLS with the same x509 error even if the client stopped verifying. Skipping verification on the client side does not help you here. It also leaves the agent unable to reconnect independently.

Does k3s not auto-renew certs?

It does, but only if the server restarts while the certs are inside the 90 day renewal window. If the cluster runs 12 months straight with no restart, or the last restart happened more than 90 days before expiry, the renewal window is missed and the certs age out. That is exactly what happened to us.

Do I need to restart etcd or anything else on a single-server k3s?

On single-server k3s with the default embedded SQLite or embedded etcd, systemctl restart k3s covers everything the server needs. On an HA k3s with external etcd or multiple servers, you rotate and restart one server at a time and let the API stay reachable through the others. That is a different story and worth its own runbook.

Will the pods restart when I bounce the k3s server?

No. Workload pods run under containerd and are not managed by the k3s systemd unit's lifecycle in a way that restarts them. During our recovery, application pods kept serving traffic the entire time the API was down. Only the control plane was affected. Anything using in-cluster kubeconfigs (operators, controllers) will reconnect once the API comes back.

Is the agent's own node cert also rotated?

It needs its own attention. k3s certificate rotate only rewrites files under /var/lib/rancher/k3s/server/tls on the machine you run it on; it does not reach into the agent's /var/lib/rancher/k3s/agent/. Restarting k3s-agent works because the agent cannot authenticate with its expired client cert, falls back to re-bootstrapping with the node token, and the server issues it a fresh one. That is a fallback path, not a rotation. The deterministic version is to run k3s certificate rotate on the agent node too, servers first, then agents. Either way step 5 is not optional.

Where we help

If your k3s cluster is past its first birthday

The two things we changed after this incident were not glamorous. We added a nightly check that runs openssl x509 -noout -enddate against every cert under /var/lib/rancher/k3s/server/tls and every embedded client-certificate-data in every kubeconfig we own, and pages if any is under 30 days from expiry. It is a 42 line shell script and it has already caught the same class of problem on a second cluster that a client acquired through a merger. We also scheduled a monthly forced restart of the k3s server during a maintenance window. Quarterly is not enough: restarts 91 days apart land inside the last-90-days renewal window at most once per 12 month cert, and only by a day or two. Monthly gives three restarts inside the window. Neither of these prevents the failure by itself; together they close the door.

The hard part of this kind of work is not running k3s certificate rotate. It is knowing before you touch anything which cert is actually expired, what order the components have to come back in for this specific k3s version, and which files the rotation does and does not rewrite. We do this often enough on inherited k3s and k0s clusters, especially ones that came along with an acquisition or a platform-team handover, to know where the traps are on each version. If you have a cluster that is past its first birthday and nobody on the team can point to the last time certs were rotated, book an infrastructure review and we will spend a 60 minute call walking the cert state with you before it fails at 3am. If you are staring at x509: certificate has expired in a log right now, say so in the request and we will be on a bridge the same day.

Originally published at https://infraforge.agency/insights/k3s-certificate-expiration-recovery/.

If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.

Why terraform plan wants to destroy 5 live failover resources

Muhammad Hassaan Javed — Thu, 13 Aug 2026 14:46:10 +0000

By 08:47 the terraform plan output was up on the shared screen and nobody wanted to be the one to type apply. The summary line read 'Plan: 5 to add, 0 to change, 5 to destroy.' Every one of the 5 destroys was a resource actively serving production. The Aurora cluster. The RDS proxy in front of it. Three security groups. The failover had happened at 02:14 that morning. SRE had brought the us-west-2 replacements up by hand while the primary region recovered. Six hours later, terraform did not know they existed.

Problem signals:

terraform plan shows -/+ replace on resources actively serving production traffic
The plan wants to destroy resources that were created out-of-band during a recent incident
aws describe commands confirm the resources exist and are healthy, but terraform state does not know about them
The team has stopped running terraform plan against this workspace because the output is too scary to act on

Five destroy-and-replace actions against live production

The plan wanted to destroy the primary database

The workspace was the one that owned the data layer for the primary application. Its state held the pre-failover Aurora cluster, the RDS proxy, and three security groups, all with us-east-1 ARNs. During the incident someone had updated the module's provider region from a variable to a hardcoded 'us-west-2', trying to bring the manually-created resources under management. That change alone did not import anything. What it did was make terraform want to replace every resource in the module, because state said us-east-1 and config now said us-west-2. Five resources in the module, five replacements, five destroys and five creates in the summary.

$ terraform plan

Terraform will perform the following actions:

  # module.data_layer.aws_rds_cluster.primary must be replaced
-/+ resource "aws_rds_cluster" "primary" {
      ~ arn                = "arn:aws:rds:us-east-1:...:cluster:app-primary" -> (known after apply)
      ~ endpoint           = "app-primary.cluster-abc.us-east-1.rds.amazonaws.com" -> (known after apply)
      ~ id                 = "app-primary" -> (known after apply)
      # (27 unchanged attributes hidden)
    }

  # module.data_layer.aws_db_proxy.primary must be replaced
-/+ resource "aws_db_proxy" "primary" {
      ~ arn      = "arn:aws:rds:us-east-1:...:db-proxy:prx-0a1b2c3d4e5f60718" -> (known after apply)
      ~ endpoint = "app-primary-proxy.proxy-abc.us-east-1.rds.amazonaws.com" -> (known after apply)
      # (14 unchanged attributes hidden)
    }

  # module.data_layer.aws_security_group.db_ingress[0] must be replaced
-/+ resource "aws_security_group" "db_ingress" {
      ~ arn = "arn:aws:ec2:us-east-1:...:security-group/sg-0abcdef1234567890" -> (known after apply)
      ~ id  = "sg-0abcdef1234567890" -> (known after apply)
    }

  # module.data_layer.aws_security_group.db_ingress[1] must be replaced
-/+ resource "aws_security_group" "db_ingress" { ... }

  # module.data_layer.aws_security_group.db_ingress[2] must be replaced
-/+ resource "aws_security_group" "db_ingress" { ... }

Plan: 5 to add, 0 to change, 5 to destroy.

The plan that stopped the room at 08:47

The apply would have done exactly what the plan said. It would have called DeleteDBCluster on the us-east-1 cluster, which was still up and still holding connections from the pools that had not been cut over yet. It would have called CreateDBCluster in us-west-2 with the identifier 'app-primary', which was already the identifier on the live cluster that SRE had created at 02:14. That second call would have returned DBClusterAlreadyExistsFault, and the state would have been mid-transaction, with the us-east-1 cluster gone and the us-west-2 cluster still not tracked. The same shape of failure was queued up for the proxy and the security groups, each with its own AWS-side name-collision error.

Why terraform state rm looked right and was wrong

What we almost did that would have made it worse

The reflex in the room was to run terraform state rm module.data_layer.aws_rds_cluster.primary on all 5 entries, then re-plan. State would forget the us-east-1 resources, plan would show 5 creates in us-west-2, and terraform would be back in a consistent-looking place. We had the commands typed. We did not run them, because someone asked the question that saved the afternoon: what does the create actually do when the resource already exists?

The answer is: it fails, and it fails deterministically. CreateDBCluster refuses an identifier that already exists in the region and returns DBClusterAlreadyExistsFault whether or not the properties you are asking for match the live resource. There is no adopt-if-identical path in the AWS API. CreateDBProxy behaves the same way, and duplicate security group names inside a VPC come back as InvalidGroup.Duplicate. Worse, a replace is a destroy followed by a create, so the destroy half lands first: the us-east-1 cluster is gone, the create fails, and state holds five addresses whose real resources terraform just deleted and could not recreate. Sorting that out is a bad afternoon and a worse changelog entry.

The right question is not 'how do we make terraform stop wanting to destroy things'. It is 'how do we get the resources that ARE running into state under terraform management, with the state entries reflecting reality'. Different question, different answer. state rm alone does not answer it, because the plan step after state rm still wants to create. Import blocks answer it, because they tell terraform 'this thing already exists, adopt it, do not create it'.

The Terraform 1.5 import block

Import blocks over state manipulation

Terraform 1.5 shipped import {} blocks in June 2023. Before that, the only tool was terraform import at the CLI, one resource at a time, with no config generation. You had to write the resource block by hand, run the import, then plan until it was empty. For 5 resources that is a slow afternoon. For 30, which is the size of the recovery we have done more often after a full-region failover, it is a week and a half.

The block form lives in HCL and terraform reads it during plan. Our module already declared all 5 resources (the only thing that had changed was the provider region), so there was nothing to generate: the plan matched each import block to the resource block already in the module and staged the adoption for the next apply. -generate-config-out is the companion flag for the other case, when you are adopting something you have no configuration for at all, and it writes a starter resource block for each import block that has no matching config.

# imports.tf (temporary, delete after one apply cycle)

import {
  to = module.data_layer.aws_rds_cluster.primary
  id = "app-primary"
}

import {
  to = module.data_layer.aws_db_proxy.primary
  id = "app-primary-proxy"
}

import {
  to = module.data_layer.aws_security_group.db_ingress[0]
  id = "sg-0a1b2c3d4e5f60718"
}

import {
  to = module.data_layer.aws_security_group.db_ingress[1]
  id = "sg-1a2b3c4d5e6f70819"
}

import {
  to = module.data_layer.aws_security_group.db_ingress[2]
  id = "sg-2b3c4d5e6f708192a"
}

One import block per resource; the id is the AWS resource ID, the to is the terraform module address

There is a state cleanup step first. The us-east-1 entries at those module addresses have to go, otherwise the import collides with an existing state entry and errors with 'resource already managed by Terraform'. We ran terraform state rm on the 5 stale entries. This is safe here because those state entries no longer match anything we intend to manage from this workspace (the us-east-1 resources are being handled by a separate decommission workspace). state rm is only dangerous when the plan step that follows tries to create things. The plan step that follows here only reads: import blocks are resolved during plan, and nothing touches AWS until apply.

$ terraform plan

Terraform will perform the following actions:

  # module.data_layer.aws_rds_cluster.primary will be imported
    resource "aws_rds_cluster" "primary" {
        cluster_identifier = "app-primary"
        engine             = "aurora-postgresql"
        engine_version     = "15.3"
        # ... 24 more attributes read from AWS
    }

  # module.data_layer.aws_db_proxy.primary will be imported
  # module.data_layer.aws_security_group.db_ingress[0] will be imported
  # module.data_layer.aws_security_group.db_ingress[1] will be imported
  # module.data_layer.aws_security_group.db_ingress[2] will be imported

Plan: 5 to import, 0 to add, 0 to change, 0 to destroy.

The successful plan after state rm + import blocks: 5 imports, 0 creates, 0 destroys

Read the summary line: 5 to import, 0 to add, 0 to change, 0 to destroy. That is the shape you want, and any add or destroy in it means an import block is pointing at the wrong address, or the module's attributes have drifted from what SRE actually built at 02:14. Fix that before you apply, not after. Commit imports.tf. Apply. The apply runs the 5 imports. After that, delete the import {} blocks. Once the address is in state they are no-ops on every subsequent plan, so leaving them in place is not an error, it is just noise that outlives the reason it was added.

How to align config to reality without overwriting the live resource

The generated HCL is 95% right and the last 5% matters

The generated config reflects what is running. That includes anything SRE added at 02:14 that was not in the pre-incident module. On this recovery, the next plan showed three drifts, all small, all load-bearing.

$ terraform plan

Terraform will perform the following actions:

  # module.data_layer.aws_rds_cluster.primary will be updated in-place
  ~ resource "aws_rds_cluster" "primary" {
      ~ backup_retention_period = 1 -> 7
      ~ tags = {
          - "incident-response" = "2024-11-14" -> null
          - "incident-owner"    = "sre-oncall"  -> null
        }
    }

  # module.data_layer.aws_db_proxy.primary will be updated in-place
  ~ resource "aws_db_proxy" "primary" {
      ~ debug_logging = true -> false
    }

Plan: 0 to add, 2 to change, 0 to destroy.

The next plan after import: the live state differs from the pre-incident module in three specific places

Two tags SRE had added at create time so the resources would be enumerable later. backup_retention_period at 1 day (the AWS default, because the incident timer was ticking and SRE clicked through the console fast) instead of the module's 7. debug_logging = true on the proxy, because SRE wanted verbose logs during the incident. Every one of these was a live state difference from what the module said the resource should look like.

For each drift item the decision is: keep the live state (add to config) or accept the plan (module wins)? On this recovery we kept the two tags for a followup and added them to the module's default tag map. We bumped backup_retention_period in the module to 7 to match the module's intent, accepting the plan on that one. We kept debug_logging=true for one more week and let the plan sit non-empty on that single field until the incident postmortem was done, then flipped it. The tempting move is to apply the whole plan at once and let the module reset everything. Do not do that without walking through each item. The point of the import was to bring the live resource under management, not to overwrite it with pre-incident config that no longer applies.

The other lesson from this recovery went into the incident runbook. When SRE creates resources by hand at 02:14, they now write the resource IDs into incident-resources.txt at the same time and commit it before the incident closes. That file becomes the source of truth for the import blocks the next morning, and cuts the recovery from 'grep CloudTrail for what got created between 02:00 and 03:00' to 'read the file, generate the imports'.

The 30-minute triage for a plan you cannot safely apply

If your terraform plan looks like this right now

The mechanical part of this recovery, the state rm and the import blocks, is a few hours of work. The hard part is the days before that, when the plan sits red and every engineer is afraid to touch it. Meanwhile someone else adds another out-of-band resource, and the next plan gets 12 lines longer, and the fear compounds. We have seen workspaces sit in this state for six months. The team stops running plan against them at all, which is its own kind of terrifying, because now nobody knows what actually differs from IaC.

We run these engagements every week. The failover-import case we have done three times this quarter alone, plus the 'someone tagged everything in the console during audit prep' variant, plus a half-dozen other shapes of out-of-band drift. Every one follows the same pattern: get the running resources into terraform without touching production, then decide what to reconcile and in what order. The reason it works is that the mechanical steps and the judgment steps are kept separate, so nobody applies the plan at 08:47 to make the red output go away.

If your workspace has a plan you cannot apply and no one is sure how it got there, book an infrastructure review and we will be on a bridge with you the same day, starting with a 30-minute diagnostic call to figure out whether the safe recovery is import blocks, moved blocks, a state file surgery, or a fresh workspace. For the class of problem this fits into, see the terraform apply fear pattern and the terraform state recovery playbook.

Originally published at https://infraforge.agency/insights/terraform-plan-destroy-live-failover-resources/.

If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.

How to recover pods a ConfigMap hook race left with empty env

Muhammad Hassaan Javed — Mon, 10 Aug 2026 03:29:46 +0000

If a Helm rollback of your service left a subset of pods in CrashLoopBackOff with empty database credentials while the rest keep serving, you are looking at a ConfigMap deletion race, not a bad rollback. The window opened during the upgrade that failed: a ConfigMap templated as a pre-upgrade hook with before-hook-creation is deleted and recreated on every upgrade, and any pod that restarted inside that gap booted with empty envFrom values and cached a broken connection pool. The rollback is when you noticed, not what caused it, and it does not clean up after itself. The fix is not kubectl rollout restart deployment. That will nuke the healthy pods too. You verify the current ConfigMap is correct, kill only the pods whose env is empty, then patch the chart so the race cannot happen again.

Problem signals:

kubectl get pods shows 3 of 12 replicas in CrashLoopBackOff with restart counts climbing (14, 17, 19) while the rest sit at 0 restarts
Application logs on the crashing pods show pq: password authentication failed for user "" or dial tcp: missing address
kubectl exec broken-pod -- printenv DATABASE_URL prints nothing and exits 1, while the same command on a healthy pod prints the DSN
helm history shows a recent rollback from revision N to N-1 within the last hour
kubectl get configmap -o yaml shows the correct DSN, so the state looks fine from the cluster's perspective and the alerting is confusing

The propagation window nobody documents

Why envFrom pods boot empty when the ConfigMap looks correct now

envFrom and env.valueFrom.configMapKeyRef resolve exactly once, at pod start. What happens when the ConfigMap is missing at that instant turns on one field: optional. With the default, optional: false, the kubelet refuses to start the container at all; the pod sits in CreateContainerConfigError with an Error: configmap not found event, which is loud and easy to diagnose. With optional: true, which plenty of charts set so that a missing config does not block a boot, the variable is simply absent, the client library reads it as an empty string, and the container starts and reports Running. That second case is the one this guide is about, and it is the only one that produces the split-brain symptom below. Nothing propagates later in either case. This is the part that confuses on-call: you look at the ConfigMap now, it is correct, so how can pods be running with empty values? Because those pods started at t=0 when the ConfigMap did not exist, and Kubernetes has no reconciliation loop that re-injects env into a running container.

The failed upgrade creates the window, not the rollback. If your ConfigMap carries a helm.sh/hook annotation (pre-install, pre-upgrade are the common ones) with a hook-delete-policy of before-hook-creation, Helm treats it as an ephemeral hook resource and deletes the existing copy before creating the new one on every upgrade. That delete-then-create is the gap. Helm fires hooks per lifecycle event, so a resource annotated pre-install,pre-upgrade is never touched by helm rollback, which runs pre-rollback and post-rollback only. Two consequences worth holding onto: the rollback did not open the window, and it does not close it either, so after rolling back to 47 the cluster is still holding revision 48's hook ConfigMap. On a healthy cluster with fast API server response, that window is 200 to 900 milliseconds. On a loaded one we have measured it at 4 seconds. Any pod that restarts inside it (crashloop backoff timer firing, HPA scale-up, node eviction) reads no ConfigMap and boots blank.

The 200ms to 4s window, opened by the upgrade's pre-upgrade hook, where a restart lands on a missing ConfigMap.

The tell that you are in this state and not something else: pods with identical spec, identical image, identical ConfigMap reference, have different runtime env. That does not happen from any cause other than the ConfigMap being absent at one pod's start time.

Confirm the rollback actually landed the correct config

Check the chart state before you touch a single pod

Do not restart anything yet. First confirm the ConfigMap now holds the correct value, otherwise you are about to restart pods into the same broken state. Run the three checks in order.

# 1. What revision are we on and did it complete?
$ helm history payments-api -n prod
REVISION  UPDATED                   STATUS      CHART              APP VERSION  DESCRIPTION
47        2024-11-03 14:22:11 UTC  superseded  payments-api-3.4.1 3.4.1        Upgrade complete
48        2024-11-03 14:41:07 UTC  failed      payments-api-3.5.0 3.5.0        Upgrade "payments-api" failed
49        2024-11-03 14:43:52 UTC  deployed    payments-api-3.4.1 3.4.1        Rollback to 47

# 2. Is the current ConfigMap the correct one?
$ kubectl get configmap payments-api-config -n prod -o jsonpath='{.data.DATABASE_URL}'
postgres://app:REDACTED@pg-primary.prod.svc:5432/payments?sslmode=require

# 3. Does its owner-reference or annotations still mark it as a hook?
$ kubectl get configmap payments-api-config -n prod -o yaml | grep -A2 'helm.sh/hook'
    helm.sh/hook: pre-install,pre-upgrade
    helm.sh/hook-delete-policy: before-hook-creation

If line 3 returns a hook annotation, the chart bug that caused this is still present and will fire again on the next upgrade. Note the annotation lists no rollback event, which is why revision 49 left this object exactly as revision 48 created it.

If the DSN value is correct, you can proceed to selective pod recovery. If it is wrong or empty, you are looking at the failed release's config, which the rollback never replaced because the object is a hook, and you need to fix the ConfigMap directly with kubectl apply before doing anything else. We keep the last-known-good ConfigMap under source control precisely for this five-minute window.

The surgical restart, not kubectl rollout restart

Restart only the pods that booted into the gap

kubectl rollout restart deployment/payments-api will restart every pod including the ones currently serving traffic correctly. If you are already down to partial capacity and Postgres has connection limits, doubling the churn is how you turn a partial outage into a full one. Identify the broken pods from their actual runtime env, or from the failure the process logged when it read that env, and delete only those.

# The pod spec cannot answer this. A variable sourced from a ConfigMap keeps
# its valueFrom stanza and never has a resolved .value written back into it,
# and envFrom keys never appear in the spec at all. Read the runtime env, or
# read what the process logged when it read the runtime env.

# Running-but-wrong pods: read the process environment directly.
$ kubectl exec -n prod payments-api-7d9c8f4b6-4kx2m -- printenv DATABASE_URL
# (prints nothing and exits 1: the variable is absent)

# CrashLoopBackOff pods cannot be exec'd into. The previous container's log
# carries the same signal, and this is the sweep that found ours.
$ for pod in $(kubectl get pods -n prod -l app=payments-api -o name); do
    if kubectl logs -n prod $pod --previous --tail=50 2>/dev/null | grep -q 'password authentication failed'; then
      echo "BROKEN: $pod"
    fi
  done
BROKEN: pod/payments-api-7d9c8f4b6-4kx2m
BROKEN: pod/payments-api-7d9c8f4b6-9jvpd
BROKEN: pod/payments-api-7d9c8f4b6-nq7wr

# Delete only those three. The Deployment will recreate them and they will
# read the current (correct) ConfigMap on start.
$ kubectl delete pod -n prod payments-api-7d9c8f4b6-4kx2m payments-api-7d9c8f4b6-9jvpd payments-api-7d9c8f4b6-nq7wr
pod "payments-api-7d9c8f4b6-4kx2m" deleted
pod "payments-api-7d9c8f4b6-9jvpd" deleted
pod "payments-api-7d9c8f4b6-nq7wr" deleted

Runtime env, or the log line the process wrote when it read that env. The pod spec is not ground truth here.

The trap worth naming, because it is the first thing most people reach for: a jsonpath filter over .spec.containers[0].env[?(@.name=="DATABASE_URL")].value looks like it should work and cannot. Kubernetes resolves ConfigMap-sourced variables in the kubelet at container start and never writes the result back onto the PodSpec, so that filter returns empty for every ConfigMap-sourced pod, healthy or broken. On a 12-replica Deployment it flags all 12. kubectl debug with an ephemeral container reading /proc/1/environ is the third way in, useful when the container image has no shell for exec.

Delete the broken pods in batches of one or two, not all at once, to protect the connection pool of the pods that are still healthy.
Watch the new pods reach Ready before deleting the next batch: kubectl get pods -n prod -l app=payments-api -w.
If your Deployment has maxUnavailable set aggressively, note that manual pod deletes bypass rollout controls, so you set the pace by hand.

Stop treating the ConfigMap as a Helm hook

The chart change that closes the window permanently

The root cause was that the ConfigMap was templated as a hook. There is almost never a reason to do this for a ConfigMap that carries runtime configuration. Hooks are for one-shot resources: pre-upgrade DB migrations, post-install seed jobs. A ConfigMap that pods depend on at start time should be a regular chart resource with a stable lifecycle across upgrades and rollbacks.

# templates/configmap.yaml
# BEFORE: this is the bug
apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ include "payments-api.fullname" . }}-config
  annotations:
    helm.sh/hook: pre-install,pre-upgrade
    helm.sh/hook-delete-policy: before-hook-creation
data:
  DATABASE_URL: {{ .Values.database.url | quote }}

# AFTER: regular resource, no hook annotations
apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ include "payments-api.fullname" . }}-config
data:
  DATABASE_URL: {{ .Values.database.url | quote }}

# templates/deployment.yaml: force a rolling restart when the ConfigMap changes
spec:
  template:
    metadata:
      annotations:
        checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}

Remove the hook annotations. Add a checksum annotation to the Deployment pod template so config changes always trigger a controlled rolling restart.

The checksum annotation is the piece most charts get wrong. Without it, changing values in the ConfigMap does not restart pods at all, because Kubernetes sees no change to the Deployment spec. Teams then work around this with a manual kubectl rollout restart after every values change, or a dummy annotation bumped by hand on each deploy, both of which are the same churn the checksum gives you for free and both of which get forgotten under pressure. With the checksum, any ConfigMap content change updates the pod template hash, which triggers a normal Deployment rollout that respects maxSurge and maxUnavailable. There is a tradeoff: you get more rollouts than before, one per config change, and if your ConfigMap has values that churn (feature flags, dynamic settings), you should move those out into a separate ConfigMap that pods read at runtime rather than at start.

We have written the broader pattern for boot-time versus runtime configuration in our Kubernetes stabilization notes, and if this is landing during or right after a migration, the same race appears with any credential source that gets templated as a hook, including some migration recovery playbooks we have run this year.

FAQ

Questions on-call keeps asking after this fires

Can I use kubectl rollout restart safely if the ConfigMap is now correct?

Yes, but only if you have the capacity to churn every pod and your database can absorb the reconnect storm. On a service with 12 replicas and a Postgres max_connections of 200 with pgbouncer in transaction mode, we have done full rollout restarts without incident. On a service that runs hot on connections, do the surgical delete instead.

Why does helm upgrade --atomic not prevent this?

--atomic rolls the release back when the upgrade fails, but the window has already opened by then. The pre-upgrade hook deletes and recreates the ConfigMap before any workload change is applied, so a pod can boot blank while the upgrade is still in flight. The rollback --atomic triggers then runs pre-rollback hooks only, which this ConfigMap is not annotated for, so it leaves the object untouched. --atomic is not a fix for this; removing the hook annotation is.

Does this affect Secret-sourced env the same way?

Yes, identically. envFrom on a Secret has the same start-time-only resolution. If you template Secrets as Helm hooks (some charts do, to inject generated passwords) you get the same race. Same fix: regular resource plus checksum annotation.

How do I know if the race actually fired versus something else broke my pods?

Two-pod comparison. Take one healthy pod and one crashing pod from the same ReplicaSet. If their runtime env for the same key differs (one has DATABASE_URL populated, the other has it empty), it is the ConfigMap race. If both have the same env and one still crashes, look elsewhere: image drift, node-local state, or a downstream dependency.

Can I use a mutating webhook or Reloader to auto-restart on ConfigMap changes instead of the checksum annotation?

Stakater Reloader works and we run it in several client clusters. The checksum annotation is simpler because it is part of the chart and needs no extra controller. If you already run Reloader for other reasons, use it. If you do not, do not install a controller to solve a one-line templating fix.

Getting the surgical recovery right the first time

If you are staring at CrashLoopBackOff right now

The hard part of this recovery is not the kubectl commands. It is deciding which pods to touch when your dashboards are red and the pressure is to do the biggest hammer available. A rollout restart in the middle of a partial outage on a service under connection pressure is how the two-hour incident becomes the six-hour incident. The judgment call is: verify config, inspect env pod-by-pod, delete only what is actually broken, and only then fix the chart so this cannot recur.

At InfraForge we do this recovery with the on-call team on the bridge, in one working session, and we leave the chart patch and the checksum annotation merged before we sign off. If you would rather not run the surgical restart on production by yourself, book a 60-minute infrastructure review and we will walk it with your team today or tomorrow, chart fix included.

Originally published at https://infraforge.agency/insights/helm-rollback-configmap-race-safe-recovery/.

If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.

Worker drops jobs intermittently: startup race, env drift, schema skew

Muhammad Hassaan Javed — Mon, 10 Aug 2026 03:15:00 +0000

If your worker is dropping jobs intermittently, sometimes crashing at boot with 'Error 111 connecting to redis:6379. Connection refused' and sometimes producing an empty result.json with no error at all, the cause is almost never a Redis performance problem. In most of the cases we get called into, it is a startup race between the worker and its dependency, hiding two quieter bugs behind it: an environment variable that silently falls back to localhost, and a job schema that got renamed on one side of the wire. This guide walks the three causes in the order they actually occur, gives you the one log grep that separates them, and shows the fix sequence that does not destroy the evidence you still need.

Problem signals:

Some runs crash at boot with redis.exceptions.ConnectionError: Error 111 connecting to redis:6379. Connection refused. and others start cleanly
result.json is written on some runs, missing on others, sometimes present but with 0 processed jobs and no error in the log
Restarting the worker alone (without the backend) makes the problem go away for a while, then it comes back after the next deploy
A grep for KeyError shows sporadic KeyError: 'job_id' traces that are being caught by a broad except block
The worker container's uid is 1000, /app/output/ is owned by root:root with mode 0755, and nobody remembers why

The symptom, and what it usually is not

Three failure modes wearing one costume

The reported symptom is almost always the same sentence: 'the worker is flaky, sometimes it processes jobs and sometimes it doesn't, and the logs don't really say why.' That sentence hides three separate bugs that compound. We rank them by the order we actually find them, not by how loud they are in the log:

1. Startup race. Worker container comes up before Redis (or the backend that populates Redis) is accepting connections. Retries are missing or set to a value that gives up in under 2 seconds. Roughly 60% of the incidents we see.
2. Environment variable drift. The backend sets REDIS_URL, the worker reads CACHE_URL, and the worker's client library silently defaults to redis://localhost:6379. It 'connects' to nothing and hangs or reads empty queues. Roughly 25%.
3. Schema skew after a rename. Someone renamed job_id to id on the producer side and missed one .get('job_id') on the consumer. .get returns None, the worker's outer except Exception swallows the downstream KeyError, the job is skipped, no error is logged. Roughly 15%, and the hardest to see.

The cause everyone blames first, and which is almost never it: 'Redis is slow' or 'the worker needs more memory'. We have not once found this to be the actual cause in this failure shape. If you are already sizing up the Redis instance, stop and read the next section first.

There is also a fourth cause that shows up as a hard PermissionError: [Errno 13] Permission denied: '/app/output/result.json' when the worker finally does try to write. It is real, but it is loud. This guide is about the quiet failures.

The discriminating check: read the logs before you touch anything

One grep that ranks the three

Before you restart, redeploy, or scale anything, capture the current state. Restarting the worker throws away the evidence that tells you which of the three you are dealing with. On docker compose, snapshot both services' logs to disk. On Kubernetes, capture kubectl logs --previous for the last crashed worker AND kubectl describe pod for its events. Do this first.

# capture before you touch anything
docker compose logs --no-color --timestamps worker  > /tmp/worker.log
docker compose logs --no-color --timestamps backend > /tmp/backend.log

# k8s equivalent
kubectl logs -n jobs worker-7c9d8f5b6-x2k4m --previous > /tmp/worker.log
kubectl describe pod -n jobs worker-7c9d8f5b6-x2k4m       > /tmp/worker-describe.txt

# then run the discriminating grep
grep -E 'Connection refused|CACHE_URL|localhost:6379|KeyError' /tmp/worker.log | head -40

Snapshot first, grep second. Anything that recreates the container (docker compose up --force-recreate worker, or a down followed by an up) discards the previous container's stdout under the default json-file driver. A plain restart keeps the log file, but it costs you the live process state that made the failure reproducible.

The grep output tells you which cause you have, in this order:

If you see Error 111 connecting to redis:6379. Connection refused in the first 5 seconds of the worker's log and never again after that, it is the startup race. The worker gave up before Redis was ready.
If you see Connecting to redis://localhost:6379 (note: localhost, not the service name), or you see NO connection log at all and the queue depth reads always return 0, it is env drift. The worker never got the right URL and its client defaulted.
If the connection logs are clean, the worker is clearly consuming messages, but the count of 'processed' log lines is lower than the count of 'received' lines, and you can find any KeyError (even one, even caught) in the traces, it is schema skew. The worker is silently dropping jobs whose payload does not match its expected shape.

Order matters. The race hides the drift, and the drift hides the schema skew. Fix them in the order they surface.

The safe fix sequence for each cause

Fix least-destructive first

Fix each cause with the smallest change that discriminates against the others. Do not batch the three fixes into one PR; you will not know which one worked, and the next incident will look identical. Ship them separately, verify each one, then move on.

Step	What it does
1. Startup race	Add a real readiness gate, not a sleep. On docker compose, use `depends_on` with `condition: service_healthy` and a healthcheck on the backend/Redis. On Kubernetes, use an initContainer that runs `nc -z redis 6379` in a loop with a bounded timeout (60s), and set the worker's client to retry with exponential backoff for at least 30s after boot. `sleep 10` in the entrypoint is the fix everyone reaches for; it papers over the problem until the day Redis takes 11 seconds.
2. Env var drift	Rename one side to match the other in a single PR. Do NOT add a fallback like `CACHE_URL or REDIS_URL`; that is how you got here. Then add a boot-time assertion: if the resolved URL is `localhost` or empty, the worker exits with a clear error instead of connecting to nothing. This is the single change that pays for itself the fastest.
3. Schema skew	Find every `.get('job_id')` and `.get('id')` across producer and consumer. Pick one name (we prefer `id` because it is what most queue libraries default to). Add a schema validation step at the consumer boundary that raises loudly on a missing key, and remove any `except Exception: pass` you find on the way. The broad except is what turned this into a silent bug.
4. Output permissions	If `PermissionError` on `/app/output/result.json` is in your logs, the fix is a Dockerfile line: `RUN mkdir -p /app/output && chown -R 1000:1000 /app/output` before the `USER 1000` directive. Do not `chmod 777`. If the directory is a mounted volume, set `fsGroup: 1000` in the pod's securityContext instead.

One thing to name explicitly: the tempting single-line fix of adding sleep 15 to the worker's entrypoint 'solves' the startup race in staging and hides all three bugs in production. We have watched this exact patch get merged, celebrated, and then paged the same team six weeks later when Redis restarted during a maintenance window and the sleep was not long enough. The cost of the right fix (a healthcheck plus retry) is roughly 40 lines of yaml and one afternoon. Pay it.

# docker-compose.yml, the readiness gate that actually works
services:
  redis:
    image: redis:7.2-alpine
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 2s
      timeout: 1s
      retries: 15

  backend:
    depends_on:
      redis:
        condition: service_healthy

  worker:
    depends_on:
      backend:
        condition: service_started
      redis:
        condition: service_healthy
    environment:
      REDIS_URL: redis://redis:6379/0    # one name, both sides
    # no sleep, no CACHE_URL fallback

The gate is condition: service_healthy plus a real healthcheck on the dependency. condition: service_started alone only waits for the container to exist, not to be ready.

Confirm the fix by running the stack from cold at least ten times in a row and checking that the processed count equals the received count on every run. If nine out of ten pass, you have not fixed it; you have improved it. Race conditions do not get 90% fixed. For deeper patterns on this kind of intermittent K8s failure, we have written up Kubernetes release failure recovery separately.

FAQ: variants readers usually ask right after

The questions that come next

Is it safe to just add restart: always and let the worker crash-loop until Redis is up? It works, but it makes your logs noisier, it costs you real seconds on every boot, and it hides the underlying dependency graph from anyone reading the compose file later. Use a healthcheck. Save restart: always for actual transient failures in steady state.

Does this apply to RabbitMQ, NATS, or Kafka workers too? Yes, the shape is identical. The verbatim error string changes (Connection refused on RabbitMQ, dial tcp: connection refused on NATS, NoBrokersAvailable on Kafka), but the three causes and the fix order are the same. The env var drift bug is especially common on Kafka clients because KAFKA_BROKERS vs BOOTSTRAP_SERVERS is a coin flip in the ecosystem.

Can I skip the healthcheck if I use Kubernetes with readiness probes? No, readiness probes gate traffic to a pod, not startup order between pods. You still need an initContainer or an application-level retry loop for the worker to wait on its dependency. Readiness alone will not save you.

Why not just fix the broad except Exception: pass and call it a day? Because removing it in isolation will surface the schema skew as a hard crash on production traffic, and if you have not fixed the startup race first, you will not be able to tell which crash is which. Fix in the order the causes appear at boot: connectivity, config, contract.

How do I stop this from recurring? Three things, in order of ROI: (1) a boot-time assertion in every service that fails fast if a required env var is missing or resolves to localhost, (2) a schema check at the consumer boundary that rejects malformed messages loudly instead of silently, (3) a pre-merge integration test that starts the stack from cold and asserts processed == received on 100 test jobs. That third one is the single highest-value test we recommend for job-processing systems.

If you are staring at an empty result.json and the clock is running

When the worker is dropping jobs right now

What makes this class of failure genuinely hard is not any single one of the three bugs. It is that they compound: the race gives you enough noise in the log that the drift looks like a symptom of the race, and by the time you fix the race, the drift has been silently corrupting queue state for hours, and the schema skew is dropping the recovery jobs you are firing to catch up. Untangling that under time pressure is where teams get stuck at 2 in the morning.

We have spent a lot of engineering hours in exactly this shape of incident, on Redis, RabbitMQ, and Kafka, across docker compose and Kubernetes. The pattern above is what we run on the call: capture logs first, grep to rank the causes, fix in dependency order, verify with cold-start runs. If you want a second set of eyes on it while it is happening, book a same-day infrastructure review and we will get on a bridge with your on-call engineer inside a few hours and work the sequence together. If the fires are already out and you want the pre-merge integration test and boot-time assertions in place before the next one, our platform reliability engagements cover exactly that.

Originally published at https://infraforge.agency/insights/worker-drops-jobs-intermittently-startup-race-env-drift/.

If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.

How a missing S3 gateway endpoint route quintupled our NAT bill

Muhammad Hassaan Javed — Thu, 16 Jul 2026 03:32:55 +0000

The finance lead asked why AWS charged us $2,100 for NAT gateway data processing last month. Our normal was around $400. Nothing in the release calendar explained it: no new services, no traffic bump in our own metrics, no scaling events. The bill just quintupled. Then someone opened VPC Flow Logs and filtered on the NAT's ENI. Roughly 71% of the bytes had destinations in 52.216.0.0/15 and 3.5.0.0/16, which is S3. We had an S3 gateway endpoint on the VPC. It was supposed to be handling that traffic on the AWS backbone for free.

Problem signals:

NAT gateway data-processing charges (NatGateway-Bytes) climb week over week with no code or workload change
VPC Flow Logs show heavy traffic to S3 IP ranges (52.216.0.0/15, 3.5.0.0/16) hitting the NAT's ENI instead of the gateway endpoint
aws ec2 describe-route-tables on a private subnet returns no route for the S3 managed prefix list (pl-xxx)
NAT gateway BytesOutToDestination stays high while VPC Flow Logs show traffic to the S3 prefix list still routing through NAT (the S3 gateway endpoint itself emits no CloudWatch metrics to check)
The S3 gateway endpoint exists in the VPC, but the bill still shows $0.045/GB on service-to-service S3 traffic

$412 to $2,103 on a flat workload, all of it NatGateway-Bytes

The bill line item that should not have existed

The NatGateway-Bytes cost for us-east-1 climbed from $412 in March to $2,103 in April. Nothing else on the bill moved. Egress was flat. EC2 was flat. RDS was flat. The whole delta was NAT data processing at $0.045 per GB.

Cost Explorer's usage-type breakdown confirmed the shape. The entire spike was NAT-Bytes on one VPC, in one region, over a six-week ramp. No cliff, no sudden jump, just a slow gradient upward that nobody watched because 'NAT costs a few hundred bucks' was our mental default.

So we mapped the destinations. AWS publishes the IP ranges for S3, DynamoDB, and every other service in ip-ranges.json. We pulled an hour of VPC Flow Logs, joined destination IPs against those ranges, and found the answer. S3 was 71% of the NAT-processed bytes. That should have been impossible. We had a gateway endpoint for S3, and gateway endpoints are free. Their entire point is to keep S3 traffic off NAT.

Cross-region buckets, then a hardcoded public endpoint, then the truth

What we thought first, and why the first two theories died fast

The first theory was that a new batch job was writing to a bucket in a different region. Cross-region S3 traffic does not hit the same-region gateway endpoint; it goes out through NAT. Reasonable theory, wrong theory. We grepped Terraform for cross-region bucket references and found nothing. We checked CloudTrail for recent bucket creations. Nothing new. All our S3 traffic was to buckets in the same region as the workload.

The second theory was that an application was talking to S3 via the public global endpoint URL instead of the regional one. That is a real failure mode: SDKs pointed at a hardcoded https://s3.amazonaws.com sometimes route to the wrong region and skip the gateway endpoint. Also wrong. Every service in the VPC used the SDK default resolver, which picks the regional endpoint.

The third theory turned out to be right. The route table for one of our private subnets was missing the S3 gateway endpoint's prefix-list association. Any pod scheduled onto a node in that subnet was reaching S3 through the NAT, at $0.045 per GB, for six weeks. Same workload the whole time. Different route table.

The route that has to exist, and the day the migration deleted it

How a gateway endpoint quietly stops covering a subnet

A gateway endpoint is not attached to your subnets the way an interface endpoint is. There is no ENI. There is no private DNS record. The gateway endpoint lives in the VPC, and to make it work for a subnet, you have to add its managed prefix list (pl-63a5400a for S3 in us-east-1) as a route in that subnet's route table, with the endpoint as the target.

The whole 'your S3 traffic bypasses NAT' behavior depends entirely on that route existing. If the route is not there, S3 traffic falls through to the default route (0.0.0.0/0), which points at the NAT gateway. There is no error, no warning, no CloudWatch alarm. The traffic just goes through NAT and gets billed per GB.

We checked all four of our private subnets' route tables for the S3 prefix-list route:

aws ec2 describe-route-tables \
  --filters "Name=association.subnet-id,Values=subnet-0abc123" \
  --query 'RouteTables[].Routes[?DestinationPrefixListId==`pl-63a5400a`]'

Repeat for each private subnet. Empty result means the S3 gateway endpoint is not covering that subnet.

Three subnets returned the S3 prefix-list route. One returned an empty array. That subnet's route table had been rewritten six weeks earlier during a network migration for another team's project. The migration rebuilt the route table from Terraform, but that Terraform module did not know about the gateway endpoint. The endpoint's association was managed by a separate stack. The rewrite silently dropped the S3 route entry.

Nothing failed. Nothing paged. The workload kept running. It just started paying NAT rates for every S3 GET and PUT the pods on those nodes made.

modify-vpc-endpoint, not create-route

The fix, and the command people reach for that does not apply here

The instinct might be to reach for aws ec2 create-route --route-table-id --vpc-endpoint-id to add the endpoint to the table by hand. That does not fit here. Interface endpoints do not use routes at all (they are ENIs with private DNS), and a gateway endpoint's prefix-list route is not added with create-route either. Gateway endpoints get added to a route table by modifying the endpoint itself and telling it which route tables to associate with.

The correct fix is one call:

aws ec2 modify-vpc-endpoint \
  --vpc-endpoint-id vpce-0abc12345 \
  --add-route-table-ids rtb-0def67890

AWS writes the prefix-list route into the route table atomically. No second command.

Verification took thirty seconds:

aws ec2 describe-route-tables --route-table-ids rtb-0def67890 \
  --query 'RouteTables[].Routes[?GatewayId==`vpce-0abc12345`]'

Should now return the pl-xxx prefix-list route with the endpoint as GatewayId.

The route was there. We then pulled a fresh five-minute slice of VPC Flow Logs and joined against S3's IP ranges again. S3 destinations were dropping out of the NAT sample. The subnet was routing them to the gateway endpoint instead. Over the next hour we watched the NAT gateway's BytesOutToDestination CloudWatch metric drop about 40% and level off.

That was the whole fix. Six weeks of overspend, roughly $1,700 in avoidable NAT charges, closed in about four minutes of actual work.

Endpoint-to-route-table coupling in Terraform, plus a rate-of-change NAT alarm

The two things we changed so this stops happening

We changed two things. First, every gateway endpoint in our Terraform is now paired with an explicit list of route-table IDs it associates with, and that list is generated from the same module that generates the private subnets. When someone adds a subnet, the endpoint's association is derived from the same variable. There is no second stack to remember.

Second, we set up a CloudWatch alarm on the NAT gateway's BytesOutToDestination metric with a per-VPC baseline. Not a fixed dollar threshold, a rate-of-change alarm: if NAT bytes for a VPC exceed three times the trailing seven-day median for two consecutive hours, we get paged. We would have caught this ramp on day four instead of week six.

We considered AWS Cost Anomaly Detection. It does catch this shape of spike, and in our archived bills it did fire, eleven days into the ramp. Our own CloudWatch alarm fires in hours because NAT bytes are a real-time metric and the cost bill is not.

For teams reading this with a similar VPC layout, the audit is roughly a ten-minute job. Enumerate every gateway endpoint, enumerate every route table that should be covered, and check for the prefix-list route in each. If you want the deeper cleanup pattern we use for accumulated cloud-cost drift, we have written more of that up in the InfraForge services overview.

This kind of drift does not fail loudly. It just bills.

If your NAT bill just went sideways and nobody deployed anything

The specific shape of this problem is one people miss because it does not fail loudly. Gateway endpoints have no failure mode that produces an error. They just silently stop covering a subnet, and the bill grows. If you have not audited your route tables against your gateway endpoints in the last six months, there is a decent chance one of your subnets is quietly paying NAT rates for S3 or DynamoDB traffic right now.

We have seen this pattern three times this quarter. Two were the same shape as ours: a route-table rewrite that dropped the prefix-list route. One was a subnet added later that never got the association at all. Each was a five-figure annual overrun that took an afternoon to identify and minutes to fix.

If your NAT gateway bill jumped and nobody deployed anything, book an infrastructure review with our team and we will start with a 30-minute diagnostic call this week. We will pull an hour of your Flow Logs, join destination IPs against AWS service ranges, and tell you which subnet is bleeding before the call ends.

Originally published at https://infraforge.agency/insights/nat-gateway-cost-spike-missing-vpc-endpoint-route/.

If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.

How one stuck PDB doubled our EKS autoscaler bill in six days

Muhammad Hassaan Javed — Thu, 16 Jul 2026 03:14:59 +0000

The AWS bill for EKS Compute went from $4,200 to $9,800 in six days. Same cluster, no launch, no traffic event, no team asking for headroom. kubectl get nodes returned 60 m5.4xlarge instances where the working set is normally 20, and forty of them were sitting under 8% CPU. The cluster autoscaler had scaled up overnight for a batch burst that finished by 6 am; it had never scaled back down.

Problem signals:

Your AWS bill for EKS Compute jumps without a matching traffic or deploy event
kubectl get nodes returns dozens more nodes than the working set, most at low CPU utilization
Cluster autoscaler logs repeat 'cannot delete: pdb blocking' and keep naming the same PDB
A single PodDisruptionBudget in the cluster shows ALLOWED DISRUPTIONS = 0 while others are 1 or higher
A workload somewhere in the cluster has a pod stuck in ImagePullBackOff that nobody is watching

The $9,800 bill and 60 nodes at 8% CPU

The Friday morning spike

The bill spike was six days old by the time the AWS Budgets alert fired. It tripped at $9,000 for the billing period; the same window the previous month was $4,200. Nothing had shipped that week, CI had been quiet since Wednesday afternoon, and no product team was asking for extra capacity.

Compute Optimizer had already flagged the cluster's managed node group as significantly overprovisioned. kubectl get nodes returned 60 m5.4xlarge nodes; the normal working set for this cluster is 20. Forty of the extras were sitting under 8% CPU, hosting kube-system DaemonSet pods and a single Deployment nobody had touched in months.

The math is straightforward. m5.4xlarge on-demand in us-east-1 is $0.768 an hour. Forty extra nodes for six days is about $4,400 of surprise burn, most of the jump from the $4,200 baseline. The overnight autoscale had happened on the previous Saturday to run a scheduled batch job; the batch finished by 6 am, and then the cluster autoscaler just stopped scaling down. Six days of that pattern is how a bill doubles.

Why HPA runaway and traffic burst were both wrong

Two theories that died fast

Two theories came up on the bridge in the first five minutes. The obvious one was HPA runaway: a Horizontal Pod Autoscaler misreading its metric and scaling a Deployment to hundreds of replicas, forcing the cluster autoscaler to hold capacity to place them. The second was a traffic burst: some overnight event pushing real request volume up, driving replicas up, then subsiding but leaving the nodes.

Both died fast. kubectl get hpa --all-namespaces showed every HPA sitting comfortably below its ceiling, none within striking distance of a scale trigger. Prometheus request-rate graphs for every ingress-fronted service were flat across the whole week. When we summed pod counts across the cluster, we got 342, roughly what a normal Friday looks like, and nowhere near the 900+ pods it would take to justify 60 m5.4xlarge nodes at typical density.

The demand side was fine. Something on the supply side was refusing to shrink.

The cluster-autoscaler log that pointed at exactly one PDB

One PDB, named in every log line

We went to the cluster autoscaler's own logs, which is where CA tells you plainly why it will not do the thing you want. kubectl -n kube-system logs deploy/cluster-autoscaler --tail=200 returned the same shape of line, forty times per scale-down cycle, each naming a specific node, all of them naming the same PDB.

I0313 09:32:04.881 static_autoscaler.go:428] Scale down status: lastScaleDownFailTime=2026-03-13 09:22:04
I0313 09:32:04.892 cluster.go:151] Fast evaluation: node ip-10-42-14-88.ec2.internal, cannot delete: pdb data-platform/nightly-rollup-pdb blocking eviction of pod data-platform/nightly-rollup-7f8b9c5d4-4kx2m (currentHealthy=39, desiredHealthy=39)
I0313 09:32:04.895 cluster.go:151] Fast evaluation: node ip-10-42-15-19.ec2.internal, cannot delete: pdb data-platform/nightly-rollup-pdb blocking eviction of pod data-platform/nightly-rollup-7f8b9c5d4-9h4tn (currentHealthy=39, desiredHealthy=39)
I0313 09:32:04.897 cluster.go:151] Fast evaluation: node ip-10-42-15-33.ec2.internal, cannot delete: pdb data-platform/nightly-rollup-pdb blocking eviction of pod data-platform/nightly-rollup-7f8b9c5d4-b7q9r (currentHealthy=39, desiredHealthy=39)
... (37 more lines, all identical shape, all naming data-platform/nightly-rollup-pdb) ...
I0313 09:32:05.114 scale_down.go:1052] Scale down considered 40 nodes, 0 removable

Every blocked node in every iteration named the same PDB. That was the whole answer in the first log grab.

We cross-checked against the cluster's full PDB inventory to confirm no other PDB was involved.

$ kubectl get poddisruptionbudgets --all-namespaces
NAMESPACE       NAME                     MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
api-gateway     api-gateway-pdb          2               N/A               3                     47d
auth            auth-service-pdb         1               N/A               2                     47d
data-platform   nightly-rollup-pdb       39              N/A               0                     213d
data-platform   warehouse-writer-pdb     1               N/A               2                     47d
payments        payments-api-pdb         2               N/A               1                     47d
search          search-indexer-pdb       1               N/A               2                     47d

Six PodDisruptionBudgets. Exactly one at zero allowed disruptions. Every other PDB had two or three to spare, which is the healthy state.

The outlier was data-platform/nightly-rollup-pdb, and it was 213 days old, while the other five had all been re-created seven weeks earlier as part of a cluster upgrade. Something about this specific PDB had been quietly wrong for a long time.

We looked at the underlying workload. The 'nightly-rollup' service started life as a CronJob and had been reimplemented at some point as a 40-replica Deployment with soft pod anti-affinity, running continuously to feed a downstream aggregation service. Its PDB carried minAvailable: 39, which is the standard 'allow one disruption at a time' setting for rolling updates. Fine when the workload is healthy.

kubectl get pods -n data-platform -l app=nightly-rollup told the rest of the story. Thirty-nine pods Running, one pod stuck in ImagePullBackOff. Six days earlier, someone had pushed a rolling update with a container image reference that pointed at an ECR registry we had migrated away from during a project the previous quarter. Every other workload had been re-pointed at the new registry during that migration; this one Deployment had been missed, and nobody had noticed because it 'just worked' for months.

The instant one pod dropped out of Ready, healthy went from 40 to 39 against a minAvailable floor of 39, and allowedDisruptions clamped to zero. From that moment on, no node hosting a nightly-rollup pod could be drained: evicting one would take healthy below the floor, and cluster autoscaler is careful about that. Forty pods, forty nodes, one stuck PDB. Each nightly-rollup pod requested about 400m CPU and 1.5 GiB of memory, which is a rounding error on an m5.4xlarge, but the pod's presence plus the PDB at zero meant the node was unremovable regardless of how much headroom the rest of the box had. Forty nodes at 5-8% CPU, all of them technically pinned by one broken image reference.

Why we deleted the PDB instead of pushing the fixed image

Delete the PDB, or fix the image

Two paths out. Fix the underlying pod: patch the Deployment's image reference, let the rolling update roll forward, healthy count returns to 40, allowedDisruptions goes positive, CA is free. Or delete the PDB directly: no PDB, no eviction block, CA can drain nodes.

We almost went with the image fix, because that is the 'correct' answer in an abstract sense; the PDB is doing what it was written to do. The problem was who owned the Deployment. The team that had originally built the nightly-rollup service had been reorganized eighteen months earlier, and the current owning team had inherited it in a spreadsheet handoff and had never shipped a change to it. To fix the image correctly we needed either to push the correct image to the old ECR registry (which required someone with write access on an account we were sunsetting) or to patch the Deployment to point at the new registry (which required knowing whether the image tag existed there and had feature-parity, which we did not, at 9:30 am).

The PDB, on the other hand, was clearly orphaned. Its author was gone. Its minAvailable: 39 was arithmetically fine for a 40-replica Deployment during healthy operation but it meant one broken pod would freeze the entire fleet, which is exactly what had happened. Deleting the PDB did not remove any replicas or affect the currently-running pods; it just removed the eviction guardrail.

kubectl delete poddisruptionbudget -n data-platform nightly-rollup-pdb
# poddisruptionbudget.policy "nightly-rollup-pdb" deleted

One command. The gate is the PDB; remove the gate.

Twelve minutes later, cluster autoscaler started removing nodes. It took about ninety minutes to work through the fleet. The nightly-rollup pods that got evicted rescheduled onto the remaining nodes without issue because the anti-affinity was preferredDuringSchedulingIgnoredDuringExecution (soft), which meant CA could consolidate the fleet onto fewer nodes when memory allowed. The pending pod stayed pending, because the underlying image reference was still wrong, but a single pending pod is a monitoring problem, not a bill problem.

By 11:30 am the cluster was back at 24 nodes. We paged the data team's on-call to fix the image reference at their leisure the following Monday. The bill for that Friday's Compute came in at $760, back inside the normal envelope.

The instinct in the first thirty minutes had been to kubectl delete node on the idle boxes. That would not have worked, and not for the reason you might expect. kubectl delete node only drops the Node object from the API server; the kubelet on that instance re-registers within seconds and the node comes back with the same name. The EC2 instance is never terminated and the orphaned PDB is untouched, so the spend does not move. Kicking nodes was never going to touch this. The PDB is the gate. Remove the gate or fix the gated pod; kicking the nodes directly is a way to spend money without changing anything.

The Kyverno policy and PDB alert that closed the gap

The rules we shipped the next week

The postmortem produced three changes we shipped inside the week.

The first was a cluster-level admission policy that rejects new PDBs unless they carry an owner annotation and an expires-at annotation. If a PDB is going to have the power to freeze half a cluster, someone needs to own it and someone needs to argue for renewing it. Kyverno was already installed for other reasons, so the rule was about thirty lines.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: pdb-lifecycle-required
spec:
  validationFailureAction: Enforce
  rules:
    - name: require-owner-and-expiry
      match:
        any:
          - resources:
              kinds:
                - PodDisruptionBudget
      validate:
        message: "PDBs must carry infraforge.io/owner and infraforge.io/expires-at annotations. See runbook: https://internal.infraforge.io/rb/pdb-lifecycle"
        pattern:
          metadata:
            annotations:
              infraforge.io/owner: "?*"
              infraforge.io/expires-at: "?*"

Kyverno ClusterPolicy blocking any PDB that arrives without an owner or an expiry.

The exemption path for genuine long-lived PDBs (data-plane primary databases, for example) is to set infraforge.io/expires-at to a far-future date and put the owning team alias in infraforge.io/owner. That does not prevent the specific failure we hit, but it does mean that in eighteen months when the next team reorganization happens, we can grep for the PDBs whose owners no longer exist before they become orphaned.

The second change was a Prometheus alert that fires when any PDB has allowedDisruptions == 0 for more than thirty minutes. Not five, not sixty. Thirty is long enough that a healthy rolling update has finished but short enough that a stuck one still gets caught inside the same business day.

The third was a workload audit that the source scenario also recommends. Over the following week we ran kubectl top pods on a rolling seven-day window against every Deployment with minReplicas > 1, and dropped the floor on five workloads whose real utilization was under 30%. Two of them we took from minReplicas: 3 to minReplicas: 1. That work did not fix the incident (the PDB was the actual bug), but it lowered the cluster's headroom baseline enough to knock another two nodes off the normal working set.

On the observability side we replaced the flat AWS Budgets threshold with anomaly detection at the EKS Compute line item, tuned to fire on a 30% week-over-week jump. The flat $9,000 threshold caught this incident six days late. The anomaly rule would have caught it inside twenty-four hours.

When your EKS bill doubles without a traffic event

When this is happening to you

Bill spikes without a matching traffic event almost always trace to a controller doing exactly what it was configured to do, in a way the config never anticipated. Cluster autoscaler is the most common source of this specific flavor, because its downscale logic is deliberately conservative around PodDisruptionBudgets; a single PDB with zero allowed disruptions can pin nodes indefinitely, and if the PDB is stuck on a pod nobody currently owns, nobody notices until the bill does. HPA misconfiguration, Karpenter provisioner drift, and stuck node-termination lifecycle hooks are the other three variants we see most often in the same shape.

We run recovery engagements with this exact shape most quarters. The fastest path to the answer is usually the cluster autoscaler logs, not a Prometheus dashboard, because CA will just tell you what it is refusing to do. If your EKS bill doubled inside a week and Compute Optimizer is flagging the node group as overprovisioned, we can be on a bridge with your platform team the same day. Book an infrastructure review and we will start with a 30-minute diagnostic call this week; the more artifact you can share ahead of the call (bill line item, kubectl get nodes output, and a kubectl -n kube-system logs deploy/cluster-autoscaler --tail=500 grab), the faster we can name the specific blocker.

If you are earlier in the shape (rising EKS bill, no obvious cause yet, no acute page), the Kubernetes and CI/CD stabilization work we do covers the workload-audit and admission-policy side of what this article described. We have written more on the general pattern of cost spikes driven by controller behavior in the cloud cost spikes problem write-up.

Originally published at https://infraforge.agency/insights/eks-autoscaler-stuck-pdb-cost-spike/.

If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.

When a hardening rollout breaks 8 layers and your own reconciler fights you

Muhammad Hassaan Javed — Mon, 13 Jul 2026 14:00:47 +0000

The first thing the on-call team tried was patching the status ConfigMap. Five apps showed Progressing in the platform's bleater-status object, the dashboard had been red for four hours, and somebody figured a kubectl patch on the status keys would at least quiet the pages while they investigated. The patch lasted about ten seconds. The internal reconciler running in the bleater-system namespace rewrote the ConfigMap on its next tick, every key back to Progressing, and the pages started again. That was the moment they called us. The hardening rollout that had run the night before had not broken one thing. It had broken eight.

Problem signals:

Apps stuck in Progressing or Degraded for hours after a hardening or security pass, with no single obvious cause in the events stream
Status ConfigMaps written by an in-cluster reconciler get rewritten within 10 to 15 seconds of any kubectl patch
kubectl delete job hangs on suspended PreSync Jobs because a hook-cleanup finalizer is still attached
Migration init containers crash-loop with pg_isready succeeding but a schema verification step failing
kubectl patch on a RoleBinding fails with 'cannot change roleRef' because roleRef is immutable, so the binding has to be recreated

Why editing the status ConfigMap was the wrong instinct

The patch that survived ten seconds

The team had built a small in-house control plane the year before. A Python reconciler Pod in bleater-system watched the managed workloads, computed health from live cluster signals, and wrote a bleater-status ConfigMap every ten to fifteen seconds. Five apps reported there: an auth service, a profile service, a timeline service, a fanout service, and a primary application that handled the user-facing API. None of them used a full GitOps platform. The reconciler followed GitOps conventions, PreSync hook Jobs, sync windows, hook-cleanup finalizers, but it operated on raw Kubernetes primitives. ConfigMaps, Jobs, Roles. No CRDs.

That detail matters because when the on-call lead patched the status ConfigMap to mark the apps healthy, the reconciler was doing its job. It read the live cluster, saw the upstream signals were still bad, and rewrote the status. The patch was not wrong because patching ConfigMaps is wrong. It was wrong because the bleater-status object was not an input to the system. It was an output. Editing an output to fix a system is the same shape of mistake as editing a Prometheus metric to fix a service.

We have seen this pattern enough times to write it down as a rule. If a controller is rewriting your patches in under a minute, the object you are patching is derived state. Find the inputs. The reconciler source was eighty lines of Python and it took two minutes to read. The health predicate was an AND-chain across eight signals: lock state, orphan hook Job (the finalizer), schema version, RBAC capability, PVC bound, ResourceQuota headroom, NetworkPolicy egress, and init-container health. Any one of the first seven returning bad meant Progressing; a failing init container returned Degraded instead. All eight were bad.

# the AND-chain we found in the reconciler
def app_health(app):
    if lock_status() == 'locked':
        return 'Progressing'
    if orphan_hook_present(app):
        return 'Progressing'
    if schema_declared_version() < required_version():
        return 'Progressing'
    if not migration_rbac_capable():
        return 'Progressing'
    if not pvc_bound(app):
        return 'Progressing'
    if quota_exhausted():
        return 'Progressing'
    if not egress_allows_db(app):
        return 'Progressing'
    if init_container_failing(app):
        return 'Degraded'
    return 'Healthy'

The reconciler's health function. Eight independent signals, all gating. Every patch to the output ConfigMap was wasted work until every signal flipped.

What the inventory pass turned up in the bleater namespace

Eight failures wearing one hat

We started with the inventory, because the ticket told us almost nothing. A real P1 page rarely enumerates faults; it tells you what is on fire and gives you the namespace. We ran the kind of get-everything pass we always run on a strange namespace.

kubectl get pods,configmaps,jobs,deployments,roles,rolebindings,serviceaccounts,pvc,resourcequota,networkpolicy -n bleater
kubectl get events -n bleater --sort-by=.lastTimestamp | tail -40
kubectl describe pod -n bleater | grep -A5 'Init Containers\|Events:'

The first three commands we ran. The namespace had about sixteen pre-existing platform workloads from other teams sharing label values with the five managed apps.

What came back was a layered mess. A suspended PreSync Job named auth-presync-migrate-legacy7r2x with a hook-cleanup finalizer and no hook-delete-policy. A second suspended Job named fanout-presync-validate that looked identical but carried the hook-delete-policy annotation and a bleater.io/owner label pointing at platform-team. A hook-reconciliation-lock ConfigMap with status: locked and a stale lock-reason from the night of the rollout. The primary application's pod in Init:CrashLoopBackOff with kubectl logs --previous showing the init container failing after pg_isready returned ok. A bleat-db-schema ConfigMap declaring version=2 with no tables-v3 key. A migration script that contained psql ... || exit 0 and had no set -e.

And then the governance layer, which is where the rollout had really gotten out of hand. A RoleBinding named migration-runner-binding pointed at migration-runner-role-v1, which had read-only verbs. A migration-runner-role-v2 existed alongside it, unbound, with create:jobs and patch:configmaps. A PersistentVolumeClaim named bleat-migration-pvc was Pending with an event saying storageclass.storage.k8s.io "fast-ssd-tier" not found, on a k3s cluster where the only storage class was local-path. A ResourceQuota set to pods: 1. A NetworkPolicy with egress: [] denying everything outbound including DNS.

Each one of those, taken alone, was a small fix. Taken together, they gated each other. The migration could not run because the RBAC was wrong. The repair Pods could not schedule because the quota was at one. The init container could not reach Postgres because the NetworkPolicy denied egress. The schema could not advance because the script swallowed errors. The reconciler refused to mark anything healthy until all of them resolved. The hardening rollout had tightened every knob at once and the knobs were not independent.

The dependency graph we drew on the bridge call. Cascade order falls out of the arrows.

The order of repair when faults gate each other

Why we raised the quota before anything else

The instinct on a multi-fault incident is to start with the most visible symptom. The CrashLoopBackOff is loud. The lock is loud. The orphan Job is loud. None of those were the right first move. The right first move was the boring one: raise the ResourceQuota, because every other fix needed to schedule a Pod. A ResourceQuota of pods: 1 caps the total number of non-terminal Pods in the namespace, not the number beyond what is already running, and about sixteen were already running. The quota was over-committed the moment the rollout landed it, so the API server admitted nothing new. We raised it to pods: 24, cpu: 8, memory: 16Gi, sized above the existing workloads plus the five managed apps and the repair Pods. Production limits. We did not delete the quota.

Then the RBAC. We described both Roles and confirmed v2 had the verbs the migration Job needed. Patching the existing RoleBinding to swing roleRef to v2 returned the error we expected.

$ kubectl patch rolebinding migration-runner-binding -n bleater \
    --type='json' -p='[{"op":"replace","path":"/roleRef/name","value":"migration-runner-role-v2"}]'
The RoleBinding "migration-runner-binding" is invalid: roleRef: Invalid value: rbac.RoleRef{...}: cannot change roleRef

$ kubectl get rolebinding migration-runner-binding -n bleater -o yaml > /tmp/rb.yaml
# edit /tmp/rb.yaml, set roleRef.name to migration-runner-role-v2
$ kubectl delete rolebinding migration-runner-binding -n bleater
$ kubectl apply -f /tmp/rb.yaml

roleRef is immutable. The only path is delete-and-recreate, with the existing object as a template so you do not lose subjects.

PVC next. We listed storage classes, saw local-path was the only one, exported the existing PVC, changed storageClassName, deleted, reapplied. The PVC sat Pending for a few more seconds until we scheduled a consumer Pod against it, because local-path on k3s binds on first consumer. Then the NetworkPolicy. We did not delete it. The deny-by-default posture was the right posture; the rollout had just forgotten to allow anything. We added three explicit egress rules: same-namespace for the Postgres reach, kube-system on UDP 53 for DNS, and the metrics endpoints the reconciler scraped. The deny-all stayed in place for everything else.

Then the lock and the orphan. The lock was a one-line patch to set status: unlocked and to replace lock-reason with resolved-2024-hardening-rollback. We left an audit value rather than blanking the field. The orphan Job hung on delete because of the finalizer. The strip-then-delete sequence is muscle memory at this point but it is worth showing because plenty of teams reach for --force first, which is the wrong tool.

# strip the finalizer first, then delete cleanly
kubectl patch job auth-presync-migrate-legacy7r2x -n bleater \
  --type=json -p='[{"op":"remove","path":"/metadata/finalizers"}]'
kubectl delete job auth-presync-migrate-legacy7r2x -n bleater

# do NOT touch fanout-presync-validate. it has hook-delete-policy set,
# carries bleater.io/owner=platform-team, and the reconciler manages it.
kubectl get job fanout-presync-validate -n bleater \
  -o jsonpath='{.metadata.annotations.argocd\.argoproj\.io/hook-delete-policy}'
# => HookSucceeded

Strip then delete. The decoy Job looks identical to the orphan from a distance; the discriminator is the hook-delete-policy annotation and the ownership label.

We have written more on cleaning up GitOps-style state safely in our Kubernetes and CI/CD stabilization playbook, including the finalizer-strip pattern and how to tell a managed Job from an orphaned one without guessing.

Don't weaken governance to silence alarms

The fixes that had to be repairs, not deletes

Halfway through the recovery the client's platform lead asked the obvious question. Why not just delete the ResourceQuota and the NetworkPolicy until things stabilize, then put them back? It would have shaved twenty minutes. We said no, and the reason is worth writing down, because it is the part of incident work that teams under pressure get wrong most often.

Governance controls exist for a reason. Someone put pods: 1 on that ResourceQuota originally because something had blown up the namespace before. Someone put the deny-all egress on because the auth service should not be able to call random external endpoints. The rollout had mangled the values, not the intent. Deleting the controls would have restored the workloads and silenced the alarms. It would have also removed two of the few real defenses that namespace had, with no scheduled work item to put them back. We have watched teams do this in March and find the controls still missing in November. The graveyard of post-incident TODOs is full of governance restore tickets that never got worked.

So we repaired. The quota went up to production limits in place. The NetworkPolicy got explicit allow rules added while the default-deny stayed. The PVC got a real storage class while the claim itself stayed at the same name and the same size. The orphan Job got deleted, because a stale suspended PreSync Job genuinely is garbage, but the cascade infrastructure stayed. Same controls, working values.

The migration script was the other repair-not-delete case. The version we found had this pattern:

# what we found
#!/bin/bash
psql -h $DB_HOST -U $DB_USER -d $DB_NAME -f /migrations/v3.sql || exit 0
echo "migration complete"

# what we replaced it with
#!/bin/bash
set -euo pipefail
psql -h $DB_HOST -U $DB_USER -d $DB_NAME \
  -v ON_ERROR_STOP=1 \
  -f /migrations/v3.sql
echo "migration complete"

|| exit 0 is the single worst line in any migration script. set -e and ON_ERROR_STOP=1 together mean a failing SQL statement actually fails the Job, which is what the reconciler was waiting to see.

After the script was patched, the migration Job ran successfully under the new RoleBinding, applied the v3 schema, and we read the tables back out of Postgres directly rather than trusting the script's exit code. The bleat-db-schema ConfigMap got tables-v3 written from observed pg_tables output. Not from the migration's stated intent. From the live database. If you ever find yourself writing schema declarations from anything other than what is actually in the database, you are setting up the next incident.

When in-house reconcilers and hardening rollouts collide

If your control plane is gaslighting your operators

The hard part of this kind of incident is not any single fault. The hard part is that an internal control plane is opinionated about state in ways that are not documented anywhere except in the reconciler's source code. When five apps are red and the dashboard says nothing changed, your team can spend an hour patching outputs that get reverted before they understand the inputs. Hardening rollouts make this worse, because they touch ResourceQuotas and NetworkPolicies and RBAC in the same change window, and the rollback path almost never accounts for the case where the controls themselves were the right idea but the values were wrong.

We run these recovery engagements every week. The in-house reconciler pattern shows up at almost every SaaS company past Series A that decided not to run ArgoCD or Flux directly. The shape of the failure is always the same: a small Python or Go service that watches a namespace and writes a status object, an operations team that does not own the reconciler code, and a control plane that fights every cosmetic fix because that is what it was built to do. We have seen the RoleBinding immutability case four times this quarter alone. The NetworkPolicy egress-without-DNS case shows up after every security audit cycle.

If you are watching a namespace where the status object keeps reverting your changes, or where a hardening pass cascaded across half a dozen layers and your team is debating whether to delete the controls to get back to green, book an infrastructure review with our team and we will be on a bridge call with you the same day. We will read your reconciler, draw the dependency graph for the cascade, and walk the repair order with your on-call. The goal is not to get the dashboard green by morning. The goal is to get it green without leaving a graveyard of governance restore tickets behind it.

Originally published at https://infraforge.agency/insights/internal-control-plane-cascade-recovery/.

If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.

Recovering a status page from a half-finished schema migration

Muhammad Hassaan Javed — Mon, 22 Jun 2026 22:19:29 +0000

The log line was 'database schema version 23 is dirty, refusing to start' and the pod exited immediately after printing it. The team had already tried a Helm rollback to the previous chart version. That pod did not get further: it printed 'database schema version 23 found, expected 21' against a binary that wanted 21. One binary refused because the schema was mid-migration, the other because the schema was now ahead of it. Both refused to start against the same database, and the company's only public status page had been down for 38 minutes. The migration job had been OOMKilled mid-run during the upgrade, and Postgres was now in a state neither binary recognized.

Problem signals:

Application logs 'schema version N found, expected M' and exits before serving traffic
Helm rollback to the previous chart version fails with the same or inverse schema error
A migration job pod shows exit code 137 or OOMKilled in kubectl describe
The schema_migrations (or equivalent) table reports a version the table DDL does not actually match
Restoring from the most recent Postgres backup would lose hours of production data the team needs to keep

Both the old and new binary refused to start against the same database

The log line that ruled out a rollback

The on-call had done the obvious thing first. The new chart was failing, so they ran helm rollback to the previous revision, revision 13, the last one before 0.91.2. The previous revision's pod came up, hit the database, and crashed too, for a different reason. The new binary expected schema 23 and found the version row marked dirty mid-migration. The old binary expected 21 and found the row already advanced to 23. One saw a migration that never finished, the other saw a database from the future. Both were sort of right.

That pattern is what told us the database was the problem, not the chart. A clean rollback should have produced a running pod. If both binaries reject the same database, the database is not in either of the states they expect. It is in a third state nobody coded for.

$ kubectl logs -n statuspage statuspage-app-7b9f-xq2vk   # 0.91.2 pod, pre-rollback
INFO  starting statuspage v0.91.2
INFO  connecting to postgres at postgres.statuspage.svc:5432
ERROR database schema version 23 is dirty, refusing to start
FATAL refusing to start with a dirty schema version

$ helm rollback statuspage 13 -n statuspage   # 13 is the last revision before 0.91.2
Rollback was a success! Happy Helming!

$ helm history statuspage -n statuspage
REVISION  UPDATED                   STATUS      CHART               APP VERSION  DESCRIPTION
13        Tue Jun 16 09:12:04 2026  superseded  statuspage-0.90.78  0.90.78      Upgrade complete
14        Fri Jun 19 14:31:47 2026  superseded  statuspage-0.91.2   0.91.2       Upgrade complete
15        Fri Jun 19 14:48:12 2026  deployed    statuspage-0.90.78  0.90.78      Rollback to 13

$ kubectl logs -n statuspage statuspage-app-6c4d-7m9pz   # rolled-back 0.90.78 pod
INFO  starting statuspage v0.90.78
ERROR database schema version 23 found, expected 21
FATAL refusing to start with schema version mismatch

Different errors from the two chart revisions, same root cause: the database, not the chart, was the problem.

The version row claimed 23. The table DDL was somewhere between 22 and 23.

What the schema_migrations table actually said

We dropped into psql against the application database and pulled the migration tracking table. The row said version 23, applied. The dirty flag was true, which on most migration libraries means 'a migration started running and never reported success'. That single boolean was the thread we pulled on for the next hour.

statuspage=> select * from schema_migrations;
 version | dirty
---------+-------
      23 | t
(1 row)

statuspage=> \d incidents
          Table "public.incidents"
   Column    |  Type   | Nullable | Default
-------------+---------+----------+---------
 id          | bigint  | not null |
 service_id  | bigint  | not null |
 started_at  | timestamp without time zone |          |
 resolved_at | timestamp without time zone |          |
 title       | text    |          |
-- expected per migration 0023: severity column, incident_updates FK, partial index on resolved_at IS NULL
-- present: none of the above

The version row was lying. The table structure was a partial 23.

We pulled the migration files out of the chart's image and read them. Migration 0023 was three statements: add a severity column, create an incident_updates table with a foreign key back, create a partial index on unresolved incidents. None of the three were present in the live schema. The OOM had hit after the migration library wrote the version row and before any DDL statement actually committed. Or possibly between statements. The order was implementation-specific and we did not care which exact moment because the answer was the same: none of 0023's DDL had landed, but the bookkeeping said it had.

This is the specific failure mode that makes partial migrations dangerous. The migration library and the actual schema disagree, and the application trusts the library. The library trusts a row it wrote in a different transaction than the DDL it was supposed to be tracking. Whether the version row and the DDL share a transaction is a per-tool design choice, not a version threshold you can date. The version/dirty pair in the table above is golang-migrate's, and golang-migrate deliberately writes version=N with dirty=true in its own transaction, runs the migration body separately, then writes dirty=false. The dirty column exists precisely because those two are not atomic, and that has not changed. Any tool that ships a dirty flag is telling you the same thing about itself. At the other end, tools that wrap the migration and its history row in one transaction on Postgres (Flyway, Rails, Django, Alembic) never leave the state described here at all. Check which one you are running before you trust the version row, because that is what decides whether this failure mode can reach you.

The base backup was 6 hours stale and uptime data is the product

Why we did not restore from backup

The instinct, and the safe move on most days, is to restore Postgres from the last known-good base backup and replay WAL up to a point just before the migration started. We checked the backup. It was a nightly pg_basebackup, 6 hours old, and WAL archiving had been configured but never tested for PITR. We could probably have done it. We were not willing to bet the status page on 'probably' while the status page was already down.

More importantly, the uptime check history is the product. A PITR replay to just before the migration would have cost only the few minutes since the migration started, but if the untested WAL replay failed we would be down to the base backup alone, and a status page that loses 6 hours of check data after an outage is worse than a status page that takes another hour to come back. We talked it through with the team lead and decided the database in front of us was recoverable, and recovering it was lower risk than the restore path. That decision is worth naming because it goes against the usual 'just restore from backup' instinct. When the data itself is the value, finishing a half-migration by hand is often the right call.

Step	What it does
Restore from base backup + WAL	PITR to a point just before the migration would lose only the few minutes since the migration started, but the PITR path was configured and never tested. If the replay failed, the fallback is the base backup alone, which loses up to 6 hours of uptime check history. Estimated 45-90 minutes if it worked the first try, much longer if not.
Finish migration 0023 by hand, fix version row	Three DDL statements, all idempotent-ish if we wrote them with IF NOT EXISTS guards. Preserves all data. Estimated 20 minutes including verification. Chose this.

Three DDL statements, a version-row correction, and a careful restart

Finishing the migration by hand

We pulled migration 0023 verbatim from the chart image, rewrote each statement with IF NOT EXISTS guards so a re-run could not double-apply, and ran them inside a single transaction so any failure left the database where we found it. Before touching anything we took a pg_dump of the application schema and data to a local file. That dump was our 'we can always undo this' insurance, separate from the production backup system.

-- 1. snapshot first, outside any transaction
$ kubectl exec -n statuspage postgres-0 -- \
    pg_dump -U statuspage -Fc statuspage > /tmp/statuspage-pre-repair.dump

-- 2. complete migration 0023 inside one transaction
BEGIN;

ALTER TABLE incidents
  ADD COLUMN IF NOT EXISTS severity smallint NOT NULL DEFAULT 3;

CREATE TABLE IF NOT EXISTS incident_updates (
  id           bigserial PRIMARY KEY,
  incident_id  bigint NOT NULL REFERENCES incidents(id) ON DELETE CASCADE,
  body         text NOT NULL,
  created_at   timestamp without time zone NOT NULL DEFAULT now()
);

CREATE INDEX IF NOT EXISTS incidents_unresolved_idx
  ON incidents (started_at)
  WHERE resolved_at IS NULL;

-- 3. clear the dirty flag, version row already says 23
UPDATE schema_migrations SET dirty = false WHERE version = 23;

-- 4. sanity-check before commit
SELECT version, dirty FROM schema_migrations;
\d incidents
\d incident_updates

COMMIT;

All four statements committed in 180ms: the three DDL statements plus the version-row correction. The read-only checks below them commit nothing. The dirty flag was the last thing to flip.

After the commit we rolled forward to the chart whose binary expects 23, because the deployment sitting at 0 replicas was still the rolled-back 0.90.78 one and it would have refused a clean schema 23 exactly the way it refused a dirty one: helm upgrade statuspage statuspage/statuspage --version 0.91.2 --set replicaCount=0 --no-hooks. The --no-hooks is what actually skips the chart's pre-upgrade migration job, since 0023 was now applied by hand; --set replicaCount=0 only sets the Deployment's replica count and would not have stopped the job on its own. Worth stating that --no-hooks suppresses every other hook in the chart too, so read what else is hooked before you reach for it. The --set is the other flag that matters. We had scaled the deployment to 0 earlier to stop the CrashLoopBackOff noise, but kubectl scale only writes spec.replicas on the live object; the upgrade renders the chart's Deployment manifest with replicas from replicaCount and the merge patch puts the live value straight back, so a plain helm upgrade would have rolled a pod during the upgrade itself, before anyone looked at anything. Carrying the 0 through the upgrade is what kept the gate. Only then did we admit traffic, kubectl scale deploy/statuspage-app --replicas=1, and watch the pod logs. It connected, read schema_migrations, found 23 clean, started serving. We curled the health endpoint, got a 200, then hit /api/services and confirmed all the configured uptime checks were present with their full history intact. Total time from 'database is the problem' to 'status page is back': 51 minutes.

The thing worth saying out loud about this kind of repair: it works because the migration was small and the failure was clean. If 0023 had been a data migration that rewrote a million rows halfway, the recovery would have looked very different and the backup-restore path would have won. Always read the failed migration before you decide which recovery to attempt. We have written more about that decision in the migration recovery playbook.

A schema snapshot before every migration, and migration jobs that cannot be OOMKilled

The pre-upgrade hook we shipped the next day

Two changes went in within 24 hours. First, the Helm chart now has a pre-upgrade hook that writes two dumps to an object storage bucket, tagged with the chart version it is about to migrate to: pg_dump --schema-only for the structure, plus pg_dump --data-only --table=schema_migrations for the version and dirty values. Both are needed, because --schema-only emits DDL only and dumps no table rows at all, so on its own it would capture everything except the version/dirty pair, which is the single field this recovery turned on. If a future migration goes sideways, the recovery starts from a known structure-level snapshot taken seconds before the migration began, not from the nightly backup. The hook adds about 4 seconds to every upgrade and has paid for itself once already.

Second, the migration job spec got real resource requests and limits, and the limit is now 2x the largest migration we have ever observed in staging plus a 50% buffer. The OOM happened because the chart shipped with a 256Mi limit and the migration that finally hit the production data size needed about 380Mi. We also removed the liveness probe from the migration job entirely. A migration that takes longer than expected should not be killed by Kubernetes; it should be allowed to either finish or fail on its own terms so the migration library can write a clean dirty=true and exit.

The shape of every upgrade now. The branch on the right is the one that saved us. The logical schema snapshot is a reference for what the structure should look like; only the physical base backup can take a WAL replay.

We have stopped recommending that teams skip the pre-migration schema snapshot just because their database is 'small enough to restore from nightly'. The nightly backup answers a different question than the schema snapshot. The nightly tells you what the data looked like 6 hours ago. The schema snapshot tells you what shape the database was in seconds before this specific migration began, which is the thing you need when a migration is the thing that broke.

Recovering a partial migration without losing the data behind it

When a status page is the thing that is down

The reason this kind of incident is hard is not the SQL. The SQL is usually three or four statements you can read off the migration file. The hard part is deciding whether to finish by hand or restore from backup, and that decision depends on details most teams have not catalogued: how their migration library handles the version row, whether PITR has ever been tested, whether the failed migration is structural or data-rewriting, and whether the data between the last backup and now is recoverable some other way.

We run these recovery engagements regularly. We have seen the OOMKilled-migration-job pattern four times in the last year, two of them on status pages or monitoring tools where the data IS the product, and we have a checklist for the decision now. If you are staring at a CrashLoopBackOff with a schema version mismatch in the logs and a Helm rollback that did not help, book an infrastructure review and we will be on a bridge with you the same day to work through the finish-by-hand versus restore decision before you commit to either.

Originally published at https://infraforge.agency/insights/recovering-status-page-half-finished-schema-migration/.

If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.

When a validating webhook blocks the ConfigMap that would fix it

Muhammad Hassaan Javed — Tue, 16 Jun 2026 20:19:47 +0000

The kubectl patch came back as a webhook call failure, connection refused, not a credentials error. That was the moment the incident stopped being about a rotated MongoDB password and started being about the admission layer. A ValidatingWebhookConfiguration with failurePolicy: Fail was pointed at a webhook pod that was crash-looping on a bad liveness probe, and the only way to fix the webhook pod was to patch a ConfigMap that the webhook itself was supposed to validate. The safety mechanism had become the outage. Our profile service was down because its database credentials were stale, the fix for the credentials was a one-line patch, and that one-line patch could not be applied because the thing that was supposed to keep ConfigMaps safe was rejecting every write in the namespace.

Problem signals:

kubectl patch or apply on a ConfigMap returns a failed calling webhook error (connection refused, or a timeout) instead of a normal validation error
An admission webhook pod is CrashLoopBackOff while its ValidatingWebhookConfiguration is set to failurePolicy: Fail
ArgoCD shows sync pending or OutOfSync on resources in the affected namespace and the sync will not progress
A workload reads stale config from a ConfigMap that was supposedly already updated, and a Deployment-level env var is shadowing the mounted value
Compliance requires that admission webhooks remain failurePolicy: Fail in production, so flipping to Ignore as a workaround is itself an audit event

We thought it was a credentials incident for the first 20 minutes

The patch that came back as a webhook call failure

The page that started the call was a profile service returning 500s on /health. The cause looked obvious. The data layer team had rotated the MongoDB credentials the day before, and the live ConfigMap in the application namespace still held the old password. There was a backup ConfigMap sitting next to it with the rotated values, labelled exactly the way the runbook described. The fix was supposed to be a thirty second kubectl patch.

It was not. The patch came back with this:

$ kubectl -n app patch configmap profile-mongodb-config \
    --type merge --patch-file rotated.yaml
Error from server (InternalError): Internal error occurred:
failed calling webhook "configmap-validator.app.svc":
Post "https://configmap-validator.app.svc:443/validate?timeout=10s":
dial tcp 10.42.7.83:443: connect: connection refused

The actual error. Not a credentials problem, an admission control problem.

That error string is the whole story. The cluster had a ValidatingWebhookConfiguration named configmap-validator that intercepted every ConfigMap write in the namespace. The webhook pod was supposed to enforce a schema policy that the compliance team owned. Right now the webhook pod was not answering on its service IP, which meant every ConfigMap write was failing closed, which meant our credential fix was failing closed, which meant the profile service stayed down.

We had walked into this kind of shape before, but usually on the cert-manager side. This time the trap was tighter: the webhook was supposed to validate the very ConfigMaps that controlled the workloads in its own namespace, and one of those workloads happened to be down for an unrelated reason. Two independent failures had stacked into a deadlock.

Why the pod was crash-looping and why nothing could fix it in place

The webhook was guarding its own ConfigMap

kubectl describe on the configmap-validator pod told us the liveness probe was failing. The pod was getting killed every 30 seconds, restarting, getting killed again. The probe was hitting /healthz on port 8443. The actual application served its health endpoint on /health, no z. Someone had copy-pasted a probe spec from an older service months ago and nobody had noticed because the webhook had been running fine until a recent image bump shifted the health route.

Fixing a Deployment in Kubernetes is normally a kubectl edit deploy or a kubectl patch on the probe spec and you are done. That was not available to us. The webhook configuration intercepted ConfigMap writes, not Deployment writes, so technically we could have patched the Deployment directly. Except the Deployment mounted a ConfigMap for its own startup arguments, and our platform team had a hard rule against in-cluster edits that drifted from the GitOps source. ArgoCD would self-heal the Deployment back to the broken probe spec inside 90 seconds.

We needed the correct health path, and the compliance team kept the canonical values in a separate namespace. Their ConfigMap held the approved liveness path, the approved annotation policy, and the compliance acknowledgement token that any incident response was required to reference. We pulled it:

$ kubectl -n compliance get configmap webhook-standards -o yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: webhook-standards
  namespace: compliance
data:
  liveness-path: "/health"
  readiness-path: "/ready"
  failure-policy: "Fail"
  ack-token: "COMP-ACK-7c3f9a-2024Q4"
  required-annotations: |
    incident.compliance/id
    incident.compliance/services
    incident.compliance/ack-token

The compliance source of truth. We read these values; we did not retype them.

Flip failurePolicy to Ignore, or delete the ValidatingWebhookConfiguration?

Choosing the smaller blast radius

There were two ways to unblock ConfigMap writes. We could delete the ValidatingWebhookConfiguration entirely, fix everything, and recreate it from the GitOps source. Or we could patch failurePolicy from Fail to Ignore for the length of the recovery and patch it back when we were done.

Step	What it does
Option A. Delete the webhook configuration	Cleanest cut. ConfigMap writes unblock instantly. Risk: an unrelated team applies an out-of-policy ConfigMap during the window and we do not catch it. Also generates a louder audit event because the object disappears from etcd.
Option B. Patch failurePolicy to Ignore	Webhook is still called; if the pod is up it still validates; if it is down, writes pass. Smaller blast radius. Audit log shows a field change, not a delete. We picked this one.

Option B won because of the audit trail. The compliance team would rather see one field flip and one field flip back, with the same controller object identity across the incident, than see a delete and a recreate with a new resourceVersion lineage. That is the kind of preference you only learn by sitting through an audit. We have written more about this kind of constraint in our infrastructure audit readiness work.

# Step 1. Snapshot the current webhook config so we can prove what we changed.
kubectl get validatingwebhookconfiguration configmap-validator \
  -o yaml > /tmp/vwc-before.yaml

# Step 2. Flip failurePolicy to Ignore, scoped to this single webhook entry.
kubectl patch validatingwebhookconfiguration configmap-validator \
  --type='json' \
  -p='[{"op":"replace","path":"/webhooks/0/failurePolicy","value":"Ignore"}]'

# Step 3. Confirm the change before touching any ConfigMap.
kubectl get validatingwebhookconfiguration configmap-validator \
  -o jsonpath='{.webhooks[0].failurePolicy}'
# expect: Ignore

The unblock. Three commands, one of which is a snapshot for the post-incident review.

We patched the ConfigMap and the service was still broken

The env var that made the credential fix invisible

With failurePolicy on Ignore, the credential patch went through. We pulled the rotated values from the backup ConfigMap, applied them to the live one, and watched the profile service pods restart. /health still returned 500. The MongoDB connection error in the application logs still showed the old username.

That was the second moment in the incident where the model of the world had to change. The ConfigMap held the new credentials. The pod environment did not. Something else was setting MONGODB_URI on the container at runtime, and it was winning.

$ kubectl -n app get deploy profile-service \
    -o jsonpath='{.spec.template.spec.containers[0].env}' | jq
[
  { "name": "MONGODB_URI",
    "valueFrom": {
      "configMapKeyRef": {
        "name": "profile-mongodb-config",
        "key": "uri"
      }
    }
  },
  { "name": "PROFILE_MONGODB_URI_OVERRIDE",
    "value": "mongodb://oldapp:oldpw@mongo-old.app.svc:27017/profiles"
  }
]

The override. Set during a migration test six weeks earlier and never removed.

The application code read PROFILE_MONGODB_URI_OVERRIDE if it was set and otherwise read MONGODB_URI. The override had been added during a migration drill six weeks ago, never cleaned up, and was now silently shadowing every ConfigMap update we tried to apply. We have stopped accepting break-glass env overrides on production Deployments for this exact reason. If the override is worth setting, it is worth its own ConfigMap with an expiry annotation that a controller cleans up. Naked env values on the Deployment spec are invisible to the operators who do not know to look for them.

We removed the env var, the pod rolled, and /health came back as 200 on the third pod we curled.

Putting the webhook back, and making sure this never happens the same way again

Restoring failurePolicy: Fail without re-creating the trap

Before we restored failurePolicy to Fail, we fixed the webhook pod. The liveness probe path went from /healthz to /health, the value we had read from the compliance ConfigMap. The pod came up healthy and stayed up. We confirmed the webhook was actually answering by sending a deliberately invalid ConfigMap and watching the validation rejection come back cleanly. Only then did we flip failurePolicy back.

The harder problem was structural. A failurePolicy: Fail webhook that gates ConfigMap writes in a namespace is fine. A failurePolicy: Fail webhook that gates ConfigMap writes in a namespace that also contains the webhook itself, where the webhook's own Deployment depends on ConfigMaps in that namespace, is a bootstrap hazard. The first time something goes wrong, you cannot fix it without breaking your own policy.

The deadlock in one picture. Every fix path routes through a ConfigMap write the webhook itself is blocking.

We made two changes before we left. First, we moved the webhook's own Deployment, Service, and startup ConfigMap out of the app namespace and into a dedicated webhooks namespace, then set a namespaceSelector that excludes webhooks from validation, so the webhook can be rebuilt from its own in-cluster ConfigMaps even when it is the thing that is broken. Every ConfigMap in app is still validated, which is the whole point of the policy. Second, we added an objectSelector that excludes ConfigMaps carrying a specific break-glass label, so an on-call engineer with the right RBAC can apply a labelled emergency patch without flipping failurePolicy at all. Both changes were reviewed by the compliance team before we merged them, because relaxing the scope of a Fail-policy webhook is itself an audit decision.

The recovery script we left behind reads every value it needs (the health path, the ack token, the affected service list) from cluster state rather than hardcoding. Hardcoded recovery scripts go stale within a quarter; scripts that read from a compliance-owned ConfigMap stay correct as long as the source of truth is maintained. The script is idempotent: rerunning it on an already-recovered cluster is a no-op, which matters because the on-call engineer who runs it at 3 am should not have to think about whether they are the first or the third person to run it that night.

When to call us, and what we will look at first

If your admission layer is gating its own recovery

The thing that makes this incident shape hard is not the webhook itself. It is that the recovery path is non-obvious, the audit consequences of the obvious workaround (flipping policy or deleting the webhook config) are real, and the second-order trap (an env var on a Deployment shadowing the ConfigMap you just fixed) only shows up after you have already burned the credibility from the first workaround. Teams who hit this for the first time usually solve the immediate outage but leave the structural deadlock in place, and then it happens again on a different webhook six months later.

We run these recovery engagements every week. The admission-webhook-blocks-its-own-config shape has come up four times this year for us, once with cert-manager involved, twice with policy webhooks like this one, once with a service mesh sidecar injector that depended on a ConfigMap in its own namespace. The env-override-shadowing-a-ConfigMap-fix pattern is even more common; we see some version of it in roughly half of the credential rotation incidents we are called into.

If your cluster has a Fail-policy admission webhook today and you have never tested what happens when its pod is down, book an infrastructure review with our team and we will start with a 30-minute diagnostic call this week. We will walk your webhook configurations, identify the ones that gate their own dependencies, and give you a concrete plan to break the loop before an incident finds it for you.

Originally published at https://infraforge.agency/insights/admission-webhook-configmap-deadlock-recovery/.

If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.

ArgoCD CVE-2022-24348: a Secret leak that hid in log volume

Muhammad Hassaan Javed — Tue, 02 Jun 2026 02:05:12 +0000

The first thing we saw in Loki was a fanout service log line that contained the string 'a2V5Y2xvYWstY2xpZW50' repeated about 40 times in a single minute. Base64 decode: 'keycloak-client'. The fanout service had no business reading anything from the keycloak namespace. It had been emitting fragments of another namespace's client-secret for three days, quietly, while Grafana OnCall sat on a low-priority log volume alert that nobody clicked. The vector turned out to be CVE-2022-24348, the ArgoCD directory traversal bug, riding in on a ConfigMap key that an automation script had committed without anyone noticing.

Problem signals:

A low-priority alert on log volume spikes that nobody investigated for days
ConfigMap keys with URL values that contain '../' segments
Application logs containing base64 strings that decode to credential-shaped prefixes
ArgoCD Application source.repoURL values that point outside the expected repo root
ConfigMap changes in the cluster that have no matching Git commit

Why a fanout service was emitting Keycloak credential fragments

The log line that should not have existed

An on-call engineer was triaging an unrelated paging storm and, out of habit, ran a Loki query against the noisiest service of the previous week. The fanout service had spiked from roughly 200 log lines per minute to 6,400 per minute three days earlier and had stayed there. The lines looked like garbage. They were not garbage.

{app="fanout-service"} |= "" | line_format "{{.message}}" | json | __error__=""

# Sample line (sanitized):
level=info msg="resolved source repo" repo="a2V5Y2xvYWstY2xpZW50LXNlY3JldDovL2NsaWVudC1pZD1ibGVhdGVyLWFwaQ==" component=helm-renderer

The base64 in the repo field decodes to 'keycloak-client-secret://client-id=bleater-api'. The fanout service was logging the resolved value of a config key that should never have resolved to a Secret.

We pulled the live ConfigMap. The offending key was named ARGOCD_APP_SOURCE_REPO_URL and its value was 'gitea.internal/platform/fanout/../keycloak-secrets'. That single '../' segment is the entire CVE-2022-24348 exposure. The ArgoCD Helm renderer, in vulnerable versions, would normalize the path after resolving it, walk out of the intended repo root, and read whatever Helm values or Secret references it found in the sibling directory. In this case the sibling directory was a Helm chart that templated the Keycloak client-secret Secret into its values. The fanout service's own application code, which logged its resolved configuration on startup and on every reconcile, then dumped fragments of that Secret into Loki as base64.

Three days. The fanout service itself was healthy the entire time. RabbitMQ consumers were running, distribution was working, the SLO board was green. The exposure was completely silent from a functional standpoint.

Log volume alerts without log content are noise generators

Why the alert sat for three days

The Grafana OnCall alert that fired three days earlier said, in effect, 'fanout-service log volume is 30x baseline'. It was tagged P3 and routed to a Slack channel that the team treats as a digest. The runbook attached to the alert said to check for retry loops. The on-call engineer at the time did check, saw no retries in the RabbitMQ metrics, and silenced the alert for 24 hours. The silence got renewed twice by the rotation handoff.

This is the part of the story we keep seeing across client engagements. A log volume alert that does not inspect log content tells you something changed, not what changed. If the alert had matched on the byte pattern of base64 strings longer than 32 characters in a service that does not normally emit base64, the page would have been P1 and would have gone to a human within minutes. Volume alone is not a signal anyone can act on in under an hour, so it gets silenced.

We have written more on this in our GitOps and ArgoCD recovery cluster, where the same pattern shows up under different vectors. The constant is that GitOps systems concentrate trust in the manifest pipeline, and any leak in that pipeline tends to surface first as 'weird logs' before it surfaces as anything else.

The pod restart that the ConfigMap patch alone does not give you

Patching the ConfigMap was not the fix

The instinct, once we identified the bad key, was to kubectl edit configmap and delete the line. We did not do that, for two reasons. First, the ConfigMap was managed by ArgoCD; a live edit would last until the next sync. Second, even after the ConfigMap was clean, the existing pods would still have the malicious URL in their environment because envFrom only resolves at pod start. The leak would continue until the pods were rolled.

The correct sequence had four steps and the order mattered.

Step	What it does
1. Commit the fix to Git first	We removed the ARGOCD_APP_SOURCE_REPO_URL key from the ConfigMap manifest in the platform repo and opened a PR. ArgoCD was the source of truth, so any cluster-side edit would be reverted. The PR also added a comment explaining the CVE so the next person reading the repo would understand the deletion.
2. Sync ArgoCD with prune disabled	We forced the sync immediately rather than waiting for the next polling interval. We left prune disabled because we wanted to confirm exactly one diff: the deletion of the bad key. Surprise prunes during a security remediation are how secondary incidents start.
3. Roll the pods explicitly	kubectl rollout restart deployment/fanout-service. The ConfigMap was clean but the pod environments still held the resolved value. Until the pods restarted, every reconcile loop in the running process kept logging the leaked fragment. The rollout took 90 seconds.
4. Verify in Loki before declaring done	We ran the same Loki query that found the leak, scoped to the time window after the rollout completed. Zero matches. Then we ran it across a 30-minute window to be sure we were not just hitting log buffer lag. Still zero. That was the moment we stopped holding our breath.

# Verify the ConfigMap is clean
kubectl get configmap fanout-service-config -n bleater -o json \
  | jq '.data | keys[]' | grep -i repo_url
# (should return nothing)

# Verify pods restarted after the ConfigMap fix timestamp
kubectl get pods -n bleater -l app=fanout-service \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.startTime}{"\n"}{end}'

# Verify the ArgoCD Application source has no traversal
kubectl get application fanout-service -n argocd \
  -o jsonpath='{.spec.source.repoURL}' | grep -F '../' \
  && echo 'STILL VULNERABLE' || echo 'clean'

# Loki check, post-restart window only
logcli query --since=15m \
  '{app="fanout-service"} |~ "a2V5Y2xvYWs|keycloak"'

The four checks we ran in order. Any non-empty result on any of them would have meant the remediation was not complete.

Treating the leak as breached until proven otherwise

Auditing whether the Secret was actually read

The harder question was not 'did we stop the leak' but 'did anyone read the leaked data while it was leaking'. The fragments were in Loki, which meant anyone with Loki read access to the bleater namespace logs could have seen them. We pulled the Loki audit log for the three-day window and listed every query that matched fanout-service logs. Twelve queries from four engineers, all of them looking at unrelated debugging work, none of them filtering on the byte pattern that would have exposed the credential.

That was reassuring but not sufficient. The Keycloak client-secret had to be treated as compromised regardless, because we could not prove the absence of external log exfiltration with high confidence. We rotated the client-secret, redeployed the services that used it, and audited Keycloak's own access log for any token issuance using the old secret from an unexpected source IP in the exposure window. We found none. The rotation took about 25 minutes including service redeployment.

We then went back to the original question that nobody had asked yet: how did the malicious ConfigMap key get there in the first place. The automation script that applied it was a 'config sync' job that pulled key-value pairs from a shared spreadsheet and wrote them into the ConfigMap. The spreadsheet was editable by a wider group than the cluster was. Somebody had added the URL three days earlier, probably as a copy-paste mistake from a different document, and the sync job had faithfully applied it. There was no Git commit, no PR, no review.

Three controls that close this class of failure

What we changed so it cannot happen quietly again

We made three changes after this incident, in priority order.

The first was an admission webhook that rejects any ConfigMap apply where a string value contains '../' or matches the shape of a URL pointing outside an allowlist of internal domains. The rule is 12 lines of OPA policy. We tested it against six months of historical ConfigMap diffs and it would have caught this exact incident on day zero. It also catches the more common case of someone pasting a localhost URL into a shared config.

The second was retiring the spreadsheet-driven sync job. Every ConfigMap that lands in the cluster now comes from a Git commit, has a commit SHA annotation, and fails admission if the annotation is missing or does not match a real commit in the repo. The work to migrate the existing key-value pairs took about a week. The job is gone and is not coming back.

The third was rewriting the log volume alert. The new version fires when fanout-service log lines contain base64-encoded strings longer than 24 characters at a rate above 5 per minute, scoped to services that do not normally emit base64. It is a Loki recording rule with a regex match and it pages a human at P1. The first week it ran it caught two false positives (both were legitimate JWT logging that we then removed) and zero real incidents. We consider that a healthy signal-to-noise ratio for a security alert.

We also upgraded ArgoCD past the CVE-2022-24348 fix line. That should have happened a year earlier. If you are running an ArgoCD version older than 2.3.0, 2.2.9, or 2.1.15, stop reading this and go check, because the same vector is sitting in your cluster waiting for an unlucky ConfigMap edit.

The control surface after the incident. The two new gates are the admission webhook on apply and the content-aware log alert at runtime.

When a CVE has been silently active in your cluster for days

If you are staring at a similar exposure window

The hard part of this kind of incident is not the patch. The patch is one line. The hard part is reconstructing the exposure window with enough confidence to know what to rotate, what to disclose, and what to audit. That work requires log retention you can query precisely, audit trails for the systems that read those logs, and the discipline to treat any leaked credential as compromised until the access logs say otherwise. Teams that have not rehearsed this work tend to do all three poorly the first time.

We run GitOps and ArgoCD recovery engagements every month. We have seen the path traversal CVE three times in the last year, all on clusters running ArgoCD versions the operators thought were current, and we have seen the same 'silent ConfigMap injection via shared spreadsheet' antipattern more often than that. The remediation pattern is the same. The audit pattern is the same. The controls that close the gap are the same.

If you suspect a similar exposure in your cluster right now, request an infrastructure review and we will start with a 30-minute diagnostic call this week to scope the audit window and the rotation list. If the exposure is active, we will be on a bridge with your team the same day.

Originally published at https://infraforge.agency/insights/argocd-cve-2022-24348-path-traversal-secret-leak-recovery/.

If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.

Why Grafana OnCall acknowledgments hang after a Helm upgrade migration

Muhammad Hassaan Javed — Mon, 01 Jun 2026 02:59:51 +0000

The call did not come from our on-call rotation. It came from a customer who noticed two unrelated degradations on their side and asked why we had not paged. We had not paged because Grafana OnCall had been silently dropping alerts for roughly 72 hours. Alertmanager's deliveries were still getting 200 OK, but the Celery task that turns a delivery into an incident failed on every one of them, so nothing new was being recorded at all, and every attempt to acknowledge or resolve the two incidents left over from the upgrade returned HTTP 500. The on-call engineer who first tried to clear one of them that morning had assumed the spinner was a UI bug and moved on. The thing meant to wake us up was the thing that was broken.

Problem signals:

OnCall UI Acknowledge and Resolve buttons spin and time out with a generic 500
New alerts from real degradations never appear at all, even though Alertmanager still gets 200 OK: the Celery alert-creation task fails and nothing is recorded
OnCall pod logs show ORM errors referencing a column that does not exist in the table
The Helm pre-upgrade migration job reported success but Postgres logs show a lock_timeout on one ALTER TABLE
There is no Prometheus alert on OnCall's own API error rate, so the regression went undetected

72 hours of dropped alerts and two incidents stuck firing since the upgrade

The alerting platform was the incident

When we got on the bridge, OnCall's incident list would not render at all. The page came back 500, and so did the API behind it. The last state anyone had seen was two incidents in firing state, both written the night of the Helm upgrade three days earlier, both with zero acknowledgment events. Nothing had been escalated since, even though the runbook said any firing incident over 15 minutes old gets escalated, because nothing new had appeared. Nothing new had appeared because nothing new was being recorded: for roughly 72 hours OnCall's ingest endpoint had been answering Alertmanager with 200 OK and then losing the payload in the Celery worker that creates the incident, so every alert it sent was dropped silently. The two stale incidents were not absorbing anything. They were simply the last rows written before the upgrade.

The first thing we tried was the obvious one: clear those two incidents. Acknowledge returned a 500. So did Resolve, so did Snooze, and so did the list endpoint that was supposed to render them, whether we drove it from the UI or called it directly with curl. The web pods were up, the database was reachable, Redis was fine. Nothing in any dashboard suggested a problem, because nobody had built a dashboard that watched OnCall itself.

# note the header: the OnCall HTTP API takes the token raw, with no
# Bearer prefix. Prefixing it gets you a 403, not the 500 we were chasing.
$ curl -s -X POST -H "Authorization: $TOKEN" \
    https://oncall.internal/api/v1/alert_groups/I8KZ.../acknowledge/
{"detail": "Internal server error"}

# from the oncall-engine pod. grep -i, and match traceback too: the
# Django traceback carries no uppercase ERROR token to match on.
$ kubectl logs deploy/oncall-engine -c engine --tail=50 | grep -iA2 -e error -e traceback
Traceback (most recent call last):
  File "/etc/app/src/engine/apps/api/views/alert_group.py", line 412, in acknowledge
    alert_group.acknowledge_by_user(request.user, action_source=ActionSource.WEB)
--
DatabaseError: column alerts_alertgroup.acknowledged_by_confirmation_phone does not exist
LINE 1: ...ledged_by_user_id", "alerts_alertgroup"."acknowledged_by_co...

The ORM was reaching for a column the table did not have.

A silent ALTER TABLE timeout the Helm hook never noticed

Why the migration job exited 0 with a half-finished schema

Our first guess was a bad release. The previous Helm upgrade had bumped OnCall by a minor version, and we assumed the new application code was looking at a field that genuinely had not shipped yet. That was half right. The running code did expect the column, but the schema change that was supposed to create it had never landed, and django_migrations on the OnCall database had no row for that migration at all. That absence is the actual signature of a half-applied schema, not the thing people go looking for. Django records a migration only after its operations return cleanly, so a cancelled statement leaves the row missing while the new pods carry on expecting the column. It is also why migrate would have retried the migration on the next run: with no row recorded, Django still considers it unapplied. That cuts both ways, and it is exactly why the hand-applied fix further down has to be reconciled with django_migrations afterwards using --fake, or the next run re-applies DDL that is already in the table. One note before the artifacts below: the column and trigger names here are anonymized from the client's environment, so do not match them against upstream Grafana OnCall field names.

The clue was in Postgres logs from three days earlier, exactly when the Helm pre-upgrade hook ran the migration job. One line, easy to miss, in the middle of dozens of normal statement logs:

2026-05-11 02:14:07 UTC ERROR:  canceling statement due to lock timeout
2026-05-11 02:14:07 UTC STATEMENT:  ALTER TABLE alerts_alertgroup
    ADD COLUMN acknowledged_by_confirmation_phone varchar(20) NULL;
2026-05-11 02:14:07 UTC LOG:  duration: 30001.114 ms

alerts_alertgroup is one of the highest-write tables in OnCall. At 02:14 a backlog of inserts was holding row locks, the ALTER hit the lock_timeout we had set globally to 30 seconds (a sensible default we put in years ago to stop one bad migration from wedging the whole database), and Postgres killed the statement. Django did not swallow that. The DatabaseError propagated out of the schema editor, manage.py migrate aborted with a traceback, and the process exited non-zero. What swallowed it was our own job entrypoint: a shell wrapper that ran migrate alongside two other setup steps with no set -e, so the failing command's status was discarded and the script exited 0 on its last line. The Helm hook checked the job's exit code, saw 0, and let the upgrade proceed. ArgoCD synced. The new pods rolled. And from that moment, every code path that touched AlertGroup broke. Django names every concrete field of a model in its default SELECT and INSERT, so a single missing column takes out the ack and resolve path, the list endpoint, and the task that creates alert groups, all alike. The synchronous paths returned 500. Ingest did not: the integration endpoint resolves the AlertReceiveChannel by token, which lives in a different table that still had all its columns, hands the payload to a Celery task with apply_async, and returns 200 immediately. So Alertmanager got a 200 OK for all 72 hours; the AlertGroup row, the only thing that touches alerts_alertgroup and therefore the only thing that hits the missing column, is written inside the worker, and that is where the alert died. That is the diagnostic trap. alertmanager_notifications_failed_total stayed at zero and there were no delivery errors on Alertmanager's side at all, so anyone who checks Alertmanager first rules out the exact cause. The evidence lives in the oncall-celery worker logs and its task-failure counts, carrying the same DatabaseError, which is precisely why the drop was silent.

A plain retry would not have worked either, for a second and separate reason. That migration runs with atomic = False, so the operations that had already succeeded before the ALTER were not rolled back with it, and one of them had left a pgtrigger object behind on alerts_alertgroup, which we only found by querying pg_trigger directly. Django's migration would have blown up on replay trying to create a trigger that already existed. That leftover trigger did not cause the lock_timeout (an existing trigger does not block an ALTER TABLE; only conflicting locks held by other sessions do). It just meant the migration could not be re-run cleanly until we removed it, and the ALTER itself still needed a window where the write backlog was not holding the table.

Why we forward-fixed instead of rolling the Helm release back

Drop the trigger, add the column, then unstick the zombies

We considered rolling back to the previous OnCall version. It looked clean on paper: the old image did not need the missing column, so the schema would match again and acks would work. We talked ourselves out of it for two reasons. First, the new pods had been running for three days and had written data shaped for the new version, including new fields in adjacent tables. A rollback would have meant either accepting writes that the old code did not understand or restoring a 72-hour-old database snapshot, which would erase three days of incident history including the zombies we wanted to clean up. Second, a rollback would leave that migration still unapplied, so the next upgrade attempt would re-run the same ALTER against the same hot table and hit the same lock_timeout the same way. We would be back here in a week.

Forward-fix it was. The sequence had to be careful, because the table was still taking writes and we were going to ALTER it. ADD COLUMN and DROP TRIGGER both take an ACCESS EXCLUSIVE lock, and that conflicts with the ACCESS SHARE lock every plain SELECT takes, so leaving the web tier up would have done two bad things at once: kept a supply of conflicting lock holders sitting in front of us, and then queued every subsequent read behind our waiting ALTER, because Postgres hands out lock requests in order. So we picked a low-write window and drained both tiers, the Celery workers that wrote to alerts_alertgroup and the web pods serving the API, and took a short deliberate outage on the ack path rather than an unbounded stall on the hottest table in OnCall.

-- 1. confirm the column is genuinely missing
SELECT column_name FROM information_schema.columns
WHERE table_name = 'alerts_alertgroup'
  AND column_name = 'acknowledged_by_confirmation_phone';
-- (0 rows)

-- 2. find the blocking trigger left over from the failed attempt
SELECT tgname FROM pg_trigger
WHERE tgrelid = 'alerts_alertgroup'::regclass
  AND tgname LIKE 'pgtrigger_%';

-- 3. add_column.sql: drop it and ALTER in one transaction, with a
--    SHORT lock_timeout. Fail fast, never hold the queue open.
BEGIN;
SET LOCAL lock_timeout = '2s';
DROP TRIGGER IF EXISTS pgtrigger_oncall_protect_finished
  ON alerts_alertgroup;
ALTER TABLE alerts_alertgroup
  ADD COLUMN acknowledged_by_confirmation_phone varchar(20) NULL;
COMMIT;

# 4. the waiting happens out here, not in the lock manager
for i in $(seq 1 30); do
  psql -v ON_ERROR_STOP=1 -f add_column.sql && break
  sleep 5
done

# 5. reconcile django_migrations with the DDL we just applied by hand,
#    and reinstall the pgtrigger object step 3 dropped. Skip this and the
#    next pre-upgrade hook re-runs the migration and dies on DuplicateColumn.
python manage.py migrate alerts 0043_alertgroup_ack_confirmation_phone --fake
python manage.py pgtrigger install alerts.AlertGroup

Short lock_timeout plus an outside retry loop, then a --fake so django_migrations matches the schema.

We did not change the global lock_timeout. Setting it LOCAL keeps the change inside this one transaction, and setting it short is the whole point: a two-second ceiling means a failed attempt lets go immediately instead of parking an ACCESS EXCLUSIVE request at the head of the queue with every read and write piling up behind it. A five-minute wait would have converted a fast failure into a five-minute freeze of the busiest table in the product. The retry loop does the waiting instead. The fourth attempt took the lock. Once the column existed, we brought the web pods and the Celery workers back and watched the engine pod logs. The 500s stopped within seconds, and the next Alertmanager delivery was actually recorded instead of dying in the worker.

Step 5 is the one that is easy to skip and expensive to skip. Applying the DDL by hand leaves django_migrations with no row for that migration, so the next pre-upgrade hook runs manage.py migrate, Django still sees the migration as unapplied, and it dies on psycopg2.errors.DuplicateColumn: column "acknowledged_by_confirmation_phone" of relation "alerts_alertgroup" already exists. Faking the row is what closes that loop. And because step 3 dropped the pgtrigger object the same migration was supposed to install, the trigger has to be reinstalled in the same pass, or the schema stays half-right in the other direction. Hand-applied DDL is not done until django_migrations agrees with it.

That left the two stale incidents. Acknowledging them was not enough. An acknowledged incident still sits in the firing state from OnCall's deduplication perspective, so now that ingest was working again, new alerts matching their labels would fold into them. We had to mark them resolved. We did it through the API first to make sure the lifecycle hooks fired and downstream integrations got the resolved webhook. That worked for one of the two. The API still refused the other for an unrelated reason: its integration had been deleted, so OnCall could not look up the routing to fire the webhook. For that single record we set resolved=TRUE and resolved_at to the current timestamp in the database directly, with a note in the incident's raw payload explaining the manual close.

We then fired a synthetic alert from Alertmanager and watched a new incident appear, ack it from the UI in under two seconds, resolve it, and confirm a follow-up alert created a fresh incident instead of folding into the resolved one. That was the real all-clear.

Meta-monitoring for the platform that does the monitoring

What we wired up so the next silent migration trips an alarm

The thing that kept us up afterward was not the migration. Migrations fail. Database locks happen. The thing that kept us up was that OnCall had been broken for three days and not one signal in our monitoring stack had told us. We had alerts on Prometheus being down, on Alertmanager being down, on Grafana being down, on every customer-facing service. We had nothing watching the incident management platform itself.

We added two rules the same week. The first is a straight error-rate alert on OnCall's API. If more than 1% of requests to /api/v1/ return 5xx for five minutes, page the platform team at critical severity. Five minutes is short enough that a real outage gets caught but long enough that a single bad deploy rolling does not page. We picked critical because if OnCall is degraded, nothing else paging matters; alerts get swallowed.

# --- Prometheus rules ---
groups:
- name: oncall-meta
  rules:
  - alert: OncallApiErrorRateHigh
    expr: |
      sum(rate(django_http_responses_total_by_status_total{job="oncall",status=~"5.."}[5m]))
      /
      sum(rate(django_http_responses_total_by_status_total{job="oncall"}[5m]))
      > 0.01
    for: 5m
    labels:
      severity: critical
      service: oncall
    annotations:
      summary: "OnCall API returning >1% 5xx for 5m"
      runbook: "https://internal/runbooks/oncall-api-errors"

# --- Loki ruler rules (loki-rules ConfigMap, NOT Prometheus) ---
# A job that exits 0 is Complete, so kube_job_status_failed never
# moves. The log stream is the only place the failure exists. Match it
# with |~ and (?i): |= is a case-sensitive literal substring filter and
# the Django traceback contains no uppercase ERROR token anywhere.
groups:
- name: oncall-meta-logs
  rules:
  - alert: OncallMigrationJobErrors
    expr: |
      count_over_time({namespace="oncall", app="migration"} |~ "(?i)traceback|error" [10m]) > 0
    for: 1m
    labels:
      severity: critical
      service: oncall

The second rule is a Loki ruler rule, not a Prometheus one: job status cannot see an exit-0 failure.

The second rule is the lesson from this specific incident. Helm trusts the exit code, and here the exit code was a lie told by our own wrapper script, not by Django. Kubernetes agreed with the lie: the Job was Complete, kube_job_status_failed stayed at zero, and no Prometheus rule built on job status could have caught it. The only place the truth lived was the job's log stream, which is why the rule runs in the Loki ruler and matches on the text of the failure itself, from any pod with the migration label in the oncall namespace, regardless of whether the job reported success. The match expression matters as much as the ruler does. LogQL's |= is a case-sensitive literal substring filter, and what we needed to catch was a Python traceback reading django.db.utils.OperationalError: canceling statement due to lock timeout, which contains no uppercase ERROR token at all. A |= "ERROR" rule would have matched nothing on the exact failure it was written for, which is the same silent no-op we were trying to alarm on in the first place. The same trap catches you at the keyboard, which is why the grep further up runs with -i and an alternated -e traceback: a case-sensitive grep ERROR on that pod returns nothing and reads as "the engine is clean." Use a case-insensitive regex filter instead, |~ "(?i)traceback|error", or the tighter |~ "(?i)(traceback $most recent call last$|django\.db\.utils\.\w*Error)", and verify it against a captured log from a real failing job before you trust it. If you copy it, also make sure your migration pods actually ship logs to Loki; the rule matches nothing otherwise. We have caught two real issues with it in the months since (neither as bad as this one, both worth knowing about within minutes instead of days).

The broader pattern, and one we now apply on every recovery engagement we run, is that any tool you depend on to notice problems needs an independent way to notice when that tool itself is the problem. We have written more about this category of failure in our migration recovery work, because the same shape appears in database cutovers, queue platform upgrades, and identity provider migrations: the system you rely on to tell you the truth is the system that has stopped telling the truth, and you only find out from a customer.

When acks are silently 500ing and you cannot tell what data is real

If your OnCall is doing this right now

The hard part of this incident is not the SQL. The hard part is making the call between forward-fix and rollback when your incident history, your zombie state, and your live alert routing are all entangled in a database that is currently being written to by application code that expects a schema it does not have. Roll back without a plan and you lose three days of incident records. Forward-fix without checking for leftover triggers and migration locks and your second attempt fails the same way as the first. Run an ALTER on a hot table with a long lock_timeout and the whole table queues behind you until it finishes or gives up.

We do these engagements every few weeks. Partial Django migrations on Grafana OnCall is the specific case we have now seen three times this year, twice from lock_timeout and once from a leftover pgtrigger object that made the migration fail on replay. Adjacent variants we have handled: Sentry post-deploy migrations that left a column nullable when the code expected NOT NULL, Mattermost upgrades where one index creation timed out, Keycloak realm migrations that completed on the primary but failed on a replica. The pattern is identical and the recovery sequence rhymes.

If your team is staring at a 500 on every ack and trying to decide whether to roll back the Helm release, book an infrastructure review with our team and we will be on a bridge with you the same day. We will help you confirm the schema delta, plan the forward-fix or the rollback with the data implications spelled out, and clean up the stuck incidents without losing the history you need for the postmortem.

Originally published at https://infraforge.agency/insights/grafana-oncall-stuck-incidents-partial-migration/.

If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.