DEV Community: Muhammad Hassaan Javed

How to recover a Helm 3 release stuck in pending-upgrade

Muhammad Hassaan Javed — Thu, 03 Sep 2026 16:43:46 +0000

A Helm release stuck in pending-upgrade blocks helm upgrade with another operation (install/upgrade/rollback) is in progress, and the useful surprise is that it does not block helm rollback. The pending check lives in prepareUpgrade and guards concurrent upgrades; pkg/action/rollback.go has no equivalent. So try helm rollback to the last good revision FIRST. Deleting a release-history Secret is a real mutation of Helm's storage and it is almost never the answer. A stuck pending-install on revision 1 has no earlier revision to roll back to, but helm uninstall has no pending check either and clears the abandoned objects with it, so that case is an uninstall and a reinstall. On a payments platform we work with, that was revision 47 sitting pending for 22 minutes after the CI runner was killed mid-upgrade, with a chart bump that had added a values.schema.json the production values file no longer satisfied waiting to break the retry. This guide is the order we run it in, with the two moves that will cost you a resource if you get them backwards.

Problem signals:

helm upgrade exits immediately with Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress
helm history -o json shows the newest revision as pending-upgrade with no deployed revision above it
The release does not appear in helm list -n <ns> at all, only in helm list -a -n <ns>
A Secret created by the half-finished upgrade exists but its key decodes to 0 bytes, and the pods CrashLoop on auth
helm upgrade with the existing production values file returns values don't meet the specifications of the schema(s) in the following chart(s)

Why does helm upgrade say another operation is in progress?

Read the history before you type rollback

The lock is a stored revision record, not a process holding a mutex.

This guide assumes Helm 3 driving the chart directly, from CI or from a laptop, with the default secret storage driver, against EKS 1.29. If Argo CD renders your chart and applies the manifests itself, none of what follows applies, because in that mode there is no Helm release history to inspect and the recovery is a git and sync-status problem instead.

Helm does not hold a lock in memory. It writes a Secret per revision, named sh.helm.release.v1.<release>.v<n>, and every operation reads the newest one first. If that newest record says pending-upgrade, helm upgrade concludes an operation is still running somewhere and refuses to start another. The CI job that wrote it may have been killed 22 minutes ago. Helm has no way to know that, so the release stays wedged until you change the record.

The first surprise is that the release can look like it does not exist. helm list -n settlement filters to deployed and failed releases, so a pending one is simply absent from the output, which sends people looking for a deleted release that is sitting right there. Run helm list -a -n settlement instead and it appears.

# The two cheap reads. Do both before you touch anything.
helm history payments-worker -n settlement -o json \
  | jq -r '.[] | "\(.revision)\t\(.status)\t\(.description)"'

# Same answer straight from the storage layer, no Helm binary involved:
kubectl get secret -n settlement \
  -l owner=helm,name=payments-worker \
  -L version,status --sort-by=.metadata.creationTimestamp

helm history renders an aligned table by default, which is useless in a ticket. The -o json form pipes cleanly, and the kubectl form works when the Helm binary is arguing with you.

In our case the JSON came back with revision 46 still marked deployed and revision 47 as pending-upgrade, description Preparing upgrade. That is the shape you are looking for: a head revision in a pending state, with the last good revision still deployed beneath it. Helm demotes the previous revision to superseded only on the success path, so a run killed mid-apply leaves 46 exactly where it was. If instead the head revision says failed, you are in a different and much easier situation, because Helm will happily accept a helm rollback against a failed release without any of the surgery below.

The rollback is the first branch. A stuck pending-install on revision 1 has no earlier revision to roll back to, so it goes to helm uninstall, not to a hand-edit of Helm storage.

Try the rollback first, then clear the record only if you must

Back up every release Secret first

Start with the rollback, because the pending guard does not apply to it. helm rollback payments-worker 46 -n settlement --wait --timeout 5m writes a new revision rather than destroying one. If it returns cleanly, you are done.

We did not start there. On the night this happened we went straight to clearing the record by hand, and the rollback we ran afterwards is the one that fixed it. Reading pkg/action/rollback.go later made the point uncomfortable: there is no pending check in the rollback path, so that rollback would have worked on its own and the Secret delete bought us nothing. It is worth being plain about that, because the procedure below circulates as folklore and most of the people running it do not need to.

You need it in one situation, and it is narrower than the folklore suggests. When the head revision is pending-install on revision 1 there is no earlier revision to roll back to, but that does not make a hand-edit of storage the answer. helm uninstall carries no pending check either, uninstall.go tests only for an already-uninstalled release, and it clears both the release record and the objects the half-finished install left behind. Uninstall, then install again. Deleting the Secret by hand is defensible only when those objects have to stay in place, and then you still owe Helm an adoption step: --take-ownership on the next install or upgrade, which landed in Helm 3.17. A failed rollback is NOT that situation. Rollback.Run calls Releases.Create with pending-rollback before it touches any resource, and on failure records that same revision as failed, so after a rollback that errors your head is a failed rollback revision and the fix is at the resource level, not in storage. Re-read helm history before you conclude otherwise.

Before deleting a revision record, take the whole history to a file. This is the one step that turns an irreversible mistake into an inconvenient one, and it costs six seconds.

kubectl get secret -n settlement \
  -l owner=helm,name=payments-worker -o json \
  | jq 'del(.items[].metadata.resourceVersion, .items[].metadata.uid,
         .items[].metadata.creationTimestamp, .items[].metadata.managedFields)' \
  > /tmp/payments-worker-helm-history.json

# Confirm the file has every revision that still exists, not just the ones you remember.
# With the default --history-max 10 this lists about ten, not one per revision.
jq -r '.items[].metadata.name' /tmp/payments-worker-helm-history.json

The jq filter is the point. A plain get -o yaml carries resourceVersion, which the API server refuses outright on a create, plus uid and creationTimestamp that it would silently overwrite, so the untouched dump restores nothing at the moment you need it.

Now confirm which revision is actually stuck, and confirm it from the payload rather than the label. The release body inside that Secret is not JSON and you cannot patch it as JSON. Helm gzips the release JSON, base64 encodes the result itself, and then Kubernetes base64 encodes the whole thing again into data.release. Open the Secret in an editor and you get an opaque blob. Anyone who tells you to flip the status field from pending-upgrade to failed with a kubectl patch has not tried it.

kubectl get secret sh.helm.release.v1.payments-worker.v47 \
  -n settlement -o jsonpath='{.data.release}' \
  | base64 -d | base64 -d | gunzip | jq -r '.info.status, .info.description'

Two base64 decodes, then gunzip. One decode gives you more base64 and people assume the data is corrupt.

That printed pending-upgrade and Preparing upgrade, which matched the label. Read what follows as the record of what we ran that night, not as the step to copy. We deleted that single Secret, which made revision 46 the head of the history again and let Helm treat the release as one it could operate on. With 46 sitting there deployed, the rollback on its own would have done the same work, so the delete bought us nothing. It is written out because these are the commands that circulate as folklore, and the two judgment calls buried in them are worth having before you are somewhere you need them. If your own head revision is pending and an earlier revision is beneath it, stop at the rollback.

kubectl delete secret sh.helm.release.v1.payments-worker.v47 -n settlement

helm rollback payments-worker 46 -n settlement --wait --timeout 5m

Delete the record, not the release: helm uninstall here would take the workload down with it, which is why the uninstall route belongs to a stuck pending-install on revision 1 and not to this. This pair is what we ran, not what this case needs. A pending head with an earlier revision under it stops at the rollback.

Two judgment calls in that pair of commands. Delete exactly one Secret, the pending one, and name it in full. A label selector delete against owner=helm,name=payments-worker removes the entire history including the revision you are about to roll back to, and then your only route home is the backup file you just wrote. Second, name the target revision explicitly. helm rollback payments-worker -n settlement with no revision argument rolls back to the revision before the head. That is the trap after a delete: with 47 gone the head IS 46, so a bare rollback targets 45 and quietly skips the revision you were aiming for. Type the number.

We use --wait --timeout 5m rather than a bare rollback because a rollback that returns success while pods are still terminating tells you nothing. With --wait, Helm returns only after the Deployment reports its expected replicas ready, so a non-zero exit is real information. The cost is honest: for as long as the rollback actually runs, up to that five minute ceiling, no helm upgrade against that release will start. A second rollback or an uninstall still would, since neither carries the pending guard.

What 'no ConfigMap with the name X found' means during a rollback

When rollback fails on a live resource Helm has no record of

The second failure people hit is a rollback that gets past the lock and then dies on a specific resource. The message reads like Helm is confused about what exists. It is not.

Error: no ConfigMap with the name "payments-worker-broker" found

Helm 3 emits this from its update path when the object exists in the cluster but has no entry in the release record it is diffing against. A rollback prints it bare, as here, while the same condition reached through helm upgrade arrives prefixed with UPGRADE FAILED:. Match on the no <Kind> with the name "<name>" found body, which is stable across both, rather than on the prefix. Rollback records its own Rollback "payments-worker" failed: ... as the revision description, not on stderr.

Get the direction right, because getting it backwards sends you looking in the wrong place for an hour. Update in Helm 3's pkg/kube/client.go takes the previous release's manifest and the target manifest, and walks the target. It looks each resource up live first, and a miss there is harmless: Helm creates the object and carries on. The error comes one step later, when it looks the same resource up in the previous release's manifest and finds nothing. So the object is in the cluster, and in the manifest being applied, and absent from the record Helm is comparing against. That is what a rollback to revision 46 runs into when something outside the release created the object, or an earlier half-finished upgrade left it behind without recording it.

The recovery is to delete the live object and run the rollback again. Deleting it makes Helm's live lookup miss, and a miss is the harmless path: Helm creates the resource from revision 46's manifest and records it properly this time. Read what revision 46 expects first with helm get manifest payments-worker --revision 46 -n settlement, so you know what is about to be recreated and can confirm nothing else depends on the object's current contents. Do not reach for --force on the rollback, and the reason is sharper than "it is risky": on this path it does nothing at all. force is only consulted inside updateResource, and this error never gets that far: the visitor checks the cluster with helper.Get first, then looks the resource up in the previous release's manifest, and returns no %s with the name %q found from that second lookup. The flag sits downstream of the point where it fails. Where --force does apply it sends a full replace (helper.Replace, a PUT) instead of patching, which discards fields another controller owns, an HPA-managed replicas being the usual casualty, and fails outright on immutable resources such as a Job or a Service clusterIP.

There is a related trap on the workload itself. A half-finished upgrade frequently creates a Secret from a template whose input value never rendered, so the Secret exists, the Deployment mounts it, and the key inside is an empty string. Kubernetes is perfectly happy with that. The pods are not, and you get a CrashLoopBackOff whose logs blame the broker rather than the chart. Check the length, not the presence.

# Presence proves nothing. Length does.
kubectl get secret payments-worker-broker-auth -n settlement \
  -o jsonpath='{.data.password}' | base64 -d | wc -c
# 0

# After a clean rollback or upgrade, the same command returns the real byte count.

A Secret that decodes to 0 bytes passes every existence check and fails every connection.

Verification is four reads and they should all agree. helm status payments-worker -n settlement -o json | jq -r '.info.status' returns deployed. helm history payments-worker -n settlement -o json | jq -r '.[-1].status' returns deployed for the head revision. The old pending entry is still listed if you got here by rolling back, because a rollback appends a revision rather than removing one; it is gone only if you deleted its Secret. kubectl get pods -n settlement -l app.kubernetes.io/name=payments-worker shows every pod Running with a restart count that stops climbing over the next few minutes. And helm get manifest against the head revision matches what is live, which is the check that catches a rollback that succeeded on paper while something else was quietly reconciling the cluster back.

How do you fix a chart schema error without deleting the schema?

Getting the values file past values.schema.json

The release is deployable again, and now you still have the original problem: the chart version you were upgrading to ships a values.schema.json that your production values file does not satisfy. Helm validates values against that schema in prepareUpgrade, before it writes a revision or touches the API server. That ordering matters more than it looks: a run that fails schema validation creates no release record at all, so a schema error can never be what left you in pending-upgrade. Revision 47 wedged because the runner was killed; the schema was waiting to break the retry, which is exactly what it did.

Error: UPGRADE FAILED: values don't meet the specifications of the schema(s) in the following chart(s):
payments-worker:
- broker.pool.maxIdle: Invalid type. Expected: integer, given: string

The chart did not change what the field means. It started enforcing a type that was previously accepted as free text.

That is the whole class of failure. A field that was unstructured for two years held "25" in quotes because someone templated it out of a CI variable, and the new schema declares it an integer. Nothing about the running workload was wrong. The schema simply started looking.

Fix it locally before you go near the cluster. helm lint ./charts/payments-worker -f values/production.yaml runs the same schema validation without a Kubernetes connection, which turns a ten minute deploy-and-fail cycle into a two second one. Once lint is clean, helm upgrade payments-worker ./charts/payments-worker -n settlement -f values/production.yaml --dry-run=server renders with a cluster connection, so lookup functions resolve and .Capabilities.APIVersions reflects what the cluster actually serves instead of Helm's built-in defaults. That catches a template reaching for an API version the cluster has dropped. It does not submit anything for admission review, so a Gatekeeper, Kyverno or pod-security rejection is still waiting for you on the real upgrade; to cover that, run helm template ... | kubectl apply --server-side --dry-run=server -f -. The server-side form of --dry-run arrived in Helm 3.13, so on older clients you get the client-side render only.

We do not delete or blank out the values.schema.json in a vendored copy of the chart to make the error go away. We have inherited two clusters where someone did exactly that, and in both the next chart bump reintroduced the schema and the same incident happened again with a different on-call engineer and no memory of the first one. Fix the values file. If the schema itself is genuinely wrong for your use, pin the chart version, open the issue upstream, and write the pin's reason in the values file where the next person will read it.

For prevention, helm upgrade --atomic helps, but not with the failure in this story. It implies --wait and rolls the release back if the upgrade does not converge, and that rollback is issued by the Helm client process itself. Kill that process and nothing is left to issue it, so a hard-killed CI job still leaves the release pending. What guards against this specific case is the opposite ordering to the one people reach for: Helm's --timeout must expire BEFORE the CI runner's own job timeout. Then Helm gives up on its own terms, unwinds, and exits. Set the runner's limit shorter and it kills Helm partway through, which is precisely how the release ends up pending. On Helm 4 the flag is --rollback-on-failure; --atomic is deprecated on helm upgrade and is an unknown flag on helm install. The tradeoff is real and we tell clients about it up front: --atomic holds the release for the entire timeout window, so on a rollout that takes four minutes with a ten minute timeout, a failed deploy blocks the next one for ten minutes rather than failing fast. That is usually the right trade for a payments path and usually the wrong one for a batch worker that deploys thirty times a day. We walk through where that line sits per service in our Kubernetes and CI/CD stabilization work.

Common questions about clearing a stuck Helm release

What people ask after the first rollback lands

Will helm rollback work while the release is still in pending-upgrade? Yes, and it is the first thing to try. The guard is upgrade-only: prepareUpgrade returns errPending when lastRelease.Info.Status.IsPending(), and its own comment scopes it to concurrent upgrades acting as a pessimistic lock. pkg/action/rollback.go carries no such check, so a rollback to a known-good revision runs and leaves a new deployed revision as the head. The pending row keeps its pending-upgrade status in the history, because performRollback supersedes only revisions that were already deployed, and it blocks nothing once the guard reads a deployed head. When there is no prior revision at all, a pending-install on revision 1, the answer is still not storage surgery: helm uninstall has no pending check either and takes the abandoned objects with it, so uninstall and install again. A rollback that ERRORS is not that case: it has already written its own failed revision, so re-read the history before touching storage.
Is deleting a sh.helm.release.v1 Secret safe? Deleting one revision record removes Helm's memory of that revision. It does not touch a single running resource. The danger is scope, not the act: delete by full name, never by label selector, and keep the backup file until helm status reports deployed.
Can I just patch the status field inside the release Secret? The payload is gzipped JSON, base64 encoded by Helm and then base64 encoded again by Kubernetes, so there is no JSON there to patch. If you want to inspect it, use the decode round trip above. To change the release state, use the supported route: clear the pending revision, then roll back.
Does any of this apply if Argo CD manages the app? No. When Argo CD renders the chart and applies the manifests itself, there is no Helm release history in the cluster to repair. The failure looks similar and the fix is entirely different.
My rollback succeeded but the pods still CrashLoop. Check the Secret and ConfigMap byte lengths, not their existence. A key that decodes to 0 bytes satisfies every reference check the Deployment does and fails at connection time, which is why the logs blame a downstream service.

When the stuck release is on the payments path and nobody wants to type delete

If the release has been pending for an hour

The hard part of this procedure is not the commands. It is that the correct move is kubectl delete secret against a revision record on a production release, with an engineering lead watching, at the point in an incident where confidence is lowest. The failure modes are unforgiving in a specific way: a label-selector delete that takes the whole history, a rollback to an implicit revision that lands one further back than you meant, a --force that replaces a StatefulSet and drops fields you needed left alone. Every one of those is recoverable if the backup exists and unrecoverable if it does not.

We do this work with teams who deploy through Helm from CI and have never had to open a release Secret before. Our part is being the second pair of eyes on the delete, reading the history with you before anything is typed, and then leaving behind the --atomic and timeout settings that stop the same job from wedging the release next quarter. Most of these calls run under an hour once we can see helm history output.

If a release is pending right now and you would rather not be the one running the delete, book an infrastructure review and we will get on a call and work through the history with you the same day. If it is already recovered and you want the CI path hardened so it does not recur, that is the same conversation with less adrenaline, and it is the shape of work described in our Kubernetes release failure playbook.

Originally published at https://infraforge.agency/insights/helm-rollback-failed-release-recovery/.

If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.

ServerlessDatabaseCapacity pinned: Aurora v2 will not scale down

Muhammad Hassaan Javed — Tue, 01 Sep 2026 17:02:09 +0000

ServerlessDatabaseCapacity sitting flat near max_capacity while pg_stat_activity is clean almost always means one of two things: a min_capacity or max_capacity change someone made in the console and never reverted, or a logical replication slot whose restart_lsn has stopped advancing. It is almost never a runaway query. Check the scaling configuration first, because that is one API call and it rules out half the problem. Then read pg_replication_slots, before you restart or drop anything.

Problem signals:

ServerlessDatabaseCapacity is a flat line for days where it used to be a sawtooth, with no matching rise in DatabaseConnections
Cost Explorer shows Aurora:ServerlessV2Usage stepping up on one calendar day and holding, while every other RDS usage type is flat
Capacity at 04:00 UTC is the same number as capacity at 14:00 UTC
pg_stat_activity has nothing older than 60 seconds and pg_stat_progress_vacuum returns zero rows
pg_replication_slots shows an active slot with hundreds of GB between restart_lsn and pg_current_wal_lsn()

What ServerlessDatabaseCapacity flat near max_capacity actually costs

218 hours at 148 ACU and not one page

A claims-analytics platform we work with runs one Aurora PostgreSQL Serverless v2 cluster in us-east-1: a writer and a single reader, roughly 15 services in front of it. For 218 hours the pair sat at a combined 148 ACU. The writer floated between 96 and 128, the reader between 32 and 48. Nothing paged. Latency was fine, the on-call rotation had a quiet week, and the only artifact of the whole thing was a CloudWatch chart that had gone flat where it used to be a sawtooth.

At the us-east-1 list rate of $0.12 per ACU-hour, 148 ACU-hours per hour for 218 hours is $3,872 of capacity. The same window at that cluster's normal 6.4 ACU average would have been $167. The gap was found by a monthly close-out preview, not by anything in the observability stack, which is the part worth sitting with. This article assumes Aurora PostgreSQL 15.x on Serverless v2, AWS CLI v2, and that you have psql access to the writer.

The flat line is most of the diagnosis. Aurora Serverless v2 scales up in seconds and comes down gradually, because shrinking capacity means giving back memory that the buffer cache is holding. A cluster with a genuinely quiet overnight window draws a sawtooth. A cluster that draws a flat line at or near its ceiling is either being told to stay there, or being kept from leaving. Those are two different bugs with two different fixes, and they are frequently both present at once.

aws ce get-cost-and-usage \
  --time-period Start=2026-07-15,End=2026-08-01 \
  --granularity DAILY \
  --metrics UnblendedCost UsageQuantity \
  --filter '{"Dimensions":{"Key":"SERVICE","Values":["Amazon Relational Database Service"]}}' \
  --group-by Type=DIMENSION,Key=USAGE_TYPE

The dollar total is smoothed by every other RDS instance in the account. The Aurora:ServerlessV2Usage line is not. In regions other than us-east-1 the usage type carries a region prefix.

Run that before you run anything on the database, because it dates the step change to a single day. Knowing the cluster went from 6 to 148 ACU-hours per hour on a Tuesday morning and never came back is worth more than an hour of query analysis. It converts an open-ended performance question into "what happened on that day", and somebody's memory usually answers it in about four minutes.

Which three causes pin Aurora Serverless v2 at high capacity?

pg_stat_activity was clean, which is the useful part

We rank these by how often we actually find them, not by how interesting they are. The first one accounts for more of these calls than the other two combined, and it is the one nobody wants to check because it feels too dumb to be the answer.

Step	What it does
1. A scaling change made in the console and never reverted	Someone raised max_capacity for a load test, a migration cutover, or a seasonal peak, and raised min_capacity at the same time so the cluster would not warm up between iterations. The work finished, the revert became a mental note, and the mental note lost to an afternoon incident. On the cluster above, min_capacity had gone from 2 to 16 on both instances. That floor alone bills $80.64 a day whether anyone queries the database or not.
2. A logical replication slot whose restart_lsn stopped advancing	A CDC connector, a DMS task, or a hand-rolled pglogical consumer holds a slot open permanently. That is normal. What is not normal is restart_lsn standing still: the cluster retains every WAL segment behind it, the walsender stays attached, and the writer never reaches the quiet state that scale-down needs. This one hides well because the consumer's own health check is green.
3. A resident working set plus a job that never lets it go idle	A rollup that runs every five minutes and briefly touches a large table will hold the buffer cache warm and reset the scale-down evaluation before it completes. Capacity ratchets up on each burst and never gets a long enough gap to come down. You see this as a capacity floor that is high but not at the ceiling, with a saw-tooth of a few ACU riding on top of it.

The two things teams reach for instead are a runaway query and a connection leak. Both are real failure modes and both are visible in a single query, so spend the ninety seconds and rule them out: if nothing in pg_stat_activity is older than 60 seconds, if the connection count matches last month, and if pg_stat_progress_vacuum is empty, stop looking at the workload. The other reflex is to blame the service, as in "Serverless v2 just does not scale down properly". It scales down. Something on this cluster is asking it not to, and the next section tells you which.

The check that separates a console change from a stuck replication slot

Two calls, ninety seconds, and the branch is decided

# 1. What is the ceiling, and what is the floor you pay for around the clock?
aws rds describe-db-clusters \
  --db-cluster-identifier claims-analytics-prod \
  --query 'DBClusters[0].ServerlessV2ScalingConfiguration'

# 2. Is anything holding WAL? Run this on the writer.
SELECT slot_name,
       plugin,
       slot_type,
       active,
       active_pid,
       restart_lsn,
       pg_size_pretty(
         pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)
       ) AS retained_wal
FROM pg_replication_slots
ORDER BY pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) DESC;

One call returns the ceiling you set. One query returns the thing that ignores it.

The first call returns a small object with MinCapacity and MaxCapacity. If MinCapacity reads 16 and your Terraform says 2, you are done with half the investigation: that is your floor, you have been paying it every hour since it changed, and CloudTrail will tell you who set it and when. Do not stop there, though. A raised floor explains a floor. It does not explain a writer sitting at 128.

The second query is the one that decides the rest. On this cluster it returned exactly one row: a logical slot on the pgoutput plugin, active true, with a live active_pid, and retained_wal of 241 GB. The number by itself is not proof of anything, because a healthy consumer can be legitimately behind during a backfill. Run the query twice, sixty seconds apart, and compare restart_lsn itself rather than the size sitting beside it. retained_wal is a distance from a moving target: pg_current_wal_lsn() advances with every write, so a constant retained_wal means restart_lsn is advancing at exactly the write rate, which is a consumer holding a steady lag, not a stuck one. The stuck condition is restart_lsn that has not moved at all between the two samples, and retained_wal growing is its confirming symptom. Falling means the consumer is catching up and what you have is a throughput problem.

Two branches, and the common case is that both fire. Fixing only the scaling config leaves the cluster pinned and makes it look like the fix failed.

Here is the part a generic answer misses. A slot can be stuck while the connector is entirely healthy, because restart_lsn only advances when the consumer commits an offset for a change it captured. If the tables in the publication are quiet while the rest of the database is hot, the connector has nothing to commit, so it never acknowledges any position, so the cluster keeps every WAL segment written since the last real event. Every dashboard is green. Every health check passes. The slot has been standing still for nine days. We check the captured tables' write rate against the cluster's total write rate whenever the two look decoupled, and it is the fastest way to catch this.

How do you bring capacity down without forcing a CDC re-snapshot?

Lower the ceiling before you go anywhere near the slot

Capture before you change. A writer reboot or a failover clears the buffer cache and resets pg_stat_statements, which destroys the memory-pressure evidence and the query history you will want if the capacity climbs back. Dropping the slot destroys the lag measurement that proves what happened. Before any mutation, save the ServerlessDatabaseCapacity series for the past 14 days, the full output of the slot query, a snapshot of pg_stat_statements ordered by total_exec_time (the column is total_exec_time on PostgreSQL 13 and later), and the CloudTrail event for the scaling change if there is one.

aws cloudwatch get-metric-statistics \
  --namespace AWS/RDS \
  --metric-name ServerlessDatabaseCapacity \
  --dimensions Name=DBInstanceIdentifier,Value=claims-analytics-prod-writer \
  --start-time 2026-07-07T00:00:00Z \
  --end-time 2026-07-21T00:00:00Z \
  --period 3600 \
  --statistics Average Maximum \
  --output json > capacity-before.json

Two weeks of hourly capacity, on disk, before anything changes. Note where the window ENDS: on the step-change day Cost Explorer already handed you, 2026-07-21 here. Run it through to today instead and the Maximum series hands back the pinned value you are trying to remove, 128, rather than the ceiling the cluster genuinely used before the change.

The scaling configuration comes down first, because it is the cheapest and most reversible move you have. It is a soft limit that takes effect at the next capacity evaluation and restarts nothing. Set max_capacity to a number the cluster actually reached before the change, which is the Maximum in capacity-before.json now that the window stops at the step change, and not to a guess. Here that Maximum read 32, and capacity fell from 112 to 32 within a few minutes with no query errors and a small, settled rise in p99.

aws rds modify-db-cluster \
  --db-cluster-identifier claims-analytics-prod \
  --serverless-v2-scaling-configuration MinCapacity=2,MaxCapacity=32 \
  --apply-immediately

# Then land the same values in code so the next console edit reads as drift.
resource "aws_rds_cluster" "claims_analytics" {
  # ...
  serverlessv2_scaling_configuration {
    min_capacity = 2
    max_capacity = 32
  }
}

The CLI stops the bleeding in one call. The Terraform block is what stops the same person doing it again next quarter.

The slot is where people cause real damage, so go slowly. Postgres will refuse to drop a slot a consumer is attached to, and that refusal is a favour: dropping it forces most CDC connectors to re-snapshot the source tables, which on a 180 GB dataset is a multi-hour outage for everything downstream.

ERROR:  replication slot "cdc_claim_events" is active for PID 24817

Read this as a warning, not an obstacle. The slot is load-bearing for a pipeline somebody owns.

Work in this order instead. Confirm who owns the consumer and whether it is running. If it is running and behind, let it catch up and re-measure; a backfill that finishes takes the pressure off by itself. If it is running but restart_lsn is not moving, the offset commit is the problem, and the fix is to make the connector emit something to acknowledge even when the captured tables are quiet. Debezium's PostgreSQL connector exposes a heartbeat interval and a heartbeat action query for exactly this (heartbeat.interval.ms and heartbeat.action.query); check the property names against the connector version you run, and point the action query at a small scratch table. One requirement decides whether this works at all: on pgoutput, decoding only emits changes for tables in the publication, so a scratch table outside it generates WAL and produces no decoding output, the connector still has nothing to acknowledge, and the slot does not move. Add it first, ALTER PUBLICATION <name> ADD TABLE <scratch>, then watch restart_lsn advance. Skip that and the fix looks like it failed when it was never wired up. A controlled pause and resume, after verifying the connector has flushed its offsets, is the safe way to nudge a slot that has drifted. Dropping the slot is the last option, taken deliberately, with the re-snapshot scheduled.

Confirm the fix on the shape of the curve, not on a single reading. Watch one full traffic cycle: the number you care about is the off-peak floor, not the peak. On this cluster the writer settled to 3 ACU at 04:00 UTC and rode up to 12 ACU during the business-hours ramp, which is a sawtooth again, and the day's total came to $19. Check that against the baseline rather than against relief: with the reader back on its restored floor of 2, the pair averages about 6.6 ACU, which is the 6.4 this cluster ran at before any of this started. A day that lands at $47 is 16 ACU average and still nearly three times baseline, and it will feel like success because it is so much better than $426. If the off-peak floor still matches the daytime peak, one of the two causes is still live and you have fixed the visible one.

Alarm on ServerlessDatabaseCapacity itself: average above roughly twice your known peak for 6 consecutive hourly datapoints. On this cluster that threshold would have paged inside a day instead of at monthly close.
Watch the scaling configuration, not just the capacity. An EventBridge rule matching CloudTrail's ModifyDBCluster events from aws.rds, routed to a channel with the calling IAM identity in the message, turns a silent console edit into a named change in minutes rather than at the next nightly plan.
Export retained WAL bytes per slot from the writer as a custom metric on a one-minute schedule. It is the single number that predicts this bill, and it is cheap to ship.
PostgreSQL 13 and later expose max_slot_wal_keep_size, which invalidates a slot once its retained WAL passes the limit. Confirm whether your Aurora parameter group exposes it before you plan around it, and be clear with the pipeline owner that invalidation means a re-snapshot.
Give temporary parameter overrides an expiry. A short-lived branch with an expiry date and a scheduled job that opens the revert PR the next day costs almost nothing and closes the exact hole that produced this.

One control worth more than the rest: if the drift alert exists and nobody sees it, it does not exist. On this engagement the nightly drift plan had been running the whole time and posting into a channel that had been muted weeks earlier after a noisy false-positive run. Muting a channel to silence one bad alert is how a working detection system becomes decoration. When we audit a cost incident we now check which alerting channels have received zero human reads recently, and it turns up more than it should. We have written about the wider pattern in our work on cloud cost spikes.

Aurora Serverless v2 scale-down questions we get asked next

What people type into search right after this one

Short answers to the follow-ups that arrive within an hour of the first fix.

Does a high max_capacity cost anything if the cluster never reaches it? You are billed for ACUs in use, with min_capacity as the floor, so the ceiling itself is not the charge. The risk is that a high ceiling removes the guardrail that would have capped the climb, which is precisely what happened here.
Can I just drop the replication slot to make capacity fall? Yes, and capacity will fall. Most CDC consumers will then re-snapshot the source tables, so treat it as a scheduled outage for the downstream pipeline rather than a quick fix. Confirm the owner first.
Will a failover or a writer reboot clear this? It will reset capacity for a while and it will clear your buffer cache and your pg_stat_statements history, which is evidence you want. The slot backlog is not cleared by a failover, so if the slot is the cause, the capacity climbs straight back.
Does any of this apply to Aurora MySQL Serverless v2? The scaling behaviour and the console-drift failure mode carry over directly. The replication mechanism does not: on MySQL the equivalent thing to look at is binlog retention and whoever is consuming it.
What min_capacity should we set? We have stopped setting min_capacity above 4 ACU on clusters with a genuinely quiet overnight window. The honest cost of that position: the first minute of the morning ramp runs against a cold cache, and on this cluster that read as roughly 23ms of extra p99 before it settled. Against $80.64 a day for a floor nobody uses, we take the 23ms.
Why did capacity stay high even after the connector caught up? Scale-down is gradual by design, and it is bounded by memory that the buffer cache is still holding. Give it a full off-peak window before you conclude the fix did not work.

If you arrived here because a CDC pipeline you inherited is doing something you cannot explain, the replication slot is usually the least documented part of the whole system. We cover that ground in our migration recovery work, where a half-finished cutover leaving a live slot behind is one of the more common things we walk into.

When the ACU line is flat and the invoice is not waiting for you

If the cycle closes before the capacity comes down

The hard version of this is not the diagnosis. It is the moment you have found a 241 GB slot, the pipeline that owns it belongs to a team that is mid-sprint on something else, the billing cycle closes in 36 hours, and the safe fix and the fast fix are not the same fix. Getting that call right needs someone who has seen both outcomes: the controlled restart that costs 78 seconds, and the slot drop that costs a six-hour re-snapshot and an apology to three downstream consumers.

We do this work on live Aurora clusters with the pipeline owners in the room, and we care as much about the guardrail you land afterwards as about the number on the dashboard tonight. Most of these engagements end with two alarms, one Terraform PR, and a slot-lag metric that nobody had before. If your capacity chart has been flat for days and the close is coming, book an infrastructure review and we will be on a call with you the same day to work the two branches in order.

Originally published at https://infraforge.agency/insights/aurora-serverless-v2-not-scaling-down/.

If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.

Karpenter picked an instance family our Savings Plan never covered

Muhammad Hassaan Javed — Thu, 27 Aug 2026 09:04:57 +0000

Savings Plan coverage across the fleet went from 100% in June to about 41% two weeks into July, and not one thing on the platform team's dashboards moved. Node count flat. Pod count flat. p99 latency flat. Karpenter had quietly replaced most of a 4-cluster fleet with m7a nodes, a family none of the three EC2 Instance Savings Plans reaches, while the m5 EC2 Instance Savings Plan kept billing $18.40/hr for m5 hours nobody was using anymore. An EC2 Instance plan is scoped to a single family in a single region; the m5 plan buys m5. The answer was in the NodePool selector, not in Karpenter.

Problem signals:

Savings Plan coverage percent drops sharply while total instance-hours stay inside normal variance
A brand new instance-family line item appears in Cost Explorer that was at zero the prior month
kubectl get nodes shows a family nobody added to any NodePool by name
Cost Anomaly Detection has no notion of Savings Plan coverage; it models spend, so the stranded commitment is structurally invisible to it and the new on-demand family only alerts if it clears your subscription threshold
Node count, pod count and latency are all flat across the window where the bill moved

What we checked first when Savings Plan coverage fell 59 points

Three explanations that all failed the arithmetic

The FinOps analyst at this healthtech SaaS (about 230 engineers, single us-east-1 footprint, four EKS clusters, EC2 running around $58k of a $92k monthly bill) pulls a coverage report on the first Monday of the month. June came back at 100%, consistent with the prior eight months. The partial-July pull came back at about 41%. Total instance-hours had moved from roughly 1,850 to 1,880 per day, which is inside the noise of ordinary customer onboarding. Only the coverage number had broken.

The platform lead's first answer was that nothing had changed, and it was an honest answer. Karpenter had gone from 1.0.6 to 1.1.4 in the June 30 maintenance window, two engineers had reviewed that PR, and the diff was a version string in a Terraform manifest. No NodePool CR had been edited. No large service had shipped. So we went looking for a billing-side explanation, and we burned most of a day on three of them.

The first was untagged usage falling out of the coverage denominator. That one dies on the API surface before you run a command: get-savings-plans-coverage exposes no tag dimension at all, so coverage data is never tag-scoped and tagging cannot move the number in either direction. Wrong tree. The second was a new linked account joining the Organization without Savings Plan sharing turned on, which genuinely does dilute aggregate coverage. aws organizations list-accounts returned the same eight accounts as June. Checking the sharing setting is not something you can do from a member account, which is worth saying because we tried it first: aws savingsplans describe-savings-plans returns only the plans the calling account owns, so a linked account never sees the payer's commitments through it, and the call says nothing about discount sharing either way. Sharing is a payer-level billing preference. We read it in the management account under Billing preferences, "Reserved Instance and Savings Plans discount sharing", and confirmed the discount was actually landing in the linked accounts from the payer, one call per account, filtering rather than grouping: aws ce get-savings-plans-coverage --time-period Start=2026-07-01,End=2026-07-22 --filter '{"Dimensions":{"Key":"LINKED_ACCOUNT","Values":["<acct-id>"]}}' --granularity MONTHLY. LINKED_ACCOUNT is a filter dimension on that API, not a GroupBy attribute; GroupBy accepts only INSTANCE_FAMILY, REGION or SERVICE, and --granularity is legal precisely because there is no GroupBy on the call. It was on. The third explanation was a service migrating from EC2-backed nodes onto Fargate, which an EC2 Instance-family plan does not reach. Both Fargate profiles were the same two that had existed for a year, and neither had picked up new work.

We also confirmed the boring one, that a plan had quietly retired. The account holds three EC2 Instance Savings Plans, one each for m5, m6i and r5, and all three had end dates in 2027 and 2028. Four explanations, four dead ends, and half a day gone against a billing cycle that closed on August 1.

The per-family coverage breakdown that ended the search in one query

22,250 hours on a family with zero coverage

The report the analyst pulls monthly is an aggregate. Aggregates hide exactly this class of problem, because a fleet swapping one family for another keeps every total roughly constant. Mid-afternoon we re-ran the coverage query grouped by instance family instead. That is not one extra argument on the same call, which cost us twenty minutes: get-savings-plans-coverage rejects --granularity when --group-by is set, so you issue one call per window and diff them yourself. It is also a spend API rather than a usage one, so the hours had to come from a second call to get-cost-and-usage.

# get-savings-plans-coverage rejects --granularity when --group-by is set,
# so run one call per window and diff them yourself.
aws ce get-savings-plans-coverage \
  --time-period Start=2026-06-01,End=2026-07-01 \
  --group-by Type=DIMENSION,Key=INSTANCE_FAMILY --output json

aws ce get-savings-plans-coverage \
  --time-period Start=2026-07-01,End=2026-07-22 \
  --group-by Type=DIMENSION,Key=INSTANCE_FAMILY --output json

# Coverage is spend-based. Each group returns only
# {CoveragePercentage, OnDemandCost, SpendCoveredBySavingsPlans, TotalCost}:
#   m5   "CoveragePercentage": "100" -> "100"
#   m7a  no usage in June            -> "0"   (~$10.3k OnDemandCost)
#   m6i  "CoveragePercentage": "100" -> "100"
#   r5   "CoveragePercentage": "100" -> "100"

# Hours are a different API, and get-cost-and-usage cannot group by
# INSTANCE_TYPE_FAMILY at all -- there it is a filter dimension only.
# Group by INSTANCE_TYPE and roll the sizes up into families yourself,
# or issue one call per family with
#   --filter '{"Dimensions":{"Key":"INSTANCE_TYPE_FAMILY","Values":["m7a"]}}'
aws ce get-cost-and-usage \
  --time-period Start=2026-06-01,End=2026-07-22 \
  --granularity MONTHLY --metrics UsageQuantity \
  --group-by Type=DIMENSION,Key=INSTANCE_TYPE --output json

# Sizes rolled up into families:
#          June, 30 days        Jul 1-22, 21 days
#   m5     47,000 (1,567/day)   11,280   (537/day)
#   m7a         0     (0/day)   22,250 (1,060/day)
#   m6i     2,300    (77/day)    1,610    (77/day)
#   r5      6,200   (207/day)    4,340   (207/day)
#   total  55,500 (1,850/day)   39,480 (1,880/day)

The aggregate said the fleet was fine. Grouped by family, 22,250 hours had walked onto a family with zero coverage while total hours per day did not move.

The July bucket is 21 days against June's 30, so read the per-day column and not the totals. Per day, m6i and r5 did not move at all, and the fleet total went from 1,850 to 1,880 hours a day, which is the variance the analyst had already dismissed. m5 had settled at roughly 29% of its June run rate; the 21-day average still reads higher than that because the swap itself sits inside the first day or two of the window. And a family that did not appear in June at all was carrying 22,250 hours at zero coverage. Confirming it in the cluster took one command, and the count told us how far it had gone.

kubectl get nodes \
  -o jsonpath='{range .items[*]}{.metadata.labels.node\.kubernetes\.io/instance-type}{"\n"}{end}' \
  | sort | uniq -c | sort -rn

     41 m7a.2xlarge
      7 m5.2xlarge

Nobody had typed m7a into a manifest anywhere in the repo. It was on 85% of the general-purpose pool.

The NodePool that governed that pool asked for instance-category In ["m", "r"], a set of allowed CPU counts, and an instance-family NotIn ["m5a", "m6a"] exclusion added in early 2024 after a numerical-precision problem with an AMD-based workload that had since been decommissioned. There was no generation constraint and no positive allow-list. The selector said, in effect, any m or r family AWS offers us, minus these two names. m7a sat inside that selector and no rule in the cluster objected, and it had been permitted long before the June 30 upgrade; nothing in that upgrade widened it. Karpenter's AWS provider enumerates the families it may pick from at runtime, from ec2:DescribeInstanceTypes and ec2:DescribeInstanceTypeOfferings against this account's own subnets and zones, then prices them from the AWS Pricing API. A family becomes selectable the day EC2 offers it in your zones: no controller release, no restart, no manifest change. The exclusion list was not out of date either, which is the uncomfortable part. m7a and m7i had both been generally available in us-east-1 since August 2023, five months before that NotIn was written. It was incomplete on the day it shipped, against families that already existed, because it named the two AMD generations the author had in mind and left everything else permitted.

Karpenter then did precisely its job, though not for the reason we assumed on day one. Our first theory was that it had found m7a cheaper. It had not, and the rate card says so plainly: in us-east-1, m7a.2xlarge lists at about $0.4637/hr against $0.384/hr for both m5.2xlarge and m6i.2xlarge, for the identical 8 vCPU / 32 GiB shape. Karpenter ranks candidate types on hourly price and, with consolidationPolicy: WhenEmptyOrUnderutilized, only replaces an underutilized node when the replacement is cheaper. A price-ranked replacement can never move a fleet from m5 onto on-demand m7a, and these hours billed at full on-demand list, so spot pricing was not in play either. What moved the fleet was capacity, and what asked for that much capacity at once was drift. The controller version bump was coincident rather than causal: bumping the chart restarts the controller pods and leaves existing NodeClaims alone, and Karpenter guards against mass drift across releases with the karpenter.sh/nodepool-hash-version annotation precisely so an upgrade does not roll the fleet. What did roll it went into the same maintenance window: the EC2NodeClass used an amiSelectorTerms alias rather than a pinned AMI ID, the alias resolved to a newly released AMI, and every node whose resolved AMI no longer matched was marked drifted and replaced. That needs no NodePool CR edit, which is why the version-string diff two engineers reviewed showed nothing. Karpenter hands EC2 Fleet a list of acceptable types in price order and Fleet launches what is actually available in the zone. m7a was on that list the whole time, ranked behind m5 and m6i on price; drift churned nodes fleet-wide, every replacement re-ran instance selection, and with that many launches compressed into one window the m5 and m6i capacity the fleet asked for was not there, so the requests fell through to the family that was. The hour table says how fast it finished. Steady-state m5 plus m7a is 1,596 hr/day once you subtract the flat 284 for m6i and r5, so with m5 settled at 29% of its June rate (454 hr/day), steady-state m7a is about 1,142 hr/day. The measured m7a average is 1,060 hr/day, 93% of steady state across the entire 21-day window, which only happens if the swap completed inside the first day or two of July. A migration trickling across three weeks would have landed near 12,000 hours, not 22,250. The m5 side agrees: 11,280 hours is only about a day and a half of full-rate m5 above the settled rate. What the provisioner cannot see either way is the Savings Plan portfolio, because commitments live in the billing system and are not part of the pricing signal it reads. Cheaper on the hourly rate card and cheaper after commitment amortization are two different questions, and Karpenter only answers the first.

Both halves of the loss run through a fleet that looks completely healthy from the cluster side.

That double charge is the part people miss. The commitment does not stop when you stop using it. $18.40/hr kept flowing whether or not m5 nodes existed, and with m5 usage down to 29%, roughly $310 a day of it bought nothing. Meanwhile the m7a hours billed at full on-demand list. Neither half appears as a line item labeled waste. The first shows up only as a coverage percent under 100, and the second shows up as a new usage type that nobody was watching for.

How we chose the fix with 9 days left on the billing cycle

Constrain the pool, buy a second commitment, or take the haircut

By late afternoon on day one we had three real options on the table and a cycle closing August 1. The analyst was on PTO from the 25th and the platform lead lost the 24th to a customer-facing latency incident, so the working window was about five days, not nine.

Step	What it does
1. Constrain the NodePool back to covered families	Fastest path, fully reversible, restores coverage inside a day. The cost is real: it gives up whatever genuine capacity headroom m7a offers on the hours that sit outside the commitment anyway. We took this one.
2. Buy an m7a Instance-family Savings Plan	Locks in a discounted rate on the family the fleet had already drifted onto. Rejected. It commits us for a year to a family the provisioner might migrate away from the next time zone capacity tightens, which is the same trap with a different name on it.
3. Replace all three plans with Compute Savings Plans	A Compute plan flexes across families, so this failure mode stops existing structurally. It cannot be done as a conversion, though: Savings Plans cannot be converted, exchanged, modified or cancelled once past the 7-day return window. The routes are layering a Compute plan on top of current spend, or waiting out the 11 months left on the nearest of the three EC2 Instance plans and replacing them as they expire. Either way it costs roughly 7 points of discount, so it was never a this-week move. It went to the Q4 architecture review.
What we did not do: revert Karpenter	The upgrade was coincident rather than the trigger: a chart bump restarts the controller pods and leaves existing NodeClaims alone. What churned every node and re-ran instance selection fleet-wide in one window was AMI drift, the EC2NodeClass alias resolving to a newly released AMI. Rolling the version bump back would still have looked like a fix and changed nothing. The release was never what admitted m7a; the controller reads the available families from EC2 at runtime, so the open-ended selector would have gone on permitting m7a on 1.0.6 exactly as it did on 1.1.4, and the next fleet-wide replacement of any kind reproduces this. The selector was the defect.

The change itself was small, and it was deliberately not another round of exclusions. A NotIn list plus a generation ceiling still permits every m and r family at generation 6 or below that nobody thought to name: m4, m5n, m5d, m5dn, m5zn, m6id, m6in, m6idn, r4, r5a, r5b, r5d, r5n, r6a, r6i, r6id, r6in. Not one of those is covered by any of the three plans, and the next thin-capacity hour lands the fleet on one of them exactly as it landed on m7a. m6a is itself a generation-6 family, which is why it still has to be named by hand, and that is the proof the generation bound cannot carry this on its own. So we replaced the deny-list with a positive allow-list naming exactly the three covered families, and kept the generation bound behind it as a second guard.

requirements:
  - key: karpenter.k8s.aws/instance-category
    operator: In
    values: ["m", "r"]
  - key: karpenter.k8s.aws/instance-family      # the load-bearing line
    operator: In
    values: ["m5", "m6i", "r5"]
  - key: karpenter.k8s.aws/instance-generation  # second guard, not the fix
    operator: Lt
    values: ["7"]

A NotIn list is a list of yesterday's problems, and a generation bound still admits every uncovered family under the ceiling. The In list is the one that holds against families nobody has named yet.

We planned it, had a second engineer read the plan, and applied to the largest cluster at 15:40 on day one. Karpenter picked up the new constraints on its next reconcile, well under a minute. Then the second-order problem showed up: unwinding this is itself a fleet-wide node replacement, and it moves no faster than the disruption budget and the drain times allow. The pool had disruption.budgets set to nodes: "10%", which is a setting we would defend on any production cluster, but read it correctly before you plan a window around it. That field is a concurrency cap, not a rate: at most 10% of the pool may be undergoing voluntary disruption at any one time, roughly four or five nodes here. There is no hourly-rate form of it; if you want a genuine time-based limit, that is a budgets entry with schedule and duration. Elapsed time is therefore set by how long each node takes to drain, PDB-gated pod eviction plus the replacement node registering, not by a per-hour quota. Draining 41 m7a nodes back onto m5 and m6i took a little over two hours. PDBs held, nothing paged, and every one of those hours billed m7a at on-demand while the m5 commitment was still under-consumed. We could have raised the budget to move faster. We did not, because trading a stable rollback for a couple of hundred dollars is a bad trade in a system where the failure mode we are recovering from is invisible to health checks.

The other thing that did not snap back was the number we were watching. Day two we finished the remaining three clusters by end of business. Day three, the trailing-window coverage query read 91.4%, not 100%, and that was correct rather than alarming. The coverage report looks back over a rolling window that still contained the uncovered m7a hours from earlier in July. Anyone treating a 24-hour coverage spot check as a pass/fail gate will either panic or declare victory early. Read the trend across several days, not the number on the day you shipped the fix.

FinOps closed the cycle at about $22,500 of overage across the 21 days before detection, plus roughly $2,400 during the mitigation week. Just under $25,000 on a month where the compute bill should have looked exactly like June's.

The 4 controls we shipped so the next new family cannot do this

Deny-lists inherit whatever the vendor ships next

The durable lesson is not about Karpenter and it is not about AMD. It is that a selector written as an exclusion list silently accepts every family you did not think to name, including the ones that already existed on the day you wrote it. Our NodePools were written as deny-lists, and a deny-list is a bet that you enumerated a set you do not control correctly, and that you will keep re-enumerating it every time AWS extends it. That bet loses on a long enough timeline, and the loss shows up in the billing system weeks later rather than in a failing test.

Four things landed in the platform IaC repo and the on-call docs over the two weeks after. The first is a rule in CI requiring instance-family to use operator: In with an enumerated list instead of NotIn. That flips the failure mode from silently adopting new families to silently ignoring them until a human adds one, which is the direction you want a cost-bearing default to fail in. We shipped it soft-failing for two weeks so teams could migrate, then made it blocking. It touched 11 NodePools across the four clusters. The second is a companion OPA policy, about 30 lines of rego, that fails the plan on any NodePool without an explicit karpenter.k8s.aws/instance-generation constraint. It is a backstop rather than the control that closes this hole, since a generation ceiling on its own still admits uncovered families below it, but the discovery set is now stated by us rather than inherited.

The third is the detection gap, and it is the one we would fix first if we could only do one. AWS Cost Anomaly Detection was watching the raw EC2 compute line at a $500/day threshold and never fired. The explanation we reached for first, that the m5 commitment kept billing and offset the new on-demand spend, is backwards and worth correcting out loud: a Savings Plan charges its fixed hourly amount whether or not you consume it, so a constant charge cannot offset anything. The raw line did move. Of the roughly $22,500 of overage, about $6,500 was commitment buying nothing at $310 a day, and the other $16,000 or so was genuinely new on-demand spend, about $760 a day, comfortably above the threshold, and it arrived as a step change in the first days of July rather than as a slow drift. That monitor should have caught it and did not, and we never got an account of its baselining out of it that we were willing to trust. What we could establish is what it does not evaluate at any threshold: coverage percent. So we stopped trying to tune a spend monitor into a coverage monitor.

So we wrote a nightly Lambda, about 80 lines of Python on an EventBridge schedule, that pulls the trailing seven days of coverage grouped by instance family and posts to the FinOps Slack channel on three conditions: a family carrying material uncovered spend that had none the week before, aggregate coverage moving more than 5 percentage points week over week, and Savings Plan utilization falling below 100%. Per-family coverage percentage is deliberately not one of them, and we learned that by writing the naive version first. Through this entire incident m5, m6i and r5 all read 100% and m7a read 0% from its first hour to its last, so a week-over-week delta on per-family coverage percent would never have crossed any threshold. What moved was which families carried the volume, and the aggregate, which went 100 to 41. Against this incident the alert as shipped would have fired within a day or two of the migration instead of 22, because m7a went from no usage at all to a full day of it. The fourth is a one-page runbook for the phrase "SP coverage dropped": run the coverage query grouped by family once per window and diff the two, pull the matching hours from get-cost-and-usage grouped by INSTANCE_TYPE and rolled up into families yourself (that API takes INSTANCE_TYPE_FAMILY only as a filter, never as a GroupBy), and if any family went from near zero to non-zero, go read the NodePool selector before you go read anything else.

We now treat any Kubernetes autoscaler as a component with a billing blast radius, not just a scheduling one, and we review its selectors during cost reviews rather than only during platform reviews. That framing is the same one we bring to Kubernetes and CI/CD stabilization work generally: the controller doing exactly what it was told is the most expensive kind of correct. If the shape of this feels familiar from the invoice side rather than the cluster side, the pattern is covered from that angle in cloud cost spikes.

Karpenter and Savings Plans: 4 questions worth answering before your next NodePool review

The questions the retro kept circling back to

These came up in the retro and have come up in every conversation we have had about this since.

Does an EC2 Instance-family Savings Plan cover a different family in the same category? No. m5 and m7a are distinct families under the plan's scope rules, not size or generation variants of one another. An EC2 Instance Savings Plan commits to one instance family in one region, flexes only across size, OS and tenancy inside it, and buys a deeper discount in exchange for that narrowness. A Compute Savings Plan is the one that flexes across families, at a lower rate.
Would Cost Anomaly Detection have caught this? Ours did not, at a $500/day threshold, and not for the reason we first told ourselves. The commitment did not mask the delta: a Savings Plan bills the same fixed amount whether you consume it or not, so it cannot offset anything. The raw EC2 line really did rise, by about $760/day of new on-demand spend, and it rose as a step change in the first days of July. It should have fired and it did not. Check what your monitor actually evaluates before you rely on it here; Cost Anomaly Detection models spend and has no notion of Savings Plan coverage, so a coverage collapse is structurally invisible to it, and the control that covers this failure is the per-family coverage alert.
Is it safe to change a NodePool selector on a busy production cluster? It was for us, and the reason is disruption.budgets plus PDBs. Read the budget correctly first: nodes: "10%" caps how many nodes may be under voluntary disruption at once, not how many per hour. Karpenter replaced 41 nodes with four or five draining concurrently, a little over two hours end to end, with no service impact. Budget the elapsed time from your own drain and registration times, and resist raising the cap to finish faster; that is exactly where a cost incident turns into an availability incident.
Should we just move to Compute Savings Plans? It removes this failure mode structurally, and it costs roughly 7 points of discount. Note that it is not a conversion: Savings Plans cannot be exchanged, modified or cancelled after the 7-day return window, so you either layer a Compute plan on top of current spend or replace the EC2 Instance plans as they expire. Nobody on the team was happy about the 7 points and nobody had a better answer. If your fleet is under an autoscaler that is free to pick families, price the immunity honestly rather than assuming the deeper discount is the cheaper option.

When the cluster looks healthy and the invoice does not agree

If your coverage number moved and your dashboards did not

The hard part of this class of incident is not the fix, which was a handful of lines in a NodePool. It is that every signal a platform team looks at daily stays green while the money leaves, and the one signal that would have caught it lives in a monthly FinOps artifact that the platform team does not own or read. Two functions each holding half the picture is how a 22-day detection lag happens to competent people.

We do this work on live clusters: reading autoscaler selectors against a commitment portfolio, finding the gap between what the scheduler optimizes and what the invoice charges, and leaving behind the CI policy and the alert so it does not come back the next time a new family lands in your zones.

If a coverage number just moved on you and the cluster looks fine, book an infrastructure review and we will get on a call the same day to find where your fleet went.

Originally published at https://infraforge.agency/insights/karpenter-savings-plan-coverage-drop/.

If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.

How a do-not-disrupt annotation broke Karpenter consolidation

Muhammad Hassaan Javed — Tue, 18 Aug 2026 11:36:38 +0000

Karpenter launched 2,800 nodes over one weekend and consolidated only 180 of them, and our EC2 on-demand bill jumped 2.2x. The cause was one line added to a shared internal Helm chart on Friday afternoon: karpenter.sh/do-not-disrupt: true, applied to every pod template the chart rendered. Renovate auto-merged the bump across 34 tenant repos over the weekend, and by Monday about 1,450 pods carried the annotation. Karpenter's consolidation loop disqualifies any node holding at least one annotated pod, so roughly 85% of nodes became unconsolidatable while new pods kept arriving. Here is how we spotted it and rolled it back without triggering a fleet-wide restart of every workload the chart owned.

Problem signals:

Karpenter's launched instance events run 5-10x baseline while disrupting via consolidation events sit at half of normal
EC2 on-demand line items on the two largest instance sizes double while spot line items stay flat
Node count climbs steadily through a quiet weekend with no matching increase in pod count or RPS
kubectl finds karpenter.sh/do-not-disrupt: true on a majority of pods even though only a handful of workloads legitimately need it
On-demand share of total node hours drifts from ~30% to >60% even though the NodePool weights spot at 100

The metric that redirected us away from HPAs

The launch rate was 7x normal; the consolidation rate was half

Our first read was wrong. The Monday FinOps digest fired at 08:15 UTC showing EC2 on-demand up 2.2x for the trailing 72 hours, and the platform lead's instinct was 'someone shipped an HPA that went sideways over the weekend.' We checked requests per second on the top 20 services; nothing moved more than 8% week over week. Total pod count was up about 40 pods against a fleet of 2,400. Not an HPA problem. Something was provisioning nodes without a matching demand signal, and the extra capacity was mostly on-demand even though the NodePool weighted spot at 100.

The evidence that pointed us at the right layer came from Karpenter's own controller logs. We asked it to summarise its own actions over the incident window:

kubectl -n karpenter logs deployment/karpenter --since=72h \
  | grep -E 'launched instance|disrupting via consolidation' \
  | awk '{print $6}' | sort | uniq -c

 2812 launched
  178 disrupting

# Baseline weekend, same window a fortnight prior:
  412 launched
  347 disrupting

Karpenter was doing both halves of its job. It was just doing one seven times too often and the other half as often as it should.

The provisioning path was healthy. There were no failed provisioning or unschedulable events. Karpenter was launching nodes about seven times as often as a normal weekend and consolidating them at half the usual rate, so capacity was arriving faster than it was leaving. Over 60 hours that gap accumulated into 98 extra nodes, most of them on-demand, sitting quietly and costing money.

One line in a shared chart, 34 tenants downstream

The chart shipped Friday at 15:04 UTC and Renovate did the rest

Once the question was 'why isn't consolidation firing', the answer came out fast. Karpenter's consolidation logic evaluates candidate nodes by checking whether every pod on the node can be safely rescheduled. Any node holding a pod with karpenter.sh/do-not-disrupt: true is immediately disqualified from consolidation until that pod is gone. The annotation exists for legitimate reasons: Kafka Streams state, long-running batches with expensive warmup, workloads that lose data on SIGTERM. In our cluster, four workloads owned it and were the only ones that should have had it.

The reality was different:

kubectl get pods --all-namespaces -o json \
  | jq -r '[.items[] | select(.metadata.annotations["karpenter.sh/do-not-disrupt"] == "true")] | length'

1450

kubectl get pods --all-namespaces -o json \
  | jq -r '[.items[] | select(.metadata.annotations["karpenter.sh/do-not-disrupt"] == "true") | .metadata.namespace] | unique | length'

38

1,450 pods across 38 namespaces carried the annotation. With pods distributed by the default scheduler, that pinned roughly 85% of nodes.

Our first suspicion was a mutating admission webhook. kubectl get mutatingwebhookconfigurations returned nothing new. The annotation was in the pod specs at the source. We picked one pod, walked back to its ReplicaSet, saw the annotation in the pod template, and ran git blame on the tenant's Deployment manifest. The blame line pointed at an internal chart bump: internal-charts/base-deployment moved from 2.7.4 to 2.7.5 on Friday at 19:47 UTC. The diff was three lines:

+ annotations:
+   karpenter.sh/do-not-disrupt: "true"

Commit message: 'add do-not-disrupt to prevent midday restarts during batch runs.'

The chart owner had one tenant whose Kafka Streams StatefulSet was losing ~90 seconds of state on every consolidation event. The narrow fix was to annotate that one StatefulSet, which required coordinating with the tenant. The wide fix was to put the annotation in the shared chart, which required coordinating with no one. They picked the wide fix. Our Renovate config had trust minor bumps from internal charts on auto-merge. Thirty-four tenant repos consumed base-deployment. Over the weekend, thirty-four PRs opened, thirty-four PRs merged, thirty-four ArgoCD syncs rolled deployments, and every rolled pod inherited the annotation.

A single opinionated chart plus a permissive auto-merge is the shape of the whole incident. Nothing in the loop was malicious. Every step was doing what it was configured to do.

The right recovery is on the pods, not the templates

The patch we almost ran would have rolled 1,450 pods

By 09:00 UTC Monday we had the diagnosis and a bad plan. The obvious move was to patch the Deployment template in each of the 34 tenants, removing the annotation at the source:

kubectl patch deployment <name> -n <ns> \
  -p '{"spec":{"template":{"metadata":{"annotations":{"karpenter.sh/do-not-disrupt":null}}}}}'

The command that looked surgical and was not.

We almost ran the loop. Then one of us asked the question that matters: does changing a template annotation trigger a rollout? It does. The Deployment controller's ComputeHash runs DeepHashObject over the entire PodTemplateSpec, and metadata.annotations is part of the template. Change any pod-template annotation and the template hash changes, which creates a new ReplicaSet, which rolls every pod. The cleanest proof of this is that kubectl rollout restart triggers a rollout precisely by stamping kubectl.kubernetes.io/restartedAt into spec.template.metadata.annotations. If template annotations were excluded from the hash, rollout restart could not work.

If we had run the patch loop across 34 tenants at 09:00 UTC on a Monday, we would have rolled about 1,450 pods in flight, cascaded PDB blocks into deploy pipelines, and (worst) triggered even more Karpenter provisioning as the rollouts churned. The recovery would have looked identical to a second, larger incident.

The correct move was to strip the annotation from the running pods, not from the template. A ReplicaSet reconciles pod count and pod ownership, not annotation drift on pods that already exist. So mutating a live pod's annotations is safe and does not trigger replacement. Karpenter re-checks consolidation eligibility on its next loop (roughly every 30 seconds) and starts disrupting the newly-eligible nodes on its own.

Here is what we ran, iterating one namespace at a time so we could abort if anything looked off:

for ns in $(kubectl get ns -o jsonpath='{.items[*].metadata.name}'); do
  case "$ns" in kube-*|karpenter) continue ;; esac
  kubectl annotate pods -n "$ns" --all \
    karpenter.sh/do-not-disrupt- --overwrite 2>/dev/null || true
done

The trailing dash on the annotation key removes it. Total wall time across ~2,440 pods was 47 seconds.

Verification was two commands, one for the pod count and one for Karpenter's response:

kubectl get pods --all-namespaces -o json \
  | jq -r '[.items[] | select(.metadata.annotations["karpenter.sh/do-not-disrupt"] == "true")] | length'

24

kubectl -n karpenter logs deployment/karpenter --tail=200 \
  | grep 'disrupting via consolidation' | tail -5

...disrupting via consolidation, 3 candidates
...disrupting via consolidation, 5 candidates
...disrupting via consolidation, 2 candidates
...disrupting via consolidation, 4 candidates
...disrupting via consolidation, 3 candidates

Down to 24 annotated pods (the four legitimate workloads plus a 20-pod fraud-scoring batch six hours into its run) and Karpenter firing consolidation events within four minutes.

Over the next 90 minutes node count dropped from 198 to 141 and plateaued. The remaining 24 pods were pinning about 22 nodes: 20 fraud-batch pods spread across 18 nodes, and 4 legitimate pods on 4 more. Consolidation could not touch those. We coordinated a restart of the fraud batch with its owning team. It checkpointed cleanly and restarted onto 5 nodes, tight-packed by Karpenter's provisioning-time bin packing, which freed 13 nodes and took us to 128. Over the next 90 minutes natural pod churn shifted another 20 pods off nodes that had held only transient workloads, and Karpenter consolidated those too. Node count landed at 108 by 15:00 UTC, back inside baseline range.

Then we rolled out the fixed chart. base-deployment 2.7.6 removed the blanket annotation and gated it behind an explicit values.yaml opt-in (karpenter.doNotDisrupt: false default, true for the four legitimate tenants). This rollout DID replace pods, because it changed the template hash, and we scheduled it deliberately. Renovate opened PRs but no longer auto-merged them. The platform team merged them in batches of five with a ten-minute soak between batches so any interaction with PDBs or startup probes surfaced before the next wave. Six hours end to end, no SLO impact.

The changes that landed in the two weeks after

Three controls that would have caught this on Saturday

The postmortem produced three durable controls. Each one closes a specific step in the causal chain above.

The first is a Kyverno ClusterPolicy that only permits the annotation on pods that explicitly opt in via a label. It ran in Audit mode for two weeks, we watched the drift metric go to zero, then we flipped it to Enforce:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-karpenter-do-not-disrupt
spec:
  validationFailureAction: Enforce
  rules:
  - name: require-opt-in-label
    match:
      any:
      - resources:
          kinds: [Pod]
    preconditions:
      all:
      - key: "{{ request.object.metadata.annotations.\"karpenter.sh/do-not-disrupt\" || '' }}"
        operator: Equals
        value: "true"
    validate:
      message: "karpenter.sh/do-not-disrupt requires label karpenter.platform/opt-in=true"
      pattern:
        metadata:
          labels:
            karpenter.platform/opt-in: "true"

A future chart change that adds the annotation without the matching label is rejected at admission and never lands.

The second is an SLO alert on Karpenter's own consolidation metric. karpenter_disruption_actions_performed_total (the metric renamed from karpenter_deprovisioning_actions_performed_total around v0.32; the semantics are the same) was already in our Prometheus scrape config; nobody had put it on a dashboard. We added a Grafana panel showing consolidation actions per hour filtered to action="consolidation", and paged if the rate stayed below 20% of the 7-day rolling median for more than two hours during business hours. Backtesting against the incident, that alert would have fired around 04:00 UTC Saturday, roughly 8 hours in, instead of 60 hours in.

The third is a check in the shared chart's CI pipeline. Any minor bump that touches a scheduling-related annotation key (karpenter.sh/*, karpenter.k8s.aws/*, scheduler.alpha.kubernetes.io/*, node.kubernetes.io/*, cluster-autoscaler.kubernetes.io/*) sets a platform-review-required flag on the PR, and Renovate is configured to leave those PRs open rather than auto-merge them. Non-scheduling changes still flow through the previous auto-merge path unchanged. The tradeoff we now pay is about one platform-team review per quarter on this chart; the incident that check would have caught cost us roughly $3,500 in on-demand overage across the 60-hour window.

We considered tightening the NodePool's disruption.budgets and did not. The root cause was not consolidation being too eager; consolidation was fine when it was allowed to fire. Tightening budgets would slow legitimate consolidation without preventing another annotation-blast. We put a comment in the NodePool YAML pointing to the postmortem so a future engineer does not try to 'fix' the budgets in response to reading it.

FAQ: Karpenter consolidation and the do-not-disrupt annotation

Common questions we get about this pattern

Does karpenter.sh/do-not-disrupt pin just the pod or the whole node? It effectively pins the node. Consolidation eligibility is evaluated per-node, and any node holding at least one annotated pod is disqualified until that pod is gone. One annotated pod on a 32-vCPU node prevents the node from ever consolidating.

Would tightening PodDisruptionBudgets have prevented this? No. PDBs govern voluntary disruption of pods that Karpenter has already decided to touch. The annotation blocks node candidacy upstream of that check, so the eviction call PDBs guard never happens. PDBs sit downstream of the gate that closed here.

Can we detect this via CloudWatch or Cost Explorer alone? Not fast enough. Cost signals arrive at daily granularity at the earliest, and our FinOps digest ran weekly. The signal you want is Karpenter's own consolidation-actions rate from /metrics, which moves within minutes of the problem.

Does this pattern apply to Karpenter v1? The mechanism is the same. The annotation moved under different API groups across versions, and v1 uses karpenter.sh/do-not-disrupt on pods with identical semantics. The Deployment controller's PodTemplateSpec hash behavior is a Kubernetes property, not a Karpenter property, and it does not change.

What if we want every pod in a Deployment to be do-not-disrupt by design? Put the annotation in the template and accept that changing it triggers a rollout. Use maxSurge and PDBs to control that rollout the way you would for any other template change. The mistake in our incident was not that a template had the annotation; it was that 34 templates got it without the owning teams knowing.

Where Karpenter regressions get stuck

If your on-demand line jumped and no one shipped anything

The awkward thing about this class of Karpenter regression is that every piece looks correct in isolation. The annotation was a real feature intended for real workloads. The chart bump was a legitimate response to a real production pain. The Renovate auto-merge policy was tuned to reduce toil on well-behaved chart bumps. The NodePool was configured the way the docs recommend. It took the combination, plus a weekend, plus a weekly cost digest, to produce a doubled node count and a mid-four-figure overage. The diagnostic move that mattered was comparing Karpenter's launch rate against its consolidation rate. The recovery move that mattered was knowing which kubectl mutations trigger a rollout and which do not.

We have written more on this shape of failure in the Kubernetes and CI/CD stabilization pillar, and the specific pattern of a shared chart change cascading through GitOps sits in the ArgoCD and GitOps recovery cluster. If your Karpenter cluster is growing and you cannot see why, book an infrastructure review and we will pull the consolidation-actions metric together, walk your controller logs the same way we walked ours, and get you a diagnosis inside a single working day.

Originally published at https://infraforge.agency/insights/karpenter-consolidation-broken-by-do-not-disrupt-annotation/.

If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.

How we recovered a k3s cluster after its client certs expired

Muhammad Hassaan Javed — Thu, 13 Aug 2026 14:54:52 +0000

119 ephemeral preview namespaces should have been reaped between 2am and 6am. None were. The teardown cron had been failing on x509: certificate has expired or not yet valid for six hours before anyone looked at the log. The k3s server was 366 days old, nobody owned certificate rotation, and the first fix we tried made zero difference because we misread which cert was expired. This is how we got kubectl working again, in what order we restarted the control plane, and the two things we changed so the next cluster does not do this to us.

Problem signals:

kubectl get nodes returns 'Unable to connect to the server: x509: certificate has expired or not yet valid'
A CI or lifecycle script that used to work is now failing on every kubectl call, and the cluster has been up more than 11 months
systemctl status k3s shows the service is active but the k3s.service journal has TLS handshake errors from the agent
The kubeconfig at /etc/rancher/k3s/k3s.yaml decodes to a client cert whose notAfter is in the past
Ephemeral or preview environments are stuck: nothing is being created, nothing is being deleted, and the queue is growing

What we walked into

119 namespaces that should have been gone by 6am

The cluster ran preview environments for a mid-size SaaS product. Every merged PR got a namespace, every closed PR got its namespace reaped by a cron that shelled out to kubectl. Simple, effective, and completely dependent on kubectl being able to talk to the API server. On the morning of the incident the reap job had been erroring every minute for about six hours before the platform on-call noticed the namespace count on the Grafana panel had gone flat instead of sawtoothing.

The error in the cron log was one line, repeated:

Unable to connect to the server: x509: certificate has expired or not yet valid: current time 2025-04-12T06:14:22Z is after 2025-04-11T18:02:17Z

The API server had been unreachable for twelve hours and twelve minutes by the time we opened the log.

This is a single-server k3s install with one agent node, running on two small EC2 instances. It had been up for 366 days, one day longer than its own certificates were valid for. Nobody had ever rotated a certificate on it. Nobody had written down that k3s issues its own internal client certs with a 12 month lifetime and quietly renews them on restart if they are within 90 days of expiry. Ours were nowhere near that window when the server last restarted, because the last restart was the kernel patch cycle that brought this cluster up in the first place, twelve months earlier. So they aged out on schedule and the cluster locked itself out.

The first thing we tried, and why it did not work

The kubeconfig regen that fixed nothing

The instinct on the first responder was correct-looking and wrong. They assumed the kubeconfig on the CI runner had drifted, copied a fresh /etc/rancher/k3s/k3s.yaml from the server, rewrote the server URL, and reran the reap. Same x509 error. They tried it a second time with KUBECONFIG pointed explicitly at the new file. Same error.

The reason it did not work is the piece that catches people out with k3s specifically. The k3s.yaml on disk is not a pointer to cert files; it is a self-contained kubeconfig with client-certificate-data and client-key-data base64-embedded directly in it. Regenerating the file by copying it from the server just copies the same expired bytes to a new path. The cert material lives at /var/lib/rancher/k3s/server/tls/client-admin.crt, and that is what had actually expired. Until that file was rotated, every kubeconfig on every machine, freshly copied or not, was carrying the same dead cert.

This is the point where we stopped guessing and asked openssl what it saw.

How we confirmed the actual expiry before touching anything

openssl said notAfter=Apr 11

Before running any rotation command we wanted to see the cert dates ourselves. Two commands, run on the k3s server:

$ sudo openssl x509 -noout -dates -in /var/lib/rancher/k3s/server/tls/client-admin.crt
notBefore=Apr 11 18:02:17 2024 GMT
notAfter=Apr 11 18:02:17 2025 GMT

$ for f in /var/lib/rancher/k3s/server/tls/client-*.crt /var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt; do
>   echo "$f"
>   sudo openssl x509 -noout -enddate -in "$f"
> done

The client-admin cert expired at 18:02 the previous evening. Every other client cert on the server had the same notAfter within a few seconds.

The serving cert for the API server itself was also expired. That mattered for the restart order later, because a k3s server with an expired serving cert will start, but agents cannot re-establish TLS to it until the rotation writes new material and the server picks it up. If we had restarted the agent first, or in parallel, we would have watched it fail to rejoin and then chased a second ghost.

Related reading: the same failure mode shows up during cluster migrations when a snapshot from an old server gets restored past the cert lifetime. We wrote about that pattern in the migration recovery notes.

The recovery sequence that actually worked

Rotate, restart server, restart agent, in that order

k3s ships a rotation subcommand that regenerates the internal certs on disk. It does not restart the service, and it does not touch k3s.yaml; the restart in step 3 is what rewrites that file. The second trap is everything downstream of the server, which we get to below. The full sequence, run as root on the server, was:

# 1. Stop the server so nothing is holding the old TLS material.
sudo systemctl stop k3s

# 2. Rotate all internal certs. On k3s v1.28+ this rewrites everything
# under /var/lib/rancher/k3s/server/tls including client-admin.crt.
sudo k3s certificate rotate

# 3. Bring the server back up. It will pick up the new certs on start.
sudo systemctl start k3s

# 4. Wait until the API is responsive with the local (root-owned) kubeconfig.
sudo KUBECONFIG=/etc/rancher/k3s/k3s.yaml k3s kubectl get nodes

# 5. On the agent node, restart so it re-handshakes with the new server cert.
sudo systemctl restart k3s-agent

Stop first, rotate, start, verify, then touch the agent. The k3s docs are explicit that the server has to be stopped before rotating, and we have separately seen a rotate against a live server leave the API server holding the old serving cert in memory until a restart anyway.

Step 4 worked from root on the server. On the CI runner, kubectl was still failing. This is where the k3s.yaml quirk from earlier comes back, with the sign flipped. k3s rewrites /etc/rancher/k3s/k3s.yaml on every start, so by the time step 4 ran, the server's own kubeconfig already carried the new embedded material; that is exactly why step 4 worked. Nothing rewrites the copies. Every kubeconfig we had ever scp'd to a runner or a laptop still held the dead bytes, and no amount of restarting the server was going to reach them. We had to export the fresh one and re-seed each consumer by hand:

# On the server, export the kubeconfig k3s rewrote when it started.
# config view --raw only prints what is already on disk; the restart
# in step 3 is what put the fresh cert material there.
sudo k3s kubectl config view --raw > /tmp/k3s-fresh.yaml

# Verify the embedded client cert is not the dead one.
grep client-certificate-data /tmp/k3s-fresh.yaml \
  | awk '{print $2}' | base64 -d \
  | openssl x509 -noout -dates
# notAfter=Apr 12 06:41:07 2026 GMT   <-- future date, good.

# Distribute to the CI runner's kubeconfig path. In our case:
scp /tmp/k3s-fresh.yaml ci-runner:/etc/lifecycle/kube/config
ssh ci-runner 'sudo chown lifecycle:lifecycle /etc/lifecycle/kube/config && sudo chmod 600 /etc/lifecycle/kube/config'

The server's own kubeconfig is correct after the restart. The copies are not, and nothing on the cluster knows they exist. Every consumer holding one has to be re-seeded by hand.

Elapsed time from opening the log to a successful kubectl get nodes on the CI runner was 47 minutes. Most of that was the first-fix detour. The rotation itself took about 90 seconds; the server restart took under 20.

The left branch is where we spent 30 wasted minutes. The right branch is what the runbook now says to do first.

FAQ

Questions the team asked when we posted the internal postmortem

Can I just add insecure-skip-tls-verify to the kubeconfig and move on?

No. The API server is presenting an expired serving cert, but it is also validating incoming client certs. Once the client-admin cert is past notAfter, the server rejects the request during TLS with the same x509 error even if the client stopped verifying. Skipping verification on the client side does not help you here. It also leaves the agent unable to reconnect independently.

Does k3s not auto-renew certs?

It does, but only if the server restarts while the certs are inside the 90 day renewal window. If the cluster runs 12 months straight with no restart, or the last restart happened more than 90 days before expiry, the renewal window is missed and the certs age out. That is exactly what happened to us.

Do I need to restart etcd or anything else on a single-server k3s?

On single-server k3s with the default embedded SQLite or embedded etcd, systemctl restart k3s covers everything the server needs. On an HA k3s with external etcd or multiple servers, you rotate and restart one server at a time and let the API stay reachable through the others. That is a different story and worth its own runbook.

Will the pods restart when I bounce the k3s server?

No. Workload pods run under containerd and are not managed by the k3s systemd unit's lifecycle in a way that restarts them. During our recovery, application pods kept serving traffic the entire time the API was down. Only the control plane was affected. Anything using in-cluster kubeconfigs (operators, controllers) will reconnect once the API comes back.

Is the agent's own node cert also rotated?

It needs its own attention. k3s certificate rotate only rewrites files under /var/lib/rancher/k3s/server/tls on the machine you run it on; it does not reach into the agent's /var/lib/rancher/k3s/agent/. Restarting k3s-agent works because the agent cannot authenticate with its expired client cert, falls back to re-bootstrapping with the node token, and the server issues it a fresh one. That is a fallback path, not a rotation. The deterministic version is to run k3s certificate rotate on the agent node too, servers first, then agents. Either way step 5 is not optional.

Where we help

If your k3s cluster is past its first birthday

The two things we changed after this incident were not glamorous. We added a nightly check that runs openssl x509 -noout -enddate against every cert under /var/lib/rancher/k3s/server/tls and every embedded client-certificate-data in every kubeconfig we own, and pages if any is under 30 days from expiry. It is a 42 line shell script and it has already caught the same class of problem on a second cluster that a client acquired through a merger. We also scheduled a monthly forced restart of the k3s server during a maintenance window. Quarterly is not enough: restarts 91 days apart land inside the last-90-days renewal window at most once per 12 month cert, and only by a day or two. Monthly gives three restarts inside the window. Neither of these prevents the failure by itself; together they close the door.

The hard part of this kind of work is not running k3s certificate rotate. It is knowing before you touch anything which cert is actually expired, what order the components have to come back in for this specific k3s version, and which files the rotation does and does not rewrite. We do this often enough on inherited k3s and k0s clusters, especially ones that came along with an acquisition or a platform-team handover, to know where the traps are on each version. If you have a cluster that is past its first birthday and nobody on the team can point to the last time certs were rotated, book an infrastructure review and we will spend a 60 minute call walking the cert state with you before it fails at 3am. If you are staring at x509: certificate has expired in a log right now, say so in the request and we will be on a bridge the same day.

Originally published at https://infraforge.agency/insights/k3s-certificate-expiration-recovery/.

If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.

Why terraform plan wants to destroy 5 live failover resources

Muhammad Hassaan Javed — Thu, 13 Aug 2026 14:46:10 +0000

By 08:47 the terraform plan output was up on the shared screen and nobody wanted to be the one to type apply. The summary line read 'Plan: 5 to add, 0 to change, 5 to destroy.' Every one of the 5 destroys was a resource actively serving production. The Aurora cluster. The RDS proxy in front of it. Three security groups. The failover had happened at 02:14 that morning. SRE had brought the us-west-2 replacements up by hand while the primary region recovered. Six hours later, terraform did not know they existed.

Problem signals:

terraform plan shows -/+ replace on resources actively serving production traffic
The plan wants to destroy resources that were created out-of-band during a recent incident
aws describe commands confirm the resources exist and are healthy, but terraform state does not know about them
The team has stopped running terraform plan against this workspace because the output is too scary to act on

Five destroy-and-replace actions against live production

The plan wanted to destroy the primary database

The workspace was the one that owned the data layer for the primary application. Its state held the pre-failover Aurora cluster, the RDS proxy, and three security groups, all with us-east-1 ARNs. During the incident someone had updated the module's provider region from a variable to a hardcoded 'us-west-2', trying to bring the manually-created resources under management. That change alone did not import anything. What it did was make terraform want to replace every resource in the module, because state said us-east-1 and config now said us-west-2. Five resources in the module, five replacements, five destroys and five creates in the summary.

$ terraform plan

Terraform will perform the following actions:

  # module.data_layer.aws_rds_cluster.primary must be replaced
-/+ resource "aws_rds_cluster" "primary" {
      ~ arn                = "arn:aws:rds:us-east-1:...:cluster:app-primary" -> (known after apply)
      ~ endpoint           = "app-primary.cluster-abc.us-east-1.rds.amazonaws.com" -> (known after apply)
      ~ id                 = "app-primary" -> (known after apply)
      # (27 unchanged attributes hidden)
    }

  # module.data_layer.aws_db_proxy.primary must be replaced
-/+ resource "aws_db_proxy" "primary" {
      ~ arn      = "arn:aws:rds:us-east-1:...:db-proxy:prx-0a1b2c3d4e5f60718" -> (known after apply)
      ~ endpoint = "app-primary-proxy.proxy-abc.us-east-1.rds.amazonaws.com" -> (known after apply)
      # (14 unchanged attributes hidden)
    }

  # module.data_layer.aws_security_group.db_ingress[0] must be replaced
-/+ resource "aws_security_group" "db_ingress" {
      ~ arn = "arn:aws:ec2:us-east-1:...:security-group/sg-0abcdef1234567890" -> (known after apply)
      ~ id  = "sg-0abcdef1234567890" -> (known after apply)
    }

  # module.data_layer.aws_security_group.db_ingress[1] must be replaced
-/+ resource "aws_security_group" "db_ingress" { ... }

  # module.data_layer.aws_security_group.db_ingress[2] must be replaced
-/+ resource "aws_security_group" "db_ingress" { ... }

Plan: 5 to add, 0 to change, 5 to destroy.

The plan that stopped the room at 08:47

The apply would have done exactly what the plan said. It would have called DeleteDBCluster on the us-east-1 cluster, which was still up and still holding connections from the pools that had not been cut over yet. It would have called CreateDBCluster in us-west-2 with the identifier 'app-primary', which was already the identifier on the live cluster that SRE had created at 02:14. That second call would have returned DBClusterAlreadyExistsFault, and the state would have been mid-transaction, with the us-east-1 cluster gone and the us-west-2 cluster still not tracked. The same shape of failure was queued up for the proxy and the security groups, each with its own AWS-side name-collision error.

Why terraform state rm looked right and was wrong

What we almost did that would have made it worse

The reflex in the room was to run terraform state rm module.data_layer.aws_rds_cluster.primary on all 5 entries, then re-plan. State would forget the us-east-1 resources, plan would show 5 creates in us-west-2, and terraform would be back in a consistent-looking place. We had the commands typed. We did not run them, because someone asked the question that saved the afternoon: what does the create actually do when the resource already exists?

The answer is: it fails, and it fails deterministically. CreateDBCluster refuses an identifier that already exists in the region and returns DBClusterAlreadyExistsFault whether or not the properties you are asking for match the live resource. There is no adopt-if-identical path in the AWS API. CreateDBProxy behaves the same way, and duplicate security group names inside a VPC come back as InvalidGroup.Duplicate. Worse, a replace is a destroy followed by a create, so the destroy half lands first: the us-east-1 cluster is gone, the create fails, and state holds five addresses whose real resources terraform just deleted and could not recreate. Sorting that out is a bad afternoon and a worse changelog entry.

The right question is not 'how do we make terraform stop wanting to destroy things'. It is 'how do we get the resources that ARE running into state under terraform management, with the state entries reflecting reality'. Different question, different answer. state rm alone does not answer it, because the plan step after state rm still wants to create. Import blocks answer it, because they tell terraform 'this thing already exists, adopt it, do not create it'.

The Terraform 1.5 import block

Import blocks over state manipulation

Terraform 1.5 shipped import {} blocks in June 2023. Before that, the only tool was terraform import at the CLI, one resource at a time, with no config generation. You had to write the resource block by hand, run the import, then plan until it was empty. For 5 resources that is a slow afternoon. For 30, which is the size of the recovery we have done more often after a full-region failover, it is a week and a half.

The block form lives in HCL and terraform reads it during plan. Our module already declared all 5 resources (the only thing that had changed was the provider region), so there was nothing to generate: the plan matched each import block to the resource block already in the module and staged the adoption for the next apply. -generate-config-out is the companion flag for the other case, when you are adopting something you have no configuration for at all, and it writes a starter resource block for each import block that has no matching config.

# imports.tf (temporary, delete after one apply cycle)

import {
  to = module.data_layer.aws_rds_cluster.primary
  id = "app-primary"
}

import {
  to = module.data_layer.aws_db_proxy.primary
  id = "app-primary-proxy"
}

import {
  to = module.data_layer.aws_security_group.db_ingress[0]
  id = "sg-0a1b2c3d4e5f60718"
}

import {
  to = module.data_layer.aws_security_group.db_ingress[1]
  id = "sg-1a2b3c4d5e6f70819"
}

import {
  to = module.data_layer.aws_security_group.db_ingress[2]
  id = "sg-2b3c4d5e6f708192a"
}

One import block per resource; the id is the AWS resource ID, the to is the terraform module address

There is a state cleanup step first. The us-east-1 entries at those module addresses have to go, otherwise the import collides with an existing state entry and errors with 'resource already managed by Terraform'. We ran terraform state rm on the 5 stale entries. This is safe here because those state entries no longer match anything we intend to manage from this workspace (the us-east-1 resources are being handled by a separate decommission workspace). state rm is only dangerous when the plan step that follows tries to create things. The plan step that follows here only reads: import blocks are resolved during plan, and nothing touches AWS until apply.

$ terraform plan

Terraform will perform the following actions:

  # module.data_layer.aws_rds_cluster.primary will be imported
    resource "aws_rds_cluster" "primary" {
        cluster_identifier = "app-primary"
        engine             = "aurora-postgresql"
        engine_version     = "15.3"
        # ... 24 more attributes read from AWS
    }

  # module.data_layer.aws_db_proxy.primary will be imported
  # module.data_layer.aws_security_group.db_ingress[0] will be imported
  # module.data_layer.aws_security_group.db_ingress[1] will be imported
  # module.data_layer.aws_security_group.db_ingress[2] will be imported

Plan: 5 to import, 0 to add, 0 to change, 0 to destroy.

The successful plan after state rm + import blocks: 5 imports, 0 creates, 0 destroys

Read the summary line: 5 to import, 0 to add, 0 to change, 0 to destroy. That is the shape you want, and any add or destroy in it means an import block is pointing at the wrong address, or the module's attributes have drifted from what SRE actually built at 02:14. Fix that before you apply, not after. Commit imports.tf. Apply. The apply runs the 5 imports. After that, delete the import {} blocks. Once the address is in state they are no-ops on every subsequent plan, so leaving them in place is not an error, it is just noise that outlives the reason it was added.

How to align config to reality without overwriting the live resource

The generated HCL is 95% right and the last 5% matters

The generated config reflects what is running. That includes anything SRE added at 02:14 that was not in the pre-incident module. On this recovery, the next plan showed three drifts, all small, all load-bearing.

$ terraform plan

Terraform will perform the following actions:

  # module.data_layer.aws_rds_cluster.primary will be updated in-place
  ~ resource "aws_rds_cluster" "primary" {
      ~ backup_retention_period = 1 -> 7
      ~ tags = {
          - "incident-response" = "2024-11-14" -> null
          - "incident-owner"    = "sre-oncall"  -> null
        }
    }

  # module.data_layer.aws_db_proxy.primary will be updated in-place
  ~ resource "aws_db_proxy" "primary" {
      ~ debug_logging = true -> false
    }

Plan: 0 to add, 2 to change, 0 to destroy.

The next plan after import: the live state differs from the pre-incident module in three specific places

Two tags SRE had added at create time so the resources would be enumerable later. backup_retention_period at 1 day (the AWS default, because the incident timer was ticking and SRE clicked through the console fast) instead of the module's 7. debug_logging = true on the proxy, because SRE wanted verbose logs during the incident. Every one of these was a live state difference from what the module said the resource should look like.

For each drift item the decision is: keep the live state (add to config) or accept the plan (module wins)? On this recovery we kept the two tags for a followup and added them to the module's default tag map. We bumped backup_retention_period in the module to 7 to match the module's intent, accepting the plan on that one. We kept debug_logging=true for one more week and let the plan sit non-empty on that single field until the incident postmortem was done, then flipped it. The tempting move is to apply the whole plan at once and let the module reset everything. Do not do that without walking through each item. The point of the import was to bring the live resource under management, not to overwrite it with pre-incident config that no longer applies.

The other lesson from this recovery went into the incident runbook. When SRE creates resources by hand at 02:14, they now write the resource IDs into incident-resources.txt at the same time and commit it before the incident closes. That file becomes the source of truth for the import blocks the next morning, and cuts the recovery from 'grep CloudTrail for what got created between 02:00 and 03:00' to 'read the file, generate the imports'.

The 30-minute triage for a plan you cannot safely apply

If your terraform plan looks like this right now

The mechanical part of this recovery, the state rm and the import blocks, is a few hours of work. The hard part is the days before that, when the plan sits red and every engineer is afraid to touch it. Meanwhile someone else adds another out-of-band resource, and the next plan gets 12 lines longer, and the fear compounds. We have seen workspaces sit in this state for six months. The team stops running plan against them at all, which is its own kind of terrifying, because now nobody knows what actually differs from IaC.

We run these engagements every week. The failover-import case we have done three times this quarter alone, plus the 'someone tagged everything in the console during audit prep' variant, plus a half-dozen other shapes of out-of-band drift. Every one follows the same pattern: get the running resources into terraform without touching production, then decide what to reconcile and in what order. The reason it works is that the mechanical steps and the judgment steps are kept separate, so nobody applies the plan at 08:47 to make the red output go away.

If your workspace has a plan you cannot apply and no one is sure how it got there, book an infrastructure review and we will be on a bridge with you the same day, starting with a 30-minute diagnostic call to figure out whether the safe recovery is import blocks, moved blocks, a state file surgery, or a fresh workspace. For the class of problem this fits into, see the terraform apply fear pattern and the terraform state recovery playbook.

Originally published at https://infraforge.agency/insights/terraform-plan-destroy-live-failover-resources/.

If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.

How to recover pods a ConfigMap hook race left with empty env

Muhammad Hassaan Javed — Mon, 10 Aug 2026 03:29:46 +0000

If a Helm rollback of your service left a subset of pods in CrashLoopBackOff with empty database credentials while the rest keep serving, you are looking at a ConfigMap deletion race, not a bad rollback. The window opened during the upgrade that failed: a ConfigMap templated as a pre-upgrade hook with before-hook-creation is deleted and recreated on every upgrade, and any pod that restarted inside that gap booted with empty envFrom values and cached a broken connection pool. The rollback is when you noticed, not what caused it, and it does not clean up after itself. The fix is not kubectl rollout restart deployment. That will nuke the healthy pods too. You verify the current ConfigMap is correct, kill only the pods whose env is empty, then patch the chart so the race cannot happen again.

Problem signals:

kubectl get pods shows 3 of 12 replicas in CrashLoopBackOff with restart counts climbing (14, 17, 19) while the rest sit at 0 restarts
Application logs on the crashing pods show pq: password authentication failed for user "" or dial tcp: missing address
kubectl exec broken-pod -- printenv DATABASE_URL prints nothing and exits 1, while the same command on a healthy pod prints the DSN
helm history shows a recent rollback from revision N to N-1 within the last hour
kubectl get configmap -o yaml shows the correct DSN, so the state looks fine from the cluster's perspective and the alerting is confusing

The propagation window nobody documents

Why envFrom pods boot empty when the ConfigMap looks correct now

envFrom and env.valueFrom.configMapKeyRef resolve exactly once, at pod start. What happens when the ConfigMap is missing at that instant turns on one field: optional. With the default, optional: false, the kubelet refuses to start the container at all; the pod sits in CreateContainerConfigError with an Error: configmap not found event, which is loud and easy to diagnose. With optional: true, which plenty of charts set so that a missing config does not block a boot, the variable is simply absent, the client library reads it as an empty string, and the container starts and reports Running. That second case is the one this guide is about, and it is the only one that produces the split-brain symptom below. Nothing propagates later in either case. This is the part that confuses on-call: you look at the ConfigMap now, it is correct, so how can pods be running with empty values? Because those pods started at t=0 when the ConfigMap did not exist, and Kubernetes has no reconciliation loop that re-injects env into a running container.

The failed upgrade creates the window, not the rollback. If your ConfigMap carries a helm.sh/hook annotation (pre-install, pre-upgrade are the common ones) with a hook-delete-policy of before-hook-creation, Helm treats it as an ephemeral hook resource and deletes the existing copy before creating the new one on every upgrade. That delete-then-create is the gap. Helm fires hooks per lifecycle event, so a resource annotated pre-install,pre-upgrade is never touched by helm rollback, which runs pre-rollback and post-rollback only. Two consequences worth holding onto: the rollback did not open the window, and it does not close it either, so after rolling back to 47 the cluster is still holding revision 48's hook ConfigMap. On a healthy cluster with fast API server response, that window is 200 to 900 milliseconds. On a loaded one we have measured it at 4 seconds. Any pod that restarts inside it (crashloop backoff timer firing, HPA scale-up, node eviction) reads no ConfigMap and boots blank.

The 200ms to 4s window, opened by the upgrade's pre-upgrade hook, where a restart lands on a missing ConfigMap.

The tell that you are in this state and not something else: pods with identical spec, identical image, identical ConfigMap reference, have different runtime env. That does not happen from any cause other than the ConfigMap being absent at one pod's start time.

Confirm the rollback actually landed the correct config

Check the chart state before you touch a single pod

Do not restart anything yet. First confirm the ConfigMap now holds the correct value, otherwise you are about to restart pods into the same broken state. Run the three checks in order.

# 1. What revision are we on and did it complete?
$ helm history payments-api -n prod
REVISION  UPDATED                   STATUS      CHART              APP VERSION  DESCRIPTION
47        2024-11-03 14:22:11 UTC  superseded  payments-api-3.4.1 3.4.1        Upgrade complete
48        2024-11-03 14:41:07 UTC  failed      payments-api-3.5.0 3.5.0        Upgrade "payments-api" failed
49        2024-11-03 14:43:52 UTC  deployed    payments-api-3.4.1 3.4.1        Rollback to 47

# 2. Is the current ConfigMap the correct one?
$ kubectl get configmap payments-api-config -n prod -o jsonpath='{.data.DATABASE_URL}'
postgres://app:REDACTED@pg-primary.prod.svc:5432/payments?sslmode=require

# 3. Does its owner-reference or annotations still mark it as a hook?
$ kubectl get configmap payments-api-config -n prod -o yaml | grep -A2 'helm.sh/hook'
    helm.sh/hook: pre-install,pre-upgrade
    helm.sh/hook-delete-policy: before-hook-creation

If line 3 returns a hook annotation, the chart bug that caused this is still present and will fire again on the next upgrade. Note the annotation lists no rollback event, which is why revision 49 left this object exactly as revision 48 created it.

If the DSN value is correct, you can proceed to selective pod recovery. If it is wrong or empty, you are looking at the failed release's config, which the rollback never replaced because the object is a hook, and you need to fix the ConfigMap directly with kubectl apply before doing anything else. We keep the last-known-good ConfigMap under source control precisely for this five-minute window.

The surgical restart, not kubectl rollout restart

Restart only the pods that booted into the gap

kubectl rollout restart deployment/payments-api will restart every pod including the ones currently serving traffic correctly. If you are already down to partial capacity and Postgres has connection limits, doubling the churn is how you turn a partial outage into a full one. Identify the broken pods from their actual runtime env, or from the failure the process logged when it read that env, and delete only those.

# The pod spec cannot answer this. A variable sourced from a ConfigMap keeps
# its valueFrom stanza and never has a resolved .value written back into it,
# and envFrom keys never appear in the spec at all. Read the runtime env, or
# read what the process logged when it read the runtime env.

# Running-but-wrong pods: read the process environment directly.
$ kubectl exec -n prod payments-api-7d9c8f4b6-4kx2m -- printenv DATABASE_URL
# (prints nothing and exits 1: the variable is absent)

# CrashLoopBackOff pods cannot be exec'd into. The previous container's log
# carries the same signal, and this is the sweep that found ours.
$ for pod in $(kubectl get pods -n prod -l app=payments-api -o name); do
    if kubectl logs -n prod $pod --previous --tail=50 2>/dev/null | grep -q 'password authentication failed'; then
      echo "BROKEN: $pod"
    fi
  done
BROKEN: pod/payments-api-7d9c8f4b6-4kx2m
BROKEN: pod/payments-api-7d9c8f4b6-9jvpd
BROKEN: pod/payments-api-7d9c8f4b6-nq7wr

# Delete only those three. The Deployment will recreate them and they will
# read the current (correct) ConfigMap on start.
$ kubectl delete pod -n prod payments-api-7d9c8f4b6-4kx2m payments-api-7d9c8f4b6-9jvpd payments-api-7d9c8f4b6-nq7wr
pod "payments-api-7d9c8f4b6-4kx2m" deleted
pod "payments-api-7d9c8f4b6-9jvpd" deleted
pod "payments-api-7d9c8f4b6-nq7wr" deleted

Runtime env, or the log line the process wrote when it read that env. The pod spec is not ground truth here.

The trap worth naming, because it is the first thing most people reach for: a jsonpath filter over .spec.containers[0].env[?(@.name=="DATABASE_URL")].value looks like it should work and cannot. Kubernetes resolves ConfigMap-sourced variables in the kubelet at container start and never writes the result back onto the PodSpec, so that filter returns empty for every ConfigMap-sourced pod, healthy or broken. On a 12-replica Deployment it flags all 12. kubectl debug with an ephemeral container reading /proc/1/environ is the third way in, useful when the container image has no shell for exec.

Delete the broken pods in batches of one or two, not all at once, to protect the connection pool of the pods that are still healthy.
Watch the new pods reach Ready before deleting the next batch: kubectl get pods -n prod -l app=payments-api -w.
If your Deployment has maxUnavailable set aggressively, note that manual pod deletes bypass rollout controls, so you set the pace by hand.

Stop treating the ConfigMap as a Helm hook

The chart change that closes the window permanently

The root cause was that the ConfigMap was templated as a hook. There is almost never a reason to do this for a ConfigMap that carries runtime configuration. Hooks are for one-shot resources: pre-upgrade DB migrations, post-install seed jobs. A ConfigMap that pods depend on at start time should be a regular chart resource with a stable lifecycle across upgrades and rollbacks.

# templates/configmap.yaml
# BEFORE: this is the bug
apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ include "payments-api.fullname" . }}-config
  annotations:
    helm.sh/hook: pre-install,pre-upgrade
    helm.sh/hook-delete-policy: before-hook-creation
data:
  DATABASE_URL: {{ .Values.database.url | quote }}

# AFTER: regular resource, no hook annotations
apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ include "payments-api.fullname" . }}-config
data:
  DATABASE_URL: {{ .Values.database.url | quote }}

# templates/deployment.yaml: force a rolling restart when the ConfigMap changes
spec:
  template:
    metadata:
      annotations:
        checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}

Remove the hook annotations. Add a checksum annotation to the Deployment pod template so config changes always trigger a controlled rolling restart.

The checksum annotation is the piece most charts get wrong. Without it, changing values in the ConfigMap does not restart pods at all, because Kubernetes sees no change to the Deployment spec. Teams then work around this with a manual kubectl rollout restart after every values change, or a dummy annotation bumped by hand on each deploy, both of which are the same churn the checksum gives you for free and both of which get forgotten under pressure. With the checksum, any ConfigMap content change updates the pod template hash, which triggers a normal Deployment rollout that respects maxSurge and maxUnavailable. There is a tradeoff: you get more rollouts than before, one per config change, and if your ConfigMap has values that churn (feature flags, dynamic settings), you should move those out into a separate ConfigMap that pods read at runtime rather than at start.

We have written the broader pattern for boot-time versus runtime configuration in our Kubernetes stabilization notes, and if this is landing during or right after a migration, the same race appears with any credential source that gets templated as a hook, including some migration recovery playbooks we have run this year.

FAQ

Questions on-call keeps asking after this fires

Can I use kubectl rollout restart safely if the ConfigMap is now correct?

Yes, but only if you have the capacity to churn every pod and your database can absorb the reconnect storm. On a service with 12 replicas and a Postgres max_connections of 200 with pgbouncer in transaction mode, we have done full rollout restarts without incident. On a service that runs hot on connections, do the surgical delete instead.

Why does helm upgrade --atomic not prevent this?

--atomic rolls the release back when the upgrade fails, but the window has already opened by then. The pre-upgrade hook deletes and recreates the ConfigMap before any workload change is applied, so a pod can boot blank while the upgrade is still in flight. The rollback --atomic triggers then runs pre-rollback hooks only, which this ConfigMap is not annotated for, so it leaves the object untouched. --atomic is not a fix for this; removing the hook annotation is.

Does this affect Secret-sourced env the same way?

Yes, identically. envFrom on a Secret has the same start-time-only resolution. If you template Secrets as Helm hooks (some charts do, to inject generated passwords) you get the same race. Same fix: regular resource plus checksum annotation.

How do I know if the race actually fired versus something else broke my pods?

Two-pod comparison. Take one healthy pod and one crashing pod from the same ReplicaSet. If their runtime env for the same key differs (one has DATABASE_URL populated, the other has it empty), it is the ConfigMap race. If both have the same env and one still crashes, look elsewhere: image drift, node-local state, or a downstream dependency.

Can I use a mutating webhook or Reloader to auto-restart on ConfigMap changes instead of the checksum annotation?

Stakater Reloader works and we run it in several client clusters. The checksum annotation is simpler because it is part of the chart and needs no extra controller. If you already run Reloader for other reasons, use it. If you do not, do not install a controller to solve a one-line templating fix.

Getting the surgical recovery right the first time

If you are staring at CrashLoopBackOff right now

The hard part of this recovery is not the kubectl commands. It is deciding which pods to touch when your dashboards are red and the pressure is to do the biggest hammer available. A rollout restart in the middle of a partial outage on a service under connection pressure is how the two-hour incident becomes the six-hour incident. The judgment call is: verify config, inspect env pod-by-pod, delete only what is actually broken, and only then fix the chart so this cannot recur.

At InfraForge we do this recovery with the on-call team on the bridge, in one working session, and we leave the chart patch and the checksum annotation merged before we sign off. If you would rather not run the surgical restart on production by yourself, book a 60-minute infrastructure review and we will walk it with your team today or tomorrow, chart fix included.

Originally published at https://infraforge.agency/insights/helm-rollback-configmap-race-safe-recovery/.

If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.

Worker drops jobs intermittently: startup race, env drift, schema skew

Muhammad Hassaan Javed — Mon, 10 Aug 2026 03:15:00 +0000

If your worker is dropping jobs intermittently, sometimes crashing at boot with 'Error 111 connecting to redis:6379. Connection refused' and sometimes producing an empty result.json with no error at all, the cause is almost never a Redis performance problem. In most of the cases we get called into, it is a startup race between the worker and its dependency, hiding two quieter bugs behind it: an environment variable that silently falls back to localhost, and a job schema that got renamed on one side of the wire. This guide walks the three causes in the order they actually occur, gives you the one log grep that separates them, and shows the fix sequence that does not destroy the evidence you still need.

Problem signals:

Some runs crash at boot with redis.exceptions.ConnectionError: Error 111 connecting to redis:6379. Connection refused. and others start cleanly
result.json is written on some runs, missing on others, sometimes present but with 0 processed jobs and no error in the log
Restarting the worker alone (without the backend) makes the problem go away for a while, then it comes back after the next deploy
A grep for KeyError shows sporadic KeyError: 'job_id' traces that are being caught by a broad except block
The worker container's uid is 1000, /app/output/ is owned by root:root with mode 0755, and nobody remembers why

The symptom, and what it usually is not

Three failure modes wearing one costume

The reported symptom is almost always the same sentence: 'the worker is flaky, sometimes it processes jobs and sometimes it doesn't, and the logs don't really say why.' That sentence hides three separate bugs that compound. We rank them by the order we actually find them, not by how loud they are in the log:

1. Startup race. Worker container comes up before Redis (or the backend that populates Redis) is accepting connections. Retries are missing or set to a value that gives up in under 2 seconds. Roughly 60% of the incidents we see.
2. Environment variable drift. The backend sets REDIS_URL, the worker reads CACHE_URL, and the worker's client library silently defaults to redis://localhost:6379. It 'connects' to nothing and hangs or reads empty queues. Roughly 25%.
3. Schema skew after a rename. Someone renamed job_id to id on the producer side and missed one .get('job_id') on the consumer. .get returns None, the worker's outer except Exception swallows the downstream KeyError, the job is skipped, no error is logged. Roughly 15%, and the hardest to see.

The cause everyone blames first, and which is almost never it: 'Redis is slow' or 'the worker needs more memory'. We have not once found this to be the actual cause in this failure shape. If you are already sizing up the Redis instance, stop and read the next section first.

There is also a fourth cause that shows up as a hard PermissionError: [Errno 13] Permission denied: '/app/output/result.json' when the worker finally does try to write. It is real, but it is loud. This guide is about the quiet failures.

The discriminating check: read the logs before you touch anything

One grep that ranks the three

Before you restart, redeploy, or scale anything, capture the current state. Restarting the worker throws away the evidence that tells you which of the three you are dealing with. On docker compose, snapshot both services' logs to disk. On Kubernetes, capture kubectl logs --previous for the last crashed worker AND kubectl describe pod for its events. Do this first.

# capture before you touch anything
docker compose logs --no-color --timestamps worker  > /tmp/worker.log
docker compose logs --no-color --timestamps backend > /tmp/backend.log

# k8s equivalent
kubectl logs -n jobs worker-7c9d8f5b6-x2k4m --previous > /tmp/worker.log
kubectl describe pod -n jobs worker-7c9d8f5b6-x2k4m       > /tmp/worker-describe.txt

# then run the discriminating grep
grep -E 'Connection refused|CACHE_URL|localhost:6379|KeyError' /tmp/worker.log | head -40

Snapshot first, grep second. Anything that recreates the container (docker compose up --force-recreate worker, or a down followed by an up) discards the previous container's stdout under the default json-file driver. A plain restart keeps the log file, but it costs you the live process state that made the failure reproducible.

The grep output tells you which cause you have, in this order:

If you see Error 111 connecting to redis:6379. Connection refused in the first 5 seconds of the worker's log and never again after that, it is the startup race. The worker gave up before Redis was ready.
If you see Connecting to redis://localhost:6379 (note: localhost, not the service name), or you see NO connection log at all and the queue depth reads always return 0, it is env drift. The worker never got the right URL and its client defaulted.
If the connection logs are clean, the worker is clearly consuming messages, but the count of 'processed' log lines is lower than the count of 'received' lines, and you can find any KeyError (even one, even caught) in the traces, it is schema skew. The worker is silently dropping jobs whose payload does not match its expected shape.

Order matters. The race hides the drift, and the drift hides the schema skew. Fix them in the order they surface.

The safe fix sequence for each cause

Fix least-destructive first

Fix each cause with the smallest change that discriminates against the others. Do not batch the three fixes into one PR; you will not know which one worked, and the next incident will look identical. Ship them separately, verify each one, then move on.

Step	What it does
1. Startup race	Add a real readiness gate, not a sleep. On docker compose, use `depends_on` with `condition: service_healthy` and a healthcheck on the backend/Redis. On Kubernetes, use an initContainer that runs `nc -z redis 6379` in a loop with a bounded timeout (60s), and set the worker's client to retry with exponential backoff for at least 30s after boot. `sleep 10` in the entrypoint is the fix everyone reaches for; it papers over the problem until the day Redis takes 11 seconds.
2. Env var drift	Rename one side to match the other in a single PR. Do NOT add a fallback like `CACHE_URL or REDIS_URL`; that is how you got here. Then add a boot-time assertion: if the resolved URL is `localhost` or empty, the worker exits with a clear error instead of connecting to nothing. This is the single change that pays for itself the fastest.
3. Schema skew	Find every `.get('job_id')` and `.get('id')` across producer and consumer. Pick one name (we prefer `id` because it is what most queue libraries default to). Add a schema validation step at the consumer boundary that raises loudly on a missing key, and remove any `except Exception: pass` you find on the way. The broad except is what turned this into a silent bug.
4. Output permissions	If `PermissionError` on `/app/output/result.json` is in your logs, the fix is a Dockerfile line: `RUN mkdir -p /app/output && chown -R 1000:1000 /app/output` before the `USER 1000` directive. Do not `chmod 777`. If the directory is a mounted volume, set `fsGroup: 1000` in the pod's securityContext instead.

One thing to name explicitly: the tempting single-line fix of adding sleep 15 to the worker's entrypoint 'solves' the startup race in staging and hides all three bugs in production. We have watched this exact patch get merged, celebrated, and then paged the same team six weeks later when Redis restarted during a maintenance window and the sleep was not long enough. The cost of the right fix (a healthcheck plus retry) is roughly 40 lines of yaml and one afternoon. Pay it.

# docker-compose.yml, the readiness gate that actually works
services:
  redis:
    image: redis:7.2-alpine
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 2s
      timeout: 1s
      retries: 15

  backend:
    depends_on:
      redis:
        condition: service_healthy

  worker:
    depends_on:
      backend:
        condition: service_started
      redis:
        condition: service_healthy
    environment:
      REDIS_URL: redis://redis:6379/0    # one name, both sides
    # no sleep, no CACHE_URL fallback

The gate is condition: service_healthy plus a real healthcheck on the dependency. condition: service_started alone only waits for the container to exist, not to be ready.

Confirm the fix by running the stack from cold at least ten times in a row and checking that the processed count equals the received count on every run. If nine out of ten pass, you have not fixed it; you have improved it. Race conditions do not get 90% fixed. For deeper patterns on this kind of intermittent K8s failure, we have written up Kubernetes release failure recovery separately.

FAQ: variants readers usually ask right after

The questions that come next

Is it safe to just add restart: always and let the worker crash-loop until Redis is up? It works, but it makes your logs noisier, it costs you real seconds on every boot, and it hides the underlying dependency graph from anyone reading the compose file later. Use a healthcheck. Save restart: always for actual transient failures in steady state.

Does this apply to RabbitMQ, NATS, or Kafka workers too? Yes, the shape is identical. The verbatim error string changes (Connection refused on RabbitMQ, dial tcp: connection refused on NATS, NoBrokersAvailable on Kafka), but the three causes and the fix order are the same. The env var drift bug is especially common on Kafka clients because KAFKA_BROKERS vs BOOTSTRAP_SERVERS is a coin flip in the ecosystem.

Can I skip the healthcheck if I use Kubernetes with readiness probes? No, readiness probes gate traffic to a pod, not startup order between pods. You still need an initContainer or an application-level retry loop for the worker to wait on its dependency. Readiness alone will not save you.

Why not just fix the broad except Exception: pass and call it a day? Because removing it in isolation will surface the schema skew as a hard crash on production traffic, and if you have not fixed the startup race first, you will not be able to tell which crash is which. Fix in the order the causes appear at boot: connectivity, config, contract.

How do I stop this from recurring? Three things, in order of ROI: (1) a boot-time assertion in every service that fails fast if a required env var is missing or resolves to localhost, (2) a schema check at the consumer boundary that rejects malformed messages loudly instead of silently, (3) a pre-merge integration test that starts the stack from cold and asserts processed == received on 100 test jobs. That third one is the single highest-value test we recommend for job-processing systems.

If you are staring at an empty result.json and the clock is running

When the worker is dropping jobs right now

What makes this class of failure genuinely hard is not any single one of the three bugs. It is that they compound: the race gives you enough noise in the log that the drift looks like a symptom of the race, and by the time you fix the race, the drift has been silently corrupting queue state for hours, and the schema skew is dropping the recovery jobs you are firing to catch up. Untangling that under time pressure is where teams get stuck at 2 in the morning.

We have spent a lot of engineering hours in exactly this shape of incident, on Redis, RabbitMQ, and Kafka, across docker compose and Kubernetes. The pattern above is what we run on the call: capture logs first, grep to rank the causes, fix in dependency order, verify with cold-start runs. If you want a second set of eyes on it while it is happening, book a same-day infrastructure review and we will get on a bridge with your on-call engineer inside a few hours and work the sequence together. If the fires are already out and you want the pre-merge integration test and boot-time assertions in place before the next one, our platform reliability engagements cover exactly that.

Originally published at https://infraforge.agency/insights/worker-drops-jobs-intermittently-startup-race-env-drift/.

If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.

How a missing S3 gateway endpoint route quintupled our NAT bill

Muhammad Hassaan Javed — Thu, 16 Jul 2026 03:32:55 +0000

The finance lead asked why AWS charged us $2,100 for NAT gateway data processing last month. Our normal was around $400. Nothing in the release calendar explained it: no new services, no traffic bump in our own metrics, no scaling events. The bill just quintupled. Then someone opened VPC Flow Logs and filtered on the NAT's ENI. Roughly 71% of the bytes had destinations in 52.216.0.0/15 and 3.5.0.0/16, which is S3. We had an S3 gateway endpoint on the VPC. It was supposed to be handling that traffic on the AWS backbone for free.

Problem signals:

NAT gateway data-processing charges (NatGateway-Bytes) climb week over week with no code or workload change
VPC Flow Logs show heavy traffic to S3 IP ranges (52.216.0.0/15, 3.5.0.0/16) hitting the NAT's ENI instead of the gateway endpoint
aws ec2 describe-route-tables on a private subnet returns no route for the S3 managed prefix list (pl-xxx)
NAT gateway BytesOutToDestination stays high while VPC Flow Logs show traffic to the S3 prefix list still routing through NAT (the S3 gateway endpoint itself emits no CloudWatch metrics to check)
The S3 gateway endpoint exists in the VPC, but the bill still shows $0.045/GB on service-to-service S3 traffic

$412 to $2,103 on a flat workload, all of it NatGateway-Bytes

The bill line item that should not have existed

The NatGateway-Bytes cost for us-east-1 climbed from $412 in March to $2,103 in April. Nothing else on the bill moved. Egress was flat. EC2 was flat. RDS was flat. The whole delta was NAT data processing at $0.045 per GB.

Cost Explorer's usage-type breakdown confirmed the shape. The entire spike was NAT-Bytes on one VPC, in one region, over a six-week ramp. No cliff, no sudden jump, just a slow gradient upward that nobody watched because 'NAT costs a few hundred bucks' was our mental default.

So we mapped the destinations. AWS publishes the IP ranges for S3, DynamoDB, and every other service in ip-ranges.json. We pulled an hour of VPC Flow Logs, joined destination IPs against those ranges, and found the answer. S3 was 71% of the NAT-processed bytes. That should have been impossible. We had a gateway endpoint for S3, and gateway endpoints are free. Their entire point is to keep S3 traffic off NAT.

Cross-region buckets, then a hardcoded public endpoint, then the truth

What we thought first, and why the first two theories died fast

The first theory was that a new batch job was writing to a bucket in a different region. Cross-region S3 traffic does not hit the same-region gateway endpoint; it goes out through NAT. Reasonable theory, wrong theory. We grepped Terraform for cross-region bucket references and found nothing. We checked CloudTrail for recent bucket creations. Nothing new. All our S3 traffic was to buckets in the same region as the workload.

The second theory was that an application was talking to S3 via the public global endpoint URL instead of the regional one. That is a real failure mode: SDKs pointed at a hardcoded https://s3.amazonaws.com sometimes route to the wrong region and skip the gateway endpoint. Also wrong. Every service in the VPC used the SDK default resolver, which picks the regional endpoint.

The third theory turned out to be right. The route table for one of our private subnets was missing the S3 gateway endpoint's prefix-list association. Any pod scheduled onto a node in that subnet was reaching S3 through the NAT, at $0.045 per GB, for six weeks. Same workload the whole time. Different route table.

The route that has to exist, and the day the migration deleted it

How a gateway endpoint quietly stops covering a subnet

A gateway endpoint is not attached to your subnets the way an interface endpoint is. There is no ENI. There is no private DNS record. The gateway endpoint lives in the VPC, and to make it work for a subnet, you have to add its managed prefix list (pl-63a5400a for S3 in us-east-1) as a route in that subnet's route table, with the endpoint as the target.

The whole 'your S3 traffic bypasses NAT' behavior depends entirely on that route existing. If the route is not there, S3 traffic falls through to the default route (0.0.0.0/0), which points at the NAT gateway. There is no error, no warning, no CloudWatch alarm. The traffic just goes through NAT and gets billed per GB.

We checked all four of our private subnets' route tables for the S3 prefix-list route:

aws ec2 describe-route-tables \
  --filters "Name=association.subnet-id,Values=subnet-0abc123" \
  --query 'RouteTables[].Routes[?DestinationPrefixListId==`pl-63a5400a`]'

Repeat for each private subnet. Empty result means the S3 gateway endpoint is not covering that subnet.

Three subnets returned the S3 prefix-list route. One returned an empty array. That subnet's route table had been rewritten six weeks earlier during a network migration for another team's project. The migration rebuilt the route table from Terraform, but that Terraform module did not know about the gateway endpoint. The endpoint's association was managed by a separate stack. The rewrite silently dropped the S3 route entry.

Nothing failed. Nothing paged. The workload kept running. It just started paying NAT rates for every S3 GET and PUT the pods on those nodes made.

modify-vpc-endpoint, not create-route

The fix, and the command people reach for that does not apply here

The instinct might be to reach for aws ec2 create-route --route-table-id --vpc-endpoint-id to add the endpoint to the table by hand. That does not fit here. Interface endpoints do not use routes at all (they are ENIs with private DNS), and a gateway endpoint's prefix-list route is not added with create-route either. Gateway endpoints get added to a route table by modifying the endpoint itself and telling it which route tables to associate with.

The correct fix is one call:

aws ec2 modify-vpc-endpoint \
  --vpc-endpoint-id vpce-0abc12345 \
  --add-route-table-ids rtb-0def67890

AWS writes the prefix-list route into the route table atomically. No second command.

Verification took thirty seconds:

aws ec2 describe-route-tables --route-table-ids rtb-0def67890 \
  --query 'RouteTables[].Routes[?GatewayId==`vpce-0abc12345`]'

Should now return the pl-xxx prefix-list route with the endpoint as GatewayId.

The route was there. We then pulled a fresh five-minute slice of VPC Flow Logs and joined against S3's IP ranges again. S3 destinations were dropping out of the NAT sample. The subnet was routing them to the gateway endpoint instead. Over the next hour we watched the NAT gateway's BytesOutToDestination CloudWatch metric drop about 40% and level off.

That was the whole fix. Six weeks of overspend, roughly $1,700 in avoidable NAT charges, closed in about four minutes of actual work.

Endpoint-to-route-table coupling in Terraform, plus a rate-of-change NAT alarm

The two things we changed so this stops happening

We changed two things. First, every gateway endpoint in our Terraform is now paired with an explicit list of route-table IDs it associates with, and that list is generated from the same module that generates the private subnets. When someone adds a subnet, the endpoint's association is derived from the same variable. There is no second stack to remember.

Second, we set up a CloudWatch alarm on the NAT gateway's BytesOutToDestination metric with a per-VPC baseline. Not a fixed dollar threshold, a rate-of-change alarm: if NAT bytes for a VPC exceed three times the trailing seven-day median for two consecutive hours, we get paged. We would have caught this ramp on day four instead of week six.

We considered AWS Cost Anomaly Detection. It does catch this shape of spike, and in our archived bills it did fire, eleven days into the ramp. Our own CloudWatch alarm fires in hours because NAT bytes are a real-time metric and the cost bill is not.

For teams reading this with a similar VPC layout, the audit is roughly a ten-minute job. Enumerate every gateway endpoint, enumerate every route table that should be covered, and check for the prefix-list route in each. If you want the deeper cleanup pattern we use for accumulated cloud-cost drift, we have written more of that up in the InfraForge services overview.

This kind of drift does not fail loudly. It just bills.

If your NAT bill just went sideways and nobody deployed anything

The specific shape of this problem is one people miss because it does not fail loudly. Gateway endpoints have no failure mode that produces an error. They just silently stop covering a subnet, and the bill grows. If you have not audited your route tables against your gateway endpoints in the last six months, there is a decent chance one of your subnets is quietly paying NAT rates for S3 or DynamoDB traffic right now.

We have seen this pattern three times this quarter. Two were the same shape as ours: a route-table rewrite that dropped the prefix-list route. One was a subnet added later that never got the association at all. Each was a five-figure annual overrun that took an afternoon to identify and minutes to fix.

If your NAT gateway bill jumped and nobody deployed anything, book an infrastructure review with our team and we will start with a 30-minute diagnostic call this week. We will pull an hour of your Flow Logs, join destination IPs against AWS service ranges, and tell you which subnet is bleeding before the call ends.

Originally published at https://infraforge.agency/insights/nat-gateway-cost-spike-missing-vpc-endpoint-route/.

If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.

How one stuck PDB doubled our EKS autoscaler bill in six days

Muhammad Hassaan Javed — Thu, 16 Jul 2026 03:14:59 +0000

The AWS bill for EKS Compute went from $4,200 to $9,800 in six days. Same cluster, no launch, no traffic event, no team asking for headroom. kubectl get nodes returned 60 m5.4xlarge instances where the working set is normally 20, and forty of them were sitting under 8% CPU. The cluster autoscaler had scaled up overnight for a batch burst that finished by 6 am; it had never scaled back down.

Problem signals:

Your AWS bill for EKS Compute jumps without a matching traffic or deploy event
kubectl get nodes returns dozens more nodes than the working set, most at low CPU utilization
Cluster autoscaler logs repeat 'cannot delete: pdb blocking' and keep naming the same PDB
A single PodDisruptionBudget in the cluster shows ALLOWED DISRUPTIONS = 0 while others are 1 or higher
A workload somewhere in the cluster has a pod stuck in ImagePullBackOff that nobody is watching

The $9,800 bill and 60 nodes at 8% CPU

The Friday morning spike

The bill spike was six days old by the time the AWS Budgets alert fired. It tripped at $9,000 for the billing period; the same window the previous month was $4,200. Nothing had shipped that week, CI had been quiet since Wednesday afternoon, and no product team was asking for extra capacity.

Compute Optimizer had already flagged the cluster's managed node group as significantly overprovisioned. kubectl get nodes returned 60 m5.4xlarge nodes; the normal working set for this cluster is 20. Forty of the extras were sitting under 8% CPU, hosting kube-system DaemonSet pods and a single Deployment nobody had touched in months.

The math is straightforward. m5.4xlarge on-demand in us-east-1 is $0.768 an hour. Forty extra nodes for six days is about $4,400 of surprise burn, most of the jump from the $4,200 baseline. The overnight autoscale had happened on the previous Saturday to run a scheduled batch job; the batch finished by 6 am, and then the cluster autoscaler just stopped scaling down. Six days of that pattern is how a bill doubles.

Why HPA runaway and traffic burst were both wrong

Two theories that died fast

Two theories came up on the bridge in the first five minutes. The obvious one was HPA runaway: a Horizontal Pod Autoscaler misreading its metric and scaling a Deployment to hundreds of replicas, forcing the cluster autoscaler to hold capacity to place them. The second was a traffic burst: some overnight event pushing real request volume up, driving replicas up, then subsiding but leaving the nodes.

Both died fast. kubectl get hpa --all-namespaces showed every HPA sitting comfortably below its ceiling, none within striking distance of a scale trigger. Prometheus request-rate graphs for every ingress-fronted service were flat across the whole week. When we summed pod counts across the cluster, we got 342, roughly what a normal Friday looks like, and nowhere near the 900+ pods it would take to justify 60 m5.4xlarge nodes at typical density.

The demand side was fine. Something on the supply side was refusing to shrink.

The cluster-autoscaler log that pointed at exactly one PDB

One PDB, named in every log line

We went to the cluster autoscaler's own logs, which is where CA tells you plainly why it will not do the thing you want. kubectl -n kube-system logs deploy/cluster-autoscaler --tail=200 returned the same shape of line, forty times per scale-down cycle, each naming a specific node, all of them naming the same PDB.

I0313 09:32:04.881 static_autoscaler.go:428] Scale down status: lastScaleDownFailTime=2026-03-13 09:22:04
I0313 09:32:04.892 cluster.go:151] Fast evaluation: node ip-10-42-14-88.ec2.internal, cannot delete: pdb data-platform/nightly-rollup-pdb blocking eviction of pod data-platform/nightly-rollup-7f8b9c5d4-4kx2m (currentHealthy=39, desiredHealthy=39)
I0313 09:32:04.895 cluster.go:151] Fast evaluation: node ip-10-42-15-19.ec2.internal, cannot delete: pdb data-platform/nightly-rollup-pdb blocking eviction of pod data-platform/nightly-rollup-7f8b9c5d4-9h4tn (currentHealthy=39, desiredHealthy=39)
I0313 09:32:04.897 cluster.go:151] Fast evaluation: node ip-10-42-15-33.ec2.internal, cannot delete: pdb data-platform/nightly-rollup-pdb blocking eviction of pod data-platform/nightly-rollup-7f8b9c5d4-b7q9r (currentHealthy=39, desiredHealthy=39)
... (37 more lines, all identical shape, all naming data-platform/nightly-rollup-pdb) ...
I0313 09:32:05.114 scale_down.go:1052] Scale down considered 40 nodes, 0 removable

Every blocked node in every iteration named the same PDB. That was the whole answer in the first log grab.

We cross-checked against the cluster's full PDB inventory to confirm no other PDB was involved.

$ kubectl get poddisruptionbudgets --all-namespaces
NAMESPACE       NAME                     MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
api-gateway     api-gateway-pdb          2               N/A               3                     47d
auth            auth-service-pdb         1               N/A               2                     47d
data-platform   nightly-rollup-pdb       39              N/A               0                     213d
data-platform   warehouse-writer-pdb     1               N/A               2                     47d
payments        payments-api-pdb         2               N/A               1                     47d
search          search-indexer-pdb       1               N/A               2                     47d

Six PodDisruptionBudgets. Exactly one at zero allowed disruptions. Every other PDB had two or three to spare, which is the healthy state.

The outlier was data-platform/nightly-rollup-pdb, and it was 213 days old, while the other five had all been re-created seven weeks earlier as part of a cluster upgrade. Something about this specific PDB had been quietly wrong for a long time.

We looked at the underlying workload. The 'nightly-rollup' service started life as a CronJob and had been reimplemented at some point as a 40-replica Deployment with soft pod anti-affinity, running continuously to feed a downstream aggregation service. Its PDB carried minAvailable: 39, which is the standard 'allow one disruption at a time' setting for rolling updates. Fine when the workload is healthy.

kubectl get pods -n data-platform -l app=nightly-rollup told the rest of the story. Thirty-nine pods Running, one pod stuck in ImagePullBackOff. Six days earlier, someone had pushed a rolling update with a container image reference that pointed at an ECR registry we had migrated away from during a project the previous quarter. Every other workload had been re-pointed at the new registry during that migration; this one Deployment had been missed, and nobody had noticed because it 'just worked' for months.

The instant one pod dropped out of Ready, healthy went from 40 to 39 against a minAvailable floor of 39, and allowedDisruptions clamped to zero. From that moment on, no node hosting a nightly-rollup pod could be drained: evicting one would take healthy below the floor, and cluster autoscaler is careful about that. Forty pods, forty nodes, one stuck PDB. Each nightly-rollup pod requested about 400m CPU and 1.5 GiB of memory, which is a rounding error on an m5.4xlarge, but the pod's presence plus the PDB at zero meant the node was unremovable regardless of how much headroom the rest of the box had. Forty nodes at 5-8% CPU, all of them technically pinned by one broken image reference.

Why we deleted the PDB instead of pushing the fixed image

Delete the PDB, or fix the image

Two paths out. Fix the underlying pod: patch the Deployment's image reference, let the rolling update roll forward, healthy count returns to 40, allowedDisruptions goes positive, CA is free. Or delete the PDB directly: no PDB, no eviction block, CA can drain nodes.

We almost went with the image fix, because that is the 'correct' answer in an abstract sense; the PDB is doing what it was written to do. The problem was who owned the Deployment. The team that had originally built the nightly-rollup service had been reorganized eighteen months earlier, and the current owning team had inherited it in a spreadsheet handoff and had never shipped a change to it. To fix the image correctly we needed either to push the correct image to the old ECR registry (which required someone with write access on an account we were sunsetting) or to patch the Deployment to point at the new registry (which required knowing whether the image tag existed there and had feature-parity, which we did not, at 9:30 am).

The PDB, on the other hand, was clearly orphaned. Its author was gone. Its minAvailable: 39 was arithmetically fine for a 40-replica Deployment during healthy operation but it meant one broken pod would freeze the entire fleet, which is exactly what had happened. Deleting the PDB did not remove any replicas or affect the currently-running pods; it just removed the eviction guardrail.

kubectl delete poddisruptionbudget -n data-platform nightly-rollup-pdb
# poddisruptionbudget.policy "nightly-rollup-pdb" deleted

One command. The gate is the PDB; remove the gate.

Twelve minutes later, cluster autoscaler started removing nodes. It took about ninety minutes to work through the fleet. The nightly-rollup pods that got evicted rescheduled onto the remaining nodes without issue because the anti-affinity was preferredDuringSchedulingIgnoredDuringExecution (soft), which meant CA could consolidate the fleet onto fewer nodes when memory allowed. The pending pod stayed pending, because the underlying image reference was still wrong, but a single pending pod is a monitoring problem, not a bill problem.

By 11:30 am the cluster was back at 24 nodes. We paged the data team's on-call to fix the image reference at their leisure the following Monday. The bill for that Friday's Compute came in at $760, back inside the normal envelope.

The instinct in the first thirty minutes had been to kubectl delete node on the idle boxes. That would not have worked, and not for the reason you might expect. kubectl delete node only drops the Node object from the API server; the kubelet on that instance re-registers within seconds and the node comes back with the same name. The EC2 instance is never terminated and the orphaned PDB is untouched, so the spend does not move. Kicking nodes was never going to touch this. The PDB is the gate. Remove the gate or fix the gated pod; kicking the nodes directly is a way to spend money without changing anything.

The Kyverno policy and PDB alert that closed the gap

The rules we shipped the next week

The postmortem produced three changes we shipped inside the week.

The first was a cluster-level admission policy that rejects new PDBs unless they carry an owner annotation and an expires-at annotation. If a PDB is going to have the power to freeze half a cluster, someone needs to own it and someone needs to argue for renewing it. Kyverno was already installed for other reasons, so the rule was about thirty lines.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: pdb-lifecycle-required
spec:
  validationFailureAction: Enforce
  rules:
    - name: require-owner-and-expiry
      match:
        any:
          - resources:
              kinds:
                - PodDisruptionBudget
      validate:
        message: "PDBs must carry infraforge.io/owner and infraforge.io/expires-at annotations. See runbook: https://internal.infraforge.io/rb/pdb-lifecycle"
        pattern:
          metadata:
            annotations:
              infraforge.io/owner: "?*"
              infraforge.io/expires-at: "?*"

Kyverno ClusterPolicy blocking any PDB that arrives without an owner or an expiry.

The exemption path for genuine long-lived PDBs (data-plane primary databases, for example) is to set infraforge.io/expires-at to a far-future date and put the owning team alias in infraforge.io/owner. That does not prevent the specific failure we hit, but it does mean that in eighteen months when the next team reorganization happens, we can grep for the PDBs whose owners no longer exist before they become orphaned.

The second change was a Prometheus alert that fires when any PDB has allowedDisruptions == 0 for more than thirty minutes. Not five, not sixty. Thirty is long enough that a healthy rolling update has finished but short enough that a stuck one still gets caught inside the same business day.

The third was a workload audit that the source scenario also recommends. Over the following week we ran kubectl top pods on a rolling seven-day window against every Deployment with minReplicas > 1, and dropped the floor on five workloads whose real utilization was under 30%. Two of them we took from minReplicas: 3 to minReplicas: 1. That work did not fix the incident (the PDB was the actual bug), but it lowered the cluster's headroom baseline enough to knock another two nodes off the normal working set.

On the observability side we replaced the flat AWS Budgets threshold with anomaly detection at the EKS Compute line item, tuned to fire on a 30% week-over-week jump. The flat $9,000 threshold caught this incident six days late. The anomaly rule would have caught it inside twenty-four hours.

When your EKS bill doubles without a traffic event

When this is happening to you

Bill spikes without a matching traffic event almost always trace to a controller doing exactly what it was configured to do, in a way the config never anticipated. Cluster autoscaler is the most common source of this specific flavor, because its downscale logic is deliberately conservative around PodDisruptionBudgets; a single PDB with zero allowed disruptions can pin nodes indefinitely, and if the PDB is stuck on a pod nobody currently owns, nobody notices until the bill does. HPA misconfiguration, Karpenter provisioner drift, and stuck node-termination lifecycle hooks are the other three variants we see most often in the same shape.

We run recovery engagements with this exact shape most quarters. The fastest path to the answer is usually the cluster autoscaler logs, not a Prometheus dashboard, because CA will just tell you what it is refusing to do. If your EKS bill doubled inside a week and Compute Optimizer is flagging the node group as overprovisioned, we can be on a bridge with your platform team the same day. Book an infrastructure review and we will start with a 30-minute diagnostic call this week; the more artifact you can share ahead of the call (bill line item, kubectl get nodes output, and a kubectl -n kube-system logs deploy/cluster-autoscaler --tail=500 grab), the faster we can name the specific blocker.

If you are earlier in the shape (rising EKS bill, no obvious cause yet, no acute page), the Kubernetes and CI/CD stabilization work we do covers the workload-audit and admission-policy side of what this article described. We have written more on the general pattern of cost spikes driven by controller behavior in the cloud cost spikes problem write-up.

Originally published at https://infraforge.agency/insights/eks-autoscaler-stuck-pdb-cost-spike/.

If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.

When a hardening rollout breaks 8 layers and your own reconciler fights you

Muhammad Hassaan Javed — Mon, 13 Jul 2026 14:00:47 +0000

The first thing the on-call team tried was patching the status ConfigMap. Five apps showed Progressing in the platform's bleater-status object, the dashboard had been red for four hours, and somebody figured a kubectl patch on the status keys would at least quiet the pages while they investigated. The patch lasted about ten seconds. The internal reconciler running in the bleater-system namespace rewrote the ConfigMap on its next tick, every key back to Progressing, and the pages started again. That was the moment they called us. The hardening rollout that had run the night before had not broken one thing. It had broken eight.

Problem signals:

Apps stuck in Progressing or Degraded for hours after a hardening or security pass, with no single obvious cause in the events stream
Status ConfigMaps written by an in-cluster reconciler get rewritten within 10 to 15 seconds of any kubectl patch
kubectl delete job hangs on suspended PreSync Jobs because a hook-cleanup finalizer is still attached
Migration init containers crash-loop with pg_isready failing at name resolution while a default-deny egress policy is in place, then failing at a schema verification step once the egress allow rules land
kubectl patch on a RoleBinding fails with 'cannot change roleRef' because roleRef is immutable, so the binding has to be recreated

Why editing the status ConfigMap was the wrong instinct

The patch that survived ten seconds

The team had built a small in-house control plane the year before. A Python reconciler Pod in bleater-system watched the managed workloads, computed health from live cluster signals, and wrote a bleater-status ConfigMap every ten to fifteen seconds. Five apps reported there: an auth service, a profile service, a timeline service, a fanout service, and a primary application that handled the user-facing API. None of them used a full GitOps platform. The reconciler followed GitOps conventions, PreSync hook Jobs, sync windows, hook-cleanup finalizers, but it operated on raw Kubernetes primitives. ConfigMaps, Jobs, Roles. No CRDs.

That detail matters because when the on-call lead patched the status ConfigMap to mark the apps healthy, the reconciler was doing its job. It read the live cluster, saw the upstream signals were still bad, and rewrote the status. The patch was not wrong because patching ConfigMaps is wrong. It was wrong because the bleater-status object was not an input to the system. It was an output. Editing an output to fix a system is the same shape of mistake as editing a Prometheus metric to fix a service.

We have seen this pattern enough times to write it down as a rule. If a controller is rewriting your patches in under a minute, the object you are patching is derived state. Find the inputs. The reconciler source was eighty lines of Python and it took two minutes to read. The health predicate was an AND-chain across eight signals: lock state, orphan hook Job (the finalizer), schema version, RBAC capability, PVC bound, ResourceQuota headroom, NetworkPolicy egress, and init-container health. Any one of the first seven returning bad meant Progressing; a failing init container returned Degraded instead. All eight were bad.

# the AND-chain we found in the reconciler
def app_health(app):
    if lock_status() == 'locked':
        return 'Progressing'
    if orphan_hook_present(app):
        return 'Progressing'
    if schema_declared_version() < required_version():
        return 'Progressing'
    if not migration_rbac_capable():
        return 'Progressing'
    if not pvc_bound(app):
        return 'Progressing'
    if quota_exhausted():
        return 'Progressing'
    if not egress_allows_db(app):
        return 'Progressing'
    if init_container_failing(app):
        return 'Degraded'
    return 'Healthy'

The reconciler's health function. Eight independent signals, all gating. Every patch to the output ConfigMap was wasted work until every signal flipped.

What the inventory pass turned up in the bleater namespace

Eight failures wearing one hat

We started with the inventory, because the ticket told us almost nothing. A real P1 page rarely enumerates faults; it tells you what is on fire and gives you the namespace. We ran the kind of get-everything pass we always run on a strange namespace.

kubectl get pods,configmaps,jobs,deployments,roles,rolebindings,serviceaccounts,pvc,resourcequota,networkpolicy -n bleater
kubectl get events -n bleater --sort-by=.lastTimestamp | tail -40
kubectl describe pod -n bleater | grep -A5 'Init Containers\|Events:'

The first three commands we ran. The namespace had about sixteen pre-existing platform workloads from other teams sharing label values with the five managed apps.

What came back was a layered mess. A suspended PreSync Job named auth-presync-migrate-legacy7r2x with a hook-cleanup finalizer and no hook-delete-policy. A second suspended Job named fanout-presync-validate that looked identical but carried the hook-delete-policy annotation and a bleater.io/owner label pointing at platform-team. A hook-reconciliation-lock ConfigMap with status: locked and a stale lock-reason from the night of the rollout. The primary application's pod in Init:CrashLoopBackOff with kubectl logs --previous showing the init container failing at pg_isready itself, could not translate host name, because the egress deny was still swallowing DNS. A bleat-db-schema ConfigMap declaring version=2 with no tables-v3 key. A migration script that contained psql ... || exit 0 and had no set -e.

And then the governance layer, which is where the rollout had really gotten out of hand. A RoleBinding named migration-runner-binding pointed at migration-runner-role-v1, which had read-only verbs. A migration-runner-role-v2 existed alongside it, unbound, with create:jobs and patch:configmaps. A PersistentVolumeClaim named bleat-migration-pvc was Pending with an event saying storageclass.storage.k8s.io "fast-ssd-tier" not found, on a k3s cluster where the only storage class was local-path. A ResourceQuota set to pods: 1. A NetworkPolicy with egress: [] denying everything outbound including DNS.

Each one of those, taken alone, was a small fix. Taken together, they gated each other. The migration could not run because the RBAC was wrong. The repair Pods could not schedule because the quota was at one. The init container could not reach Postgres because the NetworkPolicy denied egress. The schema could not advance because the script swallowed errors. The reconciler refused to mark anything healthy until all of them resolved. The hardening rollout had tightened every knob at once and the knobs were not independent.

The dependency graph we drew on the bridge call. Cascade order falls out of the arrows.

The order of repair when faults gate each other

Why we raised the quota before anything else

The instinct on a multi-fault incident is to start with the most visible symptom. The CrashLoopBackOff is loud. The lock is loud. The orphan Job is loud. None of those were the right first move. The right first move was the boring one: raise the ResourceQuota, because every other fix needed to schedule a Pod. A ResourceQuota of pods: 1 caps the total number of non-terminal Pods in the namespace, not the number beyond what is already running, and about sixteen were already running. The quota was over-committed the moment the rollout landed it, so the API server admitted nothing new. We raised pods to 24, sized above the existing workloads plus the five managed apps and the repair Pods, and we changed nothing else on that object during the incident. Adding cpu and memory to the quota would not have been a bigger ceiling, it would have been a new admission requirement: those keys are aliases for requests.cpu and requests.memory, and once a namespace enforces a quota on either, every new Pod must specify requests or limits for that resource or the control plane may reject its admission. Mid-incident that rejects every repair Pod without resource requests, including the ad-hoc consumer Pod we scheduled later to bind the local-path PVC, with Failed quota ... must specify cpu,memory. Compute quota was still worth having, so it landed afterwards, behind a LimitRange carrying default requests and limits for the namespace, and only then did cpu: 8 and memory: 16Gi go onto the quota. We did not delete the quota.

Then the RBAC. We described both Roles and confirmed v2 had the verbs the migration Job needed. Patching the existing RoleBinding to swing roleRef to v2 returned the error we expected.

$ kubectl patch rolebinding migration-runner-binding -n bleater \
    --type='json' -p='[{"op":"replace","path":"/roleRef/name","value":"migration-runner-role-v2"}]'
The RoleBinding "migration-runner-binding" is invalid: roleRef: Invalid value: rbac.RoleRef{...}: cannot change roleRef

$ kubectl get rolebinding migration-runner-binding -n bleater -o yaml > /tmp/rb.yaml
# edit /tmp/rb.yaml, set roleRef.name to migration-runner-role-v2
$ kubectl delete rolebinding migration-runner-binding -n bleater
$ kubectl apply -f /tmp/rb.yaml

roleRef is immutable. The only path is delete-and-recreate, with the existing object as a template so you do not lose subjects.

PVC next. We listed storage classes, saw local-path was the only one, exported the existing PVC, changed storageClassName, deleted, reapplied. The PVC sat Pending for a few more seconds until we scheduled a consumer Pod against it, because local-path on k3s binds on first consumer. Then the NetworkPolicy. We did not delete it. The deny-by-default posture was the right posture; the rollout had just forgotten to allow anything. We added two explicit egress rules: same-namespace for the Postgres reach, and kube-system on UDP 53 for DNS. The deny-all stayed in place for everything else. The reconciler's metrics scrape needed no rule of its own: it is initiated from bleater-system toward the Pods in bleater, so relative to this policy it is ingress, not egress, and the policy only restricted egress. Once DNS resolved and the database was reachable, pg_isready inside the init container started returning ok, and the schema verification step became the remaining failure.

Then the lock and the orphan. The lock was a one-line patch to set status: unlocked and to replace lock-reason with resolved-2024-hardening-rollback. We left an audit value rather than blanking the field. The orphan Job hung on delete because of the finalizer. The strip-then-delete sequence is muscle memory at this point but it is worth showing because plenty of teams reach for --force first, which is the wrong tool.

# strip the finalizer first, then delete cleanly
kubectl patch job auth-presync-migrate-legacy7r2x -n bleater \
  --type=json -p='[{"op":"remove","path":"/metadata/finalizers"}]'
kubectl delete job auth-presync-migrate-legacy7r2x -n bleater

# do NOT touch fanout-presync-validate. it has hook-delete-policy set,
# carries bleater.io/owner=platform-team, and the reconciler manages it.
kubectl get job fanout-presync-validate -n bleater \
  -o jsonpath='{.metadata.annotations.argocd\.argoproj\.io/hook-delete-policy}'
# => HookSucceeded

Strip then delete. The decoy Job looks identical to the orphan from a distance; the discriminator is the hook-delete-policy annotation and the ownership label.

We have written more on cleaning up GitOps-style state safely in our Kubernetes and CI/CD stabilization playbook, including the finalizer-strip pattern and how to tell a managed Job from an orphaned one without guessing.

Don't weaken governance to silence alarms

The fixes that had to be repairs, not deletes

Halfway through the recovery the client's platform lead asked the obvious question. Why not just delete the ResourceQuota and the NetworkPolicy until things stabilize, then put them back? It would have shaved twenty minutes. We said no, and the reason is worth writing down, because it is the part of incident work that teams under pressure get wrong most often.

Governance controls exist for a reason. Someone put pods: 1 on that ResourceQuota originally because something had blown up the namespace before. Someone put the deny-all egress on because the auth service should not be able to call random external endpoints. The rollout had mangled the values, not the intent. Deleting the controls would have restored the workloads and silenced the alarms. It would have also removed two of the few real defenses that namespace had, with no scheduled work item to put them back. We have watched teams do this in March and find the controls still missing in November. The graveyard of post-incident TODOs is full of governance restore tickets that never got worked.

So we repaired. The quota went up to production limits in place. The NetworkPolicy got explicit allow rules added while the default-deny stayed. The PVC got a real storage class while the claim itself stayed at the same name and the same size. The orphan Job got deleted, because a stale suspended PreSync Job genuinely is garbage, but the cascade infrastructure stayed. Same controls, working values.

The migration script was the other repair-not-delete case. The version we found had this pattern:

# what we found
#!/bin/bash
psql -h $DB_HOST -U $DB_USER -d $DB_NAME -f /migrations/v3.sql || exit 0
echo "migration complete"

# what we replaced it with
#!/bin/bash
set -euo pipefail
psql -h $DB_HOST -U $DB_USER -d $DB_NAME \
  -v ON_ERROR_STOP=1 \
  -f /migrations/v3.sql
echo "migration complete"

|| exit 0 is the single worst line in any migration script. set -e and ON_ERROR_STOP=1 together mean a failing SQL statement actually fails the Job, which is what the reconciler was waiting to see.

After the script was patched, the migration Job ran successfully under the new RoleBinding, applied the v3 schema, and we read the tables back out of Postgres directly rather than trusting the script's exit code. The bleat-db-schema ConfigMap got tables-v3 written from observed pg_tables output. Not from the migration's stated intent. From the live database. If you ever find yourself writing schema declarations from anything other than what is actually in the database, you are setting up the next incident.

When in-house reconcilers and hardening rollouts collide

If your control plane is gaslighting your operators

The hard part of this kind of incident is not any single fault. The hard part is that an internal control plane is opinionated about state in ways that are not documented anywhere except in the reconciler's source code. When five apps are red and the dashboard says nothing changed, your team can spend an hour patching outputs that get reverted before they understand the inputs. Hardening rollouts make this worse, because they touch ResourceQuotas and NetworkPolicies and RBAC in the same change window, and the rollback path almost never accounts for the case where the controls themselves were the right idea but the values were wrong.

We run these recovery engagements every week. The in-house reconciler pattern shows up at almost every SaaS company past Series A that decided not to run ArgoCD or Flux directly. The shape of the failure is always the same: a small Python or Go service that watches a namespace and writes a status object, an operations team that does not own the reconciler code, and a control plane that fights every cosmetic fix because that is what it was built to do. We have seen the RoleBinding immutability case four times this quarter alone. The NetworkPolicy egress-without-DNS case shows up after every security audit cycle.

If you are watching a namespace where the status object keeps reverting your changes, or where a hardening pass cascaded across half a dozen layers and your team is debating whether to delete the controls to get back to green, book an infrastructure review with our team and we will be on a bridge call with you the same day. We will read your reconciler, draw the dependency graph for the cascade, and walk the repair order with your on-call. The goal is not to get the dashboard green by morning. The goal is to get it green without leaving a graveyard of governance restore tickets behind it.

Originally published at https://infraforge.agency/insights/internal-control-plane-cascade-recovery/.

If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.

Recovering a status page from a half-finished schema migration

Muhammad Hassaan Javed — Mon, 22 Jun 2026 22:19:29 +0000

The log line was 'database schema version 23 is dirty, refusing to start' and the pod exited immediately after printing it. The team had already tried a Helm rollback to the previous chart version. That pod did not get further: it printed 'database schema version 23 found, expected 21' against a binary that wanted 21. One binary refused because the schema was mid-migration, the other because the schema was now ahead of it. Both refused to start against the same database, and the company's only public status page had been down for 38 minutes. The migration job had been OOMKilled mid-run during the upgrade, and Postgres was now in a state neither binary recognized.

Problem signals:

Application logs 'schema version N found, expected M' and exits before serving traffic
Helm rollback to the previous chart version fails with the same or inverse schema error
A migration job pod shows exit code 137 or OOMKilled in kubectl describe
The schema_migrations (or equivalent) table reports a version the table DDL does not actually match
Restoring from the most recent Postgres backup would lose hours of production data the team needs to keep

Both the old and new binary refused to start against the same database

The log line that ruled out a rollback

The on-call had done the obvious thing first. The new chart was failing, so they ran helm rollback to the previous revision, revision 13, the last one before 0.91.2. The previous revision's pod came up, hit the database, and crashed too, for a different reason. The new binary expected schema 23 and found the version row marked dirty mid-migration. The old binary expected 21 and found the row already advanced to 23. One saw a migration that never finished, the other saw a database from the future. Both were sort of right.

That pattern is what told us the database was the problem, not the chart. A clean rollback should have produced a running pod. If both binaries reject the same database, the database is not in either of the states they expect. It is in a third state nobody coded for.

$ kubectl logs -n statuspage statuspage-app-7b9f-xq2vk   # 0.91.2 pod, pre-rollback
INFO  starting statuspage v0.91.2
INFO  connecting to postgres at postgres.statuspage.svc:5432
ERROR database schema version 23 is dirty, refusing to start
FATAL refusing to start with a dirty schema version

$ helm rollback statuspage 13 -n statuspage   # 13 is the last revision before 0.91.2
Rollback was a success! Happy Helming!

$ helm history statuspage -n statuspage
REVISION  UPDATED                   STATUS      CHART               APP VERSION  DESCRIPTION
13        Tue Jun 16 09:12:04 2026  superseded  statuspage-0.90.78  0.90.78      Upgrade complete
14        Fri Jun 19 14:31:47 2026  superseded  statuspage-0.91.2   0.91.2       Upgrade complete
15        Fri Jun 19 14:48:12 2026  deployed    statuspage-0.90.78  0.90.78      Rollback to 13

$ kubectl logs -n statuspage statuspage-app-6c4d-7m9pz   # rolled-back 0.90.78 pod
INFO  starting statuspage v0.90.78
ERROR database schema version 23 found, expected 21
FATAL refusing to start with schema version mismatch

Different errors from the two chart revisions, same root cause: the database, not the chart, was the problem.

The version row claimed 23. The table DDL was somewhere between 22 and 23.

What the schema_migrations table actually said

We dropped into psql against the application database and pulled the migration tracking table. The row said version 23, applied. The dirty flag was true, which on most migration libraries means 'a migration started running and never reported success'. That single boolean was the thread we pulled on for the next hour.

statuspage=> select * from schema_migrations;
 version | dirty
---------+-------
      23 | t
(1 row)

statuspage=> \d incidents
          Table "public.incidents"
   Column    |  Type   | Nullable | Default
-------------+---------+----------+---------
 id          | bigint  | not null |
 service_id  | bigint  | not null |
 started_at  | timestamp without time zone |          |
 resolved_at | timestamp without time zone |          |
 title       | text    |          |
-- expected per migration 0023: severity column, incident_updates FK, partial index on resolved_at IS NULL
-- present: none of the above

The version row was lying. The table structure was a partial 23.

We pulled the migration files out of the chart's image and read them. Migration 0023 was three statements: add a severity column, create an incident_updates table with a foreign key back, create a partial index on unresolved incidents. None of the three were present in the live schema. The OOM had hit after the migration library wrote the version row and before any DDL statement actually committed. Or possibly between statements. The order was implementation-specific and we did not care which exact moment because the answer was the same: none of 0023's DDL had landed, but the bookkeeping said it had.

This is the specific failure mode that makes partial migrations dangerous. The migration library and the actual schema disagree, and the application trusts the library. The library trusts a row it wrote in a different transaction than the DDL it was supposed to be tracking. Whether the version row and the DDL share a transaction is a per-tool design choice, not a version threshold you can date. The version/dirty pair in the table above is golang-migrate's, and golang-migrate deliberately writes version=N with dirty=true in its own transaction, runs the migration body separately, then writes dirty=false. The dirty column exists precisely because those two are not atomic, and that has not changed. Any tool that ships a dirty flag is telling you the same thing about itself. At the other end, tools that wrap the migration and its history row in one transaction on Postgres (Flyway, Rails, Django, Alembic) never leave the state described here at all. Check which one you are running before you trust the version row, because that is what decides whether this failure mode can reach you.

The base backup was 6 hours stale and uptime data is the product

Why we did not restore from backup

The instinct, and the safe move on most days, is to restore Postgres from the last known-good base backup and replay WAL up to a point just before the migration started. We checked the backup. It was a nightly pg_basebackup, 6 hours old, and WAL archiving had been configured but never tested for PITR. We could probably have done it. We were not willing to bet the status page on 'probably' while the status page was already down.

More importantly, the uptime check history is the product. A PITR replay to just before the migration would have cost only the few minutes since the migration started, but if the untested WAL replay failed we would be down to the base backup alone, and a status page that loses 6 hours of check data after an outage is worse than a status page that takes another hour to come back. We talked it through with the team lead and decided the database in front of us was recoverable, and recovering it was lower risk than the restore path. That decision is worth naming because it goes against the usual 'just restore from backup' instinct. When the data itself is the value, finishing a half-migration by hand is often the right call.

Step	What it does
Restore from base backup + WAL	PITR to a point just before the migration would lose only the few minutes since the migration started, but the PITR path was configured and never tested. If the replay failed, the fallback is the base backup alone, which loses up to 6 hours of uptime check history. Estimated 45-90 minutes if it worked the first try, much longer if not.
Finish migration 0023 by hand, fix version row	Three DDL statements, all idempotent-ish if we wrote them with IF NOT EXISTS guards. Preserves all data. Estimated 20 minutes including verification. Chose this.

Three DDL statements, a version-row correction, and a careful restart

Finishing the migration by hand

We pulled migration 0023 verbatim from the chart image, rewrote each statement with IF NOT EXISTS guards so a re-run could not double-apply, and ran them inside a single transaction so any failure left the database where we found it. Before touching anything we took a pg_dump of the application schema and data to a local file. That dump was our 'we can always undo this' insurance, separate from the production backup system.

-- 1. snapshot first, outside any transaction
$ kubectl exec -n statuspage postgres-0 -- \
    pg_dump -U statuspage -Fc statuspage > /tmp/statuspage-pre-repair.dump

-- 2. complete migration 0023 inside one transaction
BEGIN;

ALTER TABLE incidents
  ADD COLUMN IF NOT EXISTS severity smallint NOT NULL DEFAULT 3;

CREATE TABLE IF NOT EXISTS incident_updates (
  id           bigserial PRIMARY KEY,
  incident_id  bigint NOT NULL REFERENCES incidents(id) ON DELETE CASCADE,
  body         text NOT NULL,
  created_at   timestamp without time zone NOT NULL DEFAULT now()
);

CREATE INDEX IF NOT EXISTS incidents_unresolved_idx
  ON incidents (started_at)
  WHERE resolved_at IS NULL;

-- 3. clear the dirty flag, version row already says 23
UPDATE schema_migrations SET dirty = false WHERE version = 23;

-- 4. sanity-check before commit
SELECT version, dirty FROM schema_migrations;
\d incidents
\d incident_updates

COMMIT;

All four statements committed in 180ms: the three DDL statements plus the version-row correction. The read-only checks below them commit nothing. The dirty flag was the last thing to flip.

After the commit we rolled forward to the chart whose binary expects 23, because the deployment sitting at 0 replicas was still the rolled-back 0.90.78 one and it would have refused a clean schema 23 exactly the way it refused a dirty one: helm upgrade statuspage statuspage/statuspage --version 0.91.2 --set replicaCount=0 --no-hooks. The --no-hooks is what actually skips the chart's pre-upgrade migration job, since 0023 was now applied by hand; --set replicaCount=0 only sets the Deployment's replica count and would not have stopped the job on its own. Worth stating that --no-hooks suppresses every other hook in the chart too, so read what else is hooked before you reach for it. The --set is the other flag that matters. We had scaled the deployment to 0 earlier to stop the CrashLoopBackOff noise, but kubectl scale only writes spec.replicas on the live object; the upgrade renders the chart's Deployment manifest with replicas from replicaCount and the merge patch puts the live value straight back, so a plain helm upgrade would have rolled a pod during the upgrade itself, before anyone looked at anything. Carrying the 0 through the upgrade is what kept the gate. Only then did we admit traffic, kubectl scale deploy/statuspage-app --replicas=1, and watch the pod logs. It connected, read schema_migrations, found 23 clean, started serving. We curled the health endpoint, got a 200, then hit /api/services and confirmed all the configured uptime checks were present with their full history intact. Total time from 'database is the problem' to 'status page is back': 51 minutes.

The thing worth saying out loud about this kind of repair: it works because the migration was small and the failure was clean. If 0023 had been a data migration that rewrote a million rows halfway, the recovery would have looked very different and the backup-restore path would have won. Always read the failed migration before you decide which recovery to attempt. We have written more about that decision in the migration recovery playbook.

A schema snapshot before every migration, and migration jobs that cannot be OOMKilled

The pre-upgrade hook we shipped the next day

Two changes went in within 24 hours. First, the Helm chart now has a pre-upgrade hook that writes two dumps to an object storage bucket, tagged with the chart version it is about to migrate to: pg_dump --schema-only for the structure, plus pg_dump --data-only --table=schema_migrations for the version and dirty values. Both are needed, because --schema-only emits DDL only and dumps no table rows at all, so on its own it would capture everything except the version/dirty pair, which is the single field this recovery turned on. If a future migration goes sideways, the recovery starts from a known structure-level snapshot taken seconds before the migration began, not from the nightly backup. The hook adds about 4 seconds to every upgrade and has paid for itself once already.

Second, the migration job spec got real resource requests and limits, and the limit is now 2x the largest migration we have ever observed in staging plus a 50% buffer. The OOM happened because the chart shipped with a 256Mi limit and the migration that finally hit the production data size needed about 380Mi. We also removed the liveness probe from the migration job entirely. A migration that takes longer than expected should not be killed by Kubernetes; it should be allowed to either finish or fail on its own terms so the migration library can write a clean dirty=true and exit.

The shape of every upgrade now. The branch on the right is the one that saved us. The logical schema snapshot is a reference for what the structure should look like; only the physical base backup can take a WAL replay.

We have stopped recommending that teams skip the pre-migration schema snapshot just because their database is 'small enough to restore from nightly'. The nightly backup answers a different question than the schema snapshot. The nightly tells you what the data looked like 6 hours ago. The schema snapshot tells you what shape the database was in seconds before this specific migration began, which is the thing you need when a migration is the thing that broke.

Recovering a partial migration without losing the data behind it

When a status page is the thing that is down

The reason this kind of incident is hard is not the SQL. The SQL is usually three or four statements you can read off the migration file. The hard part is deciding whether to finish by hand or restore from backup, and that decision depends on details most teams have not catalogued: how their migration library handles the version row, whether PITR has ever been tested, whether the failed migration is structural or data-rewriting, and whether the data between the last backup and now is recoverable some other way.

We run these recovery engagements regularly. We have seen the OOMKilled-migration-job pattern four times in the last year, two of them on status pages or monitoring tools where the data IS the product, and we have a checklist for the decision now. If you are staring at a CrashLoopBackOff with a schema version mismatch in the logs and a Helm rollback that did not help, book an infrastructure review and we will be on a bridge with you the same day to work through the finish-by-hand versus restore decision before you commit to either.

Originally published at https://infraforge.agency/insights/recovering-status-page-half-finished-schema-migration/.

If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.