Karpenter launched 2,800 nodes over one weekend and consolidated only 180 of them, and our EC2 on-demand bill jumped 2.2x. The cause was one line added to a shared internal Helm chart on Friday afternoon: karpenter.sh/do-not-disrupt: true, applied to every pod template the chart rendered. Renovate auto-merged the bump across 34 tenant repos over the weekend, and by Monday about 1,450 pods carried the annotation. Karpenter's consolidation loop disqualifies any node holding at least one annotated pod, so roughly 85% of nodes became unconsolidatable while new pods kept arriving. Here is how we spotted it and rolled it back without triggering a fleet-wide restart of every workload the chart owned.
Problem signals:
- Karpenter's launched instance events run 5-10x baseline while disrupting via consolidation events sit at half of normal
- EC2 on-demand line items on the two largest instance sizes double while spot line items stay flat
- Node count climbs steadily through a quiet weekend with no matching increase in pod count or RPS
- kubectl finds karpenter.sh/do-not-disrupt: true on a majority of pods even though only a handful of workloads legitimately need it
- On-demand share of total node hours drifts from ~30% to >60% even though the NodePool weights spot at 100
The metric that redirected us away from HPAs
The launch rate was 7x normal; the consolidation rate was half
Our first read was wrong. The Monday FinOps digest fired at 08:15 UTC showing EC2 on-demand up 2.2x for the trailing 72 hours, and the platform lead's instinct was 'someone shipped an HPA that went sideways over the weekend.' We checked requests per second on the top 20 services; nothing moved more than 8% week over week. Total pod count was up about 40 pods against a fleet of 2,400. Not an HPA problem. Something was provisioning nodes without a matching demand signal, and the extra capacity was mostly on-demand even though the NodePool weighted spot at 100.
The evidence that pointed us at the right layer came from Karpenter's own controller logs. We asked it to summarise its own actions over the incident window:
kubectl -n karpenter logs deployment/karpenter --since=72h \
| grep -E 'launched instance|disrupting via consolidation' \
| awk '{print $6}' | sort | uniq -c
2812 launched
178 disrupting
# Baseline weekend, same window a fortnight prior:
412 launched
347 disrupting
Karpenter was doing both halves of its job. It was just doing one seven times too often and the other half as often as it should.
The provisioning path was healthy. There were no failed provisioning or unschedulable events. Karpenter was launching nodes about seven times as often as a normal weekend and consolidating them at half the usual rate, so capacity was arriving faster than it was leaving. Over 60 hours that gap accumulated into 98 extra nodes, most of them on-demand, sitting quietly and costing money.
One line in a shared chart, 34 tenants downstream
The chart shipped Friday at 15:04 UTC and Renovate did the rest
Once the question was 'why isn't consolidation firing', the answer came out fast. Karpenter's consolidation logic evaluates candidate nodes by checking whether every pod on the node can be safely rescheduled. Any node holding a pod with karpenter.sh/do-not-disrupt: true is immediately disqualified from consolidation until that pod is gone. The annotation exists for legitimate reasons: Kafka Streams state, long-running batches with expensive warmup, workloads that lose data on SIGTERM. In our cluster, four workloads owned it and were the only ones that should have had it.
The reality was different:
kubectl get pods --all-namespaces -o json \
| jq -r '[.items[] | select(.metadata.annotations["karpenter.sh/do-not-disrupt"] == "true")] | length'
1450
kubectl get pods --all-namespaces -o json \
| jq -r '[.items[] | select(.metadata.annotations["karpenter.sh/do-not-disrupt"] == "true") | .metadata.namespace] | unique | length'
38
1,450 pods across 38 namespaces carried the annotation. With pods distributed by the default scheduler, that pinned roughly 85% of nodes.
Our first suspicion was a mutating admission webhook. kubectl get mutatingwebhookconfigurations returned nothing new. The annotation was in the pod specs at the source. We picked one pod, walked back to its ReplicaSet, saw the annotation in the pod template, and ran git blame on the tenant's Deployment manifest. The blame line pointed at an internal chart bump: internal-charts/base-deployment moved from 2.7.4 to 2.7.5 on Friday at 19:47 UTC. The diff was three lines:
+ annotations:
+ karpenter.sh/do-not-disrupt: "true"
Commit message: 'add do-not-disrupt to prevent midday restarts during batch runs.'
The chart owner had one tenant whose Kafka Streams StatefulSet was losing ~90 seconds of state on every consolidation event. The narrow fix was to annotate that one StatefulSet, which required coordinating with the tenant. The wide fix was to put the annotation in the shared chart, which required coordinating with no one. They picked the wide fix. Our Renovate config had trust minor bumps from internal charts on auto-merge. Thirty-four tenant repos consumed base-deployment. Over the weekend, thirty-four PRs opened, thirty-four PRs merged, thirty-four ArgoCD syncs rolled deployments, and every rolled pod inherited the annotation.
A single opinionated chart plus a permissive auto-merge is the shape of the whole incident. Nothing in the loop was malicious. Every step was doing what it was configured to do.
The right recovery is on the pods, not the templates
The patch we almost ran would have rolled 1,450 pods
By 09:00 UTC Monday we had the diagnosis and a bad plan. The obvious move was to patch the Deployment template in each of the 34 tenants, removing the annotation at the source:
kubectl patch deployment <name> -n <ns> \
-p '{"spec":{"template":{"metadata":{"annotations":{"karpenter.sh/do-not-disrupt":null}}}}}'
The command that looked surgical and was not.
We almost ran the loop. Then one of us asked the question that matters: does changing a template annotation trigger a rollout? It does. The Deployment controller's ComputeHash runs DeepHashObject over the entire PodTemplateSpec, and metadata.annotations is part of the template. Change any pod-template annotation and the template hash changes, which creates a new ReplicaSet, which rolls every pod. The cleanest proof of this is that kubectl rollout restart triggers a rollout precisely by stamping kubectl.kubernetes.io/restartedAt into spec.template.metadata.annotations. If template annotations were excluded from the hash, rollout restart could not work.
If we had run the patch loop across 34 tenants at 09:00 UTC on a Monday, we would have rolled about 1,450 pods in flight, cascaded PDB blocks into deploy pipelines, and (worst) triggered even more Karpenter provisioning as the rollouts churned. The recovery would have looked identical to a second, larger incident.
The correct move was to strip the annotation from the running pods, not from the template. A ReplicaSet reconciles pod count and pod ownership, not annotation drift on pods that already exist. So mutating a live pod's annotations is safe and does not trigger replacement. Karpenter re-checks consolidation eligibility on its next loop (roughly every 30 seconds) and starts disrupting the newly-eligible nodes on its own.
Here is what we ran, iterating one namespace at a time so we could abort if anything looked off:
for ns in $(kubectl get ns -o jsonpath='{.items[*].metadata.name}'); do
case "$ns" in kube-*|karpenter) continue ;; esac
kubectl annotate pods -n "$ns" --all \
karpenter.sh/do-not-disrupt- --overwrite 2>/dev/null || true
done
The trailing dash on the annotation key removes it. Total wall time across ~2,440 pods was 47 seconds.
Verification was two commands, one for the pod count and one for Karpenter's response:
kubectl get pods --all-namespaces -o json \
| jq -r '[.items[] | select(.metadata.annotations["karpenter.sh/do-not-disrupt"] == "true")] | length'
24
kubectl -n karpenter logs deployment/karpenter --tail=200 \
| grep 'disrupting via consolidation' | tail -5
...disrupting via consolidation, 3 candidates
...disrupting via consolidation, 5 candidates
...disrupting via consolidation, 2 candidates
...disrupting via consolidation, 4 candidates
...disrupting via consolidation, 3 candidates
Down to 24 annotated pods (the four legitimate workloads plus a 20-pod fraud-scoring batch six hours into its run) and Karpenter firing consolidation events within four minutes.
Over the next 90 minutes node count dropped from 198 to 141 and plateaued. The remaining 24 pods were pinning about 22 nodes: 20 fraud-batch pods spread across 18 nodes, and 4 legitimate pods on 4 more. Consolidation could not touch those. We coordinated a restart of the fraud batch with its owning team. It checkpointed cleanly and restarted onto 5 nodes, tight-packed by Karpenter's provisioning-time bin packing, which freed 13 nodes and took us to 128. Over the next 90 minutes natural pod churn shifted another 20 pods off nodes that had held only transient workloads, and Karpenter consolidated those too. Node count landed at 108 by 15:00 UTC, back inside baseline range.
Then we rolled out the fixed chart. base-deployment 2.7.6 removed the blanket annotation and gated it behind an explicit values.yaml opt-in (karpenter.doNotDisrupt: false default, true for the four legitimate tenants). This rollout DID replace pods, because it changed the template hash, and we scheduled it deliberately. Renovate opened PRs but no longer auto-merged them. The platform team merged them in batches of five with a ten-minute soak between batches so any interaction with PDBs or startup probes surfaced before the next wave. Six hours end to end, no SLO impact.
The changes that landed in the two weeks after
Three controls that would have caught this on Saturday
The postmortem produced three durable controls. Each one closes a specific step in the causal chain above.
The first is a Kyverno ClusterPolicy that only permits the annotation on pods that explicitly opt in via a label. It ran in Audit mode for two weeks, we watched the drift metric go to zero, then we flipped it to Enforce:
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: restrict-karpenter-do-not-disrupt
spec:
validationFailureAction: Enforce
rules:
- name: require-opt-in-label
match:
any:
- resources:
kinds: [Pod]
preconditions:
all:
- key: "{{ request.object.metadata.annotations.\"karpenter.sh/do-not-disrupt\" || '' }}"
operator: Equals
value: "true"
validate:
message: "karpenter.sh/do-not-disrupt requires label karpenter.platform/opt-in=true"
pattern:
metadata:
labels:
karpenter.platform/opt-in: "true"
A future chart change that adds the annotation without the matching label is rejected at admission and never lands.
The second is an SLO alert on Karpenter's own consolidation metric. karpenter_disruption_actions_performed_total (the metric renamed from karpenter_deprovisioning_actions_performed_total around v0.32; the semantics are the same) was already in our Prometheus scrape config; nobody had put it on a dashboard. We added a Grafana panel showing consolidation actions per hour filtered to action="consolidation", and paged if the rate stayed below 20% of the 7-day rolling median for more than two hours during business hours. Backtesting against the incident, that alert would have fired around 04:00 UTC Saturday, roughly 8 hours in, instead of 60 hours in.
The third is a check in the shared chart's CI pipeline. Any minor bump that touches a scheduling-related annotation key (karpenter.sh/*, karpenter.k8s.aws/*, scheduler.alpha.kubernetes.io/*, node.kubernetes.io/*, cluster-autoscaler.kubernetes.io/*) sets a platform-review-required flag on the PR, and Renovate is configured to leave those PRs open rather than auto-merge them. Non-scheduling changes still flow through the previous auto-merge path unchanged. The tradeoff we now pay is about one platform-team review per quarter on this chart; the incident that check would have caught cost us roughly $3,500 in on-demand overage across the 60-hour window.
We considered tightening the NodePool's disruption.budgets and did not. The root cause was not consolidation being too eager; consolidation was fine when it was allowed to fire. Tightening budgets would slow legitimate consolidation without preventing another annotation-blast. We put a comment in the NodePool YAML pointing to the postmortem so a future engineer does not try to 'fix' the budgets in response to reading it.
FAQ: Karpenter consolidation and the do-not-disrupt annotation
Common questions we get about this pattern
Does karpenter.sh/do-not-disrupt pin just the pod or the whole node? It effectively pins the node. Consolidation eligibility is evaluated per-node, and any node holding at least one annotated pod is disqualified until that pod is gone. One annotated pod on a 32-vCPU node prevents the node from ever consolidating.
Would tightening PodDisruptionBudgets have prevented this? No. PDBs govern voluntary disruption of pods that Karpenter has already decided to touch. The annotation blocks node candidacy upstream of that check, so the eviction call PDBs guard never happens. PDBs sit downstream of the gate that closed here.
Can we detect this via CloudWatch or Cost Explorer alone? Not fast enough. Cost signals arrive at daily granularity at the earliest, and our FinOps digest ran weekly. The signal you want is Karpenter's own consolidation-actions rate from /metrics, which moves within minutes of the problem.
Does this pattern apply to Karpenter v1? The mechanism is the same. The annotation moved under different API groups across versions, and v1 uses karpenter.sh/do-not-disrupt on pods with identical semantics. The Deployment controller's PodTemplateSpec hash behavior is a Kubernetes property, not a Karpenter property, and it does not change.
What if we want every pod in a Deployment to be do-not-disrupt by design? Put the annotation in the template and accept that changing it triggers a rollout. Use maxSurge and PDBs to control that rollout the way you would for any other template change. The mistake in our incident was not that a template had the annotation; it was that 34 templates got it without the owning teams knowing.
Where Karpenter regressions get stuck
If your on-demand line jumped and no one shipped anything
The awkward thing about this class of Karpenter regression is that every piece looks correct in isolation. The annotation was a real feature intended for real workloads. The chart bump was a legitimate response to a real production pain. The Renovate auto-merge policy was tuned to reduce toil on well-behaved chart bumps. The NodePool was configured the way the docs recommend. It took the combination, plus a weekend, plus a weekly cost digest, to produce a doubled node count and a mid-four-figure overage. The diagnostic move that mattered was comparing Karpenter's launch rate against its consolidation rate. The recovery move that mattered was knowing which kubectl mutations trigger a rollout and which do not.
We have written more on this shape of failure in the Kubernetes and CI/CD stabilization pillar, and the specific pattern of a shared chart change cascading through GitOps sits in the ArgoCD and GitOps recovery cluster. If your Karpenter cluster is growing and you cannot see why, book an infrastructure review and we will pull the consolidation-actions metric together, walk your controller logs the same way we walked ours, and get you a diagnosis inside a single working day.
Originally published at https://infraforge.agency/insights/karpenter-consolidation-broken-by-do-not-disrupt-annotation/.
If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.
Top comments (0)