kubectl drain stuck? Why drains hang on PodDisruptionBudgets, and how to check before you start

You cordon a node, run kubectl drain, and the terminal prints the same line every five seconds:

error when evicting pods/"api-7d9c6b8f5-x2k4q" -n payments (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.

Nothing is broken. Drain is doing what it was told: asking the API server to evict pods one at a time and waiting. A drain hangs for a small number of reasons, and almost all of them can be read from the cluster before you start. This post goes through them with commands you can run against any cluster.

What drain actually does

kubectl drain <node> first cordons the node (marks it unschedulable), then evicts each pod through the Eviction API. Eviction is not delete. The API server checks the pod's PodDisruptionBudget (PDB) and refuses with HTTP 429 if the eviction would take the workload below its budget. kubectl keeps retrying, which is the "hang".

Drain also refuses to start on some pods, and it can succeed on eviction while the pod then has nowhere to run. Those are three different failure classes:

1. Eviction is refused (PDB).
2. Drain refuses the pod up front (unmanaged pods, local storage, DaemonSets).
3. Eviction works, but the replacement pod stays Pending.

1. PDB math

List every budget and look at one column:

kubectl get pdb -A

NAMESPACE   NAME   MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
payments    api    1               N/A               0                     41d
payments    web    N/A             1                 1                     41d

ALLOWED DISRUPTIONS is the number of pods the Eviction API will let go right now. If it is 0 for a workload that has a pod on your node, that pod cannot be evicted and the drain waits.

To get just the blocking budgets:

kubectl get pdb -A -o json | jq -r '
  .items[] | select(.status.disruptionsAllowed == 0)
  | "\(.metadata.namespace)/\(.metadata.name)  healthy=\(.status.currentHealthy)/\(.status.desiredHealthy) expected=\(.status.expectedPods)"'

minAvailable vs maxUnavailable

Both express the same idea from opposite sides.

• minAvailable: N means at least N pods must stay healthy. Allowed disruptions = healthy pods minus N.
• maxUnavailable: N means at most N pods may be unavailable. Allowed disruptions = N minus pods already unavailable.

Percentages are rounded up. With 10 replicas, minAvailable: 80% allows 2 disruptions; maxUnavailable: 25% allows 3 (2.5 rounds up).

The two behave differently when replicas change. minAvailable: 3 on a 3-replica Deployment allows zero disruptions forever; maxUnavailable: 1 keeps allowing one disruption at any size. If you want a budget that scales with the workload, prefer maxUnavailable.

A budget also counts only healthy pods. If one of three replicas is already crash-looping and the budget is minAvailable: 2, allowed disruptions is 0 even though no one touched anything. A PDB can set unhealthyPodEvictionPolicy: AlwaysAllow (beta since 1.27, GA in 1.31) so that pods which are not ready can be evicted regardless of the budget; check your cluster version before relying on it.

The single-replica Deployment with a PDB

This is the most common self-inflicted drain block:

apiVersion: apps/v1
kind: Deployment
metadata: { name: report-worker, namespace: payments }
spec:
  replicas: 1
  selector: { matchLabels: { app: report-worker } }
  template:
    metadata: { labels: { app: report-worker } }
    spec:
      containers: [{ name: app, image: registry.example/report-worker:1.4 }]
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: report-worker, namespace: payments }
spec:
  minAvailable: 1
  selector: { matchLabels: { app: report-worker } }

One replica, one required. Allowed disruptions is 0, always. maxUnavailable: 0 has the same effect. Every drain that touches the node hosting this pod will hang until someone deletes the budget or scales up.

The honest options are: run at least two replicas, set maxUnavailable: 1 and accept a brief gap, or drop the PDB for a workload that cannot meet it anyway. A PDB that can never be satisfied protects nothing and blocks every node upgrade.

Also check the selector. A PDB whose selector matches no pods shows expectedPods: 0, and a selector that matches pods from several controllers produces unspecified results; the API documentation says to keep one PDB per workload.

2. Pods that drain refuses to evict

Before any eviction, kubectl drain checks each pod and aborts with a message if it finds one of these. These are flags you have to choose deliberately.

Pods without a controller

A bare pod (no ReplicaSet, StatefulSet, Job or similar owner) will not be recreated anywhere. Drain refuses unless you pass --force, and then the pod is simply gone.

Find them:

kubectl get pods -A -o json | jq -r '
  .items[] | select((.metadata.ownerReferences // []) | length == 0)
  | "\(.spec.nodeName)\t\(.metadata.namespace)/\(.metadata.name)"'

Static (mirror) pods show up here too; those are owned by the node and drain ignores them. Anything else on the list is a pod you need to decide about: recreate it as a Deployment, or accept losing it.

emptyDir and local storage

Pods with an emptyDir volume lose that data on eviction. Drain refuses unless you pass --delete-emptydir-data (older versions called it --delete-local-data). That flag is a decision, not a fix. If the emptyDir holds a cache, fine. If it holds the only copy of something, it is not.

kubectl get pods -A -o json | jq -r '
  .items[] | select(any(.spec.volumes[]?; has("emptyDir")))
  | "\(.spec.nodeName)\t\(.metadata.namespace)/\(.metadata.name)"'

A different local-storage trap: a PersistentVolume with node affinity (local PVs, or zonal disks) pins the pod to a node or zone. The pod can be evicted, but the replacement will not schedule anywhere else. That belongs in section 3.

DaemonSets

DaemonSet pods are recreated on the same node immediately, so evicting them is pointless. Drain refuses until you pass --ignore-daemonsets, which skips them. This is the one flag nearly everyone passes:

kubectl drain prod-eu-1-node-a --ignore-daemonsets --delete-emptydir-data --timeout=10m

Put --timeout on every drain you script. Without it, a PDB block waits forever.

3. Pods that cannot reschedule

Eviction succeeds, the drain moves on, and the replacement pod sits in Pending. For a single-replica workload that is an outage; for a multi-replica one it is lost capacity. Check this before you drain, because the scheduler will not tell you until it is too late.

Look at what is already pending:

kubectl get pods -A --field-selector=status.phase=Pending
kubectl describe pod -n payments <pod> | sed -n '/Events:/,$p'

The event reads like 0/6 nodes are available: 2 node(s) didn't match Pod's node affinity/selector, 3 Insufficient cpu, 1 node(s) had untolerated taint. Each clause is a reason, and you can test for each one ahead of time.

Node selectors and affinity. List what the pods on your node require:

kubectl get pods -A --field-selector spec.nodeName=prod-eu-1-node-a -o json | jq -r '
  .items[] | "\(.metadata.namespace)/\(.metadata.name)\t\(.spec.nodeSelector // {})\t\(.spec.affinity != null)"'

If a pod selects disktype=ssd and this is the only node with that label, there is nowhere for it to go. The same applies to required pod anti-affinity: a pod that must not share a node with its siblings needs one free node per replica.

Taints. Compare node taints to the tolerations of the pods you are about to move:

kubectl get nodes -o custom-columns='NODE:.metadata.name,TAINTS:.spec.taints[*].key'

Capacity. Look at what the remaining nodes can absorb:

kubectl describe nodes | grep -A6 "Allocated resources"
kubectl top nodes

Requests, not usage, decide scheduling. A node that looks idle in top can be full in Allocated resources.

Zones and volumes. A pod bound to a zonal volume can only restart in that zone. If the zone has no spare capacity, it stays Pending.

None of this replaces a real drain in a staging cluster, and the scheduler has more inputs than any manual check covers (topology spread constraints, priorities, extended resources). But the checks above catch the usual causes.

A pre-drain checklist

Run these in order:

1. kubectl get pdb -A and look for ALLOWED DISRUPTIONS of 0 on workloads with pods on the node.
2. List unmanaged pods and emptyDir pods on the node, and decide on --force and --delete-emptydir-data knowingly.
3. For each workload on the node, confirm it has a place to go: labels, taints, free requests, zone.
4. Cordon first (kubectl cordon), then drain with --ignore-daemonsets --timeout.
5. If it hangs, read the retry message. It names the pod and the namespace.

Doing this by hand for one node is fine. Doing it for every node in a pool before an upgrade is tedious and easy to get wrong.

Doing the same check with a tool

Shoalcheck is a read-only Kubernetes audit tool (CLI and desktop app) that installs nothing in the cluster. shoalcheck preflight drain <node> answers this one question: whether that node can be evacuated, and if not, which object stops it (a budget at zero, a pod nothing recreates, a volume pinned to the node, work with nowhere to go). It exits 0 when nothing stops the drain, 2 when something does, and 1 when RBAC refused some reads and the answer is partial. Preflight is free on every plan: shoalcheck.com, downloads on GitHub releases. Its model checks less than the scheduler does, so treat a clear result as "nothing found", not a guarantee.

Summary

A drain hangs because an eviction is refused or a replacement cannot start. A PDB with ALLOWED DISRUPTIONS of 0 is the usual cause, and the single-replica Deployment with minAvailable: 1 is the usual way to create one. Bare pods, emptyDir data and DaemonSets are things drain makes you acknowledge with flags. Unschedulable replacements are the quietest failure, so look at selectors, taints, capacity and zones before you cordon, not after.