kubectl drain exits 0. Your database sits Pending.

kubectl drain has one job: evict the pods on a node, respecting disruption budgets. When every eviction is accepted, it exits 0. It does not check that the evicted pods have anywhere to go. That is the scheduler's problem, and the scheduler only gets asked after the pod is already gone.

Most of the time that is fine. Two cases where it isn't come up on almost every production cluster: a pod whose disk lives in one zone, and a pod that holds a port on the host. Both drain cleanly, and in both the replacement pod never starts.

Case 1: a zonal disk, and a zone with no room

A pod's PersistentVolume can carry a required nodeAffinity. The scheduler's VolumeBinding filter only places the pod on a node whose labels satisfy it. For a cloud disk the affinity names a zone, because the disk physically lives there.

The part that is easy to miss: CSI drivers write that zone in their own label key, not in topology.kubernetes.io/zone:

Driver	Key on the PV and on the node
Azure Disk	topology.disk.csi.azure.com/zone
Amazon EBS	topology.ebs.csi.aws.com/zone
GCE Persistent Disk	topology.gke.io/zone

So a script that looks for topology.kubernetes.io/zone on your PVs finds nothing and concludes the disks are not pinned. They are.

Now drain a node in that zone when the other nodes in the zone can't take the pod (a taint, a selector, no room). Nodes in other zones have room, but the disk can't follow. The drain exits 0, and the pod sits Pending.

Reproduce it on kind

Four nodes: a control plane, two workers in zone za (one tainted for another team), one worker in zone zb. The PV's affinity uses the Azure Disk key, the way a real Azure disk would.

kind create cluster --name drain-demo --config - <<'EOF'
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes: [{role: control-plane}, {role: worker}, {role: worker}, {role: worker}]
EOF

for n in worker:za worker2:za worker3:zb; do
  kubectl label node drain-demo-${n%%:*} \
    topology.kubernetes.io/zone=${n##*:} topology.disk.csi.azure.com/zone=${n##*:}
done
kubectl taint node drain-demo-worker workload=db:NoSchedule

Then a PV pinned to za in the driver's key, and a one-replica StatefulSet that claims it:

apiVersion: v1
kind: PersistentVolume
metadata: {name: pv-db}
spec:
  capacity: {storage: 1Gi}
  accessModes: [ReadWriteOnce]
  storageClassName: manual
  hostPath: {path: /tmp/pv-db, type: DirectoryOrCreate}
  claimRef: {namespace: data, name: data-db-0}
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - {key: topology.disk.csi.azure.com/zone, operator: In, values: [za]}

(The StatefulSet is a pause container with a volumeClaimTemplates entry named data in namespace data; full manifest at the end.)

db-0 lands on worker2, the only untainted node in za. Drain it:

$ kubectl drain drain-demo-worker2 --ignore-daemonsets --delete-emptydir-data
...
$ echo $?
0
$ kubectl -n data get pod db-0
NAME   READY   STATUS    RESTARTS   AGE
db-0   0/1     Pending   0          20s

and the scheduler's reason:

0/4 nodes are available: 1 node(s) didn't match PersistentVolume's node affinity, 1 node(s) were unschedulable, 2 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/4 nodes are available: 4 Preemption is not helpful for scheduling.

worker3 has room, but it's in zb. worker is in za, but it's tainted. Nothing else is left.

Case 2: a host port, held on every other node

An ingress controller or a node-local agent is often run as a Deployment with hostPort (or hostNetwork: true), one replica per node. The scheduler's NodePorts filter won't put two pods holding the same host port on one node.

One detail makes this easy to miss: with hostNetwork: true, every containerPort you declare is a host port. The API server fills in hostPort for you.

Drain one of those nodes, and the evicted replica needs a node where the port is free. If every other eligible node already runs a replica, there isn't one:

$ kubectl -n edge get pods -o wide --no-headers | awk '{print $1,$3,$7}'
ingress-5bdbf6d67b-6gxqd Running drain-demo-worker3
ingress-5bdbf6d67b-8tdnz Running drain-demo-worker2

$ kubectl drain drain-demo-worker3 --ignore-daemonsets --delete-emptydir-data
...
$ echo $?
0
$ kubectl -n edge get pods
NAME                       READY   STATUS    RESTARTS   AGE
ingress-5bdbf6d67b-8tdnz   1/1     Running   0          22s
ingress-5bdbf6d67b-tjkvq   0/1     Pending   0          21s

0/4 nodes are available: 1 node(s) didn't have free ports for the requested pod ports, 1 node(s) were unschedulable, 2 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/4 nodes are available: 1 No preemption victims found for incoming pod, 3 Preemption is not helpful for scheduling.

Checking before you drain

Both checks are mechanical. Before draining node N, for each pod on it:

1. Volumes. For each PVC the pod mounts, read the bound PV's spec.nodeAffinity.required. Find the nodes, other than N, whose labels satisfy it. Match on whatever key the PV uses, not just topology.kubernetes.io/zone.
2. Host ports. Collect the pod's host ports (hostPort, or every containerPort when hostNetwork: true). Drop every node where a running pod already holds one of them, by address, protocol and number.
3. The rest. Drop the nodes the pod's nodeSelector, required node affinity and tolerations rule out, and the cordoned ones.

If no node is left, the pod has nowhere to go, and the drain will still exit 0.

Disclosure, and a bug of my own

I build Shoalcheck, a read-only tool that answers this before you drain (shoalcheck preflight drain <node>, free on every plan). It got both cases wrong until this week:

• Until v0.1.2 it read a volume's zone only from topology.kubernetes.io/zone, so a disk pinned in a CSI driver's key looked unpinned.
• Until v0.1.3 it didn't read host ports at all.

On the cluster above, the old versions answered that nothing they check stops the drain. The new ones answer:

$ shoalcheck preflight drain drain-demo-worker3       # v0.1.2
✓ drain-demo-worker3: nothing this model checks stops the drain
$ echo $?
0

$ shoalcheck preflight drain drain-demo-worker3       # v0.1.3
✗ drain-demo-worker3 cannot be evacuated: 1 thing stops the drain
1. Losing node drain-demo-worker3 leaves 1 pod with no node this model accepts
   If node drain-demo-worker3 were unavailable, 1 pod running on it would have no remaining node in this reading that satisfies the constraints this model checks.
   · Snapshot taken at 2026-10-28T12:30:28Z. Each resource kind is listed independently, so the node, pod and budget views can differ by moments; re-verify against the live cluster before acting.
   · edge/ingress-5bdbf6d67b-6gxqd requests 10m cpu and 16Mi memory and host port 8080/TCP; no surviving node in this reading satisfies the modelled constraints
   · modelled here: …
   · not modelled: …
$ echo $?
2

The checks are still a subset of the scheduler. The tool says so in every answer, with the list of what it does not model: preemption, autoscaling, inter-pod affinity, topology spread, and more. It can tell you a drain will strand something. It never tells you a drain is safe.

The drain check is one of the things in this 74-second walkthrough, filmed on the real product (a kind test cluster, every frame a capture):

Full manifest

apiVersion: v1
kind: Namespace
metadata: {name: data}
---
apiVersion: v1
kind: PersistentVolume
metadata: {name: pv-db}
spec:
  capacity: {storage: 1Gi}
  accessModes: [ReadWriteOnce]
  storageClassName: manual
  hostPath: {path: /tmp/pv-db, type: DirectoryOrCreate}
  claimRef: {namespace: data, name: data-db-0}
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - {key: topology.disk.csi.azure.com/zone, operator: In, values: [za]}
---
apiVersion: apps/v1
kind: StatefulSet
metadata: {name: db, namespace: data}
spec:
  replicas: 1
  serviceName: db
  selector: {matchLabels: {app: db}}
  template:
    metadata: {labels: {app: db}}
    spec:
      containers:
      - name: db
        image: registry.k8s.io/pause:3.10
        resources: {requests: {cpu: 10m, memory: 16Mi}}
        volumeMounts: [{name: data, mountPath: /data}]
  volumeClaimTemplates:
  - metadata: {name: data}
    spec:
      accessModes: [ReadWriteOnce]
      storageClassName: manual
      resources: {requests: {storage: 1Gi}}
---
apiVersion: v1
kind: Namespace
metadata: {name: edge}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: ingress, namespace: edge}
spec:
  replicas: 2
  selector: {matchLabels: {app: ingress}}
  template:
    metadata: {labels: {app: ingress}}
    spec:
      containers:
      - name: proxy
        image: registry.k8s.io/pause:3.10
        ports: [{containerPort: 8080, hostPort: 8080}]
        resources: {requests: {cpu: 10m, memory: 16Mi}}