If you read my previous posts on topology spread and the affinity club, we mostly talked about Pod placement: which nodes can accept a new Pod, and how topology spread constraints keep replicas balanced.
But clusters consist of nodes, and nodes need maintenance too. A worker node may need an operating-system upgrade, a kernel update, more capacity, or replacement hardware.
“Okay, then delete it lol,” right?
Sure, if you are okay with the people calling those services starting to scream >:)
Before taking a node out of service, we need to move its workload safely.
That raises three related questions:
- How do we stop new Pods arriving on the node?
- How do the Pods already running there leave?
- How many of those Pods is it safe to interrupt at once?
The first two are handled by cordon, drain, and eviction.
For the third, read the PodDisruptionBudget post ;)
Cordon: stop new placements
kubectl cordon worker-1
Cordoning marks a node as unschedulable. The scheduler will not place new Pods there, but the Pods already on the node keep running.
Think of it as closing a hotel to new guests: the guests already in their rooms are still there.
To make a node schedulable again later:
kubectl uncordon worker-1
Drain: prepare a node to go out of service
kubectl drain worker-1 --ignore-daemonsets
kubectl drain first cordons the node, then tries to evict its eligible Pods. Controllers such as Deployments and StatefulSets then create replacement Pods, which the scheduler can place on other suitable nodes.
Why --ignore-daemonsets?
In our case, kubectl drain stops before evicting the web Pods because it encounters DaemonSet Pods: kindnet and kube-proxy.
DaemonSet Pods are intended to run on every applicable node, so drain refuses to remove them by default.
--ignore-daemonsets tells drain to leave those Pods in place and continue evicting eligible workload Pods.
Those DaemonSet Pods remain until the node itself goes offline or is removed. That is exactly what we want during the drain:
networking / node-level services stay available while the "regular" workload leaves.
But wait... did you see what the command says?
Evicting pod...
Why evicting instead of deleting?
Eviction: removing pods in a graceful / policy-aware manner
When possible, kubectl drain uses Kubernetes’ Eviction API rather than directly deleting a Pod.
An eviction asks Kubernetes to terminate a Pod gracefully and subject to cluster policy.
The Pod receives its configured terminationGracePeriodSeconds, giving the application time to shut down. Whether it stops receiving traffic and completes cleanup safely depends on the workload’s readiness handling, lifecycle hooks, and termination behavior.
Most importantly for the PodDisruptionBudget post, the API checks whether a PodDisruptionBudget permits the disruption.
In a maintenance scenario, the tool should use eviction so availability safeguards are respected.
Planned versus unplanned disruption
Node maintenance is a voluntary disruption since an administrator intentionally asks Pods to leave a healthy node.
However:
- A node crash
- power failure
- network partitions are involuntary disruptions...
At this case Kubernetes cannot ask permission before a failed node becomes unavailable.
PodDisruptionBudgets protect only against voluntary disruptions. They cannot prevent a machine from failing, but they help us avoid making an existing incident worse while performing planned work.
Top comments (0)