The Six Minutes That Decide a Kubernetes Node Failure
Following a single node failure through heartbeats, controllers, pod eviction, network partitions, and the distributed systems principles that make Kubernetes resilient.
Ask any engineer what happens when a Kubernetes node fails, and you'll probably hear something like this: the node becomes unreachable, Pods are eventually evicted, replacement Pods are scheduled on healthy nodes, and the application recovers.
That's the explanation I carried for a long time. It describes the outcome, but not the reasoning behind it.
After spending the past week studying Kubernetes internals, I realized something that completely changed how I think about node failures.
Kubernetes isn't reacting to a node failure. It's reacting to uncertainty.
That might sound like a small distinction, but it explains almost every decision the control plane makes. Why does traffic stop reaching an unreachable node long before replacement Pods are created? Why do Pods continue to appear as Running even though the machine hosting them is no longer communicating with the cluster? Why does the ReplicaSet Controller intentionally wait instead of immediately restoring the desired replica count? And why is a network partition considered one of the hardest problems in distributed systems?
The answer isn't hidden inside a single Kubernetes component. It's the result of multiple independent controllers, each observing the cluster from its own perspective, making decisions based only on the information available to it. Together, those decisions produce the behavior we simply describe as "self-healing."
In this article, we'll follow a single Kubernetes node failure from the first missed heartbeat until the cluster reaches a stable state again. Along the way, we'll uncover the reasoning behind every major decision the control plane makes, not just what happens, but why Kubernetes was designed to behave that way.
By the end, you'll see that a Kubernetes node failure isn't really a story about Pods or controllers. It's a story about how distributed systems make safe decisions when they can never be completely certain about the state of the world.
A Node Doesn't Fail Overnight. It First Goes Silent.
When a worker node crashes, Kubernetes doesn't receive a notification saying, "this machine is down." There isn't a dedicated failure event broadcast across the cluster, nor does the control plane continuously verify that every node is alive.
Instead, it relies on a much simpler signal: periodic heartbeats from the kubelet.
Every node runs a kubelet, and one of its responsibilities is to regularly reassure the control plane that it's still healthy. As long as those heartbeats continue to arrive, the control plane assumes the node is available and leaves it alone. The moment they stop arriving, however, the cluster enters a very different mode. It doesn't immediately conclude that the node has failed. It starts gathering evidence.
Modern Kubernetes uses two mechanisms for these heartbeats.
The first is the NodeStatus object, which contains detailed information such as node conditions, capacity, addresses, and system information. Updating this object is relatively expensive because every update writes a large amount of metadata into the control plane.
To reduce that overhead, Kubernetes introduced Lease objects. A Lease contains very little information, essentially just a timestamp that says, "I'm still here." Since the object is small, kubelets can update it frequently without placing unnecessary write pressure on the API server and etcd. In large clusters with thousands of nodes, this optimization significantly reduces control plane load while preserving fast failure detection.
The control plane continuously watches these heartbeats. If a Lease isn't renewed within the configured grace period, currently forty seconds by default, Kubernetes treats the missing heartbeat as enough evidence to investigate further. Notice what triggered this decision. The control plane still hasn't confirmed that the machine has crashed. It only knows that communication has stopped.
That distinction becomes important because the same symptom can be caused by very different failures. A machine may have lost power. The kubelet process may have crashed. A network partition may have isolated the node from the control plane while everything on the machine continues running normally.
At this point, Kubernetes doesn't know which scenario it's dealing with. It only knows that a previously healthy node has gone silent.
Everything that follows is driven by how the control plane responds to that uncertainty.
The First Controller to Notice: Node Lifecycle Controller
Once the control plane decides that a node has stopped sending heartbeats, the first component to react is the Node Lifecycle Controller.
Its responsibility is straightforward: continuously evaluate the health of every node in the cluster and keep the node's state consistent with what the control plane currently knows. It isn't responsible for creating replacement pods or updating Services. Its job is simply to answer one question:
"Can this node still be trusted?"
After the heartbeat grace period expires, the controller updates the node's Ready condition to Unknown.
The choice of Unknown is intentional.
Kubernetes isn't declaring that the machine has failed. It is acknowledging that the control plane has lost visibility into the machine. That distinction becomes important later when we discuss network partitions, where a node may continue running workloads perfectly while remaining completely unreachable from the control plane.
Updating the node's condition alone isn't enough. Other components throughout the cluster also need to understand that scheduling new workloads onto this node is no longer safe.
To communicate that decision, the Node Lifecycle Controller applies two taints.
The first is node.kubernetes.io/unreachable:NoSchedule, which prevents the scheduler from placing any new Pods on the unreachable node.
The second is node.kubernetes.io/unreachable:NoExecute, which affects the workloads already running there. Unlike NoSchedule, this taint doesn't immediately remove Pods. Instead, it starts a countdown.
Most workloads automatically receive a default toleration of 300 seconds through Kubernetes' DefaultTolerationSeconds admission controller. During those five minutes, the Pods are allowed to remain associated with the unreachable node while Kubernetes waits to see whether communication can be restored.
This waiting period is a deliberate design choice.
Infrastructure failures aren't always permanent. Temporary network interruptions, overloaded control planes, or brief cloud networking issues can all interrupt heartbeats without the node itself failing. Immediately evicting every workload would create unnecessary rescheduling, container restarts, cache warm-ups, and additional load across the cluster.
By delaying eviction, Kubernetes gives the node an opportunity to recover before making more disruptive decisions.
At this stage, something interesting has happened. The control plane has acknowledged that the node is unreachable, prevented new workloads from being scheduled there, and started the eviction timer for the existing Pods.
Yet none of the running Pods have changed state.
From the API server's perspective, they're still running exactly as they were before.
That detail explains why the next set of controllers behaves very differently.
Three Controllers, Three Different Views of the Same Failure
Once the Node Lifecycle Controller marks the node as Unknown, the rest of Kubernetes doesn't receive a broadcast saying, "Start recovering."
That's because Kubernetes isn't built around components sending commands to one another. Instead, each controller continuously watches the Kubernetes API, observes the current state of the cluster, and independently decides whether any action is required. This reconciliation model is one of the fundamental design principles behind Kubernetes. Every controller works toward the same desired state, but each one does so using its own inputs and responsibilities.
At this point, three controllers are observing the same incident, yet each of them reaches a different conclusion.
The Endpoint Controller reacts first.
Its responsibility is to maintain the list of healthy endpoints behind every Service. Once the node becomes unreachable, it removes the Pods running on that node from the Service's EndpointSlices. This happens long before the Pods are deleted from the cluster.
That detail has an important consequence.
For applications running multiple replicas, incoming requests stop being routed to the unreachable node almost immediately after the node is marked Unknown. Existing Pods still exist in the API, but from the Service's perspective, they are no longer considered valid destinations for new traffic.
While the Endpoint Controller is updating network routing, the ReplicaSet Controller reaches a very different conclusion.
Its responsibility isn't networking. It's replica count.
When it inspects the Deployment, it still sees the expected number of Pods. None of those Pods have been deleted, and none of them have entered the Terminating state. As far as the ReplicaSet Controller is concerned, the desired replica count is still satisfied.
So it does nothing.
This often surprises engineers because they expect replacement Pods to appear as soon as a node becomes unreachable. From the ReplicaSet Controller's perspective, however, there is no shortage to correct. It isn't ignoring the failure. It simply hasn't observed any change that requires reconciliation.
This independent decision-making is one of Kubernetes' biggest strengths. Each controller remains focused on a single responsibility instead of trying to understand the entire system. The Endpoint Controller manages traffic. The ReplicaSet Controller manages replica count. The Node Lifecycle Controller manages node health. Together they produce coordinated behavior without ever calling one another directly.
For the next few minutes, the cluster enters an interesting state. Traffic has already stopped flowing to the unreachable node, the scheduler won't place any new workloads there, and the ReplicaSet Controller is intentionally waiting.
Nothing appears to be happening.
In reality, Kubernetes is waiting for one final signal before it commits to replacing those Pods.
The Six-Minute Timeline
From the outside, Kubernetes can look strangely inactive after a node becomes unreachable. Engineers often expect replacement Pods to appear immediately, but for nearly five minutes the cluster appears to be doing… almost nothing.
In reality, every important decision has already been made.
Around forty seconds after the last successful heartbeat, the Node Lifecycle Controller marks the node as Unknown and applies the NoSchedule and NoExecute taints. Almost immediately afterward, the Endpoint Controller removes the Pods on that node from the Service's EndpointSlices, ensuring that new requests are no longer routed to them.
For applications running multiple replicas, this is often the point where user impact ends. Even though the original Pods still exist in the Kubernetes API, incoming traffic is already flowing to healthy replicas running on other nodes.
The next five minutes are a waiting period.
During this time, the Pods continue to exist because of the default tolerationSeconds value of 300 seconds. The ReplicaSet Controller continues to observe the Deployment, but since those Pods haven't entered the Terminating state yet, it still considers the desired replica count to be satisfied.
Once the toleration period expires, the Node Lifecycle Controller finally begins the eviction process by sending deletion requests for the Pods on the unreachable node. The Pod objects receive a deletionTimestamp and transition into the Terminating state.
Only now does the ReplicaSet Controller observe that the Deployment has fewer active replicas than requested.
This is the signal it has been waiting for.
The controller immediately creates replacement Pods, which are then picked up by the scheduler and placed onto healthy worker nodes. As those new Pods start successfully and pass their readiness checks, the Endpoint Controller adds them back into the corresponding EndpointSlices, allowing Services to begin routing traffic to them.
Looking back at the entire sequence, the recovery process follows a very deliberate order:
~0–40 seconds: Kubernetes waits for heartbeat evidence before concluding the node is unreachable.
~40 seconds: The node is marked Unknown, scheduling is blocked, and Service traffic is redirected away from the unreachable node.
40–340 seconds: Existing Pods are intentionally left untouched while Kubernetes waits for the node to recover.
~340 seconds: Pod eviction begins, replacement Pods are created, scheduled, started, and gradually reintroduced into Service endpoints.
One of the biggest misconceptions about Kubernetes is that application recovery starts when replacement Pods are created. In practice, recovery often begins much earlier. For highly available applications, traffic is usually redirected long before new Pods exist. The later stages are primarily about restoring the cluster to its desired state rather than restoring availability.
Understanding this timeline makes many seemingly odd Kubernetes behaviors feel completely logical. The system isn't slow, and it isn't hesitating. It's following a carefully designed sequence that balances availability, stability, and the risk of making the wrong decision based on incomplete information.
Why Doesn't Kubernetes Replace the Pods Immediately?
At first glance, Kubernetes' five-minute waiting period feels unnecessarily cautious.
If the control plane already knows a node is unreachable, why not replace the Pods immediately and restore the desired state as quickly as possible?
The answer is that the control plane doesn't actually know whether the node has failed.
A missing heartbeat only tells Kubernetes that communication has stopped. The node may have lost power, the kubelet might have crashed, or a temporary network issue may have isolated the node from the control plane. From Kubernetes' perspective, all three situations look identical.
Acting immediately would be a risky assumption.
Imagine a network interruption lasting only a minute. If Kubernetes instantly evicted every Pod and recreated them on other nodes, the cluster would trigger container restarts, cache warm-ups, database failovers, and workload redistribution, only for the original node to reconnect moments later. The recovery process itself would become the source of unnecessary disruption.
Instead, Kubernetes separates protection from recovery.
As soon as the node becomes unreachable, it prevents new workloads from being scheduled there and redirects Service traffic to healthy replicas. Existing Pods are then given time to recover before the control plane commits to eviction. If communication is restored during that window, the cluster avoids an expensive and unnecessary recovery cycle.
This is a trade-off between recovery speed and confidence.
Distributed systems rarely have complete information, so irreversible decisions are postponed until there is enough evidence to justify them. Waiting a few minutes may seem slow, but recovering from an unnecessary failover or inconsistent cluster state is usually far more expensive.
This design choice naturally leads to another question.
What happens if the node never failed at all?
What if it continued running the entire time, while the control plane simply lost the ability to communicate with it?
The Hardest Failure Isn't a Dead Node. It's a Network Partition.
Everything we've discussed so far assumes the worker node actually stopped functioning.
Unfortunately, that's not always what happens.
Consider a different scenario. The node is still powered on. The kubelet is still running. Your application is still processing requests. The only thing that has failed is the network connection between that node and the control plane.
From the control plane's perspective, this situation is indistinguishable from a dead machine.
Heartbeats stop arriving.
The node is marked Unknown.
Traffic is redirected away.
The eviction timer begins.
Eventually, the control plane starts deleting the Pods and creating replacements elsewhere.
Meanwhile, the original node has no idea any of this is happening.
The kubelet continues running the existing containers because it never received the deletion request. As far as that machine is concerned, nothing has changed.
For a stateless web application, this usually isn't catastrophic. Existing client connections may continue working until they eventually disconnect, while new requests are routed to healthy replicas elsewhere in the cluster.
Stateful applications are a completely different story.
Imagine a PostgreSQL primary running inside a StatefulSet. If the control plane immediately created another Pod with the same identity while the original database was still accepting writes, two independent instances could begin acting as the same database.
This is the classic split-brain problem.
Preventing that outcome is one of the reasons StatefulSets behave much more conservatively than Deployments. They won't create a replacement Pod with the same identity until Kubernetes has enough evidence that the previous one is truly gone.
Even then, Kubernetes isn't responsible for protecting your data consistency.
Systems such as Patroni, etcd, and ZooKeeper introduce another layer of protection using leader election and lease-based fencing. Rather than trusting the isolated node to shut itself down correctly, they rely on an external authority to determine which instance is allowed to accept writes. Once a node loses ownership of its lease, another instance can safely become the leader without depending on the old primary to cooperate.
This is one of the most important design principles in distributed systems.
Safety guarantees should never depend on the component that may already be in a faulty or isolated state.
The authority to grant ownership must come from a healthy part of the system.
That's exactly why Kubernetes treats uncertainty so carefully. It isn't trying to prove that a node has failed. It's trying to avoid making a decision that could leave the cluster in a worse state than the original failure itself.
Lessons for Production Systems
Following a node failure step by step changes how you think about operating a Kubernetes cluster. Many production issues become much easier to reason about once you understand which controller is making a decision and, more importantly, what information that controller actually has.
One of the biggest takeaways is that recovery doesn't begin when replacement Pods are created. For applications running multiple replicas, the Endpoint Controller usually removes unreachable Pods from Service endpoints long before new Pods are scheduled. If your monitoring only tracks Pod creation, you're measuring the wrong stage of the recovery process. Measuring traffic recovery often provides a much more accurate picture of application availability.
Another important lesson is that redundancy matters more than recovery speed. Even if Kubernetes could replace Pods instantly, a single-replica application would still experience downtime while a new instance starts. Multiple healthy replicas allow traffic to continue flowing while the control plane works through the recovery process in the background.
Understanding the difference between Deployments and StatefulSets is equally important. Deployments prioritize restoring application capacity, making them well suited for stateless workloads. StatefulSets prioritize data safety and stable identities, accepting a slower recovery process to reduce the risk of split-brain scenarios. Neither approach is universally better. They solve different problems.
This also explains why force-deleting Pods should never be the first response during an incident. A Pod stuck in the Terminating state may indicate that the kubelet is unreachable rather than permanently gone. Verifying whether the node has actually failed before forcing cleanup can prevent far more serious issues, especially for stateful workloads.
Finally, remember that Kubernetes is only one layer of a distributed system. For applications where correctness is more important than availability, technologies such as Patroni, etcd, or ZooKeeper provide the coordination mechanisms needed to safely manage leadership and prevent conflicting writers. Kubernetes orchestrates infrastructure remarkably well, but consistency guarantees belong to the application layer and the consensus systems that support it.
I have one editorial suggestion before we write the conclusion.
The article no longer feels like a Kubernetes tutorial. It has evolved into an article about engineering decision-making under uncertainty, using Kubernetes as the case study. I think that's exactly what gives it lasting value. Someone reading it a year from now won't just remember the 40s and 340s timings. They'll remember the underlying principle:
Distributed systems make decisions based on the information they have, not the information they wish they had.
That's a much stronger takeaway to end on than simply summarizing the recovery timeline.
Conclusion
When I started exploring Kubernetes node failures, I expected to learn about heartbeats, controllers, and pod recovery. Those pieces are certainly important, but they weren't the biggest lesson.
The real lesson was understanding how a distributed system behaves when it doesn't have the complete picture.
Throughout this article, every controller made decisions using only the information available to it. The Node Lifecycle Controller monitored node health. The Endpoint Controller managed traffic. The ReplicaSet Controller maintained the desired number of replicas. None of them understood the entire incident, yet together they guided the cluster toward a consistent and safe state.
That design is what makes Kubernetes resilient.
It doesn't assume a node has failed simply because communication stopped. It doesn't immediately replace workloads because doing so could create even bigger problems. Instead, it gathers evidence, limits the blast radius, and postpones irreversible actions until it has enough confidence to proceed.
Once you understand that philosophy, many Kubernetes behaviors that initially seem slow or overly cautious begin to make sense. The five-minute waiting period isn't wasted time. Pods remaining in the Running state after a node disappears aren't a bug. StatefulSets refusing to create replacements immediately aren't being stubborn. Each of these decisions exists because acting on incomplete information is often more dangerous than waiting for better evidence.
More importantly, this principle extends far beyond Kubernetes.
You'll find the same pattern in distributed databases, consensus algorithms, message brokers, and leader election systems. Whenever multiple machines must agree on the state of the world, correctness almost always comes before speed.
The next time a Kubernetes node disappears from your cluster, you probably won't think about the six-minute timeline first.
You'll think about what the system actually knows.
And in distributed systems, that's usually the question that matters most.
🔗 Connect with Me
📖 Blog by Naresh B. A.
👨💻 Backend & AI Systems Engineer | Distributed Systems · Production ML
🌐 Portfolio: Naresh B A
📫 Let's connect on LinkedIn | GitHub: Naresh B A
Thanks for spending your precious time reading this. It's my personal take on a tech topic, and I really appreciate you being here. ❤️
Top comments (0)