When automatic scaling (autoscaling) kicks in for Kubernetes clusters, adding new nodes or removing existing ones becomes a necessity. However, the sudden eviction of pods from existing nodes as they are removed can lead to application disruptions and a drop in service quality. This situation is unacceptable, especially for critical workloads. This is where intelligent scaling solutions like Karpenter, which have become popular on platforms like Amazon EKS and Google GKE, come into play. As Karpenter dynamically manages nodes based on cluster needs, it must understand and consider mechanisms like PodDisruptionBudget (PDB) to ensure applications run without interruption. In this post, we will delve into how Karpenter utilizes PDBs during node consolidation and how it ensures secure cluster management through this process.
Automatic Scaling in Kubernetes and Its Challenges
While Kubernetes has the Horizontal Pod Autoscaler (HPA) for scaling pods, the scaling of the underlying node infrastructure is typically managed by solutions like the Cluster Autoscaler (CA). CA is triggered when pods enter a Pending state and adds new nodes as needed. However, the process of removing existing nodes (deprovisioning) for cost optimization or balancing resource utilization is one of CA's most sensitive areas. CA may decide to remove nodes when it finds more suitable placements for pods or when resources become scarce.
This removal process, known as "draining" a node, is a prioritized operation for CA. Pods on the node, if not managed by PodDisruptionBudget (PDB) or if PDB rules are lax, can be evicted abruptly. These sudden evictions can lead to unexpected application downtime. Especially in cases of network issues or high pod density, eviction processes can be delayed or fail. This can prevent CA from removing new nodes, halting cluster scaling and leading to unnecessary costs.
Introduction to Karpenter
Karpenter is an open-source node orchestration project designed to automatically manage Kubernetes nodes on AWS. Unlike the traditional Cluster Autoscaler, Karpenter adopts an event-driven approach. Instead of waiting for pods to enter a Pending state, it listens for changes in the Kubernetes API and acts immediately based on workload requirements. This minimizes the time pods spend in a Pending state, significantly improving overall cluster responsiveness.
One of Karpenter's key advantages is making the node provisioning process smarter and more efficient. Cluster administrators don't need to define specific NodePools beforehand; Karpenter analyzes the characteristics of running pods (CPU, RAM, GPU requirements, etc.) to select the most suitable node or nodes and launches them instantly. This flexibility allows for the dynamic creation of the optimal infrastructure for different workloads. Simultaneously, it contributes to cost optimization by identifying and removing unused nodes.
Karpenter's Node Consolidation Mechanism
One of Karpenter's most powerful features is its active node consolidation (merging or removing nodes) to reduce costs and utilize resources efficiently. Karpenter continuously monitors nodes in the cluster and evaluates whether pods can be shifted to nodes requiring fewer resources or if a node is no longer needed at all. This evaluation primarily relies on whether pods fit onto existing nodes and which pods can be safely evicted.
When Karpenter decides to remove a node, it first inspects the pods on that node. If the pods on the node to be removed can be easily moved to another node without causing issues across the cluster, Karpenter begins to drain that node. This process is as crucial as adding new nodes because unnecessarily running nodes increase costs. Karpenter's decision to consolidate must critically consider the overall health of the cluster and the operational continuity of applications.
The Role of Disruption Budget in Node Consolidation
PodDisruptionBudget (PDB) is a protection mechanism against voluntary disruptions in Kubernetes. Voluntary disruptions include planned operations such as cluster maintenance, upgrades, or node removals. PDBs allow you to specify how many pods of a particular application must remain running or how many can be evicted simultaneously to prevent service interruption. For example, a PDB set to maxUnavailable: 1 means that at most one pod from that PDB-governed group of pods can be evicted at any given time.
Advanced node orchestrators like Karpenter consider existing PDBs when making node removal decisions. If Karpenter decides to remove a node and a critical pod governed by a PDB is running on that node, Karpenter checks if evicting this pod would violate PDB rules. If the eviction would break the PDB, Karpenter will halt or pause the node removal process. This ensures that critical applications always run with the minimum number of pods specified, guaranteeing service continuity.
This is precisely why Karpenter needs to understand and adhere to PodDisruptionBudgets. While it's important to remove a node cost-effectively, it should not come at the expense of a critical application becoming completely unavailable. PDBs help us establish this balance.
Disruption Budget Configuration with Karpenter
Karpenter's integration with PodDisruptionBudget (PDB) is quite straightforward because Karpenter automatically recognizes and respects PDB resources in the Kubernetes API. This means if you already have PDBs defined in your cluster, Karpenter will automatically consider them when it decides to drain nodes. You don't need to perform any extra Karpenter configuration. All you need to do is create the correct PDBs for your critical applications.
You can use the kubectl command to create a PDB. For instance, assume you have a highly available application and you want at least 2 pods of this application to always be running. In this case, you could create a PDB like this:
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: my-critical-app-pdb
spec:
minAvailable: 2
selector:
matchLabels:
app: my-critical-app
This PDB definition guarantees that at least two pods with the label app: my-critical-app will always be available. If Karpenter decides to drain a node running a pod with this label, and the current number of running pods is already 2, Karpenter will not initiate the draining process for that node. It will wait until the minAvailable value of the PDB is met.
To visualize Karpenter's adherence to these PDBs, you can refer to the following diagram:
As seen in this flow, when Karpenter attempts to remove a node, it first checks the pods on that node and their associated PDBs. If PDB rules allow for eviction (e.g., if the maxUnavailable value is sufficient or minAvailable can still be maintained), the node draining process continues. However, if PDB rules would be violated, Karpenter will stop this operation and wait for the situation to reach a state permitted by the PDB.
Real-World Scenarios and Trade-offs
In real-world scenarios, the interaction between Karpenter and PDBs requires careful planning. It's important to strike the right balance between cost optimization and high availability. If PDBs are set too strictly, for example, with minAvailable: 100% in a small cluster, Karpenter might struggle to remove nodes to reduce costs. This can lead to an excessive number of nodes running unnecessarily in the cluster, resulting in higher costs.
On the other hand, if PDBs are not defined for critical applications, or if maxUnavailable is set too high, Karpenter might remove nodes too aggressively. In such cases, when a node is removed, all critical pods on it could be evicted, leading to service interruption for the application. This is a situation where the automated system makes seemingly illogical decisions, but the root cause is actually misconfigured PDBs.
Another important trade-off is the terminationGracePeriodSeconds for pods. When a pod is evicted, it can continue running for this duration before being cleanly shut down. Karpenter waits for the eviction of pods on a node to complete before removing the node. If terminationGracePeriodSeconds is too long, or if pods don't shut down within this period, the node removal process can also be delayed. This can prevent new nodes from coming online in a timely manner, especially in fast-scaling environments or during sudden traffic spikes.
To establish this balance, it's essential to understand application requirements, adjust PDBs based on cluster size and criticality, and regularly monitor Karpenter's node removal decisions. You can use commands like kubectl describe pdb <pdb-name> to check the status of existing PDBs and kubectl logs -n karpenter -l app.kubernetes.io/name=karpenter to monitor Karpenter logs.
Best Practices for Secure Node Consolidation
To make node consolidation with Karpenter secure and cost-effective, it's important to adhere to some fundamental principles. The first and most crucial step is to always define PodDisruptionBudget (PDB) for your critical workloads. This ensures your applications are protected from unexpected disruptions and allows Karpenter to consider this criticality when making node removal decisions. When setting up PDBs, you should correctly determine the minimum number of pods (minAvailable) or the maximum number of evictable pods (maxUnavailable) your application requires. These values should be adjusted based on the total number of nodes in the cluster and the distributed nature of the application.
Secondly, you should regularly monitor Karpenter's node removal decisions and PDB compliance. By examining Karpenter's logs (kubectl logs -n karpenter -l app.kubernetes.io/name=karpenter), you can see which nodes it's trying to remove, why, and if it's encountering any PDB-related incompatibilities. Additionally, reviewing pod eviction times and terminationGracePeriodSeconds settings in the cluster can be beneficial.
Thirdly, understand the trade-offs between cost and availability. Karpenter will aim to remove nodes to reduce costs. If PDBs prevent these removals, cluster costs might exceed expectations. In such cases, you can review your PDB settings and slightly loosen PDB rules to increase cost optimization, or add more nodes to distribute more pods across different nodes, allowing for consolidation.
Finally, monitoring Karpenter's drift and disruption mechanisms can help you detect unexpected behavior early. By inspecting Karpenter logs or using relevant monitoring tools, you can report which pods are being targeted for eviction and whether these evictions conflict with PDB rules.
Conclusion
Cost-effectiveness and high availability in Kubernetes clusters always require a balance. Karpenter, with its intelligence and speed in node provisioning and deprovisioning, helps us establish this balance. However, ensuring the operational continuity of critical applications during Karpenter's node consolidation relies on the correct understanding and configuration of PodDisruptionBudget (PDB) mechanisms.
By defining appropriate PDBs, you can ensure your applications run without interruption while Karpenter safely removes nodes. This allows you to both optimize costs and provide a seamless experience to your users. By effectively leveraging the integration of Karpenter and PDBs, you can make your Kubernetes infrastructure more robust, efficient, and reliable.
Top comments (0)