In practice, an elastic system starts with a minimal resource allocation so it can handle requests under normal load conditions.
Over time, certain situations can arise (for example, seasonality in user demand, sudden massive spikes, etc.) where the system's resources tend to become saturated due to a sudden increase in incoming requests.
In this situation, an elastic system expands its capacity by allocating additional resources from the Cloud to avoid service interruption or performance degradation.
When the request rate drops from its peak back to normal, some system resources remain underutilized. In this case, the system reduces its capacity by releasing some resources back to the Cloud, thus no longer paying for idle usage.
This is how an elastic system meets its performance criteria at minimal cost, in response to workload fluctuations.
Kubernetes implements this mechanism through three complementary components, each acting at a different level:
1. Horizontal Pod Autoscaler (HPA): minimal allocation and adapting to load
In Kubernetes, a HorizontalPodAutoscaler (HPA) automatically updates a workload resource (such as a Deployment or a StatefulSet) to automatically adjust capacity to meet demand.
Horizontal scaling means that the response to increased load is to deploy more Pods.
If the load decreases and the number of Pods is above the configured minimum, the HorizontalPodAutoscaler tells the workload resource (the Deployment, the StatefulSet, or any other similar resource) to scale back down.
This is called scaling out and scaling in!
⚠️ Horizontal Pod autoscaling does not apply to objects that cannot be scaled (for example: a DaemonSet).
2. Vertical Pod Autoscaler (VPA): complementary adjustment
A VerticalPodAutoscaler (VPA) automatically updates a workload management resource (such as a Deployment or a StatefulSet) to automatically adjust the infrastructure resource requests and limits to match actual usage.
Vertical scaling means that the response to increased resource demand is to allocate more resources (for example: memory or CPU) to the Pods already running the workload.
This is also known as:
- "rightsizing"
- or sometimes "autopilot"
This differs from horizontal scaling, which in Kubernetes would mean deploying more Pods to spread the load.
If resource usage decreases and the Pods' resource requests are above optimal levels, the VerticalPodAutoscaler tells the workload resource to reduce the resource requests, thus avoiding waste.
This is called scaling up and scaling down!
3. Cluster Autoscaler (CA): actual Cloud resource allocation
The Cluster Autoscaler is a tool that automatically adjusts the size of the Kubernetes cluster when one of the following conditions is met:
- Pods could not be scheduled in the cluster due to insufficient resources.
- Some nodes in the cluster have been underutilized for an extended period and their Pods can be moved to other existing nodes.
The main goal of the Cluster Autoscaler is to provide a place for pending Pods to run. It periodically checks for pending Pods and increases the size of the cluster if appropriate and if the scaled cluster remains within user-defined limits.
The provisioning time of new nodes does not depend on the Cluster Autoscaler itself, but rather on the cloud provider and other Kubernetes components.
It is now widely treated as a standard autoscaling component by Cloud providers, which often integrate it natively into their offerings (EKS, GKE, AKS).
💡 NB: There are also other tools for scaling on application-specific metrics, beyond the native CPU/memory-based HPA. Indeed, the cluster can remain idle while requests time out, if the monitored metric (e.g. CPU) is not the real bottleneck (e.g. memory, database connections, message queues, etc.)
Among these other tools are Prometheus Adapter, KEDA, and Datadog Kubernetes Autoscaling (DKA).
The power of the AWS ecosystem with Amazon EKS
Setting up a cluster autoscaler without the help of a Cloud provider is complex. Indeed, if your physical servers have reached their limits, you can't "conjure up" RAM in two minutes. Moreover, it is often very difficult to tie the cluster autoscaler to the physical hypervisor if you want to build an on-premise Kubernetes cluster.
However, as mentioned above, the cluster autoscaler integrates well with public cloud providers. We'll focus on AWS and its managed EKS solution.
So what is EKS's competitive advantage at the infrastructure layer?
On AWS, the Cluster Autoscaler uses Amazon EC2 Auto Scaling Groups to manage node groups. The Cluster Autoscaler typically runs as a Deployment in your cluster.
The Cluster Autoscaler therefore acts as the bridge between the state of the Kubernetes cluster (pending pods, underutilized nodes) and the AWS API that actually drives the EC2 instances (the ASG).
The two main loops of the Cluster Autoscaler on EKS
| Loop | Frequency | Role |
|---|---|---|
| Scale-Up | Every 10 seconds | Identifies unschedulable pending Pods. Simulates scheduling and, if necessary, increases the DesiredCapacity of the Auto Scaling group to trigger the launch of new instances. The Autoscaler selects a node group via the configured expander strategy. |
| Scale-Down | Continuous | Looks for underutilized nodes. If the Pods can be rescheduled safely, it drains the node and reduces the DesiredCapacity to terminate the instance. |
Flow diagram:
- A Pod is in pending state due to insufficient resources.
- The Kubernetes Cluster Autoscaler increases the desired number of instances in an Auto Scaling Group.
- AWS Auto Scaling provisions a new node.
- The Pod is scheduled on the new node.
The major drawbacks of the Cluster Autoscaler (CA)
-
Slow provisioning: it can take up to 10 minutes to add a node.
- Why? The CA has to query several APIs in cascade: Auto Scaling Group API → EC2 API.
-
Rigid configuration:
- Requires the prior creation of "Managed Node Groups" or "Auto Scaling Groups".
- Each group must contain instances of a fixed type. For GPU, CPU-optimized, or high-memory instances, you must create a specific group for each type.
- If a workload requires a resource type that isn't configured in an existing group, the CA won't be able to scale.
- Operational complexity: a lot of maintenance to manage the different node groups.
The Karpenter revolution
Karpenter is an open-source project originally developed by Amazon to solve the problems described above.
Key advantages of Karpenter
- Speed: it communicates directly with the cloud provider's API (e.g. EC2 on AWS), without going through intermediate layers (Node Groups / Auto Scaling Groups). Provisioning is much faster.
- Flexibility (Right-sizing): instead of launching predefined instances, Karpenter analyzes the exact needs of the pending Pods and launches the most suitable instance (cheapest, just enough CPU/RAM).
- Consolidation: Karpenter can consolidate workloads onto fewer nodes to reduce costs and avoid underutilized nodes.
-
Multi-Cloud support: AWS (native), Azure, Alibaba Cloud, and any provider supporting the Cluster API (Cluster API Provider).
- Note: GKE has its own native solution, so Karpenter is not a priority for Google Cloud.
How Karpenter works technically
Karpenter is installed via Helm and deploys several CRDs (Custom Resource Definitions).
a. The main CRDs
1. NodeClass (cloud-specific)
- Defines how to interact with the provider (e.g. EC2NodeClass for AWS, AKSNodeClass for Azure).
- Configures low-level parameters: OS type, tags, security, Pod limits per node, etc.
2. NodePool (the key resource)
- Defines needs rather than fixed instance types.
- Examples of possible constraints: "I want between 2 and 10 vCPUs", "arm64 or amd64 architecture", "eu-west-1 zone", "Spot or On-Demand".
- Karpenter automatically picks the best instance type matching those criteria.
- Allows automatically adding Taints and Labels to new nodes.
- Manages node expiration (to regularly renew instances and avoid configuration drift).
3. NodeClaim (managed automatically)
- Created and deleted dynamically by Karpenter.
- Represents a concrete request for a node.
- Serves as a historical record to understand why a node was created or deleted.
b. The lifecycle
- A Pod is in the "Pending" state.
- Karpenter evaluates the needs and creates a NodeClaim.
- It calls the Cloud API to provision the instance.
- Once the node is ready, it updates the NodeClaim and schedules the Pods onto it.
Summary
| Component | Level of action | Addresses |
|---|---|---|
| HPA | Number of Pods | Application load (scale out/in) |
| VPA | Resources of a Pod | Actual CPU/memory usage (scale up/down) |
| Cluster Autoscaler | Cluster nodes | Pending Pods / underutilized nodes |
| Karpenter | Cluster nodes (next generation) | Precise Pod needs, without Node Groups |
Kubernetes autoscaling therefore relies on a combination of complementary mechanisms ( application-level (HPA/VPA) and infrastructure-level {Cluster Autoscaler or Karpenter} ) to guarantee performance and cost control in the face of load variations.
Sources
- https://github.com/kubernetes/autoscaler/blob/
- https://kubernetes.io/docs/concepts/workloads/autoscaling/
- https://docs.aws.amazon.com/eks/latest/userguide/autoscaling.html
- https://docs.aws.amazon.com/eks/latest/userguide/autoscaling.html
- https://karpenter.sh/docs/getting-started/
- https://www.youtube.com/watch?v=THj__UYiq90
Top comments (2)
That was a concise explanation. and a very perfect one.
Thank you, I appreciate !