The cluster was using roughly 3% CPU.
Kubernetes still couldn't schedule several pods.
One workload was returning a 503, multiple application workloads were stuck in Pending, and the scheduler was reporting insufficient CPU.
At first, those facts seemed contradictory.
How could Kubernetes be short of CPU when the nodes were barely using their processors?
That question led me through CPU requests, a missing Karpenter controller, EBS Availability Zone affinity, IAM trust relationships, CRD mismatches, and finally a Helm upgrade that wasn't quite as successful as it first appeared.
This is what I found.
3% CPU utilisation did not mean 97% capacity was available
When troubleshooting capacity problems, one of the first metrics most of us look at is actual resource utilisation.
In this case, that metric was misleading.
The nodes were consuming only around 2–3% actual CPU, but CPU requests had reached approximately 98–99% of allocatable capacity.
Those two measurements describe very different things.
Actual CPU usage tells us what workloads are consuming at that moment.
CPU requests tell the Kubernetes scheduler how much capacity workloads have already reserved.
So although the underlying machines were mostly idle, Kubernetes saw very little CPU available for additional scheduling.
One important workload required roughly 2 vCPU, and no existing node had enough unreserved capacity to accommodate it.
The result was straightforward:
Pending
A useful mental model is a hotel.
Imagine a hotel with 100 rooms.
Only three guests are physically inside their rooms, but 99 rooms have already been reserved.
The hotel may look empty, but reception cannot allocate those reserved rooms to new guests.
That was effectively the situation inside the cluster.
So why didn't autoscaling add another node?
Once I understood the CPU-request situation, the next question was obvious.
If Kubernetes needed additional capacity, why wasn't the cluster scaling out?
This environment was designed to use Karpenter for dynamic node provisioning.
But during the investigation, I discovered that Karpenter wasn't actually running.
What made this more deceptive was that much of the supporting configuration was still present.
There was a NodePool.
The EC2 node configuration existed.
AWS permissions and discovery configuration were largely present.
Several Kubernetes resources associated with Karpenter were still there.
Looking only at those objects, it would have been easy to assume that autoscaling was available.
The problem was that the Karpenter controller itself was missing.
The configuration existed.
The control loop didn't.
I started thinking about it like a car that still had its steering wheel, dashboard and pedals but no driver.
Everything describing what should happen was present.
Nothing was actually there to make it happen.
So when Kubernetes could no longer schedule the workloads, no new node appeared.
CPU capacity wasn't the only scheduling constraint
At this point, manually adding compute appeared to be the fastest way to restore service.
But the workload returning the 503 had another constraint.
Its persistent storage was backed by Amazon EBS.
EBS volumes are Availability Zone specific.
The persistent volume was tied to one Availability Zone, while the existing nodes were running in other zones.
That meant simply finding a node with enough CPU wasn't sufficient.
The workload needed a node with enough capacity in the Availability Zone where its storage could attach.
So the outage was actually the result of multiple scheduling conditions interacting:
- CPU requests had consumed almost all allocatable CPU from the scheduler's perspective.
- Karpenter wasn't running, so additional capacity wasn't being provisioned automatically.
- A stateful workload needed compute in the same Availability Zone as its EBS volume.
The problem wasn't:
We need another node.
It was:
We need the right capacity, in the right Availability Zone, and the component that should provide it automatically isn't running.
Restoring service
During an incident, I prefer to separate service restoration from architectural remediation.
The first goal was therefore to get the workloads running again.
I temporarily increased the managed node capacity and ensured that the additional capacity included compute in the Availability Zone required by the persistent volume.
Once the appropriate capacity appeared, the effect was immediate.
The affected workload scheduled successfully.
The 503 disappeared.
Other workloads that had been waiting in Pending also began scheduling.
The immediate incident was resolved.
But manually increasing node capacity wasn't a permanent solution.
The next task was restoring autoscaling.
Restoring Karpenter
This turned out to be more involved than reinstalling a Helm chart.
Several objects from the previous Karpenter deployment still existed in the cluster, including service accounts, RBAC resources, services and custom resources.
Those objects needed to be reconciled with the new Helm-managed installation.
After getting the controller installed, I hit the next issue.
The controller started crashing.
The logs pointed to a missing resource type:
NodeClaim
That was a strong indication that the installed Karpenter controller and its Custom Resource Definitions (CRDs) were not aligned.
The controller expected Kubernetes to understand an API resource that wasn't available with the required schema.
After updating the relevant CRDs, the controller progressed further through startup.
Then the failure changed.
AWS returned:
AccessDenied
during:
AssumeRoleWithWebIdentity
At that point, the investigation moved from Kubernetes into AWS IAM.
Kubernetes identity and IAM trust had drifted apart
Karpenter was using a Kubernetes service account to assume an AWS IAM role.
The IAM trust relationship, however, expected that service-account identity in a different namespace from the namespace where the controller was actually running.
Both pieces looked reasonable on their own.
Together, they didn't match.
AWS therefore rejected the role assumption.
Once the trust relationship was aligned with the identity of the service account actually running the Karpenter controller, the controller started successfully.
It began discovering suitable instance types.
The node configuration became ready.
The NodePool became ready.
Autoscaling was operational again.
That was the point where the cluster finally regained the ability to react automatically when new workloads needed additional capacity.
One final surprise: the Helm upgrade looked successful
There was still one more piece of work.
Another environment was running an older Karpenter version, and I wanted both environments to behave consistently.
So I upgraded the second environment to the same version.
Helm reported a successful upgrade.
At first glance, everything looked fine.
But when I checked the controller image that was actually running, it was still using the older version.
The chart had been upgraded.
The controller image had not.
The cause was subtle.
The previous Helm configuration contained a pinned controller image tag and digest.
During the upgrade I used:
--reuse-values
Helm did exactly what I asked it to do.
It reused the previous values.
Unfortunately, that included the old image configuration.
So the release metadata moved forward while the actual controller image stayed behind.
The fix was to explicitly update the image version and remove the stale digest.
After that, both environments were finally running the same Karpenter version.
That incident reinforced something I try not to forget:
A successful deployment command is not the same thing as a verified deployment.
Always check what is actually running.
The troubleshooting sequence I would use next time
The exact commands will differ between environments, but the investigation pattern is reusable.
First, check actual node utilisation:
kubectl top nodes
Then inspect why a pod isn't scheduling:
kubectl describe pod <pod-name>
The Events section often tells you whether the scheduler is blocked by CPU, memory, taints, affinity rules, topology or something else.
Check how nodes are distributed across Availability Zones:
kubectl get nodes -L topology.kubernetes.io/zone
For Karpenter, confirm that its resources exist and are healthy:
kubectl get nodepool
kubectl get ec2nodeclass
Then check the controller itself:
kubectl get pods -n kube-system \
-l app.kubernetes.io/name=karpenter
If the controller isn't healthy, inspect its logs:
kubectl logs -n kube-system \
-l app.kubernetes.io/name=karpenter
The important shift is moving beyond:
How busy are my nodes?
and asking:
What resources has Kubernetes already reserved, what constraints must this workload satisfy, and which controllers should be reacting to the problem?
Those questions lead to very different troubleshooting paths.
What this incident reinforced for me
A few lessons stood out.
1. Low CPU utilisation does not mean schedulable CPU exists
A Kubernetes node can be almost idle while still having very little capacity available to the scheduler.
CPU usage and CPU requests answer different questions.
Usage tells you what workloads are consuming right now.
Requests tell Kubernetes what capacity it has already promised.
You need to look at both.
2. Existing configuration does not mean the control plane is functioning
The Karpenter-related resources were largely still there.
That made the environment look configured.
But without a healthy controller, the autoscaling control loop didn't exist.
A NodePool by itself cannot provision a node.
3. Storage topology matters
For workloads backed by zonal storage such as Amazon EBS, adding capacity anywhere in the region isn't necessarily enough.
The compute needs to exist in a location where the storage can attach.
That matters especially for stateful workloads.
4. Infrastructure drift can remain invisible until demand exposes it
The environment could continue operating without a Karpenter controller as long as the existing nodes had enough capacity.
Nothing necessarily failed immediately.
The missing controller became visible only when the cluster actually needed to scale.
That's what makes infrastructure drift dangerous.
It can sit quietly for a long time and only reveal itself during an incident.
5. Verify the running artifact, not just the deployment command
The Helm upgrade reported success while the controller image remained on the older version.
That is why deployment validation should include checking the actual image, pods and runtime state.
6. Be careful with --reuse-values
--reuse-values is convenient, but it can preserve configuration that you actually intended to replace.
Pinned image tags and digests are a good example.
Whenever I use it now, I want to know exactly which old values are being carried forward.
The bigger lesson
The part of this incident I found most interesting was the original contradiction:
Around 3% actual CPU usage, but Kubernetes still couldn't schedule workloads.
That contradiction disappeared once I stopped asking only:
How busy are these machines?
and started asking:
What capacity has Kubernetes already promised?
Kubernetes scheduling involves far more than current CPU utilisation.
It has to account for resource requests, topology constraints, storage placement, node availability and the controllers responsible for adding capacity when existing nodes can't satisfy a workload.
Sometimes an outage isn't caused by one spectacular failure.
It's several individually understandable conditions lining up at exactly the wrong time.
In this case:
- CPU requests had consumed almost all schedulable capacity.
- Autoscaling wasn't functioning.
- Persistent storage imposed an Availability Zone constraint.
- IAM configuration had drifted.
- CRDs needed alignment.
- A later Helm upgrade preserved an old controller image.
None of those observations on its own tells the whole story.
Together, they explain the outage.
And the next time I see pods stuck in Pending on a cluster that appears almost idle, one of my first questions will be:
What has Kubernetes already promised?
Have you ever encountered a Kubernetes cluster with low actual CPU usage but no schedulable CPU capacity?
I'd be interested to hear what caused it in your environment.
Top comments (0)