DEV Community

Cover image for Engineering Build Notes #2: When an Idle Kubernetes Cluster Still Couldn't Consolidate
Adedamola Ajibola
Adedamola Ajibola

Posted on

Engineering Build Notes #2: When an Idle Kubernetes Cluster Still Couldn't Consolidate

#devops #aws #kubernetes

How Kubernetes resource requests became a scheduling constraint in an Amazon EKS cluster.

Engineering Build Notes is a series where I document engineering decisions, platform improvements, and infrastructure trade-offs from real production environments.

In Engineering Build Notes #1, I started with what looked like a straightforward infrastructure cost problem.

The worker nodes appeared to have more capacity than the workloads were using.

The obvious optimization would have been to make them smaller.

The metrics told a different story.

One node was consistently running at approximately 98–100% CPU utilization, while memory utilization remained around 12–15%.

The bottleneck wasn't memory.

It was CPU.

So instead of downsizing the workers, I increased the node size:

t3.medium
    ↓
t3.large
Enter fullscreen mode Exit fullscreen mode

That removed the CPU constraint and gave the workloads more compute headroom.

But it also exposed another question:

If the cluster now had more capacity, why weren't we using that capacity more efficiently?

That became the next investigation.

The Problem

The cluster was running with Karpenter managing worker-node capacity.

After addressing the CPU bottleneck, I expected the additional capacity to create better opportunities for workload placement and node consolidation.

Instead, some nodes remained in the cluster even though their observed CPU and memory utilization looked relatively low.

I also saw Karpenter consolidation events while investigating the cluster.

At first, it was tempting to treat this as a Karpenter problem.

But before changing the Karpenter configuration, I wanted to understand the workloads running on those nodes.

That turned out to be the more useful path.

Utilization Wasn't the Whole Story

The first thing I compared was actual resource consumption vs Kubernetes resource requests.

One workload provided a useful example.

Its observed usage was approximately:

CPU:       1m
Memory:   17–23Mi
Enter fullscreen mode Exit fullscreen mode

But its configured request was significantly higher.

resources:
  requests:
    cpu: 250m
    memory: 250Mi
Enter fullscreen mode Exit fullscreen mode

At the time of observation, the workload was consuming far less than its configured requests. But Kubernetes doesn't schedule workloads based purely on what they happen to consume at a particular moment.

Resource requests are inputs Kubernetes uses when making scheduling decisions. That difference was important.

The Hidden Capacity Problem

This changed the way I looked at the cluster. From a monitoring dashboard, a node might look like this:

CPU:      10%
Memory:   25%
Enter fullscreen mode Exit fullscreen mode

The natural conclusion is:

This node has plenty of spare capacity.

But actual utilization isn't the only thing that matters for scheduling. Kubernetes also has to account for the resources requested by workloads, so we effectively had two different views:

Observed utilization
        ↓
      Low

Resource requests
        ↓
   Higher than
   actual usage
        ↓
 Less efficient
 workload packing
Enter fullscreen mode Exit fullscreen mode

This doesn't mean resource requests are the only factor involved in Karpenter consolidation.

Karpenter has to evaluate several scheduling and disruption constraints.

The important point was that oversized resource requests were one constraint we could directly control.

Don't Tune the Autoscaler First

This was an important engineering decision.

It would have been easy to start changing Karpenter configuration:

  • Make consolidation more aggressive
  • Change disruption settings
  • Change NodePool configuration
  • Try to force nodes out of the cluster

But that would have been changing the infrastructure layer before understanding the workload layer.

Instead, I followed the dependency chain:

Workload resource requests
            ↓
     Kubernetes scheduler
            ↓
    Workload placement
            ↓
 Consolidation opportunities
            ↓
   Karpenter decisions
Enter fullscreen mode Exit fullscreen mode

If the workload requirements aren't representative of actual behavior, changing the autoscaling layer may simply hide the underlying problem.

So I went one layer deeper.

Right-Sizing the Workloads

I reviewed the workloads and compared their configured resource requests with their observed usage.

Several services had requests that appeared significantly larger than their normal consumption.

The goal wasn't to make every request as small as possible.

It was to make the requests more representative of the workload while preserving reasonable operational headroom.

For example, one class of workload could move from something like:

Before

resources:
  requests:
    cpu: 100m
    memory: 512Mi

  limits:
    cpu: 800m
    memory: 1Gi
Enter fullscreen mode Exit fullscreen mode

toward:

After

resources:
  requests:
    cpu: 25m
    memory: 64Mi

  limits:
    cpu: 300m
    memory: 256Mi
Enter fullscreen mode Exit fullscreen mode

These were not blanket values applied to every application. Workloads were reviewed individually.

That distinction matters.

Right-Sizing Is Not Under-Sizing

There is an important difference between the two.

If an application is currently consuming 1m CPU, that doesn't automatically mean the correct production request is:

cpu: 1m
Enter fullscreen mode Exit fullscreen mode

Likewise, if a workload is currently using 20Mi memory, setting the request to exactly 20Mi could leave very little room for normal variation.

Production workloads experience:

  1. traffic changes
  2. startup behavior
  3. deployments
  4. background processing
  5. temporary bursts

So the objective was not:

Make resource requests as small as possible.

It was:

Make resource requests realistic enough to improve scheduling while retaining appropriate headroom.

That is a much safer definition of right sizing.

Validate Before Promoting

The changes were rolled out progressively and validated against workload behavior.

Staging provided useful evidence.

To keep the production environment anonymous, the workloads below
are represented by generic service names.

For example:

api-service-stg
CPU:       0m
Memory:   150–159Mi

backend-service-stg
CPU:       1m
Memory:    75Mi

web-service-stg
CPU:       0m
Memory:    17–19Mi

ussd-service-stg
CPU:       8m
Memory:   208Mi
Enter fullscreen mode Exit fullscreen mode

One observation was particularly important: not every workload was equally over-provisioned. Services had different resource profiles some were CPU-light, while others had higher memory requirements..

The ussd-service-stg workload, for example, had a noticeably different memory profile from the other staging services.

That reinforced the approach:

Right-size workloads individually instead of applying a blanket policy.

Production Validation

After the changes were promoted, production workloads continued to operate within expected ranges.

For example:

api-service-prod
CPU:       1m
Memory:   157–179Mi

backend-service-prod
CPU:       1m
Memory:    17–20Mi

web-service-prod
CPU:       0m
Memory:    31–62Mi

ussd-service-prod
CPU:      16m
Memory:   276Mi
Enter fullscreen mode Exit fullscreen mode

The numbers revealed a pattern: different applications had different resource profiles. Some used minimal CPU, others needed more memory. A workload-by-workload approach was more appropriate than a single global resource policy.

The Outcome

After right sizing the workloads and giving Kubernetes more realistic resource requirements, the worker-node footprint was reduced:

5 worker nodes
      ↓
4 worker nodes
Enter fullscreen mode Exit fullscreen mode

The important part wasn't simply that one node disappeared.

It was why the platform could operate with less capacity.

The sequence was:

CPU bottleneck removed
            ↓
More compute headroom
            ↓
Right-size workload requests
            ↓
Better scheduling flexibility
            ↓
5 worker nodes → 4 worker nodes
Enter fullscreen mode Exit fullscreen mode

We didn't need to force the autoscaler to behave differently.

We improved the scheduling inputs, which created better conditions for consolidation.

The Engineering Pattern

The investigation produced a pattern I now use when reviewing Kubernetes capacity:

1. Measure actual workload usage
           ↓
2. Compare to resource requests
           ↓
3. Adjust requests to be realistic
           ↓
4. Let the autoscaler respond
           ↓
5. Validate in production
Enter fullscreen mode Exit fullscreen mode

Don't tune the autoscaler first. Understand the workload first.

Utilization Isn't the Whole Scheduling Picture

One of the biggest lessons from this investigation was that Kubernetes utilization isn't a single number.

A monitoring dashboard answers one question:

What are workloads consuming right now?

Kubernetes scheduling asks a different question:

What resources have workloads requested when deciding where they can run?

And Karpenter then evaluates whether the existing capacity can be changed while respecting those scheduling constraints and other disruption and consolidation requirements.

So you can have:

Observed utilization  →  Low
Resource requests     →  Higher
Node appears idle     →  Consolidation may still be constrained
Enter fullscreen mode Exit fullscreen mode

That was the key insight from this investigation.

A useful way to think about the three layers is:

Observed utilization  →  What the workload is consuming

Resource requests     →  What Kubernetes schedules around

Karpenter             →  What capacity can potentially be consolidated
Enter fullscreen mode Exit fullscreen mode

The important point is that these are related, but they are not the same measurement.

A node can look idle from a monitoring perspective while still being difficult to consolidate.

A workload can look tiny while carrying oversized scheduling requirements. An autoscaler can then appear to be the problem when the real constraint exists one layer below it.

Key Takeaways

1. Requests Affect Scheduling

Resource requests influence workload placement. More realistic requests can improve workload packing and scheduling flexibility.

2. Different Workloads Need Different Profiles

Don't apply a single resource policy. Size each workload based on its actual behavior.

3. Measure Before Changing Infrastructure

Understand the scheduling constraints before changing autoscaler configuration.

4. Right-Size, Don't Under-Size

The goal isn't the smallest possible numbers. The goal is realistic capacity with appropriate operational headroom.

5. Validate the Change

Right-sizing is only an optimization if workloads remain stable and reliable after the change.

What I Took Away

Build Note #1 revealed a constraint at the node layer.

We had a CPU bottleneck, so we increased compute capacity:

t3.medium
    ↓
t3.large
Enter fullscreen mode Exit fullscreen mode

Build Note #2 revealed a constraint at the workload layer.

We reviewed and right-sized resource requests:

Oversized requests
        ↓
Realistic requests
        ↓
Better scheduling flexibility
Enter fullscreen mode Exit fullscreen mode

The pattern was the same:

Measure
   ↓
Identify the constraint
   ↓
Change the appropriate layer
   ↓
Validate
   ↓
Repeat
Enter fullscreen mode Exit fullscreen mode

The lesson wasn't that Karpenter needed more aggressive configuration.

It was that infrastructure automation operates within the constraints defined by the workloads and the platform.

Understand the constraint before changing the automation.

What's Next?

With workload requests right-sized and the worker-node footprint reduced, the next question was about the nodes themselves:

Were the worker nodes provisioned with more storage than they actually needed?

That became Engineering Build Notes #3.

200Gi → 80Gi

Measure first. Right-size second. Validate always.

Top comments (0)