A cost optimization exercise that challenged my assumptions about infrastructure sizing in Amazon EKS.
Engineering Build Notes is a series where I document engineering decisions, platform improvements, and infrastructure trade-offs from real production environments.
The Goal
Every infrastructure optimization starts with a goal.
Mine was straightforward:
Reduce the monthly AWS bill without compromising platform reliability.
The Kubernetes cluster was running on Amazon EKS with Karpenter managing worker nodes.
My first instinct was the same one many engineers would have:
Smaller EC2 instances should reduce infrastructure costs.
It sounded reasonable.
Before changing anything, I wanted the metrics to confirm the assumption.
They didn't.
Start With Evidence, Not Assumptions
Before making any infrastructure changes, I reviewed node utilization across the cluster.
One worker node immediately stood out.
Node: ip-10-0-2-31
CPU Utilization: 98–100%
Memory Utilization: 12–15%
Initial Observation
- CPU was saturated
- Memory still had significant headroom
That second metric mattered.
It's easy to look at low memory utilization and conclude that an instance is oversized.
But memory wasn't limiting the platform.
CPU was.
Reducing the instance size simply because memory looked underutilized would have made the real bottleneck even worse.
The metrics were telling a very different story.
The Engineering Decision
Instead of reducing capacity, I increased it.
t3.medium to t3.large
At first glance, that seems like the opposite of cost optimization.
Larger instances cost more.
So why would increasing infrastructure cost ever reduce an AWS bill?
Because infrastructure isn't optimized one resource at a time.
It's optimized as a system.
Looking Beyond Individual Resources
When CPU becomes saturated, the platform starts paying hidden costs.
Examples include:
- Increased scheduling pressure
- Less available compute for future workloads
- Reduced flexibility for autoscaling
- Poorer workload placement
Although the hourly cost of each worker node increased, the platform gained something far more valuable:
Headroom.
Removing the CPU bottleneck made the cluster healthier and created opportunities for future optimization.
One lesson stood out immediately:
The cheapest looking infrastructure is often the most expensive to operate.
Infrastructure rarely becomes cheaper by accident.
It becomes cheaper when bottlenecks are removed and resources are allowed to work more efficiently.
The Trade-Off
Every engineering decision involves trade-offs.
Increasing the node size wasn't free.
Each worker node became more expensive.
However, it also removed CPU contention and created room for further optimization across the cluster.
The trade-off wasn't:
Cheaper nodes vs. more expensive nodes
It was:
Higher cost per node vs. lower cost for the platform as a whole.
Those are very different engineering conversations.
One optimizes infrastructure.
The other optimizes systems.
Why Memory Wasn't the Problem
One question naturally came up during the review.
If memory utilization was only around 12%, why not choose a smaller instance?
Because utilization isn't the same as capacity planning.
CPU was consistently operating at its limit.
Memory wasn't.
Choosing infrastructure based solely on one underutilized resource ignores the actual bottleneck.
Platform engineering isn't about maximizing utilization percentages.
It's about removing the constraint that's limiting the system.
In this case, CPU not memory was the limiting factor.
The Outcome
After upgrading the worker nodes:
- CPU contention disappeared.
- Scheduling became more balanced.
- Applications gained additional compute headroom.
- The cluster became healthier under load.
If the story ended here, upgrading the instances would simply have increased infrastructure costs.
Fortunately, it didn't.
The Unexpected Result
Removing the CPU bottleneck gave the platform something it previously lacked:
Available compute headroom.
With additional compute headroom available, I reviewed the cluster more closely and uncovered two more opportunities.
The first was storage.
Every worker node was launching with significantly more root disk than it actually needed.
The second was application sizing.
Several workloads requested far more CPU and memory than they consistently consumed.
Those investigations deserve articles of their own, so I'll cover them separately in future Build Notes.
What mattered here was the outcome.
Removing the CPU bottleneck allowed Kubernetes and Karpenter to make better scheduling decisions across the cluster.
Conceptually, the optimization looked like this:
The final result wasn't simply larger EC2 instances.
It was a healthier cluster with fewer infrastructure constraints, less wasted storage, and better opportunities for workload consolidation.
That's when infrastructure costs finally started moving in the right direction.
The Bigger Lesson
One lesson stood out throughout this exercise.
Infrastructure should be optimized around bottlenecks not intuition.
Had I followed my original assumption, I would have reduced instance sizes simply because the goal was to lower costs.
The metrics showed that doing so would have increased CPU pressure while saving very little.
Instead, the data pointed toward a different decision.
Increase capacity first.
Remove the bottleneck.
Then optimize the rest of the platform.
Good engineering decisions begin with evidence.
Not assumptions.
A Simple Decision Framework
This optimization reinforced a simple framework that I now try to follow whenever reviewing infrastructure.
Notice what's missing.
Guesswork.
Key Takeaways
- Don't optimize based on assumptions.
- Measure before changing infrastructure.
- Identify the real bottleneck first.
- Every engineering decision involves trade-offs.
- Optimize the platform not individual resources.
Final Thoughts
One lesson stood out throughout this exercise.
Infrastructure cost optimization isn't about buying the smallest servers.
It's about understanding where the platform is constrained, removing those constraints, and allowing the system to operate more efficiently.
Sometimes that means buying bigger machines.
Sometimes it means buying fewer.
The important part is letting the data not intuition drive the decision.
Thanks for reading Engineering Build Notes #1.
Top comments (0)