TL;DR: In Kubernetes you don't pay for what your apps use — you pay for what they request, plus the nodes those requests force you to run. The fix is a loop, not a tool: measure allocation vs. real usage, rightsize requests, let the autoscaler actually shrink the fleet, put fault-tolerant work on spot capacity, hunt the orphaned storage, and show every team its number. Do those six things and the cluster bill drops without a single performance meeting.
Originally published on DevOps Enginer.
The bill is the best teacher
I learned cloud costs the honest way: on my own wallet. When I was teaching myself AWS, I spun up resources to try things — and the monthly bill taught me the cleanup lesson no tutorial had. Unattached volumes, old snapshots, buckets with no lifecycle rule: the waste was never in what I used, it was in what I had reserved and forgotten.
Kubernetes has the same trap, built into its core abstraction. A pod's resource requests are a reservation. The scheduler counts them whether the container uses the CPU or not, and your node fleet — the thing you actually pay for — is sized to fit the sum of reservations. A cluster where every app requests 2 CPUs and uses 0.2 is a cluster where 90% of the bill is payment for intentions.
This guide is the systematic cleanup: where Kubernetes money actually leaks, in the order I would hunt it.
Step 1: Measure allocation vs. usage before touching anything
Never optimize from vibes. You need two numbers per workload:
- Allocated — the sum of requests (this is what you pay for).
-
Used — the real consumption, from metrics-server or Prometheus (
kubectl topfor a spot check; a week of history for decisions).
The gap between them, per namespace and per workload, is your savings estimate. Cost-visibility tools exist precisely to draw this picture: OpenCost (open source, CNCF) and Kubecost are the well-known names, and every major cloud's managed Kubernetes now surfaces some per-namespace cost view. Whichever you use, the first report to read is "requested vs. used, by workload, sorted by gap." The top of that list is your roadmap. (If your requests are missing entirely, start with our requests and limits guide — unmeasured workloads can't be rightsized, only guessed at.)
Step 2: Rightsize the workloads — the biggest, safest win
Most clusters carry requests set by fear in the project's first week and never revisited. The rightsizing loop:
- Take a week (better: a month, covering a peak) of real usage per container.
- Set the request near the observed p95 usage, with headroom you can defend.
- Set limits deliberately — memory limits matter (too low = OOMKilled; see the troubleshooting guide), CPU limits are a genuine judgment call.
- Re-measure after a week. Rightsizing is a loop, not an event: usage drifts as features ship.
The Vertical Pod Autoscaler in recommendation mode ("Off") exists to do step 1–2's math for you: it watches usage and recommends requests without changing anything. Run it in observe-only mode first, take its recommendations as a starting hypothesis, and apply them through your normal change process — through git and review if you run GitOps, exactly as our Argo CD tutorial sets up.
Expect pushback: "what if we get a traffic spike?" The honest answer is that requests are not capacity insurance — replicas and autoscaling are. One replica requesting honestly + an HPA that can scale beats three replicas squatting on reservations, on both cost and resilience.
Step 3: Make the node fleet able to shrink
Rightsizing pods saves nothing until nodes can disappear. Three mechanisms, in order:
- Cluster Autoscaler / Karpenter must be allowed to scale down. Scale-down disabled "for safety" is the most expensive checkbox in Kubernetes. Keep sensible cooldowns, but let empty nodes die.
- Bin packing. Scattered half-empty nodes are usually a scheduling-shape problem: oversized requests fragment the fleet. Rightsizing (Step 2) is the fix; node consolidation features (Karpenter's is the well-known one on AWS) continuously repacks pods onto fewer nodes and terminates the empties.
- Right node shapes. A fleet of huge instances wastes more per stranded fragment than right-sized smaller ones — but too-small nodes drown in per-node overhead (system pods, DaemonSets on every node). Match instance families to your actual workload mix, and re-check after rightsizing, not before.
Step 4: Replicas, schedules, and the quiet hours
Production traffic has a shape; dev clusters have an even simpler one — nobody uses them at 3 a.m. on a Sunday.
- Non-production environments: scale to zero on a schedule. A CronJob that scales dev/staging deployments to 0 replicas at night and back up in the morning, or an event-driven autoscaler like KEDA for scale-to-zero patterns, routinely erases a third of a non-prod bill. Whole dev clusters can be deleted on Friday and recreated Monday if your infrastructure is actually code (see our Terraform state guide for doing that safely).
- HPA on the right signal. Replica counts fixed for peak hour are peak-priced all day. Autoscale on CPU/memory as a floor; on queue depth or request rate where you have them.
- Minimum replicas are a cost decision. Three replicas of an internal tool used by twelve people is a choice, not a law. Make it consciously.
Step 5: Spot capacity for work that can survive interruption
Spot (AWS), Spot VMs (Azure), and preemptible/spot instances (GCP) sell spare cloud capacity at steep discounts in exchange for one condition: it can be reclaimed with short notice.
Good candidates: batch jobs, CI runners, image processing, stateless web tiers behind a healthy on-demand base, and dev/test anything. Bad candidates: the single replica of something stateful that pages you when it dies. The mature pattern is a mixed fleet: an on-demand floor sized for minimum viable service, spot capacity absorbing the elastic part, with node labels/taints steering the interruption-tolerant pods to spot. Keep the percentage honest and revisited — spot is a discount on flexibility you actually have, not on flexibility you wish you had.
Step 6: The quiet costs — storage, snapshots, and orphans
This is my old AWS lesson wearing a Kubernetes costume. Clusters accumulate:
-
Orphaned PersistentVolumes — the StatefulSet was deleted in March; its volumes (and the cloud disks behind them) are still billing you in October. Audit
ReleasedPVs and unattached cloud volumes monthly. - Old snapshots and images — backup snapshots with no retention policy and registry images from a hundred CI runs ago. Lifecycle rules exist for both; the cost of setting them is an afternoon, once.
-
Load balancers per forgotten service — every
type: LoadBalancerService is a real cloud load balancer with a real hourly price. Staging environments are where they go to be forgotten. List them; ask each one what it fronts.
None of this is glamorous. All of it is the exact shape of waste I once paid for personally — reserved, forgotten, quietly invoiced.
Step 7: Showback — make the number visible to its owner
Cost work dies in a platform team's spreadsheet. It lives when teams see their own number:
-
Labels and namespaces are the accounting system. Consistent
app/team/envlabels (the Helm guide covers enforcing them via chart helpers) let OpenCost or your cloud console split the bill by owner. - A monthly one-page report per team: their allocation, their usage, their top three gaps. No blame — most teams fix their own waste the first month they can see it.
- Guardrails, not gates. ResourceQuotas per namespace act as soft budgets: the quota conversation ("we need more") forces the usage conversation ("here's what we're using") at exactly the right moment.
The weekly 30-minute cost routine
- Glance at allocation vs. usage for the top five namespaces — any new gaps?
- Check for nodes running under ~30% allocation for a week — consolidation or shape problem?
- List orphaned PVs, unattached volumes, and load balancers with no backends.
- Look at one rightsizing recommendation list (VPA or your cost tool) and ship one change through review.
- Note the cluster's cost per namespace trend. One line. Every week.
Small, boring, compounding — the same shape as every other operational habit that works.
Mistakes that keep clusters expensive
- Optimizing usage instead of allocation. Turning down a container's actual CPU use changes nothing if its request — the reservation you pay for — stays put.
- Rightsizing once, in a panic, before a budget review. Numbers taken from an atypical week produce OOMKills in a typical one. Measure across a peak.
- Disabling scale-down after one noisy incident. Fix the cooldown and the disruption budgets; don't amputate the mechanism that makes rightsizing pay.
- Spot for everything. The discount is real; so is the 3 a.m. page when a stateful singleton gets reclaimed.
- Deleting "unused" things without an owner check. The orphan audit finds candidates; a human who owns the namespace confirms. My CloudTrail habit from learning AWS applies verbatim: look at what created it and when, before you touch it.
- Treating cost as the platform team's job. The platform team owns visibility and guardrails. The requests live in application repos — the savings do too.
FAQ
What's the difference between this and general cloud cost optimization?
The unit of waste differs. In plain cloud, you hunt idle instances and orphaned storage. In Kubernetes you additionally hunt reservations: requests, replica counts, and node fragmentation. Our AWS cost optimization guide covers the cloud layer; this one covers the Kubernetes layer stacked on top of it.
Should I use Kubecost/OpenCost or my cloud's built-in view?
Start with whatever you can open today — a built-in view beats a perfect tool installed next quarter. Move to OpenCost/Kubecost when you need multi-cluster or multi-cloud splits, or cost by label dimensions your cloud view doesn't offer.
Will rightsizing hurt performance?
Done from measured p95 usage with headroom, no — requests near real usage don't throttle anything; limits and replica counts are where performance actually lives, and those are separate decisions made deliberately, not as a side effect of a stale request from 2023.
How much can this realistically save?
Anyone who quotes you a percentage for your cluster is selling something. The honest method: your own allocation-vs-usage report states the ceiling (the total gap), and a conservative first pass typically closes a meaningful share of it. Measure your gap; that's your number.
Is scale-to-zero safe for staging?
For stateless services with infrastructure-as-code and no irreplaceable local data, yes — and it's the single largest non-prod saving for most teams. Stateful staging data is the exception to engineer around, not a reason to keep everything warm forever.
Conclusion
Kubernetes cost is not a mystery and not a tooling purchase. It is reservations, audited regularly: requests matched to measured usage, a fleet allowed to shrink, replicas that follow the clock, spot capacity where interruption is cheap, orphans swept monthly, and every team able to see its own number. That is the same lesson my own AWS bill taught me years ago — you pay for what you reserve and forget — just running at cluster scale. Start with the allocation-vs-usage report this week; the top line of it is your first fix.
Top comments (0)