49% of organizations running Kubernetes in production report rising infrastructure costs since adoption — 17% say the increase has been significant. If you've felt that creep on your own cluster's bill and immediately started reading about spot instances, Karpenter, and multi-year commitment discounts, you're not wrong to look there eventually. But you're skipping a step.
Before any of that, there's a "Phase 1" cleanup that most teams never do properly — and on its own, it typically recovers 30–50% of wasted spend with zero architectural change and no new tooling budget. This is that cleanup.
Why Kubernetes Bleeds Money Quietly
Kubernetes doesn't waste money the way a forgotten EC2 instance does. It wastes money structurally: every pod you deploy reserves resources whether it uses them or not, every PVC you delete doesn't always delete its underlying volume, and every namespace you spin up for a two-week experiment quietly keeps billing you eight months later. None of this shows up as an alarm. It shows up as a bill that's 40% higher than your actual workload needs, with no single line item to blame.
That's exactly why "Phase 1" gets skipped — there's no dramatic failure forcing the fix, just slow, distributed waste across hundreds of pods.
Phase 1: The Cleanup
- Fix your resource requests and limits — not your node count
The single biggest lever is also the most boring: most teams set requests far above actual usage "just to be safe," which forces the scheduler to over-provision nodes for capacity nothing is using.
resources:
requests:
cpu: "250m"
memory: "256Mi"
limits:
cpu: "500m"
memory: "512Mi"
Don't guess these numbers. Pull 2–4 weeks of actual usage first:
kubectl top pods --all-namespaces --sort-by=cpu
kubectl top pods --all-namespaces --sort-by=memory
Or better, run the Vertical Pod Autoscaler in recommendation-only mode (updateMode: "Off") for a couple of weeks before touching anything manually — it will tell you exactly where your requests are inflated without changing live behavior.
- Hunt down orphaned PersistentVolumes
When a PersistentVolumeClaim is deleted, the underlying cloud disk (EBS, PD, etc.) doesn't always go with it — depending on your reclaimPolicy, it can sit there billing you indefinitely with nothing attached.
kubectl get pv --all-namespaces -o json | \
jq -r '.items[] | select(.status.phase=="Released") | .metadata.name'
Anything showing Released (not Bound) is disk you're paying for with nothing using it. This alone has recovered four-figure monthly savings on clusters that have been running for a year or more without cleanup.
- Kill idle namespaces and their entire footprint
Every ephemeral environment — feature branches, load-test namespaces, "temporary" staging clones — needs a hard expiry. If your team doesn't already do this, a simple label-based sweep gets you most of the way:
kubectl get namespaces -l environment=temporary -o json | \
jq -r '.items[] | select(.metadata.creationTimestamp < "'$(date -d '14 days ago' -Iseconds)'") | .metadata.name'
Pipe that into a cleanup job that runs weekly, and "temporary" namespaces actually stay temporary.
- Set a cluster autoscaler floor, not just a ceiling
Teams tune the autoscaler's max node count religiously to avoid runaway costs, but rarely tune the minimum. An overly generous minimum node count means you're paying for idle capacity 24/7 even during nights, weekends, and low-traffic windows — recompute what your true off-peak floor should be, not what felt safe when the cluster was first provisioned.
Cost Allocation: Tag Before You Optimize
You can't optimize what you can't attribute. Before chasing savings further, put a consistent put a consistent cost allocation and tagging taxonomy on every workload:
┌─────────────┬────────────────────┬──────────────┐
│ Label │ Purpose │ Example │
├─────────────┼────────────────────┼──────────────┤
│ team │ Who owns the cost │ payments │
├─────────────┼────────────────────┼──────────────┤
│ application │ What it belongs to │ checkout-api │
├─────────────┼────────────────────┼──────────────┤
│ environment │ Prod vs. non-prod │ production │
├─────────────┼────────────────────┼──────────────┤
│ cost-center │ Finance mapping │ cc-4021 │
└─────────────┴────────────────────┴──────────────┘
Enforce it with an admission policy (Kyverno or OPA Gatekeeper both work well) so nothing gets deployed without these labels. Without this, your Phase 2 optimization work — rightsizing by team, chargeback, showback — has nothing to stand on.
Tooling That Actually Helps at This Stage
You don't need a cost-management platform to do Phase 1 — kubectl top, VPA in recommendation mode, and a scheduled cleanup job cover most of it. But once the taxonomy above is in place, these make ongoing visibility much easier:
- Prometheus + Grafana — for resource-usage-over-time dashboards, so "right-sized" doesn't become "wrong-sized again in three months."
- Kubecost (or OpenCost, its open-source core) — for allocating actual spend back to the team/application labels above, turning your tagging taxonomy into real chargeback numbers.
Start with the open-source options before evaluating anything paid — Phase 1 is specifically the phase where tooling spend isn't the bottleneck yet. If you do eventually want a comparison of where paid platforms differentiate, this rundown of cloud cost management tools is a reasonable starting point.
The Checklist
Before you touch autoscalers, spot instances, or commitment discounts, confirm you've done this:
- [ ] Pulled 2+ weeks of real usage data before setting any resource request/limit
- [ ] Run VPA in recommendation mode on at least your top 10 highest-cost workloads
- [ ] Queried for Released PVs and reclaimed orphaned disks
- [ ] Set an expiry policy on ephemeral/temporary namespaces
- [ ] Re-evaluated your cluster autoscaler's minimum, not just its maximum
- [ ] Enforced a team/application/environment/cost-center labeling policy via admission control
Most teams find their 30–50% before they ever get to Phase 2. The advanced stuff — Karpenter, spot orchestration, savings plans — compounds on top of this, but it can't fix what a bad baseline already wasted.
For a deeper walkthrough with worked cost-savings examples for high-growth teams specifically, this guide (https://www.cloudkeeper.com/insights/blog/kubernetes-cost-optimization-complete-guide-high-growth-companies) is a solid next read.
Top comments (0)