The Kubernetes bill nobody looks at until finance does

Kubernetes does not send you a warning before it quietly costs you three times what it should. It just runs every pod exactly as requested, every replica exactly as configured and the bill arrives a month later as a single number nobody can immediately explain.

I have spent ten years across AWS, Azure, and Kubernetes (AKS, EKS, ARO) and the pattern behind most "why is our cloud bill so high" conversations is almost never one dramatic mistake. It is four small, boring, easy-to-miss things, quietly compounding across every service in the cluster.

Here they are - with a free calculator and a set of scripts at the end so you can check your own cluster in the next ten minutes.

The core thing to understand: requests, not usage, drive cost

This is the single most important idea in this article, so it goes first: Kubernetes bills you for what you ask for, not what you use.

When a pod requests 500m CPU and 512Mi memory, the scheduler reserves that exact capacity on a node - permanently, for as long as the pod runs - whether the pod is idle or maxed out. A node's size, and therefore its cost, is determined by the sum of requests scheduled onto it, not by real-time usage graphs.

This is why a Grafana dashboard showing "12% average CPU utilization" and a cloud bill that keeps climbing aren't in tension. They're both true, and they're describing the same problem from two different angles.

Mistake 1 - requests set once, in a hurry, and never revisited

Someone writes the first version of a deployment manifest. They don't know the real usage pattern yet, because the service doesn't exist yet, so they guess generously - "let's give it 1 CPU and 1Gi, we can tune later." It ships. It works. Nobody tunes it later, because it's working, and "working" doesn't create a ticket.

Multiply that single generous guess across every microservice a team ships, and you get a cluster where every workload is sized for a worst case that rarely arrives.

The fix isn't guessing better - it's measuring. Run kubectl top pods against your actual traffic for a week or two, then compare it to requests. The gap is your first real answer to "where does our cloud spend actually go."

kubectl top pods --all-namespaces --sort-by=cpu

Compare that against:

kubectl get pods --all-namespaces -o custom-columns=\
NAME:.metadata.name,CPU_REQ:.spec.containers[*].resources.requests.cpu,MEM_REQ:.spec.containers[*].resources.requests.memory

Mistake 2 - orphaned storage and empty load balancers

These are the sneakiest of the four, because they don't show up in CPU or memory dashboards at all. They're not compute problems - they're forgotten housekeeping.

A PersistentVolumeClaim survives its pod. Delete the deployment, forget the PVC existed, and the underlying disk keeps being billed every month, silently, until someone audits storage specifically. A LoadBalancer-type Service can end up with zero backing endpoints - the pods behind it were scaled down or deleted, but the cloud load balancer it provisioned is still running and still billing, whether or not a single packet reaches it.

Neither shows up as "high CPU" or "high memory." They show up as a line item nobody remembers approving.

# PVCs not currently mounted by any pod
kubectl get pvc --all-namespaces

# LoadBalancer services worth checking for zero endpoints
kubectl get svc --all-namespaces --field-selector spec.type=LoadBalancer

(The toolkit at the end automates the actual "is this one orphaned" comparison - doing it by hand across a big cluster gets tedious fast.)

Mistake 3 - non-prod environments running like it's prod

Dev and staging namespaces get created by copying the closest working example - which is usually the production manifest, replica count and all. Nobody consciously decided "this dev environment should run 3 replicas of everything, 24 hours a day, 7 days a week, including the entire weekend when literally nobody is looking at it." It just inherited that shape from prod and nobody revisited the decision.

Run at business hours only (roughly 45 hours/week) instead of continuously (168 hours/week), and a non-prod environment can cost well under half of its always-on price - often closer to a quarter - for identical infrastructure. A scheduled scale-to-zero is usually a single CronJob:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: scale-down-dev
spec:
  schedule: "0 19 * * 1-5"   # 7pm weekdays
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: kubectl
              image: bitnami/kubectl
              command: ["kubectl", "scale", "deployment", "--all", "--replicas=0", "-n", "dev"]
          restartPolicy: OnFailure

(Pair it with a matching scale-up CronJob for 8am, obviously - nobody wants to explain to the team why dev is down every morning.)

Mistake 4 - no autoscaling, so peak-sizing runs all day

Without a HorizontalPodAutoscaler, a deployment runs at whatever fixed replica count someone typed once - usually sized for the worst moment it needs to handle, running continuously even during the other 20 hours a day when traffic is a fraction of that peak.

The fix is one resource, and it pays for itself continuously once it exists:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-service
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-service
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

What this actually costs - a worked example

Take a genuinely ordinary microservice: 500m CPU and 512Mi memory requested, 3 replicas, running at roughly 35% real average utilization - not a horror story, just a typical unreviewed service. At AWS on-demand list pricing, that's roughly $35-40/month requested, of which roughly $23-26/month sits unused at that utilization level.

One service, ~$25/month wasted. Doesn't sound alarming. Now multiply it across the 40, 80, or 200 microservices a mid-sized platform team actually runs, and the number stops being trivial - it becomes the kind of line item a finance team asks about by name.

I built a small calculator so you can plug in your own numbers instead of trusting mine: PodTab → - type in requests, replicas, and real usage percentage, get an itemized monthly and annual figure.

Check your own cluster in the next ten minutes

Everything above is diagnosable with kubectl alone - no new tooling, no agent installed in your cluster, nothing sent anywhere. I packaged the checks I run into three small scripts: one flags over-provisioned pods against real usage, one finds orphaned PVCs and empty load balancers, one finds deployments with no autoscaler and non-prod namespaces quietly running at prod scale.

Free version: build the checks yourself from the kubectl commands above - everything you need is in this article. Packaged version, if you'd rather not assemble it: Kubernetes Cost Audit Toolkit →.

Either way - go check. The waste in most clusters isn't dramatic. It's four small, boring things, quietly compounding, waiting for someone to actually look.

---

I'm a DevOps engineer with 13 years across AWS, Azure, Kubernetes, Terraform and Ansible. Previously: Terraform state mistakes and a self-healing pipeline with Prometheus. I also maintain CronPort, a free cron expression converter, and sell a Terraform AWS Starter Kit.