Kubernetes Idle Cost: Find Zero-Traffic Workloads, 24/7 Dev Namespaces, and Orphaned PVCs (Then Scale Them to Zero)

💡 Originally published on devtocash.com — where this guide stays updated. I write hands-on DevOps/SRE deep-dives there weekly.

The 100-word answer

Kubernetes idle cost is money you pay for pods that request CPU and memory, hold a slot on a node, and serve nobody. It hides in three places: deployments that have not received a real request in a week, non-production namespaces that run 24/7 for a team that works 50 hours of the 168, and orphaned objects — PersistentVolumeClaims no pod mounts, LoadBalancer Services with no endpoints. Rightsizing does not fix any of these; a zero-traffic pod rightsized from 2 cores to 250m is still a pod nobody uses. The fix is replicas: 0 on a schedule, or deletion. Below: the PromQL and kubectl checks that find each bucket, the KEDA config that closes it, and the guardrails that stop it from biting.

Idle is not the same as inefficient

Every cost tool reports an efficiency ratio — usage divided by requests — and the FinOps agent I built earlier ranks workloads by it to propose smaller resources.requests. That is the right fix for a service that handles real traffic with 12% of the CPU it asked for.

It is the wrong fix for a service whose efficiency is 0.3% because its usage is genuinely zero. Shrinking its request from 2 cores to 250m saves 87% of a bill that should be 100% gone. Worse, the rightsized pod now looks healthy on the efficiency dashboard, so it survives the next review too. On the multi-team clusters I run, the idle bucket was consistently the second-largest line after oversized requests, and it was invisible precisely because rightsizing had already "handled" it.

So this post treats idle as its own category, with its own detector. Cost is computed from requests, not usage (the reason is covered in cost allocation per namespace), which means an idle pod costs exactly what it requests, every hour, until someone scales it down.

Bucket 1: deployments with zero real traffic

The signal is a seven-day average, never an instantaneous kubectl top. A batch service that runs for ten minutes a night looks idle at 3 pm and busy at 2 am; a week of history catches both.

Two metrics together separate "idle" from "quiet." CPU alone is not enough — a Java pod idles at 20 to 40 millicores just running its GC threads — so pair it with network receive bytes. Kubelet probes and Prometheus scrapes cost roughly a few hundred bytes per second; anything below about 1 KB/s over a week is a pod that only ever talks to its own health checks.

# Pods averaging under 5 millicores for a week
sum by (namespace, pod) (
  rate(container_cpu_usage_seconds_total{container!=""}[7d])
) < 0.005

# ...and receiving less than 1 KB/s (probes + scrapes only)
sum by (namespace, pod) (
  rate(container_network_receive_bytes_total[7d])
) < 1024

Calibrate the network threshold on a pod you know is idle in your cluster before trusting it. A pod with a 2-second liveness probe and three scrape jobs sits higher than one probed every 30 seconds.

The joins — pod to ReplicaSet to Deployment, plus requests for the cost line — are painful in PromQL and trivial in Python. This script prints a ranked table of idle deployments with their requested CPU and memory and a monthly cost estimate:

#!/usr/bin/env python3
"""Rank Deployments whose pods were idle (CPU + network) over the last 7 days."""
import re
import requests

PROM = "http://prometheus-server.monitoring.svc:80"
IDLE_CPU = 0.005     # cores, 7d average
IDLE_NET = 1024      # bytes/s, 7d average: probes and scrapes only
CORE_HOUR = 0.048    # blended $/vCPU-hour for your node pool (take it from OpenCost)
GIB_HOUR = 0.0065    # blended $/GiB-hour


def query(expr):
    r = requests.get(f"{PROM}/api/v1/query", params={"query": expr}, timeout=120)
    r.raise_for_status()
    return r.json()["data"]["result"]


def by_pod(expr):
    return {(s["metric"]["namespace"], s["metric"]["pod"]): float(s["value"][1])
            for s in query(expr)}


cpu = by_pod('sum by (namespace, pod) (rate(container_cpu_usage_seconds_total{container!=""}[7d]))')
net = by_pod('sum by (namespace, pod) (rate(container_network_receive_bytes_total[7d]))')
req_cpu = by_pod('sum by (namespace, pod) (kube_pod_container_resource_requests{resource="cpu"})')
req_mem = by_pod('sum by (namespace, pod) (kube_pod_container_resource_requests{resource="memory"})')
owner = {(s["metric"]["namespace"], s["metric"]["pod"]): s["metric"]["owner_name"]
         for s in query('kube_pod_owner{owner_kind="ReplicaSet"}')}

rows = {}
for key, c in cpu.items():
    if key not in owner or c >= IDLE_CPU or net.get(key, 0) >= IDLE_NET:
        continue
    ns, _pod = key
    deploy = re.sub(r"-[a-z0-9]{5,10}$", "", owner[key])   # strip the ReplicaSet hash
    row = rows.setdefault((ns, deploy), {"pods": 0, "cpu": 0.0, "gib": 0.0})
    row["pods"] += 1
    row["cpu"] += req_cpu.get(key, 0)
    row["gib"] += req_mem.get(key, 0) / 2**30


def monthly(row):
    return 730 * (row["cpu"] * CORE_HOUR + row["gib"] * GIB_HOUR)


print(f"{'namespace/deployment':48} pods  req_cpu  req_gib  usd/month")
for (ns, d), row in sorted(rows.items(), key=lambda kv: -monthly(kv[1])):
    print(f"{ns + '/' + d:48} {row['pods']:4d} {row['cpu']:8.2f} {row['gib']:8.2f} {monthly(row):10.0f}")

Two notes on running it. The [7d] range over every container is a heavy query; run it off-peak, or against Thanos or Mimir if you have long-term storage, and raise --query.max-samples if Prometheus refuses. And the dollar rates are placeholders: pull the real blended per-core and per-GiB rate for your node pool from the same allocation tool you use for showback (setup in OpenCost vs Kubecost); on-demand list price overstates the number on any cluster with Savings Plans or spot.

The first run on a shared dev cluster typically surfaces the same cast: a preview-* deployment from a branch merged in March, a second copy of a service someone deployed under a new name while migrating, and an internal admin UI that three people used once. None of them showed up in the efficiency report, because VPA had already shrunk their requests to the floor.

Bucket 2: non-production namespaces that never sleep

Do the arithmetic for a dev namespace once and it changes how you look at every cluster. A week has 168 hours. A team that works 07:00 to 19:00 on weekdays uses 60 of them. The other 108 hours — 64% of the namespace's cost — buy nothing. Staging environments that only see traffic during CI runs are worse.

The fix is scheduled scale-to-zero. Two tools do this well; pick one, not both.

KEDA's cron scaler is the right choice if you already run KEDA (the tradeoffs versus HPA and VPA are in HPA, VPA, and KEDA explained). Outside the window it holds the deployment at minReplicaCount; inside it, at desiredReplicas:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: api-office-hours
  namespace: dev-payments
spec:
  scaleTargetRef:
    name: api
  minReplicaCount: 0
  maxReplicaCount: 2
  cooldownPeriod: 300
  triggers:
    - type: cron
      metadata:
        timezone: Asia/Jakarta
        start: 0 7 * * 1-5
        end: 0 19 * * 1-5
        desiredReplicas: "2"

KEDA owns the HPA it creates for that target, so delete any hand-written HPA on the same deployment first, or the two controllers fight over the replica count.

kube-downscaler is simpler when you want one rule for a whole namespace and no per-deployment YAML. A single annotation on the namespace scales every Deployment and StatefulSet in it to zero outside the window and restores the previous replica count inside it:

kubectl annotate namespace dev-payments \
  'downscaler/uptime=Mon-Fri 07:00-19:00 Asia/Jakarta'

# Opt a specific workload out (a shared mock server other teams hit at night)
kubectl annotate deployment -n dev-payments mock-bank 'downscaler/exclude=true'

Either way, the saving only materialises when nodes go away. Scaling 40 pods to zero on a cluster whose autoscaler never removes the now-empty nodes saves exactly nothing. Check the node count over a week and look for the evening drop:

count(kube_node_info)

If the line is flat, the usual culprits are a DaemonSet-only node that Cluster Autoscaler refuses to remove because a pod has local storage, a cluster-autoscaler.kubernetes.io/safe-to-evict: "false" annotation, or a Karpenter NodePool with consolidation disabled. Karpenter's consolidation is materially better at this than Cluster Autoscaler's scale-down, and that gap is most of the cost difference I measured in Karpenter vs Cluster Autoscaler.

One honest limit: cold starts. A dev environment scaled to zero at 19:00 will take 2 to 4 minutes to come back when someone opens a PR at 21:00 and CI needs it, because a node has to be provisioned first. Set start fifteen minutes before the team's actual start time and accept that the occasional late-night deploy waits. Do not apply this pattern to anything user-facing.

Bucket 3: orphans that outlive their workloads

The third bucket does not show up in any per-pod metric because there is no pod. These are the leftovers of deleted workloads, and they cost real money every month.

PersistentVolumeClaims no pod mounts. Deleting a Deployment does not delete its PVCs, and a StatefulSet's volumeClaimTemplates outlive the StatefulSet by design. On AWS, a gp3 volume bills $0.08 per GB-month whether or not anything is attached, so a forgotten 500 GB database volume is $40 a month, forever.

# PVCs referenced by any running pod
kubectl get pods -A -o json \
  | jq -r '.items[] | .metadata.namespace as $ns
           | .spec.volumes[]? | select(.persistentVolumeClaim)
           | "\($ns)/\(.persistentVolumeClaim.claimName)"' \
  | sort -u > /tmp/pvc-in-use.txt

# Every PVC, with size and class
kubectl get pvc -A -o json \
  | jq -r '.items[] | "\(.metadata.namespace)/\(.metadata.name) \(.spec.resources.requests.storage) \(.spec.storageClassName)"' \
  | sort > /tmp/pvc-all.txt

# The difference: bound but unmounted
awk 'NR==FNR {used[$1]=1; next} !($1 in used)' /tmp/pvc-in-use.txt /tmp/pvc-all.txt

Read the output before acting on it. A StatefulSet scaled from 3 replicas to 1 leaves data-db-1 and data-db-2 unmounted, and they are reattached the moment it scales back up. A CronJob that mounts a PVC for twenty minutes a day looks orphaned 23 hours out of 24. Cross-check against kubectl get statefulset and kubectl get cronjob in that namespace, and snapshot before deleting anything you are not certain about.

LoadBalancer Services with no endpoints. Every type: LoadBalancer Service on a cloud provider is a real NLB or ALB with a fixed hourly charge, around $16 to $22 a month before traffic. When the Deployment behind it is deleted, the Service and the load balancer stay.

kubectl get endpoints -A -o json \
  | jq -r '.items[] | select((.subsets // []) | length == 0)
           | "\(.metadata.namespace)/\(.metadata.name)"' \
  | sort > /tmp/no-endpoints.txt

kubectl get svc -A -o json \
  | jq -r '.items[] | select(.spec.type == "LoadBalancer")
           | "\(.metadata.namespace)/\(.metadata.name)"' \
  | sort > /tmp/lb.txt

comm -12 /tmp/no-endpoints.txt /tmp/lb.txt

The cloud-side twins of these — unattached EBS volumes and idle load balancers the cluster no longer knows about at all — need the provider API rather than kubectl, and I covered that sweep in the AWS waste reclamation agent. The two lists overlap less than you would expect: a PVC deleted with the wrong reclaim policy vanishes from Kubernetes and lives on in EC2.

Guardrails before you automate any of it

The detection above is safe to run anywhere. The remediation is not, and the failure mode is specific: a script that scales the wrong thing to zero at 19:00 on a Friday produces an outage nobody notices until Monday. Three rules keep it boring.

Scope by label, never by name pattern. The downscaler annotation and the KEDA objects go only on namespaces carrying env=dev or env=staging. Enforce that label at namespace creation with your policy engine so a new namespace cannot be "neither," and make the idle report refuse to propose scaling anything outside those labels.

Never auto-delete storage. The PVC and LoadBalancer sweeps output a list; a human deletes. If you want an agent in the loop, hand it the report as a read-only tool and let it open a pull request that removes the manifest from git, which is the same PR-not-kubectl contract every other write on this site goes through. The reviewer sees the PVC name, its size, the last pod that mounted it, and the snapshot ID taken before the change.

Prove the nodes moved. Report the weekly node-hours alongside the pod count. A scale-to-zero program that cut pods by 60% and node-hours by 4% has found a scheduler problem, not a saving, and the honest number is the one that shows up on the invoice.

What this does not fix

Idle is one bucket. Oversized requests on busy services are a bigger one, and the FinOps agent handles those as rightsizing PRs. Poor bin-packing — nodes that are 40% requested because pod sizes do not tile — is a scheduler and NodePool problem. And none of this tells a team what they spent; that is allocation and showback, which has to come first, because the fastest way to get a namespace scaled to zero at night is to show its owner the 108 hours they are paying for.

Run the idle script once. If the first table is short, congratulations, and the sweep costs you five minutes a month. In my experience it is not short.

---

📌 Read the latest version of this guide — plus the full library of DevOps, SRE, Kubernetes, observability & cloud-cost guides — on devtocash.com.