DEV Community

Muskan Bandta
Muskan Bandta

Posted on

Which Team Is Eating Your Kubernetes Bill? Cost Attribution Without Perfect Tagging

The Kubernetes bill arrives as one number. The cluster is shared. Ten teams run on it. And when finance asks "who is spending this," the honest answer in most orgs is a shrug. That shrug is the single biggest reason Kubernetes cost never gets fixed: nobody owns a number they cannot see.

I have built cost attribution on a shared cluster the messy, real-world way, without a perfect tagging scheme, because you never get a perfect tagging scheme. Here is what actually works.

Why the cloud bill can't answer this

Your cloud provider bills you for nodes (EC2, GCE, VMs). It has no idea that node ran 40 pods belonging to 6 teams. The unit of cloud billing is the instance; the unit of Kubernetes work is the pod. Attribution is the job of translating node cost down to pod cost and back up to whoever owns the pod. Nothing in the raw cloud bill does that for you.

So the goal is: split each node's cost across the pods that ran on it, by how much of the node they actually used, then group those pods by owner.

Step 1: Split node cost down to pods

The standard method is to allocate a node's hourly cost across its pods by their resource requests (or usage, whichever is higher). If a pod requests 2 of a node's 8 CPUs, it "owns" a quarter of that node's cost for the time it ran.

This is exactly what Kubecost and its open-source core OpenCost do out of the box, and it is the sane place to start rather than hand-rolling it. They read node prices, pod requests/usage, and produce a per-pod cost. If you want the DIY version, the inputs are: node hourly price, pod CPU/memory requests, and runtime, joined from the metrics server and your cloud pricing.

One important nuance: allocate on requests, not just usage, at least partly. A pod that requests 4 CPUs and uses 0.1 still reserved 4 CPUs that nothing else could schedule onto. Billing it only for the 0.1 it used lets over-requesting teams hide, and over-requesting is the number one cause of cluster waste. Charge for what you reserved.

Step 2: Group pods by owner without perfect labels

Now the messy part. In theory every pod has a team label and you group by it. In practice half of them do not, because the label was added later, or a Helm chart does not propagate it, or someone shipped in a hurry. Do not wait for 100% label coverage. Instead, build owner attribution from a waterfall of signals, most authoritative first:

  1. An explicit team or cost-center label if present.
  2. Otherwise the namespace. On most clusters namespaces already map roughly to teams or services, so namespace is a strong fallback owner.
  3. Otherwise the controller name prefix (deployment/statefulset naming usually encodes the service).
  4. Whatever is left lands in an "unallocated" bucket that you report loudly.

That unallocated bucket is the trick. Do not hide it. Put it on the report at the top. Nothing drives label adoption faster than a team lead seeing "$4,200 unallocated" and wanting to know if it is theirs. The report becomes the forcing function that fixes your tagging, instead of you nagging people to tag first.

Step 3: Don't forget shared and idle cost

Two line items people forget, and they are usually 20-40% of the cluster:

  • Shared overhead. kube-system, the ingress controller, monitoring agents, service mesh sidecars. This is real cost that belongs to everyone. Split it across teams proportionally to their allocated cost (or per-namespace), and label it as shared so nobody thinks you invented it.
  • Idle cost. The gap between what nodes cost and what pods actually requested. If your nodes are 55% utilized, 45% of the bill is idle capacity. Idle is not a team's fault, it is a cluster-efficiency problem (bin-packing, autoscaler tuning), so report it separately and own it at the platform level. Blaming teams for idle they cannot control kills trust in the whole model.

Step 4: Turn the number into a behavior change

Attribution is worthless if it just sits in a dashboard. What made it stick for us:

  • A monthly per-team cost line, sent to team leads, showing their trend. Trend matters more than the absolute number.
  • Requests vs usage per team. The teams costing the most are almost always the ones over-requesting, and this view shows them exactly where to right-size. This is where the actual savings come from, not from the attribution itself but from the rightsizing it triggers.
  • The unallocated number shrinking as a shared team goal.

We wire scheduling and rightsizing recommendations off the same allocation data (that closed loop is part of what ZopNight does), but even a spreadsheet built from OpenCost output beats the shrug.

The point

You do not need perfect tags to attribute Kubernetes cost. You need a sensible node-to-pod split, a namespace-first owner waterfall, honest handling of shared and idle cost, and an unallocated bucket loud enough to embarrass the labels into existence. Start reporting a rough number this month rather than a perfect number never.

How does your team split idle and shared cluster cost, evenly, by usage, or eat it at the platform level? That is the part everyone seems to solve differently.

Top comments (0)