DEV Community: Kestrion

8 Kubernetes Cost Leaks Small Teams Miss

Kestrion — Tue, 01 Sep 2026 14:28:01 +0000

The over-provisioning, logging, and idle-infrastructure defaults that quietly inflate a small team's Kubernetes bill—and the order I'd fix them in.

Half of teams report their costs went up after adopting Kubernetes (CNCF FinOps microsurvey), with over-provisioning cited as the top cause. That matches every complaint thread: "our observability costs are destroying our cloud budget," "$80K/month just for APM/logging/metrics." The pattern is consistent: the cluster itself isn't the leak — the defaults around it are.

Here are the eight leaks I'd check first, roughly in order of typical savings.

1. Resource requests copied from tutorials

Requests reserve capacity whether you use it or not, and the cluster autoscaler buys nodes to satisfy requests, not usage. Most small-team clusters request 3–5× what they consume, because every deployment inherited 500m / 512Mi from a blog post in 2022.

Fix: measure a week of real usage per workload, reset requests to P95 + headroom. Tools: OpenCost, kubectl top, Goldilocks/VPA in recommendation mode. This alone is often 30–50% of compute.

2. Log ingestion with no retention policy

Observability pricing is per-GB-ingested and per-GB-retained. Default behavior — every pod's stdout, debug level, shipped to a SaaS, kept forever — is how a 10-person company gets a five-figure logging bill.

Fix: drop debug logs at the collector (Fluent Bit/OTel Collector filter), sample high-volume noise, set 7–30 day retention with cold archive to object storage for compliance. Bills routinely drop 60–80%.

3. Non-prod running nights and weekends

Staging and dev clusters at full size, 168 hours a week, used maybe 50. That's paying for 3× the environment you use.

Fix: scale non-prod to zero (or minimum) outside working hours. A CronJob or your cloud's scheduler is enough; kube-downscaler exists precisely for this.

4. One load balancer per service

Every Service type: LoadBalancer is a billed cloud load balancer. Ten services = ten LBs = real money for zero benefit.

Fix: one ingress controller behind one LB, route by host/path. This is a one-day change.

5. Cross-AZ traffic you never asked for

Kubernetes spreads pods across zones, then services route across zones by default, and clouds bill every cross-AZ gigabyte. Chatty microservices + 3 AZs = a silent network line item that grows with traffic.

Fix: topology-aware routing / internalTrafficPolicy, colocate chatty services, and check whether every environment truly needs 3-AZ HA (staging doesn't). Note the trade-off: internalTrafficPolicy: Local keeps traffic in-zone and off the bill, but it also stops balancing across zones — fine for a chatty backend with even load, riskier if one zone can end up overloaded while another sits idle.

6. Orphaned volumes, IPs, and snapshots

Deleting workloads doesn't always delete their PersistentVolumes (depends on reclaim policy), and never deletes old snapshots or static IPs. These accumulate for years.

Fix: quarterly sweep: unattached volumes, unassociated IPs, snapshots older than policy. kubectl get pv -o json | jq '.items[] | select(.status.phase=="Released") | .metadata.name' finds volumes still billing after their claim is gone — start there. Usually a few hundred/month found in ten minutes.

7. Zero spot/preemptible usage

Stateless, replicated workloads — the exact thing Kubernetes is good at rescheduling — are ideal for spot instances at 60–90% discount. Small teams avoid them out of fear; the fear is mostly outdated for fault-tolerant workloads.

Fix: a spot node pool for stateless/batch/CI workloads with on-demand fallback (Karpenter or managed node groups make this nearly trivial). Karpenter's consolidation policy actively bin-packs and reshuffles onto cheaper capacity as prices move, not just at initial scheduling — worth enabling on purpose, it isn't the default. Keep databases on on-demand.

8. Nobody owns the number

The meta-leak. CNCF's survey found 45% blame "lack of awareness/ownership" for overspend. If cost is nobody's metric, every leak above regenerates within a quarter of being fixed.

Fix: install OpenCost (free, CNCF), put cost-per-namespace on a dashboard the team actually sees, review it monthly in an existing meeting. Visibility changes behavior more than any optimization does.

The order I'd run this as an audit

OpenCost + one week of data (leak 8 enables seeing 1–7).
Requests vs. usage report → resize (biggest single win).
Logging ingestion/retention review (second biggest, fastest to fix).
LB/ingress consolidation and orphan sweep (quick wins).
Non-prod schedule + spot pools (structural wins).

A realistic outcome for a small cluster that's never been through this: 30–60% off the total bill, most of it in the first two steps, with no architecture changes and no new paid tools.

The uncomfortable truth: none of this is advanced. It's unglamorous hygiene that nobody owns because everyone is shipping features. Which is exactly why it's worth an afternoon.

Cost is one of seven categories in the 50-question production readiness scorecard I built. It runs in the browser and turns this checklist into a score you can track.

References

The First 10 Things I Would Check in a Messy Kubernetes Cluster

Kestrion — Tue, 25 Aug 2026 20:49:39 +0000

A practical first-pass assessment for an undocumented Kubernetes cluster—focused on risk, cost, recoverability, and ownership.

A practical first-pass assessment for an undocumented Kubernetes cluster—focused on risk, cost, recoverability, and ownership.

Imagine being handed a Kubernetes cluster with no documentation, one engineer who half-remembers building it, a growing bill, and a business that depends on it.

Nothing is obviously broken. Deployments work—most days. Certificates renew—probably. Backups exist—somewhere. This is not a failed platform; it is a platform whose risks nobody can clearly name.

I would not begin by redesigning it or introducing another tool. I would first build a map of reality. These are the ten areas I would inspect, in this order.

1. Is the foundation still supported?

First, I want to know whether the Kubernetes version and its critical add-ons are still supported. An old cluster turns every other improvement into part of an upgrade project.

I would also look for node pressure, recurring instability, and conditions the team has learned to call “normal.” Persistent warnings are findings, not personality traits of the cluster.

2. Does Git describe what is actually running?

Next, I would compare the live cluster with the repositories that supposedly define it. The difference is the team's tribal-knowledge inventory: manual hotfixes, temporary jobs that became permanent, and releases upgraded outside the normal path.

Anything that exists only in the cluster is difficult to review, explain, or rebuild.

3. Who can do what?

I would map human and service-account access, especially broad administrative permissions. The problem is rarely one dramatic mistake. It is accumulated access: former experiments, CI credentials, shared accounts, and permissions nobody felt safe removing.

The useful question is not only “who has access?” but “who still needs it?”

4. Where do secrets actually live?

Secrets often reveal the operating model faster than any architecture diagram. Are they stored in plain YAML, created manually, shared between applications, or managed from a clear source of truth?

If nobody can explain how a secret is rotated without breaking production, it probably has not been rotated.

5. Do resource requests reflect reality?

I would compare declared requests with observed usage. Messy clusters often contain both extremes: workloads reserving far more than they need and workloads declaring nothing at all.

This affects more than cost. Scheduling, autoscaling, capacity planning, and stability all depend on resource assumptions that resemble reality.

6. What is actually driving the bill?

Compute gets most of the attention, but the expensive surprise may be log ingestion, retention, cross-zone traffic, idle load balancers, or orphaned storage.

The goal is not an impressive savings percentage. It is a cost map the team can explain: which workloads and operational choices create the bill, and which of them deliver value.

7. What expires next?

Certificates, domains, credentials, and tokens are quiet until they are urgent. I would identify what expires, how renewal works, and whether a failed renewal reaches a human before customers notice.

Automation is not enough. Unobserved automation is only a more sophisticated assumption.

8. What happens when something disappears?

For the most important service, I would ask what happens during a pod restart, deployment, or node drain. Are new instances ready before receiving traffic? Do old instances finish in-flight requests? Is there enough capacity to lose one replica?

This is where “zero-downtime deployment” stops being a YAML setting and becomes observable behaviour.

9. Can the team restore, not merely back up?

I would find the latest successful backups, where they are stored, and what they actually contain. Cluster resources and application data may need different recovery mechanisms.

Then comes the decisive question: when was the last restore tested? A successful backup job proves that something was written. A restore test proves that the business can recover.

10. What does only one person know?

Finally, I would sit with the person who knows the cluster best and ask: “What are you the only person who knows how to do?”

The answer might include upgrades, DNS changes, incident recovery, certificate renewal, or a fragile deployment sequence. That list is not a documentation problem. It is operational risk—and often the most valuable output of the assessment.

What should come out of this review?

Not a 40-page report and not a shopping list of new tools. The useful result is three small artifacts:

A risk map: what could interrupt production, and why.
A cost map: where the bill comes from and what deserves investigation.
A prioritized backlog: what the team should fix first, with a clear owner.

The first actions are usually unglamorous: test a restore, confirm the rollback path, reduce unnecessary access, document critical knowledge, and schedule the next upgrade.

Messy clusters rarely need another layer of complexity. They need a shared picture of reality, explicit ownership, and an order of operations.

I turned this kind of assessment into a 50-question Kubernetes production readiness scorecard. It runs in the browser and helps a team find the first conversation worth having.

The Kubernetes Checklist for Teams Without a Platform Team

Kestrion — Thu, 13 Aug 2026 21:24:54 +0000

Most Kubernetes advice assumes you have a platform team: specialists who own upgrades, ingress, security policies, and the 2 a.m. pages.

The teams I am writing for usually have three to ten engineers, one of whom “knows Kubernetes,” and no dedicated platform team. They depend on a cluster that nobody fully owns.

I work in enterprise environments where platform teams are large and everything is process. This article is the opposite exercise: what is the minimum discipline a small team needs to run Kubernetes in production—and what enterprise baggage should it refuse to copy?

The question that matters more than any tool

Before any checklist: who owns the platform after the migration is finished?

Not “who set it up.” Who owns upgrades next year, certificate renewals, the CNI version, and deprecated APIs?

If the answer is one person's name, you do not have a platform. You have key-person risk with YAML on top. If the answer is “nobody, really,” Kubernetes is invisible operational debt accumulating interest.

The rest of this checklist exists to make that ownership small enough for a small team to carry.

For each item, score 0 if it does not exist, 1 if it exists but is informal or untested, and 2 if it is documented and tested. The purpose is not to produce a flattering number. It is to expose the next few conversations the team needs to have.

1. Deployments: Git is the source of truth

Treat Git as the source of truth for workloads and cluster configuration, including temporary fixes.
Use one reconciliation path—for example, Argo CD or Flux—so production changes are reviewed and reproducible. Keep emergency access, but reconcile every emergency change back into Git.
Define and test a rollback path for every service. A Git revert is useful only if your delivery process can deploy it safely.

This converts your cluster from a mystery into a diff. Every other practice gets easier once “what is running?” has an answer.

2. The rollout basics that prevent late-night incidents

Readiness probes that answer one question: can this pod serve traffic now? Include a dependency only when the application genuinely cannot serve a useful response without it.
Graceful termination: handle SIGTERM and allow in-flight requests to drain. Add a short preStop delay only when your routing path needs time to stop sending new traffic, then test the behavior under load.
Resource requests based on measured usage so scheduling and capacity planning reflect reality. Add limits deliberately; the right choice differs for CPU, memory, and workload behavior.
PodDisruptionBudgets for replicated workloads that must remain available during voluntary disruptions such as node drains. Size them from the replica count and failure tolerance.

3. Security: the boring baseline, not the service mesh

Small teams can over-index on advanced security machinery while skipping basics that matter in a customer security review:

RBAC scoped per human and per workload. Nobody uses cluster-admin for daily work.
Secrets via External Secrets Operator, Vault, or SOPS. Never store plaintext secrets in Git, and check who can run kubectl get secret in shared namespaces.
Admission policy: block :latest, block privileged workloads, and require resource requests. Kyverno is one approachable option.
NetworkPolicies around sensitive namespaces and workloads, starting with the traffic paths you actually understand.
Restrict control-plane access to the networks and identities that need it; use the private-endpoint options of your managed provider where they fit your operating model.

That is a stronger starting point than adding advanced machinery while basic access and secret handling remain unclear.

4. Observability: answers, not data

You do not need every log, metric, and trace. You need to answer three questions quickly: is it broken, where, and since when?

Prometheus—or a managed equivalent—plus golden-signal dashboards for your most important services.
Alerts on symptoms users feel: error rate, latency, and saturation. Regularly review alerts that never produce action.
Centralized logs with an explicit retention policy.
Tracing when you have real multi-service debugging pain and the budget. “Not yet” can be a valid architecture decision.

5. Backup: a backup you have not restored is a rumor

Velero—or a cloud-native equivalent—for Kubernetes resources and supported persistent volumes; database-native backups for databases.
Backups stored outside the cluster and account they protect.
Test a restore. Restore tests regularly expose missing permissions, credentials, data, or instructions. Find that out on a Tuesday afternoon, not during an incident.
Write one page titled “the cluster is gone”: how long rebuilding from Git and backups should take, and who does what.

6. Upgrades: a schedule, not an emergency

Kubernetes releases regularly and managed providers enforce their own support windows. Teams that wait until an upgrade is forced take on more changes at once and have less recent practice.

Set a cadence that stays comfortably inside your provider's support window, and put it on the calendar.
Run pluto or kubent for deprecated APIs before every upgrade.
Treat add-ons—ingress controller, cert-manager, CSI, and CNI—as part of the upgrade. Their compatibility can be as consequential as the control-plane version.
Rehearse on a throwaway or staging cluster first.

What not to copy from enterprises

Multi-cluster, multi-region setups before you have multi-customer problems.
A service mesh “because security.” Start with NetworkPolicies.
An internal developer portal for five engineers.
Change advisory boards. For many small teams, Git history and pull-request review are the change process.
Every logo in the CNCF landscape. The landscape is a map, not a shopping list.

Copy the discipline: ownership, rollback, tested restores, and an upgrade cadence.

Refuse the complexity: architecture shaped by headcount you do not have.

The one-hour exercise

You do not need to implement all of this at once. The useful outcome is a short, ordered backlog—not a perfect score.

Sit the team down and score each item from 0 to 2. Anything scoring 0 in backup, rollback, or secrets is a candidate for the next sprint—before another platform feature.

I turned this exercise into a 50-question production readiness scorecard that runs in the browser. It covers deployment safety, ownership, cost, security, observability, disaster recovery, and upgrades.

The goal is not a perfect platform. The goal is Kubernetes boring enough that your team can go back to shipping product.