DEV Community: Kestrion

The Kubernetes Audit Worksheet

Kestrion — Tue, 15 Sep 2026 09:15:18 +0000

Do it right from the beginning ----> Print this, block four hours, and work through it with whoever knows the cluster best. Every command is read-only.

Section 0 — Context (15 min, no terminal)

Write down before looking at anything:

Business: what runs on this cluster, and what does one hour of downtime cost?
People: who can operate it? Who is the single point of failure?
History: last incident, last upgrade, last restore test (dates, not vibes).
Bill: monthly cloud + observability spend, best guess.

You'll compare these answers with reality at the end. The gap is your executive summary.

Section 1 — Inventory (30 min)

kubectl version                          # in support?
kubectl get nodes -o wide                # count, versions, ages, pressure
kubectl get ns                           # what exists
kubectl get deploy,sts,ds,cronjob -A     # workload inventory
helm ls -A                               # what Helm thinks is installed

Record: K8s version and support status, node count/type, workload count, and — critically — which workloads are NOT represented in any Git repo. That list is your unreproducible surface.

Section 2 — Deployment path (30 min)

Trace one real service from commit to production and write each step down. Then answer:

Can anyone deploy by hand to prod? (If yes: who has, recently? Check kubectl rollout history.)
Rollback: exact commands/steps, and when it was last exercised.
Probes and shutdown: for the top 3 services, check readiness probes, preStop, terminationGracePeriodSeconds, rollout strategy, PDBs.

kubectl get deploy <top-service> -o yaml | grep -A5 -E "readinessProbe|preStop|strategy"

Section 3 — Access & secrets (45 min)

kubectl get clusterrolebindings -o wide | grep -iE "admin|edit"
kubectl auth can-i get secrets -n <shared-ns> --as=<a-normal-user>
kubectl get secrets -A | wc -l

Record: humans with cluster-admin (target: ~0 for daily use), service accounts with broad rights, secret source of truth (ESO/Vault/SOPS vs. hand-created), whether the API server is internet-reachable, and whether any Git repo contains a base64 secret (git grep -i "kind: Secret" across infra repos). Also check kubectl get validatingwebhookconfigurations,clusterpolicies -A — if nothing comes back, there's no admission-time guardrail catching a bad manifest before RBAC and secrets hygiene even get tested.

Section 4 — Cost (45 min)

If OpenCost/Kubecost isn't installed, note it as finding #1 and use what you have:

kubectl top nodes
kubectl top pods -A --sort-by=memory | head -30
kubectl describe nodes | grep -A8 "Allocated resources"

Record: requested vs. actually-used CPU/memory ratio (the over-provisioning factor), workloads with no requests, count of type: LoadBalancer services, log ingestion GB/day and retention, non-prod running out-of-hours, unattached volumes in the cloud console.

Section 5 — Observability (30 min)

Where do engineers look first during an incident? (Ask two people separately; different answers = finding.)
Open the alert history for the last 2 weeks: which alerts fired, which led to action? Non-actionable ones are noise debt.
Do golden-signal dashboards exist for the top 3 services?
Certificates: kubectl get certificates -A -o json | jq -r '.items[] | select(.status.conditions[]?.status!="True") | .metadata.name' lists cert-manager certs not currently Ready. Is renewal automated AND alerted on failure?

Section 6 — Backup & DR (30 min)

kubectl get backups -A          # velero, if present

Record: what is backed up (cluster state? volumes? databases?), where backups live (outside the blast radius?), last successful backup, last tested restore, and estimated time to rebuild everything from Git + backups. If the restore has never been tested, the audit's #1 recommendation writes itself.

Section 7 — Upgrade readiness (20 min)

kubent   # or: pluto detect-all-in-cluster    # deprecated APIs

Record: versions behind latest, deprecated API usage, add-on versions (ingress, cert-manager, CNI, CSI) and whether anyone owns their lifecycle, existence of a staging cluster or IaC to create one.

Section 8 — Synthesis (45 min)

Turn notes into three artifacts:

Risk map — every finding rated Impact (1–3) × Likelihood (1–3). Anything 6+ goes on page one. Example: an untested restore is Impact 3 (the business can't recover) × Likelihood 2 (backups exist but nobody's proven they work) = 6.
Cost opportunity — over-provisioning factor + logging retention + orphans, expressed in currency/month.
Remediation backlog — max 10 items, ordered by risk score, each with an effort guess (hours/days/weeks). Default priority when tied: tested restore → rollback → secrets/RBAC → requests cleanup → cert automation → upgrade plan.

Finish by re-reading Section 0. The distance between what the team believed and what you found is the story the leadership summary should tell.

A note on scope

This worksheet deliberately excludes application code quality, CI pipeline internals, and cloud account security outside the cluster. Those matter, but mixing them in is how audits become three-week projects nobody finishes. Half a day, cluster-focused, three artifacts out. Done beats comprehensive.

A version of this with 0–100 scoring runs in the browser: the free production readiness scorecard.

Why Kubernetes Rolling Updates Still Give You 503s

Kestrion — Wed, 09 Sep 2026 20:46:30 +0000

"We use rolling updates, so deploys are zero-downtime." Then you deploy, and for 2–3 seconds users get 503s. This exact complaint shows up constantly on Stack Overflow and Reddit, and it has a precise, fixable cause — actually four of them, stacked.

The core misunderstanding: rolling updates guarantee pod replacement order, not traffic correctness. Kubernetes replaces pods gracefully; whether requests survive depends on details it leaves to you.

Cause 1: traffic arrives before the app is ready

A pod becomes a load-balancing target when its readiness probe passes. If you have no readiness probe, that's the moment the container starts — before your app has loaded config, connected to the database, or warmed anything. First requests: connection refused.

Fix: a readiness probe that checks what "ready to serve" actually means for your app:

readinessProbe:
  httpGet: { path: /ready, port: 8080 }
  periodSeconds: 2
  failureThreshold: 2

And /ready should verify dependencies (DB connection up), not just return 200 unconditionally.

If startup is slow and variable (JVM warmup, a large cache preload), pair this with a startupProbe. Without one, a slow-but-healthy app can hit the readiness probe's failureThreshold and get killed mid-boot — the opposite symptom, same root cause: nothing told Kubernetes "still starting" is different from "broken."

Cause 2: traffic keeps arriving after the pod is told to die

This is the one that causes the classic "2–3 seconds of 503s," and it's the least intuitive. When a pod terminates, two things happen in parallel, not in sequence:

The pod gets SIGTERM.
Endpoint controllers start removing the pod from load balancers.

Step 2 takes time to propagate — kube-proxy rules on every node, cloud LB target deregistration. For seconds after SIGTERM, traffic is still routed to a pod that may already be shutting down. If your app exits immediately on SIGTERM, every one of those requests fails.

Fix — the famous sleep:

lifecycle:
  preStop:
    exec:
      command: ["sleep", "8"]

The preStop hook runs before SIGTERM is sent. Sleeping a few seconds keeps the app fully serving while endpoints propagate. It looks like a hack; it's the standard, load-bearing pattern (Kubernetes docs and every ingress vendor recommend some form of it).

How long to sleep depends on the slowest link in your specific path — in-cluster kube-proxy rules update quickly, but a cloud load balancer sitting in front can be far slower (AWS ALB's deregistration_delay defaults to 300 seconds). Check what's actually in your traffic path before picking a number.

Cause 3: the app doesn't drain on SIGTERM

After preStop, SIGTERM arrives. The app must: stop accepting new connections, finish in-flight requests, then exit. Many frameworks do this only if you ask — and some servers keep-alive connections open forever unless told to close them.

Fix: enable your framework's graceful shutdown (Spring: server.shutdown=graceful; Go: http.Server.Shutdown(); Node: close the server and track sockets), and make sure terminationGracePeriodSeconds (default 30) exceeds preStop + your longest request. If it doesn't, SIGKILL wins and kills in-flight work.

Keep-alive HTTP/2 and gRPC clients add one more wrinkle: a client holding a long-lived connection open won't notice the pod is going away until that connection actually breaks. The server needs to signal it explicitly — gRPC servers should send GOAWAY, not just stop accepting new streams — or the client keeps sending requests into a pod that's already draining.

Cause 4: the rollout replaces capacity too aggressively

Defaults (maxUnavailable: 25%) can briefly leave you under-provisioned under load: old pods dying while new pods are technically Ready but cold (empty caches, JIT not warmed, connection pools empty). Latency spikes read as downtime.

Fix:

strategy:
  rollingUpdate: { maxUnavailable: 0, maxSurge: 1 }

New capacity comes up before old capacity goes away. Slower rollout, no dip. Add a PodDisruptionBudget so node drains during upgrades obey the same rule.

The complete checklist

For every user-facing service:

[ ] Readiness probe that checks real dependencies
[ ] preStop sleep (5–10s)
[ ] Graceful shutdown on SIGTERM, verified (kill a pod, watch error rates)
[ ] terminationGracePeriodSeconds > preStop + longest request
[ ] maxUnavailable: 0, maxSurge: 1 for critical services
[ ] PodDisruptionBudget
[ ] Actually test it: kubectl rollout restart under synthetic load and watch the error rate

That last item is the real gap. Teams configure all of this and never verify it. A deploy under load in staging with a load generator running takes fifteen minutes and tells you the truth.

Why this matters beyond deploys

The same mechanics run during node upgrades, autoscaler scale-downs, and spot instance reclaims. If deploys cause 503s, so does every cluster maintenance event — your "deployment problem" is actually an "any pod ever moves" problem. Fix it once and cluster upgrades, autoscaling, and spot instances all become safe as a side effect. It's one of the highest ROI afternoons available in Kubernetes.

Deployment safety is one of seven categories in the free production readiness scorecard we built.

8 Kubernetes Cost Leaks Small Teams Miss

Kestrion — Tue, 01 Sep 2026 14:28:01 +0000

The over-provisioning, logging, and idle-infrastructure defaults that quietly inflate a small team's Kubernetes bill—and the order I'd fix them in.

Half of teams report their costs went up after adopting Kubernetes (CNCF FinOps microsurvey), with over-provisioning cited as the top cause. That matches every complaint thread: "our observability costs are destroying our cloud budget," "$80K/month just for APM/logging/metrics." The pattern is consistent: the cluster itself isn't the leak — the defaults around it are.

Here are the eight leaks I'd check first, roughly in order of typical savings.

1. Resource requests copied from tutorials

Requests reserve capacity whether you use it or not, and the cluster autoscaler buys nodes to satisfy requests, not usage. Most small-team clusters request 3–5× what they consume, because every deployment inherited 500m / 512Mi from a blog post in 2022.

Fix: measure a week of real usage per workload, reset requests to P95 + headroom. Tools: OpenCost, kubectl top, Goldilocks/VPA in recommendation mode. This alone is often 30–50% of compute.

2. Log ingestion with no retention policy

Observability pricing is per-GB-ingested and per-GB-retained. Default behavior — every pod's stdout, debug level, shipped to a SaaS, kept forever — is how a 10-person company gets a five-figure logging bill.

Fix: drop debug logs at the collector (Fluent Bit/OTel Collector filter), sample high-volume noise, set 7–30 day retention with cold archive to object storage for compliance. Bills routinely drop 60–80%.

3. Non-prod running nights and weekends

Staging and dev clusters at full size, 168 hours a week, used maybe 50. That's paying for 3× the environment you use.

Fix: scale non-prod to zero (or minimum) outside working hours. A CronJob or your cloud's scheduler is enough; kube-downscaler exists precisely for this.

4. One load balancer per service

Every Service type: LoadBalancer is a billed cloud load balancer. Ten services = ten LBs = real money for zero benefit.

Fix: one ingress controller behind one LB, route by host/path. This is a one-day change.

5. Cross-AZ traffic you never asked for

Kubernetes spreads pods across zones, then services route across zones by default, and clouds bill every cross-AZ gigabyte. Chatty microservices + 3 AZs = a silent network line item that grows with traffic.

Fix: topology-aware routing / internalTrafficPolicy, colocate chatty services, and check whether every environment truly needs 3-AZ HA (staging doesn't). Note the trade-off: internalTrafficPolicy: Local keeps traffic in-zone and off the bill, but it also stops balancing across zones — fine for a chatty backend with even load, riskier if one zone can end up overloaded while another sits idle.

6. Orphaned volumes, IPs, and snapshots

Deleting workloads doesn't always delete their PersistentVolumes (depends on reclaim policy), and never deletes old snapshots or static IPs. These accumulate for years.

Fix: quarterly sweep: unattached volumes, unassociated IPs, snapshots older than policy. kubectl get pv -o json | jq '.items[] | select(.status.phase=="Released") | .metadata.name' finds volumes still billing after their claim is gone — start there. Usually a few hundred/month found in ten minutes.

7. Zero spot/preemptible usage

Stateless, replicated workloads — the exact thing Kubernetes is good at rescheduling — are ideal for spot instances at 60–90% discount. Small teams avoid them out of fear; the fear is mostly outdated for fault-tolerant workloads.

Fix: a spot node pool for stateless/batch/CI workloads with on-demand fallback (Karpenter or managed node groups make this nearly trivial). Karpenter's consolidation policy actively bin-packs and reshuffles onto cheaper capacity as prices move, not just at initial scheduling — worth enabling on purpose, it isn't the default. Keep databases on on-demand.

8. Nobody owns the number

The meta-leak. CNCF's survey found 45% blame "lack of awareness/ownership" for overspend. If cost is nobody's metric, every leak above regenerates within a quarter of being fixed.

Fix: install OpenCost (free, CNCF), put cost-per-namespace on a dashboard the team actually sees, review it monthly in an existing meeting. Visibility changes behavior more than any optimization does.

The order I'd run this as an audit

OpenCost + one week of data (leak 8 enables seeing 1–7).
Requests vs. usage report → resize (biggest single win).
Logging ingestion/retention review (second biggest, fastest to fix).
LB/ingress consolidation and orphan sweep (quick wins).
Non-prod schedule + spot pools (structural wins).

A realistic outcome for a small cluster that's never been through this: 30–60% off the total bill, most of it in the first two steps, with no architecture changes and no new paid tools.

The uncomfortable truth: none of this is advanced. It's unglamorous hygiene that nobody owns because everyone is shipping features. Which is exactly why it's worth an afternoon.

Cost is one of seven categories in the 50-question production readiness scorecard I built. It runs in the browser and turns this checklist into a score you can track.

References

The First 10 Things I Would Check in a Messy Kubernetes Cluster

Kestrion — Tue, 25 Aug 2026 20:49:39 +0000

A practical first-pass assessment for an undocumented Kubernetes cluster—focused on risk, cost, recoverability, and ownership.

A practical first-pass assessment for an undocumented Kubernetes cluster—focused on risk, cost, recoverability, and ownership.

Imagine being handed a Kubernetes cluster with no documentation, one engineer who half-remembers building it, a growing bill, and a business that depends on it.

Nothing is obviously broken. Deployments work—most days. Certificates renew—probably. Backups exist—somewhere. This is not a failed platform; it is a platform whose risks nobody can clearly name.

I would not begin by redesigning it or introducing another tool. I would first build a map of reality. These are the ten areas I would inspect, in this order.

1. Is the foundation still supported?

First, I want to know whether the Kubernetes version and its critical add-ons are still supported. An old cluster turns every other improvement into part of an upgrade project.

I would also look for node pressure, recurring instability, and conditions the team has learned to call “normal.” Persistent warnings are findings, not personality traits of the cluster.

2. Does Git describe what is actually running?

Next, I would compare the live cluster with the repositories that supposedly define it. The difference is the team's tribal-knowledge inventory: manual hotfixes, temporary jobs that became permanent, and releases upgraded outside the normal path.

Anything that exists only in the cluster is difficult to review, explain, or rebuild.

3. Who can do what?

I would map human and service-account access, especially broad administrative permissions. The problem is rarely one dramatic mistake. It is accumulated access: former experiments, CI credentials, shared accounts, and permissions nobody felt safe removing.

The useful question is not only “who has access?” but “who still needs it?”

4. Where do secrets actually live?

Secrets often reveal the operating model faster than any architecture diagram. Are they stored in plain YAML, created manually, shared between applications, or managed from a clear source of truth?

If nobody can explain how a secret is rotated without breaking production, it probably has not been rotated.

5. Do resource requests reflect reality?

I would compare declared requests with observed usage. Messy clusters often contain both extremes: workloads reserving far more than they need and workloads declaring nothing at all.

This affects more than cost. Scheduling, autoscaling, capacity planning, and stability all depend on resource assumptions that resemble reality.

6. What is actually driving the bill?

Compute gets most of the attention, but the expensive surprise may be log ingestion, retention, cross-zone traffic, idle load balancers, or orphaned storage.

The goal is not an impressive savings percentage. It is a cost map the team can explain: which workloads and operational choices create the bill, and which of them deliver value.

7. What expires next?

Certificates, domains, credentials, and tokens are quiet until they are urgent. I would identify what expires, how renewal works, and whether a failed renewal reaches a human before customers notice.

Automation is not enough. Unobserved automation is only a more sophisticated assumption.

8. What happens when something disappears?

For the most important service, I would ask what happens during a pod restart, deployment, or node drain. Are new instances ready before receiving traffic? Do old instances finish in-flight requests? Is there enough capacity to lose one replica?

This is where “zero-downtime deployment” stops being a YAML setting and becomes observable behaviour.

9. Can the team restore, not merely back up?

I would find the latest successful backups, where they are stored, and what they actually contain. Cluster resources and application data may need different recovery mechanisms.

Then comes the decisive question: when was the last restore tested? A successful backup job proves that something was written. A restore test proves that the business can recover.

10. What does only one person know?

Finally, I would sit with the person who knows the cluster best and ask: “What are you the only person who knows how to do?”

The answer might include upgrades, DNS changes, incident recovery, certificate renewal, or a fragile deployment sequence. That list is not a documentation problem. It is operational risk—and often the most valuable output of the assessment.

What should come out of this review?

Not a 40-page report and not a shopping list of new tools. The useful result is three small artifacts:

A risk map: what could interrupt production, and why.
A cost map: where the bill comes from and what deserves investigation.
A prioritized backlog: what the team should fix first, with a clear owner.

The first actions are usually unglamorous: test a restore, confirm the rollback path, reduce unnecessary access, document critical knowledge, and schedule the next upgrade.

Messy clusters rarely need another layer of complexity. They need a shared picture of reality, explicit ownership, and an order of operations.

I turned this kind of assessment into a 50-question Kubernetes production readiness scorecard. It runs in the browser and helps a team find the first conversation worth having.

The Kubernetes Checklist for Teams Without a Platform Team

Kestrion — Thu, 13 Aug 2026 21:24:54 +0000

Most Kubernetes advice assumes you have a platform team: specialists who own upgrades, ingress, security policies, and the 2 a.m. pages.

The teams I am writing for usually have three to ten engineers, one of whom “knows Kubernetes,” and no dedicated platform team. They depend on a cluster that nobody fully owns.

I work in enterprise environments where platform teams are large and everything is process. This article is the opposite exercise: what is the minimum discipline a small team needs to run Kubernetes in production—and what enterprise baggage should it refuse to copy?

The question that matters more than any tool

Before any checklist: who owns the platform after the migration is finished?

Not “who set it up.” Who owns upgrades next year, certificate renewals, the CNI version, and deprecated APIs?

If the answer is one person's name, you do not have a platform. You have key-person risk with YAML on top. If the answer is “nobody, really,” Kubernetes is invisible operational debt accumulating interest.

The rest of this checklist exists to make that ownership small enough for a small team to carry.

For each item, score 0 if it does not exist, 1 if it exists but is informal or untested, and 2 if it is documented and tested. The purpose is not to produce a flattering number. It is to expose the next few conversations the team needs to have.

1. Deployments: Git is the source of truth

Treat Git as the source of truth for workloads and cluster configuration, including temporary fixes.
Use one reconciliation path—for example, Argo CD or Flux—so production changes are reviewed and reproducible. Keep emergency access, but reconcile every emergency change back into Git.
Define and test a rollback path for every service. A Git revert is useful only if your delivery process can deploy it safely.

This converts your cluster from a mystery into a diff. Every other practice gets easier once “what is running?” has an answer.

2. The rollout basics that prevent late-night incidents

Readiness probes that answer one question: can this pod serve traffic now? Include a dependency only when the application genuinely cannot serve a useful response without it.
Graceful termination: handle SIGTERM and allow in-flight requests to drain. Add a short preStop delay only when your routing path needs time to stop sending new traffic, then test the behavior under load.
Resource requests based on measured usage so scheduling and capacity planning reflect reality. Add limits deliberately; the right choice differs for CPU, memory, and workload behavior.
PodDisruptionBudgets for replicated workloads that must remain available during voluntary disruptions such as node drains. Size them from the replica count and failure tolerance.

3. Security: the boring baseline, not the service mesh

Small teams can over-index on advanced security machinery while skipping basics that matter in a customer security review:

RBAC scoped per human and per workload. Nobody uses cluster-admin for daily work.
Secrets via External Secrets Operator, Vault, or SOPS. Never store plaintext secrets in Git, and check who can run kubectl get secret in shared namespaces.
Admission policy: block :latest, block privileged workloads, and require resource requests. Kyverno is one approachable option.
NetworkPolicies around sensitive namespaces and workloads, starting with the traffic paths you actually understand.
Restrict control-plane access to the networks and identities that need it; use the private-endpoint options of your managed provider where they fit your operating model.

That is a stronger starting point than adding advanced machinery while basic access and secret handling remain unclear.

4. Observability: answers, not data

You do not need every log, metric, and trace. You need to answer three questions quickly: is it broken, where, and since when?

Prometheus—or a managed equivalent—plus golden-signal dashboards for your most important services.
Alerts on symptoms users feel: error rate, latency, and saturation. Regularly review alerts that never produce action.
Centralized logs with an explicit retention policy.
Tracing when you have real multi-service debugging pain and the budget. “Not yet” can be a valid architecture decision.

5. Backup: a backup you have not restored is a rumor

Velero—or a cloud-native equivalent—for Kubernetes resources and supported persistent volumes; database-native backups for databases.
Backups stored outside the cluster and account they protect.
Test a restore. Restore tests regularly expose missing permissions, credentials, data, or instructions. Find that out on a Tuesday afternoon, not during an incident.
Write one page titled “the cluster is gone”: how long rebuilding from Git and backups should take, and who does what.

6. Upgrades: a schedule, not an emergency

Kubernetes releases regularly and managed providers enforce their own support windows. Teams that wait until an upgrade is forced take on more changes at once and have less recent practice.

Set a cadence that stays comfortably inside your provider's support window, and put it on the calendar.
Run pluto or kubent for deprecated APIs before every upgrade.
Treat add-ons—ingress controller, cert-manager, CSI, and CNI—as part of the upgrade. Their compatibility can be as consequential as the control-plane version.
Rehearse on a throwaway or staging cluster first.

What not to copy from enterprises

Multi-cluster, multi-region setups before you have multi-customer problems.
A service mesh “because security.” Start with NetworkPolicies.
An internal developer portal for five engineers.
Change advisory boards. For many small teams, Git history and pull-request review are the change process.
Every logo in the CNCF landscape. The landscape is a map, not a shopping list.

Copy the discipline: ownership, rollback, tested restores, and an upgrade cadence.

Refuse the complexity: architecture shaped by headcount you do not have.

The one-hour exercise

You do not need to implement all of this at once. The useful outcome is a short, ordered backlog—not a perfect score.

Sit the team down and score each item from 0 to 2. Anything scoring 0 in backup, rollback, or secrets is a candidate for the next sprint—before another platform feature.

I turned this exercise into a 50-question production readiness scorecard that runs in the browser. It covers deployment safety, ownership, cost, security, observability, disaster recovery, and upgrades.

The goal is not a perfect platform. The goal is Kubernetes boring enough that your team can go back to shipping product.