The First 10 Things I Would Check in a Messy Kubernetes Cluster

A practical first-pass assessment for an undocumented Kubernetes cluster—focused on risk, cost, recoverability, and ownership.

A practical first-pass assessment for an undocumented Kubernetes cluster—focused on risk, cost, recoverability, and ownership.

Imagine being handed a Kubernetes cluster with no documentation, one engineer who half-remembers building it, a growing bill, and a business that depends on it.

Nothing is obviously broken. Deployments work—most days. Certificates renew—probably. Backups exist—somewhere. This is not a failed platform; it is a platform whose risks nobody can clearly name.

I would not begin by redesigning it or introducing another tool. I would first build a map of reality. These are the ten areas I would inspect, in this order.

1. Is the foundation still supported?

First, I want to know whether the Kubernetes version and its critical add-ons are still supported. An old cluster turns every other improvement into part of an upgrade project.

I would also look for node pressure, recurring instability, and conditions the team has learned to call “normal.” Persistent warnings are findings, not personality traits of the cluster.

2. Does Git describe what is actually running?

Next, I would compare the live cluster with the repositories that supposedly define it. The difference is the team's tribal-knowledge inventory: manual hotfixes, temporary jobs that became permanent, and releases upgraded outside the normal path.

Anything that exists only in the cluster is difficult to review, explain, or rebuild.

3. Who can do what?

I would map human and service-account access, especially broad administrative permissions. The problem is rarely one dramatic mistake. It is accumulated access: former experiments, CI credentials, shared accounts, and permissions nobody felt safe removing.

The useful question is not only “who has access?” but “who still needs it?”

4. Where do secrets actually live?

Secrets often reveal the operating model faster than any architecture diagram. Are they stored in plain YAML, created manually, shared between applications, or managed from a clear source of truth?

If nobody can explain how a secret is rotated without breaking production, it probably has not been rotated.

5. Do resource requests reflect reality?

I would compare declared requests with observed usage. Messy clusters often contain both extremes: workloads reserving far more than they need and workloads declaring nothing at all.

This affects more than cost. Scheduling, autoscaling, capacity planning, and stability all depend on resource assumptions that resemble reality.

6. What is actually driving the bill?

Compute gets most of the attention, but the expensive surprise may be log ingestion, retention, cross-zone traffic, idle load balancers, or orphaned storage.

The goal is not an impressive savings percentage. It is a cost map the team can explain: which workloads and operational choices create the bill, and which of them deliver value.

7. What expires next?

Certificates, domains, credentials, and tokens are quiet until they are urgent. I would identify what expires, how renewal works, and whether a failed renewal reaches a human before customers notice.

Automation is not enough. Unobserved automation is only a more sophisticated assumption.

8. What happens when something disappears?

For the most important service, I would ask what happens during a pod restart, deployment, or node drain. Are new instances ready before receiving traffic? Do old instances finish in-flight requests? Is there enough capacity to lose one replica?

This is where “zero-downtime deployment” stops being a YAML setting and becomes observable behaviour.

9. Can the team restore, not merely back up?

I would find the latest successful backups, where they are stored, and what they actually contain. Cluster resources and application data may need different recovery mechanisms.

Then comes the decisive question: when was the last restore tested? A successful backup job proves that something was written. A restore test proves that the business can recover.

10. What does only one person know?

Finally, I would sit with the person who knows the cluster best and ask: “What are you the only person who knows how to do?”

The answer might include upgrades, DNS changes, incident recovery, certificate renewal, or a fragile deployment sequence. That list is not a documentation problem. It is operational risk—and often the most valuable output of the assessment.

What should come out of this review?

Not a 40-page report and not a shopping list of new tools. The useful result is three small artifacts:

1. A risk map: what could interrupt production, and why.
2. A cost map: where the bill comes from and what deserves investigation.
3. A prioritized backlog: what the team should fix first, with a clear owner.

The first actions are usually unglamorous: test a restore, confirm the rollback path, reduce unnecessary access, document critical knowledge, and schedule the next upgrade.

Messy clusters rarely need another layer of complexity. They need a shared picture of reality, explicit ownership, and an order of operations.

I turned this kind of assessment into a 50-question Kubernetes production readiness scorecard. It runs in the browser and helps a team find the first conversation worth having.