123 Things Wrong With Your Kubernetes Cluster Right Now: Single Replicas, Pinned HPAs and Missing Probes

Here's an uncomfortable exercise: pick any production Kubernetes cluster older than a year and audit every workload against a full checklist of known misconfigurations. The typical result isn't five findings or ten. Across reliability, autoscaling, security, storage, and hygiene rules, multiplied across EKS, GKE, and AKS conventions, a serious rulebook runs to well over a hundred distinct checks, and mature clusters fail dozens of them simultaneously.

None of these failures pages anyone, which is the whole problem. A single-replica deployment is fine until the node drains. A missing readiness probe is fine until the next rollout routes traffic to a pod that isn't ready. An HPA pinned at max is fine until the day it needed to scale and couldn't. Kubernetes misconfigurations don't fail; they wait.

The taxonomy: what a complete audit actually checks

Reliability. Missing resource requests and limits (the scheduler places pods blind, and the OOM killer chooses for you). Single-replica Deployments in production with no HPA (one voluntary node drain is an outage). Degraded workloads running below desired replicas long after rollout completed. HPAs sitting at max replicas (that's not autoscaling anymore; it's a fixed fleet at its ceiling that you believe is elastic). Missing liveness and readiness probes (deadlocked containers never restart; rollouts route to unready pods). Containers running image:latest (unrepeatable deploys, unrewindable rollbacks). Failed Jobs nobody cleaned up or noticed.

Security basics. Privileged containers, containers running as root, host-network pods, ingress without TLS, services exposed wider than intended. Each is one line of YAML away from a finding that ends up in a pentest report.

Idle and orphan. Deployments scaled to zero months ago and never deleted, CronJobs suspended and forgotten (or never successfully scheduled at all), unbound PVCs holding claims against nothing, released PersistentVolumes lingering for weeks, Services whose endpoints are empty because everything behind them is gone. This category is where reliability audit meets cost audit: orphans bill.

Rightsizing. Requests set to double what the workload uses (paying for reserved capacity the scheduler can't give anyone else), and the subtler inverse: autoscalers whose own status reports they're constrained (a scaling-limited condition) while everyone assumes capacity is fine.

Run the multiplication (a rule per workload kind, per failure mode, adapted per provider) and you land in the low hundreds. The specific number in this post's title isn't rhetorical: 123 is a real rulebook's count, and it's worth knowing where it comes from. ZopNight ships 123 Kubernetes workload rules, 41 unique checks run across EKS, GKE, and AKS, covering exactly the categories above; a detail worth copying from how it's built is that rules it can't yet evaluate reliably (like OOM detection needing real time-series) are shipped disabled rather than fired on weak evidence, and its rightsizing rule only flags what the HPA's own status condition reports rather than inventing a utilization window. That's the audit-quality bar whether you buy one or build one: every finding names its evidence, and no rule fires on a guess.

The eight worst offenders, with one-liners

1. Single-replica production Deployments: kubectl get deploy -A -o json | jq -r '.items[] | select(.spec.replicas==1) | .metadata.namespace + "/" + .metadata.name'
2. HPAs pinned at max: compare status.currentReplicas to spec.maxReplicas across all HPAs; anything equal for days has hit its ceiling silently. (Also flag minReplicas == maxReplicas, which is an HPA cosplaying as a constant.)
3. Missing probes: select containers without readinessProbe or livenessProbe; the readiness gap breaks rollouts, the liveness gap immortalizes deadlocks.
4. Missing requests/limits: select containers with no resources.requests; these pods are invisible to capacity planning and first against the wall in contention.
5. image:latest: grep pod templates for the tag; every hit is a deploy you can't reproduce and a rollback you can't trust.
6. CrashLoop and restart counts: sort pods by restartCount; triple digits in a namespace nobody watches is an incident on layaway.
7. Unbound PVCs and released PVs: kubectl get pvc -A for Pending, kubectl get pv for Released older than a week; both are storage bills with no consumer.
8. Empty-endpoint Services: Services selecting zero pods, which means something upstream still resolves a name that leads nowhere.

Two habits turn the list from trivia into practice. Gate findings on stability: don't flag a deployment as degraded mid-rollout; check that the rollout completed and the degradation persisted, or the audit cries wolf weekly. And rank by blast radius times likelihood: a single-replica payment service outranks forty missing labels, and an audit that can't rank is a report nobody reads twice.

Why nobody catches these

Each item is too small to page on, too boring for sprint planning, and invisible in the dashboards teams actually watch (which show traffic and latency, not configuration posture). The failure is structural: clusters have continuous deployment and point-in-time review. New workloads land daily; the audit, if it ever happened, happened once. Whatever tooling you use, the fix is the same shape: the checklist has to run on a cadence, against every namespace, with findings that carry evidence, or the number in this post's title just grows back.

FAQ

What are the most dangerous Kubernetes misconfigurations?

By incident frequency: single-replica production workloads (any node drain becomes an outage), missing readiness probes (rollouts route traffic to unready pods), missing resource requests (scheduler places blind, OOM killer decides), and HPAs pinned at their max (elasticity that silently ended). All four are invisible until an ordinary event triggers them.

Why is an HPA at max replicas a problem?

Because it means the autoscaler wanted to scale further and couldn't. You're at a fixed ceiling while believing you're elastic: the next traffic increase has nowhere to go, and the condition doesn't alert by default. Persistent max-pinning means the ceiling is wrong, the sizing is wrong, or the demand changed and nobody was told.

Do missing liveness and readiness probes really matter?

Differently, and yes. Without readiness, Kubernetes routes traffic to pods that aren't ready, which turns every deploy into a brief outage. Without liveness, a deadlocked container simply stays deadlocked forever. The reverse failure exists too: aggressive liveness probes that restart slow-starting containers in a loop, so probes need tuning, not just presence.

How often should a cluster configuration audit run?

Continuously, or at worst weekly. Clusters change daily with every deploy, so a quarterly audit describes a cluster that no longer exists. The practical bar: every new workload gets checked within a day of landing, findings persist with evidence until fixed, and the trend (findings per namespace over time) is visible to the teams that own them.

Are these checks different on EKS vs GKE vs AKS?

The workload-level rules (replicas, probes, requests, image tags) are identical Kubernetes-native checks everywhere. Providers differ at the edges: ingress and TLS conventions, storage classes and volume behavior, and each platform's autoscaling integration, which is why serious rulebooks maintain per-provider variants of the same underlying checklist.