Seeing STATUS: Running and READY: 1/1 in kubectl get pods is a satisfying milestone, especially when standing up a new microservice. However, a workload running cleanly in a quiet staging cluster is not the same thing as a workload ready for production traffic, node drains, AZ outages, or sudden traffic spikes.
$ kubectl get pods -n payments
NAME READY STATUS RESTARTS AGE
checkout-api-6d9f8b7c4d-k8x2p 1/1 Running 0 4m
Behind that single healthy pod might be missing memory limits that trigger cascading OOMKills under load, a default service account with overly broad permissions, or a liveness probe that restarts the container before it finishes warming up.
Treating readiness as a repeatable engineering workflow—rather than a last-minute gut check before launch—bridges the gap between "it deploys" and "we can operate it safely at 3 AM." In this tutorial, we will walk through a practical production readiness assessment workflow for Kubernetes, covering cluster baselines, workload manifests, security hardening, observability, and structured troubleshooting.
Defining Kubernetes Production Readiness Criteria
Before running commands or auditing manifests, align your team on what "production-ready" actually means for your environment. While exact thresholds vary depending on whether you run stateful databases or stateless APIs, every reliable production service must satisfy six core domains:
- Availability & Resilience: Can the workload survive pod restarts, rolling upgrades, and single-node or Availability Zone (AZ) failures without dropping user requests?
- Resource Management: Are CPU and memory requests/limits defined accurately enough to prevent noisy-neighbor starvation and runaway cloud spend?
- Security & Isolation: Does the workload run with least-privilege RBAC, non-root container contexts, restricted network paths, and scanned images?
- Networking & Traffic Control: Are ingress routing, TLS termination, service discovery, and graceful connection draining properly configured?
- Observability: Can on-call engineers immediately determine whether the service is healthy, diagnose latency spikes, and trace failing requests?
- Backup & Recovery: Are persistent volumes, cluster state, and disaster recovery runbooks tested—not just documented?
Step 1: Cluster-Level Baseline Checks
Workload manifests rely on the underlying cluster behaving predictably. Before auditing individual deployments, verify that the cluster itself provides a stable foundation.
Node Health, Capacity, and Allocatable Resources
Inspect node status, Kubernetes versions, and actual resource allocation across your worker pool:
# Check node readiness, roles, and kubelet versions
kubectl get nodes -o wide
# Inspect CPU and memory requests/limits committed per node
kubectl describe nodes | grep -A 5 "Allocated resources"
Look closely at the Allocated resources section rather than raw node capacity. Managed Kubernetes providers reserve CPU and memory for the kubelet, container runtime, and OS daemons. If CPU requests consistently exceed 80–85% of allocatable capacity across nodes, the cluster scheduler will struggle to reschedule pods during an AZ outage or node upgrade.
Control-Plane and Version Compatibility
Check whether your cluster version is approaching end-of-support and verify whether workloads rely on deprecated APIs before scheduling upgrades:
# Verify client and server Kubernetes versions
kubectl version
# Check for deprecations or core component health events
kubectl get events -n kube-system --sort-by='.lastTimestamp'
Trade-off note: Tightening cluster-wide admission controllers (such as enforcing Pod Security Admission restricted profiles) improves baseline hygiene across all namespaces, but applying them abruptly to legacy namespaces can break existing deployments. Roll out cluster policies in warn or audit mode first.
Step 2: Workload-Level Configuration Checks
Once cluster health is verified, inspect individual workload specifications. Most preventable Kubernetes outages stem from misconfigured probes, missing resource bounds, or aggressive rollout settings.
Probes, Resource Bounds, and Availability Controls
Below is an illustrative configuration showing common production patterns for a stateless HTTP service. Adjust CPU/memory values and probe timings to match your application's actual startup and runtime profile rather than treating these numbers as universal defaults:
apiVersion: apps/v1
kind: Deployment
metadata:
name: checkout-api
namespace: payments
spec:
replicas: 3
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 0
maxSurge: 1
selector:
matchLabels:
app: checkout-api
template:
metadata:
labels:
app: checkout-api
spec:
serviceAccountName: checkout-api-sa
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: checkout-api
containers:
- name: api
image: registry.example.com/payments/checkout-api:v1.8.2
ports:
- containerPort: 8080
resources:
requests:
cpu: "250m"
memory: "512Mi"
limits:
memory: "512Mi"
startupProbe:
httpGet:
path: /healthz/startup
port: 8080
failureThreshold: 12
periodSeconds: 5
readinessProbe:
httpGet:
path: /healthz/ready
port: 8080
initialDelaySeconds: 5
periodSeconds: 10
livenessProbe:
httpGet:
path: /healthz/live
port: 8080
periodSeconds: 15
failureThreshold: 3
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: checkout-api-pdb
namespace: payments
spec:
minAvailable: 2
selector:
matchLabels:
app: checkout-api
Key Decisions and Trade-offs in Workload Specs
-
Separate
startupProbe,readinessProbe, andlivenessProbe: If a slow-starting JVM or Python service only has alivenessProbe, Kubernetes may kill it mid-boot, causing an infinite restart loop. UsestartupProbeto protect initialization,readinessProbeto control traffic routing, andlivenessProbestrictly to detect unrecoverable deadlocks. -
CPU limits vs. Memory limits: Setting
requests.memoryequal tolimits.memorygives predictable memory allocation and avoids sudden OOMKills caused by node overcommitment. Conversely, strictlimits.cpucan induce unnecessary CFS CPU throttling during brief traffic bursts; many teams set CPU requests accurately for scheduling while leaving CPU limits unset or generous, depending on workload isolation needs. -
PodDisruptionBudgets (PDBs): A PDB protects availability during voluntary disruptions like node drains. However, setting
minAvailable: 100%(ormaxUnavailable: 0on a single-replica deployment) will block cluster upgrades and node drains indefinitely. Always leave room for at least one pod to move. -
Autoscaling (
HorizontalPodAutoscaler): If you pair a Deployment with an HPA, avoid hardcoding static replica counts in GitOps syncs without ignoring.spec.replicas, or every deployment sync will reset active autoscaled pods back to the baseline count.
Step 3: Security Validation
Security checks should verify that a compromised container cannot easily escalate privileges or move laterally inside the cluster.
Workload Security Context and Least-Privilege Identity
Verify that pods explicitly disable privilege escalation, drop unnecessary Linux capabilities, and avoid using the namespace's default ServiceAccount:
securityContext:
runAsNonRoot: true
runAsUser: 10001
seccompProfile:
type: RuntimeDefault
containers:
- name: api
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop:
- ALL
Audit RBAC bindings and NetworkPolicies across target namespaces:
# Check what permissions a workload's ServiceAccount actually holds
kubectl auth can-i --list --as=system:serviceaccount:payments:checkout-api-sa -n payments
# Verify whether NetworkPolicies exist to restrict east-west pod traffic
kubectl get networkpolicies -n payments
Secrets and Image Hygiene
-
Secrets: Base64-encoded Kubernetes
Secretobjects are not encrypted by default unless KMS encryption at rest is enabled onetcd. Ensure secrets are injected via an external secrets operator or CSI driver rather than committed to Git. -
Image Integrity: Avoid mutable tags like
:latest. Pin immutable release tags or SHA256 digests, and verify that vulnerability scanning gates align with your team's broader DevSecOps implementation roadmap.
Step 4: Observability Readiness
Deploying an application without telemetry forces engineers to debug blind during incidents. Evaluating your telemetry stack against an observability maturity model helps ensure signals actually support rapid triage rather than creating alert noise.
Verify four essentials before go-live:
-
Metrics: Confirm Prometheus/OpenTelemetry scrape targets are active (
up == 1) and expose the four Golden Signals: latency, traffic, errors, and saturation (CPU/memory throttling, connection pool exhaustion). -
Structured Logs: Ensure containers write JSON-formatted logs to
stdout/stderrwith correlation fields (trace_id,request_id,service,version) rather than writing to local container filesystems that vanish on pod termination. - Distributed Tracing: Verify trace context propagation across upstream/downstream HTTP or gRPC calls so cross-service latency bottlenecks are visible.
- Actionable Alerts: Alert on user-impacting symptoms (e.g., error budget burn rate or sustained 5xx spikes) rather than noisy cause-based metrics like a single pod restart. Every paging alert should link to a runbook.
Step 5: Practical Kubernetes CrashLoopBackOff Troubleshooting
Even well-prepared workloads fail in production. Having a repeatable diagnostic sequence saves critical minutes during on-call triage. Below is a practical Kubernetes CrashLoopBackOff troubleshooting walkthrough.
Symptom
After deploying a configuration update to checkout-api, pods enter CrashLoopBackOff and traffic stops routing to the new ReplicaSet:
$ kubectl get pods -n payments
NAME READY STATUS RESTARTS AGE
checkout-api-7f8c9d4b2a-9m4lp 0/1 CrashLoopBackOff 4 (52s ago) 3m
Investigation Workflow
1. Inspect pod state, exit codes, and recent kubelet events:
kubectl describe pod checkout-api-7f8c9d4b2a-9m4lp -n payments
Look specifically at the Last State block and the Events table at the bottom of the output:
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: OOMKilled
Exit Code: 137
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Pulled 3m kubelet Container image already present on machine
Normal Created 92s (x4 over 3m) kubelet Created container api
Normal Started 92s (x4 over 3m) kubelet Started container api
Warning BackOff 48s (x6 over 2m) kubelet Back-off restarting failed container
2. Inspect logs from the crashed container instance:
Running kubectl logs without flags often shows an empty log stream if the container just restarted. Always pass --previous (-p) to inspect the terminated container's output:
kubectl logs checkout-api-7f8c9d4b2a-9m4lp -n payments --previous --tail=50
3. Cross-check namespace events for cluster-level triggers:
kubectl get events -n payments --sort-by='.lastTimestamp'
Interpreting Exit Codes and Remediating
In our example above, Reason: OOMKilled with Exit Code: 137 indicates the Linux kernel terminated the container because it exceeded its 512Mi memory limit (or node memory became exhausted).
-
If
Exit Code: 137(OOMKilled): Inspect actual memory consumption usingkubectl top pod -n paymentsor historical Prometheus container memory metrics. Determine whether the new release introduced a memory leak (e.g., unbounded in-memory caching) or legitimately requires a higherresources.limits.memoryallocation. -
If
Exit Code: 1or2: The application process exited due to a runtime exception, missing environment variable, invalid ConfigMap mount, or failed database connection. Check--previouslogs for stack traces. -
If
Exit Code: 143(SIGTERM): The container received a graceful termination signal—frequently triggered when an overly aggressivelivenessProbefails while the application is under heavy load or still booting.
Remediation: Roll back the offending release immediately (kubectl rollout undo deployment/checkout-api -n payments) to restore service, fix the memory leak or adjust memory limits in Git, and verify pod stability in staging before re-deploying.
Reusable Kubernetes Production Readiness Checklist
Use this Kubernetes production readiness checklist during design reviews or pre-release gates:
Cluster & Infrastructure
- [ ] Worker nodes have sufficient unallocated CPU/memory headroom to survive an AZ failure.
- [ ] Kubernetes control-plane and node versions are within supported vendor windows.
- [ ] Cluster autoscaler or Karpenter node pools are tested for scale-up latency.
Workload Resilience & Scaling
- [ ]
startupProbe,readinessProbe, andlivenessProbeare configured with realistic timeouts. - [ ]
resources.requests(CPU/memory) andresources.limits.memoryare defined based on load testing. - [ ] At least 2–3 replicas run across multiple Availability Zones (
topologySpreadConstraintsor anti-affinity). - [ ]
PodDisruptionBudgetis configured without blocking voluntary node drains. - [ ] Graceful shutdown (
preStophook andterminationGracePeriodSeconds) drains in-flight requests cleanly.
Security & Networking
- [ ] Workload uses a dedicated
ServiceAccountwith least-privilege RBAC rules. - [ ]
securityContextenforcesrunAsNonRoot: true,allowPrivilegeEscalation: false, and drops unused capabilities. - [ ] Container images use immutable tags/digests from a trusted registry with automated CVE scanning.
- [ ]
NetworkPolicyrestricts ingress and egress traffic to approved namespaces and endpoints. - [ ] Secrets are managed externally and never hardcoded in ConfigMaps or environment manifests.
Observability & Operations
- [ ] Golden signal metrics, structured JSON logs, and distributed traces are verified in production dashboards.
- [ ] Alerts target SLO burn rates or user-facing errors and link directly to tested runbooks.
- [ ] Backup and restore procedures for stateful data and cluster configuration have been tested.
Turning Readiness Gaps Into a Prioritized Improvement Roadmap
Running a thorough audit almost always uncovers more gaps than a single sprint can absorb. Trying to fix everything at once leads to stalled initiatives and developer friction. Instead, classify findings along two dimensions: Operational Risk and Implementation Effort.
- Immediate Blockers (High Risk, Low-to-Medium Effort): Fix these before production traffic hits the service. Examples include missing liveness/readiness probes, undefined memory limits, wide-open RBAC permissions, or absent error alerting.
-
Short-Term Hardening (High Risk, Higher Effort): Schedule these into the next 1–2 sprints. Examples include implementing multi-AZ topology spread constraints, adding
NetworkPolicies, or configuring automated secret rotation. - Platform & Process Investments (Medium Risk, Broad Leverage): Standardize recurring fixes into reusable Helm charts, CI policy checks, or golden-path templates as part of your platform engineering adoption roadmap, SRE adoption roadmap, or DevOps maturity assessment.
- Continuous Optimization (Lower Immediate Risk): Layer finOps rightsizing from a cloud cost optimization roadmap and refine post-incident review loops using an incident management maturity model once core stability baselines hold steady.
For engineering teams looking to evaluate operational health systematically across Kubernetes, reliability, security, and platform workflows without building assessment frameworks from scratch, resources like OpsRoadmaps provide structured readiness assessments and prioritized improvement roadmaps.
Conclusion
Production readiness in Kubernetes is not a one-time checkbox ticked before launch day—it is a continuous feedback loop between how workloads are configured and how they behave under real failure conditions. By validating cluster capacity, enforcing sensible workload defaults, locking down container privileges, verifying observability signals, and practicing structured failure triage, teams replace launch-day guesswork with repeatable operational confidence.
Next step: Pick one critical production service this week, run through the checklist above against its live deployment manifest, and turn the top three gaps into concrete engineering tickets.

Top comments (0)