DEV Community

Cover image for Running Doesn't Mean Ready: A Kubernetes Troubleshooting Lab
Roger Oliveira
Roger Oliveira

Posted on

Running Doesn't Mean Ready: A Kubernetes Troubleshooting Lab

#kubernetes #devops #cloudnative #sre

A Kubernetes Pod can be running without being ready to serve traffic.

This distinction matters when troubleshooting applications. A container process may remain alive while the application is temporarily unable to handle requests safely.

In a hands-on Kubernetes experiment, I explored this behavior using an NGINX Deployment, a ConfigMap, a readiness probe, and a Service.

The goal was simple: make a running Pod become unready, observe what Kubernetes changes, and restore readiness without restarting the container.

1. Understanding Running vs. Ready

In Kubernetes, the Pod phase and its readiness condition represent different aspects of application health.

  • Running means the Pod has been scheduled and at least one container is running, starting, or restarting.
  • Ready indicates whether the Pod meets its readiness conditions and can be considered for Service traffic.
  • Readiness probes determine whether individual containers are ready to serve requests.

A Pod can therefore remain in the Running phase while reporting Ready=False.

This is an important distinction in production troubleshooting.

A running container is not necessarily a ready application.

2. The laboratory architecture

The experiment uses a minimal Kubernetes configuration:

  • One NGINX Deployment with a single replica.
  • One ConfigMap controlling the readiness state.
  • One fictional Secret injected as an environment variable.
  • One readiness probe checking a mounted file.
  • One Service exposing the workload.
  • EndpointSlice inspection to observe endpoint readiness.

The recorded experiment was executed in an isolated Minikube environment using Kubernetes v1.30.0.

The repository also contains instructions for reproducing the exercise in a disposable Kubernetes playground.

The ConfigMap

The ConfigMap initially contains:

apiVersion: v1
kind: ConfigMap
metadata:
  name: web-readiness
data:
  status: ready
Enter fullscreen mode Exit fullscreen mode

The Deployment mounts this configuration as a file.

The readiness probe

The NGINX container uses an exec readiness probe:

readinessProbe:
  exec:
    command:
      - grep
      - -qx
      - ready
      - /etc/lab-readiness/status
  periodSeconds: 3
  failureThreshold: 2
Enter fullscreen mode Exit fullscreen mode

The probe succeeds when the mounted file contains exactly ready.

If that value changes, the probe begins to fail after Kubernetes propagates the ConfigMap update to the mounted volume.

The Secret in this lab contains a fictional value. It demonstrates workload configuration but does not control readiness.

3. Establishing a healthy baseline

Before introducing a failure, we need to know what healthy behavior looks like.

After confirming the Kubernetes context, creating the namespace, and applying the laboratory manifests, wait for the Deployment:

kubectl -n lab-workload-config rollout status \
  deployment/web --timeout=180s
Enter fullscreen mode Exit fullscreen mode

Next, inspect the Pod and record its identity:

kubectl -n lab-workload-config get pods \
  -l app=workload-config-web \
  -o 'custom-columns=NAME:.metadata.name,UID:.metadata.uid,RESTARTS:.status.containerStatuses[0].restartCount'
Enter fullscreen mode Exit fullscreen mode

Finally, inspect the Service endpoints:

kubectl -n lab-workload-config get endpointslices \
  -l kubernetes.io/service-name=web \
  -o jsonpath='{.items[*].endpoints[*].conditions.ready}{"\n"}'
Enter fullscreen mode Exit fullscreen mode

The expected healthy baseline is:

  • The Pod is Running.
  • The Pod is Ready.
  • The Service has one ready endpoint.
  • The container has not restarted.

The Pod UID and restart count are important because we will compare them after recovery.

4. Simulating a readiness failure

Now comes the interesting part.

Instead of stopping NGINX, deleting the Pod, or introducing a container crash, we change the ConfigMap:

kubectl -n lab-workload-config patch configmap \
  web-readiness --type=merge \
  -p '{"data":{"status":"blocked"}}'
Enter fullscreen mode Exit fullscreen mode

The mounted configuration file eventually changes from:

ready
Enter fullscreen mode Exit fullscreen mode

to:

blocked
Enter fullscreen mode Exit fullscreen mode

Since the readiness probe expects the exact value ready, it starts failing.

We can wait for the Pod to become unready:

kubectl -n lab-workload-config wait \
  --for=condition=Ready=false \
  pod -l app=workload-config-web \
  --timeout=180s
Enter fullscreen mode Exit fullscreen mode

Then investigate:

kubectl -n lab-workload-config get pods \
  -l app=workload-config-web

kubectl -n lab-workload-config describe pod \
  -l app=workload-config-web
Enter fullscreen mode Exit fullscreen mode

The key observation from the recorded experiment was:

The Pod remained Running, but its readiness condition became false.

The container process did not need to crash for Kubernetes to stop treating the endpoint as ready.

What happened to the Service?

The laboratory has only one replica.

When that Pod became unready, the Service had no ready endpoints available for normal traffic routing.

We inspected this behavior through the EndpointSlice API:

kubectl -n lab-workload-config get endpointslices \
  -l kubernetes.io/service-name=web \
  -o jsonpath='{.items[*].endpoints[*].conditions.ready}{"\n"}'
Enter fullscreen mode Exit fullscreen mode

The endpoint readiness condition reflected the failure.

EndpointSlice updates can lag briefly behind Pod readiness changes, so observing the transition requires allowing time for reconciliation.

Checking only the Pod phase is not enough when investigating service availability.

5. Recovering without restarting the Pod

Next, restore the ConfigMap:

kubectl -n lab-workload-config patch configmap \
  web-readiness --type=merge \
  -p '{"data":{"status":"ready"}}'
Enter fullscreen mode Exit fullscreen mode

After the mounted file updates, the readiness probe can succeed again.

Wait for recovery:

kubectl -n lab-workload-config wait \
  --for=condition=Ready \
  pod -l app=workload-config-web \
  --timeout=180s
Enter fullscreen mode Exit fullscreen mode

Then inspect the Pod identity and restart count again:

kubectl -n lab-workload-config get pods \
  -l app=workload-config-web \
  -o 'custom-columns=NAME:.metadata.name,UID:.metadata.uid,RESTARTS:.status.containerStatuses[0].restartCount'
Enter fullscreen mode Exit fullscreen mode

In the recorded Minikube execution:

  • The Pod UID remained unchanged.
  • The container restart count remained zero.
  • The Pod returned to Ready=True.
  • The Service endpoint became ready again.

No Pod replacement or container restart was required.

The workload recovered because its readiness condition changed, not because Kubernetes recreated the application.

6. What this teaches us about troubleshooting

This small experiment illustrates several useful Kubernetes behaviors.

Pod state is not application readiness

A running container may still be unsuitable for traffic.

When investigating application availability, inspect readiness conditions rather than relying exclusively on the Running phase.

Readiness and liveness solve different problems

A readiness probe determines whether a container should be considered ready for traffic.

A liveness probe can trigger a container restart when its failure threshold is reached.

In this experiment, the failure was intentionally limited to readiness. No liveness probe was configured to restart the workload.

EndpointSlices provide another important signal

Pod conditions describe workload readiness.

The EndpointSlice API shows how that readiness affects the endpoints associated with a Service.

Inspecting both helps connect workload health to traffic eligibility.

Recovery does not always require restarting workloads

Not every application failure should trigger a restart.

Sometimes the underlying process is healthy, but a temporary condition makes it unsafe to receive traffic.

A properly designed readiness probe allows Kubernetes to react to that distinction.

7. Reproduce the experiment

The complete laboratory is available in my Kubernetes From Scratch repository:

Lab 02 — Workload Configuration and Readiness

It includes:

  • Kubernetes manifests.
  • Initial deployment instructions.
  • Readiness failure simulation.
  • Pod and EndpointSlice inspection.
  • Recovery verification.
  • Cleanup procedures.

The exercise is designed for isolated, disposable Kubernetes environments.

It should not be applied to production workloads without an appropriate change process.

Final thoughts

Kubernetes troubleshooting becomes more effective when we understand what each signal represents.

Running tells us about the Pod's execution state.

Ready helps determine whether the workload is eligible to receive traffic.

They are related, but they are not interchangeable.

The takeaway: A running Pod is not necessarily ready to handle requests.

For Platform Engineering and SRE, understanding this difference is a valuable step toward building systems that fail predictably and recover safely.

This article is based on a reproducible Kubernetes laboratory developed as part of my independent technical portfolio.

GitHub: Kubernetes From Scratch

Top comments (0)