DEV Community

Rafa Terrero
Rafa Terrero

Posted on

Kubernetes: Deployment Success Is Not Production Readiness

A hands-on Kubernetes lab demonstrating why healthy Pods, successful rollouts, and working business operations are not the same thing.

A Kubernetes Deployment can report a successful rollout while a business-critical endpoint is returning HTTP 503.

No broken YAML. No crashing container. No failed readiness probe.

Just an application that Kubernetes considers available but cannot perform an important operation.

This is not necessarily a Kubernetes bug. It is a misunderstanding of what Kubernetes actually guarantees.

A successful rollout is an infrastructure-level observation, not an end-to-end verification of business functionality.

I wanted to demonstrate this distinction with something more useful than a theoretical explanation.

So I built a reproducible Kubernetes lab, deliberately introduced four different failure scenarios, observed how Kubernetes responded, and verified the recovery process.

The experiments were executed on Kubernetes 1.37.0 using Kind. The repository also includes a successful GitHub Actions workflow that independently validates the healthy deployment.

Let's walk through the architecture, experiments, and results.

1. What Kubernetes actually checks

Kubernetes provides three health probes, each with a different responsibility.

Probe Primary question Consequence of repeated failure
startupProbe Has the application started? Restart the container
readinessProbe Can this container receive traffic? Mark the container unready and remove it from normal Service traffic eligibility
livenessProbe Should this container be restarted? Restart the container

These distinctions matter.

A liveness probe should not restart an otherwise functional application simply because a shared external dependency is temporarily unavailable.

A readiness probe should represent the application's ability to handle the traffic it is expected to receive.

And a startup probe can protect a slow-starting application from premature liveness restarts. When configured, it prevents readiness and liveness checks from executing until startup succeeds.

But none of these probes automatically understands your business logic.

Kubernetes knows what your application reports. It does not independently know whether a payment, booking, authentication, or order-processing workflow is correct.

Kubernetes manages the health signals we expose. It does not define what business health means for us.

2. Build a reproducible lab

We will deploy a deliberately simple Python HTTP server using a Kubernetes ConfigMap.

The application exposes four endpoints:

Endpoint Purpose
/startup Simulate successful application initialization
/live Report whether the process should remain running
/ready Report whether the application should receive traffic
/checkout Simulate an important business operation

The server supports artificial failure flags stored in /tmp.

This allows us to demonstrate different failure modes without involving real databases, payment providers, or external infrastructure.

Requirements

You'll need:

  • Docker
  • Kind
  • kubectl
  • Git
  • Sufficient resources for two application Pods

Use a disposable Kubernetes cluster. Do not execute these experiments against a production environment.

The complete source code is available here:

Kubernetes Production Readiness Lab — GitHub

Clone the repository

git clone https://github.com/rafaterrero/kubernetes-production-readiness-lab.git

cd kubernetes-production-readiness-lab
Enter fullscreen mode Exit fullscreen mode

Create an isolated Kind cluster

We'll use a separate kubeconfig so the lab doesn't interfere with existing Kubernetes configurations.

mkdir -p "$HOME/.kube"

export KUBECONFIG="$HOME/.kube/readiness-lab-kind.yaml"

kind create cluster \
  --name readiness-lab \
  --image kindest/node:v1.37.0 \
  --kubeconfig "$KUBECONFIG" \
  --wait 180s
Enter fullscreen mode Exit fullscreen mode

This command assumes a cluster with that name doesn't already exist.

Verify the context:

kubectl config current-context

kubectl get nodes
Enter fullscreen mode Exit fullscreen mode

Expected context:

kind-readiness-lab
Enter fullscreen mode Exit fullscreen mode

A single-node Kind cluster is sufficient for this experiment, provided it has enough resources for the original Pod and the additional Pod created during a rolling update.

The application manifest

The repository includes the complete manifest at:

manifests/readiness-lab.yaml

For clarity, here is the actual configuration used in the experiments:

apiVersion: v1
kind: Namespace
metadata:
  name: readiness-lab
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: probe-demo-code
  namespace: readiness-lab
data:
  app.py: |
    import os
    from pathlib import Path
    from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

    class Handler(BaseHTTPRequestHandler):
        def do_GET(self):
            routes = {
                "/startup": True,
                "/live": not Path("/tmp/not-live").exists(),
                "/ready": (
                    os.getenv("READY_ON_START", "true") == "true"
                    and not Path("/tmp/not-ready").exists()
                ),
                "/checkout": not Path("/tmp/checkout-down").exists(),
            }

            if self.path not in routes:
                status = 404
            else:
                status = 200 if routes[self.path] else 503

            self.send_response(status)
            self.send_header("Content-Type", "text/plain")
            self.end_headers()
            self.wfile.write(
                f"{self.path}: {status}\n".encode()
            )

    ThreadingHTTPServer(
        ("0.0.0.0", 8080), Handler
    ).serve_forever()
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: readiness-demo
  namespace: readiness-lab
spec:
  replicas: 1
  revisionHistoryLimit: 3
  minReadySeconds: 5
  progressDeadlineSeconds: 90
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: readiness-demo
  template:
    metadata:
      labels:
        app: readiness-demo
    spec:
      containers:
        - name: web
          image: python:3.12-alpine
          command:
            - python
            - -u
            - /app/app.py
          env:
            - name: READY_ON_START
              value: "true"
          ports:
            - name: http
              containerPort: 8080
          volumeMounts:
            - name: code
              mountPath: /app
              readOnly: true
          startupProbe:
            httpGet:
              path: /startup
              port: http
            periodSeconds: 2
            timeoutSeconds: 2
            failureThreshold: 30
          readinessProbe:
            httpGet:
              path: /ready
              port: http
            periodSeconds: 3
            timeoutSeconds: 2
            failureThreshold: 1
          livenessProbe:
            httpGet:
              path: /live
              port: http
            periodSeconds: 5
            timeoutSeconds: 2
            failureThreshold: 3
      volumes:
        - name: code
          configMap:
            name: probe-demo-code
---
apiVersion: v1
kind: Service
metadata:
  name: readiness-demo
  namespace: readiness-lab
spec:
  selector:
    app: readiness-demo
  ports:
    - name: http
      port: 80
      targetPort: http
Enter fullscreen mode Exit fullscreen mode

A few configuration decisions are intentional.

maxUnavailable: 0 prevents the rolling update from intentionally reducing the number of available replicas below the desired count.

maxSurge: 1 permits one additional Pod during the update. The cluster therefore needs enough resources for that extra Pod.

minReadySeconds: 5 requires a newly ready Pod to remain ready for five seconds before it counts as available.

progressDeadlineSeconds: 90 defines the Deployment controller's deadline for detecting stalled progress.

These settings influence rollout behavior.

They do not validate business functionality.

Deploy the application

From the repository root:

kubectl apply -f manifests/readiness-lab.yaml

kubectl -n readiness-lab rollout status \
  deployment/readiness-demo \
  --timeout=240s

kubectl -n readiness-lab get deployments,pods,svc
Enter fullscreen mode Exit fullscreen mode

The rollout should finish successfully, assuming the image can be pulled and the cluster has sufficient resources.

In my verified environment, the initial Deployment reached:

NAME             READY   UP-TO-DATE   AVAILABLE
readiness-demo   1/1     1            1
Enter fullscreen mode Exit fullscreen mode

Verify the baseline

Instead of relying on a local port-forward, we can query the application through its Kubernetes Service from inside the cluster.

kubectl -n readiness-lab exec -i \
  deployment/readiness-demo -- python - <<'PY'
from urllib.request import urlopen

base = "http://readiness-demo.readiness-lab.svc.cluster.local"

for path in ("/ready", "/checkout"):
    with urlopen(base + path, timeout=5) as response:
        print(f"{path}: HTTP {response.status}")
PY
Enter fullscreen mode Exit fullscreen mode

Expected result:

/ready: HTTP 200
/checkout: HTTP 200
Enter fullscreen mode Exit fullscreen mode

This verifies Service DNS resolution and HTTP access through the Kubernetes Service from a Pod.

It does not, by itself, validate access from external clients.

We now have a healthy baseline.

3. Failure mode #1: A healthy Deployment with broken business functionality

Let's simulate a business-operation failure.

kubectl -n readiness-lab exec \
  deployment/readiness-demo \
  -- touch /tmp/checkout-down
Enter fullscreen mode Exit fullscreen mode

This creates a flag inside the application container.

The application will now return HTTP 503 from /checkout, while /ready remains unaffected.

Inspect both endpoints

kubectl -n readiness-lab exec -i \
  deployment/readiness-demo -- python - <<'PY'
from urllib.request import urlopen
from urllib.error import HTTPError

for path in ("/ready", "/checkout"):
    try:
        with urlopen(
            f"http://127.0.0.1:8080{path}",
            timeout=5
        ) as response:
            status = response.status
    except HTTPError as exc:
        status = exc.code

    print(f"{path}: HTTP {status}")
PY
Enter fullscreen mode Exit fullscreen mode

Actual result observed:

/ready: HTTP 200
/checkout: HTTP 503
Enter fullscreen mode Exit fullscreen mode

Now inspect the Deployment:

kubectl -n readiness-lab get deployments

kubectl -n readiness-lab get pods
Enter fullscreen mode Exit fullscreen mode

The actual output showed:

NAME             READY   UP-TO-DATE   AVAILABLE
readiness-demo   1/1     1            1
Enter fullscreen mode Exit fullscreen mode

And the Pod remained:

READY   STATUS    RESTARTS
1/1     Running   0
Enter fullscreen mode Exit fullscreen mode

What happened?

The readiness probe continued succeeding because /ready was unaffected.

The liveness probe continued succeeding too.

From Kubernetes' perspective, the application had not reported a health problem.

Yet our simulated business operation was failing.

This is the central lesson of the lab: Pod readiness is only as meaningful as the application-level conditions it represents.

There is no universal rule that every business operation must be part of readiness.

In fact, making readiness depend on every external integration can create other problems.

Imagine every replica failing readiness simultaneously because a shared payment provider experiences a brief outage.

Removing all Pods from Service routing might turn a partial failure into a broader outage.

A more robust design could keep the application available for unaffected operations, return a controlled error for the failed integration, and alert the relevant team.

The correct readiness contract depends on the application's responsibilities.

We should therefore distinguish between:

  • Infrastructure availability.
  • Application readiness.
  • Dependency health.
  • Business functionality.
  • User-facing reliability.

These are related signals, not interchangeable ones.

Restore functionality

kubectl -n readiness-lab exec \
  deployment/readiness-demo \
  -- rm /tmp/checkout-down
Enter fullscreen mode Exit fullscreen mode

4. Failure mode #2: Readiness failure without container restart

Now let's simulate an application that is running but temporarily unable to receive traffic.

kubectl -n readiness-lab exec \
  deployment/readiness-demo \
  -- touch /tmp/not-ready
Enter fullscreen mode Exit fullscreen mode

Wait for Kubernetes to detect the condition:

kubectl -n readiness-lab wait \
  --for=condition=Ready=false \
  pod -l app=readiness-demo \
  --timeout=45s
Enter fullscreen mode Exit fullscreen mode

Inspect the Pod and Deployment

kubectl -n readiness-lab get pods

kubectl -n readiness-lab get deployments
Enter fullscreen mode Exit fullscreen mode

Observed results:

NAME                              READY   STATUS    RESTARTS
readiness-demo-6b9f5b684b-44hnp   0/1     Running   0
Enter fullscreen mode Exit fullscreen mode

And:

NAME             READY   UP-TO-DATE   AVAILABLE
readiness-demo   0/1     1            0
Enter fullscreen mode Exit fullscreen mode

The container had not restarted.

The process was still running.

But Kubernetes no longer considered the Pod ready to receive normal Service traffic.

Inspect the EndpointSlice

kubectl -n readiness-lab get endpointslices \
  -l kubernetes.io/service-name=readiness-demo \
  -o yaml
Enter fullscreen mode Exit fullscreen mode

The actual EndpointSlice contained:

conditions:
  ready: false
  serving: false
  terminating: false
Enter fullscreen mode Exit fullscreen mode

The endpoint still existed in the EndpointSlice, but it was no longer eligible for normal Service routing.

With only one replica, our Service had no ready backend.

In a multi-replica workload, other ready replicas could continue serving traffic.

This distinction is particularly important during rolling updates and temporary application initialization.

Readiness failure changes traffic eligibility. It does not necessarily require restarting the process.

Restore readiness

kubectl -n readiness-lab exec \
  deployment/readiness-demo \
  -- rm /tmp/not-ready
Enter fullscreen mode Exit fullscreen mode

Wait for recovery:

kubectl -n readiness-lab wait \
  --for=condition=Ready \
  pod -l app=readiness-demo \
  --timeout=45s

kubectl -n readiness-lab rollout status \
  deployment/readiness-demo \
  --timeout=60s
Enter fullscreen mode Exit fullscreen mode

In the verified experiment, readiness recovered without a container restart.

5. Failure mode #3: A rollout blocked by readiness

Now let's simulate releasing a version that starts but never becomes ready.

Our Deployment has:

replicas: 1
strategy:
  type: RollingUpdate
  rollingUpdate:
    maxSurge: 1
    maxUnavailable: 0
Enter fullscreen mode Exit fullscreen mode

This allows Kubernetes to create an additional Pod while preserving the previous available replica.

It is an important property, but not an unconditional zero-downtime guarantee.

Introduce a defective version

Change the environment variable in the Deployment template:

kubectl -n readiness-lab set env \
  deployment/readiness-demo \
  READY_ON_START=false
Enter fullscreen mode Exit fullscreen mode

This triggers a new rollout.

The new Pod starts.

Its /startup and /live endpoints remain healthy.

However, /ready returns HTTP 503.

Inspect the Pods

kubectl -n readiness-lab get pods \
  -l app=readiness-demo
Enter fullscreen mode Exit fullscreen mode

In my experiment, the new and previous versions coexisted:

NAME                              READY   STATUS
readiness-demo-65f8578575-qp4gg   0/1     Running
readiness-demo-6b9f5b684b-44hnp   1/1     Running
Enter fullscreen mode Exit fullscreen mode

The previous version remained available.

The new version was running but never became ready.

Wait for the failed rollout

kubectl -n readiness-lab rollout status \
  deployment/readiness-demo \
  --timeout=130s
Enter fullscreen mode Exit fullscreen mode

Because the new replica remains unavailable, the Deployment controller should eventually exceed its configured 90-second progress deadline.

The actual result was:

Waiting for deployment "readiness-demo" rollout to finish:
1 old replicas are pending termination...

error: deployment "readiness-demo" exceeded its progress deadline
Enter fullscreen mode Exit fullscreen mode

Two different timeouts

There is an important distinction here.

The --timeout=130s option belongs to the kubectl client. It limits how long the command waits.

The Deployment's progressDeadlineSeconds: 90 belongs to the Kubernetes controller. It determines when the controller reports stalled rollout progress.

A client-side timeout does not automatically mean Kubernetes has declared the Deployment failed.

That is why we should inspect the controller's actual conditions.

Inspect the Deployment state

kubectl -n readiness-lab describe \
  deployment readiness-demo
Enter fullscreen mode Exit fullscreen mode

The verified experiment reported:

Replicas: 1 desired | 1 updated | 2 total |
          1 available | 1 unavailable
Enter fullscreen mode Exit fullscreen mode

And:

Type           Status  Reason
----           ------  ------
Available      True    MinimumReplicasAvailable
Progressing    False   ProgressDeadlineExceeded
Enter fullscreen mode Exit fullscreen mode

This is one of the most interesting observations of the entire lab.

The Deployment was still available.

But its latest rollout had failed.

In other words:

Available=True does not mean the newest Deployment revision was successfully rolled out.

The previous healthy version was maintaining availability while Kubernetes reported that the update could not progress.

Verify the Service endpoints

kubectl -n readiness-lab get endpointslices \
  -l kubernetes.io/service-name=readiness-demo \
  -o jsonpath='{range .items[*].endpoints[*]}{.targetRef.name}{" ready="}{.conditions.ready}{"\n"}{end}'
Enter fullscreen mode Exit fullscreen mode

Observed result:

readiness-demo-6b9f5b684b-44hnp ready=true
readiness-demo-65f8578575-qp4gg ready=false
Enter fullscreen mode Exit fullscreen mode

The old version remained eligible for Service traffic.

The new version did not.

Roll back the failed revision

kubectl -n readiness-lab rollout undo \
  deployment/readiness-demo
Enter fullscreen mode Exit fullscreen mode

This restores the previous Pod template.

However, it does not automatically reverse external side effects such as database migrations or changes to shared state.

A Kubernetes rollback is not necessarily an application rollback.

There is another operational detail worth highlighting.

Immediately after rollout undo, my first rollout status check still reported the previous progress-deadline failure.

But the previous healthy Pod was already available.

A subsequent inspection showed that the Deployment controller had completed reconciliation.

We can inspect the relevant conditions with:

kubectl -n readiness-lab get deployment readiness-demo \
  -o jsonpath='{range .status.conditions[*]}{.type}{" = "}{.status}{" | "}{.reason}{"\n"}{end}'
Enter fullscreen mode Exit fullscreen mode

And verify the desired generation:

kubectl -n readiness-lab get deployment readiness-demo \
  -o jsonpath='generation={.metadata.generation} observed={.status.observedGeneration} replicas={.status.replicas} updated={.status.updatedReplicas} available={.status.availableReplicas}{"\n"}'
Enter fullscreen mode Exit fullscreen mode

The recovered Deployment reported:

Available = True | MinimumReplicasAvailable
Progressing = True | NewReplicaSetAvailable
Enter fullscreen mode Exit fullscreen mode

And:

generation=3 observed=3 replicas=1 updated=1 available=1
Enter fullscreen mode Exit fullscreen mode

The defective ReplicaSet had been scaled to zero.

One additional consideration: kubectl rollout undo does not update the last-applied-configuration annotation used by kubectl apply. Subsequent declarative changes should account for this distinction.

The lesson is clear:

Initiating a rollback is not the same as verifying successful recovery.

A release pipeline should confirm the intended revision, controller conditions, available replicas, and application behavior before declaring recovery complete.

6. Failure mode #4: Liveness failure and automatic restart

So far, we have intentionally avoided using liveness to detect temporary readiness or business-functionality failures.

Now let's demonstrate what happens when liveness actually fails.

Inject a liveness failure

kubectl -n readiness-lab exec \
  deployment/readiness-demo \
  -- touch /tmp/not-live
Enter fullscreen mode Exit fullscreen mode

The /live endpoint now returns HTTP 503.

Our Deployment uses:

livenessProbe:
  httpGet:
    path: /live
    port: http
  periodSeconds: 5
  timeoutSeconds: 2
  failureThreshold: 3
Enter fullscreen mode Exit fullscreen mode

After three consecutive failures, kubelet should restart the container.

Observe the restart

kubectl -n readiness-lab get pods -w
Enter fullscreen mode Exit fullscreen mode

Stop watching with Ctrl+C after recovery.

Then inspect the Pod:

kubectl -n readiness-lab describe pods \
  -l app=readiness-demo
Enter fullscreen mode Exit fullscreen mode

The actual result showed that the container restart count increased:

RESTARTS
1
Enter fullscreen mode Exit fullscreen mode

The kubelet events provided the reason:

Warning  Unhealthy  Liveness probe failed:
HTTP probe failed with statuscode: 503

Normal   Killing    Container web failed
liveness probe, will be restarted
Enter fullscreen mode Exit fullscreen mode

The previous container had:

Last State: Terminated
Reason: Error
Exit Code: 137
Enter fullscreen mode Exit fullscreen mode

Exit code 137 alone does not prove an out-of-memory failure.

In this experiment, the kubelet events explicitly identified the failing liveness probe as the reason for the restart.

After restarting, the Pod recovered:

READY   STATUS    RESTARTS
1/1     Running   1
Enter fullscreen mode Exit fullscreen mode

The temporary failure flag disappeared because it was stored in the container's writable filesystem.

Why this distinction matters

In production, a failing liveness probe might indicate a deadlock or another condition from which restarting the process is an appropriate recovery action.

But restarting an application because a remote database is temporarily unavailable can be counterproductive.

If the shared dependency is failing, indiscriminately restarting every dependent application could increase load, erase useful in-memory state, and slow recovery.

Liveness is a recovery mechanism, not a general-purpose dependency monitor.

7. The missing layer: Post-deployment verification

Our first experiment demonstrated a problem Kubernetes health probes alone did not detect.

The application was ready, but an important operation failed.

This is where deployment verification must move beyond infrastructure signals.

A simple release gate should verify both the Kubernetes state and representative application behavior.

Check rollout status

kubectl -n readiness-lab rollout status \
  deployment/readiness-demo \
  --timeout=180s
Enter fullscreen mode Exit fullscreen mode

This confirms that the rollout has completed according to the Deployment controller.

It does not verify business functionality.

Verify an application endpoint through the Service

For our lab, we can query the simulated business endpoint using Kubernetes Service DNS.

kubectl -n readiness-lab exec \
  deployment/readiness-demo -- \
  python -c 'from urllib.request import urlopen; print(urlopen("http://readiness-demo.readiness-lab.svc.cluster.local/checkout", timeout=5).status)'
Enter fullscreen mode Exit fullscreen mode

A successful response prints:

200
Enter fullscreen mode Exit fullscreen mode

These checks answer different questions.

The first asks whether the Kubernetes rollout completed.

The second asks whether a particular application operation is functioning.

For this experiment, the endpoint represents a safe synthetic operation.

In production, application-level checks should exercise representative functionality without causing unwanted side effects.

Additional verification layers

A production-grade verification process might include:

Layer Example verification
Deployment Desired replicas updated and available
Application Synthetic transaction or critical API request
Reliability Error rate, latency, availability
Dependencies Connectivity and dependency-specific behavior
Data Schema and version compatibility
Security Expected authentication and authorization behavior
Rollback Known recovery procedure and compatibility checks

Depending on the service, the process can also include canary releases, progressive traffic shifting, automated analysis of service-level indicators, and human approval for high-risk changes.

There is no universal release gate.

But there should be an explicit difference between:

  • Kubernetes accepted this version.
  • The application passed our operational acceptance criteria.

8. Automating healthy-state verification

The repository includes a verification script:

scripts/verify.sh

It checks:

  • The expected Kubernetes context.
  • Deployment rollout status.
  • Deployment and Pod state.
  • The /ready and /checkout endpoints.
  • EndpointSlice readiness.

Execute it from the repository root:

export KUBECONFIG="$HOME/.kube/readiness-lab-kind.yaml"

./scripts/verify.sh
Enter fullscreen mode Exit fullscreen mode

The script was executed successfully in the original lab.

Its final output included:

=== Application Service Check ===
/ready: HTTP 200
/checkout: HTTP 200

=== EndpointSlices ===
readiness-demo-6b9f5b684b-44hnp ready=true

Healthy application verification completed.
Enter fullscreen mode Exit fullscreen mode

The script verifies the healthy state.

It does not intentionally introduce failures.

The four failure scenarios are documented separately for manual reproduction.

Independent verification using GitHub Actions

To make the lab independently reproducible, I also added a GitHub Actions workflow.

The workflow:

  1. Checks out the repository.
  2. Creates a fresh Kind cluster.
  3. Verifies the Kubernetes context.
  4. Deploys the application.
  5. Waits for the rollout.
  6. Executes scripts/verify.sh.
  7. Collects diagnostics if a step fails.

The first GitHub Actions execution completed successfully.

You can inspect the actual run here:

Successful Kubernetes Readiness Lab CI execution

This independently verifies the healthy deployment.

The four failure-injection scenarios were validated manually in the original Kind cluster and are not yet automated in CI.

That distinction matters: a green CI pipeline proves only what its tests actually exercise.

9. Production recommendations

Here are the principles I would apply when designing readiness and deployment controls for critical services.

Define probe contracts with application developers

A platform team should not guess what application readiness means.

The application owner should establish which internal conditions prevent the service from handling its intended traffic.

Separate temporary unavailability from process failure

Readiness can stop traffic without forcing a restart.

Liveness should be reserved for failures where restarting is an appropriate recovery mechanism.

Use startup probes where startup time is variable

They provide an explicit startup window instead of forcing liveness checks to tolerate lengthy initialization.

Treat successful rollouts as necessary but insufficient evidence

Deployment status should be combined with application-specific validation.

Design rollback before release

Consider configuration changes, schema compatibility, messages in queues, persistent state, and external dependencies.

Validate the deployment strategy against real capacity

maxSurge requires additional resources.

maxUnavailable constrains availability during rolling updates.

Neither setting alone proves zero downtime.

Harden the workload independently of health checks

Our lab uses a convenient Python image and an intentionally minimal manifest.

A production deployment should additionally consider:

  • Pinned image versions or digests.
  • Resource requests and limits.
  • Non-root execution.
  • Restricted container permissions.
  • Dependency management.
  • Network policies.
  • Supply-chain controls.
  • Suitable logging and observability.

Health probes do not replace these safeguards.

10. What this means for Platform Engineering

A Platform Engineering team should aim to make reliable deployment practices reusable.

Instead of forcing every development team to reinvent probes, release validation, rollback procedures, and observability, an internal platform can establish sensible defaults and clear extension points.

That might include:

  • Standardized deployment templates.
  • Documented probe contracts.
  • Reusable post-deployment verification.
  • Consistent metrics, logs, and traces.
  • Progressive delivery capabilities.
  • Automated and manual recovery procedures.

The objective is not to hide every operational decision from developers.

It is to reduce unnecessary complexity while making the important decisions visible, testable, and repeatable.

A good internal platform should make the safe path easier to follow, not merely provide more YAML to maintain.

11. Verified Lab Results — Kubernetes 1.37.0

All four failure scenarios described in this article were executed on a disposable Kind cluster running Kubernetes 1.37.0, hosted on Ubuntu 24.04 with Docker 29.10.0.

The results below are actual observations from the lab, not fabricated terminal outputs or production benchmarks.

Experiment Verified result
Business failure /ready returned HTTP 200 while /checkout returned HTTP 503. The Deployment remained available.
Readiness failure The Pod became 0/1 Ready without restarting. Its EndpointSlice reported ready=false and serving=false.
Failed rollout The new Pod remained unready while the previous Pod stayed available. Kubernetes reported ProgressDeadlineExceeded.
Rollback The healthy template was restored. The Deployment subsequently reported Progressing=True with NewReplicaSetAvailable.
Service recovery Both /ready and /checkout returned HTTP 200 through Kubernetes Service DNS.
Liveness failure Three HTTP 503 probe failures triggered a container restart. The Pod recovered with Ready=True.
CI verification GitHub Actions successfully deployed and verified the healthy application in a separate Kind cluster.

A particularly important observation

During the failed-rollout experiment, the Deployment simultaneously reported:

Available = True
Progressing = False
Reason = ProgressDeadlineExceeded
Enter fullscreen mode Exit fullscreen mode

The previous Pod was still eligible for Service traffic.

The newest Pod was not.

This demonstrates that application availability and deployment progression are separate operational signals.

During recovery, another distinction became visible.

Immediately after initiating the rollback, the rollout status command still reported the earlier failure.

Only after the controller reconciled the restored revision did the Deployment report:

Available = True | MinimumReplicasAvailable
Progressing = True | NewReplicaSetAvailable
Enter fullscreen mode Exit fullscreen mode

This reinforces why rollback initiation and verified recovery must be treated as separate steps.

12. Reproduce the experiments

The complete source code, Kubernetes manifests, experiment instructions, and verified results are publicly available:

Kubernetes Production Readiness Lab — GitHub

The repository includes:

File or directory Purpose
manifests/readiness-lab.yaml Kubernetes application and probes
scripts/verify.sh Healthy-state validation
docs/architecture.md Architecture and deployment strategy
docs/experiments.md Instructions for all four experiments
docs/verified-results.md Results observed in the original lab
evidence/final-recovery.txt Captured final recovery state
.github/workflows/kind-smoke.yml GitHub Actions verification workflow

The project is open source under the MIT License.

Cleanup

When you finish the experiments, remove the lab namespace:

kubectl delete namespace readiness-lab
Enter fullscreen mode Exit fullscreen mode

This removes the application resources from the test cluster.

You can also delete the entire disposable Kind cluster:

kind delete cluster --name readiness-lab
Enter fullscreen mode Exit fullscreen mode

The original testing environment was cleaned up after validation. The source code and documented evidence remain available in GitHub.

Final thoughts

Kubernetes is very good at managing desired state.

It can schedule Pods, restart unhealthy containers, update replicas, and manage traffic eligibility based on readiness.

But its view of application health depends on the signals and contracts we provide.

Our experiments demonstrated that:

  • A Deployment can be available while a critical business operation fails.
  • A container can be alive without being ready.
  • A Pod can be ready while an external integration is broken.
  • A Deployment can remain available while its latest rollout has failed.
  • A rollback can be initiated before recovery has been fully verified.
  • A liveness failure can trigger a restart without requiring the Pod to be recreated.

These are not reasons to distrust Kubernetes.

They are reasons to understand precisely what Kubernetes does and does not guarantee.

Production readiness is not a status field. It is an engineering contract supported by verification, observability, and recovery.

That is the distinction worth designing for.


References

The behavior discussed in this article is based on the official Kubernetes documentation:

  1. Kubernetes — Liveness, Readiness, and Startup Probes
  2. Kubernetes — Configure Liveness, Readiness and Startup Probes
  3. Kubernetes — Deployments
  4. Kubernetes — EndpointSlices
  5. Kubernetes — kubectl rollout status
  6. Kubernetes — kubectl rollout undo

Source code and reproducible experiments:

Kubernetes Production Readiness Lab — GitHub


I'm Rafa Terrero, a Senior Platform Engineer working across cloud infrastructure, DevOps, Kubernetes, and production reliability. I write about architecture decisions, engineering trade-offs, and the challenges of building dependable platforms, including infrastructure for AI-powered systems.

Top comments (0)