<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Rafa Terrero</title>
    <description>The latest articles on DEV Community by Rafa Terrero (@rafaterrero).</description>
    <link>https://dev.to/rafaterrero</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F2054871%2F7e0e2bc1-8f00-4bf5-9fa3-3a39d569d102.png</url>
      <title>DEV Community: Rafa Terrero</title>
      <link>https://dev.to/rafaterrero</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/rafaterrero"/>
    <language>en</language>
    <item>
      <title>Terraform Is Not Your Internal Platform</title>
      <dc:creator>Rafa Terrero</dc:creator>
      <pubDate>Thu, 19 Nov 2026 08:53:14 +0000</pubDate>
      <link>https://dev.to/rafaterrero/terraform-is-not-your-internal-platform-28jj</link>
      <guid>https://dev.to/rafaterrero/terraform-is-not-your-internal-platform-28jj</guid>
      <description>&lt;p&gt;&lt;em&gt;Why reusable infrastructure modules are not the same as a reusable developer experience.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;You can have well-designed Terraform modules, remote state, CI/CD pipelines, automated provisioning, and standardized security configurations.&lt;/p&gt;

&lt;p&gt;And still force every development team to understand far more infrastructure than they should need to.&lt;/p&gt;

&lt;p&gt;That's an important distinction.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Infrastructure as Code makes infrastructure reproducible. An internal developer platform should make supported engineering workflows easier to consume.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Terraform can be a critical component of that platform. But it isn't the whole platform.&lt;/p&gt;

&lt;h2&gt;
  
  
  The infrastructure is automated. Why is delivery still difficult?
&lt;/h2&gt;

&lt;p&gt;Imagine an engineering organization running applications on AWS.&lt;/p&gt;

&lt;p&gt;Its infrastructure team has invested in:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Reusable Terraform modules for networking, IAM, compute, and databases.&lt;/li&gt;
&lt;li&gt;Remote state management.&lt;/li&gt;
&lt;li&gt;Automated infrastructure deployment pipelines.&lt;/li&gt;
&lt;li&gt;Standardized tagging and security configurations.&lt;/li&gt;
&lt;li&gt;Code review and approval workflows.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;From an infrastructure perspective, this is meaningful progress.&lt;/p&gt;

&lt;p&gt;Now imagine a developer joining the organization to deploy a new backend service.&lt;/p&gt;

&lt;p&gt;They need to understand which modules to use, how those modules interact, which IAM permissions are required, how networking is configured, where application secrets belong, and how deployments work across environments.&lt;/p&gt;

&lt;p&gt;Infrastructure provisioning has become repeatable.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;But software delivery has not necessarily become self-service.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;This is where the distinction between Infrastructure as Code and Platform Engineering becomes particularly relevant.&lt;/p&gt;

&lt;h2&gt;
  
  
  Terraform solves an important problem - but not every problem
&lt;/h2&gt;

&lt;p&gt;Terraform provides a declarative approach to managing infrastructure.&lt;/p&gt;

&lt;p&gt;Reusable modules allow infrastructure teams to encapsulate configurations, reduce duplication, and establish common patterns.&lt;/p&gt;

&lt;p&gt;These are valuable capabilities, especially in organizations operating across multiple teams and environments.&lt;/p&gt;

&lt;p&gt;But developers generally think about applications rather than infrastructure resources.&lt;/p&gt;

&lt;p&gt;A developer might say:&lt;/p&gt;

&lt;p&gt;"I need to deploy a containerized service to staging, with the standard observability configuration."&lt;/p&gt;

&lt;p&gt;They are describing an application requirement.&lt;/p&gt;

&lt;p&gt;They are not necessarily asking to select Terraform modules, configure provider-specific parameters, or manage relationships between infrastructure resources.&lt;/p&gt;

&lt;p&gt;An internal developer platform should help translate those requirements into supported operational capabilities.&lt;/p&gt;

&lt;p&gt;Terraform may still do much of the work underneath.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The architectural question is which complexity the platform should own, and which complexity developers actually need to understand.&lt;/strong&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The difference is in the interface
&lt;/h2&gt;

&lt;p&gt;Consider two approaches that eventually rely on the same infrastructure provisioning technology.&lt;/p&gt;

&lt;h3&gt;
  
  
  Approach A: Exposing infrastructure configuration
&lt;/h3&gt;

&lt;p&gt;In this approach, application teams interact directly with Terraform inputs.&lt;/p&gt;

&lt;p&gt;A simplified configuration might look like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight hcl"&gt;&lt;code&gt;&lt;span class="n"&gt;application_name&lt;/span&gt;      &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"payments-api"&lt;/span&gt;
&lt;span class="n"&gt;environment&lt;/span&gt;           &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"staging"&lt;/span&gt;
&lt;span class="n"&gt;runtime&lt;/span&gt;               &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"container"&lt;/span&gt;
&lt;span class="n"&gt;observability_profile&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"standard"&lt;/span&gt;

&lt;span class="n"&gt;standard_tags&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;managed&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;by&lt;/span&gt;  &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"example-platform"&lt;/span&gt;
  &lt;span class="n"&gt;environment&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"staging"&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This approach is not inherently wrong.&lt;/p&gt;

&lt;p&gt;For infrastructure-focused teams, direct Terraform access may provide exactly the flexibility and transparency they need.&lt;/p&gt;

&lt;p&gt;But when the same model is applied indiscriminately across development teams, it can transfer infrastructure-specific responsibilities to people whose primary concern is delivering application functionality.&lt;/p&gt;

&lt;h3&gt;
  
  
  Approach B: Exposing application capabilities
&lt;/h3&gt;

&lt;p&gt;Now consider a platform-owned contract:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;platform.example.io/v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Application&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;payments-api&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;runtime&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;container&lt;/span&gt;
  &lt;span class="na"&gt;environment&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;staging&lt;/span&gt;
  &lt;span class="na"&gt;observability&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;profile&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;standard&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The application team describes what it needs through a supported interface.&lt;/p&gt;

&lt;p&gt;A platform-owned component validates the request and translates it into infrastructure configuration.&lt;/p&gt;

&lt;p&gt;The platform can enforce supported environments, naming conventions, operational profiles, and standardized metadata without requiring every development team to configure those details independently.&lt;/p&gt;

&lt;p&gt;Terraform remains underneath.&lt;/p&gt;

&lt;p&gt;The difference is the interface and the ownership of the provisioning decisions.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The platform does not eliminate infrastructure complexity. It gives that complexity a deliberate owner.&lt;/strong&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  A practical reference implementation
&lt;/h2&gt;

&lt;p&gt;To make this distinction concrete, I built a small reference implementation:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;a href="https://github.com/rafaterrero/terraform-is-not-a-platform" rel="noopener noreferrer"&gt;terraform-is-not-a-platform — GitHub&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The project demonstrates two ways of representing similar application requirements.&lt;/p&gt;

&lt;p&gt;In the direct Terraform example, consumers provide infrastructure-oriented variables.&lt;/p&gt;

&lt;p&gt;In the platform abstraction example, consumers declare application capabilities using YAML.&lt;/p&gt;

&lt;p&gt;A small Python CLI validates the contract and generates native Terraform JSON configuration.&lt;/p&gt;

&lt;p&gt;The architecture is deliberately minimal:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Approach A

Application team
      |
      v
Terraform variables
      |
      v
Terraform configuration


Approach B

Application team
      |
      v
Application contract (YAML)
      |
      v
Platform-owned Python adapter
      |
      +-- Contract validation
      +-- Supported values
      +-- Standard metadata
      |
      v
Generated Terraform JSON
      |
      v
Terraform configuration
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The Python adapter provides a narrow set of platform-owned constraints:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Supported environments: &lt;code&gt;dev&lt;/code&gt;, &lt;code&gt;staging&lt;/code&gt;, and &lt;code&gt;production&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Supported runtime: &lt;code&gt;container&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Observability profiles: &lt;code&gt;standard&lt;/code&gt; and &lt;code&gt;enhanced&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Restricted application naming conventions.&lt;/li&gt;
&lt;li&gt;Rejection of unknown contract fields.&lt;/li&gt;
&lt;li&gt;Automatic injection of standardized tags.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For example, the application team does not need to define the &lt;code&gt;managed-by&lt;/code&gt; tag. The adapter generates it as part of the infrastructure configuration.&lt;/p&gt;

&lt;p&gt;The generated output is a valid &lt;code&gt;main.tf.json&lt;/code&gt; file that Terraform can consume.&lt;/p&gt;

&lt;h3&gt;
  
  
  Try it locally
&lt;/h3&gt;

&lt;p&gt;The project requires Python and, optionally, Terraform for configuration validation.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;git clone \
  https://github.com/rafaterrero/terraform-is-not-a-platform.git

&lt;span class="nb"&gt;cd&lt;/span&gt; terraform-is-not-a-platform

python3 -m venv .venv
&lt;span class="nb"&gt;source&lt;/span&gt; .venv/bin/activate

python -m pip install -r requirements.txt

python -m pytest -q

python -m platform_demo \
  examples/platform-abstraction/service.yaml \
  --output build/main.tf.json
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;You can then validate the generated configuration:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;cd&lt;/span&gt; build

terraform init -backend=false
terraform validate
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The implementation has passed its eight Python tests and Terraform configuration validation, including an automated GitHub Actions run.&lt;/p&gt;

&lt;h3&gt;
  
  
  An important limitation
&lt;/h3&gt;

&lt;p&gt;This is intentionally a local-only demonstration.&lt;/p&gt;

&lt;p&gt;It uses Terraform's built-in &lt;code&gt;terraform_data&lt;/code&gt; resource rather than deploying real AWS services.&lt;/p&gt;

&lt;p&gt;No AWS account, cloud credentials, or paid resources are required.&lt;/p&gt;

&lt;p&gt;The example demonstrates contract validation, abstraction boundaries, and configuration generation.&lt;/p&gt;

&lt;p&gt;It does &lt;strong&gt;not&lt;/strong&gt; implement a production-ready developer platform, authorization, secret management, infrastructure deployment, policy-as-code, service lifecycle management, or operational observability.&lt;/p&gt;

&lt;p&gt;Those responsibilities would require additional architecture and engineering.&lt;/p&gt;

&lt;p&gt;The purpose of the repository is not to show that a YAML file and a Python script constitute a complete internal platform.&lt;/p&gt;

&lt;p&gt;It is to demonstrate how &lt;strong&gt;the interface developers consume can be separated from the infrastructure implementation beneath it&lt;/strong&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  A lesson from working on an internal platform in banking
&lt;/h2&gt;

&lt;p&gt;During my time contributing to the AWS architecture of Gluon Platform at Santander Digital Services, I worked with an internal application development and deployment platform where development teams could request infrastructure capabilities through an abstraction layer without interacting directly with Terraform.&lt;/p&gt;

&lt;p&gt;That separation is an important architectural principle.&lt;/p&gt;

&lt;p&gt;Developers could consume platform capabilities while the underlying infrastructure provisioning approach remained an implementation concern.&lt;/p&gt;

&lt;p&gt;My work in this environment involved AWS architectural patterns, Terraform and CloudFormation, governance considerations, secure deployment models, and technical documentation.&lt;/p&gt;

&lt;p&gt;It reinforced the importance of treating reusable infrastructure and developer-facing capabilities as related but distinct concerns.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Terraform can be part of the platform's implementation without becoming the interface developers are required to understand.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The GitHub example above is an independent educational implementation of that general principle. It does not reproduce Gluon Platform's internal architecture or code.&lt;/p&gt;

&lt;h2&gt;
  
  
  Golden paths should reduce friction, not create another bottleneck
&lt;/h2&gt;

&lt;p&gt;A platform should make common engineering workflows easier and safer.&lt;/p&gt;

&lt;p&gt;That often means providing golden paths: supported approaches to deploying and operating applications, with sensible defaults and established operational practices.&lt;/p&gt;

&lt;p&gt;For example:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Workload identities and appropriate IAM permissions.&lt;/li&gt;
&lt;li&gt;Approved networking patterns.&lt;/li&gt;
&lt;li&gt;Controlled secrets management.&lt;/li&gt;
&lt;li&gt;Standard deployment workflows.&lt;/li&gt;
&lt;li&gt;Logging, metrics, and troubleshooting conventions.&lt;/li&gt;
&lt;li&gt;Clear ownership and support boundaries.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;These capabilities are especially important in regulated environments where security, traceability, and operational consistency matter.&lt;/p&gt;

&lt;p&gt;But standardization alone is not enough.&lt;/p&gt;

&lt;p&gt;A golden path that is too restrictive can become another source of friction.&lt;/p&gt;

&lt;p&gt;Developers need supported defaults, but they also need documented ways to handle legitimate requirements outside the standard path.&lt;/p&gt;

&lt;p&gt;A good platform should reduce unnecessary decisions without removing the information and control teams need to operate their services.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Abstraction should simplify consumption, not make systems impossible to understand or troubleshoot.&lt;/strong&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Platform Engineering is also a product discipline
&lt;/h2&gt;

&lt;p&gt;Infrastructure automation is a technical capability.&lt;/p&gt;

&lt;p&gt;An internal developer platform is also a product consumed by engineering teams.&lt;/p&gt;

&lt;p&gt;That changes how its effectiveness should be evaluated.&lt;/p&gt;

&lt;p&gt;Instead of focusing only on the number of Terraform modules, pipelines, or reusable components, we should ask:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Can a development team onboard a new service without extensive platform-team intervention?&lt;/li&gt;
&lt;li&gt;How long does it take to reach a first successful deployment?&lt;/li&gt;
&lt;li&gt;How much infrastructure-specific knowledge is required?&lt;/li&gt;
&lt;li&gt;Are security and operational standards consistently applied?&lt;/li&gt;
&lt;li&gt;Can teams understand failures and recover without unnecessary escalation?&lt;/li&gt;
&lt;li&gt;Are platform capabilities actually adopted by their intended users?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;These questions should lead to measurable indicators such as onboarding time, supported-path adoption, deployment lead time, support requests, and developer feedback.&lt;/p&gt;

&lt;p&gt;None of these metrics alone proves that a platform is successful.&lt;/p&gt;

&lt;p&gt;But together they provide a more useful picture than simply counting automation artifacts.&lt;/p&gt;

&lt;p&gt;The platform should be evaluated by the engineering outcomes it enables, not merely by the infrastructure components it contains.&lt;/p&gt;

&lt;h2&gt;
  
  
  You may not need a developer portal
&lt;/h2&gt;

&lt;p&gt;Another mistake is assuming that every internal developer platform needs a dedicated web portal.&lt;/p&gt;

&lt;p&gt;A portal can improve discoverability, service ownership visibility, and access to supported workflows.&lt;/p&gt;

&lt;p&gt;But a portal is not the platform itself.&lt;/p&gt;

&lt;p&gt;Depending on organizational needs, a combination of versioned contracts, APIs, command-line tools, templates, CI/CD workflows, and documentation may offer an effective developer experience.&lt;/p&gt;

&lt;p&gt;The right choice depends on the problems engineering teams actually face.&lt;/p&gt;

&lt;p&gt;Building a sophisticated portal before identifying those problems risks creating another system that requires maintenance without delivering meaningful value.&lt;/p&gt;

&lt;p&gt;Start with a repetitive, painful workflow.&lt;/p&gt;

&lt;p&gt;Standardize it.&lt;/p&gt;

&lt;p&gt;Make it secure, observable, and easy to consume.&lt;/p&gt;

&lt;p&gt;Then expand based on actual usage and feedback.&lt;/p&gt;

&lt;h2&gt;
  
  
  Final thoughts
&lt;/h2&gt;

&lt;p&gt;Terraform is not your internal platform.&lt;/p&gt;

&lt;p&gt;Neither is Kubernetes, a collection of CI/CD pipelines, or a developer portal on its own.&lt;/p&gt;

&lt;p&gt;All of them can contribute to a platform.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A platform becomes valuable when those capabilities are assembled into a coherent, supported developer experience that makes software delivery safer and more repeatable.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Terraform can make infrastructure reproducible.&lt;/p&gt;

&lt;p&gt;Platform Engineering should make supported delivery workflows easier to consume and operate.&lt;/p&gt;

&lt;p&gt;And the most useful question may not be how many Terraform modules an organization has built.&lt;/p&gt;

&lt;p&gt;It may be this:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How much infrastructure complexity do developers still need to manage just to ship software safely?&lt;/strong&gt;&lt;/p&gt;




&lt;p&gt;&lt;strong&gt;Reference implementation:&lt;/strong&gt; &lt;a href="https://github.com/rafaterrero/terraform-is-not-a-platform" rel="noopener noreferrer"&gt;Terraform Is Not Your Internal Platform — GitHub&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Further reading:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://developer.hashicorp.com/terraform/language/modules" rel="noopener noreferrer"&gt;HashiCorp — Terraform Modules&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.cncf.io/blog/2025/11/19/what-is-platform-engineering/" rel="noopener noreferrer"&gt;CNCF — Platform Engineering&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://cloud.google.com/blog/products/application-development/common-myths-about-platform-engineering" rel="noopener noreferrer"&gt;Google Cloud — Common myths about Platform Engineering&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>platformengineering</category>
      <category>terraform</category>
      <category>aws</category>
      <category>devops</category>
    </item>
    <item>
      <title>Kubernetes: Deployment Success Is Not Production Readiness</title>
      <dc:creator>Rafa Terrero</dc:creator>
      <pubDate>Mon, 16 Nov 2026 20:42:19 +0000</pubDate>
      <link>https://dev.to/rafaterrero/kubernetes-deployment-success-is-not-production-readiness-cl9</link>
      <guid>https://dev.to/rafaterrero/kubernetes-deployment-success-is-not-production-readiness-cl9</guid>
      <description>&lt;p&gt;&lt;em&gt;A hands-on Kubernetes lab demonstrating why healthy Pods, successful rollouts, and working business operations are not the same thing.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;A Kubernetes Deployment can report a successful rollout while a business-critical endpoint is returning HTTP 503.&lt;/p&gt;

&lt;p&gt;No broken YAML. No crashing container. No failed readiness probe.&lt;/p&gt;

&lt;p&gt;Just an application that Kubernetes considers available but cannot perform an important operation.&lt;/p&gt;

&lt;p&gt;This is not necessarily a Kubernetes bug. It is a misunderstanding of what Kubernetes actually guarantees.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A successful rollout is an infrastructure-level observation, not an end-to-end verification of business functionality.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;I wanted to demonstrate this distinction with something more useful than a theoretical explanation.&lt;/p&gt;

&lt;p&gt;So I built a reproducible Kubernetes lab, deliberately introduced four different failure scenarios, observed how Kubernetes responded, and verified the recovery process.&lt;/p&gt;

&lt;p&gt;The experiments were executed on Kubernetes 1.37.0 using Kind. The repository also includes a successful GitHub Actions workflow that independently validates the healthy deployment.&lt;/p&gt;

&lt;p&gt;Let's walk through the architecture, experiments, and results.&lt;/p&gt;

&lt;h2&gt;
  
  
  1. What Kubernetes actually checks
&lt;/h2&gt;

&lt;p&gt;Kubernetes provides three health probes, each with a different responsibility.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Probe&lt;/th&gt;
&lt;th&gt;Primary question&lt;/th&gt;
&lt;th&gt;Consequence of repeated failure&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;startupProbe&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Has the application started?&lt;/td&gt;
&lt;td&gt;Restart the container&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;readinessProbe&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Can this container receive traffic?&lt;/td&gt;
&lt;td&gt;Mark the container unready and remove it from normal Service traffic eligibility&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;livenessProbe&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Should this container be restarted?&lt;/td&gt;
&lt;td&gt;Restart the container&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;These distinctions matter.&lt;/p&gt;

&lt;p&gt;A liveness probe should not restart an otherwise functional application simply because a shared external dependency is temporarily unavailable.&lt;/p&gt;

&lt;p&gt;A readiness probe should represent the application's ability to handle the traffic it is expected to receive.&lt;/p&gt;

&lt;p&gt;And a startup probe can protect a slow-starting application from premature liveness restarts. When configured, it prevents readiness and liveness checks from executing until startup succeeds.&lt;/p&gt;

&lt;p&gt;But none of these probes automatically understands your business logic.&lt;/p&gt;

&lt;p&gt;Kubernetes knows what your application reports. It does not independently know whether a payment, booking, authentication, or order-processing workflow is correct.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Kubernetes manages the health signals we expose. It does not define what business health means for us.&lt;/strong&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  2. Build a reproducible lab
&lt;/h2&gt;

&lt;p&gt;We will deploy a deliberately simple Python HTTP server using a Kubernetes ConfigMap.&lt;/p&gt;

&lt;p&gt;The application exposes four endpoints:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Endpoint&lt;/th&gt;
&lt;th&gt;Purpose&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;/startup&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Simulate successful application initialization&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;/live&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Report whether the process should remain running&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;/ready&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Report whether the application should receive traffic&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;/checkout&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Simulate an important business operation&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The server supports artificial failure flags stored in &lt;code&gt;/tmp&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;This allows us to demonstrate different failure modes without involving real databases, payment providers, or external infrastructure.&lt;/p&gt;

&lt;h3&gt;
  
  
  Requirements
&lt;/h3&gt;

&lt;p&gt;You'll need:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Docker&lt;/li&gt;
&lt;li&gt;Kind&lt;/li&gt;
&lt;li&gt;kubectl&lt;/li&gt;
&lt;li&gt;Git&lt;/li&gt;
&lt;li&gt;Sufficient resources for two application Pods&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Use a disposable Kubernetes cluster. &lt;strong&gt;Do not execute these experiments against a production environment.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The complete source code is available here:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;a href="https://github.com/rafaterrero/kubernetes-production-readiness-lab" rel="noopener noreferrer"&gt;Kubernetes Production Readiness Lab — GitHub&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  Clone the repository
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;git clone https://github.com/rafaterrero/kubernetes-production-readiness-lab.git

&lt;span class="nb"&gt;cd&lt;/span&gt; kubernetes-production-readiness-lab
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Create an isolated Kind cluster
&lt;/h3&gt;

&lt;p&gt;We'll use a separate kubeconfig so the lab doesn't interfere with existing Kubernetes configurations.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;mkdir -p &lt;span class="s2"&gt;"$HOME/.kube"&lt;/span&gt;

&lt;span class="kd"&gt;export&lt;/span&gt; &lt;span class="nv"&gt;KUBECONFIG&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"$HOME/.kube/readiness-lab-kind.yaml"&lt;/span&gt;

kind create cluster \
  --name readiness-lab \
  --image kindest/node:v1.37.0 \
  --kubeconfig &lt;span class="s2"&gt;"$KUBECONFIG"&lt;/span&gt; \
  --wait 180s
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This command assumes a cluster with that name doesn't already exist.&lt;/p&gt;

&lt;p&gt;Verify the context:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl config current-context

kubectl get nodes
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Expected context:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;kind-readiness-lab
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A single-node Kind cluster is sufficient for this experiment, provided it has enough resources for the original Pod and the additional Pod created during a rolling update.&lt;/p&gt;

&lt;h3&gt;
  
  
  The application manifest
&lt;/h3&gt;

&lt;p&gt;The repository includes the complete manifest at:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;manifests/readiness-lab.yaml&lt;/code&gt;&lt;/p&gt;

&lt;p&gt;For clarity, here is the actual configuration used in the experiments:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Namespace&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;readiness-lab&lt;/span&gt;
&lt;span class="nn"&gt;---
&lt;/span&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ConfigMap&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;probe-demo-code&lt;/span&gt;
  &lt;span class="na"&gt;namespace&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;readiness-lab&lt;/span&gt;
&lt;span class="na"&gt;data&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;app.py&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;|&lt;/span&gt;
    &lt;span class="s"&gt;import os&lt;/span&gt;
    &lt;span class="s"&gt;from pathlib import Path&lt;/span&gt;
    &lt;span class="s"&gt;from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer&lt;/span&gt;

    &lt;span class="na"&gt;class Handler(BaseHTTPRequestHandler)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;def do_GET(self)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="s"&gt;routes = {&lt;/span&gt;
                &lt;span class="na"&gt;"/startup"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;True,&lt;/span&gt;
                &lt;span class="na"&gt;"/live"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;not Path("/tmp/not-live").exists(),&lt;/span&gt;
                &lt;span class="na"&gt;"/ready"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;(&lt;/span&gt;
                    &lt;span class="s"&gt;os.getenv("READY_ON_START", "true") == "true"&lt;/span&gt;
                    &lt;span class="s"&gt;and not Path("/tmp/not-ready").exists()&lt;/span&gt;
                &lt;span class="s"&gt;),&lt;/span&gt;
                &lt;span class="na"&gt;"/checkout"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;not Path("/tmp/checkout-down").exists(),&lt;/span&gt;
            &lt;span class="p"&gt;}&lt;/span&gt;

            &lt;span class="na"&gt;if self.path not in routes&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                &lt;span class="s"&gt;status = 404&lt;/span&gt;
            &lt;span class="na"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                &lt;span class="s"&gt;status = 200 if routes[self.path] else 503&lt;/span&gt;

            &lt;span class="s"&gt;self.send_response(status)&lt;/span&gt;
            &lt;span class="s"&gt;self.send_header("Content-Type", "text/plain")&lt;/span&gt;
            &lt;span class="s"&gt;self.end_headers()&lt;/span&gt;
            &lt;span class="s"&gt;self.wfile.write(&lt;/span&gt;
                &lt;span class="na"&gt;f"{self.path}&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s"&gt;status&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s"&gt;\n".encode()&lt;/span&gt;
            &lt;span class="s"&gt;)&lt;/span&gt;

    &lt;span class="s"&gt;ThreadingHTTPServer(&lt;/span&gt;
        &lt;span class="s"&gt;("0.0.0.0", 8080), Handler&lt;/span&gt;
    &lt;span class="s"&gt;).serve_forever()&lt;/span&gt;
&lt;span class="nn"&gt;---
&lt;/span&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;apps/v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Deployment&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;readiness-demo&lt;/span&gt;
  &lt;span class="na"&gt;namespace&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;readiness-lab&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;replicas&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
  &lt;span class="na"&gt;revisionHistoryLimit&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;
  &lt;span class="na"&gt;minReadySeconds&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;
  &lt;span class="na"&gt;progressDeadlineSeconds&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;90&lt;/span&gt;
  &lt;span class="na"&gt;strategy&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;RollingUpdate&lt;/span&gt;
    &lt;span class="na"&gt;rollingUpdate&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;maxSurge&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
      &lt;span class="na"&gt;maxUnavailable&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
  &lt;span class="na"&gt;selector&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;matchLabels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;app&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;readiness-demo&lt;/span&gt;
  &lt;span class="na"&gt;template&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;app&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;readiness-demo&lt;/span&gt;
    &lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;containers&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;web&lt;/span&gt;
          &lt;span class="na"&gt;image&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;python:3.12-alpine&lt;/span&gt;
          &lt;span class="na"&gt;command&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="s"&gt;python&lt;/span&gt;
            &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="s"&gt;-u&lt;/span&gt;
            &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="s"&gt;/app/app.py&lt;/span&gt;
          &lt;span class="na"&gt;env&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;READY_ON_START&lt;/span&gt;
              &lt;span class="na"&gt;value&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"true"&lt;/span&gt;
          &lt;span class="na"&gt;ports&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;http&lt;/span&gt;
              &lt;span class="na"&gt;containerPort&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;8080&lt;/span&gt;
          &lt;span class="na"&gt;volumeMounts&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;code&lt;/span&gt;
              &lt;span class="na"&gt;mountPath&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;/app&lt;/span&gt;
              &lt;span class="na"&gt;readOnly&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
          &lt;span class="na"&gt;startupProbe&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="na"&gt;httpGet&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
              &lt;span class="na"&gt;path&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;/startup&lt;/span&gt;
              &lt;span class="na"&gt;port&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;http&lt;/span&gt;
            &lt;span class="na"&gt;periodSeconds&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;
            &lt;span class="na"&gt;timeoutSeconds&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;
            &lt;span class="na"&gt;failureThreshold&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;30&lt;/span&gt;
          &lt;span class="na"&gt;readinessProbe&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="na"&gt;httpGet&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
              &lt;span class="na"&gt;path&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;/ready&lt;/span&gt;
              &lt;span class="na"&gt;port&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;http&lt;/span&gt;
            &lt;span class="na"&gt;periodSeconds&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;
            &lt;span class="na"&gt;timeoutSeconds&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;
            &lt;span class="na"&gt;failureThreshold&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
          &lt;span class="na"&gt;livenessProbe&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="na"&gt;httpGet&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
              &lt;span class="na"&gt;path&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;/live&lt;/span&gt;
              &lt;span class="na"&gt;port&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;http&lt;/span&gt;
            &lt;span class="na"&gt;periodSeconds&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;
            &lt;span class="na"&gt;timeoutSeconds&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;
            &lt;span class="na"&gt;failureThreshold&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;
      &lt;span class="na"&gt;volumes&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;code&lt;/span&gt;
          &lt;span class="na"&gt;configMap&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;probe-demo-code&lt;/span&gt;
&lt;span class="nn"&gt;---
&lt;/span&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Service&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;readiness-demo&lt;/span&gt;
  &lt;span class="na"&gt;namespace&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;readiness-lab&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;selector&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;app&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;readiness-demo&lt;/span&gt;
  &lt;span class="na"&gt;ports&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;http&lt;/span&gt;
      &lt;span class="na"&gt;port&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;80&lt;/span&gt;
      &lt;span class="na"&gt;targetPort&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;http&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A few configuration decisions are intentional.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;maxUnavailable: 0&lt;/code&gt; prevents the rolling update from intentionally reducing the number of available replicas below the desired count.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;maxSurge: 1&lt;/code&gt; permits one additional Pod during the update. The cluster therefore needs enough resources for that extra Pod.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;minReadySeconds: 5&lt;/code&gt; requires a newly ready Pod to remain ready for five seconds before it counts as available.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;progressDeadlineSeconds: 90&lt;/code&gt; defines the Deployment controller's deadline for detecting stalled progress.&lt;/p&gt;

&lt;p&gt;These settings influence rollout behavior.&lt;/p&gt;

&lt;p&gt;They do not validate business functionality.&lt;/p&gt;

&lt;h3&gt;
  
  
  Deploy the application
&lt;/h3&gt;

&lt;p&gt;From the repository root:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl apply -f manifests/readiness-lab.yaml

kubectl -n readiness-lab rollout status \
  deployment/readiness-demo \
  --timeout=240s

kubectl -n readiness-lab get deployments,pods,svc
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The rollout should finish successfully, assuming the image can be pulled and the cluster has sufficient resources.&lt;/p&gt;

&lt;p&gt;In my verified environment, the initial Deployment reached:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;NAME             READY   UP-TO-DATE   AVAILABLE
readiness-demo   1/1     1            1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Verify the baseline
&lt;/h3&gt;

&lt;p&gt;Instead of relying on a local port-forward, we can query the application through its Kubernetes Service from inside the cluster.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl -n readiness-lab &lt;span class="nb"&gt;exec&lt;/span&gt; -i \
  deployment/readiness-demo -- python - &lt;span class="o"&gt;&amp;lt;&amp;lt;&lt;/span&gt;&lt;span class="s1"&gt;'PY'&lt;/span&gt;
from urllib.request import urlopen

base &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"http://readiness-demo.readiness-lab.svc.cluster.local"&lt;/span&gt;

&lt;span class="k"&gt;for&lt;/span&gt; path &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"/ready"&lt;/span&gt;, &lt;span class="s2"&gt;"/checkout"&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;:
    with urlopen&lt;span class="o"&gt;(&lt;/span&gt;base + path, &lt;span class="nv"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;5&lt;span class="o"&gt;)&lt;/span&gt; as response:
        print&lt;span class="o"&gt;(&lt;/span&gt;f&lt;span class="s2"&gt;"{path}: HTTP {response.status}"&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
PY
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Expected result:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;/ready: HTTP 200
/checkout: HTTP 200
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This verifies Service DNS resolution and HTTP access through the Kubernetes Service from a Pod.&lt;/p&gt;

&lt;p&gt;It does not, by itself, validate access from external clients.&lt;/p&gt;

&lt;p&gt;We now have a healthy baseline.&lt;/p&gt;

&lt;h2&gt;
  
  
  3. Failure mode #1: A healthy Deployment with broken business functionality
&lt;/h2&gt;

&lt;p&gt;Let's simulate a business-operation failure.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl -n readiness-lab &lt;span class="nb"&gt;exec&lt;/span&gt; \
  deployment/readiness-demo \
  -- touch /tmp/checkout-down
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This creates a flag inside the application container.&lt;/p&gt;

&lt;p&gt;The application will now return HTTP 503 from &lt;code&gt;/checkout&lt;/code&gt;, while &lt;code&gt;/ready&lt;/code&gt; remains unaffected.&lt;/p&gt;

&lt;h3&gt;
  
  
  Inspect both endpoints
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl -n readiness-lab &lt;span class="nb"&gt;exec&lt;/span&gt; -i \
  deployment/readiness-demo -- python - &lt;span class="o"&gt;&amp;lt;&amp;lt;&lt;/span&gt;&lt;span class="s1"&gt;'PY'&lt;/span&gt;
from urllib.request import urlopen
from urllib.error import HTTPError

&lt;span class="k"&gt;for&lt;/span&gt; path &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"/ready"&lt;/span&gt;, &lt;span class="s2"&gt;"/checkout"&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;:
    try:
        with urlopen&lt;span class="o"&gt;(&lt;/span&gt;
            f&lt;span class="s2"&gt;"http://127.0.0.1:8080{path}"&lt;/span&gt;,
            &lt;span class="nv"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;5
        &lt;span class="o"&gt;)&lt;/span&gt; as response:
            status &lt;span class="o"&gt;=&lt;/span&gt; response.status
    except HTTPError as exc:
        status &lt;span class="o"&gt;=&lt;/span&gt; exc.code

    print&lt;span class="o"&gt;(&lt;/span&gt;f&lt;span class="s2"&gt;"{path}: HTTP {status}"&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
PY
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Actual result observed:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;/ready: HTTP 200
/checkout: HTTP 503
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Now inspect the Deployment:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl -n readiness-lab get deployments

kubectl -n readiness-lab get pods
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The actual output showed:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;NAME             READY   UP-TO-DATE   AVAILABLE
readiness-demo   1/1     1            1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;And the Pod remained:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;READY   STATUS    RESTARTS
1/1     Running   0
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  What happened?
&lt;/h3&gt;

&lt;p&gt;The readiness probe continued succeeding because &lt;code&gt;/ready&lt;/code&gt; was unaffected.&lt;/p&gt;

&lt;p&gt;The liveness probe continued succeeding too.&lt;/p&gt;

&lt;p&gt;From Kubernetes' perspective, the application had not reported a health problem.&lt;/p&gt;

&lt;p&gt;Yet our simulated business operation was failing.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;This is the central lesson of the lab: Pod readiness is only as meaningful as the application-level conditions it represents.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;There is no universal rule that every business operation must be part of readiness.&lt;/p&gt;

&lt;p&gt;In fact, making readiness depend on every external integration can create other problems.&lt;/p&gt;

&lt;p&gt;Imagine every replica failing readiness simultaneously because a shared payment provider experiences a brief outage.&lt;/p&gt;

&lt;p&gt;Removing all Pods from Service routing might turn a partial failure into a broader outage.&lt;/p&gt;

&lt;p&gt;A more robust design could keep the application available for unaffected operations, return a controlled error for the failed integration, and alert the relevant team.&lt;/p&gt;

&lt;p&gt;The correct readiness contract depends on the application's responsibilities.&lt;/p&gt;

&lt;p&gt;We should therefore distinguish between:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Infrastructure availability.&lt;/li&gt;
&lt;li&gt;Application readiness.&lt;/li&gt;
&lt;li&gt;Dependency health.&lt;/li&gt;
&lt;li&gt;Business functionality.&lt;/li&gt;
&lt;li&gt;User-facing reliability.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;These are related signals, not interchangeable ones.&lt;/p&gt;

&lt;h3&gt;
  
  
  Restore functionality
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl -n readiness-lab &lt;span class="nb"&gt;exec&lt;/span&gt; \
  deployment/readiness-demo \
  -- rm /tmp/checkout-down
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  4. Failure mode #2: Readiness failure without container restart
&lt;/h2&gt;

&lt;p&gt;Now let's simulate an application that is running but temporarily unable to receive traffic.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl -n readiness-lab &lt;span class="nb"&gt;exec&lt;/span&gt; \
  deployment/readiness-demo \
  -- touch /tmp/not-ready
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Wait for Kubernetes to detect the condition:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl -n readiness-lab &lt;span class="nb"&gt;wait&lt;/span&gt; \
  --for=condition=Ready=false \
  pod -l &lt;span class="nv"&gt;app&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;readiness-demo \
  --timeout=45s
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Inspect the Pod and Deployment
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl -n readiness-lab get pods

kubectl -n readiness-lab get deployments
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Observed results:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;NAME                              READY   STATUS    RESTARTS
readiness-demo-6b9f5b684b-44hnp   0/1     Running   0
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;And:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;NAME             READY   UP-TO-DATE   AVAILABLE
readiness-demo   0/1     1            0
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The container had not restarted.&lt;/p&gt;

&lt;p&gt;The process was still running.&lt;/p&gt;

&lt;p&gt;But Kubernetes no longer considered the Pod ready to receive normal Service traffic.&lt;/p&gt;

&lt;h3&gt;
  
  
  Inspect the EndpointSlice
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl -n readiness-lab get endpointslices \
  -l kubernetes.io/service-name=readiness-demo \
  -o yaml
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The actual EndpointSlice contained:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;conditions&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;ready&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;
  &lt;span class="na"&gt;serving&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;
  &lt;span class="na"&gt;terminating&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The endpoint still existed in the EndpointSlice, but it was no longer eligible for normal Service routing.&lt;/p&gt;

&lt;p&gt;With only one replica, our Service had no ready backend.&lt;/p&gt;

&lt;p&gt;In a multi-replica workload, other ready replicas could continue serving traffic.&lt;/p&gt;

&lt;p&gt;This distinction is particularly important during rolling updates and temporary application initialization.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Readiness failure changes traffic eligibility. It does not necessarily require restarting the process.&lt;/strong&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  Restore readiness
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl -n readiness-lab &lt;span class="nb"&gt;exec&lt;/span&gt; \
  deployment/readiness-demo \
  -- rm /tmp/not-ready
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Wait for recovery:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl -n readiness-lab &lt;span class="nb"&gt;wait&lt;/span&gt; \
  --for=condition=Ready \
  pod -l &lt;span class="nv"&gt;app&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;readiness-demo \
  --timeout=45s

kubectl -n readiness-lab rollout status \
  deployment/readiness-demo \
  --timeout=60s
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;In the verified experiment, readiness recovered without a container restart.&lt;/p&gt;

&lt;h2&gt;
  
  
  5. Failure mode #3: A rollout blocked by readiness
&lt;/h2&gt;

&lt;p&gt;Now let's simulate releasing a version that starts but never becomes ready.&lt;/p&gt;

&lt;p&gt;Our Deployment has:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;replicas&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
&lt;span class="na"&gt;strategy&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;RollingUpdate&lt;/span&gt;
  &lt;span class="na"&gt;rollingUpdate&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;maxSurge&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
    &lt;span class="na"&gt;maxUnavailable&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This allows Kubernetes to create an additional Pod while preserving the previous available replica.&lt;/p&gt;

&lt;p&gt;It is an important property, but not an unconditional zero-downtime guarantee.&lt;/p&gt;

&lt;h3&gt;
  
  
  Introduce a defective version
&lt;/h3&gt;

&lt;p&gt;Change the environment variable in the Deployment template:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl -n readiness-lab &lt;span class="nb"&gt;set&lt;/span&gt; env \
  deployment/readiness-demo \
  &lt;span class="nv"&gt;READY_ON_START&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;false&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This triggers a new rollout.&lt;/p&gt;

&lt;p&gt;The new Pod starts.&lt;/p&gt;

&lt;p&gt;Its &lt;code&gt;/startup&lt;/code&gt; and &lt;code&gt;/live&lt;/code&gt; endpoints remain healthy.&lt;/p&gt;

&lt;p&gt;However, &lt;code&gt;/ready&lt;/code&gt; returns HTTP 503.&lt;/p&gt;

&lt;h3&gt;
  
  
  Inspect the Pods
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl -n readiness-lab get pods \
  -l &lt;span class="nv"&gt;app&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;readiness-demo
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;In my experiment, the new and previous versions coexisted:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;NAME                              READY   STATUS
readiness-demo-65f8578575-qp4gg   0/1     Running
readiness-demo-6b9f5b684b-44hnp   1/1     Running
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The previous version remained available.&lt;/p&gt;

&lt;p&gt;The new version was running but never became ready.&lt;/p&gt;

&lt;h3&gt;
  
  
  Wait for the failed rollout
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl -n readiness-lab rollout status \
  deployment/readiness-demo \
  --timeout=130s
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Because the new replica remains unavailable, the Deployment controller should eventually exceed its configured 90-second progress deadline.&lt;/p&gt;

&lt;p&gt;The actual result was:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Waiting for deployment "readiness-demo" rollout to finish:
1 old replicas are pending termination...

error: deployment "readiness-demo" exceeded its progress deadline
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Two different timeouts
&lt;/h3&gt;

&lt;p&gt;There is an important distinction here.&lt;/p&gt;

&lt;p&gt;The &lt;code&gt;--timeout=130s&lt;/code&gt; option belongs to the &lt;code&gt;kubectl&lt;/code&gt; client. It limits how long the command waits.&lt;/p&gt;

&lt;p&gt;The Deployment's &lt;code&gt;progressDeadlineSeconds: 90&lt;/code&gt; belongs to the Kubernetes controller. It determines when the controller reports stalled rollout progress.&lt;/p&gt;

&lt;p&gt;A client-side timeout does not automatically mean Kubernetes has declared the Deployment failed.&lt;/p&gt;

&lt;p&gt;That is why we should inspect the controller's actual conditions.&lt;/p&gt;

&lt;h3&gt;
  
  
  Inspect the Deployment state
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl -n readiness-lab describe \
  deployment readiness-demo
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The verified experiment reported:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Replicas: 1 desired | 1 updated | 2 total |
          1 available | 1 unavailable
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;And:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Type           Status  Reason
----           ------  ------
Available      True    MinimumReplicasAvailable
Progressing    False   ProgressDeadlineExceeded
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;This is one of the most interesting observations of the entire lab.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The Deployment was still available.&lt;/p&gt;

&lt;p&gt;But its latest rollout had failed.&lt;/p&gt;

&lt;p&gt;In other words:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;Available=True&lt;/code&gt; does not mean the newest Deployment revision was successfully rolled out.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The previous healthy version was maintaining availability while Kubernetes reported that the update could not progress.&lt;/p&gt;

&lt;h3&gt;
  
  
  Verify the Service endpoints
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl -n readiness-lab get endpointslices \
  -l kubernetes.io/service-name=readiness-demo \
  -o &lt;span class="nv"&gt;jsonpath&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;'{range .items[*].endpoints[*]}{.targetRef.name}{" ready="}{.conditions.ready}{"\n"}{end}'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Observed result:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;readiness-demo-6b9f5b684b-44hnp ready=true
readiness-demo-65f8578575-qp4gg ready=false
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The old version remained eligible for Service traffic.&lt;/p&gt;

&lt;p&gt;The new version did not.&lt;/p&gt;

&lt;h3&gt;
  
  
  Roll back the failed revision
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl -n readiness-lab rollout undo \
  deployment/readiness-demo
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This restores the previous Pod template.&lt;/p&gt;

&lt;p&gt;However, it does not automatically reverse external side effects such as database migrations or changes to shared state.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A Kubernetes rollback is not necessarily an application rollback.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;There is another operational detail worth highlighting.&lt;/p&gt;

&lt;p&gt;Immediately after &lt;code&gt;rollout undo&lt;/code&gt;, my first &lt;code&gt;rollout status&lt;/code&gt; check still reported the previous progress-deadline failure.&lt;/p&gt;

&lt;p&gt;But the previous healthy Pod was already available.&lt;/p&gt;

&lt;p&gt;A subsequent inspection showed that the Deployment controller had completed reconciliation.&lt;/p&gt;

&lt;p&gt;We can inspect the relevant conditions with:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl -n readiness-lab get deployment readiness-demo \
  -o &lt;span class="nv"&gt;jsonpath&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;'{range .status.conditions[*]}{.type}{" = "}{.status}{" | "}{.reason}{"\n"}{end}'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;And verify the desired generation:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl -n readiness-lab get deployment readiness-demo \
  -o &lt;span class="nv"&gt;jsonpath&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;'generation={.metadata.generation} observed={.status.observedGeneration} replicas={.status.replicas} updated={.status.updatedReplicas} available={.status.availableReplicas}{"\n"}'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The recovered Deployment reported:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Available = True | MinimumReplicasAvailable
Progressing = True | NewReplicaSetAvailable
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;And:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;generation=3 observed=3 replicas=1 updated=1 available=1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The defective ReplicaSet had been scaled to zero.&lt;/p&gt;

&lt;p&gt;One additional consideration: &lt;code&gt;kubectl rollout undo&lt;/code&gt; does not update the &lt;code&gt;last-applied-configuration&lt;/code&gt; annotation used by &lt;code&gt;kubectl apply&lt;/code&gt;. Subsequent declarative changes should account for this distinction.&lt;/p&gt;

&lt;p&gt;The lesson is clear:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Initiating a rollback is not the same as verifying successful recovery.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;A release pipeline should confirm the intended revision, controller conditions, available replicas, and application behavior before declaring recovery complete.&lt;/p&gt;

&lt;h2&gt;
  
  
  6. Failure mode #4: Liveness failure and automatic restart
&lt;/h2&gt;

&lt;p&gt;So far, we have intentionally avoided using liveness to detect temporary readiness or business-functionality failures.&lt;/p&gt;

&lt;p&gt;Now let's demonstrate what happens when liveness actually fails.&lt;/p&gt;

&lt;h3&gt;
  
  
  Inject a liveness failure
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl -n readiness-lab &lt;span class="nb"&gt;exec&lt;/span&gt; \
  deployment/readiness-demo \
  -- touch /tmp/not-live
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;/live&lt;/code&gt; endpoint now returns HTTP 503.&lt;/p&gt;

&lt;p&gt;Our Deployment uses:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;livenessProbe&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;httpGet&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;path&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;/live&lt;/span&gt;
    &lt;span class="na"&gt;port&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;http&lt;/span&gt;
  &lt;span class="na"&gt;periodSeconds&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;
  &lt;span class="na"&gt;timeoutSeconds&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;
  &lt;span class="na"&gt;failureThreshold&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;After three consecutive failures, kubelet should restart the container.&lt;/p&gt;

&lt;h3&gt;
  
  
  Observe the restart
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl -n readiness-lab get pods -w
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Stop watching with &lt;code&gt;Ctrl+C&lt;/code&gt; after recovery.&lt;/p&gt;

&lt;p&gt;Then inspect the Pod:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl -n readiness-lab describe pods \
  -l &lt;span class="nv"&gt;app&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;readiness-demo
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The actual result showed that the container restart count increased:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;RESTARTS
1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The kubelet events provided the reason:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Warning  Unhealthy  Liveness probe failed:
HTTP probe failed with statuscode: 503

Normal   Killing    Container web failed
liveness probe, will be restarted
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The previous container had:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Last State: Terminated
Reason: Error
Exit Code: 137
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Exit code 137 alone does not prove an out-of-memory failure.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;In this experiment, the kubelet events explicitly identified the failing liveness probe as the reason for the restart.&lt;/p&gt;

&lt;p&gt;After restarting, the Pod recovered:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;READY   STATUS    RESTARTS
1/1     Running   1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The temporary failure flag disappeared because it was stored in the container's writable filesystem.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why this distinction matters
&lt;/h3&gt;

&lt;p&gt;In production, a failing liveness probe might indicate a deadlock or another condition from which restarting the process is an appropriate recovery action.&lt;/p&gt;

&lt;p&gt;But restarting an application because a remote database is temporarily unavailable can be counterproductive.&lt;/p&gt;

&lt;p&gt;If the shared dependency is failing, indiscriminately restarting every dependent application could increase load, erase useful in-memory state, and slow recovery.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Liveness is a recovery mechanism, not a general-purpose dependency monitor.&lt;/strong&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  7. The missing layer: Post-deployment verification
&lt;/h2&gt;

&lt;p&gt;Our first experiment demonstrated a problem Kubernetes health probes alone did not detect.&lt;/p&gt;

&lt;p&gt;The application was ready, but an important operation failed.&lt;/p&gt;

&lt;p&gt;This is where deployment verification must move beyond infrastructure signals.&lt;/p&gt;

&lt;p&gt;A simple release gate should verify both the Kubernetes state and representative application behavior.&lt;/p&gt;

&lt;h3&gt;
  
  
  Check rollout status
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl -n readiness-lab rollout status \
  deployment/readiness-demo \
  --timeout=180s
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This confirms that the rollout has completed according to the Deployment controller.&lt;/p&gt;

&lt;p&gt;It does not verify business functionality.&lt;/p&gt;

&lt;h3&gt;
  
  
  Verify an application endpoint through the Service
&lt;/h3&gt;

&lt;p&gt;For our lab, we can query the simulated business endpoint using Kubernetes Service DNS.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl -n readiness-lab &lt;span class="nb"&gt;exec&lt;/span&gt; \
  deployment/readiness-demo -- \
  python -c &lt;span class="s1"&gt;'from urllib.request import urlopen; print(urlopen("http://readiness-demo.readiness-lab.svc.cluster.local/checkout", timeout=5).status)'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A successful response prints:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;200
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;These checks answer different questions.&lt;/p&gt;

&lt;p&gt;The first asks whether the Kubernetes rollout completed.&lt;/p&gt;

&lt;p&gt;The second asks whether a particular application operation is functioning.&lt;/p&gt;

&lt;p&gt;For this experiment, the endpoint represents a safe synthetic operation.&lt;/p&gt;

&lt;p&gt;In production, application-level checks should exercise representative functionality without causing unwanted side effects.&lt;/p&gt;

&lt;h3&gt;
  
  
  Additional verification layers
&lt;/h3&gt;

&lt;p&gt;A production-grade verification process might include:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Layer&lt;/th&gt;
&lt;th&gt;Example verification&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Deployment&lt;/td&gt;
&lt;td&gt;Desired replicas updated and available&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Application&lt;/td&gt;
&lt;td&gt;Synthetic transaction or critical API request&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Reliability&lt;/td&gt;
&lt;td&gt;Error rate, latency, availability&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Dependencies&lt;/td&gt;
&lt;td&gt;Connectivity and dependency-specific behavior&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Data&lt;/td&gt;
&lt;td&gt;Schema and version compatibility&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Security&lt;/td&gt;
&lt;td&gt;Expected authentication and authorization behavior&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Rollback&lt;/td&gt;
&lt;td&gt;Known recovery procedure and compatibility checks&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Depending on the service, the process can also include canary releases, progressive traffic shifting, automated analysis of service-level indicators, and human approval for high-risk changes.&lt;/p&gt;

&lt;p&gt;There is no universal release gate.&lt;/p&gt;

&lt;p&gt;But there should be an explicit difference between:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Kubernetes accepted this version.&lt;/li&gt;
&lt;li&gt;The application passed our operational acceptance criteria.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  8. Automating healthy-state verification
&lt;/h2&gt;

&lt;p&gt;The repository includes a verification script:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;scripts/verify.sh&lt;/code&gt;&lt;/p&gt;

&lt;p&gt;It checks:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The expected Kubernetes context.&lt;/li&gt;
&lt;li&gt;Deployment rollout status.&lt;/li&gt;
&lt;li&gt;Deployment and Pod state.&lt;/li&gt;
&lt;li&gt;The &lt;code&gt;/ready&lt;/code&gt; and &lt;code&gt;/checkout&lt;/code&gt; endpoints.&lt;/li&gt;
&lt;li&gt;EndpointSlice readiness.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Execute it from the repository root:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="kd"&gt;export&lt;/span&gt; &lt;span class="nv"&gt;KUBECONFIG&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"$HOME/.kube/readiness-lab-kind.yaml"&lt;/span&gt;

./scripts/verify.sh
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The script was executed successfully in the original lab.&lt;/p&gt;

&lt;p&gt;Its final output included:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;=== Application Service Check ===
/ready: HTTP 200
/checkout: HTTP 200

=== EndpointSlices ===
readiness-demo-6b9f5b684b-44hnp ready=true

Healthy application verification completed.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The script verifies the healthy state.&lt;/p&gt;

&lt;p&gt;It does not intentionally introduce failures.&lt;/p&gt;

&lt;p&gt;The four failure scenarios are documented separately for manual reproduction.&lt;/p&gt;

&lt;h3&gt;
  
  
  Independent verification using GitHub Actions
&lt;/h3&gt;

&lt;p&gt;To make the lab independently reproducible, I also added a GitHub Actions workflow.&lt;/p&gt;

&lt;p&gt;The workflow:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Checks out the repository.&lt;/li&gt;
&lt;li&gt;Creates a fresh Kind cluster.&lt;/li&gt;
&lt;li&gt;Verifies the Kubernetes context.&lt;/li&gt;
&lt;li&gt;Deploys the application.&lt;/li&gt;
&lt;li&gt;Waits for the rollout.&lt;/li&gt;
&lt;li&gt;Executes &lt;code&gt;scripts/verify.sh&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Collects diagnostics if a step fails.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The first GitHub Actions execution completed successfully.&lt;/p&gt;

&lt;p&gt;You can inspect the actual run here:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;a href="https://github.com/rafaterrero/kubernetes-production-readiness-lab/actions/runs/43944585227" rel="noopener noreferrer"&gt;Successful Kubernetes Readiness Lab CI execution&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;This independently verifies the healthy deployment.&lt;/p&gt;

&lt;p&gt;The four failure-injection scenarios were validated manually in the original Kind cluster and are not yet automated in CI.&lt;/p&gt;

&lt;p&gt;That distinction matters: &lt;strong&gt;a green CI pipeline proves only what its tests actually exercise.&lt;/strong&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  9. Production recommendations
&lt;/h2&gt;

&lt;p&gt;Here are the principles I would apply when designing readiness and deployment controls for critical services.&lt;/p&gt;

&lt;h3&gt;
  
  
  Define probe contracts with application developers
&lt;/h3&gt;

&lt;p&gt;A platform team should not guess what application readiness means.&lt;/p&gt;

&lt;p&gt;The application owner should establish which internal conditions prevent the service from handling its intended traffic.&lt;/p&gt;

&lt;h3&gt;
  
  
  Separate temporary unavailability from process failure
&lt;/h3&gt;

&lt;p&gt;Readiness can stop traffic without forcing a restart.&lt;/p&gt;

&lt;p&gt;Liveness should be reserved for failures where restarting is an appropriate recovery mechanism.&lt;/p&gt;

&lt;h3&gt;
  
  
  Use startup probes where startup time is variable
&lt;/h3&gt;

&lt;p&gt;They provide an explicit startup window instead of forcing liveness checks to tolerate lengthy initialization.&lt;/p&gt;

&lt;h3&gt;
  
  
  Treat successful rollouts as necessary but insufficient evidence
&lt;/h3&gt;

&lt;p&gt;Deployment status should be combined with application-specific validation.&lt;/p&gt;

&lt;h3&gt;
  
  
  Design rollback before release
&lt;/h3&gt;

&lt;p&gt;Consider configuration changes, schema compatibility, messages in queues, persistent state, and external dependencies.&lt;/p&gt;

&lt;h3&gt;
  
  
  Validate the deployment strategy against real capacity
&lt;/h3&gt;

&lt;p&gt;&lt;code&gt;maxSurge&lt;/code&gt; requires additional resources.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;maxUnavailable&lt;/code&gt; constrains availability during rolling updates.&lt;/p&gt;

&lt;p&gt;Neither setting alone proves zero downtime.&lt;/p&gt;

&lt;h3&gt;
  
  
  Harden the workload independently of health checks
&lt;/h3&gt;

&lt;p&gt;Our lab uses a convenient Python image and an intentionally minimal manifest.&lt;/p&gt;

&lt;p&gt;A production deployment should additionally consider:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Pinned image versions or digests.&lt;/li&gt;
&lt;li&gt;Resource requests and limits.&lt;/li&gt;
&lt;li&gt;Non-root execution.&lt;/li&gt;
&lt;li&gt;Restricted container permissions.&lt;/li&gt;
&lt;li&gt;Dependency management.&lt;/li&gt;
&lt;li&gt;Network policies.&lt;/li&gt;
&lt;li&gt;Supply-chain controls.&lt;/li&gt;
&lt;li&gt;Suitable logging and observability.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Health probes do not replace these safeguards.&lt;/p&gt;

&lt;h2&gt;
  
  
  10. What this means for Platform Engineering
&lt;/h2&gt;

&lt;p&gt;A Platform Engineering team should aim to make reliable deployment practices reusable.&lt;/p&gt;

&lt;p&gt;Instead of forcing every development team to reinvent probes, release validation, rollback procedures, and observability, an internal platform can establish sensible defaults and clear extension points.&lt;/p&gt;

&lt;p&gt;That might include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Standardized deployment templates.&lt;/li&gt;
&lt;li&gt;Documented probe contracts.&lt;/li&gt;
&lt;li&gt;Reusable post-deployment verification.&lt;/li&gt;
&lt;li&gt;Consistent metrics, logs, and traces.&lt;/li&gt;
&lt;li&gt;Progressive delivery capabilities.&lt;/li&gt;
&lt;li&gt;Automated and manual recovery procedures.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The objective is not to hide every operational decision from developers.&lt;/p&gt;

&lt;p&gt;It is to reduce unnecessary complexity while making the important decisions visible, testable, and repeatable.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A good internal platform should make the safe path easier to follow, not merely provide more YAML to maintain.&lt;/strong&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  11. Verified Lab Results — Kubernetes 1.37.0
&lt;/h2&gt;

&lt;p&gt;All four failure scenarios described in this article were executed on a disposable Kind cluster running Kubernetes 1.37.0, hosted on Ubuntu 24.04 with Docker 29.10.0.&lt;/p&gt;

&lt;p&gt;The results below are actual observations from the lab, not fabricated terminal outputs or production benchmarks.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Experiment&lt;/th&gt;
&lt;th&gt;Verified result&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Business failure&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;/ready&lt;/code&gt; returned HTTP 200 while &lt;code&gt;/checkout&lt;/code&gt; returned HTTP 503. The Deployment remained available.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Readiness failure&lt;/td&gt;
&lt;td&gt;The Pod became &lt;code&gt;0/1 Ready&lt;/code&gt; without restarting. Its EndpointSlice reported &lt;code&gt;ready=false&lt;/code&gt; and &lt;code&gt;serving=false&lt;/code&gt;.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Failed rollout&lt;/td&gt;
&lt;td&gt;The new Pod remained unready while the previous Pod stayed available. Kubernetes reported &lt;code&gt;ProgressDeadlineExceeded&lt;/code&gt;.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Rollback&lt;/td&gt;
&lt;td&gt;The healthy template was restored. The Deployment subsequently reported &lt;code&gt;Progressing=True&lt;/code&gt; with &lt;code&gt;NewReplicaSetAvailable&lt;/code&gt;.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Service recovery&lt;/td&gt;
&lt;td&gt;Both &lt;code&gt;/ready&lt;/code&gt; and &lt;code&gt;/checkout&lt;/code&gt; returned HTTP 200 through Kubernetes Service DNS.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Liveness failure&lt;/td&gt;
&lt;td&gt;Three HTTP 503 probe failures triggered a container restart. The Pod recovered with &lt;code&gt;Ready=True&lt;/code&gt;.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;CI verification&lt;/td&gt;
&lt;td&gt;GitHub Actions successfully deployed and verified the healthy application in a separate Kind cluster.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h3&gt;
  
  
  A particularly important observation
&lt;/h3&gt;

&lt;p&gt;During the failed-rollout experiment, the Deployment simultaneously reported:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Available = True
Progressing = False
Reason = ProgressDeadlineExceeded
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The previous Pod was still eligible for Service traffic.&lt;/p&gt;

&lt;p&gt;The newest Pod was not.&lt;/p&gt;

&lt;p&gt;This demonstrates that &lt;strong&gt;application availability and deployment progression are separate operational signals&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;During recovery, another distinction became visible.&lt;/p&gt;

&lt;p&gt;Immediately after initiating the rollback, the rollout status command still reported the earlier failure.&lt;/p&gt;

&lt;p&gt;Only after the controller reconciled the restored revision did the Deployment report:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Available = True | MinimumReplicasAvailable
Progressing = True | NewReplicaSetAvailable
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This reinforces why rollback initiation and verified recovery must be treated as separate steps.&lt;/p&gt;

&lt;h2&gt;
  
  
  12. Reproduce the experiments
&lt;/h2&gt;

&lt;p&gt;The complete source code, Kubernetes manifests, experiment instructions, and verified results are publicly available:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;a href="https://github.com/rafaterrero/kubernetes-production-readiness-lab" rel="noopener noreferrer"&gt;Kubernetes Production Readiness Lab — GitHub&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The repository includes:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;File or directory&lt;/th&gt;
&lt;th&gt;Purpose&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;manifests/readiness-lab.yaml&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Kubernetes application and probes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;scripts/verify.sh&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Healthy-state validation&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;docs/architecture.md&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Architecture and deployment strategy&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;docs/experiments.md&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Instructions for all four experiments&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;docs/verified-results.md&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Results observed in the original lab&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;evidence/final-recovery.txt&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Captured final recovery state&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;.github/workflows/kind-smoke.yml&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;GitHub Actions verification workflow&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The project is open source under the MIT License.&lt;/p&gt;

&lt;h3&gt;
  
  
  Cleanup
&lt;/h3&gt;

&lt;p&gt;When you finish the experiments, remove the lab namespace:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl delete namespace readiness-lab
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This removes the application resources from the test cluster.&lt;/p&gt;

&lt;p&gt;You can also delete the entire disposable Kind cluster:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kind delete cluster --name readiness-lab
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The original testing environment was cleaned up after validation. The source code and documented evidence remain available in GitHub.&lt;/p&gt;

&lt;h2&gt;
  
  
  Final thoughts
&lt;/h2&gt;

&lt;p&gt;Kubernetes is very good at managing desired state.&lt;/p&gt;

&lt;p&gt;It can schedule Pods, restart unhealthy containers, update replicas, and manage traffic eligibility based on readiness.&lt;/p&gt;

&lt;p&gt;But its view of application health depends on the signals and contracts we provide.&lt;/p&gt;

&lt;p&gt;Our experiments demonstrated that:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A Deployment can be available while a critical business operation fails.&lt;/li&gt;
&lt;li&gt;A container can be alive without being ready.&lt;/li&gt;
&lt;li&gt;A Pod can be ready while an external integration is broken.&lt;/li&gt;
&lt;li&gt;A Deployment can remain available while its latest rollout has failed.&lt;/li&gt;
&lt;li&gt;A rollback can be initiated before recovery has been fully verified.&lt;/li&gt;
&lt;li&gt;A liveness failure can trigger a restart without requiring the Pod to be recreated.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;These are not reasons to distrust Kubernetes.&lt;/p&gt;

&lt;p&gt;They are reasons to understand precisely what Kubernetes does and does not guarantee.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Production readiness is not a status field. It is an engineering contract supported by verification, observability, and recovery.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;That is the distinction worth designing for.&lt;/p&gt;




&lt;h3&gt;
  
  
  References
&lt;/h3&gt;

&lt;p&gt;The behavior discussed in this article is based on the official Kubernetes documentation:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;a href="https://kubernetes.io/docs/concepts/workloads/pods/probes/" rel="noopener noreferrer"&gt;Kubernetes — Liveness, Readiness, and Startup Probes&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/" rel="noopener noreferrer"&gt;Kubernetes — Configure Liveness, Readiness and Startup Probes&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://kubernetes.io/docs/concepts/workloads/controllers/deployment/" rel="noopener noreferrer"&gt;Kubernetes — Deployments&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://kubernetes.io/docs/concepts/services-networking/endpoint-slices/" rel="noopener noreferrer"&gt;Kubernetes — EndpointSlices&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://kubernetes.io/docs/reference/kubectl/generated/kubectl_rollout/kubectl_rollout_status/" rel="noopener noreferrer"&gt;Kubernetes — kubectl rollout status&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://kubernetes.io/docs/reference/kubectl/generated/kubectl_rollout/kubectl_rollout_undo/" rel="noopener noreferrer"&gt;Kubernetes — kubectl rollout undo&lt;/a&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;strong&gt;Source code and reproducible experiments:&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://github.com/rafaterrero/kubernetes-production-readiness-lab" rel="noopener noreferrer"&gt;Kubernetes Production Readiness Lab — GitHub&lt;/a&gt;&lt;/p&gt;




&lt;p&gt;&lt;em&gt;I'm Rafa Terrero, a Senior Platform Engineer working across cloud infrastructure, DevOps, Kubernetes, and production reliability. I write about architecture decisions, engineering trade-offs, and the challenges of building dependable platforms, including infrastructure for AI-powered systems.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>kubernetes</category>
      <category>devops</category>
      <category>platformengineering</category>
      <category>sre</category>
    </item>
  </channel>
</rss>
