DEV Community

iapilgrim
iapilgrim

Posted on

GCP The Hard Way — Part 5: Canary Deployments Against a Delayed-Failure Bug

#gcp #kubernetes #gke #cicd

Introduction

A deployment that reports success and a release that doesn't harm
users are not the same thing. This distinction matters most when a bug
doesn't manifest immediately — memory leaks, connection pool
exhaustion, and similar failure modes often pass initial health checks
before degrading. This post deploys a workload to Google Kubernetes
Engine with correctly configured liveness and readiness probes,
introduces a version with a 30-second delayed failure, and walks
through rollback and canary rollout as complementary mitigation
strategies.

Solution overview 

┌─────────────┐      ┌──────────────────────┐
│   Ingress /   │────▶│   Service: myapp       │
│ Load Balancer │      └──────────┬───────────┘
└─────────────┘                  │
                    ┌─────────────┴─────────────┐
                    ▼                             ▼
          ┌───────────────────┐        ┌───────────────────┐
          │ Deployment: stable  │        │ Deployment: canary  │
          │  (9 replicas, v1)   │        │  (1 replica, v2)     │
          └───────────────────┘        └───────────────────┘
Enter fullscreen mode Exit fullscreen mode

Prerequisites

  • A GKE Standard cluster with kubectl configured
  • Artifact Registry for container image storage

Walkthrough

Step 1: Provision the cluster and registry 

gcloud services enable container.googleapis.com artifactregistry.googleapis.com

gcloud artifacts repositories create app-repo \
  --repository-format=docker --location=asia-southeast1

gcloud container clusters create app-cluster \
  --zone=asia-southeast1-a --num-nodes=2 --machine-type=e2-medium

gcloud container clusters get-credentials app-cluster --zone=asia-southeast1-a
Enter fullscreen mode Exit fullscreen mode

Step 2: Build an application with a controllable failure mode 

from flask import Flask
import time, os

app = Flask(__name__)
START_TIME = time.time()
BUG_MODE = os.environ.get("BUG_MODE", "false") == "true"

@app.route("/health")
def health():
    if BUG_MODE and (time.time() - START_TIME) > 30:
        return "UNHEALTHY", 500
    return "OK", 200
Enter fullscreen mode Exit fullscreen mode

This mirrors real-world delayed failures more closely than a bug that
fails immediately at startup, which would be caught before any traffic
is served.

Step 3: Deploy with production-grade probes 

readinessProbe:
  httpGet: {path: /health, port: 8080}
  periodSeconds: 5
  failureThreshold: 2
livenessProbe:
  httpGet: {path: /health, port: 8080}
  periodSeconds: 10
  failureThreshold: 3
Enter fullscreen mode Exit fullscreen mode 
kubectl apply -f deployment.yaml
kubectl get pods -w
Enter fullscreen mode Exit fullscreen mode

Step 4: Roll out the faulty version and observe 

kubectl apply -f deployment-v2-bugged.yaml
kubectl get pods -w
Enter fullscreen mode Exit fullscreen mode

For the first ~30 seconds, pods report Running and Ready — this is
expected, and is precisely the window in which a naive smoke test
would report success.

Diagnosing probe behavior: After 30 seconds, inspect events:

kubectl describe pod <pod-name>
Enter fullscreen mode Exit fullscreen mode

readinessProbe failures remove a pod from the Service's Endpoints
list without restarting the container; livenessProbe failures cause
the kubelet to kill and restart the container, visible as an
increasing RESTARTS count in kubectl get pods. Because this
example uses the same path for both probes, expect a repeating
restart cycle consistent with CrashLoopBackOff.

Step 5: Roll back 

kubectl rollout undo deployment/myapp
kubectl rollout status deployment/myapp
Enter fullscreen mode Exit fullscreen mode

Measure the elapsed time from issuing the rollback to full traffic
stability — this is your empirical Mean Time to Recovery (MTTR) for
this failure class.

Step 6: Implement canary rollout for the next release

Rather than replacing the entire fleet at once, split traffic across
two Deployments sharing a Service selector:

kubectl apply -f deployment-stable.yaml   # 9 replicas, v1
kubectl apply -f deployment-canary.yaml   # 1 replica, v2 (bug fixed)
Enter fullscreen mode Exit fullscreen mode 
for i in $(seq 1 50); do
  curl -s http://<EXTERNAL_IP> | grep -o "v[12]-[a-z]*"
done | sort | uniq -c
Enter fullscreen mode Exit fullscreen mode

Confirm the observed traffic split approximates the 9:1 replica ratio
before progressively shifting more replicas to the canary track.

Clean up resources 

kubectl delete deployment myapp-stable myapp-canary --ignore-not-found
kubectl delete service myapp-service --ignore-not-found
gcloud container clusters delete app-cluster --zone=asia-southeast1-a --quiet
Enter fullscreen mode Exit fullscreen mode

Conclusion

Readiness and liveness probes solve different problems and produce
different Kubernetes behaviors — conflating the two is a common source
of confusion when debugging rollout issues. Combined with a canary
rollout strategy, this pattern limits the blast radius of exactly the
kind of delayed-failure bug demonstrated here.

In Part 6, we validate disaster recovery assumptions by
deliberately deleting a region's infrastructure and measuring actual
recovery time against a documented plan.

Top comments (0)