DEV Community

Metronom
Metronom

Posted on

Zero-Downtime Kubernetes Deployment: Killing Deploy 5xx With k3d Before Prod

Zero-Downtime Kubernetes Deployment: Killing Deploy 5xx With k3d Before Prod

Every deploy of our payments API leaked 5xx. A dozen requests, every time, right at the rollout. They retried and succeeded, so nobody paged. We process money. "A dozen dropped requests, they retried" is not a sentence I say to a customer.

This is a zero-downtime kubernetes deployment story. It has no service mesh in it. The fix was four boring things — readiness probes, real SIGTERM handling, a preStop sleep, and maxUnavailable: 0 — and I proved every one of them on a laptop k3d cluster before prod ever saw the change.

Symptom

billing-api. Python, FastAPI, Postgres. Two replicas. Green on every dashboard: pods Running, CPU flat, error rate a rounding error.

Then the synthetic checks:

2025-... GET /charge 200
2025-... GET /charge 503   <-- rollout starts
2025-... GET /charge 502
2025-... GET /charge 503
2025-... GET /charge 200   <-- rollout ends
Enter fullscreen mode Exit fullscreen mode

Every deploy. A burst of 502/503 that lined up exactly with the new ReplicaSet coming up. For a year I filed it under "cost of rolling updates." Wrong on every count.

Root cause

I read our Deployment against a production-readiness checklist. It was embarrassing.

spec:
  replicas: 2
  template:
    spec:
      containers:
        - name: billing-api
          image: ghcr.io/acme/billing-api:sha-9f2c1a
          # no probes
          # no resources
          # no graceful shutdown
Enter fullscreen mode Exit fullscreen mode

Four independent bugs, each producing 5xx on its own:

  1. No readiness probe. Kubernetes routed traffic to pods before FastAPI had opened its DB pool.
  2. The container launched via sh -c "uvicorn ...". PID 1 was the shell. The app never received SIGTERM.
  3. Endpoint removal and SIGTERM happen in parallel on pod deletion. Traffic hit pods that had already started shutting down.
  4. Default RollingUpdate allowed 25% unavailable — a real capacity dip on two replicas.

There is a solid, end-to-end breakdown of why each of these matters at this writeup on making a local cluster production-like; I won't re-derive the theory here. I'll show the fixes and how I tested them in k3d — CNCF's k3s in Docker — before shipping.

The fix

1. Readiness is the gate

Split health into two endpoints. Liveness answers "is the process alive." Readiness answers "can I serve a request right now."

@app.get("/healthz")
def healthz():
    # liveness: process is up. Nothing else.
    return {"status": "ok"}

@app.get("/ready")
def ready():
    # readiness: DB pool reachable? If not, 503.
    # Pod leaves endpoints. It is NOT restarted.
    ...
    return {"status": "ready"}
Enter fullscreen mode Exit fullscreen mode 
livenessProbe:
  httpGet: { path: /healthz, port: 8080 }
  initialDelaySeconds: 15
  periodSeconds: 10
  failureThreshold: 3
readinessProbe:
  httpGet: { path: /ready, port: 8080 }
  periodSeconds: 5
Enter fullscreen mode Exit fullscreen mode

The model: liveness restarts, readiness reroutes. A readiness failure pulls the pod out of the Service endpoints without killing it — the Kubernetes probe docs say so directly. Do not put a DB check in liveness. Postgres flickers, liveness fails, Kubernetes restarts a healthy app, and now you have a restart storm on top of a DB blip. Dependency checks go in readiness. Always.

Verified on k3d in about ten seconds:

kubectl get endpointslices -n billing -w
Enter fullscreen mode Exit fullscreen mode

Kill the DB connection. Watch the pod drop out of endpoints. Restore it. Watch it come back. No restart. That one experiment taught me more than a year of prod Grafana.

2. SIGTERM, PID 1, and the shell that ate the signal

CMD sh -c "uvicorn app.main:app ..." makes sh PID 1. sh does not forward SIGTERM. The app ran until the 30s grace period elapsed, then got SIGKILL'd mid-request.

One line. Exec form. uvicorn becomes PID 1.

CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8080"]
Enter fullscreen mode Exit fullscreen mode

Modern uvicorn drains cleanly on SIGTERM: stop accepting, finish in-flight, close the pool, exit — but only if it receives the signal. The Pod lifecycle docs spell out the sequence: kubelet runs preStop, sends SIGTERM to PID 1, waits terminationGracePeriodSeconds, then SIGKILLs. A shell at PID 1 eats step two.

Check what your PID 1 actually is:

kubectl exec -it deploy/billing-api -n billing -- ps -o pid,comm
#   PID COMMAND
#     1 uvicorn      <- good
#     1 sh           <- your signal is going nowhere
Enter fullscreen mode Exit fullscreen mode

3. The endpoints race, and the preStop sleep

Here is the one nobody tells you. On pod deletion, endpoint removal and SIGTERM fire in parallel. For a short window kube-proxy and the ingress still route to a pod that has already begun shutdown. Those requests die. That was the rest of our 5xx. The CNCF pod-termination walkthrough spells out the race and why a short preStop sleep is the standard fix.

spec:
  terminationGracePeriodSeconds: 30
  containers:
    - name: billing-api
      lifecycle:
        preStop:
          exec:
            command: ["sh", "-c", "sleep 15"]
Enter fullscreen mode Exit fullscreen mode

The preStop sleep buys time for routing to converge before the app tears down.

Gotcha that nearly bit me: preStop counts against the grace period. sleep 30 with a 30s grace budget leaves zero seconds to drain — SIGKILL again. Keep the sleep well under budget. sleep 15, grace 30.

4. maxUnavailable: 0 

spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  minReadySeconds: 5
Enter fullscreen mode Exit fullscreen mode

maxUnavailable: 0 means the ready-pod count never dips during a rollout — the rolling-update guide documents the surge-then-retire dance. maxSurge: 1 brings up a new pod, waits for its readiness, retires an old one, loops. minReadySeconds guards against pods that report ready and immediately crash. None of it works without the readiness probe from fix 1 — the rollout waits on it.

Verifying zero downtime locally

Runbook I ran on k3d before touching prod:

  1. Spin the cluster: k3d cluster create billing --agents 2.
  2. Apply the manifests, wait for kubectl rollout status deployment/billing-api -n billing.
  3. Hammer it in one terminal: 
while true; do curl -s -o /dev/null -w "%{http_code}\n" http://billing.localhost:8081/; done
Enter fullscreen mode Exit fullscreen mode
  1. Trigger a rollout in another: kubectl set image deploy/billing-api billing-api=...:sha-new -n billing.
  2. Watch the curl loop. Any non-200 is a bug still in the manifest.

The first few passes still threw 5xx — which told me my readiness-plus-preStop combo wasn't done. I iterated on the laptop, for free, on a Tuesday, instead of debugging it in prod on a Friday night. When it held zero errors through a dozen rollouts locally, I shipped. Prod matched: zero, sustained.

Guardrail

  • Memory limit too low = death, not slowdown. CPU is compressible (throttled). Memory is not — OOMKilled, exit code 137. Set the request near P99 + ~20% headroom.
  • limit without request sets request = limit silently. Inflates scheduler demand until pods sit Pending.
  • replicas: 1 is downtime by definition. Any restart, drain, or rollout is an outage.
  • Slow starters want a startup probe, not a giant initialDelaySeconds. The startup probe holds off liveness/readiness until boot completes, so liveness can stay strict.
  • The production-readiness checklist is now a required review item on every Deployment PR. No probes, no shutdown story, no merge.

What I'd do differently

Fold all of this into the Helm chart as defaults so no service can ship without probes and a drain story. And stop treating "it retries" as "it works." A retried 5xx is still a 5xx that reached a customer.

If your deploys leak errors and the team shrugs it off as normal — it isn't. Grab a local cluster and break your own rollout until it stops breaking.

Bottom line: zero-downtime rollouts aren't a mesh feature, they're four manifest details, and a laptop k3d cluster catches all four before prod does.

Sources

Top comments (0)