Why Kubernetes Rolling Updates Still Give You 503s

"We use rolling updates, so deploys are zero-downtime." Then you deploy, and for 2–3 seconds users get 503s. This exact complaint shows up constantly on Stack Overflow and Reddit, and it has a precise, fixable cause — actually four of them, stacked.

The core misunderstanding: rolling updates guarantee pod replacement order, not traffic correctness. Kubernetes replaces pods gracefully; whether requests survive depends on details it leaves to you.

Cause 1: traffic arrives before the app is ready

A pod becomes a load-balancing target when its readiness probe passes. If you have no readiness probe, that's the moment the container starts — before your app has loaded config, connected to the database, or warmed anything. First requests: connection refused.

Fix: a readiness probe that checks what "ready to serve" actually means for your app:

readinessProbe:
  httpGet: { path: /ready, port: 8080 }
  periodSeconds: 2
  failureThreshold: 2

And /ready should verify dependencies (DB connection up), not just return 200 unconditionally.

If startup is slow and variable (JVM warmup, a large cache preload), pair this with a startupProbe. Without one, a slow-but-healthy app can hit the readiness probe's failureThreshold and get killed mid-boot — the opposite symptom, same root cause: nothing told Kubernetes "still starting" is different from "broken."

Cause 2: traffic keeps arriving after the pod is told to die

This is the one that causes the classic "2–3 seconds of 503s," and it's the least intuitive. When a pod terminates, two things happen in parallel, not in sequence:

1. The pod gets SIGTERM.
2. Endpoint controllers start removing the pod from load balancers.

Step 2 takes time to propagate — kube-proxy rules on every node, cloud LB target deregistration. For seconds after SIGTERM, traffic is still routed to a pod that may already be shutting down. If your app exits immediately on SIGTERM, every one of those requests fails.

Fix — the famous sleep:

lifecycle:
  preStop:
    exec:
      command: ["sleep", "8"]

The preStop hook runs before SIGTERM is sent. Sleeping a few seconds keeps the app fully serving while endpoints propagate. It looks like a hack; it's the standard, load-bearing pattern (Kubernetes docs and every ingress vendor recommend some form of it).

How long to sleep depends on the slowest link in your specific path — in-cluster kube-proxy rules update quickly, but a cloud load balancer sitting in front can be far slower (AWS ALB's deregistration_delay defaults to 300 seconds). Check what's actually in your traffic path before picking a number.

Cause 3: the app doesn't drain on SIGTERM

After preStop, SIGTERM arrives. The app must: stop accepting new connections, finish in-flight requests, then exit. Many frameworks do this only if you ask — and some servers keep-alive connections open forever unless told to close them.

Fix: enable your framework's graceful shutdown (Spring: server.shutdown=graceful; Go: http.Server.Shutdown(); Node: close the server and track sockets), and make sure terminationGracePeriodSeconds (default 30) exceeds preStop + your longest request. If it doesn't, SIGKILL wins and kills in-flight work.

Keep-alive HTTP/2 and gRPC clients add one more wrinkle: a client holding a long-lived connection open won't notice the pod is going away until that connection actually breaks. The server needs to signal it explicitly — gRPC servers should send GOAWAY, not just stop accepting new streams — or the client keeps sending requests into a pod that's already draining.

Cause 4: the rollout replaces capacity too aggressively

Defaults (maxUnavailable: 25%) can briefly leave you under-provisioned under load: old pods dying while new pods are technically Ready but cold (empty caches, JIT not warmed, connection pools empty). Latency spikes read as downtime.

Fix:

strategy:
  rollingUpdate: { maxUnavailable: 0, maxSurge: 1 }

New capacity comes up before old capacity goes away. Slower rollout, no dip. Add a PodDisruptionBudget so node drains during upgrades obey the same rule.

The complete checklist

For every user-facing service:

• [ ] Readiness probe that checks real dependencies
• [ ] preStop sleep (5–10s)
• [ ] Graceful shutdown on SIGTERM, verified (kill a pod, watch error rates)
• [ ] terminationGracePeriodSeconds > preStop + longest request
• [ ] maxUnavailable: 0, maxSurge: 1 for critical services
• [ ] PodDisruptionBudget
• [ ] Actually test it: kubectl rollout restart under synthetic load and watch the error rate

That last item is the real gap. Teams configure all of this and never verify it. A deploy under load in staging with a load generator running takes fifteen minutes and tells you the truth.

Why this matters beyond deploys

The same mechanics run during node upgrades, autoscaler scale-downs, and spot instance reclaims. If deploys cause 503s, so does every cluster maintenance event — your "deployment problem" is actually an "any pod ever moves" problem. Fix it once and cluster upgrades, autoscaling, and spot instances all become safe as a side effect. It's one of the highest ROI afternoons available in Kubernetes.

Deployment safety is one of seven categories in the free production readiness scorecard we built.

Comments

[Raknaos]

Cause 4 is the one people dismiss as "a problem for companies bigger than mine" until the first flash crowd.

I run a single-region service on a modest VPS and still eat this: a deploy that's technically correct can amplify traffic to the survivors the moment pods start cycling. The rolling update makes it look safe because each individual step is safe. The system-level failure only shows up in the interaction between steps and traffic.

I also learned the hard way that the readiness probe should measure the thing you actually promise. /ready returning 200 when the DB is down isn't a readiness probe, it's a liveness probe wearing a better outfit — and during a DB failover it actively harms you by routing traffic to pods that will 500 anyway.

Good writeup of the stacked causes. The framing that rolling updates guarantee pod replacement order but not traffic correctness should be the headline of every deploy runbook.