TL;DR: Graceful pod shutdown on EKS needs a preStop delay plus tuned ALB deregistration, otherwise the load balancer keeps routing to terminating pods and returns 502s mid-rollout. Here's the fix we shipped and how we tested it.
During a routine deploy of our build orchestration API on EKS last month, the ALB logged roughly 1,400 HTTP 502s across the 90 seconds it took to roll three replicas. Nobody got paged, but our availability SLO dashboard showed a clear dip on every deploy, and we deploy a fair few times a day. We run EKS 1.29 with the AWS Load Balancer Controller in IP target mode, and the root cause was a race in graceful pod shutdown that nobody had tuned for. No worries if you've hit this; it's common and the fix is small once you see it.
What graceful pod shutdown actually means on EKS
Graceful pod shutdown is the sequence Kubernetes runs when terminating a pod: it removes the pod from Service endpoints, executes any preStop hook, sends SIGTERM to the container, then SIGKILL once the grace period expires. The pod termination lifecycle is well documented, but the catch on EKS is that endpoint removal inside the cluster and target deregistration at the ALB happen on different clocks.
When the controller runs in IP mode, the ALB sends traffic straight to pod IPs, not through kube-proxy. So the in-cluster endpoint can vanish while the ALB still has that pod IP registered as a healthy target. For a few seconds, the load balancer routes requests at a pod that's already mid-shutdown, and you get 502s.
Why the load balancer keeps sending traffic to terminating pods
The timing gap is the whole problem. The moment a pod enters Terminating, kubelet starts the shutdown clock and the container can begin refusing connections. Meanwhile the ALB only stops sending new requests after it processes the deregistration, and connection draining respects the deregistration delay, which defaults to 300 seconds.
So you have two failure shapes. Either the app stops accepting connections before the ALB has stopped sending them, which is the 502 we saw, or you set the grace period too short and SIGKILL lands while requests are still in flight. The trick is keeping the pod alive and serving until the ALB has actually deregistered it. That's what the preStop hook buys you.
The preStop hook and deregistration delay fix
We added a preStop sleep so the container stays up and keeps serving in-flight requests while deregistration propagates, and we set a grace period longer than that sleep. The values came from watching how long deregistration took during a rollout, which for us was about 10 to 12 seconds.
spec:
terminationGracePeriodSeconds: 40
containers:
- name: orchestrator
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "sleep 20"]
The sleep does nothing clever. It just delays SIGTERM so the pod keeps answering during the window where the ALB might still route to it. We also enabled pod readiness gates via the AWS Load Balancer Controller, so a new pod isn't marked Ready until the ALB reports its target as healthy. That closes the matching race on the scale-up side, where a fresh pod takes traffic before it's actually registered.
A cleaner long-term option is handling SIGTERM in the application itself: stop accepting new connections, finish in-flight ones, then exit. We're moving that way, but the preStop sleep got the 502s to zero on the same afternoon, so it shipped first.
Testing it with a game day
I don't trust a fix I haven't watched fail under load, so we ran a small game day. We pointed a constant load generator at the service and triggered a rolling deploy while watching the status codes. A simple loop with hey does the job:
hey -z 120s -c 50 https://orchestrator.internal/healthz
kubectl rollout restart deployment/orchestrator
Before the change, every rollout produced a burst of 502s in the hey summary that lined up exactly with the Terminating events in kubectl get pods -w. After adding the preStop hook and readiness gates, three consecutive rollouts under the same 50-concurrent load showed zero 502s. That's the only evidence I'll accept that graceful pod shutdown is genuinely working, because "never had a bad deploy" usually just means nobody measured one.
Trade-offs and limitations
A preStop sleep is a blunt instrument. It adds fixed latency to every pod termination, so a 20 second sleep across a large deployment slows your rollouts and your scale-down. If you're cost-sensitive on spot capacity, that delay is real money.
It also masks the better fix. Sleeping the container is a workaround for an app that doesn't drain connections on SIGTERM, and if your app holds long-lived connections, a flat sleep won't cover them. Readiness gates help, but they add a dependency on the AWS Load Balancer Controller being healthy, and they make pod startup slower because Ready now waits on ALB registration. Pick the grace period from observed deregistration time, not a number you reckon looks safe.
Where to go next
If your EKS deploys throw 502s, instrument the rollout first: run constant load, watch status codes against pod Terminating events, and measure your actual deregistration time before picking a preStop value. Then decide whether a sleep is enough or whether you want proper SIGTERM draining in the app. The next thing I'd try is replacing the static sleep with application-level connection draining, so the pod exits as soon as in-flight requests finish instead of always waiting the full window.
Top comments (0)