14:32 UTC. PagerDuty fires on the checkout service's error-rate alarm. Datadog shows the 502 rate on
POST /checkoutjumping from a baseline of 0.02% to 4.8%, right in the middle of a rolling deploy that had shipped a dozen times before without anyone noticing it. By 14:36 the deploy finishes, the 502s stop, and the dashboard looks normal again. Nobody asked for this one to rotate twelve pods for a routine dependency bump, and yet here it was, the third deploy that week to show the exact same spike, each time clean before and after, dirty for about four minutes in the middle.
the scramble
First theory in #incidents: the new image is bad. That's the obvious read when a deploy and an error spike land on top of each other. Someone pulls up the new revision's pod logs looking for stack traces, crash loops, anything. Nothing. Zero restarts across all twelve pods. Zero 5xx logged by the app itself. Whatever's producing these 502s, it isn't the checkout service throwing an error, because the checkout service never saw the request at all.
Second theory: a health check regression is marking targets unhealthy mid-rollout and the ALB is handling the transition badly. Someone checks the target group in the AWS console. Zero unhealthy targets logged for the entire deploy window. By every signal AWS itself can show, nothing was ever wrong with any target. That's the part that stalls the room for a few minutes, because if the load balancer never thought a target was unhealthy, where are the 502s even coming from?
the hunt
With both easy theories dead, someone pulls the ALB's own access logs from S3 and greps for the deploy window instead of trusting the target-group health view. ALB access logs record a target_status_code of - for exactly one case: it picked a target, tried to open the connection, and got nothing back, connection refused or reset. That's a different failure mode than "target failed its health check," and it doesn't show up on the target group's health dashboard at all.
alb access logs, filtered to the deploy window
aws s3 cp s3://alb-logs/checkout/2026/10/20/ . --recursive
zcat *.log.gz | awk '$8 == "-" && $6 ~ /checkout/' | cut -d' ' -f4,8,14
14:33:02.418481Z - 10.4.21.117:3000
14:33:04.901223Z - 10.4.21.117:3000
14:33:05.550112Z - 10.4.21.117:3000
14:34:11.009887Z - 10.4.21.94:3000
14:34:12.733410Z - 10.4.21.94:3000
Every failed request maps to a target IP, and those IPs belong to pods. Cross-reference that against kubectl get events history for the same window: each burst of failures lines up within a handful of seconds of that exact pod's termination. 10.4.21.117 got its SIGTERM at 14:33:00. The last 502 mapped to it is at 14:33:05.55, five seconds after the process was told to shut down.
That's the actual question now: why was the ALB still sending live traffic to a pod five seconds after Kubernetes told it to die? Checking the AWS Load Balancer Controller's own logs for the same window answers it. The controller doesn't poll for pod state, it watches EndpointSlice updates and reacts by calling DeregisterTargets on the target group. The controller's reconcile loop timestamps show a four-to-seven-second gap between the pod going NotReady and the target group actually being told to drop that IP. The checkout app had no SIGTERM handler, so Node's default behavior kicked in: no listener means the process exits almost immediately, well under a second, as soon as the signal arrives.
The pod was dying in under a second. The signal that it was dying took four to seven seconds to reach the one system still routing traffic to it. The 300-second deregistration_delay.timeout_seconds on the target group, set months earlier to drain long-lived connections gracefully once a target is actually marked for removal, was never the problem and never fired, because the controller hadn't called DeregisterTargets yet when the pod went away. That setting governs what happens after deregistration starts. It has nothing to say about the gap before it starts.
the find
Root cause: the checkout app shut down faster than Kubernetes' own control plane could tell the load balancer it was going away. A bare SIGTERM with no handler killed the process in under a second. In this cluster it took four to seven seconds for that pod's IP to drop from the EndpointSlice, for the AWS Load Balancer Controller to notice, and for DeregisterTargets to actually fire on the ALB. Every request the ALB routed to that pod during that gap hit a socket nobody was listening on anymore.
the fix
Two changes, working together. First, the app needed to survive SIGTERM long enough to stop accepting new connections on its own terms instead of just dying:
checkout service, graceful shutdown handler
const server = app.listen(3000);
process.on('SIGTERM', () => {
console.log('SIGTERM received, draining connections');
server.close(() => {
console.log('server closed, no in-flight requests remain');
process.exit(0);
});
// belt and suspenders: force exit if close() hangs on a stuck connection
setTimeout(() => process.exit(1), 25_000).unref();
});
Second, and this is the half that actually fixes the race: the pod needed to delay sending SIGTERM in the first place, long enough for the EndpointSlice update and the ALB controller's deregister call to actually land before the process started shutting down.
deployment.yaml, checkout container
spec:
terminationGracePeriodSeconds: 45
containers:
- name: checkout
lifecycle:
preStop:
exec:
command: ["sh", "-c", "sleep 10"]
readinessProbe:
httpGet:
path: /healthz
port: 3000
periodSeconds: 5
failureThreshold: 1
The preStop hook runs before SIGTERM is ever sent to the container, and the pod is marked NotReady the moment termination begins, well before the hook finishes sleeping. Ten seconds was comfortably above the four-to-seven-second gap measured in the controller logs. terminationGracePeriodSeconds went up to 45 so the sleep plus the app's own graceful shutdown both have room to finish before Kubernetes gives up and sends SIGKILL.
the aftermath
4.8% Peak 502 rate on checkout during the deploy, against a 0.02% baseline
4-7s Measured gap between a pod going NotReady and the ALB controller deregistering its target
1,240 Failed checkout POSTs across the 14-minute rolling deploy before the fix
0 Failed requests observed across the next 30 deploys after the preStop hook shipped
- A target group's
deregistration_delayonly governs draining after a target has been told it's leaving. It does nothing about the time it takes the control plane to decide and propagate that a target is leaving in the first place, and that gap is where this bug lived. - A process with no
SIGTERMhandler doesn't die gracefully, it just dies fast. During a rolling deploy that's the wrong kind of fast, fast enough to beat the signal telling everyone else it's on the way out. - ALB access logs record connection-level failures that never touch the target group's health dashboard. A target can pass every health check the dashboard tracks and still spend a few seconds quietly refusing live connections underneath it.
- A synthetic check now hits
/checkoutevery two seconds during every deploy and pages if a single 502 shows up, specifically so this class of bug can't hide behind "the dashboards looked fine" again.
Nothing about the checkout service's code was ever broken, and nothing about the ALB's configuration was wrong on its own terms. The bug lived entirely in the few seconds between two correct systems finding out about each other.
Top comments (1)
The target_status_code of - detail is the useful one: the target group health view said nothing was wrong while the access log showed refused connections, so the symptom hid in a source nobody opens first. For the fix, a preStop sleep that outlasts the controller's deregistration lag, plus a grace period longer than the sleep and the drain, is the usual shape. A cheap regression guard is to run a steady request loop through the ALB during a test rollout and fail the check if any request returns 502, so the next dependency bump catches it before production does.