If your error tracker shows a tight cluster of 502 Bad Gateway or ECONNRESET errors that starts the second a deploy begins and stops about ten seconds later, your app is almost certainly being killed while it still holds in-flight requests. The fix is not one setting. Four separate things have to be true — the process has to receive SIGTERM, handle it, stop accepting new connections while draining old ones, and stay alive long enough for the load balancer to stop routing to it — and most teams have exactly one or two of those in place.
I have debugged this on Kubernetes, on ECS, and on a plain Docker host, and the symptom is identical every time: a burst of errors that is too small to page anyone and too regular to be a coincidence. Here is how to find which of the four links is broken.
Why do deploys produce 502s at all?
Two clocks are running during a rollout, and nothing synchronizes them.
Clock one is the orchestrator killing your container. Kubernetes sends SIGTERM, waits terminationGracePeriodSeconds (30 by default), then sends SIGKILL. docker stop sends SIGTERM and waits 10 seconds before SIGKILL.
Clock two is the routing layer forgetting about your pod. In Kubernetes, removing the pod from the Service endpoints propagates asynchronously to every kube-proxy and every ingress controller. On ECS behind an ALB, the task is deregistered from the target group and then the deregistration delay runs down.
These two happen concurrently, not in sequence. The orchestrator does not wait for the routing layer to catch up before it starts killing your process. So for some window — usually a fraction of a second to a couple of seconds — traffic is still being sent to a container that has already been told to die. If that container exits immediately, every request in that window becomes a 502.
The takeaway: a 502 during deploy is not a crash, it is a race between the kill signal and the routing update.
Is your process even receiving SIGTERM?
Before touching shutdown logic, confirm the signal arrives. The most common reason it does not is the container's PID 1.
# Shell form: your process runs as a child of /bin/sh, which does NOT forward SIGTERM.
CMD npm start
# Exec form: your process IS PID 1 and receives the signal.
CMD ["node", "server.js"]
The shell form runs /bin/sh -c "npm start". The shell becomes PID 1, receives SIGTERM, and does nothing with it — your Node process never hears about it and dies 10 or 30 seconds later by SIGKILL. Wrapper commands are the other frequent culprit: anything that spawns your server as a child process has to explicitly forward signals, and process-manager wrappers are a common place for them to get swallowed.
Verify it in about fifteen seconds, on any machine:
docker run -d --name shutdowntest myimage:latest
docker stop shutdowntest
docker inspect shutdowntest --format '{{.State.ExitCode}} {{.State.OOMKilled}}'
Exit code 0 means you handled the signal and exited cleanly. 143 is 128 + 15, meaning SIGTERM was delivered and the default handler killed you — the signal arrived but nothing handled it. 137 is 128 + 9: you were SIGKILLed after the grace period expired, which means either you never got the signal or your shutdown handler hung.
If you need a real init process — because your app legitimately spawns children, or you inherited a shell-form entrypoint you cannot change — Docker's built-in init flag runs tini as PID 1 and forwards signals to your process correctly. The drawback is that it is a runtime flag, so it only helps where you control the run command; on Kubernetes you add tini or dumb-init to the image instead.
The takeaway: exit code 143 or 137 after a docker stop means your shutdown handler is either missing or hanging, and no amount of load balancer tuning will fix that.
What a correct Node shutdown handler looks like
Calling server.close() is necessary but famously not sufficient. It stops accepting new connections and waits for active ones to finish — but an idle keep-alive connection counts as active, so on a busy service the callback may never fire. Node 18.2 added the two methods that close that gap:
const http = require('http');
const server = http.createServer(app);
server.listen(3000);
let shuttingDown = false;
// The readiness endpoint flips first, before anything stops working.
app.get('/readyz', (req, res) => {
res.status(shuttingDown ? 503 : 200).send(shuttingDown ? 'draining' : 'ok');
});
function shutdown(signal) {
if (shuttingDown) return;
shuttingDown = true;
console.log(`${signal} received, draining`);
// Give the routing layer time to notice /readyz is failing.
setTimeout(() => {
server.close(async (err) => {
if (err) console.error('close error', err);
await pool.end(); // drain the DB pool
process.exit(err ? 1 : 0);
});
// Idle keep-alive sockets would otherwise hold close() open forever.
server.closeIdleConnections();
}, 5000);
// Hard ceiling, comfortably inside terminationGracePeriodSeconds.
setTimeout(() => {
console.error('drain timed out, forcing exit');
server.closeAllConnections();
process.exit(1);
}, 20000).unref();
}
['SIGTERM', 'SIGINT'].forEach((sig) => process.on(sig, () => shutdown(sig)));
Three details matter more than the rest. The 5-second delay before server.close() is the whole point — you keep serving normally while the load balancer notices you are unready. The forced-exit timer must be shorter than the orchestrator's grace period, or you get SIGKILLed mid-drain and lose the requests you were trying to protect. And attaching a SIGTERM listener removes Node's default exit behavior, so if your handler has a bug the process now hangs until it is killed — you have made things worse, not better.
If you would rather not hand-roll this, the stoppable and http-terminator packages wrap the same connection-draining logic behind one call. The tradeoff is a production dependency in the shutdown path, which is the last place you want a surprise.
The takeaway: sleep first, then close — a shutdown handler that starts closing sockets the instant SIGTERM lands is just a faster way to drop requests.
Where the delay belongs on each platform
The "wait before you stop serving" step can live in the app or in the platform. Pick one place and be explicit about it.
| Platform | What removes you from routing | Where to put the drain delay | Grace period knob |
|---|---|---|---|
| Kubernetes | Endpoint removal, propagated async to kube-proxy / ingress |
preStop sleep hook, or in-app delay after SIGTERM
|
terminationGracePeriodSeconds (default 30) |
| ECS + ALB | Target deregistration, then deregistration delay | Rely on deregistration delay; keep serving until then |
stopTimeout (task def), ALB deregistration delay |
| Plain Docker / Compose | Nothing — you are the routing layer | In-app delay |
docker stop -t, STOPSIGNAL
|
| Managed PaaS | Provider-controlled | In-app delay only | Usually fixed, check provider docs |
On Kubernetes the preStop hook is the version most teams should start with, because it works for any language without touching application code:
lifecycle:
preStop:
exec:
command: ["sleep", "5"]
terminationGracePeriodSeconds: 30
preStop runs before SIGTERM is sent, and the grace period clock only starts after it finishes — so your app keeps serving normally for those 5 seconds while endpoint removal propagates. Its real drawback is that sleep must exist in the image, which it does not in distroless or scratch builds; there you either use the built-in sleep action available in recent Kubernetes versions or move the delay into the app.
One rule that is easy to miss: readiness and liveness probes must behave differently during shutdown. Readiness should fail immediately so traffic stops. Liveness must keep passing, or the kubelet restarts a container that was in the middle of a clean exit.
The takeaway: put the drain delay in exactly one layer, and make sure your grace period is longer than your in-app forced-exit timer.
How do you prove it is fixed?
Do not trust a quiet dashboard — deploy under load and watch. Run a steady, low-rate request stream against the service and trigger a rollout:
# Any constant-rate client works; the point is a request every 100ms during the rollout.
while true; do
curl -s -o /dev/null -w "%{http_code}\n" https://your-service/healthz
sleep 0.1
done | sort | uniq -c
Roll the deployment in another terminal. A correct setup shows 200 for every line. If you see a handful of 502s, note how many: at 10 requests per second, six 502s means roughly a 600ms hole, which points at routing propagation rather than a missing handler. Dozens of them across the whole rollout points at the process dying instantly, and you should go back to the docker stop exit-code check.
The takeaway: the only convincing test for graceful shutdown is a real rollout under continuous traffic, because the bug only exists inside a window that idle traffic never hits.
FAQ
Why does my Kubernetes pod get 502 errors during rolling updates?
Because endpoint removal and SIGTERM happen concurrently, so traffic keeps arriving for a short window after your container starts shutting down. Add a preStop sleep of a few seconds so the pod keeps serving normally while the routing update propagates.
What is exit code 143 in Docker?
143 is 128 + 15, meaning the process was terminated by SIGTERM and used the default handler instead of shutting down cleanly. It confirms the signal was delivered — the problem is in your application, not in the container runtime.
Does server.close() in Node wait for in-flight requests?
Yes, but it also waits for idle keep-alive connections, so on a busy server the callback can be delayed indefinitely. Call server.closeIdleConnections() right after server.close() (available from Node 18.2) and keep a forced-exit timer as a backstop.
Bottom line
If you only do one thing, check the exit code after docker stop — that single number tells you whether you have a signal problem or a draining problem, and the two have completely different fixes. On Kubernetes, add a preStop sleep and a readiness endpoint that fails immediately on SIGTERM; that combination fixes the majority of deploy-time 502s without any application rewrite. Write the in-app handler when you have real cleanup to do — draining a database pool, finishing a queue job — and always give it a forced-exit timer shorter than the platform grace period. Then prove it with a rollout under constant traffic, because this is a bug that only exists during a window you will never hit by hand.
Top comments (0)