Deploys were producing about a dozen 502s each, concentrated in a two-second window, and only on requests that took longer than five seconds. The service had preStop hooks, a terminationGracePeriodSeconds, and a SIGTERM handler. All of it was there. The order was wrong.
What shutdown actually did
def shutdown(signum, frame):
server.stop_accepting()
time.sleep(5)
sys.exit(0)
That reads like "stop taking new work, let in-flight work finish, exit". The third line is the problem: sys.exit does not wait for anything. It raises SystemExit in the main thread, which unwinds the stack and ends the process with open transactions, open sockets, and any in-flight requests still being handled on worker threads.
Five seconds happened to be shorter than the slowest write path. So the requests that mattered — the ones holding an open transaction — were exactly the ones killed.
Graceful shutdown is four ordered steps
1. Fail readiness before you stop serving. The load balancer needs time to notice. If the readiness probe runs every ten seconds, the pod must fail it and then wait at least one full interval before refusing connections. Otherwise the LB keeps sending traffic for up to ten seconds after you stopped accepting it, and those requests fail at the TCP layer. This is not a shutdown problem, it is a "the LB never heard you" problem, and it is the most common cause of 502s during rollout.
2. Stop accepting new connections, keep the listener open just long enough. In Kubernetes, preStop: sleep 10 before SIGTERM is the crude version and it works. The precise version is a readiness endpoint that returns 503 immediately on the signal, a sleep of two probe intervals, then the socket closes.
3. Drain in-flight requests with a counter, not a sleep. Track started/finished requests around the handler. Wait until the counter is zero or the deadline expires. A sleep is a guess about your tail latency; the counter is the actual number.
4. Close the database pool last, and never mid-transaction. The pool's close should wait for checked-out connections. If a handler is between BEGIN and COMMIT, killing the process rolls back work the client already believes succeeded — and if there is no idempotency key, the client retries and you get a double write instead of a clean failure.
The budget has to be smaller than the killer
The orchestrator sends SIGKILL at the end of the grace period, and that is not negotiable. So the app's own deadline must be strictly shorter:
| Stage | Budget |
|---|---|
| Readiness turns 503 | t+0 |
| Two probe intervals | t+0 → t+20s |
| Drain in-flight | t+20s → t+45s |
| Close pool, flush logs | t+45s → t+50s |
terminationGracePeriodSeconds |
60s |
If the drain deadline is 30s and the grace period is 30s, you have written code that assumes the drain always finishes instantly. It does not. The failing requests are always the slow ones — that is what "slowest" means.
What to test
Shutdown is testable and almost nobody tests it. Send SIGTERM to a running instance under load, and assert two things: zero requests return a connection error, and every request that was mid-transaction either committed or was cleanly rolled back and retried.
The version that finally worked here was eleven lines: a signal handler that flips readiness, a counter around the handler, a drain loop with a deadline, and a pool.close() that waits. The 502s went to zero. The original five-second sleep was not too short — it was in the wrong place, doing the job of a counter.
I write about production failures in Postgres, queues, and distributed systems.
Top comments (0)