Every deploy produced a small spike of 502s. Maybe forty errors, over about ten seconds, then clean. Small enough that it lived on a dashboard for a year as "deploy noise" and small enough that nobody had the appetite to chase it, until a customer with a batch integration started retrying those failures into a much larger problem.
The cause was in the Dockerfile. The entrypoint was written in shell form, CMD npm start, which means the container's PID 1 is /bin/sh -c, and the application is a child of it. Shell does not forward signals to children. Kubernetes sent SIGTERM at the start of termination, the shell received it, the Node process never heard about it, and thirty seconds later the kubelet sent SIGKILL. Every request the pod was serving at that moment died mid-flight.
We had written a graceful shutdown handler in the application. It had never once executed in production. That's the detail I find worth repeating: the code was correct, tested, and unreachable, and nothing in our tooling could tell us that.
The fix has two halves and both were needed. Exec form in the Dockerfile, CMD ["node", "server.js"], so the process is PID 1 and gets the signal directly. Then a shutdown handler that stops accepting new connections, waits for in-flight requests to finish with a deadline shorter than terminationGracePeriodSeconds, and exits.
That still left a smaller spike, because pod termination and endpoint removal happen in parallel. The kubelet starts killing while the proxy on some node is still routing to that address. A preStop sleep of five seconds, doing nothing except delaying the SIGTERM until the endpoint change has propagated, removed the rest of it.
What I took away is that shutdown is the least exercised path in most services and the one your users notice on every single release. We test startup constantly, by accident, because nothing works otherwise. Nobody tests the last two seconds. Now our smoke test sends traffic while it rolls a deployment, and a single dropped connection fails the build.
Graceful shutdown you have never observed working is a comment, not a feature.
– Sergey Shinder
Top comments (0)