I went looking for dropped requests during a rolling restart and found something more annoying than dropped requests: a deploy that looks perfect and isn't.
Setup is deliberately boring. Two Node/Express replicas behind nginx, ten clients hammering an endpoint that takes three seconds, docker stop on one replica halfway through. The app is the version most of us have shipped at some point, with no signal handling at all:
app.get('/work', async (req, res) => {
await new Promise(r => setTimeout(r, 3000));
res.json({ ok: true, pid: process.pid });
});
app.listen(8080);
No SIGTERM handler. Docker sends the signal, Node exits, and anything mid-flight dies with it. I expected a pile of 502s.
total=65 ok=65 failed=0
Zero. Three runs, zero every time.
The failure was there, nginx just paid for it
The requests did die. nginx caught the upstream connection dropping before any response headers had gone out, so it quietly opened a connection to the other replica and ran the whole thing again. The client never knew.
That behaviour is proxy_next_upstream, it's on by default, and I want to be fair to it because it is doing exactly what you'd want a reverse proxy to do when a backend disappears mid-request. It is also the reason your dashboard can report a flawless deploy while the thing being deployed is quietly broken, which is a strange position for a metric to be in.
The only place it shows up is latency:
naive app: p50 = 3.01s p99 = 5.94s
graceful app: p50 = 3.02s p99 = 3.04s
Same zero-error result, same load, same everything. The affected requests took twice as long, because they were executed twice. If you are watching error rate you see nothing. If you are watching p99 you see a spike at every deploy that you have probably learned to ignore.
Three seconds of extra latency is survivable. Doing the work twice might not be, and that depends entirely on what the work is: a retried search query costs you nothing, a retried outbound email costs you a duplicate, and a retried payment authorisation costs you a phone call from someone in finance. nginx has no idea which of those it just re-ran.
Take the safety net away
Plenty of setups don't have that retry. A Kubernetes Service is iptables or IPVS, and it does not re-run your request. An L4 load balancer won't. A client talking straight to your app certainly won't. Once headers are on the wire, even nginx can't.
Same test, proxy_next_upstream off:
naive app: 5 / 70 requests failed (7%)
There is the pile of 502s I went looking for. Nothing about the app changed. The only difference is whether something upstream was covering for it.
The fix, and why it is only most of a fix
The app-level fix is the one everybody writes about. Stop accepting new connections, let the in-flight ones finish, then exit:
const server = app.listen(8080);
process.on('SIGTERM', () => {
server.close(() => process.exit(0));
});
Same test:
naive: 5 / 70 failed
graceful: 1 / 71 failed
Better. Not fixed. That last failure is stubborn, it showed up on all three runs, and it is the interesting one.
The in-flight requests are safe now. What's left is the requests arriving in the gap between server.close() and the load balancer working out that this instance is gone, and during that gap nginx is still holding the address in its upstream list, so it does the reasonable thing and opens a fresh connection to a socket that has just stopped accepting them, which produces a 502 for a client who did nothing wrong.
No amount of application code fixes that. The app has already done the right thing. The load balancer is the one still pointing at it.
Take it out of rotation first
Remove the instance from the load balancer, give the change a second to settle, and only then send SIGTERM:
graceful + drained from the LB first: 0 / 70 failed (3 runs)
That's the whole ordering. Stop routing to it, then stop it. In Kubernetes this is what a preStop hook buys you, and it is why preStop: sleep 5 looks like a hack and isn't. The sleep isn't for your app, which is already finished. It's to let the endpoint removal propagate before the container goes away.
What I got wrong on the way
My first attempt at the drain test came back with 3, 1 and 1 failures, and I nearly wrote a paragraph explaining that draining doesn't help as much as you'd hope.
It was my bug. I had mounted nginx.conf read-only, so the command that swapped in the drained config failed without complaining, no drain ever happened, and what I had actually done was run the same test twice and then write an explanation for the difference between two identical things. With a writable mount it's zero out of seventy, three runs in a row.
Worth saying out loud because the failure mode is so ordinary: my test harness was broken in a way that produced plausible numbers.
The whole thing
failed/total p50 p99
naive, nginx retry on (default) 0 / 65 3.01s 5.94s
graceful, nginx retry on 0 / 70 3.02s 3.04s
naive, no retry 5 / 70 3.01s 3.05s
graceful, no retry 1 / 71 3.02s 3.05s
graceful + drained from LB first 0 / 70 3.02s 3.05s
Three runs of each, identical every run.
What I'd check on Monday
Error rate is not going to tell you whether you have this. If there is a retrying proxy in front of your app, error rate is exactly the metric that will hide it.
Look at p99 during a deploy instead. A tail that roughly doubles for a few seconds and then settles back is the signature, because that is the shape of a request being run, killed, and run again somewhere else.
Then check the two halves separately, because they fail independently. Does your app have a SIGTERM handler that finishes in-flight work? And does your platform stop routing to the instance before it sends that signal? The first without the second still leaks requests, just fewer of them.
All of it runs on a laptop. Two containers, nginx, and a load generator that counts outcomes. No cloud account and nothing to sign up for.
Top comments (0)