Our customer portal runs on four application servers behind nginx, in an upstream block with nothing but addresses on its server lines:
upstream portal {
server 10.20.1.11:8080;
server 10.20.1.12:8080;
server 10.20.1.13:8080;
server 10.20.1.14:8080;
}
Nothing but addresses means the defaults, and two of them matter here. max_fails is 1 and fail_timeout is ten seconds, so one failed attempt to reach a server marks it unavailable for ten seconds. And proxy_next_upstream is error timeout, so when an attempt fails or times out, nginx passes the request to the next server, for any request that is safe to repeat, which includes every GET. A timeout always counts as a failed attempt, whatever proxy_next_upstream says.
On the first working day of November our branch managers opened the month end statement report. It is a GET, it builds a PDF inside the request, and with October's volumes it ran past proxy_read_timeout, sixty seconds, before sending a byte. nginx counted a failure against the first server, marked it down, and sent the same request to the second, which also took more than a minute, and so on through all four.
Fourteen managers did this between nine and ten, several of them more than once when nothing came back. Each request walked across the whole pool, marking servers down as it went. For most of that hour at least one server was out of rotation and the rest carried its share, and at moments all four were marked at once and nginx answered everyone else with 502, logging no live upstreams. About one request in six failed. The servers themselves were fine throughout, at forty percent CPU. A slow report was being recorded as four sick servers, by a load balancer doing what its defaults said.
The statement report is now generated in the background and emailed, so no request runs for more than a few seconds. The /reports location points at its own upstream block with the same servers, max_fails=0 so its timeouts are never counted, and proxy_next_upstream off. The main upstream keeps passive checks but needs three failures in thirty seconds. We alert on no live upstreams in the error log, and our load test now includes one request that runs past the timeout, to show us what a slow request does to everyone else.
A failure count is only useful if what it counts is the server failing.
– Sergey Shinder
Top comments (0)