Checkout errors sat between three and eight percent for most of a Thursday. Not enough to page, because our threshold is one percent sustained across the service and the service was at 0.4. Enough that retries covered most of it and support saw a thin, steady trickle of people who had tried twice and given up.
It took three hours to establish that one instance out of twenty-four was producing nearly all of it. The host had a degraded network interface. Roughly one connection in ten to that instance stalled and eventually timed out. Everything else about it looked completely ordinary: it was serving traffic, using CPU, reporting metrics, and answering the load balancer's health check every ten seconds without missing one, because a small GET to a cheap endpoint usually got through.
Two assumptions failed together. The health check asked the instance about itself rather than looking at what real requests were experiencing, and a process on a machine with a bad NIC is perfectly capable of saying it feels fine. And every dashboard we owned aggregated across the fleet, where one instance failing eight percent of its own share is a rounding error in the total.
The fixes are structural. Per instance error rate and latency now sit on the service dashboard, with an alert that fires when any single instance runs at five times the fleet median for five minutes. That is a comparison rather than a threshold, so it needs no tuning and it works at any traffic level. The load balancer does passive health checking alongside the active kind, ejecting a target on the failures of real requests instead of the success of a synthetic one. And our provider's hardware health events, which had raised a degraded hardware notice for that host ninety minutes before the first customer complaint, now arrive in the same channel as everything else.
The habit that changed is smaller and matters more. A suspected instance gets removed first and investigated afterwards. Most of that Thursday went on deciding whether a machine we could have replaced in ninety seconds was really the problem.
Our failure model had two states in it, working and gone. Nearly everything expensive I have dealt with since lives between them.
– Sergey Shinder
Top comments (0)