Everyone worries about the health check that answers 200 while the server is dead. Nobody worries about the one that answers 503 while the server is fine. Last week, the second one helped take down Microsoft 365.
On November 12, Exchange Online servers started failing automated health checks and were pulled out of service. The servers that were left took the load and overloaded. Servers that restarted failed the checks again immediately. Mailboxes stopped opening in Outlook on the desktop, on the web, and on phones. Teams calendars, SharePoint, and Copilot went with them. Microsoft lists more than 15 affected services. The outage ran just under 24 hours, from about 16:20 UTC on November 12 to 16:10 UTC on November 13.
Microsoft's preliminary cause is an issue during an earlier network capacity upgrade that surfaced after a later network change, a latent bug in the systems that establish secure connections. It was not an attack and it had nothing to do with AI. The full post-incident report is still to come, so the cause may get sharper. But the shape of the failure is already clear, and the shape is the lesson.
The health check did what it was written to do. The servers genuinely could not establish secure connections, so the check said so and the load balancer pulled them out. The problem is that nothing in the system could tell "this server is broken" apart from "the thing this server depends on is broken". The dependency was shared, so the check failed everywhere at once, and the load balancer dutifully drained the whole fleet. Every server it removed pushed its connections onto servers that were already short of capacity. The check was the amplifier. It turned a bad hour into a lost day.
This is the failure mode in the other direction from the one I usually write about. Two days before the outage I posted a thread about health checks that answer 200 while the database is unreachable. That check is a lie: the process is running, so it says yes, while the server cannot serve a single request, and the load balancer keeps sending it traffic until a user reports the error. My check opens a connection to the database and runs a query that touches a real table. It checks the queue the workers read from. If any of them fails, the endpoint answers 503 and the server leaves rotation before users notice. I learned the shape of this from email work, where /health returned 200 while SES was throttling, and the workflow was broken while the status page was green. A check has to validate the real dependencies: queue depth, the SMTP handshake, the bounce and complaint spikes. A health check is a promise that the service can do its job. Make it check the job.
But a check that tests the job can fail in the other direction too, and that is what Microsoft ran into. A readiness check that touches a shared dependency will fail on every server at the same time when that dependency breaks. The load balancer cannot see the difference between one dead server and fifty healthy servers with the same broken dependency. It only sees fifty failing checks.
While the pool shrank, the clients kept retrying. That is the second mechanism in this outage, and it is the same one from yesterday's webhook extra: a retry with no backoff and no limit is a sender hammering your endpoint while your handler is already struggling. Retries pour load onto the servers that are still standing, which is exactly the load they cannot take. My rule for webhooks is a limit, backoff, and a dead-letter queue at the end, because a webhook that has failed ten times is not going to succeed on the eleventh. Microsoft reached for the same instinct at planetary scale: it extended the email timeout window to 48 hours, so delayed mail kept retrying gently instead of bouncing. Give the queue time instead of giving up on it.
Three rules come out of the wreckage.
First, check the job, not the process. Liveness and readiness are different endpoints. Liveness asks whether the process is running. Readiness asks whether it can do its work. A readiness check touches the database, the queue, the dependency that matters, and answers 503 when any of them fails.
Second, separate "this instance is broken" from "a shared dependency is down". If one dependency fails everywhere, failing every health check drains the fleet. Check shared dependencies on a separate endpoint that the load balancer does not use to drain servers, or degrade instead of failing. A server that cannot reach one dependency may still serve the requests that do not need it.
Third, give retries a budget. Backoff, a limit, a dead-letter queue. A retry that never stops is load, not reliability. The handler that can run twice without harm is the safety net; the retry only gives it another chance to answer.
Go look at your own /health endpoint. If it only checks that the process is running, it is not checking anything. What does yours check?
If you build systems like this, follow along. I write about the failure modes that only show up when real traffic arrives.
Top comments (0)