Our contract said 99.9 percent availability. When a customer asked us to prove we'd met it for the previous quarter, we discovered we had no idea. We had uptime checks, which reported 99.97, and we had an incident log, which suggested something closer to 99.4, and the two numbers were measuring different things badly. Neither of them measured what the customer experienced.
The uptime check was an HTTP GET against a health endpoint, from one region, every sixty seconds. That endpoint returned 200 as long as the process was alive. It stayed green through a forty minute outage where every real request failed on a database timeout, because the health handler didn't touch the database. It also stayed green during the incident where we served correct responses at eleven seconds each, since it had no latency threshold. We were measuring whether a process existed, and calling it availability.
Writing a real SLI took longer than I expected, and most of the time went into deciding what counts as a failed request. Health checks and bot traffic are excluded. A 400 is the client's problem, not ours, so it counts as served. A 429 from our own rate limiter counts as failed, which was argued about for a week and I still think is right. Anything slower than two seconds counts as failed even if it eventually returned, because from the user's side a response nobody waited for is not a response.
Once it was defined, the measurement came from the load balancer's own request logs rather than from a synthetic prober, because those are the actual requests actual users made. The first honest number for the quarter was 99.72. Below our commitment, above the incident log's guess, and for the first time a figure I'd defend in a room.
The useful part wasn't the number. It was the error budget it implied: about two hours of failed-request time per month, which turned "should we ship this risky change" from a debate into arithmetic. When the budget was spent, we stopped shipping features and fixed things.
An SLO you can't compute from real traffic is not a target. It's a sentence in a contract.
– Sergey Shinder
Top comments (0)