The arithmetic of four nines, why pager-first response can't meet it, and three shifts: burn-rate alerting, automated first brakes, and humans for governance.99.99% uptime sounds like a percentage. It is actually an unforgiving clock.
Across a rolling 30-day window (43,200 minutes):
SLO Downtime budget
99.9% 43.2 minutes
99.95% 21.6 minutes
99.99% 4.32 minutes
4.3 minutes is less than it takes many on-call engineers to open a laptop and find the right VPN token. If your SLO is four nines and your first response is a human reading a pager, the math says you have already breached.
(Quick terminology note: the SLO is your internal target and the SLA is the contractual promise, usually looser. This post is about the SLO.)
Shift 1: Burn rate over static thresholds
Paging on a single spike is both noisy and slow. Alert on consumption velocity instead:
burn_rate = observed_error_ratio / (1 - SLO)
At 99.99% the allowed error ratio is 0.0001:
Burn rate 1: you use exactly the whole budget by day 30.
Burn rate 14.4: a 0.144% error ratio, spending 2% of the monthly budget per hour.
Burn rate 10,000: a total outage, which empties the budget in 4.32 minutes.
Pair a long window with a short one, as described in the Google SRE Workbook, so the alert is both real and still happening:
Action Burn rate Long window Short window Budget spent
Page 14.4 1h 5m 2%
Page 6 6h 30m 5%
Ticket 1 3d 6h 10%
A tiny calculator to build intuition:
python
def burn_rate(error_ratio: float, slo: float) -> float:
return error_ratio / (1 - slo)
def minutes_to_exhaust(burn: float, window_minutes: int = 43_200) -> float:
return window_minutes / burn
slo = 0.9999
print(burn_rate(0.00144, slo)) # 14.4
print(minutes_to_exhaust(14.4) / 60) # ~50 hours to empty the month
print(minutes_to_exhaust(10_000)) # 4.32 minutes: total outage
Shift 2: Automated first brakes
At four nines, the first responder can't be a human. It must be an automated gate that can:
shed traffic,
trip a circuit breaker,
or roll back an unhealthy release,
before the 4-minute budget disappears. A minimal decision loop:
python
def first_brake(alarm, ctx):
if not ctx.flags["first_brakes_enabled"]: # human-controlled kill switch
return ctx.page("Automation disabled: manual response")
if ctx.actions_last_hour >= ctx.max_actions: # prevent flapping
return ctx.page("Automation budget exhausted: human takeover")
if ctx.deployed_within(minutes=30):
ctx.rollback_last_deploy()
else:
ctx.shed_load()
ctx.record(alarm)
ctx.page("First brake applied. Review and decide next step.")
Humans are still paged. The difference is the order: the brake is applied first, and the engineer arrives to a stabilizing system with a record of what happened.
Automation can cause outages too. Limit blast radius, rate-limit actions, keep an independent kill switch, prefer reversible actions, and rehearse the brakes with game days.
Shift 3: Reserve humans for governance
Humans can't beat a raw latency clock, but they are excellent at deciding the safe operating envelope. Shift the engineer's role from manual button-pusher at 3 AM to designing and reviewing:
the SLOs and burn-rate thresholds,
which actions automation may take, and under what conditions,
post-action reviews that tune the guardrails,
game days that prove the brakes work.
The honest limit
Detection time is bounded by metric resolution and evaluation period. A 1-minute metric can't catch a 5-second event, however good the alert math is. For four nines, invest in high-resolution metrics and short fast-burn windows.
Takeaway
If your SLA is 99.99%, automation isn't an efficiency upgrade. It is an arithmetic requirement.
What is the tightest SLO your production services run against today, and how much of your first response is fully hands-off? I'd like to hear which automated actions you trust to run unattended.

Top comments (0)