Originally published on kuryzhev.cloud
SLO burn-rate alerts are supposed to tell an on-call engineer one thing clearly: how fast the error budget is being consumed. A typical failure scenario looks like this: a marketing campaign or a bot crawler sends a sudden burst of traffic, request volume triples for ten minutes, and within minutes a page fires claiming the service will exhaust its monthly error budget in a couple of hours at the current burn rate. The dashboard looks alarming. The actual error rate, when checked manually, is well within normal bounds. The alert was arguably correct about the math and not very useful about the reality.
This is a common misunderstanding about SLO burn-rate alerts: they measure a rate of consumption, not an absolute failure count, and ratios can move sharply when volume changes even if underlying reliability has not meaningfully degraded. Understanding what burn-rate alerts actually compute — and where the traffic-spike blind spot comes from — is the difference between trusting the page and quietly routing it to a muted channel.
Failure scenario
Consider a service with a 99.9% availability SLO over a 30-day window. That target allows 0.1% of requests to fail; if traffic were perfectly uniform, it corresponds to roughly 43 minutes of full unavailability across the month. Most implementations express the budget in failed requests rather than wall-clock minutes, because request-based SLIs are what the alerting rules actually query.
A burn-rate alert compares the ratio of failed requests to total requests over a short window (say 5 minutes) and a longer window (say 1 hour), then compares that ratio against a multiple of the error budget. A burn rate of 1 means the budget is being consumed exactly as fast as the window allows; a burn rate of 14.4 means a 30-day budget would be gone in roughly two hours if the rate held.
During a traffic spike, the denominator grows quickly while the numerator can grow for reasons unrelated to a code regression. Suppose baseline traffic is 1,000 requests per minute with 1 error — a 0.1% error rate, exactly at the target, so the baseline burn rate is 1x. If traffic climbs to 5,000 requests per minute and errors climb to 72 because a connection pool sized for baseline load starts rejecting, the ratio is 1.44%: a 14.4x burn rate against the 0.1% budget, which is enough to trip a paging threshold on its own. Note what the ratio does and does not tell you here: errors grew 72x in absolute terms while the ratio grew only 14.4x, because the denominator grew alongside them. A rule keyed only to the ratio cannot distinguish that shape from a sustained regression at steady traffic, and may page on behavior the service returns from as soon as autoscaling catches up.
Watch out for alerts configured with a single short window (like 5 minutes). They react fastest to spikes and are the most likely to fire on transient traffic shape changes rather than genuine reliability degradation.
Why it happens
Burn-rate alerting, as described in the Google SRE workbook's multiwindow, multi-burn-rate approach, is designed to balance two competing needs: catching fast, severe outages quickly, and avoiding pages for noise. Implementations commonly use a short window (5m–1h) for fast detection and a long window (1h–6h) for confirmation, requiring both to breach a threshold before paging.
The traffic-spike problem appears when the short window is sensitive enough to catch a real two-minute outage, while the underlying infrastructure — connection pools, downstream rate limits, autoscaling lag — genuinely does produce more errors under sudden load. In that case the alert is not wrong: the error ratio did increase, and if it held for the full budget period it would exhaust the SLO. The weakness is the implicit assumption that current conditions are representative of the next hour, which often fails for a spike that resolves once autoscaling catches up or campaign traffic tapers off.
A second, quieter cause is how the query is constructed. If the burn-rate expression recalculates from raw counters at every evaluation cycle with no confirmation window, a five-minute burst dominates the short-window ratio while the rolling hour it is compared against has barely moved. Verify with the query itself rather than assuming: run the short-window and long-window expressions against Prometheus for the spike interval using promtool query instant, or the HTTP /api/v1/query_range endpoint for a time series, and compare the short-window ratio against the long-window ratio over the same period. If the short window jumped far beyond what the hour confirms, the alert shape is the problem rather than the service.
One more detail worth checking: the selector must match how failures are actually recorded. A rule that matches only status=~"5.." will miss client-side timeouts that never produced a response status, so the SLI definition and the alert expression need to agree on what counts as a failed request.
The fix
The standard fix is not to remove burn-rate alerting but to implement it with multiple windows and multiple burn-rate thresholds, as documented in the Google SRE workbook. A fast-burn alert (high threshold, short window) should require confirmation from a longer window before paging; a single-window alert should not page on its own.
Here is a Prometheus recording rule and alert pair implementing a multi-window burn-rate check for a 99.9% SLO:
groups:
- name: slo-burn-rate
rules:
# Single source of truth for the SLO error budget (1 - 0.999)
- record: slo:error_budget:ratio
expr: vector(0.001)
# Fast window: catches severe short outages
- record: slo:requests_errors:ratio_rate5m
expr: |
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))
# Slow window: confirms the trend isn't a transient spike
- record: slo:requests_errors:ratio_rate1h
expr: |
sum(rate(http_requests_total{status=~"5.."}[1h]))
/
sum(rate(http_requests_total[1h]))
- name: slo-burn-rate-alerts
rules:
- alert: ErrorBudgetFastBurn
# A 14.4x burn rate exhausts a 30-day budget in roughly
# two hours if sustained. Page only when BOTH windows
# agree, not just the noisy 5m one.
expr: |
slo:requests_errors:ratio_rate5m
> on() group_left() (14.4 * slo:error_budget:ratio)
and
slo:requests_errors:ratio_rate1h
> on() group_left() (14.4 * slo:error_budget:ratio)
for: 2m
labels:
severity: page
annotations:
summary: "Fast error budget burn confirmed over 5m and 1h windows"
Requiring agreement across windows filters out the common case where a spike produces a sharp five-minute ratio jump that the one-hour window does not confirm. For teams building this in Grafana, the Grafana Alerting documentation covers multi-condition alert rules; depending on the Grafana version, two queries can be combined with a boolean expression in the rule editor rather than maintaining two separate alert definitions.
A complementary fix is gating on absolute error volume alongside the ratio, since a true incident usually produces both a ratio increase and a meaningful rise in raw error count:
sum(increase(http_requests_total{status=~"5.."}[5m])) > 50
and
slo:requests_errors:ratio_rate5m
> on() group_left() (14.4 * slo:error_budget:ratio)
Pick the absolute threshold from the service's own baseline rather than copying 50; it is a floor that says "too few failures to wake anyone," and the right value depends on traffic volume.
Watch out for hardcoding the numeric SLO target in every alert expression. When the target changes — say from 99.9% to 99.95% — every burn-rate rule needs updating in sync, and it is easy to miss one. Recording the budget once as its own rule (as above), or templating the rule files, keeps the threshold in one place; see the Alertmanager routing notes on DevOps_DayS for how the resulting labels affect routing.
Prevention checklist
A few structural choices prevent most traffic-spike false pages before they happen:
- Use multi-window, multi-burn-rate alerts (5m+1h, 30m+6h) instead of a single window
- Require BOTH windows to breach threshold before paging, not either
- Alert on absolute error count as a secondary gate, not ratio alone
- Set separate burn-rate thresholds for page vs. ticket severity
(e.g. 14.4x for page, 6x for a lower-urgency ticket)
- Keep the SLI selector aligned with the SLO definition (timeouts included)
- Review burn-rate alert history after every known traffic spike
and note whether it paged unnecessarily
- Keep the SLO target as a single source of truth (recording rule or template),
not copy-pasted across every alert expression
- Document the expected behavior during known high-traffic events
(product launches, sales, scheduled batch jobs) in the runbook
None of this eliminates every noisy page. Capacity problems that only appear under load — a connection pool sized for baseline traffic, a downstream API with its own rate limit — are real reliability risks and may well deserve a page, even if they only reveal themselves during a spike. The goal of tuning SLO burn-rate alerts is not silence; it is making sure the page reflects a sustained trend the error budget actually cares about, rather than a five-minute ratio artifact. The burn-rate math and window-selection tables live in the Google SRE workbook chapter on alerting on SLOs, and the Prometheus alerting best practices guide covers the more general question of what belongs in a page at all — both are worth reading before finalizing thresholds for a specific SLO.
Top comments (0)