Nijo George Payyappilly

Posted on Aug 10

Multi-Window Burn Rate Alerting: A Formal Analysis of the AND-Gate Logic Behind Google's Alert Model

#kubernetes #sre #reliability #devops

Alerting on Service Level Objectives is a solved problem in the sense that the solution exists, has been published, and works reliably when implemented correctly. It is an unsolved problem in the sense that most organisations implementing SLOs do not implement it correctly — they alert on threshold breaches of individual metrics rather than on error budget consumption rates, and they use single measurement windows rather than the dual-window AND-gate structure that distinguishes a genuine sustained burn from a transient spike.

The cost of getting this wrong is operational. An alerting system that pages too frequently trains engineers to treat pages as background noise — the alert equivalent of a car alarm that everyone has learned to ignore. An alerting system that pages too infrequently allows budget burns to proceed undetected until the SLO is already breached. Both failure modes are self-reinforcing: the first erodes the operational culture that makes on-call sustainable; the second erodes the error budget that makes deployment velocity permissible.

The Google SRE Workbook's multi-window burn rate alerting model solves both failure modes simultaneously. This post derives it formally — not as a recipe to follow, but as a logical structure to understand. Understanding why the model works is what enables practitioners to adapt it correctly to their specific SLO windows, service characteristics, and operational environments.

The Single-Window Alerting Trade-off

Before deriving the multi-window model, the problem it solves must be precise. The fundamental constraint of single-window alerting is this: for any fixed measurement window W and any fixed alerting threshold T, the alert exhibits an irreducible trade-off between sensitivity (detecting genuine budget consumption early) and specificity (not firing on transient spikes that self-correct).

────────────────────────────────────────────────────────────────────────────
SINGLE-WINDOW ALERTING: THE UNAVOIDABLE TRADE-OFF

Given:
  SLO window: 28 days
  SLO target: 99.9% (error budget = 0.1%)
  Single measurement window W: T minutes

SHORT WINDOW (W = 5 minutes):
  Sensitivity: HIGH — detects burn rate changes quickly
  Specificity: LOW — every brief error spike triggers the alert
    Example: A 30-second traffic anomaly causing 5% errors for 30s
    Error rate in 5-minute window: (0.3/5) × 5% = ~0.3%
    This is 3× the error budget rate → alert fires
    But: the anomaly self-corrected; budget consumed was negligible
    Result: PAGE AT 2 AM for a self-correcting 30-second spike

LONG WINDOW (W = 1 hour):
  Sensitivity: LOW — requires sustained errors to trigger
  Specificity: HIGH — short spikes diluted across the window
    Example: A 10-minute complete outage (100% error rate)
    Error rate in 60-minute window: 10/60 = 16.7% → 167× budget rate
    This exceeds any reasonable threshold → alert fires
    But: by the time the 60-minute window accumulates enough signal,
    4.2% of the 28-day budget has already been consumed
    At severe burns, detection is too late to prevent material impact

FUNDAMENTAL CONSTRAINT:
  For any single window W:
    Shorter W → more false positives (specificity decreases)
    Longer W → slower detection (sensitivity decreases)
    No single W value optimises both simultaneously
────────────────────────────────────────────────────────────────────────────

The dual-window AND-gate escapes this constraint by using two windows simultaneously: a long window to confirm that a burn is sustained (not a transient spike), and a short window to confirm that the burn is still ongoing (not a historical artifact). The AND of the two conditions provides the noise suppression of the long window with the recency validation of the short window.

Formal Definition: Burn Rate

Before the AND-gate logic can be analysed, the burn rate must be defined precisely.

────────────────────────────────────────────────────────────────────────────
BURN RATE: FORMAL DEFINITION

Let:
  E     = error budget as a proportion (e.g., 0.001 for 99.9% SLO)
  W_SLO = SLO measurement window in hours (e.g., 672h for 28 days)
  r(t)  = error rate at time t (proportion of requests failing)

Budget consumption rate at 1× burn:
  r_baseline = E / W_SLO
  (the error rate at which the budget would be exactly exhausted at window end)

  For 99.9% SLO over 28 days:
    r_baseline = 0.001 / 672 = 0.00000149 per hour
               = 0.00149 per 1000 hours (negligible hourly)

Burn rate B at time t:
  B(t) = r(t) / r_baseline

  Interpretation:
    B = 1   → error rate on pace to exactly exhaust budget in 28 days
    B = 14  → error rate 14× r_baseline; budget exhausted in 28/14 = 2 days
    B = 6   → budget exhausted in 28/6 ≈ 4.7 days
    B = 3   → budget exhausted in 28/3 ≈ 9.3 days
    B = 0   → no errors; budget not being consumed

Time to budget exhaustion at constant burn rate B:
  t_exhaustion = W_SLO / B

  B = 14: t_exhaustion = 672h / 14 = 48h (2 days)
  B = 6:  t_exhaustion = 672h / 6  = 112h (~4.7 days)
  B = 3:  t_exhaustion = 672h / 3  = 224h (~9.3 days)
  B = 1:  t_exhaustion = 672h / 1  = 672h (28 days, exactly at window end)

IMPORTANT DISTINCTION:
  t_exhaustion is the time until budget runs out IF burn continues.
  The detection window (1h, 6h) is the window used to MEASURE B(t).
  These are not the same. Detection at B=14 over a 1-hour window means:
    → The burn has been at 14× for the last 1 hour
    → If it continues, budget exhausts in ~47 more hours
    → The 1 hour of burning has consumed 14/672 ≈ 2.1% of total budget
────────────────────────────────────────────────────────────────────────────

The AND-Gate: Formal Analysis

For each alert tier, two windows are defined: a long window W_L that estimates the sustained burn rate, and a short window W_S that validates the burn is current. The alert fires if and only if the burn rate exceeds threshold B_threshold in BOTH windows simultaneously.

────────────────────────────────────────────────────────────────────────────
AND-GATE FORMAL SPECIFICATION

Alert condition:
  FIRE if: B(W_L) > B_threshold  AND  B(W_S) > B_threshold

Where:
  B(W) = mean burn rate over window W
  W_L  = long window (1h, 6h, 1d, 3d depending on tier)
  W_S  = short window (5m, 30m, 2h, 6h depending on tier)

NOISE SUPPRESSION PROPERTY:
  A transient spike of duration d < W_S that creates burn rate B_spike:
    B(W_S) = (d/W_S) × B_spike + ((W_S-d)/W_S) × 0
           = (d/W_S) × B_spike

  For the AND condition to fire:
    (d/W_S) × B_spike > B_threshold
    Required spike: B_spike > B_threshold × (W_S/d)

  Example: W_S = 5 minutes, B_threshold = 14, spike duration d = 30 seconds
    Required spike rate: 14 × (5/0.5) = 14 × 10 = 140× burn rate
    A 30-second spike must create a 140× burn rate to trigger the short window
    At 99.9% SLO, 140× burn rate = 14% error rate for 30 seconds
    Most transient anomalies fall below this threshold → noise suppressed

RECENCY VALIDATION PROPERTY:
  A historical burn event that ended T hours ago:
    For T > W_S: B(W_S) ≈ 0 → AND condition fails → no stale alert
    For T < W_S: B(W_S) > 0 proportional to recency → may still page
    For T > W_L: B(W_L) ≈ 0 → AND condition fails definitively

  This property ensures that a burn event that resolved T hours ago
  stops generating pages once T > W_S — typically within 5 minutes
  of the error rate returning to normal.

MINIMUM DETECTION LATENCY:
  Minimum time before alert fires after burn begins (at constant B > B_threshold):
    t_min_detection = W_S (must accumulate W_S of signal in short window)

  For the 14× tier with W_S = 5 minutes:
    t_min_detection = 5 minutes
    Budget consumed before detection: 14 × (5/672/60) × 100% ≈ 0.17%
    Fraction of 28-day budget consumed before page: 17% (of total 0.1% budget)
────────────────────────────────────────────────────────────────────────────

Deriving the Four Tiers: Why These Multipliers?

The four burn rate thresholds — 14×, 6×, 3×, 1× — are not arbitrary. Each is calibrated to a specific combination of urgency (time to budget exhaustion) and detection window (how much budget is consumed before the alert fires).

────────────────────────────────────────────────────────────────────────────
TIER DERIVATION: CALIBRATION CRITERIA

DESIGN CRITERIA FOR EACH TIER:
  1. Time to budget exhaustion at threshold burn rate (urgency)
  2. Budget consumed before detection (detection cost)
  3. Appropriate organisational response (page vs ticket)
  4. Detection window feasibility (window short enough to fire early)

TIER 1 — PAGE IMMEDIATELY (14× burn, 1h/5m windows)

  Urgency:
    t_exhaustion = 28 days / 14 = 2 days
    "If nothing changes, SLO is breached in 2 days"
    → Requires immediate human response; cannot wait for business hours

  Detection cost:
    Budget consumed before short-window detection (5 min): ~0.17% of budget
    Budget consumed before long-window confirmation (1 hr): ~2.1% of budget
    This is acceptable: detection while most of the budget is still intact

  Why 14×, not 20× or 10×:
    20× → t_exhaustion = 1.4 days; detection before 1 hour often impossible
    10× → t_exhaustion = 2.8 days; urgency threshold ambiguous for paging
    14× provides a clean "2-day" exhaustion horizon with 1-hour detection window

TIER 2 — PAGE (WITHIN 30 MIN) (6× burn, 6h/30m windows)

  Urgency:
    t_exhaustion = 28 days / 6 = 4.67 days
    "Budget exhausted in under 5 days; needs response today"
    → Requires on-call page; response within 30 minutes

  Detection cost:
    Budget consumed before short-window detection (30 min): ~0.74% of budget
    Budget consumed before long-window confirmation (6 hr): ~8.9% of budget
    At 6× burn, 6-hour confirmation costs ~8.9% of total budget;
    acceptable given the lower urgency vs Tier 1

  Why 6h long window:
    6h window smooths out periods of elevated but not catastrophic error rate
    A service spiking to high error rates for 1-2 hours dilutes to below
    threshold in a 6-hour window → prevents Tier 2 pages for recoverable spikes

TIER 3 — TICKET (BUSINESS HOURS) (3× burn, 1d/2h windows)

  Urgency:
    t_exhaustion = 28 days / 3 = 9.3 days
    "Budget exhausted in ~9 days; needs attention this week"
    → Does not warrant waking someone up; create a ticket

  Detection cost:
    Budget consumed before 2h short-window detection: ~0.89% of budget
    At 3× sustained burn, this is manageable
    The 1-day long window provides very high noise suppression

  Why 1-day long window:
    A 3× burn rate is only ~3× the normal error rate
    Day-to-day traffic variation can produce sustained periods at 2-3×
    without representing a genuine reliability problem
    1-day window ensures the ticket tier only fires on genuinely
    sustained moderate burns, not daily traffic variation

TIER 4 — TREND REVIEW (1× burn, 3d/6h windows)

  Urgency:
    t_exhaustion = 28 days / 1 = 28 days (exactly exhausts at window end)
    "On pace to breach SLO; needs review but not urgent"
    → Weekly SRE sync item; not a ticket or a page

  Why 3-day long window:
    1× burn rate is definitionally "on pace to exhaust at window end"
    This is the background noise rate — the service is barely over its
    error budget allocation
    A 3-day window provides extremely high noise suppression;
    only genuinely sustained 1× burns generate the trend alert

────────────────────────────────────────────────────────────────────────────
TIER SUMMARY TABLE (28-day window, 99.9% SLO)

Tier  Multiplier  Long    Short  t_exhaust  Budget at  Response
                  Window  Window            Long Det.
────  ──────────  ──────  ─────  ─────────  ─────────  ──────────────
1     14×         1h      5m     2 days     ~2.1%      Page immediately
2     6×          6h      30m    4.7 days   ~8.9%      Page within 30min
3     3×          1d      2h     9.3 days   ~10.7%     Ticket this week
4     1×          3d      6h     28 days    ~10.7%     Weekly review
────────────────────────────────────────────────────────────────────────────

Complete Prometheus Implementation

# Multi-Window Burn Rate Alerting — Complete Implementation
# SLO: 99.9% availability over 28-day rolling window
# All four tiers with AND-gate dual windows

groups:
  - name: slo.burnrate.recording
    interval: 30s
    rules:

      # Base SLI: request success ratio
      - record: sli:http_request_success:ratio_rate5m
        expr: |
          sum(rate(http_requests_total{
            job="api-server",
            status!~"5.."
          }[5m]))
          /
          sum(rate(http_requests_total{job="api-server"}[5m]))

      # Burn rate at multiple windows — the key recording rules
      # These power all four alert tiers
      - record: slo:error_budget_burn_rate:ratio_rate5m
        expr: (1 - sli:http_request_success:ratio_rate5m) / (1 - 0.999)

      - record: slo:error_budget_burn_rate:ratio_rate30m
        expr: |
          (1 - sum(rate(http_requests_total{status!~"5.."}[30m]))
               / sum(rate(http_requests_total[30m])))
          / (1 - 0.999)

      - record: slo:error_budget_burn_rate:ratio_rate1h
        expr: |
          (1 - sum(rate(http_requests_total{status!~"5.."}[1h]))
               / sum(rate(http_requests_total[1h])))
          / (1 - 0.999)

      - record: slo:error_budget_burn_rate:ratio_rate2h
        expr: |
          (1 - sum(rate(http_requests_total{status!~"5.."}[2h]))
               / sum(rate(http_requests_total[2h])))
          / (1 - 0.999)

      - record: slo:error_budget_burn_rate:ratio_rate6h
        expr: |
          (1 - sum(rate(http_requests_total{status!~"5.."}[6h]))
               / sum(rate(http_requests_total[6h])))
          / (1 - 0.999)

      - record: slo:error_budget_burn_rate:ratio_rate1d
        expr: |
          (1 - sum(rate(http_requests_total{status!~"5.."}[1d]))
               / sum(rate(http_requests_total[1d])))
          / (1 - 0.999)

      - record: slo:error_budget_burn_rate:ratio_rate3d
        expr: |
          (1 - sum(rate(http_requests_total{status!~"5.."}[3d]))
               / sum(rate(http_requests_total[3d])))
          / (1 - 0.999)

      # Budget remaining — the dashboard and gate metric
      - record: slo:error_budget_remaining:ratio
        expr: |
          1 - (
            (1 - sli:http_request_success:ratio_rate5m)
            / (1 - 0.999)
          )

  - name: slo.burnrate.alerts
    rules:

      # ── TIER 1: PAGE IMMEDIATELY ─────────────────────────────────────────
      # 14× burn, 1h long / 5m short AND-gate
      # Budget exhausts in ~2 days if burn continues
      - alert: SLO_BurnRate_P1_14x
        expr: |
          slo:error_budget_burn_rate:ratio_rate1h  > 14
          AND
          slo:error_budget_burn_rate:ratio_rate5m  > 14
        for: 2m
        labels:
          severity: critical
          tier: "1"
          burn_multiplier: "14"
        annotations:
          summary: >
            SLO burn rate at 14× — budget exhausts in ~2 days if sustained.
            Budget remaining: {{ with query "slo:error_budget_remaining:ratio" }}
            {{ . | first | value | humanizePercentage }}{{ end }}
          runbook: "https://wiki.internal/sre/runbooks/slo-burn-p1"
          dashboard: "https://grafana.internal/d/slo-burn/burn-rate"

      # ── TIER 2: PAGE WITHIN 30 MINUTES ───────────────────────────────────
      # 6× burn, 6h long / 30m short AND-gate
      # Budget exhausts in ~4.7 days if burn continues
      - alert: SLO_BurnRate_P2_6x
        expr: |
          slo:error_budget_burn_rate:ratio_rate6h   > 6
          AND
          slo:error_budget_burn_rate:ratio_rate30m  > 6
        for: 5m
        labels:
          severity: page
          tier: "2"
          burn_multiplier: "6"
        annotations:
          summary: >
            SLO burn rate at 6× — budget exhausts in ~4.7 days if sustained.
          runbook: "https://wiki.internal/sre/runbooks/slo-burn-p2"

      # ── TIER 3: TICKET (BUSINESS HOURS) ──────────────────────────────────
      # 3× burn, 1d long / 2h short AND-gate
      # Budget exhausts in ~9.3 days if burn continues
      - alert: SLO_BurnRate_Ticket_3x
        expr: |
          slo:error_budget_burn_rate:ratio_rate1d  > 3
          AND
          slo:error_budget_burn_rate:ratio_rate2h  > 3
        for: 15m
        labels:
          severity: warning
          tier: "3"
          burn_multiplier: "3"
        annotations:
          summary: >
            SLO burn rate at 3× — budget exhausts in ~9.3 days if sustained.
            Review in next engineering standup.

      # ── TIER 4: WEEKLY TREND REVIEW ──────────────────────────────────────
      # 1× burn, 3d long / 6h short AND-gate
      # Budget on pace to exhaust at window end
      - alert: SLO_BurnRate_Trend_1x
        expr: |
          slo:error_budget_burn_rate:ratio_rate3d  > 1
          AND
          slo:error_budget_burn_rate:ratio_rate6h  > 1
        for: 1h
        labels:
          severity: info
          tier: "4"
          burn_multiplier: "1"
        annotations:
          summary: >
            SLO burn on pace to exhaust budget at 28-day window end.
            Review budget trend in weekly SRE sync.

Istio STRICT mTLS: The SLI Source Matters

In environments running Istio in STRICT mTLS mode, the choice of SLI source — application metrics versus Envoy proxy metrics — determines whether the burn rate calculation captures all failure modes or only the ones the application can see.

# Istio-aware burn rate: captures mTLS-layer rejections
# that application-level metrics cannot see

- record: sli:http_request_success:ratio_rate5m
  expr: |
    sum(rate(istio_requests_total{
      reporter="destination",
      response_code!~"5.."
    }[5m]))
    /
    sum(rate(istio_requests_total{
      reporter="destination"
    }[5m]))

# Why reporter="destination" (not "source"):
#   source = measured at the caller's sidecar
#   destination = measured at the called service's sidecar
#
#   mTLS handshake failures at the destination sidecar:
#     → Appear as errors in destination metrics
#     → Do NOT appear in source metrics (connection never established)
#     → Do NOT appear in application metrics (request never reached app)
#
#   Using reporter="destination" captures:
#     ✓ Application-level 5xx errors
#     ✓ mTLS policy rejection errors
#     ✓ Sidecar proxy errors
#
#   Using reporter="source" OR application metrics misses:
#     ✗ mTLS-layer rejections (certificates expired, policy violation)
#     ✗ These create phantom budget consumption invisible to burn rate

Common Antipatterns

The Single-Window Implementation antipattern → Implementing burn rate alerting with only the long window, skipping the short-window AND-gate. This produces alerts that fire on historical burn events long after the error rate has recovered — operators respond to an ongoing emergency that resolved thirty minutes ago. The short window is not optional; it is the recency validation that prevents this failure mode.
The Threshold Without Derivation antipattern → Using the 14×/6×/3×/1× thresholds without verifying they are calibrated to your actual SLO window. The thresholds in this post are derived for a 28-day window. For a 7-day window, a 14× burn rate exhausts the budget in 12 hours — which changes the appropriate detection window and the response urgency. Derive your thresholds from your window.
The Missing Tier 4 antipattern → Implementing only the page tiers (Tier 1 and 2) and skipping the ticket and trend tiers. Tier 4 is the early warning system for budget degradation that has not yet reached urgent levels. A service that runs at sustained 1× burn for two weeks will exhaust its budget at the 28-day window end — but will never have triggered a page. Without Tier 4, this pattern is invisible until the budget is gone.
The App-Metric SLI in Istio antipattern → Computing burn rates from application-level HTTP response codes in an Istio service mesh environment. As described above, mTLS-layer failures are invisible to application metrics. In STRICT mTLS mode, a certificate rotation gone wrong or a PeerAuthentication policy misconfiguration will consume error budget without generating any application-level error signals. Use Envoy proxy metrics.
The for Duration Misconfiguration antipattern → Setting the for duration on Tier 1 alerts to more than 2 minutes. The for field adds a minimum firing duration — it requires the condition to be true for the specified time before the alert fires. For Tier 1 at 14× burn rate, every minute of for duration adds burn at 14× before the page is sent. A for: 15m on a Tier 1 alert delays notification by 15 minutes at a rate that consumes significant budget.

Maturity Progression

────────────────────────────────────────────────────────────────────────────
STAGE        ALERTING MATURITY STATE             NORTH STAR SIGNAL
────────────────────────────────────────────────────────────────────────────
Reactive     Threshold alerts on individual      Alert volume is high.
             metrics. No SLO concept.            Engineers filter noise
             No burn rate concept.               by experience. P95
                                                 latency alerts fire
                                                 on every deploy.

Defined      SLOs defined. Single-window         First burn rate alert
             burn rate alerts implemented        implemented. Some noise
             for Tier 1 only. App-level          reduction vs. threshold
             SLI source.                         alerting.

Measured     Dual-window AND-gate for all        Tier 2 and 3 alerts
             four tiers. Istio proxy SLI.        implemented. Alert
             Budget remaining recorded.          volume tracking.
             False positive rate tracked.        False positive rate
                                                 measured and declining.

Optimised    All tiers calibrated to actual      Zero false positives
             SLO window. Short window            on Tier 1 in last
             tuned for service traffic           30 days. Tier 4 trend
             characteristics. Multi-service      alerts surfacing budget
             burn rate dashboard.                degradation proactively.

Generative   Burn rate alerting extended         Burn rate model used
             to infrastructure layer:            to govern deployment
             database, cache, queue.             gates. Budget policy
             Alert quality metrics in            tiers automated from
             SRE quarterly report.               burn rate state.
────────────────────────────────────────────────────────────────────────────

Five Action Items for This Week

Verify your Tier 1 alert threshold is calibrated to your actual SLO window. Take your SLO window in hours, divide by 14. That is the time-to-exhaustion at your Tier 1 threshold. If your SLO window is 7 days (168 hours), your 14× burn rate exhausts the budget in 12 hours — which changes whether a 1-hour long window is the right detection window.
Check that your current alerting implementation uses the AND-gate, not an OR-gate or single window. Pull your alert rules and verify that both the long-window and short-window conditions are required simultaneously. A single-window rule or an OR-gate rule does not have the noise suppression properties derived in this post.
Switch your SLI source from application metrics to Istio Envoy proxy metrics if you are running a service mesh. Query istio_requests_total{reporter="destination"} versus your application-level HTTP response metrics for the same service over the same window. If the numbers differ, the delta is budget consumption that your current alerting cannot see.
Add the budget remaining recording rule and verify it is accessible from your deployment gate. The budget remaining metric is the input to the error budget policy tier classification. If your Argo CD PreSync hook cannot query it, the deployment gate has no connection to the SLO state it is supposed to enforce.
Measure your Tier 1 false positive rate over the last 30 days. Count the number of Tier 1 alerts that fired and then resolved without human intervention within 30 minutes. Each of those is a false positive. If the count is above zero, examine whether the AND-gate is correctly configured or whether the short-window threshold is too sensitive for your service's normal traffic variance.

"The burn rate model is not a set of magic numbers. It is a logical structure derived from the properties you want your alerting system to have: page at high urgency, not on transient spikes; ticket at moderate urgency, reliably; surface trends before they become emergencies. Understanding the derivation — rather than just copying the thresholds — is what enables you to adapt the model to services that are unusual: high-cardinality SLIs, multi-modal traffic patterns, services with known daily error rate variance. The AND-gate is not Google's magic. It is the logical solution to a well-specified alerting design problem."

DEV Community