AWS & SRE Field Manual (Part 9): SRE Framework — SLI, SLO, SLA & Error Budget Engineering

Enes Guler · 2026-09-13T13:57:03Z

1. TL;DR & Problem Statement Definition: A foundational Site Reliability Engineering (SRE) framework that defines, measures, tracks, and legally/operationally guarantees system reliability across three distinct abstraction layers. Problem Solved: Eliminates ambiguity around the question "Is the system healthy and stable?" by aligning engineering velocity (shipping features fast) with product stability (keeping systems up) through shared, objective mathematical agreements. Category: Site Reliability Engineering / Observability & Telemetry 2. Core Architecture & Key Components SLI (Service Level Indicator) ---> "What is the actual measurement right now?" (Prometheus Metric) │ ▼ SLO (Service Level Objective) ---> "What internal target must engineering hit?" (Team Goal / Error Budget) │ ▼ SLA (Service Level Agreement) ---> "What formal commitment do we make to customers?" (Legal / Financial Contract) Enter fullscreen mode Exit fullscreen mode 2.1. SLI (Service Level Indicator) - Empirical & Raw Telemetry The actual, quantifiable metric representing real-time system performance and health emitted by telemetry platforms (Prometheus, CloudWatch, Datadog). Availability / Success Rate: Ratio of successful requests to total requests (e.g., non-5xx responses vs. total requests). Latency: Execution duration of incoming transactions (e.g., p95, p99 percentiles). Throughput / Saturation: Ingested requests per second (RPS) and underlying resource exhaustion. Example: "99.2% of HTTP requests returned 2xx/3xx over the last 5 minutes" or "Database p95 read latency is 120ms." 2.2. SLO (Service Level Objective) - Internal Engineering Target The precise reliability target agreed upon internally by engineering, SRE, and product management. The Golden Rule: The SLO must always be stricter than the SLA. This margin forms an operational safety buffer, allowing internal teams to remediate degradation before incurring external legal or financial penalties. Error Budget: An SLO directly calculates an allowable margin of failure: $$\text{Error Budget} = 100\% - \text{SLO}$$ For a 99.9% monthly availability SLO, the maximum allowable downtime is approximately 43.8 minutes per 30-day window. Example: "99.9% of API requests must complete successfully in under 200ms across any rolling 30-day window." 2.3. SLA (Service Level Agreement) - External Customer Contract The legally binding agreement established between the service provider and paying end users. Explicitly defines financial consequences, service credit refunds, and contractual termination remedies when breached. Audited directly by executive leadership and legal departments. Example: "If monthly availability falls below 99.0%, the customer receives a 20% service credit refund on their subsequent billing invoice." 3. Deep Dive Engineering & Operational Matrix Layer Primary Audience Metric Origin / Source Impact of a Breach SLI SRE & On-Call Engineers Telemetry & APM (Prometheus, OpenTelemetry) Real-time dashboard spikes or threshold alert triggers SLO Engineering & Product Teams Aggregate historical SLI evaluation window Feature releases frozen; team pivots to reliability sprint SLA Customers, Legal & Executives Contractual business compliance audits Service credit payouts, financial penalties, contract breach 4. Advanced SRE Mechanisms & Reliability Patterns Multi-Window Multi-Burn-Rate Alerting (Google SRE Standard) Static threshold alerting (e.g., "Alert if error rate > 1% over 5 minutes") creates false-positive alert fatigue during minor spikes and completely misses slow budget erosion. Burn Rate: The consumption velocity of an error budget relative to its standard exhaustion timeframe: 1x Burn Rate: Consumes 100% of the budget in exactly 30 days (normal baseline rate). 14.4x Burn Rate: Consumes 2% of the entire 30-day error budget in just 1 hour (requires immediate incident triage). Production Alerting Logic: Uses intersecting multi-window verification: Short Window (14.4x / 2-minute to 1-hour window): Pages on-call engineers immediately (PagerDuty/Opsgenie). Long Window (6x / 6-hour window): Automatically generates a non-paging ticket for standard working hours investigation. Composite SLIs & User Journey Mapping Monitoring single metrics (e.g., isolated CPU utilization or individual database query speed) fails to reflect true user pain. Combine discrete multi-step transactions (e.g., User Authentication → Add to Cart → Payment Confirmation) into weighted Composite SLIs verified by Synthetic Canaries and Real User Monitoring (RUM). The Over-Reliability Paradox & Chaos Injection When an underlying service consistently runs far above its target SLO (e.g., delivering 100% uptime against a 99.9% SLO target), downstream dependent systems build false assumptions of perfection and omit necessary retries, timeouts, and circuit breakers. Teams use controlled chaos experiments (Chaos Engineering / Chaos Mesh) to deliberately burn excess error budgets, exposing hidden architectural fragility. 5. Practical Notes & Configuration Snippets PromQL: Calculating 5-Minute Rolling HTTP Availability SLI ( sum(rate(http_requests_total{status!~"5.."}[5m])) / sum(rate(http_requests_total[5m])) ) * 100 Enter fullscreen mode Exit fullscreen mode OpenSLO Specification Manifest (SLO as Code) apiVersion : openslo/v1 kind : SLO metadata : name : checkout-service-availability spec : service : checkout-service description : " 99.9% availability for checkout operations over a rolling 30-day window" budgetingMethod : Occurrences objectives : - target : 0.999 timeWindow : - duration : 30d isRolling : true indicator : metadata : name : checkout-availability-sli spec : ratioMetric : good : metricSource : type : prometheus query : sum(rate(http_requests_total{service="checkout", status!~"5.."}[5m])) total : metricSource : type : prometheus query : sum(rate(http_requests_total{service="checkout"}[5m])) Enter fullscreen mode Exit fullscreen mode 6. Gotchas & Common Pitfalls 100% Reliability Is an Anti-Pattern: Aiming for 100% availability is economically unsustainable. Because user edge networks (cellular, home Wi-Fi) have availability bounds below 99.9%, spending exponential engineering and cloud infrastructure budgets chasing the final 0.01% yields zero perceived user value. Measurement Vantage Point Fallacy: Capturing SLIs exclusively inside deep backend container pods blinds the observability stack to edge networking failures, TLS handshake timeouts, and Ingress routing misconfigurations. Measure SLIs at the outermost ingress load balancer or API Gateway level. 7. Production Best Practices Codified Error Budget Policy: Document clear operational rules for budget depletion. When an error budget reaches 0%, standard feature deployments to production are automatically blocked by the CI/CD pipeline, pivoting team capacity entirely to technical debt remediation, stability fixes, and chaos testing. Eliminate Non-Actionable Alerting: Ensure production alerts are strictly actionable. If an alert does not require an immediate, human-driven operational intervention, it must not wake up on-call engineers. Adopt SLO as Code (Sloth / Pyrra): Keep SLO definitions checked into Git repositories using OpenSLO or Sloth. Use CI/CD pipelines to automatically generate Prometheus alert rules and Grafana dashboards directly from declarative YAML files.

#sre #devops #cloud #monitoring

1. TL;DR & Problem Statement

Definition: A foundational Site Reliability Engineering (SRE) framework that defines, measures, tracks, and legally/operationally guarantees system reliability across three distinct abstraction layers.
Problem Solved: Eliminates ambiguity around the question "Is the system healthy and stable?" by aligning engineering velocity (shipping features fast) with product stability (keeping systems up) through shared, objective mathematical agreements.
Category: Site Reliability Engineering / Observability & Telemetry

2. Core Architecture & Key Components

SLI (Service Level Indicator)   ---> "What is the actual measurement right now?" (Prometheus Metric)
        │
        ▼
SLO (Service Level Objective)   ---> "What internal target must engineering hit?" (Team Goal / Error Budget)
        │
        ▼
SLA (Service Level Agreement)   ---> "What formal commitment do we make to customers?" (Legal / Financial Contract)

2.1. SLI (Service Level Indicator) - Empirical & Raw Telemetry

The actual, quantifiable metric representing real-time system performance and health emitted by telemetry platforms (Prometheus, CloudWatch, Datadog).

Availability / Success Rate: Ratio of successful requests to total requests (e.g., non-5xx responses vs. total requests).
Latency: Execution duration of incoming transactions (e.g., p95, p99 percentiles).
Throughput / Saturation: Ingested requests per second (RPS) and underlying resource exhaustion.
Example: "99.2% of HTTP requests returned 2xx/3xx over the last 5 minutes" or "Database p95 read latency is 120ms."

2.2. SLO (Service Level Objective) - Internal Engineering Target

The precise reliability target agreed upon internally by engineering, SRE, and product management.

The Golden Rule: The SLO must always be stricter than the SLA. This margin forms an operational safety buffer, allowing internal teams to remediate degradation before incurring external legal or financial penalties.
Error Budget: An SLO directly calculates an allowable margin of failure: $$\text{Error Budget} = 100\% - \text{SLO}$$ For a 99.9% monthly availability SLO, the maximum allowable downtime is approximately 43.8 minutes per 30-day window.
Example: "99.9% of API requests must complete successfully in under 200ms across any rolling 30-day window."

2.3. SLA (Service Level Agreement) - External Customer Contract

The legally binding agreement established between the service provider and paying end users.

Explicitly defines financial consequences, service credit refunds, and contractual termination remedies when breached. Audited directly by executive leadership and legal departments.
Example: "If monthly availability falls below 99.0%, the customer receives a 20% service credit refund on their subsequent billing invoice."

3. Deep Dive Engineering & Operational Matrix

Layer	Primary Audience	Metric Origin / Source	Impact of a Breach
SLI	SRE & On-Call Engineers	Telemetry & APM (Prometheus, OpenTelemetry)	Real-time dashboard spikes or threshold alert triggers
SLO	Engineering & Product Teams	Aggregate historical SLI evaluation window	Feature releases frozen; team pivots to reliability sprint
SLA	Customers, Legal & Executives	Contractual business compliance audits	Service credit payouts, financial penalties, contract breach

4. Advanced SRE Mechanisms & Reliability Patterns

Multi-Window Multi-Burn-Rate Alerting (Google SRE Standard)

Static threshold alerting (e.g., "Alert if error rate > 1% over 5 minutes") creates false-positive alert fatigue during minor spikes and completely misses slow budget erosion.

Burn Rate: The consumption velocity of an error budget relative to its standard exhaustion timeframe:
- 1x Burn Rate: Consumes 100% of the budget in exactly 30 days (normal baseline rate).
- 14.4x Burn Rate: Consumes 2% of the entire 30-day error budget in just 1 hour (requires immediate incident triage).
Production Alerting Logic: Uses intersecting multi-window verification:
- Short Window (14.4x / 2-minute to 1-hour window): Pages on-call engineers immediately (PagerDuty/Opsgenie).
- Long Window (6x / 6-hour window): Automatically generates a non-paging ticket for standard working hours investigation.

Composite SLIs & User Journey Mapping

Monitoring single metrics (e.g., isolated CPU utilization or individual database query speed) fails to reflect true user pain. Combine discrete multi-step transactions (e.g., User Authentication → Add to Cart → Payment Confirmation) into weighted Composite SLIs verified by Synthetic Canaries and Real User Monitoring (RUM).

The Over-Reliability Paradox & Chaos Injection

When an underlying service consistently runs far above its target SLO (e.g., delivering 100% uptime against a 99.9% SLO target), downstream dependent systems build false assumptions of perfection and omit necessary retries, timeouts, and circuit breakers. Teams use controlled chaos experiments (Chaos Engineering / Chaos Mesh) to deliberately burn excess error budgets, exposing hidden architectural fragility.

5. Practical Notes & Configuration Snippets

PromQL: Calculating 5-Minute Rolling HTTP Availability SLI

(
  sum(rate(http_requests_total{status!~"5.."}[5m]))
  /
  sum(rate(http_requests_total[5m]))
) * 100

OpenSLO Specification Manifest (SLO as Code)

apiVersion: openslo/v1
kind: SLO
metadata:
  name: checkout-service-availability
spec:
  service: checkout-service
  description: "99.9% availability for checkout operations over a rolling 30-day window"
  budgetingMethod: Occurrences
  objectives:
    - target: 0.999
      timeWindow:
        - duration: 30d
          isRolling: true
  indicator:
    metadata:
      name: checkout-availability-sli
    spec:
      ratioMetric:
        good:
          metricSource:
            type: prometheus
            query: sum(rate(http_requests_total{service="checkout", status!~"5.."}[5m]))
        total:
          metricSource:
            type: prometheus
            query: sum(rate(http_requests_total{service="checkout"}[5m]))

6. Gotchas & Common Pitfalls

100% Reliability Is an Anti-Pattern: Aiming for 100% availability is economically unsustainable. Because user edge networks (cellular, home Wi-Fi) have availability bounds below 99.9%, spending exponential engineering and cloud infrastructure budgets chasing the final 0.01% yields zero perceived user value.
Measurement Vantage Point Fallacy: Capturing SLIs exclusively inside deep backend container pods blinds the observability stack to edge networking failures, TLS handshake timeouts, and Ingress routing misconfigurations. Measure SLIs at the outermost ingress load balancer or API Gateway level.

7. Production Best Practices

Codified Error Budget Policy: Document clear operational rules for budget depletion. When an error budget reaches 0%, standard feature deployments to production are automatically blocked by the CI/CD pipeline, pivoting team capacity entirely to technical debt remediation, stability fixes, and chaos testing.
Eliminate Non-Actionable Alerting: Ensure production alerts are strictly actionable. If an alert does not require an immediate, human-driven operational intervention, it must not wake up on-call engineers.
Adopt SLO as Code (Sloth / Pyrra): Keep SLO definitions checked into Git repositories using OpenSLO or Sloth. Use CI/CD pipelines to automatically generate Prometheus alert rules and Grafana dashboards directly from declarative YAML files.

Top comments (1)

Some comments may only be visible to logged-in visitors. Sign in to view all comments.