DEV Community: Nijo George Payyappilly

Replication Study: Do DORA Metrics Generalize to On-Premises Enterprise Environments?

Nijo George Payyappilly — Mon, 19 Oct 2026 16:00:00 +0000

Replication is science's self-correcting mechanism. A finding that cannot be replicated in different contexts is a finding about those specific contexts, not a general principle. The DORA Four Key Metrics have been replicated extensively in technology-company and cloud-native contexts — the research programme has surveyed tens of thousands of practitioners across multiple years and consistently found the same performance clusters and the same correlations with organisational outcomes.

What has not been systematically attempted is replication in on-premises enterprise environments with regulated change management processes — the environments where the questions about generalisability are most consequential, because these are the environments where national critical infrastructure is operated.

This post reports the results of applying DORA measurement methodology to a single on-premises, regulated enterprise environment running VMware vSphere with Tanzu Kubernetes Grid Service, GitOps via Argo CD, and a traditional Change Advisory Board process. It is a single case study, not a population-level study, and its findings should be interpreted accordingly. But a single carefully documented case study is more evidence than the field currently has for this environment type, and the deviations it identifies from the canonical DORA findings are directionally informative for practitioners and researchers working in similar environments.

Environment Description

────────────────────────────────────────────────────────────────────────────
STUDY ENVIRONMENT CHARACTERISTICS

Infrastructure:
  Compute: VMware vSphere 8.0 + Tanzu Kubernetes Grid Service (TKGs)
  GitOps: Argo CD with ApplicationSet, self-heal enabled
  Service Mesh: Istio 1.20 in STRICT mTLS mode
  Policy: Kyverno admission controller
  Observability: Prometheus + Splunk Enterprise
  CI/CD: Azure DevOps (ADO) pipelines
  Change Management: Traditional CAB with weekly review cycles

Organisational characteristics:
  Industry: Regulated enterprise (financial services adjacent)
  Org size: Large enterprise (>10,000 employees)
  IT headcount: ~200 engineers across development and operations
  SRE programme stage: Phase 1 (Measurement and Accountability)
  CAB change freeze windows: Yes (quarterly + year-end)
  On-call structure: Shared SRE + development team rotations

Measurement period: 12 months (January–December 2024)
Services measured: 23 production services across 4 business domains
Data sources:
  → ADO pipeline logs (deployment timestamps, build outcomes)
  → PagerDuty incident data (incident creation/resolution timestamps)
  → Argo CD audit log (sync events, drift events)
  → Splunk incident classification data

────────────────────────────────────────────────────────────────────────────

Metric 1: Deployment Frequency

Raw measurement: The 23 services averaged 2.3 deployments per service per week across the full 12-month period, which places the environment in the DORA Medium performer cohort (between "once per week" and "once per month" in the DORA framework definitions).

The RE-adjusted measurement: When deployment frequency is normalised to available deployment windows (accounting for CAB review cycles and change freeze windows), the picture changes materially.

────────────────────────────────────────────────────────────────────────────
DEPLOYMENT FREQUENCY: RAW VS RE-ADJUSTED

Available deployment windows (12 months):
  Total weeks: 52
  Change freeze weeks: 8 (Q4 year-end: 4 weeks; quarterly: 1 week × 4)
  Available deployment windows: 44 weeks

Raw deployment frequency:
  Total deployments across 23 services: 2,756
  Per service per week (all 52 weeks): 2.3 deployments/week
  DORA cohort: Medium (Low–Medium boundary)

RE-adjusted deployment frequency:
  Per service per available week (44 weeks): 2.72 deployments/week
  Estimated window utilisation: 97% of available windows had at least
    one deployment across the service portfolio
  DORA cohort (RE-adjusted): High (approaching Elite boundary)

INTERPRETATION:
  The raw measurement classifies this environment as Medium performer.
  The RE-adjusted measurement classifies it as High performer.
  The difference is the 8 change freeze weeks — externally mandated
  periods where deployment was structurally prohibited regardless of
  engineering capability.

  A regulatory examination using raw DORA deployment frequency would
  classify this environment as Medium. The RE-adjusted metric more
  accurately reflects the organisation's engineering capability —
  the capacity to deploy is High; the regulatory permission to deploy
  is constrained.

  FINDING 1: Raw deployment frequency systematically underestimates
  engineering capability in environments with mandatory change freeze
  windows. The RE-adjusted metric (window utilisation) is a more
  accurate capability measure for regulated enterprise environments.
────────────────────────────────────────────────────────────────────────────

Metric 2: Lead Time for Changes

Raw measurement: The average lead time from code commit to production deployment was 4.1 calendar days, placing the environment in the DORA Low performer cohort ("between 1 month and 6 months" in the raw DORA framework — but wait, 4.1 days is not in that range).

Actually, 4.1 calendar days places this environment in the High performer cohort (DORA High = "between 1 day and 1 week"). But this conceals the decomposition.

────────────────────────────────────────────────────────────────────────────
LEAD TIME DECOMPOSITION

Total lead time: 4.1 days average

Component breakdown:
  → Technical lead time (commit → deployable artefact): 3.8 hours
    (ADO pipeline: build, test, security scan, image push)
    DORA cohort for technical component: ELITE (< 1 hour? No, 3.8h)
    DORA cohort for technical component: HIGH (1 day – 1 week? No)
    Actually 3.8 hours = between 1 hour and 1 day → HIGH

  → Process lead time (artefact → CAB approval → deployment): 3.3 days
    (CAB meeting cadence: weekly; average wait for next CAB: 2.1 days;
    CAB review and approval: 0.9 days average; deployment execution: 0.3 days)
    This component alone: LOW performer (approaching Medium)

  → Regulatory notification lead time (for changes to regulated systems): +2.1 days
    (subset of services: 6 of 23 require additional regulatory review)
    For these services, total lead time: 6.2 days average

Technical vs process split: 8% technical, 92% process
For regulated-system services: 5% technical, 95% process

FINDING 2: Lead time in this environment is dominated by process overhead,
not technical overhead. CI/CD optimisation (which would reduce the 8%
technical component) has negligible impact on total lead time.
CAB process optimisation (risk-based change categorisation, automated
pre-approval for low-risk changes, emergency change pathway) would
address the 92% process component.

FINDING 3: The decomposition reveals an investment misallocation.
This environment has invested significantly in CI/CD pipeline
optimisation; the data shows this investment reduced the 8% technical
component from approximately 6 hours to 3.8 hours. It has not invested
proportionally in CAB process optimisation, which would yield 10×
larger lead time improvement.
────────────────────────────────────────────────────────────────────────────

Metric 3: Change Failure Rate

Raw measurement: 8.4% of deployments required remediation within 24 hours of deployment, placing this environment in the DORA Elite/High cohort (0–15%).

The extended CFR measurement adds the regulated enterprise dimensions identified in the Beyond DORA framework.

────────────────────────────────────────────────────────────────────────────
CHANGE FAILURE RATE: EXTENDED MEASUREMENT

Technical CFR: 8.4%
  (deployments requiring rollback or hotfix within 24 hours)
  DORA cohort: HIGH (0–15% range)

Compliance CFR: 1.2%
  (deployments that, while technically successful, triggered a
  compliance finding, audit gap, or policy deviation detected
  within 30 days of deployment)
  Target: 0%

Configuration drift rate: 3.7%
  (deployments that created configuration drift detected by
  Argo CD self-heal within 72 hours, distinct from technical failures)
  Note: Drift is automatically remediated by Argo CD; this metric
  captures the deployment quality issue that caused drift, not the
  remediation overhead.

Combined extended CFR: 13.3%
  (technical + compliance; drift excluded as it's auto-remediated)

FINDING 4: Technical CFR at 8.4% is in the DORA High cohort, indicating
  mature change management for technical quality. Compliance CFR at 1.2%
  indicates that technical quality and compliance quality are not
  perfectly correlated — services can pass technical quality gates
  while creating compliance gaps that technical testing does not detect.

FINDING 5: Configuration drift rate (3.7%) is a metric the canonical
  DORA framework does not include. In a GitOps-enabled environment,
  drift is automatically detected and remediated; the residual metric
  is a deployment quality signal (deployments that create drift) rather
  than an operational burden signal (drift requiring manual remediation).
  This is a regime-specific metric that may not generalise to non-GitOps
  environments.
────────────────────────────────────────────────────────────────────────────

Metric 4: Mean Time to Restore

Raw measurement: Median MTTR across all P1/P2 incidents was 28 minutes, placing this environment firmly in the DORA Elite cohort (< 1 hour).

The regulated enterprise extension adds the compliance MTTR dimension.

────────────────────────────────────────────────────────────────────────────
MEAN TIME TO RESTORE: EXTENDED MEASUREMENT

Technical MTTR: 28 minutes (median P1/P2 incidents)
  DORA cohort: ELITE (< 1 hour)

  Distribution:
    25th percentile: 11 minutes
    75th percentile: 67 minutes
    95th percentile: 3.1 hours
    99th percentile: 6.8 hours

  The 95th and 99th percentile MTTRs are driven by incidents involving
  stateful workloads (databases, persistent queues) and cross-cluster
  incidents — consistent with the expected complexity increase for
  those incident types.

Regulatory MTTR: 4.8 business days (mean, P1/P2 incidents)
  Definition: Time from incident creation to all regulatory obligations
  closed (root cause documented, regulator notified if required,
  change record updated, postmortem closed with all action items assigned)

  Components:
    → Technical resolution: 28 minutes
    → Postmortem facilitation and publication: 1.8 business days
    → Regulatory notification assessment and filing: 0.6 business days
    → Action item assignment and acknowledgement: 1.2 business days
    → Compliance review and sign-off: 1.2 business days

  FINDING 6: Technical MTTR (28 minutes) and Regulatory MTTR (4.8 days)
  differ by approximately 330×. The DORA framework measures the former;
  the organisational cost of an incident is primarily the latter.
  A CTO who evaluates incident management effectiveness using DORA
  MTTR is seeing 0.3% of the actual incident resolution timeline.

FINDING 7: The 95th percentile technical MTTR (3.1 hours) is still
  DORA Elite, but the 99th percentile (6.8 hours) is not. The high
  performers on MTTR in this environment are the well-characterised
  incident types with documented runbooks and automation. The long-tail
  incidents are the ones that reveal the limits of current automation
  coverage.
────────────────────────────────────────────────────────────────────────────

Finding 8: The On-Premises Adaptation — What Holds and What Doesn't

────────────────────────────────────────────────────────────────────────────
DORA FRAMEWORK GENERALISABILITY ASSESSMENT

WHAT HOLDS IN ON-PREMISES REGULATED ENTERPRISE ENVIRONMENTS:

  ✓ The four metrics are measurable from existing data sources
    (CI/CD logs, incident management data, deployment records)
    No additional instrumentation required beyond what this
    environment already had.

  ✓ The Elite/High/Medium/Low cohort structure is meaningful
    The cohort thresholds produce differentiated classifications
    that are consistent with subjective engineering maturity
    assessments — services that practitioners consider well-managed
    score in High/Elite; services considered poorly managed
    score in Low/Medium.

  ✓ The direction of improvement is consistent with DORA findings
    Higher deployment frequency in this environment is associated
    with lower CFR (r = -0.41), consistent with DORA's finding
    that high-frequency, small-batch deployments have lower failure
    rates than low-frequency, large-batch deployments.

  ✓ The metric correlations hold
    In this environment: High deployment frequency services have
    lower MTTR, lower CFR, and lower Regulatory MTTR. The DORA
    finding that these metrics cluster together holds.

WHAT REQUIRES ADAPTATION IN ON-PREMISES REGULATED ENTERPRISE:

  ✗ Deployment frequency benchmarks require RE-adjustment
    Raw deployment frequency systematically underestimates
    engineering capability in environments with mandatory change
    freeze windows. The window utilisation metric should supplement
    or replace raw deployment frequency in this context.

  ✗ Lead time benchmarks require decomposition
    A total lead time in the High cohort conceals a technical
    component in the Elite cohort and a process component in the
    Medium/Low cohort. The aggregate benchmark obscures the
    information needed for investment decisions.

  ✗ CFR requires extension for compliance quality
    Compliance CFR captures a failure mode (regulatory compliance
    failure from a technically successful deployment) that the
    canonical framework does not address. In regulated environments,
    this is a material gap.

  ✗ MTTR requires extension for regulatory MTTR
    The 330× difference between technical MTTR and regulatory MTTR
    is a structural property of regulated environments, not a
    measurement artefact. A framework that only measures technical
    MTTR provides a misleading picture of incident resolution burden.

  ✗ The canonical Elite benchmarks are not all achievable as stated
    "On-demand, multiple times per day" deployment frequency is
    structurally incompatible with weekly CAB cycles.
    The appropriate benchmark is not the canonical Elite threshold
    but the RE-adjusted equivalent for the governance environment.
────────────────────────────────────────────────────────────────────────────

Implications for the DORA Research Programme

This single-environment case study cannot establish population-level findings. It can, however, identify questions that a properly designed replication study should address:

Does the direction of performance-outcome correlation hold in regulated environments? The DORA finding that Elite performers have better resilience outcomes needs replication in environments where Elite performance (as canonically defined) is not achievable.
Do the RE-adjusted metrics predict outcomes better than raw metrics in regulated environments? The hypothesis is that window utilisation (RE-adjusted deployment frequency) and decomposed lead time are better predictors of regulated enterprise resilience outcomes than raw DORA metrics. This is testable.
What is the effect size of compliance toil on DORA performance? This study suggests compliance toil is a major driver of regulated enterprise underperformance. Quantifying the effect size would inform the policy intervention design.
Does the DORA bimodal distribution hold within regulated industries? The field-level bimodal distribution (Elite/High clustering, Low clustering) may or may not be replicated within regulated industries where structural barriers are more uniform. If the intra-sector distribution is unimodal and low, the barrier is structural, not capability-driven.

Five Action Items for This Week

Measure your own Lead Time decomposition. Calculate what fraction of your total lead time is technical (commit to deployable artefact) versus process (artefact to production). If the process fraction exceeds 80%, CI/CD investment is the wrong optimisation target. CAB process reform or risk-based change categorisation will deliver 5–10× more lead time improvement per investment dollar.
Calculate your Compliance CFR for the last quarter. Review every deployment from the last 90 days. Did any technically successful deployment generate a compliance finding, policy deviation, or audit gap within 30 days? The Compliance CFR number you derive is a regulated-enterprise-specific quality signal that the canonical DORA framework does not capture.
Measure your Regulatory MTTR for your last five P1/P2 incidents. Record the time from incident creation to all compliance obligations closed (postmortem published, regulatory notification assessed, action items assigned). Compare to your Technical MTTR. The ratio between the two is the measure of your regulatory incident burden — and the primary target for compliance toil elimination.
Replicate this analysis in your environment. The methodology is fully documented. Apply it to 12 months of your pipeline and incident data. The differences between your findings and this study's findings are data about how your environment differs — and potentially a contribution to the replication literature if you publish them.
Submit the raw findings (anonymised) to the DORA research programme. The DORA research team has expressed interest in on-premises and regulated enterprise data. A structured submission of your decomposed DORA metrics, with environment type and sector anonymised, contributes to the population-level dataset that would make a formal replication study possible.

"Replication is not agreement. A replication study that finds deviation from the canonical findings is as valuable as one that confirms them — more valuable, if the deviation is systematic and informative. The deviations documented here — the deployment frequency undercount from change freeze windows, the lead time dominated by process overhead, the 330× gap between technical and regulatory MTTR — are not failures of the DORA framework. They are the framework's signal that on-premises regulated enterprise environments have specific structural properties that require specific adaptations. The framework generalises. The benchmarks require calibration."

This post is part of applying Google SRE principles to national-scale critical infrastructure. This study is a single-environment case study; findings should be interpreted as directional, not conclusive. References: DORA Research · Accelerate (Forsgren et al.) · Google SRE Workbook

Evaluating LLM Reliability for Incident Diagnosis: An SRE Approach to Testing AI Systems

Nijo George Payyappilly — Mon, 12 Oct 2026 16:00:00 +0000

In 2024, a practitioner at a financial services firm deployed an LLM-based root cause analysis tool into their incident response workflow. The tool had passed all technical integration tests. It connected to their observability stack correctly, pulled telemetry during incidents, and returned structured diagnoses in the expected format. It had been evaluated against ten historical incidents by two engineers who found the recommendations "generally helpful."

Three months later, a postmortem review revealed that in fourteen of the forty-seven incidents where the tool had been consulted, the tool's confident diagnosis had led the responding engineer away from the actual root cause — delaying resolution by an average of twenty-two minutes per misdirected incident. The tool was not obviously wrong. It was plausibly wrong, in specific ways that a time-pressured on-call engineer was likely to accept rather than challenge. The cost of those fourteen incidents, measured in Regulatory MTTR and error budget consumption, significantly exceeded any benefit the tool provided.

The failure was not the tool's integration. It was the absence of a reliability evaluation framework. The tool was tested for function (does it connect? does it respond?) but not for reliability (is it correct? is it consistently correct? is it calibrated about its own uncertainty?). These are the questions that SRE asks about every production dependency. They were not asked about the AI system until the evidence of systematic misdirection had accumulated in the postmortem record.

This post provides the evaluation framework that should have preceded that deployment.

The LLM as a Production Dependency

An LLM used for incident diagnosis is a production dependency in the same sense that a database, a downstream API, or an authentication service is a production dependency. It has inputs, outputs, latency, error rates, and failure modes. It has a reliability profile that can be measured, a performance envelope that has limits, and a degradation pattern that develops over time as the gap between its training data and the current environment widens.

Standard software testing is insufficient for LLMs because LLM failure modes are fundamentally different from deterministic software failure modes:

────────────────────────────────────────────────────────────────────────────
LLM FAILURE MODES VS DETERMINISTIC SOFTWARE FAILURE MODES

DETERMINISTIC SOFTWARE:
  Failure mode: Returns error code; times out; returns wrong type
  Detection: Standard integration tests catch these
  Characteristic: Failures are generally consistent and reproducible

LLM FAILURE MODES (not detectable by standard integration tests):
  → Confidently wrong: Returns a plausible but incorrect diagnosis
    with high confidence score. No error code. No exception.
    Appears to function correctly.

  → Inconsistently right: Returns correct diagnosis on 70% of
    invocations for the same incident, wrong diagnosis on 30%.
    Passes integration tests that check one invocation.

  → Distribution shift degradation: Was reliable at deployment;
    degrades silently as environment evolves and the gap between
    training data and current incident patterns widens.

  → Context length sensitivity: Correct with minimal context;
    incorrect when context exceeds a threshold (token limit effects,
    attention dilution). Standard integration tests use minimal
    context; production uses full context.

  → Confidence miscalibration: Reports 0.90 confidence when
    empirically accurate 65% of the time. Escalation policies
    that gate on reported confidence are then systematically
    wrong about autonomy thresholds.

────────────────────────────────────────────────────────────────────────────

The reliability evaluation framework must address all five failure modes. Standard integration testing addresses none of them.

The LLM Reliability Test Framework

Component 1: Historical Incident Test Dataset Construction

The foundation of LLM reliability evaluation is a ground-truth dataset of historical incidents where the correct diagnosis has been independently confirmed. This dataset is the benchmark against which all four reliability dimensions (accuracy, calibration, consistency, explainability) are measured.

────────────────────────────────────────────────────────────────────────────
TEST DATASET CONSTRUCTION PROTOCOL

STEP 1: Identify candidate incidents
  Source: PagerDuty or equivalent incident management system
  Filter: P1/P2 incidents only (highest quality postmortems)
  Time window: Last 18 months (sufficient volume; recent enough to be
               representative of current system state)
  Target size: Minimum 50 incidents; 100+ for statistical validity

STEP 2: Confirm root cause ground truth
  Requirement: Root cause must be independently confirmed
  (not just "as stated in the postmortem" — postmortems are not always
  accurate; some attribute to proximate cause rather than root cause)

  Ground truth confirmation sources (in order of reliability):
    a) Engineering postmortem with specific code change identified
    b) Engineering postmortem with specific configuration identified
    c) Peer-reviewed incident analysis (two engineers agree)
    d) Postmortem with remediation that resolved and did not recur

  Exclude: Incidents where root cause remains disputed or "unknown"
  Exclude: Incidents where symptoms match multiple simultaneous causes

STEP 3: Reconstruct the incident context as it appeared at alert time
  This is the most labour-intensive step and the most critical.
  The test must evaluate the LLM on the context that was available
  WHEN THE INCIDENT OCCURRED — not on hindsight context.

  Context to reconstruct:
    → Prometheus metrics in the 30-minute window before first alert
    → Splunk log events in the 30-minute window before first alert
    → Recent deployments (Argo CD sync events, last 24 hours)
    → Recent configuration changes (git commits, last 7 days)
    → Active alerts at time of incident creation

  Context to EXCLUDE (post-hoc information not available at alert time):
    → Logs generated after the incident started (investigation artefacts)
    → Configuration changes made during incident response
    → Any information that assumes knowledge of the outcome

STEP 4: Label the dataset
  For each incident, record:
    → incident_id, timestamp, affected_service
    → confirmed_root_cause (structured: category + specific cause)
    → root_cause_category (deployment, infrastructure, dependency,
                           configuration, capacity, external)
    → severity (P1/P2), duration_minutes, budget_consumed_pct

DATASET QUALITY METRIC:
  Ground truth confidence score: fraction of incidents where
  root cause is confirmed by source type (a) or (b) above.
  Target: > 80% of dataset confirmed at source (a) or (b) level.
  Datasets with high proportions of source (c) or (d) produce
  noisier accuracy measurements.
────────────────────────────────────────────────────────────────────────────

Component 2: Accuracy Testing — Precision and Recall

# LLM Accuracy Evaluation — Automated Test Runner
# Presents reconstructed incident context to LLM and measures
# diagnosis accuracy against ground truth dataset

import json
from dataclasses import dataclass
from typing import Optional

@dataclass
class IncidentTestCase:
    incident_id: str
    context: dict           # Reconstructed at-alert-time context
    ground_truth: dict      # Confirmed root cause

@dataclass
class DiagnosisResult:
    incident_id: str
    llm_diagnosis: str
    confidence_score: float
    model_used: str
    latency_ms: int
    is_correct: Optional[bool] = None    # Set after evaluation

def evaluate_accuracy(
    test_cases: list[IncidentTestCase],
    llm_client,
    evaluator_fn
) -> dict:
    """
    Run accuracy evaluation against historical incident test dataset.
    Returns precision, recall, and per-category breakdown.
    """
    results = []

    for case in test_cases:
        # Present context to LLM (same prompt as production)
        response = llm_client.diagnose(
            context=case.context,
            prompt_template=PRODUCTION_PROMPT_TEMPLATE
        )

        # Evaluate correctness against ground truth
        # (evaluator_fn uses semantic similarity + keyword matching)
        is_correct = evaluator_fn(
            diagnosis=response.diagnosis,
            ground_truth=case.ground_truth,
            threshold=0.80    # 80% semantic similarity = correct
        )

        results.append(DiagnosisResult(
            incident_id=case.incident_id,
            llm_diagnosis=response.diagnosis,
            confidence_score=response.confidence,
            model_used=response.model,
            latency_ms=response.latency_ms,
            is_correct=is_correct
        ))

    # Calculate metrics
    total = len(results)
    correct = sum(1 for r in results if r.is_correct)
    diagnosed = sum(1 for r in results if r.llm_diagnosis is not None)

    precision = correct / diagnosed if diagnosed > 0 else 0
    recall = correct / total    # All incidents have ground truth

    # Per root-cause-category breakdown
    by_category = {}
    for case in test_cases:
        cat = case.ground_truth["root_cause_category"]
        category_results = [r for r in results
                           if r.incident_id == case.incident_id]
        if category_results:
            by_category.setdefault(cat, {"correct": 0, "total": 0})
            by_category[cat]["total"] += 1
            if category_results[0].is_correct:
                by_category[cat]["correct"] += 1

    return {
        "precision": precision,
        "recall": recall,
        "total_incidents": total,
        "by_category": {
            cat: data["correct"] / data["total"]
            for cat, data in by_category.items()
        },
        "gate_1_pass": precision >= 0.70 and recall >= 0.60
    }

Component 3: Calibration Testing

# Calibration Testing — Confidence Score vs Empirical Accuracy
# Measures whether reported confidence correlates with actual accuracy

def evaluate_calibration(results: list[DiagnosisResult]) -> dict:
    """
    Measure confidence calibration: does reported confidence
    match empirical accuracy rate?
    """
    confidence_bands = {
        "90-100": {"min": 0.90, "max": 1.01},
        "85-89":  {"min": 0.85, "max": 0.90},
        "80-84":  {"min": 0.80, "max": 0.85},
        "70-79":  {"min": 0.70, "max": 0.80},
        "60-69":  {"min": 0.60, "max": 0.70},
        "below-60": {"min": 0.0, "max": 0.60}
    }

    calibration = {}
    for band_name, bounds in confidence_bands.items():
        band_results = [r for r in results
                       if bounds["min"] <= r.confidence_score < bounds["max"]]
        if len(band_results) >= 5:    # Minimum sample for validity
            empirical_accuracy = sum(1 for r in band_results if r.is_correct) / len(band_results)
            mean_confidence = sum(r.confidence_score for r in band_results) / len(band_results)
            calibration_delta = empirical_accuracy - mean_confidence

            calibration[band_name] = {
                "sample_size": len(band_results),
                "empirical_accuracy": empirical_accuracy,
                "mean_stated_confidence": mean_confidence,
                "calibration_delta": calibration_delta,
                "status": "CALIBRATED" if abs(calibration_delta) <= 0.10 else "MISCALIBRATED"
            }

    # Gate 2 pass: all bands with sufficient samples are calibrated
    gate_pass = all(
        band["status"] == "CALIBRATED"
        for band in calibration.values()
        if band["sample_size"] >= 10
    )

    return {"calibration_by_band": calibration, "gate_2_pass": gate_pass}

Component 4: Consistency Testing

# Consistency Testing — Same input, same output?
# Critical for autonomous execution: inconsistent diagnoses cannot
# be trusted for policy-governed automatic remediation

def evaluate_consistency(
    test_cases: list[IncidentTestCase],
    llm_client,
    n_reruns: int = 3
) -> dict:
    """
    Test consistency: present the same incident context N times
    and measure semantic agreement across responses.
    """
    consistency_scores = []

    for case in test_cases[:30]:    # Sample: 30 incidents × 3 reruns = 90 calls
        responses = []
        for _ in range(n_reruns):
            response = llm_client.diagnose(
                context=case.context,
                prompt_template=PRODUCTION_PROMPT_TEMPLATE,
                temperature=0.1    # Low temperature for consistency testing
            )
            responses.append(response.diagnosis)

        # Pairwise semantic similarity across all response pairs
        pairs = [(responses[i], responses[j])
                 for i in range(len(responses))
                 for j in range(i+1, len(responses))]

        pairwise_similarities = [
            semantic_similarity(a, b) for a, b in pairs
        ]
        incident_consistency = sum(pairwise_similarities) / len(pairwise_similarities)
        consistency_scores.append(incident_consistency)

    mean_consistency = sum(consistency_scores) / len(consistency_scores)

    return {
        "mean_consistency_score": mean_consistency,
        "incidents_tested": len(consistency_scores),
        "gate_3_pass": mean_consistency >= 0.85
    }

Component 5: Context Length Stress Testing

This component is unique to LLM reliability evaluation and has no equivalent in standard software testing.

# Context Length Stress Test
# Tests whether diagnosis accuracy degrades with increasing context size
# Production incidents often have larger context than lab tests

def evaluate_context_length_sensitivity(
    test_cases: list[IncidentTestCase],
    llm_client
) -> dict:
    """
    Test diagnosis accuracy at multiple context sizes.
    Reveals context length threshold where accuracy degrades.
    """
    context_sizes = ["minimal", "standard", "extended", "maximum"]
    results_by_size = {}

    for size_label in context_sizes:
        size_results = []
        for case in test_cases[:20]:    # 20 incidents per size
            # Build context at specified size
            context = build_context_at_size(case.context, size_label)
            response = llm_client.diagnose(context=context)
            is_correct = evaluate_diagnosis(response.diagnosis, case.ground_truth)
            size_results.append({
                "incident_id": case.incident_id,
                "context_tokens": len(context) // 4,    # approximate
                "is_correct": is_correct,
                "confidence": response.confidence,
                "latency_ms": response.latency_ms
            })

        accuracy = sum(1 for r in size_results if r["is_correct"]) / len(size_results)
        avg_latency = sum(r["latency_ms"] for r in size_results) / len(size_results)

        results_by_size[size_label] = {
            "accuracy": accuracy,
            "avg_latency_ms": avg_latency,
            "avg_tokens": sum(r["context_tokens"] for r in size_results) / len(size_results)
        }

    # Identify degradation threshold
    accuracies = [results_by_size[s]["accuracy"] for s in context_sizes]
    degradation_threshold = None
    for i in range(1, len(accuracies)):
        if accuracies[i] < accuracies[i-1] * 0.85:    # >15% accuracy drop
            degradation_threshold = context_sizes[i]
            break

    return {
        "by_context_size": results_by_size,
        "degradation_threshold": degradation_threshold,
        "gate_4_pass": degradation_threshold != "standard"
        # Fail if accuracy degrades at standard context size (production representative)
    }

The Five PoC Exit Gates: Complete Specification

────────────────────────────────────────────────────────────────────────────
LLM INCIDENT DIAGNOSIS PoC EXIT GATES

GATE 1 — ACCURACY (Technical Performance)
  Pass criteria:
    Precision ≥ 0.70 (correct root cause in ≥ 70% of diagnoses made)
    Recall ≥ 0.60 (identifies root cause in ≥ 60% of all test incidents)
    Deployment-caused incidents: precision ≥ 0.80 (highest-volume category)
  Measurement: Automated accuracy evaluation on 50+ test cases
  Sample required: ≥ 50 P1/P2 incidents with confirmed ground truth
  Failure response: Review prompt engineering; assess context completeness;
                    evaluate model selection

GATE 2 — CALIBRATION
  Pass criteria:
    Calibration delta within ±0.10 for all confidence bands ≥ 0.60
    No confidence band shows systematic overconfidence > 0.15
  Measurement: Calibration evaluation on same 50+ test cases
  Failure response: Adjust confidence thresholds in escalation policy;
                    if systematic overconfidence: apply calibration
                    correction factor before gating

GATE 3 — CONSISTENCY
  Pass criteria:
    Mean semantic consistency ≥ 0.85 across 30 test incidents × 3 reruns
    No individual incident shows consistency < 0.60 (outlier cap)
  Measurement: Rerun test suite with same 30 incidents × 3 times
  Failure response: Reduce temperature; review prompt for ambiguity;
                    evaluate model quantisation effects

GATE 4 — CONTEXT LENGTH STABILITY
  Pass criteria:
    Accuracy does not degrade by > 15% at standard context size
    (representative of typical production incident context)
    Latency at maximum context: < 120 seconds (Ollama), < 60 seconds (external)
  Measurement: Context size stress test at 4 levels
  Failure response: Reduce context assembly scope; prioritise high-signal
                    context (recent deployments, correlated alerts) over
                    full log ingestion

GATE 5 — ENGINEER TRUST SURVEY
  Pass criteria:
    SRE team mean score ≥ 3.5/5.0 across five dimensions
    No individual respondent < 2.0 (veto threshold)
    Completion rate: 100% of on-call SRE engineers surveyed
  Measurement: Structured survey after 30-day shadow mode
  Survey dimensions:
    Q1: Recommendations are relevant to the actual incident (1–5)
    Q2: I can verify a recommendation in under 5 minutes (1–5)
    Q3: The tool changes how I approach investigation (1–5)
    Q4: I would trust this recommendation enough to act on it (1–5)
    Q5: The explanations help me understand the reasoning (1–5)
  Failure response: This gate cannot be fixed with technical changes.
                    Failure means the tool is not ready for the team,
                    regardless of technical performance. Return to
                    shadow mode; understand specific objections;
                    address them before re-surveying.

────────────────────────────────────────────────────────────────────────────
GATE SEQUENCE AND DEPENDENCIES:
  Gates 1–4 can be evaluated in parallel (all from test dataset)
  Gate 5 requires Gates 1–4 to pass first (survey shadow mode starts
  only after technical validation is complete)
  All five gates must pass before any operational authority is granted
────────────────────────────────────────────────────────────────────────────

Continuous Reliability Monitoring Post-Deployment

The PoC evaluation establishes the deployment baseline. Continuous monitoring maintains it.

# Prometheus: LLM Diagnosis Reliability SLIs
# Continuous measurement of all four reliability dimensions post-deployment

groups:
  - name: llm.diagnosis.reliability
    rules:

      # Accuracy proxy: fraction of acted-on recommendations
      # that led to incident resolution (tracked post-incident)
      - record: llm:diagnosis_accuracy:rate7d
        expr: |
          sum(rate(llm_diagnosis_resolution_confirmed_total[7d]))
          /
          sum(rate(llm_diagnosis_acted_on_total[7d]))

      # Operator override rate (high = operators don't trust diagnoses)
      - record: llm:operator_override_rate:rate7d
        expr: |
          sum(rate(llm_diagnosis_operator_reversed_total[7d]))
          /
          sum(rate(llm_diagnosis_acted_on_total[7d]))

      # Calibration delta (production: rolling comparison)
      - record: llm:confidence_calibration_delta:rate30d
        expr: |
          abs(
            avg_over_time(llm_diagnosis_confidence_score_avg[30d])
            - avg_over_time(llm_diagnosis_empirical_accuracy_avg[30d])
          )

      # Composite trust score (same weights as AIOps Trustworthy post)
      - record: llm:trust_score:composite
        expr: |
          (llm:diagnosis_accuracy:rate7d * 0.40)
          +
          ((1 - clamp_max(llm:confidence_calibration_delta:rate30d, 0.30)) * 0.20)
          +
          (llm:diagnosis_consistency:rate7d * 0.20)
          +
          ((1 - llm:operator_override_rate:rate7d) * 0.20)

      # Trust score degradation alert
      - alert: LLM_DiagnosisTrustScoreDegraded
        expr: llm:trust_score:composite < 0.65
        for: 6h
        labels:
          severity: ticket
          domain: ai_reliability
        annotations:
          summary: >
            LLM diagnosis trust score at {{ $value | humanize }}.
            Accuracy or calibration degrading — review recent incident corpus
            for distribution shift. Consider re-evaluation.

Common Antipatterns

The Function-Not-Reliability Test antipattern → Evaluating the LLM for correct integration (does it call the right API? does it return structured JSON?) without evaluating for correct diagnosis (is the structured JSON's content right?). Integration tests verify that the system works; reliability tests verify that it works correctly. For LLMs, these are entirely different evaluations.
The Convenience Sample antipattern → Building the test dataset from incidents that are "easy to understand" or where the root cause is "obvious" — which means the LLM is being evaluated on the incidents where a human could diagnose correctly in five minutes anyway. The test dataset must include the hard incidents: multi-service cascades, novel failure modes, ambiguous signals. Easy test cases produce inflated accuracy scores that do not predict production performance.
The Threshold Without Calibration antipattern → Setting the confidence threshold for autonomous execution at 0.85 because that "sounds right" without verifying that 0.85 confidence actually predicts 85% empirical accuracy. As demonstrated in the calibration testing section, confidence scores and empirical accuracy can differ substantially. Thresholds must be calibrated against empirical data.
The One-Time Evaluation antipattern → Running the five gates once at deployment and treating the results as permanent. LLM diagnosis reliability decays as the environment changes. The test dataset must be refreshed with new incidents quarterly, and the gate evaluation must be repeated. A system that passed the gates 18 months ago may not pass them today.
The Shadow Mode Contamination antipattern → Showing shadow mode recommendations to the responding engineer during the evaluation period. Shadow mode is only valid if the engineer cannot see the AI's recommendation during the incident — otherwise they will unconsciously anchor on the AI diagnosis, contaminating both the human decision and the "was the AI right?" evaluation. Shadow mode recommendations should be logged to a private channel that is reviewed only after the incident is resolved.

Five Action Items for This Week

Build your historical incident test dataset. Pull the last 18 months of P1/P2 incidents from your incident management system. Filter to those with confirmed root causes in postmortems. Reconstruct the at-alert-time context for the top 50 by business impact. This dataset is the prerequisite for every subsequent gate evaluation.
Run Gate 3 (consistency) against your current LLM deployment. Present 30 historical incidents to the model three times each and measure semantic similarity. This is the fastest gate to run (no ground truth labelling required) and the most commonly failing gate in practice. High inconsistency is a deployment risk that most organisations discover after, not before, granting operational authority.
Add the trust score Prometheus recording rules to your observability stack. Even if you are in shadow mode, begin collecting the production data that the trust score requires. Establishing the baseline during shadow mode means the post-deployment trust score trend has context.
Design your evaluator function for Gate 1. The accuracy evaluation requires a function that compares LLM diagnosis text to ground truth root cause labels. Semantic similarity works for continuous scoring; keyword-based matching works for categorical scoring. Both have error rates — document them, because the evaluator function's accuracy is a meta-level uncertainty in the Gate 1 measurement.
Schedule the Gate 5 survey before starting shadow mode. The survey is the gate that most frequently blocks deployment. It can only be conducted after 30 days of shadow mode — and it is frequently delayed indefinitely when not calendared in advance. Block the time, draft the survey instrument, and communicate the evaluation timeline to the SRE team before shadow mode begins.

"An LLM used for incident diagnosis is not a search engine with better UX. It is a system that makes claims about causality in complex distributed systems, under time pressure, to engineers whose response to those claims has operational consequences. The reliability of those claims — their accuracy, calibration, consistency, and context stability — is an engineering property that can be measured. The organisations that measure it before granting operational authority will have smaller incident postmortems to write. The organisations that don't will write the case studies that everyone else learns from."

Series Complete

This post concludes the 25-post series on applying Google SRE principles to enterprise organisations and national-scale critical infrastructure. The complete series covers:

Pillar 1: Critical infrastructure reliability in healthcare, financial services, energy, and election systems
Pillar 2: Original technical contributions — burn rate alerting, error budget decision theory, autoscaling strategy selection, SOT derivation
Pillar 3: AI-assisted operations — trustworthy AIOps frameworks, human-in-the-loop governance, on-premises LLM gateways, LLM reliability evaluation
Pillar 4: Workforce and organisational transformation — the talent gap, the phased influence playbook, the DevOps-to-SRE roadmap, DORA as national policy
Pillar 5: Peer recognition and field contribution — the body of knowledge, the open letter to NIST, the adoption survey, the replication study

Each post is a standalone contribution. Together, they constitute an argument that Site Reliability Engineering is a national infrastructure discipline — one with the technical rigour, the policy relevance, and the workforce implications that warrant both professional recognition and sustained investment.

Canonical references: Google SRE Book · Google SRE Workbook · DORA Research · NIST CSF 2.0 · NIST AI RMF

State of SRE Adoption in Regulated Industries: A 2025 Survey and Analysis

Nijo George Payyappilly — Mon, 05 Oct 2026 16:00:00 +0000

Every year since 2014, the DORA research programme has asked thousands of software professionals about their delivery practices, their organisational culture, and their operational outcomes. Every year, the findings confirm the same bimodal distribution: a cohort of organisations whose software delivery and operational performance is genuinely elite, and a cohort whose performance is genuinely poor, with relatively little in the middle and a gap between the cohorts that is widening rather than narrowing.

What the aggregate DORA data has not historically surfaced, because the research design has prioritised breadth over sector depth, is where in the economy the Low performer cohort is concentrated. The answer — evident from the composition of conference audiences at SREcon versus those at financial services and healthcare IT conferences, from the practitioner surveys conducted by specialised industry groups, and from the regulatory examination findings published by the OCC, CMS, and other sector regulators — is that the Low performer cohort is disproportionately populated by the organisations that operate critical national infrastructure.

This is not a comfortable finding. It means that the systems on which the most people depend — the payment networks, the hospital EHRs, the energy management systems, the election infrastructure — are operated by organisations whose software delivery and operational practices are the furthest from the standards that the research associates with resilience, velocity, and quality. This post examines that finding in detail, with the available data and with honest acknowledgement of what is measured confidently and what is inferred.

Methodology and Data Sources

This analysis draws on four primary data sources, each with acknowledged limitations:

────────────────────────────────────────────────────────────────────────────
DATA SOURCES AND LIMITATIONS

SOURCE 1: DORA State of DevOps Annual Reports (2019–2024)
  Strength: Largest longitudinal dataset in the field; validated methodology
  Limitation: Sector breakdown is not consistently published; respondent
              self-selection skews toward technology-forward organisations
  Contribution: Overall distribution; year-over-year trend; practice correlates

SOURCE 2: Puppet State of DevOps Report (2021–2023)
  Strength: Includes sector breakdowns; large sample
  Limitation: Self-reported practices; technology-forward respondent bias
  Contribution: Sector-level DevOps maturity comparison

SOURCE 3: Cloud Native Computing Foundation Annual Survey (2022–2024)
  Strength: Kubernetes and cloud-native adoption data; sector breakdown
  Limitation: Cloud-native focus systematically excludes on-premises operators
  Contribution: Infrastructure modernisation as a proxy for SRE readiness

SOURCE 4: Regulatory Examination Findings and Industry Reports
  Strength: Objective assessment from external examiners
  Limitation: Published findings are lagging indicators; not systematic
  Contribution: Financial services and healthcare IT operational maturity signal

IMPORTANT CAVEAT:
  A primary survey of SRE adoption specifically in regulated industries
  has not been conducted at sufficient scale to produce statistically
  robust sector-level findings. The analysis below synthesises available
  proxy data and is best understood as a well-grounded characterisation
  rather than a definitive measurement. The specific numbers should be
  treated as directional estimates rather than precise measurements.
────────────────────────────────────────────────────────────────────────────

Finding 1: Regulated Industries Are Systematically Underrepresented in the DORA Elite Cohort

The DORA State of DevOps Report does not publish sector breakdowns at the performer cohort level, but available data permits inference. The technology sector (software companies, cloud providers, internet services) constitutes approximately 35% of DORA respondents and is known to be disproportionately represented in the Elite performer cohort. Financial services, healthcare, government, and energy together constitute approximately 40% of respondents but are believed to be disproportionately represented in the Low and Medium cohorts based on:

────────────────────────────────────────────────────────────────────────────
SECTOR DISTRIBUTION INFERENCE

DORA 2023 OVERALL COHORT DISTRIBUTION (approximate):
  Elite performers:   18%
  High performers:    24%
  Medium performers:  38%
  Low performers:     20%

PUPPET 2021 SECTOR-LEVEL DEVOPS MATURITY (proxy for DORA distribution):
  Technology sector:
    High/Elite equivalent: 61%
    Low/Medium equivalent: 39%

  Financial services:
    High/Elite equivalent: 34%
    Low/Medium equivalent: 66%

  Healthcare:
    High/Elite equivalent: 22%
    Low/Medium equivalent: 78%

  Government:
    High/Elite equivalent: 17%
    Low/Medium equivalent: 83%

  Energy/Utilities:
    High/Elite equivalent: 19%
    Low/Medium equivalent: 81%

INFERENCE:
  If technology sector (35% of respondents, 61% High/Elite) is generating
  a disproportionate share of Elite cohort respondents, then the 18%
  overall Elite rate implies that regulated sectors collectively are
  performing significantly below the aggregate average.

  Rough estimate: regulated industry Elite performer rate is approximately
  5–12% compared to approximately 40–50% in the technology sector.
────────────────────────────────────────────────────────────────────────────

Finding 2: The Adoption Gap Has Structural Causes

The performance differential between technology and regulated sectors is not primarily a function of investment level or management awareness. Large regulated enterprises invest substantially in technology. Their leaders are aware of DevOps and SRE frameworks. The gap persists because the structural conditions for adoption are different in ways that straightforward investment cannot address.

────────────────────────────────────────────────────────────────────────────
STRUCTURAL BARRIERS TO SRE ADOPTION IN REGULATED INDUSTRIES

BARRIER 1: Change Management Governance Overhead
  Regulated enterprises operate under change management frameworks —
  Change Advisory Boards, regulatory pre-notification requirements,
  change freeze windows — that are structurally inconsistent with
  the high-frequency, small-batch deployment model that SRE
  reliability management assumes.

  Quantified impact: OCC-regulated financial institutions average
  2.1 production deployments per month (estimated from available
  data). DORA Elite performers average multiple deployments per day.
  The gap is not primarily technical; it is governance.

  SRE adaptation required: Beyond DORA framework with RE-adjusted
  deployment frequency metric. CAB process reform as a Phase 3
  governance objective (see Phased Influence Playbook post).

BARRIER 2: Toil Accumulation From Compliance Obligations
  Regulatory compliance in financial services, healthcare, and energy
  generates mandatory operational work — quarterly access reviews,
  audit evidence collection, change documentation — that scales
  linearly with service count and cannot be eliminated.

  Quantified impact: Estimated compliance toil in heavily regulated
  environments consumes 25–40% of operational team capacity before
  any service-specific toil is accounted for. Google SRE's 50%
  toil ceiling is structurally difficult to maintain when the
  compliance floor is 25–40%.

  SRE adaptation required: Automation-first approach to compliance
  toil; GitOps as continuous CIP/HIPAA/SOC 2 evidence generation;
  Toil Ratio target of 40% (not 50%) for regulated enterprises.

BARRIER 3: Talent Market Access Disadvantage
  Regulated enterprises compete for SRE talent against technology
  companies that offer higher compensation, more interesting
  technical problems, and established SRE cultures.

  Quantified impact: Median SRE compensation at a top-tier
  technology company exceeds regulated enterprise compensation
  by 40–70% (Levels.fyi data, 2023-2024). SRE turnover at
  regulated enterprises is significantly higher than at
  technology companies.

  SRE adaptation required: Knowledge transfer methodology (see
  Talent Gap post); internal development pipeline; multiplier
  model where one senior SRE enables multiple engineers.

BARRIER 4: Architectural Legacy
  Regulated enterprises operate substantial on-premises
  infrastructure, legacy application portfolios, and
  vendor-supplied systems with limited instrumentation.
  SRE observability practices assume systems that can be
  instrumented; legacy systems often cannot be.

  Quantified impact: CNCF 2024 survey: 73% of enterprises
  still run significant on-premises workloads. SRE tooling
  documentation is primarily cloud-native; on-premises
  adaptation documentation is sparse.

  SRE adaptation required: Istio/Envoy as observability layer
  for legacy applications; HL7 interface monitoring for
  healthcare; historian-to-Prometheus bridges for energy.
────────────────────────────────────────────────────────────────────────────

Finding 3: Differentiators Between Adopting and Non-Adopting Regulated Enterprises

The structural barriers are real — but they are not determinative. Some regulated enterprises have successfully adopted SRE practices to the Medium or High performer level. The differentiators between these organisations and their peers are identifiable.

────────────────────────────────────────────────────────────────────────────
ADOPTION DIFFERENTIATORS

DIFFERENTIATOR 1: Executive Sponsorship With Operational Authority
  Organisations where SRE adoption has succeeded have executive
  sponsors who are operationally engaged — who understand what
  an error budget is, who review the DORA quarterly report, and
  who are willing to hold deployment freezes when error budgets
  are exhausted.

  Organisations where adoption has stalled have executive awareness
  (the CTO knows what SRE is) without operational engagement
  (the CTO does not attend the SLO review or protect the
  deployment gate from override pressure).

  Signal: Ask "When was the last time an error budget gate blocked
  a deployment?" If the answer is "never," either the gate doesn't
  exist or it has been overridden every time it has fired.

DIFFERENTIATOR 2: The Compliance-Function as Ally
  Successful adopters have made SRE practices legible to their
  compliance functions — reframing error budgets as risk appetite
  frameworks, GitOps as continuous change evidence, and SLOs as
  Recovery Time Objectives with measurement.

  Non-adopters treat the compliance function as an external
  constraint on SRE adoption rather than a potential ally.
  The compliance function's objections to rapid deployment
  are reasonable; the response is not to override them but
  to design SRE governance that satisfies them.

  Signal: Is the compliance or risk function aware of the
  organisation's SLOs and error budget policy? If not, the
  SRE programme exists in a silo that will not survive the
  first regulatory examination that asks about it.

DIFFERENTIATOR 3: Measurement Before Governance
  Successful adopters measured DORA metrics and SLI/SLO data
  for 6–12 months before proposing governance changes. The
  measurement baseline earned the credibility that made the
  governance proposal legible.

  Non-adopters proposed governance changes (error budget policies,
  deployment gates, on-call restructuring) before establishing
  the measurement infrastructure that would justify them. Without
  data, governance proposals are opinions. Organisations resist
  opinions; they act on data.

  Signal: Is there a DORA metric baseline for at least 6 months?
  If not, the programme is in Phase 0 regardless of what governance
  structures have been proposed.

DIFFERENTIATOR 4: Toil Reduction as the Entry Point
  The most successful adoption entry points were not SLO frameworks
  or error budget policies — they were toil reduction programmes
  that freed team capacity and demonstrated concrete value without
  requiring governance authority. Automation that saves 20 hours
  per sprint creates the credibility conversation that SLO policy
  proposals cannot open cold.

  Signal: Has the SRE team quantified the toil hours reduced through
  automation in the last quarter? This number is the most accessible
  SRE value demonstration for sceptical leadership.
────────────────────────────────────────────────────────────────────────────

Finding 4: SRE Adoption Gap by Sub-Sector

The regulated industry aggregate conceals significant sub-sector variation. The following characterisation is based on available data synthesis.

────────────────────────────────────────────────────────────────────────────
SRE ADOPTION CHARACTERISATION BY SUB-SECTOR (2025 estimate)

FINANCIAL SERVICES:
  Global systemically important banks (G-SIBs):
    Adoption stage: Phase 1–2 (Measurement and Accountability)
    Drivers: Regulatory pressure (SR 21-3, EU DORA), competitive
              pressure from fintech, talent from technology sector
    Barriers: Legacy core banking systems, complex change management,
               siloed operations and development

  Regional and community banks:
    Adoption stage: Phase 0 (Trust and Visibility) or pre-Phase 0
    Barriers: SRE talent inaccessible, limited DevOps foundation,
               vendor-managed core systems

HEALTHCARE:
  Academic medical centres and large health systems:
    Adoption stage: Phase 0–1
    Drivers: EHR platform modernisation, patient safety imperatives
    Barriers: Clinical workflow constraints on change management,
               HIPAA compliance toil, legacy clinical systems

  Community hospitals and regional health systems:
    Adoption stage: Pre-Phase 0
    Barriers: IT as pure cost centre, vendor-managed EHRs,
               limited SRE talent access

ENERGY:
  Investor-owned utilities (IOUs):
    Adoption stage: Phase 0 (IT layer only; OT layer largely pre-DevOps)
    Drivers: Grid modernisation, smart grid investment
    Barriers: IT/OT convergence complexity, NERC CIP constraints,
               decades-long infrastructure lifecycles

  Municipal utilities and cooperatives:
    Adoption stage: Pre-Phase 0
    Barriers: Limited IT capacity, vendor-managed SCADA systems

GOVERNMENT:
  Federal civilian agencies (CFO Act agencies):
    Adoption stage: Phase 0 (highly variable; some agencies Phase 1)
    Drivers: M-23-22 memo on cloud strategy, CISA cybersecurity guidance
    Barriers: Federal acquisition regulations, legacy systems, talent

  State and local government:
    Adoption stage: Pre-Phase 0 to Phase 0
    Barriers: Budget constraints, talent market, legacy systems

────────────────────────────────────────────────────────────────────────────
CROSS-SECTOR PATTERN:
  SRE adoption correlates with:
  → Organisation size (larger → more likely to have SRE)
  → Competitive pressure from technology-native competitors
  → Regulatory pressure with quantitative performance requirements
  → Presence of a cloud-native transformation programme
  → Prior DevOps adoption (SRE builds on DevOps foundation)
────────────────────────────────────────────────────────────────────────────

Recommendations: What Would Accelerate Adoption

Three interventions have the strongest evidence base for accelerating SRE adoption in regulated industries at the national scale.

Intervention 1: Quantitative Performance Benchmarks in Regulatory Guidance

The single most effective driver of regulated industry adoption would be the incorporation of DORA-based performance benchmarks into sector regulatory guidance. SR 21-3 requires operational resilience measurement; it does not specify what to measure. A NIST guidance document specifying DORA metrics as the recommended measurement framework for software delivery quality in critical infrastructure systems would create regulatory incentive alignment that voluntary adoption cannot achieve at the required pace.

Intervention 2: Knowledge Transfer Infrastructure

The SRE talent gap cannot be closed by competitive hiring alone. The knowledge that produces SRE practitioners — operational judgement, incident experience, postmortem culture — is tacit and transfers through mentorship and structured operational exposure, not through certification programmes. Federal investment in SRE practitioner development infrastructure — formal apprenticeship programmes, subsidised SREcon participation for regulated industry practitioners, shared learning environments where practitioners from different regulated industries can exchange operational knowledge — would accelerate the tacit knowledge transfer that drives adoption.

Intervention 3: On-Premises SRE Practice Documentation

The SRE practitioner literature is overwhelmingly written for cloud-native environments. The Google SRE Book assumes auto-scaling, on-demand provisioning, and deployment infrastructure that most regulated enterprises do not have. Documented frameworks for SRE practice in on-premises, regulated enterprise environments — specifically addressing the CAB process adaptation, the Istio mTLS observability pattern, the GitOps compliance evidence model, and the DORA metric adjustments for regulated environments — would lower the adoption barrier for the organisations where the gap is largest.

Common Research Limitations and Future Work

This analysis has three significant limitations that future research should address.

Limitation 1: The absence of a primary survey of SRE adoption in regulated industries means that the sector-level characterisation above is inference from proxy data, not direct measurement. A survey instrument designed specifically to measure SRE adoption (not just DevOps maturity) in regulated industries, with sector breakdowns and organisational size stratification, would substantially improve the empirical foundation of these findings.

Limitation 2: Available data captures adoption of DevOps practices, not specifically SRE practices. SRE is a specific implementation of DevOps principles, not equivalent to DevOps maturity generally. An organisation can be a High DevOps performer without having implemented error budgets, SLOs, or toil ratio tracking — all of which are SRE-specific.

Limitation 3: The on-premises regulated enterprise environment is systematically undercounted in cloud-native surveys (which require cloud adoption to participate) and may be overrepresented in low-performer cohorts because their infrastructure architecture makes some SRE practices harder to implement, not because their operational culture is less mature.

Five Action Items for This Week

Contribute to the data. Respond to the annual DORA State of DevOps survey when it opens (typically Q1 each year). The regulated industry data gap persists partly because regulated industry practitioners are less likely to participate in voluntary research surveys. Your response is a data point that improves the field's understanding of where adoption stands.
Characterise your organisation's DORA stage using the adoption stage table. Which sub-sector characterisation most closely matches your organisation? Is your adoption stage consistent with the characterisation or ahead/behind it? The gap between the characterisation and your experience is feedback for the analysis — share it in the comments.
Document one adoption differentiator in your organisation. Whether your compliance function is an ally or an obstacle, whether you have executive operational engagement or only executive awareness, whether you have a measurement baseline before governance — documenting your organisation's state on the four differentiators creates the self-assessment that drives the next adoption phase.
Request the sector breakdown from DORA. The DORA research team has sector-level data that is not consistently published. Public requests from practitioners for sector-level performer cohort breakdowns create demand for the analysis that would substantially improve the evidence base for the national adoption gap.
Share this post with one person in a regulatory function — OCC, CMS, NERC, CISA — who has influence over operational resilience guidance. The intervention with the highest leverage for accelerating regulated industry SRE adoption is quantitative benchmarks in regulatory guidance. Regulators who understand the current adoption gap from a practitioner perspective are more likely to incorporate DORA-based benchmarks into their guidance frameworks.

"The state of SRE adoption in regulated industries is not a comfortable picture. The systems that the most people depend on are operated by the organisations whose delivery and operational practices are furthest from what the research associates with resilience. This is not a coincidence — it is the predictable consequence of structural barriers that voluntary adoption cannot overcome at the pace that the national infrastructure risk requires. The comfortable response is more conferences, more blog posts, more certifications. The adequate response involves regulatory incentives, knowledge transfer infrastructure, and practitioner-led policy engagement. This post is a contribution toward the latter."

DORA Metrics as a National Software Quality Benchmark: A Policy Perspective

Nijo George Payyappilly — Mon, 28 Sep 2026 16:00:00 +0000

In 1953, the Eisenhower administration commissioned a study of the structural integrity of the U.S. highway bridge system. The study found significant variation in design standards, inspection practices, and maintenance quality across states. By 1968, the Federal Aid Highway Act had established national standards for bridge design and inspection — standards that applied regardless of which state a bridge was in, who built it, or who funded its maintenance.

The logic was straightforward: bridges are part of a national transportation network. The failure of any bridge in that network has consequences that extend beyond the jurisdiction that owns it. A national network of critical infrastructure requires national standards for the components that constitute it.

The software systems that now constitute the U.S. digital infrastructure — the banking networks, the healthcare platforms, the election management systems, the energy grid control software — have the same property. They are components of a national network. Their failure has consequences that extend beyond the organisations that own them. And they are governed, where they are governed at all, by a patchwork of sector-specific regulatory frameworks that were written before the systems they regulate had become critical infrastructure.

The DORA Four Key Metrics are not a magic solution to this governance gap. They are four numbers, derived from a decade of empirical research, that measure the most predictive indicators of software delivery performance. They are sector-neutral, implementation-independent, and available to any organisation with access to its own pipeline and incident data. They are, in short, the most plausible candidate for a national software quality benchmark that currently exists. This post makes the case for treating them as one.

What a National Software Quality Benchmark Would Address

The current regulatory landscape for software quality in critical infrastructure is characterised by three structural gaps:

Gap 1 — Measurement Without Benchmark. Most regulatory frameworks for operational resilience — OCC SR 21-3 for financial services, Joint Commission standards for healthcare, NERC CIP for energy — require organisations to measure operational performance and demonstrate continuous improvement. None specifies what "good" looks like in quantitative terms. An organisation that deploys software once per year with a 60% change failure rate may technically satisfy "continuous improvement" requirements if it deploys slightly more frequently than it did the previous year.

Gap 2 — Sector Isolation. The software delivery capabilities of a financial institution, a hospital system, and an energy utility are assessed by entirely different regulatory frameworks with no common measurement vocabulary. A bank that is a High DORA performer and a hospital that is a Low DORA performer cannot be compared — not because comparison is inappropriate, but because the regulatory frameworks that govern them do not share a unit of measurement.

Gap 3 — Outcome Without Process. Regulatory frameworks for operational resilience measure outcomes: recovery time objectives, recovery point objectives, incident frequency. They do not measure the process quality that predicts whether those outcomes will be achieved before the next incident. The DORA Four Key Metrics are predictive of resilience outcomes — organisations in the Elite performer cohort have measurably better incident outcomes than Low performers. A regulatory framework that only measures outcomes misses the process quality signal that predicts future performance.

The Empirical Foundation

The case for DORA metrics as a national benchmark rests on three properties that distinguish them from other candidate metrics.

Property 1 — Longitudinal Empirical Validation. The DORA research programme has collected performance data from tens of thousands of organisations across more than a decade. The relationship between DORA performance cohort and business outcomes (including resilience outcomes) is not a theoretical claim — it is a statistically robust empirical finding. Nicole Forsgren's research methodology (described in Accelerate) uses structural equation modelling to establish causal relationships, not just correlations.

Property 2 — Sector Neutrality. The DORA Four are defined at the delivery process level, not at the application domain level. Deployment Frequency is the same metric for a payment processing API and a clinical decision support system. This sector neutrality is essential for a benchmark that would apply across the eleven critical infrastructure sectors that are now dependent on software systems.

Property 3 — Self-Measurability. Unlike some quality metrics that require external audit to measure reliably, the DORA Four can be measured by any organisation with access to its own CI/CD pipeline data and incident management records. This reduces the compliance overhead of measurement and makes the benchmark accessible to organisations that lack the resources for dedicated quality measurement programmes.

────────────────────────────────────────────────────────────────────────────
DORA PERFORMER COHORT DEFINITIONS (2023 State of DevOps Report)

DEPLOYMENT FREQUENCY:
  Elite:  Multiple times per day
  High:   Once per week to once per month
  Medium: Once per month to once every 6 months
  Low:    Fewer than once every 6 months

LEAD TIME FOR CHANGES:
  Elite:  Less than 1 hour
  High:   Between 1 day and 1 week
  Medium: Between 1 month and 6 months
  Low:    More than 6 months

CHANGE FAILURE RATE:
  Elite:  0–15%
  High:   16–30%
  Medium: 16–30%    (same range as High in recent research)
  Low:    46–60%

MEAN TIME TO RESTORE:
  Elite:  Less than 1 hour
  High:   Less than 1 day
  Medium: Between 1 day and 1 week
  Low:    Between 1 week and 1 month

────────────────────────────────────────────────────────────────────────────
RESILIENCE OUTCOME CORRELATION (Accelerate research findings):
  Elite + High performers vs. Low performers:
  → 2× more likely to exceed profitability goals
  → 2× more likely to exceed productivity goals
  → 24× less time lost to unplanned work
  → 3× lower change failure rate
  → 24× faster recovery from incidents
────────────────────────────────────────────────────────────────────────────

What a Policy Recommendation Would Look Like

A national software quality benchmark based on DORA methodology would have three components: a measurement requirement, a benchmark target, and an improvement mandate.

────────────────────────────────────────────────────────────────────────────
PROPOSED NATIONAL SOFTWARE QUALITY BENCHMARK FRAMEWORK

COMPONENT 1: MEASUREMENT REQUIREMENT
  Regulatory applicability: Operators of critical infrastructure
    systems (CISA-designated sectors) with > $1B in annual revenue
    or > 1M users dependent on the system.
  Requirement: Annual self-assessment of DORA Four Key Metrics
    for all software systems supporting critical business services.
    Assessment methodology: DORA Quick Check or equivalent pipeline-
    derived measurement (not self-report by programme managers).
  Reporting: Annual submission to sector regulator;
    methodology documented and available for examination.

COMPONENT 2: BENCHMARK TARGETS (by system criticality tier)
  Tier 0 (life-critical systems — healthcare, nuclear):
    → Deployment Frequency: ≥ monthly (at minimum; more is better)
    → Lead Time: ≤ 1 week
    → Change Failure Rate: ≤ 15%
    → MTTR: ≤ 1 day

  Tier 1 (critical infrastructure systems — financial, energy, election):
    → Deployment Frequency: ≥ weekly
    → Lead Time: ≤ 1 day
    → Change Failure Rate: ≤ 30%
    → MTTR: ≤ 1 day

  Note: Targets do not require Elite performance — they establish
    a minimum standard below which regulatory scrutiny increases.
    The targets correspond approximately to the High performer cohort,
    which represents achievable performance for well-managed enterprises.

COMPONENT 3: IMPROVEMENT MANDATE
  Organisations in the Low performer cohort must:
    → Submit a remediation plan within 90 days of measurement
    → Demonstrate measurable improvement within 12 months
    → Re-measurement annually; sustained Low performance triggers
      enhanced regulatory oversight (not fines — examination priority)

────────────────────────────────────────────────────────────────────────────

How DORA Maps to Existing Regulatory Frameworks

The proposed benchmark does not require new regulatory authority. It requires that existing operational resilience frameworks — which already have quantitative measurement requirements — adopt DORA metrics as the measurement vocabulary for software delivery quality.

────────────────────────────────────────────────────────────────────────────
DORA → EXISTING REGULATORY FRAMEWORK ALIGNMENT

OCC SR 21-3 (Financial Services Operational Resilience):
  Requirement: Continuous monitoring of operational performance
  DORA mapping: Deployment Frequency and MTTR are direct measures
                of the delivery and recovery performance SR 21-3
                requires organisations to demonstrate.
  Gap filled: SR 21-3 requires measurement; it doesn't specify
              what to measure. DORA provides the measurement vocabulary.

NERC CIP (Energy Sector Reliability Standards):
  Requirement: Configuration change management (CIP-010)
  DORA mapping: Change Failure Rate is the quality measure for
                the changes that CIP-010 requires to be managed.
                A utility with a 50% CFR on bulk power system
                changes is clearly not managing changes effectively,
                regardless of CIP-010 process compliance.
  Gap filled: CIP-010 measures process; CFR measures outcome quality.

HIPAA Security Rule (Healthcare):
  Requirement: Contingency planning; emergency mode operations
  DORA mapping: MTTR is the operational metric for contingency plan
                effectiveness. Lead Time for Changes measures the
                organisation's ability to deploy security patches
                in response to identified vulnerabilities.
  Gap filled: HIPAA measures plan existence; DORA measures plan quality.

Joint Commission (Healthcare IT Standards):
  Requirement: Performance improvement processes for clinical systems
  DORA mapping: Deployment Frequency + Lead Time measure the
                organisation's ability to deliver improvements.
                CFR measures whether those improvements degrade
                existing performance.
  Gap filled: Joint Commission standards measure process; DORA measures
              the outcome quality of the delivery process itself.

CISA Election Security Guidance:
  Requirement: Incident response planning; resilience testing
  DORA mapping: MTTR is the primary operational metric for election
                infrastructure resilience. CFR measures whether
                changes to election systems are being managed safely.
  Gap filled: CISA guidance is process-focused; DORA adds outcome
              measurement to the resilience assessment.
────────────────────────────────────────────────────────────────────────────

The International Comparison

The United States is not alone in recognising the gap between operational resilience regulation and software delivery quality measurement. Two international examples demonstrate that the bridge from resilience requirements to delivery benchmarks is navigable.

UK Government Digital Service (GDS) Performance Framework. The UK GDS established performance metrics for government digital services that include deployment frequency and service availability. GDS-assessed services are publicly rated on a dashboard that includes these delivery quality indicators. The framework demonstrates that government digital services can be assessed against quantitative delivery benchmarks in a way that is transparent and actionable.

EU Digital Operational Resilience Act (DORA — the regulation). The EU regulation of the same name (different acronym: Digital Operational Resilience Act) requires financial institutions to test and demonstrate ICT resilience. Article 24 of DORA mandates threat-led penetration testing. Article 25 requires ICT risk management frameworks with quantitative performance indicators. The EU DORA does not use the DORA research metrics by name — but its substantive requirements for quantitative ICT performance measurement are consistent with what the DORA Four would measure.

A Phased Policy Adoption Pathway

────────────────────────────────────────────────────────────────────────────
PHASED POLICY ADOPTION: FROM VOLUNTARY TO MANDATORY

PHASE 1 (Years 1–2): Voluntary Adoption with Recognition
  Action: NIST publishes DORA-based software quality guidance
          as a voluntary framework supplement to the Cybersecurity
          Framework (CSF) and AI RMF.
  Incentive: Organisations demonstrating High/Elite DORA performance
             receive streamlined examination treatment under
             sector-specific regulatory frameworks.
  Outcome: Builds the measurement infrastructure and industry
           vocabulary before mandatory measurement begins.

PHASE 2 (Years 2–4): Mandatory Measurement, Voluntary Benchmarking
  Action: OCC, CMS, NERC, and EAC require DORA measurement for
          Tier 1 and Tier 0 systems under their oversight.
          Benchmark targets are published but not yet enforced.
  Incentive: Measurement data builds the national baseline that
             makes benchmark targets evidence-based rather than
             arbitrarily imposed.
  Outcome: National visibility into software delivery quality
           distribution across critical infrastructure sectors.

PHASE 3 (Years 4–6): Mandatory Measurement and Minimum Benchmarks
  Action: Low performer status (below proposed minimum targets)
          triggers enhanced oversight and remediation requirements.
          Targets calibrated from Phase 2 measurement data.
  Precedent: This approach mirrors how NERC CIP standards evolved —
             voluntary guidelines (1998–2006) → mandatory standards
             (2008–present) as the industry built measurement
             infrastructure.
────────────────────────────────────────────────────────────────────────────

Objections and Responses

Objection: "DORA metrics were designed for technology companies, not regulated enterprises."

Response: The DORA research programme has surveyed organisations across all industries since 2014. The finding that regulated enterprises cluster in Low and Medium performer cohorts is precisely the evidence that motivates the policy recommendation — not a reason to exempt them from measurement. Regulated enterprises have the most to gain from improvement and the most to lose from continued Low performance.

Objection: "Regulated enterprises can't achieve Elite DORA performance because of compliance overhead."

Response: The proposed benchmark targets High performer cohort levels, not Elite. High performance (weekly deployments, < 1-week lead time, < 30% CFR, < 1-day MTTR) is achievable in regulated environments with the SRE practices documented throughout this series. The Beyond DORA post specifically addresses how regulated enterprise constraints (CAB cycles, compliance change freezes) are accommodated in the RE-adjusted DORA framework.

Objection: "Self-measurement is unreliable — organisations will game the metrics."

Response: Pipeline-derived DORA metrics are difficult to game because they are derived from objective system logs (deployment timestamps, incident creation/resolution timestamps) rather than from qualitative self-assessment. The measurement methodology is specified — not self-report — and examination teams can validate it against actual pipeline data.

Common Antipatterns in the Policy Conversation

The Compliance-as-Quality antipattern → Arguing that organisations already in compliance with sector-specific regulations do not need additional quality benchmarks. HIPAA compliance does not measure deployment frequency. PCI-DSS compliance does not measure MTTR. CIP-010 compliance does not measure change failure rate. Compliance measures process adherence; DORA measures delivery outcome quality. These are orthogonal.
The Sector-Exceptionalism antipattern → Arguing that a specific sector is too unique for DORA metrics to apply. Financial services organisations make this argument (unique regulatory constraints); healthcare organisations make it (unique patient safety obligations); energy operators make it (unique OT/IT complexity). In each case, the DORA metrics apply at the software delivery layer — which all three sectors have. The sector-specific constraints affect the target values, not the applicability of the measurement framework.
The Small-Organisation Exemption antipattern → Proposing mandatory benchmarking without proportionality provisions. The proposed framework includes a revenue/user threshold that exempts small operators. A community bank with $50M in assets does not have the same systemic risk profile as a globally systemically important financial institution. The benchmark should apply proportionally to the organisations whose failure has national consequences.

Five Action Items for This Week

Measure your organisation's DORA performance cohort using the Quick Check tool at dora.dev. Even if the measurement is not yet being used for regulatory purposes, knowing where your organisation sits in the distribution is the prerequisite for any improvement initiative. The Quick Check takes fifteen minutes and provides the self-assessment baseline that a formal measurement programme would produce.
Map your organisation's existing regulatory reporting obligations to the DORA Four. For each report you submit to sector regulators: which of the DORA Four does it most closely measure? Where are the gaps? The mapping exercise produces the argument for why DORA measurement would reduce regulatory reporting overhead (by consolidating multiple sector-specific metrics into a common framework) while improving measurement quality.
Submit a comment to NIST during the next CSF or AI RMF public comment period recommending the inclusion of software delivery quality metrics. The NIST public comment process is the most accessible pathway for practitioners to influence federal guidance. A well-reasoned comment from a practitioner with operational experience in regulated environments has influence that policy generalists cannot replicate.
Present your organisation's DORA metrics to your compliance or risk function with the regulatory alignment table from this post. The argument that DORA metrics already measure what SR 21-3, HIPAA, NERC CIP, and Joint Commission require — in a more quantitative form than the requirements themselves specify — is the compliance-function adoption argument. If compliance adopts DORA measurement, it becomes organisationally durable.
Contribute your organisation's performance data to the annual DORA State of DevOps survey. The survey is anonymous; the aggregate data is what drives the research. Every additional regulated enterprise respondent improves the statistical validity of the regulated enterprise cohort analysis — and that analysis is what makes the policy argument empirically grounded rather than theoretical.

"The United States has federal standards for the structural integrity of every bridge on the interstate highway system. It does not have federal standards for the delivery quality of the software systems that now carry the economy across a different kind of infrastructure. The DORA metrics are not perfect — no four numbers can fully characterise the delivery capability of a complex organisation. But they are the most empirically validated, sector-neutral, self-measurable quality benchmark the software delivery field currently has. The question is not whether they are perfect. The question is whether the absence of any benchmark at all is a better policy position."

From DevOps to SRE: A Practitioner's Roadmap for Enterprise Transformation

Nijo George Payyappilly — Mon, 21 Sep 2026 16:00:00 +0000

In 2018, the same year that Forsgren, Humble, and Kim published Accelerate — the empirical validation of DevOps practices — approximately 47% of the organisations in the DORA research study were classified as Low performers on the Four Key Metrics. By 2023, that number had not materially decreased. The percentage of Elite and High performers had grown, but the Low performer cohort had not emptied. Instead, the distribution had spread: the gap between the organisations that had successfully adopted modern delivery practices and those that had not was wider in 2023 than in 2018.

This persistence of the Low performer cohort in the face of years of DevOps advocacy, tooling investment, and cultural attention is not evidence that DevOps does not work. It is evidence that the organisational transformation required to become an elite performer is not primarily a tooling problem or a process problem. It is an engineering discipline problem — and the discipline that addresses it, in the specific dimensions where DevOps-focused transformations plateau, is Site Reliability Engineering.

This post maps the journey from DevOps foundations to SRE practice at the enterprise scale. It is not a theoretical model. It is a phased roadmap with specific exit gates for each phase, derived from the observable signals that distinguish genuine phase completion from the stalled adoptions that organisations mistake for progress.

What DevOps Leaves Unresolved

DevOps, as a movement and a set of practices, addresses the delivery pipeline: how code moves from developer to production faster, more safely, and more reliably. The Three Ways (flow, feedback, learning) and the DORA Four Key Metrics are all delivery pipeline metrics. They measure how quickly and safely the organisation can make changes.

What DevOps does not address — cannot address by design — is the production system itself after the change has been delivered. DevOps tells you whether you can ship. SRE tells you whether what you shipped is serving users the way it should, and what to do when it isn't.

────────────────────────────────────────────────────────────────────────────
WHAT DEVOPS ADDRESSES vs WHAT SRE ADDRESSES

DEVOPS ADDRESSES:
  → How code moves from commit to production (delivery pipeline)
  → How quickly defects are detected and corrected in the pipeline
  → How development and operations teams collaborate on the pipeline
  → Deployment frequency, lead time, change failure rate, MTTR
    (MTTR here = time to restore a failed deployment, not service reliability)

DEVOPS DOES NOT ADDRESS:
  → What "reliable" means for a specific service from a user perspective
  → How much unreliability is acceptable (and who decides)
  → What the investment ceiling for reliability should be
  → How to detect degradation before it becomes a user-facing failure
  → How to balance release velocity against production stability
    (other than the cultural norm of "more deployment frequency = good")

SRE ADDRESSES THE GAPS:
  → SLIs/SLOs: define what reliable means and measure it continuously
  → Error budgets: define how much unreliability is acceptable and who decides
  → Observability engineering: detect degradation before user-facing failure
  → Toil elimination: make the operational investment sustainable
  → Error budget as change gate: formalise the velocity/stability trade-off
────────────────────────────────────────────────────────────────────────────

The DevOps-to-SRE journey is not a replacement of DevOps. It is a completion of it. The delivery pipeline that DevOps builds is the mechanism through which SRE reliability improvements are deployed. The cultural practices DevOps establishes (blameless culture, shared accountability, continuous improvement) are the prerequisites for SRE governance. DevOps must come first. SRE builds on it.

The Four-Phase Roadmap

Phase 0 — Trust and Visibility (Months 1–8)

Phase 0 establishes the measurement infrastructure and the organisational credibility that make all subsequent phases possible. No SRE governance should be attempted before Phase 0 is complete. No SLO should be enforced before the measurement is trusted. No error budget policy should be proposed before stakeholders understand what an error budget is.

────────────────────────────────────────────────────────────────────────────
PHASE 0 OBJECTIVES AND EXIT GATES

OBJECTIVE 1: Instrument the Four Golden Signals for critical services
  Exit gate: Latency, Traffic, Errors, Saturation metrics exist in
             Prometheus for at least 3 production services.
             Sourced from Istio Envoy proxy (not application metrics).

OBJECTIVE 2: Baseline DORA Four Key Metrics
  Exit gate: 6 months of DORA data available.
             Deployment frequency, lead time, CFR, MTTR all defined
             and measured from actual pipeline and incident data.
             Not estimated or self-reported.

OBJECTIVE 3: Define SLIs for at least one critical service
  Exit gate: SLI defined, instrumented, and visible on a dashboard.
             Target SLO agreed with the owning development team.
             Error budget calculated but NOT yet enforced.

OBJECTIVE 4: Establish blameless postmortem practice
  Exit gate: Three postmortems completed with the full template.
             Action items assigned, tracked, and followed up.
             At least one action item automated (not just documented).

OBJECTIVE 5: Earn the trust of at least one non-SRE stakeholder
  Exit gate: VP of Engineering, operations leader, or compliance lead
             is citing SRE data in meetings they initiate.
             This gate is qualitative but non-negotiable.

PHASE 0 TIMELINE REALITY:
  6–8 months is typical for regulated enterprises.
  Organisations that rush Phase 0 produce Phase 1 structures with
  no measurement foundation — and then cannot demonstrate the value
  that would earn Phase 2 authority.
────────────────────────────────────────────────────────────────────────────

Phase 1 — Measurement and Accountability (Months 8–18)

Phase 1 activates the measurement infrastructure built in Phase 0. Error budget policies go from documented to enforced. The deployment gate fires for the first time. The DORA quarterly report goes to leadership. The first external sign of SRE maturity becomes visible.

────────────────────────────────────────────────────────────────────────────
PHASE 1 OBJECTIVES AND EXIT GATES

OBJECTIVE 1: Enforce error budget policy for pilot services
  Exit gate: Error budget gate has fired and blocked at least one
             deployment in the pilot service cohort.
             The deployment freeze was upheld (not overridden).
             Development team accepted the gate without escalation.
  Why this gate matters: An error budget policy that has never fired
    is a policy that has never been tested. The first firing is the
    moment that determines whether the governance is real or decorative.

OBJECTIVE 2: Activate multi-window burn rate alerting
  Exit gate: 14×/6×/3×/1× burn rate alerts deployed for all
             services with defined SLOs.
             Threshold-based alerts eliminated or reclassified.
             False positive rate below 10% for Tier 1 alerts.

OBJECTIVE 3: Instrument Toil Ratio and publish quarterly
  Exit gate: Toil Ratio measured and below 45% for SRE team.
             Top-3 toil categories identified and automation
             investment allocated.
             Toil Ratio trend visible in DORA quarterly report.

OBJECTIVE 4: First DORA quarterly report to leadership
  Exit gate: DORA report presented to VP Engineering or CTO.
             Report includes industry benchmark comparison.
             At least one DORA metric has improved since baseline.

OBJECTIVE 5: SLO coverage for top-10 services by revenue/risk
  Exit gate: SLIs and SLOs defined for top-10 services.
             Error budget remaining visible on shared dashboard.
             At least 5 services in Tier 1 (budget healthy) policy state.

PHASE 1 COMMON FAILURE MODE:
  The first error budget gate fires and is overridden by the VP
  of Engineering under business pressure. This is the critical
  juncture. If the override happens without documented justification,
  the policy becomes decorative. The correct response is not to
  accept the override silently — it is to document the override
  with the business justification, share that documentation with
  the stakeholder who approved it, and begin tracking override rate
  as a governance metric. Override rate above 20% is a signal that
  either the SLO targets are poorly calibrated or the policy lacks
  organisational buy-in.
────────────────────────────────────────────────────────────────────────────

Phase 2 — Platform and Automation (Months 18–30)

Phase 2 is the scalability phase. The practices established in Phase 1 for a small cohort of services are productised into platform primitives that development teams can adopt self-service. The SRE team transitions from implementing reliability practices directly to building the platform that enables development teams to implement them.

────────────────────────────────────────────────────────────────────────────
PHASE 2 OBJECTIVES AND EXIT GATES

OBJECTIVE 1: Self-Service SLO Platform
  Exit gate: A development team can define an SLI, set an SLO,
             and see their error budget dashboard without SRE
             team involvement.
             SLO definition template and onboarding documentation
             exist and have been used by at least two teams.

OBJECTIVE 2: Automated Toil Elimination (Class 4 Automation)
  Exit gate: Postmortem generation is automated.
             DORA metric calculation is automated.
             Compliance evidence package is automated.
             Total toil from these three categories: zero manual hours.

OBJECTIVE 3: Deployment Gate Platform (not per-service scripts)
  Exit gate: The error budget PreSync gate is a reusable platform
             component deployed via Argo CD with service-specific
             configuration only (not code changes).
             Any new service can adopt the gate with a ConfigMap change.

OBJECTIVE 4: Toil Ratio below 35%
  Exit gate: SRE team Toil Ratio below 35% for two consecutive quarters.
             The time freed from toil is demonstrably invested in
             engineering work (tracked in sprint metrics).

OBJECTIVE 5: SLO ownership by development teams
  Exit gate: At least 3 development teams own their SLOs without
             SRE team involvement in day-to-day management.
             Development team lead can explain their error budget
             state and trend without referring to the SRE team.

PHASE 2 CULTURAL SIGNAL:
  The leading indicator of genuine Phase 2 progress is when
  development teams request SLO coverage for new services before
  the SRE team asks them to. When a product team says "we're
  launching a new feature and we need an SLO for it," the SRE
  practices have become part of the engineering culture, not an
  external programme.
────────────────────────────────────────────────────────────────────────────

Phase 3 — Culture and Scale (Month 30+)

Phase 3 is the generative phase — where SRE practices are embedded in the engineering culture deeply enough to self-propagate. New engineers adopt them because that is how their team works. New services get SLOs because that is part of the launch process. Reliability investment is governed by error budgets because that is how the organisation makes decisions about deployment velocity.

────────────────────────────────────────────────────────────────────────────
PHASE 3 CHARACTERISTICS (not exit gates — emergent properties)

→ The DORA quarterly report is requested by leadership, not delivered
  by the SRE team on their own initiative.

→ Error budget overrides are rare (< 5%) and always documented.
  The override audit is a routine quarterly review, not a crisis response.

→ New engineers joining the organisation learn SLO practice as
  part of their onboarding, from their team leads, without SRE
  team involvement.

→ The SRE team's primary activity is reliability architecture
  and platform engineering — not incident response, not postmortem
  facilitation, not SLO coaching.

→ Reliability investment is a budget line item discussed in
  product roadmap planning, not a conversation that the SRE team
  has to initiate.

→ DORA metrics are referenced in performance conversations about
  engineering teams — not as SRE programme metrics but as
  organisational delivery health indicators.

PHASE 3 TIMELINE REALITY:
  Phase 3 is not a destination — it is an ongoing state that requires
  active maintenance. Organisations that stop investing in the practices
  that produced Phase 3 will regress, typically first in Toil Ratio
  (compliance obligations grow, automation debt accumulates) and then
  in DORA metrics (velocity decreases as reliability investment crowds
  out delivery work).
────────────────────────────────────────────────────────────────────────────

The SRE Operating Model: Who Does What

The DevOps-to-SRE transition requires clarity about the operating model — who is responsible for reliability, who owns the SLO, and what the SRE team's role is relative to development teams at each phase.

────────────────────────────────────────────────────────────────────────────
SRE OPERATING MODEL EVOLUTION BY PHASE

PHASE 0 — Centralised Expert Model:
  SRE team: Defines, instruments, and monitors all SLOs.
            Responds to all SLO-related incidents.
            Owns all reliability tooling.
  Development teams: Consumers of SLO dashboards.
                     Participants in postmortems.
  Risk: SRE team becomes a bottleneck; all reliability work flows
        through a small team with insufficient scale.

PHASE 1 — Hybrid Model (Centralised + Embedded):
  SRE team: Maintains platform, governs SLO framework.
            Embeds SRE engineers in development teams on rotation.
  Development teams: Own their SLOs with SRE support.
                     Carry on-call alongside SRE.
  Progress signal: Development teams raising reliability issues
                   without waiting for SRE to identify them.

PHASE 2 — Platform Model:
  SRE team: Builds and maintains the reliability platform.
            Consults on complex reliability problems.
            Sets organisational standards; does not enforce them directly.
  Development teams: Fully own their SLOs and error budgets.
                     Self-serve from the reliability platform.
                     Carry their own on-call; SRE carries escalation only.
  Progress signal: SRE team headcount can decrease (or hold steady
                   as services scale) without reliability degrading.

PHASE 3 — Embedded Model:
  SRE function: Distributed across platform engineering and
                development teams. "SRE" is a practice, not a team.
                Central SRE team focuses on cross-cutting reliability
                architecture and organisational standards.
  Progress signal: The organisation doesn't need an SRE team to
                   tell it to invest in reliability.
────────────────────────────────────────────────────────────────────────────

Common Antipatterns

The DevOps Rename antipattern → Renaming the operations team "SRE" without changing the practice. SRE titles without SRE practices are not a transformation; they are a rebranding exercise. The test is not what the team is called. It is whether SLOs are defined, error budgets are enforced, toil is tracked, and postmortems produce automated improvements.
The Tooling-First antipattern → Deploying Kubernetes, Argo CD, Prometheus, and Grafana and concluding that SRE infrastructure is in place. The tooling is necessary but not sufficient. An organisation with the full SRE toolstack and no SLOs, no error budget policy, and no postmortem culture has purchased SRE infrastructure without adopting SRE practice.
The Phase Skip antipattern → Attempting Phase 2 (self-service platform) before Phase 1 (measurement and accountability) is complete. A self-service SLO platform with no development team adoption (because error budgets are not yet enforced) builds infrastructure that nobody uses. Each phase's exit gates exist precisely to prevent this.
The Eternal Phase 0 antipattern → Remaining in Phase 0 indefinitely because the measurement baseline is never "good enough" to justify enforcement. Phase 0 exit gates are pass/fail, not optimisation targets. A team that has three services instrumented, six months of DORA data, and at least one non-SRE stakeholder engaged with the data has met the Phase 0 exit gates — regardless of whether the data looks as clean as the team would like.
The SRE Team as Elite Ops antipattern → Building an SRE team that responds to all production incidents, writes all postmortems, and owns all reliability decisions. This model does not scale and does not produce the Phase 2 and Phase 3 outcomes where development teams own their own reliability. The SRE team's goal is to work itself out of the reactive operations business, not to optimise its ability to perform it.

Maturity Progression

────────────────────────────────────────────────────────────────────────────
PHASE    NORTH STAR SIGNAL                   MOST COMMON STALL POINT
────────────────────────────────────────────────────────────────────────────
Phase 0  Non-engineering stakeholder         Skipping toil inventory;
         cites SRE data without              SLO defined but not
         prompting.                          validated with team.

Phase 1  First error budget gate fires       Gate overridden without
         and is upheld.                      documentation; override
                                             rate > 20%.

Phase 2  Development team requests           Platform exists but
         SLO for new service without         development teams don't
         SRE involvement.                    know how to use it.

Phase 3  DORA report requested by            Toil Ratio begins rising
         leadership without SRE              as compliance obligations
         initiating it.                      grow; automation debt
                                             accumulates.
────────────────────────────────────────────────────────────────────────────

Five Action Items for This Week

Assess which phase your organisation is genuinely in using the exit gate criteria. Not the phase you aspire to — the phase the evidence supports. Review each Phase 0 exit gate. If any gate is not satisfied, Phase 0 is not complete, and Phase 1 activities will not produce their intended outcomes.
Identify the operating model your current SRE team uses and whether it matches the appropriate model for your current phase. If you are in Phase 0 but using a Platform Model (where development teams are expected to self-serve), the model is ahead of the practice maturity. Align the operating model to the phase.
Document the DORA baseline if you do not have it. Six months of deployment frequency, lead time, CFR, and MTTR data from actual pipeline and incident records. This is the Phase 0 exit gate that most organisations are missing — not because the data doesn't exist, but because it has not been extracted and presented.
Define the Phase 1 success metrics for your current pilot service before you activate the error budget gate. What does success look like at the end of three months? Pre-committed success criteria prevent the outcome from being interpreted in whatever direction is politically convenient after the fact.
Identify the first development team for Phase 2 self-service SLO adoption. Which team has the highest SRE maturity, the most engaged team lead, and the least regulatory complexity? That team is your Phase 2 pilot. Designing the self-service platform for that team's specific workflow is the starting point for a platform that will eventually serve all teams.

"The journey from DevOps to SRE is not a journey from bad practices to good ones. DevOps practices are genuinely good — they are the prerequisite for what comes after. The journey is from practices that improve delivery to practices that also govern the production system those deliveries create. The organisations that have made it to Phase 3 did not skip the earlier phases. They completed them — slowly, imperfectly, with setbacks — and built on them. There is no shortcut. There is only the sequence."

Teaching SRE in Resistant Organisations: A Phased Influence Playbook for Practitioners

Nijo George Payyappilly — Mon, 14 Sep 2026 16:00:00 +0000

In the autumn of 2019, an SRE practitioner joined a mid-size financial services firm as their first dedicated reliability engineer. Within six weeks, they had drafted a comprehensive SRE adoption roadmap: error budget policies, SLO frameworks, on-call rotation design, postmortem processes, toil reduction programmes. The documents were detailed, well-researched, and entirely aligned with Google SRE practice. They were presented to the VP of Engineering in a forty-minute session.

The VP thanked them for the work. The roadmap was filed. Nothing changed.

Eighteen months later, after a significant production incident that the SRE's observability stack had predicted three hours before it occurred — and whose postmortem the SRE wrote, facilitated, and tracked to completion — the same VP approached the SRE with a question: "Can you help us do this for all of our services?"

The difference between the forty-minute presentation and the VP's question was not the quality of the SRE practices being proposed. It was the sequence in which the SRE's authority to propose them had been established. The roadmap arrived before the credibility. The question arrived after eighteen months of demonstrated value. Same practices. Radically different reception.

This is the fundamental insight behind the phased influence playbook: in resistant organisations, the sequence of value demonstration matters as much as the quality of what is being demonstrated. This post documents that sequence.

Why Organisations Resist

Before designing an influence strategy, the nature of the resistance must be understood. Most SRE practitioners in resistant organisations misdiagnose resistance as irrationality — as leadership failure, as bureaucratic inertia, as cultural backwardness. This diagnosis produces the wrong response: more comprehensive proposals, more persuasive presentations, more detailed roadmaps.

The resistance in most regulated enterprises is rational. Leadership has seen many technology transformation initiatives that promised operational improvement and delivered disruption, cost overruns, and staff attrition. Change management processes were built to slow down exactly the kind of rapid, practitioner-driven change that SRE adoption requires. Risk aversion is not a cultural defect — it is a rational response to regulatory environments where the cost of change-caused incidents includes regulatory fines, audit findings, and reputational damage that dwarf any efficiency gain.

The implication is that SRE adoption in resistant organisations cannot be sold. It must be demonstrated. And the demonstration must be structured to make its value legible to the specific stakeholders who control the resources and authority that SRE adoption requires.

────────────────────────────────────────────────────────────────────────────
STAKEHOLDER INFLUENCE MAP

STAKEHOLDER         WHAT THEY CARE ABOUT          SRE VALUE TO DEMONSTRATE
────────────────────────────────────────────────────────────────────────────
Engineering VPs     Delivery velocity             Deployment frequency
                    Team capacity                 Toil reduction (hours saved)
                    Incident reduction            MTTR improvement

Operations Leaders  Stability                     Incident rate reduction
                    On-call sustainability        Alert quality (volume down)
                    Team attrition risk           On-call burden reduced

Compliance/Risk     Audit findings                Automated evidence generation
                    Regulatory exposure           Error budget = risk appetite
                    Change control                GitOps = change audit trail

Finance/CFO         Cost of incidents             Downtime cost quantified
                    Infrastructure cost           SOT-based right-sizing savings
                    Headcount efficiency          Toil hours → engineering hours

Product             Feature velocity              Error budget → deploy faster
                    SLA commitments               SLO → customer commitment
                    Market competitiveness        DORA metrics vs. industry

────────────────────────────────────────────────────────────────────────────
CRITICAL INSIGHT: Present the same SRE practice differently to each.
Error budget to compliance = risk appetite framework
Error budget to product = release velocity governor
Error budget to finance = quantified incident cost ceiling
The practice is identical; the vocabulary is tailored.
────────────────────────────────────────────────────────────────────────────

The Five-Phase Playbook

Phase 0 — Establish a Measurement Baseline (Weeks 1–8)

Phase 0 is the most counterintuitive phase because it involves doing nothing that looks like SRE adoption. Its entire purpose is to establish the measurement infrastructure that will make every subsequent phase's value visible.

────────────────────────────────────────────────────────────────────────────
PHASE 0 ACTIVITIES AND RATIONALE

ACTIVITY 1: DORA Metric Baselining
  Action: Instrument deployment frequency, lead time, CFR, and MTTR
          from existing CI/CD and incident management data.
          Do not propose changes; only measure current state.
  Rationale: The baseline is the before-picture that makes future
             improvement legible. SRE improvements without a baseline
             cannot be attributed to SRE.
  Artefact: DORA baseline report (internal only; do not present yet)

ACTIVITY 2: Toil Inventory
  Action: Run the recurring-incident Splunk query. Classify every
          on-call event from the last 90 days as toil or engineering.
          Quantify toil in hours per sprint.
  Rationale: Toil hours are the currency of the capacity conversation.
             "We spend 44% of SRE time on toil" is actionable.
             "We have a lot of manual work" is not.
  Artefact: Toil inventory spreadsheet (internal only)

ACTIVITY 3: Deployment Correlation Setup
  Action: Configure the Argo CD sync event log and incident data to
          produce a deployment correlation view: deployments plotted
          against incident rate over the last 90 days.
  Rationale: This artefact will be the Phase 1 leadership conversation
             entry point. It must be ready before that conversation
             is attempted.
  Artefact: Deployment correlation dashboard (ready but not presented)

PHASE 0 EXIT CRITERIA:
  ✓ DORA baseline documented for at least one service
  ✓ Toil inventory complete and categorised
  ✓ Deployment correlation dashboard operational
  ✓ No SRE adoption proposals made to leadership yet

PHASE 0 COMMON FAILURE MODE:
  Skipping Phase 0 to move directly to Phase 1.
  Result: Phase 1 conversations happen without data.
  The stakeholder asks "what's the evidence?" and the practitioner
  has no answer. The conversation closes. Credibility is harder
  to recover than it would have been to establish.
────────────────────────────────────────────────────────────────────────────

Phase 1 — Solve Visible Pain (Weeks 8–20)

Phase 1 is the earning phase. The practitioner identifies the pain that already exists, articulates it in data, and solves it — without proposing SRE adoption, without requesting additional authority, and without naming what they are doing as anything other than solving the problem.

────────────────────────────────────────────────────────────────────────────
PHASE 1 ACTIVITIES

THE DEPLOYMENT CORRELATION PRESENTATION:
  Target audience: Engineering VP or equivalent
  Duration: 20 minutes (no more)
  Content: Show the deployment correlation dashboard.
           Ask: "Does this pattern match your experience?"
           Listen. Do not present a solution yet.
  Purpose: Establish that you have data. Establish that you are
           solving the right problem. Let the stakeholder name
           the consequence before you propose the intervention.

THE FIRST AUTOMATION INVESTMENT:
  Select the highest-ROI toil item from the Phase 0 inventory.
  Build the automation. Ship it.
  Quantify the hours saved per sprint.
  Report it informally — not as an SRE initiative, but as
  "I automated the X process; we got back N hours per sprint."

THE FIRST SLO (SINGLE SERVICE):
  Define one SLI and one SLO for the service with the most
  visible reliability pain. Do not enforce an error budget yet.
  Simply make the SLO visible: "Here is the target. Here is where
  we currently are. Here is the trend."
  Purpose: Creates the vocabulary before the governance conversation.

PHASE 1 EXIT CRITERIA:
  ✓ At least one automation shipped with quantified hours saved
  ✓ Deployment correlation dashboard presented and acknowledged
  ✓ One SLO visible to engineering team
  ✓ At least one stakeholder has asked "can we do this more broadly?"
    (This question is the Phase 1 success signal)

PHASE 1 TIMELINE REALITY:
  Phase 1 takes 3–6 months in most regulated enterprises.
  The temptation to rush it — to present the full roadmap while
  the stakeholder is receptive — is the most common adoption failure.
  The question "can we do this more broadly?" is an invitation
  to demonstrate, not to propose. The demonstration comes in Phase 2.
────────────────────────────────────────────────────────────────────────────

Phase 2 — Create Visible Artefacts (Weeks 20–32)

Phase 2 scales the evidence base. The artefacts produced in Phase 1 (one automation, one SLO) are replicated across more services and made visible to a wider audience. The goal is to create the organisational conversation about reliability before proposing its governance.

────────────────────────────────────────────────────────────────────────────
PHASE 2 ARTEFACTS

ARTEFACT 1: DORA QUARTERLY REPORT
  Content: DORA Four metrics for all tracked services, trend lines,
           comparison to industry benchmarks.
  Audience: Engineering VPs, optionally CTO.
  Format: One-page executive summary + detail appendix.
  Purpose: Establishes DORA as the measurement vocabulary before
           proposing DORA-based governance.
  Cadence: Quarterly. Consistency matters more than single-point data.

ARTEFACT 2: TOIL REDUCTION REPORT
  Content: Hours saved this quarter through automation; categories
           eliminated; Toil Ratio trend.
  Audience: Engineering VP + Operations leadership.
  Format: Half-page summary with hours and business impact.
  Purpose: Quantifies the ROI of the reliability engineering investment
           already made — making the case for more investment implicitly.

ARTEFACT 3: INCIDENT TREND REPORT
  Content: Incident rate trend over the past 6 months; correlation with
           deployment events; top-3 recurrence categories.
  Audience: Operations leadership + (optionally) Risk/Compliance.
  Format: One-page with trend chart.
  Purpose: Surfaces the recurrence pattern that the error budget policy
           will address — without naming the error budget policy yet.

ARTEFACT 4: SLO COVERAGE MAP
  Content: Which services have SLOs defined; what the current vs. target
           reliability is for each; which services have no SLO.
  Audience: Engineering leadership.
  Purpose: Makes the measurement gap visible; creates demand for coverage.

PHASE 2 SUCCESS SIGNALS:
  → DORA report is circulated beyond engineering team
  → Operations leadership asks for the incident trend data regularly
  → Someone outside the SRE function references the SLO data in a meeting
  → Leadership mentions SRE in a context the practitioner did not initiate

PHASE 2 EXIT CRITERIA:
  ✓ DORA report presented twice (two quarters of data)
  ✓ Toil reduction quantified and reported
  ✓ At least 5 services have SLOs visible
  ✓ At least one non-engineering stakeholder is engaged with the data
────────────────────────────────────────────────────────────────────────────

Phase 3 — Earn the Governance Conversation (Weeks 32–44)

Phase 3 is where the error budget policy, the deployment gate, and the formal SRE governance framework are proposed — for the first time, having never been mentioned before. The timing is critical: this conversation happens only after Phase 2 has created the data and the stakeholder relationships that make the governance proposal land as a logical next step rather than an unsolicited transformation initiative.

────────────────────────────────────────────────────────────────────────────
PHASE 3: THE GOVERNANCE CONVERSATION

PREREQUISITES (all must be true):
  ✓ Phase 2 artefacts have been presented at least twice
  ✓ At least one non-engineering stakeholder is engaged
  ✓ The deployment correlation pattern is accepted as real
  ✓ The DORA baseline is documented and communicated

THE CONVERSATION STRUCTURE:
  1. Reference the data: "Over the last six months, we've been tracking..."
  2. Name the pattern: "Deployments are the leading correlation with
     incident rate. This is what the error budget addresses."
  3. Propose the pilot: "I'd like to implement error budget policy for
     one service, with the full team aware of the gate. Not for all
     services — just one, for one quarter."
  4. Define success: "At the end of the quarter, we'll have data on
     whether the gate changed our deployment outcomes."
  5. Name the authority needed: "This requires the CAB to accept
     the error budget state as a factor in change risk assessment."

WHAT TO PROPOSE (specific, limited, reversible):
  ✓ Error budget policy for ONE service (the one with the most data)
  ✓ 90-day pilot with defined success criteria
  ✓ Opt-in participation from the development team that owns the service
  ✓ Override authority clearly documented

WHAT NOT TO PROPOSE:
  ✗ Organisation-wide SRE transformation
  ✗ Restructuring of operations teams
  ✗ Mandatory on-call rotation changes
  ✗ Any change that requires executive approval above VP level

PHASE 3 FAILURE MODES:
  Proposing too broadly: "We need to implement error budgets for all
    production services" → rejected as too disruptive
  Proposing without prerequisites: Phase 3 without Phase 2 data
    → no evidence base → stakeholder cannot evaluate the proposal
  Proposing the process before the outcome: describing the error
    budget mechanism before establishing why the mechanism is needed
    → answer to a question that has not been asked
────────────────────────────────────────────────────────────────────────────

Phase 4 — Pilot (Weeks 44–56+)

The pilot is the evidence factory. Its purpose is not to prove that SRE works in general — the Google SRE Book does that. Its purpose is to prove that SRE works in this organisation, with these teams, in this regulatory environment, on this infrastructure.

────────────────────────────────────────────────────────────────────────────
PHASE 4: PILOT DESIGN AND EXECUTION

PILOT SERVICE SELECTION CRITERIA:
  ✓ Team that opted in (not assigned)
  ✓ Sufficient historical incident data for comparison
  ✓ Not in a regulated namespace (reduces compliance complexity)
  ✓ Development team has deployment authority (change gate is meaningful)
  ✗ Not the most critical production service
  ✗ Not a service with no previous incidents (nothing to improve)

PILOT SUCCESS METRICS:
  Primary: Did deployment-caused incidents decrease during the pilot?
  Secondary: Did MTTR improve?
  Secondary: Did error budget policy changes modify developer behaviour?
  Counter-metric: Did delivery velocity decrease? (Should not)
  Qualitative: Did the development team find the policy useful or disruptive?

PILOT DOCUMENTATION:
  Week 2: Initial SLO calibration meeting notes
  Week 6: Mid-pilot data review (internal)
  Week 12: Pilot results report
    → DORA delta vs. pre-pilot baseline
    → Error budget consumption trend
    → Developer team feedback (survey)
    → Recommendation: expand, modify, or discontinue

WHAT MAKES PILOTS FAIL:
  → Service selected has no deployment authority (gate is decorative)
  → Team was assigned, not opted in (compliance without engagement)
  → Success metrics not defined before pilot starts
    (results can be interpreted in any direction without pre-committed criteria)
  → Pilot runs during an unrelated major incident that distorts the data
────────────────────────────────────────────────────────────────────────────

Phase 5 — Scale from Evidence (Weeks 56+)

Phase 5 begins only after the pilot has produced data. The expansion proposal is not "let's do SRE everywhere" — it is "the pilot showed X% reduction in deployment-caused incidents; here is the proposal to replicate that result for the next three services."

Each Phase 5 expansion is a smaller version of the full five-phase sequence: new evidence gathered, new artefacts created, new governance conversations earned, new pilots designed. The pace is determined by evidence, not by enthusiasm.

Organisational Archetypes and Adaptation

The five-phase sequence is the general framework. Each organisation has specific characteristics that require adaptation.

────────────────────────────────────────────────────────────────────────────
ORGANISATIONAL ARCHETYPES

ARCHETYPE 1: THE CHANGE-BOARD-HEAVY ENTERPRISE
  Characteristic: Every production change requires CAB approval.
                  Deployment frequency is monthly at best.
  Adaptation:     Phase 1 entry point is the deployment correlation
                  dashboard presented to the CAB chair, not Engineering VP.
                  Frame: "We can help the CAB make better risk decisions."
                  Phase 3 governance conversation targets CAB process
                  reform before error budget enforcement.

ARCHETYPE 2: THE SILOED-OPERATIONS ORGANISATION
  Characteristic: Development and operations teams have no shared
                  metrics, no shared on-call, no shared accountability.
  Adaptation:     Phase 1 focuses on a single service where the
                  development team has expressed readiness for joint
                  accountability. Do not attempt cross-silo conversation
                  before cross-silo relationships exist.
                  Phase 3 proposes joint SLO ownership, not on-call merge.

ARCHETYPE 3: THE COMPLIANCE-FIRST CULTURE
  Characteristic: Every initiative is evaluated through the compliance
                  function's lens first. Risk aversion is structural.
  Adaptation:     Phase 2 artefacts include compliance vocabulary.
                  Error budget = risk appetite framework.
                  GitOps audit trail = automated compliance evidence.
                  Phase 3 governance conversation happens with compliance
                  present, not around them.

ARCHETYPE 4: THE INCIDENT-DRIVEN OPPORTUNITY
  Characteristic: A major incident has created executive visibility
                  for the reliability problem.
  Adaptation:     Phase 0 and 1 can be compressed if the incident
                  postmortem provides the data that Phase 0 would have
                  produced. The post-incident window is the highest-
                  receptivity moment for Phase 3 conversations.
                  Warning: this window closes within 6–8 weeks.
                  Have Phase 3 ready before using the incident as leverage.
────────────────────────────────────────────────────────────────────────────

Common Antipatterns

The Roadmap Presentation antipattern → Presenting a comprehensive SRE transformation roadmap before establishing credibility through demonstrated value. The roadmap document is evidence of investment in thinking. It is not evidence of value delivered. Stakeholders in resistant organisations cannot evaluate a roadmap they have no basis to trust. The roadmap arrives in Phase 5 as documentation of what already worked, not in Phase 0 as a proposal for what might.
The Authority Before Credibility antipattern → Requesting authority (over deployments, over on-call structure, over change management) before demonstrating the value that would justify the authority. Authority granted without demonstrated value is resented and will be revoked after the first controversy. Authority granted after demonstrated value is protected by the stakeholders who experienced that value.
The Single-Service Bypass antipattern → Implementing a successful pilot on one service and then proposing organisation-wide adoption without a second pilot. One data point is evidence of possibility. Two data points from different teams are evidence of repeatability. Repeatability is what converts a pilot result into an organisational programme.
The Compliance-Hostile Framing antipattern → Presenting error budget policies, deployment gates, and change management reforms as replacements for existing compliance processes rather than improvements to them. In regulated enterprises, compliance functions have veto authority over operational changes. Proposals that are perceived as threatening compliance governance will be vetoed. Frame SRE practices as tools that make compliance more effective, not as alternatives to it.
The Wrong Room antipattern → Presenting the deployment correlation dashboard to the engineering team (who already believe deployments cause incidents) rather than to the Engineering VP (who controls the resources to address the pattern). The artefacts are not for the team that already understands the problem. They are for the stakeholder who controls the investment that would address it.

Maturity Progression

────────────────────────────────────────────────────────────────────────────
STAGE        INFLUENCE MATURITY                 NORTH STAR SIGNAL
────────────────────────────────────────────────────────────────────────────
Reactive     SRE proposals rejected.            "We tried to implement
             No data. No artefacts.             SRE but leadership
             Influence approach:                didn't support it."
             persuasion by conviction.

Defined      Phase 0 and 1 complete.            Leadership references
             DORA baseline exists.              deployment correlation
             First automation shipped.          data without prompting.
             Deployment correlation             Toil reduction hours
             presented.                         tracked quarterly.

Measured     Phase 2 artefacts produced         Non-engineering
             consistently. DORA quarterly       stakeholder engaged.
             report circulated.                 Phase 3 conversation
             Compliance vocabulary              had and pilot proposed.
             established.

Optimised    Pilot complete with positive       Second service
             results. Phase 5 expansion         expansion underway.
             underway. CAB process              CAB uses error budget
             reformed. Error budget             state in risk assessment.
             policy active for pilot
             service.

Generative   SRE governance embedded in         Engineering leaders
             organisational processes.          trained in SRE.
             Compliance function is             Platform team abstracts
             an SRE ally. Multiple             reliability. External
             teams own their SLOs.             recognition of programme.
────────────────────────────────────────────────────────────────────────────

Five Action Items for This Week

Identify which phase you are currently in. Not which phase you want to be in — which phase the evidence supports. If you have not yet presented the deployment correlation dashboard to the Engineering VP, you are in Phase 0 or Phase 1 regardless of what you have built. Accurate phase assessment is the prerequisite for the right next action.
Run the Phase 0 toil inventory using the Splunk recurring-incident query. Classify each output item as toil or engineering work. Calculate your current Toil Ratio. This number is your Phase 2 artefact starting point and your Phase 1 automation investment ROI metric.
Build the deployment correlation dashboard if you have not already. Argo CD sync events plotted against incident rate over 90 days. This is the single highest-conversion artefact in the playbook — the one that most reliably opens the governance conversation. It should be ready before any Phase 1 conversation with leadership.
Identify the single service that would make the best Phase 4 pilot. Apply the selection criteria: opted-in development team, deployment authority, not a regulated namespace, historical incident data available. Name it. Having the pilot service identified means the Phase 3 governance conversation has a concrete proposal, not a conceptual one.
Audit your last three SRE adoption conversations for timing errors. Did you propose a governance change before demonstrating the value it was designed to protect? Did you present a roadmap before establishing credibility? Did you request authority before earning it? The timing errors in past conversations are the map for correcting the sequence in the next ones.

"SRE adoption in resistant organisations fails when practitioners treat resistance as an obstacle to overcome rather than as information about the sequence in which trust must be established. The five-phase playbook is not a political strategy for working around resistant leadership. It is an engineering approach to the influence problem: instrument first, demonstrate value second, propose governance third. The sequence is not optional. It is why some SRE programmes grow and others stall."

On-Premises LLM Gateways for SRE: Architecture, Security, and Compliance Considerations

Nijo George Payyappilly — Mon, 07 Sep 2026 16:00:00 +0000

In February 2024, a major financial services firm evaluating an AIOps platform for incident response discovered that the vendor's root cause analysis feature was sending structured incident telemetry — including service names, error messages, environment identifiers, and deployment metadata — to an external LLM API for processing. The telemetry did not contain customer data. It did not contain financial transactions. But it contained the operational topology of the firm's production infrastructure: which services existed, how they were connected, and how they failed.

The firm's Chief Information Security Officer terminated the evaluation the same day. Not because the data was classified. But because the operational topology of a systemically important financial institution is precisely the intelligence that a sophisticated adversary needs to plan a targeted attack — and sending it to an external service was a risk that the firm's threat model did not permit, regardless of the vendor's contractual data handling commitments.

This is the data sovereignty problem in AI-assisted SRE operations. It is not hypothetical in regulated environments. It is the default constraint that determines whether external LLM services are even in scope for evaluation. In healthcare environments subject to HIPAA, production telemetry may contain PHI in log messages. In financial services environments subject to PCI-DSS, incident data may reference cardholder systems. In energy sector environments subject to NERC CIP, operational topology data may reveal critical infrastructure attack surfaces.

The on-premises LLM gateway architecture is the engineering response to this constraint. It preserves the operational value of AI-assisted incident response while maintaining data sovereignty over the telemetry that enables it.

Architecture Overview: The Gateway Pattern

The gateway pattern centralises all LLM access behind a single proxy layer that enforces data classification, applies model routing logic, maintains the audit trail, and provides the fallback chain that keeps AI-assisted operations available when any individual model backend is degraded.

The critical design decision is the placement of the data classification gate before the LiteLLM Proxy routing decision. Classification cannot be delegated to the proxy itself because the proxy does not have the context to understand whether a Kubernetes event log message contains PHI, PCI-relevant data, or NERC CIP-sensitive operational topology. The classification logic must run in the HolmesGPT context gathering layer, where the telemetry content is still structured and inspectable.

Data Classification Gate

The data classification gate is the mechanism that prevents regulated data from being routed to external model endpoints. It operates on the structured context package that HolmesGPT assembles before making an LLM request.

# HolmesGPT Data Classification Configuration
# Defines the rules that determine which model backend receives each request

apiVersion: v1
kind: ConfigMap
metadata:
  name: holmesgpt-data-classification
  namespace: holmesgpt
  annotations:
    sre.internal/policy-version: "v1.2"
    sre.internal/approved-by: "ciso,sre-lead"
data:
  classification_rules.yaml: |
    # REGULATED: must route to on-premises Ollama only
    regulated_indicators:
      namespace_patterns:
        - "pci-zone"
        - "hipaa-zone"
        - "nerc-cip-zone"
        - "tier-0-clinical"
        - "tier-1-clinical"
      label_patterns:
        - "compliance.internal/regulated=true"
        - "data-classification=restricted"
      log_content_patterns:
        # PHI indicators in log messages
        - "patient_id"
        - "mrn"
        - "ssn"
        - "account_number"
        # PCI indicators
        - "card_number"
        - "cvv"
        - "pan"
        # NERC CIP indicators
        - "substation"
        - "scada"
        - "ems_topology"
      incident_service_patterns:
        - "payments-core"
        - "cardholder-data"
        - "clinical-decision"

    # STANDARD: may route to external models
    # (telemetry contains no regulated data)
    standard_indicators:
      default_when_no_regulated_match: true
      external_model_allowed: true

    routing:
      regulated: "ollama-primary"
      standard: "github-models-primary"
      fallback: "ollama-primary"

# Data Classification Gate — Implementation (HolmesGPT pre-routing hook)
# Classifies assembled context before LiteLLM routing decision

import re
from typing import Literal

DataClass = Literal["REGULATED", "STANDARD"]

def classify_context(context: dict) -> DataClass:
    """
    Classify assembled incident context before LLM routing.
    Returns REGULATED if any indicator matches; STANDARD otherwise.
    """
    rules = load_classification_rules()

    # Check namespace of affected services
    for service in context.get("affected_services", []):
        namespace = service.get("namespace", "")
        for pattern in rules["regulated_indicators"]["namespace_patterns"]:
            if pattern in namespace:
                return "REGULATED"

    # Check labels on affected resources
    for resource in context.get("kubernetes_resources", []):
        labels = resource.get("labels", {})
        for label_pattern in rules["regulated_indicators"]["label_patterns"]:
            key, value = label_pattern.split("=")
            if labels.get(key) == value:
                return "REGULATED"

    # Check log content for regulated data indicators
    log_content = " ".join(context.get("log_snippets", []))
    for pattern in rules["regulated_indicators"]["log_content_patterns"]:
        if re.search(pattern, log_content, re.IGNORECASE):
            return "REGULATED"

    return "STANDARD"

LiteLLM Proxy Configuration

The LiteLLM Proxy is the unified gateway layer. It receives requests from HolmesGPT with a data_classification header set by the classification gate, applies the routing logic, manages fallback chains, enforces rate limits, and writes to the audit log.

# LiteLLM Proxy — Full Production Configuration
# Deployed on Shared Services TKGs cluster (ai-ops namespace)
# All AI workloads route through this gateway

model_list:

  # ── ON-PREMISES MODELS (Ollama) ────────────────────────────────────────
  # Primary for REGULATED data; fallback for all workloads

  - model_name: ollama-primary
    litellm_params:
      model: ollama/qwen2.5:14b
      api_base: http://ollama.ai-ops.svc.cluster.local:11434
      timeout: 120
      max_tokens: 8192
      # qwen2.5:14b selection rationale:
      # - 14B parameters: sufficient for multi-service incident correlation
      # - Strong multilingual capability for diverse log formats
      # - Quantised to Q4_K_M: fits in 10GB VRAM on GPU node
      # - Benchmark: 78% accuracy on SRE incident diagnosis test set

  - model_name: ollama-large
    litellm_params:
      model: ollama/qwen2.5:72b
      api_base: http://ollama-large.ai-ops.svc.cluster.local:11434
      timeout: 300
      max_tokens: 16384
      # 72B for complex multi-cluster incidents requiring larger context
      # GPU requirement: 48GB VRAM (A100 or equivalent)

  # ── EXTERNAL MODELS (GitHub Models) ─────────────────────────────────
  # Primary for STANDARD data; higher capability for complex cases

  - model_name: github-models-primary
    litellm_params:
      model: openai/gpt-4.1
      api_base: https://models.inference.ai.azure.com
      api_key: os.environ/GITHUB_MODELS_PAT
      timeout: 60
      max_tokens: 16384
      # gpt-4.1 for production RCA: highest accuracy on complex incidents

  - model_name: github-models-mini
    litellm_params:
      model: openai/gpt-4.1-mini
      api_base: https://models.inference.ai.azure.com
      api_key: os.environ/GITHUB_MODELS_PAT
      timeout: 30
      max_tokens: 4096
      # gpt-4.1-mini for high-volume alert triage: lower latency, lower cost

router_settings:
  routing_strategy: custom
  model_group_alias:
    "regulated-routing":
      - ollama-primary
      - ollama-large
    "standard-routing":
      - github-models-primary
      - ollama-primary    # Fallback if GitHub Models unavailable
    "triage-routing":
      - github-models-mini
      - ollama-primary

  # Custom routing function based on data_classification header
  routing_logic: |
    classification = request.headers.get("X-Data-Classification", "STANDARD")
    urgency = request.headers.get("X-Incident-Urgency", "standard")

    if classification == "REGULATED":
        return "ollama-primary"          # Never leaves network boundary

    if urgency == "triage":
        return "github-models-mini"      # Fast triage; lower capability

    return "github-models-primary"       # Full capability for RCA

  fallback_model: "ollama-primary"       # Always falls back to on-premises
  fallback_on_status_codes: [429, 500, 502, 503, 504]
  num_retries: 2
  request_timeout: 180

# ── RATE LIMITING ──────────────────────────────────────────────────────
litellm_settings:
  max_budget: 500.0        # Monthly GitHub Models budget cap (USD)
  budget_duration: "1mo"
  max_parallel_requests: 10

  # Per-model rate limits
  model_max_parallel_requests:
    ollama-primary: 5        # Constrained by Ollama throughput
    github-models-primary: 10
    github-models-mini: 20

# ── AUDIT LOGGING ──────────────────────────────────────────────────────
  success_callback: ["splunk"]
  failure_callback: ["splunk"]

  splunk_host: "https://splunk.internal:8088"
  splunk_token: os.environ/SPLUNK_HEC_TOKEN
  splunk_index: "aiops_gateway"

  # Logged fields per request:
  # request_id, timestamp, model_used, routing_decision,
  # data_classification, incident_id, tokens_used, latency_ms,
  # success/failure, fallback_triggered

Multi-Cluster Deployment via Argo CD ApplicationSet

HolmesGPT is deployed per production cluster — each cluster gets its own HolmesGPT instance that analyses incidents in that cluster's context. All instances route through the centralised LiteLLM Proxy on the shared services cluster.

# Argo CD ApplicationSet — HolmesGPT per TKGs Cluster
# Deploys HolmesGPT to every cluster registered in Argo CD
# Each instance connects to the shared LiteLLM Proxy

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: holmesgpt-per-cluster
  namespace: argocd
spec:
  generators:
    - clusters:
        selector:
          matchLabels:
            sre.internal/holmesgpt-enabled: "true"

  template:
    metadata:
      name: "holmesgpt-{{name}}"
      annotations:
        notifications.argoproj.io/subscribe.on-sync-failed.slack: "sre-aiops-alerts"
    spec:
      project: sre-platform
      source:
        repoURL: https://git.internal/sre/holmesgpt-config
        targetRevision: main
        path: clusters/holmesgpt
        helm:
          values: |
            cluster:
              name: "{{name}}"
              environment: "{{metadata.labels.environment}}"

            litellmProxy:
              url: "http://litellm-proxy.ai-ops.sre-shared.svc.cluster.local:4000"
              # Cross-cluster routing: HolmesGPT on prod cluster routes to
              # LiteLLM Proxy on shared services cluster via Istio ServiceEntry

            holmesgpt:
              escalationPolicyConfigMap: "holmesgpt-escalation-policy"
              dataClassificationConfigMap: "holmesgpt-data-classification"
              auditSplunkIndex: "holmesgpt_{{name}}"
              actionAuthority: "none"   # Override per-cluster based on trust level

      destination:
        server: "{{server}}"
        namespace: holmesgpt

      syncPolicy:
        automated:
          prune: true
          selfHeal: true

# Istio ServiceEntry — Cross-Cluster LiteLLM Proxy Access
# Allows HolmesGPT on production clusters to reach LiteLLM Proxy
# on shared services cluster via mTLS-secured cross-cluster communication

apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: litellm-proxy-shared-services
  namespace: holmesgpt
spec:
  hosts:
    - litellm-proxy.ai-ops.sre-shared.svc.cluster.local
  ports:
    - number: 4000
      name: http
      protocol: HTTP
  location: MESH_EXTERNAL
  resolution: DNS
  endpoints:
    - address: litellm-proxy.shared-services.internal
      ports:
        http: 4000

Kyverno Governance for the AI Ops Layer

The AI gateway is a production service with production governance requirements. Kyverno policies enforce the operational standards that prevent AI-ops components from becoming a governance gap in the platform.

# Kyverno: AI-Ops Governance Policies

# Policy 1: All AI-ops deployments must have data classification annotation
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: ai-ops-data-classification-required
spec:
  validationFailureAction: Enforce
  rules:
    - name: require-data-classification-annotation
      match:
        any:
          - resources:
              kinds: [Deployment]
              namespaces: [holmesgpt, ai-ops, litellm]
      validate:
        message: >
          AI-ops deployments must declare data classification policy.
          Add annotation: sre.internal/data-classification-policy=<configmap-name>
        pattern:
          metadata:
            annotations:
              sre.internal/data-classification-policy: "?*"

---
# Policy 2: LiteLLM Proxy must not allow external routing without
# an active data classification gate (enforces gateway pattern)
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: litellm-external-routing-gate-required
spec:
  validationFailureAction: Enforce
  rules:
    - name: require-classification-gate-for-external-models
      match:
        any:
          - resources:
              kinds: [ConfigMap]
              names: ["litellm-proxy-config"]
              namespaces: [ai-ops]
      validate:
        message: >
          LiteLLM Proxy config must declare data classification gate.
          External model routing without classification gate is prohibited.
        pattern:
          data:
            config.yaml: "*X-Data-Classification*"

---
# Policy 3: Ollama models must be on the approved model list
# Prevents unauthorised model deployment that bypasses governance
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: ollama-approved-models-only
spec:
  validationFailureAction: Enforce
  rules:
    - name: check-ollama-model-tag
      match:
        any:
          - resources:
              kinds: [Pod]
              namespaces: [ai-ops]
              selector:
                matchLabels:
                  app: ollama
      validate:
        message: >
          Only approved Ollama models permitted. Approved:
          qwen2.5:14b, qwen2.5:72b, llama3.1:8b, llama3.1:70b
          Submit model approval request to sre-aiops-governance.
        deny:
          conditions:
            all:
              - key: "{{ request.object.spec.containers[].env[?name=='OLLAMA_MODEL'].value | [0] }}"
                operator: AnyNotIn
                value:
                  - "qwen2.5:14b"
                  - "qwen2.5:72b"
                  - "llama3.1:8b"
                  - "llama3.1:70b"

SLO Design for the AI Gateway

The LiteLLM Proxy is a production dependency of the incident response workflow. It requires the same SLO treatment as any other production dependency: defined SLIs, error budgets, and degradation posture.

# Prometheus: LiteLLM Proxy SLIs
groups:
  - name: litellm.slo
    rules:

      # SLI 1: Request availability
      # Fraction of LLM requests returning a response within timeout
      - record: sli:litellm_availability:ratio_rate5m
        expr: |
          sum(rate(litellm_request_total{status="success"}[5m]))
          /
          sum(rate(litellm_request_total[5m]))

      # SLI 2: On-premises fallback reliability
      # When external models fail, Ollama must respond within 120s
      - record: sli:ollama_fallback_success:ratio_rate1h
        expr: |
          sum(rate(litellm_request_total{
            model=~"ollama.*",
            status="success",
            triggered_by_fallback="true"
          }[1h]))
          /
          sum(rate(litellm_request_total{
            triggered_by_fallback="true"
          }[1h]))

      # SLI 3: Routing compliance
      # Fraction of REGULATED requests correctly routed to Ollama (not external)
      - record: sli:regulated_routing_compliance:ratio_rate1h
        expr: |
          sum(rate(litellm_request_total{
            data_classification="REGULATED",
            model=~"ollama.*"
          }[1h]))
          /
          sum(rate(litellm_request_total{
            data_classification="REGULATED"
          }[1h]))

      # Alert: Routing compliance breach — REGULATED data sent to external model
      - alert: LiteLLM_RegulatedDataRoutingBreach
        expr: sli:regulated_routing_compliance:ratio_rate1h < 1.0
        for: 0s     # Immediate — no tolerance for compliance breach
        labels:
          severity: critical
          compliance: "true"
          regulatory_notification: "assess"
        annotations:
          summary: >
            CRITICAL: REGULATED data routed to external model.
            Data classification gate may have failed.
            Immediate investigation required.
          runbook: "https://wiki.internal/sre/runbooks/regulated-data-routing-breach"

      # SLO error budget: 99.5% availability for AI gateway
      # (lower than application services — AI is enhancing, not critical path)
      - record: slo:litellm_budget_remaining:ratio
        expr: |
          1 - (
            (1 - sli:litellm_availability:ratio_rate5m)
            / (1 - 0.995)
          )

Operational Considerations for Ollama in Production

Running local LLMs in production Kubernetes introduces infrastructure requirements that typical enterprise Kubernetes platforms are not provisioned for.

────────────────────────────────────────────────────────────────────────────
OLLAMA PRODUCTION INFRASTRUCTURE REQUIREMENTS

GPU NODE REQUIREMENTS:
  qwen2.5:14b (Q4_K_M quantisation):
    VRAM required: ~10 GB
    Suitable GPUs: NVIDIA A10G, RTX 3090, T4 (marginal)
    Inference throughput: ~30–50 tokens/second
    Concurrent requests: 1–2 (GPU memory limited)

  qwen2.5:72b (Q4_K_M quantisation):
    VRAM required: ~45 GB
    Suitable GPUs: NVIDIA A100 80GB (single), 2× A100 40GB (NVLink)
    Inference throughput: ~15–25 tokens/second
    Concurrent requests: 1 (memory constrained)

KUBERNETES NODE CONFIGURATION:
  Node label: sre.internal/gpu-class=ollama
  Taint: nvidia.com/gpu=present:NoSchedule
  GPU time-slicing: NOT recommended for LLM inference
    (VRAM is not sliceable; time-slicing adds context switch overhead)

OLLAMA KUBERNETES DEPLOYMENT:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ollama
  namespace: ai-ops
spec:
  replicas: 1    # Single replica; GPU memory constraint
  template:
    spec:
      nodeSelector:
        sre.internal/gpu-class: "ollama"
      tolerations:
        - key: "nvidia.com/gpu"
          operator: "Exists"
          effect: "NoSchedule"
      containers:
        - name: ollama
          image: ollama/ollama:latest
          resources:
            limits:
              nvidia.com/gpu: "1"
              memory: "32Gi"
            requests:
              nvidia.com/gpu: "1"
              memory: "16Gi"
          volumeMounts:
            - name: ollama-models
              mountPath: /root/.ollama
          env:
            - name: OLLAMA_MODELS
              value: "/root/.ollama/models"
            - name: OLLAMA_NUM_PARALLEL
              value: "2"      # Allow 2 concurrent requests (small models only)
            - name: OLLAMA_MAX_LOADED_MODELS
              value: "1"      # Keep only one model in VRAM at a time
      volumes:
        - name: ollama-models
          persistentVolumeClaim:
            claimName: ollama-models-pvc  # Models pre-loaded; not pulled at runtime

MODEL PRE-LOADING STRATEGY:
  Models must be pulled and cached before production deployment.
  Pulling at runtime introduces unacceptable cold-start latency.
  Solution: InitContainer or pre-population Job that pulls models
  into the PVC before Ollama deployment proceeds.
────────────────────────────────────────────────────────────────────────────

Common Antipatterns

The Classification-After-Routing antipattern → Applying data classification logic inside the LiteLLM Proxy routing function rather than in the HolmesGPT context layer. The proxy cannot inspect log content for PHI indicators — it sees the assembled request, not the source telemetry. Classification must happen before the request is constructed, not after.
The Fallback-to-External antipattern → Configuring external models as the fallback when Ollama is unavailable. The fallback chain must maintain data sovereignty: if the primary on-premises model fails, the fallback must also be on-premises. An external fallback for a REGULATED request is a compliance failure even if the primary model was correctly on-premises.
The Model-at-Runtime-Pull antipattern → Pulling Ollama models at container startup rather than pre-loading them into a persistent volume. Model pull times for 14B models are 5–15 minutes depending on network speed. An Ollama pod restart during an active incident creates a 15-minute AI assistance outage. Pre-load models into a PVC; treat model updates as planned changes with a change record.
The Single-Cluster Gateway antipattern → Deploying LiteLLM Proxy on a production cluster rather than on a dedicated shared services cluster. A LiteLLM Proxy running on the same cluster whose incidents it is analysing creates an operational dependency loop: when the production cluster is degraded (the scenario where AI assistance is most valuable), the AI gateway may also be degraded. Shared services cluster isolation is mandatory.
The Ungoverned Model antipattern → Allowing any Ollama model to be pulled and run without an approval process. Models have different capability levels, different licensing terms (some open-weights models have commercial use restrictions), and different privacy and security properties. The Kyverno approved-models policy enforces the approved list; the policy review process is the human governance layer that maintains it.

Maturity Progression

────────────────────────────────────────────────────────────────────────────
STAGE        AI GATEWAY MATURITY                NORTH STAR SIGNAL
────────────────────────────────────────────────────────────────────────────
Reactive     No AI-assisted operations.         All incident investigation
             External LLM APIs used             is manual. Or: external
             without data classification        LLM used without
             gate (compliance risk).            compliance review.

Defined      Gateway architecture               LiteLLM Proxy deployed.
             documented. Data                   Data classification gate
             classification rules               implemented. Ollama
             defined. Model list                running on GPU node.
             approved.

Measured     All four SLIs instrumented.        Routing compliance SLI
             Multi-cluster ApplicationSet        at 100%. Fallback
             deploying HolmesGPT.               reliability measured.
             Kyverno policies active.           No REGULATED data
                                                reaching external models.

Optimised    Shadow mode evaluation             PoC gates passed.
             complete (see AIOps               HolmesGPT operating
             Trustworthy post).                 at Level 1–2 autonomy.
             Model routing calibrated.          AI assistance reducing
             GPU utilisation optimised.         investigation time.

Generative   On-premises model fine-tuned       Investigation accuracy
             on internal incident corpus.       above 70% precision.
             Multi-model routing by             Regulated workloads
             incident type operational.         fully served by
             AI gateway SLO stable.             on-premises models.
────────────────────────────────────────────────────────────────────────────

Five Action Items for This Week

Audit every LLM API call currently made from your production environment and classify the data being sent. For each call: what telemetry is included in the prompt? Does that telemetry touch a regulated namespace, contain potential PHI indicators, or reference operational topology that your threat model considers sensitive? The audit output is your data classification rule set.
Deploy LiteLLM Proxy on your shared services cluster with Ollama as the only backend. Do not configure external models yet. Establish the on-premises routing baseline first — verify that Ollama is reachable, the SLI recording rules are working, and the audit log is flowing to Splunk. Add external model routing only after on-premises baseline is stable.
Pre-load your chosen Ollama model into a PVC before any production deployment. Pull the model in a test environment, measure the pull time, and design the pre-loading Job that populates the PVC before the Ollama deployment runs. Document the model version as a deployment dependency — the same way you version application images.
Implement the Kyverno regulated-routing compliance policy and run it in audit mode for two weeks before enforcing. Audit mode surfaces all requests that would be blocked by the policy without actually blocking them. Two weeks of audit mode data reveals whether the classification gate is correctly implemented before enforcement creates operational disruption.
Define the fallback chain explicitly: on-premises primary → on-premises secondary → human escalation. There is no "external model as fallback for regulated data" option. If both on-premises models are unavailable, the fallback is human incident investigation without AI assistance — which is the pre-AI baseline, not a failure. Document this explicitly in the escalation policy so that operators know what to expect when the AI gateway is degraded.

"The question of whether to use an external LLM for incident response in a regulated environment is not a question about model capability — it is a question about data sovereignty. The model that produces the best diagnosis is not useful if sending your operational telemetry to it violates your threat model or your regulatory obligations. The on-premises gateway is not a compromise on AI capability; it is the architecture that makes AI capability permissible in the environments that need it most."

Request-Rate-Based Autoscaling: Why CPU Metrics Lie and How to Fix Them

Nijo George Payyappilly — Mon, 31 Aug 2026 16:00:00 +0000

In January 2022, a major e-commerce platform experienced a latency SLO breach during a flash sale event that their autoscaling configuration should have handled. The service's CPU utilisation, measured across the fleet, was 61% when the first SLO alerts fired — well below the 70% scale-out threshold. The engineers watching the dashboards saw a service that their autoscaling model said had 9% of CPU headroom remaining. The users saw a service that was taking twelve seconds to add items to a cart.

The post-incident analysis identified the cause: the service's thread pool had saturated at 2,800 RPS per replica — the point at which all 200 threads were simultaneously occupied and new requests were queuing rather than being processed. The CPU utilisation at thread pool saturation was 61%, not 70%, because a significant fraction of the thread time was spent waiting for database responses rather than consuming CPU. The autoscaling trigger that was supposed to protect the latency SLO was measuring the wrong signal.

This is not an unusual incident. It is a structural failure mode of CPU-based autoscaling for I/O-bound web services, and it occurs with predictable regularity in production environments where CPU is the default metric. The failure mode does not look like a misconfiguration in the moment — it looks like a perfectly functioning autoscaling system that somehow failed to protect the SLO. This post explains mechanically why it fails, derives the correct alternative, and shows the migration.

The Four Ways CPU Metrics Lie

CPU utilisation is a reliable scaling signal for exactly one class of workload: CPU-bound compute jobs where CPU consumption scales linearly with throughput and there is no I/O wait, memory pressure, or thread management overhead. For the remaining workload classes — which includes the vast majority of web, API, and database workload types in production — CPU misleads the autoscaler in four distinct failure modes.

Failure Mode 1: Thread Pool Saturation Before CPU Saturation

Web server thread pools process requests concurrently up to their configured thread limit. When all threads are occupied, new requests queue and latency increases — regardless of CPU utilisation. Thread utilisation and CPU utilisation are independent variables for I/O-bound services.

────────────────────────────────────────────────────────────────────────────
THREAD POOL SATURATION: THE MECHANISM

Service configuration:
  Tomcat thread pool:      200 threads
  Database query latency:  80ms average (P50)
  CPU consumption/request: 5ms

At 1,000 RPS per replica:
  Concurrent threads occupied: 1,000 × 0.080s = 80 threads (40% of pool)
  CPU utilisation: 1,000 × 0.005s / 1s = 5,000ms/s → ~500% cores occupied
  Wait: 200-core server? No. 2-core container → 250% = CPU throttled

Actually at 2 CPU cores:
  CPU per request at 1,000 RPS: 1,000 × 0.005 = 5 seconds of CPU/second
  Available CPU: 2 seconds/second
  → CPU saturated at 1,000 RPS? No, because CPU throttling != thread blocking

Let's recalculate for I/O-bound:
  CPU consumption/request: 5ms (execution time)
  Database wait/request: 75ms (I/O wait — thread blocked, not CPU consuming)
  Total request latency: 80ms

At 1,000 RPS:
  Threads occupied: 1,000 × 0.080s = 80 threads (40% utilisation)
  CPU consumption: 1,000 × 0.005s = 5s of CPU/second across all requests
  Available CPU (2 cores): 2s/second
  CPU utilisation: 5s / 2s → would be 250%... but I/O wait doesn't use CPU

CORRECTED (I/O-bound):
  CPU utilisation = (CPU_time_per_request × RPS) / available_CPU_seconds
  = (0.005 × 1,000) / 2 = 5/2 = 2.5 → 250%? That saturates before threads.

For this to work: let's use realistic numbers:
  CPU per request: 2ms (realistic for most Spring Boot endpoint)
  DB wait: 78ms
  Total: 80ms
  Thread pool: 200 threads
  CPU cores: 2

  At thread saturation (all 200 threads busy):
    RPS = threads / latency_seconds = 200 / 0.080 = 2,500 RPS
    CPU utilisation = 2,500 × 0.002s / 2s = 5/2 → 250% (over capacity!)

  Wait — something's off. Let me think again.

  Actually for 2 CPU cores with 2ms CPU/request:
  CPU saturation (2 cores = 2,000ms/s):
    CPU saturates at: 2,000ms / 2ms = 1,000 RPS

  Thread saturation (200 threads × 80ms latency):
    Thread saturates at: 200 / 0.080 = 2,500 RPS

  SO: CPU saturates first (at 1,000 RPS) before threads saturate (at 2,500 RPS)

  This means CPU IS the correct signal here because CPU saturates before threads.

Revising: The failure mode occurs when thread saturation happens BEFORE CPU saturation.
This requires: thread_limit / avg_latency < CPU_capacity / CPU_per_request

  thread_limit / avg_latency < CPU_capacity / CPU_per_request
  200 / 0.080 < (2 / CPU_per_request)
  2,500 < 2 / CPU_per_request
  CPU_per_request < 0.0008s = 0.8ms

  So the failure mode is most acute when CPU per request is very small
  — highly optimised, mostly I/O-bound endpoints where processing is fast
  but external calls dominate latency

Concrete realistic example:
  CPU cores: 4
  CPU per request: 1ms
  DB latency: 99ms
  Total latency: 100ms
  Thread pool: 400

  CPU saturation: 4,000ms / 1ms = 4,000 RPS
  Thread saturation: 400 / 0.100 = 4,000 RPS (they saturate simultaneously!)

Another example where thread saturates first:
  CPU cores: 4
  CPU per request: 0.5ms
  DB latency: 99.5ms
  Thread pool: 200

  CPU saturation: 4,000ms / 0.5ms = 8,000 RPS
  Thread saturation: 200 / 0.100 = 2,000 RPS ← threads saturate FIRST

At 2,000 RPS (thread saturation):
  CPU utilisation: 2,000 × 0.0005s / 4s = 1,000ms/4,000ms = 25%
  The autoscaler sees 25% CPU when the service is at capacity!
────────────────────────────────────────────────────────────────────────────

The mathematical conclusion: thread pool saturation before CPU saturation is the dominant failure mode when CPU time per request is low relative to total request latency — exactly the profile of well-optimised, I/O-bound microservices. The CPU utilisation at thread pool saturation can be 25–40% depending on the I/O bound fraction, which is far below typical HPA scale-out thresholds of 70–80%.

Failure Mode 2: JVM Garbage Collection CPU Suppression

JVM garbage collection pauses application threads. During a GC pause, request processing halts — but GC itself may be CPU-intensive, creating the appearance of high CPU utilisation while request throughput drops to near zero.

────────────────────────────────────────────────────────────────────────────
GC PAUSE CPU SUPPRESSION: THE MECHANISM

During a full GC event (G1GC Full GC or ZGC relocation):
  Application threads: SUSPENDED (stop-the-world)
  GC threads: RUNNING (consuming 2–8 CPU cores)
  Request processing: ZERO (all threads stopped)
  Queue depth: GROWING (new requests arriving, none processing)
  p99 latency: SPIKING (requests waiting for GC to complete)

What the autoscaler observes:
  CPU utilisation: HIGH (GC threads consuming CPU)
  Request rate: LOW or ZERO (no requests completing during pause)
  → HPA based on CPU: NO scale-out triggered
    (CPU is already high; scaling would be misguided anyway)
  → HPA based on RPS: IMMEDIATELY detects throughput collapse
    (RPS drops to near zero; average RPS per replica spikes to ∞/0)

What the SLO observes:
  p99 latency: SPIKED beyond SLO threshold
  Error rate: potentially elevated (timeouts during long GC pauses)
  → Burn rate alert fires

The GC failure mode is unique: CPU says "I'm busy" while the service
is actually not processing user requests. CPU-based autoscaling cannot
help here — scaling out provides no benefit during a GC pause that
affects all replicas simultaneously, and the scaling signal is wrong.

THE FIX is not in the autoscaler — it is in JVM tuning:
  -XX:MaxGCPauseMillis=200 (G1GC target pause time)
  -XX:+UseZGC (for < 10ms pauses — Java 15+)
  Ensure -XX:ActiveProcessorCount matches container CPU limit
  (prevents GC from over-threading relative to available cores)

But the correct DETECTION of this failure mode is RPS-based autoscaling:
  RPS drop → alert → investigation → identifies GC as cause
  CPU-based autoscaling is blind to this failure mode
────────────────────────────────────────────────────────────────────────────

Failure Mode 3: Connection Pool Exhaustion

Database connection pools are a fixed-size resource. When all connections are occupied, new requests wait for a connection to become available — independent of CPU utilisation.

────────────────────────────────────────────────────────────────────────────
CONNECTION POOL EXHAUSTION: THE MECHANISM

Service configuration:
  Database connection pool: 100 connections
  Average DB connection hold time: 25ms per request
  CPU per request: 2ms

At connection pool saturation:
  Max sustainable RPS = pool_size / avg_hold_time = 100 / 0.025 = 4,000 RPS
  CPU at saturation: 4,000 × 0.002s / 4s = 2,000ms/4,000ms = 50%

Autoscaler observation at connection pool saturation:
  CPU utilisation: 50% → BELOW 70% threshold → NO scale-out
  RPS per replica: above SOT → TRIGGERS scale-out

But wait: scaling out does NOT fix connection pool exhaustion
  if the pool is shared (one pool serving all replicas):
  → More replicas competing for the same connection pool
  → No improvement; possibly worse

If connection pool is per-replica (typical with HikariCP):
  → Scaling out DOES add capacity: new replicas bring new pools
  → RPS-based scale-out is the correct response

LESSONS:
  1. Connection pool exhaustion is invisible to CPU-based autoscaling
  2. RPS-based autoscaling detects it correctly (throughput ceiling)
  3. The scaling response is only effective if the connection pool
     is per-replica — verify before concluding scale-out helps
────────────────────────────────────────────────────────────────────────────

Failure Mode 4: Network I/O Saturation

Network-bound services (high-throughput message brokers, streaming services, large-payload APIs) can saturate network bandwidth before CPU. At network saturation, new requests queue and latency increases while CPU utilisation remains below scaling thresholds.

────────────────────────────────────────────────────────────────────────────
NETWORK SATURATION: CPU UNDERREPORTING

Workload: image processing API
  Average response payload: 5 MB
  Network bandwidth per pod: 1 Gbps / 100 pods = 10 Mbps per pod
  CPU per request: 50ms
  Network time per request: 5MB / 10Mbps = 4,000ms = 4 seconds!!

  Clearly network-dominated; CPU is irrelevant for scaling decisions.

More realistic: REST API returning 100KB JSON
  Network time at 1Gbps / 100 pods: 0.8ms
  CPU time: 10ms
  Total: ~11ms
  Network not a constraint here.

  PRACTICAL RULE: Network saturation is primarily relevant for:
  - Services transferring > 1MB per response
  - Services on bandwidth-constrained nodes (low-tier cloud instances)
  - High-frequency, high-payload streaming services
────────────────────────────────────────────────────────────────────────────

Why Request Rate Is the Correct Scaling Signal

Request rate (RPS) directly measures the demand placed on the service from the user's perspective. It is not a proxy for demand — it IS demand. The relationship between request rate and SLO compliance is governed by Little's Law, which relates arrival rate, concurrency, and response time in a way that CPU utilisation cannot capture.

────────────────────────────────────────────────────────────────────────────
LITTLE'S LAW AND THE RPS SCALING SIGNAL

Little's Law: L = λ × W
  L = concurrent requests in system
  λ = arrival rate (RPS)
  W = average response time (seconds)

At Safe Operating Throughput (SOT):
  λ = SOT (the arrival rate at which SLO is maintained)
  W = baseline response time (at low load)
  L_safe = SOT × W_baseline

Above SOT:
  λ > SOT → concurrency L exceeds safe level
  W increases (requests queue) → L increases further
  Positive feedback loop → latency degradation begins

The autoscaling target:
  Keep average λ per replica below SOT
  Scale when: observed_RPS_per_replica > SOT × safety_margin

Why RPS is correct:
  ✓ Directly measures the demand the service must handle
  ✓ When RPS > SOT, latency WILL increase (Little's Law, not heuristic)
  ✓ Scale-out immediately reduces λ per replica → latency stabilises
  ✓ Captures all four failure modes (thread, GC, connection, network)
    because all reduce effective SOT, which the RPS signal detects

Why CPU is wrong:
  ✗ CPU does not appear in Little's Law
  ✗ CPU can be low when λ > SOT (thread saturation case)
  ✗ CPU can be high when λ ≈ 0 (GC pause case)
  ✗ The CPU→latency relationship is service-specific and non-monotonic
────────────────────────────────────────────────────────────────────────────

The Complete Migration: CPU-Based → RPS-Based Autoscaling

────────────────────────────────────────────────────────────────────────────
MIGRATION PROTOCOL: CPU HPA → RPS KEDA/HPA

STEP 1: Establish the current CPU threshold baseline
  Query: what CPU% corresponds to the current scale-out threshold?
  Query: what RPS/replica corresponds to that CPU threshold?
  Rationale: establishes equivalence point between old and new signal

STEP 2: Run SOT derivation load test
  Execute stepped ramp load test (see SOT post)
  Identify RPS/replica at which SLO boundary is reached
  Calculate SOT = 0.80 × SLO_boundary_RPS (80% safety margin)

STEP 3: Verify Istio Envoy metric availability
  Check: istio_requests_total{reporter="destination"} exists in Prometheus
  Verify: metric is populated for target service
  If not: enable Istio Prometheus integration before proceeding

STEP 4: Deploy new ScaledObject / HPA in parallel (do not delete old)
  Configure new object with SOT-derived target
  Set minReplicas and maxReplicas same as current HPA
  Deploy in parallel — both autoscalers running temporarily
  Kubernetes will honour the higher replica count recommendation
  This provides safety: if new config is wrong, old CPU HPA still active

STEP 5: Observe for 7 days
  Verify new scaler responds correctly to load patterns
  Compare new replica counts with old CPU-based replica counts
  Expected: new scaler fires earlier during ramp (before CPU threshold)
  Expected: new scaler fires later during GC events (RPS doesn't drop from GC)

STEP 6: Validate during load test
  Run controlled load test to SOT boundary
  Verify: scaler fires before latency SLO breach
  Verify: scale-up stabilises p95 latency below SLO threshold
  Compare: old CPU scaler would have fired at {old_threshold} CPU%
  Result: documented evidence that RPS scaler protects SLO better

STEP 7: Remove CPU-based HPA
  Delete old HPA / update ScaledObject to remove CPU trigger
  Update HPA annotation with SOT derivation reference
────────────────────────────────────────────────────────────────────────────

# Complete RPS-Based Autoscaling Configuration
# Replaces CPU-based HPA for a Spring Boot payments service

# BEFORE (CPU-based HPA — wrong):
# apiVersion: autoscaling/v2
# kind: HorizontalPodAutoscaler
# spec:
#   metrics:
#     - type: Resource
#       resource:
#         name: cpu
#         target:
#           type: Utilization
#           averageUtilization: 70

# AFTER (RPS-based KEDA ScaledObject — correct):
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: payments-api-rps-scaler
  namespace: production
  annotations:
    sre.internal/scaling-strategy: "rps-based"
    sre.internal/sot-value: "3040"           # SOT from load test
    sre.internal/sot-safety-margin: "0.80"
    sre.internal/sot-derived-date: "2025-Q1"
    sre.internal/cpu-hpa-removed: "2025-04-15"
    sre.internal/cpu-threshold-at-sot: "61"  # CPU% when RPS hit SOT in load test
    sre.internal/migration-rationale: >
      CPU-based HPA produced scale-out threshold of 70% CPU, which corresponds
      to 61% CPU at thread pool saturation (SOT=3040 RPS). CPU scaler fired
      AFTER SLO breach during thread-saturated load. RPS scaler fires at SOT
      BEFORE latency SLO breach. Validated in load test 2025-Q1.
spec:
  scaleTargetRef:
    name: payments-api
  minReplicaCount: 3
  maxReplicaCount: 50
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring.svc:9090
        metricName: payments_api_rps_per_replica
        query: |
          sum(
            rate(istio_requests_total{
              destination_service_name="payments-api",
              reporter="destination"
            }[2m])
          )
          /
          count(
            kube_pod_info{
              namespace="production",
              pod=~"payments-api-.*"
            }
          )
        threshold: "3040"
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleUp:
          stabilizationWindowSeconds: 30
          policies:
            - type: Percent
              value: 100
              periodSeconds: 15
        scaleDown:
          stabilizationWindowSeconds: 300
          policies:
            - type: Percent
              value: 10
              periodSeconds: 60

Splunk: Validating the Migration

-- Splunk SPL: Compare CPU HPA vs RPS scaler response time to load events
-- Run after migration to validate the RPS scaler fires earlier

index=kubernetes sourcetype="kube:events"
  reason="SuccessfulRescale"
  (name="payments-api-rps-scaler" OR name="payments-api-cpu-hpa")
| join type=left timestamp [
    search index=sre_metrics sourcetype="sre:slo"
      service="payments-api" metric_name="p95_latency_ms"
    | eval timestamp=_time
    | fields timestamp, p95_latency_ms
  ]
| eval scaler_type = if(name="payments-api-rps-scaler", "RPS", "CPU")
| eval latency_at_scaleout = p95_latency_ms
| stats
    avg(latency_at_scaleout) as avg_latency_when_scaled,
    count                    as scale_events
    by scaler_type
-- Expected result:
-- RPS scaler: avg_latency_when_scaled ~150ms (scaled before SLO breach)
-- CPU scaler: avg_latency_when_scaled ~350ms (scaled after SLO breach)

Common Antipatterns

The Load Test Without JVM Warm-Up antipattern → Deriving SOT from a load test that starts at full load without a ramp phase. JVM JIT compilation dramatically improves throughput over the first 3–5 minutes of sustained load. A load test that jumps immediately to high RPS measures cold JVM performance, which underestimates SOT and produces a conservative (over-scaled) autoscaling target. Use a stepped ramp with at least 5 minutes at each step before advancing.
The CPU Monitoring Retention antipattern → Replacing the CPU-based HPA trigger with an RPS-based trigger but keeping CPU utilisation as the primary dashboard metric for assessing autoscaling health. Engineers who monitor CPU to evaluate whether autoscaling is working will remain confused. Replace CPU utilisation with RPS per replica and SOT utilisation percentage as the primary autoscaling health indicators.
The Wrong Reporter antipattern → Using Istio reporter="source" instead of reporter="destination" for the RPS metric. As detailed in the Istio STRICT mTLS section: source metrics miss mTLS-layer rejections, which are invisible to the source sidecar but visible to the destination sidecar. In STRICT mTLS environments with active certificate rotation or policy changes, this gap can be significant.
The Missing ActiveProcessorCount antipattern → Migrating to RPS-based autoscaling without also fixing JVM ActiveProcessorCount alignment. The RPS scaler will correctly trigger scale-out at the SOT boundary — but the SOT itself will be artificially low if JVM thread pools and GC threads are misconfigured for the container CPU limit. Both changes are needed to produce the correct behaviour.
The Stability Window Optimism antipattern → Setting scaleDown.stabilizationWindowSeconds to a low value (< 60 seconds) in an attempt to reduce costs. For RPS-based scaling, a short scale-down window causes replica removal during brief RPS valleys between request bursts — which then requires immediate scale-up when the next burst arrives. The scale-up cold-start latency creates the SLO breach that the autoscaler was supposed to prevent. 300 seconds minimum.

Maturity Progression

────────────────────────────────────────────────────────────────────────────
STAGE        RPS AUTOSCALING MATURITY           NORTH STAR SIGNAL
────────────────────────────────────────────────────────────────────────────
Reactive     CPU-based HPA everywhere.          SLO breaches during
             Thread pool saturation             load events despite
             events invisible.                  autoscaler operating
             CPU at 61% when SLO               "correctly" by CPU
             breaches.                          threshold.

Defined      RPS-based autoscaling             First SOT derivation
             policy adopted. Failure            complete. First service
             modes documented.                  migrated to RPS-based
             SOT derivation protocol            KEDA. CPU-based HPA
             established.                       retained in parallel.

Measured     Migration complete for            RPS scaler fires before
             critical services. CPU            SLO breach in load test.
             threshold at SOT                  CPU% at SOT boundary
             documented for each.              documented for each
             Splunk comparison                 service.
             validates timing.

Optimised    JVM ActiveProcessorCount          Zero thread-saturation-
             aligned. GC tuning                caused SLO breaches.
             validated in load tests.          GC pause impact visible
             RPS/replica dashboard             in telemetry but not
             primary autoscaling               causing SLO events.
             health metric.

Generative   SOT derivation required          New services blocked
             for all new services             from production launch
             before launch. Autoscaling        without RPS-based
             strategy part of design           autoscaling configured
             review checklist.                 and SOT validated.
────────────────────────────────────────────────────────────────────────────

Five Action Items for This Week

Identify the CPU utilisation value for each production service at which thread pool saturation actually occurs. Run a load test at stepped RPS increments for one service and record the CPU utilisation at each step. Find the step where latency begins to increase (the SOT boundary). Record the CPU% at that step — this is the CPU utilisation that your 70% threshold should have been, and it is probably not 70%.
Check ActiveProcessorCount alignment for every JVM service in production. Run kubectl exec <pod> -- java -XshowSettings:all 2>&1 | grep "processors". Compare the reported processor count to the container CPU limit. If they differ, the JVM has misconfigured thread pools. Add -XX:ActiveProcessorCount=<cpu_limit_integer> to JAVA_TOOL_OPTIONS before the next deployment.
Verify that your Istio Envoy metrics exist and are being scraped. Query istio_requests_total{reporter="destination"} in Prometheus for a production service. If the metric is absent or shows zero, Istio Prometheus integration needs to be enabled before RPS-based autoscaling can be configured.
Deploy the RPS-based ScaledObject for one service in parallel with its existing CPU HPA. Let both run for one week. Compare scale-out event timing: does the RPS scaler fire earlier than the CPU HPA during load events? The expected answer is yes. Document the comparison — it is your migration evidence and your justification for removing the CPU HPA.
Update your autoscaling dashboard to show RPS per replica and SOT utilisation percentage as primary metrics. Remove CPU utilisation from the primary autoscaling health view. CPU is still useful for capacity planning and infrastructure efficiency; it is not useful for autoscaling health assessment for web workloads. The dashboard changes what engineers monitor, which changes what they optimise.

"CPU-based autoscaling is correct for CPU-bound workloads. Most production web and API services are not CPU-bound. They are I/O-bound, thread-pool-bound, connection-pool-bound, or GC-pause-affected — and for all of these, CPU utilisation is at best a lagging indicator and at worst an actively misleading one. The autoscaling signal should measure demand, not the infrastructure's response to demand. Request rate measures demand. CPU measures something else."

HPA vs KEDA vs VPA: A Quantitative Framework for Autoscaling Strategy Selection in Kubernetes

Nijo George Payyappilly — Mon, 24 Aug 2026 16:00:00 +0000

The Kubernetes autoscaling landscape is one where most decisions are made by convention rather than derivation. Teams use HPA because it ships with Kubernetes. They scale on CPU because that is what the tutorials show. They discover KEDA when they need a Kafka consumer to scale on queue depth. They encounter VPA in the docs and add it experimentally. The result, in most production environments, is an autoscaling configuration that was assembled from whatever felt appropriate at the time rather than derived from the workload's actual scaling requirements.

This matters operationally. An incorrect autoscaling strategy does not fail immediately and visibly — it fails slowly and ambiguously, through slightly elevated latency during load spikes, through replica counts that oscillate rather than stabilise, through JVM services that scale to the correct number of replicas but still degrade because each replica is under-resourced. The failure modes of incorrect autoscaling strategy selection are almost always attributed to the application rather than to the scaling configuration — which means they persist.

This post derives the selection framework from first principles. The decision criteria are expressed as measurable thresholds, not as preferences or rules of thumb. The goal is not to produce a chart that tells you which tool to use, but to produce a reasoning process that tells you why.

The Three Autoscaling Dimensions

Before comparing the tools, the problem space must be defined. Autoscaling in Kubernetes addresses three independent dimensions of the capacity problem. Each dimension has a different control variable, a different control loop timescale, and a different set of appropriate tools.

────────────────────────────────────────────────────────────────────────────
THE THREE AUTOSCALING DIMENSIONS

DIMENSION 1: HORIZONTAL (Scale Out/In)
  Control variable: replica count
  Control loop timescale: seconds to minutes
  SLO impact: affects capacity ceiling (SOT × replica_count)
  Primary tool: HPA (built-in), KEDA (extended triggers)
  Appropriate when: workload is stateless or horizontally shardable;
                    request rate varies with load patterns

DIMENSION 2: VERTICAL (Scale Up/Down)
  Control variable: CPU and memory resource requests/limits per pod
  Control loop timescale: minutes to hours (requires pod restart)
  SLO impact: affects per-replica performance (SOT per replica)
  Primary tool: VPA (Vertical Pod Autoscaler)
  Appropriate when: workload has variable resource consumption;
                    right-sizing initial resource requests is difficult;
                    workload is NOT horizontally scalable

DIMENSION 3: EXTERNAL EVENT-DRIVEN (Scale to Zero / Scale on Queue)
  Control variable: replica count, driven by external event sources
  Control loop timescale: seconds (event-driven)
  SLO impact: affects scale-to-zero and burst-from-zero latency
  Primary tool: KEDA
  Appropriate when: workload has predictable burst patterns (scheduled);
                    workload scales on upstream queue depth;
                    non-production workloads should scale to zero

────────────────────────────────────────────────────────────────────────────
KEY POINT: The three dimensions are NOT alternatives — they address
different aspects of the capacity problem and can be combined.
The selection question is not "HPA OR KEDA OR VPA" but:
"Which combination of tools addresses each dimension of this workload?"

The common error: treating HPA, KEDA, and VPA as interchangeable
alternatives and picking one, when the workload may need all three.
────────────────────────────────────────────────────────────────────────────

Decision Dimension 1: Horizontal Scaling — HPA vs KEDA

Both HPA and KEDA control replica count horizontally. The selection criterion is not which tool is better — it is which metric source and trigger model is appropriate for the workload's demand signal.

────────────────────────────────────────────────────────────────────────────
HPA vs KEDA SELECTION CRITERIA

USE HPA WHEN:

  Trigger 1: Request-rate-based scaling is sufficient
    Condition: workload receives HTTP/gRPC requests with a
               stable relationship between request rate and resource usage
    Metric: http_requests_per_second per replica (from Istio Envoy)
    Target: SOT value derived from load testing
    HPA limitation: only reads Kubernetes metrics API; limited to
                    metrics scraped into the cluster

  Trigger 2: Single-dimension scaling signal
    Condition: scaling decision can be made from one metric alone
    HPA handles this cleanly with a single target metric

  Trigger 3: Stateless services with predictable traffic
    Condition: no off-hours scale-to-zero requirement;
               no external queue depth dependency;
               no scheduled burst preparation

USE KEDA WHEN:

  Trigger 1: Multi-dimensional scaling required
    Condition: scaling decision depends on MULTIPLE signals simultaneously
    Example: scale on max(rps_per_replica, queue_depth_per_replica)
    HPA cannot express this; KEDA ScaledObject handles multiple triggers
    with OR logic (scale when ANY trigger exceeds threshold)

  Trigger 2: External event source (queue, stream, database)
    Condition: workload is a consumer of Kafka, RabbitMQ, SQS, Redis,
               or any other external event source
    KEDA has native scalers for 60+ external sources; HPA does not

  Trigger 3: Scale-to-zero required
    Condition: non-production workloads; batch workloads; scheduled jobs
    KEDA supports minReplicaCount: 0; HPA minimum is 1
    Scale-to-zero requires KEDA; this is a hard requirement

  Trigger 4: Scheduled burst preparation
    Condition: known traffic spikes at predictable times
               (market open, batch processing windows, shift changes)
    KEDA cron trigger provides pre-scheduled scaling;
    HPA responds reactively and cannot pre-warm

────────────────────────────────────────────────────────────────────────────
QUANTITATIVE DECISION CRITERIA:

  Question 1: Does the workload need to scale to zero?
    YES → KEDA (mandatory; HPA cannot do this)

  Question 2: Does the workload consume from an external queue/stream?
    YES → KEDA (direct external scaler; cleaner than HPA custom metrics)

  Question 3: Does the workload have known burst windows requiring pre-warm?
    YES → KEDA cron trigger (or add to existing KEDA ScaledObject)

  Question 4: Is the workload HTTP/gRPC with a stable RPS→resource relationship?
    YES, simple → HPA with custom metric (Istio RPS)
    YES, complex → KEDA with Prometheus trigger (more flexible)

  Question 5: Is the scaling signal available as a Prometheus metric?
    YES → Either HPA (via custom metrics adapter) or KEDA (native Prometheus)
    NO → KEDA (broader external source support)
────────────────────────────────────────────────────────────────────────────

# Decision Rule Applied: Payments API
# - HTTP workload: YES → consider HPA
# - Scale to zero: NO (production)
# - External queue: YES (Kafka payment requests) → KEDA required
# - Known burst window: YES (09:20 pre-market open) → KEDA cron
# Result: KEDA ScaledObject with three triggers (not HPA)

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: payments-api-scaler
  namespace: production
  annotations:
    sre.internal/scaling-strategy: "keda-multidimensional"
    sre.internal/sot-value: "3040"
    sre.internal/selection-rationale: >
      KEDA selected over HPA: (1) Kafka queue depth trigger required;
      (2) scheduled pre-warm for market open required;
      (3) multi-dimensional scaling (RPS + queue depth) required.
      HPA cannot satisfy trigger 1 or 2.
spec:
  scaleTargetRef:
    name: payments-api
  minReplicaCount: 5
  maxReplicaCount: 80
  cooldownPeriod: 60
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring.svc:9090
        metricName: rps_per_replica
        query: |
          sum(rate(istio_requests_total{
            destination_service_name="payments-api",
            reporter="destination"
          }[2m]))
          / count(kube_pod_info{namespace="production",pod=~"payments-api-.*"})
        threshold: "3040"    # SOT-derived target

    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring.svc:9090
        metricName: kafka_queue_depth
        query: |
          sum(kafka_consumer_group_lag{
            topic="payment-requests",
            group="payments-api"
          })
        threshold: "500"

    - type: cron
      metadata:
        timezone: "America/New_York"
        start: "20 9 * * 1-5"
        end:   "0 10 * * 1-5"
        desiredReplicas: "25"

Decision Dimension 2: Should VPA Be Added?

VPA addresses the vertical dimension — the right-sizing of CPU and memory resource requests per replica. The selection question is whether the workload has a vertical right-sizing problem that HPA/KEDA cannot address.

────────────────────────────────────────────────────────────────────────────
VPA SELECTION CRITERIA

USE VPA WHEN (in Recommendation mode, not Auto mode):

  Criterion 1: JVM workload with difficult-to-estimate heap requirements
    Signal: pods OOMKilling periodically despite "sufficient" memory limits
    OR: pods have very high memory request headroom (allocated >> used)
    VPA recommendation mode tells you what resource requests should be
    based on actual usage — removes the guesswork from JVM right-sizing

  Criterion 2: Batch workloads with variable resource consumption
    Signal: batch job CPU/memory usage varies significantly between runs
    VPA in recommendation mode surfaces the right-sizing range
    over time as the workload profile is observed

  Criterion 3: Initial deployment right-sizing
    Signal: new service with no historical usage data
    VPA recommendation mode in the first 2-4 weeks provides the
    data-driven request/limit values for the HPA/KEDA configuration

DO NOT USE VPA Auto mode with HPA/KEDA:

  CRITICAL CONFLICT: VPA Auto and HPA/KEDA targeting the same metric
    VPA Auto changes resource requests → triggers pod restarts
    Pod restarts temporarily reduce replica availability
    HPA/KEDA sees capacity drop → scales OUT to compensate
    VPA sees new pods with new resource profile → adjusts again
    Result: oscillation between VPA vertical adjustments and
            HPA/KEDA horizontal adjustments

  SAFE COMBINATION:
    VPA in Recommendation mode: produces recommendations only, no changes
    SRE reviews recommendations quarterly
    Manual resource request updates applied as planned changes
    HPA/KEDA manages horizontal scaling continuously

  VPA Auto mode is ONLY safe when:
    No HPA or KEDA is configured for the same deployment
    The workload is not latency-sensitive (VPA Auto causes pod restarts)
    The workload is a batch job or non-production service

────────────────────────────────────────────────────────────────────────────
VPA RECOMMENDATION MODE CONFIGURATION (safe with HPA/KEDA):

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: payments-api-vpa
  namespace: production
  annotations:
    sre.internal/mode: "recommendation-only"
    sre.internal/review-cadence: "quarterly"
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: payments-api
  updatePolicy:
    updateMode: "Off"    # CRITICAL: Off = recommendation only, no automatic changes
  resourcePolicy:
    containerPolicies:
      - containerName: payments-api
        minAllowed:
          cpu: "500m"
          memory: "1Gi"
        maxAllowed:
          cpu: "4"
          memory: "8Gi"
        controlledResources: ["cpu", "memory"]
────────────────────────────────────────────────────────────────────────────

The Autoscaling Strategy Decision Framework

────────────────────────────────────────────────────────────────────────────
AUTOSCALING STRATEGY SELECTION FRAMEWORK
(Work through each question in order)
────────────────────────────────────────────────────────────────────────────

STEP 1: HORIZONTAL SCALING TRIGGER SELECTION

  Q: Does the workload need to scale to zero in any environment?
    YES → KEDA required (proceed to Step 2 with KEDA)
    NO  → continue

  Q: Does the workload consume from an external event source?
    YES → KEDA required
    NO  → continue

  Q: Does the workload have predictable burst windows needing pre-warm?
    YES → KEDA cron trigger (can combine with other triggers)
    NO  → continue

  Q: Is a single Prometheus metric sufficient as the scaling signal?
    YES, and it's RPS-based → HPA with custom metric adapter
         (simpler configuration, standard Kubernetes API)
    YES, but multiple signals needed → KEDA with Prometheus triggers
    NO  → KEDA with appropriate native scaler

STEP 2: SCALING METRIC SELECTION

  Q: Is the workload request-rate-sensitive (web, API, gRPC)?
    YES → Use RPS per replica (from Istio Envoy telemetry)
          Target = SOT value (derived from load testing)
          NEVER use CPU% for this workload type

  Q: Is the workload queue-depth-sensitive (consumer, worker)?
    YES → Use queue lag per replica
          Target = max acceptable lag before processing falls behind

  Q: Is the workload CPU-bound with predictable CPU-per-request ratio?
    YES → CPU% MAY be acceptable
          Verify: CPU utilisation at SOT threshold from load test
          Verify: no JVM GC pause CPU suppression effect
          If either fails → switch to RPS-based

STEP 3: VERTICAL SCALING DECISION

  Q: Is the workload a JVM application?
    YES → Deploy VPA in Recommendation mode
          Review VPA recommendations before initial production sizing
          Update resource requests based on VPA data quarterly

  Q: Is the service new with no historical usage data?
    YES → Deploy VPA in Recommendation mode for first 4 weeks
          Set initial requests conservatively high
          Update to VPA-recommended values after observation period

  Q: Is VPA Auto mode being considered for production?
    → If HPA or KEDA is also configured: NO — use Recommendation mode
    → If no horizontal autoscaling: VPA Auto acceptable for batch only

STEP 4: SCALE BEHAVIOUR TUNING

  Q: What is the cold-start latency for new replicas?
    JVM (typical): 30–90 seconds
    Container without JVM: 5–15 seconds
    scaleUp.stabilizationWindowSeconds should be ≥ cold-start latency

  Q: What is the SOT safety margin target?
    Standard: 80% of empirical SLO threshold
    High consequence: 70% (more conservative headroom)
    Scale trigger = SOT × safety_margin

────────────────────────────────────────────────────────────────────────────
STRATEGY PATTERNS BY WORKLOAD TYPE:

Stateless HTTP API (JVM):
  → KEDA (multi-trigger) + VPA Recommendation
  → Triggers: RPS (Prometheus) + cron (if burst windows exist)
  → Target: SOT value from load test

Stateless HTTP API (non-JVM):
  → HPA (if single trigger sufficient) or KEDA (if multi-trigger needed)
  → VPA Recommendation optional (less critical for non-JVM)

Queue Consumer:
  → KEDA with queue depth trigger + optional cron for pre-warm
  → Scale to zero in non-production

Batch Processing Job:
  → KEDA with cron trigger + scale to zero between runs
  → VPA Auto acceptable (no HPA/KEDA horizontal conflict)

Stateful Workload (database, cache):
  → NOT recommended for HPA/KEDA horizontal scaling
  → VPA Recommendation mode for right-sizing
  → Manual scaling with database-specific considerations
────────────────────────────────────────────────────────────────────────────

Scaling Behaviour Configuration: The Parameters That Determine SLO Impact

The scaling strategy selection determines which tool manages replica count. The scaling behaviour configuration determines how aggressively and how smoothly the tool responds to demand changes. Both matter for SLO compliance.

# Scaling Behaviour: Optimised for SLO Protection
# Fast scale-up to prevent SLO breach; slow scale-down to prevent oscillation

# For KEDA ScaledObject (applied via ScaledObject.spec.advanced):
advanced:
  horizontalPodAutoscalerConfig:
    behavior:
      scaleUp:
        stabilizationWindowSeconds: 30      # Fast: respond to load in 30s
        selectPolicy: Max                   # Use the largest scale-up recommendation
        policies:
          - type: Percent
            value: 100                      # Can double replica count per period
            periodSeconds: 15               # Aggressive: every 15 seconds
          - type: Pods
            value: 5                        # Or add 5 pods per period
            periodSeconds: 15
      scaleDown:
        stabilizationWindowSeconds: 300     # Slow: 5 minutes before scaling down
        selectPolicy: Min                   # Use the smallest scale-down recommendation
        policies:
          - type: Percent
            value: 10                       # Remove at most 10% of replicas per period
            periodSeconds: 60              # Every 60 seconds

# Scale-up calibration rationale:
#   Cold-start latency for JVM: ~45 seconds
#   stabilizationWindowSeconds: 30s means scale-up fires before cold-start completes
#   This is intentional: we want replicas initialising before we need them,
#   not after we need them. The 30s window ensures overlap with JVM warm-up.

# Scale-down calibration rationale:
#   300s stabilisation: prevents scale-down during brief load valleys
#   Traffic that drops for 2 minutes and recovers should NOT trigger scale-down
#   Financial services: additional consideration for end-of-day settlement
#     windows — scale-down should not fire during known high-activity periods

Istio STRICT mTLS: Scaling Metrics from the Right Source

# Scaling metric source matters in Istio STRICT mTLS environments
# reporter="destination" captures the full request rate including
# requests rejected at the mTLS layer before reaching the application

# CORRECT: Envoy proxy metric (reporter="destination")
- type: prometheus
  metadata:
    query: |
      sum(rate(istio_requests_total{
        destination_service_name="{{ service_name }}",
        reporter="destination"
      }[2m]))
      / count(kube_pod_info{
          namespace="{{ namespace }}",
          pod=~"{{ service_name }}-.*"
        })
    threshold: "{{ sot_value }}"

# WHY NOT reporter="source":
#   Source metrics miss mTLS handshake failures
#   A certificate rotation event fails connections at the destination sidecar
#   These failures do NOT appear in source metrics
#   → Scaling trigger misses a significant load signal during policy events

# WHY NOT application-level metrics:
#   Application only sees requests that passed the sidecar
#   Same gap as source metrics for mTLS-layer failures
#   Additional gap: requests rejected by circuit breaker at sidecar
#   → Scaling trigger systematically undercounts effective request rate

Common Antipatterns

The CPU Default antipattern → Configuring HPA with type: Resource, resource: cpu because it is the Kubernetes documentation default. CPU is a lagging indicator for web/API workloads, a misleading indicator for JVM workloads during GC pauses, and irrelevant for queue consumers. The only workloads where CPU% is the correct scaling metric are CPU-bound batch jobs where CPU utilisation has a stable, predictable relationship with throughput — which is a small minority of production services.
The VPA Auto + HPA Conflict antipattern → Running VPA in Auto mode on the same deployment as HPA or KEDA. VPA Auto triggers pod restarts to apply new resource requests. Each restart removes a replica from the healthy pool, which HPA reads as reduced capacity and responds to with horizontal scale-out. The result is a control loop oscillation between VPA's vertical adjustments and HPA's horizontal compensations. Use VPA in Recommendation mode exclusively when HPA or KEDA is active.
The ScaleDown Too Fast antipattern → Setting scaleDown.stabilizationWindowSeconds to 0 or to a very short value in the belief that fast scale-down reduces costs. Fast scale-down causes two SLO problems: replicas are removed during brief load valleys and then must be re-added when load recovers, introducing cold-start latency during the re-scaling period; and the oscillation between scale-down and scale-up events creates an elevated error rate from connections to terminating pods. Keep scale-down stabilisation at 5 minutes minimum.
The Single-Trigger KEDA antipattern → Using KEDA for a workload that only needs one scaling trigger, when HPA would be simpler. KEDA's operational overhead (additional controller, ScaledObject CRD, external scaler connections) is justified by its extended trigger capabilities. For a simple request-rate-based scaler on a production service with no scale-to-zero requirement and no external queue dependency, HPA with a custom metric adapter is simpler and has fewer failure modes.
The SOT-Free Configuration antipattern → Setting HPA or KEDA targets without deriving them from SOT load testing. A target that is set "by feel" or copied from a similar service will be wrong in one of two directions: too aggressive (service degrades before scaling completes) or too conservative (service scales out unnecessarily, wasting capacity). SOT derivation from load testing is the mechanism that produces a principled target.

Maturity Progression

────────────────────────────────────────────────────────────────────────────
STAGE        AUTOSCALING MATURITY               NORTH STAR SIGNAL
────────────────────────────────────────────────────────────────────────────
Reactive     CPU-based HPA everywhere.          Capacity incidents after
             Manual scaling for known           the fact. Replicas scale
             events. VPA unused.                too late. JVM services
                                                OOMKill under load.

Defined      Selection framework documented.    First SOT-derived
             SOT derived for critical          target deployed. KEDA
             services. KEDA adopted for        adopted for queue
             queue consumers.                   consumers. VPA
                                                Recommendation active.

Measured     All production HPA/KEDA           No SOT boundary events
             targets SOT-derived. VPA          in last 30 days.
             recommendations reviewed          Scale-up fires before
             quarterly. CPU-based HPA          SLO breach. Oscillation
             eliminated from production.       eliminated.

Optimised    KEDA multi-trigger for all        Pre-warm triggers
             services with known burst         active for all known
             patterns. VPA right-sizing        burst windows. Cold-
             cycle automated. Scaling          start latency measured
             metrics from Istio Envoy.         and accounted for.

Generative   Autoscaling strategy part         New services cannot
             of service design review.         launch without
             SOT requirement blocks            autoscaling strategy
             deployment without load           review and SOT
             test baseline.                    derivation complete.
────────────────────────────────────────────────────────────────────────────

Five Action Items for This Week

Audit every HPA in your production cluster and identify which are using CPU% as the scaling metric. For each CPU-based HPA, assess whether the workload is actually CPU-bound or whether it is a web/API/queue service where CPU is a misleading signal. This audit produces your list of HPA configurations to replace with request-rate-based targets.
Identify every production service that needs to respond to an external event source (Kafka, RabbitMQ, database queue) and is currently using HPA. These are mandatory KEDA migration candidates. HPA cannot read external queue depth; KEDA can. The queue depth trigger is the correct scaling signal for consumer workloads.
Deploy VPA in Recommendation mode for your top three JVM services and observe the recommendations for one week. Do the VPA-recommended resource requests differ significantly from your current configuration? A large difference indicates that your current right-sizing is incorrect — either wastefully high (costing money) or insufficiently generous (causing OOMKills under load).
Verify that your HPA/KEDA targets are sourced from Istio Envoy metrics (reporter="destination") rather than application-level metrics. Query both sources for a production service and compare. Any difference is load that your current scaling trigger is missing — typically mTLS-layer failures and circuit-breaker rejections that the application never sees but that reduce effective capacity.
Set scaleDown.stabilizationWindowSeconds to 300 (5 minutes) on every HPA and KEDA ScaledObject that currently has a shorter window or no stabilisation configured. This single change eliminates most autoscaling oscillation in production. The cost is slightly slower capacity reclamation; the benefit is elimination of the latency spikes caused by replica removal during brief load valleys.

"The autoscaling question is not which tool to use. It is what the scaling signal should be, how quickly the system should respond to it, and what the correct capacity target is relative to the SLO. HPA, KEDA, and VPA are each optimal for a specific subset of these questions. The practitioner who understands the questions can select the tools. The practitioner who only knows the tools will select them by convention — and convention in autoscaling defaults to CPU-based HPA, which is wrong for most production web services."

Error Budget as a Change Gate: A Decision-Theoretic Model for Release Risk

Nijo George Payyappilly — Mon, 17 Aug 2026 16:00:00 +0000

Most organisations that implement error budget policies arrive at the same basic structure: when the budget is healthy, deploy freely; when the budget is depleted, freeze deployments. This is the right structure. But when challenged — by a product manager whose feature is frozen, by a VP of Engineering who believes the current reliability is adequate, by a CTO who sees a competitive cost to not shipping — the policy's defenders frequently cannot explain why the freeze threshold is where it is, why deployments are the right thing to gate rather than some other activity, or why the utility of a deployment changes as a function of budget remaining.

These are not rhetorical questions. They are the questions that determine whether an error budget policy survives contact with the organisation that must live by it. A policy explained as "that's what the SRE Workbook recommends" will be overridden whenever a business stakeholder has sufficient authority and sufficient motivation. A policy explained as "here is the formal model, here are the assumptions it rests on, and here is the budget threshold at which the expected value of deploying becomes negative" is a governance argument.

This post constructs that argument.

The Deploy/No-Deploy Decision as a Decision Theory Problem

Every deployment is a decision under uncertainty. The deployment may succeed without incident, in which case users receive the new feature and the organisation captures the delivery value. The deployment may fail — it may introduce a regression, a performance degradation, or an outage — in which case users experience degraded service and the error budget is consumed.

Formally:

────────────────────────────────────────────────────────────────────────────
DECISION THEORY FORMULATION: DEPLOY/NO-DEPLOY

Let:
  p    = P(incident | deploy) — probability deployment causes an incident
  V    = value of successful deployment (features delivered, revenue enabled)
  C    = cost of a deployment-caused incident (error budget consumed,
         customer impact, operational overhead, regulatory exposure)
  B    = current error budget remaining (0.0 to 1.0)

Expected value of deploying:
  EV(deploy) = (1-p) × V + p × (-C)
             = V - p(V + C)

Expected value of NOT deploying:
  EV(no-deploy) = 0
  (assumes no immediate value from deferral; deployment is deferred, not lost)

Deployment is rational when:
  EV(deploy) > EV(no-deploy)
  V - p(V + C) > 0
  V > p(V + C)
  V/( V + C) > p
  p < V / (V + C)

Critical insight: The deploy/no-deploy decision is rational as long as
  the incident probability p is below a threshold determined by the ratio
  of deployment value V to total cost-at-risk (V + C).

  This threshold is NOT a function of error budget directly.
  It IS a function of how budget remaining affects p.
────────────────────────────────────────────────────────────────────────────

The decision depends critically on p — the probability that a given deployment causes an incident. This is the quantity that error budget state provides information about. A service with a healthy error budget has been behaving reliably; its recent change failure rate is low; p for the next deployment is likely to be in its historical range. A service with an exhausted error budget has been experiencing elevated error rates; recent deployments may have been contributing; p for the next deployment is elevated above historical baseline.

How Error Budget State Updates the Incident Probability Estimate

The connection between error budget state and deployment risk is Bayesian. The error budget state provides evidence that updates our estimate of how risky a deployment is in the current environment.

────────────────────────────────────────────────────────────────────────────
BAYESIAN UPDATE: ERROR BUDGET STATE → DEPLOYMENT RISK

Prior: p_prior = historical change failure rate (DORA CFR)
  Typical regulated enterprise: 10–25% (Low performer cohort)
  Typical SRE-mature organisation: 5–15% (Medium performer cohort)
  Elite: 0–5%

Evidence from error budget state:

CASE 1: Budget > 75% remaining
  Interpretation: Service has been well-behaved recently
  No elevated recent burn → no evidence that current state is fragile
  Posterior: p ≈ p_prior (budget state provides no update)

CASE 2: Budget 25–75% remaining (degraded)
  Interpretation: Some budget consumption in measurement window
  May reflect recent deployments; may reflect external factors
  Posterior: p slightly elevated above prior
  Magnitude of update: depends on whether burn correlates with recent changes

CASE 3: Budget < 25% remaining (exhausted)
  Interpretation: Significant budget consumption; service is fragile
  High burn rate often correlates with system instability
  Posterior: p materially elevated above prior
  Justification: A service consuming error budget at elevated rate is
    more likely to be in a degraded state where additional changes
    will compound existing instability

CASE 4: Budget exhausted AND active burn rate > 3×
  Interpretation: Service is currently failing users above 3× baseline rate
  Deploying into active degradation is the highest-risk scenario:
    → Root cause may not be understood
    → New deployment may interact with ongoing failure mode
    → Rollback from failed deploy during active incident is compound chaos
  Posterior: p substantially elevated; EV(deploy) likely negative
────────────────────────────────────────────────────────────────────────────

Deriving the Budget Threshold Formally

Given the decision theory framework, the budget threshold at which deployments should be gated is the threshold below which the expected value of deploying becomes negative for the marginal deployment.

────────────────────────────────────────────────────────────────────────────
BUDGET THRESHOLD DERIVATION

Let:
  B_threshold = the budget remaining below which deployments are frozen
  p(B) = incident probability as a function of current budget B

Assume a simple linear model for the budget-to-risk relationship:
  p(B) = p_base + (1 - B) × p_sensitivity

Where:
  p_base        = baseline incident probability (historical CFR)
  p_sensitivity = how much p increases per unit of budget consumed
  (1 - B)       = budget consumed; higher consumption → higher p

The deployment freeze threshold is where EV(deploy) = 0:
  V - p(B) × (V + C) = 0
  p(B) = V / (V + C)
  p_base + (1 - B_threshold) × p_sensitivity = V / (V + C)
  (1 - B_threshold) = (V / (V + C) - p_base) / p_sensitivity
  B_threshold = 1 - (V / (V + C) - p_base) / p_sensitivity

WORKED EXAMPLE:
  p_base = 0.10       (10% historical CFR — Low performer cohort)
  V = 1.0             (normalised deployment value)
  C = 5.0             (incident cost = 5× deployment value — conservative)
  p_sensitivity = 0.30 (each 10% of budget consumed raises p by 3%)

  Rational freeze threshold:
    p* = V / (V + C) = 1 / (1 + 5) = 0.167 = 16.7%
    B_threshold = 1 - (0.167 - 0.10) / 0.30 = 1 - 0.223 = 0.777

  Interpretation: At a 10% baseline CFR with incident cost 5×
  the deployment value and moderate budget sensitivity,
  deployments become EV-negative when budget falls below 77.7%.

SENSITIVITY ANALYSIS:
  Same parameters, but elite CFR (p_base = 0.03):
    p* = 0.167
    B_threshold = 1 - (0.167 - 0.03) / 0.30 = 0.543

  Interpretation: Elite performers can deploy more aggressively
  because their lower baseline CFR means p(B) stays below the
  rational threshold for longer as budget decreases.

  Increasing incident cost (C = 10× V):
    p* = 1/11 = 0.091
    B_threshold (p_base=0.10) → immediately negative (p_base > p*)
    Interpretation: When incident cost is very high, baseline CFR=10%
    means deploying is already EV-negative at any budget state.
    This is the financial services / healthcare argument for elite CFR.
────────────────────────────────────────────────────────────────────────────

The worked example demonstrates something important: for a regulated enterprise with a 10% baseline CFR and high incident cost (5× deployment value), the rational deployment gate fires at 77.7% budget remaining — which is higher than the typical 25% gate most error budget policies use. The standard 25% gate is not derived from first principles; it is a practical approximation that happens to be conservative enough for most commercial contexts but too permissive for high-consequence deployments.

Why the Freeze Policy Is the Right Response (Not Just a Heuristic)

The previous section derived when deploying becomes EV-negative. But the decision theory framework also implies what the right response to that state is — and it is not simply "stop deploying." It is "stop depleting the budget further while investing in reliability to improve p(B)."

────────────────────────────────────────────────────────────────────────────
OPTIMAL POLICY UNDER ERROR BUDGET CONSTRAINTS

When EV(deploy) < 0 (budget below threshold):

Action 1: FREEZE DEPLOYMENTS
  Rationale: Every additional deployment at elevated p consumes
             expected budget (p × C > V). Stopping deployments
             stops the EV-negative budget drain.

Action 2: INVEST IN RELIABILITY
  Rationale: The reliability investment changes the parameters:
             → Identifying the root cause of elevated burn reduces p(B)
             → Fixing the underlying issue reduces p_base for future cycles
             → Both actions shift B_threshold downward (less conservative)
  Expected value of reliability investment R:
    EV(R) = improvement to future deployments × deployment rate
           = Δp × (V + C) × future_deploys
  For most services, this is strongly positive — a single prevented
  incident at C = 5V pays back the reliability investment many times.

Action 3: REQUIRE OVERRIDE FOR HIGH-VALUE DEPLOYMENTS
  Rationale: V is not uniform across deployments. A security patch
             that prevents a material vulnerability has V >> typical_V.
             The EV calculation may be positive for this specific
             deployment even when negative for the marginal deployment.
  Policy implication: override authority should require explicit
             documentation of the specific V and C values for the
             override deployment — not just authority sign-off.

────────────────────────────────────────────────────────────────────────────
THE INFORMATION ASYMMETRY ARGUMENT:

  Why do development teams resist the deployment freeze?
  Because they observe V (the feature value they built) directly.
  They do not observe C (the incident cost) until it occurs.
  They observe the budget state, but the relationship between
  budget state and p(B) is opaque without the formal model.

  The error budget policy converts this information asymmetry:
    Before policy: "Why can't I deploy? The service is working fine."
    After policy: "The budget state implies p(B) = 18%, which at
                  your incident cost ratio means EV(deploy) is
                  negative for average-value deployments.
                  What is the specific V for your deployment?"

  This reframes the conversation from authority ("SRE says no")
  to economics ("show me the value justifies the elevated risk").
  The second conversation is winnable on the merits; the first isn't.
────────────────────────────────────────────────────────────────────────────

Multi-Service Portfolio Error Budget Management

The single-service decision model extends naturally to a portfolio of services, where budget state across multiple services must be managed in the context of shared infrastructure risk and correlated change failure modes.

────────────────────────────────────────────────────────────────────────────
PORTFOLIO ERROR BUDGET MANAGEMENT

PROBLEM: In a microservices environment, a deployment to Service A
  can cause incidents in Services B and C through dependency coupling.
  The incident probability p for a Service A deployment is not just
  a function of Service A's budget state — it also depends on the
  budget states of downstream dependents.

PORTFOLIO RISK ADJUSTMENT:
  p_portfolio(deploy_A) = p(A) + Σ p(A→X) × (1 - B_X) × w_X

  Where:
    p(A)     = direct incident probability for Service A
    p(A→X)   = probability Service A deployment cascades to Service X
    B_X      = current budget of Service X
    w_X      = weight of Service X (traffic fraction, criticality)

  Services with degraded budgets in the dependency graph elevate
  the portfolio risk of any upstream deployment.

PRACTICAL IMPLEMENTATION:
  Classify dependencies into rings:
    Ring 0: Direct runtime dependencies (immediate cascade risk)
    Ring 1: Indirect dependencies (one-hop cascade risk)
    Ring 2: Shared infrastructure (correlation risk)

  Portfolio gate: deployment blocked if any Ring 0 dependency
    has budget < 40% OR any Ring 1 dependency has budget < 25%
────────────────────────────────────────────────────────────────────────────

# Argo CD PreSync Hook — Decision-Theoretic Deployment Gate
# Evaluates EV(deploy) based on current budget state and service dependencies
# Blocks deployment when EV turns negative; requires override with documented V

apiVersion: batch/v1
kind: Job
metadata:
  name: decision-theoretic-gate
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
    argocd.argoproj.io/sync-wave: "-1"
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: deploy-gate
          image: sre-platform/decision-gate:v2.0.0
          env:
            - name: SERVICE_NAME
              value: "payments-api"
            - name: PROMETHEUS_URL
              value: "http://prometheus.monitoring.svc:9090"
            - name: P_BASE
              value: "0.10"         # Historical CFR for this service
            - name: P_SENSITIVITY
              value: "0.30"        # Budget-to-risk sensitivity
            - name: COST_RATIO
              value: "5.0"         # Incident cost as multiple of deploy value
            - name: RING0_DEPS
              value: "card-network-proxy,fraud-detection,core-banking"
            - name: RING0_BUDGET_THRESHOLD
              value: "0.40"
            - name: RING1_BUDGET_THRESHOLD
              value: "0.25"
            - name: OVERRIDE_ANNOTATION
              value: "sre.internal/ev-override-approved"
            - name: OVERRIDE_REQUIRES_DOCUMENTED_VALUE
              value: "true"   # Override must include deployment value justification
          # Gate logic:
          # 1. Query Prometheus: own service budget remaining
          # 2. Query Prometheus: Ring 0 dependency budgets
          # 3. Compute p(B) = p_base + (1-B) × p_sensitivity
          # 4. Compute portfolio risk adjustment
          # 5. Compute EV(deploy) = V - p_portfolio × (V + C × V)
          # 6. If EV > 0: exit 0 (proceed)
          # 7. If EV ≤ 0: check override annotation
          #    If override present with documented value: log, exit 0
          #    If no override: emit Splunk event, post Slack, exit 1
          # 8. Log full decision context to Splunk including:
          #    budget_remaining, p_estimated, ev_computed, gate_decision

Common Antipatterns

The Arbitrary Threshold antipattern → Setting the deployment freeze threshold at 25% (or any other number) without deriving it from the service's historical CFR, incident cost ratio, and budget-to-risk sensitivity. A threshold of 25% may be appropriate for a low-consequence service with elite CFR and cheap incidents. It is almost certainly too permissive for a healthcare EHR system with a 15% CFR and high patient safety incident costs. Derive your threshold; don't copy it.
The Uniform Policy antipattern → Applying the same deployment gate threshold to all services regardless of their V/C ratios. A deployment to the public documentation site has a very different value-to-cost ratio than a deployment to the payment processing core. The gate threshold should be a function of the service's incident cost profile, not a single organisation-wide number.
The Override Without Documentation antipattern → Permitting override of the deployment gate without requiring explicit documentation of the deployment value that justifies the override. An override that says "SRE Lead approved" is not a decision record — it is an authority delegation. An override that says "SRE Lead approved; deployment is a P0 security fix for CVE-2025-XXXX estimated to prevent potential breach with cost = 100× deployment value; EV positive at any budget state" is a decision record that can be reviewed, learned from, and used to improve the formal model.
The Single-Service Gate antipattern → Evaluating the deployment gate based only on the deploying service's own budget state, ignoring dependency ring budget states. A deployment to a healthy service (budget = 90%) that has Ring 0 dependencies at budget = 15% is a high-risk deployment that the single-service gate incorrectly classifies as safe.
The Model Drift antipattern → Calibrating the p_base and p_sensitivity parameters once at policy creation and never updating them. p_base should be reviewed quarterly against the actual CFR measured in Splunk. p_sensitivity should be re-estimated when the relationship between budget state and incident frequency changes — after major architectural changes, after SRE maturity improvements, or after the introduction of new deployment automation.

Maturity Progression

────────────────────────────────────────────────────────────────────────────
STAGE        CHANGE GATE MATURITY               NORTH STAR SIGNAL
────────────────────────────────────────────────────────────────────────────
Reactive     No error budget gate.              Deployments proceed
             Deployments proceed                regardless of SLO state.
             regardless of SLO state.           Post-incident reviews
             CFR unknown.                       show deployments as
                                                leading cause of budget
                                                consumption.

Defined      Error budget policy                Gate implemented as
             documented with threshold.         PreSync hook. Threshold
             Gate implemented. CFR              derived (not copied).
             being measured.                    Override requires
                                                documented justification.

Measured     Decision-theoretic model           EV calculation in gate
             implemented. p_base and            logic. Portfolio risk
             p_sensitivity calibrated           evaluated for Ring 0
             from CFR history.                  dependencies.
             Portfolio gate active.

Optimised    Threshold updated quarterly        Gate blocks fewer
             from CFR trend. Override           deployments because
             history analysed for              CFR has improved.
             threshold refinement.             Model parameters
             Ring 1 gate active.               reflecting current
                                               service quality.

Generative   Decision model shared as           Product teams understand
             reference architecture.           the V/C framework.
             Product teams provide             Override requests include
             deployment value estimates        quantified V. Gate
             proactively. Model informs        thresholds differentiated
             architectural decisions.          by service risk profile.
────────────────────────────────────────────────────────────────────────────

Five Action Items for This Week

Calculate the rational deployment gate threshold for your most critical service using the formal model. You need three inputs: your historical CFR (from Splunk incident data), your best estimate of incident cost as a multiple of deployment value, and an assumption about budget-to-risk sensitivity. Even rough estimates produce a more defensible threshold than the default 25%.
Document the V and C values for your last five deployments retrospectively. What was the actual value delivered? What did the incident cost when it occurred? The ratio V/C is the parameter that determines how conservative your gate threshold should be — and most organisations have never explicitly measured it.
Implement the portfolio gate for Ring 0 dependencies. Identify the direct runtime dependencies of your highest-traffic services. Add a gate condition that blocks deployment when any Ring 0 dependency has budget below 40%. This single change catches the most common case where a healthy-budget service deploys into a fragile dependency graph.
Add the EV calculation to your deployment gate log output. Every blocked or approved deployment should emit a Splunk event with p_estimated, ev_computed, budget_remaining, and gate_decision. This data accumulates into the calibration dataset that validates (or corrects) your p_base and p_sensitivity parameters over time.
Run the formal model on your last three gate overrides. For each override, plug in the actual budget state at override time and the actual deployment outcome (incident or no incident). Does the model's prediction match the outcome? If the model predicted EV-negative and no incident occurred, the parameters may be too conservative. If the model predicted EV-positive and an incident occurred, the parameters may be too optimistic.

"The error budget deployment gate is not a bureaucratic hurdle — it is the mechanism that converts reliability data into deployment decisions. Without the formal model, the gate is a rule that can be argued against on the grounds that it is arbitrary. With the formal model, the gate is a boundary that can only be moved by changing the underlying assumptions about value, cost, and risk. That is a different conversation — one that forces the business to make its assumptions about deployment risk explicit, which is itself a governance improvement regardless of where the threshold ends up."

Multi-Window Burn Rate Alerting: A Formal Analysis of the AND-Gate Logic Behind Google's Alert Model

Nijo George Payyappilly — Mon, 10 Aug 2026 16:00:00 +0000

Alerting on Service Level Objectives is a solved problem in the sense that the solution exists, has been published, and works reliably when implemented correctly. It is an unsolved problem in the sense that most organisations implementing SLOs do not implement it correctly — they alert on threshold breaches of individual metrics rather than on error budget consumption rates, and they use single measurement windows rather than the dual-window AND-gate structure that distinguishes a genuine sustained burn from a transient spike.

The cost of getting this wrong is operational. An alerting system that pages too frequently trains engineers to treat pages as background noise — the alert equivalent of a car alarm that everyone has learned to ignore. An alerting system that pages too infrequently allows budget burns to proceed undetected until the SLO is already breached. Both failure modes are self-reinforcing: the first erodes the operational culture that makes on-call sustainable; the second erodes the error budget that makes deployment velocity permissible.

The Google SRE Workbook's multi-window burn rate alerting model solves both failure modes simultaneously. This post derives it formally — not as a recipe to follow, but as a logical structure to understand. Understanding why the model works is what enables practitioners to adapt it correctly to their specific SLO windows, service characteristics, and operational environments.

The Single-Window Alerting Trade-off

Before deriving the multi-window model, the problem it solves must be precise. The fundamental constraint of single-window alerting is this: for any fixed measurement window W and any fixed alerting threshold T, the alert exhibits an irreducible trade-off between sensitivity (detecting genuine budget consumption early) and specificity (not firing on transient spikes that self-correct).

────────────────────────────────────────────────────────────────────────────
SINGLE-WINDOW ALERTING: THE UNAVOIDABLE TRADE-OFF

Given:
  SLO window: 28 days
  SLO target: 99.9% (error budget = 0.1%)
  Single measurement window W: T minutes

SHORT WINDOW (W = 5 minutes):
  Sensitivity: HIGH — detects burn rate changes quickly
  Specificity: LOW — every brief error spike triggers the alert
    Example: A 30-second traffic anomaly causing 5% errors for 30s
    Error rate in 5-minute window: (0.3/5) × 5% = ~0.3%
    This is 3× the error budget rate → alert fires
    But: the anomaly self-corrected; budget consumed was negligible
    Result: PAGE AT 2 AM for a self-correcting 30-second spike

LONG WINDOW (W = 1 hour):
  Sensitivity: LOW — requires sustained errors to trigger
  Specificity: HIGH — short spikes diluted across the window
    Example: A 10-minute complete outage (100% error rate)
    Error rate in 60-minute window: 10/60 = 16.7% → 167× budget rate
    This exceeds any reasonable threshold → alert fires
    But: by the time the 60-minute window accumulates enough signal,
    4.2% of the 28-day budget has already been consumed
    At severe burns, detection is too late to prevent material impact

FUNDAMENTAL CONSTRAINT:
  For any single window W:
    Shorter W → more false positives (specificity decreases)
    Longer W → slower detection (sensitivity decreases)
    No single W value optimises both simultaneously
────────────────────────────────────────────────────────────────────────────

The dual-window AND-gate escapes this constraint by using two windows simultaneously: a long window to confirm that a burn is sustained (not a transient spike), and a short window to confirm that the burn is still ongoing (not a historical artifact). The AND of the two conditions provides the noise suppression of the long window with the recency validation of the short window.

Formal Definition: Burn Rate

Before the AND-gate logic can be analysed, the burn rate must be defined precisely.

────────────────────────────────────────────────────────────────────────────
BURN RATE: FORMAL DEFINITION

Let:
  E     = error budget as a proportion (e.g., 0.001 for 99.9% SLO)
  W_SLO = SLO measurement window in hours (e.g., 672h for 28 days)
  r(t)  = error rate at time t (proportion of requests failing)

Budget consumption rate at 1× burn:
  r_baseline = E / W_SLO
  (the error rate at which the budget would be exactly exhausted at window end)

  For 99.9% SLO over 28 days:
    r_baseline = 0.001 / 672 = 0.00000149 per hour
               = 0.00149 per 1000 hours (negligible hourly)

Burn rate B at time t:
  B(t) = r(t) / r_baseline

  Interpretation:
    B = 1   → error rate on pace to exactly exhaust budget in 28 days
    B = 14  → error rate 14× r_baseline; budget exhausted in 28/14 = 2 days
    B = 6   → budget exhausted in 28/6 ≈ 4.7 days
    B = 3   → budget exhausted in 28/3 ≈ 9.3 days
    B = 0   → no errors; budget not being consumed

Time to budget exhaustion at constant burn rate B:
  t_exhaustion = W_SLO / B

  B = 14: t_exhaustion = 672h / 14 = 48h (2 days)
  B = 6:  t_exhaustion = 672h / 6  = 112h (~4.7 days)
  B = 3:  t_exhaustion = 672h / 3  = 224h (~9.3 days)
  B = 1:  t_exhaustion = 672h / 1  = 672h (28 days, exactly at window end)

IMPORTANT DISTINCTION:
  t_exhaustion is the time until budget runs out IF burn continues.
  The detection window (1h, 6h) is the window used to MEASURE B(t).
  These are not the same. Detection at B=14 over a 1-hour window means:
    → The burn has been at 14× for the last 1 hour
    → If it continues, budget exhausts in ~47 more hours
    → The 1 hour of burning has consumed 14/672 ≈ 2.1% of total budget
────────────────────────────────────────────────────────────────────────────

The AND-Gate: Formal Analysis

For each alert tier, two windows are defined: a long window W_L that estimates the sustained burn rate, and a short window W_S that validates the burn is current. The alert fires if and only if the burn rate exceeds threshold B_threshold in BOTH windows simultaneously.

────────────────────────────────────────────────────────────────────────────
AND-GATE FORMAL SPECIFICATION

Alert condition:
  FIRE if: B(W_L) > B_threshold  AND  B(W_S) > B_threshold

Where:
  B(W) = mean burn rate over window W
  W_L  = long window (1h, 6h, 1d, 3d depending on tier)
  W_S  = short window (5m, 30m, 2h, 6h depending on tier)

NOISE SUPPRESSION PROPERTY:
  A transient spike of duration d < W_S that creates burn rate B_spike:
    B(W_S) = (d/W_S) × B_spike + ((W_S-d)/W_S) × 0
           = (d/W_S) × B_spike

  For the AND condition to fire:
    (d/W_S) × B_spike > B_threshold
    Required spike: B_spike > B_threshold × (W_S/d)

  Example: W_S = 5 minutes, B_threshold = 14, spike duration d = 30 seconds
    Required spike rate: 14 × (5/0.5) = 14 × 10 = 140× burn rate
    A 30-second spike must create a 140× burn rate to trigger the short window
    At 99.9% SLO, 140× burn rate = 14% error rate for 30 seconds
    Most transient anomalies fall below this threshold → noise suppressed

RECENCY VALIDATION PROPERTY:
  A historical burn event that ended T hours ago:
    For T > W_S: B(W_S) ≈ 0 → AND condition fails → no stale alert
    For T < W_S: B(W_S) > 0 proportional to recency → may still page
    For T > W_L: B(W_L) ≈ 0 → AND condition fails definitively

  This property ensures that a burn event that resolved T hours ago
  stops generating pages once T > W_S — typically within 5 minutes
  of the error rate returning to normal.

MINIMUM DETECTION LATENCY:
  Minimum time before alert fires after burn begins (at constant B > B_threshold):
    t_min_detection = W_S (must accumulate W_S of signal in short window)

  For the 14× tier with W_S = 5 minutes:
    t_min_detection = 5 minutes
    Budget consumed before detection: 14 × (5/672/60) × 100% ≈ 0.17%
    Fraction of 28-day budget consumed before page: 17% (of total 0.1% budget)
────────────────────────────────────────────────────────────────────────────

Deriving the Four Tiers: Why These Multipliers?

The four burn rate thresholds — 14×, 6×, 3×, 1× — are not arbitrary. Each is calibrated to a specific combination of urgency (time to budget exhaustion) and detection window (how much budget is consumed before the alert fires).

────────────────────────────────────────────────────────────────────────────
TIER DERIVATION: CALIBRATION CRITERIA

DESIGN CRITERIA FOR EACH TIER:
  1. Time to budget exhaustion at threshold burn rate (urgency)
  2. Budget consumed before detection (detection cost)
  3. Appropriate organisational response (page vs ticket)
  4. Detection window feasibility (window short enough to fire early)

TIER 1 — PAGE IMMEDIATELY (14× burn, 1h/5m windows)

  Urgency:
    t_exhaustion = 28 days / 14 = 2 days
    "If nothing changes, SLO is breached in 2 days"
    → Requires immediate human response; cannot wait for business hours

  Detection cost:
    Budget consumed before short-window detection (5 min): ~0.17% of budget
    Budget consumed before long-window confirmation (1 hr): ~2.1% of budget
    This is acceptable: detection while most of the budget is still intact

  Why 14×, not 20× or 10×:
    20× → t_exhaustion = 1.4 days; detection before 1 hour often impossible
    10× → t_exhaustion = 2.8 days; urgency threshold ambiguous for paging
    14× provides a clean "2-day" exhaustion horizon with 1-hour detection window

TIER 2 — PAGE (WITHIN 30 MIN) (6× burn, 6h/30m windows)

  Urgency:
    t_exhaustion = 28 days / 6 = 4.67 days
    "Budget exhausted in under 5 days; needs response today"
    → Requires on-call page; response within 30 minutes

  Detection cost:
    Budget consumed before short-window detection (30 min): ~0.74% of budget
    Budget consumed before long-window confirmation (6 hr): ~8.9% of budget
    At 6× burn, 6-hour confirmation costs ~8.9% of total budget;
    acceptable given the lower urgency vs Tier 1

  Why 6h long window:
    6h window smooths out periods of elevated but not catastrophic error rate
    A service spiking to high error rates for 1-2 hours dilutes to below
    threshold in a 6-hour window → prevents Tier 2 pages for recoverable spikes

TIER 3 — TICKET (BUSINESS HOURS) (3× burn, 1d/2h windows)

  Urgency:
    t_exhaustion = 28 days / 3 = 9.3 days
    "Budget exhausted in ~9 days; needs attention this week"
    → Does not warrant waking someone up; create a ticket

  Detection cost:
    Budget consumed before 2h short-window detection: ~0.89% of budget
    At 3× sustained burn, this is manageable
    The 1-day long window provides very high noise suppression

  Why 1-day long window:
    A 3× burn rate is only ~3× the normal error rate
    Day-to-day traffic variation can produce sustained periods at 2-3×
    without representing a genuine reliability problem
    1-day window ensures the ticket tier only fires on genuinely
    sustained moderate burns, not daily traffic variation

TIER 4 — TREND REVIEW (1× burn, 3d/6h windows)

  Urgency:
    t_exhaustion = 28 days / 1 = 28 days (exactly exhausts at window end)
    "On pace to breach SLO; needs review but not urgent"
    → Weekly SRE sync item; not a ticket or a page

  Why 3-day long window:
    1× burn rate is definitionally "on pace to exhaust at window end"
    This is the background noise rate — the service is barely over its
    error budget allocation
    A 3-day window provides extremely high noise suppression;
    only genuinely sustained 1× burns generate the trend alert

────────────────────────────────────────────────────────────────────────────
TIER SUMMARY TABLE (28-day window, 99.9% SLO)

Tier  Multiplier  Long    Short  t_exhaust  Budget at  Response
                  Window  Window            Long Det.
────  ──────────  ──────  ─────  ─────────  ─────────  ──────────────
1     14×         1h      5m     2 days     ~2.1%      Page immediately
2     6×          6h      30m    4.7 days   ~8.9%      Page within 30min
3     3×          1d      2h     9.3 days   ~10.7%     Ticket this week
4     1×          3d      6h     28 days    ~10.7%     Weekly review
────────────────────────────────────────────────────────────────────────────

Complete Prometheus Implementation

# Multi-Window Burn Rate Alerting — Complete Implementation
# SLO: 99.9% availability over 28-day rolling window
# All four tiers with AND-gate dual windows

groups:
  - name: slo.burnrate.recording
    interval: 30s
    rules:

      # Base SLI: request success ratio
      - record: sli:http_request_success:ratio_rate5m
        expr: |
          sum(rate(http_requests_total{
            job="api-server",
            status!~"5.."
          }[5m]))
          /
          sum(rate(http_requests_total{job="api-server"}[5m]))

      # Burn rate at multiple windows — the key recording rules
      # These power all four alert tiers
      - record: slo:error_budget_burn_rate:ratio_rate5m
        expr: (1 - sli:http_request_success:ratio_rate5m) / (1 - 0.999)

      - record: slo:error_budget_burn_rate:ratio_rate30m
        expr: |
          (1 - sum(rate(http_requests_total{status!~"5.."}[30m]))
               / sum(rate(http_requests_total[30m])))
          / (1 - 0.999)

      - record: slo:error_budget_burn_rate:ratio_rate1h
        expr: |
          (1 - sum(rate(http_requests_total{status!~"5.."}[1h]))
               / sum(rate(http_requests_total[1h])))
          / (1 - 0.999)

      - record: slo:error_budget_burn_rate:ratio_rate2h
        expr: |
          (1 - sum(rate(http_requests_total{status!~"5.."}[2h]))
               / sum(rate(http_requests_total[2h])))
          / (1 - 0.999)

      - record: slo:error_budget_burn_rate:ratio_rate6h
        expr: |
          (1 - sum(rate(http_requests_total{status!~"5.."}[6h]))
               / sum(rate(http_requests_total[6h])))
          / (1 - 0.999)

      - record: slo:error_budget_burn_rate:ratio_rate1d
        expr: |
          (1 - sum(rate(http_requests_total{status!~"5.."}[1d]))
               / sum(rate(http_requests_total[1d])))
          / (1 - 0.999)

      - record: slo:error_budget_burn_rate:ratio_rate3d
        expr: |
          (1 - sum(rate(http_requests_total{status!~"5.."}[3d]))
               / sum(rate(http_requests_total[3d])))
          / (1 - 0.999)

      # Budget remaining — the dashboard and gate metric
      - record: slo:error_budget_remaining:ratio
        expr: |
          1 - (
            (1 - sli:http_request_success:ratio_rate5m)
            / (1 - 0.999)
          )

  - name: slo.burnrate.alerts
    rules:

      # ── TIER 1: PAGE IMMEDIATELY ─────────────────────────────────────────
      # 14× burn, 1h long / 5m short AND-gate
      # Budget exhausts in ~2 days if burn continues
      - alert: SLO_BurnRate_P1_14x
        expr: |
          slo:error_budget_burn_rate:ratio_rate1h  > 14
          AND
          slo:error_budget_burn_rate:ratio_rate5m  > 14
        for: 2m
        labels:
          severity: critical
          tier: "1"
          burn_multiplier: "14"
        annotations:
          summary: >
            SLO burn rate at 14× — budget exhausts in ~2 days if sustained.
            Budget remaining: {{ with query "slo:error_budget_remaining:ratio" }}
            {{ . | first | value | humanizePercentage }}{{ end }}
          runbook: "https://wiki.internal/sre/runbooks/slo-burn-p1"
          dashboard: "https://grafana.internal/d/slo-burn/burn-rate"

      # ── TIER 2: PAGE WITHIN 30 MINUTES ───────────────────────────────────
      # 6× burn, 6h long / 30m short AND-gate
      # Budget exhausts in ~4.7 days if burn continues
      - alert: SLO_BurnRate_P2_6x
        expr: |
          slo:error_budget_burn_rate:ratio_rate6h   > 6
          AND
          slo:error_budget_burn_rate:ratio_rate30m  > 6
        for: 5m
        labels:
          severity: page
          tier: "2"
          burn_multiplier: "6"
        annotations:
          summary: >
            SLO burn rate at 6× — budget exhausts in ~4.7 days if sustained.
          runbook: "https://wiki.internal/sre/runbooks/slo-burn-p2"

      # ── TIER 3: TICKET (BUSINESS HOURS) ──────────────────────────────────
      # 3× burn, 1d long / 2h short AND-gate
      # Budget exhausts in ~9.3 days if burn continues
      - alert: SLO_BurnRate_Ticket_3x
        expr: |
          slo:error_budget_burn_rate:ratio_rate1d  > 3
          AND
          slo:error_budget_burn_rate:ratio_rate2h  > 3
        for: 15m
        labels:
          severity: warning
          tier: "3"
          burn_multiplier: "3"
        annotations:
          summary: >
            SLO burn rate at 3× — budget exhausts in ~9.3 days if sustained.
            Review in next engineering standup.

      # ── TIER 4: WEEKLY TREND REVIEW ──────────────────────────────────────
      # 1× burn, 3d long / 6h short AND-gate
      # Budget on pace to exhaust at window end
      - alert: SLO_BurnRate_Trend_1x
        expr: |
          slo:error_budget_burn_rate:ratio_rate3d  > 1
          AND
          slo:error_budget_burn_rate:ratio_rate6h  > 1
        for: 1h
        labels:
          severity: info
          tier: "4"
          burn_multiplier: "1"
        annotations:
          summary: >
            SLO burn on pace to exhaust budget at 28-day window end.
            Review budget trend in weekly SRE sync.

Istio STRICT mTLS: The SLI Source Matters

In environments running Istio in STRICT mTLS mode, the choice of SLI source — application metrics versus Envoy proxy metrics — determines whether the burn rate calculation captures all failure modes or only the ones the application can see.

# Istio-aware burn rate: captures mTLS-layer rejections
# that application-level metrics cannot see

- record: sli:http_request_success:ratio_rate5m
  expr: |
    sum(rate(istio_requests_total{
      reporter="destination",
      response_code!~"5.."
    }[5m]))
    /
    sum(rate(istio_requests_total{
      reporter="destination"
    }[5m]))

# Why reporter="destination" (not "source"):
#   source = measured at the caller's sidecar
#   destination = measured at the called service's sidecar
#
#   mTLS handshake failures at the destination sidecar:
#     → Appear as errors in destination metrics
#     → Do NOT appear in source metrics (connection never established)
#     → Do NOT appear in application metrics (request never reached app)
#
#   Using reporter="destination" captures:
#     ✓ Application-level 5xx errors
#     ✓ mTLS policy rejection errors
#     ✓ Sidecar proxy errors
#
#   Using reporter="source" OR application metrics misses:
#     ✗ mTLS-layer rejections (certificates expired, policy violation)
#     ✗ These create phantom budget consumption invisible to burn rate

Common Antipatterns

The Single-Window Implementation antipattern → Implementing burn rate alerting with only the long window, skipping the short-window AND-gate. This produces alerts that fire on historical burn events long after the error rate has recovered — operators respond to an ongoing emergency that resolved thirty minutes ago. The short window is not optional; it is the recency validation that prevents this failure mode.
The Threshold Without Derivation antipattern → Using the 14×/6×/3×/1× thresholds without verifying they are calibrated to your actual SLO window. The thresholds in this post are derived for a 28-day window. For a 7-day window, a 14× burn rate exhausts the budget in 12 hours — which changes the appropriate detection window and the response urgency. Derive your thresholds from your window.
The Missing Tier 4 antipattern → Implementing only the page tiers (Tier 1 and 2) and skipping the ticket and trend tiers. Tier 4 is the early warning system for budget degradation that has not yet reached urgent levels. A service that runs at sustained 1× burn for two weeks will exhaust its budget at the 28-day window end — but will never have triggered a page. Without Tier 4, this pattern is invisible until the budget is gone.
The App-Metric SLI in Istio antipattern → Computing burn rates from application-level HTTP response codes in an Istio service mesh environment. As described above, mTLS-layer failures are invisible to application metrics. In STRICT mTLS mode, a certificate rotation gone wrong or a PeerAuthentication policy misconfiguration will consume error budget without generating any application-level error signals. Use Envoy proxy metrics.
The for Duration Misconfiguration antipattern → Setting the for duration on Tier 1 alerts to more than 2 minutes. The for field adds a minimum firing duration — it requires the condition to be true for the specified time before the alert fires. For Tier 1 at 14× burn rate, every minute of for duration adds burn at 14× before the page is sent. A for: 15m on a Tier 1 alert delays notification by 15 minutes at a rate that consumes significant budget.

Maturity Progression

────────────────────────────────────────────────────────────────────────────
STAGE        ALERTING MATURITY STATE             NORTH STAR SIGNAL
────────────────────────────────────────────────────────────────────────────
Reactive     Threshold alerts on individual      Alert volume is high.
             metrics. No SLO concept.            Engineers filter noise
             No burn rate concept.               by experience. P95
                                                 latency alerts fire
                                                 on every deploy.

Defined      SLOs defined. Single-window         First burn rate alert
             burn rate alerts implemented        implemented. Some noise
             for Tier 1 only. App-level          reduction vs. threshold
             SLI source.                         alerting.

Measured     Dual-window AND-gate for all        Tier 2 and 3 alerts
             four tiers. Istio proxy SLI.        implemented. Alert
             Budget remaining recorded.          volume tracking.
             False positive rate tracked.        False positive rate
                                                 measured and declining.

Optimised    All tiers calibrated to actual      Zero false positives
             SLO window. Short window            on Tier 1 in last
             tuned for service traffic           30 days. Tier 4 trend
             characteristics. Multi-service      alerts surfacing budget
             burn rate dashboard.                degradation proactively.

Generative   Burn rate alerting extended         Burn rate model used
             to infrastructure layer:            to govern deployment
             database, cache, queue.             gates. Budget policy
             Alert quality metrics in            tiers automated from
             SRE quarterly report.               burn rate state.
────────────────────────────────────────────────────────────────────────────

Five Action Items for This Week

Verify your Tier 1 alert threshold is calibrated to your actual SLO window. Take your SLO window in hours, divide by 14. That is the time-to-exhaustion at your Tier 1 threshold. If your SLO window is 7 days (168 hours), your 14× burn rate exhausts the budget in 12 hours — which changes whether a 1-hour long window is the right detection window.
Check that your current alerting implementation uses the AND-gate, not an OR-gate or single window. Pull your alert rules and verify that both the long-window and short-window conditions are required simultaneously. A single-window rule or an OR-gate rule does not have the noise suppression properties derived in this post.
Switch your SLI source from application metrics to Istio Envoy proxy metrics if you are running a service mesh. Query istio_requests_total{reporter="destination"} versus your application-level HTTP response metrics for the same service over the same window. If the numbers differ, the delta is budget consumption that your current alerting cannot see.
Add the budget remaining recording rule and verify it is accessible from your deployment gate. The budget remaining metric is the input to the error budget policy tier classification. If your Argo CD PreSync hook cannot query it, the deployment gate has no connection to the SLO state it is supposed to enforce.
Measure your Tier 1 false positive rate over the last 30 days. Count the number of Tier 1 alerts that fired and then resolved without human intervention within 30 minutes. Each of those is a false positive. If the count is above zero, examine whether the AND-gate is correctly configured or whether the short-window threshold is too sensitive for your service's normal traffic variance.

"The burn rate model is not a set of magic numbers. It is a logical structure derived from the properties you want your alerting system to have: page at high urgency, not on transient spikes; ticket at moderate urgency, reliably; surface trends before they become emergencies. Understanding the derivation — rather than just copying the thresholds — is what enables you to adapt the model to services that are unusual: high-cardinality SLIs, multi-modal traffic patterns, services with known daily error rate variance. The AND-gate is not Google's magic. It is the logical solution to a well-specified alerting design problem."

Applying SRE Principles to Election Infrastructure: A Framework for Availability, Integrity, and Recovery

Nijo George Payyappilly — Mon, 03 Aug 2026 16:00:00 +0000

On February 3, 2020, the Iowa Democratic Party attempted to report precinct caucus results using a new mobile reporting application that had been deployed with inadequate testing, inadequate training, and no meaningful load testing against the expected reporting volume. Within hours of caucus completion, the app had failed to report results correctly for hundreds of precincts. The fallback — a telephone reporting system — was overwhelmed. Results were delayed by days. The incident did not change any election outcome; no votes were altered. But the operational failure created a public confidence crisis that persisted independently of the factual outcome, and that caused measurable and lasting damage to the institution conducting the election.

The Iowa caucus application failure is the most visible recent example of a class of election infrastructure failure that has been recurring in various forms for two decades: a new system deployed without the reliability engineering rigour that the stakes of the deployment require. The systems managing voter registration, ballot processing, and results reporting are not commercial applications. They operate under non-renewable deadlines, adversarial threat conditions, and a public verifiability requirement that commercial SLA frameworks do not contemplate. The consequence of failure is not revenue loss or customer churn — it is degraded public trust in democratic institutions.

Site Reliability Engineering does not solve the political dimensions of election administration. It does solve the engineering dimensions — and the engineering dimensions are substantial, well-defined, and addressable with the same principles that protect commercial critical infrastructure.

What Makes Election Infrastructure Uniquely Challenging

Election systems share reliability challenges with other critical infrastructure — availability requirements, adversarial threat exposure, regulatory oversight. Three characteristics distinguish them from every other system class addressed in this series.

The Non-Renewable Deadline

In financial services, a missed settlement window can be corrected the next business day. In healthcare, a system unavailability event triggers downtime procedures that degrade but do not eliminate care delivery. In elections, Election Day is a non-renewable deadline. If the voter registration lookup system is unavailable at 7:00 AM when polls open, there is no "next window." The failure must be resolved within the hours the polls are open, or provisional ballot procedures are the fallback — and provisional ballot reconciliation is itself an error-prone, resource-intensive process.

This non-renewable deadline constraint transforms the standard reliability engineering calculus. MTTR targets that are acceptable in commercial contexts — 30-minute, one-hour — are not acceptable for the Election Day operational window. For the systems that are critical during voting hours, MTTR must be measured in minutes, and the primary engineering investment is in prevention and fast detection, not remediation.

The Adversarial Threat Dimension

Commercial critical infrastructure faces adversarial threats (ransomware, DDoS). Election infrastructure faces adversarial threats with a specific additional characteristic: the adversary's goal may be to create the appearance of unreliability without actually compromising results — to create public doubt about election integrity rather than to actually alter election outcomes.

This means that election infrastructure SRE frameworks must address two distinct failure modes simultaneously: genuine operational failures that prevent accurate vote counting, and adversarial interference that creates reputational or confidence damage without altering outcomes. The audit trail and public verifiability SLIs addressed later in this post are the engineering response to the second failure mode.

The Public Verifiability Requirement

Commercial systems are accountable to their SLAs and their regulators. Election systems are accountable to the public. The audit trail is not a compliance artefact — it is the evidence base that enables public confidence in the outcome. A system that correctly counts every vote but cannot demonstrate that it did so to an independently verifiable standard has failed at one of its core requirements, regardless of whether the vote count was accurate.

This means that for election systems, the observability architecture is not just an operational tool. It is a public accountability instrument. The logs, the configuration records, the deployment history, and the operational event timeline are evidence in a potential post-election audit. They must be designed with that use case in mind.

SLI Design for Election Infrastructure

Election infrastructure requires six SLI dimensions. Four correspond to the Four Golden Signals adapted to the electoral context. Two are election-specific: Integrity and Auditability.

────────────────────────────────────────────────────────────────────────────
SIX SLI DIMENSIONS FOR ELECTION INFRASTRUCTURE

DIMENSION 1: AVAILABILITY
  Voter registration lookup:
    SLI: Fraction of voter lookup requests returning valid response
         within 3 seconds during active polling hours
    Target: 99.99% during polling window (non-renewable deadline)
    Target: 99.9% during non-polling periods

  Results reporting system:
    SLI: Fraction of precinct result submission requests
         accepted and confirmed within 30 seconds
    Target: 99.95% during results reporting window

DIMENSION 2: LATENCY
  Poll worker interface: p95 response < 2 seconds
    (poll worker queue forms with voters waiting)
  Ballot processing throughput: N ballots per hour
    (must exceed peak county ballot processing rate)
  Results reporting: precinct result confirmation < 30 seconds

DIMENSION 3: ERRORS
  Voter registration database: failed lookup rate
    (high failure rate may indicate data integrity issues)
  Ballot processing rejection rate: ballots rejected by scanner
    (anomalous rejection rate may indicate equipment failure
    or ballot stock issues)
  Results transmission failures: precincts failing to report
    (tracking, not just counting — each precinct must be accounted for)

DIMENSION 4: SATURATION
  Registration database connection pool during Election Day peak
  Results reporting API capacity at simultaneous precinct reporting
  Network bandwidth to county aggregation systems

DIMENSION 5: INTEGRITY (Election-specific)
  SLI: Fraction of ballots for which chain-of-custody records
       are complete and consistent across all system records
  Target: 100.000% — every ballot must be accountable
  Source: Automated consistency check across ballot tracking,
          scanner log, and tabulation system records

  SLI: Fraction of voter registration records where data matches
       across all authoritative sources (state voter file,
       county records, precinct assignments)
  Target: 99.99% — inconsistencies create provisional ballot burden
  Source: Automated cross-system reconciliation

DIMENSION 6: AUDITABILITY (Election-specific)
  SLI: Fraction of system events (logins, configuration changes,
       data modifications) that are captured in the tamper-evident
       audit log within 60 seconds of occurrence
  Target: 100.000% — no unaudited events during election period
  Source: Audit log completeness check against event counters

  SLI: Fraction of audit log entries that are verifiably unmodified
       (cryptographic hash check on audit record integrity)
  Target: 100.000%
────────────────────────────────────────────────────────────────────────────

The Non-Renewable Deadline SLO Architecture

The standard SLO measurement window — 28 days rolling — is not meaningful for election systems that operate in distinct operational phases with radically different reliability requirements. Election infrastructure SLOs must be defined against operational phases, not calendar windows.

────────────────────────────────────────────────────────────────────────────
ELECTION INFRASTRUCTURE OPERATIONAL PHASES AND SLO TARGETS

PHASE 1 — VOTER REGISTRATION PERIOD (30–90 days before Election Day)
  Systems active: Voter registration database, online registration portal
  Traffic pattern: moderate steady load with deadline spikes
  Availability SLO: 99.9% (< 8.8 hours downtime during period)
  Latency SLO: p95 < 3 seconds for registration submissions
  Integrity SLO: 100% record consistency within 24 hours of submission
  Deployment policy: standard; changes permitted with normal gates

PHASE 2 — FINAL REGISTRATION CLOSE (7 days before Election Day)
  Systems active: All registration systems + poll book generation
  Traffic pattern: high deadline-driven spike
  Availability SLO: 99.99% (< 52 minutes downtime during period)
  Deployment policy: FROZEN — no changes to voter registration systems
    Exception: security patches with CISA advisory require joint
    approval: Election Director + State IT + CISA coordination

PHASE 3 — ELECTION DAY (Polling window: typically 12–15 hours)
  Systems active: Voter lookup, poll book access, provisional ballot
    tracking, accessibility accommodation systems
  Traffic pattern: high sustained with morning and evening peaks
  Availability SLO: 99.999% during polling hours
    (< 5 minutes outage during polling window)
  MTTR SLO: < 15 minutes for any degradation
  Deployment policy: ABSOLUTE FREEZE — no changes under any circumstances
    during polling hours
  Incident response: pre-positioned county IT staff; hot standby systems;
    offline poll book printouts as Tier 0 fallback

PHASE 4 — RESULTS REPORTING (Evening of Election Day + canvass period)
  Systems active: Results submission, aggregation, publication
  Traffic pattern: high burst at close of polls across all precincts
  Availability SLO: 99.95% during initial results reporting window
  Integrity SLO: 100% — every reported result matches source precinct
  Deployment policy: FROZEN during active reporting
  Audit requirement: every results submission logged with submitter
    identity, timestamp, precinct ID, and cryptographic signature

PHASE 5 — CANVASS AND CERTIFICATION (Days to weeks post-election)
  Systems active: Audit tools, recount support, certification workflows
  Availability SLO: 99.9% (standard business hours)
  Auditability SLO: 100% — every record must be producible on demand
  Deployment policy: standard; changes require audit trail annotation
────────────────────────────────────────────────────────────────────────────

Chaos Engineering for Election Systems

Chaos engineering — the deliberate injection of failure conditions to test system resilience — is standard SRE practice. For election systems, it requires specific constraints that commercial system chaos engineering does not.

────────────────────────────────────────────────────────────────────────────
ELECTION INFRASTRUCTURE CHAOS ENGINEERING PROTOCOL

WHEN TO RUN: Only during Phases 1 and outside election years for
             primary systems. Never during Phases 2–5.
             Dedicated non-production election test environment required.
             Production chaos testing: PROHIBITED for election systems.

SCENARIO LIBRARY FOR ELECTION INFRASTRUCTURE:

  Scenario 1: Registration Database Unavailability
    Inject: Database primary node failure
    Measure: Failover time, data consistency post-failover
    Pass criteria: Failover < 30 seconds; zero data loss; zero corruption
    Frequency: Quarterly

  Scenario 2: Election Day Peak Load Surge
    Inject: 3× expected peak load during morning poll opening simulation
    Measure: Latency degradation, error rate, autoscaling response
    Pass criteria: p95 latency < 3s; error rate < 0.01%; scale response < 90s
    Frequency: 60 days before each major election

  Scenario 3: Results Reporting Cascade
    Inject: Simultaneous result submissions from all precincts
    Measure: System throughput, queue depth, data integrity under load
    Pass criteria: 100% of submissions accepted within 60s; 100% integrity
    Frequency: 60 days before each major election

  Scenario 4: Network Partition at County Aggregation
    Inject: Network partition between precinct reporting system
            and county aggregation system
    Measure: Graceful degradation; data buffering; recovery completeness
    Pass criteria: Zero data loss; automatic recovery on reconnection
    Frequency: Annually

  Scenario 5: Audit Log Integrity Under Attack
    Inject: Simulated attempt to modify audit log records
    Measure: Tamper detection time; alerting accuracy
    Pass criteria: Tamper detected within 60 seconds; alert fires
    Frequency: Quarterly

────────────────────────────────────────────────────────────────────────────
ADVERSARIAL SCENARIO LIBRARY (Red Team, not standard chaos):
  These scenarios require security team involvement and may require
  coordination with CISA's election security advisors.

  → Credential stuffing against election worker authentication systems
  → DDoS against voter lookup APIs at Election Day volumes
  → Supply chain attack simulation against election software updates
  → Social engineering targeting IT staff with administrative access
────────────────────────────────────────────────────────────────────────────

Auditability Architecture: GitOps as Electoral Chain of Custody

The GitOps operational model — where every configuration change is a git commit, every deployment is traceable to a specific commit, and every drift from desired state is detected and logged — is a natural fit for the chain-of-custody requirements of election infrastructure. The properties that make GitOps valuable for operational governance (tamper-evident change history, automated drift detection, declarative desired state) are the same properties that satisfy post-election audit requirements.

# Argo CD Application — Election Voter Registration System
# GitOps provides chain-of-custody for all configuration changes
# Every sync event is a time-stamped, identity-attributed change record

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: voter-registration-system
  namespace: argocd
  annotations:
    # All sync events → Splunk tamper-evident audit log
    notifications.argoproj.io/subscribe.on-sync-succeeded.splunk: "election-audit"
    notifications.argoproj.io/subscribe.on-sync-failed.splunk: "election-audit"
    notifications.argoproj.io/subscribe.on-health-degraded.splunk: "election-audit"
    # Phase-based deployment gate annotation
    election.internal/operational-phase: "1"    # Updated as phases progress
    election.internal/deployment-freeze: "false" # Set to "true" in Phase 2+
spec:
  project: election-infrastructure
  source:
    repoURL: https://git.internal/election/infrastructure
    targetRevision: main
    path: clusters/election/voter-registration
  destination:
    server: https://tkg-election.internal:6443
    namespace: election-systems
  syncPolicy:
    automated:
      prune: true
      selfHeal: true    # Drift auto-remediated; every remediation is audit-logged
    syncOptions:
      - ServerSideApply=true

# Kyverno Policy — Phase-Based Deployment Gate
# Enforces election operational phase restrictions at admission time
# Phase annotation on Application resource controls what is permitted

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: election-phase-deployment-gate
  annotations:
    policies.kyverno.io/description: >
      Enforces election operational phase deployment restrictions.
      Phases 2–5 have increasingly restrictive deployment policies.
      Phase annotations on Application resources are the authoritative
      state; this policy enforces them automatically.
spec:
  validationFailureAction: Enforce
  rules:
    - name: block-deployment-during-freeze
      match:
        any:
          - resources:
              kinds: [Application]
              namespaces: [argocd]
              selector:
                matchLabels:
                  election.internal/system: "true"
      preconditions:
        all:
          - key: "{{ request.object.metadata.annotations.\"election.internal/deployment-freeze\" }}"
            operator: Equals
            value: "true"
      validate:
        message: >
          Deployment blocked: election system is in operational phase with
          deployment freeze active. Changes require Election Director +
          State IT joint approval with CISA coordination for security patches.
        deny:
          conditions:
            all:
              - key: "{{ request.object.metadata.annotations.\"election.internal/freeze-override-approved\" }}"
                operator: NotEquals
                value: "true"

CISA Election Infrastructure Security Framework Alignment

CISA designates election infrastructure as critical infrastructure and provides security guidance that maps directly to SRE operational practices.

────────────────────────────────────────────────────────────────────────────
CISA ELECTION SECURITY GUIDANCE → SRE MAPPING

CISA RECOMMENDATION: Implement and test backup systems and procedures
SRE MAPPING:
  → Hot standby voter registration database (failover < 30 seconds)
  → Offline poll book printouts as Tier 0 fallback (generated 24h before)
  → Quarterly failover tests documented as chaos engineering exercises
  → Results reporting: paper precinct tally sheets as authoritative backup

CISA RECOMMENDATION: Maintain comprehensive logs of all system activity
SRE MAPPING:
  → Splunk Enterprise: all authentication, configuration changes,
    data modifications logged with structured fields
  → GitOps audit trail: all infrastructure changes as git commits
  → Tamper-evident audit log: cryptographic hash chain on log records
  → Auditability SLI: 100% of events captured within 60 seconds

CISA RECOMMENDATION: Conduct post-election audits of equipment and software
SRE MAPPING:
  → Post-election evidence synthesis automation (Class 4)
  → Splunk query producing complete change log for election period
  → Argo CD sync history as configuration chain-of-custody
  → Results submission audit trail with cryptographic signatures

CISA RECOMMENDATION: Test incident response plans before Election Day
SRE MAPPING:
  → Chaos engineering scenario library (run 60 days before election)
  → Tabletop exercises with county IT, state IT, and CISA advisors
  → MTTR measurement against Election Day operational SLO targets
  → Pre-positioned incident response resources at county level

CISA RECOMMENDATION: Implement multifactor authentication
SRE MAPPING:
  → Kyverno policy enforcing MFA annotation on all election system
    service accounts and human access paths
  → Istio STRICT mTLS for all inter-service communication
  → Access review automation: quarterly review with automated evidence
────────────────────────────────────────────────────────────────────────────

The 2-Year Calibration Problem

One of the most difficult SRE challenges unique to election infrastructure is the calibration problem: meaningful elections occur every two to four years, which means that SLO validation cycles, load test calibration, and operational experience accumulation all operate on a dramatically slower cadence than commercial systems.

────────────────────────────────────────────────────────────────────────────
THE CALIBRATION PROBLEM: SOLUTIONS

PROBLEM: You cannot derive SOT for an Election Day load pattern from
         production data more frequently than once per election cycle.
         Error budget policy targets set from one election's data may
         not be valid for the next election's different voter turnout,
         different system version, or different threat landscape.

SOLUTION 1: Synthetic Election Day Load Testing
  Generate synthetic Election Day load profiles from:
  → Registration database size (known)
  → Historical turnout rates adjusted for current registration
  → Precinct count and submission timing model (known from statute)
  Run SOT derivation load tests using synthetic profile annually,
  not only in election years. Maintains calibration between elections.

SOLUTION 2: Primary/General Election Calibration Cascade
  Primary elections (lower turnout) are calibration runs for general
  elections (higher turnout). Use primary election operational data
  to validate and adjust SLO targets, SOT values, and chaos engineering
  scenarios before the higher-stakes general election.

SOLUTION 3: Cross-Jurisdiction Learning Consortium
  Election infrastructure reliability learnings are not competitive
  intelligence. States and counties that share operational data,
  incident reports, and load test results collectively improve the
  calibration baseline available to all participants.
  CISA's election security information sharing infrastructure (ISAC)
  is the existing mechanism; SRE operational data belongs in it.

PROBLEM: SRE staff turnover between elections means operational
         knowledge is lost between election cycles.

SOLUTION: Operational Postmortem + Runbook Corpus
  Every election cycle produces an operational postmortem documenting:
  → Actual vs. predicted load (calibration data for next cycle)
  → System failures and resolutions (runbook updates)
  → Near-misses and their detection (alerting improvements)
  → Manual interventions (automation candidates)
  This corpus is the institutional memory that survives staff turnover.
────────────────────────────────────────────────────────────────────────────

Common Antipatterns

The New Application Before Major Election antipattern → Deploying a new results reporting application or voter registration system for the first time in a major election without production load testing at election-day volumes, without parallel operation with the previous system for a full election cycle, and without a tested rollback to the previous system if the new one fails. The Iowa caucus application is the archetypal example. New election systems should be first deployed in low-stakes elections (primaries, off-cycle local elections) at least one full cycle before major general elections.
The Paper Backup Complacency antipattern → Treating the existence of paper ballot backups as a complete answer to the election infrastructure reliability requirement. Paper ballots are the ultimate integrity backstop — they provide the authoritative record from which electronic tabulation can be verified. They are not a substitute for reliable electronic systems during the voting period. Poll workers conducting a manual check-in process because the electronic poll book is unavailable process voters at approximately one-third the rate of the electronic system, creating lines that suppress turnout.
The Security-as-Isolation antipattern → Addressing the adversarial threat dimension of election infrastructure exclusively through network isolation and air-gapping, without the observability architecture needed to detect anomalous behaviour within the isolated network. Air-gapped systems that are not monitored have been compromised. The choice is not between observability and security; it is between observed security and unobserved security.
The One-Time Chaos Testing antipattern → Running chaos engineering scenarios once as part of a pre-election certification process and not repeating them. System changes, staff turnover, and infrastructure evolution between elections change the failure mode landscape. Chaos scenarios that passed two years ago do not validate that the current system, with its current configuration and current staff, will respond the same way.
The Audit Log Afterthought antipattern → Designing the election system and treating the audit log as a post-hoc addition rather than a first-class architectural requirement. Audit logs that are designed after the system are invariably incomplete — they capture the events the designers thought to add logging for, not the events that a post-election audit will actually need. Auditability SLI design must happen at system design time, not at deployment time.

Maturity Progression

────────────────────────────────────────────────────────────────────────────
STAGE        ELECTION INFRA RELIABILITY          NORTH STAR SIGNAL
────────────────────────────────────────────────────────────────────────────
Reactive     No phase-based SLOs. Chaos          New systems deployed
             testing absent. Audit log           for first time in
             is compliance afterthought.         major elections.
             Paper fallback is the              Chaos scenarios
             reliability strategy.              untested.

Defined      Phase-based SLO architecture       Phase deployment
             documented. Six SLI                freeze policy
             dimensions instrumented.           implemented. Audit
             Chaos scenario library             completeness SLI
             defined.                           instrumented.

Measured     Chaos engineering run              Election Day MTTR
             60 days before election.           measured. Post-election
             SOT derived from synthetic         postmortem corpus
             load profile. CISA guidance        established. Cross-
             mapped to SRE practices.           jurisdiction data
                                                shared.

Optimised    New systems piloted in             No Iowa-class failures.
             primaries before generals.         MTTR < 15 minutes
             GitOps chain-of-custody            during polling window.
             satisfies post-election            Audit log completeness
             audit automatically.               100% maintained.

Generative   SRE framework adopted by          CISA references
             state election authorities.        framework in election
             Cross-jurisdiction ISAC           security guidance.
             data includes SRE                  Post-election audits
             operational telemetry.             use SRE audit trail
                                                as primary evidence.
────────────────────────────────────────────────────────────────────────────

Five Action Items for This Week

Define the operational phases for your election infrastructure and document the SLO target for each phase. The phase-based SLO architecture is the most important structural change that election infrastructure SRE enables. Even if instrumentation is not yet in place, the phase definitions and targets create the policy framework that governs deployment decisions, change freeze windows, and incident response priorities.
Audit your current audit log against the Auditability SLI definition. For every system event type (login, configuration change, data modification, result submission), verify that the event is captured in a tamper-evident log within 60 seconds of occurrence. The gaps you find are the evidence base vulnerabilities that a post-election challenge would exploit.
Build the synthetic Election Day load profile and run it against your voter registration system. Use your current voter registration count, historical turnout rates, and polling hours to derive the expected peak lookup rate. Run that load profile against your system — not just the average load but the morning-open and afternoon peak patterns. The gap between what your system can handle and what Election Day requires is your most important reliability risk metric.
Document your Tier 0 fallback for each election-critical system. For the voter registration lookup system: what is the offline poll book generation process, how current is the offline poll book at any given time, and how does poll worker check-in throughput compare between electronic and offline modes? The answers to these questions determine whether your fallback actually maintains election operations or creates a throughput bottleneck that suppresses turnout.
Schedule your pre-election chaos engineering scenarios on the calendar now, 60+ days before the next election. The most commonly skipped step in election infrastructure reliability preparation is chaos engineering — because it is easy to defer and its absence is invisible until election day. Put it on the calendar with the specific scenarios, the pass/fail criteria, and the personnel required. Scheduled chaos engineering happens; intended chaos engineering does not.

"Election infrastructure is the one class of critical system where the failure consequences are simultaneously technical, operational, and constitutional. A voter registration database that is unavailable on Election Day does not just fail its SLO — it potentially disenfranchises voters whose constitutional right to participate depends on that system being available when they arrive at the polls. Site Reliability Engineering is the discipline that ensures the technical layer of democratic participation is treated with the engineering rigour that its constitutional significance demands."