In 2024, a practitioner at a financial services firm deployed an LLM-based root cause analysis tool into their incident response workflow. The tool had passed all technical integration tests. It connected to their observability stack correctly, pulled telemetry during incidents, and returned structured diagnoses in the expected format. It had been evaluated against ten historical incidents by two engineers who found the recommendations "generally helpful."
Three months later, a postmortem review revealed that in fourteen of the forty-seven incidents where the tool had been consulted, the tool's confident diagnosis had led the responding engineer away from the actual root cause — delaying resolution by an average of twenty-two minutes per misdirected incident. The tool was not obviously wrong. It was plausibly wrong, in specific ways that a time-pressured on-call engineer was likely to accept rather than challenge. The cost of those fourteen incidents, measured in Regulatory MTTR and error budget consumption, significantly exceeded any benefit the tool provided.
The failure was not the tool's integration. It was the absence of a reliability evaluation framework. The tool was tested for function (does it connect? does it respond?) but not for reliability (is it correct? is it consistently correct? is it calibrated about its own uncertainty?). These are the questions that SRE asks about every production dependency. They were not asked about the AI system until the evidence of systematic misdirection had accumulated in the postmortem record.
This post provides the evaluation framework that should have preceded that deployment.
The LLM as a Production Dependency
An LLM used for incident diagnosis is a production dependency in the same sense that a database, a downstream API, or an authentication service is a production dependency. It has inputs, outputs, latency, error rates, and failure modes. It has a reliability profile that can be measured, a performance envelope that has limits, and a degradation pattern that develops over time as the gap between its training data and the current environment widens.
Standard software testing is insufficient for LLMs because LLM failure modes are fundamentally different from deterministic software failure modes:
────────────────────────────────────────────────────────────────────────────
LLM FAILURE MODES VS DETERMINISTIC SOFTWARE FAILURE MODES
DETERMINISTIC SOFTWARE:
Failure mode: Returns error code; times out; returns wrong type
Detection: Standard integration tests catch these
Characteristic: Failures are generally consistent and reproducible
LLM FAILURE MODES (not detectable by standard integration tests):
→ Confidently wrong: Returns a plausible but incorrect diagnosis
with high confidence score. No error code. No exception.
Appears to function correctly.
→ Inconsistently right: Returns correct diagnosis on 70% of
invocations for the same incident, wrong diagnosis on 30%.
Passes integration tests that check one invocation.
→ Distribution shift degradation: Was reliable at deployment;
degrades silently as environment evolves and the gap between
training data and current incident patterns widens.
→ Context length sensitivity: Correct with minimal context;
incorrect when context exceeds a threshold (token limit effects,
attention dilution). Standard integration tests use minimal
context; production uses full context.
→ Confidence miscalibration: Reports 0.90 confidence when
empirically accurate 65% of the time. Escalation policies
that gate on reported confidence are then systematically
wrong about autonomy thresholds.
────────────────────────────────────────────────────────────────────────────
The reliability evaluation framework must address all five failure modes. Standard integration testing addresses none of them.
The LLM Reliability Test Framework
Component 1: Historical Incident Test Dataset Construction
The foundation of LLM reliability evaluation is a ground-truth dataset of historical incidents where the correct diagnosis has been independently confirmed. This dataset is the benchmark against which all four reliability dimensions (accuracy, calibration, consistency, explainability) are measured.
────────────────────────────────────────────────────────────────────────────
TEST DATASET CONSTRUCTION PROTOCOL
STEP 1: Identify candidate incidents
Source: PagerDuty or equivalent incident management system
Filter: P1/P2 incidents only (highest quality postmortems)
Time window: Last 18 months (sufficient volume; recent enough to be
representative of current system state)
Target size: Minimum 50 incidents; 100+ for statistical validity
STEP 2: Confirm root cause ground truth
Requirement: Root cause must be independently confirmed
(not just "as stated in the postmortem" — postmortems are not always
accurate; some attribute to proximate cause rather than root cause)
Ground truth confirmation sources (in order of reliability):
a) Engineering postmortem with specific code change identified
b) Engineering postmortem with specific configuration identified
c) Peer-reviewed incident analysis (two engineers agree)
d) Postmortem with remediation that resolved and did not recur
Exclude: Incidents where root cause remains disputed or "unknown"
Exclude: Incidents where symptoms match multiple simultaneous causes
STEP 3: Reconstruct the incident context as it appeared at alert time
This is the most labour-intensive step and the most critical.
The test must evaluate the LLM on the context that was available
WHEN THE INCIDENT OCCURRED — not on hindsight context.
Context to reconstruct:
→ Prometheus metrics in the 30-minute window before first alert
→ Splunk log events in the 30-minute window before first alert
→ Recent deployments (Argo CD sync events, last 24 hours)
→ Recent configuration changes (git commits, last 7 days)
→ Active alerts at time of incident creation
Context to EXCLUDE (post-hoc information not available at alert time):
→ Logs generated after the incident started (investigation artefacts)
→ Configuration changes made during incident response
→ Any information that assumes knowledge of the outcome
STEP 4: Label the dataset
For each incident, record:
→ incident_id, timestamp, affected_service
→ confirmed_root_cause (structured: category + specific cause)
→ root_cause_category (deployment, infrastructure, dependency,
configuration, capacity, external)
→ severity (P1/P2), duration_minutes, budget_consumed_pct
DATASET QUALITY METRIC:
Ground truth confidence score: fraction of incidents where
root cause is confirmed by source type (a) or (b) above.
Target: > 80% of dataset confirmed at source (a) or (b) level.
Datasets with high proportions of source (c) or (d) produce
noisier accuracy measurements.
────────────────────────────────────────────────────────────────────────────
Component 2: Accuracy Testing — Precision and Recall
# LLM Accuracy Evaluation — Automated Test Runner
# Presents reconstructed incident context to LLM and measures
# diagnosis accuracy against ground truth dataset
import json
from dataclasses import dataclass
from typing import Optional
@dataclass
class IncidentTestCase:
incident_id: str
context: dict # Reconstructed at-alert-time context
ground_truth: dict # Confirmed root cause
@dataclass
class DiagnosisResult:
incident_id: str
llm_diagnosis: str
confidence_score: float
model_used: str
latency_ms: int
is_correct: Optional[bool] = None # Set after evaluation
def evaluate_accuracy(
test_cases: list[IncidentTestCase],
llm_client,
evaluator_fn
) -> dict:
"""
Run accuracy evaluation against historical incident test dataset.
Returns precision, recall, and per-category breakdown.
"""
results = []
for case in test_cases:
# Present context to LLM (same prompt as production)
response = llm_client.diagnose(
context=case.context,
prompt_template=PRODUCTION_PROMPT_TEMPLATE
)
# Evaluate correctness against ground truth
# (evaluator_fn uses semantic similarity + keyword matching)
is_correct = evaluator_fn(
diagnosis=response.diagnosis,
ground_truth=case.ground_truth,
threshold=0.80 # 80% semantic similarity = correct
)
results.append(DiagnosisResult(
incident_id=case.incident_id,
llm_diagnosis=response.diagnosis,
confidence_score=response.confidence,
model_used=response.model,
latency_ms=response.latency_ms,
is_correct=is_correct
))
# Calculate metrics
total = len(results)
correct = sum(1 for r in results if r.is_correct)
diagnosed = sum(1 for r in results if r.llm_diagnosis is not None)
precision = correct / diagnosed if diagnosed > 0 else 0
recall = correct / total # All incidents have ground truth
# Per root-cause-category breakdown
by_category = {}
for case in test_cases:
cat = case.ground_truth["root_cause_category"]
category_results = [r for r in results
if r.incident_id == case.incident_id]
if category_results:
by_category.setdefault(cat, {"correct": 0, "total": 0})
by_category[cat]["total"] += 1
if category_results[0].is_correct:
by_category[cat]["correct"] += 1
return {
"precision": precision,
"recall": recall,
"total_incidents": total,
"by_category": {
cat: data["correct"] / data["total"]
for cat, data in by_category.items()
},
"gate_1_pass": precision >= 0.70 and recall >= 0.60
}
Component 3: Calibration Testing
# Calibration Testing — Confidence Score vs Empirical Accuracy
# Measures whether reported confidence correlates with actual accuracy
def evaluate_calibration(results: list[DiagnosisResult]) -> dict:
"""
Measure confidence calibration: does reported confidence
match empirical accuracy rate?
"""
confidence_bands = {
"90-100": {"min": 0.90, "max": 1.01},
"85-89": {"min": 0.85, "max": 0.90},
"80-84": {"min": 0.80, "max": 0.85},
"70-79": {"min": 0.70, "max": 0.80},
"60-69": {"min": 0.60, "max": 0.70},
"below-60": {"min": 0.0, "max": 0.60}
}
calibration = {}
for band_name, bounds in confidence_bands.items():
band_results = [r for r in results
if bounds["min"] <= r.confidence_score < bounds["max"]]
if len(band_results) >= 5: # Minimum sample for validity
empirical_accuracy = sum(1 for r in band_results if r.is_correct) / len(band_results)
mean_confidence = sum(r.confidence_score for r in band_results) / len(band_results)
calibration_delta = empirical_accuracy - mean_confidence
calibration[band_name] = {
"sample_size": len(band_results),
"empirical_accuracy": empirical_accuracy,
"mean_stated_confidence": mean_confidence,
"calibration_delta": calibration_delta,
"status": "CALIBRATED" if abs(calibration_delta) <= 0.10 else "MISCALIBRATED"
}
# Gate 2 pass: all bands with sufficient samples are calibrated
gate_pass = all(
band["status"] == "CALIBRATED"
for band in calibration.values()
if band["sample_size"] >= 10
)
return {"calibration_by_band": calibration, "gate_2_pass": gate_pass}
Component 4: Consistency Testing
# Consistency Testing — Same input, same output?
# Critical for autonomous execution: inconsistent diagnoses cannot
# be trusted for policy-governed automatic remediation
def evaluate_consistency(
test_cases: list[IncidentTestCase],
llm_client,
n_reruns: int = 3
) -> dict:
"""
Test consistency: present the same incident context N times
and measure semantic agreement across responses.
"""
consistency_scores = []
for case in test_cases[:30]: # Sample: 30 incidents × 3 reruns = 90 calls
responses = []
for _ in range(n_reruns):
response = llm_client.diagnose(
context=case.context,
prompt_template=PRODUCTION_PROMPT_TEMPLATE,
temperature=0.1 # Low temperature for consistency testing
)
responses.append(response.diagnosis)
# Pairwise semantic similarity across all response pairs
pairs = [(responses[i], responses[j])
for i in range(len(responses))
for j in range(i+1, len(responses))]
pairwise_similarities = [
semantic_similarity(a, b) for a, b in pairs
]
incident_consistency = sum(pairwise_similarities) / len(pairwise_similarities)
consistency_scores.append(incident_consistency)
mean_consistency = sum(consistency_scores) / len(consistency_scores)
return {
"mean_consistency_score": mean_consistency,
"incidents_tested": len(consistency_scores),
"gate_3_pass": mean_consistency >= 0.85
}
Component 5: Context Length Stress Testing
This component is unique to LLM reliability evaluation and has no equivalent in standard software testing.
# Context Length Stress Test
# Tests whether diagnosis accuracy degrades with increasing context size
# Production incidents often have larger context than lab tests
def evaluate_context_length_sensitivity(
test_cases: list[IncidentTestCase],
llm_client
) -> dict:
"""
Test diagnosis accuracy at multiple context sizes.
Reveals context length threshold where accuracy degrades.
"""
context_sizes = ["minimal", "standard", "extended", "maximum"]
results_by_size = {}
for size_label in context_sizes:
size_results = []
for case in test_cases[:20]: # 20 incidents per size
# Build context at specified size
context = build_context_at_size(case.context, size_label)
response = llm_client.diagnose(context=context)
is_correct = evaluate_diagnosis(response.diagnosis, case.ground_truth)
size_results.append({
"incident_id": case.incident_id,
"context_tokens": len(context) // 4, # approximate
"is_correct": is_correct,
"confidence": response.confidence,
"latency_ms": response.latency_ms
})
accuracy = sum(1 for r in size_results if r["is_correct"]) / len(size_results)
avg_latency = sum(r["latency_ms"] for r in size_results) / len(size_results)
results_by_size[size_label] = {
"accuracy": accuracy,
"avg_latency_ms": avg_latency,
"avg_tokens": sum(r["context_tokens"] for r in size_results) / len(size_results)
}
# Identify degradation threshold
accuracies = [results_by_size[s]["accuracy"] for s in context_sizes]
degradation_threshold = None
for i in range(1, len(accuracies)):
if accuracies[i] < accuracies[i-1] * 0.85: # >15% accuracy drop
degradation_threshold = context_sizes[i]
break
return {
"by_context_size": results_by_size,
"degradation_threshold": degradation_threshold,
"gate_4_pass": degradation_threshold != "standard"
# Fail if accuracy degrades at standard context size (production representative)
}
The Five PoC Exit Gates: Complete Specification
────────────────────────────────────────────────────────────────────────────
LLM INCIDENT DIAGNOSIS PoC EXIT GATES
GATE 1 — ACCURACY (Technical Performance)
Pass criteria:
Precision ≥ 0.70 (correct root cause in ≥ 70% of diagnoses made)
Recall ≥ 0.60 (identifies root cause in ≥ 60% of all test incidents)
Deployment-caused incidents: precision ≥ 0.80 (highest-volume category)
Measurement: Automated accuracy evaluation on 50+ test cases
Sample required: ≥ 50 P1/P2 incidents with confirmed ground truth
Failure response: Review prompt engineering; assess context completeness;
evaluate model selection
GATE 2 — CALIBRATION
Pass criteria:
Calibration delta within ±0.10 for all confidence bands ≥ 0.60
No confidence band shows systematic overconfidence > 0.15
Measurement: Calibration evaluation on same 50+ test cases
Failure response: Adjust confidence thresholds in escalation policy;
if systematic overconfidence: apply calibration
correction factor before gating
GATE 3 — CONSISTENCY
Pass criteria:
Mean semantic consistency ≥ 0.85 across 30 test incidents × 3 reruns
No individual incident shows consistency < 0.60 (outlier cap)
Measurement: Rerun test suite with same 30 incidents × 3 times
Failure response: Reduce temperature; review prompt for ambiguity;
evaluate model quantisation effects
GATE 4 — CONTEXT LENGTH STABILITY
Pass criteria:
Accuracy does not degrade by > 15% at standard context size
(representative of typical production incident context)
Latency at maximum context: < 120 seconds (Ollama), < 60 seconds (external)
Measurement: Context size stress test at 4 levels
Failure response: Reduce context assembly scope; prioritise high-signal
context (recent deployments, correlated alerts) over
full log ingestion
GATE 5 — ENGINEER TRUST SURVEY
Pass criteria:
SRE team mean score ≥ 3.5/5.0 across five dimensions
No individual respondent < 2.0 (veto threshold)
Completion rate: 100% of on-call SRE engineers surveyed
Measurement: Structured survey after 30-day shadow mode
Survey dimensions:
Q1: Recommendations are relevant to the actual incident (1–5)
Q2: I can verify a recommendation in under 5 minutes (1–5)
Q3: The tool changes how I approach investigation (1–5)
Q4: I would trust this recommendation enough to act on it (1–5)
Q5: The explanations help me understand the reasoning (1–5)
Failure response: This gate cannot be fixed with technical changes.
Failure means the tool is not ready for the team,
regardless of technical performance. Return to
shadow mode; understand specific objections;
address them before re-surveying.
────────────────────────────────────────────────────────────────────────────
GATE SEQUENCE AND DEPENDENCIES:
Gates 1–4 can be evaluated in parallel (all from test dataset)
Gate 5 requires Gates 1–4 to pass first (survey shadow mode starts
only after technical validation is complete)
All five gates must pass before any operational authority is granted
────────────────────────────────────────────────────────────────────────────
Continuous Reliability Monitoring Post-Deployment
The PoC evaluation establishes the deployment baseline. Continuous monitoring maintains it.
# Prometheus: LLM Diagnosis Reliability SLIs
# Continuous measurement of all four reliability dimensions post-deployment
groups:
- name: llm.diagnosis.reliability
rules:
# Accuracy proxy: fraction of acted-on recommendations
# that led to incident resolution (tracked post-incident)
- record: llm:diagnosis_accuracy:rate7d
expr: |
sum(rate(llm_diagnosis_resolution_confirmed_total[7d]))
/
sum(rate(llm_diagnosis_acted_on_total[7d]))
# Operator override rate (high = operators don't trust diagnoses)
- record: llm:operator_override_rate:rate7d
expr: |
sum(rate(llm_diagnosis_operator_reversed_total[7d]))
/
sum(rate(llm_diagnosis_acted_on_total[7d]))
# Calibration delta (production: rolling comparison)
- record: llm:confidence_calibration_delta:rate30d
expr: |
abs(
avg_over_time(llm_diagnosis_confidence_score_avg[30d])
- avg_over_time(llm_diagnosis_empirical_accuracy_avg[30d])
)
# Composite trust score (same weights as AIOps Trustworthy post)
- record: llm:trust_score:composite
expr: |
(llm:diagnosis_accuracy:rate7d * 0.40)
+
((1 - clamp_max(llm:confidence_calibration_delta:rate30d, 0.30)) * 0.20)
+
(llm:diagnosis_consistency:rate7d * 0.20)
+
((1 - llm:operator_override_rate:rate7d) * 0.20)
# Trust score degradation alert
- alert: LLM_DiagnosisTrustScoreDegraded
expr: llm:trust_score:composite < 0.65
for: 6h
labels:
severity: ticket
domain: ai_reliability
annotations:
summary: >
LLM diagnosis trust score at {{ $value | humanize }}.
Accuracy or calibration degrading — review recent incident corpus
for distribution shift. Consider re-evaluation.
Common Antipatterns
The Function-Not-Reliability Test antipattern → Evaluating the LLM for correct integration (does it call the right API? does it return structured JSON?) without evaluating for correct diagnosis (is the structured JSON's content right?). Integration tests verify that the system works; reliability tests verify that it works correctly. For LLMs, these are entirely different evaluations.
The Convenience Sample antipattern → Building the test dataset from incidents that are "easy to understand" or where the root cause is "obvious" — which means the LLM is being evaluated on the incidents where a human could diagnose correctly in five minutes anyway. The test dataset must include the hard incidents: multi-service cascades, novel failure modes, ambiguous signals. Easy test cases produce inflated accuracy scores that do not predict production performance.
The Threshold Without Calibration antipattern → Setting the confidence threshold for autonomous execution at 0.85 because that "sounds right" without verifying that 0.85 confidence actually predicts 85% empirical accuracy. As demonstrated in the calibration testing section, confidence scores and empirical accuracy can differ substantially. Thresholds must be calibrated against empirical data.
The One-Time Evaluation antipattern → Running the five gates once at deployment and treating the results as permanent. LLM diagnosis reliability decays as the environment changes. The test dataset must be refreshed with new incidents quarterly, and the gate evaluation must be repeated. A system that passed the gates 18 months ago may not pass them today.
The Shadow Mode Contamination antipattern → Showing shadow mode recommendations to the responding engineer during the evaluation period. Shadow mode is only valid if the engineer cannot see the AI's recommendation during the incident — otherwise they will unconsciously anchor on the AI diagnosis, contaminating both the human decision and the "was the AI right?" evaluation. Shadow mode recommendations should be logged to a private channel that is reviewed only after the incident is resolved.
Five Action Items for This Week
Build your historical incident test dataset. Pull the last 18 months of P1/P2 incidents from your incident management system. Filter to those with confirmed root causes in postmortems. Reconstruct the at-alert-time context for the top 50 by business impact. This dataset is the prerequisite for every subsequent gate evaluation.
Run Gate 3 (consistency) against your current LLM deployment. Present 30 historical incidents to the model three times each and measure semantic similarity. This is the fastest gate to run (no ground truth labelling required) and the most commonly failing gate in practice. High inconsistency is a deployment risk that most organisations discover after, not before, granting operational authority.
Add the trust score Prometheus recording rules to your observability stack. Even if you are in shadow mode, begin collecting the production data that the trust score requires. Establishing the baseline during shadow mode means the post-deployment trust score trend has context.
Design your evaluator function for Gate 1. The accuracy evaluation requires a function that compares LLM diagnosis text to ground truth root cause labels. Semantic similarity works for continuous scoring; keyword-based matching works for categorical scoring. Both have error rates — document them, because the evaluator function's accuracy is a meta-level uncertainty in the Gate 1 measurement.
Schedule the Gate 5 survey before starting shadow mode. The survey is the gate that most frequently blocks deployment. It can only be conducted after 30 days of shadow mode — and it is frequently delayed indefinitely when not calendared in advance. Block the time, draft the survey instrument, and communicate the evaluation timeline to the SRE team before shadow mode begins.
"An LLM used for incident diagnosis is not a search engine with better UX. It is a system that makes claims about causality in complex distributed systems, under time pressure, to engineers whose response to those claims has operational consequences. The reliability of those claims — their accuracy, calibration, consistency, and context stability — is an engineering property that can be measured. The organisations that measure it before granting operational authority will have smaller incident postmortems to write. The organisations that don't will write the case studies that everyone else learns from."
Series Complete
This post concludes the 25-post series on applying Google SRE principles to enterprise organisations and national-scale critical infrastructure. The complete series covers:
- Pillar 1: Critical infrastructure reliability in healthcare, financial services, energy, and election systems
- Pillar 2: Original technical contributions — burn rate alerting, error budget decision theory, autoscaling strategy selection, SOT derivation
- Pillar 3: AI-assisted operations — trustworthy AIOps frameworks, human-in-the-loop governance, on-premises LLM gateways, LLM reliability evaluation
- Pillar 4: Workforce and organisational transformation — the talent gap, the phased influence playbook, the DevOps-to-SRE roadmap, DORA as national policy
- Pillar 5: Peer recognition and field contribution — the body of knowledge, the open letter to NIST, the adoption survey, the replication study
Each post is a standalone contribution. Together, they constitute an argument that Site Reliability Engineering is a national infrastructure discipline — one with the technical rigour, the policy relevance, and the workforce implications that warrant both professional recognition and sustained investment.
Canonical references: Google SRE Book · Google SRE Workbook · DORA Research · NIST CSF 2.0 · NIST AI RMF
Top comments (0)