On May 12, 2017, the WannaCry ransomware attack encrypted systems across the United Kingdom's National Health Service. Forty hospital trusts were directly affected. Approximately 19,000 appointments were cancelled. Ambulances were diverted. Operating theatres closed. The attack did not penetrate NHS clinical systems through a sophisticated zero-day exploit. It propagated through unpatched Windows XP machines running medical imaging software whose vendors had not released — and whose hospitals had not applied — a security patch that Microsoft had made available three months earlier.
The NHS WannaCry incident is not primarily a cybersecurity story. It is an operational maturity story. The systems that were compromised were unpatched because the change management processes that would have applied the patches conflicted with the uptime requirements of clinical systems — or were believed to, in the absence of a formal reliability framework that could quantify the trade-off. The downtime risk of patching was visible and immediate. The security risk of not patching was probabilistic and deferred. Without an error budget framework — without a formal mechanism for allocating planned downtime against a measured reliability target — the decision defaulted to the path of least immediate resistance. The ransomware attackers made the consequence of that decision concrete.
Healthcare IT is the domain where the stakes of operational maturity decisions are highest and the adoption of modern reliability engineering practices has been slowest. The reasons are structural: regulatory conservatism, long procurement cycles, decades of vendor lock-in to legacy clinical systems, and a compliance culture that conflates documentation with operational excellence. This post applies SRE principles directly to healthcare operational contexts — not as a theoretical exercise but as a practical framework for the engineering decisions that determine whether clinical systems are available when clinicians need them.
The Tiered Criticality Model
Healthcare IT systems are not uniformly critical. The SLO design mistake most commonly made in healthcare environments is applying either excessive conservatism to everything (treating email with the same reliability investment as the ICU monitoring system) or insufficient rigour to everything (applying the same 99.9% availability target to medication dispensing as to the cafeteria scheduling system). Neither approach allocates reliability investment correctly.
A tiered criticality model establishes three classes of healthcare IT system, each with distinct SLO targets, error budget policies, and engineering investment levels.
────────────────────────────────────────────────────────────────────────────
HEALTHCARE IT CRITICALITY TIERS
TIER 0 — LIFE-CRITICAL SYSTEMS
Definition: Systems whose failure or incorrect output directly threatens
patient safety within minutes. Failure is measured in lives, not SLAs.
Examples:
→ Clinical decision support systems (drug interaction checks)
→ Infusion pump and ventilator control software
→ ICU patient monitoring integration
→ Emergency department triage systems
→ Code team notification infrastructure
SLO approach: Traditional error budget policy does NOT apply.
Tier 0 systems require a modified framework:
→ Availability target: 99.999% (< 5.3 minutes downtime/year)
→ Zero-tolerance error budget: budget exhaustion triggers immediate
architectural review, not deployment freeze
→ No planned downtime during active clinical hours
→ Hot standby with < 30-second failover, tested monthly
→ Correctness SLI mandatory (availability is necessary but not sufficient)
TIER 1 — CLINICAL WORKFLOW SYSTEMS
Definition: Systems whose unavailability requires clinical staff to
switch to documented downtime procedures, degrading care quality
and increasing error risk.
Examples:
→ Electronic Health Record (EHR) — Epic, Oracle Health/Cerner
→ Pharmacy dispensing and verification systems
→ Clinical laboratory information systems (LIS)
→ Radiology PACS and RIS systems
→ Nursing documentation systems
→ Surgical scheduling and perioperative systems
SLO target: 99.99% availability (< 52 minutes downtime/year)
Error budget: Standard four-tier policy applies
→ Tier 3 freeze during Joint Commission survey windows
→ Override authority: CISO + CMO + VP Technology (clinical impact)
Planned maintenance: 2 AM–4 AM windows only; advance notice ≥ 72 hours
TIER 2 — CLINICAL SUPPORT SYSTEMS
Definition: Systems that support clinical operations but whose
unavailability does not immediately compromise patient safety.
Examples:
→ Revenue cycle management and billing systems
→ Patient scheduling and registration
→ Medical imaging archiving (non-active)
→ Staff scheduling and time management
→ Credentialing and HR systems
SLO target: 99.9% availability (< 8.8 hours downtime/year)
Error budget: Standard policy; deployment velocity unrestricted at Tier 1
Planned maintenance: Standard maintenance windows apply
────────────────────────────────────────────────────────────────────────────
CRITICAL DESIGN PRINCIPLE:
Tier 0 systems must be architecturally isolated from Tier 1 and Tier 2
systems. A change management failure in a Tier 2 billing system must not
be able to cascade to a Tier 0 clinical decision support system.
This isolation is the healthcare equivalent of blast radius management.
────────────────────────────────────────────────────────────────────────────
The Error Budget Paradox in Life-Critical Contexts
Standard SRE error budget theory holds that the budget represents the permissible unreliability that the business has agreed to accept in exchange for development velocity. An error budget of 0.1% means 0.1% of requests may fail over the measurement window — and that the business has explicitly decided this failure rate is acceptable.
For Tier 0 healthcare systems, this framing is ethically untenable. A drug interaction check system does not have an acceptable failure rate expressed as a percentage of checks. A single missed drug interaction that results in patient harm is not a budget item; it is a sentinel event. The error budget framework, applied without modification, produces the wrong organisational posture for systems in this class.
The modification required is a shift from budget as velocity enabler to budget as safety signal.
────────────────────────────────────────────────────────────────────────────
ERROR BUDGET — MODIFIED FRAMEWORK FOR TIER 0 SYSTEMS
STANDARD SRE FRAMING:
Error budget is a resource to be spent.
Healthy budget → deploy faster, accept more risk.
Exhausted budget → freeze deployments, invest in reliability.
TIER 0 MODIFIED FRAMING:
Error budget is a safety signal, not a resource.
Any budget consumption → immediate root cause investigation.
Budget consumption rate → the leading indicator of systemic risk.
Budget exhaustion → architectural review, not just deployment freeze.
SLI DESIGN FOR TIER 0: CORRECTNESS MANDATORY
Availability SLI (necessary but not sufficient):
fraction of clinical decision support queries returning a response
within 500ms
Target: 99.999%
Correctness SLI (the SLI that availability alone cannot capture):
fraction of drug interaction checks that return a result consistent
with the reference pharmacopeia database
Target: 100.000% — zero tolerance
Measurement: automated consistency checks against reference database
sampled at 1% of production volume, continuously
Completeness SLI (for systems with mandatory data fields):
fraction of patient records transferred between systems where
all mandatory clinical fields are present and non-null
Target: 99.9999% (one missed mandatory field per million transfers)
────────────────────────────────────────────────────────────────────────────
THE DOWNTIME PROCEDURE FALLACY:
Most healthcare organisations believe they have addressed Tier 0 system
downtime risk through documented paper-based downtime procedures.
This is a contingency plan for graceful failure, not a reliability posture.
Paper-based downtime procedures:
→ Increase medication error rate by 3–5× (multiple studies, ISMP data)
→ Cannot support the clinical decision support checks that prevent
the highest-consequence errors
→ Generate documentation backlogs that take hours to reconcile
after system recovery
→ Are exercised infrequently enough that staff compliance is unreliable
The SRE response to "we have downtime procedures" is:
"What is your MTTR for Tier 0 systems?"
"How frequently do you test failover?"
"What is your correctness SLI, not just your availability SLI?"
────────────────────────────────────────────────────────────────────────────
SLI Design for Healthcare Systems
The Four Golden Signals apply to healthcare IT with healthcare-specific interpretations. Two additional signals beyond the standard four are required for clinical systems: Correctness and Queue Safety.
────────────────────────────────────────────────────────────────────────────
SIX SIGNALS FOR HEALTHCARE IT OBSERVABILITY
SIGNAL 1: LATENCY
Clinical decision support: response within 500ms (clinician workflow)
EHR page load: < 2 seconds (KLAS benchmark for clinician satisfaction)
Lab result delivery: < 60 seconds from instrument result to EHR
PACS image load: < 5 seconds (diagnostic imaging workflow)
Medication dispense verification: < 3 seconds
SIGNAL 2: TRAFFIC
HL7 message throughput (ADT, ORU, ORM message volumes)
EHR session concurrency by clinical unit
Medication dispense events per hour (baseline vs. surge)
Lab order volume (leading indicator of system load)
SIGNAL 3: ERRORS
HL7 message delivery failures (NACK responses)
Interface engine queue backlog (messages awaiting delivery)
Clinical decision support null responses (no drug interaction check returned)
EHR login failures during shift change (highest-concurrency window)
Pharmacy verification system rejections
SIGNAL 4: SATURATION
EHR application server CPU during 7 AM and 3 PM shift changes
Database connection pool utilisation for core clinical applications
HL7 interface engine queue depth
PACS storage utilisation approaching capacity
SIGNAL 5: CORRECTNESS (Healthcare-specific)
Drug interaction check consistency vs. reference database
Allergy alert firing rate vs. expected rate from patient population
Lab result value range validation (result outside physiologically
plausible range = data integrity signal)
Patient matching accuracy (MPI — Master Patient Index match rate)
SIGNAL 6: QUEUE SAFETY (Healthcare-specific)
Medication orders awaiting verification queue age
(orders pending > 30 minutes in active clinical context = safety risk)
Stat lab order turnaround time vs. SLO
Critical value notification delivery confirmation rate
(critical lab values must reach ordering clinician within defined window)
Code response notification delivery latency
────────────────────────────────────────────────────────────────────────────
# Prometheus Recording Rules — Healthcare SLIs
# Sourced from HL7 interface engine metrics and EHR application telemetry
groups:
- name: healthcare.slo.tier1
interval: 30s
rules:
# SLI: EHR availability (Tier 1 — clinical workflow)
- record: sli:ehr_availability:ratio_rate5m
expr: |
sum(rate(ehr_http_requests_total{status!~"5.."}[5m]))
/
sum(rate(ehr_http_requests_total[5m]))
# SLI: HL7 message delivery success rate
# NACK responses = delivery failure; timeout = delivery failure
- record: sli:hl7_delivery:ratio_rate5m
expr: |
sum(rate(hl7_messages_acknowledged_total[5m]))
/
sum(rate(hl7_messages_sent_total[5m]))
# SLI: Medication order queue age (Queue Safety signal)
# Queue age > 30 minutes for active orders = safety threshold breach
- record: sli:medication_queue_safety:ratio
expr: |
count(medication_order_queue_age_minutes < 30)
/
count(medication_order_queue_age_minutes >= 0)
# SLI: Critical value notification delivery (Tier 0 adjacent)
# Critical lab values must reach ordering clinician within 60 minutes
- record: sli:critical_value_delivery:ratio_rate1h
expr: |
sum(rate(critical_value_notifications_delivered_ontime_total[1h]))
/
sum(rate(critical_value_notifications_issued_total[1h]))
# Error budget burn rate for EHR (Tier 1 — 99.99% target)
- record: slo:ehr_budget_burn_rate:ratio_rate1h
expr: |
(1 - sli:ehr_availability:ratio_rate5m)
/
(1 - 0.9999)
# Alert: Medication queue safety SLO breach
- alert: MedicationQueueSafety_SLOBreach
expr: sli:medication_queue_safety:ratio < 0.99
for: 5m
labels:
severity: page
tier: "1"
clinical_safety: "true"
annotations:
summary: >
Medication order queue age exceeding 30-minute safety threshold
for {{ $value | humanizePercentage }} of active orders.
Clinical safety risk: orders pending verification beyond safe window.
runbook: "https://wiki.internal/sre/runbooks/medication-queue-safety"
escalation: "clinical-informatics-oncall"
HIPAA Technical Safeguards as SLO Requirements
The HIPAA Security Rule's Technical Safeguards (45 CFR § 164.312) establish requirements for electronic Protected Health Information (ePHI) systems that translate directly into SRE operational obligations. The compliance framing and the SRE framing are different surfaces of the same engineering requirement.
────────────────────────────────────────────────────────────────────────────
HIPAA TECHNICAL SAFEGUARD → SRE MAPPING
§ 164.312(a)(1) — Access Control
HIPAA: Implement technical policies that allow access only to
authorised persons or software programs
SRE mapping: Kyverno admission controller policies enforcing
service account RBAC; Istio STRICT mTLS for
inter-service communication; automated access
review with Splunk audit trail
Toil eliminated: manual quarterly RBAC review → continuous
Kyverno policy enforcement
§ 164.312(b) — Audit Controls
HIPAA: Implement hardware, software, and procedural mechanisms
to record and examine activity in systems containing ePHI
SRE mapping: Splunk Enterprise ingesting structured audit events
from all clinical system access; Argo CD sync log
as change audit trail; automated audit evidence
synthesis (Class 4 automation)
Toil eliminated: manual audit evidence collection →
automated quarterly evidence package
§ 164.312(c)(1) — Integrity
HIPAA: Implement policies to protect ePHI from improper alteration
or destruction
SRE mapping: Correctness SLI on clinical data pipelines;
GitOps self-heal as configuration integrity control;
HL7 message delivery acknowledgement tracking
Note: Integrity here is the Correctness signal, not availability.
This is the requirement that mandates the sixth SRE signal
for healthcare environments.
§ 164.312(e)(1) — Transmission Security
HIPAA: Implement technical security measures to guard against
unauthorised access to ePHI transmitted over networks
SRE mapping: Istio STRICT mTLS across all inter-service
communication carrying ePHI; certificate rotation
automation; mTLS-aware SLI computation (Envoy proxy
metrics, not application metrics)
────────────────────────────────────────────────────────────────────────────
COMPLIANCE TOIL ELIMINATED BY SRE PRACTICES:
Manual RBAC audit (quarterly): → Kyverno continuous enforcement
Manual change evidence collection: → Argo CD audit log + Splunk query
Manual integrity checks: → Correctness SLI continuous monitoring
Manual encryption verification: → Istio mTLS policy + Kyverno admission
────────────────────────────────────────────────────────────────────────────
Operational Architecture: EHR Reliability on Kubernetes
# Error Budget Gate — EHR Deployment (Tier 1 Healthcare System)
# More conservative thresholds than standard enterprise deployments
# Clinical impact assessment required at Tier 2 budget state
apiVersion: v1
kind: ConfigMap
metadata:
name: ehr-error-budget-policy
namespace: clinical-systems
annotations:
sre.internal/policy-version: "v2.1"
sre.internal/approved-by: "sre-lead,ciso,cmo-delegate"
sre.internal/clinical-impact-review: "required-at-tier-2"
sre.internal/joint-commission-freeze: "active-during-survey"
data:
policy.yaml: |
slo_target: 0.9999 # 99.99% — Tier 1 clinical workflow
tiers:
tier_1_healthy:
budget_remaining_threshold: 0.80 # More conservative than standard 75%
permitted:
- "standard-release-cadence"
- "feature-flags-max-5pct"
restricted: []
tier_2_degraded:
budget_remaining_threshold: 0.40 # Conservative midpoint
restricted:
- "max-1-deploy-per-week" # Weekly, not daily — clinical context
- "requires-clinical-informatics-sre-approval"
- "requires-clinical-impact-assessment"
required_notifications:
- "clinical-informatics-team"
- "nursing-informatics"
tier_3_exhausted:
budget_remaining_threshold: 0.20 # Tighter than standard 25%
prohibited:
- "all-deployments-except-patient-safety-p0"
required:
- "joint-sre-cmo-review-within-24h"
- "patient-safety-committee-notification"
- "risk-management-notification"
special_windows:
joint_commission_survey:
description: "No deployments during accreditation survey windows"
freeze_type: "absolute"
override_authority: "CEO + CMO + CIO joint approval"
regulatory_reporting:
description: "CMS quality reporting periods — Tier 2 restrictions apply"
freeze_type: "tier_2_equivalent"
# Kyverno Policy — ePHI Namespace Isolation
# Enforces that clinical systems carrying ePHI cannot communicate
# with non-clinical namespaces without explicit policy exception
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: ephi-namespace-isolation
annotations:
policies.kyverno.io/description: >
Services in ePHI-classified namespaces (tier-0, tier-1-clinical)
may only communicate with other ePHI-classified namespaces or
explicitly approved external services. Prevents accidental ePHI
exposure through misconfigured service routing. HIPAA §164.312(e)(1).
spec:
validationFailureAction: Enforce
rules:
- name: restrict-ephi-namespace-egress
match:
any:
- resources:
kinds: [NetworkPolicy]
namespaces:
- tier-0-clinical
- tier-1-clinical
- tier-1-pharmacy
validate:
message: >
NetworkPolicy in ePHI namespace must not allow egress to
non-ePHI namespaces without explicit HIPAA exception annotation.
deny:
conditions:
all:
- key: "{{ request.object.metadata.annotations.\"hipaa.internal/ephi-exception\" }}"
operator: NotEquals
value: "approved"
- key: "{{ request.object.spec.egress[].to[].namespaceSelector }}"
operator: AnyNotIn
value:
- tier-0-clinical
- tier-1-clinical
- tier-1-pharmacy
- monitoring
- sre-platform
Toil Elimination in Clinical IT Operations
Healthcare IT operations carry some of the highest-density compliance toil in any regulated sector. The quarterly Joint Commission audit preparation, the HIPAA access review cycles, the change control documentation for clinical systems, and the manual reconciliation of HL7 interface logs are all automatable — and all generate significant toil that displaces reliability engineering investment.
Joint Commission evidence automation → Splunk queries that automatically assemble change management evidence, access review records, and system availability data into structured audit packages. The same GitOps audit trail that satisfies CIP-010 in energy environments satisfies Joint Commission IT standards in healthcare.
HL7 interface monitoring automation → Automated detection of interface engine queue backlog, NACK rate elevation, and message transformation errors via Prometheus recording rules. Eliminates the manual log review that on-call clinical informatics staff perform at shift change.
Medication order queue alerting → The Queue Safety SLI alert fires automatically when orders are aging beyond the safe window. Eliminates the manual "check the queue" workflow that charge nurses perform hourly on paper rounds.
Downtime procedure drill automation → Scheduled quarterly failover tests executed via Argo Workflows against the non-production clinical environment. Eliminates the manual coordination overhead of failover drills and ensures clinical staff maintain downtime procedure proficiency without requiring a production incident.
Common Antipatterns
The Downtime Procedure Posture antipattern → Treating documented paper-based downtime procedures as a reliability posture rather than a failure contingency. Downtime procedures accept failure; SRE prevents it. A healthcare organisation whose reliability strategy is "we know how to fail gracefully" has not adopted SRE; it has refined its failure acceptance protocol.
The Single-Tier SLO antipattern → Applying a single availability target to all clinical systems regardless of patient safety impact. A 99.9% SLO applied to a drug interaction check system means that system may be unavailable for approximately 8.8 hours per year. In an active clinical environment, that 8.8 hours represents thousands of medication orders processed without automated interaction checking.
The Compliance-as-Availability antipattern → Treating HIPAA compliance audits as evidence of system reliability. HIPAA compliance measures whether the organisation documented and controlled access to ePHI. It does not measure whether clinical systems were available when clinicians needed them, whether medication interaction checks returned correct results, or whether critical lab values were delivered within the safe notification window.
The Vendor SLA Substitution antipattern → Accepting the EHR vendor's contractual SLA as the de facto SLO. Vendor SLAs measure vendor infrastructure uptime — they do not measure end-to-end clinical workflow availability inclusive of integration layers, network infrastructure, authentication systems, and the interface engine that connects the EHR to every other clinical system in the hospital.
The Change Freeze Overcorrection antipattern → Implementing change freezes so broad and long (six-month freezes around Joint Commission surveys) that the organisation is unable to apply security patches, fix patient safety defects, or implement regulatory-required changes during the freeze window. The correct response to Joint Commission survey periods is a tighter error budget tier, not a blanket freeze.
Maturity Progression
────────────────────────────────────────────────────────────────────────────
STAGE HEALTHCARE IT RELIABILITY STATE NORTH STAR SIGNAL
────────────────────────────────────────────────────────────────────────────
Reactive Single-tier reliability model. Downtime measured in
Downtime procedures are the "events per year" not
reliability strategy. Vendor minutes. No MTTR SLO.
SLA = organisational SLO. Correctness unmeasured.
Defined Tiered criticality model Tier 0/1/2 classification
documented. SLIs defined for complete. Correctness
Tier 0 and Tier 1 systems. SLI instrumented for
HIPAA safeguards mapped Tier 0 systems.
to SRE practices.
Measured Error budget policy active Tier 1 burn rate alerts
for Tier 1. Tier 0 correctness replacing threshold
SLI monitored. HIPAA audit alerts. Medication
evidence automated. queue safety SLI live.
Optimised Tier 0 hot standby tested MTTR < 30 minutes for
monthly. Downtime drills Tier 1. Zero correctness
automated. Toil Ratio SLO breaches for Tier 0.
below 35% (clinical IT Compliance evidence
compliance overhead generated automatically.
accounted for).
Generative Healthcare reliability SRE framework adopted
framework shared across by peer health systems.
health system network. Regulatory bodies aware
Joint Commission familiar of SLO-based governance.
with SLO governance model. Patient safety metrics
correlated with SLO data.
────────────────────────────────────────────────────────────────────────────
Five Action Items for This Week
Classify every clinical IT system you are responsible for into Tier 0, 1, or 2. The classification forces the question that most healthcare IT organisations avoid: which of our systems, if unavailable or incorrect, directly threatens patient safety within minutes? That question has a shorter answer list than most clinical IT teams expect — and a longer list than most clinical IT executives believe.
Define a Correctness SLI for your highest-Tier 0 system. Availability is necessary but insufficient for life-critical systems. What is the measurable signal that tells you the drug interaction check returned the right answer, not just an answer? Defining this SLI is the first step toward instrumenting it.
Measure your actual MTTR for your most critical clinical system, not your documented RTO. RTO is what you committed to achieving. MTTR is what you have actually achieved. For most healthcare organisations, these numbers are significantly different. The gap between them is the engineering investment required to make the commitment real.
Map your HIPAA Technical Safeguard compliance activities to the toil taxonomy. Identify which HIPAA compliance activities are manual, repetitive, and automatable. The quarterly access review, the audit evidence collection, the change documentation — classify each by automation class and calculate the toil hours per quarter. This is your compliance toil reduction business case.
Test your Tier 0 failover — this quarter, not at the next planned drill. Schedule an unannounced failover test for one Tier 0 system during off-hours. Measure the actual failover time. Compare it against your documented RTO. If the test reveals a gap, you have identified the most important reliability engineering investment in your healthcare IT environment.
"Healthcare organisations that treat availability as a compliance checkbox and reliability as a vendor contractual term are systematically underinvesting in the engineering discipline that prevents the class of failures that harm patients. Downtime procedures are not a reliability posture — they are a managed acceptance of failure. Site Reliability Engineering is the discipline that moves healthcare IT from managing failure gracefully to preventing it systematically."
Top comments (0)