On April 22, 2018, TSB Bank in the United Kingdom began migrating 5.4 million customer accounts from Lloyds Banking Group's legacy IT infrastructure to its own platform. The migration had been planned for two years. Within hours of the cutover, 1.9 million customers were unable to access their accounts. Some customers could see other customers' account data. Business banking customers were seeing personal account balances that were not theirs. The fraud detection system began blocking legitimate transactions. TSB's CEO later testified to the UK Parliament that the problems stemmed from untested dependencies, inadequate migration validation, and a change management process that was not calibrated to the risk of the migration it was executing.
The TSB migration failure cost the bank approximately £330 million in remediation, compensation, and regulatory fines. It took five months to fully resolve. The CEO resigned. The UK Financial Conduct Authority opened a formal investigation.
The TSB incident is a textbook case of what happens when the reliability engineering practices governing a change are not commensurate with the blast radius of that change. The migration was not a standard deployment. It was a transaction that simultaneously affected 5.4 million accounts, moved a decade's worth of customer data, and had no tested rollback path. The change management framework applied to it did not reflect any of those characteristics. The error budget policy — had one existed — would have made the risk visible and either mandated a staged rollout or triggered the override authority escalation that would have forced the explicit risk acceptance conversation before the cutover, not five months after it.
Why Financial Services Is Different
Financial services reliability engineering operates under three constraints that distinguish it from every other sector:
Constraint 1 — Correctness is a first-class SLI, co-equal with availability. A payment processing system that is available but settling transactions to the wrong accounts is more dangerous than a system that is simply unavailable. Unavailability triggers fallback procedures and halts new transactions. Incorrect settlement propagates through the interbank network, creates counterparty credit exposure, and generates reconciliation failures that cascade for days after the originating error is corrected. The correctness SLI is not an afterthought in financial services SLO design; it is the primary safety constraint.
Constraint 2 — Latency SLOs have regulatory dimensions. For payment systems operating on NACHA ACH rules, Fedwire operating procedures, and SWIFT messaging standards, latency is not a user experience metric — it is a regulatory compliance metric. A Fedwire payment that is not settled within the operating window has legal consequences for the sending institution. A card network transaction that exceeds authorization time limits triggers a fallback path that has different liability implications than an approved transaction. The latency SLO governs legal obligation, not customer satisfaction.
Constraint 3 — Individual institution failures create systemic risk. Unlike healthcare, where a hospital's system failure harms that hospital's patients, a major financial institution's reliability failure can create systemic risk that extends across the entire financial system. Fedwire processes approximately $4 trillion in interbank settlements per business day. A settlement failure by a large participant does not stay contained to that participant — it creates overnight liquidity requirements for every institution that was expecting a settlement that did not arrive, and in extreme cases requires Federal Reserve intervention to prevent cascade effects.
The regulatory floor argument: OCC SR 21-3 establishes operational resilience expectations that, when expressed as RTOs and RPOs for critical business services, effectively require 99.99% availability or better for the systems that support those services. 99.99% is not an ambitious SRE engineering target in financial services. It is the regulatory minimum below which examiners begin asking difficult questions.
Multi-Dimensional SLO Architecture for Financial Systems
Standard SRE SLO design defines a single SLI per service behaviour. Financial services systems require multi-dimensional SLO design: multiple SLIs, each representing a distinct dimension of service correctness, must all meet their targets simultaneously for the service to be considered within its SLO.
────────────────────────────────────────────────────────────────────────────
PAYMENT PROCESSING API — MULTI-DIMENSIONAL SLO ARCHITECTURE
DIMENSION 1: AVAILABILITY
SLI: Fraction of payment initiation requests returning a non-5xx
response within 3 seconds
Target: 99.99% over 28-day rolling window
Error budget: 0.01% of requests (~4.3 minutes complete downtime/28d)
Source: Istio Envoy proxy metrics (reporter="destination")
DIMENSION 2: LATENCY
SLI: Fraction of payment initiation requests completing
authorisation response within 1,500ms (p95)
Target: 99.9% of requests
Rationale: Card network authorization timeout = 2,000ms.
1,500ms p95 provides 500ms headroom for network variance.
Source: Istio request duration histogram
DIMENSION 3: SETTLEMENT CORRECTNESS
SLI: Fraction of initiated payments where settlement record
matches authorisation record (amount, account, currency,
reference identical in both systems within 5-minute reconciliation)
Target: 100.000% — zero tolerance for settlement mismatch
Source: Automated reconciliation job comparing authorisation
database against settlement ledger
Note: Any settlement mismatch triggers immediate P0 incident
regardless of error budget state
DIMENSION 4: IDEMPOTENCY
SLI: Fraction of duplicate payment requests (same idempotency key,
retry within 24 hours) that are correctly deduplicated
rather than processed twice
Target: 99.999% — double-processing is a regulatory and customer harm event
Source: Idempotency key collision detection log
DIMENSION 5: AUDIT COMPLETENESS
SLI: Fraction of payment transactions with complete, tamper-evident
audit trail (initiation → authorisation → settlement → confirmation)
Target: 100.000% — regulatory requirement (12 CFR Part 210 for ACH)
Source: Splunk audit pipeline completeness check
────────────────────────────────────────────────────────────────────────────
SERVICE HEALTH DEFINITION:
The payments service is within SLO if and only if ALL FIVE dimensions
are within their targets simultaneously.
A service that is 99.999% available but has a 0.001% settlement
mismatch rate is NOT within SLO.
A service that is processing all requests correctly but with
p95 latency of 1,800ms is NOT within SLO.
────────────────────────────────────────────────────────────────────────────
# Prometheus: Multi-Dimensional SLO Recording Rules
# All five dimensions tracked; composite health derived from AND of all five
groups:
- name: payments.slo.multidimensional
interval: 30s
rules:
# Dimension 1: Availability
- record: sli:payments_availability:ratio_rate5m
expr: |
sum(rate(istio_requests_total{
destination_service_name="payments-api",
response_code!~"5..",
reporter="destination"
}[5m]))
/
sum(rate(istio_requests_total{
destination_service_name="payments-api",
reporter="destination"
}[5m]))
# Dimension 2: Latency (fraction within 1500ms)
- record: sli:payments_latency_1500ms:ratio_rate5m
expr: |
sum(rate(istio_request_duration_milliseconds_bucket{
destination_service_name="payments-api",
reporter="destination",
le="1500"
}[5m]))
/
sum(rate(istio_request_duration_milliseconds_count{
destination_service_name="payments-api",
reporter="destination"
}[5m]))
# Dimension 3: Settlement correctness (from reconciliation job)
- record: sli:payments_settlement_correctness:ratio_rate1h
expr: |
sum(rate(payment_settlement_matched_total[1h]))
/
sum(rate(payment_settlement_checked_total[1h]))
# Composite SLO health: all dimensions must be healthy
# (product of all ratios — any breach pulls composite below 1.0)
- record: slo:payments_composite_health:ratio
expr: |
sli:payments_availability:ratio_rate5m
*
sli:payments_latency_1500ms:ratio_rate5m
*
sli:payments_settlement_correctness:ratio_rate1h
# Immediate alert: settlement correctness ANY breach
- alert: PaymentSettlementMismatch_P0
expr: sli:payments_settlement_correctness:ratio_rate1h < 1.0
for: 0s # Immediate — no stabilisation window for correctness breach
labels:
severity: critical
escalation: immediate
regulatory_notification: required
annotations:
summary: >
CRITICAL: Payment settlement mismatch detected.
{{ with query "1 - sli:payments_settlement_correctness:ratio_rate1h" }}
{{ . | first | value | humanizePercentage }}{{ end }} of settlements
do not match authorisation records.
Regulatory notification may be required within 4 hours (EU DORA).
runbook: "https://wiki.internal/sre/runbooks/settlement-mismatch"
Error Budget Policy for Financial Services
The error budget policy for financial services systems requires two modifications to the standard four-tier structure: a higher override authority (the Chief Risk Officer must be in the override chain, not just the VP of Engineering), and an explicit regulatory notification tier that fires when budget exhaustion creates a reportable operational event.
────────────────────────────────────────────────────────────────────────────
PAYMENT PROCESSING — ERROR BUDGET POLICY v2.4
SERVICE: payments-api (Dimension 1: Availability)
SLO TARGET: 99.99% over 28-day rolling window
ERROR BUDGET: 0.01% of requests (~4.3 minutes complete downtime/28d)
APPROVED BY: SRE Lead + CTO + CRO
REVIEWED BY: Operational Risk Committee (quarterly)
TIER 1 — Budget Healthy (> 80% remaining)
✓ Normal release cadence: up to 2 deployments/day
✓ A/B testing in production: ≤ 5% traffic fraction
✓ Infrastructure changes with standard risk assessment
Note: Financial services uses 80% floor (vs. standard 75%)
due to regulatory examination sensitivity
TIER 2 — Budget Degraded (40–80% remaining)
⚠ Maximum 1 deployment per week
⚠ No A/B testing; production changes to hardened code only
⚠ Infrastructure changes require SRE Lead + Head of Technology Risk
⚠ Operational Risk daily briefing until budget recovers
Required: root cause documentation within 24 hours of tier entry
TIER 3 — Budget Exhausted (< 40% remaining)
✗ No deployments except regulatory-mandated changes or P0 remediation
✗ No infrastructure changes except emergency rollbacks
✗ No third-party integrations or dependency upgrades
Required within 24 hours:
→ Joint review: SRE Lead + CTO + CRO
→ Operational Risk Committee notification
→ Assessment: does budget exhaustion constitute a reportable
operational event under OCC SR 21-3 or local regulation?
Override authority: CTO + CRO joint written approval
All overrides: logged to regulatory audit trail
REGULATORY NOTIFICATION TIER — Budget + Settlement Events
Trigger: Any settlement correctness SLO breach (Dimension 3)
OR budget exhaustion creating customer-impacting outage
> 2 hours continuous
Required: Legal + Compliance notification within 1 hour
OCC/FRB notification assessment within 4 hours
EU DORA: notify competent authority within 4 hours
of major ICT-related incident classification
────────────────────────────────────────────────────────────────────────────
DORA Metrics in Financial Services: The Regulated Enterprise Adjustments
The DORA Four Key Metrics require the regulated enterprise adjustments described in the Beyond DORA post, but with additional financial-sector-specific calibrations.
────────────────────────────────────────────────────────────────────────────
DORA METRICS — FINANCIAL SERVICES CALIBRATION
DEPLOYMENT FREQUENCY:
Raw DORA benchmark: elite = multiple times per day
Financial services reality: CAB cycle, change freeze windows,
pre/post-market hours restrictions for trading systems
RE-adjusted benchmark: elite = ≥ 90% of available windows utilised
Additional constraint: PCI-DSS Req 6.4 change control requirements
add mandatory pre-deployment security review for cardholder data
environments → increases process lead time systematically
LEAD TIME FOR CHANGES:
Decompose into:
→ Technical lead time (commit → deployable artefact): target < 2 hours
→ Security review lead time (cardholder data env): + 1–3 days
→ CAB review lead time: + 2–5 business days
→ Regulatory pre-notification (some changes): + 5–30 days
The technical lead time and the total lead time can differ by an
order of magnitude. Optimising CI/CD without addressing governance
overhead produces negligible total lead time improvement.
CHANGE FAILURE RATE:
Standard CFR: technical failures only
Financial services extended CFR:
→ Technical CFR: production incidents from changes (target < 5%)
→ Compliance CFR: changes triggering audit findings (target 0%)
→ Settlement CFR: changes creating settlement discrepancies (target 0%)
→ Regulatory CFR: changes requiring regulatory notification (target 0%)
MEAN TIME TO RESTORE:
Standard MTTR: service degradation to restoration
Financial services extended MTTR:
→ Technical MTTR: degradation to service restoration (target < 30 min)
→ Settlement MTTR: settlement discrepancy to full reconciliation
→ Regulatory MTTR: incident to closed regulatory obligation (target < 5 days)
→ Customer MTTR: incident to all affected customers made whole
────────────────────────────────────────────────────────────────────────────
Systemic Risk as a Reliability Dimension
Individual institution SLOs do not capture the systemic risk dimension of financial services reliability — the risk that a single institution's failure propagates across the financial system through counterparty exposure, interbank settlement dependencies, and market confidence effects.
The Federal Reserve's Fedwire Funds Service and the Clearing House Interbank Payments System (CHIPS) together process approximately 85% of large-value interbank dollar payments in the United States. Both systems have participants whose failure would create systemic consequences extending far beyond that participant's own customers. For these institutions, the reliability engineering obligation extends beyond their own SLO targets to their obligations as systemic participants.
────────────────────────────────────────────────────────────────────────────
SYSTEMIC RISK RELIABILITY OBLIGATIONS (Large Financial Institutions)
FEDWIRE PARTICIPANTS:
Obligation: Must be capable of processing all queued payments within
the operating day even under adverse conditions (FRB SR 03-9)
SRE mapping:
→ Capacity planning must include worst-case settlement day scenarios
→ SOT derivation must account for peak-end-of-day settlement volume
→ Business continuity testing must include scenario where primary
data centre is unavailable on settlement day close
PAYMENT SYSTEM CONCENTRATION RISK:
If a small number of participants handle the majority of payment volume,
the reliability of the payment system depends disproportionately on
those participants' reliability.
SRE mapping:
→ Higher SLO targets for systemically important participants
→ Error budget policies that restrict changes during peak
settlement periods (end-of-quarter, year-end)
→ Mandatory capacity headroom at 2× peak volume (systemic shock)
OPERATIONAL RESILIENCE TESTING (SR 21-3 / EU DORA):
Large financial institutions must test resilience under severe but
plausible scenarios affecting critical business services.
SRE mapping:
→ Chaos engineering exercises against payments, settlement,
and data integrity systems
→ Tabletop scenarios: "primary data centre unavailable on
largest settlement day of the year"
→ Annual resilience test results reviewed by Board Risk Committee
────────────────────────────────────────────────────────────────────────────
Automated Regulatory Compliance: SR 21-3 → SRE Mapping
# Splunk: Automated SR 21-3 Operational Resilience Evidence Package
# Generates quarterly compliance evidence from operational telemetry
# Eliminates manual evidence collection (Class 4 automation)
# Query: SR 21-3 Critical Business Services — Availability Evidence
index=sre_metrics sourcetype="sre:error_budget"
service IN ("payments-api", "core-banking", "settlement-engine",
"fraud-detection", "customer-authentication")
earliest=-91d latest=now
| stats
avg(budget_remaining_pct) as avg_budget_pct,
min(budget_remaining_pct) as min_budget_pct,
count(eval(policy_tier="TIER_3")) as tier3_events,
count(eval(regulatory_notification_sent=1)) as regulatory_notifications,
avg(technical_mttr_minutes) as avg_mttr_min
by service, quarter
| eval
slo_met = if(min_budget_pct > 20, "YES", "REVIEW_REQUIRED"),
rto_met = if(avg_mttr_min < 30, "YES", "REVIEW_REQUIRED")
| eval sr213_posture = case(
slo_met="YES" AND rto_met="YES" AND tier3_events=0, "STRONG",
slo_met="YES" AND rto_met="YES", "ADEQUATE",
true(), "REQUIRES_ATTENTION"
)
| table service, quarter, avg_budget_pct, min_budget_pct,
tier3_events, avg_mttr_min, sr213_posture
| outputlookup sr213_evidence_Q1_2025.csv
Common Antipatterns
The Availability-Only SLO antipattern → Defining financial services SLOs in terms of availability alone and treating settlement correctness, idempotency, and audit completeness as implementation details. The TSB migration failure was not an availability failure in its most damaging dimension — many customers could access their accounts but saw incorrect balances or other customers' data. A pure availability SLO would have shown green while the settlement correctness SLO was catastrophically breached.
The Vendor SLA Substitution antipattern → Accepting the payment processor vendor's contractual SLA as the de facto operational resilience target. Vendor SLAs measure vendor infrastructure uptime. They do not measure end-to-end transaction correctness, settlement integrity, or the compliance posture of the organisation using the vendor's system. SR 21-3 explicitly places operational resilience accountability on the financial institution, not its vendors.
The Change Freeze as Risk Management antipattern → Implementing broad, calendar-based change freezes (year-end, quarter-end) as a risk management substitute for error budget policy. Broad change freezes create pressure for large batch changes immediately before and after the freeze window — exactly the large-batch, high-risk change pattern that produces the highest CFR. The correct response to high-risk periods is a tighter error budget tier with explicit authorisation for changes that must occur.
The Settlement Reconciliation Deferral antipattern → Running settlement reconciliation as a batch process (nightly or end-of-day) rather than as a continuous SLI. Settlement discrepancies that are not detected until end-of-day reconciliation have been propagating for hours before they are visible. The correctness SLI must run continuously, not at batch intervals, to provide the detection window that operational response requires.
The Regulatory Notification Avoidance antipattern → Delaying operational incident classification to avoid triggering regulatory notification obligations. EU DORA requires notification to competent authorities within 4 hours of a major ICT-related incident classification. Organisations that delay classification to defer notification are creating a regulatory risk that is typically larger than the operational risk they are trying to manage.
Maturity Progression
────────────────────────────────────────────────────────────────────────────
STAGE FINANCIAL SERVICES RELIABILITY NORTH STAR SIGNAL
────────────────────────────────────────────────────────────────────────────
Reactive Availability-only SLOs. Settlement reconciliation
Settlement correctness runs nightly. Regulatory
measured in batch. notifications generated
Vendor SLA = org SLO. reactively.
Defined Multi-dimensional SLO Settlement correctness
architecture documented. SLI instrumented
SR 21-3 mapped to SRE. continuously. Error
DORA metrics calibrated. budget policy includes
CRO override authority.
Measured All five SLO dimensions Composite SLO health
tracked. Regulatory MTTR metric live. SR 21-3
measured. Compliance CFR evidence automated.
tracked separately. Settlement MTTR measured.
Optimised Systemic risk scenarios Chaos engineering
tested annually. TSB-class tested against payment
migration risk assessment settlement stack.
process formalised. Regulatory MTTR
Settlement correctness < 3 business days.
zero-breach maintained.
Generative Reliability framework Framework referenced in
shared with regulators. OCC examination guidance.
SR 21-3 alignment CRO presents SLO data
demonstrated in to Board Risk Committee
examination. quarterly.
────────────────────────────────────────────────────────────────────────────
Five Action Items for This Week
Add settlement correctness as a Dimension 3 SLI to your payments service SLO design. If your current SLO only measures availability and latency, you are flying blind on the dimension that produces the most severe regulatory consequences. Define what "correct settlement" means for your service, identify the data source that would reveal a mismatch, and begin instrumentation — even at a low sampling rate — this sprint.
Calculate the actual regulatory notification obligations triggered by your last three major incidents. For each incident: did the outage duration, customer impact, or data integrity issue meet the threshold for notification under your primary regulator's operational resilience guidelines (OCC SR 21-3, EU DORA, FCA, or equivalent)? If you have had incidents that met notification thresholds and were not notified, that is a compliance risk that exists independently of the operational risk.
Decompose your Lead Time for Changes into technical and regulatory components. PCI-DSS change control requirements for cardholder data environments, pre-notification requirements for operational changes affecting systemically important functions, and CAB review cycles all contribute to process lead time in ways that cannot be reduced through CI/CD optimisation. Knowing which component dominates is the prerequisite for investing in the right improvement.
Add CRO to your error budget policy override authority chain. An error budget policy for financial services systems that can be overridden by the VP of Engineering alone is not calibrated to the regulatory risk exposure that budget exhaustion represents. The Chief Risk Officer's involvement in the override decision is both organisationally appropriate and defensible to regulators as evidence of operational risk governance.
Run the SR 21-3 evidence package Splunk query for last quarter. Even if you are not using SLO-based governance yet, understanding how your current operational telemetry maps (or fails to map) to SR 21-3 evidence requirements identifies the measurement gaps that a full SRE programme would close.
"In financial services, the question is never whether you can survive a 99.99% availability target. The question is whether you can survive a 100% settlement correctness requirement — and whether your SLO framework even knows the difference. Availability is necessary but not sufficient when the systems in question are the settlement infrastructure of the national economy. Reliability engineering in financial services begins where availability engineering ends."
Top comments (0)