Replication is science's self-correcting mechanism. A finding that cannot be replicated in different contexts is a finding about those specific contexts, not a general principle. The DORA Four Key Metrics have been replicated extensively in technology-company and cloud-native contexts — the research programme has surveyed tens of thousands of practitioners across multiple years and consistently found the same performance clusters and the same correlations with organisational outcomes.
What has not been systematically attempted is replication in on-premises enterprise environments with regulated change management processes — the environments where the questions about generalisability are most consequential, because these are the environments where national critical infrastructure is operated.
This post reports the results of applying DORA measurement methodology to a single on-premises, regulated enterprise environment running VMware vSphere with Tanzu Kubernetes Grid Service, GitOps via Argo CD, and a traditional Change Advisory Board process. It is a single case study, not a population-level study, and its findings should be interpreted accordingly. But a single carefully documented case study is more evidence than the field currently has for this environment type, and the deviations it identifies from the canonical DORA findings are directionally informative for practitioners and researchers working in similar environments.
Environment Description
────────────────────────────────────────────────────────────────────────────
STUDY ENVIRONMENT CHARACTERISTICS
Infrastructure:
Compute: VMware vSphere 8.0 + Tanzu Kubernetes Grid Service (TKGs)
GitOps: Argo CD with ApplicationSet, self-heal enabled
Service Mesh: Istio 1.20 in STRICT mTLS mode
Policy: Kyverno admission controller
Observability: Prometheus + Splunk Enterprise
CI/CD: Azure DevOps (ADO) pipelines
Change Management: Traditional CAB with weekly review cycles
Organisational characteristics:
Industry: Regulated enterprise (financial services adjacent)
Org size: Large enterprise (>10,000 employees)
IT headcount: ~200 engineers across development and operations
SRE programme stage: Phase 1 (Measurement and Accountability)
CAB change freeze windows: Yes (quarterly + year-end)
On-call structure: Shared SRE + development team rotations
Measurement period: 12 months (January–December 2024)
Services measured: 23 production services across 4 business domains
Data sources:
→ ADO pipeline logs (deployment timestamps, build outcomes)
→ PagerDuty incident data (incident creation/resolution timestamps)
→ Argo CD audit log (sync events, drift events)
→ Splunk incident classification data
────────────────────────────────────────────────────────────────────────────
Metric 1: Deployment Frequency
Raw measurement: The 23 services averaged 2.3 deployments per service per week across the full 12-month period, which places the environment in the DORA Medium performer cohort (between "once per week" and "once per month" in the DORA framework definitions).
The RE-adjusted measurement: When deployment frequency is normalised to available deployment windows (accounting for CAB review cycles and change freeze windows), the picture changes materially.
────────────────────────────────────────────────────────────────────────────
DEPLOYMENT FREQUENCY: RAW VS RE-ADJUSTED
Available deployment windows (12 months):
Total weeks: 52
Change freeze weeks: 8 (Q4 year-end: 4 weeks; quarterly: 1 week × 4)
Available deployment windows: 44 weeks
Raw deployment frequency:
Total deployments across 23 services: 2,756
Per service per week (all 52 weeks): 2.3 deployments/week
DORA cohort: Medium (Low–Medium boundary)
RE-adjusted deployment frequency:
Per service per available week (44 weeks): 2.72 deployments/week
Estimated window utilisation: 97% of available windows had at least
one deployment across the service portfolio
DORA cohort (RE-adjusted): High (approaching Elite boundary)
INTERPRETATION:
The raw measurement classifies this environment as Medium performer.
The RE-adjusted measurement classifies it as High performer.
The difference is the 8 change freeze weeks — externally mandated
periods where deployment was structurally prohibited regardless of
engineering capability.
A regulatory examination using raw DORA deployment frequency would
classify this environment as Medium. The RE-adjusted metric more
accurately reflects the organisation's engineering capability —
the capacity to deploy is High; the regulatory permission to deploy
is constrained.
FINDING 1: Raw deployment frequency systematically underestimates
engineering capability in environments with mandatory change freeze
windows. The RE-adjusted metric (window utilisation) is a more
accurate capability measure for regulated enterprise environments.
────────────────────────────────────────────────────────────────────────────
Metric 2: Lead Time for Changes
Raw measurement: The average lead time from code commit to production deployment was 4.1 calendar days, placing the environment in the DORA Low performer cohort ("between 1 month and 6 months" in the raw DORA framework — but wait, 4.1 days is not in that range).
Actually, 4.1 calendar days places this environment in the High performer cohort (DORA High = "between 1 day and 1 week"). But this conceals the decomposition.
────────────────────────────────────────────────────────────────────────────
LEAD TIME DECOMPOSITION
Total lead time: 4.1 days average
Component breakdown:
→ Technical lead time (commit → deployable artefact): 3.8 hours
(ADO pipeline: build, test, security scan, image push)
DORA cohort for technical component: ELITE (< 1 hour? No, 3.8h)
DORA cohort for technical component: HIGH (1 day – 1 week? No)
Actually 3.8 hours = between 1 hour and 1 day → HIGH
→ Process lead time (artefact → CAB approval → deployment): 3.3 days
(CAB meeting cadence: weekly; average wait for next CAB: 2.1 days;
CAB review and approval: 0.9 days average; deployment execution: 0.3 days)
This component alone: LOW performer (approaching Medium)
→ Regulatory notification lead time (for changes to regulated systems): +2.1 days
(subset of services: 6 of 23 require additional regulatory review)
For these services, total lead time: 6.2 days average
Technical vs process split: 8% technical, 92% process
For regulated-system services: 5% technical, 95% process
FINDING 2: Lead time in this environment is dominated by process overhead,
not technical overhead. CI/CD optimisation (which would reduce the 8%
technical component) has negligible impact on total lead time.
CAB process optimisation (risk-based change categorisation, automated
pre-approval for low-risk changes, emergency change pathway) would
address the 92% process component.
FINDING 3: The decomposition reveals an investment misallocation.
This environment has invested significantly in CI/CD pipeline
optimisation; the data shows this investment reduced the 8% technical
component from approximately 6 hours to 3.8 hours. It has not invested
proportionally in CAB process optimisation, which would yield 10×
larger lead time improvement.
────────────────────────────────────────────────────────────────────────────
Metric 3: Change Failure Rate
Raw measurement: 8.4% of deployments required remediation within 24 hours of deployment, placing this environment in the DORA Elite/High cohort (0–15%).
The extended CFR measurement adds the regulated enterprise dimensions identified in the Beyond DORA framework.
────────────────────────────────────────────────────────────────────────────
CHANGE FAILURE RATE: EXTENDED MEASUREMENT
Technical CFR: 8.4%
(deployments requiring rollback or hotfix within 24 hours)
DORA cohort: HIGH (0–15% range)
Compliance CFR: 1.2%
(deployments that, while technically successful, triggered a
compliance finding, audit gap, or policy deviation detected
within 30 days of deployment)
Target: 0%
Configuration drift rate: 3.7%
(deployments that created configuration drift detected by
Argo CD self-heal within 72 hours, distinct from technical failures)
Note: Drift is automatically remediated by Argo CD; this metric
captures the deployment quality issue that caused drift, not the
remediation overhead.
Combined extended CFR: 13.3%
(technical + compliance; drift excluded as it's auto-remediated)
FINDING 4: Technical CFR at 8.4% is in the DORA High cohort, indicating
mature change management for technical quality. Compliance CFR at 1.2%
indicates that technical quality and compliance quality are not
perfectly correlated — services can pass technical quality gates
while creating compliance gaps that technical testing does not detect.
FINDING 5: Configuration drift rate (3.7%) is a metric the canonical
DORA framework does not include. In a GitOps-enabled environment,
drift is automatically detected and remediated; the residual metric
is a deployment quality signal (deployments that create drift) rather
than an operational burden signal (drift requiring manual remediation).
This is a regime-specific metric that may not generalise to non-GitOps
environments.
────────────────────────────────────────────────────────────────────────────
Metric 4: Mean Time to Restore
Raw measurement: Median MTTR across all P1/P2 incidents was 28 minutes, placing this environment firmly in the DORA Elite cohort (< 1 hour).
The regulated enterprise extension adds the compliance MTTR dimension.
────────────────────────────────────────────────────────────────────────────
MEAN TIME TO RESTORE: EXTENDED MEASUREMENT
Technical MTTR: 28 minutes (median P1/P2 incidents)
DORA cohort: ELITE (< 1 hour)
Distribution:
25th percentile: 11 minutes
75th percentile: 67 minutes
95th percentile: 3.1 hours
99th percentile: 6.8 hours
The 95th and 99th percentile MTTRs are driven by incidents involving
stateful workloads (databases, persistent queues) and cross-cluster
incidents — consistent with the expected complexity increase for
those incident types.
Regulatory MTTR: 4.8 business days (mean, P1/P2 incidents)
Definition: Time from incident creation to all regulatory obligations
closed (root cause documented, regulator notified if required,
change record updated, postmortem closed with all action items assigned)
Components:
→ Technical resolution: 28 minutes
→ Postmortem facilitation and publication: 1.8 business days
→ Regulatory notification assessment and filing: 0.6 business days
→ Action item assignment and acknowledgement: 1.2 business days
→ Compliance review and sign-off: 1.2 business days
FINDING 6: Technical MTTR (28 minutes) and Regulatory MTTR (4.8 days)
differ by approximately 330×. The DORA framework measures the former;
the organisational cost of an incident is primarily the latter.
A CTO who evaluates incident management effectiveness using DORA
MTTR is seeing 0.3% of the actual incident resolution timeline.
FINDING 7: The 95th percentile technical MTTR (3.1 hours) is still
DORA Elite, but the 99th percentile (6.8 hours) is not. The high
performers on MTTR in this environment are the well-characterised
incident types with documented runbooks and automation. The long-tail
incidents are the ones that reveal the limits of current automation
coverage.
────────────────────────────────────────────────────────────────────────────
Finding 8: The On-Premises Adaptation — What Holds and What Doesn't
────────────────────────────────────────────────────────────────────────────
DORA FRAMEWORK GENERALISABILITY ASSESSMENT
WHAT HOLDS IN ON-PREMISES REGULATED ENTERPRISE ENVIRONMENTS:
✓ The four metrics are measurable from existing data sources
(CI/CD logs, incident management data, deployment records)
No additional instrumentation required beyond what this
environment already had.
✓ The Elite/High/Medium/Low cohort structure is meaningful
The cohort thresholds produce differentiated classifications
that are consistent with subjective engineering maturity
assessments — services that practitioners consider well-managed
score in High/Elite; services considered poorly managed
score in Low/Medium.
✓ The direction of improvement is consistent with DORA findings
Higher deployment frequency in this environment is associated
with lower CFR (r = -0.41), consistent with DORA's finding
that high-frequency, small-batch deployments have lower failure
rates than low-frequency, large-batch deployments.
✓ The metric correlations hold
In this environment: High deployment frequency services have
lower MTTR, lower CFR, and lower Regulatory MTTR. The DORA
finding that these metrics cluster together holds.
WHAT REQUIRES ADAPTATION IN ON-PREMISES REGULATED ENTERPRISE:
✗ Deployment frequency benchmarks require RE-adjustment
Raw deployment frequency systematically underestimates
engineering capability in environments with mandatory change
freeze windows. The window utilisation metric should supplement
or replace raw deployment frequency in this context.
✗ Lead time benchmarks require decomposition
A total lead time in the High cohort conceals a technical
component in the Elite cohort and a process component in the
Medium/Low cohort. The aggregate benchmark obscures the
information needed for investment decisions.
✗ CFR requires extension for compliance quality
Compliance CFR captures a failure mode (regulatory compliance
failure from a technically successful deployment) that the
canonical framework does not address. In regulated environments,
this is a material gap.
✗ MTTR requires extension for regulatory MTTR
The 330× difference between technical MTTR and regulatory MTTR
is a structural property of regulated environments, not a
measurement artefact. A framework that only measures technical
MTTR provides a misleading picture of incident resolution burden.
✗ The canonical Elite benchmarks are not all achievable as stated
"On-demand, multiple times per day" deployment frequency is
structurally incompatible with weekly CAB cycles.
The appropriate benchmark is not the canonical Elite threshold
but the RE-adjusted equivalent for the governance environment.
────────────────────────────────────────────────────────────────────────────
Implications for the DORA Research Programme
This single-environment case study cannot establish population-level findings. It can, however, identify questions that a properly designed replication study should address:
Does the direction of performance-outcome correlation hold in regulated environments? The DORA finding that Elite performers have better resilience outcomes needs replication in environments where Elite performance (as canonically defined) is not achievable.
Do the RE-adjusted metrics predict outcomes better than raw metrics in regulated environments? The hypothesis is that window utilisation (RE-adjusted deployment frequency) and decomposed lead time are better predictors of regulated enterprise resilience outcomes than raw DORA metrics. This is testable.
What is the effect size of compliance toil on DORA performance? This study suggests compliance toil is a major driver of regulated enterprise underperformance. Quantifying the effect size would inform the policy intervention design.
Does the DORA bimodal distribution hold within regulated industries? The field-level bimodal distribution (Elite/High clustering, Low clustering) may or may not be replicated within regulated industries where structural barriers are more uniform. If the intra-sector distribution is unimodal and low, the barrier is structural, not capability-driven.
Five Action Items for This Week
Measure your own Lead Time decomposition. Calculate what fraction of your total lead time is technical (commit to deployable artefact) versus process (artefact to production). If the process fraction exceeds 80%, CI/CD investment is the wrong optimisation target. CAB process reform or risk-based change categorisation will deliver 5–10× more lead time improvement per investment dollar.
Calculate your Compliance CFR for the last quarter. Review every deployment from the last 90 days. Did any technically successful deployment generate a compliance finding, policy deviation, or audit gap within 30 days? The Compliance CFR number you derive is a regulated-enterprise-specific quality signal that the canonical DORA framework does not capture.
Measure your Regulatory MTTR for your last five P1/P2 incidents. Record the time from incident creation to all compliance obligations closed (postmortem published, regulatory notification assessed, action items assigned). Compare to your Technical MTTR. The ratio between the two is the measure of your regulatory incident burden — and the primary target for compliance toil elimination.
Replicate this analysis in your environment. The methodology is fully documented. Apply it to 12 months of your pipeline and incident data. The differences between your findings and this study's findings are data about how your environment differs — and potentially a contribution to the replication literature if you publish them.
Submit the raw findings (anonymised) to the DORA research programme. The DORA research team has expressed interest in on-premises and regulated enterprise data. A structured submission of your decomposed DORA metrics, with environment type and sector anonymised, contributes to the population-level dataset that would make a formal replication study possible.
"Replication is not agreement. A replication study that finds deviation from the canonical findings is as valuable as one that confirms them — more valuable, if the deviation is systematic and informative. The deviations documented here — the deployment frequency undercount from change freeze windows, the lead time dominated by process overhead, the 330× gap between technical and regulatory MTTR — are not failures of the DORA framework. They are the framework's signal that on-premises regulated enterprise environments have specific structural properties that require specific adaptations. The framework generalises. The benchmarks require calibration."
This post is part of applying Google SRE principles to national-scale critical infrastructure. This study is a single-environment case study; findings should be interpreted as directional, not conclusive. References: DORA Research · Accelerate (Forsgren et al.) · Google SRE Workbook
Top comments (0)