Ask a small engineering team how delivery is going and you get adjectives: pretty good, a bit slow, kind of fragile. Ask the same team four specific questions and you get a health record: how often do you deploy, how long from merged to live, what share of deploys cause an incident, and how fast do you undo a bad one?
Those are the four DORA metrics — and they matter precisely because they fight each other. You cannot max all four by gaming one. The tension is the honesty.
You can read the full note on the HIVE80lab ops-notes site — here is the working version.
1. The four numbers, defined for a small team
- Deployment frequency — how many days per week something ships to production. Count days-with-a-deploy, not deploys-per-day: three deploys before lunch then silence is one day. Batch deploys (the Friday 4pm "week's work in one release") show up as exactly what they are: frequency 1.
- Lead time for changes — hours from merge to production, median. Not from ticket creation: that number is dominated by queue and negotiation, not by your pipeline. If a merged PR waits two days for "the next release window," lead time is 48 hours, and no re-labeling changes it.
- Change failure rate — the share of production deploys that caused an incident, a rollback, a hotfix, or a customer-visible defect. Count every deploy; mark the failed ones; divide. A rollback is not a separate successful deploy — it is the receipt for a failed one.
- Failed-deployment recovery time — time from customer impact to service restored, median. From impact, not ticket-open: a monitoring gap is part of the cost, and measuring from open flatters the queue instead of the customer.
2. The five traps that make the numbers lie
Counting effort instead of outcomes. Story points, commits, lines of code — none survive contact with a customer. If a metric has no deployment date in it, it is a work diary, not a delivery metric.
Vanity micro-deploys. A typo fix at 9am to start the frequency graph is fine once; a pattern of trivial deploys means frequency stopped measuring flow. The pair that keeps each other honest is frequency × failure rate — if frequency doubles while failure rate triples, you are shipping risk, not value.
Lead time measured from ticket creation. Then the number includes product debate and the two weeks the ticket sat in the backlog — real costs, but the wrong lever. Keep cycle time (ticket-to-merge) and lead time (merge-to-prod) as two separate numbers.
Recovery from ticket-open, not impact. The customer does not know a ticket exists. If detection relies on a user report, your recovery time includes their wait — exactly the number that should hurt.
Quarterly argue-the-data. A team that spends the retro disputing whether a deploy "counts" is spending the effort the metric was meant to save. Decide the counting rules once, on one page; imperfect-but-consistent beats precise-but-disputed.
3. Benchmarks: aim at the band, not the elite line
From the DORA (State of DevOps) research:
- Deployment frequency: low monthly+ → medium weekly-to-monthly → high at-least-daily. If you ship weekly you are already ahead of most teams your size.
- Lead time: low 1–6 months → medium week-to-a-month → high under a week → elite under a day. The gap from "a month" to "two days" is usually one bottleneck: a release ritual that could be a button.
- Change failure rate: 0–15% is low; above ~40% the fix is almost never "test more" in the abstract — it is smaller changes plus a rehearsed rollback.
- Recovery: under an hour is elite. Small teams usually lose here to detection, not repair — the fix is a monitoring row, not a war room.
Rule of use: pick one number per quarter to move; leave the other three as context. Teams that push all four at once improve none.
4. Instrumentation in one afternoon
- Deploy log (10 min): one-line append on every production deploy — timestamp, short SHA, service. A shell alias around your deploy command does it.
-
Lead time (20 min): deploy log +
git log --merges→ merge-to-deploy per release; a spreadsheet column does the arithmetic. Weekly median, not mean. - Failure flag (10 min): one checkbox in your post-incident template — "caused by deploy #N" — linked to the deploy log. Failure rate = failed/total over the window.
- Recovery clock (10 min): incident start (impact) and end (restored) already exist in your incident doc; write them in the same row.
- The weekly four-number table (5 min/week): four cells, one trend arrow each, reviewed in the meeting you already run. A number nobody reviews is a number you will stop trusting.
- One lever per quarter: frequency → shrink the release ritual; lead time → shrink PR size; failure rate → smaller changes + rollback rehearsal; recovery → detection first, rollback second.
5. Worked example
A six-person B2B scheduling app measures honestly in March: deploys 1.2 days/week (the Friday batch), lead time 9 days median, change failure rate 30% (the Friday batch has grown to forty-PR releases, so any bad PR ships with nine friends), recovery 6 hours (bad deploys found by customers, rolled back Monday).
They pick one lever — release size. PRs capped at one day's work, deploy whenever the tree is green instead of on Fridays.
Ninety days later: deploys 4.1 days/week, lead time 26 hours, failure rate 12%, recovery 40 minutes. No platform was bought; the total instrumentation is a shell alias and a spreadsheet.
The counter-example is the sister team that bought a delivery-analytics platform first, then spent a quarter disputing what a "deploy" was in the tool's config, and ended March with a beautiful dashboard of numbers nobody trusted.
From the HIVE80lab kit
- The First 30 Minutes — free incident quick-start
- Ops Starter Kit — incident response for small teams — $14
- Ops Starter Kit Vol. 2 — advanced incident response & communications — $27
- Ops Mega Bundle — all 5 kits in one download — $49
Related: the deployment rollback checklist is what recovery time is made of, the change-freeze window policy tells you when to stop shipping entirely, and the incident post-mortem template is where the failure flag lives.
Top comments (0)