Recovery Metrics Are Engineering Operating Evidence

#incidentresponse #sre #engineeringtelemetry #teamstationai

Recovery metrics are often reduced to one MTTR number. That makes the dashboard easy to read, but it can hide the part of the engineering system that actually failed.

An outage has a technical path and an operating path. The technical path includes the change, system state, alert, diagnosis, rollback, and restoration. The operating path includes ownership, decision rights, permissions, handoffs, escalation, and the control that changes after the event.

Both paths belong in the evidence.

Break the recovery clock into observable stages

A useful recovery trace should preserve at least:

detection delay
time to contain customer impact
time to mitigate the failure
time to restore a known safe state
authority wait time
rollback, disable, or reroute action
corrective-control aging
proof that the completed control worked

The distinction matters because the same total recovery time can come from very different systems. One team may spend most of the window detecting the problem. Another may diagnose it quickly and wait for production access. A third may restore service fast but repeat the same incident because the corrective action never became an executable control.

One number cannot identify those failure modes by itself.

Restoration comes before a confident explanation

Google SRE incident guidance puts service restoration ahead of a perfect root-cause story. That operating order protects the customer while the system is still unstable, then preserves the evidence needed for investigation after the service is safe.

A practical sequence is:

Detect the actual customer or system impact.
Contain the failure so it cannot spread.
Restore or roll back to a known state.
Preserve the timeline, changes, logs, decisions, and access events.
Identify contributing conditions without forcing a premature single cause.
Change a test, alert, permission, runbook, ownership rule, rollback path, or other executable control.
Verify the new control under a real or safely reproduced condition.

The last step is easy to skip. Ticket closure is not proof that the operating system changed.

Authority delay is part of engineering delay

Distributed teams make the permission gap visible. An engineer may identify the failed change and know the rollback path, yet still wait for someone in another time zone to approve the action.

The technical mitigation may take minutes. The authority path may take hours.

That does not justify unlimited access. It supports bounded authority with clear incident roles, tested rollback permissions, time-limited emergency access, action logging, explicit escalation, and production readback. Responsibility and authority need to meet at the recovery point.

Use telemetry to inspect the system, not rank people

DORA delivery measures place change failure and recovery alongside delivery flow. The value comes from reading them together over a defined window, using a consistent event classification and a known data source.

TeamStation uses the recovery trace as operating evidence. It should expose weak observability, delayed decisions, missing rollback, broken handoffs, and unverified corrective work. It should not become individual surveillance.

The full model, evidence limits, and primary references are here:
https://teamstation.dev/research/articles/failure-is-engineering-operating-evidence

Recovery Metrics source doctrine:
https://engineering.teamstation.dev/failure/recovery-metrics/