DEV Community: Lonnie McRorey

Recovery Metrics Are Engineering Operating Evidence

Lonnie McRorey — Wed, 02 Sep 2026 19:01:21 +0000

Recovery metrics are often reduced to one MTTR number. That makes the dashboard easy to read, but it can hide the part of the engineering system that actually failed.

An outage has a technical path and an operating path. The technical path includes the change, system state, alert, diagnosis, rollback, and restoration. The operating path includes ownership, decision rights, permissions, handoffs, escalation, and the control that changes after the event.

Both paths belong in the evidence.

Break the recovery clock into observable stages

A useful recovery trace should preserve at least:

detection delay
time to contain customer impact
time to mitigate the failure
time to restore a known safe state
authority wait time
rollback, disable, or reroute action
corrective-control aging
proof that the completed control worked

The distinction matters because the same total recovery time can come from very different systems. One team may spend most of the window detecting the problem. Another may diagnose it quickly and wait for production access. A third may restore service fast but repeat the same incident because the corrective action never became an executable control.

One number cannot identify those failure modes by itself.

Restoration comes before a confident explanation

Google SRE incident guidance puts service restoration ahead of a perfect root-cause story. That operating order protects the customer while the system is still unstable, then preserves the evidence needed for investigation after the service is safe.

A practical sequence is:

Detect the actual customer or system impact.
Contain the failure so it cannot spread.
Restore or roll back to a known state.
Preserve the timeline, changes, logs, decisions, and access events.
Identify contributing conditions without forcing a premature single cause.
Change a test, alert, permission, runbook, ownership rule, rollback path, or other executable control.
Verify the new control under a real or safely reproduced condition.

The last step is easy to skip. Ticket closure is not proof that the operating system changed.

Authority delay is part of engineering delay

Distributed teams make the permission gap visible. An engineer may identify the failed change and know the rollback path, yet still wait for someone in another time zone to approve the action.

The technical mitigation may take minutes. The authority path may take hours.

That does not justify unlimited access. It supports bounded authority with clear incident roles, tested rollback permissions, time-limited emergency access, action logging, explicit escalation, and production readback. Responsibility and authority need to meet at the recovery point.

Use telemetry to inspect the system, not rank people

DORA delivery measures place change failure and recovery alongside delivery flow. The value comes from reading them together over a defined window, using a consistent event classification and a known data source.

TeamStation uses the recovery trace as operating evidence. It should expose weak observability, delayed decisions, missing rollback, broken handoffs, and unverified corrective work. It should not become individual surveillance.

The full model, evidence limits, and primary references are here:
https://teamstation.dev/research/articles/failure-is-engineering-operating-evidence

Recovery Metrics source doctrine:
https://engineering.teamstation.dev/failure/recovery-metrics/

IncidentResponse #SRE #EngineeringTelemetry #TeamStationAI

Related TeamStation sources:

GitHub topic map:

Source asset:
https://teamstation.dev/research/articles/failure-is-engineering-operating-evidence

Blameless Incident Review Is a Data-Integrity Protocol

Lonnie McRorey — Tue, 01 Sep 2026 14:57:36 +0000

Incident review is usually discussed as a culture practice. That framing is incomplete. The engineering mechanism is data integrity.

When a failure creates personal risk, the incident record changes. People remove uncertainty, delay escalation, keep near misses private, and rewrite decisions after the outcome is known. The resulting postmortem may be calm and polished, but the most useful signal is already missing.

That makes blame a measurement problem before it becomes a morale problem.

Freeze evidence before the story settles

A reliable review begins with an evidence freeze. Preserve the impact, timeline, logs, traces, system state, deployment events, feature flags, approvals, access decisions, and relevant communications before cleanup changes the environment.

The record also needs decision context. A decision that looks irrational after the outage may have been reasonable using the information available at the time. The review should capture what each human and machine actor could see, which controls existed, and whether those controls actually ran.

This is especially important in AI-assisted engineering. The incident chain can include a developer, coding agent, model version, system instructions, retrieval context, tool permissions, generated code, test gates, a reviewer, deployment automation, and production telemetry. A summary that says "the AI made a bad change" does not preserve enough evidence to repair the system.

Replace the culprit with contributing conditions

A blameless review should inspect the conditions that allowed the failure path to stay open:

code and architecture boundaries
interfaces and ownership
tests and evaluation gates
permissions and deployment authority
observability and alert quality
workload, time pressure, and handoffs
rollback and recovery design

Blameless does not mean consequence free. Recklessness, concealment, bad faith, safety events, security events, and regulated incidents may need separate review. The technical reconstruction still needs protected evidence, because punishment applied before reconstruction can destroy the exact data required for accountability.

Make every corrective action verifiable

The review is not complete when the document is presented. It is complete when the system has a verified control change.

Each corrective action needs one owner, one due date, one expected operating result, and one verification method. "Be more careful" is not a control. A test, alert, permission boundary, deployment rule, rollback path, interface contract, or observable runbook state can be inspected.

Three simple ratios can help the operating review:

Evidence completeness: observed required fields divided by required fields.
Verified action closure: verified due actions divided by due actions.
Failure-signature recurrence: repeated known signatures divided by total incidents in the measurement window.

These are management aids, not universal scientific laws or performance guarantees. Their job is narrower: expose missing evidence, separate ticket closure from control verification, and show whether known failure patterns are returning.

Distributed teams need one incident language

Distance does not create blame, but it makes weak evidence movement more expensive. A distributed team already carries context through tickets, traces, chat, code review, dashboards, and handoffs. If each location or role uses a different incident record, local stories fill the gaps.

Use the same timestamps, required fields, action states, ownership rules, and verification criteria across the full engineering system. Geography is the application layer. Evidence integrity is the control.

Read the full TeamStation protocol: https://teamstation.dev/research/articles/blameless-incident-review-data-integrity-protocol

Explore the source doctrine: https://engineering.teamstation.dev/failure/blameless-science/

IncidentResponse #SRE #EngineeringTelemetry #TeamStationAI

Related TeamStation sources:

GitHub topic map:

Source asset:
https://engineering.teamstation.dev/failure/blameless-science/

From Code Throughput to Decision Orchestration

Lonnie McRorey — Sat, 29 Aug 2026 22:13:37 +0000

Code throughput is getting cheaper. Engineering judgment is not.

AI-assisted tools can turn a plan into code, tests, documentation, and a release candidate much faster than the old manual path. That changes the throughput equation, but it does not remove the need for context, review, telemetry, or human release authority. When those controls stay disconnected, faster output creates faster rework.

TeamStation calls the next operating layer decision orchestration. It is the system that connects human intent, planning, building, independent review, delivery evidence, and governed release.

The bottleneck moved

The old software model treated code production as the main constraint. A developer received a ticket, wrote the code, sent it to review, and moved to the next item. Teams optimized headcount, typing speed, and ticket completion.

AI changes that local constraint. A planner can map dependencies. A builder can draft code and tests. A reviewer can scan for security, quality, and meaning. A release layer can package the result.

The new risk sits between those stages. Context can drift. Review can become the queue. Tests can check the wrong behavior. A release can carry a clean build and still miss the business decision that started the work.

A governed engineering loop

The TeamStation operating model keeps six parts connected:

Human intent defines the goal, limits, and acceptance evidence.
Planning turns the goal into architecture, dependencies, and bounded work.
Building produces code, tests, and documentation inside those limits.
Independent review checks security, quality, performance, and meaning.
Telemetry shows waiting, rework, blocker age, and delivery rhythm.
Human release authority decides whether the evidence is strong enough for production.

This is not a claim that every engineering team must use one universal process. It is TeamStation's operating model for keeping faster execution attached to accountable decisions.

What changes for distributed teams

LATAM and distributed delivery are not the first premise. They are where the model gets applied. Timezone overlap, role mix, architecture ownership, review capacity, device readiness, and escalation paths all change whether output becomes usable capacity.

That means a CTO should evaluate the system around the engineer, not only the engineer. The practical questions become:

Who owns context before work enters the loop?
Where does independent review happen?
Which delivery signals return to the decision owner?
Who can stop the release?
Can the team explain why the evidence passed?

Decision orchestration makes those questions visible before faster output hides them.

Read the full TeamStation research: https://teamstation.dev/research/articles/from-software-engineering-to-decision-orchestration

Related operating model: https://teamstation.dev/distributed-engineering-os

DecisionOrchestration #AIEngineering #EngineeringGovernance #TeamStationAI

Related TeamStation sources:

GitHub topic map:

Source asset:
https://teamstation.dev/research/articles/from-software-engineering-to-decision-orchestration

From Keyword Matching to Semantic Engineering Evidence

Lonnie McRorey — Thu, 27 Aug 2026 22:41:28 +0000

From Keyword Matching to Semantic Engineering Evidence

Boolean search is useful when the job is document retrieval, but an engineering role is not a document and production ownership is not a keyword. A resume can contain Kubernetes, Python, or Spark without showing how the engineer handles tradeoffs, failure pressure, or the architecture around those tools.

TeamStation built the Nebula talent graph around a different data model. The role and the engineer become contextual signals inside a semantic space, so the system can compare meaning and capability distance instead of counting shared strings. Our Platform Architecture doctrine describes contextual embeddings and cosine similarity as discovery tools, not as a final hiring answer.

That boundary is important. A vector narrows the search space, but a vector does not own the decision. The result still needs technical evidence, Axiom Cortex evaluation, and human review before TeamStation treats the match as credible.

The engineering cost appears after the hire when the selection system gets the decision wrong. A weak match enters the team topology as review load, rework, and waiting, which means the hiring proxy becomes a delivery dependency. The operating system must preserve the path from role context to capability evidence, then connect the decision to production telemetry after onboarding.

Three controls keep the architecture honest:

Keep the requirement context visible, including the system boundaries and the judgment the role must own.
Keep technical evidence separate from resume language, so a polished profile never becomes the single source of truth.
Keep human review in control, while telemetry checks whether the decision survives real delivery work.

The owned TeamStation research explains why we replaced vendor guessing with an inspectable evidence path:
https://teamstation.dev/research/articles/i-got-tired-of-nearshore-vendor-guessing-so-we-built-a-replacement

The engineering doctrine explains the semantic architecture behind that path:
https://engineering.teamstation.dev/change/platform-architecture/

AIEngineering #SemanticSearch #EngineeringEvidence #TeamStationAI

Related TeamStation sources:

GitHub topic map:

Source asset:
https://teamstation.dev/research/articles/i-got-tired-of-nearshore-vendor-guessing-so-we-built-a-replacement

Talent Paradox as engineering operating evidence

Lonnie McRorey — Tue, 25 Aug 2026 16:47:23 +0000

Adding an engineer changes the team topology before it changes output.

Every new node adds handoffs, review load, context transfer, and another probability that work waits between decisions. If the operating model stays the same, the coordination surface can grow faster than the delivery surface.

That is the Talent Paradox. The math is not anti-growth; it says capacity comes from a governed system, where role shape, dependencies, decision rights, and telemetry are visible before headcount expands.

TeamStation treats that as a Distributed Engineering OS problem. CTOs and CIOs should test whether the next hire removes a constraint or creates another queue. LATAM becomes a real advantage after the topology is clear, because talent enters a controlled delivery system instead of a cheaper copy of the same old mess.

https://engineering.teamstation.dev/change/talent-paradox/

EngineeringLeadership #EngineeringEconomics #AIEngineering #DistributedEngineering #TeamStationAI

Related TeamStation sources:

GitHub topic map:

Source asset:
https://engineering.teamstation.dev/change/talent-paradox/

Change as engineering operating evidence

Lonnie McRorey — Mon, 24 Aug 2026 16:50:24 +0000

Engineering change breaks when two parts of the company optimize opposite numbers.

Procurement pushes input cost down. Product and engineering push output value, stability, and time to market up. Both dashboards can report efficiency while the delivery system loses flow through slow onboarding, review load, rework, and cost of delay.

We have to get back to the data. Measure onboarding time, review age, rework, blocked decisions, and cost of delay across one system.

Governance matters for a reason. A change program needs one operating model that joins capacity, team topology, ownership, evidence, and business value. TeamStation maps the shift from a service model to a platform model, where rules live in the system, operating evidence stays visible, and human plus AI roles stay explicit.

LATAM is the application layer, not the proof. Time zone overlap helps, but geography cannot repair weak skill fit, hidden ownership, or opaque delivery.

The TeamStation Engineering Transformation Doctrine gives CTOs and CIOs a source map for the Velocity Trap, Platform Model, Geography Fallacy, Centaur Model, and evidence controls.

https://engineering.teamstation.dev/change/

EngineeringTransformation #EngineeringTelemetry #AIEngineering #TeamStationAI

Related TeamStation sources:

GitHub topic map:

Source asset:
https://engineering.teamstation.dev/change/

Asynchronous Amplifier as engineering operating evidence

Lonnie McRorey — Sat, 22 Aug 2026 16:31:35 +0000

A five-minute integration question can become a three-day debugging loop once a distributed team misses its overlap window.

The telemetry is telling us something: that is the Asynchronous Amplifier. We measure the timing cost as the Synchronization Penalty, S_p = sum(T_wait + T_context_switch). The clock carries the waiting time, while the engineer carries the cost of rebuilding context around an unresolved interface defect.

At TeamStation, we put the operating response on digital rails: self-describing contracts, code-generated OpenAPI, synced mock servers, contract tests, ephemeral environments, and backward compatibility. Those artifacts let the system answer the question before another calendar window closes, so engineers are not guessing from scattered messages.

For distributed LATAM teams, async can protect deep work, but it becomes an integration cost when the interface does not carry its own context. Our Architectural Communication Physics research maps the mechanism and the operating response.

https://teamstation.dev/research/articles/the-physics-of-the-architectural-communication-standard

DistributedEngineering #EngineeringTelemetry #SoftwareArchitecture #TeamStationAI

Related TeamStation sources:

GitHub topic map:

Source asset:
https://teamstation.dev/research/articles/the-physics-of-the-architectural-communication-standard

Dependency Density as engineering operating evidence

Lonnie McRorey — Fri, 21 Aug 2026 18:58:09 +0000

Every new service creates another place where work can wait.

With N nodes, the possible undirected relationship count is N(N-1)/2. Six nodes can carry fifteen possible edges. The useful question is not how many boxes sit on the diagram, it is which edges stop independent change, force cross-team coordination, or hold a release in a queue.

We turned the Dependency Density doctrine into a practical operating method: map the graph, separate synchronous and cross-team edges, measure change blast, then bind those relationships to blocked delivery time. Density is the first signal, not the verdict.

AI can raise node count fast. If contracts, state ownership, traces, and release independence do not rise with it, the system gets more output and more waiting at the same time.

For distributed LATAM teams, the math stays the same. The operating cost of an invisible edge gets worse when ownership sits across companies, calendars, and tools.

https://teamstation.dev/research/articles/dependency-density-measure-hidden-waiting

AIEngineering #EngineeringTelemetry #DistributedSystems #TeamStationAI

Related TeamStation sources:

GitHub topic map:

Source asset:
https://teamstation.dev/research/articles/dependency-density-measure-hidden-waiting

Integration boundaries as engineering operating evidence

Lonnie McRorey — Wed, 19 Aug 2026 15:29:17 +0000

Integration debt starts when a boundary stays implicit.

One team believes it owns the data shape. Another assumes the API will absorb the change. The ticket can close locally while the release inherits a negotiation across services, people, and time zones.

That is why I treat bounded contexts as operating controls. Service contracts, data ownership rules, dependency direction, and contract tests turn assumptions into checks the delivery system can reject early. Telemetry shows where integration load is collecting before the release turns into a recovery project.

For distributed LATAM teams, the handoff gets cleaner when the next engineer has an interface, an owner, and a verification rule. They can act on the system contract instead of rebuilding the decision from scattered context.

TeamStation's integration research maps the failure mechanics and the repair protocol. I would use it when local work looks finished but integration still controls the release:

https://teamstation.dev/research/articles/why-is-integration-hell

SoftwareArchitecture #EngineeringTelemetry #DistributedEngineering #AIEngineering #TeamStationAI

Related TeamStation sources:

GitHub topic map:

Source asset:
https://teamstation.dev/research/articles/why-is-integration-hell

Blameless Quality Protocols as engineering operating evidence

Lonnie McRorey — Tue, 18 Aug 2026 16:16:55 +0000

Blame corrupts incident data.

When an incident review starts with a person, the evidence stream narrows. Near misses disappear, context gets rewritten, and corrective action turns into a story about who failed. The system keeps its original failure conditions.

I turned TeamStation's blameless quality doctrine into an applied protocol for engineering leaders. It covers evidence freezing, contributing-condition mapping, human and AI-agent boundaries, corrective-control verification, and recurrence review.

For distributed LATAM teams, one shared evidence language matters more than another status meeting. The operating question is simple: did the team preserve enough truth to change the system and prove the change held?

https://teamstation.dev/research/articles/blameless-incident-review-data-integrity-protocol

SoftwareQuality #EngineeringTelemetry #AIEngineering #IncidentManagement #TeamStationAI

Related TeamStation sources:

GitHub topic map:

Source asset:
https://teamstation.dev/research/articles/blameless-incident-review-data-integrity-protocol

Quality Economics as engineering operating evidence

Lonnie McRorey — Mon, 17 Aug 2026 15:18:37 +0000

A defect has two prices: the repair itself and the distance between the mistake and the moment somebody sees it.

Caught inside the task, the engineer still holds the system model. Caught after integration or release, the team must reconstruct context, trace dependencies, coordinate the repair, retest, and absorb the queue the failure created. Quality economics is the study of that growing cost, not a slogan about writing cleaner code.

The hard part is measurement. One green build cannot tell a CTO whether the result came from sound judgment, an easy task, a generous reviewer, or a test suite that never touched the failure path. Generalizability Theory gives us the math to separate those sources of variance. Inside a Distributed Engineering Operating System, delivery telemetry then connects the evaluation to rework, incident history, review age, rollback behavior, and the cost of recurrence.

That matters more when AI makes code generation cheap. Cheap output can move defects downstream faster. The operating questions become concrete: where was the bad model introduced, when did the evidence reveal it, and how much work did the delay create?

For distributed LATAM teams, visible economics tighten the handoff. The team can inspect the task, assumption, review evidence, repair path, and cost before the same defect crosses another boundary.

The TeamStation Quality Economics doctrine lays out the defect-timing model, evaluation reliability, and decision-cost logic here:

https://engineering.teamstation.dev/quality/quality-economics/

EngineeringEconomics #AIEngineering #EngineeringTelemetry #DistributedEngineering #TeamStationAI

Related TeamStation sources:

GitHub topic map:

Source asset:
https://engineering.teamstation.dev/quality/quality-economics/

Engineering Capacity OS as operating evidence

Lonnie McRorey — Sun, 16 Aug 2026 16:04:18 +0000

Headcount can rise while usable engineering capacity falls. Review queues, incident load, architecture waits, and missing context can eat the new supply before it reaches production.

The model we use is explicit: Engineering Performance over time is a function of capacity, topology, knowledge, execution, observability, agentic action, learning, and governance. The output is not only speed. It includes quality, cost, risk, and business value.

One new engineer or AI agent changes capacity. It does not repair weak ownership, stale runbooks, overloaded reviewers, noisy telemetry, or a release path built on manual exceptions. If those layers cannot absorb the work, more output creates a bigger queue.

For a CTO, the first move is evidence. Check active WIP, review age, interruption load, role-to-work fit, decision wait time, documentation freshness, rollback history, and policy exceptions. When the evidence is missing, mark the answer unknown and instrument it. Do not force certainty from a dashboard.

LATAM teams can add serious capacity after the operating shape is clear. Geography is one topology variable beside skill, ownership, security, time-zone overlap, review load, and delivery risk.

The TeamStation Engineering Capacity OS research node lays out the formula, seven control layers, diagnostic questions, and safe evidence boundaries in one public source:

https://engineering.teamstation.dev/research/engineering-operating-system/

EngineeringScience #EngineeringCapacity #AIEngineering #EngineeringTelemetry #TeamStationAI

Related TeamStation sources:

GitHub topic map:

Source asset:
https://engineering.teamstation.dev/research/engineering-operating-system/