DEV Community

NTCTech
NTCTech

Posted on • Originally published at rack2cloud.com

The System Recovered. Your Recovery Boundary Didn't.

Every disaster recovery program is built on the same unexamined assumption: that restoring the system restores the recovery boundary the organization actually needs back in service. That assumption held for twenty years. It doesn't hold anymore, and most recovery programs haven't noticed.

recovery boundary vs. dependency boundary — diverging architecture diagram

The Assumption

Ask any infrastructure team what "recovery" means and you'll get some version of the same answer: restore the data, boot the system, confirm the application responds. That's the model every DR runbook, every RTO/RPO target, and every recovery test is built around. It's also the model that made sense when a system's entire operational surface — data, compute, authentication, network — lived inside a boundary the organization owned outright.

Recovery programs were never built to ask a second question: does everything the restored system needs to actually function come back with it? For most of the industry's history, that question didn't need asking, because the answer was structurally guaranteed. If you owned the system, you owned what it depended on. Restoring the one restored the other by definition.

That guarantee is gone, and the recovery discipline hasn't caught up.

The Recovery Boundary and the Dependency Boundary Diverged

Recovery boundaries stopped expanding at the exact moment dependency boundaries started expanding.

For most of the last two decades, the two moved together. A system's recovery boundary — the set of things a DR plan was responsible for bringing back — and its dependency boundary — the set of things the system actually needed to function — were the same line, because the same organization owned both sides of it:

  • Database
  • Application
  • Authentication
  • Network Restore all four, in the right order, and the service came back. There was nothing to fail outside the boundary, because nothing the system depended on lived outside it.

That's no longer the shape of the problem. A modern production service depends on:

  • Cloud identity (Entra, Okta, Ping)
  • DNS resolution (often a third-party provider, not internal)
  • A SaaS management or control plane
  • A certificate authority
  • One or more external APIs
  • A cloud control plane the organization doesn't operate Half of that list sits outside the boundary most recovery plans define as "the system." The dependency boundary kept expanding as architecture modernized. The recovery boundary — built around asset ownership, not service composition — stayed exactly where it was twenty years ago. Organizations changed what a service depends on without changing what a recovery plan is responsible for.

This is worth distinguishing precisely from three adjacent failure modes. Recovery Design Boundary asks whether a recovery architecture was designed at all — a prerequisite question underneath this one. Recovery Authority Fragmentation asks whether the people, credentials, and operational knowledge needed to execute recovery survive the incident — a question about who can act, not what was in scope.

Continuity Execution Boundary is the closest relative, and the distinction matters enough to state directly. That framework — and its companion piece, The Dependencies Recovery Plans Forget — asks whether continuity survives once technical recovery has already been validated as executable: the failover chain has been adversarially tested, and the open question is whether the business actually resumes operating once it passes. That companion piece names identity, DNS, certificates, and network as the specific dependencies recovery plans routinely fail to model — a downstream, testing-coverage question sitting on top of a completed architecture.

This post asks a question one layer further back: was the recovery boundary architecturally defined against the service's real dependency boundary in the first place, before anyone got to the testing stage at all. The progression runs in one direction:

  • Continuity Execution Boundary — can recovery execute across its continuity boundary?
  • This post — was the recovery boundary itself scoped against the service's actual dependency boundary, or against what the organization happens to own?
  • Recoverability Gap — once that scoping mismatch is exposed, what remains unrecoverable at all? Three different questions, in sequence, not three versions of the same one. A recovery program can pass Continuity Execution Boundary's test — the failover chain works, the business resumes — and still have drawn its recovery boundary in the wrong place from day one, because "the business resumes" and "the boundary was drawn correctly" are not the same claim. This post is about the drawing, not the resuming.

That's also where this connects to Movement Authority Boundary territory, and it's not an obvious link. Movement Authority Boundary names the pattern where the thing that permits participation — routing, admission, discovery — sits structurally separate from the thing being moved. A recovered system with an unavailable identity provider or unreachable control plane is the same pattern applied to recovery: the system can be fully healthy and still unable to participate, because the authority that permits participation was never inside the recovery boundary to begin with. The recovered asset and the right to operate are not the same thing, and recovery programs have been quietly assuming they are.

The System Recovered. The Service Didn't.

The pattern is easiest to dismiss in isolation and hardest to dismiss in aggregate. It's also easy to mistake for the identity/DNS/certificate failure surfaces this pillar has already covered in detail — so the examples below deliberately sit outside that territory. Start with the case that's easiest to wave off:

The VM restored. The license activation service was unavailable. The software boots and runs. It just can't confirm its license against a vendor's activation server that was never named in the recovery plan, and depending on the vendor's grace-period logic, the application degrades or locks entirely somewhere between hour four and hour thirty-six of the incident.

The application restored. The external payment API was unavailable. The app is fully up. Every transaction that touches the third-party payment processor fails, because the processor's endpoint, its IP allowlist, or its webhook callback path was scoped as "a vendor's problem," not as part of what "recovered" means for this service.

The database restored. The message broker was unavailable. Data integrity is intact and queryable. Nothing downstream processes, because the event-driven integrations this service depends on route through an externally hosted broker that sits entirely outside the recovery boundary — the data is there, the system isn't, because the system was never just the database.

The backup platform restored. The cloud control plane was unavailable. The tool responsible for recovery itself depends on a control plane outside the organization's operational reach. This is the case that's hardest to dismiss, because it isn't the service failing to recover — it's the recovery mechanism itself sharing the same dependency-boundary problem as everything it's supposed to be recovering.

By the time you reach the last case, "just add it to the runbook" has stopped being a satisfying answer. Every one of these is the same failure at a different layer: the asset came back, the service didn't, and the reason lives entirely outside what the recovery plan was scoped to cover.

Working through this on your own recovery architecture? rack2cloud.com/recovery-boundary-dependency-failure/

Why Recovery Testing Misses This

Recovery testing validates exactly what recovery planning scopes — which is the problem. A standard DR exercise answers three questions: can we restore, can we boot, can we log in. All three questions are answered from inside the recovery boundary, using access and connectivity the test environment usually grants by default or by exception.

None of those three questions asks whether the dependencies the service needs in a real incident — the ones outside the organization's ownership — are reachable, authenticated, or even operating on the incident's timeline rather than the vendor's. A test that succeeds by restoring the asset and never touches the dependency boundary isn't validating service recovery. It's validating asset recovery, and calling it something larger than what it actually proved.

Recovery Dependency Mapper: Rack2Cloud's Recovery Dependency Mapper tests recovery sequencing and detects cycles among known dependencies. It does not determine whether a dependency belongs inside the recovery boundary — sequence the dependencies you know, then challenge the boundary that determined which dependencies you identified.

DR test boundary stops before the recovery boundary does

Quick Recoverability Test

For every service classified as critical, the recovery test should be answering all of the following — not just the first two:

Quick Recoverability Test:

  • Can it be restored?
  • Can it authenticate?
  • Can it resolve its dependencies?
  • Can it obtain the certificates it requires?
  • Can it reach the control planes it requires?
  • Can it operate if any of those dependencies are unavailable?
  • Can those dependencies be recovered by the same team responsible for recovering the service? If the honest answer to any of the last four is "we don't know" or "someone else owns that," the recovery plan has tested the asset, not the service — and the last question is the one that exposes why: the team that owns the recovery runbook usually doesn't own the systems the runbook silently depends on.

The New Recoverability Problem

Recoverability is no longer a property of the asset. It's a property of the dependency chain the asset sits inside, and most of that chain now sits outside any single team's recovery authority.

This doesn't mean recovery programs are wrong to restore what they own — that work is still necessary. It means "restored" and "recoverable" have quietly become different claims, and recovery programs are still reporting the first one as though it were the second. Until recovery scope is redrawn around what a service actually depends on, rather than what an organization happens to own, that gap will keep surfacing at incident time, in front of the people least equipped to close it on the spot.

Architect's Verdict

Recovery programs were designed for systems that owned the things they depended on. Modern services increasingly depend on systems they do not control. The next generation of recovery failures won't occur because restoration failed. They'll occur because the dependency was never part of the recovery design.

Additional Resources


Originally published at rack2cloud.com

Top comments (0)