A full cloud outage is a different failure
A cloud provider can fail in a way that leaves machines running but makes the system hard to operate. Deployment APIs, identity, DNS, service discovery, and queues may be unreachable. A backup image is not much help if the team cannot provision it or obtain credentials. This differs from an Availability Zone (AZ) failure, where the rest of the platform remains under normal control.
A stand-in asks what must remain usable when the primary control plane is unavailable. It is a separate service for a few critical operations, with enough cached state to make bounded decisions. It is a second operating path with a smaller promise, not a restore point waiting to be rehydrated.
The design brief changes. Multi-AZ and multi-region deployments reduce different risks; a stand-in needs its own failure domain and activation path. It may use another cloud or a separately managed environment, but it cannot depend on the primary control plane. AWS's reliability guidance frames disaster recovery around recovery objectives and architecture. A stand-in adds a stricter question: can the recovery system operate while the primary is unreachable?
What Monzo Stand-in gets right
Monzo's engineering post describes Stand-in as separate backup banking infrastructure. The Primary Platform runs on Amazon Web Services, while Stand-in runs on Google Cloud Platform. The platforms have independent Kubernetes clusters and separate services. Stand-in exposes limited endpoints and a simplified app experience for card spending, cash withdrawal, bank transfers, balance and transaction views, and freezing or unfreezing cards.
The important choice is separation. Monzo's post says that common operations, including card payment processing, are implemented independently in the two platforms. Stand-in is therefore not a copy of the core banking platform with another database connection. Different code reduces the chance that one software defect, deployment mistake, or assumption will take down both paths.
The design also explains why strong consistency was not applied to every replicated value. If both platforms had to acknowledge every write, losing either could block updates. Monzo keeps the primary as the system of record and treats Stand-in decisions as temporary. Disaster recovery becomes a product boundary: preserve a few useful actions, and define what happens to everything else.
Independence beats perfect replication
A common response to cloud-provider risk is to replicate the entire deployment in a second region or cloud. A full replica can fit some recovery objectives, but it is expensive and complex. It also reproduces shared assumptions: a bad identity policy, Kubernetes upgrade, or shared DNS or release system can affect both copies.
A stand-in chooses independence over feature parity. Instead of duplicating every service, the team identifies the smallest set of operations that must survive. For a bank, that may be payment processing and transfers. For a SaaS platform, it may be read-only access to critical data or the ability to start a failover process manually. The stand-in is a deliberately limited system, not a second edition of the whole product.
Traditional active-passive recovery can still be useful, but its contract is to promote the same service with the same data model and APIs. A stand-in accepts a different API and reduced functionality when that separates failure modes. The goal is a working path while the primary is unreachable, not an invisible handoff.
Sync only the state you can tolerate
During a network partition, a design cannot promise both strong consistency and availability for the same write. A stand-in needs a bounded stale view and a policy for each action. Monzo's post describes non-blocking, eventually consistent replication, immutable data copied from the Primary Platform, separate state created while Stand-in is active, and a durable queue of effects for recovery.
Monzo says its minimal state includes balances, limited transaction history, card and account details, pots, and payees. It monitors replication lag rather than pretending the view is perfect. A smaller system should carry a monotonic version or source timestamp, reject older events, and alert when freshness crosses policy.
A cached balance may be stale. A read-only view can show its last known value, while an irreversible write may require a conservative decline or review queue. Each accepted action needs an idempotency key and durable record so recovery can distinguish a new operation from a replay.
Keep the emergency surface small
A small emergency surface is easier to reason about, test, and operate under stress. Every additional service adds dependencies, configuration, and another way for activation to fail.
This is a deliberate architectural constraint. For a payments system, the minimum useful surface might authorize a transaction from cached state and record the decision for later reconciliation. For an e-commerce platform, it might serve product listings from a static cache and let authenticated users view their order history. The exact boundary comes from the customer promise, not from the list of services already running in the primary platform.
The stand-in should not include administrative or analytics features or expose the full primary API. Every endpoint is a potential defect; every synchronized table is a freshness and recovery problem. A boring service with a clear refusal mode is more useful than a broad service that fails halfway through an operation.
Design the switch as a product
Activating a stand-in is not a simple DNS cutover. It changes the customer experience, the write path, and the recovery plan. Treat the switch as a product feature rather than an operational afterthought.
The activation control should be explicit and reversible. An operator or independent health check can request the transition, but the stand-in should refuse activation when its cache is empty, sync lag is outside policy, or its dependencies are unhealthy. Record the decision and its reason in an audit log the primary cannot erase.
Observability needs the same independence. Logs, metrics, traces, and alert delivery cannot depend only on the primary monitoring stack. Customer communication also belongs in the design: users may see fewer features or delayed transactions, and support staff need a visible state that explains why.
During Stand-in operation, some values are stale, some writes are provisional, and some requests must be rejected. Making those rules visible is safer than routing traffic to a hidden copy and asking downstream services to infer what happened.
Test the path before you need it
A stand-in that has never exercised its activation path is an unverified assumption. Test promotion, degraded behavior, and return to the primary. A small matrix can run regularly:
| Scenario | Property to verify |
|---|---|
| Stale snapshot | The service rejects or limits an unsafe write and still serves the actions allowed by policy. |
| Duplicate event | Replaying an event does not double-apply an effect or change a settled decision. |
| Failed promotion | A missing dependency leaves the service in a known state and does not advertise partial availability. |
| Primary recovery | The stand-in drains new writes, emits its durable effects, and returns control without losing the audit trail. |
Fault injection should remove the dependencies the design claims not to need: block primary DNS, revoke primary identity permissions, disable a control-plane API, and interrupt sync. Run the tests against the real activation mechanism, not a test-only flag. Kubernetes readiness and liveness probes help with process health, but they do not prove fresh business state; that condition needs its own test and alert.
A small reference implementation
These examples illustrate a stand-in service. They are not production-ready and have not been tested in an outage. The service is a transaction authorizer with a bounded sync queue, explicit transitions, and a recovery record.
Start with a queue that fails loudly when it is full. A bounded queue that silently evicts old snapshots can turn a known outage into an unknown data gap.
import asyncio
from dataclasses import dataclass
from enum import Enum
from queue import Empty, Full, Queue
class StandInState(str, Enum):
IDLE = "idle"
ACTIVE = "active"
DRAINING = "draining"
FAILED = "failed"
@dataclass(frozen=True)
class AccountSnapshot:
account_id: str
balance: int
frozen: bool
version: int
class SyncQueue:
def __init__(self, max_items: int = 10_000) -> None:
self._items: Queue[AccountSnapshot] = Queue(maxsize=max_items)
def put(self, snapshot: AccountSnapshot) -> None:
try:
self._items.put_nowait(snapshot)
except Full as exc:
raise RuntimeError("sync queue is full") from exc
def drain(self) -> list[AccountSnapshot]:
snapshots = []
while True:
try:
snapshots.append(self._items.get_nowait())
except Empty:
return snapshots
The queue rejects input when full, giving the sync worker an observable failure instead of discarding the oldest state. The snapshot version lets the consumer ignore delayed events that would move an account backward.
The service owns its transitions and applies snapshots only when their versions advance:
class StandInService:
_allowed = {
StandInState.IDLE: {StandInState.ACTIVE, StandInState.FAILED},
StandInState.ACTIVE: {StandInState.DRAINING, StandInState.FAILED},
StandInState.DRAINING: {StandInState.IDLE, StandInState.FAILED},
StandInState.FAILED: {StandInState.IDLE},
}
def __init__(self) -> None:
self._state = StandInState.IDLE
self._accounts: dict[str, AccountSnapshot] = {}
self._lock = asyncio.Lock()
@property
def state(self) -> StandInState:
return self._state
async def transition(self, target: StandInState) -> None:
async with self._lock:
if target not in self._allowed[self._state]:
raise RuntimeError(f"invalid transition: {self._state} -> {target}")
if target is StandInState.ACTIVE and not self._accounts:
raise RuntimeError("cannot activate without cached state")
self._state = target
async def apply_snapshot(self, snapshot: AccountSnapshot) -> bool:
async with self._lock:
current = self._accounts.get(snapshot.account_id)
if current and snapshot.version <= current.version:
return False
self._accounts[snapshot.account_id] = snapshot
return True
async def authorize_transaction(
self,
account_id: str,
amount: int,
) -> tuple[bool, str]:
if amount <= 0:
return False, "invalid_amount"
if self._state is not StandInState.ACTIVE:
return False, "stand_in_not_active"
async with self._lock:
account = self._accounts.get(account_id)
if not account:
return False, "account_not_cached"
if account.frozen:
return False, "account_frozen"
if account.balance < amount:
return False, "insufficient_balance_cached"
return True, "approved"
The authorizer never calls the primary. The caller still needs an idempotency key and a reservation or ledger rule; this example only shows the availability decision. A real policy may decline more aggressively when the snapshot is old.
The activation check should combine primary reachability with stand-in readiness. The probes must use paths that remain available when the primary control plane is not.
from dataclasses import dataclass
from typing import Awaitable, Callable
@dataclass(frozen=True)
class Health:
primary_up: bool
cache_ready: bool
sync_lag_ok: bool
async def decide_state(
service: StandInService,
probe_primary: Callable[[], Awaitable[bool]],
cache_ready: Callable[[], bool],
sync_lag_ok: Callable[[], bool],
) -> StandInState:
health = Health(
primary_up=await probe_primary(),
cache_ready=cache_ready(),
sync_lag_ok=sync_lag_ok(),
)
if health.primary_up:
if service.state is StandInState.ACTIVE:
await service.transition(StandInState.DRAINING)
return service.state
if (
service.state is StandInState.IDLE
and health.cache_ready
and health.sync_lag_ok
):
await service.transition(StandInState.ACTIVE)
return service.state
The function refuses promotion when the cache or freshness check is unhealthy. Recovery moves to DRAINING first, giving the write path a visible stopping point.
After recovery, reconciliation should classify records before applying anything:
from dataclasses import dataclass
@dataclass(frozen=True)
class Advice:
transaction_id: str
account_id: str
amount: int
def classify_advices(
primary_ids: set[str],
already_applied: set[str],
advices: list[Advice],
) -> tuple[list[Advice], list[Advice], list[Advice]]:
seen = set(already_applied)
duplicates, review = [], []
for advice in advices:
if advice.transaction_id in seen or advice.transaction_id in primary_ids:
duplicates.append(advice)
else:
review.append(advice)
seen.add(advice.transaction_id)
return [], duplicates, review
async def begin_rollback(service: StandInService) -> None:
await service.transition(StandInState.DRAINING)
This deliberately auto-applies nothing. A duplicate is recorded, and an unseen advice goes to review or a separately designed idempotent step. A delayed payment is visible; a double-applied payment is harder to undo.
The trade-off is deliberate
Designing for a full cloud outage means choosing which failure a service must tolerate. A full replica optimizes feature parity, but it can carry the same software, identity assumptions, and operational dependencies into the recovery environment. A stand-in optimizes independence and accepts a smaller promise.
That promise has costs. Users see fewer features, some state is eventually consistent, and the team must reconcile provisional effects when the primary returns. The reward is a recovery path whose assumptions are visible: a separate platform, a bounded cache, explicit refusal modes, an independent switch, and tests that exercise the path before an emergency. Monzo's Stand-in is a concrete example of this design, but the pattern applies to any service where a short list of useful actions matters more than keeping the entire product online.
Originally published on Dispatch.
Top comments (0)