High-Availability Distributed Sagas via Active-Active Topologies

In tier-1 core banking systems, system availability is not merely an operational metric—it is a regulatory mandate. Outages in mission-critical business capabilities like Payment Execution or Position Keeping trigger immediate regulatory scrutiny, massive financial penalties, and systemic reputational damage.

Historically, banks relied on Active-Passive (Disaster Recovery) infrastructure topologies. In these setups, a primary data center handles all live traffic while a secondary data center sits idle, asynchronously replicating state. However, during a catastrophic regional failure (e.g., a total cloud region outage or physical fiber cut), Active-Passive failover mechanisms incur unacceptable recovery costs:

• Non-Zero RPO (Recovery Point Objective): Asynchronous replication logs lag behind live transactions, resulting in lost financial writes upon failover.
• High RTO (Recovery Time Objective): Manual or semi-automated DNS cutovers, database promotion scripts, and cache warm-ups take minutes—if not hours—to execute safely.
• In-Flight Saga Corruption: Distributed transactions mid-execution during a regional collapse are left in indeterminate states, leading to orphan holds on customer accounts or uncompensated ledger debits.

Under the Xenon Architecture Standards, high-availability banking platforms eliminate single-region failure risks by deploying Multi-Region Active-Active Topologies for distributed Saga orchestrators (such as Temporal) backed by Distributed SQL engines (such as CockroachDB or Google Cloud Spanner). This paper evaluates the architectural patterns, state fencing mechanisms, consensus latency mathematics, and automated DR runbooks required to achieve RPO = 0 and RTO < 1s across regional disasters.

💡 Explore the Xenon Architecture Standard For comprehensive architectural blueprints, BIAN service domain mappings, and dual-orchestration integration patterns, visit the official Xenon Architecture Guide. To inspect reference code, infrastructure templates, and open-source banking modules, explore the VecPay-Tech GitHub Organization.

---

BIAN Service Domain HA & SLA Matrix

Not all BIAN Service Domains demand identical disaster recovery profiles. Engineering active-active consensus across multi-region WANs introduces network latency trade-offs that must be categorized by domain criticality.

BIAN Service Domain	Criticality Tier	Target RPO	Target RTO	Multi-Region Topology Pattern	Database Consensus Strategy
Payment Execution	Tier-1 (Critical)	0 (Zero Loss)	< 1 Second	Multi-Region Active-Active (3+ Regions)	Serialized Quorum Consensus (Raft/Paxos)
Position Keeping	Tier-1 (Critical)	0 (Zero Loss)	< 1 Second	Multi-Region Active-Active (3+ Regions)	Multi-Region Row-Level Lease Pinning
Credit Assessment	Tier-2 (High)	< 1 Second	< 10 Seconds	Dual-Region Active-Active (Follow-the-Leader)	Asynchronous Cross-Region Group Replication
Customer Onboarding	Tier-2 (High)	< 5 Seconds	< 30 Seconds	Active-Active with Regional Sharding	Eventual Consistency / Asynchronous Outbox
Historical Reporting	Tier-3 (Standard)	< 1 Hour	< 1 Hour	Active-Passive (DR Read Replica)	Asynchronous Batch Snapshot Hydration

---

Mathematical Model: Multi-Region Consensus Latency vs. Availability

Achieving zero RPO across active-active regions requires synchronous consensus (e.g., Raft or Paxos). A transaction cannot be acknowledged to a client until a quorum of regions confirms state persistence.

The minimum commit latency $T_{\text{commit}}$ for a distributed Saga step executing across a multi-region deployment spanning $M$ regions (where $M \ge 3$ to prevent split-brain tie votes) is governed by the round-trip time (RTT) to the closest majority quorum:

$$T_{\text{commit}} = T_{\text{local-execution}} + \max\bigl( \text{RTT}{\text{region-local}}, \text{RTT}{\text{region-quorum-1}} \bigr) + T_{\text{fsync}}$$

Where $\text{RTT}_{\text{region-quorum-1}}$ represents the network latency to the geographically closest peer region required to form a majority quorum ( $\lfloor M/2 \rfloor + 1$ nodes).

The composite system availability $A_{\text{system}}$ across $M$ independent active-active regions, assuming an individual region unavailability probability of $P_{\text{region-fail}}$ , is expressed as:

$$A_{\text{system}} = 1 - \sum_{k=0}^{\lfloor M/2 \rfloor} \binom{M}{k} \cdot \bigl( P_{\text{region-fail}} \bigr)^{M-k} \cdot \bigl( 1 - P_{\text{region-fail}} \bigr)^{k}$$

By deploying across 3 geographically distinct active-active regions with independent power grids and cloud control planes, system availability reaches 99.999% ("five nines"), guaranteeing continuous execution even during the complete destruction of an entire cloud region.

---

Active-Active Multi-Region Saga Architecture Topology

In a Xenon active-active architecture, application workers and Saga orchestrators run simultaneously in all active regions. Traffic is routed via Global Anycast DNS or BGP Anycast routing.

                           MULTI-REGION ACTIVE-ACTIVE TOPOLOGY

                      +------------------------------------------+
                      | Global Anycast DNS / Cloudflare Route    |
                      +--------------------+---------------------+
                                           |
                ┌──────────────────────────┼──────────────────────────┐
                │ 33% Traffic              │ 33% Traffic              │ 33% Traffic
                v                          v                          v
  +--------------------------+ +--------------------------+ +--------------------------+
  | REGION A (us-east-1)     | | REGION B (us-west-2)     | | REGION C (eu-central-1)  |
  |                          | |                          | |                          |
  | [ App / Temporal Worker] | | [ App / Temporal Worker] | | [ App / Temporal Worker] |
  |            │             | |            │             | |            │             |
  |            v             | |            v             | |            v             |
  | [ CockroachDB Node 1 ]   | | [ CockroachDB Node 2 ]   | | [ CockroachDB Node 3 ]   |
  +------------+-------------+ +------------+-------------+ +------------+-------------+
               ▲                            ▲                            ▲
               └────────────────────────────┼────────────────────────────┘
                                Multi-Region Raft Quorum
                                (Synchronous State Replications)

1. Global Traffic Management: Requests enter through Anycast BGP routing, directing client traffic to the closest operational region.
2. Stateless App / Saga Workers: Stateless Temporal application workers run in all regions, connecting to the local region's database pool.
3. Distributed SQL Consensus Layer: CockroachDB or Google Cloud Spanner maintains row-level Raft consensus across regions. A write committed in Region A is synchronously replicated to Region B before acknowledging completion, guaranteeing RPO = 0.

---

Multi-Region Saga State Persistence & Fencing Mechanics

The primary risk during a network partition between regions is the Zombie Worker Scenario: Region A experiences a transient network partition and is temporarily isolated. Region B assumes Region A is dead and takes over execution of an in-flight Payment Saga. If Region A wakes back up mid-way, both regions might attempt to execute steps or compensations simultaneously, leading to double-spends.

To prevent double execution, Xenon Architecture mandates Epoch Fencing Tokens enforced by distributed database leases.

1. Database Schema for Multi-Region Saga State (sagas_schema.sql)

CREATE TABLE saga_instances (
    saga_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    bian_domain VARCHAR(255) NOT NULL,
    current_state VARCHAR(255) NOT NULL,
    epoch_token BIGINT NOT NULL DEFAULT 1,
    lease_owner_region VARCHAR(64) NOT NULL,
    lease_expires_at TIMESTAMPTZ NOT NULL,
    payload JSONB NOT NULL,
    updated_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP
) LOCALITY REGIONAL BY ROW;

-- Ensure high-concurrency balance updates execute near primary account region
ALTER TABLE saga_instances CONFIGURE ZONE USING 
    num_replicas = 3,
    constraints = '{"+region=us-east-1": 1, "+region=us-west-2": 1, "+region=eu-central-1": 1}';

2. Idempotent Epoch-Fenced Saga Execution Engine (Go / Temporal SDK)

The Go code below illustrates how a Temporal worker in Region B acquires a lease, increments the epoch_token, and invalidates any lingering execution attempts from Region A:

package sagas

import (
    "context"
    "database/sql"
    "fmt"
    "time"

    "go.temporal.io/sdk/workflow"
)

type SagaLeaseManager struct {
    db           *sql.DB
    regionName   string
    leaseTimeout time.Duration
}

func NewSagaLeaseManager(db *sql.DB, regionName string) *SagaLeaseManager {
    return &SagaLeaseManager{
        db:           db,
        regionName:   regionName,
        leaseTimeout: 5 * time.Second,
    }
}

// AcquireOrRenewLease uses conditional CAS (Compare-And-Swap) to fence out old workers
func (m *SagaLeaseManager) AcquireOrRenewLease(ctx context.Context, sagaID string) (int64, error) {
    tx, err := m.db.BeginTx(ctx, &sql.TxOptions{Isolation: sql.LevelSerializable})
    if err != nil {
        return 0, fmt.Errorf("failed to begin serializable transaction: %w", err)
    }
    defer tx.Rollback()

    var currentEpoch int64
    var currentOwner string
    var leaseExpires time.Time

    // 1. Read current saga fencing state
    query := `SELECT epoch_token, lease_owner_region, lease_expires_at FROM saga_instances WHERE saga_id = $1 FOR UPDATE`
    err = tx.QueryRowContext(ctx, query, sagaID).Scan(&currentEpoch, &currentOwner, &leaseExpires)
    if err != nil {
        return 0, fmt.Errorf("failed to fetch saga state: %w", err)
    }

    now := time.Now().UTC()

    // 2. Verify if lease is held by another active region
    if currentOwner != m.regionName && now.Before(leaseExpires) {
        return 0, fmt.Errorf("lease currently held by active region %s until %s", currentOwner, leaseExpires)
    }

    // 3. Increment Epoch Token to fence out zombie workers from previous region
    newEpoch := currentEpoch + 1
    newExpiry := now.Add(m.leaseTimeout)

    updateQuery := `
        UPDATE saga_instances 
        SET epoch_token = $1, lease_owner_region = $2, lease_expires_at = $3, updated_at = $4 
        WHERE saga_id = $5 AND epoch_token = $6`

    res, err := tx.ExecContext(ctx, updateQuery, newEpoch, m.regionName, newExpiry, now, sagaID, currentEpoch)
    if err != nil {
        return 0, fmt.Errorf("fencing update failed: %w", err)
    }

    rows, err := res.RowsAffected()
    if err != nil || rows == 0 {
        return 0, fmt.Errorf("CAS fencing failure: epoch token mutated concurrently by another region")
    }

    if err := tx.Commit(); err != nil {
        return 0, fmt.Errorf("failed to commit lease acquisition: %w", err)
    }

    return newEpoch, nil
}

// PaymentSagaWorkflow executes with cross-region fencing checks
func HighAvailabilityPaymentSaga(ctx workflow.Context, sagaID string, input PaymentSagaInput) error {
    options := workflow.ActivityOptions{
        StartToCloseTimeout: 10 * time.Second,
        RetryPolicy: &workflow.RetryPolicy{
            MaximumAttempts: 3,
        },
    }
    ctx = workflow.WithActivityOptions(ctx, options)

    // Step 1: Claim Lease and Fence Execution Context
    var epochToken int64
    err := workflow.ExecuteActivity(ctx, AcquireLeaseActivity, sagaID).Get(ctx, &epochToken)
    if err != nil {
        return fmt.Errorf("aborting saga execution: failed to acquire multi-region lease: %w", err)
    }

    // Step 2: Pass Epoch Token to Downstream Microservices for Strict Mutex Checking
    err = workflow.ExecuteActivity(ctx, ExecuteFencedDebitActivity, input.SourceAccount, input.Amount, epochToken).Get(ctx, nil)
    if err != nil {
        // Run Compensations with Epoch Token
        _ = workflow.ExecuteActivity(ctx, ExecuteFencedCompensationActivity, input.SourceAccount, input.Amount, epochToken).Get(ctx, nil)
        return fmt.Errorf("saga failed, compensations executed: %w", err)
    }

    return nil
}

---

Automated DR Runbooks & Failover Mechanics

In an active-active topology, regional failover does not require manual human intervention. Continuous automated health checking at the ingress mesh layer detects regional unresponsiveness and reroutes traffic in sub-seconds.

                    AUTOMATED SUB-SECOND FAILOVER TIMELINE

  T=0ms           T=150ms                     T=300ms                 T=450ms
  Region A        Health Probes Fail          Envoy / Anycast         Region B Workers
  Infrastructure  (3x 50ms Timeout)           Reroutes Traffic        Claim Leases &
  Collapses       ──────►                     to Region B             Resume Pending Sagas
  ─────────────────────────────────────────────────────────────────────────────────►

Automated Envoy Multi-Region Route Failover Configuration

The following Envoy Proxy configuration snippet illustrates automatic client-side load balancer eviction of a failing active region:

static_resources:
  clusters:
    - name: bian_payment_execution_cluster
      connect_timeout: 0.25s
      type: STRICT_DNS
      lb_policy: ROUND_ROBIN
      common_lb_config:
        healthy_panic_threshold:
          value: 50.0
      health_checks:
        - timeout: 0.1s
          interval: 0.2s
          unhealthy_threshold: 2
          healthy_threshold: 1
          http_health_check:
            path: "/healthz/liveness"
      outlier_detection:
        consecutive_5xx: 3
        interval: 1s
        base_ejection_time: 30s
        max_ejection_percent: 100
      load_assignment:
        cluster_name: bian_payment_execution_cluster
        endpoints:
          - priority: 0 # Local Region Priority
            locality:
              region: us-east-1
            lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: payment-api.us-east-1.internal.xenon
                      port_value: 8443
          - priority: 1 # Failover Region Priority (Sub-second automatic fallback)
            locality:
              region: us-west-2
            lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: payment-api.us-west-2.internal.xenon
                      port_value: 8443

---

Chaos Engineering, Validation & DR Testing

To verify RPO = 0 and RTO < 1s compliance for regulatory bodies (e.g., Fed, EBA, MAS), core banking architectures must execute continuous, automated chaos injection experiments in staging and production environments.

Chaos Mesh Experiment: Region Isolation Simulation (region_blackhole.yaml)

This experiment drops all network packets between Region A (us-east-1) and the remaining cluster nodes while an active load test of 10,000 TPS payment Sagas is executing:

apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: simulate-region-a-blackhole
  namespace: xenon-chaos
spec:
  action: partition
  mode: all
  selector:
    namespaces:
      - bian-payment-execution
    labelSelectors:
      topology.kubernetes.io/region: "us-east-1"
  direction: both
  target:
    selector:
      namespaces:
        - bian-payment-execution
      labelSelectors:
        topology.kubernetes.io/region: "us-west-2"
  duration: "5m"
  scheduler:
    cron: "0 2 * * 6" # Run automatically every Saturday at 02:00 UTC

Verification Criteria During Chaos Experiments

During a simulated regional blackout, automated monitoring systems must verify three non-negotiable assertions:

1. Zero Corrupted States: Database consistency checkers verify that no account balance exhibits a split-brain mismatch or missing ledger transaction (RPO = 0).
2. Sub-Second RTO: Ingress metrics demonstrate that 99.9% of HTTP/gRPC requests resume successful processing in peer regions within < 850ms of network partition onset.
3. No Double-Compensations: Temporal history logs verify that no Saga executed compensating actions twice due to race conditions between surviving regions.

---

Deploying high-availability distributed Sagas via multi-region Active-Active topologies guarantees that core banking platforms survive full cloud-region disasters without data loss or prolonged downtime. By combining distributed SQL Raft consensus, epoch-fenced Saga execution, and automated ingress failover routing, financial systems achieve strict RPO = 0 and RTO < 1s operational compliance across all critical BIAN service domains.