Pradeep Kandepaneni

Posted on Jul 21

Preventing Unsafe Regional Failover in Amazon EKS with Policy-Gated Recovery

#aws #devops #kubernetes #sre

This article was originally published on AWS Builder Center.

A secondary Amazon EKS cluster can appear healthy while still being unsafe to receive production traffic. This article presents a policy-gated recovery model that validates data replication, compute capacity, application health, container images, secrets, certificates, and synthetic business transactions before promoting a recovery Region.

This article was originally published on [AWS Builder Center]

https://builder.aws.com/content/3GnlrIlqWy7Hel5zlqaIMsg2rPM/preventing-unsafe-regional-failover-in-amazon-eks-with-policy-gated-recovery

Why a Healthy Standby Region Can Still Be Unsafe

Running an application across multiple Availability Zones protects against many infrastructure failures, but it does not provide complete protection from a regional outage. A common recovery design is to maintain a second Amazon Elastic Kubernetes Service (Amazon EKS) cluster in another AWS Region and redirect traffic when the primary Region becomes unavailable.

The problem is that a reachable cluster is not necessarily ready to operate as the active production Region.

Kubernetes readiness probes may succeed while the database replica is behind, required container images are missing, regional secrets contain incorrect endpoints, certificates are unavailable, or the recovery cluster lacks enough compute capacity. A DNS or load-balancer health check cannot detect all of these conditions.

For stateful workloads, premature promotion can be more damaging than a temporary outage. It can introduce data loss, conflicting writes, authentication failures, partial application availability, or a recovery environment that collapses as soon as production traffic arrives.

A safer approach is to treat regional promotion as a controlled engineering decision rather than a simple routing change.

This article presents a policy-gated recovery model for Amazon EKS. The model uses mandatory safety gates and a weighted recovery-readiness score to validate infrastructure capacity, application health, data replication, container images, secrets, certificates, GitOps state, writer fencing, and synthetic business transactions before traffic and write authority move to the recovery Region.

Reference Architecture

The architecture uses two AWS Regions: a primary Region serving production traffic and a warm-standby recovery Region maintained at reduced capacity.

Each Region contains an independent Amazon EKS cluster, regional networking, application load balancing, monitoring, secrets, certificates, and container images. Application configuration is synchronized through a GitOps workflow so that the recovery cluster continuously receives approved Kubernetes manifests and deployment changes.

Stateful data is replicated using Amazon Aurora Global Database. Container images are copied through Amazon Elastic Container Registry (Amazon ECR) cross-Region replication, while AWS Secrets Manager provides regional secret replicas.

These replication mechanisms must be validated independently. Successful replication configuration does not prove that every required image, secret value, certificate, endpoint, or database change is available and usable in the recovery Region.

Amazon Application Recovery Controller (ARC) Region switch coordinates the recovery workflow. Region switch uses recovery plans composed of ordered or parallel execution steps. The recovery plan can be initiated manually or through approved automation, depending on the workload's operational requirements.

A separate recovery-readiness evaluator checks mandatory safety conditions before database promotion, workload scaling, or traffic movement is permitted.

The design separates four concerns:

Infrastructure readiness — The recovery Region has sufficient networking, compute, quota, and worker-node capacity.
Application readiness — Critical workloads are synchronized, schedulable, healthy, and able to complete synthetic business transactions.
Data readiness — Replication lag is within the approved recovery-point objective, and conflicting writes are prevented.
Dependency readiness — Required images, secrets, certificates, identity services, queues, caches, and external integrations are available.

A regional endpoint health check alone cannot prove these conditions. Promotion is allowed only after every mandatory safety gate passes.

Why Endpoint Health Is Not Recovery Readiness

Traditional failover designs often rely on a small number of signals:

Load balancer endpoint availability
DNS health-check status
Kubernetes pod readiness
Application response codes
Infrastructure resource availability

These signals are valuable, but they answer only narrow questions.

A successful readiness probe confirms that a container can respond to a configured health endpoint. It does not prove that the application can authenticate users, perform database writes, retrieve secrets, process queue messages, contact third-party services, or sustain production traffic.

Consider the following failure conditions:

Application pods are healthy, but the Aurora secondary cluster has replication lag beyond the accepted recovery-point objective.
Kubernetes Deployments exist, but worker nodes cannot scale because of service quotas or insufficient subnet address capacity.
The image tag exists in the recovery Region, but the required immutable image digest does not.
Secrets were replicated, but a connection string still references a primary-Region endpoint.
The recovery cluster is synchronized, but the GitOps controller reverses emergency replica changes.
TLS certificates are missing, expired, or not attached to the recovery load balancer.
The database is promoted, but the original writer continues accepting requests, creating a split-brain risk.
The login endpoint responds, but a complete business transaction fails.

Regional recovery readiness must therefore be evaluated as a combined system property, not as a single health-check result.

Separate Hard Gates from Readiness Scores

The proposed model uses two types of controls:

Hard promotion gates
Weighted readiness signals

This separation is critical.

A weighted score is useful for summarizing operational health, but it must never compensate for a failed data-safety requirement. A recovery Region with excellent application health but unsafe database state should not receive production writes.

Hard Promotion Gates

Hard gates represent conditions that are non-negotiable. If any hard gate fails, promotion stops.

Database Replication

Replication status must be healthy.
Replication lag must remain below the approved threshold.
The target database cluster must be eligible for failover or switchover.
Database engine versions must be compatible with the selected recovery operation.

Writer Fencing

The previous writer must be unavailable, isolated, or explicitly fenced before the recovery Region accepts writes.
Applications must not maintain writable connections to both Regions.
Connection pools and DNS caches must be considered during the transition.

Amazon EKS Capacity

The required Kubernetes resources must exist in the recovery cluster.
Sufficient worker-node capacity must be available or capable of scaling.
Subnets, IP addresses, instance quotas, and Availability Zone distribution must support the desired replica count.
Critical pods must become schedulable within the recovery timeout.

Application Health

Required Deployments must have the minimum number of ready replicas.
Pod disruption budgets, topology constraints, and health probes must be valid.
Critical services must pass internal and external health validation.

Container Image Availability

Required repositories must exist in the recovery Region.
The exact image digests referenced by production manifests must be available.
Image pull permissions and node access must be validated.

Amazon ECR replication rules should be configured before images are pushed. Images that existed before replication was configured should not be assumed to appear automatically in the destination Region.

Secret and Configuration Readiness

Required secrets must exist in the recovery Region.
Secret versions must match the expected deployment version.
Regional endpoints embedded in secret values must be validated.
AWS Key Management Service keys and permissions must be available in the recovery Region.

Certificate Readiness

Certificates must be valid and available in the required Region.
Load balancer listeners must reference the correct certificate.
Certificate expiration and domain validation status must be checked.

GitOps Synchronization

Critical applications must be synchronized.
No unresolved high-severity drift should exist.
GitOps tools must not reverse recovery-time scaling changes.

Synthetic Business Transactions

A synthetic test should validate more than a basic HTTP response. Depending on the application, it can perform steps such as:

Authenticate a test user.
Create a temporary transaction.
Read the newly created record.
Update the record.
Delete or clean up the test data.
Verify that logs and metrics were produced.

ARC Recovery-Plan Evaluation

Before execution, the Region switch plan should pass its available plan-evaluation checks. Custom policy gates should complement—not replace—the validations performed by ARC.

Weighted Recovery-Readiness Score

After all hard gates pass, weighted signals can provide a consolidated operational score.

A reference weighting model could use:

25% application health
20% infrastructure capacity
20% data health
15% dependency health
10% observability health
10% policy compliance

The weights must reflect the workload's actual risk profile. A transaction-processing system may assign greater weight to data health, while a stateless content-delivery workload may emphasize capacity and dependency availability.

The score should not be interpreted as a universal standard. It is an operational decision aid.

A simple decision policy is:

IF any hard gate fails:
    BLOCK PROMOTION

ELSE IF readiness score is below 90:
    REQUIRE OPERATOR REVIEW

ELSE:
    ALLOW THE APPROVED RECOVERY WORKFLOW

A score of 95 must never override a failed writer-fencing gate, excessive database replication lag, or a missing production image.

Example Recovery Policy

The following example represents a simplified policy definition. Production implementations should store thresholds in version-controlled configuration and require peer review for changes.

recoveryPolicy:
  targetRegion: us-west-2

  hardGates:
    maximumDatabaseLagSeconds: 300
    minimumAvailableReplicasPercent: 95

    requiredSyntheticTests:
      - login
      - create-order
      - read-order
      - update-order

    requireDatabasePromotionEligibility: true
    requireImageDigestValidation: true
    requireRegionalSecretValidation: true
    requireCertificateValidation: true
    requireGitOpsSynchronization: true
    requireWriterFencing: true
    requireArcPlanEvaluation: true

  readinessScore:
    minimumAutomaticApprovalScore: 90

    weights:
      applicationHealth: 25
      infrastructureCapacity: 20
      dataHealth: 20
      dependencyHealth: 15
      observabilityHealth: 10
      policyCompliance: 10

The threshold values above are examples. They must be aligned with the organization's recovery-time objective, recovery-point objective, application behavior, data-loss tolerance, and operational approval model.

Example Readiness Evaluation

The following Python example demonstrates the decision logic. It intentionally keeps hard-gate evaluation separate from weighted scoring.

from dataclasses import dataclass
from typing import Iterable


@dataclass(frozen=True)
class HardGate:
    name: str
    passed: bool
    details: str = ""


@dataclass(frozen=True)
class WeightedSignal:
    name: str
    value: float
    weight: float


def evaluate_recovery_readiness(
    hard_gates: Iterable[HardGate],
    weighted_signals: Iterable[WeightedSignal],
    minimum_score: float = 90.0,
) -> dict:
    failures = [
        {
            "name": gate.name,
            "details": gate.details,
        }
        for gate in hard_gates
        if not gate.passed
    ]

    if failures:
        return {
            "decision": "BLOCK",
            "score": None,
            "failed_gates": failures,
        }

    signals = list(weighted_signals)

    if not signals:
        return {
            "decision": "BLOCK",
            "score": None,
            "failed_gates": [
                {
                    "name": "weighted-signals",
                    "details": "No weighted readiness signals were provided.",
                }
            ],
        }

    total_weight = sum(signal.weight for signal in signals)

    if total_weight <= 0:
        raise ValueError("Total signal weight must be greater than zero.")

    normalized_score = sum(
        signal.value * signal.weight
        for signal in signals
    ) / total_weight

    decision = (
        "ALLOW"
        if normalized_score >= minimum_score
        else "REVIEW"
    )

    return {
        "decision": decision,
        "score": round(normalized_score, 2),
        "failed_gates": [],
    }

In a production implementation, each signal should include:

Collection timestamp
Source system
Region
Resource identifier
Evaluation result
Threshold
Evidence location
Expiration period

Stale evidence should fail closed. A database-lag measurement collected 30 minutes earlier should not approve a current regional promotion.

Recovery Workflow with AWS ARC Region Switch

A human-approved automated workflow is usually more defensible than fully automatic promotion for business-critical stateful workloads.

A reference recovery sequence is:

An operator declares a regional recovery event or initiates a controlled exercise.
ARC Region switch evaluates the configured recovery plan.
The recovery-readiness evaluator collects current evidence from both Regions.
Hard promotion gates are evaluated.
The recovery EKS cluster scales critical Kubernetes resources.
Worker-node autoscaling increases node capacity as required.
The GitOps controller verifies application synchronization.
Aurora Global Database performs the appropriate failover operation.
Applications reconnect to the new database writer.
Synthetic transactions run against the recovery Region.
Writer fencing is confirmed.
ARC routing controls or the approved traffic-management mechanism moves traffic.
Post-promotion monitoring verifies error rate, latency, saturation, and business transactions.
The workflow completes, pauses, or rolls back based on validation results.

ARC Region switch can scale supported EKS workload resources as part of a recovery plan. However, increasing Kubernetes replica counts does not independently create worker-node capacity. The cluster still depends on a node-provisioning mechanism such as Karpenter, Cluster Autoscaler, or EKS Auto Mode.

Recovery testing must therefore validate both layers:

Desired Kubernetes replica capacity
Actual schedulable worker-node capacity

Prevent GitOps from Reversing Recovery Changes

GitOps systems continuously reconcile actual cluster state with the state stored in Git. This is normally desirable, but it can conflict with disaster-recovery actions.

For example, ARC may increase a Deployment from three replicas to twelve replicas during recovery. If the Git repository still specifies three replicas, the GitOps controller may attempt to restore the lower count.

The same conflict can occur when a recovery workflow modifies Horizontal Pod Autoscaler behavior to prevent immediate scale-down.

Recovery design should explicitly define which system owns temporary scaling changes.

Options include:

Configure GitOps to ignore the Deployment replica field.
Configure GitOps to ignore specific Horizontal Pod Autoscaler fields.
Commit recovery-state changes to a dedicated Git branch.
Temporarily suspend reconciliation for selected resources.
Use environment-specific overlays that represent active and standby capacity.

The selected approach must be tested. "GitOps is configured" is not sufficient evidence that GitOps and the recovery workflow will cooperate.

Kubernetes Resilience Controls

The recovery Region should enforce the same—or stricter—workload-resilience controls used in the primary Region.

The following example spreads application replicas across Availability Zones:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout-api
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: checkout-api
  template:
    metadata:
      labels:
        app: checkout-api
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: checkout-api

      containers:
        - name: checkout-api
          image: ACCOUNT_ID.dkr.ecr.REGION.amazonaws.com/checkout-api@sha256:IMAGE_DIGEST

          resources:
            requests:
              cpu: "500m"
              memory: "512Mi"
            limits:
              cpu: "1"
              memory: "1Gi"

          readinessProbe:
            httpGet:
              path: /health/ready
              port: 8080
            initialDelaySeconds: 10
            periodSeconds: 5

          livenessProbe:
            httpGet:
              path: /health/live
              port: 8080
            initialDelaySeconds: 30
            periodSeconds: 10

Using immutable image digests reduces ambiguity during recovery. The promotion gate can verify that the exact digest referenced by the workload exists in the recovery Region.

Database Promotion and Write Safety

Aurora Global Database supports planned switchovers and disaster-recovery failovers. These operations should not be treated as equivalent.

A planned switchover is used when the current primary and secondary Regions are healthy and the organization intentionally moves the primary role. An unplanned failover is used when the primary Region is impaired or unavailable.

The recovery workflow must account for:

Current replication lag
Engine-version compatibility
Promotion eligibility
Application reconnection behavior
DNS caching
Connection-pool expiration
Transaction retries
Potential data loss during unplanned failure
Reintroduction of the previous primary as a secondary
Failback sequencing

Applications should use retry logic that distinguishes transient connectivity failure from permanent transaction rejection. Idempotency controls are especially important for payment, order, provisioning, and messaging workflows.

Writer fencing must be validated before the recovery Region accepts writes. This can involve isolating the previous writer, revoking application access, disabling routing, applying security controls, or otherwise proving that dual-writer operation cannot occur.

Container Image Validation

Amazon ECR supports cross-Region and cross-account private-image replication. However, replication configuration is not equivalent to complete image readiness.

The recovery workflow should verify:

Destination repository existence
Expected image digest
Multi-architecture manifest availability
Image-pull permissions
Encryption configuration
Vulnerability-scanning status, where required
Node access to the destination repository

Images pushed before a replication rule was configured should not be assumed to exist in the destination Region. Validate the exact digests required by active Kubernetes manifests.

A simple validation process can:

Extract all image references from production workloads.
Resolve tags to immutable digests.
Query the destination ECR registry.
Compare expected and available digests.
Block promotion when any required digest is missing.

Secret and Certificate Validation

AWS Secrets Manager can replicate encrypted secret data and metadata across Regions. Replication is useful, but it does not understand the semantic meaning of the values stored inside a secret.

A replicated secret may still contain:

A primary-Region database endpoint
A Region-specific queue URL
A private service hostname unavailable from the recovery Region
A certificate identifier valid only in the primary Region
An external allowlist that excludes recovery egress addresses

The recovery gate must validate both existence and correctness.

A secret-readiness test should verify:

Expected secret version
Recovery-Region encryption key
Resource policy
Application IAM access
Regional endpoint values
Rotation status
Successful application retrieval

Certificates should also be validated by Region, expiration date, domain name, listener association, and deployment status.

Observability During Recovery

Recovery without observability is guesswork.

The recovery Region must provide independent visibility into:

Kubernetes replica availability
Unschedulable pods
Worker-node provisioning
Application error rate
Request latency
Database replication lag
Database connection failures
Queue depth and consumer lag
Cache availability
Synthetic transaction success
DNS or routing state
ARC recovery-plan execution
GitOps synchronization
Certificate errors
Secret-access failures

Do not make the recovery workflow dependent solely on a monitoring system hosted in the Region being deactivated.

Critical dashboards, alarms, logs, and operational evidence should remain available during a regional impairment.

Post-promotion validation should continue for a defined stabilization period. Passing a single synthetic transaction immediately after traffic movement does not prove sustained recovery.

Recovery-Time Budget

A recovery-time objective should be decomposed into measurable stages.

The following is a worked reference target—not a production benchmark:

Recovery stage	Target
Event declaration and operator approval	3 minutes
ARC plan evaluation and policy checks	4 minutes
EKS workload and node-capacity scaling	8 minutes
Database promotion and application reconnection	6 minutes
Synthetic transaction validation	4 minutes
Traffic movement and stabilization	5 minutes
Total target recovery budget	30 minutes

Each stage should produce timestamps and evidence.

A recovery exercise should record:

Start and completion time
Duration of each execution step
Manual interventions
Failed gates
Overrides
Database lag
Replica scaling time
Node-provisioning time
Synthetic-test results
Routing-change time
Stabilization results
Corrective actions

With a small number of exercises, publish every result along with the minimum, median, and maximum. Avoid statistically weak percentile claims from insufficient samples.

Failback Is Part of the Design

Failover is only half of disaster recovery.

After the original Region becomes healthy, the organization must decide whether and when to return production traffic. Immediate failback can introduce additional risk while teams are still stabilizing the environment.

A controlled failback process should:

Confirm that the original Region is stable.
Rebuild or rejoin the original database cluster as required.
Re-establish replication in the intended direction.
Validate that no divergent writes exist.
Synchronize GitOps state.
Validate images, secrets, certificates, queues, caches, and dependencies.
Restore sufficient EKS and worker-node capacity.
Run synthetic business transactions.
Fence the current writer before changing write authority.
Perform a planned database switchover when supported.
Move application traffic.
Monitor the restored primary Region through a stabilization period.
Return the other Region to the intended standby capacity.

Failback should use the same policy discipline as failover. "The original Region is back online" is not sufficient evidence that it is safe to become primary again.

Security Considerations

A recovery platform has permission to alter critical infrastructure and routing. Its security model deserves the same scrutiny as a production deployment system.

Recommended controls include:

Dedicated IAM roles for recovery-plan execution
Least-privilege EKS access entries
Namespace-scoped Kubernetes permissions where practical
Separate permissions for evaluation and execution
Multi-person approval for high-risk recovery actions
AWS CloudTrail logging
Immutable audit records
Encryption for data and secrets in both Regions
Restricted emergency-access procedures
Periodic credential and role review
Protection against unauthorized policy-threshold changes
Signed or reviewed recovery configuration
Separation of duties between application, platform, and security teams

Manual override capability may be necessary, but overrides should require explicit authorization, justification, timestamped evidence, and post-event review.

Data-safety gates such as writer fencing should not support routine bypass.

Limitations and Trade-Offs

This architecture introduces operational discipline, but it does not eliminate complexity.

Cost

A warm-standby environment costs more than a pilot-light design because a functional copy of the platform remains active at reduced capacity.

Recovery Speed

A pilot-light architecture can reduce cost but generally requires more provisioning and scaling during recovery.

DNS Behavior

DNS-based traffic movement is affected by resolver and client caching. Configured TTL values do not guarantee that every client immediately honors the new destination.

Application Architecture

Applications with hard-coded regional dependencies, non-idempotent workflows, or tightly coupled external services may require significant redesign.

Data Consistency

Asynchronous cross-Region replication can introduce data-loss exposure during an unplanned failure. The organization must define an acceptable recovery-point objective.

Automation Risk

Fully automatic failover may be inappropriate for stateful systems when failure detection is uncertain or write safety cannot be proven automatically.

Service Availability

AWS service features, quotas, instance types, and database-engine support vary by Region and version. Validate the target Regions before implementation.

Scoring Limitations

A readiness score is a decision aid. It is not a substitute for hard safety gates, engineering review, or repeated recovery exercises.

Recommended Adoption Approach

Do not begin by applying this model to every production workload.

Start with one representative application that has:

Clear recovery objectives
Defined business transactions
Known dependencies
Testable data-replication behavior
Manageable compliance constraints
An engaged application owner

A practical adoption sequence is:

Inventory all regional dependencies.
Define recovery-time and recovery-point objectives.
Classify hard promotion gates.
Build the recovery Region with infrastructure as code.
Configure application and data replication.
Implement evidence collection.
Implement synthetic transactions.
Create the ARC Region switch plan.
Integrate policy evaluation.
Run recovery exercises without traffic movement.
Run controlled failover and failback exercises.
Correct gaps before expanding to additional applications.

The goal is not to automate every step immediately. The goal is to make every decision explicit, measurable, repeatable, and auditable.

Conclusion

A secondary Amazon EKS cluster should not be promoted merely because its API endpoint and pods are reachable.

Safe regional recovery requires evidence that compute capacity, data, application state, identity, container images, certificates, dependencies, routing, and business transactions are ready together.

Hard promotion gates protect non-negotiable safety conditions. A weighted readiness score gives operators a consolidated view of the remaining operational signals. Amazon Application Recovery Controller Region switch provides an orchestration layer for executing the recovery workflow, while GitOps, infrastructure as code, replication services, synthetic testing, and observability provide the supporting controls.

The result is not zero-risk failover. No architecture can honestly promise that.

The objective is a recovery process in which unsafe conditions block promotion, operational decisions are supported by current evidence, and every recovery exercise improves the next one.

Disclaimer

The architecture, code examples, thresholds, and recovery-time budget in this article are provided as a reference scenario. They must be adapted and validated against each organization's workload characteristics, recovery objectives, security requirements, compliance obligations, service quotas, AWS Region availability, and operational procedures.

This article represents the author's technical perspective. It is not an AWS-supported solution, an official architecture from AWS, or a guarantee of availability, recovery time, or data protection.