DEV Community: Pradeep Kandepaneni

Making a Service Survive an Availability Zone Outage — and Proving It for $0

Pradeep Kandepaneni — Tue, 21 Jul 2026 08:01:47 +0000

A reproducible multi-AZ resilience walkthrough: spread a service across simulated zones, kill one under load, and measure the dropped requests — plus the parts that only show up in real production.

Originally published on the AWS Builder Center: https://builder.aws.com/content/3GnvKe3PBz3fP3m3pFFDFghjIxX/making-a-service-survive-an-availability-zone-outage-and-proving-it-for-dollar0
Repo: https://github.com/PradeepKandepaneni/golden-path-resilience

Most articles about availability zones tell you to "spread across three AZs" and stop there, as if the words alone did the work. They don't. Spreading pods across zones is easy to write and easy to get subtly wrong, and the only way to know whether your setup actually survives a zone going dark is to make one go dark and watch what happens.

So that's what this does. It takes a small service, spreads it across three simulated zones, then kills one zone while a stream of requests is hitting the service — and counts how many of those requests fail. If the architecture holds, that number is zero. And because the whole thing runs on a local cluster and free CI, you can clone it and watch your own zone die without spending a cent or waiting for a real outage.

This is a sequel to a earlier piece where I built a "golden path" — a free, end-to-end pipeline from a git push to a deployed Kubernetes workload. Same tiny service here; the interesting part this time isn't the pipeline, it's the topology. Repo: github.com/PradeepKandepaneni/golden-path-resilience.

I'll be honest up front about one boundary, because it matters: the demo simulates zones on a local cluster. That reproduces the Kubernetes-level behaviour — scheduling, spread, eviction, rescheduling, traffic routing — which is most of what you actually tune. It does not reproduce real cross-zone latency or partial network partitions. I'll cover those separately, from the parts of this that only show up in production. Keeping that line clear is the difference between a demo that teaches and one that quietly lies.

The setup: three zones on one laptop

Real AWS worker nodes get a topology.kubernetes.io/zone label automatically. Locally, I fake it: a kind cluster with three worker nodes, and a script that labels them az-a, az-b, az-c. Now Kubernetes believes it has three zones to reason about, and every zone-aware feature behaves as it would in a real account.

That single label is the hinge the whole thing turns on. Everything below — the spread, the eviction behaviour, the rescheduling — keys off it.

The three pieces that actually do the work

Zone resilience for a stateless service isn't one setting. It's three, and each covers a gap the others don't.

Topology spread constraints distribute the pods evenly across the zones. Six replicas, three zones, two per zone. Without this, the scheduler is free to pile most of your pods into one zone, and "we run in three AZs" becomes a lie the first time the wrong one fails.

The important detail is one field, and it's the field people get wrong:

yaml
topologySpreadConstraints:

maxSkew: 1 topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: ScheduleAnyway labelSelector: matchLabels: app: golden-path

whenUnsatisfiable: ScheduleAnyway is deliberate, and it's the opposite of what a lot of people reach for. The stricter DoNotSchedule sounds safer — enforce the even spread, always. But think about what happens during an actual outage: a zone dies, its pods are evicted, and they need somewhere to go. With DoNotSchedule, rescheduling those pods onto the two surviving zones would break the even-spread rule, so the scheduler refuses, and the pods sit Pending. You've traded away availability to preserve a tidy distribution during the exact moment availability matters most. ScheduleAnyway says: spread evenly when you can, but during degradation, keep the service up. That's the resilient choice, and it's a one-word decision that most tutorials never explain.

A PodDisruptionBudget is the second piece. Draining a node is a voluntary disruption, and Kubernetes will happily evict everything on it unless you tell it not to. minAvailable: 4 (of six) means the drain of a single zone can proceed — it only needs to evict two — while four pods are guaranteed to stay serving the entire time. The PDB is what turns "drain the node" from a gamble into a guarantee.

Readiness probes are the quiet third piece. When the evicted pods reschedule onto surviving zones, they're not instantly ready. The readiness probe gates traffic so a pod only receives requests once /healthz is green. Without it, you'd route requests at pods that are still starting and serve errors during the recovery — snatching a small outage from the jaws of a successful failover.

None of these three is exotic. The point is that resilience is the combination, and dropping any one of them leaves a gap that only shows up when a zone actually fails.

Killing a zone, and counting the damage

Description is cheap, so the repo doesn't describe — it measures. scripts/az-outage-demo.sh starts a load generator firing a few requests a second at the service, waits for steady state, then simulates the outage:

bash
kubectl cordon "$NODE"
kubectl drain "$NODE" --ignore-daemonsets --delete-emptydir-data --force

Cordon stops new pods landing on the doomed node; drain evicts the ones already there. From Kubernetes' point of view, that zone is gone. The load generator keeps firing the whole time, logging each request as ok or FAIL, and at the end the script tallies them:

==================================================
Requests sent during the zone outage : 175

Requests that FAILED : 0

PASS: the service survived the loss of zone az-a.

Zero failures across the outage. The four pods in the surviving zones never stopped serving; the two evicted pods rescheduled onto those zones and rejoined once ready; the Service only ever routed traffic to pods that were actually up. The client saw nothing.

The CI job asserts this on every push — it fails the build if more than a couple of requests drop. That's the part that makes this more than a story: a green check means the failover genuinely worked on real (if simulated) infrastructure, and you can read the run yourself instead of trusting my screenshot. Because it's a public repo, the whole thing — multi-node cluster, zone outage, load test — runs on free GitHub runners at no cost.

Why "ScheduleAnyway" is the whole lesson in one field

If you take one thing from the reproducible half of this, make it that field. It's a perfect small example of how resilience decisions are counterintuitive: the setting that sounds stricter and safer (DoNotSchedule) is the one that hurts you when it counts, and the "looser" one (ScheduleAnyway) is what keeps you up. You cannot reason your way to that from the words "spread across three AZs." You get there by asking what happens after a zone is already gone — which is the question most resilience writing forgets to ask.

Where the simulation ends and production begins

Everything above is reproducible on your laptop, and that's exactly why it's limited to the orchestration layer. Real multi-AZ adds things a local cluster can't fake, and pretending otherwise would undercut the honest parts. Three in particular are worth naming.

Cross-zone latency is real and it's not free. In a real account, every hop between zones adds a millisecond or two, and traffic that ping-pongs across zones under load adds up — in latency and, on AWS, in cross-AZ data transfer cost. Local kind has effectively zero inter-node latency, so it can't show you this. In production it shapes real decisions, like topology-aware routing to keep traffic in-zone when possible.

Partial failures are messier than a clean drain. My demo removes a zone cleanly. Real zones fail in uglier ways — a network partition where the zone is reachable but slow, or intermittently up. Those are the scenarios that expose whether your health checks and timeouts are tuned correctly, and a clean kubectl drain never triggers them.

The data layer is the hard half, and it's deliberately not in the reproducible demo. Stateless failover — what this repo shows — is the easy part precisely because any pod can serve any request. The moment state is involved, resilience gets genuinely hard: a multi-AZ database failover has a measurable failover window, in-flight transactions to reason about, and connection pools that need to survive the primary moving.

Those three are why the honest framing of this whole exercise is "I reproduced the orchestration behaviour for free, and here's where production goes further" — not "I built a production multi-AZ system on my laptop." The reproducible part is genuinely useful and genuinely verifiable. The rest is where real accounts and real incidents teach you things a simulation can't.

Takeaways

Resilience is a combination, not a setting. Spread, disruption budget, and readiness each cover a gap the others leave. Ship one without the others and you've built a service that looks resilient right up until a zone fails.

Ask what happens after the failure, not before. The ScheduleAnyway decision only makes sense if you reason about the moment a zone is already gone. Most resilience mistakes come from designing for the healthy state and never simulating the broken one.

Prove it, don't describe it. "Spread across three AZs" is a claim. "Here's the demo that kills a zone under load and reports zero dropped requests, and here's the CI run" is evidence — and evidence is worth more than any architecture diagram.

The repo is public and the demo runs green for $0: github.com/PradeepKandepaneni/golden-path-resilience. Clone it, run make up && make demo, and kill a zone yourself.

Preventing Unsafe Regional Failover in Amazon EKS with Policy-Gated Recovery

Pradeep Kandepaneni — Tue, 21 Jul 2026 07:30:42 +0000

This article was originally published on AWS Builder Center.

A secondary Amazon EKS cluster can appear healthy while still being unsafe to receive production traffic. This article presents a policy-gated recovery model that validates data replication, compute capacity, application health, container images, secrets, certificates, and synthetic business transactions before promoting a recovery Region.

This article was originally published on [AWS Builder Center]

https://builder.aws.com/content/3GnlrIlqWy7Hel5zlqaIMsg2rPM/preventing-unsafe-regional-failover-in-amazon-eks-with-policy-gated-recovery

Why a Healthy Standby Region Can Still Be Unsafe

Running an application across multiple Availability Zones protects against many infrastructure failures, but it does not provide complete protection from a regional outage. A common recovery design is to maintain a second Amazon Elastic Kubernetes Service (Amazon EKS) cluster in another AWS Region and redirect traffic when the primary Region becomes unavailable.

The problem is that a reachable cluster is not necessarily ready to operate as the active production Region.

Kubernetes readiness probes may succeed while the database replica is behind, required container images are missing, regional secrets contain incorrect endpoints, certificates are unavailable, or the recovery cluster lacks enough compute capacity. A DNS or load-balancer health check cannot detect all of these conditions.

For stateful workloads, premature promotion can be more damaging than a temporary outage. It can introduce data loss, conflicting writes, authentication failures, partial application availability, or a recovery environment that collapses as soon as production traffic arrives.

A safer approach is to treat regional promotion as a controlled engineering decision rather than a simple routing change.

This article presents a policy-gated recovery model for Amazon EKS. The model uses mandatory safety gates and a weighted recovery-readiness score to validate infrastructure capacity, application health, data replication, container images, secrets, certificates, GitOps state, writer fencing, and synthetic business transactions before traffic and write authority move to the recovery Region.

Reference Architecture

The architecture uses two AWS Regions: a primary Region serving production traffic and a warm-standby recovery Region maintained at reduced capacity.

Each Region contains an independent Amazon EKS cluster, regional networking, application load balancing, monitoring, secrets, certificates, and container images. Application configuration is synchronized through a GitOps workflow so that the recovery cluster continuously receives approved Kubernetes manifests and deployment changes.

Stateful data is replicated using Amazon Aurora Global Database. Container images are copied through Amazon Elastic Container Registry (Amazon ECR) cross-Region replication, while AWS Secrets Manager provides regional secret replicas.

These replication mechanisms must be validated independently. Successful replication configuration does not prove that every required image, secret value, certificate, endpoint, or database change is available and usable in the recovery Region.

Amazon Application Recovery Controller (ARC) Region switch coordinates the recovery workflow. Region switch uses recovery plans composed of ordered or parallel execution steps. The recovery plan can be initiated manually or through approved automation, depending on the workload's operational requirements.

A separate recovery-readiness evaluator checks mandatory safety conditions before database promotion, workload scaling, or traffic movement is permitted.

The design separates four concerns:

Infrastructure readiness — The recovery Region has sufficient networking, compute, quota, and worker-node capacity.
Application readiness — Critical workloads are synchronized, schedulable, healthy, and able to complete synthetic business transactions.
Data readiness — Replication lag is within the approved recovery-point objective, and conflicting writes are prevented.
Dependency readiness — Required images, secrets, certificates, identity services, queues, caches, and external integrations are available.

A regional endpoint health check alone cannot prove these conditions. Promotion is allowed only after every mandatory safety gate passes.

Why Endpoint Health Is Not Recovery Readiness

Traditional failover designs often rely on a small number of signals:

Load balancer endpoint availability
DNS health-check status
Kubernetes pod readiness
Application response codes
Infrastructure resource availability

These signals are valuable, but they answer only narrow questions.

A successful readiness probe confirms that a container can respond to a configured health endpoint. It does not prove that the application can authenticate users, perform database writes, retrieve secrets, process queue messages, contact third-party services, or sustain production traffic.

Consider the following failure conditions:

Application pods are healthy, but the Aurora secondary cluster has replication lag beyond the accepted recovery-point objective.
Kubernetes Deployments exist, but worker nodes cannot scale because of service quotas or insufficient subnet address capacity.
The image tag exists in the recovery Region, but the required immutable image digest does not.
Secrets were replicated, but a connection string still references a primary-Region endpoint.
The recovery cluster is synchronized, but the GitOps controller reverses emergency replica changes.
TLS certificates are missing, expired, or not attached to the recovery load balancer.
The database is promoted, but the original writer continues accepting requests, creating a split-brain risk.
The login endpoint responds, but a complete business transaction fails.

Regional recovery readiness must therefore be evaluated as a combined system property, not as a single health-check result.

Separate Hard Gates from Readiness Scores

The proposed model uses two types of controls:

Hard promotion gates
Weighted readiness signals

This separation is critical.

A weighted score is useful for summarizing operational health, but it must never compensate for a failed data-safety requirement. A recovery Region with excellent application health but unsafe database state should not receive production writes.

Hard Promotion Gates

Hard gates represent conditions that are non-negotiable. If any hard gate fails, promotion stops.

Database Replication

Replication status must be healthy.
Replication lag must remain below the approved threshold.
The target database cluster must be eligible for failover or switchover.
Database engine versions must be compatible with the selected recovery operation.

Writer Fencing

The previous writer must be unavailable, isolated, or explicitly fenced before the recovery Region accepts writes.
Applications must not maintain writable connections to both Regions.
Connection pools and DNS caches must be considered during the transition.

Amazon EKS Capacity

The required Kubernetes resources must exist in the recovery cluster.
Sufficient worker-node capacity must be available or capable of scaling.
Subnets, IP addresses, instance quotas, and Availability Zone distribution must support the desired replica count.
Critical pods must become schedulable within the recovery timeout.

Application Health

Required Deployments must have the minimum number of ready replicas.
Pod disruption budgets, topology constraints, and health probes must be valid.
Critical services must pass internal and external health validation.

Container Image Availability

Required repositories must exist in the recovery Region.
The exact image digests referenced by production manifests must be available.
Image pull permissions and node access must be validated.

Amazon ECR replication rules should be configured before images are pushed. Images that existed before replication was configured should not be assumed to appear automatically in the destination Region.

Secret and Configuration Readiness

Required secrets must exist in the recovery Region.
Secret versions must match the expected deployment version.
Regional endpoints embedded in secret values must be validated.
AWS Key Management Service keys and permissions must be available in the recovery Region.

Certificate Readiness

Certificates must be valid and available in the required Region.
Load balancer listeners must reference the correct certificate.
Certificate expiration and domain validation status must be checked.

GitOps Synchronization

Critical applications must be synchronized.
No unresolved high-severity drift should exist.
GitOps tools must not reverse recovery-time scaling changes.

Synthetic Business Transactions

A synthetic test should validate more than a basic HTTP response. Depending on the application, it can perform steps such as:

Authenticate a test user.
Create a temporary transaction.
Read the newly created record.
Update the record.
Delete or clean up the test data.
Verify that logs and metrics were produced.

ARC Recovery-Plan Evaluation

Before execution, the Region switch plan should pass its available plan-evaluation checks. Custom policy gates should complement—not replace—the validations performed by ARC.

Weighted Recovery-Readiness Score

After all hard gates pass, weighted signals can provide a consolidated operational score.

A reference weighting model could use:

25% application health
20% infrastructure capacity
20% data health
15% dependency health
10% observability health
10% policy compliance

The weights must reflect the workload's actual risk profile. A transaction-processing system may assign greater weight to data health, while a stateless content-delivery workload may emphasize capacity and dependency availability.

The score should not be interpreted as a universal standard. It is an operational decision aid.

A simple decision policy is:

IF any hard gate fails:
    BLOCK PROMOTION

ELSE IF readiness score is below 90:
    REQUIRE OPERATOR REVIEW

ELSE:
    ALLOW THE APPROVED RECOVERY WORKFLOW

A score of 95 must never override a failed writer-fencing gate, excessive database replication lag, or a missing production image.

Example Recovery Policy

The following example represents a simplified policy definition. Production implementations should store thresholds in version-controlled configuration and require peer review for changes.

recoveryPolicy:
  targetRegion: us-west-2

  hardGates:
    maximumDatabaseLagSeconds: 300
    minimumAvailableReplicasPercent: 95

    requiredSyntheticTests:
      - login
      - create-order
      - read-order
      - update-order

    requireDatabasePromotionEligibility: true
    requireImageDigestValidation: true
    requireRegionalSecretValidation: true
    requireCertificateValidation: true
    requireGitOpsSynchronization: true
    requireWriterFencing: true
    requireArcPlanEvaluation: true

  readinessScore:
    minimumAutomaticApprovalScore: 90

    weights:
      applicationHealth: 25
      infrastructureCapacity: 20
      dataHealth: 20
      dependencyHealth: 15
      observabilityHealth: 10
      policyCompliance: 10

The threshold values above are examples. They must be aligned with the organization's recovery-time objective, recovery-point objective, application behavior, data-loss tolerance, and operational approval model.

Example Readiness Evaluation

The following Python example demonstrates the decision logic. It intentionally keeps hard-gate evaluation separate from weighted scoring.

from dataclasses import dataclass
from typing import Iterable


@dataclass(frozen=True)
class HardGate:
    name: str
    passed: bool
    details: str = ""


@dataclass(frozen=True)
class WeightedSignal:
    name: str
    value: float
    weight: float


def evaluate_recovery_readiness(
    hard_gates: Iterable[HardGate],
    weighted_signals: Iterable[WeightedSignal],
    minimum_score: float = 90.0,
) -> dict:
    failures = [
        {
            "name": gate.name,
            "details": gate.details,
        }
        for gate in hard_gates
        if not gate.passed
    ]

    if failures:
        return {
            "decision": "BLOCK",
            "score": None,
            "failed_gates": failures,
        }

    signals = list(weighted_signals)

    if not signals:
        return {
            "decision": "BLOCK",
            "score": None,
            "failed_gates": [
                {
                    "name": "weighted-signals",
                    "details": "No weighted readiness signals were provided.",
                }
            ],
        }

    total_weight = sum(signal.weight for signal in signals)

    if total_weight <= 0:
        raise ValueError("Total signal weight must be greater than zero.")

    normalized_score = sum(
        signal.value * signal.weight
        for signal in signals
    ) / total_weight

    decision = (
        "ALLOW"
        if normalized_score >= minimum_score
        else "REVIEW"
    )

    return {
        "decision": decision,
        "score": round(normalized_score, 2),
        "failed_gates": [],
    }

In a production implementation, each signal should include:

Collection timestamp
Source system
Region
Resource identifier
Evaluation result
Threshold
Evidence location
Expiration period

Stale evidence should fail closed. A database-lag measurement collected 30 minutes earlier should not approve a current regional promotion.

Recovery Workflow with AWS ARC Region Switch

A human-approved automated workflow is usually more defensible than fully automatic promotion for business-critical stateful workloads.

A reference recovery sequence is:

An operator declares a regional recovery event or initiates a controlled exercise.
ARC Region switch evaluates the configured recovery plan.
The recovery-readiness evaluator collects current evidence from both Regions.
Hard promotion gates are evaluated.
The recovery EKS cluster scales critical Kubernetes resources.
Worker-node autoscaling increases node capacity as required.
The GitOps controller verifies application synchronization.
Aurora Global Database performs the appropriate failover operation.
Applications reconnect to the new database writer.
Synthetic transactions run against the recovery Region.
Writer fencing is confirmed.
ARC routing controls or the approved traffic-management mechanism moves traffic.
Post-promotion monitoring verifies error rate, latency, saturation, and business transactions.
The workflow completes, pauses, or rolls back based on validation results.

ARC Region switch can scale supported EKS workload resources as part of a recovery plan. However, increasing Kubernetes replica counts does not independently create worker-node capacity. The cluster still depends on a node-provisioning mechanism such as Karpenter, Cluster Autoscaler, or EKS Auto Mode.

Recovery testing must therefore validate both layers:

Desired Kubernetes replica capacity
Actual schedulable worker-node capacity

Prevent GitOps from Reversing Recovery Changes

GitOps systems continuously reconcile actual cluster state with the state stored in Git. This is normally desirable, but it can conflict with disaster-recovery actions.

For example, ARC may increase a Deployment from three replicas to twelve replicas during recovery. If the Git repository still specifies three replicas, the GitOps controller may attempt to restore the lower count.

The same conflict can occur when a recovery workflow modifies Horizontal Pod Autoscaler behavior to prevent immediate scale-down.

Recovery design should explicitly define which system owns temporary scaling changes.

Options include:

Configure GitOps to ignore the Deployment replica field.
Configure GitOps to ignore specific Horizontal Pod Autoscaler fields.
Commit recovery-state changes to a dedicated Git branch.
Temporarily suspend reconciliation for selected resources.
Use environment-specific overlays that represent active and standby capacity.

The selected approach must be tested. "GitOps is configured" is not sufficient evidence that GitOps and the recovery workflow will cooperate.

Kubernetes Resilience Controls

The recovery Region should enforce the same—or stricter—workload-resilience controls used in the primary Region.

The following example spreads application replicas across Availability Zones:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout-api
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: checkout-api
  template:
    metadata:
      labels:
        app: checkout-api
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: checkout-api

      containers:
        - name: checkout-api
          image: ACCOUNT_ID.dkr.ecr.REGION.amazonaws.com/checkout-api@sha256:IMAGE_DIGEST

          resources:
            requests:
              cpu: "500m"
              memory: "512Mi"
            limits:
              cpu: "1"
              memory: "1Gi"

          readinessProbe:
            httpGet:
              path: /health/ready
              port: 8080
            initialDelaySeconds: 10
            periodSeconds: 5

          livenessProbe:
            httpGet:
              path: /health/live
              port: 8080
            initialDelaySeconds: 30
            periodSeconds: 10

Using immutable image digests reduces ambiguity during recovery. The promotion gate can verify that the exact digest referenced by the workload exists in the recovery Region.

Database Promotion and Write Safety

Aurora Global Database supports planned switchovers and disaster-recovery failovers. These operations should not be treated as equivalent.

A planned switchover is used when the current primary and secondary Regions are healthy and the organization intentionally moves the primary role. An unplanned failover is used when the primary Region is impaired or unavailable.

The recovery workflow must account for:

Current replication lag
Engine-version compatibility
Promotion eligibility
Application reconnection behavior
DNS caching
Connection-pool expiration
Transaction retries
Potential data loss during unplanned failure
Reintroduction of the previous primary as a secondary
Failback sequencing

Applications should use retry logic that distinguishes transient connectivity failure from permanent transaction rejection. Idempotency controls are especially important for payment, order, provisioning, and messaging workflows.

Writer fencing must be validated before the recovery Region accepts writes. This can involve isolating the previous writer, revoking application access, disabling routing, applying security controls, or otherwise proving that dual-writer operation cannot occur.

Container Image Validation

Amazon ECR supports cross-Region and cross-account private-image replication. However, replication configuration is not equivalent to complete image readiness.

The recovery workflow should verify:

Destination repository existence
Expected image digest
Multi-architecture manifest availability
Image-pull permissions
Encryption configuration
Vulnerability-scanning status, where required
Node access to the destination repository

Images pushed before a replication rule was configured should not be assumed to exist in the destination Region. Validate the exact digests required by active Kubernetes manifests.

A simple validation process can:

Extract all image references from production workloads.
Resolve tags to immutable digests.
Query the destination ECR registry.
Compare expected and available digests.
Block promotion when any required digest is missing.

Secret and Certificate Validation

AWS Secrets Manager can replicate encrypted secret data and metadata across Regions. Replication is useful, but it does not understand the semantic meaning of the values stored inside a secret.

A replicated secret may still contain:

A primary-Region database endpoint
A Region-specific queue URL
A private service hostname unavailable from the recovery Region
A certificate identifier valid only in the primary Region
An external allowlist that excludes recovery egress addresses

The recovery gate must validate both existence and correctness.

A secret-readiness test should verify:

Expected secret version
Recovery-Region encryption key
Resource policy
Application IAM access
Regional endpoint values
Rotation status
Successful application retrieval

Certificates should also be validated by Region, expiration date, domain name, listener association, and deployment status.

Observability During Recovery

Recovery without observability is guesswork.

The recovery Region must provide independent visibility into:

Kubernetes replica availability
Unschedulable pods
Worker-node provisioning
Application error rate
Request latency
Database replication lag
Database connection failures
Queue depth and consumer lag
Cache availability
Synthetic transaction success
DNS or routing state
ARC recovery-plan execution
GitOps synchronization
Certificate errors
Secret-access failures

Do not make the recovery workflow dependent solely on a monitoring system hosted in the Region being deactivated.

Critical dashboards, alarms, logs, and operational evidence should remain available during a regional impairment.

Post-promotion validation should continue for a defined stabilization period. Passing a single synthetic transaction immediately after traffic movement does not prove sustained recovery.

Recovery-Time Budget

A recovery-time objective should be decomposed into measurable stages.

The following is a worked reference target—not a production benchmark:

Recovery stage	Target
Event declaration and operator approval	3 minutes
ARC plan evaluation and policy checks	4 minutes
EKS workload and node-capacity scaling	8 minutes
Database promotion and application reconnection	6 minutes
Synthetic transaction validation	4 minutes
Traffic movement and stabilization	5 minutes
Total target recovery budget	30 minutes

Each stage should produce timestamps and evidence.

A recovery exercise should record:

Start and completion time
Duration of each execution step
Manual interventions
Failed gates
Overrides
Database lag
Replica scaling time
Node-provisioning time
Synthetic-test results
Routing-change time
Stabilization results
Corrective actions

With a small number of exercises, publish every result along with the minimum, median, and maximum. Avoid statistically weak percentile claims from insufficient samples.

Failback Is Part of the Design

Failover is only half of disaster recovery.

After the original Region becomes healthy, the organization must decide whether and when to return production traffic. Immediate failback can introduce additional risk while teams are still stabilizing the environment.

A controlled failback process should:

Confirm that the original Region is stable.
Rebuild or rejoin the original database cluster as required.
Re-establish replication in the intended direction.
Validate that no divergent writes exist.
Synchronize GitOps state.
Validate images, secrets, certificates, queues, caches, and dependencies.
Restore sufficient EKS and worker-node capacity.
Run synthetic business transactions.
Fence the current writer before changing write authority.
Perform a planned database switchover when supported.
Move application traffic.
Monitor the restored primary Region through a stabilization period.
Return the other Region to the intended standby capacity.

Failback should use the same policy discipline as failover. "The original Region is back online" is not sufficient evidence that it is safe to become primary again.

Security Considerations

A recovery platform has permission to alter critical infrastructure and routing. Its security model deserves the same scrutiny as a production deployment system.

Recommended controls include:

Dedicated IAM roles for recovery-plan execution
Least-privilege EKS access entries
Namespace-scoped Kubernetes permissions where practical
Separate permissions for evaluation and execution
Multi-person approval for high-risk recovery actions
AWS CloudTrail logging
Immutable audit records
Encryption for data and secrets in both Regions
Restricted emergency-access procedures
Periodic credential and role review
Protection against unauthorized policy-threshold changes
Signed or reviewed recovery configuration
Separation of duties between application, platform, and security teams

Manual override capability may be necessary, but overrides should require explicit authorization, justification, timestamped evidence, and post-event review.

Data-safety gates such as writer fencing should not support routine bypass.

Limitations and Trade-Offs

This architecture introduces operational discipline, but it does not eliminate complexity.

Cost

A warm-standby environment costs more than a pilot-light design because a functional copy of the platform remains active at reduced capacity.

Recovery Speed

A pilot-light architecture can reduce cost but generally requires more provisioning and scaling during recovery.

DNS Behavior

DNS-based traffic movement is affected by resolver and client caching. Configured TTL values do not guarantee that every client immediately honors the new destination.

Application Architecture

Applications with hard-coded regional dependencies, non-idempotent workflows, or tightly coupled external services may require significant redesign.

Data Consistency

Asynchronous cross-Region replication can introduce data-loss exposure during an unplanned failure. The organization must define an acceptable recovery-point objective.

Automation Risk

Fully automatic failover may be inappropriate for stateful systems when failure detection is uncertain or write safety cannot be proven automatically.

Service Availability

AWS service features, quotas, instance types, and database-engine support vary by Region and version. Validate the target Regions before implementation.

Scoring Limitations

A readiness score is a decision aid. It is not a substitute for hard safety gates, engineering review, or repeated recovery exercises.

Recommended Adoption Approach

Do not begin by applying this model to every production workload.

Start with one representative application that has:

Clear recovery objectives
Defined business transactions
Known dependencies
Testable data-replication behavior
Manageable compliance constraints
An engaged application owner

A practical adoption sequence is:

Inventory all regional dependencies.
Define recovery-time and recovery-point objectives.
Classify hard promotion gates.
Build the recovery Region with infrastructure as code.
Configure application and data replication.
Implement evidence collection.
Implement synthetic transactions.
Create the ARC Region switch plan.
Integrate policy evaluation.
Run recovery exercises without traffic movement.
Run controlled failover and failback exercises.
Correct gaps before expanding to additional applications.

The goal is not to automate every step immediately. The goal is to make every decision explicit, measurable, repeatable, and auditable.

Conclusion

A secondary Amazon EKS cluster should not be promoted merely because its API endpoint and pods are reachable.

Safe regional recovery requires evidence that compute capacity, data, application state, identity, container images, certificates, dependencies, routing, and business transactions are ready together.

Hard promotion gates protect non-negotiable safety conditions. A weighted readiness score gives operators a consolidated view of the remaining operational signals. Amazon Application Recovery Controller Region switch provides an orchestration layer for executing the recovery workflow, while GitOps, infrastructure as code, replication services, synthetic testing, and observability provide the supporting controls.

The result is not zero-risk failover. No architecture can honestly promise that.

The objective is a recovery process in which unsafe conditions block promotion, operational decisions are supported by current evidence, and every recovery exercise improves the next one.

Disclaimer

The architecture, code examples, thresholds, and recovery-time budget in this article are provided as a reference scenario. They must be adapted and validated against each organization's workload characteristics, recovery objectives, security requirements, compliance obligations, service quotas, AWS Region availability, and operational procedures.

This article represents the author's technical perspective. It is not an AWS-supported solution, an official architecture from AWS, or a guarantee of availability, recovery time, or data protection.

References

I Built a Platform Engineering Golden Path That Runs for $0 — Every Decision, and Everything That Broke

Pradeep Kandepaneni — Tue, 21 Jul 2026 05:33:22 +0000

Originally published on the AWS Builder Center: https://builder.aws.com/content/3GnbxJrdbgQ5sXNJTRXvHIB5OaV/i-built-a-platform-engineering-golden-path-that-runs-for-dollar0-every-decision-and-everything-that-broke
Repo: https://github.com/PradeepKandepaneni/golden-path

Most "platform engineering" articles show you a diagram. A tidy box labeled Internal Developer Platform, some arrows pointing at happy little developer icons, a paragraph about reducing cognitive load. Then the article ends, and you're holding a concept you can't run, can't fork, and can't check.

I wanted the opposite. The smallest thing a developer could actually use — push code, get back a tested, hardened, deployed workload, no tickets, no clicking — and I wanted it to cost nothing to run, so anyone reading this could clone it and watch it work instead of trusting me. No AWS bill to reproduce it. No "works on my cluster, promise."

So that's what I built, and this is the honest write-up. The repo is public and green on every push, so none of what follows requires taking my word for it: github.com/PradeepKandepaneni/golden-path. Every design decision, why I threw out the alternatives, and — the part actually worth your time — the one bug that took the whole thing down and what it taught me about hardened containers.

What a "golden path" actually is

The term gets thrown around, so let me pin it down. A golden path is the paved road a platform team hands its developers: one opinionated, well-lit route from "I have code" to "it's running," with the boring-but-critical parts already handled. Tests. Immutable image tags. Health checks. A pod that doesn't run as root. The developer doesn't wire those up per service, and doesn't get to forget them either. They're in the road.

Platform engineering, stripped of the conference gloss, is mostly the work of building and maintaining those roads so product teams stop rebuilding CI pipelines and Kubernetes manifests from scratch every time they ship something. Pave the road once; every team that drives on it inherits consistency, security, and speed for free. That's the whole pitch.

The trap — and I nearly walked straight into it — is believing you need a giant portal to deliver any of that. You don't. You need one road that works.

Decision 1: A template repo, not Backstage

The flashy way to demo an Internal Developer Platform is Backstage. It's the name everyone knows, it has a slick UI, and "built an IDP with Backstage" looks great on a repo.

I skipped it, and I'd tell you to skip it too, at least at the start. Backstage is a full-stack app you have to run, host, and babysit. For a solo project meant to show a pattern, that's weeks of wrestling with the portal before you've paved a single road — and a half-finished portal is worse than nothing, because it advertises that you started something big and quit.

A template repository plus GitHub Actions gets you the same idea — one automated path that scaffolds, tests, scans, and deploys a service — with zero portal code. It's also honestly closer to what a lot of real teams run day to day. Golden paths don't need a portal. They need an opinionated, automated pipeline. That's the thing I built, and it's a thing I could finish.

If I had to compress the lesson to four words: ship what you can finish.

Decision 2: kind in CI, not a cloud cluster

Here's the constraint that shaped everything else: the project had to run for free, in CI and on my laptop.

That killed running a real EKS cluster as the default. EKS is the right answer in production — it's on the roadmap as the documented production path — but the control plane alone runs about $0.10 an hour before you add a single node, load balancer, or NAT gateway. Leave one up while you iterate and it quietly turns into a $150–300/month surprise. Worse than the money: nobody can cheaply reproduce your work, which defeats the entire point of a demo repo.

So the default cluster is kind — Kubernetes running inside Docker. Locally, make up creates a cluster, builds the image, loads it, deploys, and smoke-tests it. In CI, the pipeline spins a kind cluster up inside the GitHub Actions runner, deploys to it, checks it, and throws it away when the job ends. My real run created the cluster and had the control plane Ready in 19 seconds:

Ready after 19s 💚 Set kubectl context to "kind-golden-path"

The economics are the quiet superpower. On a public repo, Actions minutes are unlimited and free, and the cluster lives and dies inside that free runner. The entire pipeline — a live Kubernetes deploy plus a real HTTP smoke test — costs nothing. Every push. Forever.

yaml

name: Create kind cluster uses: helm/kind-action@v1 with: cluster_name: golden-path config: kind/kind-config.yaml

That's the whole "provision Kubernetes" step, and it runs on GitHub's dime.

Decision 3: The image tag is the git SHA. Never :latest

Small, and non-negotiable. The image is tagged with the short git SHA of the commit that built it, and the manifest is pinned to that exact tag before the deploy.

yaml

name: Compute image tag from git SHA
id: vars
run: echo "sha=${GITHUB_SHA::7}" >> "$GITHUB_OUTPUT"
name: Build image
run: |
docker build \
-t golden-path:${{ steps.vars.outputs.sha }} \
--build-arg VERSION=${{ steps.vars.outputs.sha }} .

:latest is a lie. It's a mutable label pointing at different bytes over time, so "what's running" and "what I think is running" drift apart the second someone pushes again. A content-addressable tag means a running pod can tell you exactly which commit it came from — and this service does, because I bake the SHA into the binary at build time with an ldflags override. You'll see it pay off at the end of the war story below, in the actual smoke-test output.

Tiny discipline. Enormous payoff the first time you're staring at a production incident at 2 a.m. wondering what's actually deployed.

Decision 4: Distroless and non-root, by default

A golden path only earns its name if good practice is the default, not a thing you remember to bolt on. So the service runs on a distroless base — no shell, no package manager, almost nothing to pivot into — and the pod is hardened right in the manifest:

yaml
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]

Non-root. No privilege escalation. Read-only root filesystem. Every Linux capability dropped. It's the posture you'd eventually want a cluster-wide policy engine to enforce (Kyverno or OPA, on the roadmap), but here it starts as the baked-in default. The distroless choice also just produces a tiny image — the final artifact came out at 10.4MB, which the pipeline confirmed after loading it onto the node:

docker.io/library/golden-path fbab3d2 ae676b1d9427 10.4MB

And this decision is exactly where the project bit me. Which is the part I actually want to talk about.

The bug: CreateContainerConfigError

I pushed the first real version, opened the Actions tab, and watched it fail. Not at build. Not at cluster creation. At the deploy:

Waiting for deployment "golden-path" rollout to finish: 0 of 2 updated replicas are available...
error: timed out waiting for the condition
Error: Process completed with exit code 1

Zero of two replicas. A three-minute hang, then a red X.

Here's the first trap, and it's worth internalizing: that error is useless. "Timed out waiting for the condition" tells you the rollout didn't finish. It says nothing about why. The pods could be failing to pull an image, crashing on boot, flunking a health check, any of a dozen things. kubectl rollout status flattens all of them into the same timeout.

My honest first guess was an image problem — the tag hadn't matched, the load had failed, something like that. Wrong, as it turned out. But I couldn't even tell I was wrong, because the pipeline wasn't telling me anything.

So my first change wasn't a fix. It was making the pipeline talk. I added a diagnostics step that only fires when the deploy fails and dumps everything about the broken pods:

yaml

name: Diagnostics (only if the deploy failed) if: failure() run: | kubectl get pods -o wide kubectl describe pods -l app=golden-path kubectl get events --sort-by=.lastTimestamp | tail -40 kubectl logs -l app=golden-path --all-containers --tail=100 || true

Pushed again. Same timeout — but this time the log told the truth right underneath it:

NAME READY STATUS RESTARTS AGE
golden-path-6445746597-2czz8 0/1 CreateContainerConfigError 0 3m
golden-path-6445746597-svdct 0/1 CreateContainerConfigError 0 3m

CreateContainerConfigError. Not an image problem at all — the image had loaded fine (so much for my first guess). Not a crash — the container never even started. Kubernetes was refusing to create it from its config.

The cause is a genuinely sharp edge. I'd set runAsNonRoot: true, which tells the kubelet to refuse to start a container that would run as UID 0. To enforce that at create time, the kubelet needs to know the numeric user ID the container will run as. My distroless image runs as a non-root user — but it doesn't declare a numeric UID the kubelet can read at that moment. No number to check against, so the kubelet can't prove the container is non-root, and rather than risk running something as root, it refuses. Config error. Pods parked. Rollout hangs. Generic timeout.

One line fixes it: pin the UID so the check has a number. The distroless nonroot variant runs as UID 65532:

yaml
securityContext:
runAsNonRoot: true
runAsUser: 65532 # <- the fix
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]

Pushed. Green. And this time the rollout log shows the recovery happening in real time — two replicas coming up over about eight seconds:

Waiting for deployment "golden-path" rollout to finish: 0 of 2 updated replicas are available...
Waiting for deployment "golden-path" rollout to finish: 1 of 2 updated replicas are available...
deployment "golden-path" successfully rolled out

Then the in-cluster smoke test hit the service and got a real response back:

{"message":"hello from the golden path","version":"fbab3d2"}

Look at that version field: fbab3d2. That's the git SHA from Decision 3, baked into the binary, reported by the running pod. The immutable-tag discipline isn't theory in that line — the workload is telling me exactly which commit it came from.

Why give a whole section to one missing line of YAML? Because it's the shape of nearly every hardening bug you'll meet. Doing the secure thing — refusing to run as root — is what caused the failure. Had I left the container running as root like a lazy default, it would've started instantly, and I'd have shipped something worse. Security correctness cost me an explicit config I didn't know I owed, and the failure mode (CreateContainerConfigError, hiding behind a generic rollout timeout) pointed nowhere near the actual cause. That gap — between where it breaks and why it breaks — is where most of the real work in this field lives.

Why "free" is a feature, not a shortcut

It'd be easy to read the zero-cost constraint as me being cheap. It's the reverse. Free-to-run is what makes the work verifiable, and verifiable is the whole point.

Piece Cost
GitHub Actions (public repo) Free, unlimited
Local kind cluster Free (your CPU)
All tooling — Go, Terraform, Argo CD, Syft, Trivy, Kyverno Free / open source
EKS (roadmap, optional) Paid — destroy when done

Because the pipeline deploys to a throwaway cluster and fails the build unless the app actually serves traffic, a green check here doesn't mean "the code compiled." It means "the golden path works, end to end, and here's the run that proves it." Anyone can open the workflow, click into a run, and see the deploy happen — and clone it to reproduce the whole thing locally without spending a cent. That's a different kind of credibility than a screenshot.

What's deliberately not done yet

I want to be straight about scope, because the temptation with a piece like this is to pretend the toy is a platform. It isn't. This is the thin slice: one service, end to end, green. The value is that it's finished and runs, not that it's complete.

Here's what bolts on next, each a self-contained addition to a spine that already works:

Supply-chain security. Generate an SBOM with Syft and scan the image with Trivy in CI, failing the build on critical CVEs. One of the loudest DevOps themes of 2026, and it drops straight into the existing pipeline.
Policy as code. Enforce that hardening posture cluster-wide with Kyverno or OPA Gatekeeper, so the golden path is guaranteed rather than merely suggested. (Which, satisfyingly, would have caught my non-root bug at admission time with a far clearer message than CreateContainerConfigError.)
GitOps delivery. Hand deploys to Argo CD so a commit becomes the source of truth instead of an imperative kubectl apply.
Real infrastructure. A Terraform/OpenTofu module for an EKS cluster as the documented production path — spun up to capture evidence, then terraform destroy'd, because it isn't free and nobody should leave it running.

Building them one at a time, on top of something that already works, is the entire method. A finished thin slice beats a half-built everything, every single time.

Three things I'd hand to anyone building their own

Ship the thin slice first. The failure mode of ambitious platform projects isn't bad architecture — it's never finishing. Get one service deploying end to end with a single green check, then add layers. Everything I built hangs off that spine.

Make your pipeline talk before you make it work. The hours I'd have burned guessing at 0 of 2 replicas available collapsed to minutes the moment I added a diagnostics step. Instrument the failure path early. You'll lean on it constantly.

Hardening forces explicitness, and that's the job. runAsNonRoot without a runAsUser is the small, sharp, real edge between "I read a tutorial" and "I ran this and hit the wall." Secure defaults make you spell things out; learning where they make you spell things out is most of the craft.

The repo is here, green and free to run: github.com/PradeepKandepaneni/golden-path. Clone it, run make up, break it, and tell me what you'd pave next.