DEV Community: Pradeep Kandepaneni

Blast Radius First: A Trust-Tier Model for Letting AI Agents Touch Production

Pradeep Kandepaneni — Tue, 04 Aug 2026 06:20:34 +0000

How to decide what an autonomous agent is allowed to do before you decide how clever it is.
Originally published on AWS Builder Center
[https://builder.aws.com/content/3HRGyQKfP1sAjbyhm1WkJsBDca8/blast-radius-first-a-trust-tier-model-for-letting-ai-agents-touch-production]
Most conversations about AI in operations start in the wrong place. They start with the model - how good is it, how much can we trust its judgment, will it hallucinate a kubectl delete at 3 a.m. The capability question feels like the important one, so it gets all the attention.
It's the wrong question to lead with.
The question that actually determines whether AI-driven operations is safe or reckless isn't how good is the agent - it's what happens when it's wrong. Not if. When. Every automated actor, human or machine, eventually takes the wrong action against a live system. The engineers who've run production at scale know this in their bones: you don't design for the day everything works, you design for the day it doesn't. AI changes nothing about that principle and everything about its urgency, because an agent can be wrong faster, more confidently, and at higher frequency than any on-call human.
So this piece is about designing from blast radius first. Decide what an agent is allowed to touch based on what breaks when it's wrong - and only then worry about how capable it needs to be. I'll lay out a trust-tier model I've found useful for that, grounded in the same reliability discipline - SLOs, error budgets, progressive delivery - that we already trust to govern human change.
The capability trap
Here's the seductive logic that gets teams into trouble. The model is getting better every quarter. Once it's good enough, we'll let it do more. So autonomy becomes a function of trust in the model's judgment, and trust grows with benchmarks and vibes.
The problem is that model capability and blast radius are completely independent axes, and teams collapse them into one. A highly capable agent restarting a stateless pod is low-risk. A mediocre agent modifying an IAM policy is catastrophic-risk. Capability tells you how often the agent is right. Blast radius tells you how bad it is when the agent is wrong. You cannot buy down the second with the first. A 99.9%-accurate agent acting on something with an irreversible, estate-wide blast radius is still one bad call away from an incident you can't take back - and at machine frequency, 99.9% accuracy produces a wrong action surprisingly often.
This is why "let it do more once it's smarter" quietly inverts the safety model. It ties the size of the blast radius to the thing that varies most (model quality) instead of the thing you actually control (permissions). Reliability engineering solved this for humans decades ago and we forgot to apply it: we don't give a new SRE root on the billing database because they're clever. We give them scoped access that widens as the cost of their mistakes proves survivable. Same principle. Different actor.
Trust tiers: permissions scoped to reversibility
The model I use scopes agent authority to a single question asked of every possible action: how expensive is it to undo this if it's wrong? That gives four tiers.
Tier 0 - Observe
The agent can read everything and act on nothing. It surfaces correlations, drafts a diagnosis, proposes a remediation - and stops. A human executes. Blast radius: zero. This is where every agent starts and where most should live longer than teams want them to. The value here is real and underrated: a good Tier 0 agent that turns forty dashboards into one ranked hypothesis has already paid for itself, with none of the risk.
Tier 1 - Reversible action
The agent can take actions that fully self-heal or roll back with no residual state. Restart a stateless pod. Scale a deployment within pre-set bounds. Drain and cordon a node the cluster will replace. Flush a cache. The defining test: if the agent is wrong, the system returns to its prior state automatically or with a trivial, well-understood undo. Blast radius: bounded and recoverable. This is the sweet spot where autonomy earns its keep on toil.
Tier 2 - Semi-reversible action
Actions that can be undone, but at a cost - a config change that requires a redeploy to revert, a traffic shift, a scale-up that costs real money. Reversible in principle, but the undo has latency, blast radius, or a price tag. Agents operate here only with a guardrail outside the model: a policy engine that bounds the action, a canary that must stay healthy, an error-budget check that halts the action if the service is already burning budget. The agent proposes and initiates; the system, not the model's confidence, decides whether it proceeds.
Tier 3 - Irreversible / high-consequence
Deleting persistent data. Modifying IAM or security groups. Anything touching secrets, billing, or the control plane itself. Anything a human would need a second reviewer for. The agent's authority here is exactly zero, permanently, regardless of how capable it becomes. Not "not yet" - no. These actions get a human in the loop by design, because the cost of being wrong is unbounded and no accuracy figure justifies unbounded downside. The point of naming this tier explicitly is to stop the quiet drift of Tier 3 actions into Tier 2 because the demo looked impressive.
Mapping this onto real platform work makes it concrete. Automated pod restarts and bounded autoscaling sit comfortably at Tier 1 - reversible, well understood, safe to let run. Traffic shifts and configuration changes that need a redeploy to undo belong at Tier 2, gated behind canary health and an error-budget check rather than the agent's own confidence. And the things that warrant the tightest governance in any DevSecOps context - CI/CD pipeline changes, tool-access controls, IAM policy - are unambiguously Tier 3: no autonomous authority, human review by design, regardless of how good the tooling becomes.
Wiring the tiers to reliability primitives you already have
The elegant part - and the part that makes this practical rather than aspirational - is that you don't need new machinery to enforce these tiers. The primitives already exist in any mature SRE practice.
Error budgets become the throttle. A Tier 2 action shouldn't fire when the target service is already burning its error budget - that's precisely when an automated change is most likely to compound an incident. Gate agent actions on burn rate the same way you gate deploys. When the budget is healthy, the agent has room to act; when it's exhausted, autonomy contracts automatically. The error budget stops being just a release-cadence signal and becomes a real-time autonomy governor.
Progressive delivery becomes the safety envelope. Any Tier 2 agent action ships the way a risky deploy ships: to a canary first, watched against SLIs, with automated rollback if the signal degrades. The agent doesn't get a special fast path around the safeguards humans use - it gets the same envelope, which is the whole point. If your rollback automation is good enough to trust with a human's deploy, it's good enough to backstop an agent's action.
Policy-as-code becomes the tier boundary. The line between tiers can't live in a prompt - prompts are suggestions, and an agent under pressure will rationalise past a suggestion. The boundary lives in an external policy engine (OPA, admission control, IAM itself) that the agent physically cannot talk its way around. This is the single most important design commitment in the whole model: the tier boundary must be enforced by something the model cannot reason with. Guardrails the agent can argue with are decoration.
Put together, these give you an agent whose authority expands and contracts with the real-time health of the system, bounded by machinery it can't override - which is a far more honest safety story than "the model is good, trust it."
Why this beats "trust the model"
Three reasons this framing holds up where capability-first thinking doesn't.
It's auditable. "The agent can take Tier 0 and Tier 1 actions on these services, gated on error budget, with everything above requiring a human" is a sentence you can put in front of a security review, an auditor, or a nervous VP and have it mean something. "We trust the model's judgment" is not.
It's stable under model change. Swap the underlying model, upgrade it, downgrade it to save cost - the safety envelope doesn't move, because it was never a function of the model in the first place. Your risk posture doesn't silently drift every time a vendor ships a new version.
It decouples value from danger. Most of the operational value of AIOps lives at Tiers 0 and 1 - faster diagnosis, less toil, quicker recovery on reversible actions. Most of the danger lives at Tier 3. Teams that lead with capability chase the danger to get the value. Teams that lead with blast radius harvest the value and leave the danger on the table where it belongs.
Where to start on Monday
You don't roll this out as a platform program. You start by asking one question of whatever automation or agent you already have, or are about to build: what's the worst irreversible thing this could do, and what physically stops it? If the answer is "the prompt tells it not to," you have a tier boundary enforced by a suggestion, and that's the first thing to fix.
From there: inventory your candidate remediation actions and sort them into the four tiers by reversibility, not by how confident you are in the model. Enforce the Tier 2/3 line in policy-as-code, not prose. Gate Tier 2 on error budget. Keep everything at Tier 0 longer than feels necessary, and promote actions up a tier only when the tier below has proven boring. Boring is the goal. Boring is what safe automation looks like from the outside.
The agents will keep getting better. That's exactly why the discipline has to come from somewhere other than the agent. Blast radius first, capability second - and the reliability practice you already trust to govern human change is more than enough to govern the machine kind too.

This trust-tier model sits above two companion pieces I've written: pricing agent autonomy against error budgets and dollars, and building a production-safe AI remediation firewall for Amazon EKS.

The Autonomy Boundary: Pricing Your AIOps Agent in Both Error Budget and Dollars

Pradeep Kandepaneni — Wed, 29 Jul 2026 05:23:49 +0000

Autonomous SRE agents like AWS DevOps Agent can already investigate and remediate incidents — the unsolved problem is the boundary: which actions run unsupervised. A first-principles framework for drawing that line by reversibility, evidence-based confidence, and metered cost, with a reproducible lab behind every claim.

The full lab — the blast-radius router, the Terraform guardrails, the fault-injection scripts, and the capture tooling behind every figure above — is at

github.com/PradeepKandepaneni/autonomy-boundary-lab

Originally published on the AWS Builder Center:

AWS Builder Center

Connect with builders who understand your journey. Share solutions, influence AWS product development, and access useful content that accelerates your growth. Your community starts here.

builder.aws.com

Every number in this article traces to a test, a captured log, or the budget model in that repo.

The claim, up front

Autonomy is no longer a capability problem. Your agent can already investigate an incident and act on it. AWS DevOps Agent went generally available on March 31, 2026, built on Amazon Bedrock AgentCore, and it correlates telemetry, code, and deployment data to triage issues without a human driving each step. The interesting question moved somewhere else.

The binding constraint is now a decision boundary: which actions may an agent execute unsupervised, which must it escalate, and how do you encode that line so it survives change-management policy, a budget ceiling, and a 2 a.m. failure? Almost every "look, it self-healed" demo skips this. Industry data for 2026 puts it bluntly — only about one in five organizations has a mature governance model for autonomous agents. That gap, not model capability, is where production incidents and surprise bills come from.

This article gives you a first-principles framework for drawing that boundary, and it insists on a second axis most treatments ignore: the boundary is simultaneously a reliability decision and a cost decision. An autonomous investigation triggered by a flapping, false-positive alert burns error budget and agent-minutes. If you draw the line using reliability logic alone, you will leak money; if you draw it using cost logic alone, you will suppress investigations you needed. You have to price both.

First principles: why this is a decision-theory problem, not a tooling problem

Strip away the vendor framing and every autonomous remediation reduces to one comparison. For a given detected condition, you weigh the expected cost of the agent acting on its own against the expected cost of escalating to a human.

Acting alone has an expected cost of roughly P(agent is wrong) × (blast radius of a wrong action). Escalating has a cost too, and it is not zero: human response latency, on-call toil, and — the part people forget — the agent-minutes and tokens already spent investigating before the escalation. AI SRE agents make far more model calls than a chatbot because a single incident triggers planning, tool selection, evidence gathering, hypothesis revision, and reporting. That work has a meter running on it.

So the boundary is defined by three quantities, and you can reason about all three concretely:

Reversibility and blast radius of the action. Restarting a crash-looping pod is contained and reversible. Scaling a Karpenter node group within a ceiling is reversible. Modifying a production database, deleting a volume, or rolling a schema is neither.
The agent's confidence — defined as evidence completeness, not model self-report. A hypothesis backed by a specific file, line, log record, and a clean before/after correlation is high-confidence. "The model said 87%" is not evidence; agents hallucinate, and a confidently wrong root-cause analysis is the dangerous failure mode.
The economic weight of investigating. Investigation-based pricing punishes noisy alerts: an agent investigating a false positive still costs you. Alert quality is therefore a direct input to agent spend, which couples your SLO hygiene to your FinOps.

Everything below is just these three quantities, made operational.

The framework: a blast-radius router, not a self-healing switch

Do not ship an on/off "autonomy" toggle. Ship a router that classifies each candidate action into one of four dispositions, parameterised by the three quantities above.

Agent confidence (evidence completeness) Reversible + contained blast radius Irreversible or wide blast radius
High (specific evidence, clean correlation) Auto-execute, then log an immutable audit record and open a prevention item Execute-with-preview: agent proposes the exact change; human approves before it runs
Low (thin or conflicting evidence) Auto-investigate under a budget cap, gather evidence, act only if confidence crosses the threshold; otherwise escalate Escalate immediately. Do not act. Attach evidence and page a human

Two design rules make this router safe rather than decorative:

Confidence is a gate, not a suggestion. Below your threshold, the disposition can never be auto-execute regardless of blast radius. The threshold is a policy value you tune from measured calibration, not a vibe.

The router's ceilings are enforced below the agent, in infrastructure, not inside the agent's own logic. This is the single most important architectural decision in the whole design, and I will justify it next.

Deep mechanics: guardrails as code, enforced beneath the agent

An agent that polices its own limits is a single point of failure — the same LLM that might hallucinate the root cause is the thing deciding whether it is allowed to act. Put the hard limits where the agent cannot reason its way past them.

Scope IAM so the blast radius is physically bounded. The agent's execution role should be able to restart pods, cordon a node, or trigger a rollback of a specific deployment, and should have no path to delete persistent volumes, alter a production database, or touch IAM. If the router mis-classifies, the permission boundary is the backstop.

Encode budget and scale ceilings in Terraform / policy, not prose. A Karpenter provisioner gets a hard node ceiling. A cost guardrail caps agent-minutes per incident and per day. The failure this prevents is real: a misconfigured AI agent can generate in hours a bill that would take months to accumulate under ordinary provisioning, and there is at least one publicly reported near-half-billion-dollar AI-misconfiguration incident on record. The ceiling is the difference between a bounded experiment and an unbounded liability.

Encode change-freeze windows as policy. An agent that scales or patches outside an approved change window has produced a governance violation even if the action itself was correct. This is one of the named risks in the 2026 governance-gap data — autonomous actions that conflict with change-management policy — and it is trivially preventable with a deny window the agent honours.

Keep an immutable audit trail. Every auto-executed action writes an audit record the agent cannot later modify. AWS DevOps Agent produces immutable investigation timelines for exactly this reason; if you are wiring your own remediation layer, replicate the property. Auditability is what lets you widen autonomy later with evidence instead of hope.

Route the low-severity and known-noisy alerts away from auto-investigation. This is the cost-hygiene mechanism. Deduplicate first, suppress flappers, and only spend agent-minutes on signals that clear a quality bar. Then feed the noise metrics back into tightening your SLOs, closing the loop between reliability and spend.

The failure modes (this is the part your readers actually need)

Anyone can show the happy path. Credibility comes from the failure modes, and there are four worth naming.

Auto-resolution that masks the root cause. The most seductive failure. The alarm returns to OK, the graph looks clean, and the underlying defect is untouched and will recur. This is not hypothetical — AWS's own AgentCore observability walkthrough shows an incident that self-resolved in about a minute while the triggering vulnerability remained live. Mitigation: every auto-resolve must still open a prevention-backlog item. Recovery is not resolution.
The noisy-alert cost trap. Investigation-based billing means an agent chewing on false positives is a silent budget leak. Mitigation: gate investigations on alert quality and dedupe aggressively before the meter starts. Your alert hygiene is now a line item.
Change-management conflict. A technically correct scale-up during a freeze is still an incident of its own kind. Mitigation: freeze windows as enforced policy, not documentation.
Confidence miscalibration. The agent is confidently wrong. Mitigation: require concrete evidence artifacts — the specific file, line, and log record — as a precondition for auto-execute, and treat the model's self-reported confidence as untrusted input.

A note on metrics — read this before you quote anyone

The market is saturated with unverifiable numbers. You will see claims of 90%-plus alert-noise reduction, 40–60% MTTR improvement across enterprise deployments, and one vendor advertising 94% root-cause accuracy with up to 75% MTTR improvement. These are third-party and vendor-reported figures. Cite them as such or not at all, and never present them as your own results.

The only numbers you should attach your name to are ones you measured, with a stated baseline, a stated method, and a stated time window. A senior reviewer — or a senior interviewer — will ask you all three, and a borrowed figure collapses under the first follow-up. This is why the rest of this article is a lab you run, not a benchmark I assert.

Reproduce this: the autonomy-boundary-lab

Everything above is buildable at low cost. The repo is deliberately named to stand apart from generic "golden path" demos, because its contribution is the policy layer, not the platform.

What you build:

A small service on a kind cluster (or a single-node EKS if you want the DevOps Agent integration end to end), instrumented so a CloudWatch alarm fires on a defined failure.
A reproducible fault-injection script — a crash loop, a bad deploy, a dependency timeout — so the same incident is repeatable and your logs are real, not staged.
The blast-radius router as a small policy component: it reads the alarm and the agent's evidence, classifies the action into one of the four dispositions, and enforces the confidence gate.
The guardrails as code: a scoped IAM role, a Karpenter node ceiling, an agent-minute budget cap, and a change-freeze deny window, all in Terraform.
What the lab actually produced (this run)

The fault-injection script moves the app's listen port to :9999 while the readiness and liveness probes still target :5678. That is a single, locatable configuration change — the high-evidence path the router routes to auto_execute. The captured Kubernetes event stream shows the root cause in plain text:

Warning Unhealthy Liveness probe failed: dial tcp 10.244.0.7:5678: connect: connection refused
Warning Unhealthy Readiness probe failed: dial tcp 10.244.0.7:5678: connect: connection refused
Normal Killing Container app failed liveness probe, will be restarted

The failing pod's restart count climbed on a clean cadence — 1, 2, 3, 4 at roughly 30-second intervals, matching the probe's periodSeconds: 10 × failureThreshold: 3 — before Kubernetes backed it off into CrashLoopBackOff. This is the two halves of high-confidence evidence in one capture: an exact location (probes hitting a port the app no longer serves) and a clean before/after correlation (the failure begins precisely at the config change).

Equally important is what did not break. The previous ReplicaSet held its two Ready replicas throughout, so the service stayed up while the bad revision failed to roll out. The incident was reversible and contained — exactly the quadrant the router sends to auto_execute. The lab's behaviour and the router's classification agree, on the same run.

The cost dimension (modeled, not measured)

Investigation is metered, so the boundary has a price. The rate is AWS-official — $0.0083 per agent-second, i.e. $0.498 per agent-minute, per AWS's DevOps Agent pricing page, billed only for active agent time (idle is free). AWS's own example puts a typical 8-minute investigation at about $3.98. What is modeled below is how many agent-minutes a given incident consumes, since these figures come from the lab's budget.py ceilings rather than a live agent run — read them as a model of the dynamics, not a measurement of this specific fault:

A per-incident ceiling of 15 agent-minutes caps a single investigation at about $7.47 — a deliberate ceiling set above AWS's ~$3.98 typical, not a prediction of typical spend. A per-day ceiling of 240 agent-minutes caps a runaway day at about $119.52. The ceiling is the difference between a bounded experiment and an unbounded liability.
The router suppresses a known-noisy alert before the meter starts, spending $0.00 on it. An ungated agent investigating the same false positive spends up to the per-incident cap. So each suppressed false positive avoids up to $7.47 — which is why alert hygiene is a FinOps lever, not just a reliability one.

That is one measured claim (the incident timeline, reproducible from the repo) and a modeled cost claim at an AWS-official rate (clearly labelled, traceable to real code output). Both are smaller and less flashy than the vendor headlines, and that is exactly why they hold up. The measured version — real agent-minutes from a live DevOps Agent investigation of this same fault — is the subject of the follow-up.

Tradeoffs, stated plainly

When full autonomy is correct: contained, reversible, well-understood operations where the cost of a wrong action is low and recovery is trivial — pod restarts, in-ceiling scaling, rollbacks of a specific known-bad deploy. Some domains already run closed-loop remediation in production, notably patch management and certain automated security responses.

When full autonomy is negligent: irreversible or wide-blast-radius actions, anything touching the data plane, low-confidence situations, and anything inside a change freeze. Here the correct disposition is escalate-with-evidence, and "the agent was confident" is not a defence.

The multi-cloud wrinkle. The GA release of AWS DevOps Agent extended investigations beyond AWS into Azure and on-premises workloads. If your estate is hybrid, your boundary policy has to be cloud-agnostic — the router classifies by blast radius and evidence, which are portable concepts, while the guardrail enforcement is implemented per-environment. Do not hard-wire the policy to one provider's primitives.

The one-line version

The agents can act. The engineering that separates a production-grade AIOps posture from a demo is the boundary — drawn from reversibility, evidence-based confidence, and the metered cost of investigation — and enforced in infrastructure the agent cannot override. Draw that line well and the autonomy is an asset. Skip it and you have automated your way into a governance violation with a bill attached.

The full lab — the blast-radius router, the Terraform guardrails, the fault-injection scripts, and the capture tooling behind every figure above — is at github.com/PradeepKandepaneni/autonomy-boundary-lab. Every number in this article traces to a test, a captured log, or the budget model in that repo.

Building a Production-Safe AI Remediation Firewall for Amazon EKS

Pradeep Kandepaneni — Wed, 29 Jul 2026 05:10:46 +0000

A reproducible multi-AZ resilience walkthrough: spread a service across simulated zones, kill one under load, and measure the dropped requests — plus the parts that only show up in real production.

Originally published on the AWS Builder Center:

AWS Builder Center

Connect with builders who understand your journey. Share solutions, influence AWS product development, and access useful content that accelerates your growth. Your community starts here.

builder.aws.com

Repo:

PradeepKandepaneni / golden-path-resilience

golden-path-resilience

Make a service survive the loss of an entire availability zone — and prove it, on free infrastructure, with a demo that kills a zone under load and counts how many requests drop. Spoiler: it should be zero.

This is a sequel to golden-path Same tiny service; now the interesting part is the topology: how it's spread so that one zone can vanish without taking the service with it.

Runs end-to-end on a local multi-node kind cluster and in CI on GitHub's free runners. Total cost: $0.

What this proves

Three worker nodes are labelled as three simulated zones (az-a, az-b, az-c). Six replicas spread two-per-zone. Then scripts/az-outage-demo.sh:

starts a continuous stream of requests at the service,
cordons and drains one zone's node (a stand-in for that AZ going dark),
watches the evicted pods reschedule onto the surviving zones,
tallies how many…

View on GitHub

The claim, up front

First principles: why this is a decision-theory problem, not a tooling problem

So the boundary is defined by three quantities, and you can reason about all three concretely:

Everything below is just these three quantities, made operational.

The framework: a blast-radius router, not a self-healing switch

Do not ship an on/off "autonomy" toggle. Ship a router that classifies each candidate action into one of four dispositions, parameterised by the three quantities above.

High confidence (specific evidence, clean before/after correlation):

Reversible + contained blast radius → Auto-execute, then log an immutable audit record and open a prevention item.
Irreversible or wide blast radius → Execute-with-preview: the agent proposes the exact change; a human approves before it runs.

Low confidence (thin or conflicting evidence):

Reversible + contained blast radius → Auto-investigate under a budget cap; gather evidence and act only if confidence crosses the threshold, otherwise escalate.
Irreversible or wide blast radius → Escalate immediately — do not act; attach evidence and page a human.

Two design rules make this router safe rather than decorative:

Deep mechanics: guardrails as code, enforced beneath the agent

The failure modes (this is the part your readers actually need)

Anyone can show the happy path. Credibility comes from the failure modes, and there are four worth naming.

Auto-resolution that masks the root cause. The most seductive failure. The alarm returns to OK, the graph looks clean, and the underlying defect is untouched and will recur. This is not hypothetical — AWS's own AgentCore observability walkthrough shows an incident that self-resolved in about a minute while the triggering vulnerability remained live. Mitigation: every auto-resolve must still open a prevention-backlog item. Recovery is not resolution.
The noisy-alert cost trap. Investigation-based billing means an agent chewing on false positives is a silent budget leak. Mitigation: gate investigations on alert quality and dedupe aggressively before the meter starts. Your alert hygiene is now a line item.
Change-management conflict. A technically correct scale-up during a freeze is still an incident of its own kind. Mitigation: freeze windows as enforced policy, not documentation.
Confidence miscalibration. The agent is confidently wrong. Mitigation: require concrete evidence artifacts — the specific file, line, and log record — as a precondition for auto-execute, and treat the model's self-reported confidence as untrusted input.

A note on metrics — read this before you quote anyone

Reproduce this: the autonomy-boundary-lab

Everything above is buildable at low cost. The repo is deliberately named to stand apart from generic "golden path" demos, because its contribution is the policy layer, not the platform.

What you build:

The cost dimension (modeled, not measured)

Tradeoffs, stated plainly

The one-line version

A reproducible multi-AZ resilience walkthrough: spread a service across simulated zones, kill one under load, and measure the dropped requests — plus the parts that only show up in real production.

Originally published on the AWS Builder Center: https://builder.aws.com/content/3GnvKe3PBz3fP3m3pFFDFghjIxX/making-a-service-survive-an-availability-zone-outage-and-proving-it-for-dollar0
Repo:

PradeepKandepaneni / golden-path-resilience

golden-path-resilience

This is a sequel to golden-path Same tiny service; now the interesting part is the topology: how it's spread so that one zone can vanish without taking the service with it.

Runs end-to-end on a local multi-node kind cluster and in CI on GitHub's free runners. Total cost: $0.

What this proves

Three worker nodes are labelled as three simulated zones (az-a, az-b, az-c). Six replicas spread two-per-zone. Then scripts/az-outage-demo.sh:

starts a continuous stream of requests at the service,
cordons and drains one zone's node (a stand-in for that AZ going dark),
watches the evicted pods reschedule onto the surviving zones,
tallies how many…

View on GitHub

Building a Production-Safe AI Remediation Firewall for Amazon EKS

Pradeep Kandepaneni — Wed, 29 Jul 2026 03:32:51 +0000

Originally published on the AWS Builder Center:

AWS Builder Center

Connect with builders who understand your journey. Share solutions, influence AWS product development, and access useful content that accelerates your growth. Your community starts here.

builder.aws.com

AI can reduce the time required to investigate incidents, correlate telemetry, and identify probable root causes. But giving an AI system unrestricted production access creates a new reliability risk: the remediation platform itself can become the cause of the next outage.

This article presents an AI Remediation Firewall for workloads running on Amazon Elastic Kubernetes Service (Amazon EKS).

The design follows one principle:

AI proposes. Deterministic policy authorises. Controlled automation executes. Independent telemetry verifies.

The architecture uses:

AWS DevOps Agent for investigation and root-cause analysis
A structured remediation proposal as the handoff contract
AWS Step Functions for workflow control
Amazon Verified Permissions for deterministic authorisation
AWS Systems Manager Automation for controlled execution
Amazon CloudWatch Application Signals and CloudWatch Synthetics for independent verification
Kubernetes RBAC and IAM to restrict blast radius
Progressive delivery and rollback to contain failed changes

The objective is not unrestricted autonomous operations. It is bounded autonomy.

The production incident

Consider a digital commerce platform running on Amazon EKS.

The application contains:

API gateway
Checkout service
Inventory service
Payment service
Pricing service
Amazon Aurora PostgreSQL
Amazon ElastiCache for Redis
Amazon SQS
Application Load Balancer
CloudWatch Application Signals
CloudWatch Synthetics

A deployment changes the checkout configuration:

database:
  poolSize: 80
  connectionTimeoutMilliseconds: 500

retry:
  maximumAttempts: 8
  initialBackoffMilliseconds: 25

The previous configuration was:

database:
  poolSize: 30
  connectionTimeoutMilliseconds: 2000

retry:
  maximumAttempts: 3
  initialBackoffMilliseconds: 250

Within minutes:

Checkout p99 latency increases from about 220 ms to more than four seconds
HTTP 5xx errors increase
Aurora connections approach the configured maximum
SQS queue depth grows
HPA activity increases
CPU usage rises across checkout and payment services
Pods remain Running
Liveness probes continue to pass

This is a difficult incident because infrastructure health looks acceptable at a superficial level.

Restarting pods provides only temporary relief. Increasing replicas creates more database connections and makes the problem worse.

AWS DevOps Agent investigates:

CloudWatch metrics and logs
Application Signals topology
Kubernetes events
Deployment history
CloudTrail configuration changes
Aurora metrics
SQS metrics
Approved runbooks and previous incidents

It identifies the latest configuration change as the probable trigger and proposes:

Roll back the checkout service to the previous known-good revision.

That recommendation may be correct, but it is not yet authorised.

Why an AI recommendation must be treated as untrusted input

An AI-generated remediation can be technically plausible and still be unsafe because:

The root-cause analysis may be incomplete
Multiple deployments may be active
The target may have changed after the investigation started
The rollback revision may contain a security defect
The proposed blast radius may be too large
The service may already be recovering
The action may affect a critical dependency
The rollback artefact may no longer exist
The recommendation may be based on stale telemetry

The AI system must not decide:

Its own permissions
Whether its evidence is sufficient
Whether the blast radius is acceptable
Whether human approval is required
Whether the remediation succeeded

Those decisions belong outside the AI reasoning boundary.

Reference architecture

CloudWatch SLO alarm or incident alert
                 |
                 v
          AWS DevOps Agent
                 |
       Investigation and RCA
                 |
                 v
      Remediation Proposal Adapter
                 |
                 v
          Amazon EventBridge
                 |
                 v
          AWS Step Functions
                 |
       +---------+----------+
       |                    |
       v                    v
Live-context collector   Schema validator
       |                    |
       +---------+----------+
                 |
                 v
       Deterministic risk engine
                 |
                 v
    Amazon Verified Permissions
                 |
     +-----------+-----------+
     |           |           |
     v           v           v
Auto-execute  Approval      Deny
     |           |
     +-----+-----+
           |
           v
AWS Systems Manager Automation
           |
           v
Restricted EKS executor
           |
           v
Canary or bounded rollout
           |
           v
Application Signals and Synthetics
           |
     +-----+------+
     |            |
     v            v
 Promote       Roll back

The trust boundaries are:

Investigation boundary
AI gathers evidence and proposes an action.
Authorisation boundary
Deterministic logic decides whether that action is allowed.
Execution boundary
A restricted runbook performs one allowlisted operation.
Verification boundary
Independent telemetry decides whether recovery occurred.

1. Keep diagnostic access read-only

The investigation layer should be able to:

Read CloudWatch metrics and logs
Read Kubernetes events
Read deployment metadata
Read CloudTrail changes
Read service topology
Read approved incident history
Invoke narrow diagnostic tools

It should not have broad write access to:

Kubernetes resources
IAM
Security groups
Databases
Secrets
CI/CD pipelines
DNS

Expose bounded tools such as:

get_kubernetes_events(namespace, workload)
get_deployment_revision(namespace, deployment)
collect_eks_node_diagnostics(instance_id)
query_application_errors(service, start_time, end_time)
compare_configuration(current_revision, previous_revision)

Do not expose:

execute_shell(command)

A generic shell endpoint turns a controlled diagnostic integration into remote code execution.

2. Require a structured remediation proposal

Do not send free-form AI text to an executor.

Use a versioned contract:

{
  "schemaVersion": "1.0",
  "proposalId": "rp-01JZ8A17R2K5",
  "incidentId": "inc-checkout-0017",
  "createdAt": "2026-07-28T20:12:31Z",
  "expiresAt": "2026-07-28T20:17:31Z",
  "target": {
    "environment": "production",
    "cluster": "commerce-prod",
    "region": "us-east-1",
    "namespace": "checkout",
    "resourceType": "ArgoRollout",
    "resourceName": "checkout-api"
  },
  "action": {
    "type": "ROLLBACK_DEPLOYMENT",
    "parameters": {
      "targetRevision": 41
    },
    "requestedBlastRadiusPercent": 5
  },
  "evidence": [
    {
      "source": "APPLICATION_SIGNALS",
      "type": "LATENCY_INCREASE"
    },
    {
      "source": "CLOUDTRAIL",
      "type": "RECENT_CHANGE"
    },
    {
      "source": "KUBERNETES",
      "type": "CONFIGURATION_CHANGE"
    },
    {
      "source": "AURORA",
      "type": "RESOURCE_SATURATION"
    }
  ],
  "rollback": {
    "available": true,
    "knownGoodRevision": 41
  },
  "agentAssessment": {
    "confidencePercent": 92,
    "summary": "Revision 42 introduced retry amplification"
  }
}

Reject proposals containing:

Unknown actions
Wildcard targets
Free-form commands
Missing rollback data
Expired timestamps
Unregistered resources
Unknown revisions
Excessive blast radius
Insufficient evidence

The proposal contains intent, not executable commands.

Accept:

{
  "action": {
    "type": "ROLLBACK_DEPLOYMENT",
    "parameters": {
      "targetRevision": 41
    }
  }
}

Reject:

{
  "command": "kubectl delete pods --all --all-namespaces"
}

3. Separate action risk from incident urgency

A critical incident does not make a dangerous remediation safe.

Calculate two independent values.

Action risk

Action Risk =
    Action Severity       × 0.25
  + Resource Criticality  × 0.20
  + Blast Radius          × 0.20
  + Evidence Uncertainty  × 0.15
  + Change Collision      × 0.10
  + Rollback Weakness     × 0.10

Example action severity:

Action	Severity
Restart one stateless pod	10
Scale within approved limits	20
Progressive rollback	30
Modify HPA policy	45
Change database parameters	70
Modify routing	85
Modify IAM	95
Delete data	100

Example resource criticality:

Resource	Criticality
Development service	10
Internal production service	35
Customer-facing API	70
Authentication service	90
Payment platform	100

Incident urgency

Incident Urgency =
    SLO Burn Rate      × 0.40
  + Customer Impact    × 0.35
  + Incident Duration  × 0.25

Urgency may shorten approval time or escalate responders. It must not lower the action-risk threshold.

4. Do not trust AI confidence directly

The control plane should calculate validated confidence from evidence quality.

from dataclasses import dataclass
from datetime import datetime, timezone
from typing import Any

ALLOWLISTED_ACTIONS = {
    "RESTART_SINGLE_POD",
    "SCALE_WITHIN_LIMIT",
    "ROLLBACK_DEPLOYMENT",
}

@dataclass(frozen=True)
class RiskResult:
    risk_score: int
    validated_confidence: int
    route: str
    reasons: list[str]

def clamp(value: int) -> int:
    return max(0, min(100, value))

def validated_confidence(proposal: dict[str, Any]) -> int:
    evidence = proposal.get("evidence", [])
    sources = {item.get("source") for item in evidence if item.get("source")}
    types = {item.get("type") for item in evidence if item.get("type")}

    agent_confidence = clamp(
        int(proposal.get("agentAssessment", {}).get("confidencePercent", 0))
    )

    has_change = "RECENT_CHANGE" in types
    has_telemetry = bool(
        {"LATENCY_INCREASE", "ERROR_RATE_INCREASE", "RESOURCE_SATURATION"}
        & types
    )

    if len(sources) >= 3 and has_change and has_telemetry:
        cap = 95
    elif len(sources) >= 2 and has_telemetry:
        cap = 75
    else:
        cap = 50

    return min(agent_confidence, cap)

def calculate_action_risk(
    proposal: dict[str, Any],
    context: dict[str, Any],
) -> RiskResult:
    action = proposal["action"]["type"]

    if action not in ALLOWLISTED_ACTIONS:
        return RiskResult(100, 0, "DENY", ["Action is not allowlisted"])

    expires_at = datetime.fromisoformat(
        proposal["expiresAt"].replace("Z", "+00:00")
    )

    if datetime.now(timezone.utc) >= expires_at:
        return RiskResult(100, 0, "DENY", ["Proposal expired"])

    confidence = validated_confidence(proposal)
    uncertainty = 100 - confidence

    risk = round(
        clamp(context["actionSeverity"]) * 0.25
        + clamp(context["resourceCriticality"]) * 0.20
        + clamp(context["blastRadiusPercent"]) * 0.20
        + clamp(uncertainty) * 0.15
        + (100 if context["concurrentChange"] else 0) * 0.10
        + (0 if context["rollbackAvailable"] else 100) * 0.10
    )

    reasons = []

    if not context["rollbackAvailable"]:
        risk = max(risk, 70)
        reasons.append("No verified rollback point")

    if context["concurrentChange"]:
        risk = max(risk, 65)
        reasons.append("Concurrent change detected")

    if context["blastRadiusPercent"] > 25:
        risk = max(risk, 65)
        reasons.append("Blast radius exceeds 25 percent")

    if confidence < 70:
        risk = max(risk, 61)
        reasons.append("Evidence confidence below threshold")

    if risk <= 25 and confidence >= 90:
        route = "AUTO_EXECUTE"
    elif risk <= 60:
        route = "REQUIRE_APPROVAL"
    else:
        route = "DENY"

    return RiskResult(clamp(risk), confidence, route, reasons)

The weights are organisational policy, not universal values.

The critical properties are:

Inputs are bounded
Logic is deterministic
Algorithms are versioned
Decisions are reproducible
Every factor is recorded
AI cannot change thresholds
Failure defaults to denial

5. Refresh production context before authorisation

Do not authorise using only the data captured during investigation.

Immediately before execution, verify:

The incident is still active
The target revision is still deployed
No other production change is running
The previous revision still exists
The previous revision is security-compliant
The workload has not already recovered
The target resource still maps to the same owner and criticality
The SLO is still burning
The proposal has not expired

This prevents time-of-check/time-of-use failures.

A five-minute proposal expiration is a reasonable starting point for autonomous actions.

6. Authorise with Amazon Verified Permissions

Model the remediation controller as an application:

Principal: remediation orchestrator
Action: auto-execute or request-approval
Resource: production workload
Context: risk, confidence, blast radius, rollback status, action type

Evaluate AutoExecute first. If denied, evaluate RequestApproval. If both are denied, block the proposal.

Automatic execution policy:

permit (
    principal == RemediationControl::Service::"orchestrator",
    action == RemediationControl::Action::"AutoExecute",
    resource
)
when {
    context.riskScore <= 25 &&
    context.validatedConfidence >= 90 &&
    context.blastRadiusPercent <= 10 &&
    context.rollbackAvailable &&
    !context.concurrentChange &&
    (
        context.actionType == "RESTART_SINGLE_POD" ||
        context.actionType == "SCALE_WITHIN_LIMIT" ||
        context.actionType == "ROLLBACK_DEPLOYMENT"
    )
};

Approval policy:

permit (
    principal == RemediationControl::Service::"orchestrator",
    action == RemediationControl::Action::"RequestApproval",
    resource
)
when {
    context.riskScore <= 60 &&
    context.validatedConfidence >= 70 &&
    context.blastRadiusPercent <= 25 &&
    context.rollbackAvailable &&
    !context.concurrentChange
};

Explicitly forbidden actions:

forbid (
    principal,
    action,
    resource
)
when {
    context.actionType == "MODIFY_IAM" ||
    context.actionType == "DELETE_DATA" ||
    context.actionType == "DISABLE_AUDIT_LOGGING" ||
    context.actionType == "OPEN_PUBLIC_NETWORK_ACCESS"
};

The authorisation request can be submitted from Lambda:

import boto3

client = boto3.client("verifiedpermissions")

def is_authorized(
    policy_store_id: str,
    route_action: str,
    workload_id: str,
    context: dict,
) -> bool:
    response = client.is_authorized(
        policyStoreId=policy_store_id,
        principal={
            "entityType": "RemediationControl::Service",
            "entityId": "orchestrator",
        },
        action={
            "actionType": "RemediationControl::Action",
            "actionId": route_action,
        },
        resource={
            "entityType": "RemediationControl::Workload",
            "entityId": workload_id,
        },
        context={
            "contextMap": {
                "riskScore": {"long": context["riskScore"]},
                "validatedConfidence": {
                    "long": context["validatedConfidence"]
                },
                "blastRadiusPercent": {
                    "long": context["blastRadiusPercent"]
                },
                "rollbackAvailable": {
                    "boolean": context["rollbackAvailable"]
                },
                "concurrentChange": {
                    "boolean": context["concurrentChange"]
                },
                "actionType": {"string": context["actionType"]},
            }
        },
    )

    return response["decision"] == "ALLOW"

Store with every decision:

Risk-engine version
Policy-store ID
Policy version
Risk score
Validated confidence
Decision route
Determining policy IDs

7. Orchestrate with AWS Step Functions

The state machine should coordinate:

Proposal validation
Expiration check
Live-context collection
Risk calculation
Verified Permissions evaluation
Human approval where required
Runbook execution
Canary verification
Promotion or rollback
Audit completion

A simplified workflow:

{
  "StartAt": "ValidateProposal",
  "States": {
    "ValidateProposal": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Next": "CollectLiveContext"
    },
    "CollectLiveContext": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Next": "CalculateRisk"
    },
    "CalculateRisk": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Next": "AuthoriseRoute"
    },
    "AuthoriseRoute": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Next": "RouteDecision"
    },
    "RouteDecision": {
      "Type": "Choice",
      "Choices": [
        {
          "Variable": "$.decision.route",
          "StringEquals": "AUTO_EXECUTE",
          "Next": "ExecuteRunbook"
        },
        {
          "Variable": "$.decision.route",
          "StringEquals": "REQUIRE_APPROVAL",
          "Next": "ApprovalRunbook"
        }
      ],
      "Default": "Denied"
    },
    "ExecuteRunbook": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken",
      "Next": "VerifyCanary"
    },
    "ApprovalRunbook": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken",
      "Next": "VerifyCanary"
    },
    "VerifyCanary": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Next": "VerificationDecision"
    },
    "VerificationDecision": {
      "Type": "Choice",
      "Choices": [
        {
          "Variable": "$.verification.passed",
          "BooleanEquals": true,
          "Next": "Promote"
        }
      ],
      "Default": "Rollback"
    },
    "Promote": {
      "Type": "Succeed"
    },
    "Rollback": {
      "Type": "Fail"
    },
    "Denied": {
      "Type": "Succeed"
    }
  }
}

Production implementations also need:

Retries with bounded backoff
Catch handlers
Idempotency
Dead-letter handling
Timeouts
Duplicate suppression
Per-workload locking
Compensating actions
Audit-write failure handling

8. Execute only allowlisted runbooks

Map each action to a versioned Systems Manager runbook.

RUNBOOK_CATALOG = {
    "RESTART_SINGLE_POD": {
        "document": "AIOps-RestartSinglePod-v3",
        "maximumBlastRadius": 10,
    },
    "SCALE_WITHIN_LIMIT": {
        "document": "AIOps-ScaleDeployment-v4",
        "maximumBlastRadius": 20,
    },
    "ROLLBACK_DEPLOYMENT": {
        "document": "AIOps-RollbackDeployment-v7",
        "maximumBlastRadius": 25,
    },
}

The AI selects intent from the catalogue. It does not generate commands.

Example runbook:

description: Roll back an approved EKS workload
schemaVersion: '0.3'

assumeRole: '{{ AutomationAssumeRole }}'

parameters:
  AutomationAssumeRole:
    type: AWS::IAM::Role::Arn

  ExecutorFunctionName:
    type: String

  ClusterName:
    type: String

  Namespace:
    type: String

  WorkloadName:
    type: String

  TargetRevision:
    type: Integer

  TrafficPercent:
    type: Integer
    allowedValues:
      - 5
      - 10
      - 25

  ProposalId:
    type: String

mainSteps:
  - name: ExecuteRestrictedRollback
    action: aws:invokeLambdaFunction
    timeoutSeconds: 120
    maxAttempts: 1
    onFailure: Abort
    inputs:
      FunctionName: '{{ ExecutorFunctionName }}'
      InvocationType: RequestResponse
      Payload: >
        {
          "operation": "ROLLBACK_DEPLOYMENT",
          "cluster": "{{ ClusterName }}",
          "namespace": "{{ Namespace }}",
          "workload": "{{ WorkloadName }}",
          "targetRevision": {{ TargetRevision }},
          "trafficPercent": {{ TrafficPercent }},
          "proposalId": "{{ ProposalId }}"
        }

For medium-risk actions, add an aws:approve step before invoking the child runbook.

High-risk actions such as IAM changes, data deletion, network exposure, or disabling audit controls should remain outside the autonomous remediation catalogue.

9. Restrict EKS access

Use a dedicated IAM role and EKS access entry for the executor.

Do not grant cluster-admin.

Map the role to a Kubernetes group:

aiops-remediation-executor

Namespace-scoped Role:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: aiops-remediation-executor
  namespace: checkout
rules:
  - apiGroups:
      - apps
      - argoproj.io
    resources:
      - deployments
      - rollouts
    verbs:
      - get
      - list
      - watch
      - patch
      - update

  - apiGroups:
      - ""
    resources:
      - pods
    verbs:
      - get
      - list
      - watch

RoleBinding:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: aiops-remediation-executor
  namespace: checkout
subjects:
  - kind: Group
    name: aiops-remediation-executor
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: aiops-remediation-executor
  apiGroup: rbac.authorization.k8s.io

The executor must not be able to:

Read Secrets
Modify ClusterRoles
Create privileged pods
Execute commands in arbitrary containers
Modify other namespaces
Delete namespaces
Change its own IAM access
Modify EKS access entries

For stronger isolation, use separate executor roles for each action category.

10. Limit blast radius with progressive delivery

An approved action should not immediately affect every replica.

Start with:

Initial traffic: 5%
Observation window: 5 minutes
Required successful synthetic runs: 3
Maximum 5xx rate: 1%
Maximum p99 latency: 750 ms
Required business validation success: 100%

Stop promotion when:

Error rate rises
Latency exceeds the threshold
New Kubernetes errors appear
Database pressure worsens
SLO burn rate increases
Business validation fails
Another production change begins

Promote in stages:

5% → 25% → 50% → 100%

Each stage requires independent verification.

11. Verify business behaviour, not only infrastructure health

A successful rollout does not prove recovery.

Pods may be healthy and return HTTP 200 while producing an incorrect checkout total.

CloudWatch Synthetics should validate both technical and business invariants.

const https = require("https");
const log = require("SyntheticsLogger");

const API_HOST = process.env.API_HOST;

function sendRequest(options, body) {
  return new Promise((resolve, reject) => {
    const request = https.request(options, response => {
      let data = "";

      response.on("data", chunk => {
        data += chunk;
      });

      response.on("end", () => {
        resolve({
          statusCode: response.statusCode,
          body: data
        });
      });
    });

    request.on("error", reject);
    request.write(body);
    request.end();
  });
}

exports.handler = async () => {
  const body = JSON.stringify({
    customerId: "synthetic-customer",
    items: [
      {
        sku: "SYNTHETIC-SKU-001",
        quantity: 2
      }
    ]
  });

  const options = {
    hostname: API_HOST,
    method: "POST",
    path: "/api/v1/checkout/quote",
    headers: {
      "Content-Type": "application/json",
      "Content-Length": Buffer.byteLength(body),
      "X-Synthetic-Transaction": "true"
    },
    timeout: 5000
  };

  const started = Date.now();
  const response = await sendRequest(options, body);
  const duration = Date.now() - started;

  if (response.statusCode !== 200) {
    throw new Error(`Unexpected HTTP ${response.statusCode}`);
  }

  if (duration > 750) {
    throw new Error(`Latency exceeded threshold: ${duration} ms`);
  }

  const payload = JSON.parse(response.body);

  const subtotal = payload.lines.reduce(
    (total, line) => total + line.unitPrice * line.quantity,
    0
  );

  const expected = subtotal + payload.tax + payload.shipping;

  if (Math.abs(payload.total - expected) > 0.01) {
    throw new Error(
      `Business invariant failed. Expected ${expected}, received ${payload.total}`
    );
  }

  if (payload.currency !== "USD") {
    throw new Error(`Unexpected currency: ${payload.currency}`);
  }

  log.info(`Synthetic checkout passed in ${duration} ms`);

  return {
    passed: true,
    durationMilliseconds: duration
  };
};

Synthetic transactions must not create real payments, shipments, inventory consumption, or customer notifications.

Use:

Reserved synthetic identities
Test inventory
Payment-provider test mode
Cleanup workflows
Explicit synthetic headers
Analytics filters

12. Verification contract

The verification layer should query authoritative telemetry directly.

{
  "proposalId": "rp-01JZ8A17R2K5",
  "observationWindowSeconds": 300,
  "requirements": {
    "minimumSuccessfulSyntheticRuns": 3,
    "maximumErrorRatePercent": 1,
    "maximumP99LatencyMilliseconds": 750,
    "maximumDatabaseConnectionUtilisationPercent": 75,
    "maximumSloBurnRate": 2,
    "minimumHealthyReplicaPercent": 100
  }
}

Example result:

{
  "passed": true,
  "observations": {
    "successfulSyntheticRuns": 5,
    "errorRatePercent": 0.4,
    "p99LatencyMilliseconds": 420,
    "databaseConnectionUtilisationPercent": 61,
    "sloBurnRate": 0.8,
    "healthyReplicaPercent": 100
  }
}

Promotion requires every mandatory condition to pass.

Do not average away a critical business failure.

End-to-end execution

Step 1: Detect

Application Signals detects an elevated SLO burn rate and triggers an investigation.

Step 2: Investigate

AWS DevOps Agent correlates telemetry, configuration history, deployment metadata, and service dependencies.

Step 3: Propose

The proposal adapter creates:

Action: ROLLBACK_DEPLOYMENT
Target revision: 41
Initial traffic: 5%
Agent confidence: 92%
Rollback available: Yes

Step 4: Refresh context

The control plane confirms:

Revision 42 is still active
Revision 41 exists
No concurrent deployment is running
The incident is still active
The workload matches the inventory record

Step 5: Calculate risk

Factor	Value
Action severity	30
Resource criticality	70
Blast radius	5
Evidence uncertainty	8
Change collision	0
Rollback weakness	0

30 × 0.25 = 7.50
70 × 0.20 = 14.00
 5 × 0.20 = 1.00
 8 × 0.15 = 1.20
--------------------
Risk score = 23.70, rounded to 24

Validated confidence is 92%.

Step 6: Authorise

Verified Permissions allows automatic execution because:

Risk is at or below 25
Confidence is at least 90
Blast radius is at or below 10%
Rollback exists
No concurrent change is active
The action is allowlisted

Step 7: Execute

Systems Manager invokes the restricted rollback executor.

The executor can modify only:

commerce-prod
└── checkout namespace
    └── checkout-api

Step 8: Verify

Five percent of traffic moves to revision 41.

CloudWatch Synthetics validates the checkout path. Application Signals validates latency, errors, and SLO burn rate. Database and queue metrics confirm recovery.

Step 9: Promote or roll back

If every gate passes, traffic moves progressively to 100%.

If any gate fails, the workflow restores the prior state and marks the remediation unsuccessful.

Step 10: Prevent recurrence

The incident generates preventive work:

Add retry budgets
Enforce exponential backoff
Validate connection-pool settings
Add configuration contract tests
Add load tests
Add deployment policies
Improve SLO coverage

A platform that repeatedly fixes the same incident without creating preventive engineering work is only automating toil.

Failure scenarios to test

1. Retry amplification

Inject excessive retries, minimal backoff, and aggressive database pool settings.

Expected result:

AI correlates deployment and database saturation
Scaling is not selected as the first response
Rollback is proposed
Connection pressure decreases after remediation

2. Silent network failure

Inject an invalid route or packet-filtering rule.

Expected result:

Pods may remain running
Service-to-service calls fail
Node diagnostics provide evidence
The system does not restart every workload blindly

3. HPA thrashing

Configure an aggressive target with insufficient stabilisation.

Expected result:

Oscillation is detected
Unbounded scaling is denied
HPA changes require approval
Verification uses a longer window

4. Partial secret rotation

Rotate credentials for only part of the workload.

Expected result:

Pod-level error distribution reveals the split
Secret changes remain outside autonomous execution
A predefined security runbook is used

5. Phantom success

Return HTTP 200 with an incorrect checkout total.

Expected result:

Infrastructure checks pass
Business synthetic fails
Promotion stops
Automatic rollback begins

6. Concurrent deployment

Start another deployment after the proposal is generated.

Expected result:

Context refresh detects the collision
Automatic execution is denied
The proposal becomes stale

7. Missing rollback artefact

Delete the previous image or revision.

Expected result:

Rollback weakness increases
Risk exceeds the threshold
The proposal is denied

Security controls

IAM separation

Use separate roles for:

Investigation
Proposal ingestion
Context collection
Risk evaluation
Policy evaluation
Orchestration
Runbook execution
EKS modification
Verification
Audit writing

Do not allow one role to modify the proposal, policy, executor, and audit record.

Immutable evidence

Store decisions and evidence in encrypted Amazon S3.

Consider:

Versioning
Object Lock where required
AWS KMS customer-managed keys
Restricted deletion
CloudTrail data events
Payload checksums

Idempotency

Create an idempotency key:

incidentId + target + actionType + targetRevision

If an identical action is:

Running: reject the duplicate
Successful: return the previous result
Failed: require a new proposal
Rolled back: block automatic retry

Concurrency locking

Use DynamoDB conditional writes to lock a workload:

LOCK#commerce-prod#checkout#checkout-api

Only one remediation workflow should modify a workload at a time.

Input integrity

Hash the proposal before authorisation.

Before execution, verify:

hash(authorised proposal) == hash(execution proposal)

Network controls

Where possible:

Use private subnets
Use a private EKS endpoint
Use VPC endpoints
Restrict outbound access
Restrict security-group egress
Avoid general internet access for the executor

Secrets

The proposal must not contain:

Passwords
Tokens
Private keys
Database credentials
Customer data

Pass references, not values.

Observability for the remediation platform

The remediation control plane needs its own SLOs.

Recommended metrics:

RemediationProposalReceived
RemediationProposalRejected
RemediationProposalExpired
RemediationRiskScore
RemediationAutoExecuted
RemediationApprovalRequested
RemediationDenied
RemediationExecutionSucceeded
RemediationExecutionFailed
RemediationVerificationPassed
RemediationVerificationFailed
RemediationRollbackSucceeded
RemediationRollbackFailed
RemediationDuplicateSuppressed
RemediationConcurrentChangeDetected

Recommended dimensions:

Environment
Cluster
Namespace
Service
ActionType
RiskBand
DecisionRoute
RunbookVersion
PolicyVersion
Outcome

Do not use proposal IDs as metric dimensions. Store them in logs.

Alarm on:

Executor failure
Rollback failure
Workflow timeout
Policy-evaluation failure
Approval timeout
Audit-write failure
Unexpected increases in automatic remediation
Repeated remediation of the same workload

When the control plane fails internally, deny or pause remediation. Do not bypass the control.

Metrics that prove value

Do not publish invented performance claims.

Measure:

Metric	Definition
Mean time to detect	Incident start to alert
Mean time to investigate	Alert to probable root cause
Mean time to authorise	Proposal to policy decision
Mean time to remediate	Authorised action to recovery
Mean time to verify	Execution to verified recovery
Unsafe-action prevention rate	Unsafe proposals blocked
Automatic-remediation success rate	Auto actions passing verification
False-remediation rate	Executed actions based on incorrect RCA
Rollback success rate	Failed remediations reversed
Change-collision prevention rate	Conflicting changes blocked
Cost per investigation	AI, telemetry, workflow, and execution cost
Repeat-incident rate	Recurrence of the same failure mode

Compare:

Manual incident response
AI-assisted investigation with manual execution
AI-assisted investigation with risk-gated execution

The objective is:

Reduce recovery time without increasing the rate or blast radius of unsafe production changes.

Limitations

This pattern does not eliminate risk.

Root-cause analysis may be incomplete
Telemetry may be missing
Deterministic policies can still be wrong
Runbooks can contain defects
Human approval can delay recovery
Cross-account approval may require additional workflow design
Business validation may be difficult to automate safely
Break-glass actions still require human ownership

Do not expand autonomous permissions during an outage. Maintain a documented and audited break-glass process.

Production-readiness checklist

Investigation

Diagnostic access is read-only
MCP tools are narrowly scoped
Generic shell access is prohibited
Evidence sources are recorded
Output uses a versioned schema

Authorisation

Actions are allowlisted
Risk and urgency are separated
Live context is refreshed
Proposals expire
Default is deny
Forbidden actions have explicit deny policies
Policy changes use CI/CD
AI cannot change its own permissions

Execution

Runbooks are versioned
Parameters are validated
Free-form commands are rejected
IAM uses least privilege
Kubernetes RBAC is namespace-scoped
Secrets are inaccessible
Blast radius is bounded
Actions are idempotent
Concurrent changes are locked

Verification

Verification is independent
Infrastructure and business checks are included
Multiple successful observations are required
Failure triggers rollback
Rollback is verified
Success requires evidence

Audit

Proposal is immutable
Risk factors are stored
Policy and runbook versions are stored
Approvals are recorded
Execution logs are retained
Verification results are retained
Denied actions are searchable

Conclusion

AI can accelerate incident investigation, but investigation and production authority are different capabilities.

A system that proposes a change, authorises it, executes it, and judges its own success has collapsed too many trust boundaries.

The safer design separates responsibilities:

AWS DevOps Agent investigates
A structured proposal captures intent
A deterministic risk engine evaluates danger
Amazon Verified Permissions authorises
AWS Step Functions coordinates
Systems Manager executes allowlisted runbooks
IAM and Kubernetes RBAC contain access
Progressive delivery limits blast radius
Application Signals and Synthetics verify recovery

The result is not uncontrolled autonomy. It is bounded autonomy:

AI for investigation
Policy for authority
Runbooks for execution
SLOs and business tests for truth
Humans for ambiguity and high-risk decisions

AI proposes. Deterministic policy authorises. Controlled automation executes. Independent telemetry verifies.

That is the difference between an impressive AIOps demo and a production-safe reliability platform.

Disclaimer

The architecture, code samples, scoring weights, thresholds, IAM boundaries, Cedar policies, Kubernetes permissions, approval rules, and verification criteria in this article are reference examples.

Adapt and test them against your workload characteristics, security policies, compliance requirements, incident processes, service quotas, recovery objectives, and operational maturity.

Do not enable automatic production remediation until the actions have passed controlled failure injection, rollback testing, security review, and operational readiness exercises.

Making a Service Survive an Availability Zone Outage — and Proving It for $0

Pradeep Kandepaneni — Tue, 21 Jul 2026 08:01:47 +0000

A reproducible multi-AZ resilience walkthrough: spread a service across simulated zones, kill one under load, and measure the dropped requests — plus the parts that only show up in real production.

Originally published on the AWS Builder Center: https://builder.aws.com/content/3GnvKe3PBz3fP3m3pFFDFghjIxX/making-a-service-survive-an-availability-zone-outage-and-proving-it-for-dollar0
Repo: https://github.com/PradeepKandepaneni/golden-path-resilience

Preventing Unsafe Regional Failover in Amazon EKS with Policy-Gated Recovery

Pradeep Kandepaneni — Tue, 21 Jul 2026 07:30:42 +0000

This article was originally published on AWS Builder Center.

A secondary Amazon EKS cluster can appear healthy while still being unsafe to receive production traffic. This article presents a policy-gated recovery model that validates data replication, compute capacity, application health, container images, secrets, certificates, and synthetic business transactions before promoting a recovery Region.

This article was originally published on [AWS Builder Center]

https://builder.aws.com/content/3GnlrIlqWy7Hel5zlqaIMsg2rPM/preventing-unsafe-regional-failover-in-amazon-eks-with-policy-gated-recovery

Why a Healthy Standby Region Can Still Be Unsafe

Running an application across multiple Availability Zones protects against many infrastructure failures, but it does not provide complete protection from a regional outage. A common recovery design is to maintain a second Amazon Elastic Kubernetes Service (Amazon EKS) cluster in another AWS Region and redirect traffic when the primary Region becomes unavailable.

The problem is that a reachable cluster is not necessarily ready to operate as the active production Region.

Kubernetes readiness probes may succeed while the database replica is behind, required container images are missing, regional secrets contain incorrect endpoints, certificates are unavailable, or the recovery cluster lacks enough compute capacity. A DNS or load-balancer health check cannot detect all of these conditions.

For stateful workloads, premature promotion can be more damaging than a temporary outage. It can introduce data loss, conflicting writes, authentication failures, partial application availability, or a recovery environment that collapses as soon as production traffic arrives.

A safer approach is to treat regional promotion as a controlled engineering decision rather than a simple routing change.

This article presents a policy-gated recovery model for Amazon EKS. The model uses mandatory safety gates and a weighted recovery-readiness score to validate infrastructure capacity, application health, data replication, container images, secrets, certificates, GitOps state, writer fencing, and synthetic business transactions before traffic and write authority move to the recovery Region.

Reference Architecture

The architecture uses two AWS Regions: a primary Region serving production traffic and a warm-standby recovery Region maintained at reduced capacity.

Each Region contains an independent Amazon EKS cluster, regional networking, application load balancing, monitoring, secrets, certificates, and container images. Application configuration is synchronized through a GitOps workflow so that the recovery cluster continuously receives approved Kubernetes manifests and deployment changes.

Stateful data is replicated using Amazon Aurora Global Database. Container images are copied through Amazon Elastic Container Registry (Amazon ECR) cross-Region replication, while AWS Secrets Manager provides regional secret replicas.

These replication mechanisms must be validated independently. Successful replication configuration does not prove that every required image, secret value, certificate, endpoint, or database change is available and usable in the recovery Region.

Amazon Application Recovery Controller (ARC) Region switch coordinates the recovery workflow. Region switch uses recovery plans composed of ordered or parallel execution steps. The recovery plan can be initiated manually or through approved automation, depending on the workload's operational requirements.

A separate recovery-readiness evaluator checks mandatory safety conditions before database promotion, workload scaling, or traffic movement is permitted.

The design separates four concerns:

Infrastructure readiness — The recovery Region has sufficient networking, compute, quota, and worker-node capacity.
Application readiness — Critical workloads are synchronized, schedulable, healthy, and able to complete synthetic business transactions.
Data readiness — Replication lag is within the approved recovery-point objective, and conflicting writes are prevented.
Dependency readiness — Required images, secrets, certificates, identity services, queues, caches, and external integrations are available.

A regional endpoint health check alone cannot prove these conditions. Promotion is allowed only after every mandatory safety gate passes.

Why Endpoint Health Is Not Recovery Readiness

Traditional failover designs often rely on a small number of signals:

Load balancer endpoint availability
DNS health-check status
Kubernetes pod readiness
Application response codes
Infrastructure resource availability

These signals are valuable, but they answer only narrow questions.

A successful readiness probe confirms that a container can respond to a configured health endpoint. It does not prove that the application can authenticate users, perform database writes, retrieve secrets, process queue messages, contact third-party services, or sustain production traffic.

Consider the following failure conditions:

Application pods are healthy, but the Aurora secondary cluster has replication lag beyond the accepted recovery-point objective.
Kubernetes Deployments exist, but worker nodes cannot scale because of service quotas or insufficient subnet address capacity.
The image tag exists in the recovery Region, but the required immutable image digest does not.
Secrets were replicated, but a connection string still references a primary-Region endpoint.
The recovery cluster is synchronized, but the GitOps controller reverses emergency replica changes.
TLS certificates are missing, expired, or not attached to the recovery load balancer.
The database is promoted, but the original writer continues accepting requests, creating a split-brain risk.
The login endpoint responds, but a complete business transaction fails.

Regional recovery readiness must therefore be evaluated as a combined system property, not as a single health-check result.

Separate Hard Gates from Readiness Scores

The proposed model uses two types of controls:

Hard promotion gates
Weighted readiness signals

This separation is critical.

A weighted score is useful for summarizing operational health, but it must never compensate for a failed data-safety requirement. A recovery Region with excellent application health but unsafe database state should not receive production writes.

Hard Promotion Gates

Hard gates represent conditions that are non-negotiable. If any hard gate fails, promotion stops.

Database Replication

Replication status must be healthy.
Replication lag must remain below the approved threshold.
The target database cluster must be eligible for failover or switchover.
Database engine versions must be compatible with the selected recovery operation.

Writer Fencing

The previous writer must be unavailable, isolated, or explicitly fenced before the recovery Region accepts writes.
Applications must not maintain writable connections to both Regions.
Connection pools and DNS caches must be considered during the transition.

Amazon EKS Capacity

The required Kubernetes resources must exist in the recovery cluster.
Sufficient worker-node capacity must be available or capable of scaling.
Subnets, IP addresses, instance quotas, and Availability Zone distribution must support the desired replica count.
Critical pods must become schedulable within the recovery timeout.

Application Health

Required Deployments must have the minimum number of ready replicas.
Pod disruption budgets, topology constraints, and health probes must be valid.
Critical services must pass internal and external health validation.

Container Image Availability

Required repositories must exist in the recovery Region.
The exact image digests referenced by production manifests must be available.
Image pull permissions and node access must be validated.

Amazon ECR replication rules should be configured before images are pushed. Images that existed before replication was configured should not be assumed to appear automatically in the destination Region.

Secret and Configuration Readiness

Required secrets must exist in the recovery Region.
Secret versions must match the expected deployment version.
Regional endpoints embedded in secret values must be validated.
AWS Key Management Service keys and permissions must be available in the recovery Region.

Certificate Readiness

Certificates must be valid and available in the required Region.
Load balancer listeners must reference the correct certificate.
Certificate expiration and domain validation status must be checked.

GitOps Synchronization

Critical applications must be synchronized.
No unresolved high-severity drift should exist.
GitOps tools must not reverse recovery-time scaling changes.

Synthetic Business Transactions

A synthetic test should validate more than a basic HTTP response. Depending on the application, it can perform steps such as:

Authenticate a test user.
Create a temporary transaction.
Read the newly created record.
Update the record.
Delete or clean up the test data.
Verify that logs and metrics were produced.

ARC Recovery-Plan Evaluation

Before execution, the Region switch plan should pass its available plan-evaluation checks. Custom policy gates should complement—not replace—the validations performed by ARC.

Weighted Recovery-Readiness Score

After all hard gates pass, weighted signals can provide a consolidated operational score.

A reference weighting model could use:

25% application health
20% infrastructure capacity
20% data health
15% dependency health
10% observability health
10% policy compliance

The weights must reflect the workload's actual risk profile. A transaction-processing system may assign greater weight to data health, while a stateless content-delivery workload may emphasize capacity and dependency availability.

The score should not be interpreted as a universal standard. It is an operational decision aid.

A simple decision policy is:

IF any hard gate fails:
    BLOCK PROMOTION

ELSE IF readiness score is below 90:
    REQUIRE OPERATOR REVIEW

ELSE:
    ALLOW THE APPROVED RECOVERY WORKFLOW

A score of 95 must never override a failed writer-fencing gate, excessive database replication lag, or a missing production image.

Example Recovery Policy

The following example represents a simplified policy definition. Production implementations should store thresholds in version-controlled configuration and require peer review for changes.

recoveryPolicy:
  targetRegion: us-west-2

  hardGates:
    maximumDatabaseLagSeconds: 300
    minimumAvailableReplicasPercent: 95

    requiredSyntheticTests:
      - login
      - create-order
      - read-order
      - update-order

    requireDatabasePromotionEligibility: true
    requireImageDigestValidation: true
    requireRegionalSecretValidation: true
    requireCertificateValidation: true
    requireGitOpsSynchronization: true
    requireWriterFencing: true
    requireArcPlanEvaluation: true

  readinessScore:
    minimumAutomaticApprovalScore: 90

    weights:
      applicationHealth: 25
      infrastructureCapacity: 20
      dataHealth: 20
      dependencyHealth: 15
      observabilityHealth: 10
      policyCompliance: 10

The threshold values above are examples. They must be aligned with the organization's recovery-time objective, recovery-point objective, application behavior, data-loss tolerance, and operational approval model.

Example Readiness Evaluation

The following Python example demonstrates the decision logic. It intentionally keeps hard-gate evaluation separate from weighted scoring.

from dataclasses import dataclass
from typing import Iterable


@dataclass(frozen=True)
class HardGate:
    name: str
    passed: bool
    details: str = ""


@dataclass(frozen=True)
class WeightedSignal:
    name: str
    value: float
    weight: float


def evaluate_recovery_readiness(
    hard_gates: Iterable[HardGate],
    weighted_signals: Iterable[WeightedSignal],
    minimum_score: float = 90.0,
) -> dict:
    failures = [
        {
            "name": gate.name,
            "details": gate.details,
        }
        for gate in hard_gates
        if not gate.passed
    ]

    if failures:
        return {
            "decision": "BLOCK",
            "score": None,
            "failed_gates": failures,
        }

    signals = list(weighted_signals)

    if not signals:
        return {
            "decision": "BLOCK",
            "score": None,
            "failed_gates": [
                {
                    "name": "weighted-signals",
                    "details": "No weighted readiness signals were provided.",
                }
            ],
        }

    total_weight = sum(signal.weight for signal in signals)

    if total_weight <= 0:
        raise ValueError("Total signal weight must be greater than zero.")

    normalized_score = sum(
        signal.value * signal.weight
        for signal in signals
    ) / total_weight

    decision = (
        "ALLOW"
        if normalized_score >= minimum_score
        else "REVIEW"
    )

    return {
        "decision": decision,
        "score": round(normalized_score, 2),
        "failed_gates": [],
    }

In a production implementation, each signal should include:

Collection timestamp
Source system
Region
Resource identifier
Evaluation result
Threshold
Evidence location
Expiration period

Stale evidence should fail closed. A database-lag measurement collected 30 minutes earlier should not approve a current regional promotion.

Recovery Workflow with AWS ARC Region Switch

A human-approved automated workflow is usually more defensible than fully automatic promotion for business-critical stateful workloads.

A reference recovery sequence is:

An operator declares a regional recovery event or initiates a controlled exercise.
ARC Region switch evaluates the configured recovery plan.
The recovery-readiness evaluator collects current evidence from both Regions.
Hard promotion gates are evaluated.
The recovery EKS cluster scales critical Kubernetes resources.
Worker-node autoscaling increases node capacity as required.
The GitOps controller verifies application synchronization.
Aurora Global Database performs the appropriate failover operation.
Applications reconnect to the new database writer.
Synthetic transactions run against the recovery Region.
Writer fencing is confirmed.
ARC routing controls or the approved traffic-management mechanism moves traffic.
Post-promotion monitoring verifies error rate, latency, saturation, and business transactions.
The workflow completes, pauses, or rolls back based on validation results.

ARC Region switch can scale supported EKS workload resources as part of a recovery plan. However, increasing Kubernetes replica counts does not independently create worker-node capacity. The cluster still depends on a node-provisioning mechanism such as Karpenter, Cluster Autoscaler, or EKS Auto Mode.

Recovery testing must therefore validate both layers:

Desired Kubernetes replica capacity
Actual schedulable worker-node capacity

Prevent GitOps from Reversing Recovery Changes

GitOps systems continuously reconcile actual cluster state with the state stored in Git. This is normally desirable, but it can conflict with disaster-recovery actions.

For example, ARC may increase a Deployment from three replicas to twelve replicas during recovery. If the Git repository still specifies three replicas, the GitOps controller may attempt to restore the lower count.

The same conflict can occur when a recovery workflow modifies Horizontal Pod Autoscaler behavior to prevent immediate scale-down.

Recovery design should explicitly define which system owns temporary scaling changes.

Options include:

Configure GitOps to ignore the Deployment replica field.
Configure GitOps to ignore specific Horizontal Pod Autoscaler fields.
Commit recovery-state changes to a dedicated Git branch.
Temporarily suspend reconciliation for selected resources.
Use environment-specific overlays that represent active and standby capacity.

The selected approach must be tested. "GitOps is configured" is not sufficient evidence that GitOps and the recovery workflow will cooperate.

Kubernetes Resilience Controls

The recovery Region should enforce the same—or stricter—workload-resilience controls used in the primary Region.

The following example spreads application replicas across Availability Zones:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout-api
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: checkout-api
  template:
    metadata:
      labels:
        app: checkout-api
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: checkout-api

      containers:
        - name: checkout-api
          image: ACCOUNT_ID.dkr.ecr.REGION.amazonaws.com/checkout-api@sha256:IMAGE_DIGEST

          resources:
            requests:
              cpu: "500m"
              memory: "512Mi"
            limits:
              cpu: "1"
              memory: "1Gi"

          readinessProbe:
            httpGet:
              path: /health/ready
              port: 8080
            initialDelaySeconds: 10
            periodSeconds: 5

          livenessProbe:
            httpGet:
              path: /health/live
              port: 8080
            initialDelaySeconds: 30
            periodSeconds: 10

Using immutable image digests reduces ambiguity during recovery. The promotion gate can verify that the exact digest referenced by the workload exists in the recovery Region.

Database Promotion and Write Safety

Aurora Global Database supports planned switchovers and disaster-recovery failovers. These operations should not be treated as equivalent.

A planned switchover is used when the current primary and secondary Regions are healthy and the organization intentionally moves the primary role. An unplanned failover is used when the primary Region is impaired or unavailable.

The recovery workflow must account for:

Current replication lag
Engine-version compatibility
Promotion eligibility
Application reconnection behavior
DNS caching
Connection-pool expiration
Transaction retries
Potential data loss during unplanned failure
Reintroduction of the previous primary as a secondary
Failback sequencing

Applications should use retry logic that distinguishes transient connectivity failure from permanent transaction rejection. Idempotency controls are especially important for payment, order, provisioning, and messaging workflows.

Writer fencing must be validated before the recovery Region accepts writes. This can involve isolating the previous writer, revoking application access, disabling routing, applying security controls, or otherwise proving that dual-writer operation cannot occur.

Container Image Validation

Amazon ECR supports cross-Region and cross-account private-image replication. However, replication configuration is not equivalent to complete image readiness.

The recovery workflow should verify:

Destination repository existence
Expected image digest
Multi-architecture manifest availability
Image-pull permissions
Encryption configuration
Vulnerability-scanning status, where required
Node access to the destination repository

Images pushed before a replication rule was configured should not be assumed to exist in the destination Region. Validate the exact digests required by active Kubernetes manifests.

A simple validation process can:

Extract all image references from production workloads.
Resolve tags to immutable digests.
Query the destination ECR registry.
Compare expected and available digests.
Block promotion when any required digest is missing.

Secret and Certificate Validation

AWS Secrets Manager can replicate encrypted secret data and metadata across Regions. Replication is useful, but it does not understand the semantic meaning of the values stored inside a secret.

A replicated secret may still contain:

A primary-Region database endpoint
A Region-specific queue URL
A private service hostname unavailable from the recovery Region
A certificate identifier valid only in the primary Region
An external allowlist that excludes recovery egress addresses

The recovery gate must validate both existence and correctness.

A secret-readiness test should verify:

Expected secret version
Recovery-Region encryption key
Resource policy
Application IAM access
Regional endpoint values
Rotation status
Successful application retrieval

Certificates should also be validated by Region, expiration date, domain name, listener association, and deployment status.

Observability During Recovery

Recovery without observability is guesswork.

The recovery Region must provide independent visibility into:

Kubernetes replica availability
Unschedulable pods
Worker-node provisioning
Application error rate
Request latency
Database replication lag
Database connection failures
Queue depth and consumer lag
Cache availability
Synthetic transaction success
DNS or routing state
ARC recovery-plan execution
GitOps synchronization
Certificate errors
Secret-access failures

Do not make the recovery workflow dependent solely on a monitoring system hosted in the Region being deactivated.

Critical dashboards, alarms, logs, and operational evidence should remain available during a regional impairment.

Post-promotion validation should continue for a defined stabilization period. Passing a single synthetic transaction immediately after traffic movement does not prove sustained recovery.

Recovery-Time Budget

A recovery-time objective should be decomposed into measurable stages.

The following is a worked reference target—not a production benchmark:

Recovery stage	Target
Event declaration and operator approval	3 minutes
ARC plan evaluation and policy checks	4 minutes
EKS workload and node-capacity scaling	8 minutes
Database promotion and application reconnection	6 minutes
Synthetic transaction validation	4 minutes
Traffic movement and stabilization	5 minutes
Total target recovery budget	30 minutes

Each stage should produce timestamps and evidence.

A recovery exercise should record:

Start and completion time
Duration of each execution step
Manual interventions
Failed gates
Overrides
Database lag
Replica scaling time
Node-provisioning time
Synthetic-test results
Routing-change time
Stabilization results
Corrective actions

With a small number of exercises, publish every result along with the minimum, median, and maximum. Avoid statistically weak percentile claims from insufficient samples.

Failback Is Part of the Design

Failover is only half of disaster recovery.

After the original Region becomes healthy, the organization must decide whether and when to return production traffic. Immediate failback can introduce additional risk while teams are still stabilizing the environment.

A controlled failback process should:

Confirm that the original Region is stable.
Rebuild or rejoin the original database cluster as required.
Re-establish replication in the intended direction.
Validate that no divergent writes exist.
Synchronize GitOps state.
Validate images, secrets, certificates, queues, caches, and dependencies.
Restore sufficient EKS and worker-node capacity.
Run synthetic business transactions.
Fence the current writer before changing write authority.
Perform a planned database switchover when supported.
Move application traffic.
Monitor the restored primary Region through a stabilization period.
Return the other Region to the intended standby capacity.

Failback should use the same policy discipline as failover. "The original Region is back online" is not sufficient evidence that it is safe to become primary again.

Security Considerations

A recovery platform has permission to alter critical infrastructure and routing. Its security model deserves the same scrutiny as a production deployment system.

Recommended controls include:

Dedicated IAM roles for recovery-plan execution
Least-privilege EKS access entries
Namespace-scoped Kubernetes permissions where practical
Separate permissions for evaluation and execution
Multi-person approval for high-risk recovery actions
AWS CloudTrail logging
Immutable audit records
Encryption for data and secrets in both Regions
Restricted emergency-access procedures
Periodic credential and role review
Protection against unauthorized policy-threshold changes
Signed or reviewed recovery configuration
Separation of duties between application, platform, and security teams

Manual override capability may be necessary, but overrides should require explicit authorization, justification, timestamped evidence, and post-event review.

Data-safety gates such as writer fencing should not support routine bypass.

Limitations and Trade-Offs

This architecture introduces operational discipline, but it does not eliminate complexity.

Cost

A warm-standby environment costs more than a pilot-light design because a functional copy of the platform remains active at reduced capacity.

Recovery Speed

A pilot-light architecture can reduce cost but generally requires more provisioning and scaling during recovery.

DNS Behavior

DNS-based traffic movement is affected by resolver and client caching. Configured TTL values do not guarantee that every client immediately honors the new destination.

Application Architecture

Applications with hard-coded regional dependencies, non-idempotent workflows, or tightly coupled external services may require significant redesign.

Data Consistency

Asynchronous cross-Region replication can introduce data-loss exposure during an unplanned failure. The organization must define an acceptable recovery-point objective.

Automation Risk

Fully automatic failover may be inappropriate for stateful systems when failure detection is uncertain or write safety cannot be proven automatically.

Service Availability

AWS service features, quotas, instance types, and database-engine support vary by Region and version. Validate the target Regions before implementation.

Scoring Limitations

A readiness score is a decision aid. It is not a substitute for hard safety gates, engineering review, or repeated recovery exercises.

Recommended Adoption Approach

Do not begin by applying this model to every production workload.

Start with one representative application that has:

Clear recovery objectives
Defined business transactions
Known dependencies
Testable data-replication behavior
Manageable compliance constraints
An engaged application owner

A practical adoption sequence is:

Inventory all regional dependencies.
Define recovery-time and recovery-point objectives.
Classify hard promotion gates.
Build the recovery Region with infrastructure as code.
Configure application and data replication.
Implement evidence collection.
Implement synthetic transactions.
Create the ARC Region switch plan.
Integrate policy evaluation.
Run recovery exercises without traffic movement.
Run controlled failover and failback exercises.
Correct gaps before expanding to additional applications.

The goal is not to automate every step immediately. The goal is to make every decision explicit, measurable, repeatable, and auditable.

Conclusion

A secondary Amazon EKS cluster should not be promoted merely because its API endpoint and pods are reachable.

Safe regional recovery requires evidence that compute capacity, data, application state, identity, container images, certificates, dependencies, routing, and business transactions are ready together.

Hard promotion gates protect non-negotiable safety conditions. A weighted readiness score gives operators a consolidated view of the remaining operational signals. Amazon Application Recovery Controller Region switch provides an orchestration layer for executing the recovery workflow, while GitOps, infrastructure as code, replication services, synthetic testing, and observability provide the supporting controls.

The result is not zero-risk failover. No architecture can honestly promise that.

The objective is a recovery process in which unsafe conditions block promotion, operational decisions are supported by current evidence, and every recovery exercise improves the next one.

Disclaimer

The architecture, code examples, thresholds, and recovery-time budget in this article are provided as a reference scenario. They must be adapted and validated against each organization's workload characteristics, recovery objectives, security requirements, compliance obligations, service quotas, AWS Region availability, and operational procedures.

This article represents the author's technical perspective. It is not an AWS-supported solution, an official architecture from AWS, or a guarantee of availability, recovery time, or data protection.

References

I Built a Platform Engineering Golden Path That Runs for $0 — Every Decision, and Everything That Broke

Pradeep Kandepaneni — Tue, 21 Jul 2026 05:33:22 +0000

Originally published on the AWS Builder Center: https://builder.aws.com/content/3GnbxJrdbgQ5sXNJTRXvHIB5OaV/i-built-a-platform-engineering-golden-path-that-runs-for-dollar0-every-decision-and-everything-that-broke
Repo: https://github.com/PradeepKandepaneni/golden-path

Most "platform engineering" articles show you a diagram. A tidy box labeled Internal Developer Platform, some arrows pointing at happy little developer icons, a paragraph about reducing cognitive load. Then the article ends, and you're holding a concept you can't run, can't fork, and can't check.

I wanted the opposite. The smallest thing a developer could actually use — push code, get back a tested, hardened, deployed workload, no tickets, no clicking — and I wanted it to cost nothing to run, so anyone reading this could clone it and watch it work instead of trusting me. No AWS bill to reproduce it. No "works on my cluster, promise."

So that's what I built, and this is the honest write-up. The repo is public and green on every push, so none of what follows requires taking my word for it: github.com/PradeepKandepaneni/golden-path. Every design decision, why I threw out the alternatives, and — the part actually worth your time — the one bug that took the whole thing down and what it taught me about hardened containers.

What a "golden path" actually is

The term gets thrown around, so let me pin it down. A golden path is the paved road a platform team hands its developers: one opinionated, well-lit route from "I have code" to "it's running," with the boring-but-critical parts already handled. Tests. Immutable image tags. Health checks. A pod that doesn't run as root. The developer doesn't wire those up per service, and doesn't get to forget them either. They're in the road.

Platform engineering, stripped of the conference gloss, is mostly the work of building and maintaining those roads so product teams stop rebuilding CI pipelines and Kubernetes manifests from scratch every time they ship something. Pave the road once; every team that drives on it inherits consistency, security, and speed for free. That's the whole pitch.

The trap — and I nearly walked straight into it — is believing you need a giant portal to deliver any of that. You don't. You need one road that works.

Decision 1: A template repo, not Backstage

The flashy way to demo an Internal Developer Platform is Backstage. It's the name everyone knows, it has a slick UI, and "built an IDP with Backstage" looks great on a repo.

I skipped it, and I'd tell you to skip it too, at least at the start. Backstage is a full-stack app you have to run, host, and babysit. For a solo project meant to show a pattern, that's weeks of wrestling with the portal before you've paved a single road — and a half-finished portal is worse than nothing, because it advertises that you started something big and quit.

A template repository plus GitHub Actions gets you the same idea — one automated path that scaffolds, tests, scans, and deploys a service — with zero portal code. It's also honestly closer to what a lot of real teams run day to day. Golden paths don't need a portal. They need an opinionated, automated pipeline. That's the thing I built, and it's a thing I could finish.

If I had to compress the lesson to four words: ship what you can finish.

Decision 2: kind in CI, not a cloud cluster

Here's the constraint that shaped everything else: the project had to run for free, in CI and on my laptop.

That killed running a real EKS cluster as the default. EKS is the right answer in production — it's on the roadmap as the documented production path — but the control plane alone runs about $0.10 an hour before you add a single node, load balancer, or NAT gateway. Leave one up while you iterate and it quietly turns into a $150–300/month surprise. Worse than the money: nobody can cheaply reproduce your work, which defeats the entire point of a demo repo.

So the default cluster is kind — Kubernetes running inside Docker. Locally, make up creates a cluster, builds the image, loads it, deploys, and smoke-tests it. In CI, the pipeline spins a kind cluster up inside the GitHub Actions runner, deploys to it, checks it, and throws it away when the job ends. My real run created the cluster and had the control plane Ready in 19 seconds:

Ready after 19s 💚 Set kubectl context to "kind-golden-path"

The economics are the quiet superpower. On a public repo, Actions minutes are unlimited and free, and the cluster lives and dies inside that free runner. The entire pipeline — a live Kubernetes deploy plus a real HTTP smoke test — costs nothing. Every push. Forever.

yaml

name: Create kind cluster uses: helm/kind-action@v1 with: cluster_name: golden-path config: kind/kind-config.yaml

That's the whole "provision Kubernetes" step, and it runs on GitHub's dime.

Decision 3: The image tag is the git SHA. Never :latest

Small, and non-negotiable. The image is tagged with the short git SHA of the commit that built it, and the manifest is pinned to that exact tag before the deploy.

yaml

name: Compute image tag from git SHA
id: vars
run: echo "sha=${GITHUB_SHA::7}" >> "$GITHUB_OUTPUT"
name: Build image
run: |
docker build \
-t golden-path:${{ steps.vars.outputs.sha }} \
--build-arg VERSION=${{ steps.vars.outputs.sha }} .

:latest is a lie. It's a mutable label pointing at different bytes over time, so "what's running" and "what I think is running" drift apart the second someone pushes again. A content-addressable tag means a running pod can tell you exactly which commit it came from — and this service does, because I bake the SHA into the binary at build time with an ldflags override. You'll see it pay off at the end of the war story below, in the actual smoke-test output.

Tiny discipline. Enormous payoff the first time you're staring at a production incident at 2 a.m. wondering what's actually deployed.

Decision 4: Distroless and non-root, by default

A golden path only earns its name if good practice is the default, not a thing you remember to bolt on. So the service runs on a distroless base — no shell, no package manager, almost nothing to pivot into — and the pod is hardened right in the manifest:

yaml
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]

Non-root. No privilege escalation. Read-only root filesystem. Every Linux capability dropped. It's the posture you'd eventually want a cluster-wide policy engine to enforce (Kyverno or OPA, on the roadmap), but here it starts as the baked-in default. The distroless choice also just produces a tiny image — the final artifact came out at 10.4MB, which the pipeline confirmed after loading it onto the node:

docker.io/library/golden-path fbab3d2 ae676b1d9427 10.4MB

And this decision is exactly where the project bit me. Which is the part I actually want to talk about.

The bug: CreateContainerConfigError

I pushed the first real version, opened the Actions tab, and watched it fail. Not at build. Not at cluster creation. At the deploy:

Waiting for deployment "golden-path" rollout to finish: 0 of 2 updated replicas are available...
error: timed out waiting for the condition
Error: Process completed with exit code 1

Zero of two replicas. A three-minute hang, then a red X.

Here's the first trap, and it's worth internalizing: that error is useless. "Timed out waiting for the condition" tells you the rollout didn't finish. It says nothing about why. The pods could be failing to pull an image, crashing on boot, flunking a health check, any of a dozen things. kubectl rollout status flattens all of them into the same timeout.

My honest first guess was an image problem — the tag hadn't matched, the load had failed, something like that. Wrong, as it turned out. But I couldn't even tell I was wrong, because the pipeline wasn't telling me anything.

So my first change wasn't a fix. It was making the pipeline talk. I added a diagnostics step that only fires when the deploy fails and dumps everything about the broken pods:

yaml

name: Diagnostics (only if the deploy failed) if: failure() run: | kubectl get pods -o wide kubectl describe pods -l app=golden-path kubectl get events --sort-by=.lastTimestamp | tail -40 kubectl logs -l app=golden-path --all-containers --tail=100 || true

Pushed again. Same timeout — but this time the log told the truth right underneath it:

NAME READY STATUS RESTARTS AGE
golden-path-6445746597-2czz8 0/1 CreateContainerConfigError 0 3m
golden-path-6445746597-svdct 0/1 CreateContainerConfigError 0 3m

CreateContainerConfigError. Not an image problem at all — the image had loaded fine (so much for my first guess). Not a crash — the container never even started. Kubernetes was refusing to create it from its config.

The cause is a genuinely sharp edge. I'd set runAsNonRoot: true, which tells the kubelet to refuse to start a container that would run as UID 0. To enforce that at create time, the kubelet needs to know the numeric user ID the container will run as. My distroless image runs as a non-root user — but it doesn't declare a numeric UID the kubelet can read at that moment. No number to check against, so the kubelet can't prove the container is non-root, and rather than risk running something as root, it refuses. Config error. Pods parked. Rollout hangs. Generic timeout.

One line fixes it: pin the UID so the check has a number. The distroless nonroot variant runs as UID 65532:

yaml
securityContext:
runAsNonRoot: true
runAsUser: 65532 # <- the fix
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]

Pushed. Green. And this time the rollout log shows the recovery happening in real time — two replicas coming up over about eight seconds:

Waiting for deployment "golden-path" rollout to finish: 0 of 2 updated replicas are available...
Waiting for deployment "golden-path" rollout to finish: 1 of 2 updated replicas are available...
deployment "golden-path" successfully rolled out

Then the in-cluster smoke test hit the service and got a real response back:

{"message":"hello from the golden path","version":"fbab3d2"}

Look at that version field: fbab3d2. That's the git SHA from Decision 3, baked into the binary, reported by the running pod. The immutable-tag discipline isn't theory in that line — the workload is telling me exactly which commit it came from.

Why give a whole section to one missing line of YAML? Because it's the shape of nearly every hardening bug you'll meet. Doing the secure thing — refusing to run as root — is what caused the failure. Had I left the container running as root like a lazy default, it would've started instantly, and I'd have shipped something worse. Security correctness cost me an explicit config I didn't know I owed, and the failure mode (CreateContainerConfigError, hiding behind a generic rollout timeout) pointed nowhere near the actual cause. That gap — between where it breaks and why it breaks — is where most of the real work in this field lives.

Why "free" is a feature, not a shortcut

It'd be easy to read the zero-cost constraint as me being cheap. It's the reverse. Free-to-run is what makes the work verifiable, and verifiable is the whole point.

Piece Cost
GitHub Actions (public repo) Free, unlimited
Local kind cluster Free (your CPU)
All tooling — Go, Terraform, Argo CD, Syft, Trivy, Kyverno Free / open source
EKS (roadmap, optional) Paid — destroy when done

Because the pipeline deploys to a throwaway cluster and fails the build unless the app actually serves traffic, a green check here doesn't mean "the code compiled." It means "the golden path works, end to end, and here's the run that proves it." Anyone can open the workflow, click into a run, and see the deploy happen — and clone it to reproduce the whole thing locally without spending a cent. That's a different kind of credibility than a screenshot.

What's deliberately not done yet

I want to be straight about scope, because the temptation with a piece like this is to pretend the toy is a platform. It isn't. This is the thin slice: one service, end to end, green. The value is that it's finished and runs, not that it's complete.

Here's what bolts on next, each a self-contained addition to a spine that already works:

Supply-chain security. Generate an SBOM with Syft and scan the image with Trivy in CI, failing the build on critical CVEs. One of the loudest DevOps themes of 2026, and it drops straight into the existing pipeline.
Policy as code. Enforce that hardening posture cluster-wide with Kyverno or OPA Gatekeeper, so the golden path is guaranteed rather than merely suggested. (Which, satisfyingly, would have caught my non-root bug at admission time with a far clearer message than CreateContainerConfigError.)
GitOps delivery. Hand deploys to Argo CD so a commit becomes the source of truth instead of an imperative kubectl apply.
Real infrastructure. A Terraform/OpenTofu module for an EKS cluster as the documented production path — spun up to capture evidence, then terraform destroy'd, because it isn't free and nobody should leave it running.

Building them one at a time, on top of something that already works, is the entire method. A finished thin slice beats a half-built everything, every single time.

Three things I'd hand to anyone building their own

Ship the thin slice first. The failure mode of ambitious platform projects isn't bad architecture — it's never finishing. Get one service deploying end to end with a single green check, then add layers. Everything I built hangs off that spine.

Make your pipeline talk before you make it work. The hours I'd have burned guessing at 0 of 2 replicas available collapsed to minutes the moment I added a diagnostics step. Instrument the failure path early. You'll lean on it constantly.

Hardening forces explicitness, and that's the job. runAsNonRoot without a runAsUser is the small, sharp, real edge between "I read a tutorial" and "I ran this and hit the wall." Secure defaults make you spell things out; learning where they make you spell things out is most of the craft.

The repo is here, green and free to run: github.com/PradeepKandepaneni/golden-path. Clone it, run make up, break it, and tell me what you'd pave next.