Amazon's March 2026 AI outages — two separate incidents within three days, totaling more than 6 million lost orders — have done something unusual for the SRE community: they've made a failure mode visible that most teams have been quietly carrying in their production systems without acknowledging.
The incidents were traced to AI-generated code changes deployed without adequate approval gates. Amazon's response was a 90-day code safety reset across 335 critical systems, with a new requirement that AI-assisted code changes be reviewed by senior engineers before deployment.
That response is SRE discipline. Applied reactively. This article is about applying it proactively — and about a compounding failure mode most teams haven't modeled yet.
The Double-Exposure Problem
The SRE concept of blast radius asks: when a component fails, what is the maximum scope of impact? Most blast radius models assume that the failing component is one thing — a service, a database, a network partition.
In 2026 production environments, a new blast radius scenario is emerging that most models don't account for:
What happens when your AI agent and the AI-generated code it runs on fail simultaneously?
This is the double-exposure problem. It has three components:
Exposure 1 — AI runtime behavior. Your AI agent operates non-deterministically. Its decisions, tool selections, and reasoning paths vary across invocations. Standard observability — latency, error rate, availability — does not instrument this layer. The semantic failure modes (wrong decisions, context drift, tool compensation) are invisible to your dashboards.
Exposure 2 — AI-generated code changes. Your CI/CD pipeline uses AI assistance to generate infrastructure changes, configuration updates, or application code. According to Lightrun's 2026 survey of 200 senior SRE and DevOps leaders, 43% of these changes require manual debugging in production even after passing QA. Not a single survey respondent expressed "very confidence" that AI-generated code would behave correctly in production.
Exposure 3 — The interaction.** When an AI-generated code change deploys to the same environment your agent is operating in, you have two non-deterministic systems interacting. The code change may alter the agent's tool environment, context window, or available action space in ways that manifest as behavioral drift — drift that your current instrumentation will miss because it's measuring infrastructure, not agent behavior.
The result: a production incident that looks like agent degradation. The root cause is a code change. The RCA takes hours because the investigation starts at the wrong layer.
Why Standard Observability Misses This
IEEE Spectrum described this failure class in their recent article on quiet AI failures: every monitoring dashboard reads healthy while users report that system decisions are becoming wrong.
This is structurally identical to what happens in the double-exposure scenario. A code change that subtly alters an agent's tool environment produces no infrastructure-layer signal. The agent's HTTP responses stay at 200. Latency stays within SLO. Error budget stays unburned.
What changes is the agent's Decision Quality Rate — the percentage of decisions falling within expected behavioral bounds. And Tool Invocation Efficiency — the ratio of tool calls per task completion. And eventually Human Escalation Rate — the percentage of tasks requiring intervention.
None of these are instrumented in a standard observability stack. All of them detect the double-exposure failure mode before it reaches user impact.
The Governance Framework
Amazon's 90-day reset is a retroactive version of what proactive SRE governance looks like. Here are the four components that matter, drawn from first principles rather than post-incident response:
1. The AI Code Change Approval Gate
Every code change touching an AI agent's runtime environment — its tools, configuration, action space, or infrastructure — should require explicit approval before deployment. Not because AI code generation is untrustworthy, but because non-deterministic code changes interacting with non-deterministic runtime systems have a compounding failure surface that standard CI/CD testing cannot fully cover.
This is not a new concept. Amazon has now required it. The cost of implementing it proactively is hours. The cost of discovering it's missing is incidents.
Implementation: A dedicated approval stage in your deployment pipeline for changes flagged as AI-generated or agent-environment-adjacent. This is distinct from your standard peer review — it specifically evaluates: does this change touch any agent's tool environment, context configuration, or action space?
2. Behavioral Baseline Snapshots Around Code Deployments
Apply the same framework version governance pattern to AI code changes: snapshot your agent's behavioral baselines before the change deploys, and compare post-deployment behavior against them.
Specifically, capture per-task-class TIE and DQR baselines immediately before any deployment that touches your agent's environment. Run the deployment in a shadow environment for a minimum review period. If TIE drifts more than 15% or DQR drops more than 15%, flag for human review before promoting to production.
This is the instrumentation that would have surfaced Amazon's failure earlier in the pipeline — not at the infrastructure layer, but at the behavioral layer where the actual impact manifested.
from agentsre import AgentSLICollector, TaskRecord
from agentsre.sprawl import FrameworkVersionGovernance
# Capture baseline before deployment
gov = FrameworkVersionGovernance(
tie_drift_threshold=1.15,
dqr_drift_threshold=0.85,
min_shadow_samples=30,
)
gov.snapshot_baseline(
agent_id="your-agent",
task_class="your-task-class",
framework_version="pre-ai-code-change",
tie_values=current_tie_samples,
dqr_values=current_dqr_samples,
)
# After shadow deployment — evaluate before promoting
result = gov.evaluate_upgrade(
agent_id="your-agent",
task_class="your-task-class",
production_version="pre-ai-code-change",
shadow_version="post-ai-code-change",
)
if result.decision == UpgradeDecision.BLOCK:
block_deployment(result.block_reason)
3. A Blast Radius Model for Double-Exposure
Most blast radius models assume one failing component. Run the double-exposure calculation explicitly:
- Which of your production services depend on AI agents?
- Which code paths in those services are AI-generated?
- If both the agent's semantic behavior and the underlying code fail simultaneously, what is the maximum scope of user impact?
- What is the safe degradation sequence — which agent capabilities can you reduce autonomously, and in what order?
This calculation should exist as a named document, owned by a named person, reviewed quarterly. It is the blast radius equivalent of a fire drill — done in advance so the answer is known before the incident.
4. A Proactive Runbook — Not Amazon's Retroactive Reset
Amazon's 90-day reset is a retroactive runbook. Write yours proactively. A minimum viable AI code reliability runbook covers:
- Detection: Which metrics signal that an AI code change has degraded agent behavior? (Answer: TIE drift, DQR drop, HER increase — not latency or error rate)
- Attribution: How do you determine whether the degradation is agent behavior, code change, or model drift? (Answer: compare against behavioral baseline snapshots captured pre-deployment)
- Containment: What is the fastest path to reverting the code change while maintaining partial agent operation? (Answer: the progressive autonomy constraint ladder — not a binary kill switch)
- Recovery criteria: When is it safe to redeploy? (Answer: shadow behavioral baselines within ±15% of production baseline for 30 consecutive minutes)
The SRE Perspective on AI Code Generation
The Lightrun finding that 88% of SRE leaders need two to three redeploy cycles to verify an AI-generated fix suggests something straightforward: the testing and verification frameworks for AI-generated code have not kept pace with the adoption of AI code generation.
This is the same lag that produced Amazon's incidents. And it's the same lag that the SRE community has closed before — with microservices, with Kubernetes, with cloud-native architectures. Each time, capability arrived before governance. The SRE discipline developed the governance.
The governance for AI-generated code in agent environments exists. Error budgets, blast radius models, approval gates, behavioral baseline comparison — these are standard SRE tools. They need to be applied to a new layer of the stack.
The open-source implementation is at github.com/Ajay150313/agentsre. The FrameworkVersionGovernance module handles behavioral baseline capture and comparison. The progressive constraint ladder handles safe degradation. Both work for AI code change governance as directly as they do for framework upgrades.
Amazon spent 6.3 million lost orders learning this lesson. Most teams can learn it for the cost of an afternoon.
What To Do This Week
If you're running AI agents in production and using AI-assisted code generation in the same environment:
Today: Identify which code changes in your last 30 days touched your agent's tool environment, configuration, or action space. Determine whether any were AI-generated. If yes — were they reviewed specifically for agent-environment impact?
This week: Add an AI code change flag to your deployment pipeline. Start capturing TIE and DQR baselines around any deployment flagged as agent-environment-adjacent.
This month: Run the double-exposure blast radius calculation. Document the result. Assign an owner. Review it with your team.
The Amazon incidents happened in March. The Lightrun survey data was collected in January. IEEE Spectrum is calling quiet failure one of the defining challenges of the year.
The signal is clear. The governance frameworks exist.
Open-source implementation: github.com/Ajay150313/agentsre
LinkedIn discussion: https://www.linkedin.com/posts/ajay-devineni_sre-agenticai-reliability-activity-7458330530212835328-36__?utm_source=share&utm_medium=member_desktop&rcm=ACoAACIp55QBRGVmAcEbf0D-1PaR5vEbm2yMcJU
What does your current approval gate for AI-generated code look like? Or is this the first time you've run the double-exposure calculation?
Top comments (0)