This is post #8 of the OWASP Agentic AI Top 10: What Builders on AWS Need to Know series.
A search tool starts returning errors. 2% failure rate. Nothing dramatic.
Your agent retries. Makes sense. But the agent's retry logic lives inside a probabilistic reasoning engine that doesn't understand backpressure. It retries immediately. And again. And again. Each retry burns tokens, adds latency, and puts more load on the already-struggling search service.
Meanwhile, three other agents are doing the same thing. Against the same endpoint. At the same time.
Within minutes, that 2% tool error rate has become a 20% agent failure rate. The search service is now completely overwhelmed by your own agents' retry storm. Downstream services that depend on search results start timing out. The cascade spreads.
You started with a minor blip. You ended with a system-wide outage. And nobody shipped a bug.
Welcome to Cascading Failures.
What Are Cascading Failures in Agentic Systems?
I wrote about AI as a single point of failure recently. Cascading failures in agentic systems are that same risk, amplified by autonomy.
ASI08 describes what happens when a single fault propagates across autonomous agents, compounding into system-wide harm. The initial trigger can be anything: a hallucination, a poisoned memory entry, a tool failure, a corrupted message from another agent.
What makes this different from regular distributed systems failures?
Agents don't understand backpressure. A microservice hitting a 503 backs off. An agent hitting a failure asks its LLM "what should I try next?" and the LLM says "I don't care, try again" or worse, "try a different approach" which hammers a different service.
Agents persist state. A corrupted decision doesn't just produce one bad response. It gets saved to memory, influences the next planning cycle, and propagates to every agent that reads from shared state.
Agents delegate autonomously. One compromised planner can emit unsafe steps to an executor, which runs them without validation, which triggers more agents, which... you get it.
The OWASP document puts it bluntly: there's a "potential discrepancy in the speed and scale of fault propagation in a multi-agent system and the ability of humans to keep up with them."
Translation: cascades happen faster than humans can intervene.
How Cascading Failures Actually Happen
1. The Retry Storm
This is the most common pattern. And it's devastating.
A detailed analysis from early 2026 showed the math: if each of 10 sequential tool calls has a 2% error rate, you get an 18.3% per-session failure rate before retries even kick in. Add three retry layers (SDK, gateway, agent loop) and that 2% blip becomes a full-blown siege on your weakest dependency.
Real incident: In June 2026, a Microsoft Copilot outage was caused by a token exchange service rejecting valid credentials. The resulting cascade of retry storms overloaded the remaining functional nodes until engineers manually rolled back.
Real incident: GitHub's 30x infrastructure crisis. By February 2026, agentic CI/CD workflows had pushed GitHub to 90 million PRs merged per month and 1.4 billion commits. A study of 208,000 CI/CD runs found that agent PRs fail more often, and each failure triggers more retries, creating a platform-wide feedback loop. GitHub's CTO announced they needed to redesign for 30x capacity, up from the 10x plan they'd started just months earlier. Agent retry storms were literally breaking one of the world's largest development platforms.
2. Feedback Loop Amplification
Two agents rely on each other's outputs. Agent A generates a report. Agent B validates it. Agent A uses B's validation to refine the report. B validates again. Each cycle amplifies any initial error.
OWASP documents a specific pattern: an auto-remediation agent suppresses alerts to meet latency SLAs. A planning agent interprets fewer alerts as "success" and widens automation scope. The blind spots compound across regions.
3. Corrupted Shared State Propagation
One agent writes bad data to a shared memory store (DynamoDB, Redis, a vector database). Every other agent that reads from that store now operates on corrupted context. And because each agent might write its own derived conclusions back, the corruption multiplies.
OWASP scenario: A financial Market Analysis agent gets poisoned via prompt injection. It inflates risk limits. The Position agent auto-trades larger positions. The Execution agent places orders. Compliance stays blind because everything is "within parameters." The cascade runs at machine speed through the entire trading chain.
Real incident: In April 2026, a Cursor AI agent running Claude Opus 4.6 deleted PocketOS's entire production database in 9 seconds. The agent was assigned to fix a database connection issue. Instead of diagnosing it, the agent decided to delete and recreate the volume. Because Railway stores backups inside the same volume, a single API call wiped everything. The cascade: one bad decision → production data gone → backups gone → company operations halted. Nine seconds from "minor fix" to total data loss.
4. Auto-Deployment Cascade
An orchestrator agent pushes a faulty update. It propagates automatically to all connected agents. Every agent in the network is now running corrupted logic. And because they're all making decisions based on the same bad update, the errors correlate.
Mitigating Cascading Failures on AWS
The good news: distributed systems engineering solved most of these patterns decades ago. The challenge is applying those patterns to systems where the "retry logic" lives inside an LLM.
1. Step Functions: Error Handling That Actually Works
AWS Step Functions gives you the circuit breakers and blast-radius controls that agent frameworks typically lack. Here's a state machine with proper error handling:
{
"Comment": "Agent tool call guarded by a DynamoDB-backed circuit breaker",
"StartAt": "CheckBreaker",
"States": {
"CheckBreaker": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:123456789012:function:circuit-breaker-check",
"Parameters": { "tool_name.$": "$.tool_name" },
"ResultPath": "$.breaker",
"Retry": [
{ "ErrorEquals": ["States.TaskFailed"], "IntervalSeconds": 1, "MaxAttempts": 2, "BackoffRate": 2.0 }
],
"Next": "BreakerDecision"
},
"BreakerDecision": {
"Type": "Choice",
"Choices": [
{ "Variable": "$.breaker.decision", "StringEquals": "short_circuit", "Next": "GracefulDegradation" }
],
"Default": "AgentToolCall"
},
"AgentToolCall": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:123456789012:function:agent-tool-executor",
"Parameters": { "tool_name.$": "$.tool_name", "args.$": "$.args" },
"TimeoutSeconds": 30,
"Retry": [
{ "ErrorEquals": ["ToolTransientError"], "IntervalSeconds": 5, "MaxAttempts": 2, "BackoffRate": 3.0 }
],
"Catch": [
{ "ErrorEquals": ["States.ALL"], "ResultPath": "$.error", "Next": "RecordFailure" }
],
"ResultPath": "$.tool_result",
"Next": "RecordSuccess"
},
"RecordSuccess": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:123456789012:function:circuit-breaker-record",
"Parameters": { "tool_name.$": "$.tool_name", "outcome": "success" },
"ResultPath": "$.breaker_update",
"Retry": [
{ "ErrorEquals": ["States.TaskFailed"], "IntervalSeconds": 1, "MaxAttempts": 2, "BackoffRate": 2.0 }
],
"Next": "Succeeded"
},
"RecordFailure": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:123456789012:function:circuit-breaker-record",
"Parameters": { "tool_name.$": "$.tool_name", "outcome": "failure" },
"ResultPath": "$.breaker_update",
"Retry": [
{ "ErrorEquals": ["States.TaskFailed"], "IntervalSeconds": 1, "MaxAttempts": 2, "BackoffRate": 2.0 }
],
"Next": "GracefulDegradation"
},
"GracefulDegradation": {
"Type": "Pass",
"Result": {
"status": "degraded",
"message": "Tool unavailable - returning cached/default response"
},
"ResultPath": "$.tool_result",
"Next": "Succeeded"
},
"Succeeded": { "Type": "Succeed" }
}
}
The circuit breaker (state lives in DynamoDB, not in $.failureCount)
This is what makes it a real breaker: the state survives across executions, so once it trips OPEN it short-circuits everyone until the cooldown, then lets exactly one trial through
(HALF_OPEN) to test recovery.
import os
import time
import boto3
from botocore.exceptions import ClientError
ddb = boto3.resource("dynamodb")
TABLE = ddb.Table(os.environ["BREAKER_TABLE"])
FAILURE_THRESHOLD = int(os.environ.get("FAILURE_THRESHOLD", "5"))
OPEN_SECONDS = int(os.environ.get("OPEN_SECONDS", "30")) # cooldown + half-open staleness window
def _conditional(update):
"""Run a conditional update. Return True if it applied, False if the condition failed."""
try:
update()
return True
except ClientError as e:
if e.response["Error"]["Code"] == "ConditionalCheckFailedException":
return False
raise
def check_handler(event, _ctx):
"""Called BEFORE the tool runs. Decides whether to allow the call or short-circuit."""
tool = event["tool_name"]
now = int(time.time())
item = TABLE.get_item(Key={"tool_name": tool}).get("Item") or {}
state = item.get("state", "CLOSED")
if state == "CLOSED":
return {"decision": "allow", "state": "CLOSED"}
if state == "OPEN":
opened_at = int(item.get("opened_at", 0))
if now - opened_at < OPEN_SECONDS:
return {"decision": "short_circuit", "state": "OPEN"}
# Cooldown elapsed: exactly one caller wins the single trial slot (OPEN -> HALF_OPEN).
won = _conditional(lambda: TABLE.update_item(
Key={"tool_name": tool},
UpdateExpression="SET #s = :half, half_open_at = :now",
ConditionExpression="#s = :open AND opened_at = :oa",
ExpressionAttributeNames={"#s": "state"},
ExpressionAttributeValues={":half": "HALF_OPEN", ":open": "OPEN", ":oa": opened_at, ":now": now},
))
return {"decision": "allow", "state": "HALF_OPEN"} if won \
else {"decision": "short_circuit", "state": "OPEN"}
# HALF_OPEN: a trial is in flight, short-circuit everyone else until it reports back,
# unless the trial went stale (crashed without recording) -> grab a fresh trial slot.
half_open_at = int(item.get("half_open_at", now))
if now - half_open_at < OPEN_SECONDS:
return {"decision": "short_circuit", "state": "HALF_OPEN"}
won = _conditional(lambda: TABLE.update_item(
Key={"tool_name": tool},
UpdateExpression="SET half_open_at = :now",
ConditionExpression="half_open_at = :prev",
ExpressionAttributeValues={":now": now, ":prev": half_open_at},
))
return {"decision": "allow" if won else "short_circuit", "state": "HALF_OPEN"}
def record_handler(event, _ctx):
"""Called AFTER the tool runs (or fails). Updates the breaker state."""
tool = event["tool_name"]
now = int(time.time())
if event["outcome"] == "success":
# Success closes the breaker and clears the counter.
TABLE.update_item(
Key={"tool_name": tool},
UpdateExpression="SET #s = :closed, failure_count = :zero REMOVE opened_at, half_open_at",
ExpressionAttributeNames={"#s": "state"},
ExpressionAttributeValues={":closed": "CLOSED", ":zero": 0},
)
return {"state": "CLOSED", "failure_count": 0}
# Failure: atomically bump the counter.
resp = TABLE.update_item(
Key={"tool_name": tool},
UpdateExpression="SET failure_count = if_not_exists(failure_count, :zero) + :one",
ExpressionAttributeValues={":zero": 0, ":one": 1},
ReturnValues="UPDATED_NEW",
)
failures = int(resp["Attributes"]["failure_count"])
# Trip open at the threshold (or when a HALF_OPEN trial fails). Only stamp opened_at on the
# transition INTO open, so repeated failures don't keep resetting the cooldown.
if failures >= FAILURE_THRESHOLD:
_conditional(lambda: TABLE.update_item(
Key={"tool_name": tool},
UpdateExpression="SET #s = :open, opened_at = :now REMOVE half_open_at",
ConditionExpression="attribute_not_exists(#s) OR #s <> :open",
ExpressionAttributeNames={"#s": "state"},
ExpressionAttributeValues={":open": "OPEN", ":now": now},
))
return {"state": "OPEN", "failure_count": failures}
return {"state": "CLOSED", "failure_count": failures}
The table and IAM (don't skip this)
The breaker needs one DynamoDB table, keyed by tool:
aws dynamodb create-table \
--table-name agent-circuit-breakers \
--attribute-definitions AttributeName=tool_name,AttributeType=S \
--key-schema AttributeName=tool_name,KeyType=HASH \
--billing-mode PAY_PER_REQUEST
Both breaker Lambdas need dynamodb:GetItem and dynamodb:UpdateItem on that table ARN, and the state machine's execution role needs lambda:InvokeFunction on the three functions. The atomic UpdateItem with conditional writes is doing the real work here — it's what makes the breaker correct when hundreds of executions hit it concurrently.
- This is a count-based breaker, not time-windowed. It trips after N total failures, not "N failures in the last 60s." For bursty traffic you'll want a rolling window or a failure-rate calculation. Easy to add, but say so.
- "Blast-radius controls" still needs more than this. The breaker plus
TimeoutSecondsgives you fault isolation and bounded call duration. Real blast-radius work is concurrency caps (Step Functions Map MaxConcurrency, Lambda reserved concurrency), per-tenant queues, and IAM scoping. If you want the section to keep that phrase, show one of those too — otherwise call it "fault isolation and graceful degradation," which is what the code actually delivers.
2. CloudWatch Composite Alarms for Cascade Detection
Individual alarms are useless for detecting cascades. You need to detect correlated failures across agents. CloudWatch Composite Alarms let you do exactly that:
import boto3
cloudwatch = boto3.client("cloudwatch")
AGENTS = ["market-analysis", "position-sizing", "execution", "compliance"]
ERROR_THRESHOLD = 5 # > 5 errors in a 60s period => that agent is "erroring"
CASCADE_THRESHOLD = 2 # >= 2 agents failing at once => cascade
def build_metrics(agents):
"""Build the metric-math graph: a per-agent 'failing' flag, summed into a count."""
metrics = []
breach_ids = []
for idx, name in enumerate(agents, start=1):
err_id, beat_id, breach_id = f"e{idx}", f"h{idx}", f"b{idx}"
breach_ids.append(breach_id)
# Raw error count for this agent (feeds the expression, not returned)
metrics.append({
"Id": err_id,
"MetricStat": {
"Metric": {"Namespace": "AgentPlatform", "MetricName": "ErrorCount",
"Dimensions": [{"Name": "AgentName", "Value": name}]},
"Period": 60, "Stat": "Sum"},
"ReturnData": False})
# Heartbeat the agent emits on a timer. Absent/0 => the agent has gone silent.
metrics.append({
"Id": beat_id,
"MetricStat": {
"Metric": {"Namespace": "AgentPlatform", "MetricName": "Heartbeat",
"Dimensions": [{"Name": "AgentName", "Value": name}]},
"Period": 60, "Stat": "Sum"},
"ReturnData": False})
# An agent is "failing" if it is erroring OR it has gone silent.
# FILL(...,0) turns missing data into 0, so a DEAD agent scores as a failure
# instead of quietly sitting in INSUFFICIENT_DATA and being ignored.
metrics.append({
"Id": breach_id,
"Expression": f"(FILL({err_id},0) > {ERROR_THRESHOLD}) OR (FILL({beat_id},0) < 1)",
"Label": f"{name}-failing", "ReturnData": False})
# cascade = number of agents currently failing. This is the series the alarm watches.
metrics.append({
"Id": "cascade",
"Expression": " + ".join(breach_ids),
"Label": "AgentsInFailureState", "ReturnData": True})
return metrics
cloudwatch.put_metric_alarm(
AlarmName="cascade-detection-trading-agents",
AlarmDescription="Fires when 2 or more agents are erroring or silent at the same time.",
Metrics=build_metrics(AGENTS),
EvaluationPeriods=1,
DatapointsToAlarm=1,
Threshold=CASCADE_THRESHOLD,
ComparisonOperator="GreaterThanOrEqualToThreshold",
TreatMissingData="notBreaching",
AlarmActions=[
"arn:aws:lambda:us-east-1:123456789012:function:emergency-circuit-breaker"
],
)
- The heartbeat is instrumentation you have to add. Each agent must emit a
Heartbeatmetric on a timer (say every 30s), independent of whether it's doing work. If you reuse Lambda Invocations instead, a legitimately idle agent looks identical to a dead one and you'll get false cascades. For continuously-active trading agents that may be fine; say which you mean. - This caps at ~5 agents. The 10-metric hard limit means error + heartbeat per agent tops out around five agents in a single alarm. Beyond that, switch to a Metrics Insights query alarm
(SELECT ... GROUP BY AgentName), which aggregates across many agents without the per-metric cap, or tier the detection. Worth a sentence so nobody builds this for 30 agents and hits the wall.
3. Circuit Breaker Lambda
import os
import boto3
from botocore.config import Config
from botocore.exceptions import ClientError
# Adaptive retries add client-side rate limiting so a burst of StopExecution calls
# rides out throttling instead of aborting the halt.
_CFG = Config(retries={"max_attempts": 10, "mode": "adaptive"})
sfn = boto3.client("stepfunctions", config=_CFG)
sns = boto3.client("sns")
# Comma-separated list, so "halt all agents" can span more than one workflow.
STATE_MACHINE_ARNS = [a.strip() for a in os.environ["STATE_MACHINE_ARNS"].split(",") if a.strip()]
INCIDENT_TOPIC_ARN = os.environ["INCIDENT_TOPIC_ARN"]
# Safety valve: DRY_RUN=true logs what WOULD be stopped without stopping anything.
DRY_RUN = os.environ.get("DRY_RUN", "false").lower() == "true"
def emergency_circuit_breaker(event, context):
"""Triggered by the cascade alarm. Stop EVERY running execution of the workflow(s),
then notify. Paginates so it truly reaches all of them, and keeps going if an
individual stop fails."""
found = stopped = failed = 0
paginator = sfn.get_paginator("list_executions")
for sm_arn in STATE_MACHINE_ARNS:
for page in paginator.paginate(stateMachineArn=sm_arn, statusFilter="RUNNING"):
for execution in page["executions"]:
found += 1
arn = execution["executionArn"]
if DRY_RUN:
print(f"[DRY_RUN] would stop {arn}")
continue
try:
sfn.stop_execution(
executionArn=arn,
error="CascadeDetected",
cause="Cascade alarm triggered - correlated failures across agents",
)
stopped += 1
except ClientError as e:
# Race (execution just finished) or throttling that exhausted retries.
# Log and keep going - one bad stop must not abandon the rest.
failed += 1
code = e.response["Error"]["Code"]
print(f"[ERROR] could not stop {arn}: {code} - {e}")
result = {"found": found, "stopped": stopped, "failed": failed, "dry_run": DRY_RUN}
_notify(result)
return result
def _notify(r):
if r["dry_run"]:
subject = f"[DRY RUN] Agent cascade: {r['found']} executions would be halted"
elif r["failed"]:
subject = f"AGENT CASCADE - stopped {r['stopped']}/{r['found']}, {r['failed']} FAILED"
else:
subject = f"AGENT CASCADE - stopped all {r['stopped']} executions"
sns.publish(
TopicArn=INCIDENT_TOPIC_ARN,
Subject=subject[:100], # SNS Subject hard limit is 100 chars
Message=(
"Cascade alarm fired.\n"
f"Running executions found: {r['found']}\n"
f"Successfully stopped: {r['stopped']}\n"
f"Failed to stop: {r['failed']}\n"
f"Dry run: {r['dry_run']}\n\n"
"Note: StopExecution halts the state machine but does not force-kill a Lambda "
"task that is already running - those finish or hit their own timeout."
),
)
This is your "big red button." When the cascade alarm fires, everything stops. Immediately. You can investigate and restart selectively once you understand what happened.
Three caveats:
- "Immediately" has an asterisk.
StopExecutionstops the orchestration — no new steps get scheduled — but a Lambda task that's already mid-flight keeps running until it returns or hits its own timeout. The button stops the machine, not the gun already fired. That's why the SNS message spells it out. If you need to truly cut off in-flight work, the agent Lambdas need short timeouts (tie back to your earlierTimeoutSecondssection). - Standard workflows only.
ListExecutionsandStopExecutiondon't work for Express workflows (the API is explicitly "not supported by EXPRESS"). Say so, because a reader on Express gets nothing from this. - This auto-halts all trading on one alarm. The blast radius of a false-positive cascade is total.
DRY_RUNlets you validate before arming it, but the post should state plainly that you're trading "occasionally halt on a false alarm" for "never run blind during a real one" — and that's a deliberate choice, not a free lunch.
4. AWS Fault Injection Service: Test Your Resilience
You can't know if your circuit breakers work unless you test them. AWS Fault Injection Service (FIS) lets you deliberately inject failures into your agent infrastructure:
- Inject latency into tool API calls
- Force Lambda throttling to simulate overloaded backends
- Kill a percentage of agent tasks to test graceful degradation
- Inject error responses from downstream services
Run these experiments in staging regularly. Discover where your cascades happen before production does.
5. Rate Limiting at the Agent Level
Don't let agents decide their own retry behavior. Enforce rate limits externally:
# In your agent's tool execution layer
import threading
import time
class AgentRateLimiter:
"""External limits for agent tool calls, enforced OUTSIDE the LLM's loop.
Caps calls-per-minute per tool, and retries per operation. Thread-safe.
Scope: one process. If the agent runs across several instances, back this with
shared state (Redis/DynamoDB) so the limit is global instead of per-instance.
"""
def __init__(self, max_calls_per_minute=10, max_retries=2):
self.max_calls = max_calls_per_minute
self.max_retries = max_retries
self.calls = {} # tool_name -> recent call timestamps
self.attempts = {} # operation_id -> attempts so far
self._lock = threading.Lock()
def can_call(self, tool_name: str) -> bool:
"""True if this tool is under its per-minute limit (and records the call)."""
now = time.monotonic()
with self._lock:
recent = [t for t in self.calls.get(tool_name, []) if t > now - 60]
if len(recent) >= self.max_calls:
self.calls[tool_name] = recent
return False
recent.append(now)
self.calls[tool_name] = recent
return True
def can_retry(self, operation_id: str) -> bool:
"""True while an operation still has retries left. The agent doesn't get to
decide 'I should retry' - this does. max_retries=2 allows 3 attempts total."""
with self._lock:
self.attempts[operation_id] = self.attempts.get(operation_id, 0) + 1
return self.attempts[operation_id] <= self.max_retries + 1
The critical insight: this rate limiter lives outside the LLM's reasoning loop. The agent doesn't get to decide "I should retry." The infrastructure decides whether the agent is allowed to call the tool at all.
Key Takeaway
Design your agent systems for failure, not just for success. Bounded retries, hard timeouts, circuit breakers, graceful degradation, correlated failure detection. If your agents can retry without limit, delegate without checks, or propagate errors without containment, you don't have a multi-agent system. You have a cascade waiting to happen.
Up Next
Post 9: Human-Agent Trust Exploitation (ASI09) - What happens when humans stop questioning AI recommendations, and how Amazon A2I and Step Functions approval gates keep a human genuinely in the loop.
I would be very interested to hear your thoughts or comments, so please feel free to ping me on LinkedIn or Twitter, or drop them below. If you've experienced a cascade in an agent system, I genuinely want to hear the war story.
Onward!!
Top comments (0)