In a single-agent system, failure is simple: the agent errors, you retry.
In multi-agent systems, failure is a graph problem.
The Cascade Failure Problem
Agent A: ✅ Success
Agent B: ❌ Timeout (depends on A)
Agent C: ❌ Skipped (depends on B)
Agent D: ❌ Partial data (depends on C)
One timeout propagates through the entire pipeline. Without recovery, your system is fragile.
Our Recovery Strategy
AgentForge implements 3 recovery layers:
Layer 1: Retry with Exponential Backoff
@retry(max_attempts=3, backoff=exponential(base=2, max=60))
def agent_call(params):
return llm.invoke(params)
Layer 2: Circuit Breaker
If an agent fails 5 times in 10 minutes, we stop calling it and return a degraded response:
{
"status": "degraded",
"agent": "market_data",
"fallback": "cached_data",
"warning": "Real-time data unavailable, using 15-min delayed feed"
}
Layer 3: Pipeline Re-planning
When a critical agent fails, the orchestrator can re-plan:
- Skip the failed step if non-critical
- Substitute with a backup agent
- Halt and alert with full context trace
A Real Incident
Last month, our market data API went down during trading hours. Here's what happened:
- 14:32 — Market data agent timeout (Layer 1: 3 retries failed)
- 14:33 — Circuit breaker opened for market data agent
- 14:33 — Pipeline automatically switched to cached data + warning flag
- 14:35 — Full report generated with "delayed data" disclaimer
- 15:00 — Market data API recovered, circuit breaker closed automatically
Zero manual intervention. Zero missed reports.
This Is Table Stakes
If your multi-agent system can't handle one agent failing, it's not production-ready.
AgentForge makes this the default, not an afterthought.
https://github.com/agentforge-cyber/agentforge-mvp
Posted on 2026-07-17 by the AgentForge team.
Top comments (1)
The graph-failure framing is the part that matters here. Once one agent times out, the real question stops being "retry or not" and becomes "what downstream guarantees still hold after the fallback path?" I also liked that your examples separate skip, substitute, and halt behaviors, because production systems usually need all three and the hard part is making the policy explicit enough to debug later. This is exactly the kind of workflow where a local execution trace helps, since you want to reconstruct the failing hop, the recovery branch, and the warning surfaced to the caller without guessing after the fact. Tools like agent-inspect are useful in that gap for the same reason. Curious whether you have found a good threshold for when cached-data substitution should still count as success versus a degraded outcome that needs separate reporting.