If you've deployed an AI agent to production, you know the feeling: everything works perfectly during testing, then at 3 AM on a Saturday, your agent hits an API rate limit, silently fails, and your users wake up to a broken experience.
The difference between a demo agent and a production agent isn't the LLM — it's how it handles failure.
I've been running AI agents 24/7 for months now, and I've learned (the hard way) that resilience engineering is the single most underrated skill in the AI agent space. In this article, I'll share 7 battle-tested patterns that transformed my agents from fragile toys into systems I actually trust to run unsupervised.
Why AI Agents Fail Differently Than Traditional Software
Traditional software fails predictably. A database timeout throws an exception. A null pointer crashes the process. You can write unit tests for these.
AI agents fail creatively. They:
- Hallucinate a tool name that doesn't exist, then try to call it
- Get stuck in infinite loops ("Let me try that again... Let me try that again...")
- Produce valid-looking output that's completely wrong
- Silently degrade when an upstream API changes its response format
- Burn through your API budget in minutes during a retry storm
This means traditional error handling (try/catch, retries, circuit breakers) is necessary but not sufficient. You need patterns designed specifically for the non-deterministic nature of LLM-powered systems.
Pattern 1: The Layered Retry with Exponential Backoff + Jitter
Everyone knows about retries. But naive retries on LLM APIs are dangerous — they can amplify failures and drain your budget fast.
The pattern:
import random
import time
def resilient_llm_call(prompt, max_retries=3):
base_delay = 1.0
for attempt in range(max_retries):
try:
response = call_llm(prompt)
validate_response(response) # Critical: validate BEFORE returning
return response
except RateLimitError:
delay = base_delay * (2 ** attempt) + random.uniform(0, 1)
time.sleep(delay)
except ValidationError as e:
# Don't retry the same prompt — rephrase it
prompt = rephrase_with_error_context(prompt, str(e))
except Exception as e:
log_unexpected_failure(e, attempt)
if attempt == max_retries - 1:
return fallback_response(prompt)
return fallback_response(prompt)
The key insight: different error types need different retry strategies. Rate limits need backoff. Validation failures need prompt rephrasing. Unknown errors need logging and fallback.
The jitter (random component) prevents the "thundering herd" problem when multiple agent instances retry simultaneously.
Pattern 2: The Circuit Breaker for External Tools
When your agent calls external APIs (web search, databases, third-party services), a single failing dependency can cascade and take down the entire agent.
The circuit breaker pattern tracks failure rates and "opens" the circuit when failures exceed a threshold, preventing further calls to the failing service:
CLOSED (normal) → failures exceed threshold → OPEN (blocking)
↓
timeout expires
↓
HALF-OPEN (testing)
↓ ↓
success failure
↓ ↓
CLOSED OPEN
In practice, this means your agent gracefully skips a broken tool instead of hanging or crashing:
class ToolCircuitBreaker:
def __init__(self, failure_threshold=5, recovery_timeout=60):
self.failures = 0
self.threshold = failure_threshold
self.timeout = recovery_timeout
self.state = "closed"
self.last_failure = None
def call(self, tool_fn, *args):
if self.state == "open":
if time.time() - self.last_failure > self.timeout:
self.state = "half-open"
else:
raise CircuitOpenError(f"Tool unavailable, retry after {self.timeout}s")
try:
result = tool_fn(*args)
self.failures = 0
self.state = "closed"
return result
except Exception as e:
self.failures += 1
self.last_failure = time.time()
if self.failures >= self.threshold:
self.state = "open"
raise
When I implemented this for my agents' web search tool, downtime incidents dropped by ~80%. The agent simply tells the user "web search is temporarily unavailable, working from cached knowledge" instead of hanging.
Pattern 3: The Watchdog Timer (Anti-Infinite-Loop)
LLM agents love infinite loops. An agent decides to "try one more approach," which fails, so it tries "one more approach," which fails...
The fix is brutally simple: a watchdog timer that kills runaway executions.
import signal
class AgentWatchdog:
def __init__(self, max_steps=50, max_time_seconds=300):
self.max_steps = max_steps
self.max_time = max_time_seconds
self.step_count = 0
def tick(self):
self.step_count += 1
if self.step_count > self.max_steps:
raise WatchdogError(
f"Agent exceeded {self.max_steps} steps. "
f"Likely stuck in a loop. Terminating."
)
def run_with_timeout(self, agent_fn):
signal.alarm(self.max_time)
try:
return agent_fn()
except TimeoutError:
return "Task timed out. Here's what I completed so far..."
Two limits work together:
- Step limit: Catches logical loops (agent keeps calling the same tool)
- Time limit: Catches slow failures (each step takes too long)
I set my production agents to 50 steps max and 5 minutes timeout. If your agent needs more than 50 steps for a task, the task is probably too complex and should be decomposed.
Pattern 4: Graceful Degradation Chains
This is the pattern that changed everything for me. Instead of binary success/failure, define a chain of increasingly degraded responses:
class DegradationChain:
def __init__(self):
self.levels = [
self.full_capability, # Level 0: Everything works
self.reduced_tools, # Level 1: Skip non-essential tools
self.cached_knowledge, # Level 2: Use only cached data
self.template_response, # Level 3: Pre-written templates
self.honest_failure, # Level 4: "I can't do this right now"
]
def execute(self, task):
for level, handler in enumerate(self.levels):
try:
result = handler(task)
if level > 0:
result.add_note(
f"Running in degraded mode (level {level}). "
f"Some features may be limited."
)
return result
except Exception as e:
log(f"Level {level} failed: {e}, falling to level {level+1}")
return self.catastrophic_fallback(task)
The magic: your agent always responds with something useful, even when everything is on fire. A Level 3 template response is infinitely better than a silent failure or a cryptic error message.
Real-world example: My monitoring agent normally fetches live data, analyzes it, and generates a report. When the data API is down, it falls back to cached data with a "data may be stale" warning. When the LLM is overloaded, it returns a pre-formatted template. The user always gets something.
Pattern 5: Output Validation Guards
This is the pattern most people skip, and it's the one that causes the most production incidents.
LLMs can return syntactically valid but semantically wrong output. Your agent might generate a SQL query that runs perfectly — and deletes your production database. Or return a JSON response with all the right fields but nonsensical values.
class OutputValidator:
def __init__(self):
self.validators = []
def add_rule(self, name, check_fn, severity="block"):
self.validators.append({
"name": name,
"check": check_fn,
"severity": severity # "block", "warn", "log"
})
def validate(self, output):
issues = []
for v in self.validators:
if not v["check"](output):
issues.append(v)
blockers = [i for i in issues if i["severity"] == "block"]
if blockers:
raise ValidationError(
f"Output blocked: {[b['name'] for b in blockers]}"
)
warnings = [i for i in issues if i["severity"] == "warn"]
if warnings:
output.add_warnings([w["name"] for w in warnings])
return output
# Example validators
validator = OutputValidator()
validator.add_rule(
"no_destructive_sql",
lambda o: not any(kw in o.upper() for kw in ["DROP", "DELETE", "TRUNCATE"]),
severity="block"
)
validator.add_rule(
"response_length_reasonable",
lambda o: 10 < len(o) < 50000,
severity="warn"
)
validator.add_rule(
"no_hallucinated_urls",
lambda o: all(url_exists(u) for u in extract_urls(o)),
severity="warn"
)
Key validators I run on every agent output:
- No destructive operations (SQL deletes, file removals, API deletions)
- Response length sanity (too short = probably failed, too long = probably looping)
- URL validation (LLMs love hallucinating URLs)
- Format compliance (if you expect JSON, validate the schema)
- Cost sanity check (flag if a single response cost > $X)
Pattern 6: The Dead Man's Switch
Here's a failure mode nobody talks about: the agent that silently stops working.
No errors. No crashes. It just... stops processing. Maybe the event loop died. Maybe it's stuck waiting on a response that will never come. Maybe the process is alive but the agent logic is deadlocked.
The dead man's switch pattern: your agent must actively prove it's alive at regular intervals.
class DeadManSwitch:
def __init__(self, interval_seconds=300, alert_fn=None):
self.interval = interval_seconds
self.alert_fn = alert_fn or self.default_alert
self.last_heartbeat = time.time()
def heartbeat(self):
"""Agent calls this after completing each task"""
self.last_heartbeat = time.time()
def check(self):
"""External monitor calls this periodically"""
silence = time.time() - self.last_heartbeat
if silence > self.interval:
self.alert_fn(
f"Agent silent for {silence:.0f}s "
f"(threshold: {self.interval}s). "
f"Possible deadlock or crash."
)
return False
return True
This is exactly how frameworks like OpenClaw handle it — with a built-in heartbeat system. If the agent doesn't check in within the expected interval, the system knows something is wrong and can restart it or alert the operator.
The key insight: monitoring the absence of activity is harder than monitoring errors, but it catches the most dangerous failures.
Pattern 7: The Audit Trail (Your Future Self Will Thank You)
When (not if) something goes wrong in production, you need to reconstruct exactly what happened. Every agent action should be logged with enough context to replay the decision chain.
class AgentAuditLog:
def __init__(self, session_id):
self.session_id = session_id
self.entries = []
def log(self, action, input_data, output_data, metadata=None):
entry = {
"timestamp": datetime.utcnow().isoformat(),
"session_id": self.session_id,
"action": action,
"input": self.sanitize(input_data),
"output": self.sanitize(output_data),
"token_usage": metadata.get("tokens", {}),
"cost_usd": metadata.get("cost", 0),
"latency_ms": metadata.get("latency", 0),
"model": metadata.get("model", "unknown"),
"error": metadata.get("error", None),
}
self.entries.append(entry)
self.persist(entry)
def sanitize(self, data):
"""Remove PII and secrets before logging"""
# Never log API keys, passwords, or personal data
return redact_sensitive(str(data)[:10000])
Minimum fields to log:
- Timestamp (obviously)
- Input/Output (sanitized — never log secrets or PII)
- Token usage and cost (you will need this for budgeting)
- Latency (performance degradation is often the first sign of trouble)
- Model version (behavior changes between model versions)
- Error details (full stack trace, not just the message)
Putting It All Together: The Resilience Stack
These patterns aren't independent — they form layers:
┌─────────────────────────────────────┐
│ Dead Man's Switch (Pattern 6) │ ← Catches silent failures
├─────────────────────────────────────┤
│ Watchdog Timer (Pattern 3) │ ← Catches infinite loops
├─────────────────────────────────────┤
│ Audit Trail (Pattern 7) │ ← Records everything
├─────────────────────────────────────┤
│ Degradation Chain (Pattern 4) │ ← Always responds
├─────────────────────────────────────┤
│ Circuit Breaker (Pattern 2) │ ← Isolates tool failures
├─────────────────────────────────────┤
│ Output Validation (Pattern 5) │ ← Catches bad outputs
├─────────────────────────────────────┤
│ Layered Retry (Pattern 1) │ ← Handles transient errors
├─────────────────────────────────────┤
│ Your AI Agent │
└─────────────────────────────────────┘
Each layer handles a different failure mode. Together, they create an agent that can run unsupervised for days without intervention.
Quick-Start Resources
If you're building production AI agents and want to skip the trial-and-error phase:
Free AI Agent Starter Pack — Templates and checklists for deploying your first agent, including a pre-built resilience configuration.
AI Agent Complete Guide — Deep dive into agent architecture, from SOUL.md persona design to production deployment patterns like the ones in this article.
Key Takeaways
- AI agents fail differently than traditional software — plan for non-deterministic failures
- Different errors need different strategies — don't just retry everything the same way
- Always have a fallback — your agent should degrade gracefully, never crash silently
- Validate outputs before acting — the most dangerous failures look like successes
- Monitor the silence — a quiet agent might be a dead agent
- Log everything — you will need the audit trail, guaranteed
The agents that survive production aren't the smartest ones — they're the ones that handle failure gracefully. Build for resilience first, intelligence second.
Recommended Tools
- Vultr — cloud VPS hosting
- ElevenLabs — AI voice generation
What error handling patterns have you found essential for your AI agents? Drop a comment — I'd love to compare notes.
Top comments (0)