When a major LLM provider goes down, your app shouldn't. Elevated error rates across model APIs are a when, not an if - so the resilience layer matters as much as the integration itself.
The Idea: Circuit Breakers and Fallback Chains
Most teams wire up an LLM API and handle errors with a basic try/except. That works fine until a real outage hits - then every request hangs or fails hard, and users see a broken product instead of a graceful degradation.
The pattern worth building is a fallback chain with a circuit breaker. Track consecutive failures, and once a threshold is crossed, stop hammering the primary provider and route to a backup - whether that's a different provider, a cached response, or a simplified deterministic answer. A circuit breaker (a pattern borrowed from distributed systems) sits in front of your API call and activates when failure rate spikes, protecting your app from compounding timeouts.
The two pieces together - fallback routing and circuit breaking - let you serve something useful even during a full provider outage.
Real Example
Here's a minimal Python pattern using tenacity for retries and a simple circuit breaker flag, with provider fallback:
import openai
import anthropic
from tenacity import retry, stop_after_attempt, wait_exponential
CIRCUIT_OPEN = False
FAILURE_COUNT = 0
THRESHOLD = 3
def call_primary(prompt):
client = anthropic.Anthropic()
return client.messages.create(
model="claude-3-5-sonnet-20241022",
max_tokens=256,
messages=[{"role": "user", "content": prompt}]
).content[0].text
@retry(stop=stop_after_attempt(2), wait=wait_exponential(min=1, max=4))
def call_fallback(prompt):
client = openai.OpenAI()
return client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": prompt}]
).choices[0].message.content
def get_response(prompt):
global CIRCUIT_OPEN, FAILURE_COUNT
if not CIRCUIT_OPEN:
try:
result = call_primary(prompt)
FAILURE_COUNT = 0
return result
except Exception:
FAILURE_COUNT += 1
if FAILURE_COUNT >= THRESHOLD:
CIRCUIT_OPEN = True
return call_fallback(prompt)
In production you'd replace the global flags with something like pybreaker or a Redis-backed counter so the state persists across workers. The routing logic stays the same.
Key Takeaways
- API outages from any LLM provider are recurring events - building for them upfront is cheaper than scrambling during one.
- A circuit breaker prevents cascading timeouts by stopping requests to a failing endpoint once a failure threshold is hit.
- A fallback chain (primary provider → secondary provider → cached/static response) keeps your product functional at degraded quality rather than fully broken.
Have you actually tested your LLM integration with the primary endpoint completely unreachable - what happened?
Sources referenced: HackerNews discussion on Claude elevated errors incident, Anthropic status page
Top comments (0)