Your dashboards go red. Not one model. All of them. Retries spike. Latency climbs. Nothing recovers.
When every Claude call starts failing at once
If you ship anything serious on top of LLM APIs, you have lived this moment.
Requests that worked minutes ago now return errors. You flip the model flag. Same thing. You increase retries. Error rate goes up, not down. Queues back up. Downstream services feel it.
This is not a prompt issue. This is not a bad deploy. This is what a platform-level incident looks like from the client side.
The recent Anthropic incident labeled "Elevated error rate across multiple models" is a clean example of the failure mode that matters most in production. Multi-model impact. No numbers. No knobs to turn. Just errors.
The mistake teams make is assuming model diversity equals resilience. It does not.
Why model fallback quietly fails during platform incidents
Most LLM client stacks are built around a simple idea: if model A fails, try model B.
That works when failures are model-scoped. It fails when the blast radius is shared infrastructure.
Under the hood, models often share front-door API gateways, auth and quota systems, routing layers, and shared inference clusters or schedulers.
When error rate rises across multiple models, it usually means something above the model is degraded. Switching from Opus to Sonnet does not bypass the problem because the problem is not Opus or Sonnet.
The docs do not emphasize this. SDK examples actively encourage retry and fallback loops that assume independence. In production, this assumption is wrong often enough to matter.
What this failure looks like in real systems
Here is the common sequence I see during incidents like this. Error rate jumps from sub-1 percent to double digits. Clients retry aggressively. Retry traffic amplifies load on the already degraded system. Latency increases, triggering timeouts. Queues back up. Background jobs fall behind. Engineers start manually throttling.
The most dangerous part is that nothing is technically "down." Health checks pass. Some requests succeed. This keeps retry storms alive.
If you only built retry logic, you built a traffic amplifier.
The naive retry loop that makes things worse
This is the pattern I still see in production code:
from anthropic import Anthropic
import time
client = Anthropic()
def call_claude(prompt):
for attempt in range(5):
try:
return client.messages.create(
model="claude-3-sonnet",
max_tokens=512,
messages=[{"role": "user", "content": prompt}]
)
except Exception:
time.sleep(2 ** attempt)
raise RuntimeError("Claude failed after retries")
This looks reasonable. It is not.
During a platform incident, all retries hit the same failing surface, backoff still creates synchronized retry waves, and you contribute to global load at the worst possible moment. This pattern is survivable at low volume. At scale, it cascades.
The real fix starts with circuit breakers, not retries
The first control you need is not more retries. It is a circuit breaker that can say "stop calling this API."
A circuit breaker turns repeated failure into a fast local decision. Here is a minimal breaker using Redis as shared state:
import redis
import time
r = redis.Redis()
def claude_available():
return r.get("claude_circuit") != b"open"
def trip_claude():
r.setex("claude_circuit", 60, "open")
def call_claude_with_breaker(fn):
if not claude_available():
raise RuntimeError("Claude circuit open")
try:
return fn()
except Exception:
trip_claude()
raise
The breaker is shared across workers. The open state expires automatically. Fail fast beats slow failure every time. This alone will save your system from self-inflicted damage.
Why retries still matter, but only inside boundaries
Retries are not evil. Unbounded retries are.
The pattern that works is a small retry count, a tight timeout, wrapped inside a breaker:
def safe_claude_call(prompt):
def attempt():
return client.messages.create(
model="claude-3-sonnet",
max_tokens=512,
timeout=5,
messages=[{"role": "user", "content": prompt}]
)
for _ in range(2):
try:
return call_claude_with_breaker(attempt)
except Exception:
time.sleep(0.5)
raise RuntimeError("Claude unavailable")
Notice what is missing: no exponential backoff to 30 seconds, no retry count in the teens, no blind faith that persistence equals success. At incident scale, shorter and fewer retries win.
Model fallback is not resilience, provider fallback is
If multiple models fail together, the only real isolation is at the provider boundary. That means your fallback graph needs to cross vendors, not model families.
def generate(prompt):
try:
return safe_claude_call(prompt)
except RuntimeError:
return call_openai(prompt)
This is not about preference. It is about failure domains. If your business cannot tolerate degraded LLM output, you must pay the complexity tax of multi-provider support. There is no shortcut here.
Bulkheads stop one failing feature from drowning everything
Another failure pattern during incidents is resource starvation. Your summarization jobs eat all workers. User-facing requests time out. Everyone is unhappy.
Bulkheads prevent this by isolating workloads:
queues:
llm_user_requests:
concurrency: 50
llm_background_jobs:
concurrency: 10
During elevated error rates, user traffic stays responsive and background jobs slow down, not everything. If you only have one queue, you do not have isolation.
Idempotency saves you from retry side effects
Retries are dangerous when requests have side effects. If your LLM call triggers writes, notifications, or billing events, retries can duplicate work.
Use idempotency keys consistently:
idempotency_key = f"summary:{document_id}"
client.messages.create(
model="claude-3-sonnet",
idempotency_key=idempotency_key,
messages=[{"role": "user", "content": prompt}]
)
When incidents happen, duplicate requests are common. Idempotency turns chaos into correctness.
What monitoring actually matters during elevated error rates
Most teams monitor request count and average latency. That is not enough. During incidents, you need error rate by provider, circuit breaker state, retry volume, and queue depth growth.
A simple metric that pays for itself is retry amplification:
select
sum(retry_attempts) / sum(original_requests) as retry_factor
from llm_request_metrics
where timestamp > now() - interval '5 minutes';
If that number goes above 1.5, you are probably making things worse.
Why status pages do not help you in the moment
Status pages tell you two things: something is wrong, and someone else is fixing it. They do not tell you how long retries will fail, which endpoints are safe, or whether partial recovery is real.
Your system must assume uncertainty. That is why local control mechanisms matter more than external signals. If your only reaction is waiting for green, you are already behind.
Graceful degradation beats hard failure
When LLMs are unavailable, many products default to errors. This is often unnecessary. You can return cached responses, use smaller local models for basic tasks, skip non-critical enrichment, or defer generation to async jobs.
A simple cache fallback:
cached = cache.get(prompt_hash)
if cached:
return cached
response = generate(prompt)
cache.set(prompt_hash, response, ttl=3600)
return response
During incidents, cache hit rates go up. That buys you time.
Where this advice breaks
There are cases where none of this helps: if your workload is real-time and uncachable, if output correctness is legally critical, or if you are contractually bound to one provider.
In those cases, your only option is admission control. Say no early. Protect the rest of the system. Rejecting requests cleanly is better than timing out everything.
Admission control and load shedding when breakers are not enough
Circuit breakers stop calls after failure. Admission control stops calls before failure. During platform incidents, the worst thing you can do is accept unlimited work you cannot complete. You need a hard cap.
A simple token bucket per provider works well:
import time
from collections import deque
class RateGate:
def __init__(self, max_inflight):
self.max_inflight = max_inflight
self.inflight = 0
def acquire(self):
if self.inflight >= self.max_inflight:
return False
self.inflight += 1
return True
def release(self):
self.inflight -= 1
claude_gate = RateGate(max_inflight=20)
Usage:
if not claude_gate.acquire():
raise RuntimeError("Claude overloaded locally")
try:
return safe_claude_call(prompt)
finally:
claude_gate.release()
During partial outages, latency often doubles before error rate spikes. Inflight requests pile up silently. Your worker pool gets saturated even if the provider eventually errors.
Concrete numbers I have seen: normal p95 latency 1.2s, incident p95 latency 8 to 12s, worker exhaustion within 90 seconds. Load shedding keeps tail latency bounded and preserves capacity for critical paths. It is blunt, but it works.
Hedged requests sound smart and usually backfire with LLM APIs
Some teams try hedged requests: fire the same prompt at two models or providers and take the first response. This can help with single-request latency variance. During incidents, it is gasoline.
Example of what not to do:
def hedged_generate(prompt):
return first_completed([
lambda: call_claude(prompt),
lambda: call_claude(prompt)
])
You double traffic exactly when capacity is constrained, trigger rate limits faster, and amplify retry storms upstream. Even cross-provider hedging is risky if you do not gate it:
def cautious_hedge(prompt):
if system_healthy():
return call_claude(prompt)
return first_completed([
lambda: call_claude(prompt),
lambda: call_openai(prompt)
])
Rules that keep this safe: hedge only after a latency threshold, never hedge inside retries, and disable hedging automatically when error rate rises. Hedging is an optimization. Incidents are not the time to optimize.
How to actually test this before production does it for you
Most LLM outage handling code is untested. That is why it fails under stress. You can simulate platform incidents locally with three levers: forced error responses, artificial latency, and partial success rates.
import random
import time
def flaky_claude(prompt):
r = random.random()
if r < 0.3:
raise RuntimeError("500")
if r < 0.6:
time.sleep(10)
return "ok"
Then run load tests and watch circuit breaker open rate, queue depth over time, admission rejections, and the retry amplification metric. You want to see breakers opening quickly, inflight count staying flat, and non-critical queues slowing first. If your test shows recovery only after traffic stops, your controls are too weak.
This is not chaos for chaos' sake. It is validating that your failure handling actually reduces load instead of redistributing it.
Final Thoughts
Elevated error rates across multiple models are not edge cases anymore. They are normal failure modes for platform-scale AI services.
If your resilience strategy is model switching and hope, you will relive the same outage every time.
The fix is not clever prompts or better SDK defaults. It is the same discipline that keeps payment systems, search backends, and message brokers alive under stress.
Treat LLMs like the critical infrastructure they are. Your future incidents will still hurt, but they will not take everything down with them.
Resources & References
- Anthropic Incident: Elevated error rate across multiple models
- AWS Architecture Blog: Exponential Backoff and Jitter
- Google SRE Book: Handling Overload
- Stripe Engineering: Designing APIs for failure
Stay in Touch
Short takes and discussions on X → https://x.com/sebuzdugan
Practical AI / ML videos on YouTube → https://www.youtube.com/@sebuzdugan/
Partnerships & collabs → sebuzdugan@gmail.com
Originally published on Medium.
Top comments (0)