The 3 AM Phone Call
Two years ago, I woke up to my phone buzzing at 3:14 AM. Our payment processing API was down — again. The third-party provider had another "unexpected outage," and our system was retrying requests in a tight loop, hammering their already-dead endpoint while our users saw nothing but spinning wheels.
That night cost us 47 abandoned carts and a very angry customer support ticket that started with "I tried 11 times and..."
I swore I'd never get that call again. Here's what I built instead — and what you should build too if your application depends on external APIs.
The Problem: Most API Clients Are Optimistic to a Fault
The default HTTP client in most languages does exactly one thing when a request fails: nothing useful. No retry. No backoff. No circuit breaker. Just a stack trace and a confused user.
Here's what a naive API call looks like in most codebases:
import requests
def charge_customer(amount, token):
resp = requests.post("https://api.paymentprovider.com/v1/charges", json={
"amount": amount,
"token": token
})
return resp.json()
This function has three failure modes, and it handles exactly zero of them:
| Failure | What Happens | User Sees |
|---|---|---|
| Timeout (API hangs) |
requests waits forever (default: no timeout) |
Infinite spinner |
| 5xx error | Exception propagates up | "Something went wrong" |
| Network blip (transient) | Instant failure, no retry | Frustrating "try again" |
Layer 1: Timeouts (The One-Liner That Saves You)
The single highest-ROI change you can make:
# Before: no timeout — can hang forever
resp = requests.post(url, json=payload)
# After: connect timeout + read timeout
resp = requests.post(url, json=payload, timeout=(3.0, 10.0))
The tuple is (connect_timeout, read_timeout). Connect timeout handles DNS/network delays. Read timeout handles slow responses. If your API hasn't sent bytes in 10 seconds, something is wrong — fail fast so your retry logic can kick in.
Real numbers from our system: Adding timeouts reduced average failure recovery time from 45 seconds to 6 seconds. Users stopped rage-quitting during upstream outages.
Layer 2: Retry With Exponential Backoff and Jitter
Not all failures are permanent. Network blips, DNS hiccups, and load balancer reboots last seconds. A retry with the right backoff strategy catches these.
import time
import random
def retry_with_backoff(func, max_retries=3, base_delay=1.0):
for attempt in range(max_retries + 1):
try:
return func()
except (ConnectionError, Timeout, ServerError) as e:
if attempt == max_retries:
raise
# Exponential backoff with jitter
delay = base_delay * (2 ** attempt)
jitter = random.uniform(0, delay * 0.3)
time.sleep(delay + jitter)
Why jitter matters: If 10,000 clients all retry at exactly 1s, 2s, 4s, you create thundering herds that can knock over a recovering service. Jitter spreads the load. AWS has a great writeup on this in their architecture blog — their internal services all use jittered backoff.
The retry delays look like this:
- Attempt 1: ~1.0-1.3 seconds
- Attempt 2: ~2.0-2.6 seconds
- Attempt 3: ~4.0-5.2 seconds
Total: ~7-9 seconds before giving up. Fast enough that users notice the delay, slow enough to catch most transient failures.
Layer 3: Circuit Breaker (Stop Hitting a Dead Horse)
Retry with backoff works for transient failures. But when a service is down for 30 minutes, retrying every 9 seconds means 200 failed requests per endpoint — wasting resources and potentially making things worse.
Enter the circuit breaker pattern:
import threading
import time
class CircuitBreaker:
def __init__(self, failure_threshold=5, recovery_timeout=30):
self.failure_threshold = failure_threshold
self.recovery_timeout = recovery_timeout
self.failures = 0
self.last_failure_time = 0
self.state = "CLOSED" # CLOSED, OPEN, HALF_OPEN
self.lock = threading.Lock()
def call(self, func, fallback=None):
with self.lock:
if self.state == "OPEN":
if time.time() - self.last_failure_time > self.recovery_timeout:
self.state = "HALF_OPEN"
else:
return fallback() if fallback else None
try:
result = func()
with self.lock:
self.failures = 0
self.state = "CLOSED"
return result
except Exception:
with self.lock:
self.failures += 1
self.last_failure_time = time.time()
if self.failures >= self.failure_threshold:
self.state = "OPEN"
raise
# Usage
breaker = CircuitBreaker(failure_threshold=5, recovery_timeout=30)
def get_product_price(product_id):
return breaker.call(
lambda: requests.get(f"https://api.pricing.com/v1/products/{product_id}", timeout=5),
fallback=lambda: {"price": get_cached_price(product_id)}
)
Three states:
- CLOSED: Normal operation. Requests flow through.
- OPEN: Circuit is tripped. All requests immediately return the fallback (or fail fast). No requests reach the downstream service.
- HALF_OPEN: After the recovery timeout, one request is allowed through as a probe. If it succeeds, the circuit closes. If it fails, it opens again.
This one pattern eliminated our 3 AM pages. When the payment provider went down, the circuit opened within 5 failures (~45 seconds), and from that point on, every request returned a "payment temporarily unavailable — we'll notify you when it's back" message instead of spinning forever. Support tickets about payment failures dropped from "I tried 11 times" to "I got a clear message and an email when it was back."
Layer 4: Graceful Degradation (The Fallback Chain)
Not every API call is critical. If your product recommendation engine is down, showing a static "popular products" list is better than a blank page.
Build fallback chains:
def get_recommendations(user_id):
# Tier 1: Personalized ML model
try:
return call_recommendation_api(user_id)
except Exception:
pass
# Tier 2: Cached recent recommendations
cached = get_cached_recommendations(user_id)
if cached:
return cached
# Tier 3: Static popular products
return get_popular_products(category="default")
Map your API dependencies on a criticality spectrum:
| Criticality | Examples | Strategy |
|---|---|---|
| Critical (must succeed) | Payment processing, auth | Retry + circuit breaker + alerting |
| Important (can degrade) | Product search, recommendations | Fallback chain + stale cache |
| Nice-to-have (optional) | Analytics, logging, A/B tests | Fire-and-forget, async queue |
Putting It All Together: A Production-Ready API Client
Here's what a resilient API call looks like after all four layers:
from dataclasses import dataclass
from typing import Optional, Callable
import requests
import time
import random
import threading
@dataclass
class ResilientClient:
timeout: tuple = (3.0, 10.0)
max_retries: int = 3
base_delay: float = 1.0
circuit_threshold: int = 5
circuit_recovery: int = 30
def __post_init__(self):
self._breakers = {}
def _get_breaker(self, key: str) -> CircuitBreaker:
if key not in self._breakers:
self._breakers[key] = CircuitBreaker(
failure_threshold=self.circuit_threshold,
recovery_timeout=self.circuit_recovery
)
return self._breakers[key]
def call(self, method: str, url: str, *,
fallback: Optional[Callable] = None,
breaker_key: Optional[str] = None,
**kwargs) -> requests.Response:
breaker_key = breaker_key or f"{method}:{url}"
breaker = self._get_breaker(breaker_key)
def make_request():
return requests.request(
method, url, timeout=self.timeout, **kwargs
)
def attempt_with_retry():
for attempt in range(self.max_retries + 1):
try:
return make_request()
except (requests.ConnectionError, requests.Timeout) as e:
if attempt == self.max_retries:
raise
delay = self.base_delay * (2 ** attempt)
jitter = random.uniform(0, delay * 0.3)
time.sleep(delay + jitter)
raise RuntimeError("unreachable")
return breaker.call(attempt_with_retry, fallback=fallback)
# Usage
client = ResilientClient()
try:
resp = client.call("POST", "https://api.payment.com/v1/charges",
json={"amount": 1999, "token": "tok_xxx"},
fallback=lambda: queue_for_retry(amount=1999))
except Exception:
# Circuit is open and no fallback — show user a clear message
show_message("Payment is temporarily unavailable. We'll retry automatically.")
The Results After 6 Months
Since implementing these four layers across our 40+ external API dependencies:
- 3 AM pages: Zero. Down from 2-3 per month.
- Failed payment recovery: 94% of transient failures now resolve automatically via retry (was ~30% with naive retry).
- User-facing error rate: Down 82%. Users see fallback content or clear messages instead of blank screens.
- Upstream load during outages: Down 96%. Circuit breakers stop the barrage.
The best part? Most of this is 50 lines of code. The CircuitBreaker class is 30 lines. The retry decorator is 15 lines. Timeouts are one parameter.
What I'd Do Differently
Start with timeouts, not retries. I built an elaborate retry system first, then realized I had no timeout set — so retries were waiting for the default infinity-second timeout before even starting. Backwards.
Log every circuit trip. When a circuit opens, log it with the endpoint, failure count, and timestamp. You'll discover patterns — like "the inventory API always goes down Tuesdays at 2 AM during their maintenance window."
Circuit breakers need per-endpoint granularity, not per-service.
POST /chargesGET /productswill fail. Key your breakers by endpoint, not by hostname.Test your fallbacks. Easy to write a fallback function, easy to forget it breaks when the upstream API changes its response format. Run chaos engineering tests monthly.
TL;DR
Four layers, from simplest to most impactful:
- Timeouts — 1 line of code, prevents infinite hangs
- Retry with jittered backoff — 15 lines, catches transient failures
- Circuit breaker — 30 lines, stops hammering dead services
- Fallback chain — Business logic, keeps users happy during outages
Build these before you need them. Because at 3 AM, you don't want to be the person debugging a retry loop.
What's your approach to handling API failures? I'd love to hear what patterns have worked for you — drop a comment below.
Top comments (0)