It was a normal Friday afternoon until the tickets started: "Is your API down?" Our status page was green. Our logs were clean. Our servers were fine. The thing that was actually down belonged to a vendor we had no visibility into, no pager for, and no way to fix.
That was the day I stopped treating upstream providers as a background assumption and started treating failure as a design input. If your application calls a third-party API — a payment processor, an AI model provider, a geocoding service — then their outage is your outage. The only question is how much of it leaks to your users.
Here are the four patterns that actually kept us online after that day, with code you can adapt.
Pattern 1: Cut failures off early with timeouts everywhere
The most common outage amplifier is not a slow response. It is no response at all. A connection that hangs open consumes a worker thread, then another, then your whole pool, and suddenly your API is "down" even though every upstream is fine.
Rule one: every outbound call gets a connect timeout and a read timeout, and you set them deliberately rather than relying on defaults.
import httpx
# connect timeout: how long to wait for the TCP/TLS handshake
# read timeout: how long to wait between bytes of the response
client = httpx.Client(timeout=httpx.Timeout(connect=3.0, read=10.0))
# For streaming responses, a single read timeout is not enough.
# You want a per-byte budget, not a per-response budget:
stream_timeout = httpx.Timeout(connect=3.0, read=30.0, write=10.0, pool=3.0)
The number matters less than the existence of the limit. Pick a budget that fits your user-facing latency goal, then make it smaller than your platform's request timeout, so you decide when a call is dead — not the load balancer upstream of you.
Pattern 2: A circuit breaker so you stop hammering a dead provider
Timeouts stop a single request from hanging forever. But if the provider is down for twenty minutes, every request will still wait out its full timeout before failing. Under load, that is a self-inflicted pileup: your users wait, your queues fill, your own error rate spikes.
A circuit breaker fixes that by failing fast once you know the upstream is unhealthy. The state machine is small enough to hold in your head:
- Closed: calls go through normally. Failures are counted.
- Open: after N consecutive failures, all calls fail immediately without touching the network.
- Half-open: after a cooldown, one probe request is allowed through. If it succeeds, the circuit closes; if it fails, it opens again.
import time
class CircuitBreaker:
def __init__(self, failure_threshold=5, cooldown=30.0):
self.failure_threshold = failure_threshold
self.cooldown = cooldown
self.failures = 0
self.state = "closed" # closed | open | half_open
self.opened_at = None
@property
def allowed(self) -> bool:
if self.state == "closed":
return True
if self.state == "open":
if time.monotonic() - self.opened_at >= self.cooldown:
self.state = "half_open"
return True
return False
return True # half_open: exactly one probe request
def record_success(self) -> None:
self.failures = 0
self.state = "closed"
def record_failure(self) -> None:
self.failures += 1
if self.failures >= self.failure_threshold or self.state == "half_open":
self.state = "open"
self.opened_at = time.monotonic()
Wiring it around a call is a few lines:
def call_with_breaker(breaker: CircuitBreaker, fn):
if not breaker.allowed:
raise UpstreamUnavailable("failing fast: circuit is open")
try:
result = fn()
except Exception:
breaker.record_failure()
raise
breaker.record_success()
return result
Two details that mattered in practice. First, count only meaningful failures — a 400 validation error from the provider is not an outage signal and should not trip the circuit; a timeout, connection error, or 5xx should. Second, give each upstream dependency its own breaker. A shared breaker means one flaky endpoint takes down the healthy ones with it.
Pattern 3: Serve the last good response instead of an error page
Fast failure is good. Not failing at all is better. For data that changes slowly — a currency list, a weather forecast, a model's cached output — the most reliable source during an outage is often the last successful response you already have.
The pattern is stale-while-error: serve from cache when it is fresh; when the upstream fails, fall back to the stale entry instead of surfacing the error.
import time
_cache: dict[str, tuple[float, object]] = {}
def get_with_fallback(key: str, fetch, ttl_seconds: float = 300.0):
now = time.monotonic()
entry = _cache.get(key)
if entry and entry[0] > now:
return entry[1] # fresh cache: happy path
try:
payload = fetch()
except Exception:
if entry is not None:
return entry[1] # upstream down: stale is better than nothing
raise
_cache[key] = (now + ttl_seconds, payload)
return payload
A real implementation should store an explicit fetched_at timestamp with the payload, log loudly whenever it serves a stale fallback, and expire entries entirely after a much longer absolute horizon — you do not want to serve data from last quarter because the vendor was down for a month. The key idea is that your users should never see a provider's incident page; they should see slightly old data and a healthy API.
Pattern 4: Fail over to a second provider — only where it is safe
If your dependency is a commodity API — several providers offer the same interface, like the OpenAI-compatible chat endpoints many model hosts expose — a secondary provider gives you an escape hatch. The loop is simple:
def call_with_failover(providers, request):
last_error = None
for provider in providers:
try:
return provider.chat(request)
except (TimeoutError, httpx.TransportError) as exc:
last_error = exc
continue
raise UpstreamUnavailable(f"all providers failed: {last_error}")
The warning I wish someone had given me earlier: failover is only safe when the operation is read-only or idempotent. If you retry a non-idempotent payment charge on a second provider, you do not get resilience — you get two charges and a very angry customer. For writes, failover is a product decision, not a code change. When in doubt, fail over only for reads, and only across providers that share a compatible contract.
Test the failure modes before the failure
Every one of these patterns rots if it is never exercised. The cheap way to keep them honest is to simulate the outage in tests by monkeypatching the transport:
def test_circuit_opens_after_timeouts(monkeypatch):
breaker = CircuitBreaker(failure_threshold=3, cooldown=30.0)
def flaky_call():
raise httpx.ReadTimeout("simulated upstream hang")
for _ in range(3):
with pytest.raises(httpx.ReadTimeout):
call_with_breaker(breaker, flaky_call)
assert breaker.state == "open"
# While open, we fail fast without ever touching the network:
with pytest.raises(UpstreamUnavailable):
call_with_breaker(breaker, flaky_call)
The test that caught the most for us: asserting the circuit behavior, not just the HTTP behavior. Anyone can write a test that a timeout raises; the valuable test proves the breaker opened, the fallback served stale data, and the whole chain degraded gracefully instead of collapsing.
The checklist I keep next to every integration
- Every outbound call has an explicit connect and read timeout.
- Each upstream has its own circuit breaker with a sane threshold and cooldown.
- Cacheable data has a stale-while-error fallback with a hard expiry.
- Non-idempotent calls never get automatic retries or failover.
- The failure path is tested with simulated timeouts, not just happy-path mocks.
- When the upstream degrades, my API degrades by design — never by accident.
That Friday outage ended up being a gift, honestly. It forced us to treat failure as a normal state of the system rather than an exception to be handled later. Your dependencies will go down. Your users do not have to know.
Top comments (0)