Splitting a system into services buys you fault isolation only if the calls between services are built for failure. Otherwise one slow dependency quietly eats the caller’s connections, then its event loop, and then endpoints that never touch the slow service start timing out too. The Google SRE book calls this a cascading failure: “a failure that grows over time as a result of positive feedback.”
The circuit breaker pattern is the usual answer, and it is a good one. In my lab it was also the least important of the six changes. This post builds a small three-service stand in Python, injects a slow downstream, and adds microservices resilience patterns one at a time: timeouts, retries with exponential backoff and full jitter, a retry budget, idempotency keys, a circuit breaker, a bulkhead and a fallback. After every step the same incident is replayed and measured.
What you will get:
- A reproducible fault-injection stand (orders → inventory, orders → pricing) with open-loop load.
- Measurements of the cascade with no protection, including one I did not expect: the HTTP client’s connection pool blocking the event loop.
- Under 200 lines of dependency-free resilience primitives and 12 pytest tests, two of which caught real bugs in my first circuit breaker.
- A table after each step, so you can see what each pattern fixes and what it does not.
Stack: Python 3.13, httpx 0.28.1, Starlette 1.10.0, uvicorn 0.54.0, aiohttp 3.14.5 (load generator only), pytest 9.2.0. Every number and every output block below comes from real runs on a 2-vCPU Linux sandbox.
The lab: three services and one bad afternoon
Diagram of the lab: a load generator sends 60 rps to /product and 20 rps to /catalog on the orders service; orders calls inventory (16 DB slots, 0.5 s per query from 3 s to 10 s) and pricing (always 10 ms) through one shared httpx pool of 100 connections

During the slow phase inventory can serve 16 / 0.5 s = 32 requests per second while 60 arrive. That gap is the incident. The load generator is open-loop on purpose: real users do not wait for your backlog to clear before clicking again.
The fault injection is just a mutable dict that the benchmark flips over HTTP:

To run everything yourself:
How to read the tables: “inventory calls” counts every attempt that reached inventory (960 requests were offered), “conns busy/waiting” is the peak of busy connections in the orders → inventory pool and of requests queued for one, “loop stall” is the worst event-loop lag measured inside orders, and “recovery” is how long after inventory heals (t=10 s) it takes until a 0.5 s window of /product is at least 95% successful with p99 under 200 ms.
Step 0. No protection: watch the cascade
The baseline is what a lot of service code looks like: one shared client for every dependency and no timeout. httpx is actually safer than this by default (it raises after “5 seconds of network inactivity”), but requests will not time out at all “unless a timeout value is set explicitly”, so timeout=None is a realistic starting point.
Timeline of the baseline run: /product median latency climbs to 3.7 s during the slow phase, then /catalog, which only calls the healthy pricing service, climbs to 3.1 s once the shared pool hits 100 busy connections and up to 166 waiting requests
Three things happened, in order:
Inventory latency rose, so requests to it held connections longer. By t=5.7 s all 100 connections in the shared pool were busy.
/catalog requests, which only need pricing, queued behind them for a free connection. Pricing answered in 10 ms the whole time, yet /catalog p99 reached 3.16 s.
The backlog outlived the fault. Inventory healed at t=10 s, but orders needed another 4.5 s to drain its queue of waiting requests.
The 96% success rate is the trap: almost everything succeeded, just seconds too late.
The surprise: the pool itself burned CPU
The run-to-run spread was large. Across three baseline runs /catalog p99 was 3.16 s, 16.81 s and 11.57 s. The “loop stall” column explains it: in the bad runs the orders event loop was blocked for up to 6.8 s. Nothing in my code was CPU-heavy, so I read httpcore 1.0.9’s connection_pool.py. Every time a request enters or leaves the pool, _assign_requests_to_connections() walks the queued requests and, for each one, scans the connection list. With 100 connections and 400+ queued requests, that work runs on every pool change, and the event loop pays for it. Once it starts, it feeds itself: a slower loop means slower responses, which means more queued requests.
The lesson is not “httpx is bad”: an unbounded queue in front of a bounded resource is a cascade waiting for a trigger, wherever that queue lives.
Step 1. Timeouts: a per-attempt limit and a deadline
Two different limits are needed. A per-attempt timeout says how long one HTTP call may take. A deadline says how long the whole operation may take, including retries that come later. httpx covers the first one, asyncio.timeout() (Python 3.11+) the second.
How to choose 1 s? The AWS Builders’ Library suggests picking an acceptable false-timeout rate (say 0.1%) and setting the timeout at the matching downstream latency percentile (p99.9). Our healthy inventory answers in about 20 ms, so 1 s is generous.

Latency is now bounded and /catalog is back to normal, because /product can no longer hold more than about 60 connections (60 rps × 1 s). The success rate dropped to 63%, which is the honest number: during the slow phase inventory could not answer in time, and now we say so in one second instead of four.
Look at the peak inventory queue, though: 214, more than double the baseline’s 100. A timeout frees the caller’s connection, but inventory keeps working on the abandoned request. With no protection, the pool limit of 100 was accidentally acting as a cap on inventory. Timeouts removed that cap. The next steps are about putting a deliberate one back.
Step 2. Retries: exponential backoff with jitter, and a budget
The 5% of fast 503s are transient, so retrying looks like free success rate. First, the naive way: three immediate retries.
This is the worst row in the experiment. /product success fell to 24% and /catalog to 46%. Every failed attempt was immediately replaced by another one, the pool hit 100 busy connections again with 154 waiting, the loop stalled for 6.1 s, and /catalog failed with PoolTimeout and ConnectTimeout. Inventory received fewer calls (590) than in the baseline only because orders was too busy to send them. Two more runs gave the same 24% for /product and 37% for /catalog.
Retries multiply. The AWS Builders’ Library gives the stack-wide version: five layers that each retry three times put 243x the load on the database. Our single layer was enough.
The fix has three parts:
- Exponential backoff with full jitter. Wait a random time between 0 and min(cap, base * 2**attempt). Marc Brooker’s AWS Architecture Blog post simulated clients contending on an optimistic-concurrency database: backoff without jitter did more work and took more time than every jittered variant, and Full Jitter did the least work among them.
- A retry budget. I modeled it on gRPC’s retry throttling: every failure costs a token, every success refunds 0.1, and retries are allowed only while the bucket is above half. When most calls fail, retries stop by themselves.
- Respect the deadline. Never start a backoff sleep that would end after the deadline.
The budget’s effect is easy to pin down in a unit test with a dependency that is down: 100 requests make 400 calls without a budget and 103 with one.
Retrying a POST needs an idempotency key
Retrying GET /stock is safe. Retrying POST /reserve is not: per MDN, POST and PATCH “are not guaranteed to be idempotent.” The dangerous case is a lost response: the server commits, the reply arrives after the client’s timeout, and the client retries a write that already happened.
idempotency_demo.py reproduces it: 10% of /reserve calls commit and then answer after 2 s, past the caller’s 0.5 s timeout. The client retries with the same backoff as above.
Without a key, all 200 orders were “confirmed” and 26 of them reserved stock twice. With a UUID per order, sent as Idempotency-Key and reused across retries, inventory replays the stored result instead of committing again:
The 409 for a key that is still in progress follows the IETF Idempotency-Key header draft (draft-ietf-httpapi-idempotency-key-header-08, still an Internet-Draft), which also recommends 422 when a key is reused with a different payload. Stripe’s API works the same way and lets keys be pruned after at least 24 hours. In production the key store must be shared and durable (a table with a unique constraint, written in the same transaction as the reservation), not a dict in memory.
Step 3. The circuit breaker pattern
What is the circuit breaker pattern? Martin Fowler’s description: wrap the remote call, count failures, and after a threshold “trip” so further calls fail immediately without touching the dependency. After a timeout the breaker goes half-open and lets a trial call through. If it succeeds, the circuit closes; if it fails, the timer restarts. Michael Nygard popularized the pattern in Release It!.
Circuit breaker states with the transitions recorded in the run: open at 4.87 s, half-open at 5.87 s, open again at 6.87 s when the probe timed out, half-open at 7.89 s, closed at 8.39 s; below, inventory queue depth with the breaker drops to 0 while open, compared with up to 203 with backoff only
My version is count-based (5 consecutive failures), with a 1 s open period and one probe in half-open:

My first version did not have the _generation check or the CancelledError branch, and two tests caught real bugs:
- A late success closed the circuit. A request admitted while the breaker was closed can finish seconds later, after the breaker opened and went half-open. Its success then closed the circuit without any real probe. The generation counter discards results from calls admitted before the last transition.
- A cancelled probe jammed half-open. When the outer deadline cancels the probe, neither the success nor the failure path ran, so the single probe slot stayed taken and every later call got CircuitOpenError forever. Cancellation now counts as a failure.
The breaker sits inside the retry loop, so each attempt is counted and a retry against an open circuit fails instantly (retry() only retries TransientError, and CircuitOpenError is not one).
With the breaker, inventory got 769 calls instead of 985 and its peak queue was 68 instead of 218. While the circuit was open, the inventory queue drained to zero, which is the point: the struggling service gets room to recover. Two details from the transition log are worth knowing. The first probe at 5.87 s timed out, so the breaker reopened. At 8.39 s it closed while inventory was still in its slow phase, because by then the queue had drained and 0.5 s answers fit under the 1 s timeout. A breaker reacts to symptoms it can see, not to the fault itself.
Production breakers usually use a failure rate over a sliding window instead of consecutive failures. Resilience4j’s defaults, for reference: a window of 100 calls, a 50% failure-rate threshold, a 60 s wait in the open state and 10 permitted calls in half-open. Microsoft’s pattern guide describes the same three states.
Step 4. The bulkhead pattern
The bulkhead pattern is named after the partitions in a ship’s hull: if one compartment floods, the others stay dry. Microsoft’s guide suggests, for example, a separate connection pool per downstream service. In orders that means a separate httpx client per dependency plus a hard cap on concurrent inventory calls. Calls over the cap are rejected instead of queued:
The bulkhead rejected 197 calls, yet /product success went up to 79%. With at most 24 calls in flight, inventory’s queue never exceeded 24, its latency stayed under the 1 s timeout, and the requests that were admitted succeeded. The breaker never tripped in this run. That is not a reason to remove it: the bulkhead protects against “slow”, the breaker against “down”. A dependency that returns errors instantly never fills a bulkhead.
The number 24 came from the dependency, not from the caller: inventory has 16 DB slots, so a few more than 16 in flight keeps it busy without building a long queue. A bulkhead sized from the caller’s side (“we can afford 100 connections”) does not protect the callee.
Step 5. Fallback: degrade instead of fail
The last layer decides what the user sees when every other layer says no. For stock levels, the last known value is usually better than an error page, as long as it is labeled:
Every /product request got an answer: 80% fresh, 20% marked stale, zero errors. A fallback is a product decision: stale stock on a product page is fine, a stale balance in a payment flow is not. Without a safe fallback, fail fast with a clear error.
All steps side by side
Cheat sheet of the wrapper order in orders: fallback, deadline, bulkhead, retry with backoff and budget, circuit breaker, per-attempt timeout, then the HTTP call, with what each layer protects against
The order of the wrappers matters. In this lab, from the outside in: fallback, deadline, bulkhead, retry, circuit breaker, per-attempt timeout. The bulkhead is outside the retries so one request holds one slot for all its attempts. The breaker is inside the retries so every attempt is counted and an open circuit ends the retry loop at once.
Gotchas I hit
- Timeouts move work, they do not cancel it. Inventory’s queue doubled after Step 1 because abandoned requests kept running. Propagate deadlines downstream (the SRE book recommends deadline propagation) and let the server drop work whose caller is gone.
- Naive retries are worse than no retries. 24% vs 96% success in this lab, with collateral damage on /catalog.
- A shared client is a shared failure domain. One httpx.AsyncClient for all dependencies means one pool and one queue. Use one client per dependency.
- Breakers need tests for concurrency, not just state. Both bugs above only show up with overlapping calls and cancellation.
- Measure open-loop. My first load generator used httpx too. It fell behind schedule as soon as the system slowed, which hid the cascade. A closed-loop benchmark slows itself down exactly when production would not.
Production checklist
- Per-attempt timeout from the dependency’s p99.9, plus a deadline per incoming request that is propagated downstream.
- Retries only on transient errors: exponential backoff retry with full jitter, a retry budget, and retries at one layer of the stack only.
- Idempotency keys on every retried write, stored durably with a unique constraint.
- A circuit breaker per dependency, failure rate over a window, with its state exported as a metric and alerting on “open”.
- A bulkhead per dependency: separate pools, concurrency cap sized from the callee’s capacity.
- An explicit fallback or an explicit fast failure for each call site.
- A fault-injection test like this one in CI or staging, replayed after every change to these settings.
Wrap-up
The circuit breaker pattern is the most famous of the microservices resilience patterns, but in this experiment the timeout fixed the cascade, the retry budget prevented the retry storm, the bulkhead protected the slow service from its caller, and the fallback turned the remaining errors into labeled stale data. The breaker’s job was narrower: give a struggling dependency room to recover. Each pattern covers a different failure, and they only work together if they are applied in a deliberate order.
How do you size bulkheads and breaker thresholds in your services: from load tests, from production metrics, or by gut feeling? I would like to hear what works for you.



















Top comments (1)
The shared httpx pool detail really stood out to me — one slow dependency quietly starving /catalog is exactly the kind of cascade that's hard to spot from dashboards alone. I also liked that naive retries made things worse (24% vs 96% success); the retry budget modelled on gRPC throttling feels like the part most teams skip. Do you size the bulkhead cap of 24 from the callee's DB slots in production too, or do you start from load-test numbers and tune from there?