Setting a timeout does not give you a bounded operation. It gives you a number. Whether that number ever turns into an actual limit depends on three things the documentation rarely mentions: which phase of the call the timeout applies to, how much budget is left by the time the call runs, and whether the process really stops work when the limit fires. Most timeout bugs live in the gap between the number and the behavior, and they fail silently — the operation completes, just later than every promise you made upstream.
This article is a field guide to closing that gap. It covers the three timeout types that are actually different from each other, why per-call timeouts compose into unbounded totals, how to carry a deadline through a call graph instead, and what has to happen after a deadline fires for the whole mechanism to mean anything. None of it requires new infrastructure. All of it is a change in where the number is chosen and what the code does with it.
The Three Timeouts That Are Actually Different
When an HTTP client says "timeout", it usually means one of two things. A connect timeout bounds the phase before the request is sent: DNS resolution, TCP handshake, TLS negotiation. A read timeout bounds the silence between bytes once the request is in flight. Neither bounds the whole operation. A call can use its entire connect budget, then sit under its read budget, and the user experiences a duration that is the sum of both — or worse, the product, if the code retries.
The third kind is the one most clients hide: a total timeout that bounds everything, from the first byte of the request to the last byte of the response. Some libraries expose it explicitly. Some do not expose it at all, which means the only way to get one is to wrap the call yourself.
import httpx
# connect, read, write and pool are separate budgets
client = httpx.Client(
timeout=httpx.Timeout(
connect=3.0, # DNS + TCP + TLS
read=10.0, # silence between bytes
write=10.0, # silence while uploading
pool=3.0, # waiting for a free connection
)
)
Note what the configuration above does not contain: an overall limit. read=10.0 means ten seconds of silence, not a ten-second call. A server that trickles one byte every nine seconds for an hour satisfies that read timeout forever. If your mental model was "the request dies after ten seconds", the model was wrong, and nothing on the wire will tell you that. The total timeout is a different feature, and when the library does not ship one, it is your job to add it:
import asyncio
async def call_with_total(fn, total: float):
# one cap around everything, including the time spent waiting to send
return await asyncio.wait_for(asyncio.to_thread(fn), timeout=total)
A Per-Call Timeout Is Not a Deadline
Per-call timeouts look like they compose, and they do not. An HTTP handler that calls a database with a five-second timeout, inside a worker that has its own ten-second timeout, inside a queue consumer that waits fifteen seconds — that is a call graph whose worst case is the sum of every level, not the slowest one. Each layer protects itself, and none of them protects the user, because the user is the only participant with a real deadline.
The distinction that matters: a timeout is a limit on one operation; a deadline is a point in time by which a whole chain of operations must finish. If the chain has four hops and each hop gets its own generous timeout, the total is generous four times. The system is not slow because the timeouts are wrong. It is slow because the timeouts are permissive in the exact place where the sum is what the user experiences.
The fix is to stop treating timeouts as independent knobs and start treating them as a budget that is allocated once, at the entry point, and consumed as the work travels.
import asyncio
async def handle_order(request):
# the only place a real number is chosen
deadline = asyncio.get_running_loop().time() + 30.0
await charge_card(deadline)
await book_inventory(deadline)
async def charge_card(deadline: float):
remaining = deadline - asyncio.get_running_loop().time()
if remaining <= 0:
raise DeadlineExceeded()
await asyncio.wait_for(payment_provider.charge(), timeout=remaining)
Carry the Deadline, Convert at the Boundary
The pattern above is the whole trick: every function receives the deadline, computes the remaining budget when it actually needs a timeout, and converts that remaining budget into the timeout it passes to whatever it calls. The conversion happens at the boundary, so each layer still speaks the language of its own library — wait_for wants seconds, a socket wants a deadline, a database driver wants its own timeout object — while the total stays bounded by the number chosen at the entry point.
Two properties make this work. First, the deadline is never reset: no function along the path gets to start a fresh countdown, because a fresh countdown is exactly how the sum of the per-call timeouts sneaks back in. Second, the check happens before the call, not after: waiting until a timeout fires to discover the budget is gone is waiting for the failure you were trying to prevent.
The subtle part is the check itself. remaining <= 0 looks like a boundary check, but the real cost is inside the call: a call that starts with one millisecond of budget left will burn the entire underlying timeout machinery before failing, and it will fail with a confusing error instead of the clean decision the caller needs. Subtract a safety margin. If the remaining budget is smaller than what a meaningful operation needs, fail fast with the degraded path, not with the timeout handler.
The Fan-Out Trap and the Retry Trap
Parallelism hides the same problem in the other direction. Ten calls issued with asyncio.gather, each with its own five-second timeout, complete in five seconds in the worst case — the budget is not multiplied, because the calls overlap. But the moment a retry loop wraps those calls, the arithmetic changes. Three retries of a five-second call is fifteen seconds of worst case, and if the retry loop sits inside a caller that also retries, the totals compound exactly like the nested timeouts above.
Retries are where budgets die. The standard loop — try, sleep, try again, with a backoff factor — has no concept of a deadline, so it will happily spend fifteen seconds when the product promised three. The fix is to make the retry consume the same budget as the call.
async def call_with_budget(operation, deadline: float, attempts: int = 3):
loop = asyncio.get_running_loop()
for attempt in range(attempts):
remaining = deadline - loop.time()
if remaining <= 0:
raise DeadlineExceeded()
try:
return await asyncio.wait_for(operation(), timeout=remaining)
except (TimeoutError, TransientError):
if attempt == attempts - 1:
raise
# backoff comes out of the same budget
await asyncio.sleep(min(0.5 * (2 ** attempt), remaining / 2))
The sleep is the detail that usually gets skipped. It is real time, it is inside the user's deadline, and it must be charged to the budget like everything else. A backoff that waits two seconds after the budget expired is a bug that only shows up in production, because in tests the retry loop always succeeds on the first attempt and nobody ever sees the sleep.
Cancellation Is the Other Half
A timeout that fires but does not stop work is a timer, not a limit. In asyncio, wait_for cancels the task when the time is up, and cancellation is the mechanism that actually enforces the contract — the coroutine is asked to unwind, its finally blocks run, and control returns to the caller. The enforcement fails when the code being cancelled refuses to be cancelled. Catching CancelledError and continuing is possible, and it converts a timeout into a stall that the process can never escape.
# the anti-pattern: catching CancelledError defeats every timeout above you
try:
result = await slow_operation()
except asyncio.CancelledError:
cleanup() # fine
await finish_work() # not fine — you are now the hung process
raise
The same logic applies to CPU-bound work. A timeout cancels a coroutine at its next suspension point; a function that spins in a loop never suspends, so the cancellation waits at the gate. The answer is not a bigger timeout — it is to put CPU-bound work where the process can actually abandon it: a worker process, or explicit progress checks inside the loop that respect the deadline and raise on their own.
What Happens When the Deadline Fires
The last question is the one most designs forget: what does the caller receive? The answer is never "nothing". A user waiting on an operation that will not complete needs a decision — a degraded result, a cached value, an error the UI can render — and the deadline is the moment that decision gets made on your terms instead of the user's.
There is a second layer to this. A dependency that is slow is often a dependency that is down, and if every request waits out the full timeout before failing, a slow dependency becomes an outage: the timeout is spent, then the retry spends another, and the process fills with requests that are technically not stuck but are not doing anything either. A circuit breaker caps the damage by failing fast once the timeout has fired a few times in a row, instead of paying the full price on every request while the dependency recovers.
class CircuitBreaker:
def __init__(self, threshold: int = 3, cooldown: float = 30.0):
self.threshold = threshold
self.cooldown = cooldown
self.failures = 0
self.opened_at = 0.0
def _open(self) -> bool:
return self.failures >= self.threshold
async def call(self, operation):
if self._open():
raise ServiceUnavailable()
try:
result = await operation()
self.failures = 0
return result
except TimeoutError:
self.failures += 1
raise
Measure the Contract, Not the Knob
A timeout value is a promise with a number on it, and promises need measurement. The two numbers that matter are the timeout itself and the percentile of call durations: if the p99 of a call sits close to its timeout, the system is returning errors that nobody is counting, because the requests that time out are precisely the ones that never make it into the duration histogram. The timeout is not a performance target; it is an upper bound, and durations that live near it are a finding, not a configuration.
Timeout outcomes deserve their own counters, separately from success and failure. How many requests hit the deadline? How many of those hit it on the first attempt versus the third? How many were saved by the degraded path? These are the numbers that tell you whether the contract is being kept, and they are invisible in a dashboard that only shows average latency — the average is pulled down by fast successes while the timeout stream stays hidden in the tail.
Tests have the same blind spot. A timeout test that waits for real seconds is slow, flaky, and never exercises the interesting cases — the deadline consumed by retries, the cancellation that cleans up mid-flight. Time-based code is testable by faking the clock: inject a clock that can be advanced, and every timeout path becomes a deterministic test instead of a sleep-and-hope.
def test_deadline_is_consumed_by_retries(fake_clock):
calls = []
async def flaky():
calls.append(fake_clock.now())
raise TransientError()
with pytest.raises(DeadlineExceeded):
fake_clock.run(
call_with_budget(flaky, deadline=fake_clock.now() + 2.0, attempts=10)
)
# the backoff ate the whole budget: the loop exited before the final attempt
assert len(calls) == 2
A timeout is a contract between the process and its caller. The number is the easy part; the contract is the phase it applies to, the budget it consumes, the work it actually cancels, and the answer the caller gets when it fires. Set the number once, carry the deadline through the call graph, cancel real work when it trips, and measure the distance between the promise and the behavior. That distance, not the knob, is the thing worth engineering.
Originally published on Dispatch.
Top comments (0)