I had a tool that usually answered in a few seconds, and sometimes took 44. For a batch job that's annoying. For a tool called by AI agents, which typically wait around 30 seconds for an answer, it's broken: the agent gets "still running" and the user gets nothing.
The upstream service was rate-limiting me with HTTP 429 Too Many Requests. My first instinct was "add better backoff". That was the wrong instinct. Here's what was actually happening, and what fixed it.
Where the 44 seconds went
I pulled the run log and lined up timestamps. The time broke down roughly like this:
- ~18 s: opening the first connection through a proxy session stalled. Nothing failed; it just hung.
- ~12 s: after a 429, opening a replacement session was slow again.
- ~7.5 s: at the very end, the process waited up to 5 seconds for background listeners before exiting.
The actual rate-limit waits were a small part of it. Most of the time was spent waiting on things that were never going to succeed quickly.
The sneaky one was the first. I had a 15-second timeout configured, but it was a read timeout. It limited each read, not the whole request. A connection that trickled bytes slowly, or stalled during setup, could blow straight past it.
Fix 1: Total-time budgets, not just read timeouts
Most HTTP clients let you set separate timeouts for connect, read, write and pool. Set all of them, and also wrap the whole operation in an overall deadline:
import asyncio
import httpx
TIMEOUT = httpx.Timeout(connect=5.0, read=7.0, write=5.0, pool=5.0)
async def fetch(client: httpx.AsyncClient, url: str) -> httpx.Response:
# Hard cap on the whole request, including slow trickles.
return await asyncio.wait_for(client.get(url, timeout=TIMEOUT), timeout=8.0)
I also capped each query (which needs several requests) at 16 seconds. If it can't finish, the tool returns a clear error item instead of hanging. A fast, honest failure is better than a slow success the caller never sees.
Fix 2: On 429, switch, don't wait
Classic advice for 429 is exponential backoff: wait 1 s, 2 s, 4 s... That's right when you are the problem and the server needs you to slow down globally. It's wrong when the limit is tied to an identity (an IP or session) and you have another identity available.
My tool runs through rotating proxy sessions, so the fix was:
- Keep a spare session already open.
- On a 429, a stall or a network error, switch to the spare immediately and start opening a new spare in the background.
- Only back off (briefly) if all sessions are limited.
async def get_with_failover(pool, url):
for attempt in range(3):
session = await pool.current()
try:
resp = await fetch(session, url)
if resp.status_code != 429:
return resp
except (httpx.TransportError, asyncio.TimeoutError):
pass
pool.rotate() # promote the warm spare
pool.warm_spare() # start opening the next one, don't await
await asyncio.sleep(0.5) # everything limited: short pause, then last try
return await fetch(await pool.current(), url)
The important part is that the replacement is already warm. Switching to a session you still have to open just moves the wait somewhere else.
Fix 3: Race the start
Opening the first session was the slowest, least predictable step. So I open two at startup and use whichever is ready first; the other becomes the spare.
async def first_ready(*coros):
tasks = [asyncio.create_task(c) for c in coros]
done, pending = await asyncio.wait(tasks, return_when=asyncio.FIRST_COMPLETED)
winner = done.pop().result()
return winner, pending # keep the pending one as the spare
This costs a little extra (one more connection per run) and removes most of the tail.
Note what I didn't parallelize: the main requests themselves. In my case the second request needs a token returned by the first, so they're inherently sequential. Racing works on independent steps, not on a dependency chain.
Fix 4: Don't let shutdown eat your budget
The last 5–7 seconds were the process waiting for background tasks to finish before exiting. I capped that wait at about 1 second and made closing old sessions happen in the background. Exit now takes around 0.3 s.
It sounds trivial, but for short tools, teardown can be a surprisingly large share of total time.
Results
| Before | After (10 test runs) | |
|---|---|---|
| Typical run | a few seconds | 6.5 s (p50) |
| Slow runs | up to 40–44 s | 8.0 s (p95) |
| Average cost per run | $0.00035 | $0.00011 |
Typical runs were already fine before. The tail collapsed, and that's what matters for anything with a deadline. Cost also dropped by about two-thirds, because runs that finish faster use less compute.
Takeaways
- Measure where the time goes before tuning backoff. My problem was stalls, not rate limits.
- Set total-time budgets, not just read timeouts.
- On 429, switch identity if you can; back off only when you can't.
- Keep a warm spare so switching is instant.
- Race independent slow steps; keep dependent steps sequential.
- Cap shutdown time.
- Fail fast and clearly when the budget is gone.
This came out of building agent-friendly data tools (the one above is a Google Trends API on Apify). If you've found other ways to tame tail latency against rate-limited APIs, I'd love to hear them.
Top comments (0)