With one user on the server, my voice AI answered in 0.9 seconds. With eight users, the last person in line waited 11.4 seconds, and it sounded broken. The model, the network and the speech vendor were all fine. The problem was one line, client = Anthropic(), called from inside an async def. A sync Anthropic client was blocking the asyncio event loop, and every other session on the box stopped while it ran.
I spent two days blaming the wrong things. Here's the whole story, with the numbers.
TL;DR
- Calling the sync
Anthropic()client inside anasync defhandler blocks the asyncio event loop for the whole generation. No other coroutine runs until it finishes: no WebSocket frames, no audio, nothing. - Testing with one user hides it completely. Lag scales with how many sessions are active at once: my p95 time-to-first-audio went from 0.9s with 1 session to 11.4s with 8.
- Find it with
loop.set_debug(True)plusloop.slow_callback_duration = 0.1, or with a 10-line loop-lag monitor in production. - Fix: switch to
AsyncAnthropicandasync forover the stream. Wrapping the sync call inasyncio.to_threadonly helps until the default thread pool fills (6 threads on a 2-vCPU box). - CPU-bound work like PDF parsing belongs in a
ProcessPoolExecutor, not on the loop and not in a thread.
What was I building?
The server runs voice interview sessions. Full disclosure: I built it. It's Preterview, which runs realistic voice interviews with three interviewer styles, scores a resume or portfolio, and writes a report at the end. Each session is a WebSocket: audio comes in, a transcript comes back from speech-to-text, Claude writes the interviewer's next line, and text-to-speech streams it out.
It's one FastAPI process under uvicorn on a 2-vCPU machine. That's one event loop serving every session at once. If you've written async Python, you probably know where this is going.
Why did single-user testing look perfect?
Blocking the event loop costs nothing when there's nothing else waiting. With one user, a 2.3-second blocking call just looks like a 2.3-second LLM call. Nobody else is queued behind it, so nothing looks wrong.
My local runs, my staging runs and my demo calls all had one person on the line: me. Everything measured 0.9s to first audio. I shipped it.
What did the bug look like in production?
The first complaints were "the interviewer froze for a few seconds" and "the audio stuttered." Not every session. Only at certain times of day.
My first guesses, in order:
- The speech-to-text vendor was slow at peak hours. (I moved regions. Nothing changed.)
- TTS chunks were too small. (I made them bigger. Stutter got worse.)
- The LLM was slow. (Claude's own timing for each call was flat across the day.)
That third clue was the one that mattered. The model took the same time at 3 a.m. and at 8 p.m. But the user waited much longer at 8 p.m. So the extra time was being spent inside my process, before or after the API call.
How do you detect a blocked asyncio event loop?
Turn on asyncio debug mode and lower the slow-callback threshold. asyncio will then log every step that holds the loop longer than the threshold, with the task that did it.
import asyncio
import logging
logging.basicConfig(level=logging.WARNING)
async def on_startup():
loop = asyncio.get_running_loop()
loop.set_debug(True)
loop.slow_callback_duration = 0.1 # seconds; default is 0.1 in debug mode
On staging, under a replayed load test, the logs filled with lines like:
Executing <Task pending name='Task-412' coro=<session_turn() running at app/session.py:88>>
took 2.312 seconds
Debug mode adds overhead, so I didn't want it in production. Instead I added a lag monitor. It sleeps for 100ms and checks how late it woke up. If the loop is free, it wakes on time. If something is hogging the loop, it wakes late, and the delay tells you how long the loop was blocked.
async def loop_lag_monitor(interval: float = 0.1, warn_ms: float = 50):
loop = asyncio.get_running_loop()
while True:
start = loop.time()
await asyncio.sleep(interval)
lag_ms = (loop.time() - start - interval) * 1000
if lag_ms > warn_ms:
log.warning("event loop lag %.0f ms", lag_ms)
In one week of production logs, I got 1,412 warnings over 100ms. When I matched them to the debug traces from staging, they broke down like this:
| Source of the block | Warnings | Share |
|---|---|---|
| Sync Anthropic client calls | 868 | 61% |
pypdf resume parsing |
325 | 23% |
A sync requests.post webhook + a time.sleep in a retry helper |
219 | 16% |
The longest single block was 7.9 seconds: pypdf grinding through a 14-page resume while every live interview on the box sat silent.
Why does a sync client inside async def block everything?
An async def only yields control at an await. A sync network call has no await, so the loop is stuck until the socket read returns. My code looked harmless:
client = Anthropic() # sync client
async def session_turn(ws, history):
with client.messages.stream(model=MODEL, max_tokens=400, messages=history) as stream:
for text in stream.text_stream: # blocks the loop waiting for each chunk
await ws.send_text(text) # yields, but only for a moment
The await ws.send_text made it look async. But for text in stream.text_stream is a sync iterator doing a blocking socket read between every chunk. In practice, the loop was frozen for the whole generation, apart from a few microseconds per send.
Every other session's incoming audio frames, outgoing TTS chunks and heartbeats waited in line. With eight people mid-answer at once, the last one in line waited for seven generations before their own started.
How bad was it under load?
I wrote a load test that replays recorded answers across N concurrent fake sessions and measures time-to-first-audio (TTFA). It counts from the moment the user stops talking to the moment the first TTS byte goes out. Here's p95 over 200 turns at each level:
| Concurrent sessions | Sync client | Sync client in to_thread
|
AsyncAnthropic |
|---|---|---|---|
| 1 | 0.9s | 0.9s | 0.9s |
| 4 | 5.2s | 1.0s | 0.9s |
| 8 | 11.4s | 3.4s | 1.0s |
The middle column is the part I nearly got wrong.
Is asyncio.to_thread enough to fix a blocking SDK call?
Only up to a point. asyncio.to_thread runs the call on the loop's default ThreadPoolExecutor, which since Python 3.8 has min(32, os.cpu_count() + 4) workers. On my 2-vCPU box, that's 6 threads.
My first fix was to wrap the sync stream in to_thread, and at 4 sessions it looked solved. At 8 sessions, two turns sat in the executor queue waiting for a free thread, and p95 rose to 3.4s. The loop was no longer frozen, but a new bottleneck had appeared: the thread pool.
You can make the pool bigger. But at that point you're running a thread per in-flight LLM call so you can keep using a client that wasn't built for your runtime. The SDK ships an async client. Use it.
What's the correct fix?
Use AsyncAnthropic and iterate the stream with async for, so each wait for the next chunk is a real await:
from anthropic import AsyncAnthropic
client = AsyncAnthropic()
async def session_turn(ws, history):
async with client.messages.stream(model=MODEL, max_tokens=400, messages=history) as stream:
async for text in stream.text_stream:
await ws.send_text(text)
That's a three-word diff, and it took p95 at 8 sessions from 11.4s to 1.0s.
The other 39% of the warnings needed different fixes:
-
pypdfresume parsing is CPU-bound pure Python. Moving it toto_threadkeeps the loop mostly responsive, but it still competes for the GIL with the loop thread. I moved it to aProcessPoolExecutor(max_workers=2)withloop.run_in_executor. Parsing a resume now costs other sessions nothing. -
requests.postto a webhook: replaced it withhttpx.AsyncClient. -
time.sleep(2)in a retry helper: changed toawait asyncio.sleep(2). That one was mine, it was in a "utils" file, and it had been freezing the server for two full seconds on every retry.
After the changes, the lag monitor logged 9 warnings over 100ms in the following week, all under 180ms, all from garbage collection pauses. That's down from 1,412.
How do I stop this from coming back?
Make it fail loudly instead of degrading quietly:
- Keep the lag monitor in production and alert when the p99 loop lag goes over 100ms. It's 10 lines and costs almost nothing.
- Load test with concurrency, not just latency. A single-session test can't catch this kind of bug. My CI now runs the replay at 1, 4 and 8 sessions and fails if p95 at 8 is more than 1.5x the value at 1.
-
Grep for the usual suspects in async code paths:
Anthropic(,OpenAI(,requests.,time.sleep, and sync file or PDF parsing. A lint rule that flags the sync clients in modules full ofasync defcaught two more before they shipped.
The lesson I keep relearning: in async Python, the slowest blocking call sets the latency for every user on the box, not just the one who triggered it.
So why did a sync Anthropic client block my asyncio event loop?
A sync Anthropic client blocks the asyncio event loop because its network reads are ordinary blocking calls with no await. Even inside an async def, the single loop thread is stuck until the response or the next stream chunk arrives, and every other coroutine waits. One user never notices. Concurrent users queue up behind each other: my voice server went from 0.9s to 11.4s p95 time-to-first-audio at 8 sessions. Detect it with loop.set_debug(True) and slow_callback_duration, or a loop-lag monitor. Fix it with AsyncAnthropic and async for, not asyncio.to_thread, which just moves the queue into a 6-thread pool. Send CPU-bound work to a process pool.
Written by the developer behind Preterview, an interview prep platform.
Top comments (0)