The setup
Our voice stack per call opens a handful of persistent connections: a WebSocket to our STT provider for streaming audio in, a WebSocket to our TTS provider for streaming audio out, a gRPC channel to the model provider for the dialog turns, and a connection out of a pool to Postgres for per-call state (transcript so far, slot-filling progress, escalation flags). Four things, per call, all needing to stay alive for the duration of the conversation.
In testing, four connections times fifteen calls is sixty connections. Nothing blinks.
Week 2 after the traffic ramp
We onboarded a partner whose call volume was meaningfully higher than anything we'd load-tested against. Not planned that way. Sales closed it, ops didn't loop us in until the contract was signed, standard story.
First day of real volume, we hit somewhere around 180-200 simultaneous calls during a lunch-hour spike. That's when things started falling over.
The symptoms didn't look like "connection pool exhausted" at first. They looked like:
- Calls connecting fine, audio starting fine, then going silent 8-15 seconds in, no error on the client side
- A subset of calls where the bot's response would start, cut off mid-sentence, and never resume
- Our internal dashboards showing p50 latency looking normal while p99 looked like it belonged to a different system entirely
The actual error, once we found it in the logs instead of the dashboards, was a lot more boring than the symptoms suggested: TimeoutError: QueuePool limit of size 20 overflow 10 reached, connection timed out, timeout 30.
Our Postgres pool. Twenty connections, ten overflow, sized for a service that in every test we'd run never asked for more than a dozen at once.
What was actually happening
Every call opened a connection at call-start to write initial state and held a reference to reuse for the duration, because we didn't want per-turn connection churn on every dialog step (that's a real cost, and avoiding it wasn't the wrong instinct on its own). At 200 concurrent calls, that's 200 things wanting a connection out of a pool sized for 30.
The 170 or so calls that couldn't get a connection didn't fail loudly. They sat in an await waiting for the pool, which from the audio pipeline's point of view looked exactly like a hang. The STT stream was still open, still buffering audio, but nothing was reading the transcript out of it because the coroutine handling that call was blocked upstream waiting on a database connection that wasn't coming.
That's the "silent 8-15 seconds in" symptom. That's also, separately, why the cut-off-mid-sentence calls happened: those were calls that got a connection, made progress, then lost it again when the pool churned under contention and a later query in the same call's turn couldn't get back in.
Concurrency where it broke, in our case: right around 180. Below that, degraded but recoverable. Above it, the failure mode compounded, because calls stuck waiting on the pool held their gRPC and WebSocket connections open too, which meant those pools started filling up with connections attached to calls that weren't making progress. One exhausted pool dragged the other three down with it inside maybe 90 seconds.
The fix: size the pool honestly, then bound everything with a semaphore anyway
Two changes. First, obvious one: the pool was undersized for the actual concurrency ceiling we needed, so we sized it properly against our target max concurrent calls, with headroom, and moved short-lived queries off the long-held per-call connection where we could (most per-turn writes didn't actually need to hold a dedicated connection for the whole call; only a few pieces of state did).
Second, less obvious one, and the one that actually mattered when the next traffic spike came: we stopped assuming the pool size was the only thing that needed a limit. We added an explicit concurrency gate in front of the whole per-call resource acquisition step, so that call number 201 doesn't get to start racing for a database connection and a gRPC channel at the same moment as calls 1 through 200. It waits, visibly, in a queue we control, instead of invisibly, in a queue Postgres controls.
import asyncio
import logging
import time
logger = logging.getLogger("call_admission")
MAX_CONCURRENT_CALLS = 220 # matched to pool sizing + provider concurrent-session caps
ADMISSION_TIMEOUT_S = 4.0
call_semaphore = asyncio.Semaphore(MAX_CONCURRENT_CALLS)
async def admit_call(call_id: str):
"""
Gate on total concurrent calls before we touch any downstream
resource (db pool, STT socket, gRPC channel). Waiting here is cheap
and visible. Waiting inside the db pool was neither.
"""
start = time.monotonic()
try:
await asyncio.wait_for(
call_semaphore.acquire(),
timeout=ADMISSION_TIMEOUT_S,
)
except asyncio.TimeoutError:
waited = time.monotonic() - start
logger.warning("call_admission_rejected call_id=%s waited=%.2fs", call_id, waited)
raise CallCapacityExceeded(call_id)
logger.info("call_admitted call_id=%s waited=%.2fs", call_id, time.monotonic() - start)
return True
def release_call(call_id: str):
call_semaphore.release()
logger.info("call_released call_id=%s", call_id)
class CallCapacityExceeded(Exception):
pass
async def handle_call(call_id: str, setup_fn, teardown_fn):
"""
Wraps the existing per-call setup (db connection, STT/TTS sockets,
gRPC channel) with the admission gate. On CallCapacityExceeded, the
caller routes to a degraded response (queue message or immediate
human transfer) instead of accepting the call and hanging it.
"""
try:
await admit_call(call_id)
except CallCapacityExceeded:
return "reject_to_overflow_queue"
try:
await setup_fn(call_id)
# ... normal call handling happens here ...
finally:
await teardown_fn(call_id)
release_call(call_id)
MAX_CONCURRENT_CALLS at 220 isn't a round number we picked for style. It's set slightly under the smallest of our four resource ceilings (db pool capacity, STT provider's concurrent-session cap, TTS provider's concurrent-session cap, and a gRPC channel limit we impose ourselves), so that admission control is always the first thing to say no, not the fourth.
The important part isn't the number. It's that there's now one place that says no, early and loudly, instead of four places that each partially say no by hanging.
What changed, measured
Before the fix: system degraded starting around 150-180 concurrent calls, and past roughly 200 it compounded into the kind of failure where new calls and existing calls both suffered, because starved resources cascaded across the four connection types.
After: we've run sustained load tests at 300 concurrent calls (above our real traffic ceiling on purpose, for margin) with the admission gate rejecting overflow cleanly into a degraded path (a "please hold, high volume" message plus a faster route to a human queue) rather than hanging silently. In our load tests, zero silent hangs at 300 concurrent, versus a system that started audibly breaking at 180 before the change. That's our number, from our tests, on our stack. Your mileage depends entirely on what your downstream providers cap you at.
What I'd tell myself before the partner call closed
Load test at the concurrency your contracts imply, not the concurrency your last test happened to use. That's obvious in hindsight and was apparently not obvious three weeks before it happened.
More specifically: any time a call holds a resource for its full duration instead of per-request, go check what happens at 10x your test concurrency before a partner's lunch-hour traffic checks for you. Fifteen calls times four persistent connections looked like nothing on our dashboards. Two hundred calls times four persistent connections found our weakest pool in about ninety seconds, and the only warning we got was a support lead asking why the bot had gone quiet mid-sentence.
Top comments (0)