I Fixed Routing. Then I Found Out Routing Was Never the Real Problem
From "the pods keep dying" to "oh, it's the architecture"
Where We Left Off
In my last post I replaced a Redis session-affinity lookup with Istio consistent hashing. WebRTC drops stopped. We saved money. I felt clever.
Then we tried to actually scale Jarvis.
Jarvis is our real-time voice agent. It brokers a Daily.co WebRTC room per call and runs a pipecat pipeline: user speaks → VAD → STT → LLM → TTS → bot speaks. The LLM lives in Stark, our orchestrator. Jarvis is the audio edge. One pod served several calls at once, like any normal web service.
Here is what "normal web service" thinking got us:
jarvis-xxxxx RESTARTS: 1 LAST STATE: OOMKilled
heap: pressing against the memory limit
OS threads: in the hundreds
live calls: a handful
A handful of calls. Hundreds of threads. Consistent hashing had nothing to do with it.
This post is the autopsy: three problems, what each one did to CPU and memory, and the one variable they all share. Part 2 is the rebuild with Agones and Karpenter, one pod per call.
What One Call Costs
Per call: one WebRTC peer, two vendor WebSockets, one SSE stream, one ONNX model scoring audio about 30 times a second, and a pile of resampling. All inside one Python process, under one CPU limit, sharing one memory limit.
That limit was small. A few hundred millicores and well under a gigabyte, because that's what a web service pod looks like. Everything below is a fight over it.
Here is what that fight looked like in prod, back when jarvis was a plain Deployment. Three pods sharing calls, two days, real data:
Two things to hold onto. The CPU panel sits near zero and jumps in short spikes, and every spike is a call doing something: someone talking, a call starting, a TTS burst going out. The memory panels are a staircase. We'll come back to both.
Problem 1: VAD Isn't CPU-Hungry. It's CPU-Sensitive.
VAD decides when the user has stopped talking. In pipecat that's Silero, an ONNX model scoring every short chunk, plus a stop_secs timer: once the score drops below confidence for stop_secs seconds, the turn ends and the transcript goes to the LLM.
@dataclass
class VAD:
confidence: float # how sure the that this is speech
start_secs: float # how long speech must persist before "user started"
stop_secs: float # how long silence must persist before "user stopped"
min_volume: float # below this, don't even ask the model
The catch: stop_secs is wall-clock time, but the chunks arrive through a process that can be throttled. Late chunks look like silence.
What happened on our pods:
- Loading Silero costs tens of ms of CPU per call. Under the pod's CPU limit that stretched to several times longer in wall time, on the start-call path.
- Look at the CPU panel again. Between turns a pod sits near zero. When someone talks it spikes, and the spike is short. The kernel doesn't bill the average. It bills 100 ms slices against the quota.
- When that happens, the logs look like this:
[VAD] user stopped speaking (stop_secs of "silence": really stop_secs of late chunks)
[STT] flush → LLM "I was going to say that"
[VAD] user started speaking (the chunks finally arrive)
[STT] flush → LLM "we should probably"
→ two LLM turns for one sentence. User hears two replies.
We called these "duplicate messages". The first fix is in git history and I'm not proud of it:
bump STTToTextProcessor response timeout (band-aid for duplicate messages)
Over the next few months we also loosened confidence and min_volume and shortened stop_secs, because VAD "wasn't firing". Some of that was real. Some of it was us tuning a model to compensate for a starved process.
CPU: VAD's own cost is small. Its sensitivity to throttling is total. It's the canary.
RAM: a few MB per analyzer. Not the memory problem.
Problem 2: STT/TTS Traffic Is Where the CPU Goes
My first theory was Python. Single-threaded, GIL, two calls step on each other, done. I had that paragraph written. Then I remembered we'd run TTS to a bunch of devices off one process without anything choking.
So I measured it. Two threads of pure Python loops take turns, textbook GIL. But the audio work a call does, resampling and VAD inference, is C and C++ underneath. It drops the GIL and runs on two cores just fine. The Daily client's threads don't ask Python at all.
Python wasn't the bottleneck. The pod's CPU budget was. Parallel threads don't get around a budget. They spend it faster.
Where a call's CPU goes:
| Stage | Work | When |
|---|---|---|
| Mic → STT | chunk, frame, send, parse transcripts, decide when to flush | while the user speaks |
| TTS → pod | receive, resample, cut into 20 ms frames, hand to the native client | while the bot speaks |
| Native client | Opus encode/decode, jitter buffer, WebRTC | always, both legs |
| Vendor failover | tear down one WebSocket, open another, mid-turn | rare, expensive |
None of it is heavy alone. Each one is a spike on that CPU panel. Put calls together on one pod and it goes like this:
Our load tester Voldemort, exists because of this table. One Locust user is one whole call: real STT, mock TTS and LLM with measured vendor latencies baked in. An instant mock makes a pod look several times more capable than it is.
CPU: the bulk of the in-call burst.
RAM: a bounded working set per live call. If this were the only memory cost we'd never have OOMed.
So why did we OOM?
Problem 3: The Daily.co Thread Leak
Python asyncio code doesn't make hundreds of threads. Something native does.
daily-python wraps Daily's C++ WebRTC client. Every call creates one, and every client starts roughly ten OS threads. You'll see names like SoupSignalling and mediasoup-manag in /proc/<pid>/task/*/comm. Those threads and the client's audio buffers are freed by exactly one thing: daily.release().
In pipecat, release() runs at the very end of pipeline teardown, after an EndFrame has flowed through every processor. Our teardown under load:
end-call arrives
→ stop_pipeline()
→ queue EndFrame
→ wait for runner ... timeout ...
→ task.cancel() ← "fallback if the pipeline does not finish"
→ drop transport reference
→ gc.collect()
task.cancel() cancels the Python coroutine. It does not call release() on the C++ client. The Python object gets collected. The ten threads and the buffers don't. They were never Python's to collect.
And once we dropped our reference, our "active transports" gauge went back to zero. Our metric said clean. The kernel said hundreds of threads.
Back to the screenshot from the top. CPU is quiet. Memory is a staircase, and there isn't a single step down in two days. Every step is a call that ended and left something behind. Three pods, same slope, so not one bad pod or one bad user.
The Fixes That Weren't
Every one of these felt like progress at the time:
| Attempt | What it did | Why it wasn't enough |
|---|---|---|
| Double the memory limit | OOMed slower | Leak is unbounded |
memray in prod |
Clean heap snapshots of Python objects | Leak was native |
gc.collect() after every cleanup |
Reclaimed the Python cycle | No effect on C++ threads |
| jemalloc | RSS tracked real usage | Can't free what was never freed |
Redis snapshots + /resume-call
|
Calls survived a pod crash | We built resume-on-crash instead of not crashing. Deleted later |
| Istio consistent hashing |
end-call reaches the right pod |
Until a scale event moves the ring. Then end-call lands on the wrong pod and the call runs to the watchdog |
That last row is my previous post. Hashing was correct. It only mattered because one pod held state for many calls. We solved routing to a stateful pod without asking why the pod was stateful for more than one call.
The actual leak fix is small:
# daily.release() is the only thing that reclaims the native client's
# threads and buffers. Never cancel it.
release = spawn(transport.cleanup(), name=f"daily-release-{session_id}")
for timeout in (_RELEASE_TIMEOUT_SECONDS, _RELEASE_GRACE_SECONDS):
try:
await asyncio.wait_for(asyncio.shield(release), timeout=timeout)
return
except TimeoutError:
logger.warning("[MEMORY] Daily release slow, still waiting")
Shield the release. Prune the transport entry only after it completes, so the gauge stays honest. That stopped the bleeding. It didn't change the fact that one pod was juggling several realtime pipelines on a web-service CPU limit.
The Comparison That Changed the Architecture
| Problem | Bottleneck | Per call | Scales with | Shows up as |
|---|---|---|---|---|
| VAD jitter | CPU (sensitivity) | tiny, first to feel a throttle | calls per pod, superlinearly | duplicate turns |
| STT/TTS traffic | CPU (volume) | most of the burst | calls per pod, linearly | throttled periods, late TTS |
| Daily thread leak | RAM + threads | tens of MiB, ~10 threads per leaked call | calls ever served by the pod | OOMKilled |
This reframed it to:
"What if a pod only ever served one call? Like a game server. One match, then it exits."
Then nothing leaks across calls, because there is no across calls. VAD never competes with another call's TTS, because there are no neighbours. CPU per pod is one call's spikes and nothing else. Memory per pod is one call's working set, and then the process is gone. Routing is "the pod I was given".
The problem collapses from "make a multi-tenant realtime Python process behave" to "hand out fresh processes fast enough". That's a scheduling problem, and the game industry solved it years ago.
If You're In My Shoes
If you run a native SDK in Python on Kubernetes and you see threads creeping up while your metrics say clean, memory that only goes up, or endpointing that's flaky only under load, don't start with the model params. Count the threads.
# 1. threads, not memory
kubectl exec <pod> -- ls /proc/1/task | wc -l
# 2. what the threads are
kubectl exec <pod> -- sh -c 'cat /proc/1/task/*/comm | sort | uniq -c | sort -rn | head'
# 3. real heap vs cache
kubectl exec <pod> -- grep -E '^(anon|file) ' /sys/fs/cgroup/memory.stat
For CPU, watch throttled periods, not the usage average. The average is what the spikes look like after Grafana flattens them:
rate(container_cpu_cfs_throttled_periods_total{container="<app>"}[5m])
/ rate(container_cpu_cfs_periods_total{container="<app>"}[5m])
And one more, because container_oom_events_total read zero in our cluster even for pods that were genuinely OOMKilled:
max_over_time(kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}[14d]) > 0
Part 2: how we rebuilt Jarvis as a one-pod-one-call fleet on Agones with Karpenter supplying nodes, the warm-pool formula N = x + 3√x, the three ways a pod can leak out of the pool, and the arithmetic to 10,000 concurrent calls.
It's not as clever as consistent hashing. It's a lot more boring. In the best way.
Top comments (4)
The thread leak hitting the C++ WebRTC client rather than the Python pipeline makes complete sense. When a pipeline teardown cancels the outer asyncio task on a timeout, Python treats the coroutine as dropped and decrements its own object references. But native threads spawned inside a shared library like mediasoup or WebRTC run entirely outside the runtime's awareness, so unless the wrapper's destructor or an explicit release function runs to completion, those worker threads stay attached to the Linux process table indefinitely.A similar trap happens with local audio buffers and resampling contexts when workers are terminated abruptly. If the native C++ layer allocates ring buffers through raw malloc or internal thread-local pools, Python garbage collection never triggers the corresponding free calls. Over twenty or thirty terminated calls, you end up with hundreds of ghost threads and a steadily climbing RSS footprint even though memory profilers report a flat heap.
Exactly this. The part that cost us the most time was the "flat heap" bit: memray in prod showed nothing wrong, because it only sees Python allocations. The thread count from /proc was the first metric that disagreed with our own gauges, and it was right.
Part 2 covers what we did so a pod never lives long enough to accumulate this in the first place.
Did the OS thread count come from the pipecat pipeline itself, or from the STT and TTS clients? Curious which one you ended up pinning down before Part 2.
It came from the daily.co. Sure I'll drop the part 2 this week only.