Scaling Voice Agents to 10,000+ Calls - Part 2: One Pod, One Call, with Agones

The Game Industry Solved My Problem Years Ago. I Found Out This Year.

Part 2: how we stopped fighting a multi-tenant Python process and started handing out fresh ones

Where We Left Off

Part 1 ended with a table. Three problems, VAD jitter, STT/TTS CPU bursts and a Daily.co native thread leak, and every one of them scaled with the same variable: how many calls one pod touches.

pod serving several calls:   hundreds of threads, heap at the limit, noisy neighbours, OOMKilled
pod serving one call:        one call's spikes, bounded memory, then it's gone

"The question that reframed it was simple: what if a pod only ever served one call, like a game server?"

So I looked at how game servers do it. The answer is Agones, a Kubernetes controller from Google and Ubisoft whose whole job is to keep a pool of ready game-server pods, hand exactly one to a match, and make sure nobody kills it mid-game.

Swap "match" for "call" and read that again. It fits, down to the failure modes.

This post is the build: the shape, the warm-pool math, the Karpenter settings that will drop a live call if you get them wrong, the two ways a pod leaks out of the pool, and copy-paste configs at the end.

The Shape of It

Part	Owns
Agones Fleet	N Ready pods and the Ready → Allocated → Shutdown state machine
FleetAutoscaler	keeps replicas = allocated + bufferSize
Karpenter NodePool	arm64 nodes appear when pods are Pending, vanish when empty
Stark (orchestrator)	allocates a pod, talks to it directly at nodeIP:hostPort, remembers the address
Jarvis	Ready() when warm, Health() heartbeats, Shutdown() when the call ends

Gone: the Istio DestinationRule from my last post, the Service as the call path, the HPA, the Redis routing map. Affinity stopped being a hash ring and became a stored address. Stark writes down the address it was given and end-call goes there.

The Lifecycle Is the Whole Design

Scheduled ──▶ Ready ──▶ Allocated ──▶ Shutdown
   boot        warm      in a call      deleted, backfilled

Two properties make this safe for calls:

• Allocation is atomic. Ready → Allocated flips once. Agones never hands the same pod to two callers, so "one pod one call" is an invariant, not a convention.
• Release is cooperative. Only the pod can leave Allocated, by calling Shutdown(). Agones never reclaims an Allocated pod for a scale-down or a rollout. A scale-down can never hang up on a user.

On the jarvis side that's three verbs on the SDK, ready(), health() and shutdown(), plus one rule that took me a while to internalize: the pod never exits itself. It calls Shutdown(), Agones deletes it.

The Warm Pool: N Ready Pods, Always

The first question everyone asks is "how many idle pods do you keep?" The intuitive answer, "enough for concurrent calls", is wrong. The Fleet already grows to allocated + N. Concurrency is handled by growth. The warm pool only has to cover the calls that arrive while a replacement pod is still booting.

So N is a function of two things: how fast calls arrive, and how long a fresh pod takes to reach Ready. Today N is a fixed number for us, tuned from load tests with headroom on top. Making it follow arrival rate automatically is the next step, and the FleetAutoscaler has a webhook policy for exactly that.

What this says about cost: cold start is the cost driver, not traffic. Halve the time to Ready and you halve the idle pool. So that is where the effort went:

Cold-start item	Before	After
pipecat import	lazy, on the first call	eager, at boot, before Ready()
Silero VAD load	on the start-call path	loaded at boot, first call adopts it
Daily room + token	REST round-trip on every start-call	prewarmed at boot
GC	everything tracked	gc.collect(); gc.freeze() after boot, so per-call collects scan only per-call garbage
Node provision	about a minute on top of pod boot	still about a minute, which is what the pool is sized for

In production, picking a pod takes tens of milliseconds at p95. The user doesn't know the pod didn't exist for them a moment ago.

Sizing a Pod, and Then a Node

Part 1's lesson was that CPU trouble lives in the spikes, not the average, and spikes from neighbouring calls collide. One pod per call removes the neighbours. What's left is sizing the pod so its own spikes never get throttled:

# Guaranteed QoS: requests == limits, set from one call profiled on its own.
resources:
  requests: { cpu: <one call's burst>, memory: <one call's peak working set> }
  limits:   { cpu: <one call's burst>, memory: <one call's peak working set> }

The node shape was a decision, not a default. One call's profile is lopsided: a meaningful slice of CPU for a modest amount of memory, roughly a 1:2 ratio of cores to GiB once the pod is sized Guaranteed. General-purpose instances come at 1:4, so a node packed with jarvis pods runs out of CPU with half its memory still empty, and you pay for RAM nobody uses. Compute-optimized Graviton, the C family, is 1:2.

Burstable (small request, bigger limit) packs about twice as many pods per node, and puts far more spike demand on the node than it has cores. The extra nodes are the price of never throttling a live call.

Karpenter: Four Settings That Will Drop a Live Call

A NodePool is a rule: "if a pod needs capacity like this, you may build a node like this." Nodes appear when pods are Pending and vanish when they empty.

1. Only remove a node when it is empty. Karpenter has two modes for cleaning up nodes. WhenEmptyOrUnderutilized says: if a node is only half used, move its pods somewhere else and delete it. That works for web servers, because any pod can be replaced by any other. It does not work here. "Moving" a jarvis pod means killing it, and there is a person on the call. So we use WhenEmpty: Karpenter may delete a node only when nothing is running on it.

2. Fill nodes up before starting new ones. Rule 1 has a catch. If Agones hands out calls evenly across all nodes, no node ever becomes empty, so no node is ever removed, and you keep paying for every node you ever created. scheduling: Packed on the Fleet tells Agones to put new calls on the fullest node first.

3. Tell Karpenter directly: do not touch this pod. Agones already marks a pod in a call as "do not evict", but it uses the annotation the old cluster-autoscaler understands. Karpenter ignores that one.

4. No spot instances. Spot gives about two minutes of warning before AWS takes the machine back. A warm pod turns into a live call in milliseconds, and a call lasts longer than two minutes. There is no pod in this pool that is safe to lose, so every node is on-demand.

The Two Ways a Pod Leaks Out of the Pool

"Only the pod can release itself" is a feature right up until the pod doesn't. A missed Shutdown() is permanent. The GameServer stays Allocated, still counts toward allocatedReplicas, and the autoscaler provisions a replacement for it.

How often does a pod get claimed and then not used? More than you'd guess. Here's one week of prod, about ten thousand calls started versus calls where the user actually reached the room:

The gap has two causes, and we see both every day:

(a) Started, but never joined. Stark allocated the pod, start-call landed, the pipeline is up and the Daily room is waiting. The participant never arrives. The idle timeout is what saves it, the pod tears the session down and calls shutdown() itself.

(b) Joined, then hung up before the call finished starting. The participant comes in and leaves while the pipeline is still being built, so end-call races start-call.

The alert that catches both: Allocated count versus Stark's live-call count. A persistent gap is a leaked pod.

What Changed

Same production environment, shared pods before, one-call pods after:

	Before	After
Threads per pod	hundreds	dozens
Heap per pod	at the limit, OOMKilled	bounded, no OOMKills since cutover
CPU	spikes colliding across calls	one call's spikes, Guaranteed QoS, no neighbours
Pod affinity	Istio hash ring, reshuffles on scale	the address Agones returned
Allocation latency	n/a, Service round-robin	tens of ms at p95
Capacity model	"how many calls fit in a pod?"	pods per node, linear
Risk to live calls from scaling	HPA + default scheduling	WhenEmpty + Packed + do-not-disrupt + on-demand

The architecture didn't fix the thread leak. The shielded release() from Part 1 did. What the architecture did is make the class of bug irrelevant. A process that serves one call and dies can't accumulate anything.

Three Things I Got Wrong

1. initialDelaySeconds lower than the cold start. The Agones health check counts from pod start. The heartbeat starts when lifespan startup finishes, tens of seconds in. Agones started checking, saw nothing, and killed every pod a few seconds before it would have gone Ready. No error anywhere. Just a Fleet at 0 Ready, forever.

2. Leaving the managed node group next to the Karpenter pool. In dev we kept an old node group with the same label and taint "while it's phased out". Nothing scales a managed node group, so overflow landed on Karpenter and Karpenter nodes stuck around because Packed kept filling them. Two schedulers, one label. Pick one.

3. Agones' Helm pre-delete hook. Upstream ships a hook that deletes every GameServer in the managed namespaces on chart delete. ArgoCD runs it whenever the Application is deleted, including an ApplicationSet regenerating it, and preserveResourcesOnDeletion doesn't stop hooks. On prod that is every live call, ended by a GitOps refactor. agones.crds.cleanupOnDelete: false, for good.

If You're In My Shoes

If you run realtime, stateful, one-user-at-a-time workloads, voice agents, video rooms, browser sessions, anything with a native SDK that holds threads, stop trying to make a multi-tenant process behave and look at Agones. It was built for exactly "hand out one fresh process per session and never kill it mid-session".

Three rules:

1. The pod never exits itself. Shutdown(), then let SIGTERM run your teardown.
2. Every path to an Allocated pod ends in Shutdown(). Failed starts included. Alert on Allocated versus your own live-session count.
3. WhenEmpty + Packed + do-not-disrupt + on-demand. Drop any one and Karpenter will eventually hang up on someone.

Jarvis: the lifecycle. Python, the shape of it:

# lifespan startup
await transport_service.prewarm()     # Daily room + token, off the critical path
prewarm_vad()                          # Silero, off the critical path
gc.collect(); gc.freeze()
await agones.ready()                   # LAST: joins the pool only when actually warm

# every call-ending path converges here
await publish_call_terminated(...)     # the end event first, billing depends on it
await cleanup_session(...)             # EndFrame → Daily release(), shielded, never cancelled
await agones.shutdown()                # hand the GameServer back. Do NOT os._exit.

Watch these

# warm pool depth, alert when it gets thin
sum(agones_gameservers_count{fleet_name="<fleet>", type="Ready"})

# leaked Allocated pods, alert when this stays above your live-call count
sum(agones_gameservers_count{fleet_name="<fleet>", type="Allocated"})

# the autoscaler is actually allowed to scale (0 = frozen at current replicas)
min(agones_fleet_autoscalers_able_to_scale)

# throttling on a Guaranteed pod should be ~0
sum(rate(container_cpu_cfs_throttled_periods_total{container="<app>"}[5m]))
  / sum(rate(container_cpu_cfs_periods_total{container="<app>"}[5m]))

# nodes the fleet is holding
count(count by (node) (container_memory_working_set_bytes{container="<app>"}))

---

Two posts ago the answer was "hash the session header". One post ago it was "count the threads". This time it was "stop sharing the process".

Every one of those was right for the problem in front of me. None was the last answer. That's how infrastructure evolves, and honestly, watching a Fleet backfill a pod in the time it takes to say hello is a pretty good feeling.