Serving 500 concurrent LLM chats on one 4-core box with tier-aware queueing

zhenjie zhang — Fri, 07 Aug 2026 09:32:04 +0000

TL;DR — When traffic spikes on a shared LLM backend, a naive concurrency limit lets free-tier users starve paying users. This post walks through why our first solution (a global asyncio.Semaphore) broke, and how a small Redis-backed tier-aware slot manager brought p99 latency during peaks from 20+ seconds down to under 2. The core code is ~40 lines. There's no magic — just a bit of fairness.

What we're running

An AI companion app: users chat with LLM-powered personas, with pgvector-backed long-term memory. Nothing exotic on the infra side:

FastAPI behind Uvicorn (8 workers)
Postgres 16 + pgvector (via PgBouncer)
Redis for cache and queueing
R2 for media
A single 4-core / 8 GB box

At peak we see a few hundred users online and around 50 concurrent LLM calls in flight. Costs are modest, everything is boringly self-hosted, and I like it that way.

This post is about one specific thing that broke: fair queueing of LLM slots across free and paying tiers. If you're building on top of hosted LLM APIs and you have a free tier, you'll likely run into this eventually.

The symptom

Users on our paid tier started saying that chats felt "unresponsive" during peak hours. Monitoring confirmed it: p99 request latency sat comfortably at ~800 ms most of the day, then spiked to 20+ seconds for 30–60 minute windows.

The bottleneck wasn't the LLM. Anthropic was responding in ~1.5 s like always. The problem was our own outbound concurrency: we had capped in-flight LLM calls per worker at 4, and a wave of free-tier users was saturating those slots. Paid users' requests queued behind free-tier ones for tens of seconds. FIFO, no fairness — the classic noisy-neighbor pattern.

Attempt 1: a global `asyncio.Semaphore`

The obvious first pass:

_LLM_SEM = asyncio.Semaphore(30)

async def call_llm(...):
    async with _LLM_SEM:
        return await client.messages.create(...)

Two problems with this, both worth understanding before you reach for the same tool:

1. It's per-process. With 8 uvicorn workers, we actually had 8 × 30 = 240 slots, not 30. When traffic peaked we'd occasionally hit the provider hard enough to get rate-limited (429s).

2. It's FIFO. Free and paying users compete on equal footing. When free users outnumber paying users 20 to 1, paying users lose the race almost every time.

Neither of these is a subtle bug — they're both design-level. A per-process semaphore is fundamentally the wrong shape when you have multiple workers, and a FIFO queue is fundamentally the wrong shape when you want to prioritize.

Attempt 2: a Redis-backed, tier-aware semaphore

Design goals we landed on:

Total in-flight LLM calls capped globally across all workers (protects the upstream provider)
Per-tier caps so free and guest users can't consume the entire budget
Different wait behavior per tier — free/guest fail fast (better UX to say "try again" than hang), paid tiers queue patiently

The slot budget we ended up with:

GLOBAL_MAX = 30

TIER_CAPS = {
    "premium": 30,   # can use the entire global budget
    "pro":     20,
    "free":    15,
    "guest":    5,
}

MAX_WAIT_BY_TIER = {
    "premium": 60,   # wait, don't fail — they paid
    "pro":     45,
    "free":    12,   # fail fast so FE shows "retry" quickly
    "guest":    8,
}

The key insight: the guest cap of 5 isn't there to throttle guests. It's there to guarantee that even if 500 guests hit us at once, there are always 25 slots reserved for paying users.

The atomic bit

Implementation is a small Lua script that does atomic check-and-increment on two counters (global + per-tier). This matters — if you increment one and then the other in separate calls, you can race and end up over-committed:

-- KEYS: global_key, tier_key
-- ARGV: global_max, tier_cap, ttl
local g = tonumber(redis.call('GET', KEYS[1])) or 0
local t = tonumber(redis.call('GET', KEYS[2])) or 0
if g < tonumber(ARGV[1]) and t < tonumber(ARGV[2]) then
    redis.call('INCR', KEYS[1])
    redis.call('INCR', KEYS[2])
    redis.call('EXPIRE', KEYS[1], tonumber(ARGV[3]))
    redis.call('EXPIRE', KEYS[2], tonumber(ARGV[3]))
    return 1
end
return 0

The EXPIRE calls are a safety net: if a worker crashes mid-call, the counter drains itself after 5 minutes instead of leaking a slot forever. In production I've never actually seen this fire in the healthy path, but it saved me during an early deploy when I had to kill a hung worker.

Wrapped in an async context manager, the call site stays boring:

async with tier_slot(user.tier):
    reply = await generate_reply_async(...)

Internally it polls every 100 ms, logs a warning if wait exceeds 3 s (queue.slow_wait), and raises TimeoutError at the tier's max_wait. The route layer catches the timeout and returns a structured error like {"code": "SERVER_BUSY", "retry_after": 5} — the frontend has a localized string for that code.

The Redis-is-down fallback

If Redis dies, the context manager silently falls back to a per-process asyncio.Semaphore(30). You lose cross-worker isolation, but you don't 500 the user. A single log line (queue.redis_unavailable) fires our alert.

This one detail is what actually lets me sleep at night. Redis outages are rare, but they tend to happen at the exact moment when your queueing infrastructure failing catastrophically would compound the incident.

Three related lessons that made the difference

The queueing change was the headline, but three unrelated tweaks did as much heavy lifting:

1. `statement_cache_size=0` on the async Postgres engine

We run behind PgBouncer in transaction pooling mode. asyncpg's prepared-statement cache is per-connection, but transaction pooling hands you a different connection every query. Without disabling the cache, you get sporadic prepared statement "__asyncpg_stmt_0__" does not exist errors — which look like DB corruption but aren't.

If you use asyncpg + PgBouncer + transaction mode, set this before you deploy. It's a one-line fix for a genuinely confusing bug.

2. A 3-second embedding timeout

For every chat message we do a pgvector similarity search over the NPC's long-term memory. That requires embedding the user's message first. When the embedding endpoint is slow, this used to block the entire chat response.

Our current rule: if embedding takes more than 3 s, we skip the memory retrieval and reply without long-term context. A fast, slightly less contextual reply beats a stuck one. Users don't notice; nobody has ever complained about the "missing memory" fallback.

3. Cross-provider fallback

Every LLM call is wrapped in a retry that swaps providers on the second attempt (Anthropic ↔ OpenAI). When one provider has a bad 10 minutes, users don't see it. This one is well-known but underimplemented — most teams bolt on retry logic and stop there. Adding the cross-provider fallback took an afternoon and has quietly paid for itself dozens of times.

Where we ended up

p99 on chat endpoints: ~1.8 s steady, ~4 s at peak (was 20+ s during spikes)
Free-tier abuse no longer affects paying users
We can absorb roughly a 10× traffic spike without going down — free/guest users start seeing SERVER_BUSY sooner than pro/premium do
Single box still handles it. Vertical scale to 8 cores is our next headroom step; horizontal is the one after that

The new tradeoff this created

You now have to tune the tier caps. If you launch a new tier, or your traffic mix shifts, the caps drift out of alignment. There's no principled formula — we started with "premium gets everything, guests get scraps" and adjusted based on the queue.slow_wait warnings we saw in logs.

I've thought about auto-tuning based on rolling 5-minute wait percentiles per tier, but so far manual tuning has been fine. The alerting is what matters — if slow_wait starts firing for pro or premium, we know the caps need adjusting before users notice.

One honest limitation: this only works if your bottleneck is your own concurrency, not the provider's. If Anthropic starts rate-limiting you at 15 concurrent, no clever local queueing will help — you'd need actual rate-limit-aware backpressure (react to the provider's own retry-after headers). We're not there yet.

Takeaways

If you're building on top of hosted LLM APIs and expect to serve a mix of free and paying users:

A per-process semaphore doesn't scale past one worker. If you have multiple workers, you need a distributed counter.
FIFO is unfair by construction. Reserved slots for paying tiers cost you little and prevent the noisy-neighbor pattern.
Fail fast on the free tier, queue patiently on the paid tier. The right timeout is very different depending on who's waiting.
Always have a fallback for when your queueing layer itself fails. Silent degradation beats cascading outages.

Happy to answer questions in the comments — full source isn't open, but the queue module is small enough that this post is basically the whole thing.

Built at [platos.me]

Real Friend AI — Platos

Create the AI version of anyone. Chat, roleplay, and generate images and videos with private AI companions that reach out, remember, and stay present.

platos.me

if you want to see what it powers.

DEV Community: zhenjie zhang