TL;DR — When traffic spikes on a shared LLM backend, a naive concurrency limit lets free-tier users starve paying users. This post walks through why our first solution (a global asyncio.Semaphore) broke, and how a small Redis-backed tier-aware slot manager brought p99 latency during peaks from 20+ seconds down to under 2. The core code is ~40 lines. There's no magic — just a bit of fairness.
What we're running
An AI companion app: users chat with LLM-powered personas, with pgvector-backed long-term memory. Nothing exotic on the infra side:
- FastAPI behind Uvicorn (8 workers)
- Postgres 16 + pgvector (via PgBouncer)
- Redis for cache and queueing
- R2 for media
- A single 4-core / 8 GB box
At peak we see a few hundred users online and around 50 concurrent LLM calls in flight. Costs are modest, everything is boringly self-hosted, and I like it that way.
This post is about one specific thing that broke: fair queueing of LLM slots across free and paying tiers. If you're building on top of hosted LLM APIs and you have a free tier, you'll likely run into this eventually.
The symptom
Users on our paid tier started saying that chats felt "unresponsive" during peak hours. Monitoring confirmed it: p99 request latency sat comfortably at ~800 ms most of the day, then spiked to 20+ seconds for 30–60 minute windows.
The bottleneck wasn't the LLM. Anthropic was responding in ~1.5 s like always. The problem was our own outbound concurrency: we had capped in-flight LLM calls per worker at 4, and a wave of free-tier users was saturating those slots. Paid users' requests queued behind free-tier ones for tens of seconds. FIFO, no fairness — the classic noisy-neighbor pattern.
Attempt 1: a global asyncio.Semaphore
The obvious first pass:
_LLM_SEM = asyncio.Semaphore(30)
async def call_llm(...):
async with _LLM_SEM:
return await client.messages.create(...)
Two problems with this, both worth understanding before you reach for the same tool:
1. It's per-process. With 8 uvicorn workers, we actually had 8 × 30 = 240 slots, not 30. When traffic peaked we'd occasionally hit the provider hard enough to get rate-limited (429s).
2. It's FIFO. Free and paying users compete on equal footing. When free users outnumber paying users 20 to 1, paying users lose the race almost every time.
Neither of these is a subtle bug — they're both design-level. A per-process semaphore is fundamentally the wrong shape when you have multiple workers, and a FIFO queue is fundamentally the wrong shape when you want to prioritize.
Attempt 2: a Redis-backed, tier-aware semaphore
Design goals we landed on:
- Total in-flight LLM calls capped globally across all workers (protects the upstream provider)
- Per-tier caps so free and guest users can't consume the entire budget
- Different wait behavior per tier — free/guest fail fast (better UX to say "try again" than hang), paid tiers queue patiently
The slot budget we ended up with:
GLOBAL_MAX = 30
TIER_CAPS = {
"premium": 30, # can use the entire global budget
"pro": 20,
"free": 15,
"guest": 5,
}
MAX_WAIT_BY_TIER = {
"premium": 60, # wait, don't fail — they paid
"pro": 45,
"free": 12, # fail fast so FE shows "retry" quickly
"guest": 8,
}
The key insight: the guest cap of 5 isn't there to throttle guests. It's there to guarantee that even if 500 guests hit us at once, there are always 25 slots reserved for paying users.
The atomic bit
Implementation is a small Lua script that does atomic check-and-increment on two counters (global + per-tier). This matters — if you increment one and then the other in separate calls, you can race and end up over-committed:
-- KEYS: global_key, tier_key
-- ARGV: global_max, tier_cap, ttl
local g = tonumber(redis.call('GET', KEYS[1])) or 0
local t = tonumber(redis.call('GET', KEYS[2])) or 0
if g < tonumber(ARGV[1]) and t < tonumber(ARGV[2]) then
redis.call('INCR', KEYS[1])
redis.call('INCR', KEYS[2])
redis.call('EXPIRE', KEYS[1], tonumber(ARGV[3]))
redis.call('EXPIRE', KEYS[2], tonumber(ARGV[3]))
return 1
end
return 0
The EXPIRE calls are a safety net: if a worker crashes mid-call, the counter drains itself after 5 minutes instead of leaking a slot forever. In production I've never actually seen this fire in the healthy path, but it saved me during an early deploy when I had to kill a hung worker.
Wrapped in an async context manager, the call site stays boring:
async with tier_slot(user.tier):
reply = await generate_reply_async(...)
Internally it polls every 100 ms, logs a warning if wait exceeds 3 s (queue.slow_wait), and raises TimeoutError at the tier's max_wait. The route layer catches the timeout and returns a structured error like {"code": "SERVER_BUSY", "retry_after": 5} — the frontend has a localized string for that code.
The Redis-is-down fallback
If Redis dies, the context manager silently falls back to a per-process asyncio.Semaphore(30). You lose cross-worker isolation, but you don't 500 the user. A single log line (queue.redis_unavailable) fires our alert.
This one detail is what actually lets me sleep at night. Redis outages are rare, but they tend to happen at the exact moment when your queueing infrastructure failing catastrophically would compound the incident.
Three related lessons that made the difference
The queueing change was the headline, but three unrelated tweaks did as much heavy lifting:
1. statement_cache_size=0 on the async Postgres engine
We run behind PgBouncer in transaction pooling mode. asyncpg's prepared-statement cache is per-connection, but transaction pooling hands you a different connection every query. Without disabling the cache, you get sporadic prepared statement "__asyncpg_stmt_0__" does not exist errors — which look like DB corruption but aren't.
If you use asyncpg + PgBouncer + transaction mode, set this before you deploy. It's a one-line fix for a genuinely confusing bug.
2. A 3-second embedding timeout
For every chat message we do a pgvector similarity search over the NPC's long-term memory. That requires embedding the user's message first. When the embedding endpoint is slow, this used to block the entire chat response.
Our current rule: if embedding takes more than 3 s, we skip the memory retrieval and reply without long-term context. A fast, slightly less contextual reply beats a stuck one. Users don't notice; nobody has ever complained about the "missing memory" fallback.
3. Cross-provider fallback
Every LLM call is wrapped in a retry that swaps providers on the second attempt (Anthropic ↔ OpenAI). When one provider has a bad 10 minutes, users don't see it. This one is well-known but underimplemented — most teams bolt on retry logic and stop there. Adding the cross-provider fallback took an afternoon and has quietly paid for itself dozens of times.
Where we ended up
- p99 on chat endpoints: ~1.8 s steady, ~4 s at peak (was 20+ s during spikes)
- Free-tier abuse no longer affects paying users
- We can absorb roughly a 10× traffic spike without going down — free/guest users start seeing
SERVER_BUSYsooner than pro/premium do - Single box still handles it. Vertical scale to 8 cores is our next headroom step; horizontal is the one after that
The new tradeoff this created
You now have to tune the tier caps. If you launch a new tier, or your traffic mix shifts, the caps drift out of alignment. There's no principled formula — we started with "premium gets everything, guests get scraps" and adjusted based on the queue.slow_wait warnings we saw in logs.
I've thought about auto-tuning based on rolling 5-minute wait percentiles per tier, but so far manual tuning has been fine. The alerting is what matters — if slow_wait starts firing for pro or premium, we know the caps need adjusting before users notice.
One honest limitation: this only works if your bottleneck is your own concurrency, not the provider's. If Anthropic starts rate-limiting you at 15 concurrent, no clever local queueing will help — you'd need actual rate-limit-aware backpressure (react to the provider's own retry-after headers). We're not there yet.
Takeaways
If you're building on top of hosted LLM APIs and expect to serve a mix of free and paying users:
- A per-process semaphore doesn't scale past one worker. If you have multiple workers, you need a distributed counter.
- FIFO is unfair by construction. Reserved slots for paying tiers cost you little and prevent the noisy-neighbor pattern.
- Fail fast on the free tier, queue patiently on the paid tier. The right timeout is very different depending on who's waiting.
- Always have a fallback for when your queueing layer itself fails. Silent degradation beats cascading outages.
Happy to answer questions in the comments — full source isn't open, but the queue module is small enough that this post is basically the whole thing.
Built at [platos.me]
Real Friend AI — Platos
Create the AI version of anyone. Chat, roleplay, and generate images and videos with private AI companions that reach out, remember, and stay present.
platos.me
if you want to see what it powers.
Top comments (0)