Start a big batch job on the server your chatbot uses, and the first word of every chat reply can take over a minute. Two separate fixes are needed, because the server never stops a running request to let a new one in.
Engineer's view. This is a database connection pool with nothing reserved. A nightly report job takes every connection, and the login page waits behind it. An LLM server runs a fixed number of requests at once and never evicts one for a newcomer. So chat waits for a batch request to finish, and your latency alert fires.
TL;DR. A batch job slows chat on a shared LLM server because the server fills every running slot and never makes room for a new arrival. Priority decides who gets the next free slot, and headroom makes sure one is free, so use both.
Explain Like I'm 5
A restaurant has 64 tables. A tour bus pulls up with 4,000 people, and they all join the line.
A regular walks in for a quick lunch. The host could let them skip the line. That helps, but they still wait until some table finishes eating, because the restaurant never asks seated diners to leave.
The better host also keeps a few tables empty while the tour group eats. Now the regular sits down at once. The tour group finishes a little later, and nobody else waits at all.
The Problem
You have shipped this bug before, and it had nothing to do with AI.
A nightly report job ran against your production database. It opened as many connections as the pool allowed and held each one for a long query. At 2 a.m. nobody noticed. Then someone moved the job to business hours. The login page started timing out, even though the database had plenty of CPU left. Every connection was taken, and login waited in line behind the reports.
An LLM server has the same pool, and it is smaller than you think. vLLM, the common open source engine, runs at most --max-num-seqs requests per replica at a time. Call each place in that batch a slot. A request holds its slot until it writes its last token. The engine does not stop a running request to make room for a new one. It only does that when GPU memory runs out.
So a batch job that sends thousands of requests takes every slot. A chat request that arrives next waits for a slot to free. Then it waits behind every batch request queued before it.
Engineers at Google measured this on the llm-d serving stack. A chat user's time to first token was 45 milliseconds on an idle server. With a 4,000-request batch job pushing in at full concurrency, it was 84 seconds.
The fix: use priority to decide who gets the next free slot, and cap the batch job's requests in flight below the slot count so that a slot is usually free.
Read the full post on The AI Commit
How to Keep a Batch Job From Slowing Chat Users on a Shared LLM Server carries the rest: the fix and why it works, the code to change for each role, how to tell it worked, when it is the wrong tool, and an interactive model you can play with.
Originally published at theaicommit.com, a daily AI engineering lab for software engineers. The original has an interactive version you can play with, and the full runnable code is on GitHub.
Top comments (0)