Mayur Agnani

Posted on Nov 1 Originally published at theaicommit.com

How to Stop a Long LLM Answer From Freezing When the Server Runs Out of Memory

#ai #llm #performance

Engineer's view. This is memory overcommit. Back every page up front and half your RAM sits idle. Overcommit, and an out-of-memory killer stops a process at peak. An LLM server faces the same choice, because each request's memory grows with its answer. You pay in idle GPU memory or in frozen streams.

TL;DR. An LLM server must decide how much memory to hold for an answer before the model has written it. Hold too much and you waste capacity. Hold too little and a running answer gets paused to make room.

Explain Like I'm 5

Picture a restaurant where nobody knows how long each family will stay. You could save a giant
table for every family, just in case. Most families leave early, so half your seats sit empty while
people wait outside.

Or you could seat everyone at small tables and hope. When the room gets full, you ask the newest
family to stand up and wait, even in the middle of dinner. They keep their food. They just can't
eat for a while.

An AI server runs that restaurant, and every answer is a family.

The Problem

You have shipped this bug already, in a system with no model in it.

You ran a service on Kubernetes. You set its memory request to the worst case you had ever seen.
Nothing crashed. But the cluster reported itself full while the real usage sat near 40%, and new
pods stayed pending.

So you lowered the request to the average. The cluster filled up nicely. Then a busy afternoon
came, several pods grew at once, and the kernel killed one in the middle of its work.

You were guessing a number before you knew the answer. An LLM server guesses that number for every
request.

The memory here is the KV cache: the model's saved work for every token so far. Without it, the
model would reread the whole conversation for each new token. It grows by one token per token
generated. The model only decides the length by stopping, so at the moment a request starts, nobody
knows its final size. The caller sends max_tokens, but that is a ceiling, not a forecast.

Under load, that guess shows up in two ways. Your GPU memory looks full while throughput sits far
below what you paid for. Or a streaming answer stops mid-sentence for many seconds, then carries on
as if nothing happened. Neither one logs an error.

The fix: find out which rule your server uses, measure how long your answers really are, then set
max_tokens and your concurrency limit from those lengths. New to the KV cache? Start at
How to Work Out How Much GPU Memory One Request Really Needs.

The Fix: Match the Limits to Your Server's Admission Rule

Admission is the moment a server lets a waiting request start. Three popular servers give three
different answers to one question: what must be free first?

What does each server hold back?

TGI counts the prompt plus the full max_new_tokens against its budget. It cannot run out mid-answer.

SGLang counts max_new_tokens times a ratio, clipped at 4,096 tokens. The ratio starts at 0.7 and
falls to about 0.1 while things go well. When memory runs out, it resets from what requests really
used.

vLLM counts only what exists now. Its --watermark setting can keep some memory free, but the default
is zero.

What happens when memory runs out?

The server evicts a running request. It frees that request's cache and sends it to the front of
the waiting line. vLLM picks the newest request. SGLang picks the one with the least output.

The answer isn't lost. Tokens already generated are kept. The request must rebuild its cache by
rereading the prompt and its own output, and only then does the stream move again. In vLLM, nothing
new starts during a step that evicted.

Is the redone work the expensive part?

I expected it to be. It wasn't. My first simulation was built to show a server thrashing, and
throughput never collapsed. Rereading is much faster than generating.

Eviction redid 80% of the prefill work and still delivered about five times the throughput. The
price landed elsewhere: 26% of finished requests were evicted, and their median freeze was 58
seconds. Throughput hid that completely.

What This Means for You

When this matters. You run your own model server, or you are choosing one. It matters most when
answers vary widely in length, as they do with reasoning models and agents. If you only call a hosted
API, this is the provider's problem. You can still detect it from your side, and your max_tokens
may still matter.

How it affects you. The same GPU can give very different results depending on which rule it runs.
On a reserving server, one team's careless limit costs everyone. In my chat simulation, moving
max_tokens from 8,192 to 4,096 raised throughput 29%, from 351 to 454 tokens per second. Answers
were short, so about 0.5% would have been cut. On an evicting server, average throughput looks
healthy while a few users stare at a stopped stream. Your dashboard's mean hides them.

What to do about it. Start with no setup. Log the largest gap between chunks on your streaming
calls. A gap of many seconds in the middle of an answer is your first sign.

Next, log how many tokens your answers really use. Compare that with the max_tokens you send.
Most teams send a default nobody chose.

Then cap concurrency from those lengths. On an evicting server, that cap removed more freezing than
any other setting I tried. Finally, check the eviction counter and alert on it.

Implementing It

The change.

Four jobs, and one person may do all of them.

Job 1, whoever owns the client. Measure the freeze from the outside. This works against any server.
In your streaming call:

gap, last = 0.0, None
for chunk in client.chat.completions.create(model=MODEL, messages=msgs, stream=True):
    now = time.monotonic()
    gap, last = max(gap, now - last) if last else 0.0, now
metrics.observe("stream_max_gap_seconds", gap)

Start timing after the first chunk, as this does, so that queue time isn't counted as a freeze.

Job 2, whoever sets max_tokens. Choose it from logged lengths, not from the model's limit. Take a
value a little above your longest normal answer, then count what it would cut:

limit = sorted(lengths)[int(0.999 * len(lengths))]
cut = sum(n > limit for n in lengths) / len(lengths)

A wrong limit shows up as finish_reason: "length". Alert on that rate.

Job 3, whoever runs the server. Work out two limits from the same logs. safe_concurrency() in
code_example.py does this:

never = pool / (mean_prompt + p95_answer)     # almost no evictions, low throughput
average = pool / (mean_prompt + mean_answer)  # the memory is full on an average mix

pool is your KV cache size in tokens. Those limits came out at 4 and 15 for the long-answer case. The upper
limit is the useful one here. It gave up 17% of throughput, cut evicted requests from 26% to 7%, and
cut the median freeze from 58 to 17 seconds.

Job 4, the setting for your server. On vLLM, lower admission without shrinking the batch buffers:

vllm serve "$MODEL" --max-num-active-seqs 15

--max-num-seqs also works, and the docs recommend it. Be careful with --watermark: at 0.2 it cut
redone work from 80% to 53%, but the median freeze rose to 247 seconds, because an evicted request
then waits for extra room.

On SGLang, watch for the pool-full warning. Its tuning guide says to raise
--schedule-conservativeness to about 1.3 if the warning fires often. Lower it to about 0.3 if token
usage stays low while requests queue. On TGI, set --max-total-tokens as low as your longest real
request. TGI is in maintenance mode, so weigh that for a new deployment.

How you know it worked.

On vLLM, this should sit near zero:

rate(vllm:num_preemptions_total[5m])

vllm:kv_cache_usage_perc should stay high, or your cap is too low. On any server, p99 of
stream_max_gap_seconds should fall while total tokens per second holds. If usage drops and waiting
requests climb, you set the cap too low. Raise it until evictions just reappear.

When Tuning Admission Is the Wrong Tool

When you never fill the pool. If vllm:kv_cache_usage_perc stays well under 1 at peak, no rule is
making a difference. Fix something else.

When batch work doesn't care. A nightly job nobody watches can freeze for a minute at no cost. Leave
concurrency high, and take the throughput.

When you are simply out of capacity. A cap moves waiting from the stream to the queue. It doesn't
make more GPU. In my runs, the queue still grew, and your time to first token will show that.

When a limit would cut real work. A low max_tokens on a reasoning model can end the answer before
it starts. Check finish_reason first.

Treat my numbers as a shape, not a measurement. They come from a simulation that follows each
server's source code, with invented lengths and an assumed prefill speed. Single runs also varied widely: the same setup froze for 26 to 170 seconds across nine seeds. It
ignores prefix caching, which can make a reread much cheaper. I did not run these servers on a GPU. If prefill is slow, or
contexts are long, eviction costs more than it did here.

Three questions before you change anything:

How much longer is your p95 answer than your mean answer?
Does your server hold memory for max_tokens, or only for what exists?
Would a user notice a ten-second pause, or only a slower batch?

Glossary

KV cache — the model's saved work for every token so far, which grows as the answer grows.
admission — the moment a server lets a waiting request start using memory.
evicts — frees a running request's cache and sends it back to wait.
prefill — processing input tokens to build the cache, much faster than generating.
max_tokens — the most tokens a caller allows for one answer; a ceiling, not a forecast.
watermark — a share of memory vLLM can keep free when admitting requests.

DEV Community