DEV Community

Qaiser Mehdi
Qaiser Mehdi

Posted on

Two vLLM servers on one GPU: the startup check that bites, and the fix

Someone asked on the vLLM tracker how to run a small embedding model next to a chat model on one GPU, so the card isn't half idle. I tested it on a rented RTX 3090 (24 GB) with vLLM 0.27.1: BAAI/bge-small-en-v1.5 for embeddings and Qwen2.5-1.5B-Instruct for chat, each in its own vllm serve.

It works

Embedding server at --gpu-memory-utilization 0.10, chat server at 0.70. Both came up and served with 0 errors. Together they used 18,317 of 24,576 MiB.

The startup check

Start the chat server first at 0.9, then the embedding server at 0.10, and the second one refuses to start:

ValueError: Free memory on device cuda:0 (0.95/23.8 GiB) on startup is less than desired GPU memory utilization (0.1, 2.38 GiB).

At startup vLLM checks that free memory is at least your fraction of the whole card. The chat server had taken almost everything, so 0.10 (2.38 GiB) didn't fit, even though the embedding model never uses that much.

The fix: give the small model a small number

The embedding model has no KV cache to size. vLLM runs it as a pooling model, its log has no "Available KV cache memory" line, and it stayed at about 450 to 580 MiB whether I gave it 0.10, 0.05 or 0.03.

At 0.03 (0.71 GiB) it started fine next to the 0.9 chat server: 23,110 MiB + 552 MiB on one card. With --gpu-memory-utilization 0.03 --max-model-len 512 --max-num-seqs 32 it used 454 MiB at the same throughput as at 0.10.

What sharing costs

Each server busy alone, then both busy at the same time (30 seconds each, 0 errors):

  • embeddings: 1,761 to 1,609 texts/s (-8.6%)
  • chat: 2,607 to 2,298 output tokens/s (-11.8%)

There's no isolation between them, so that's the price.

Leave headroom

The chat server used more than its fraction: 17,710 MiB at 0.70, where 0.70 of the card is 17,203 MiB. Its own log says so too: 12.53 GiB of KV cache in use, 11.9 GiB to fit the request.

Limits

One card, one vLLM version, two small models, short runs, one try each. Starting both servers at the same moment worked once here, but one run isn't proof, so start them one at a time.

Script and raw output: https://github.com/qaisermehdi3-coder/qvunex/tree/main/benchmarks/data/coloc-3090-2026-10-23

The thread: https://github.com/vllm-project/vllm/issues/60599

If you run vLLM on your own GPUs and want to see how much of the paid time serves nothing, there's a 2-minute guide here: https://github.com/qaisermehdi3-coder/qvunex/blob/main/TRY-IT.md

If you want it measured on your setup, I take it on as paid work: qvunexaudit@gmail.com

Top comments (8)

Collapse
 
analista_83 profile image
Sammi De Blas •

Sorry for the slow reply. I only saw the drop on the second server, the first one stayed steady. Recording the KV cache reservation each server prints at startup shows whether the pair fits the card before decode drops.

Collapse
 
qaisermehdi profile image
Qaiser Mehdi •

Thanks, that helps. The test script already saves those lines for each server ("Available KV cache memory", "GPU KV cache size"), and the embedding server printed none, because vLLM runs it as a pooling model with no KV cache. A decode check on both servers after a same-moment start is a good addition. Do you run two servers on one card in production?

Collapse
 
analista_83 profile image
Sammi De Blas •

Startup order hides the real failure: if the second server starts before the first holds the memory, the check passes and the decode degrades later.

Collapse
 
qaisermehdi profile image
Qaiser Mehdi •

Good point, and it's why the post says to start them one at a time. Honest limit: I ran the same-moment start only once (0.70 + 0.10) and only checked that both came up, I didn't measure decode speed after. vLLM reserves its KV cache at start, so if the two together ask for more than the card has I'd expect an out-of-memory error rather than slow decode, but I haven't tested that. Have you seen the slowdown yourself? Which vLLM version and GPU?

Collapse
 
analista_83 profile image
Sammi De Blas •

Sorry for the slow reply. I only saw the drop on the second server, the first one stayed steady. Recording the KV cache reservation each server prints at startup shows whether the pair fits the card before decode drops.

Collapse
 
analista_83 profile image
Sammi De Blas •

Not in production. On one card I keep a single server and send the embedding calls to CPU.

Collapse
 
analista_83 profile image
Sammi De Blas •

Only on the second server. The KV reservation printed at startup shows if the pair fits the card before decode drops.

Collapse
 
analista_83 profile image
Sammi De Blas •

Only in tests. In production one server per card, so the KV cache check is a startup gate, not a daily problem.