Someone asked on the vLLM tracker how to run a small embedding model next to a chat model on one GPU, so the card isn't half idle. I tested it on a rented RTX 3090 (24 GB) with vLLM 0.27.1: BAAI/bge-small-en-v1.5 for embeddings and Qwen2.5-1.5B-Instruct for chat, each in its own vllm serve.
It works
Embedding server at --gpu-memory-utilization 0.10, chat server at 0.70. Both came up and served with 0 errors. Together they used 18,317 of 24,576 MiB.
The startup check
Start the chat server first at 0.9, then the embedding server at 0.10, and the second one refuses to start:
ValueError: Free memory on device cuda:0 (0.95/23.8 GiB) on startup is less than desired GPU memory utilization (0.1, 2.38 GiB).
At startup vLLM checks that free memory is at least your fraction of the whole card. The chat server had taken almost everything, so 0.10 (2.38 GiB) didn't fit, even though the embedding model never uses that much.
The fix: give the small model a small number
The embedding model has no KV cache to size. vLLM runs it as a pooling model, its log has no "Available KV cache memory" line, and it stayed at about 450 to 580 MiB whether I gave it 0.10, 0.05 or 0.03.
At 0.03 (0.71 GiB) it started fine next to the 0.9 chat server: 23,110 MiB + 552 MiB on one card. With --gpu-memory-utilization 0.03 --max-model-len 512 --max-num-seqs 32 it used 454 MiB at the same throughput as at 0.10.
What sharing costs
Each server busy alone, then both busy at the same time (30 seconds each, 0 errors):
- embeddings: 1,761 to 1,609 texts/s (-8.6%)
- chat: 2,607 to 2,298 output tokens/s (-11.8%)
There's no isolation between them, so that's the price.
Leave headroom
The chat server used more than its fraction: 17,710 MiB at 0.70, where 0.70 of the card is 17,203 MiB. Its own log says so too: 12.53 GiB of KV cache in use, 11.9 GiB to fit the request.
Limits
One card, one vLLM version, two small models, short runs, one try each. Starting both servers at the same moment worked once here, but one run isn't proof, so start them one at a time.
Script and raw output: https://github.com/qaisermehdi3-coder/qvunex/tree/main/benchmarks/data/coloc-3090-2026-10-23
The thread: https://github.com/vllm-project/vllm/issues/60599
If you run vLLM on your own GPUs and want to see how much of the paid time serves nothing, there's a 2-minute guide here: https://github.com/qaisermehdi3-coder/qvunex/blob/main/TRY-IT.md
If you want it measured on your setup, I take it on as paid work: qvunexaudit@gmail.com
Top comments (8)
Sorry for the slow reply. I only saw the drop on the second server, the first one stayed steady. Recording the KV cache reservation each server prints at startup shows whether the pair fits the card before decode drops.
Thanks, that helps. The test script already saves those lines for each server ("Available KV cache memory", "GPU KV cache size"), and the embedding server printed none, because vLLM runs it as a pooling model with no KV cache. A decode check on both servers after a same-moment start is a good addition. Do you run two servers on one card in production?
Startup order hides the real failure: if the second server starts before the first holds the memory, the check passes and the decode degrades later.
Good point, and it's why the post says to start them one at a time. Honest limit: I ran the same-moment start only once (0.70 + 0.10) and only checked that both came up, I didn't measure decode speed after. vLLM reserves its KV cache at start, so if the two together ask for more than the card has I'd expect an out-of-memory error rather than slow decode, but I haven't tested that. Have you seen the slowdown yourself? Which vLLM version and GPU?
Sorry for the slow reply. I only saw the drop on the second server, the first one stayed steady. Recording the KV cache reservation each server prints at startup shows whether the pair fits the card before decode drops.
Not in production. On one card I keep a single server and send the embedding calls to CPU.
Only on the second server. The KV reservation printed at startup shows if the pair fits the card before decode drops.
Only in tests. In production one server per card, so the KV cache check is a startup gate, not a daily problem.