Someone asked on the vLLM tracker how to run a small embedding model next to a chat model on one GPU, so the card isn't half idle. I tested it on a rented RTX 3090 (24 GB) with vLLM 0.27.1: BAAI/bge-small-en-v1.5 for embeddings and Qwen2.5-1.5B-Instruct for chat, each in its own vllm serve.
It works
Embedding server at --gpu-memory-utilization 0.10, chat server at 0.70. Both came up and served with 0 errors. Together they used 18,317 of 24,576 MiB.
The startup check
Start the chat server first at 0.9, then the embedding server at 0.10, and the second one refuses to start:
ValueError: Free memory on device cuda:0 (0.95/23.8 GiB) on startup is less than desired GPU memory utilization (0.1, 2.38 GiB).
At startup vLLM checks that free memory is at least your fraction of the whole card. The chat server had taken almost everything, so 0.10 (2.38 GiB) didn't fit, even though the embedding model never uses that much.
The fix: give the small model a small number
The embedding model has no KV cache to size. vLLM runs it as a pooling model, its log has no "Available KV cache memory" line, and it stayed at about 450 to 580 MiB whether I gave it 0.10, 0.05 or 0.03.
At 0.03 (0.71 GiB) it started fine next to the 0.9 chat server: 23,110 MiB + 552 MiB on one card. With --gpu-memory-utilization 0.03 --max-model-len 512 --max-num-seqs 32 it used 454 MiB at the same throughput as at 0.10.
What sharing costs
Each server busy alone, then both busy at the same time (30 seconds each, 0 errors):
- embeddings: 1,761 to 1,609 texts/s (-8.6%)
- chat: 2,607 to 2,298 output tokens/s (-11.8%)
There's no isolation between them, so that's the price.
Leave headroom
The chat server used more than its fraction: 17,710 MiB at 0.70, where 0.70 of the card is 17,203 MiB. Its own log says so too: 12.53 GiB of KV cache in use, 11.9 GiB to fit the request.
Limits
One card, one vLLM version, two small models, short runs, one try each. Starting both servers at the same moment worked once here, but one run isn't proof, so start them one at a time.
Script and raw output: https://github.com/qaisermehdi3-coder/qvunex/tree/main/benchmarks/data/coloc-3090-2026-10-23
The thread: https://github.com/vllm-project/vllm/issues/60599
If you run vLLM on your own GPUs and want to see how much of the paid time serves nothing, there's a 2-minute guide here: https://github.com/qaisermehdi3-coder/qvunex/blob/main/TRY-IT.md
If you want it measured on your setup, I take it on as paid work: qvunexaudit@gmail.com
Top comments (1)
Startup order hides the real failure: if the second server starts before the first holds the memory, the check passes and the decode degrades later.