A retrieval application needs three things running at once: something that serves a model, something that stores vectors, and your code. On a workstation that is one compose file — provided you get the GPU reservation and the start ordering right, which are the two stanzas people copy wrong.
What the stack is
Three services. vLLM serving an OpenAI-compatible HTTP API on port 8000, Qdrant storing vectors with its REST API on 6333 and gRPC on 6334, and an application container that talks to both by service name over the compose network. Compose creates that network and registers each service’s name in DNS, so the app reaches http://vllm:8000 and http://qdrant:6333 with no addresses anywhere.
Ports published to the host are for you, not for the services. The application container does not need ports: to reach Qdrant; publishing 6333 is only so you can curl it from your own shell. Leaving out unnecessary ports: entries avoids collisions with whatever else is running locally.
The compose file
services:
vllm:
image: vllm/vllm-openai:latest
command: ["--model", "Qwen/Qwen3-0.6B", "--max-model-len", "8192"]
ipc: host
environment:
HF_TOKEN: ${HF_TOKEN:?set HF_TOKEN in .env}
volumes:
- hf-cache:/root/.cache/huggingface
ports:
- "8000:8000"
deploy:
resources:
reservations:
devices:
- capabilities: ["gpu"]
count: all
healthcheck:
test: ["CMD-SHELL", "python -c \"import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://localhost:8000/v1/models').status==200 else 1)\""]
interval: 15s
timeout: 5s
start_period: 600s
retries: 5
qdrant:
image: qdrant/qdrant:latest
volumes:
- qdrant-storage:/qdrant/storage
ports:
- "6333:6333"
- "6334:6334"
app:
build: .
environment:
OPENAI_BASE_URL: http://vllm:8000/v1
OPENAI_API_KEY: not-used-locally
QDRANT_URL: http://qdrant:6333
depends_on:
vllm:
condition: service_healthy
qdrant:
condition: service_started
volumes:
hf-cache:
qdrant-storage:
The two volumes are the difference between a stack you can restart and one you cannot. hf-cache holds downloaded weights, so a docker compose down && up does not re-download several gigabytes; vLLM’s own documentation mounts the Hugging Face cache for exactly this reason. qdrant-storage maps the path Qdrant persists to, so your collections survive. Omit either and you will rebuild your index every morning.
ipc: host is the compose spelling of --ipc=host. vLLM documents needing it because PyTorch uses shared memory between processes and the default /dev/shm in a container is too small; the alternative is raising shm_size on the service.
vLLM documentation on running the OpenAI-compatible server in Docker
Giving a container a GPU
Compose does not have a --gpus flag. GPUs are requested through the generic device reservation under deploy.resources.reservations.devices, where capabilities is the only required field:
deploy:
resources:
reservations:
devices:
- capabilities: ["gpu"]
count: all
To pin specific GPUs, use device_ids with the UUIDs or indices instead. The compose specification documents count and device_ids as mutually exclusive — set both and the file is rejected. This whole stanza does nothing unless the NVIDIA Container Toolkit is installed and the daemon configured, and the failure when it is not is a container that starts and sees no devices rather than a clear error.
The surprising part is that the stanza lives under deploy: at all. Most of that key describes orchestration — replicas, restart policies, placement constraints — and is meaningful to Swarm rather than to a local docker compose up. The device reservation is one of the parts that is honoured locally, which is why a file can look like it is asking for Swarm features and still work on a workstation. If you find yourself adding deploy.replicas next to it expecting three containers, that is the half that will be ignored.
Compose specification: deploy.resources
Start order and health conditions
Bare depends_on only orders container starts. It waits for the container to be running, which for a model server means the process exists and has not yet read a single weight file. Your application then starts, sends its first request and gets a connection error before the model is loaded.
The long-form depends_on with condition: service_healthy waits for the healthcheck to report healthy instead, which is why the vLLM service above defines one. Note the start_period: during that window failures do not count against the retry budget, which is precisely what a service that spends minutes loading weights needs. That mechanism, and how to make the check reflect real readiness rather than a listening socket, is the subject of writing a HEALTHCHECK for a model-serving container.
The health test above is written in Python rather than with curl deliberately. Slim and distroless images frequently do not contain curl or wget, and a healthcheck that shells out to a binary that is not there reports unhealthy forever with an error nobody reads. Use an interpreter the image definitely has. The same caution applies to the Qdrant service, which is why it is left on service_started here rather than given a check that may not run.
The collection, and the dimension that must match
Qdrant does not create collections implicitly, and the create call fixes the vector width permanently. This is where a local stack most often goes wrong, because the number has to agree with whatever your embedding model emits and nothing checks that for you until the first insert.
curl -sS -X PUT http://localhost:6333/collections/docs \
-H 'content-type: application/json' \
-d '{"vectors": {"size": 1024, "distance": "Cosine"}}'
Take size from the model rather than from a tutorial — ask the embedding model for one vector and read its length. Get it wrong and the upsert fails with a dimension error naming both numbers, which is at least a clear message; the worse outcome is picking a distance that does not match how the model was trained, because that fails silently as merely poor retrieval. Qdrant accepts Cosine, Dot, Euclid and Manhattan, and most sentence-embedding models expect cosine similarity.
Changing either value later means creating a new collection and re-embedding every document, so it is worth deciding deliberately at this point. That is also the argument for keeping the collection name versioned — docs-v2 rather than docs — so a model swap does not require deleting data you may want to compare against.
Run it
- Put
HF_TOKENin a.envfile next to the compose file. The:?syntax in the environment value makes compose fail immediately with your message if it is unset, instead of starting a server that fails to download. - Start the dependencies first and watch the model load:
docker compose up vllm qdrant. The first run downloads weights; subsequent runs read the cache volume. - Confirm health from your own shell:
curl -s localhost:8000/v1/modelsshould list the served model, andcurl -s localhost:6333/collectionsshould return an empty collection list. - Check compose agrees:
docker compose psshows a health status column. If vllm sits atstartingpast yourstart_period, the test command is wrong — run it manually withdocker compose exec vllm ...and read the error. - Bring up the application:
docker compose up app. It now starts only once vLLM reports healthy, so its first embedding call has somewhere to go. - Tear down without losing state:
docker compose downkeeps named volumes.docker compose down -vdeletes them, including your weight cache — which is the command people run by reflex and then wait twenty minutes for.
One thing worth deciding early: the application above is written against an OpenAI-shaped API, so the local vLLM service and a hosted provider are interchangeable to it — only OPENAI_BASE_URL changes. That is worth preserving deliberately, because the moment application code branches on which provider it is talking to, local development and production stop being the same program. A gateway such as Multigrid is one way to hold that seam: one base URL and one key locally and in production, with the routing decision behind it rather than in the compose file.
Top comments (0)