Day 18: Docker Compose for Multi-Container AI Stacks

A serious AI application is a small distributed system. A RAG pipeline cannot be one monolithic process, because inference, data indexing and the user interface have different needs and change at different speeds. Docker Compose lets you describe all of them in one file and start them together on a single host.

In plain terms: Compose is a seating plan for a dinner party. You say who sits where, who can talk to whom, and who must arrive before the others, and Docker sets the table.

The core topology: a 3-tier RAG stack

Diagram: A self-contained RAG deployment has three isolated services: an application layer that orchestrates, a vector database that holds the embeddings, and a model server such as vLLM or Ollama that generates the answer. Docker Compose starts and connects them on a single host. See the animated version.

• Model server (vLLM or Ollama). A dedicated compute container that manages GPU or system memory and exposes inference over an internal port.
• Vector database (Qdrant, Milvus or pgvector). An independent, stateful service that holds the embeddings and answers nearest-neighbour queries.
• Application layer (FastAPI, plus Streamlit or Next.js). The orchestrator: it takes the user's question, fetches matching context from the vector database, builds the prompt and asks the model server for the answer.

Three configuration practices

1. A private bridge network. Put every container on one user-defined network. Docker then resolves service names automatically, so the app can call http://ollama:11434 or http://qdrant:6333. Publish only the app's port to the host, and the database and model server stay unreachable from outside.

Diagram: Containers on a user-defined bridge network find each other by service name through Docker's built-in DNS, for example http://ollama:11434 or http://qdrant:6333. Only the app's port is published to the host, so the database and model server are not reachable from outside. See the animated version.

2. Volumes for state. Mount separate volumes for the vector database storage and the model cache (/root/.ollama, or /root/.cache/huggingface for Hugging Face models). Containers can then be updated, restarted or rebuilt without losing the index or downloading the weights again.

Diagram: Mounting separate volumes for the vector database storage and the model cache lets you rebuild, restart or upgrade containers without losing the index or downloading the model weights again. See the animated version.

3. Healthchecks and start order. Add a healthcheck to each service and use depends_on with condition: service_healthy, so the app never queries a database or model server that is not listening yet.

Diagram: With a healthcheck on each service and depends_on with condition service_healthy, the application container waits until the database and model server are actually listening, so it never queries a service that is not ready. See the animated version.

Here is a compact example. It assumes your app lives in ./app with a Dockerfile like yesterday's, and that the healthcheck commands suit the image versions you use. Check each image's documentation, since minimal images do not always include the tools a check needs. In production, pin a specific image tag instead of latest.

services:
  app:
    build: ./app
    ports:
      - "8000:8000"            # the only port published to the host
    environment:
      OLLAMA_URL: http://ollama:11434
      QDRANT_URL: http://qdrant:6333
    depends_on:
      ollama:
        condition: service_healthy
      qdrant:
        condition: service_healthy
    networks: [ai-network]

  qdrant:
    image: qdrant/qdrant:latest
    volumes:
      - qdrant_data:/qdrant/storage
    healthcheck:
      test: ["CMD-SHELL", "bash -c ':> /dev/tcp/127.0.0.1/6333' || exit 1"]
      interval: 10s
      retries: 10
    networks: [ai-network]

  ollama:
    image: ollama/ollama:latest
    volumes:
      - ollama_models:/root/.ollama
    healthcheck:
      test: ["CMD", "ollama", "list"]
      interval: 10s
      retries: 10
    networks: [ai-network]
    # For an NVIDIA GPU, add a device reservation:
    # deploy:
    #   resources:
    #     reservations:
    #       devices:
    #         - driver: nvidia
    #           count: all
    #           capabilities: [gpu]

volumes:
  qdrant_data:
  ollama_models:

networks:
  ai-network:

Start it with docker compose up -d, and stop it with docker compose down (add -v only if you want the volumes deleted too). Decoupling the stack this way means any one component can be upgraded, scaled or replaced without destabilizing the rest.

Coming Up Next

Day 19: Kubernetes for AI 101: pods, deployments and services for model serving.

DockerCompose #Microservices #RAG #SystemDesign #DevOps

---

Originally published at https://sureshpallapothu.in/blog/day-18-docker-compose, where this post includes animated diagrams.