The 2026 Guide to Self-Hosting a Private Local RAG Stack (Ollama + DeepSeek + Qdrant + Open WebUI)

Sending proprietary source code, internal documents, and client contracts to closed AI APIs (OpenAI, Anthropic, Google) is a massive compliance and security hazard for tech teams.

With open-weight models like DeepSeek-R1 (Distill 7B/8B/14B) and Llama 3.3, and vector databases like Qdrant, you can host a complete enterprise-grade AI chatbot with document search on a single dedicated server or VPS.

Here is the exact production Docker Compose setup to run a local RAG stack with zero telemetry.

---

The Stack Architecture

User -> Open WebUI (Frontend & Document Processor)
           |
           +---> Ollama (Inference: DeepSeek / Llama 3 / Nomic Embed)
           |
           +---> Qdrant (High-performance Vector Database)

Component	Role	Resource Requirement
Open WebUI	Chat interface, document parsing, user auth	~300 MB RAM
Ollama	LLM & Embedding inference engine	4GB–16GB RAM (CPU or GPU)
Qdrant	Rust-based Vector Search Database	~150 MB RAM
Caddy v2	Reverse proxy, automatic SSL / TLS	~30 MB RAM

---

1. Production Docker Compose (docker-compose.yml)

version: "3.8"

networks:
  ai-net:
    driver: bridge

volumes:
  ollama-data:
  openwebui-data:
  qdrant-data:
  caddy-data:
  caddy-config:

services:
  # --- Reverse Proxy ---
  caddy:
    image: caddy:2-alpine
    container_name: ai-caddy
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - caddy-data:/data
      - caddy-config:/config
    networks:
      - ai-net
    depends_on:
      - open-webui

  # --- Inference Engine (Ollama) ---
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    volumes:
      - ollama-data:/root/.ollama
    networks:
      - ai-net
    # If using NVIDIA GPU, uncomment the deploy block:
    # deploy:
    #   resources:
    #     reservations:
    #       devices:
    #         - driver: nvidia
    #           count: all
    #           capabilities: [gpu]
    environment:
      - OLLAMA_KEEP_ALIVE=24h
      - OLLAMA_NUM_PARALLEL=4

  # --- Vector Database (Qdrant) ---
  qdrant:
    image: qdrant/qdrant:v1.12.0
    container_name: qdrant
    restart: unless-stopped
    volumes:
      - qdrant-data:/qdrant/storage
    networks:
      - ai-net
    environment:
      - QDRANT__SERVICE__GRPC_PORT=6334

  # --- Web Interface & RAG Orchestration ---
  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    restart: unless-stopped
    volumes:
      - openwebui-data:/app/backend/data
    networks:
      - ai-net
    depends_on:
      - ollama
      - qdrant
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
      - VECTOR_DB=qdrant
      - QDRANT_URI=http://qdrant:6333
      - RAG_EMBEDDING_ENGINE=ollama
      - RAG_EMBEDDING_MODEL=nomic-embed-text
      - ENABLE_SIGNUP=false # Set to true on first run to create admin
      - WEBUI_SECRET_KEY=${WEBUI_SECRET_KEY}

---

2. Reverse Proxy & Domain (Caddyfile)

ai.yourdomain.com {
    encode gzip zstd

    # Increase body limit for large PDF document uploads
    request_body {
        max_size 100MB
    }

    reverse_proxy open-webui:8080 {
        header_up X-Real-IP {remote_host}
        header_up X-Forwarded-Proto {scheme}
    }
}

---

3. Pulling Models and Starting Up

1. Start the stack:

   docker compose up -d

1. Download your reasoning model & embedding model:

   # Pull DeepSeek-R1 Distill (7B or 8B)
   docker exec -it ollama ollama pull deepseek-r1:8b

   # Pull high-performance embedding model for RAG document chunking
   docker exec -it ollama ollama pull nomic-embed-text

1. Visit https://ai.yourdomain.com, register the master account, and upload your team's PDFs, markdown docs, or codebase to start private semantic search!

---

4. Hardware Recommendations

Setup	Recommended Specs	Cloud Cost Equivalent
CPU Only (Budget/Homelab)	4-8 vCPU, 16GB DDR5 RAM (Hetzner CCX23 ~€24/mo)	$20/user/mo ($240/yr/user)
GPU Accelerated	1x NVIDIA RTX 4090 / A4000 (16-24GB VRAM) (~$80/mo)	$30-$50/user/mo Enterprise SaaS

---

Explore More Self-Hosted Stacks

Looking for more curated open-source alternatives?

• Browse our ChatGPT & AI Assistants Directory.
• Find the best cloud VPS provider with our 2026 VPS Guide.
• Assemble your full open-source stack with our Interactive Stack Builder.

Want 50+ battle-tested Docker Compose templates with Caddy TLS and backup scripts? Get the Self-Hosted Starter Stack Pack ($29).