Elder Fernandes

Posted on Aug 23 Originally published at selfhoststack-8z4.pages.dev

Self-Hosted LLM Inference & Serving in 2026: vLLM vs Ollama vs LocalAI vs TGI (+ Open WebUI)

#ai #docker #devops #selfhosted

Sending confidential company data, source code, and customer records to commercial LLM APIs exposes organizations to privacy breaches, unexpected rate limits, and runaway monthly bills.

With open-weight models like DeepSeek-R1 / V3, Llama 3.3 70B, Mistral NeMo, and Qwen 2.5, running local inference on your own hardware (or dedicated GPU cloud instances like Hetzner, RunPod, or Lambda) is faster and more cost-effective than ever.

However, selecting the right inference engine determines whether you achieve 5 tokens/sec or 120 tokens/sec per GPU.

In this guide, we break down the four leading self-hosted LLM serving engines and provide production-ready Docker Compose configurations.

1. Engine Comparison Matrix

Engine	Primary Focus	Best For	Model Formats	Concurrency & Batching
vLLM	High-throughput production serving	Multi-user API backends & enterprise	AWQ, GPTQ, FP8, SafeTensors	PagedAttention (Massive throughput)
Ollama	Developer simplicity & desktop/homelab	Single-user & small team prototyping	GGUF	Continuous batching (Basic)
LocalAI	Drop-in OpenAI API replacement	CPU + GPU fallback & multimodal	GGUF, GGML, ONNX, Diffusers	Standard queueing
TGI (Hugging Face)	Production enterprise deployment	Hugging Face ecosystem pipelines	SafeTensors, AWQ, EETQ	Dynamic batching & FlashAttention

2. Choosing Your Serving Architecture

When to Choose vLLM

You have modern NVIDIA GPUs (RTX 3090, 4090, A100, H100) or AMD ROCm hardware.
You need to serve multiple concurrent users or background agent workflows without throughput collapse.
You want state-of-the-art memory management via PagedAttention, eliminating VRAM waste from KV caching.

When to Choose Ollama

You want zero-configuration model pulling (ollama run llama3.3).
You are running on consumer hardware, Apple Silicon (M1/M2/M3/M4), or hybrid CPU/GPU setups using GGUF quantization.
You want seamless integration with Open WebUI, AnythingLLM, or Cursor.

When to Choose LocalAI

You need a unified engine capable of running text generation, text-to-speech (TTS), audio transcription (Whisper), and image generation (Stable Diffusion) inside a single container.

3. Production Deployment Configurations

Stack 1: High-Performance vLLM + Open WebUI (GPU Production Stack)

This stack exposes an OpenAI-compatible API on port 8000 and the premier ChatGPT-like interface (Open WebUI) on port 3000:

services:
  vllm:
    image: vllm/vllm-openai:latest
    container_name: vllm_server
    restart: unless-stopped
    environment:
      - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-}
    volumes:
      - ~/.cache/huggingface:/root/.cache/huggingface
    ports:
      - "127.0.0.1:8000:8000"
    ipc: host
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    command: >
      --model Qwen/Qwen2.5-Coder-7B-Instruct
      --gpu-memory-utilization 0.90
      --max-model-len 8192
      --dtype auto
      --api-key ${VLLM_API_KEY:-super-secret-key-123}

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open_webui
    restart: unless-stopped
    ports:
      - "127.0.0.1:3000:8080"
    volumes:
      - open_webui_data:/app/backend/data
    environment:
      - OPENAI_API_BASE_URL=http://vllm:8000/v1
      - OPENAI_API_KEY=${VLLM_API_KEY:-super-secret-key-123}
      - WEBUI_AUTH=true
      - ENABLE_SIGNUP=false # Disable public registration
    depends_on:
      - vllm

volumes:
  open_webui_data:

Stack 2: Ollama + Open WebUI (CPU & GPU Homelab Stack)

For mixed environments and easy GGUF model handling:

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    ports:
      - "127.0.0.1:11434:11434"
    volumes:
      - ollama_models:/root/.ollama
    # Uncomment deploy block if NVIDIA GPU is present:
    # deploy:
    #   resources:
    #     reservations:
    #       devices:
    #         - driver: nvidia
    #           count: all
    #           capabilities: [gpu]

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open_webui
    restart: unless-stopped
    ports:
      - "127.0.0.1:3000:8080"
    volumes:
      - open_webui_data:/app/backend/data
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
      - WEBUI_AUTH=true
    depends_on:
      - ollama

volumes:
  ollama_models:
  open_webui_data:

4. Hardware Sizing & VRAM Rules of Thumb

Model Size	Quantization	Minimum VRAM / RAM	Recommended Hardware
7B / 8B (Llama 3.1 / Qwen 2.5)	Q4_K_M (GGUF)	6 GB VRAM	RTX 3060 / Apple M1 16GB
7B / 8B	FP16 / AWQ 4-bit	8–16 GB VRAM	RTX 4070 / RTX 3090
14B / 32B (DeepSeek / Qwen)	Q4_K_M / AWQ	16–24 GB VRAM	RTX 3090 (24GB) / RTX 4090
70B (Llama 3.3)	Q4_K_M (GGUF)	48 GB RAM / VRAM	2x RTX 3090 (NVLink) or Mac Studio 64GB
70B	FP8 / AWQ	48–80 GB VRAM	2x RTX 4090 or 1x A100 (80GB)

5. Production Hardening Checklist

Authentication First: Never expose unauthenticated Ollama (11434) or vLLM (8000) ports to the public internet. Anyone finding the IP can drain your compute or inject prompts.
Reverse Proxy & SSL: Wrap your Open WebUI behind Traefik or Caddy with Let's Encrypt certificates.
Model Weights Cache: Persist ~/.cache/huggingface and ~/.ollama on fast NVMe volumes to prevent re-downloading 20GB+ weights across container restarts.

Conclusion & Architecture Roadmap

Self-hosting your AI stack provides full data governance, zero compliance risk, and fixed compute costs.

Looking for battle-tested Docker Compose templates for local AI, RAG pipelines, vector databases, and reverse proxies? Check out the complete guides on SelfHostStack or grab the Self-Hosted Starter Stack Pack ($29).

DEV Community