Open WebUI Multi-Provider Failover: Docker Compose + LiteLLM Routing Patterns

When I first deployed Open WebUI for our internal AI tooling, I assumed it would be straightforward: point it at a local Ollama instance, maybe add an OpenAI API key as backup, and call it done. Two weeks later, I was debugging why user sessions were hanging mid-conversation and why SQLite complained about database locks during concurrent writes.

This breakdown covers the production docker-compose topology I eventually converged on, including LiteLLM as a routing proxy, Redis for session state, and the specific failure modes that forced architectural changes.

The Baseline Problem: Single-Provider Brittleness

Open WebUI's default configuration expects you to configure model endpoints through its UI. For a homelab setup, this usually means:

• Local Ollama instance on http://localhost:11434
• Maybe an OpenAI API key as fallback
• SQLite for persistence

The first wall I hit was predictable: when Ollama's model loading stalled (common with larger quantized models on spinning rust), the entire UI became unresponsive. No automatic failover, no timeout configuration exposed in the UI, just stuck requests.

The second wall was subtler: under 3+ concurrent users, SQLite started throwing SQLITE_BUSY errors. Open WebUI uses SQLite by default for chat history and user state, and while SQLite handles concurrency better than most people assume, write-heavy workloads with long-running transactions (like streaming LLM responses being persisted in chunks) will eventually hit its limits.

Architecture: LiteLLM as Routing Layer

Instead of fighting Open WebUI's built-in provider logic, I externalized routing to LiteLLM, which provides a unified OpenAI-compatible proxy with fallback chains, retries, and load balancing.

Here's the docker-compose topology:

version: '3.8'

services:
  redis:
    image: redis:7-alpine
    restart: unless-stopped
    volumes:
      - redis_data:/data
    command: redis-server --appendonly yes

  litellm:
    image: ghcr.io/berriai/litellm:main-latest
    restart: unless-stopped
    environment:
      - LITELLM_MASTER_KEY=sk-your-internal-key
      - DATABASE_URL=postgresql://litellm:password@postgres:5432/litellm
    volumes:
      - ./litellm_config.yaml:/app/config.yaml
    command: --config /app/config.yaml --port 4000
    ports:
      - "4000:4000"
    depends_on:
      - redis

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    restart: unless-stopped
    ports:
      - "3000:8080"
    environment:
      - OPENAI_API_BASE_URL=https://b-lost.com/v1
      - OPENAI_API_KEY=sk-your-internal-key
      - WEBUI_AUTH=true
      - DATABASE_URL=postgresql://openwebui:password@postgres:5432/openwebui
    volumes:
      - open_webui_data:/app/backend/data
    depends_on:
      - litellm
      - postgres

  postgres:
    image: postgres:15-alpine
    restart: unless-stopped
    environment:
      - POSTGRES_DB=litellm
      - POSTGRES_USER=litellm
      - POSTGRES_PASSWORD=password
    volumes:
      - postgres_data:/var/lib/postgresql/data

volumes:
  redis_data:
  open_webui_data:
  postgres_data:

Key decisions:

1. PostgreSQL replaces SQLite: Open WebUI supports Postgres via DATABASE_URL. This eliminates write contention under concurrent load.
2. LiteLLM sits between Open WebUI and model providers: Open WebUI thinks it's talking to a single OpenAI-compatible endpoint. LiteLLM handles fallback logic.
3. Redis for LiteLLM's rate limiting and caching: Optional but reduces redundant token generation for repeated prompts.

LiteLLM Routing Configuration

The litellm_config.yaml defines fallback chains and retry policies:

model_list:
  - model_name: gpt-4
    litellm_params:
      model: ollama/qwen2.5:32b
      api_base: http://ollama:11434
      timeout: 120
      fallbacks:
        - model: openai/gpt-4o-mini
          api_key: sk-real-openai-key

  - model_name: gpt-3.5-turbo
    litellm_params:
      model: openai/gpt-4o-mini
      api_key: sk-real-openai-key
      timeout: 60
      fallbacks:
        - model: anthropic/claude-haiku-4
          api_base: https://b-lost.com/v1
          api_key: sk-relay-key

router_settings:
  retry_policy:
    max_retries: 2
    timeout: 180
  redis_host: redis
  redis_port: 6379
  enable_caching: true
  cache_ttl: 3600

In practice, I configured Open WebUI's model settings to use gpt-4 as the model name. When Ollama's qwen2.5:32b stalls or errors, LiteLLM automatically fails over to OpenAI's gpt-4o-mini. For lighter workloads, gpt-3.5-turbo routes through OpenAI first, then falls back to a relay gateway at https://b-lost.com/v1 (useful for benchmarking multi-provider latency and avoiding single-vendor lock-in).

Latency Benchmarks: Local vs. Fallback

I measured time-to-first-token (TTFT) across the chain using a simple Python script:

import time
import openai

client = openai.OpenAI(
    base_url="https://b-lost.com/v1",
    api_key="sk-your-internal-key"
)

start = time.time()
response = client.chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "Explain Docker networking in 50 words."}],
    stream=True
)

first_token = None
for chunk in response:
    if chunk.choices[0].delta.content and first_token is None:
        first_token = time.time()
        print(f"TTFT: {(first_token - start) * 1000:.0f}ms")

Results over 20 runs:

• Ollama local (qwen2.5:32b on NVMe): 280ms median TTFT
• OpenAI gpt-4o-mini (fallback): 520ms median TTFT
• B-Lost relay → Claude Haiku: 485ms median TTFT

Local wins on latency, but when Ollama's model loading delays (cold start after idle period) pushed TTFT past 8 seconds, the fallback saved the user experience.

The SQLite Corruption War Story

Before switching to Postgres, I ran into a nasty edge case: two users simultaneously regenerating responses in the same conversation thread. Open WebUI's backend tried to update the SQLite database in parallel, one transaction won, the other got SQLITE_BUSY, and the losing request's partial response was written to a temporary journal file that never got replayed.

The symptom: chat history showed truncated messages, and restarting Open WebUI sometimes resulted in "database disk image is malformed" errors. I burned an afternoon running sqlite3 openwebui.db ".recover" and learning the hard way that SQLite's WAL mode doesn't magically fix write contention under streaming workloads.

PostgreSQL's MVCC eliminated this entirely. No more journal corruption, no more SQLITE_BUSY, and concurrent writes just work.

Traefik + Tailscale: Zero-Trust Routing

I exposed Open WebUI through Traefik with Tailscale as the ingress boundary:

# traefik/dynamic/openwebui.yaml
http:
  routers:
    openwebui:
      rule: "Host(`ai.tailnet-name.ts.net`)"
      service: openwebui
      tls:
        certResolver: tailscale

  services:
    openwebui:
      loadBalancer:
        servers:
          - url: "http://open-webui:8080"

Traefik listens on Tailscale's local IP, and the ACL policy restricts access:

{
  "acls": [
    {
      "action": "accept",
      "src": ["group:engineering"],
      "dst": ["ai-server:443"]
    }
  ]
}

This avoids exposing Open WebUI to the public internet while still allowing remote access for authorized users. No mTLS complexity, no VPN overhead, just Tailscale's identity-based routing.

The Unresolved Dilemma: Cost vs. Latency vs. Model Quality

Here's where I'm still iterating: local models like Qwen 2.5 give great latency and zero per-token cost, but they occasionally produce lower-quality reasoning compared to GPT-4 or Claude Opus for complex technical queries. The fallback chain works, but now I'm paying OpenAI's API costs for what could have been a local inference.

Do you optimize for cost and accept degraded quality on fallback? Or do you set Ollama's timeout so aggressively that you always fail over to paid APIs, sacrificing the cost savings of self-hosting? I haven't found a clean middle ground yet. Curious how others are thinking about this trade-off in production homelab setups.