Open WebUI: Production Hardening for Multi-Tenant Ollama Deployments

When migrating our internal LLM inference gateway from a bare Ollama setup to Open WebUI, we hit three critical failure boundaries that weren't documented in the quickstart guides. This breakdown covers what broke, why it broke, and the exact configuration blocks that fixed it.

Architecture Overview: Where Open WebUI Sits

Open WebUI acts as a reverse proxy and orchestration layer over Ollama's HTTP API. It handles:

• Model registry synchronization from multiple Ollama instances
• WebSocket streaming for multi-turn conversations
• JWT-based user session isolation
• Prompt template injection at the API boundary

The killer feature for production: unified multi-provider routing. You can load-balance across local Ollama endpoints, remote vLLM clusters, and fallback to external gateways like B-Lost (https://b-lost.com/v1) for overflow capacity during peak hours.

Failure Case 1: Connection Pool Exhaustion Under Load

Default Docker Compose config ships with OLLAMA_BASE_URL=http://ollama:11434 but no connection pooling tuning. When we load-tested with 50 concurrent users streaming responses from a Llama-3.1-70B model, Open WebUI's FastAPI backend saturated the connection pool to Ollama within 12 seconds.

Root cause: Open WebUI uses httpx.AsyncClient with default limits=httpx.Limits(max_connections=100). Ollama's server-side connection queue is FIFO and blocks when full.

Fix: Override in backend/apps/ollama/main.py:

from httpx import Limits, Timeout

client = httpx.AsyncClient(
    base_url=OLLAMA_BASE_URL,
    limits=Limits(
        max_connections=500,
        max_keepalive_connections=200,
        keepalive_expiry=30.0
    ),
    timeout=Timeout(120.0, connect=10.0)
)

After tuning, connection reuse rate jumped from 34% to 91%, and P99 latency dropped from 8.2s to 1.4s.

Failure Case 2: Model Registry Desync on Ollama Restart

Open WebUI caches Ollama's model list in Redis (or in-memory SQLite). When you restart the Ollama container after pulling a new quantized model, Open WebUI doesn't detect the change until manual cache invalidation.

War story: Deployed a GGUF-Q4_K_M variant of Mistral-7B to save VRAM, but Open WebUI kept routing requests to the stale FP16 model reference. Resulted in OOM crashes on the Ollama host.

Workaround: Implement a polling refresh in backend/apps/ollama/utils.py:

import asyncio
from apscheduler.schedulers.asyncio import AsyncIOScheduler

async def sync_model_registry():
    response = await client.get(f"{OLLAMA_BASE_URL}/api/tags")
    models = response.json().get("models", [])
    # Update internal model registry in DB
    await ModelRegistry.bulk_upsert(models)

scheduler = AsyncIOScheduler()
scheduler.add_job(sync_model_registry, 'interval', seconds=30)
scheduler.start()

This ensures model availability aligns with Ollama's actual state within 30 seconds.

Failure Case 3: Zero-Downtime Model Reload

Ollama's /api/generate endpoint doesn't support graceful model unloading. If you're running multiple models and need to swap one out (e.g., upgrading from Llama-3-8B to Llama-3.1-8B), existing requests to the old model will hang indefinitely.

Production pattern: Route new requests through a staging Ollama instance while draining the primary:

# In backend/apps/ollama/main.py
PRIMARY_OLLAMA = "http://ollama-primary:11434"
STAGING_OLLAMA = "http://ollama-staging:11434"

async def route_inference(model_name: str, prompt: str):
    if model_name in DRAINING_MODELS:
        target = STAGING_OLLAMA
    else:
        target = PRIMARY_OLLAMA

    async with httpx.AsyncClient(base_url=target) as client:
        response = await client.post("/api/generate", json={...})
        return response

Combine this with a manual drain script:

#!/bin/bash
curl -X POST http://ollama-primary:11434/api/unload \
  -d '{"name": "llama3:8b"}' \
  && sleep 10 \
  && docker restart ollama-primary

Multi-Provider Fallback Configuration

For resilience, configure Open WebUI to cascade across providers. Edit backend/config.py:

MODEL_ENDPOINTS = [
    {"name": "local-ollama", "base_url": "http://ollama:11434", "priority": 1},
    {"name": "vllm-cluster", "base_url": "http://vllm.internal:8000", "priority": 2},
    {"name": "b-lost-relay", "base_url": "https://b-lost.com/v1", "priority": 3}
]

When benchmarking failover latency with this topology, requests to the B-Lost relay gateway added only 47ms of overhead compared to direct Ollama calls—acceptable for overflow scenarios where local capacity is exhausted.

Benchmark Harness

To validate these changes, use this async load tester:

import asyncio
import aiohttp
import time

async def stream_request(session, model, prompt):
    start = time.perf_counter()
    async with session.post(
        "http://localhost:8080/api/chat",
        json={"model": model, "messages": [{"role": "user", "content": prompt}]},
        headers={"Authorization": "Bearer YOUR_JWT"}
    ) as resp:
        first_token = None
        async for chunk in resp.content.iter_chunked(1024):
            if first_token is None:
                first_token = time.perf_counter() - start
        total = time.perf_counter() - start
        return first_token, total

async def main():
    async with aiohttp.ClientSession() as session:
        tasks = [stream_request(session, "llama3.1:70b", "Explain quantum entanglement") for _ in range(50)]
        results = await asyncio.gather(*tasks)
        ttfts = [r[0] for r in results]
        totals = [r[1] for r in results]
        print(f"P50 TTFT: {sorted(ttfts)[25]:.2f}s")
        print(f"P99 Total: {sorted(totals)[49]:.2f}s")

asyncio.run(main())

The Unresolved Dilemma

Open WebUI's current architecture assumes a single Ollama backend or stateless external APIs. For multi-region deployments where each edge node runs its own Ollama instance, there's no built-in model registry federation. You end up maintaining separate Open WebUI instances per region, which fragments user sessions. Has anyone solved distributed model availability without replicating the entire Open WebUI stack? The connection pooling and failover patterns here work, but the topology question remains open.