DEV Community

Cover image for Running Unsloth in Docker with OpenAI-Compatible API Relay and Unbuffered SSE
Dave Mille
Dave Mille

Posted on

Running Unsloth in Docker with OpenAI-Compatible API Relay and Unbuffered SSE

#ai #ollama #machinelearning #opensource

Running Unsloth in Docker with OpenAI-Compatible API Relay and Unbuffered SSE

When testing unslothai/unsloth for local LLM inference and fine-tuning, I needed a production-grade Docker deployment that could:

  1. Run the Unsloth container with GPU pass-through
  2. Expose an OpenAI-compatible API for downstream tools (Claude Code, Codex, etc.)
  3. Integrate third-party API relays (OpenRouter, DeepSeek) as fallback routes
  4. Prevent SSE (Server-Sent Events) buffering at the reverse proxy layer

This post documents the exact configs that worked in my homelab.

Docker Compose with Health Checks and GPU Support

Here's the minimal docker-compose.yml for running Unsloth with NVIDIA GPU runtime:

version: '3.8'

services:
  unsloth:
    image: unsloth/unsloth:latest
    container_name: unsloth
    restart: unless-stopped
    runtime: nvidia
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
      - UNSLOTH_API_KEY=${UNSLOTH_API_KEY}
      - OPENAI_BASE_URL=http://localhost:8000/v1
      - OPENAI_API_KEY=${RELAY_API_KEY}
    ports:
      - "8000:8000"
    volumes:
      - ./models:/root/.cache/huggingface
      - ./config:/app/config
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 60s
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
Enter fullscreen mode Exit fullscreen mode

Key points:

  • runtime: nvidia + NVIDIA_VISIBLE_DEVICES=all ensures GPU access
  • Health check probes /health endpoint (adjust if Unsloth uses a different path)
  • Volumes persist model weights and config across restarts
  • OPENAI_BASE_URL allows chaining to external relays (e.g., OpenRouter)

Caddy Reverse Proxy with Unbuffered SSE

To prevent response buffering (critical for streaming LLM outputs), here's the Caddyfile:

unsloth.local {
    reverse_proxy localhost:8000 {
        flush_interval -1
        header_up X-Real-IP {remote_host}
        header_up X-Forwarded-For {remote_host}
    }
}
Enter fullscreen mode Exit fullscreen mode

Why flush_interval -1 matters:
By default, reverse proxies buffer responses to optimize throughput. For SSE endpoints (/v1/chat/completions with stream=true), buffering causes 10–30 second delays before the first token appears. Setting flush_interval -1 disables buffering.

Nginx Alternative (if you prefer Nginx) 

server {
    listen 80;
    server_name unsloth.local;

    location / {
        proxy_pass http://localhost:8000;
        proxy_buffering off;
        proxy_set_header X-Accel-Buffering no;
        proxy_set_header Connection '';
        proxy_http_version 1.1;
        chunked_transfer_encoding off;
    }
}
Enter fullscreen mode Exit fullscreen mode

Critical directives:

  • proxy_buffering off;
  • X-Accel-Buffering: no;
  • chunked_transfer_encoding off; (prevents Nginx from rechunking the stream)

Verifying Unbuffered Streaming from Terminal

Once deployed, test that tokens stream immediately:

curl -N -X POST http://unsloth.local/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $UNSLOTH_API_KEY" \
  -d '{
    "model": "unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL",
    "messages": [{"role": "user", "content": "Count to 10 slowly"}],
    "stream": true
  }'
Enter fullscreen mode Exit fullscreen mode

Expected behavior:
You should see data: {"choices":[...]} lines appear in real-time, not all at once after 10+ seconds.

If responses are buffered, double-check:

  1. Caddy/Nginx flush settings
  2. Docker healthcheck isn't blocking traffic during startup
  3. Upstream relay (if used) supports streaming

Integrating Third-Party API Relays

To route requests through OpenRouter or similar gateways, set OPENAI_BASE_URL in the container:

OPENAI_BASE_URL=https://openrouter.ai/api/v1
OPENAI_API_KEY=sk-or-v1-...
Enter fullscreen mode Exit fullscreen mode

Unsloth's OpenAI-compatible server will forward requests to the specified base URL, allowing you to:

  • Use models not available locally (Claude, GPT-4, etc.)
  • Offload compute during peak load
  • A/B test local vs. cloud inference

Lessons Learned

  1. Health checks prevent premature traffic routing – Without start_period: 60s, Caddy routes requests before model loading completes, causing 502 errors.
  2. SSE buffering is silent – Your logs will show 200 OK, but the client waits forever. Always test with curl -N.
  3. GPU memory fragmentation – Restart the container daily if running multiple models; VRAM doesn't always release cleanly.

References

Top comments (0)