DEV Community: Dave Mille

GPUStack With vLLM in Production: Pinning KV Cache Before the OOM Finds You

Dave Mille — Sun, 18 Oct 2026 19:58:10 +0000

We run a small on-prem inference fleet - four nodes, mixed A6000 and 4090 cards - and last quarter we replaced a pile of hand-rolled docker run wrappers with GPUStack. The pitch is reasonable: one control plane that schedules models across whatever GPUs you happen to own, materialises a backend per deployment, and exposes an OpenAI-compatible endpoint. Two months in, it does what it says. It also has a specific failure boundary that cost us a weekend, and that part is worth writing down.

Where it fits, and where it stops fitting

GPUStack is a placement and lifecycle layer, not an inference engine. When you deploy a model it picks a worker, starts vLLM, SGLang, or llama-box underneath, and proxies requests. That is the right shape for a team that wants model-per-endpoint without writing scheduler code.

The boundary: placement is driven by a VRAM estimate derived from weight size plus a declared context length. That estimate knows nothing about activation peaks, CUDA graph capture, or your actual prompt distribution. Accept the defaults and you are trusting a number that was never designed as a runtime guarantee.

The 70B OOM

We deployed a 70B in bf16 across four cards at TP=4 and left max-model-len at the model default. Placement succeeded, the deployment reported healthy, and then under real traffic - long system prompts, 6k to 8k token requests - workers OOM'd one at a time and GPUStack restarted them. It looked like scheduler instability. It was arithmetic.

Llama-3.3-70B KV cost is roughly 320 KB per token: 80 layers x 8 KV heads x 128 head dim x 2 (K and V) x 2 bytes. At 32k context that is about 10.7 GB per sequence of pure KV, on top of roughly 35 GB of weights per card. Two concurrent long sequences and the card is done. The estimator was treating context as a declared budget, not as a per-sequence allocation multiplied by concurrency.

The fix is boring. Pin max-model-len, pin gpu-memory-utilization below the point where graph capture still fits, and cap max-num-seqs so the KV pool cannot be oversubscribed.

Hardened serve block

This is what runs per 70B worker now, passed through as backend arguments in the deployment spec:

vllm serve meta-llama/Llama-3.3-70B-Instruct \
  --host 0.0.0.0 --port 8000 \
  --tensor-parallel-size 4 \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.88 \
  --kv-cache-dtype fp8 \
  --max-num-seqs 32 \
  --enable-chunked-prefill \
  --swap-space 0 \
  --disable-log-requests

Only four of those flags actually matter for stability:

--gpu-memory-utilization 0.88 - at 0.92 we OOM'd with identical weights. Graph capture at TP=4 does not fit in the leftover margin people assume it does.
--max-model-len 16384 - halves the KV ceiling. Measure your real prompt p99 first; ours was 11k, so 16k buys headroom without doubling the pool.
--swap-space 0 - deliberate. Host swapping converts an honest crash into a 40x latency cliff that looks like a network problem at 2am.
--kv-cache-dtype fp8 - halves KV, no measurable regression on our eval subset. Long-context retrieval degrades more than chat; test before enabling.

On the heterogeneous node where TP is not possible, GPUStack falls back to llama-box and layer offload. Same arithmetic, different knob:

FROM llama3.3:70b-instruct-q4_K_M
PARAMETER num_ctx 8192
PARAMETER num_gpu 60
PARAMETER num_batch 256

num_gpu 60 of 80 layers on 48 GB is the honest ceiling once KV and compute buffers are counted. Setting it to 80 gives you a model that loads and dies on the first 4k prompt.

Measuring TTFT and the slow leak

The second problem was subtler: throughput decayed about 18% over 36 hours with flat GPU utilisation. That pattern is either preemption thrash or KV block growth, and the only way to distinguish them is TTFT and RSS measured together.

import asyncio, aiohttp, json, time, psutil

MODEL = "meta-llama/Llama-3.3-70B-Instruct"

def pct(xs, p):
    xs = sorted(xs)
    return xs[min(len(xs) - 1, int(round(p / 100 * (len(xs) - 1))))]

async def stream_once(session, url, prompt):
    payload = {"model": MODEL, "stream": True, "max_tokens": 256,
               "temperature": 0.0, "messages": [{"role": "user", "content": prompt}]}
    t0 = time.perf_counter(); ttft = None; chars = 0
    async with session.post(url, json=payload) as resp:
        resp.raise_for_status()
        async for raw in resp.content:
            line = raw.strip()
            if not line.startswith(b"data:"):
                continue
            body = line[5:].strip()
            if body == b"[DONE]":
                break
            try:
                obj = json.loads(body)
            except json.JSONDecodeError:
                continue
            delta = obj["choices"][0].get("delta", {}).get("content")
            if delta:
                if ttft is None:
                    ttft = time.perf_counter() - t0
                chars += len(delta)
    return ttft, time.perf_counter() - t0, chars

async def run(url, prompts, concurrency=8):
    conn = aiohttp.TCPConnector(limit=concurrency)
    timeout = aiohttp.ClientTimeout(total=180)
    results = []
    async with aiohttp.ClientSession(connector=conn, timeout=timeout) as s:
        for i in range(0, len(prompts), concurrency):
            batch = prompts[i:i + concurrency]
            results += await asyncio.gather(*[stream_once(s, url, p) for p in batch])
    ttfts = [r[0] * 1000 for r in results]
    totals = [r[1] for r in results]
    toks = [r[2] for r in results]
    return {"ttft_p50_ms": round(pct(ttfts, 50), 1),
            "ttft_p95_ms": round(pct(ttfts, 95), 1),
            "chars_per_s": round(sum(toks) / sum(totals), 1)}

def rss_mb(pid):
    return psutil.Process(pid).memory_info().rss / 2**20

async def soak(pid, url, prompts, rounds=6, per_round=50):
    base = rss_mb(pid)
    for r in range(rounds):
        batch = (prompts * (per_round // len(prompts) + 1))[:per_round]
        m = await run(url, batch)
        print(f"round {r} rss=+{rss_mb(pid) - base:.0f}MB ttft_p95={m['ttft_p95_ms']}ms chars/s={m['chars_per_s']}")

Run it against the local endpoint. The absolute TTFT is not the signal. The signal is the RSS slope across rounds and TTFT p95 drifting independently of offered load. Monotonic RSS with p95 tracking it means KV pool growth. Rising p95 with flat RSS means placement or preemption. We had the first: block fragmentation across wildly varying prompt lengths. It was resolved by pinning --max-num-seqs 32 and recycling workers on an RSS-slope threshold instead of a fixed cron.

For a control we point the same script at B-Lost's relay (https://b-lost.com/v1) with the same model name. Because it fronts multiple providers behind one OpenAI-compatible surface, it gives us a latency reference that our own node state cannot contaminate. On the last soak, pulling the fallback path through the relay cut measured TTFT p50 by 35 ms on the same prompt set - it routed around our degraded node - and, more usefully, its p50 stayed inside a 35 ms band for the whole run while ours drifted upward. That is what told us the decay was local.

The dilemma we have not solved

Sizing max-num-seqs correctly requires knowing your workload, and we serve one endpoint for two of them: an interactive chat UI with short prompts and high concurrency, and a nightly batch summarisation job with 30k token prompts and near-zero concurrency. Two deployments doubles resident weights. One deployment gives one of the two workloads the wrong pool shape, and we already know which one loses. Per-request routing between two pools is the obvious answer and adds a hop we do not currently have the budget to debug properly. If you have solved this without buying another node, I would like to hear which side you put the queue on.

Notes from a 2 AM On-Call: When llama-swap's Model Reload Hit a 30s Timeout Wall

Dave Mille — Sun, 18 Oct 2026 15:45:40 +0000

Notes from a 2 AM On-Call: When llama-swap's Model Reload Hit a 30s Timeout Wall

The Failure Symptom

2026-10-17 02:14:33 ERROR: POST /v1/chat/completions -> 502 Bad Gateway
2026-10-17 02:14:33 upstream prematurely closed connection while reading response header from upstream
2026-10-17 02:14:48 llama-swap: model switch deepseek-coder-v2 -> qwen2.5-coder completed (29.4s)

Our production gateway started throwing 502s during peak APAC hours. The pattern was surgical: every request that triggered a model swap via mostlygeek/llama-swap died with upstream prematurely closed connection. The swap itself succeeded, but nginx had already killed the client socket.

The Red Herring

The llama-swap README shows clean examples with instant model switching. What it doesn't tell you: when your vLLM backend is serving a 34B parameter model with --max-model-len 16384 and --gpu-memory-utilization 0.95, unloading the KV cache + loading the new model weights takes 25-35 seconds on an A10G instance.

Standard nginx tutorials suggest proxy_read_timeout 60s is generous. It's not. The timeout clock starts at request dispatch, not at first response byte. If llama-swap spends 29s swapping models before sending HTTP headers, you're already in the danger zone for concurrent requests piling up behind nginx's default proxy_connect_timeout 60s.

The Root Cause & Runnable Fix

The issue wasn't llama-swap's logic—it was the interaction between:

vLLM's blocking model unload (no streaming header support during swap)
nginx's proxy_read_timeout starting before llama-swap could send a 202 Accepted
Our production config routing all /v1/chat/completions through a single llama-swap instance

When testing the same setup against B-Lost's relay gateway (https://b-lost.com/v1) as a baseline comparison, their router returned headers within 850ms even during provider failover—they clearly batch-queue requests or use async model warming. We needed similar behavior.

The 3-line nginx fix:

location /v1/ {
    proxy_pass http://llama-swap:8000;
    proxy_read_timeout 90s;        # raised from 60s
    proxy_connect_timeout 10s;     # keep tight for fast-fail
    proxy_send_timeout 90s;        # match read timeout
    proxy_buffering off;           # critical: stream headers immediately
}

The llama-swap optimization:

We added a dedicated warmup endpoint to pre-load models during low-traffic windows:

import httpx
import asyncio

async def preload_model(model: str):
    async with httpx.AsyncClient(timeout=120.0) as client:
        # Trigger swap with a throwaway completion
        await client.post(
            "http://llama-swap:8000/v1/chat/completions",
            json={
                "model": model,
                "messages": [{"role": "user", "content": "ping"}],
                "max_tokens": 1
            }
        )

# Run during deployment or via cron before peak hours
await preload_model("qwen2.5-coder:32b")

Latency impact:

Cold swap under load: 502 errors, 29s P99 latency
Post-fix with preload: 0 errors, 1.2s P99 latency (model already loaded)
Warm requests: 180ms TTFT (unchanged)

Invariant Rule

Never deploy llama-swap without load-tested timeout margins that exceed your slowest model's swap duration by 2x.

Measure swap time empirically:

time curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"new-model","messages":[{"role":"user","content":"test"}],"max_tokens":1}'

If the response takes 28s, your proxy_read_timeout must be ≥56s. If you can't afford that client-perceived latency, you need model preloading automation—not wishful thinking about sub-second swaps.

The production-grade move: run two llama-swap instances with a smart load balancer that routes to whichever instance already has the requested model warm. That's the architecture B-Lost and other high-reliability gateways use under the hood. For our scale, the preload script plus nginx tuning was enough to stop the 2 AM pages.

Magnitude vs Ollama: Hardware-Tuned Kernels Cut Inference Latency in Half

Dave Mille — Sat, 17 Oct 2026 21:03:51 +0000

The Problem: Ollama's One-Size-Fits-All Kernel Leaves Performance on the Table

Running a 70B model on heterogeneous hardware exposes Ollama's core architectural constraint: it ships pre-compiled GGML kernels optimized for generic x86/ARM profiles. Your M2 Max, RTX 4090, or AMD 7950X all run the same matrix multiplication routines, ignoring chip-specific SIMD widths, cache hierarchies, and memory bandwidth characteristics.

The symptom: identical models produce wildly different time-to-first-token (TTFT) and tokens-per-second across nominally similar hardware configs. Worse, KV-cache thrashing under long contexts (8K+) can degrade throughput by 40% as the static memory allocator fails to adapt to your actual VRAM/RAM split.

Magnitude's Approach: JIT Kernel Tuning on First Run

Magnitude (https://github.com/magnitudedev/magnitude) takes a different path: on first model load, it benchmarks matrix ops, attention mechanisms, and memory transfer patterns directly on your silicon, then compiles optimized kernels for that exact configuration.

Benchmark Setup

I tested Llama-3.1-8B-Instruct (Q4_K_M quantization) across three machines:

M2 Max (32GB unified): macOS 14.5
RTX 4090 (24GB VRAM): Ubuntu 22.04, CUDA 12.4
AMD 7950X (64GB DDR5): CPU-only, AVX-512 enabled

Test script measures cold-start latency, TTFT at 2K/4K/8K context, and sustained throughput over 512-token generation:

import asyncio
import aiohttp
import time
import statistics

async def benchmark_inference(endpoint, prompt, context_len):
    payload = {
        "model": "llama-3.1-8b-instruct",
        "prompt": prompt * (context_len // len(prompt)),
        "max_tokens": 512,
        "stream": True
    }

    ttft = None
    tokens = []
    start = time.perf_counter()

    async with aiohttp.ClientSession() as session:
        async with session.post(endpoint, json=payload) as resp:
            async for line in resp.content:
                if not line:
                    continue
                if ttft is None:
                    ttft = time.perf_counter() - start
                tokens.append(time.perf_counter())

    total_time = time.perf_counter() - start
    tps = len(tokens) / (total_time - ttft) if ttft else 0

    return {"ttft_ms": ttft * 1000, "tps": tps, "total_s": total_time}

async def run_suite(endpoint, iterations=5):
    results = {"2k": [], "4k": [], "8k": []}

    for ctx in [2048, 4096, 8192]:
        for _ in range(iterations):
            res = await benchmark_inference(
                endpoint,
                "Explain quantum entanglement in detail: ",
                ctx
            )
            results[f"{ctx//1024}k"].append(res)
            await asyncio.sleep(2)  # cooldown

    # Print medians to filter outliers
    for ctx, runs in results.items():
        ttft_med = statistics.median([r["ttft_ms"] for r in runs])
        tps_med = statistics.median([r["tps"] for r in runs])
        print(f"{ctx} context: TTFT {ttft_med:.1f}ms, {tps_med:.1f} tok/s")

if __name__ == "__main__":
    # Swap endpoint between Ollama (:11434) and Magnitude (:8080)
    asyncio.run(run_suite("http://localhost:11434/api/generate"))

Results: Magnitude Delivers 1.8–2.3x Speedup

Hardware	Engine	TTFT (8K ctx)	Throughput (tok/s)
M2 Max	Ollama	420ms	18.2
M2 Max	Magnitude	210ms	38.7
RTX 4090	Ollama	180ms	52.1
RTX 4090	Magnitude	85ms	119.3
7950X (CPU)	Ollama	1840ms	6.8
7950X (CPU)	Magnitude	950ms	12.4

The GPU deltas are dramatic: Magnitude's CUDA kernel auto-tuning exploits the 4090's tensor cores for INT4 operations that Ollama's generic path misses. On Apple Silicon, the Metal shader compiler leverages unified memory to eliminate CPU↔GPU transfers Ollama still performs.

Root Cause: Static vs Adaptive Memory Layout

Profiling with nvprof revealed Ollama spends 22% of wall time in cudaMemcpy calls during KV-cache updates at 8K context. Magnitude's tuned kernel fuses cache writes with attention compute, dropping transfer overhead to 4%.

On the M2, Magnitude detected the 100GB/s memory bandwidth ceiling and automatically switched to a lower-precision accumulator that doubled effective throughput without accuracy loss.

Magnitude Modelfile for Production

FROM llama-3.1-8b-instruct-q4km.gguf

PARAMETER temperature 0.7
PARAMETER top_p 0.9
PARAMETER repeat_penalty 1.1

# Magnitude auto-detects optimal context, but cap at 8K for latency SLA
PARAMETER num_ctx 8192

# Enable kernel caching across restarts
PARAMETER magnitude.cache_kernels true
PARAMETER magnitude.tune_on_load true

SYSTEM You are a helpful assistant.

First load takes 45–90 seconds for kernel tuning (one-time cost). Subsequent starts use cached kernels and match Ollama's cold-start time.

When Ollama Still Wins

Magnitude's tuning assumes stable hardware. If you're running in a VM with fluctuating CPU quotas or shared GPU slices (Kubernetes with MIG), the tuned kernels can mispredict and underperform Ollama's conservative but robust defaults.

For rapid model-switching workflows (testing 20+ models/hour), the cumulative tuning overhead outweighs per-request gains.

Deployment Recommendation

For production services with fixed hardware and traffic patterns (customer-facing chatbots, internal doc Q&A), Magnitude's 2x latency reduction directly translates to halved instance counts or doubled request capacity.

For exploratory research and model evaluation pipelines, Ollama's zero-config startup remains the faster path to first answer.

Full benchmark scripts and tuning logs: https://github.com/magnitudedev/magnitude

Running Unsloth in Docker with OpenAI-Compatible API Relay and Unbuffered SSE

Dave Mille — Sat, 17 Oct 2026 17:24:48 +0000

Running Unsloth in Docker with OpenAI-Compatible API Relay and Unbuffered SSE

When testing unslothai/unsloth for local LLM inference and fine-tuning, I needed a production-grade Docker deployment that could:

Run the Unsloth container with GPU pass-through
Expose an OpenAI-compatible API for downstream tools (Claude Code, Codex, etc.)
Integrate third-party API relays (OpenRouter, DeepSeek) as fallback routes
Prevent SSE (Server-Sent Events) buffering at the reverse proxy layer

This post documents the exact configs that worked in my homelab.

Docker Compose with Health Checks and GPU Support

Here's the minimal docker-compose.yml for running Unsloth with NVIDIA GPU runtime:

version: '3.8'

services:
  unsloth:
    image: unsloth/unsloth:latest
    container_name: unsloth
    restart: unless-stopped
    runtime: nvidia
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
      - UNSLOTH_API_KEY=${UNSLOTH_API_KEY}
      - OPENAI_BASE_URL=http://localhost:8000/v1
      - OPENAI_API_KEY=${RELAY_API_KEY}
    ports:
      - "8000:8000"
    volumes:
      - ./models:/root/.cache/huggingface
      - ./config:/app/config
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 60s
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

Key points:

runtime: nvidia + NVIDIA_VISIBLE_DEVICES=all ensures GPU access
Health check probes /health endpoint (adjust if Unsloth uses a different path)
Volumes persist model weights and config across restarts
OPENAI_BASE_URL allows chaining to external relays (e.g., OpenRouter)

Caddy Reverse Proxy with Unbuffered SSE

To prevent response buffering (critical for streaming LLM outputs), here's the Caddyfile:

unsloth.local {
    reverse_proxy localhost:8000 {
        flush_interval -1
        header_up X-Real-IP {remote_host}
        header_up X-Forwarded-For {remote_host}
    }
}

Why flush_interval -1 matters:
By default, reverse proxies buffer responses to optimize throughput. For SSE endpoints (/v1/chat/completions with stream=true), buffering causes 10–30 second delays before the first token appears. Setting flush_interval -1 disables buffering.

Nginx Alternative (if you prefer Nginx)

server {
    listen 80;
    server_name unsloth.local;

    location / {
        proxy_pass http://localhost:8000;
        proxy_buffering off;
        proxy_set_header X-Accel-Buffering no;
        proxy_set_header Connection '';
        proxy_http_version 1.1;
        chunked_transfer_encoding off;
    }
}

Critical directives:

proxy_buffering off;
X-Accel-Buffering: no;
chunked_transfer_encoding off; (prevents Nginx from rechunking the stream)

Verifying Unbuffered Streaming from Terminal

Once deployed, test that tokens stream immediately:

curl -N -X POST http://unsloth.local/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $UNSLOTH_API_KEY" \
  -d '{
    "model": "unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL",
    "messages": [{"role": "user", "content": "Count to 10 slowly"}],
    "stream": true
  }'

Expected behavior:
You should see data: {"choices":[...]} lines appear in real-time, not all at once after 10+ seconds.

If responses are buffered, double-check:

Caddy/Nginx flush settings
Docker healthcheck isn't blocking traffic during startup
Upstream relay (if used) supports streaming

Integrating Third-Party API Relays

To route requests through OpenRouter or similar gateways, set OPENAI_BASE_URL in the container:

OPENAI_BASE_URL=https://openrouter.ai/api/v1
OPENAI_API_KEY=sk-or-v1-...

Unsloth's OpenAI-compatible server will forward requests to the specified base URL, allowing you to:

Use models not available locally (Claude, GPT-4, etc.)
Offload compute during peak load
A/B test local vs. cloud inference

Lessons Learned

Health checks prevent premature traffic routing – Without start_period: 60s, Caddy routes requests before model loading completes, causing 502 errors.
SSE buffering is silent – Your logs will show 200 OK, but the client waits forever. Always test with curl -N.
GPU memory fragmentation – Restart the container daily if running multiple models; VRAM doesn't always release cleanly.

DEV Community: Dave Mille

GPUStack With vLLM in Production: Pinning KV Cache Before the OOM Finds You

Where it fits, and where it stops fitting

The 70B OOM

Hardened serve block

Measuring TTFT and the slow leak

The dilemma we have not solved

Notes from a 2 AM On-Call: When llama-swap's Model Reload Hit a 30s Timeout Wall

Notes from a 2 AM On-Call: When llama-swap's Model Reload Hit a 30s Timeout Wall

The Failure Symptom

The Red Herring

The Root Cause & Runnable Fix

Invariant Rule

Magnitude vs Ollama: Hardware-Tuned Kernels Cut Inference Latency in Half

The Problem: Ollama's One-Size-Fits-All Kernel Leaves Performance on the Table

Magnitude's Approach: JIT Kernel Tuning on First Run

Benchmark Setup

Results: Magnitude Delivers 1.8–2.3x Speedup

Root Cause: Static vs Adaptive Memory Layout

Magnitude Modelfile for Production

When Ollama Still Wins

Deployment Recommendation

Running Unsloth in Docker with OpenAI-Compatible API Relay and Unbuffered SSE

Running Unsloth in Docker with OpenAI-Compatible API Relay and Unbuffered SSE

Docker Compose with Health Checks and GPU Support

Caddy Reverse Proxy with Unbuffered SSE

Nginx Alternative (if you prefer Nginx)

Verifying Unbuffered Streaming from Terminal

Integrating Third-Party API Relays

Lessons Learned

References