GPUStack With vLLM in Production: Pinning KV Cache Before the OOM Finds You

We run a small on-prem inference fleet - four nodes, mixed A6000 and 4090 cards - and last quarter we replaced a pile of hand-rolled docker run wrappers with GPUStack. The pitch is reasonable: one control plane that schedules models across whatever GPUs you happen to own, materialises a backend per deployment, and exposes an OpenAI-compatible endpoint. Two months in, it does what it says. It also has a specific failure boundary that cost us a weekend, and that part is worth writing down.

Where it fits, and where it stops fitting

GPUStack is a placement and lifecycle layer, not an inference engine. When you deploy a model it picks a worker, starts vLLM, SGLang, or llama-box underneath, and proxies requests. That is the right shape for a team that wants model-per-endpoint without writing scheduler code.

The boundary: placement is driven by a VRAM estimate derived from weight size plus a declared context length. That estimate knows nothing about activation peaks, CUDA graph capture, or your actual prompt distribution. Accept the defaults and you are trusting a number that was never designed as a runtime guarantee.

The 70B OOM

We deployed a 70B in bf16 across four cards at TP=4 and left max-model-len at the model default. Placement succeeded, the deployment reported healthy, and then under real traffic - long system prompts, 6k to 8k token requests - workers OOM'd one at a time and GPUStack restarted them. It looked like scheduler instability. It was arithmetic.

Llama-3.3-70B KV cost is roughly 320 KB per token: 80 layers x 8 KV heads x 128 head dim x 2 (K and V) x 2 bytes. At 32k context that is about 10.7 GB per sequence of pure KV, on top of roughly 35 GB of weights per card. Two concurrent long sequences and the card is done. The estimator was treating context as a declared budget, not as a per-sequence allocation multiplied by concurrency.

The fix is boring. Pin max-model-len, pin gpu-memory-utilization below the point where graph capture still fits, and cap max-num-seqs so the KV pool cannot be oversubscribed.

Hardened serve block

This is what runs per 70B worker now, passed through as backend arguments in the deployment spec:

vllm serve meta-llama/Llama-3.3-70B-Instruct \
  --host 0.0.0.0 --port 8000 \
  --tensor-parallel-size 4 \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.88 \
  --kv-cache-dtype fp8 \
  --max-num-seqs 32 \
  --enable-chunked-prefill \
  --swap-space 0 \
  --disable-log-requests

Only four of those flags actually matter for stability:

• --gpu-memory-utilization 0.88 - at 0.92 we OOM'd with identical weights. Graph capture at TP=4 does not fit in the leftover margin people assume it does.
• --max-model-len 16384 - halves the KV ceiling. Measure your real prompt p99 first; ours was 11k, so 16k buys headroom without doubling the pool.
• --swap-space 0 - deliberate. Host swapping converts an honest crash into a 40x latency cliff that looks like a network problem at 2am.
• --kv-cache-dtype fp8 - halves KV, no measurable regression on our eval subset. Long-context retrieval degrades more than chat; test before enabling.

On the heterogeneous node where TP is not possible, GPUStack falls back to llama-box and layer offload. Same arithmetic, different knob:

FROM llama3.3:70b-instruct-q4_K_M
PARAMETER num_ctx 8192
PARAMETER num_gpu 60
PARAMETER num_batch 256

num_gpu 60 of 80 layers on 48 GB is the honest ceiling once KV and compute buffers are counted. Setting it to 80 gives you a model that loads and dies on the first 4k prompt.

Measuring TTFT and the slow leak

The second problem was subtler: throughput decayed about 18% over 36 hours with flat GPU utilisation. That pattern is either preemption thrash or KV block growth, and the only way to distinguish them is TTFT and RSS measured together.

import asyncio, aiohttp, json, time, psutil

MODEL = "meta-llama/Llama-3.3-70B-Instruct"

def pct(xs, p):
    xs = sorted(xs)
    return xs[min(len(xs) - 1, int(round(p / 100 * (len(xs) - 1))))]

async def stream_once(session, url, prompt):
    payload = {"model": MODEL, "stream": True, "max_tokens": 256,
               "temperature": 0.0, "messages": [{"role": "user", "content": prompt}]}
    t0 = time.perf_counter(); ttft = None; chars = 0
    async with session.post(url, json=payload) as resp:
        resp.raise_for_status()
        async for raw in resp.content:
            line = raw.strip()
            if not line.startswith(b"data:"):
                continue
            body = line[5:].strip()
            if body == b"[DONE]":
                break
            try:
                obj = json.loads(body)
            except json.JSONDecodeError:
                continue
            delta = obj["choices"][0].get("delta", {}).get("content")
            if delta:
                if ttft is None:
                    ttft = time.perf_counter() - t0
                chars += len(delta)
    return ttft, time.perf_counter() - t0, chars

async def run(url, prompts, concurrency=8):
    conn = aiohttp.TCPConnector(limit=concurrency)
    timeout = aiohttp.ClientTimeout(total=180)
    results = []
    async with aiohttp.ClientSession(connector=conn, timeout=timeout) as s:
        for i in range(0, len(prompts), concurrency):
            batch = prompts[i:i + concurrency]
            results += await asyncio.gather(*[stream_once(s, url, p) for p in batch])
    ttfts = [r[0] * 1000 for r in results]
    totals = [r[1] for r in results]
    toks = [r[2] for r in results]
    return {"ttft_p50_ms": round(pct(ttfts, 50), 1),
            "ttft_p95_ms": round(pct(ttfts, 95), 1),
            "chars_per_s": round(sum(toks) / sum(totals), 1)}

def rss_mb(pid):
    return psutil.Process(pid).memory_info().rss / 2**20

async def soak(pid, url, prompts, rounds=6, per_round=50):
    base = rss_mb(pid)
    for r in range(rounds):
        batch = (prompts * (per_round // len(prompts) + 1))[:per_round]
        m = await run(url, batch)
        print(f"round {r} rss=+{rss_mb(pid) - base:.0f}MB ttft_p95={m['ttft_p95_ms']}ms chars/s={m['chars_per_s']}")

Run it against the local endpoint. The absolute TTFT is not the signal. The signal is the RSS slope across rounds and TTFT p95 drifting independently of offered load. Monotonic RSS with p95 tracking it means KV pool growth. Rising p95 with flat RSS means placement or preemption. We had the first: block fragmentation across wildly varying prompt lengths. It was resolved by pinning --max-num-seqs 32 and recycling workers on an RSS-slope threshold instead of a fixed cron.

For a control we point the same script at B-Lost's relay (https://b-lost.com/v1) with the same model name. Because it fronts multiple providers behind one OpenAI-compatible surface, it gives us a latency reference that our own node state cannot contaminate. On the last soak, pulling the fallback path through the relay cut measured TTFT p50 by 35 ms on the same prompt set - it routed around our degraded node - and, more usefully, its p50 stayed inside a 35 ms band for the whole run while ours drifted upward. That is what told us the decay was local.

The dilemma we have not solved

Sizing max-num-seqs correctly requires knowing your workload, and we serve one endpoint for two of them: an interactive chat UI with short prompts and high concurrency, and a nightly batch summarisation job with 30k token prompts and near-zero concurrency. Two deployments doubles resident weights. One deployment gives one of the two workloads the wrong pool shape, and we already know which one loses. Per-request routing between two pools is the obvious answer and adds a hop we do not currently have the budget to debug properly. If you have solved this without buying another node, I would like to hear which side you put the queue on.