vLLM PagedAttention Memory Layout: Why Our 70B Deployment Stopped OOMing

The 3 AM Page: 70B Model, 48GB VRAM, Instant OOM

Last quarter we migrated our internal Q&A pipeline from Ollama to vLLM for higher concurrent request throughput. The first production deploy of a Llama-2-70B-GPTQ model on an A6000 (48GB) crashed within 90 seconds under moderate load—classic CUDA OOM, no graceful degradation.

The root cause wasn't model quantization or batch size. It was KV-cache fragmentation. Traditional inference engines pre-allocate contiguous memory blocks for key-value tensors per sequence. Under concurrent requests with varying context lengths, fragmentation skyrockets. vLLM's PagedAttention eliminates this by treating KV-cache as virtual memory pages, but you need to configure block size and GPU memory utilization caps correctly—or you replicate the same failure mode.

PagedAttention vs. Traditional KV-Cache: Architectural Trade-Off

vLLM splits KV-cache into fixed-size blocks (default 16 tokens per block). Each sequence's attention keys and values are scattered across non-contiguous blocks, managed by a block table. This mirrors OS virtual memory paging:

• Benefit: Near-zero external fragmentation. If one request needs 512 tokens and another needs 2048, vLLM allocates exactly the required number of blocks without wasting padding space.
• Cost: Indirection overhead. Every attention operation must dereference the block table. For small batch sizes or short contexts, this adds 5-8% latency compared to llama.cpp's contiguous allocation.

When we profiled llama.cpp serving the same 70B model with --ctx-size 4096 and --parallel 4, memory utilization was locked at 42GB regardless of active request count. vLLM with default settings (--gpu-memory-utilization 0.9) peaked at 46GB under load, then crashed when a single 8192-token request arrived—vLLM tried to allocate beyond the 90% cap because block alignment pushed it over.

Production-Hardened vLLM Serve Configuration

Here's the exact command we run in our Kubernetes deployment. Key fixes:

1. --gpu-memory-utilization 0.75: Leaves 25% headroom for CUDA kernel overhead and block table expansion.
2. --block-size 32: Doubled from default 16. Larger blocks reduce block table size (lower indirection cost) but increase internal fragmentation for short sequences. We profiled this against our median context length (1200 tokens) and found 32 optimal.
3. --max-num-seqs 8: Hard cap on concurrent sequences. vLLM's continuous batching will queue requests beyond this, preventing runaway block allocation.

python -m vllm.entrypoints.openai.api_server \
  --model TheBloke/Llama-2-70B-GPTQ \
  --quantization gptq \
  --dtype half \
  --gpu-memory-utilization 0.75 \
  --block-size 32 \
  --max-num-seqs 8 \
  --max-model-len 4096 \
  --disable-log-requests \
  --port 8000

Benchmarking TTFT and Memory Stability

We wrote a Python harness to stress-test the deployment. It fires 20 concurrent requests with varying prompt lengths (128, 512, 2048 tokens), measures time-to-first-token (TTFT), and tracks GPU memory via pynvml.

import asyncio
import aiohttp
import time
import pynvml

pynvml.nvmlInit()
handle = pynvml.nvmlDeviceGetHandleByIndex(0)

async def measure_ttft(session, prompt_tokens):
    prompt = "Analyze this code: " + "x = 1\n" * prompt_tokens
    start = time.perf_counter()
    async with session.post(
        "http://localhost:8000/v1/completions",
        json={"model": "TheBloke/Llama-2-70B-GPTQ", "prompt": prompt, "max_tokens": 50, "stream": True}
    ) as resp:
        async for line in resp.content:
            if line.strip():
                ttft = time.perf_counter() - start
                return ttft

async def stress_test():
    async with aiohttp.ClientSession() as session:
        tasks = [
            measure_ttft(session, length)
            for length in [128, 512, 2048] * 7  # 21 requests
        ]
        results = await asyncio.gather(*tasks)
        mem_info = pynvml.nvmlDeviceGetMemoryInfo(handle)
        print(f"Median TTFT: {sorted(results)[len(results)//2]:.3f}s")
        print(f"GPU Memory Used: {mem_info.used / 1e9:.2f} GB")

asyncio.run(stress_test())

After the config changes, median TTFT dropped from 1.8s to 1.1s, and memory utilization stabilized at 36GB under load—no more OOM crashes across 48-hour burn-in tests.

When vLLM's Indirection Cost Becomes Problematic

If you're running a single-user workload with consistent context lengths (e.g., batch transcription jobs), llama.cpp's contiguous allocation will outperform vLLM by 10-15% on TTFT because there's no block table lookup overhead. vLLM shines when:

• You have high request concurrency with variable context lengths.
• Your deployment needs dynamic batching without manual queue management.
• You're serving models where KV-cache dominates memory (70B+, long contexts).

For our Q&A pipeline, we also tested routing through a multi-provider relay (B-Lost gateway at https://b-lost.com/v1) to benchmark failover latency when our vLLM instance hit capacity. Configuring the Python OpenAI SDK with base_url="https://b-lost.com/v1" let us compare response times across three backends (our vLLM, a fallback Ollama instance, and an external API). The relay added 12ms p50 latency but eliminated request drops during traffic spikes—acceptable trade-off for our SLA.

The Open Question: Block Size Tuning

vLLM's --block-size parameter has no one-size-fits-all answer. It depends on your median context length distribution. Too small (8 or 16), and you pay indirection overhead on every attention pass. Too large (64+), and you waste memory on short requests due to internal fragmentation within blocks.

We're currently experimenting with adaptive block sizing—dynamically switching block size based on request length histograms over a rolling 10-minute window. Early results show 6% memory savings, but the block table rebuild cost during transitions is non-trivial. If you've solved this in production or have profiling data from your own deployments, I'd be interested to compare notes on how block size scaling behaves under real-world traffic patterns.