Self-Hosting Your First LLM: What the Tutorials Skip About GPU Memory

Here is the short version: the model weights are the smallest GPU-memory surprise you'll hit. A 7B model in FP16 needs about 14GB just for weights, but the KV cache — the per-request memory that grows with context length and batch size — is what actually decides whether your setup survives real traffic. Most "run an LLM on your GPU" tutorials load the weights, run one short prompt, and declare victory. Then you send a 6,000-token document at a batch of eight and it OOMs.

I've set this up enough times to know the failure isn't random. It's arithmetic you can do before you rent the GPU. This post is that arithmetic.

Why does a model that "fits" still run out of memory?

GPU memory for inference is four separate buckets, and tutorials only mention the first:

1. Model weights — fixed, predictable, load once.
2. KV cache — scales with context_length × batch_size. This is the one that bites.
3. Activations and CUDA overhead — the runtime itself, plus temporary tensors during a forward pass.
4. Fragmentation — memory that's technically free but unusable because it's in the wrong-sized holes.

The weights number is the one everyone quotes because it's easy. Parameter count × bytes-per-parameter:

Precision	Bytes/param	7B model	13B model	70B model
FP16/BF16	2	~14 GB	~26 GB	~140 GB
INT8	1	~7 GB	~13 GB	~70 GB
INT4 (GPTQ/AWQ/GGUF Q4)	~0.5	~4 GB	~7 GB	~38 GB

So a 7B model in 4-bit "fits" on a 24GB card with 20GB to spare, and the tutorial ends there. That leftover 20GB is not spare — it's your working budget for everything in buckets 2 through 4, and it disappears faster than you'd guess.

Takeaway: Weights tell you if the model loads; they tell you nothing about whether it serves traffic.

How big is the KV cache, really?

The KV cache stores the key and value tensors for every token already in the context so the model doesn't recompute them each step. Its size, per request, is roughly:

kv_bytes = 2 × num_layers × num_kv_heads × head_dim × seq_len × bytes_per_element

The 2 is for keys and values. Note num_kv_heads, not the full attention-head count — modern models use grouped-query attention (GQA), which is the single biggest reason KV cache is smaller than older formulas suggest. Worth checking your model's config, because it swings the number by 4-8×.

Worked example, Llama-3-8B-class architecture (32 layers, 8 KV heads, head_dim 128, FP16):

per_token = 2 × 32 × 8 × 128 × 2 bytes = 131,072 bytes ≈ 128 KB/token

That's about 1 GB for a single 8,192-token request. Now the part tutorials skip: this is per concurrent request. Serve a batch of 16 at that context length and you've spent ~16 GB on KV cache alone — more than the weights. On a 24GB card holding a 4-bit 8B model (~5 GB with overhead), you have maybe 17-18GB left, and you just watched a modest batch eat all of it.

Two levers shrink this:

• Shorter max_model_len. If you don't need 32k context, don't provision for it. Serving frameworks pre-reserve cache based on this ceiling.
• KV-cache quantization (FP8 or INT8 KV). Roughly halves the cache at a small quality cost. vLLM and TGI both support it; it's off by default.

Takeaway: Budget KV cache as per_token_KB × max_context × expected_concurrency before you pick a card — it often dwarfs the weights.

What does the runtime itself cost?

Even before a single token, loading CUDA and your inference framework claims memory. The CUDA context alone is typically a few hundred MB to over a gigabyte depending on driver and GPU. PyTorch's caching allocator reserves more. Frameworks like vLLM deliberately grab a large fraction of remaining VRAM up front (controlled by gpu_memory_utilization, default 0.9) to manage the KV cache themselves — which is great for throughput but means "nvidia-smi shows 90% used" is expected, not a leak.

Then there's fragmentation. Naive allocators hand out one contiguous block per request's KV cache. When requests of varying lengths come and go, you get Swiss-cheese memory: 4GB free, but no single 2GB hole. This is exactly the problem vLLM's PagedAttention solves — it pages the KV cache like an OS pages RAM, so non-contiguous free memory is usable. If you're comparing serving frameworks, this is the practical reason vLLM sustains higher concurrency than a plain transformers loop on the same card.

Takeaway: Reserve 1-2GB for runtime overhead as a floor, and prefer a paged-attention serving stack the moment you have concurrent requests.

Which framework for which situation?

These are the four I actually reach for, with honest limitations:

Tool	Best for	Real drawback
Ollama	Local dev, single user, "just run it"	Not built for high-concurrency serving; batching is limited
llama.cpp	CPU/GPU hybrid, low VRAM, edge boxes	GGUF quant setup is fiddly; peak throughput trails GPU-native servers
vLLM	Production serving, high concurrency	Heavier setup; needs a proper CUDA GPU; startup VRAM grab surprises people
TGI (Text Generation Inference)	Production serving with HF ecosystem	Tighter model-support window; also GPU-hungry at start

For a first self-host on a single 24GB consumer card (RTX 3090/4090), a 7-8B model quantized to 4-bit via Ollama or llama.cpp is the reliable starting point. When you move past one user, switch to vLLM and set max_model_len and gpu_memory_utilization deliberately rather than accepting defaults.

Takeaway: Match the framework to concurrency, not to model size — the model fits on the card either way; only one of them survives real traffic.

A quick sizing pass before you commit

Rather than guess, do this arithmetic with your actual model config (config.json has num_hidden_layers, num_key_value_heads, head_dim or hidden_size / num_attention_heads):

def vram_estimate_gb(params_b, bytes_per_param,
                     num_layers, num_kv_heads, head_dim,
                     max_ctx, concurrency,
                     kv_bytes=2, overhead_gb=2.0):
    weights = params_b * 1e9 * bytes_per_param / 1e9
    per_tok = 2 * num_layers * num_kv_heads * head_dim * kv_bytes
    kv = per_tok * max_ctx * concurrency / 1e9
    return round(weights + kv + overhead_gb, 1)

# Llama-3-8B, 4-bit weights, FP16 KV, 8k context, 8 concurrent
print(vram_estimate_gb(8, 0.5, 32, 8, 128, 8192, 8))  # ~14.5 GB

Bump concurrency to 24 and that same setup crosses 30GB — past a 24GB card. This ten-line function has saved me more grief than any benchmark, because it turns "will it work?" into a number you check before spending money on a bigger GPU or a cloud instance. Treat it as an estimate with ±15% slack for allocator behavior, not a guarantee.

Bottom line

If you're self-hosting your first LLM, size for the KV cache and overhead, not just the weights — the weights are the part that always fits. For a single user on one consumer GPU, a 4-bit 7-8B model under Ollama or llama.cpp is the safe first step. The moment you have concurrent requests, move to vLLM or TGI, cap max_model_len to what you actually need, and consider FP8 KV cache. And before you upgrade to a pricier card because you hit OOM, run the arithmetic above — nine times out of ten the fix is a smaller context ceiling or KV quantization, not more VRAM.

Related reading

• Add Full-Text Search to Your App Before Reaching for Elasticsearch
• Postman vs Bruno vs Hoppscotch: Does Your API Client Really Need a Cloud Account?
• Vercel vs Netlify vs Cloudflare Pages: Where Your Side Project Should Actually Live

Comments

[Reid Marlow]

The KV-cache point is the one that bites people. I usually budget the first deploy around admission control, not just whether the weights load. A single long-context request can be a better stress test than a pile of short chats. Do you cap context at the API edge, or let the scheduler reject once the cache budget is gone?

[Libme]

Thank you so much for leaving a comment.
The KV-cache-as-real-budget framing is exactly right, and I like your instinct to stress-test with a single long-context request instead of a swarm of short ones — that's where the sequence-length term in the cache math actually shows up. On your question: I lean toward letting the scheduler reject once the cache budget is gone rather than capping context at the edge, because a hard edge cap either has to be set pessimistically (wasting VRAM most of the time) or it lies about what the GPU can actually hold at that moment given concurrent sequences. The one thing I'd add is to watch queue-time latency as your admission signal, not just outright rejections — a request that's technically admitted but stuck waiting for cache pages to free up can degrade the whole batch's tokens/sec long before you ever see a 503. Curious what you're using to observe cache pressure in practice — are you reading it off the serving framework's own metrics, or inferring it from batch occupancy?