How much GPU do you need to run an LLM locally? The rules of thumb we use

Frank — Sat, 14 Nov 2026 12:07:35 +0000

Every few weeks a client asks us the same question: "We want to use AI on our own documents, but nothing may leave the building. What hardware do we need?"

The answer almost always comes down to one number: GPU memory (VRAM). Here are the rules of thumb we use when we size machines for clients and for our own lab. (We also wrote a longer, non-technical version of this guide in German: Lokale KI im Unternehmen: Welche Hardware brauchen Sie wirklich?)

Rule 1: the model has to fit into VRAM

A language model is billions of parameters. To answer fast, all of them need to sit in the GPU's memory. If the model spills over into system RAM, it still runs, but it gets many times slower.

With 4-bit quantization (e.g. Q4_K_M in GGUF, the usual trade-off between quality and size), a model needs roughly:

≈ 0.6 GB of VRAM per billion parameters

That number comes from Q4_K_M averaging a bit under 5 bits per weight. So:

Model size	VRAM at 4-bit	Typical hardware
7–8B	~5–6 GB	almost any current GPU
14B	~9–10 GB	a 12–16 GB card
32B	~20 GB	one 24 GB card, e.g. RTX 3090
70B	~40–45 GB	two 24 GB cards
600B+ (e.g. full DeepSeek)	several hundred GB	server with multiple datacenter GPUs

These are weights only, without a long context. Exact numbers depend on the model, the quantization and the inference engine.

Rule 2: don't forget the KV cache

The weights are only half the story. Every token in the context (your documents, the chat history) needs memory too: the KV cache. For long documents or several users at once, this easily adds several gigabytes.

You can estimate it from the model's config.json:

KV bytes per token = 2 × layers × kv_heads × head_dim × bytes_per_value

Example: Qwen2.5-32B has 64 layers, 8 KV heads and a head dimension of 128. At FP16 (2 bytes):

2 × 64 × 8 × 128 × 2 = 262,144 bytes ≈ 256 KB per token
32,768 tokens × 256 KB ≈ 8 GB

So a 32B model with a full 32k context needs about 20 GB for the weights plus 8 GB for the
cache. That no longer fits on a single 24 GB card. Your options:

use a shorter context (most chat turns don't need 32k),
quantize the KV cache to 8-bit (halves it, usually with little quality loss),
or add a second GPU.

Models with grouped-query attention (few KV heads, like the example above) are much cheaper here than older models with one KV head per attention head.

Rule 3: start with the smallest model that does the job

Bigger isn't automatically better, and it's always slower and more expensive.

Small (up to ~8B): summaries, classification, extraction, simple Q&A. Efficient models like Liquid AI's LFM family even run on weaker hardware.
Medium (14–32B): the sweet spot for most companies. A good assistant, good search over your own documents (RAG), useful coding help. Qwen has strong models in this class.
Large (70B+): demanding writing, complex analysis, many concurrent users.

New open-weight models come out every few months, and the medium class catches up fast. A 32B modetoday does what needed 70B not long ago.

Three typical setups

1. Workstation with one GPU. One 24 GB card handles models up to ~32B. Good for a small team with an assistant and document search.

2. Workstation with several GPUs. Two to four cards give you 48–96 GB. That's enough for the 70B class, several models side by side, or image and video generation. With llama.cpp, splitting a model across cards is one flag:

llama-server -m model-Q4_K_M.gguf \
  -ngl 99 \
  --split-mode layer --tensor-split 1,1,1,1 \
  -c 16384

-ngl 99 offloads all layers to the GPUs, --tensor-split sets the share per card.

3. A dedicated GPU server in an EU data centre. No hardware in the office, but still no AI provider in between. We use dedicated servers (e.g. at Hetzner in Finland) for clients who need GDPR-friendly setups without running their own machines.

What people forget

Power and heat. An RTX 3090 is rated for up to 350 W. Four of them pull more than 1.4 kW under load, just for the GPUs. You need a matching power supply, circuit and cooling.
Noise. A multi-GPU workstation doesn't belong next to a desk. Put it in a server or utility room.
System RAM and CPU. Plenty of RAM helps with loading models and with processing large document batches, e.g. OCR on scans.
Operations. Updates, monitoring and swapping in new models take ongoing time. Plan for it, whether you do it yourself or someone does it for you.

Our own lab

We only recommend what we run ourselves. Our lab workstation has four RTX 3090s (96 GB VRAM total), a Threadripper 3960X and 256 GB of DDR4. We use it to test new open-weight models such as
Qwen, DeepSeek and Liquid AI for text, code, images and video, mostly with llama.cpp. For documents
like PDFs and scans we use Docling, also in production for clients. More about the setup: our lab.

TL;DR

4-bit model ≈ 0.6 GB VRAM per billion parameters, plus the KV cache for your context.
For most teams, one 24 GB GPU or a rented GPU server in the EU is enough to start.
Pick the model that fits the task, not the biggest one, and plan for running it, not just buying it.

I'm Frank, founder of web3.berlin, a small engineering team in Berlin. We set up private AI on clients' own servers and build blockchain infrastructure. Questions about sizing a setup? Ask in the comments.

DEV Community: Frank