DEV Community

Cover image for How much GPU memory does an LLM need? The whole sum, worked out on real cards
RONI DAS
RONI DAS

Posted on Originally published at systemdesign.academy

How much GPU memory does an LLM need? The whole sum, worked out on real cards

Before a team rents or buys a GPU to serve a language model, someone asks the same question: how much memory does this model need? The answer you usually find online is "the number of parameters times two bytes". That sum is correct, and it is also the reason so many first deployments disappoint. It tells you whether the model loads. It does not tell you how many people the card can serve, and that is the number your users care about.

In this post I work through the whole sum the way I would at a whiteboard with a new engineer on the team. Every number comes from real starts of vLLM 0.33 on rented GPUs, or from the course's GPU planning game, where each plan was checked on real cards. Nothing here is a rule of thumb that I have not seen hold on a card.

Three things share the card

A model being served is not alone in GPU memory. Three things share it.

  1. The weights. These are the numbers the model learned in training. Their size is fixed for a given model and a given number of bits.
  2. The working space. This is the memory the model needs for its arithmetic while it runs. vLLM measures it with a trial run when it starts and sets it aside.
  3. The KV cache. This holds notes on every token of every conversation the model is answering at that moment. It grows with every person and every token.

The first two are fixed. Only the third grows with your users, so the third decides how many people one card can serve. Keep that in mind, because everything below is really about how much room is left for it.

Step one: the weights

Multiply the number of parameters by the bytes each one takes: 2 bytes in 16 bits, 1 byte in 8 bits, and about half a byte in 4 bits.

Qwen3-8B has about 8.2 billion parameters. In 16 bits that is 16.4 GB, which is 15.26 GiB. A GiB is 1,073,741,824 bytes, a little more than a GB, and vLLM counts in GiB, so it is worth doing the sum in the same unit. The sum on paper was close to what the server reported: vLLM's own start-up log counted 15.27 GiB once Qwen3-8B was on the card.

Step two: the working space

This one surprises people because it is not one fixed number. On our RTX 4090 with Qwen3-8B it was about 1.80 GiB on a first start, which also compiles the model, and about 1.15 GiB on a later start that reused the compiled program.

The practical lesson is simple. Read the cache size from vLLM's log on a second start before you plan capacity, not the first.

Step three: what is left is the KV cache

Here is the whole budget for Qwen3-8B in 16 bits on one rented RTX 4090 with 24 GB of memory.

A waterfall of the memory budget for Qwen3-8B in 16 bits on the RTX 4090, drawn to scale in GiB. The card reports 23.52 GiB. vLLM leaves 8% alone, 1.88 GiB, leaving its share of 21.64 GiB. The weights take 15.26 GiB and the working space about 1.80 GiB. The last bar, about 4.6 GiB, is the KV cache, room for roughly 33,000 tokens.

The card reported 23.52 GiB to vLLM. By default vLLM 0.33 uses 92% of it, which is 21.64 GiB, and leaves the rest alone. Take away 15.27 GiB of weights and about 1.8 GiB of working space, and the KV cache got 4.56 GiB in vLLM's log. (The figure above is our sizing script's version of the same sum, which lands within a few hundred tokens of the log.)

How many tokens is that? For Qwen3-8B, every token of every conversation needs 144 KiB of notes. At 144 KiB a token, 4.56 GiB is room for 33,216 tokens, which vLLM keeps as 2,076 pages of 16 tokens each.

Now turn tokens into people. That is enough for about one conversation of 32,768 tokens, or 8 of 4,000, or 16 of 2,000, if they all reach their full length at the same time. So the model fits on a 24 GB card, with modest room for conversations. The weights took most of the card; the users get what is left.

This is also why two cards that both hold the weights can serve very different numbers of people. The weights take the same room whether one person or a hundred are asking. Every extra person adds their own notes.

Every setting moves the cache, and only the cache

Once you see that the weights and working space are paid first, every tuning knob becomes easy to reason about. Whatever you change, the KV cache absorbs the difference.

Four columns drawn to one scale in GiB, one for each way of serving Qwen3-8B. 16 bits at 92% on the RTX 4090: 33,216 tokens. 16 bits at 80%: 12,656 tokens. 8-bit weights at 92%: 79,184 tokens. 16 bits at 92% on a 44 GiB card, on paper: 170,560 tokens.

The share of the card. The --gpu-memory-utilization setting is 0.92 by default in vLLM 0.33. Because the weights and working space take the same room whatever the share, the whole change comes out of the cache. On our card, going from 0.92 to 0.80 cut the cache from 4.56 GiB to 1.74 GiB, and the room for conversations from 33,216 tokens to 12,656. A small change in the share is a large change in capacity.

Weights in 8 bits. Whatever the weights give up goes straight to the cache. Qwen3-8B in 8 bits (FP8) took 8.9 GiB instead of 15.27 GiB, and the cache grew to 10.88 GiB, room for 79,184 tokens instead of 33,216. That is about 2.38 times as many. Fewer bits can change the quality of the answers, so test the smaller copy on your own questions before you rely on it.

The cache itself in 8 bits. With --kv-cache-dtype fp8, each token's notes take half the bytes: 73,728 instead of 147,456 for Qwen3-8B. In our run the cache held 59,856 tokens instead of 33,216. That is 1.80 times as many rather than 2, because that start needed a little more working space. Check answer quality on long conversations before you keep it.

A bigger card. The fourth column is on paper, drawn dashed: the same model on a card with 44 GiB available gets 170,560 tokens of cache. A bigger card helps more than its size suggests, because only the cache grows.

Why vLLM sometimes refuses to start

A common first error looks like a memory problem but is really a check. At start-up, vLLM makes sure that at least one conversation of the longest allowed length fits in the cache, and refuses to start if it does not. Qwen3-8B allows 40,960 tokens, which needs 5.62 GiB of cache, and our card had 4.56 GiB.

The fix is to start with a shorter --max-model-len, such as 8,192. That does not make the cache any bigger. It only changes the check, and it means no single conversation can grow past that length.

Big models: fitting is not the same as serving

For a 70B model the question changes from "how many people fit?" to "does the model fit at all with room for anyone?"

A 70B model on one 80 GB card. Llama 3.1 70B in 8 bits loaded into 67.7 GiB on one H100 80 GB. That left 0.07 GiB for the KV cache. vLLM refused to start with room for 2,560 tokens, so the card could not hold even one ordinary conversation, let alone sixteen people. The model fitted. It could not serve.

The same 70B on two A100s. The fix is to split the model across two cards with tensor parallelism, which divides each of the model's large weight tables between the GPUs so that every layer is shared out. In vLLM that is --tensor-parallel-size 2. The two cards talk at every step, so they need a fast link between them; our A100 80 GB SXM cards were joined by NVLink. Llama 3.1 70B in 8 bits then took 33.89 GiB on each card, and the KV cache got 37.48 GiB in all. Sixteen people at once got a new word every 48 ms each, inside the game's promise of 60 ms. One person alone got a word every 26.5 ms.

Two gaming cards instead. You can split a model across two RTX 4090s the same way, but it is much slower than it looks on paper. Those cards talk over PCIe, which moves at most 32 GB a second each way on that card. On cards without NVLink it is usually better to run separate copies of a smaller model, one per card, and split the people between them.

A 120B model on one 80 GB card. This works when the model is stored in very few bits from the start. OpenAI's gpt-oss-120b is published with its expert layers in MXFP4, a 4-bit format with one shared scale for every 32 numbers. On one A100 80 GB it loaded in 66.14 GiB and left 4.6 GiB for the KV cache. It is also a mixture-of-experts model, where only some groups of weights work on each token, which is why it writes quickly for its size: 32 people at once got a word every 29 ms each in our run.

CPU offload makes room, and makes every token wait

When memory is short, the tempting fix is to keep part of the weights in the computer's ordinary memory instead of on the GPU.

A hand-drawn architecture of CPU offload on one RTX 4090. 6.02 GiB of weights sit in the computer's memory and cross PCIe, at most 32 GB a second, to the GPU's cores for every token, while the other 9.25 GiB stay in GPU memory at 919 GB a second. One person got 2.6 tokens a second, and one token took 388 ms offloaded against 17.4 ms without.

With --cpu-offload-gb 6, vLLM moved 6.02 GiB of Qwen3-8B's weights off our RTX 4090, and the cache grew from 33,216 to 76,912 tokens. That looks like a win until you time it. One person's answer fell from 56.4 to 2.6 tokens a second, about 22 times slower, because the offloaded weights cross the PCIe link for every token. Use it only when the model will not fit any other way and speed does not matter.

Two reasons the picture is better than it first looks

Newer models need less memory per user. In grouped-query attention, several query heads share one key and one value, and only keys and values go in the cache. Qwen3-8B shares each one among 4 heads, so it keeps 144 KiB a token; the older Qwen1.5-7B keeps 512 KiB. The same memory therefore holds about 3.6 times as many tokens of the newer model, even though it has more parameters. You can check any model by comparing num_key_value_heads with num_attention_heads in its config.json.

vLLM does not reserve room for the longest conversation. It stores the cache in pages of 16 tokens, taken one at a time as a conversation grows, so a conversation wastes at most part of its last page. This is called PagedAttention. On our card, 64 people with conversations of about 340 tokens used only 66.8% of the cache at once, where reserving 32,768 tokens for each of them would have fitted only one.

How to check the number for your own model

Start the model with vLLM and read its start-up log. It prints the memory the weights took, the KV cache it set aside, the number of tokens that fit, and a maximum concurrency figure, which is how many conversations of the full allowed length would fit at once. Divide the tokens by the length of your own users' conversations for a realistic count.

So the short version is this. Count three things: the weights, the working space and the KV cache. The first two are fixed, so subtract them from the share of the card your server uses, and whatever is left is the room for conversations. Divide that by the notes one token needs and by the length of your users' conversations, and you know how many people fit. When nothing is left, use fewer bits or more GPUs.

The full guide, with every question and the lesson each number comes from, is on our site: How much GPU memory does an LLM need?. If you want a first estimate before you rent anything, the free GPU memory calculator does the same sum in the browser.

Go deeper: Hands-On Inference Engineering

These numbers come from a course I teach on System Design Academy, Hands-On Inference Engineering. In it you rent the GPU yourself, start vLLM with real models, read its memory budget line by line, and change each setting to see how many people fit. It goes on into PagedAttention, the scheduler, quantization, speculative decoding and serving at scale, each one run and measured on a rented GPU, and it is written for backend and AI Engineering teams who want to serve their own models: see the Hands-On Inference Engineering course.

Top comments (0)