DEV Community

Cover image for How Much GPU Memory Do I Need?
Sachin Kr. Rajput
Sachin Kr. Rajput

Posted on Originally published at blog.sachinrajput.dev

How Much GPU Memory Do I Need?

So far in this series we've met the two big things that live in a GPU's memory. In Part 1 it was the model's tables, hauled through the factory for every single word. In Part 5 it was the notebook, the KV cache that every user fills up one word at a time.

That's almost everything you need to answer the question every team asks before buying or renting GPUs: how much memory do I need? Today we put the pieces into one small worksheet, fill it in for real models and real cards, and measure the one line everybody forgets.

The worksheet

Two lines of arithmetic. That's all:

room for notebooks = GPU memory − model tables − everything else

users that fit = room for notebooks ÷ notebook per token ÷ tokens per chat

Here it is filled in for the example from Part 5, Llama 3.1 8B on one 80 GB NVIDIA H100, with chats of up to 8,192 tokens. I wrote it as a small script, gpu_memory.py, so every number below comes from the same place:

1) worksheet: Llama 3.1 8B, BF16, one H100, 8K-token chats
   GPU memory                        80.00 GB
 - model tables  8.03B x 2 bytes     16.06 GB
 - everything else  2 GB + 5%        6.00 GB
 = room for notebooks                57.94 GB
 / notebook per token                  128 KB  -> 442K tokens
 / tokens per chat                  8,192     -> 54 users
Enter fullscreen mode Exit fullscreen mode

54 people chatting at once. Same answer as Part 5, because it's the same arithmetic. Now let's look at each line, because each one hides a decision.

Line 1: the model's tables

This one is easy: number of parameters × bytes per number. An 8-billion-parameter model stored with 2-byte numbers (BF16) needs 16 GB. With 1-byte numbers (FP8), 8 GB. With half-byte numbers (INT4), 4 GB.

2) model tables (GB) by bytes per number
   model            BF16 (2 B)  FP8 (1 B)  INT4 (0.5 B)  notebook/token
   Llama 3.1 8B           16.1        8.0           4.0          128 KB
   Qwen2.5 7B             15.2        7.6           3.8           56 KB
   Mixtral 8x7B           93.4       46.7          23.4          128 KB
   Llama 3.1 70B         141.2       70.6          35.3          320 KB
Enter fullscreen mode Exit fullscreen mode

Two things worth noticing:

  • Mixtral 8x7B is a "mixture of experts". Each word only uses about 13 billion of its 46.7 billion parameters, but all of them must sit in memory, because the next word may need different experts. For memory, count every parameter, not just the ones a word uses.
  • Look at the last column. Qwen2.5 7B and Llama 3.1 8B are almost the same size, but Qwen keeps 4 notebook heads instead of 8, so its notebook is less than half as big. Hold that thought.

(Smaller numbers save memory but cost a little accuracy. Part 10 of this series is all about number formats.)

Line 2: "everything else"

This is the line people forget. Besides the tables and the notebooks, a GPU server needs memory for the runtime itself, for some headroom, and for scratch space: the temporary results a step creates while it works, like the attention scores from Part 5, which exist only while the step runs.

How big is that scratch space? I measured it on the same tiny factory as before (activations.py): push a prompt through in one go, then again in chunks of 512 tokens, and record the peak extra memory each way.

weight tables on the 12 floors: 340 MB, notebook: 72 KB per token

peak scratch memory for ONE step (on top of the model and the notebook)
    prompt  notebook    all at once    in chunks of 512
       512     38 MB          57 MB               57 MB
      1024     75 MB         215 MB              107 MB
      2048    151 MB         835 MB              209 MB
      4096    302 MB        3288 MB              411 MB
Enter fullscreen mode Exit fullscreen mode
  • All at once, a 4,096-token prompt needed 3.3 GB of scratch space. That's almost ten times the model's own tables. Doubling the prompt roughly quadruples it, because the attention scores compare every word with every other word.
  • In chunks of 512, it needed 411 MB. Same answer, an eighth of the scratch space.

Real servers do both tricks: an attention kernel (FlashAttention) that never builds the full score table, and a cap on how many tokens one step may process (in vLLM, max_num_batched_tokens). That's what keeps "everything else" small and predictable.

So the worksheet uses a simple rule of thumb: 2 GB + 5% of the card, per GPU. That's 6 GB on an 80 GB card, the same allowance as Part 5. But it's an assumption, so check it on your own server. vLLM, for example, measures its own peak at startup and prints how much memory it has left for the KV cache. By default it also uses only 90% of the card, which makes this line closer to 10 GB:

7) Llama 3.1 8B BF16, one H100, 8K chats, with different 'everything else' allowances
   everything else  6 GB -> room 57.94 GB -> 54.0 users
   everything else 10 GB -> room 53.94 GB -> 50.2 users
Enter fullscreen mode Exit fullscreen mode

Four users fewer. Not a disaster, but exactly the kind of gap that turns a capacity plan into an incident.

Line 3: the notebooks

From Part 5: notebook per token = 2 × floors × notebook heads × head size × bytes per number. That's 128 KB for Llama 3.1 8B in BF16, and 56 KB for Qwen2.5 7B, which shares each notebook between more heads.

That one difference is worth a lot. On the same single H100, with the same 8K chats:

   Llama 3.1 8B BF16 tables     H100 : 1 GPU  (54 users fit)
   Qwen2.5 7B BF16 tables       H100 : 1 GPU  (125 users fit)
Enter fullscreen mode Exit fullscreen mode

Two models of about the same size, and one fits more than twice as many users. When you compare models for serving, compare their notebooks, not just their parameter counts.

Same model, different cards

Here's Llama 3.1 8B in BF16 on four different GPUs, for chats of different lengths:

3) Llama 3.1 8B in BF16: users that fit on ONE GPU
   GPU             room       4K       8K      32K     128K
   L4            4.7 GB      8.8      4.4      1.1      0.3
   L40S         27.5 GB     51.3     25.6      6.4      1.6
   H100         57.9 GB    107.9     54.0     13.5      3.4
   H200        115.9 GB    215.9    107.9     27.0      6.7
Enter fullscreen mode Exit fullscreen mode

Read across a row and every 4× longer chat means 4× fewer users. The model is 128K-capable, but on one H100 only about 3 people can actually use all 128K at the same time.

Read down a column and the 24 GB L4 is the surprise: the 16 GB model leaves it under 5 GB of room. It can load the model, but it can hold only a handful of 8K chats.

When the model doesn't fit on one card

Llama 3.1 70B in BF16 needs 141 GB just for its tables. One 80 GB H100 can't hold it, so we split the model across several GPUs. Engineers call this tensor parallelism: every floor's tables are cut into slices, one slice per GPU, and the notebooks are split the same way (each GPU keeps some of the 8 notebook heads). Each GPU still needs its own "everything else".

4) Llama 3.1 70B on H100s, 8K-token chats
   GPUs    tables per GPU  room (all GPUs)  users BF16  users FP8 tables
   1             141.2 GB      doesn't fit         0.0               1.3
   2              70.6 GB           6.8 GB         2.5              28.8
   4              35.3 GB         154.8 GB        57.7              84.0
   8              17.6 GB         450.8 GB       167.9             194.2
Enter fullscreen mode Exit fullscreen mode

Llama 3.1 70B in BF16 on 80 GB H100s, 8K-token chats. One GPU: the 141 GB model doesn't fit. Two GPUs: 6.8 GB of room, 2.5 chats. Four GPUs: 154.8 GB of room, 57.7 chats. Eight GPUs: 450.8 GB of room, 167.9 chats.

This is my favourite result in the post. Two H100s fit the model and almost nothing else: 2.5 users. Add two more and you get 57.7 users, 23 times more, because the first 141 GB are already paid for and every new byte goes to notebooks.

The FP8 column shows the other lever. Storing the tables in 1-byte numbers makes the model fit on one H100, but with room for only 1.3 users. On two H100s, FP8 tables take you from 2.5 users to 28.8.

One shared copy, or many copies?

If a model fits on one card, should you run one copy per GPU, or split one copy across several? Memory has a clear answer:

6) Llama 3.1 8B BF16 on four L4s, 8K-token chats
   one copy split across all 4 (tensor parallel):  62.5 users
   four separate copies, one per GPU:              17.7 users  (4.4 each)
Enter fullscreen mode Exit fullscreen mode

Four copies pay for the 16 GB of tables four times. One shared copy pays once and gives the rest to notebooks, so 3.5 times more users fit.

Memory isn't the whole story, though. A split model has to pass results between GPUs on every step, and L4s talk to each other over the slower PCIe bus, not the fast NVLink links that H100 servers use. So the memory math says "split", and a real test decides whether the speed is still good enough. That's a topic for Part 9.

Planning backwards

The worksheet also runs backwards: start from how many users you need and find the smallest setup. Here it is for 50 users with 8K chats:

5) smallest setup for 50 users at 8K (BF16 notebooks)
   Llama 3.1 8B BF16 tables     L4   : 4 GPUs  (63 users fit)
   Llama 3.1 8B BF16 tables     H100 : 1 GPU  (54 users fit)
   Qwen2.5 7B BF16 tables       L4   : 2 GPUs  (56 users fit)
   Qwen2.5 7B BF16 tables       H100 : 1 GPU  (125 users fit)
   Mixtral 8x7B BF16 tables     L4   : 8 GPUs  (68 users fit)
   Mixtral 8x7B BF16 tables     H100 : 2 GPUs  (51 users fit)
   Llama 3.1 70B BF16 tables    L4   : more than 8
   Llama 3.1 70B BF16 tables    H100 : 4 GPUs  (58 users fit)
   Llama 3.1 70B FP8 tables     L4   : more than 8
   Llama 3.1 70B FP8 tables     H100 : 4 GPUs  (84 users fit)
Enter fullscreen mode Exit fullscreen mode

Change the model, the card, the chat length or the number of users at the top of the script and it answers in a second.

What the worksheet doesn't tell you

It tells you what fits, not how fast it runs. Fifty-four users fitting on one H100 doesn't mean all fifty-four get a fast answer: every decode step still reads the 16 GB of tables plus every user's notebook (Part 4's roofline). Memory sets the ceiling on users. Speed decides whether you can actually serve that many within your latency goal.

It also assumes every user uses their whole chat length at once. Real chats are mostly shorter, so a server that hands out notebook space as it's needed can fit many more people. That's exactly what Part 7 is about.

Cheat sheet

Line How to fill it in
GPU memory The card's memory, times the number of GPUs
Model tables Parameters × bytes per number (count every expert in a mixture of experts)
Everything else Runtime + scratch space + headroom: about 2 GB + 5% per GPU, then check what your server reports
Notebook per token 2 × floors × notebook heads × head size × bytes per number
Users that fit Room for notebooks ÷ notebook per token ÷ tokens per chat
Biggest lever Chat length: 4× longer chats, 4× fewer users

What's next

The worksheet assumes every chat gets its full notebook space reserved up front. Real servers don't do that. Part 7: Continuous Batching and PagedAttention shows how they hand out notebook space page by page, and swap users in and out of a batch on every step, so the same GPU serves far more people.


Every number in this post comes from two scripts: gpu_memory.py (the worksheet, using each model's published config.json, 1 GB = 1e9 bytes) and activations.py (the scratch-space measurement, on the same tiny GPT-2-shaped factory as Part 5, float32 NumPy on a 2-core cloud CPU). Real GPU servers manage memory differently in the details, so treat "everything else" as a starting point and check it against what your own server reports.


This is Part 6 of Inference Foundations, a 10-part series that explains how AI models answer you, one measured step at a time. Next up is Part 7: Continuous Batching and PagedAttention. Follow me here on DEV to get it when it's out, or read the whole series on blog.sachinrajput.dev.

Get each new part by email: Inference Notes, a free newsletter. One idea every two weeks, with the numbers and the code.

Top comments (0)