So far in this series we've met the two big things that live in a GPU's memory. In Part 1 it was the model's tables, hauled through the factory for every single word. In Part 5 it was the notebook, the KV cache that every user fills up one word at a time.
That's almost everything you need to answer the question every team asks before buying or renting GPUs: how much memory do I need? Today we put the pieces into one small worksheet, fill it in for real models and real cards, and measure the one line everybody forgets.
The worksheet
Two lines of arithmetic. That's all:
room for notebooks = GPU memory − model tables − everything else
users that fit = room for notebooks ÷ notebook per token ÷ tokens per chat
Here it is filled in for the example from Part 5, Llama 3.1 8B on one 80 GB NVIDIA H100, with chats of up to 8,192 tokens. I wrote it as a small script, gpu_memory.py, so every number below comes from the same place:
1) worksheet: Llama 3.1 8B, BF16, one H100, 8K-token chats
GPU memory 80.00 GB
- model tables 8.03B x 2 bytes 16.06 GB
- everything else 2 GB + 5% 6.00 GB
= room for notebooks 57.94 GB
/ notebook per token 128 KB -> 442K tokens
/ tokens per chat 8,192 -> 54 users
54 people chatting at once. Same answer as Part 5, because it's the same arithmetic. Now let's look at each line, because each one hides a decision.
Line 1: the model's tables
This one is easy: number of parameters × bytes per number. An 8-billion-parameter model stored with 2-byte numbers (BF16) needs 16 GB. With 1-byte numbers (FP8), 8 GB. With half-byte numbers (INT4), 4 GB.
2) model tables (GB) by bytes per number
model BF16 (2 B) FP8 (1 B) INT4 (0.5 B) notebook/token
Llama 3.1 8B 16.1 8.0 4.0 128 KB
Qwen2.5 7B 15.2 7.6 3.8 56 KB
Mixtral 8x7B 93.4 46.7 23.4 128 KB
Llama 3.1 70B 141.2 70.6 35.3 320 KB
Two things worth noticing:
- Mixtral 8x7B is a "mixture of experts". Each word only uses about 13 billion of its 46.7 billion parameters, but all of them must sit in memory, because the next word may need different experts. For memory, count every parameter, not just the ones a word uses.
- Look at the last column. Qwen2.5 7B and Llama 3.1 8B are almost the same size, but Qwen keeps 4 notebook heads instead of 8, so its notebook is less than half as big. Hold that thought.
(Smaller numbers save memory but cost a little accuracy. Part 10 of this series is all about number formats.)
Line 2: "everything else"
This is the line people forget. Besides the tables and the notebooks, a GPU server needs memory for the runtime itself, for some headroom, and for scratch space: the temporary results a step creates while it works, like the attention scores from Part 5, which exist only while the step runs.
How big is that scratch space? I measured it on the same tiny factory as before (activations.py): push a prompt through in one go, then again in chunks of 512 tokens, and record the peak extra memory each way.
weight tables on the 12 floors: 340 MB, notebook: 72 KB per token
peak scratch memory for ONE step (on top of the model and the notebook)
prompt notebook all at once in chunks of 512
512 38 MB 57 MB 57 MB
1024 75 MB 215 MB 107 MB
2048 151 MB 835 MB 209 MB
4096 302 MB 3288 MB 411 MB
- All at once, a 4,096-token prompt needed 3.3 GB of scratch space. That's almost ten times the model's own tables. Doubling the prompt roughly quadruples it, because the attention scores compare every word with every other word.
- In chunks of 512, it needed 411 MB. Same answer, an eighth of the scratch space.
Real servers do both tricks: an attention kernel (FlashAttention) that never builds the full score table, and a cap on how many tokens one step may process (in vLLM, max_num_batched_tokens). That's what keeps "everything else" small and predictable.
So the worksheet uses a simple rule of thumb: 2 GB + 5% of the card, per GPU. That's 6 GB on an 80 GB card, the same allowance as Part 5. But it's an assumption, so check it on your own server. vLLM, for example, measures its own peak at startup and prints how much memory it has left for the KV cache. By default it also uses only 90% of the card, which makes this line closer to 10 GB:
7) Llama 3.1 8B BF16, one H100, 8K chats, with different 'everything else' allowances
everything else 6 GB -> room 57.94 GB -> 54.0 users
everything else 10 GB -> room 53.94 GB -> 50.2 users
Four users fewer. Not a disaster, but exactly the kind of gap that turns a capacity plan into an incident.
Line 3: the notebooks
From Part 5: notebook per token = 2 × floors × notebook heads × head size × bytes per number. That's 128 KB for Llama 3.1 8B in BF16, and 56 KB for Qwen2.5 7B, which shares each notebook between more heads.
That one difference is worth a lot. On the same single H100, with the same 8K chats:
Llama 3.1 8B BF16 tables H100 : 1 GPU (54 users fit)
Qwen2.5 7B BF16 tables H100 : 1 GPU (125 users fit)
Two models of about the same size, and one fits more than twice as many users. When you compare models for serving, compare their notebooks, not just their parameter counts.
Same model, different cards
Here's Llama 3.1 8B in BF16 on four different GPUs, for chats of different lengths:
3) Llama 3.1 8B in BF16: users that fit on ONE GPU
GPU room 4K 8K 32K 128K
L4 4.7 GB 8.8 4.4 1.1 0.3
L40S 27.5 GB 51.3 25.6 6.4 1.6
H100 57.9 GB 107.9 54.0 13.5 3.4
H200 115.9 GB 215.9 107.9 27.0 6.7
Read across a row and every 4× longer chat means 4× fewer users. The model is 128K-capable, but on one H100 only about 3 people can actually use all 128K at the same time.
Read down a column and the 24 GB L4 is the surprise: the 16 GB model leaves it under 5 GB of room. It can load the model, but it can hold only a handful of 8K chats.
When the model doesn't fit on one card
Llama 3.1 70B in BF16 needs 141 GB just for its tables. One 80 GB H100 can't hold it, so we split the model across several GPUs. Engineers call this tensor parallelism: every floor's tables are cut into slices, one slice per GPU, and the notebooks are split the same way (each GPU keeps some of the 8 notebook heads). Each GPU still needs its own "everything else".
4) Llama 3.1 70B on H100s, 8K-token chats
GPUs tables per GPU room (all GPUs) users BF16 users FP8 tables
1 141.2 GB doesn't fit 0.0 1.3
2 70.6 GB 6.8 GB 2.5 28.8
4 35.3 GB 154.8 GB 57.7 84.0
8 17.6 GB 450.8 GB 167.9 194.2
This is my favourite result in the post. Two H100s fit the model and almost nothing else: 2.5 users. Add two more and you get 57.7 users, 23 times more, because the first 141 GB are already paid for and every new byte goes to notebooks.
The FP8 column shows the other lever. Storing the tables in 1-byte numbers makes the model fit on one H100, but with room for only 1.3 users. On two H100s, FP8 tables take you from 2.5 users to 28.8.
One shared copy, or many copies?
If a model fits on one card, should you run one copy per GPU, or split one copy across several? Memory has a clear answer:
6) Llama 3.1 8B BF16 on four L4s, 8K-token chats
one copy split across all 4 (tensor parallel): 62.5 users
four separate copies, one per GPU: 17.7 users (4.4 each)
Four copies pay for the 16 GB of tables four times. One shared copy pays once and gives the rest to notebooks, so 3.5 times more users fit.
Memory isn't the whole story, though. A split model has to pass results between GPUs on every step, and L4s talk to each other over the slower PCIe bus, not the fast NVLink links that H100 servers use. So the memory math says "split", and a real test decides whether the speed is still good enough. That's a topic for Part 9.
Planning backwards
The worksheet also runs backwards: start from how many users you need and find the smallest setup. Here it is for 50 users with 8K chats:
5) smallest setup for 50 users at 8K (BF16 notebooks)
Llama 3.1 8B BF16 tables L4 : 4 GPUs (63 users fit)
Llama 3.1 8B BF16 tables H100 : 1 GPU (54 users fit)
Qwen2.5 7B BF16 tables L4 : 2 GPUs (56 users fit)
Qwen2.5 7B BF16 tables H100 : 1 GPU (125 users fit)
Mixtral 8x7B BF16 tables L4 : 8 GPUs (68 users fit)
Mixtral 8x7B BF16 tables H100 : 2 GPUs (51 users fit)
Llama 3.1 70B BF16 tables L4 : more than 8
Llama 3.1 70B BF16 tables H100 : 4 GPUs (58 users fit)
Llama 3.1 70B FP8 tables L4 : more than 8
Llama 3.1 70B FP8 tables H100 : 4 GPUs (84 users fit)
Change the model, the card, the chat length or the number of users at the top of the script and it answers in a second.
What the worksheet doesn't tell you
It tells you what fits, not how fast it runs. Fifty-four users fitting on one H100 doesn't mean all fifty-four get a fast answer: every decode step still reads the 16 GB of tables plus every user's notebook (Part 4's roofline). Memory sets the ceiling on users. Speed decides whether you can actually serve that many within your latency goal.
It also assumes every user uses their whole chat length at once. Real chats are mostly shorter, so a server that hands out notebook space as it's needed can fit many more people. That's exactly what Part 7 is about.
Cheat sheet
| Line | How to fill it in |
|---|---|
| GPU memory | The card's memory, times the number of GPUs |
| Model tables | Parameters × bytes per number (count every expert in a mixture of experts) |
| Everything else | Runtime + scratch space + headroom: about 2 GB + 5% per GPU, then check what your server reports |
| Notebook per token | 2 × floors × notebook heads × head size × bytes per number |
| Users that fit | Room for notebooks ÷ notebook per token ÷ tokens per chat |
| Biggest lever | Chat length: 4× longer chats, 4× fewer users |
What's next
The worksheet assumes every chat gets its full notebook space reserved up front. Real servers don't do that. Part 7: Continuous Batching and PagedAttention shows how they hand out notebook space page by page, and swap users in and out of a batch on every step, so the same GPU serves far more people.
Every number in this post comes from two scripts: gpu_memory.py (the worksheet, using each model's published config.json, 1 GB = 1e9 bytes) and activations.py (the scratch-space measurement, on the same tiny GPT-2-shaped factory as Part 5, float32 NumPy on a 2-core cloud CPU). Real GPU servers manage memory differently in the details, so treat "everything else" as a starting point and check it against what your own server reports.
This is Part 6 of Inference Foundations, a 10-part series that explains how AI models answer you, one measured step at a time. Next up is Part 7: Continuous Batching and PagedAttention. Follow me here on DEV to get it when it's out, or read the whole series on blog.sachinrajput.dev.
Get each new part by email: Inference Notes, a free newsletter. One idea every two weeks, with the numbers and the code.

Top comments (0)