In Part 3 the Word Factory had two speed limits. The forklift brings tables from the warehouse, and the workbench does the math. Reading your question got stuck on the workbench. Typing the answer got stuck on the forklift.
Today we draw both limits on one picture. Engineers call it the roofline model, and it's the most useful sketch I know for LLM inference. Once you have it, you can look at any chip, any model and any batch size and say which limit you'll hit, before you run anything.
The whole model fits in one line:
speed = the smaller of (workbench speed) and (forklift speed × math per byte)
Let's unpack that, measure it, and then use it.
One number per job: math per byte
Every job the factory does has a recipe: how many math steps (FLOPs) it needs, and how many bytes the forklift must bring for it. Divide one by the other and you get the job's math per byte. Engineers call it arithmetic intensity.
A few jobs from earlier parts:
- One token through a big table. Each table number is 4 bytes and gets used for 2 FLOPs (one multiply, one add). That's 0.5 FLOPs per byte.
- 256 tokens through the same table. The table is fetched once and used 256 times: about 122 FLOPs per byte.
- Adding two lists of numbers. Fetch two numbers (8 bytes), write one (4 bytes), do 1 FLOP: about 0.08 FLOPs per byte.
Same factory, same tables, numbers that are 1,500 times apart. That spread is the whole story of LLM speed.
Drawing the roof
Now picture a chart. Left to right is math per byte. Bottom to top is speed (FLOPs per second).
The slanted roof is the forklift. If a job does 1 FLOP per byte and the forklift brings 15 billion bytes per second, the job can't go faster than 15 billion FLOPs per second, however fast the workbench is. Do 2 FLOPs per byte and the limit doubles. So this roof is a slope: forklift speed × math per byte.
The flat roof is the workbench. Past a certain point the forklift keeps up easily, and the workbench's top speed is the limit.
Where they meet is the ridge. It's the workbench speed divided by the forklift speed. Jobs to the left of the ridge are memory-bound. Jobs to the right are compute-bound.
In code, that's all of it:
def roof(intensity): # FLOPs per byte -> best possible FLOPs per second
return min(PEAK, BW * intensity)
I measured the two roofs the same way as in Part 3, with roofline.py: one token through a 400 MB table for the forklift, and a big multiply that fits near the workbench for the math:
memory roof : 15.0 GB/s
math roof : 164.3 GFLOP/s
ridge point : 11.0 FLOPs per byte
(The cloud CPU ran a little faster today than for Part 3, which measured 12 GB/s and 121 GFLOP/s. The ridge barely moved: 11 instead of 10. That's a hint for later.)
So on this computer, any job that does fewer than 11 FLOPs per byte is waiting on the forklift.
Putting real jobs under the roof
The roof is a prediction. Let's test it. The same script times three kinds of jobs, each one sized so it can't hide in the chip's cache:
# 1) the model's tables: a 400 MB table, N tokens on the trip
for B in [1, 2, 4, 8, 16, 32, 64, 128, 256, 512]:
X = np.random.rand(N, B).astype(np.float32)
sec = timed(lambda: W @ X)
# 2) typing for many users: each one reads their OWN 25 MB notebook
sec = timed(lambda: [k @ q for k, q in zip(K, Q)])
# 3) a small step: add two lists of 50 million numbers
sec = timed(lambda: np.add(u, v, out=out))
Each test ran several times and I kept the middle result:
operation FLOP/B GFLOP/s roof of roof
table x 1 tokens 0.50 7.7 7.5 103%
table x 2 tokens 1.00 6.4 15.0 43%
table x 4 tokens 2.00 14.2 29.9 47%
table x 8 tokens 3.99 27.7 59.7 46%
table x 16 tokens 7.97 49.5 119.3 41%
table x 32 tokens 15.90 77.6 164.3 47%
table x 64 tokens 31.60 103.1 164.3 63%
table x 128 tokens 62.40 118.8 164.3 72%
table x 256 tokens 121.77 143.2 164.3 87%
table x 512 tokens 232.22 156.0 164.3 95%
notebook read, 16 users 0.50 6.9 7.5 92%
notebook read, 64 users 0.50 6.6 7.5 89%
add two lists 0.08 0.8 1.2 66%
And here's the same thing as a picture:
Four things to notice.
1. More tokens walk you up the slope. One token sits at the bottom left, stuck on the forklift. Every time you double the tokens on the trip, the job moves right and climbs. Past the ridge it reaches the flat roof: at 512 tokens the table multiply ran at 95% of the workbench's top speed. (The 1-token row sits right on the roof because it's the same job I used to measure the forklift.)
2. The roof is a ceiling, not a promise. Between 2 and 16 tokens, the points sit at about half their roof. Nothing in the hardware forces that. The math library just uses a less efficient method for "skinny" multiplies with only a few tokens. Real GPU servers ship special code for exactly this small-batch zone, because that's where typing lives. When a job sits far below its roof, the roofline tells you there's speed left on the table, and roughly how much.
3. Notebooks don't walk. With 16 users or 64 users, reading notebooks stays at 0.5 FLOPs per byte. Batching moved the table jobs to the right, but it can't move these, because every user brings their own notebook and nothing is shared. This is Part 3's "the notebooks don't share", seen on the roofline. (Real models share each notebook between a few attention heads, a trick called grouped-query attention, which lifts this to a few FLOPs per byte. Still far left of the ridge.)
4. Small steps live in the basement. Adding two lists does 0.08 FLOPs per byte, the lowest point on the chart. That's why, in Part 2, the tiny steps between the big tables took far more time than their share of the math. It's also why real engines glue small steps together into one pass (kernel fusion): fewer trips for the forklift.
Where LLM inference lands
Put the pieces of an LLM on this chart and the two phases from Part 3 fall right out:
| Job | Math per byte | Where it lands |
|---|---|---|
| Typing, 1 user, the model's tables | ~0.5 (float32), ~1 (2-byte numbers) | Far left. Forklift. |
| Typing, many users, the tables | grows with the number of users | Climbs the slope as the batch grows |
| Reading a 256-token question | ~120 | Right of the ridge. Workbench. |
| Typing, reading each user's notebook | ~0.5 to a few, for any batch size | Stuck on the left |
| Small steps (adds, norms) | ~0.1 | The basement |
One handy shortcut: when the tables are stored as 2-byte numbers (BF16, how most models are served), every number does 2 FLOPs per token and costs 2 bytes. So math per byte ≈ the number of tokens on the trip. Want to know if a batch keeps the math busy? Compare the batch size with the ridge.
Predict before you run
Now the payoff. To draw a roofline for any chip you need just two numbers from its spec sheet: memory bandwidth and dense math speed.
- NVIDIA H100 SXM: 3.35 TB/s and 989 TFLOP/s (BF16, without the "sparsity" bonus that some spec sheets add). Ridge: about 295 FLOPs per byte.
- NVIDIA B200: 8 TB/s and 2,250 TFLOP/s (BF16, dense). Ridge: about 281.
That's a forklift about 220 times faster than my CPU's and a workbench about 6,000 times faster. The ridge is 27 times further right than my CPU's 11. On a big GPU the workbench is so fast that almost everything is waiting on memory.
Here's predict.py applying the one-line model to an 8-billion-parameter model in BF16 (16 GB of tables) on one H100, counting only the tables:
an 8-billion-parameter model in BF16 (16 GB of weights) on one H100
tokens step ms fetch ms math ms tokens/s limit
1 4.78 4.78 0.02 209 memory
8 4.78 4.78 0.13 1,675 memory
64 4.78 4.78 1.04 13,400 memory
256 4.78 4.78 4.14 53,600 memory
512 8.28 4.78 8.28 61,812 math
1024 16.57 4.78 16.57 61,812 math
Read it like a chatbot company would:
- One user can never get more than ~209 words per second from this model on this chip. The forklift needs 4.78 ms to bring 16 GB, and the math takes 0.02 ms of that. No amount of workbench helps.
- Up to a few hundred users ride for free. From 1 to 256 tokens per trip, the step time doesn't change at all. Every extra user is pure profit, until you pass the ridge.
- Past the ridge, the bill arrives. At 512 and 1,024 tokens per trip the step gets longer, because now you're waiting on the workbench.
And two predictions that surprise people:
same model, 1 user, different number sizes on the H100
BF16, 2 bytes 4.78 ms per token -> at most 209 tokens/s; ridge needs ~295 tokens per trip
FP8, 1 byte 2.39 ms per token -> at most 419 tokens/s; ridge needs ~295 tokens per trip
- Smaller numbers make one user type twice as fast. Storing the tables in 1-byte numbers (FP8) halves the bytes, so the forklift brings the whole model in half the time. That's why so much effort goes into quantization, the last part of this series.
- But the ridge doesn't move. The chip also does FP8 math twice as fast, so you still need ~295 tokens per trip to keep the workbench busy. Newer chips are the same: the B200 raised both roofs, and the ridge stayed around 280 tokens.
How good is the prediction?
Remember, these are ceilings. Real engines land below them, and the notebooks add bytes the table-only math above ignores. Part 3 showed how fast that grows: at 64 users with 512 notes each, the notebooks were seven times bigger than the model. So real servers hit the forklift sooner and harder than this table says.
We can check the ceiling on our tiny factory, though. In Part 3, typing one word for one user meant fetching 377 MB (tables plus one notebook) with a forklift that, that day, did 12 GB/s. The roofline says that can't take less than 31 ms. The measurement was 37.5 ms, about 83% of the roof. One line of arithmetic, done before running anything, got within a fifth of the real answer.
How to use the roofline
- Find the two roofs. Memory bandwidth and dense math speed, from the spec sheet or a quick measurement.
- Find the ridge. Math speed ÷ bandwidth.
- Find your job's math per byte. For the model's tables with 2-byte numbers, it's about the number of tokens on the trip.
- Compare. Left of the ridge: you're paying for bytes, so fetch fewer (smaller numbers, shared prefixes) or share each fetch with more tokens (batching). Right of the ridge: you're paying for math, so you need a faster workbench or less math.
Cheat sheet
| Idea | One line |
|---|---|
| Roofline | speed = min(math roof, memory roof × math per byte) |
| Ridge | math roof ÷ memory roof. My CPU: 11. H100: ~295. B200: ~281 |
| Memory-bound | Left of the ridge. Typing, notebooks, small steps |
| Compute-bound | Right of the ridge. Reading long questions, big batches |
| Batching | Moves the table work right. Doesn't move the notebooks |
| Smaller numbers | Fewer bytes, faster typing. The ridge stays put |
| Below the roof | Software leaving speed on the table. Go look |
What's next
The notebooks kept showing up as the job batching can't fix. Part 5: The KV Cache From Scratch builds the notebook by hand: what's in it, why it grows with every word, how big it gets for real models, and the tricks servers use to make it smaller.
Every number in this post comes from two scripts: roofline.py (measured on a 2-core cloud CPU with float32 NumPy) and predict.py (the one-line model applied to published chip specs). GPU figures are predictions from dense peak specs, not measurements, so treat them as ceilings.
Top comments (0)