A transformer is a stack of matrix multiplications with a small amount of glue between them. Once you can count the arithmetic in a matrix multiply, you can count the arithmetic in a whole model — and the widely quoted rule that inference costs about 2 × params × tokens stops being a rule you memorise and becomes a number you derive.
What one matrix multiply costs
Multiply an m × k matrix by a k × n matrix and you get an m × n result. Each output element is a dot product of length k: k multiplications and k - 1 additions. Hardware counts a multiply and an add as two floating-point operations, and the off-by-one on the additions is noise at these sizes, so the standard count is:
FLOPs(m x k @ k x n) = 2 * m * k * n
Example: 1 x 4096 @ 4096 x 4096
= 2 * 1 * 4096 * 4096
= 33,554,432 FLOPs (33.6 million, for one token)
Note what the 2 is and what it is not. It is one multiply plus one add per weight touched, and it is the only place that factor comes from. Everybody who quotes 2 × params is quoting this line.
Note also that m = 1 is the interesting case at inference time. One token at a time means every “matrix multiply” in a decoding step is really a vector-matrix product, which is why generation is slower than reading a prompt even though the FLOP count per token is identical.
A transformer layer, with its actual shapes
Take a model of a size you can hold in your head. Every number below is an assumption; they are typical of a 7B-class decoder-only model, and the arithmetic works for any other set you substitute.
| Assumption | Description |
|---|---|
| d_model = 4096 | The residual stream width. |
| n_layers = 32 | Identical blocks stacked. |
| n_heads = 32 | Each of head dimension 128, so 32 x 128 = 4096. Multi-head attention, no grouping. |
| d_ff = 16384 | The feed-forward hidden width, the usual 4 x d_model. |
| vocab = 128,000 | Tokens in and out. Input and output embedding matrices tied, so counted once. |
Inside one block there are exactly six weight matrices that a token passes through, and every one of them is a matmul against the residual stream:
per layer, per token, with x of shape 1 x 4096
Q = x @ Wq Wq: 4096 x 4096
K = x @ Wk Wk: 4096 x 4096
V = x @ Wv Wv: 4096 x 4096
... attention happens ...
o = a @ Wo Wo: 4096 x 4096
h = o @ W_up W_up: 4096 x 16384
y = h @ W_down W_down: 16384 x 4096
Everything else in the block — the residual additions, the layer norms, the activation function — touches a few thousand numbers rather than a few hundred million. It is real work and it matters for numerical stability, but at these shapes it rounds to zero in a FLOP count.
Counting one layer, then thirty-two
Parameters first, because FLOPs are twice parameters and it is easier to check the smaller number.
Wq 4096 x 4096 = 16,777,216
Wk 4096 x 4096 = 16,777,216
Wv 4096 x 4096 = 16,777,216
Wo 4096 x 4096 = 16,777,216
W_up 4096 x 16384 = 67,108,864
W_down 16384 x 4096 = 67,108,864
-----------
per layer 201,326,592 (201.3M)
x 32 layers 6,442,450,944 (6.44B)
+ embedding
128,000 x 4096 = 524,288,000 (0.52B)
-------------
total 6,966,738,944 (6.97B)
So this set of assumptions describes a 6.97B-parameter model. That is not a coincidence dressed up: d_model = 4096 with 32 layers is what a “7B” is, and the arithmetic is why.
Now the forward pass, which is twice that:
FLOPs per token = 2 * 6,966,738,944
= 13,933,477,888
~ 13.9 GFLOP
Thirteen point nine billion floating-point operations to produce one token. A 500-token answer to a 2,000-token prompt is 2,500 token positions through the model, so 2,500 × 13.9 = 34.8 TFLOP for one request.
Where 2 x params x tokens comes from
It came from the two lines above, and it is worth being explicit about what makes it true rather than approximately-often-true.
- Every parameter is used exactly once per token. Each weight participates in one multiply-accumulate for each token passing through its layer. That is what makes the count linear in parameters and linear in tokens at the same time.
- The 2 is a multiply and an add. Not a fudge factor, not a safety margin.
- Training is roughly 6, not 2. A training step is a forward pass plus a backward pass, and the backward pass computes two gradients per weight — one with respect to the input, one with respect to the weight. So
2 + 4 = 6 × params × tokensper training token, which is where that other widely quoted constant comes from.
The rule breaks in one common case and it is worth naming. A mixture-of-experts model does not use every parameter per token — a router picks a few experts of many. Substitute active parameters for total parameters and the rule works again, which is the entire reason anybody builds one.
The part the rule leaves out
2 × params × tokens counts the weight matmuls. It does not count attention itself, because attention multiplies activations by activations, and there are no parameters in Q @ K.T. That work depends on context length, not on model size.
For one query token attending over n cached keys:
scores = Q @ K.T 1 x 4096 @ 4096 x n -> 2 * 4096 * n
out = A @ V 1 x n @ n x 4096 -> 2 * n * 4096
--------------
4 * 4096 * n per layer
At n = 2,048: 4 * 4096 * 2048 = 33,554,432 per layer
x 32 layers = 1,073,741,824 = 1.07 GFLOP
Against 13.9 GFLOP of weight work: 1.07 / 13.9 = 7.7%
Under eight per cent at 2k context, so ignoring it is fine. At 32k context the same sum gives 17.2 GFLOP per token, which is larger than the entire rest of the model. The rule of thumb has a context length beyond which it is simply wrong, and that crossover is worth knowing where it is.
From FLOPs to seconds, and why it disappoints
The obvious next step is to divide by a chip’s throughput. Assume an accelerator that sustains 400 TFLOP/s of useful bf16 arithmetic — this is an assumption, roughly forty per cent utilisation of a current data-centre part, and you should substitute your own.
13.9 GFLOP per token / 400e12 FLOP/s = 34.8 microseconds
= 28,700 tokens/second (?)
Nobody sees 28,700 tokens per second from a 7B model on one chip, and the reason is the most useful thing on this page. Generating a single token requires reading every weight out of memory. At two bytes per weight that is 13.9 GB of memory traffic per token, and at a memory bandwidth of, say, 3.3 TB/s:
bytes moved per token = 6.97e9 params * 2 bytes = 13.9 GB
time = 13.9e9 / 3.3e12 = 4.2 milliseconds
= ~238 tokens/second
Arithmetic intensity of a decoding step:
13.9e9 FLOPs / 13.9e9 bytes = 1 FLOP per byte
The hardware can do roughly 400e12 / 3.3e12 = 121 FLOPs per byte
before arithmetic becomes the limit. One is not 121.
Decoding one token at a time is memory-bound by a factor of about a hundred. This is why batching helps so much — the weights are read once and used for every sequence in the batch, so the arithmetic per byte rises in direct proportion to batch size — and why quantisation speeds up generation even though it does not reduce the FLOP count at all. Halving the bytes per weight halves the traffic that is actually the bottleneck.
The FLOP count is the same wherever a model runs; the tokens per second are not, because they depend on batch pressure, hardware and how a provider serves it. Multigrid records time to first token and total time per request and reports p50 and p95 separately, which is the only way to see the difference between a model that is slow and a route that is occasionally slow.
Every model figure here is an assumption stated at the top of the section, not a specification of any released model. The point of writing them down is that you can replace all five and redo the sum in two minutes.
Top comments (0)