One forward pass through a dense transformer costs about two floating-point operations per parameter per token. That single rule, worked through, tells you what an answer costs in arithmetic, and its failure modes tell you almost everything interesting about inference.
The rule, and where the 2 comes from
Nearly all of the arithmetic in a transformer is matrix multiplication, and nearly all of the weights sit in those matrices. Multiplying an input vector by a weight matrix touches every weight exactly once, and each touch is one multiply and one add — a multiply–accumulate, conventionally counted as two floating-point operations.
Forward pass (inference): FLOPs ~= 2 * N * T
Training step: FLOPs ~= 6 * N * T
N = number of parameters
T = number of tokens processed
The 6 for training is 2 for the forward pass plus roughly 4 for the
backward pass, which computes gradients with respect to both the inputs
and the weights and therefore does about twice the work of the forward.
Both forms are approximations that ignore layer normalisation, the activation functions, the softmax and the residual additions. Those are element-wise operations on activations rather than on weights, and in a model of any size they are a rounding error against the matrix multiplies. What the rule does not ignore harmlessly is attention, which is dealt with below.
A worked answer, end to end
Take a dense 70-billion-parameter model, a prompt of 1,000 tokens and an answer of 500 tokens. Every one of those 1,500 tokens is pushed through the full stack of weights once.
Model: N = 70e9 parameters, dense
Request: 1,000 input tokens, 500 output tokens
Prefill (the prompt, processed in one parallel pass):
2 * 70e9 * 1000 = 1.40e14 FLOPs = 140 TFLOP
Decode (500 sequential passes, one token each):
2 * 70e9 * 500 = 0.70e14 FLOPs = 70 TFLOP
Total per answer:
2 * 70e9 * 1500 = 2.10e14 FLOPs = 210 TFLOP
Two hundred and ten teraflops for one short answer. For scale, that is roughly the arithmetic a high-end consumer graphics card can do in a few seconds at its theoretical peak, and it explains immediately why inference at scale is a hardware problem rather than a software one.
Note that input and output tokens cost the same number of FLOPs each. They do not cost the same amount of time or money, and the reason is in the latency section below — it is also why providers price input and output separately.
The term the rule leaves out
Attention does not multiply by weights; it multiplies queries by keys, both of which are activations whose size grows with the sequence. Its cost per generated token therefore scales with how much context is already there, and the standard estimate adds a second term:
FLOPs per token ~= 2 * N + 2 * n_layers * n_ctx * d_model
n_layers = transformer blocks
n_ctx = tokens already in the context
d_model = model width (hidden size)
The interesting question is where the second term catches the first, because below that point you can ignore attention entirely and above it you cannot. Set them equal and solve:
2 * n_layers * n_ctx * d_model = 2 * N
n_ctx = N / (n_layers * d_model)
For a 70B model with 80 layers and d_model = 8192:
n_ctx = 70e9 / (80 * 8192)
= 70e9 / 655,360
= 106,811 tokens
Check the two ends:
at 1,000 tokens of context: 2*80*1000*8192 = 1.31e9 vs 2N = 1.40e11
attention is 0.9% of the total
at 128,000 tokens of context: 2*80*128000*8192 = 1.68e11 vs 2N = 1.40e11
attention is now the larger term
So for a model of that shape, attention is negligible until roughly a hundred thousand tokens of context and dominant beyond it. That single number explains why long-context pricing behaves the way it does, why million-token context windows are expensive out of proportion to their length, and why prefill cost stops being linear in prompt length once prompts get very long.
The layer count and width above are a plausible shape for a 70B dense model and are used to make the algebra concrete. Substitute the real values from the model card you care about — the crossover formula, not the 106,811, is the part to carry away.
Checking it against a GPU
A FLOP count becomes meaningful when divided by a rate. Accelerator datasheets quote a peak, and real workloads reach a fraction of it; that fraction is called model FLOPs utilisation, and for large, well-tuned training runs published values have sat in the range of roughly 35 to 50 per cent. Inference prefill can reach similar levels; inference decode cannot, for reasons in the next section.
NVIDIA's published specification for the H100 SXM gives a dense BF16
peak of about 989 TFLOP/s (the doubled figure some tables show assumes
structured sparsity, which general inference does not use).
Take 40% utilisation as a working assumption:
effective rate = 0.40 * 989e12 = 3.96e14 FLOP/s
time for our 210 TFLOP answer:
2.10e14 / 3.96e14 = 0.53 seconds of one accelerator's arithmetic
Half a second of GPU time for a 500-token answer. That is the cost side of the equation, and it is roughly what the economics of serving are built on: a provider fills the accelerator with many concurrent requests so that this half-second of work is being done for dozens of users at once.
Why this does not predict latency
The half-second above is arithmetic time, and a single user waiting for that answer will wait far longer. Generating one token requires reading the model’s weights out of memory, and at batch size one the accelerator spends nearly all of its time waiting on memory rather than computing. The FLOP count is not the binding constraint; memory bandwidth is.
Same 70B model at BF16, batch size 1, on the same accelerator.
Bytes that must be read per generated token:
70e9 params * 2 bytes = 1.40e11 bytes
At the H100 SXM's published 3.35 TB/s of memory bandwidth:
1.40e11 / 3.35e12 = 0.0418 s per token -> ~24 tokens/second
500 output tokens -> ~21 seconds of wall clock,
against 0.53 seconds of arithmetic.
The arithmetic units are idle roughly 97% of the time.
This is the single most important thing the FLOP count does not tell you, and it is why batching exists: adding a second concurrent request costs almost no extra memory traffic, because the same weights are already being read. Throughput scales with batch size until the arithmetic finally becomes the constraint. The full version of this calculation is in the tokens-per-second page.
When the rule is wrong
- Mixture-of-experts models. Use active parameters, not total. A model with 400B total and 40B active does
2 * 40e9 * TFLOPs per token and needs memory for all 400B. Applying the rule to the headline number over-states the arithmetic by ten times — the two parameter counts exist precisely because of this. - Reasoning models. The rule counts tokens, and a reasoning model emits tokens you never see. An answer that shows 200 tokens may have cost 4,000, so the FLOPs per visible token can be an order of magnitude off. Count reasoning tokens as output.
- Cached prefixes. A cache hit skips the prefill arithmetic for the cached span entirely. The FLOPs are not reduced, they are simply not performed again.
- Quantisation does not change the FLOP count. Running at INT4 rather than BF16 does the same number of operations on cheaper units. It changes bytes moved and operations per second available, not operations required.
- Speculative decoding changes the accounting entirely. A draft model proposes several tokens and the large model verifies them in one pass, so tokens produced and full forward passes stop being the same quantity.
Used within those limits, 2 * N * T is the most useful single equation in applied machine learning. It sizes hardware, it bounds cost, it explains pricing, and unlike almost every other number on this subject it cannot go out of date.
Top comments (0)