Nothing on this page was measured. It is arithmetic from two numbers you can look up — a model’s parameter count and a CPU’s vector throughput — and its value is that you can see which assumption to change when your machine disagrees with it.
FLOPs per document
A transformer forward pass costs approximately two floating-point operations per parameter per token — one multiply and one add for each weight in each matrix multiplication. The important refinement for embedding models is which parameters count: the token embedding table is a lookup, not a multiplication, so it must be excluded. That is a large correction for multilingual models, where the table is two-thirds of the parameters.
For a 12-layer, 768-wide encoder, the transformer weights are 12 x 12d² = 84.9M regardless of vocabulary. Attention over the sequence adds a term that does not involve weights at all: the score matrix and the value aggregation together cost about 4 x layers x s² x d. For a 512-token document:
weights term = 2 x 84.9e6 x 512 = 86.9 GFLOP
attention term = 4 x 12 x 512^2 x 768 = 9.7 GFLOP
total = 96.6 GFLOP per document
at 128 tokens:
weights term = 2 x 84.9e6 x 128 = 21.7 GFLOP
attention term = 4 x 12 x 128^2 x 768 = 0.6 GFLOP
total = 22.3 GFLOP per document
Two things to read off that. The weights term is linear in sequence length and the attention term is quadratic, so at 512 tokens attention is only a tenth of the bill and at 8192 tokens it would dominate completely — 4 x 12 x 8192² x 768 = 2,474 GFLOP against a weights term of 1,391 GFLOP. That crossover is why long-context embedding models are far more expensive per document than their parameter count suggests.
The second thing to read off is that documents per second is the wrong unit for planning if your documents vary in length. Tokens per second is close to constant across lengths for the weights term, and it is the number that lets you estimate a corpus pass: total tokens divided by tokens per second. Convert to documents at the end if you want a headline, using your corpus’s actual mean token count rather than the model’s maximum sequence length — substituting the maximum is the usual mistake and it overstates the cost by whatever fraction of the window your documents do not fill.
What a CPU can do per second
Peak floating-point throughput is a property of the instruction set, the core count and the clock, and it is arithmetic rather than a benchmark:
FLOP/s = cores x clock x FLOP-per-cycle-per-core
FLOP-per-cycle, fp32:
AVX2 2 FMA units x 8 lanes x 2 (multiply+add) = 32
AVX-512 2 FMA units x 16 lanes x 2 = 64
example: 8 cores at 4.0 GHz with AVX2
8 x 4.0e9 x 32 = 1,024 GFLOP/s peak
Check which applies with lscpu | grep -o 'avx[^ ]*' on Linux, and check whether your CPU actually has two FMA units — some parts have one, halving the figure — and remember that sustained all-core clocks under vector load are lower than the boost clock on the box.
Peak is not achievable. Well-tuned dense matrix multiplication on a CPU reaches perhaps 60–80% of peak for large operands; a transformer’s matrices at modest batch sizes are not large operands, and the pass is interrupted by layer norms, softmaxes and activations that are memory-bound rather than arithmetic-bound. A sustained fraction of 40% is a reasonable planning assumption and is the assumption used below. If your measurement lands at half the estimate, this is the term that was wrong.
Putting them together
assumed sustained throughput = 0.40 x 1,024 = 410 GFLOP/s
512-token documents: 96.6 / 410 = 0.236 s -> ~4.2 docs/s
128-token documents: 22.3 / 410 = 0.054 s -> ~18.4 docs/s
Every one of those numbers is the output of this arithmetic and none of them is a measurement. The full assumption list, so you can see which to change: 2 FLOPs per parameter per token; 84.9M non-embedding parameters from the 12-layer, 768-wide geometry; the attention term as 4 x layers x s² x d; an 8-core 4.0 GHz AVX2 CPU with two FMA units; 40% of peak sustained; fp32 throughout; and no tokenization, I/O or padding overhead.
The useful conclusions are ratios rather than absolutes, and they are robust to the sustained-fraction assumption being wrong because it cancels:
- Quartering the document length gives roughly 4.4x the documents per second — slightly better than linear because the quadratic attention term shrinks faster than the linear one.
- A large model (24 layers, 1024 wide: 24 x 12 x 1024² = 302M transformer parameters) costs 3.6x the weights term of a base model, so expect roughly 1.2 docs/s at 512 tokens under the same assumptions.
- A small model (12 layers, 384 wide: 21.2M) totals 26.5 GFLOP per 512-token document, a little over a quarter of base, so roughly 15 docs/s under the same assumptions.
- AVX-512 doubles FLOP per cycle, so an otherwise identical machine with it should roughly double these figures — in practice less, because AVX-512 workloads often run at a reduced clock.
What the estimate misses
Four things, and in a real pipeline they frequently matter more than everything above.
- Padding. A batch is padded to its longest member and every padded position costs full FLOPs. Batch a 30-token title with a 500-token article and you compute 500 tokens twice. On a corpus with mixed lengths this alone can double or triple the real cost, and sorting the corpus by token length before batching recovers most of it for the price of one sort.
- Batch size one. At batch one the matrix multiplications become matrix-vector products, which read the whole weight matrix to produce a small output. That is memory-bandwidth bound, not arithmetic bound, and the FLOP estimate does not apply at all — throughput will be far below it. Batch size is the single largest lever on CPU embedding throughput.
- Threads. Set
OMP_NUM_THREADSandtorch.set_num_threads()deliberately. The default often oversubscribes, and two libraries each spawning a full thread pool is a common and expensive mistake. - Tokenization and I/O. Fast tokenizers are quick but not free, and on short documents the pipeline around the model can take a comparable share of wall time.
Measuring instead
The estimate exists to be replaced. This takes a minute:
OMP_NUM_THREADS=8 python - <<'PY'
import time, torch
from sentence_transformers import SentenceTransformer
torch.set_num_threads(8)
model = SentenceTransformer("BAAI/bge-base-en-v1.5", device="cpu")
docs = [l.strip() for l in open("corpus_sample.txt")][:512]
model.encode(docs[:16]) # warm up
for bs in (1, 8, 32, 64):
t = time.perf_counter()
model.encode(docs, batch_size=bs, normalize_embeddings=True)
dt = time.perf_counter() - t
print("batch", bs, "->", round(len(docs) / dt, 1), "docs/s")
PY
Run it on your own documents, not on repeated copies of one sentence, because length distribution is part of the answer. The batch-size sweep is the important part: the gap between batch 1 and batch 32 is usually several-fold and tells you whether you are bandwidth-bound or arithmetic-bound, which is the thing that decides whether to reach for an int8 ONNX export next.
Top comments (0)