DEV Community

Cover image for How to benchmark LLM latency: 3 measurement traps I hit on 116 models
HAOHAOnovai
HAOHAOnovai

Posted on

How to benchmark LLM latency: 3 measurement traps I hit on 116 models

I ran a streaming TTFT probe against one OpenAI-compatible endpoint at three different times on the same day, same prompt, same server. Here is what the first measurement would have told me, and what the third one did.

model                 03:10 median   05:15 median   07:00 median
deepseek-v4-flash         0.41s          0.50s          0.24s
deepseek-v4-pro           0.20s          4.99s          4.39s
glm-4.7-flash             3.74s          1.26s         80.46s
kimi-k3                   2.26s          1.68s          3.16s
hy4-flash                 1.00s          0.92s          1.12s
Enter fullscreen mode Exit fullscreen mode

deepseek-v4-pro measured 0.20s at 03:10 and 4.39s four hours later. Same model, same endpoint, same prompt, same 16 max tokens. That is a 22x difference and it is not a typo.

If I had published the 03:10 number, it would have been accurate and useless. This post is about the three ways that happens.

Trap 1: n=1

A single request is not a measurement, it is a sample. The glm-4.7-flash row above is the reason to care: 0.61 / 3.74 / 4.29 in one session, 8.45 / 1.26 / 0.70 in another, 80.46 / 52.15 / timeout in a third. A single probe of that model can return anything from 0.6s to 80s depending on which request you happen to catch.

The cheap fix is three requests and a median, which costs almost nothing and immediately separates the two populations:

  • hy4-flash: 1.00 / 0.92 / 0.91 — spread 0.09s. Boring, publishable.
  • glm-4.7-flash: 8.45 / 1.26 / 0.70 — spread 7.75s. Not a latency number, a distribution with a long right tail.

A model whose median is 0.6s and whose p90 is 4.3s feels broken to users far more often than its median suggests. If your table only publishes the median, you have hidden the number that predicts the complaint.

Trap 2: measuring a model instead of a route

The 22x swing on deepseek-v4-pro is the interesting one because it did not happen to deepseek-v4-flash on the same vendor. Flash stayed between 0.24s and 0.50s all day. So the swing is not "DeepSeek is slow today", it is specific to that model's instance or queue.

That distinction is only visible if you measure more than one model per vendor and more than one model per category. A benchmark that measures "the API" and reports one number per provider will average a stable route with a degraded one and call the result a provider comparison.

What I would report instead, per model:

  1. median over n>=3
  2. spread (max - min), because the spread is the user-visible risk
  3. the time of measurement, because without it the number is not comparable to anything

The third item is the one everybody drops, and it is the one that makes a public leaderboard defensible or not.

Trap 3: assuming the catalogue is honest

Latency only matters for models that answer. I sent one max_tokens: 1 request to every id in a 116-entry catalogue and classified the responses by error type. 52 answered, 40 correctly complained that they are image/video/audio models and told me which endpoint to use instead, and:

2 entries were listed but not usable.

minimax-text-01
  HTTP 404  "[MiniMax] The model \"MiniMax-Text-01\" is no longer available. (2069)"
             type=upstream_provider_error

minimax-m2.5
  HTTP 402  "The free trial quota for the service has been exhausted and postpaid billing is ..."
Enter fullscreen mode Exit fullscreen mode

The first is the worst kind of catalogue defect: a reader sees 116 models, picks one, and gets a 404 that says the model was retired upstream. The second is cheaper to fix but the same category — listed, and unusable for a reason the listing does not mention.

Two out of 116 is 1.7%, which sounds small until you notice that a reader who picks either one concludes the whole catalogue is stale.

The audit costs one request per model and no generation spend, which is the point: you can check liveness with deliberately invalid requests, because a dead model fails differently from a wrong-endpoint model. 404 no longer available and 400 use /v1/images/generations instead are different answers and both are informative.

What I would do differently

Measure on a schedule, not once. Three timepoints on one day already changed the conclusion. A daily three-sample probe would have caught the deepseek-v4-pro regression the first morning it appeared instead of four hours into it.

Publish the spread next to the median. It is one extra number and it is the one that predicts complaints.

Verify liveness separately from latency. They fail differently and the check costs almost nothing.

Write the measurement time into the table header. I have read probably a hundred latency comparisons and the ones that omit it are, without exception, the ones I cannot use.

Limits, stated plainly

  • n=3 per model per timepoint. Enough to show the variance is real; not enough to publish a p90, which is what you would actually want to alert on.
  • One server, one network path. TTFT is client-location dependent, and this probe originates from the same region the endpoint is hosted in. Numbers from a different continent would be larger and differently shaped.
  • One endpoint. The 22x swing may be specific to this route, and treating it as a general claim about that vendor would be exactly the mistake this post is about.
  • The catalogue audit covered text endpoints only. The 40 non-text models were verified only as far as their error messages, not by running them.

The honest summary: the first number I measured was wrong in a way I could not have detected without measuring again later. That is an argument for scheduling measurements, not for measuring more carefully once.

Top comments (0)