My benchmark reported 4,393,158 tokens per second

My API benchmark once reported 4,393,158 tokens per second. The useful result was finding out what that number actually measured.

The harness timed a streaming response and calculated throughput like this:

generation_time = total_request_time - time_to_first_token
tokens_per_second = output_tokens / generation_time

That calculation assumes the client observes tokens arriving over a meaningful interval after the first token. Some endpoints in the run instead held the response and delivered it in a burst. The first observed content arrived close to the end of the request. Subtracting those two timestamps left an interval close to zero.

Dividing the entire output by that tiny interval produced the spectacular number. It described the burst arriving at the client, not the model generating millions of tokens per second.

A streaming request is not proof of observable generation

Asking for streaming does not guarantee that the timing seen by the client exposes the underlying generation process. From that trace alone, I could not locate the buffering inside the serving path. I could only see that the arrivals did not support the throughput calculation I was making.

This is the distinction I needed in the results:

• Post-first-content throughput: output divided by the observed interval after first content, when the stream provides a useful interval.
• End-to-end throughput: output divided by the whole request duration, including waiting before content arrives.

The second number is useful for the caller, but it is a different metric. Quietly putting both in one ranking would hide the difference.

The guard I added

The benchmark now falls back to end-to-end timing when a streaming response has fewer than three data chunks or the calculated generation interval is below 0.2 seconds. It also withholds the first-token metric for responses classified as buffered. Empty output becomes an error rather than a row claiming zero tokens per second.

Here is the relevant logic, simplified from the harness:

generation_time = total_time - first_content_time
buffered = chunk_count < 3 or generation_time < 0.2

if buffered:
    generation_time = total_time

throughput = output_tokens / generation_time

Those thresholds are conservative measurement guards, not a universal detector of server buffering. A short, fast response can also produce a small interval. The harness has a separate streaming capability probe that uses chunk count; its comments explicitly warn against treating a short duration alone as evidence that streaming is fake.

The fallback gives the run a usable wall-clock measurement. It does not recover the hidden generation speed, and I should not label it as though it did.

Reproduce the failure without calling a model

For a synthetic illustration, suppose 1,000 output tokens arrive after a ten-second wait, with only 0.001 seconds between the first and last observed content:

post-first-content calculation: 1,000 / 0.001 = 1,000,000 tokens/s
end-to-end calculation:         1,000 / 10    =       100 tokens/s

These are illustrative inputs, not the raw timings behind my 4,393,158 result. They show why the denominator matters: the same output can produce radically different rates depending on the interval being measured.

Before trusting an unusually fast row, I now want the total duration, first-content timing, output token count, chunk count, and the metric actually used beside it. If the trace only exposes a final burst, the honest result is end-to-end throughput with the generation rate left unknown.