Benchmarking LLM Inference: TTFT, ITL & Throughput

When deploying large language models to production, measuring performance accurately is critical. Whether you're using vLLM, SGLang, TensorRT-LLM, or a custom inference stack, you need to understand:

• Throughput: How many requests per second can your system handle?
• Latency metrics: Time to First Token (TTFT), Inter-Token Latency (ITL), and end-to-end latency
• Token generation speed: Tokens per second under different concurrency levels
• Tail latency: P95 and P99 values that affect user experience

In this post, I'll walk through the key metrics for benchmarking LLMs and share why I built llmperf-rs, a Rust-based benchmarking tool that takes a different approach to measuring them.

Why Existing Tools Fell Short

While working with ray-project/llmperf, which is now archived, I noticed it calculates Inter-Token Latency (ITL) by averaging per-request first, then aggregating those averages. That works for many use cases, but I needed to preserve individual latency spikes during testing.

There's also genai-perf, which was very in-depth. My only problem was not being able to run it natively on Ubuntu 22.04 without a Docker container. As of an edit (Apr 2026), they've sunsetted genai-perf in favor of aiperf, which I haven't tried but looks comprehensive. vllm-bench is solid too, but requires installing vllm.

The goal was a simple binary that runs almost anywhere with minimal dependencies. It was also a learning project.

Metrics

Time To First Token (TTFT)

TTFT measures how quickly the model begins responding after receiving your request. For interactive apps, this is the perceived latency before any output appears. It's also important for RAG-based applications where a large chunk of processing happens at the prefill stage.

TTFT = first_token_timestamp - request_start_timestamp

Lower is better.

Inter-Token Latency (ITL)

ITL is the time between consecutive tokens during generation. Spikes can come from multiple issues, most commonly network problems. ITL is usually consistent due to how KV caches and the computation works.

When testing against vLLM, I noticed high ITL spikes happen when you benchmark close to the context limit. I suspect this is due to vLLM evicting requests that exceed the KV cache size. If 3 requests come in with 0.8x context length and 0.2x for generation, but the GPU only has room for 2.8x, one request will be preempted. vllm preemption docs

Aggregation: concatenate ALL ITL values across all responses, then compute statistics. Each response produces N-1 ITL values (where N is the token count). By aggregating raw values instead of per-request averages, you preserve the true distribution including outliers.

Throughput Metrics

Prefill TPS counts tokens processed per second during the prefill phase:

Prefill TPS = input_tokens / TTFT

However, prefill TPS doesn't accurately reflect system performance, because TTFT includes queue wait time, not just actual processing time. Under load, your request might sit in a queue before prefill starts, so a lower prefill TPS often reflects queue contention, not the system's processing capability.

Decode TPS is tokens generated per second during the decode phase:

Decode TPS = output_tokens / (final_time - decode_start_time)

What Matters Most

For production serving, focus on TTFT, ITL stats, and maybe RPM.

TTFT is the perceived responsiveness of your system. ITL statistics reveal decode-phase issues that throughput hides: the 99th percentile and max ITL expose preemption events from KV cache limits and network issues. ITL matters less for batch jobs or non-streaming APIs where users don't watch tokens arrive in real-time.

Token Counting

Accurate metrics need accurate token counts. llmperf-rs handles this two ways:

1. API response gets priority: most OpenAI-compatible endpoints return token counts in the usage field.
2. Tokenizer (optional): the default Llama tokenizer ships inside the binary, so no network is needed. For exact input counts, override with a model-specific tokenizer. Note that chat templates can cause <10 token variance.

The original llmperf uses a single tokenizer for all models. Different models use different tokenizers, so llmperf-rs lets you specify the correct one. For example, Llama-2 has a vocab size of 32000, while Qwen3-4B has 151936. In my testing, setting input tokens to 8192 against a Qwen endpoint while using the default llama tokenizer returned values around 7363-7376 tokens. That's a real error if you care about exact numbers.

Validating Your Results

Benchmark runs should ideally end with finish_reason = length, meaning the model hit the max_tokens limit.

An edit (July 2026): with modern models, finish_reason = stop is common even with a high max_tokens, because the model may refuse the request or simply end early. Treat stops as acceptable, but be aware they add variance to output tokens, which then adds noise to RPM and E2E latency. I generally check the ratio of length vs stop rather than rejecting every stop outright.

When to Use llmperf-rs

Use it when: running benchmarks with minimal dependencies (single binary, Docker image), testing OpenAI-compatible endpoints (vLLM, Ollama, local APIs), wanting low overhead (Rust, no Ray/ZMQ), or just needing a quick level-above-curl way to test an endpoint.

Consider alternatives when: you need GPU-level metrics (trtllm-bench or aiperf), vLLM-specific features, extensive reporting dashboards, a Python-first workflow, or distributed testing.

Why ITL Matters Even When Throughput Looks Good

High throughput with bad ITL means tokens arrive in bursts, and chat users notice the choppy streaming. ITL spikes (p99 > 100ms) often indicate preemption or network issues. For non-user-facing cases like agentic coding, throughput may matter more than ITL specifics.

The full version with the detailed metrics documentation, installation steps, and example output is on my blog.