DEV Community: Wayne

Learnings of the Poor

Wayne — Sun, 26 Apr 2026 06:44:18 +0000

Necessity is the mother of invention

I was already GPU poor, but a recent job change combined with rising component prices have also made me RAM and NVMe poor.

While I am nowhere close to the experts of optimisations in the early 2000s or 90s, I took this time to brush up on some fundamentals and key concepts in Python. As the saying goes:

"Premature optimisation is the root of all evil"

We are not looking for very deep level optimisations, these changes aim to follow the Pareto Principle where 80% of the outcome comes from 20% of the effort. The changes below may or may not be 20% effort but I would consider them low-effort.

As such, there won't be any discussion on performance profiling, where we are determining hot loops, cache misses, memory reallocations etc.

Iterators

Frankly I think this is an important concept that has a great carryover regardless of languages. Understanding iterators also helps if you need to think of channels, which is very important in Go.

The typical approach collects results at every stage into lists:

data = read_file("data.jsonl")
data = first_filter(data)
data = second_processing(data)
write_processed_data(data, "output.jsonl")

The issue: if data.jsonl is bigger than your RAM, you run OOM very fast. Using yield instead keeps memory usage low:

from collections.abc import Iterator

def read_file(file: str) -> Iterator[dict]:
    with open(file, "r") as f:
        for line in f:
            yield json.loads(line)

def first_filter(input_data: Iterator[dict]) -> Iterator[dict]:
    for record in input_data:
        if is_good(record):
            yield record

Each function in the pipeline takes an Iterator[dict] and yields records one at a time. Memory usage drops significantly.

Caveats:

Files are held open throughout the pipeline, so unintentional edits or moves will break it.
json.dumps does not add a trailing newline, so f.write(json.dumps(record) + '\n') is intentional when writing JSONL.

Learning points

I find that iterators are a step before understanding pipelines, channels, or pub/sub patterns. When you understand iterators, you understand the bottlenecks of your code. They are fundamentally all iterators that consume and yield.

If process_data is slow (1 line per second) while reading and filtering is fast (4 lines per second), the pipeline is bounded by 1 line per second. The solution is more processing workers bridged through queues or channels:

Read-worker-1 -> Filter-worker-1 -> Process-worker-{1..4} -> Write-worker-1

Compression

In my Compression post, I mentioned that benchmarks should be done to know whether your use case supports compressions. For write once, read many scenarios, higher compression values may help.

Here is a measurement for an IO-constrained scenario (reading a JSONL file from NAS):

ZST: 100000it [00:05, 17220.01it/s]  (9.47 MB/s)
Raw: 100000it [00:40, 2492.39it/s]  (11.15 MB/s)

Because data is compressed, you can read more data per buffer. More lines are stored per MB of compressed JSONL compared to its raw form.

Less is more

Less work means more efficient processing. It's about eliminating wasted work, not always adding a cache everywhere.

If filtering takes 1s per line and processing takes 5s per line:

Process then filter on 10000 lines: 10000 * 6s = 60000s
Filter then process on 10000 lines (50% bad): 10000 * 1s + 5000 * 5s = 35000s

No complex code, no need for compiled languages. Algorithmic complexity matters too. Choosing the right data structure — a set for membership checks instead of a list, a deque instead of a list for queue operations — can eliminate entire classes of wasted work regardless of language.

The full version with code examples and benchmarks is on my blog.

How to Benchmark LLM Inference Performance: TTFT, ITL, and Throughput Metrics

Wayne — Sun, 26 Apr 2026 05:05:46 +0000

When deploying large language models to production, measuring performance accurately is critical. Whether you're using vLLM, SGLang, TensorRT-LLM, or a custom inference stack, you need to understand:

Throughput: How many requests per second can your system handle?
Latency metrics: Time to First Token (TTFT), Inter-Token Latency (ITL), and end-to-end latency
Token generation speed: Tokens per second under different concurrency levels
Tail latency: P95 and P99 values that affect user experience

In this post, I'll walk through the key metrics for benchmarking language models and share why I built llmperf-rs, a Rust-based benchmarking tool that takes a different approach to measuring these metrics.

The Problem with Existing Tools

While working with ray-project/llmperf (now archived), I noticed that Inter-Token Latency (ITL) was calculated by averaging per-request first, then aggregating those averages. This approach works well for many use cases, but I needed to preserve individual latency spikes during testing.

There's also genai-perf, which is very comprehensive. My only issue was running it on Ubuntu 22.04 without Docker. As of this update, they've sunsetted genai-perf in favor of aiperf.

vllm-bench is solid too, but requires installing vllm.

The goal was to build a simple binary that runs almost anywhere with minimal dependencies. It was also a learning project.

Metrics

This is a summary of the full metrics documentation.

Time To First Token (TTFT)

TTFT measures how quickly the model begins responding after receiving your request. For interactive applications, this is the perceived latency before the user sees any output. It's also important for RAG-based applications where a large chunk of processing happens at the prefill stage.

TTFT = first_token_timestamp - request_start_timestamp

Lower is better.

Inter-Token Latency (ITL)

ITL is the time between consecutive tokens during generation. Spikes can reveal multiple issues, most commonly network problems. ITL is usually consistent due to how KV caches and the computation works.

When testing against vLLM, I noticed that high ITL spikes happen when you benchmark close to the context limit. I suspect this is due to vLLM's eviction of requests if they exceed the KV cache size.

For example, if 3 requests come in with 0.8x context length and 0.2x for generation, but the GPU has space for only 2.8x context length, one of the requests will be preempted.

Aggregation: concatenate ALL ITL values across all responses, then compute statistics. Each response produces (N-1) ITL values (where N is the token count). By aggregating raw values instead of per-request averages, you preserve the true distribution including outliers.

Throughput Metrics

Prefill TPS — tokens processed per second during the prefill phase:

Prefill TPS = input_tokens / TTFT

However, prefill TPS doesn't accurately reflect system performance because TTFT includes queue wait time, not just actual processing time. When a server is under load, your request might sit in a queue waiting for resources. The lower prefill TPS in that case reflects queue contention, not the system's processing capability.

Decode TPS — tokens generated per second during the decode phase:

Decode TPS = output_tokens / (final_time - decode_start_time)

This is the generation speed: how fast the model produces output.

What Matters Most

For production serving, focus on TTFT, ITL stats, and maybe RPM.

TTFT measures how quickly users see their first token — this is the perceived responsiveness of your system.

ITL statistics reveal decode-phase issues that throughput metrics hide. The 99th percentile and max ITL values expose preemption events from KV cache limits and network issues between components.

ITL matters less for batch jobs or non-streaming APIs where users don't watch tokens arrive in real-time.

Token Counting

Accurate metrics require accurate token counts. llmperf-rs handles this in two ways:

API response — Most OpenAI-compatible endpoints return token counts in the usage field. By default, llmperf-rs uses this as priority.
Tokenizer — For exact input counts, pass a HuggingFace tokenizer. Note that chat templates may cause <10 token variance.

The original llmperf uses a single tokenizer for all models. Different models use different tokenizers, so llmperf-rs lets you specify the correct one or rely on API-reported counts.

For example, Llama-2 has a vocab size of 32000, while Qwen3-4B has 151936. In my own testing, setting input tokens to 8192 against a Qwen endpoint while using the default llama tokenizer returned values around 7363-7376 tokens.

Validating Your Results

All benchmark runs should end with finish_reason = length (meaning the model hit the max_tokens limit). If you see finish_reason = stop, the model stopped early. This affects metrics like RPM and E2E latency. Higher rejection rates can produce higher RPMs and lower latency due to shorter responses.

When to Use llmperf-rs

Use llmperf-rs when: running benchmarks with minimal dependencies, testing OpenAI-compatible endpoints, wanting low overhead (Rust, no Ray/ZMQ), or needing a quick way to test endpoints.

Consider alternatives when: you need GPU-level metrics (use trtllm-bench or aiperf), testing vLLM-specific features, requiring extensive reporting dashboards, or needing distributed testing.

Why ITL Matters Even When Throughput Looks Good

High throughput with bad ITL means tokens arrive in bursts, and chat users notice the choppy streaming. ITL spikes (p99 >100ms) often indicate preemption, network issues, or other problems. For non-user-facing use cases like agentic coding, throughput may matter more than ITL specifics.

The full version with code examples, benchmarks, and installation instructions is on my blog.