DEV Community: Wayne

Measuring LLM Prefix Caching: The Cache Hit Rate Metric

Wayne — Wed, 05 Aug 2026 02:49:08 +0000

Prefix caching is one of the biggest cost levers in LLM serving. vLLM, SGLang, TGI, and most hosted providers all do some version of it: during prefill they compute a key-value (KV) cache, and if a later request shows up with the same prompt prefix, they reuse that cache instead of recomputing it. Done well, a lot of expensive prefill compute turns into a cheap cache lookup.

Whether it helps depends on how much of your traffic re-sends the same prefix, and most benchmarking runs don't tell you. This is part of my LLM benchmarking guide. Here I want to focus on how to actually measure cache effectiveness: the metric, why it matters for agentic workloads, and the cost angle.

What prefix caching reuses

During prefill, the server computes the KV cache for the input prompt. If a later request sends a prefix the server has already seen, it can skip recomputing that part and just serve it from cache. The classic example is a multi-turn conversation: each turn re-sends the entire prior history, and ideally everything except the newest user message comes back from cache.

This is also why cache reuse only shows up in multi-turn or shared-prefix workloads. A single isolated request has nothing to reuse. Turn 0 is always a cold start. So if you want to measure caching, you have to re-send history, which means multi-turn requests.

Why this matters most for agentic workloads

A chat conversation re-sends some history every turn, but an agentic coding loop does this on fast-forward, and at a scale where caching stops being optional and starts dominating both latency and cost.

Here's how a coding agent actually runs (Claude Code, Cursor, Cline, that sort of thing). It loops: read the task, decide on an action, call a tool to read a file or run a command, get the result back, decide the next action, call another tool. Each one of those iterations is a new API request, and every request re-sends the entire accumulated context. The system prompt, the original task, all the prior reasoning, every previous tool call and its result. The only genuinely new content is the latest tool result and the model's next decision. Everything before that is a prefix the server has already computed.

Latency. Without caching, each step's time-to-first-token includes recomputing prefill for the whole context. As the conversation grows, every step gets a little slower, so the agent loop itself drags as the session wears on. With good caching, only the new tool result triggers prefill, and TTFT stays roughly flat across the session instead of climbing.
Cost. Prefill compute scales with context length, so re-paying for the full context on every tool call gets expensive fast. Agentic sessions are notoriously long, often tens of thousands of tokens and well past 100k. With caching you pay for each token's prefill once instead of on every subsequent step.

So an agentic session is really just a long multi-turn conversation where history gets re-sent every turn, which is exactly the case cache_hit_rate was built for. If you're picking a serving setup for agentic workloads, cache hit rate under a realistic multi-turn load is one of the most telling numbers you can collect.

One nuance worth knowing: some providers also offer explicit prompt caching, where the client marks cache breakpoints (Anthropic's cache_control is the example). That's a different mechanism from the automatic prefix caching most OpenAI-compatible endpoints do, and llmperf-rs only measures the automatic kind. For a standard tool-call loop against a vLLM-style endpoint, automatic prefix caching is what applies.

The cache hit rate metric

The metric I use measures cache reuse against the content that was previously sent, not the whole request. Caching only reuses what the server has already seen: the assistant's prior outputs and earlier user prompts that get echoed back in the next request. New tokens in the current turn can never be cached, because the server hasn't seen them before.

Implemented in llmperf-rs:

cache_hit_rate = sum(cached_tokens) / sum(total_tokens_of_non_final_turns)

Numerator: the sum of cached_tokens reported by the endpoint on each turn, read from prompt_tokens_details.cached_tokens in the streamed usage object. None means the endpoint didn't report the field.
Denominator: the sum of each turn's total tokens (input + output) for every turn except the last turn. The last turn's content is never re-sent in a later request, so it can never be served from cache and is excluded.

100% means every previously-sent token came back from cache. In a perfect cache, cached_tokens equals the prior-turn total on every warm turn.

Edge cases

A few that bite in practice:

Single-turn runs report None. With only one turn there's no history to re-send, so the denominator is zero. Cache hit rate is a multi-turn concept.
An all-None run reports None. If the endpoint never reports cached_tokens, you get None, not zero. That's deliberate: None means "not measurable", which is different from 0.0 (a cache that's just never hit).
A mismatched endpoint dilutes instead of nulling. If some turns report cached_tokens and others don't, the unobserved turns are left out of the numerator but their re-sent history still counts in the denominator. So a noisy endpoint just pulls the ratio down rather than wiping it out.
Cold start contributes nothing. Turn 0 reports cached_tokens = 0 or just omits it, so it doesn't move the numerator either way.

How to run it

You need multi-turn requests, which in llmperf-rs is --multi-turn N:

export OPENAI_API_BASE=http://localhost:8000/v1   # vLLM with prefix caching enabled
llmperf --model Qwen/Qwen3-4B-Instruct-2507 \
        --multi-turn 5 \
        --max-num-completed-requests 10

The summary then includes a cache_hit_rate field (alongside the TTFT/ITL/throughput metrics covered in the main guide):

Example value only, illustrative and not from a real run.

"cache_hit_rate": 0.91

That single summary number aggregates across the whole run. Per-turn cached_tokens and turn_index are also written to the individual-responses file if you want to see how the cache builds up over turns after the cold start.

A note on reasoning content

There's a subtlety if you're benchmarking reasoning models. The common guidance is to discard a model's reasoning_content from the message history you send back, to save tokens. llmperf-rs does the opposite for multi-turn runs: it echoes the previous turn's reasoning_content on the assistant message.

The reason is exactly this topic. Providers that support prefix caching over reasoning (Z.ai's "Preserved thinking" with clear_thinking: false, for example) can reuse the KV cache across turns only if the reasoning is re-sent. Dropping it to save on echoed-input tokens throws away the cache reuse, which usually costs more than it saves. Providers that don't understand reasoning_content just ignore the field, so it's safe to send.

So if you're measuring cache hit rate on a reasoning model, make sure you're re-sending the reasoning. Otherwise you're measuring a workload that disables its own cache.

Caveats

Token-count accuracy matters less here than elsewhere, because the ratio is between two token sums rather than a token count against a wall-clock time. Chat-template token variance affects numerator and denominator similarly, so it mostly cancels out.
Cache behavior depends on server config, not just the model. vLLM's prefix caching can be on or off; KV-cache size and eviction policy affect whether a warm turn actually hits cache. A low cache hit rate under load can point at preemption rather than a broken cache.
This measures endpoint-level prefix caching, not GPU-level cache statistics. For kernel-level breakdowns you'd want a tool like aiperf or trtllm-bench. See my notes on llmperf alternatives.

Wrapping up

If you've enabled prefix caching, cache hit rate is how you confirm it's earning its keep. The key thing to get right is the denominator: measure cache reuse against the history you re-sent, not against the whole request, or you'll understate a cache that's working fine. And remember it's strictly a multi-turn metric. A single-turn benchmark tells you nothing about caching.

The full version with the exact math and more detail is on my blog.

llmperf Is Archived: Alternatives for LLM Benchmarking

Wayne — Wed, 05 Aug 2026 02:47:21 +0000

If you've been using ray-project/llmperf, you may have noticed it's now in archive mode. No new updates, no fixes, no responses to issues. If you're evaluating it for the first time, that's worth knowing before you build anything on top of it.

This page is part of my LLM benchmarking guide, which covers the metrics themselves (TTFT, ITL, throughput). Here I want to focus on the tools — what's out there now that llmperf is effectively done, and what I ended up building.

What llmperf was good at

Credit where it's due — llmperf was the go-to open-source option for benchmarking OpenAI-compatible endpoints. It measured the metrics that matter (TTFT, ITL, throughput), handled concurrency, and came out of the Anyscale/Ray team, so it had credibility. For a lot of teams it did the job well.

Why I started looking around

Two things pushed me to look at alternatives, and both are about fit rather than flaws:

ITL aggregation. llmperf computes Inter-Token Latency by averaging within each request first, then aggregating those per-request averages. That's a reasonable choice and works well for many use cases. But I was specifically trying to catch latency spikes during the decode phase, and per-request averaging smooths exactly those out. I needed the raw distribution.
Startup overhead. Runs spawn Ray workers, so there's a meaningful spin-up cost before the first request fires. When I just want to poke at an endpoint quickly, that's more ceremony than I want — I was after something closer to curl than a cluster setup.

Neither is a flaw. They're design decisions that matched llmperf's goals and didn't match mine.

The alternatives

Tool	Single binary / low deps	GPU-level metrics	Distributed	Notes
aiperf	No	Yes	Yes	Python package, very comprehensive
vllm-bench	No	Yes	No	vLLM-specific
trtllm-bench	No	Yes	No	TensorRT-LLM specific
GuideLLM	No	Partial	No	Strong reporting and dashboards
llmperf-rs	Yes	No	No	Single Rust binary

There's also a note for genai-perf: NVIDIA sunsetted it and moved development to aiperf.

When to use which

You need GPU-level metrics (prefix caching, kernel-level breakdown) → aiperf or trtllm-bench. This is where llmperf-rs won't help you.
You're testing vLLM-specific behavior → vllm-bench.
You want dashboards and visual reporting → GuideLLM.
You need distributed load generation → aiperf.
You just want to hit an OpenAI-compatible endpoint quickly, with minimal setup, and see TTFT/ITL/throughput → llmperf-rs.

That last one is the gap I was trying to fill — something I could drop onto a box and run in seconds, that preserved raw ITL values so spikes weren't hidden.

What I built: llmperf-rs

llmperf-rs is a single Rust binary that benchmarks any OpenAI-compatible endpoint (vLLM, Ollama, local APIs). It:

Keeps raw ITL values across all responses before computing percentiles, so p99 and max actually reflect spikes.
Uses API-reported token counts from the usage field when available, falling back to a tokenizer you specify. The original llmperf used one tokenizer for everything, which gets inaccurate across model families.
Outputs console summaries plus JSON for digging in with pandas.

It's not trying to compete with the GPU-deep tools. I think of it as one level above curl — fast to start, low dependency, good enough for most "how's this endpoint doing" questions.

If that trade-off sounds right for you: grab it from the releases page, or cargo install --git https://github.com/wheynelau/llmperf-rs.

The takeaway

llmperf being archived doesn't mean the tooling disappeared — aiperf, vllm-bench, GuideLLM, and others are all actively maintained. The choice mostly comes down to how deep you need to go (GPU metrics vs endpoint metrics) and how much setup you're willing to tolerate.

You can find this post and more on my blog.

Switching to Secondary Is Faster

Wayne — Sat, 02 May 2026 06:57:07 +0000

Remember, switching to your pistol is always faster than reloading.

The same idea applies to LLM workflows.

Most of the time, you don't need a flagship model to scaffold a project. Boilerplate, spec drafts, and initial plans are all tasks where a smaller model can do the heavy lifting. Then you pass the result to a larger model for review.

Why this works

Prefill is usually a single forward pass (not counting advanced stuff like chunking and sequence parallelism). Fundamentally, the next token is just model.forward(). How does this help?

Say your initial prompt is 16k tokens (a rough ballpark for a Claude Code session) and you need to generate another 16k tokens of output (tool calls, reads, edits included). If your large model generates at 50t/s, a small model can easily hit 200t/s. That's 80 seconds versus 320 seconds for the same 16k tokens.

The concept is the same as speculative decoding. Modern decoders use a small draft model to propose multiple tokens at once, then the large model verifies them in parallel. Using a secondary model for the first pass is just speculative decoding scaled to 16k tokens.

The workflow

Here's what I've been doing:

Plan — either with a small model for speed, or directly with a large model for precision. The large model is more accurate but spends more tokens on the planning stage.
Review — pass the plan to a large model, fix what's wrong.
Generate code — small model implements from the refined spec.
Review again — catch what the small model missed.

For the small model, I've been using Qwen 3.6 35b MoE. It's fast enough to run locally and produces reasonable boilerplate. The large model then acts as a reviewer rather than a first-pass generator.

This hasn't been tested on novel codebases. For truly new problems, I write the code myself and use the small model for repetitive tasks like generating tests and boilerplate.

You can find this post and more on my blog.

Ansible at Home

Wayne — Mon, 27 Apr 2026 05:17:16 +0000

Introduction

While some concepts in Engineering should not be brought back to home, I find that Ansible was one of the few tools that is actually useful in a home environment.

Why Ansible?

Ansible is a powerful automation tool that can help manage and configure systems efficiently. Another key important aspect commonly missed out is documentation. In the past, I would usually SSH into my home server and make changes directly. If I remember to document it, I would save code snippets into a README.md or obsidian note. However, this approach is prone to human error and can lead to inconsistencies over time. Most forms of IaC (Infrastructure as Code) tools are self documenting, as the code itself serves as documentation.

Setup and Configuration

Before diving into use cases, it's important to set up Ansible properly for a home environment. The configuration is straightforward and makes running playbooks much more convenient.

I keep two files in the project directory: an ansible.cfg pointing to my inventory file and enabling become_ask_pass so it prompts for sudo passwords rather than storing credentials (security first, even at home), and an inventory.ini with at least localhost ansible_connection=local so playbooks run locally without SSH overhead. With those in place, ansible-playbook playbook.yml just works.

Use Cases

System configuration

I am using a consumer intel CPU with a stock cooler for my homelab. As I don't expect it to run heavy workloads, I don't need it to run at full power. I set the PL1 and PL2 power limits through Ansible rather than using the intel-undervolt tool. This way, if I ever need to re-install the OS or set up a new server, I can easily apply the same configuration without having to remember the exact commands or settings.

The playbook validates that PL1 and PL2 values fall within acceptable ranges (hard lower/upper limits) and ensures PL2 >= PL1 before applying them. It then writes to the sysfs powercap interfaces to set sustained and burst power limits, and creates a systemd service for persistence across reboots.

It's easy to make mistakes when setting raw power values -- adding one extra zero can be disastrous. With Ansible, I can specify values like 65W and 90W instead of 65000000 and 90000000, and the validation layer catches out-of-range inputs before they get applied.

Restic setup

Restic is a great backup tool that can be used to back up data to various locations. The backup scripts are manually written, but the cron jobs and log rotation are managed by Ansible. If the below process were to be done manually, it would be prone to human error and inconsistencies, as multiple files are involved, such as cron jobs, log rotation configuration, and the backup scripts themselves.

The playbook ensures restic is installed, makes the backup scripts executable, sets up two daily cron jobs (one for immich data at 2 AM, one for documents at 3 AM), and configures logrotate to keep 21 days of compressed logs with daily rotation.

Conclusion

As you add more services and configurations to your home environment, the benefits of using Ansible become even more apparent. It helps maintain consistency, reduces the risk of human error, and serves as documentation for your setup. Whether you're managing a single server or multiple devices, Ansible can streamline your home automation tasks effectively.

The full version with the complete playbook examples is on my blog.

Making Compression a Habit with zstd

Wayne — Sun, 26 Apr 2026 13:47:27 +0000

With zstd being added to Python 3.14, I've been using compressed files more often in my workflow. Here's what I've learned about making compression a habit.

Python Data Processing with Compression

Python 3.14 adds native zstd.open() support, which is a big step forward. Here's the comparison:

Before 3.14 (with zstandard package):

import zstandard as zstd
import json
import io

# Writing compressed JSONL with Zstandard
data = [
    {"id": 1, "name": "Alice", "score": 95},
    {"id": 2, "name": "Bob", "score": 87},
    {"id": 3, "name": "Charlie", "score": 92}
]

# Write
with open('data.jsonl.zst', 'wb') as f:
    cctx = zstd.ZstdCompressor(level=3)
    with cctx.stream_writer(f) as writer:
        for record in data:
            line = (json.dumps(record) + '\n').encode('utf-8')
            writer.write(line)

# Read
with open('data.jsonl.zst', 'rb') as f:
    dctx = zstd.ZstdDecompressor()
    with dctx.stream_reader(f) as reader:
        text_stream = io.TextIOWrapper(reader, encoding='utf-8')
        for line in text_stream:
            record = json.loads(line)
            print(record)

Python 3.14+ is much simpler:

from compression import zstd
import json

# Read and print first record
with zstd.open('data.jsonl.zst', "rt") as f:
    for line in f:
        data = json.loads(line)
        print(data)
        break  # Remove break to read all lines

The API mirrors regular open() -- just use zstd.open() instead.

Key points:

Use 'wt' mode for writing text, 'rt' for reading
Typical compression ratio: 6-7x size reduction at zstd-3

Benchmarking Your Workload

You should benchmark compression according to your workload to determine your trade-offs.

For archival of logs or long-term storage, you can use higher compression levels of zstd. Archives like Pushshift Reddit typically use level 22. For most use cases, zstd-3 is a good default.

Working with Compressed Files

Zstd includes tools for viewing, searching, and processing compressed files without manual decompression.

Quick commands:

zstdcat data.json.zst -- view the contents
zstdless data.json.zst -- page through like less
zstdgrep "error" events.json.zst -- search inside compressed files
zstdgrep -c "timeout" events.json.zst -- count occurrences

You can also pipe to other tools: zstdcat events.json.zst | grep ERROR | jq '.timestamp'

Transferring files

Rsync

The -z flag compresses data during transfer. On highly compressible files, rsync may report speedup > 1.0x. Here's a test with about 66GB of JSONL files:

sent 50,322 bytes  received 12,451,737,167 bytes  19,290,143.28 bytes/sec
total size is 66,857,841,487  speedup is 5.37

That speedup means the data sent was much less than the original file size. This is highly beneficial if you're network bound or concerned about egress costs.

S3 and Cloud Storage

AWS charges for outbound data transfer (egress). Compressing data before storage can significantly reduce these costs. With a 7.0x compression ratio, a $14,000 egress bill drops to roughly $2,000.

Here's an upload comparison on a gigabit connection with a 4GB JSONL file:

# Compressed upload (zstd -k -c ... | s5cmd pipe)
real    0m8.139s
# Result in S3: 363.7MB

# Uncompressed upload (s5cmd cp)
real    0m57.547s
# Result in S3: 4.0GB

The same principle works for downloading: s5cmd cat s3://bucket/data.zst | zstd -d > data.jsonl. Compression takes longer than decompression, but the speedup is usually worth it.

Conclusion

I use zstdcat to read files and rarely need to edit them in an IDE. This habit cut my text storage by up to 80%. There's a balance between convenience, speed, and storage - and this works for me. More optimized formats like protobuf or arrow exist, but most text processing still uses JSON.

The full version with code examples and benchmarks is on my blog.

Using hf tokenizers in Rust

Wayne — Sun, 26 Apr 2026 13:43:42 +0000

The tokenizers library from Hugging Face provides an efficient way to work with text tokenization in Rust. This guide shows you how to get started with pretrained tokenizers.

Setup

First, add the tokenizer library to your project:

cargo add tokenizers --features http,hf-hub

Basic Usage

Here's a complete example that loads a pretrained tokenizer and processes text:

use tokenizers::Tokenizer;

fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
    // Load a pretrained tokenizer
    let tokenizer = Tokenizer::from_pretrained("hf-internal-testing/llama-tokenizer", None)?;

    let text = "This is a sample string to tokenize";

    // Encode the text (false = no special tokens)
    let encoding = tokenizer.encode(text, false)?;

    // Get token IDs
    let token_ids = encoding.get_ids();
    println!("Token IDs: {:?}", token_ids);

    // Get token text
    let tokens = encoding.get_tokens();
    println!("Tokens: {:?}", tokens);

    println!("Original: {}", text);
    println!("Number of tokens: {}", token_ids.len());

    let decoded = tokenizer.decode(token_ids, true)?;
    println!("Original: {}", text);
    println!("Decoded: {}", decoded);

    Ok(())
}

Working with Different Models

You can use various pretrained models:

// GPT-2 tokenizer
let gpt_tokenizer = Tokenizer::from_pretrained("gpt2", None)?;

// BERT tokenizer
let bert_tokenizer = Tokenizer::from_pretrained("bert-base-uncased", None)?;

// Llama tokenizer
let llama_tokenizer = Tokenizer::from_pretrained("hf-internal-testing/llama-tokenizer", None)?;

Configuration

To change the cache directory for downloaded models, set the HF_HOME environment variable:

export HF_HOME=/path/to/your/cache

Setting environment variables programmatically is not recommended as it requires an unsafe block.

Private Repositories

If you encounter this error:

Error: RequestError(Status(401, Response[status: 401, status_text: Unauthorized, url: https://huggingface.co/google/gemma-3-12b-it/resolve/main/tokenizer.json]))

It means you are not authenticated and may require a token. There are two ways to achieve this:

Write your token to $HF_HOME/token, usually $HOME/.cache/huggingface
Within Rust code:

use tokenizers::{Tokenizer, FromPretrainedParameters};

let params = FromPretrainedParameters {
    token: Some("<your very secret token>".to_string()),
    ..Default::default()
};
let tokenizer = Tokenizer::from_pretrained("google/gemma-3-4b-it", Some(params))?;

Note that you may still need to get permission to access the repos.

Branches

You can specify a specific branch or revision:

use tokenizers::{Tokenizer, FromPretrainedParameters};

let params = FromPretrainedParameters {
    revision: "main".to_string(),  // or specific commit hash
    ..Default::default()
};
let tokenizer = Tokenizer::from_pretrained("google/gemma-3-4b-it", Some(params))?;

User-Agent

Params have another variable called user_agent for customizing the HTTP client user agent string.

use tokenizers::{Tokenizer, FromPretrainedParameters};

let params = FromPretrainedParameters {
    user_agent: Some("my-rust-app/1.0".to_string()),
    ..Default::default()
};
let tokenizer = Tokenizer::from_pretrained("gpt2", Some(params))?;

Summary

The Hugging Face tokenizers library provides a robust, production-ready solution for text processing in Rust applications. With support for pretrained models, authentication for private repositories, and flexible configuration options, it's an excellent choice for NLP workflows in Rust.

You can find this post and more on my blog.

Setting Up Docker CI for Rust with cargo-dist

Wayne — Sun, 26 Apr 2026 13:29:54 +0000

Rust CI

The Core Idea

Building Rust inside Docker is slow. A typical multi-stage Dockerfile compiles the binary in one stage and copies it into a minimal image in another. That works fine for local builds, but in CI it takes a long time, especially when you're emulating arm64 through QEMU.

The better approach: let cargo-dist handle the compilation as part of the release workflow. By the time the Docker job runs, the binaries are already built and available as GitHub Actions artifacts. Docker just copies them in. QEMU is still needed for the final multi-arch manifest, but it's only moving files around rather than running a compiler through emulation, so arm64 builds don't take nearly as long.

The Setup

The starting point is the cargo-dist quickstart guide. Once that's in place, you need a few configuration pieces to trigger the Docker build after the release.

In release.yml, add a custom-docker-publish job that calls your docker-publish workflow and passes the plan output and binary name as inputs.

In dist-workspace.toml, set post-announce-jobs to point at your docker workflow:

post-announce-jobs = ["./docker-publish"]
github-custom-job-permissions = { "docker-publish" = { packages = "write", contents = "read" } }
allow-dirty = ["ci"]

The permissions block was needed because my docker workflow didn't have enough access by default.

The Docker Workflow

The workflow runs as a workflow_call and takes the dist plan JSON, binary name, and target triple suffix as inputs. Here's the overall structure:

on:
  workflow_call:
    inputs:
      plan:
        required: true
        type: string
      binary_name:
        required: true
        type: string
      target_triple_suffix:
        required: false
        type: string
        default: "unknown-linux-musl"

The job itself:

Set up QEMU and Docker Buildx
Log in to GHCR
Extract the version from the dist plan's announcement_tag
Generate Docker metadata (semver tags, major.minor, major, and latest for non-prereleases)
Download the amd64 and arm64 artifacts produced by cargo-dist
Extract and normalize the artifacts, moving binaries into the right folders
Build and push with docker/build-push-action@v6 targeting both linux/amd64 and linux/arm64

The version tags are pulled from the dist plan, so they stay in sync with cargo-dist's release process. The latest tag is skipped for prereleases.

- name: Build and push
  uses: docker/build-push-action@v6
  with:
    context: .
    platforms: linux/amd64,linux/arm64
    push: true
    tags: ${{ steps.meta.outputs.tags }}
    build-args: BINARY_NAME=${{ inputs.binary_name }}

The Dockerfile

The Dockerfile depends on what your binary needs. I used distroless images and determined the right base image by running ldd on the compiled binary:

linux-vdso.so.1 (0x00007ffdfb764000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6
/lib64/ld-linux-x86-64.so.2

Since this binary needed libc and libm, I went with gcr.io/distroless/cc-debian13:nonroot.

FROM gcr.io/distroless/cc-debian13:nonroot

ARG TARGETARCH
ARG BINARY_NAME

COPY --chmod=755 artifacts/${TARGETARCH}/${BINARY_NAME} /usr/local/bin/app

EXPOSE 8000

USER nonroot:nonroot

ENTRYPOINT ["/usr/local/bin/app"]

The full version with the complete workflow YAML and more context is on my blog.

Learnings of the Poor

Wayne — Sun, 26 Apr 2026 06:44:18 +0000

Necessity is the mother of invention

I was already GPU poor, but a recent job change combined with rising component prices have also made me RAM and NVMe poor.

While I am nowhere close to the experts of optimisations in the early 2000s or 90s, I took this time to brush up on some fundamentals and key concepts in Python. As the saying goes:

"Premature optimisation is the root of all evil"

We are not looking for very deep level optimisations, these changes aim to follow the Pareto Principle where 80% of the outcome comes from 20% of the effort. The changes below may or may not be 20% effort but I would consider them low-effort.

As such, there won't be any discussion on performance profiling, where we are determining hot loops, cache misses, memory reallocations etc.

Iterators

Frankly I think this is an important concept that has a great carryover regardless of languages. Understanding iterators also helps if you need to think of channels, which is very important in Go.

The typical approach collects results at every stage into lists:

data = read_file("data.jsonl")
data = first_filter(data)
data = second_processing(data)
write_processed_data(data, "output.jsonl")

The issue: if data.jsonl is bigger than your RAM, you run OOM very fast. Using yield instead keeps memory usage low:

from collections.abc import Iterator

def read_file(file: str) -> Iterator[dict]:
    with open(file, "r") as f:
        for line in f:
            yield json.loads(line)

def first_filter(input_data: Iterator[dict]) -> Iterator[dict]:
    for record in input_data:
        if is_good(record):
            yield record

Each function in the pipeline takes an Iterator[dict] and yields records one at a time. Memory usage drops significantly.

Caveats:

Files are held open throughout the pipeline, so unintentional edits or moves will break it.
json.dumps does not add a trailing newline, so f.write(json.dumps(record) + '\n') is intentional when writing JSONL.

Learning points

I find that iterators are a step before understanding pipelines, channels, or pub/sub patterns. When you understand iterators, you understand the bottlenecks of your code. They are fundamentally all iterators that consume and yield.

If process_data is slow (1 line per second) while reading and filtering is fast (4 lines per second), the pipeline is bounded by 1 line per second. The solution is more processing workers bridged through queues or channels:

Read-worker-1 -> Filter-worker-1 -> Process-worker-{1..4} -> Write-worker-1

Compression

In my Compression post, I mentioned that benchmarks should be done to know whether your use case supports compressions. For write once, read many scenarios, higher compression values may help.

Here is a measurement for an IO-constrained scenario (reading a JSONL file from NAS):

ZST: 100000it [00:05, 17220.01it/s]  (9.47 MB/s)
Raw: 100000it [00:40, 2492.39it/s]  (11.15 MB/s)

Because data is compressed, you can read more data per buffer. More lines are stored per MB of compressed JSONL compared to its raw form.

Less is more

Less work means more efficient processing. It's about eliminating wasted work, not always adding a cache everywhere.

If filtering takes 1s per line and processing takes 5s per line:

Process then filter on 10000 lines: 10000 * 6s = 60000s
Filter then process on 10000 lines (50% bad): 10000 * 1s + 5000 * 5s = 35000s

No complex code, no need for compiled languages. Algorithmic complexity matters too. Choosing the right data structure — a set for membership checks instead of a list, a deque instead of a list for queue operations — can eliminate entire classes of wasted work regardless of language.

The full version with code examples and benchmarks is on my blog.

Benchmarking LLM Inference: TTFT, ITL & Throughput

Wayne — Sun, 26 Apr 2026 05:05:46 +0000

When deploying large language models to production, measuring performance accurately is critical. Whether you're using vLLM, SGLang, TensorRT-LLM, or a custom inference stack, you need to understand:

Throughput: How many requests per second can your system handle?
Latency metrics: Time to First Token (TTFT), Inter-Token Latency (ITL), and end-to-end latency
Token generation speed: Tokens per second under different concurrency levels
Tail latency: P95 and P99 values that affect user experience

In this post, I'll walk through the key metrics for benchmarking LLMs and share why I built llmperf-rs, a Rust-based benchmarking tool that takes a different approach to measuring them.

Why Existing Tools Fell Short

While working with ray-project/llmperf, which is now archived, I noticed it calculates Inter-Token Latency (ITL) by averaging per-request first, then aggregating those averages. That works for many use cases, but I needed to preserve individual latency spikes during testing.

There's also genai-perf, which was very in-depth. My only problem was not being able to run it natively on Ubuntu 22.04 without a Docker container. As of an edit (Apr 2026), they've sunsetted genai-perf in favor of aiperf, which I haven't tried but looks comprehensive. vllm-bench is solid too, but requires installing vllm.

The goal was a simple binary that runs almost anywhere with minimal dependencies. It was also a learning project.

Metrics

Time To First Token (TTFT)

TTFT measures how quickly the model begins responding after receiving your request. For interactive apps, this is the perceived latency before any output appears. It's also important for RAG-based applications where a large chunk of processing happens at the prefill stage.

TTFT = first_token_timestamp - request_start_timestamp

Lower is better.

Inter-Token Latency (ITL)

ITL is the time between consecutive tokens during generation. Spikes can come from multiple issues, most commonly network problems. ITL is usually consistent due to how KV caches and the computation works.

When testing against vLLM, I noticed high ITL spikes happen when you benchmark close to the context limit. I suspect this is due to vLLM evicting requests that exceed the KV cache size. If 3 requests come in with 0.8x context length and 0.2x for generation, but the GPU only has room for 2.8x, one request will be preempted. vllm preemption docs

Aggregation: concatenate ALL ITL values across all responses, then compute statistics. Each response produces N-1 ITL values (where N is the token count). By aggregating raw values instead of per-request averages, you preserve the true distribution including outliers.

Throughput Metrics

Prefill TPS counts tokens processed per second during the prefill phase:

Prefill TPS = input_tokens / TTFT

However, prefill TPS doesn't accurately reflect system performance, because TTFT includes queue wait time, not just actual processing time. Under load, your request might sit in a queue before prefill starts, so a lower prefill TPS often reflects queue contention, not the system's processing capability.

Decode TPS is tokens generated per second during the decode phase:

Decode TPS = output_tokens / (final_time - decode_start_time)

What Matters Most

For production serving, focus on TTFT, ITL stats, and maybe RPM.

TTFT is the perceived responsiveness of your system. ITL statistics reveal decode-phase issues that throughput hides: the 99th percentile and max ITL expose preemption events from KV cache limits and network issues. ITL matters less for batch jobs or non-streaming APIs where users don't watch tokens arrive in real-time.

Token Counting

Accurate metrics need accurate token counts. llmperf-rs handles this two ways:

API response gets priority: most OpenAI-compatible endpoints return token counts in the usage field.
Tokenizer (optional): the default Llama tokenizer ships inside the binary, so no network is needed. For exact input counts, override with a model-specific tokenizer. Note that chat templates can cause <10 token variance.

The original llmperf uses a single tokenizer for all models. Different models use different tokenizers, so llmperf-rs lets you specify the correct one. For example, Llama-2 has a vocab size of 32000, while Qwen3-4B has 151936. In my testing, setting input tokens to 8192 against a Qwen endpoint while using the default llama tokenizer returned values around 7363-7376 tokens. That's a real error if you care about exact numbers.

Validating Your Results

Benchmark runs should ideally end with finish_reason = length, meaning the model hit the max_tokens limit.

An edit (July 2026): with modern models, finish_reason = stop is common even with a high max_tokens, because the model may refuse the request or simply end early. Treat stops as acceptable, but be aware they add variance to output tokens, which then adds noise to RPM and E2E latency. I generally check the ratio of length vs stop rather than rejecting every stop outright.

When to Use llmperf-rs

Use it when: running benchmarks with minimal dependencies (single binary, Docker image), testing OpenAI-compatible endpoints (vLLM, Ollama, local APIs), wanting low overhead (Rust, no Ray/ZMQ), or just needing a quick level-above-curl way to test an endpoint.

Consider alternatives when: you need GPU-level metrics (trtllm-bench or aiperf), vLLM-specific features, extensive reporting dashboards, a Python-first workflow, or distributed testing.

Why ITL Matters Even When Throughput Looks Good

High throughput with bad ITL means tokens arrive in bursts, and chat users notice the choppy streaming. ITL spikes (p99 > 100ms) often indicate preemption or network issues. For non-user-facing cases like agentic coding, throughput may matter more than ITL specifics.

The full version with the detailed metrics documentation, installation steps, and example output is on my blog.