LLMeter: Measuring the Model You Actually Run

A local-first benchmark CLI, and why half of a real result is more useful than all of a borrowed one.

---

The number you cannot look up

Model leaderboards are measurements of somebody else's machine. They are useful for comparing architectures and useless for answering the only question that matters when you run models yourself: what does this thing do on my hardware, with this quantization, behind this provider.

That gap is wider than it sounds. A 2B model in an Ollama install, an LM Studio install, and a bare llama-server are three different systems with three different latency profiles. The first request of a session usually pays for loading weights into VRAM, and on a 6 GB laptop GPU that payment is not small. Quantization changes both speed and, sometimes, whether structured output works at all. None of that shows up in a leaderboard, because none of it is the model. It is the rig around the model.

LLMeter exists to measure the rig.

What it is, and what it refuses to do

LLMeter is a single Rust binary. You point it at an OpenAI-compatible /v1 endpoint and it measures what comes back. It ships with presets for twelve providers, from first-class targets like ollama, lmstudio, and llama-cpp through to vllm, sglang, localai, litellm, tgi, jan, and mlx-lm, with a custom preset for anything else that speaks the API.

The boundary is deliberate and stated up front in the manual: LLMeter does not start provider servers, install models, or manage provider processes. You start the provider. LLMeter attaches to it, checks reachability, and measures. That decision removes a whole class of failure where a benchmark tool quietly restarts your server mid-run, or where a result depends on which process supervisor happened to be running.

There are two ways in. One is an interactive terminal menu for provider setup, model inventory, and benchmark workspaces. The other is a scriptable command set (status, models, bench, report, quality) meant for automation and CI, with exit codes that mean something: status returns 0 when the provider is reachable and 1 when it is not, and a non-interactive invocation without a subcommand prints help and exits 2 rather than guessing.

What gets measured

The standard llm suite covers six benchmarks: chat generation, the /v1/responses path, response consistency across repeated identical prompts, timing across short, medium, and long prompts, JSON-schema structured output, and tool calling. A separate embeddings suite covers /v1/embeddings. When a provider does not support something, that call is recorded as a per-record error so the rest of the run continues.

Above that sits a performance surface aimed at people tuning a deployment rather than testing a model: profiles called smoke, latency, throughput, and sweep. sweep builds a matrix across prompt tokens, output tokens, and concurrency. The important detail is that you can print the plan before anything is sent:

llmeter bench perf --models llama3.1 --profile sweep \
  --prompt-tokens 128,512 --output-tokens 64,128 --concurrency 1,2 --dry-run

That prints the selected models, the matrix dimensions, warmup and measured request counts, and the total request estimate. The default ceiling is 500 requests, and crossing it requires an explicit --allow-large-matrix. For anyone who has ever pointed a load generator at their own machine and walked away, this is a sensible guard rail.

The quality command deserves a note because it is deliberately unfinished. It prints dry-run adapter commands for external evaluators (lighteval, inspect-ai, lm-eval-harness, SWE-bench) and stops there. It does not install or execute them. It is a planning surface, and the documentation says so instead of implying integration that does not exist.

A real run, including the parts that failed

I ran the standard suite against qwen3.5:2b through Ollama on a laptop with a 6 GB RTX 3060, temperature 0.0, max_tokens 16, one run per benchmark. The whole run is committed to the repository as JSON, CSV, Markdown, and HTML, which is why I can quote it precisely.

The first number is the one worth looking at. Chat generation took 64.28 seconds to produce 16 tokens, an effective 0.25 tokens per second. That is not the model being slow. That is the first request of the session paying to load weights into VRAM on a 6 GB card. Every request after it ran between 328 ms and 730 ms, with throughput between 21.9 and 48.8 tokens per second depending on prompt size. Consistency across repeated identical prompts was a pairwise similarity of 1.00, which is what greedy decoding should produce.

Then two benchmarks failed:

• structured-output recorded schema_valid: false
• tool-calling recorded tool_call_valid: false

A 2B model failing to hold a JSON schema and failing to emit a well-formed function call is not a surprise. What matters is that the report says so. The run finished with eight records, zero errors, and a green executive summary, and the two capability failures are still visible in the table rather than buried under a pass count. If you are choosing a model to sit behind an agent that depends on tool calls, this is the single most useful line in the output, and most benchmark tooling does not give it to you.

The honest caveat is scale. This was one run per benchmark. Percentiles are computed with a nearest-rank estimator, which means P95 stays unpublished below 20 samples and P99 below 100. A smoke run is exploratory, not conclusive. If you want to compare two quantizations properly, you need repeated runs on the same machine, ideally on the same thermal footing.

Semantics you are allowed to argue with

Most benchmark tools hand you a number and let you assume it means what you hope. LLMeter documents where its numbers are proxies, and the list is worth reading:

• TTFT is the client-observed time to the first non-empty streamed chunk. It is not a tokenizer-confirmed token, and it includes provider scheduling and prompt processing.
• Inter-chunk latency is measured separately and must not be read as inter-token latency. ITL is only reported when streaming timing exists and the provider reports output usage for at least two tokens.
• estimated_load_overhead_ms is a first-request versus warm-request estimate. It does not measure provider restart, cache eviction, or model loading from the provider's own point of view.
• Token throughput uses scenario wall-clock time; per-request timing stays latency-only.

This is the part I find most convincing about the tool. A measurement surfaces its own error bars and says which of its four numbers is an estimate rather than a measurement. It makes the results harder to screenshot into a marketing claim, which is probably the point.

Reports worth keeping

Each run produces a JSON record, a flattened CSV, a Markdown summary, and a self-contained HTML report. The JSON is the canonical artifact and it is versioned: schema 3.0 is enforced by identity, and a result file from an older schema is rejected rather than silently upgraded. That is the right call for anything you intend to compare across months, because a silent upgrade is how a benchmark series quietly stops being a series.

The defaults lean toward not leaking things. Response previews are omitted unless you pass --include-response-preview, and credential-shaped values are redacted before anything is written to disk. Token-count settings survive, since they are the point of the exercise, but a bearer token that a provider echoed back does not.

Installing it

Windows x86-64 is the primary target, shipped as a .zip. Linux x86-64 arrives as a .tar.gz and needs a compatible glibc runtime, since it is not a fully static build. macOS gets both Intel and Apple silicon archives. Each release publishes a SHA256SUMS file plus GitHub artifact attestations, so gh attestation verify works on the archives and the checksum file.

cargo install llmeter --locked

The README documents that line as the conventional path, and the crate metadata is complete, but the crate is not actually on crates.io yet. The name is unclaimed and cargo publish --dry-run passed on the release commit, so this is a pending manual step rather than an unsolved problem. Until then, cargo install --path . --locked from a checkout, or the release archives, are the working routes.

Reproducing this

Against a provider already running on http://localhost:11434/v1:

llmeter --provider ollama status
llmeter --provider ollama models
llmeter --provider ollama bench run --suite llm --models all \
  --benchmarks all --runs 1 --max-tokens 16 --temperature 0.0 \
  --export both --report both
llmeter report list

The raw artifacts behind every figure in this article are in the repository at assets/QA/runtime-ollama/benchmark_results/, dated 2026-09-10, alongside the release validation record for 0.4.0.

Who it is for

If you run models locally and you have ever wondered whether the quantization you switched to is actually faster, or whether the small model you picked can hold a JSON schema, LLMeter answers that in one command and writes it down in a format you can diff later. It is also useful in the less pleasant direction: it will tell you that a model you already committed to cannot do tool calling, and it will do so without softening the result.

It is not a leaderboard tool, and it will not manage your inference stack for you. It measures a provider you already trust to run, which is a smaller job than it sounds and a more useful one than it looks.

---

LLMeter is MIT licensed. Source, releases, and the run artifacts quoted above are at github.com/CTCycle/LLMeter-local-benchmarks.