Eight green rows and an empty string

I wanted a narrow answer this morning: does the 2B model I keep on this laptop return a function call when I ask for one, or does it wander off mid-sentence?

I got a report with eight green rows, zero errors, and eight empty response strings. I noticed the first two facts and missed the third for a while.

The laptop

6 GB RTX 3060, laptop version. Ollama loads qwen3.5:2b at 5.0 GB of VRAM with a 131072-token context window, and reports the split as 25 percent CPU, 75 percent GPU. It does not fit, so it spills, and that is a lot of cache to be holding next to weights that already overflow.

Your machine will differ in every one of those numbers, which is the whole reason for measuring at home.

The run

The tool is LLMeter, one Rust binary that benchmarks whatever OpenAI-compatible endpoint you point it at. Standard suite, temperature 0.0, sixteen tokens per answer:

llmeter --provider ollama bench run --suite llm \
  --models qwen3.5:2b --benchmarks all \
  --runs 1 --max-tokens 16 --temperature 0.0 \
  --export both --report both

Benchmark	Prompt	Wall time	Output tok/s	Valid output
chat generation	short	103,621 ms	0.15	n/a
responses API	responses	834 ms	19.2	n/a
consistency	x2	844 ms	38.5	similarity 1.0000
prompt sizes	short (22 tok in)	389 ms	41.1	n/a
prompt sizes	medium (49 tok in)	723 ms	22.1	n/a
prompt sizes	long (124 tok in)	2,144 ms	7.5	n/a
structured JSON	schema required	812 ms	19.7	0% schema valid
tool call	function required	522 ms	30.7	0% tool call valid

The first row: 103.6 seconds for sixteen tokens. Everything after it ran between 389 ms and 2.1 s. Short prompts pushed 41 tokens per second, the long one managed 7.5 because the prefill had 124 input tokens to chew before generation could start.

Two of the eight requests came back flagged: structured JSON invalid, tool call invalid. At sixteen output tokens this model had no room to do either.

Then I stopped believing the first row

My first reading was that 103 seconds told me something about the model. To check that, I unloaded the weights and measured the load itself.

I unloaded the weights (ollama stop qwen3.5:2b, then ollama ps to confirm nothing was resident) and ran the same single request again: twelve point six seconds, then eight point seven, then nine point seven.

Same card, same model, same command, twenty minutes apart. The first measurement of the session was twelve times slower than the fastest reload, and nothing about the model changed in between. The first load pulled 2.7 GB of weights off a cold disk. By the third run the operating system had those files in the page cache and the card had less work to do.

So the biggest number in that chart mostly measures how fast this disk handed those files over. The model's own contribution to it was small.

The empty string

I turned on response previews to see what the model actually said, and the preview column came back empty. I re-ran the same request with the token budget raised to 32, 64 and 200: still empty, at every budget. The provider returned usage numbers the whole time, 22 tokens in and 16 out, and the CLI computed throughput from them correctly, 19.7 tokens per second.

Then the consistency row, the one with 1.0000 next to it. Two identical prompts, both answered with an empty string, compared pairwise and scored as a perfect match. That row reads like the healthiest number in the report, and it is the emptiest.

Nothing failed. Eight records, zero errors, successful records: eight. Had I stopped at the executive summary I would have written down that this model produces about twenty tokens per second and I would have been describing the speed of an empty response.

I still do not know why the content field was empty. Something about how this model's output arrives through the OpenAI-compatible endpoint, or the token budget. I did not chase it down. The part that mattered was already in the report, in a column I could read.

What the tool got right

The capability flags stayed visible. schema_valid: false, tool_call_valid: false, json_parse_success: false, each in its own column instead of folded into a pass rate. That is the difference between a report I can act on and a green tick I have to trust.

It attaches to the endpoint and measures what comes back. It does not start your servers or load your models. That sounds like a missing feature until you have had a load generator restart the thing it was supposed to be measuring, mid-run, and then report the result as a number.

Twelve provider presets, three of them first-class (ollama, lmstudio, llama-cpp), then vLLM, SGLang, LocalAI, LiteLLM, TGI, text-generation-webui, Jan, MLX-LM and a generic OpenAI-compatible target. Subcommands are scriptable and the exit codes are meaningful, so this can run in CI rather than in a screenshot.

Two smaller decisions that mattered more than they sound. Results are written as schema 3.0 with a mandatory run kind, and a result file without that version is refused on load rather than silently upgraded. That refusal is what keeps a series of results a series. Previews are off by default and credential-shaped values are redacted before anything reaches disk, so turning previews on was a deliberate act on my part rather than something I found later in a JSON file.

The export behind the chart near the top is in a scratch folder on my laptop. The repository ships the code and the manual, not the artifacts, and its QA directory is gitignored on purpose. The run ID and the command are in this post, so you can repeat the measurement on hardware you own, which is the only comparison worth making.

What I do differently now

I read the content field before any average. OK in the status column means the request completed, not that the answer was usable, and this run is the cleanest example of that gap I have produced.

The capability flags are the result, the tokens-per-second column is the footnote. My question was answered by two zeros, not by 41 tokens per second.

One run is a sample of one. The same request on the same laptop covered a twelvefold range inside twenty minutes, so three runs with the model unloaded in between, median reported, is the minimum I would put in front of another person.

And I stopped comparing across machines. The 25/75 CPU-GPU split is the number that explains everything above, and it exists only on this machine.

The model still is not good enough for the agent I had in mind. I found that out in one command, and I have a report I can keep and diff against the next run instead of a leaderboard entry for somebody else's hardware.

---

LLMeter is MIT licensed: github.com/CTCycle/LLMeter-local-benchmarks. Release archives cover Windows, Linux and both macOS architectures with checksums. There is no crates.io package yet, so the archives or a checkout are the two routes.