Why Local LLM Benchmarks Need Power Data

Tokens per second is useful, but it is not enough to choose a local model setting. Power draw tells you what the faster setting costs while it runs. On my RTX 5090, two Gemma 26B settings both passed the same fixed tasks, yet the faster one used much more power for a small speed gain.

What did the RTX 5090 run measure?

The Aug. 1, 2026 report records a workshop mix of three 200-token generation tasks and four code tasks. The model was gemma4:26b in Ollama with a 2,048-token context. The fixed-task quality result was 100% at both num_batch=256 and num_batch=512.

The report also records the machine state: an NVIDIA RTX 5090 with 32,607 MiB total memory, 73 C at the snapshot, and 401.68 W drawn against a 575 W limit. Those fields make the result easier to compare with a later run on the same machine or a different card.

What changed when I raised num_batch?

At num_batch=256, Gemma produced 215.7 tok/s and averaged 187.6 W. At num_batch=512, it produced 218.5 tok/s and averaged 234.3 W. The second setting added 46.7 W for 2.8 tok/s.

That is a poor trade if both settings meet the same quality bar and the workload is not waiting on prompt processing. The higher setting may still win for a batch job where wall-clock time matters more than power. The point is that the benchmark should expose the choice instead of hiding it behind one speed number.

Why does speed alone give the wrong answer?

A speed-only table encourages a single ranking. The fastest row looks like the winner. That ranking changes when you add power, quality, context length, and memory use.

The same report shows this in a smaller model. The llama3.1:8b agent-code task ran at 227.79 tok/s with a 7,826 MiB VRAM peak. Gemma 26B ran the same named workload at 207.41 tok/s with a 20,233 MiB VRAM peak. These are not interchangeable rows. They describe different memory and quality choices.

I keep the measurement fields together because local model selection is a fit problem first. My earlier notes on VRAM fit and runtime support cover the load gate. The 5090 phase timing receipt covers why one end-to-end latency number can hide the slow phase.

What fields should a local LLM benchmark include?

Record these fields for every meaningful run:

1. Model and quantization.
2. Runtime and version.
3. Context length, prompt size, and output limit.
4. Task name and fixed quality result.
5. Tokens per second.
6. Average and peak power when available.
7. VRAM peak and the GPU memory ceiling.
8. A timestamp and the exact machine.

Do not compare rows that change several of these fields at once. If the model, runtime, prompt, and task all change, the speed difference cannot tell you which change mattered.

What should I change in my own benchmark loop?

Add power and quality as first-class columns. Keep the prompt and task set fixed. Run the same setting more than once when the result will drive a production choice. Store the raw output with the summary so a surprising row can be checked later.

The useful question is not “Which setting is fastest?” It is “Which setting meets the quality bar at an acceptable power and memory cost?” That answer belongs in the report beside tok/s.

Accompanying prompt

What the prompt does: It turns a local model run into a comparable benchmark row with speed, quality, power, and memory fields.

Copy/paste this prompt:

Role:
You are a local LLM benchmark reviewer.

Context:
Model: [model and quantization]
Runtime: [Ollama, llama.cpp, or other]
GPU: [GPU name and memory]
Run log: [paste the log or report path]

Task:
1. Extract the task, context length, prompt size, and output limit.
2. Extract tokens/sec, average and peak power, VRAM peak, and quality result.
3. Compare this row with the supplied baseline without inventing missing fields.

Output:
- One markdown benchmark row.
- A short pass/fail quality note.
- One sentence on the speed, power, and memory tradeoff.

Constraints:
- Keep it short.
- Use exact numbers and file paths when available.
- Do not invent missing measurements.

Copy the block above.

Get the artifact-backed local AI lab notes by email: https://bmdpat.com/5090-reports

---

Get the local AI lab notes (benchmark rows, VRAM fit, quant choices, what runs on consumer GPUs), M-F only when there is something worth sending: https://bmdpat.com/newsletter?utm_source=blog_md&utm_medium=aeo&utm_campaign=local-llm-benchmark-power-data-2026

---

Originally published on bmdpat.com. I run a one-person AI agent company and write about what actually works.

Want these in your inbox? Subscribe to the newsletter - no spam, unsubscribe anytime.