I had eight models sitting on my laptop and no idea which one I should actually be using.
I could get tokens per second out of llama-bench. That told me llama3.2 was fast. It told me nothing about whether llama3.2 was good — whether the quantization I'd pulled had quietly broken something, whether the 8B model I'd been avoiding because it felt sluggish was worth the wait.
So I built the thing that measures both. It's called homebench, and most of the month I spent on it went not into the code but into deciding what the numbers were allowed to mean. Here's what turned out to be hard.
Tokens per second has a denominator problem
"Output tokens divided by time" sounds like it settles the question. It doesn't, because which time?
A generation request involves loading the model (sometimes), processing the prompt, then generating tokens. Include model load and a cold first run looks catastrophic. Include prompt processing and your number moves depending on how long your prompt was, which means it's partly measuring your benchmark rather than the model.
I settled on: output tokens divided by generation time, excluding both prompt processing and model load. That's the number that answers "how fast will text appear once it starts appearing," which is the thing you actually feel when using a model.
But there's a second split underneath. Ollama reports server-side eval timing directly, it tells you how long generation took, measured inside the server. Other backends don't, so you time the token stream client-side and eat whatever HTTP and serialization overhead sits between you and the model.
Those two paths are not strictly comparable. A number measured inside the server will look slightly better than the same model measured across a socket. I could have hidden that. Instead it's in the README, because a benchmark that compares two different measurements is worse than no benchmark.
Memory has no single honest answer
I wanted one number: how much RAM does this model cost me?
There isn't one. What exists is:
- The resident size of the model weights, which some runners will tell you (Ollama exposes it via
/api/ps, LM Studio via/api/v0) and others won't. - The actual resident set size of the backend process, which includes weights plus KV cache plus whatever else the server is doing, and which you have to sample from outside.
- The theoretical size from the model card, which is neither of the above and is usually wrong in practice.
I report the first where the runner exposes it, and fall back to a best-effort peak-RSS sample of the backend's processes otherwise. Both are labelled as what they are.
The temptation was to blend them into one confident-looking figure. Resisting that is, I think, the single most important design decision in the whole tool. A number that looks precise and isn't is worse than a number that admits its own fuzziness.
Single-stream throughput lies on batching servers
The leaderboard measures one request at a time. That's the right default, it's what you experience when you're the only person using your model.
But servers that batch requests behave completely differently under concurrency. A vLLM instance handling eight simultaneous requests does far more total work than eight times its single-stream rate would suggest, because batching amortizes the expensive parts across requests.
So there's a separate throughput sweep that fires N requests at each concurrency level and reports aggregate tokens per second, speedup versus concurrency 1, and p95 latency. On a batching server you see aggregate throughput climb sublinearly while per-request latency degrades, the classic tradeoff, made visible.
The result I like most is the negative one. On a non-batching setup, aggregate throughput stays flat while latency climbs linearly. Your server is just queueing. That's genuinely useful to know and you can't see it from a single-stream number.
Quality grading has to be boring to be worth anything
The obvious approach to measuring quality is to ask a bigger model to judge the outputs. It's also the approach that makes your benchmark unreproducible, since now your scores depend on a second model's mood.
So the default suite is 31 tasks that are graded deterministically: exact numeric match, multiple-choice letter, valid JSON with required keys, regex. Temperature 0, fixed seed. Run it twice on the same model and you get the same number.
This is much less impressive than an LLM judge and much more trustworthy. There is an optional --judge flag for open-ended tasks: summaries, explanations, anything where "correct" isn't a string match, but it's marked in the docs as a signal rather than a score, because that's what it is.
And here's the caveat I put directly in the README rather than burying: 31 tasks is a smoke test, not a leaderboard. It will catch "this quantization broke the model." It will not reliably separate two models that are three points apart, and I make no claims about contamination. For real evaluation rigor, use lm-evaluation-harness. For "did I break something," this is the right size.
You can point it at your own tasks with a YAML file, which is what I'd suggest for anything you'd actually make a decision on.
What it ended up as
One command that finds the models already installed in your runner, measures speed and memory, runs the quality suite, and renders a live terminal leaderboard.
pip install homebench
homebench
Works with Ollama, LM Studio, llama.cpp, vLLM, or any OpenAI-compatible server. Saves every run, so homebench diff answers "did that quantization actually help?" There's also a fit command that reads your hardware and tells you which models you could run before you spend an hour downloading one that OOMs.
MIT: github.com/david-g-3654/homebench
If you run models locally, I'd genuinely like to know whether the quality suite tests the right things. I picked 31 tasks and I'm not confident I picked well. Issues and PRs welcome.
Top comments (0)