DEV Community: Alex Fank

Why averaging LLM benchmarks gives the wrong leaderboard

Alex Fank — Mon, 05 Oct 2026 15:20:16 +0000

Every few weeks new open-weight model is released with a table of benchmark results, and every few weeks we asked the same practical question: is it better than the one we already run? A single ranked list should answer that. Building one turned out to be harder than we expected, and our first attempt failed in an instructive way. This is the first of three posts about that work.

The problem space

There is no shortage of benchmarks. Epoch AI's capabilities index alone draws on more than 50 distinct benchmarks. The trouble is that no model has been measured on all of them, and numbers that do exist are not always comparable.

Vendors report results with their own prompts, harnesses and reasoning-effort settings. Hugging Face found that different implementations of MMLU "give widely different numbers and even change the ranking order of the models". Independent boards run models themselves, which helps, but every board makes its own choices, and a model appears only on the boards whose maintainers got to it.

Benchmarks also age. The authors of MMLU-Pro note that performance on earlier benchmarks had begun to plateau, making differences between models hard to discern. GSM8K is a familiar example: a freshly written look-alike set, GSM1k, showed accuracy drops of up to 8% for some model families. New, harder benchmarks appear to separate the frontier models, and the older ones stay on the books.

Some boards rotate their questions on purpose. LiveBench replaces about one sixth of its questions in each update, so snapshot from one month is not strictly comparable to one from another. We treat each LiveBench snapshot as a unit and never mix scores across snapshots.

The result is that every model is measured on a different, sparse subset of the available boards. Any composite has to decide what to do about the holes.

What we built first

Our first version, which launched in June 2026, was deliberately plain. We normalized each result with 0 to 1 range between the board's random-guess baseline and its ceiling. Within each category (reasoning, coding, math and so on) we averaged whatever normalized scores a model had. The overall number was then an equal-weight mean of the category sub-scores. Missing entries were never filled with zeros; category a model had no data for was simply absent.

To keep thin evidence out of the ranking we added a coverage gate: a model needed results in at least three categories, and at least one of them had to be reasoning or coding. There was no minimum number of benchmarks.

This looked reasonable. It resembles the way several public leaderboards combine results, it is easy to explain, and no benchmark is privileged. We shipped it.

How it broke

On 2026-09-20 we looked at the open-model ranking after adding several new boards and found something we could not defend. Phi 3.5 MoE Instruct was measured on three benchmarks: PIQA (0.886), GSM8K (0.887) and BoolQ (0.846). Its composite was 83.5. Kimi K3 had results on eighteen benchmarks, including some of the hardest current ones, and its composite was 66.7. The ranking put the three-benchmark model well above the eighteen-benchmark one. Overall, Kimi K3 sat at number 8, behind Llama 2 70B and Phi 3.5 MoE. The top three open entries each had exactly three boards.

The mechanism is straightforward once it is visible. Hard modern boards produce low scores, even for excellent models. A model that has been run on many of them gets lower mean for it. A model that was only ever submitted to saturated classics, where nearly everyone scores near the ceiling, is never exposed to that penalty. The average rewards avoiding difficult tests.

We checked that this was not an artifact of the new boards. Recomputing without the six boards added that day still left the older models on top, with Kimi K3 at number 13.

Two fixes we measured and rejected

The first obvious fix is a minimum benchmark count. We tried it. Models with four to eleven easy boards still beat flagships, because the problem is which boards a model has, not how many.

The second is to standardize each benchmark with a z-score across all models, so an easy board stops being easy. This also failed. A legacy board has a legacy population: a 2023 model compared against other 2023-era models still earns a high z-score. With z-scores over the full field, Kimi K3 only reached number 2, and Falcon 180B and StableBeluga2 were still in the top six.

We kept a floor of four benchmarks anyway, partly because it matched how we already labeled confidence: three-benchmark composites swing wildly.

The fix that worked for v2: an active panel

What the failures had in common was that old boards were still voting. A benchmark carries information about today's models only while today's models are still being submitted to it. So we changed the rule: a benchmark counts toward the composite only if the newest result on it comes from a model released within the last 365 days.

The rule is derived from the data rather than curated by hand. Nobody on our side decides that a board is obsolete; if a board goes quiet upstream, it leaves the panel by itself. With this rule, 27 boards were active, and ten saturated classics dropped out, among them GSM8K, HellaSwag, PIQA, BoolQ and MMLU. The four-benchmark floor stayed.

The chart shows the panel as it stands on 2026-10-05: with more boards imported since, 40 are active, and the same ten classics remain frozen.

There was a cost, and we accepted it deliberately. 60 of 126 previously ranked models lost their overall rank, because they had been scored mostly on the frozen boards and fell below the floor of four active ones. Qwen2.5-7B-Instruct, with 9.7 million downloads, was among them. We chose not to hide this: a model's benchmarks page now says plainly that it no longer qualifies, instead of silently dropping the section. An absent score means there is not enough current evidence, not that the model is weak.

After the change Kimi K3 ranked number 1 of 67.

Why v2 was still not enough

The active panel fixed the visible absurdity, but it is a patch on the same estimator, and the weakness remained. Averaging still penalizes models that were submitted to hard boards. We saw this directly later on: when we considered folding in two new Epoch boards, our check showed that doing so would move 107 standings, because a drop-and-average penalizes the frontier models that a hard, narrowly covered board was run on. We held those boards out at weight 0, which treats the symptom.

It also cannot compare models whose benchmark sets do not overlap. If one model was tested on boards A and B, and another on boards C and D, an average of raw scores says nothing about which is stronger. The two sets of boards may differ in difficulty, and nothing in the average tells us by how much.

That is not a new problem. It is the situation psychometrics has dealt with for decades: people sit different exams, and the exams have to be placed on one scale, with the difficulty of each test estimated from the responses rather than assumed. The tool for that is item response theory. In the second part we will describe how we applied it to benchmark results, and what it changed in the rankings.

The live leaderboard

The current leaderboard, and the written methodology behind it, are here: llmrun Score leaderboard. The full method is documented on the methodology page.

Estimating tokens/s for Mixture-of-Experts models: active parameters, plus a routing term

Alex Fank — Mon, 05 Oct 2026 14:33:15 +0000

llmrun.dev estimates whether a given LLM can be run locally and how fast it will decode. The first part of that question is arithmetic. The second required several iterations, and Mixture-of-Experts models were the case where the initial formula was least accurate. This note describes the estimator in its current form, the calibration behind it, and the cases in which it is known to fail.

Decode is a bandwidth problem

During generation, each output token requires reading the weights from memory once. Batch size is 1 on a local machine, so there is almost no arithmetic over which to amortize that read, and speed is set by how fast bytes stream out of VRAM or unified memory. The dense formula used is the standard one:

tok/s = (bandwidth GB/s / model size GB) x efficiency

Efficiency is a per-platform constant covering software overhead and the gap between spec-sheet and achievable bandwidth. In the code it is 0.65 for NVIDIA (CUDA), 0.60 for AMD (ROCm 7.x), 0.50 for Intel (oneAPI) and 0.70 for Apple (Metal). These values were taken from community llama.cpp and Ollama results and are treated as calibration constants, not physical quantities. The basis of the estimate is dated in the code (last reviewed 2026-09-21, against llama.cpp / Ollama with GGUF weights), and the methodology page prints that date.

Why dense math fails for MoE

A Mixture-of-Experts model holds all its experts in memory but routes each token through only a few. VRAM therefore scales with total parameters, while bytes read per token scale with active parameters. gpt-oss-20b has 20.9B total parameters and 3.6B active. Applying the dense formula to its 12.9 GB footprint on an RTX 4090 (1008 GB/s, from the spec sheet) gives about 51 tok/s, whereas llama.cpp's llama-bench measures about 222 on that card. The dense formula is off by more than 4x, in the pessimistic direction.

The first correction is to scale the bytes read by the active fraction: the in-memory model size is multiplied by active/total parameters. This is an approximation, since attention and shared weights are not routed and the active count already includes them, but it allows a single ratio taken from the model's config to be used instead of modelling tensors separately. A model takes this path only if active parameters are below 90% of total. Anything denser is treated as dense, in which case the function returns exactly the dense result.

The routing term

Scaling bytes alone overshoots. When only 3.6B parameters are read per token, the bandwidth term becomes very small, and a fixed per-layer cost that the formula ignored begins to dominate: evaluating the router and dispatching to expert kernels, once per layer, per token. This cost is modelled as an additive time per layer:

seconds/token = active_GB / (bandwidth x efficiency)
              + layers x per_layer_overhead_ms / 1000

The per-layer overhead constants are 0.055 ms for NVIDIA and 0.2 ms for Apple, AMD and Intel, with 0.1 ms when the brand is unknown. The NVIDIA value is much lower because CUDA graphs amortize launch cost better than the other llama.cpp backends do. If the layer count is missing, a fallback of 48 is used, the median across MoE models in the catalogue at the time of calibration. The constants were fitted against llama-bench tg128 (decode) results for gpt-oss-20b and gpt-oss-120b on 8 hardware combinations. They are kept in the test suite as fixtures, so a change to any constant that breaks one of them fails the build.

Two worked examples follow, computed from the constants above. The bandwidth values are the GPU/SoC spec values used in the fixtures. The measured figures are the llama-bench results in the fixtures.

	RTX 4090, gpt-oss-20b	M2 Ultra, gpt-oss-120b
Model in memory	12.9 GB	65 GB
Active / total params	3.6B / 20.9B	5.1B / 116.8B
Bytes read per token	2.22 GB	2.84 GB
Bandwidth (spec) x efficiency	1008 x 0.65	800 x 0.70
Bandwidth term	3.39 ms	5.07 ms
Layers x overhead	24 x 0.055 = 1.32 ms	36 x 0.2 = 7.20 ms
Estimate	212 tok/s	81.5 tok/s
Measured (llama-bench)	222	80

The 4090 case is dominated by bandwidth. In the M2 Ultra case, routing overhead accounts for more than half of the per-token time.

Accuracy

The test suite asserts that every fixture lies within 18% of its measured value. The public page states an expected error of roughly plus or minus 20% on decode tok/s, and quoting a tighter figure is not justified. Eight fixtures on two model families constitute a small sample. The constants fit gpt-oss well. They have not been validated on every MoE shape (many fine-grained experts, large shared experts), and misses of more than 20% are expected for some of them.

Where it breaks

CPU and RAM offload is the largest gap. Once part of the model resides in system RAM, decode speed is pulled toward the slowest tier from which weights are read, and a single bandwidth number is no longer meaningful. The methodology page lists offload as not modelled. Fit still indicates whether the model loads, but the tok/s figure assumes that everything is in fast memory.

Prefill is a different regime. Reading the prompt is compute-bound, so it is estimated separately: prefill tok/s = dense FP16 TFLOPS x 1e12 x MFU / (2 x active parameters), with MFU of 0.45 for NVIDIA, 0.22 for AMD, 0.18 for Intel and 0.20 for Apple, calibrated on llama.cpp pp512 runs with Q4_K_M weights. Time to first token is the number of prompt tokens divided by that rate. MoE benefits prefill as well, since FLOPs scale with active parameters, but the error bar is wider, about 30%, and wider still where no verified compute figure is available. A fast-decoding card can therefore still impose a long wait before the first token.

Context length is a further, less visible source of error. The decode formula accounts for the weights and nothing else, so it ignores the growing KV cache that attention also reads on every token. The cache itself is straightforward to size: 2 (K and V) x KV heads x head dim x layers x 2 bytes (FP16) per token. Some architectures keep a full cache on only some layers (linear attention, sliding window), and the VRAM code accounts for that. The speed estimate, however, does not decrease with context, so long-context numbers are optimistic. Quantized KV cache is also not modelled.

Quantization carries its own overhead. Model size is derived from effective bits per weight, so Q4_K_M files are not exactly 4 bits per weight, and measured sizes from Ollama or llama.cpp are preferred over the computed estimate whenever they exist. Batching, speculative decoding, multi-GPU interconnects and flash-attention variants are likewise outside the model.

Fit, briefly

The fit side is simpler. VRAM is weights (parameters x bits per weight / 8) plus KV cache plus about 0.3 GB of framework overhead. When architecture data is unavailable, the fallback is a flat 10% overhead. "Fits" is judged with room reserved for a 16K-token context, because a model that loads with a 2K cache is not usable for a real conversation.

The arithmetic for an individual model is printed on its page, for example gpt-oss-20b. Measurements from llama-bench that disagree with an estimate by more than the stated error are welcome, since this is how the constants are corrected.