DEV Community: Alex Fank

Averages, Elo, IRT: how composite LLM indices differ

Alex Fank — Thu, 22 Oct 2026 06:45:14 +0000

Part 1 demonstrated that simply averaging benchmarks can push a model tested on just three easy boards higher than flagships evaluated across eighteen. Part 2 swapped this average for an IRT model instead. This final section takes a wider view. Many groups release one score to answer "how good is this model," yet they arrive there through very different methods.

An easy way to see that the approach matters: consider the 137 models with both an LMArena rating and an llmrun Composite Score. Across all of them, the two rankings align closely, showing a rank correlation of 0.91. Among the 20 strongest models, however, they disagree completely: -0.02.

Neither figure is flawed. They address different questions, and the gap appears exactly where people examine it most closely.

Three ways to get one number

Average a fixed set of evals. The publisher runs every model on identical tests and calculates a weighted mean. The Artificial Analysis Intelligence Index uses 10 evaluations split into four weighted categories. The now-retired Hugging Face Open LLM Leaderboard relied on six benchmarks, removing the random-guess baseline from each before scoring. This approach stays clean and controlled, yet it covers only what the publisher can afford to run; plus, swapping out for a new set of evals creates a fresh version that no longer compares fairly with previous results.

Let people vote. LMArena presents users with two anonymous answers and fits a Bradley-Terry model to the votes, which is essentially an Elo system. No benchmark grid is required; only head-to-head battles suffice. The catch? It measures which answer audiences prefer, not what a model can actually do. Length emerged as the strongest style effect. LMArena's style control adjusts for length and markdown, yet a vote remains just a vote. An independent audit, The Leaderboard Illusion, also revealed that closed models get sampled into more battles than open ones.

Infer a hidden ability. Item response theory treats each benchmark result as evidence about a model's ability, assigning its own difficulty to every test. The Epoch Capabilities Index applies this method across more than 50 benchmarks, just like our score over 40. A major benefit is handling sparse data: two models that never shared a benchmark can still be compared via intermediate ones that overlap both. However, an estimate is only as strong as the boards supporting it, and refitting may shift old values unless the scale remains frozen.

At a glance

Artificial Analysis Intelligence Index. This is a weighted average of 10 evals it runs itself, lacking any interval for the index. It works best when comparing current models within a single harness.
LMArena. Uses Bradley-Terry to analyze human votes, complete with confidence intervals. Ideal for comparing chat preferences.
Epoch Capabilities Index. IRT across more than 50 benchmarks, using a 90% bootstrap interval. Ideal for tracking progress along the frontier over time.
llmrun Composite Score. It applies IRT across 40 boards using a fixed scale and sets a 90% spread for each model. This is ideal for open-weight models; check it alongside your hardware compatibility.
LiveBench. It runs its own evaluations using rotated questions and includes 95% bootstrap intervals. This suite works best when you need to minimize data contamination.

Methods as each index documents them, checked on 2026-10-22.

Reading one score without fooling yourself

Here are a few rules of thumb, using figures from our leaderboard dated 2026-10-22.

If two ranges overlap, call it a tie. MiMo V2.6 Pro RL scores 133.4 while Kimi K3 comes in at 132.8. MiMo operates on a 90% range from 130.6 to 136.2, based on 8 boards; Kimi's sits between 131.2 and 134.4 based on 30. Given the current data, they appear evenly matched, though Kimi's figure feels more reliable.
If you care about one skill, ignore the overall rank. DeepSeek V4.1 Flash ranks fifth among open models with a score of 129.5; however, its coding ability sits at 122.2, roughly 12 points behind Kimi K3's 134.4. The model shines in reasoning, scoring 134.0.
If it needs to run on your machine, check memory first. Kimi K3 requires around 1.7 TB of memory even at 4-bit precision. None of the top open models fit on a single 24 GB card; the 24 GB page ranks the ones that do.

Which one to read

Choosing a model for local use. Begin with per-skill scores for open-weight options, then narrow down based on what your GPU can handle. That's exactly where our scoring system applies.
Following the frontier. ECI or the Artificial Analysis index.
How a chat model feels. LMArena, preferably with style controls engaged.
Worried the model saw the test? LiveBench swaps out roughly one-sixth of its questions with every update.
Any of them. Examine the interval and determine what it actually measures. When a narrow range comes from multiple runs, it reveals little about how those specific benchmarks were chosen.

Where ours fits

ECI counts as its nearest relative, with distinctions lying mainly in scope:

We don't run any models; everything stems from published benchmarks like Epoch's hub, LiveBench, the SWE-bench leaderboards, and MMLU-Pro.
Open-weight models are the focus here, with a look at how each one matches your hardware.
The scale remains locked to a 2026-Q4 reference, meaning scores update solely when fresh results for a given model come in.
A board stays active only as long as it has results from the past year. This solves the issue mentioned in Part 1.

Known gaps: most benchmarks focus on English, with no vision or multilingual domain yet; we rely entirely on what upstream sources publish. You can find the full method and changelog on the methodology page, while the current ranking is on the leaderboard.

Fitting one score across 40 sparse benchmarks with item response theory

Alex Fank — Fri, 09 Oct 2026 12:39:19 +0000

Part 1 described how our averaged composite failed: a model measured on three saturated legacy boards outranked flagships measured across eighteen, because hard boards lower a mean and easy boards raise it. Minimum counts and per-board z-scores did not fix it, so we changed the model. This part covers the replacement and the decisions behind it.

The idea from psychometrics

Psychometrics has a standard way to compare people who sat different exams. Each person has one latent ability, written θ. Each test item has a difficulty b and a discrimination a, how sharply it separates weaker from stronger test takers. In the two-parameter logistic (2PL) model, the expected result is a sigmoid of the gap between ability and difficulty.

We treat each benchmark as one item. A result is first rescaled to 0–1 between the board's random-guess baseline and its ceiling, and the model is then:

y(i, j) = sigmoid( a_j * (θ_i − b_j) )

Here y is the normalized result of model i on board j, θ_i is the model's ability, b_j is the board's difficulty and a_j its discrimination (we parameterize a_j = exp(α_j) so it stays positive).

With this form a hard board stops penalizing whoever was run on it: it sits higher on the difficulty axis, and a low raw score there is what the model expects. Models that share no board also become comparable. If A and B have no board in common but each shares boards with a third model C, the fitted a and b values tie all three to one axis. Most models in our data appear on a fraction of the boards, so the whole scale rests on this linking; we check below whether it holds.

Epoch AI's Capabilities Index uses the same family of model, performance = σ(α_b [C_m − D_b]), fitted at the benchmark level; its methodology is at https://epoch.ai/data/eci-documentation/methodology. IRT has also been applied to LLM evaluation at the item level, to estimate ability from far fewer questions: tinyBenchmarks (arXiv 2402.14992), metabench (arXiv 2407.12844) and Fluid Benchmarking (arXiv 2509.11106). Like Epoch, we work with per-board aggregates, since leaderboards do not publish per-question responses.

The figure shows four boards with their fitted reference parameters. MMLU-Pro is easy (b = −1.05, a = 1.44) and separates models only at the low end. Humanity's Last Exam is hard (b = 1.87, a = 1.03), and only high-ability models leave the floor. ARC-AGI-2 is the most discriminating board in the reference (a = 3.64, b = 1.05). Its curve is nearly a step. ForecastBench sits at the other extreme (a = 0.31, b = −0.82); its curve is almost flat, and θ depends on it only weakly. A plain average would give all four the same weight.

Decisions

Fitting, and what one board can do

We fit by maximum a posteriori estimation with weakly informative priors: θ ~ N(0, 1), b ~ N(0, 2²) and log a ~ N(0, 0.5²). The last keeps discrimination near 1 unless a board's data say otherwise. The optimizer is full-batch Adam with no random initialization, so the same data always produce the same numbers.

The prototype showed us how little a standard error from one board is worth. Hy4 Preview ranked first among open models on the strength of a single math board, and its standard error came out small. With one observation and fixed item parameters, the Laplace approximation returns a number that is too tight. The prototype report proposed gating on standard error below a threshold; we did not adopt it, since that gate relies on the very quantity that had misled us. We gate on coverage instead: a model needs at least 4 boards, spanning at least 2 categories, including reasoning or coding.

Every published score carries a 90% range from the same approximation, θ ± 1.645 standard errors mapped to the display scale. On the methodology page we give a hypothetical pair, both at 130.0: the well-covered model has a range of 126–134 and the thinly covered one 112–149.

The scale: changed once, then frozen

We first shipped the score as a bounded 0–100 number, the expected result on a typical benchmark. It saturated near 100, and the top models were hard to tell apart. On 2026-10-04, a day after launch, we moved to an open linear scale, 100 + 25θ. A score of 100 is the average model of the 2026-Q4 reference field (θ = 0), 25 points is one standard deviation of ability, and there is no ceiling. Every number changed and the order of models did not.

A free refit re-standardizes θ each time, so one new hard board would shift every model's score. We froze the a and b of 40 boards as a reference named 2026-Q4. Live fits hold those values as anchors and estimate θ, plus parameters for any board not yet in the snapshot, and a model's number moves only when new data about that model arrive. The price is that 100 means the average of a fixed field, so scores compare within one version; a rebase would ship as a new version with an old-to-new table.

Which boards count

Elo-style boards such as LMArena are relative ratings with no baseline or ceiling to normalize against, so they cannot be placed on the 0–1 result axis. They stay on separate leaderboards; Epoch's ECI excludes them on the same grounds. We also drop a board if more than 25% of its normalized results sit exactly at 0 or 1, since that suggests the baseline or ceiling we assumed is wrong.

Some boards that our earlier composite held out at weight 0 are admitted again. The weights existed because a plain average let a hard, narrowly covered board penalize the few frontier models run on it, which the IRT model does not do.

Domain scores for coding, agents, math, science and reasoning use the same global board parameters. Each is the ability estimated from that domain's boards alone, needs at least 2 of them, and is directly comparable with the overall score.

Validation

We hold out results and predict them. In 5-fold cross-validation, the IRT model reaches a held-out RMSE of 0.093 on the 0–1 scale. A baseline with a per-model offset plus a per-board mean reaches 0.177, and the plain board average 0.246.

Coverage improved too: in the prototype, 239 open models were estimable, against 78 that passed the earlier composite's gate. In a leave-one-board-out check, dropping any single large board moved the top 20 open models by at most about 5 places. The largest moves came from one board (5 places) and from ARC-AGI-2 (4).

Some rankings changed a great deal, and we went through the large moves one by one. DeepSeek R1 0528 fell from 5th under the earlier composite to 36th in the prototype. Under the average its rank depended on which boards it happened to be measured on; per-board difficulty places the same results below newer models that did well on those boards. Qwen3 235B A22B moved from 14th to 46th, and Seed OSS 36B, which had only 4 boards, from 7th to 37th. In the other direction MiniMax M2.7 rose from 76th to 29th and Inkling Small from 51st to 9th.

What readers of Part 1 asked

Two readers asked in the comments on Part 1 about things we had assumed and not measured. One asked whether models measured on disjoint benchmark sets end up on one scale at all, whether neighbouring ranks can be told apart, and which date the 365-day rule for active boards uses. The other raised contamination and missing data, which we come back to at the end. We ran the checks on 2026-10-09 with the production fitting code and data; our refit reproduced the live open ranking exactly.

Is it one scale?

We built the bipartite graph of models and boards and counted connected components. There is one, holding all 1,076 entities with data and all 40 boards. No entity sits outside it, gate-passing or otherwise.

Had there been a second component, a free fit would have located it only through the priors. θ ~ N(0, 1) would pull its models toward θ = 0, which displays as 100, the reference average, and that 100 would look like a measurement without being one. In the frozen fit this cannot happen for a board in the 2026-Q4 reference, because its a and b are fixed and a model is placed on the scale through its own boards whatever the graph looks like. The risk returns only for a new non-reference board observed solely by models with no anchored boards; today all 40 boards are anchored.

Can neighbours be told apart?

The published ranges come from a Laplace approximation and ignore which boards a model happened to be run on. To capture that, we resampled boards: 500 replicates, each drawing the 40 boards with replacement, with item parameters frozen at their reference values and each model's θ refit on the drawn boards (a result counts as many times as its board was drawn). The eligible set was held fixed at the 99 open models that pass the gate on full data, and ranks are among those 99.

Kimi K3 is first in 93.8% of replicates and never below second, for a 90% rank interval of 1–2. All 14 adjacent pairs in the open top 15 have overlapping 90% rank intervals. If we ask instead how often the higher-placed model stays ahead, only one pair clears 95%: DeepSeek V4 Pro 0813 over DeepSeek V4 Flash 0731 (#3 over #4), at 97.4%. Kimi K3 over GLM 5.3 comes close at 94.4%, and from #9 down each adjacent order holds in 48–59% of replicates, about a coin flip.

The top three stay within ranks 1–4, #4 to #7 form a second block, and from #8 down the intervals spread over roughly 6 to 16. Width tracks coverage more than score: DeepSeek V4.1 Flash, scored on 10 boards, has an interval of #2–11, and MiMo V2.5 Pro, on 6 boards, #9–23, while GLM 5.2 with 24 boards sits at #5–8. We now read the open leaderboard in these tiers and would not defend a one-place gap below the top.

What date the 365-day rule uses

A board counts toward the score while it is still receiving current results, meaning its newest date is at most 365 days old. That date is the latest of three per-board values: the newest as_of on an open-model result, the newest as_of on an external result, and the newest release date of an open model scored on the board. Our code comments call the release date a fallback, but the code takes the maximum.

That makes the rule looser than it reads. We counted, for each of the 40 active boards, the entities with a result dated on or after 2025-10-10. Four boards have fewer than five: Fiction.LiveBench and SWE-bench Lite one each, Aider Polyglot and MATH Level 5 two each. SWE-bench Lite is the weakest case. Its one recent entry is DeepSeek V3.2, whose result is dated 2025-01-11; the board stays active because that model's catalogue release date is 2025-12-01, and under an as_of-only rule it would already be out. MATH Level 5 ages out on 2026-10-15 unless new results arrive. We treat this as a known weakness and intend to tighten the rule, for example by requiring a minimum number of recent results per board.

Contamination and missing data

The second reader pointed out that which boards a model is run on is not random, and that uneven contamination could distort how discriminating a board appears. Non-random coverage is what broke the average in Part 1; in the IRT model a missing board carries no penalty and only widens the range, and the resampling above shows how far ranks move with the choice of boards. We list contamination as a limitation and have not measured its effect on a. The a values are fitted once on the 2026-Q4 field and then frozen, so they reflect that field's published results as they stand.

Where it stands, 2026-10-09

On the live fit, 40 reference boards and 4,344 observations produce a published overall score for 384 models, 99 of them open. The open top three are Kimi K3 at 133.6 (90% range 131.9–135.4), GLM 5.3 at 129.9 (127.9–131.8) and DeepSeek V4 Pro 0813 at 129.2 (127.4–131.0); given the rank intervals above, the order of the second and third is not settled. The best model overall, GPT 6 Astra at 157.1, is 23.5 points ahead of the best open one, about 0.94 standard deviations of ability. It has results on only 6 boards, and its range of 150.6–163.7 is more than three times as wide as Kimi K3's.

Limitations

Contamination inflates scores whenever test questions reach training data; rotating or private sets reduce it and do not remove it. Some leaderboards accept developer-submitted runs with their own scaffolds, and we use each source's published figure as is. Where a proprietary model has several effort variants, we use the best published one. The inputs are English-centric, and knowledge, long context, instruction following, multilingual ability and vision are not covered yet. The scale is tied to a frozen snapshot of the field. The score also says nothing about whether a model fits your hardware; we estimate that separately.

Part 3 looks at how other composite indices approach the same problem (Epoch's ECI, Artificial Analysis, Arena and others) and what each choice trades away.

The full method is on the methodology page, and current scores are on the leaderboard.

Why averaging LLM benchmarks gives the wrong leaderboard

Alex Fank — Mon, 05 Oct 2026 15:20:16 +0000

Every few weeks new open-weight model is released with a table of benchmark results, and every few weeks we asked the same practical question: is it better than the one we already run? A single ranked list should answer that. Building one turned out to be harder than we expected, and our first attempt failed in an instructive way. This is the first of three posts about that work.

The problem space

There is no shortage of benchmarks. Epoch AI's capabilities index alone draws on more than 50 distinct benchmarks. The trouble is that no model has been measured on all of them, and numbers that do exist are not always comparable.

Vendors report results with their own prompts, harnesses and reasoning-effort settings. Hugging Face found that different implementations of MMLU "give widely different numbers and even change the ranking order of the models". Independent boards run models themselves, which helps, but every board makes its own choices, and a model appears only on the boards whose maintainers got to it.

Benchmarks also age. The authors of MMLU-Pro note that performance on earlier benchmarks had begun to plateau, making differences between models hard to discern. GSM8K is a familiar example: a freshly written look-alike set, GSM1k, showed accuracy drops of up to 8% for some model families. New, harder benchmarks appear to separate the frontier models, and the older ones stay on the books.

Some boards rotate their questions on purpose. LiveBench replaces about one sixth of its questions in each update, so snapshot from one month is not strictly comparable to one from another. We treat each LiveBench snapshot as a unit and never mix scores across snapshots.

The result is that every model is measured on a different, sparse subset of the available boards. Any composite has to decide what to do about the holes.

What we built first

Our first version, which launched in June 2026, was deliberately plain. We normalized each result with 0 to 1 range between the board's random-guess baseline and its ceiling. Within each category (reasoning, coding, math and so on) we averaged whatever normalized scores a model had. The overall number was then an equal-weight mean of the category sub-scores. Missing entries were never filled with zeros; category a model had no data for was simply absent.

To keep thin evidence out of the ranking we added a coverage gate: a model needed results in at least three categories, and at least one of them had to be reasoning or coding. There was no minimum number of benchmarks.

This looked reasonable. It resembles the way several public leaderboards combine results, it is easy to explain, and no benchmark is privileged. We shipped it.

How it broke

On 2026-09-20 we looked at the open-model ranking after adding several new boards and found something we could not defend. Phi 3.5 MoE Instruct was measured on three benchmarks: PIQA (0.886), GSM8K (0.887) and BoolQ (0.846). Its composite was 83.5. Kimi K3 had results on eighteen benchmarks, including some of the hardest current ones, and its composite was 66.7. The ranking put the three-benchmark model well above the eighteen-benchmark one. Overall, Kimi K3 sat at number 8, behind Llama 2 70B and Phi 3.5 MoE. The top three open entries each had exactly three boards.

The mechanism is straightforward once it is visible. Hard modern boards produce low scores, even for excellent models. A model that has been run on many of them gets lower mean for it. A model that was only ever submitted to saturated classics, where nearly everyone scores near the ceiling, is never exposed to that penalty. The average rewards avoiding difficult tests.

We checked that this was not an artifact of the new boards. Recomputing without the six boards added that day still left the older models on top, with Kimi K3 at number 13.

Two fixes we measured and rejected

The first obvious fix is a minimum benchmark count. We tried it. Models with four to eleven easy boards still beat flagships, because the problem is which boards a model has, not how many.

The second is to standardize each benchmark with a z-score across all models, so an easy board stops being easy. This also failed. A legacy board has a legacy population: a 2023 model compared against other 2023-era models still earns a high z-score. With z-scores over the full field, Kimi K3 only reached number 2, and Falcon 180B and StableBeluga2 were still in the top six.

We kept a floor of four benchmarks anyway, partly because it matched how we already labeled confidence: three-benchmark composites swing wildly.

The fix that worked for v2: an active panel

What the failures had in common was that old boards were still voting. A benchmark carries information about today's models only while today's models are still being submitted to it. So we changed the rule: a benchmark counts toward the composite only if the newest result on it comes from a model released within the last 365 days.

The rule is derived from the data rather than curated by hand. Nobody on our side decides that a board is obsolete; if a board goes quiet upstream, it leaves the panel by itself. With this rule, 27 boards were active, and ten saturated classics dropped out, among them GSM8K, HellaSwag, PIQA, BoolQ and MMLU. The four-benchmark floor stayed.

The chart shows the panel as it stands on 2026-10-05: with more boards imported since, 40 are active, and the same ten classics remain frozen.

There was a cost, and we accepted it deliberately. 60 of 126 previously ranked models lost their overall rank, because they had been scored mostly on the frozen boards and fell below the floor of four active ones. Qwen2.5-7B-Instruct, with 9.7 million downloads, was among them. We chose not to hide this: a model's benchmarks page now says plainly that it no longer qualifies, instead of silently dropping the section. An absent score means there is not enough current evidence, not that the model is weak.

After the change Kimi K3 ranked number 1 of 67.

Why v2 was still not enough

The active panel fixed the visible absurdity, but it is a patch on the same estimator, and the weakness remained. Averaging still penalizes models that were submitted to hard boards. We saw this directly later on: when we considered folding in two new Epoch boards, our check showed that doing so would move 107 standings, because a drop-and-average penalizes the frontier models that a hard, narrowly covered board was run on. We held those boards out at weight 0, which treats the symptom.

It also cannot compare models whose benchmark sets do not overlap. If one model was tested on boards A and B, and another on boards C and D, an average of raw scores says nothing about which is stronger. The two sets of boards may differ in difficulty, and nothing in the average tells us by how much.

That is not a new problem. It is the situation psychometrics has dealt with for decades: people sit different exams, and the exams have to be placed on one scale, with the difficulty of each test estimated from the responses rather than assumed. The tool for that is item response theory. In the second part we will describe how we applied it to benchmark results, and what it changed in the rankings.

The live leaderboard

The current leaderboard, and the written methodology behind it, are here: llmrun Score leaderboard. The full method is documented on the methodology page.

Estimating tokens/s for Mixture-of-Experts models: active parameters, plus a routing term

Alex Fank — Mon, 05 Oct 2026 14:33:15 +0000

llmrun.dev estimates whether a given LLM can be run locally and how fast it will decode. The first part of that question is arithmetic. The second required several iterations, and Mixture-of-Experts models were the case where the initial formula was least accurate. This note describes the estimator in its current form, the calibration behind it, and the cases in which it is known to fail.

Decode is a bandwidth problem

During generation, each output token requires reading the weights from memory once. Batch size is 1 on a local machine, so there is almost no arithmetic over which to amortize that read, and speed is set by how fast bytes stream out of VRAM or unified memory. The dense formula used is the standard one:

tok/s = (bandwidth GB/s / model size GB) x efficiency

Efficiency is a per-platform constant covering software overhead and the gap between spec-sheet and achievable bandwidth. In the code it is 0.65 for NVIDIA (CUDA), 0.60 for AMD (ROCm 7.x), 0.50 for Intel (oneAPI) and 0.70 for Apple (Metal). These values were taken from community llama.cpp and Ollama results and are treated as calibration constants, not physical quantities. The basis of the estimate is dated in the code (last reviewed 2026-09-21, against llama.cpp / Ollama with GGUF weights), and the methodology page prints that date.

Why dense math fails for MoE

A Mixture-of-Experts model holds all its experts in memory but routes each token through only a few. VRAM therefore scales with total parameters, while bytes read per token scale with active parameters. gpt-oss-20b has 20.9B total parameters and 3.6B active. Applying the dense formula to its 12.9 GB footprint on an RTX 4090 (1008 GB/s, from the spec sheet) gives about 51 tok/s, whereas llama.cpp's llama-bench measures about 222 on that card. The dense formula is off by more than 4x, in the pessimistic direction.

The first correction is to scale the bytes read by the active fraction: the in-memory model size is multiplied by active/total parameters. This is an approximation, since attention and shared weights are not routed and the active count already includes them, but it allows a single ratio taken from the model's config to be used instead of modelling tensors separately. A model takes this path only if active parameters are below 90% of total. Anything denser is treated as dense, in which case the function returns exactly the dense result.

The routing term

Scaling bytes alone overshoots. When only 3.6B parameters are read per token, the bandwidth term becomes very small, and a fixed per-layer cost that the formula ignored begins to dominate: evaluating the router and dispatching to expert kernels, once per layer, per token. This cost is modelled as an additive time per layer:

seconds/token = active_GB / (bandwidth x efficiency)
              + layers x per_layer_overhead_ms / 1000

The per-layer overhead constants are 0.055 ms for NVIDIA and 0.2 ms for Apple, AMD and Intel, with 0.1 ms when the brand is unknown. The NVIDIA value is much lower because CUDA graphs amortize launch cost better than the other llama.cpp backends do. If the layer count is missing, a fallback of 48 is used, the median across MoE models in the catalogue at the time of calibration. The constants were fitted against llama-bench tg128 (decode) results for gpt-oss-20b and gpt-oss-120b on 8 hardware combinations. They are kept in the test suite as fixtures, so a change to any constant that breaks one of them fails the build.

Two worked examples follow, computed from the constants above. The bandwidth values are the GPU/SoC spec values used in the fixtures. The measured figures are the llama-bench results in the fixtures.

	RTX 4090, gpt-oss-20b	M2 Ultra, gpt-oss-120b
Model in memory	12.9 GB	65 GB
Active / total params	3.6B / 20.9B	5.1B / 116.8B
Bytes read per token	2.22 GB	2.84 GB
Bandwidth (spec) x efficiency	1008 x 0.65	800 x 0.70
Bandwidth term	3.39 ms	5.07 ms
Layers x overhead	24 x 0.055 = 1.32 ms	36 x 0.2 = 7.20 ms
Estimate	212 tok/s	81.5 tok/s
Measured (llama-bench)	222	80

The 4090 case is dominated by bandwidth. In the M2 Ultra case, routing overhead accounts for more than half of the per-token time.

Accuracy

The test suite asserts that every fixture lies within 18% of its measured value. The public page states an expected error of roughly plus or minus 20% on decode tok/s, and quoting a tighter figure is not justified. Eight fixtures on two model families constitute a small sample. The constants fit gpt-oss well. They have not been validated on every MoE shape (many fine-grained experts, large shared experts), and misses of more than 20% are expected for some of them.

Where it breaks

CPU and RAM offload is the largest gap. Once part of the model resides in system RAM, decode speed is pulled toward the slowest tier from which weights are read, and a single bandwidth number is no longer meaningful. The methodology page lists offload as not modelled. Fit still indicates whether the model loads, but the tok/s figure assumes that everything is in fast memory.

Prefill is a different regime. Reading the prompt is compute-bound, so it is estimated separately: prefill tok/s = dense FP16 TFLOPS x 1e12 x MFU / (2 x active parameters), with MFU of 0.45 for NVIDIA, 0.22 for AMD, 0.18 for Intel and 0.20 for Apple, calibrated on llama.cpp pp512 runs with Q4_K_M weights. Time to first token is the number of prompt tokens divided by that rate. MoE benefits prefill as well, since FLOPs scale with active parameters, but the error bar is wider, about 30%, and wider still where no verified compute figure is available. A fast-decoding card can therefore still impose a long wait before the first token.

Context length is a further, less visible source of error. The decode formula accounts for the weights and nothing else, so it ignores the growing KV cache that attention also reads on every token. The cache itself is straightforward to size: 2 (K and V) x KV heads x head dim x layers x 2 bytes (FP16) per token. Some architectures keep a full cache on only some layers (linear attention, sliding window), and the VRAM code accounts for that. The speed estimate, however, does not decrease with context, so long-context numbers are optimistic. Quantized KV cache is also not modelled.

Quantization carries its own overhead. Model size is derived from effective bits per weight, so Q4_K_M files are not exactly 4 bits per weight, and measured sizes from Ollama or llama.cpp are preferred over the computed estimate whenever they exist. Batching, speculative decoding, multi-GPU interconnects and flash-attention variants are likewise outside the model.

Fit, briefly

The fit side is simpler. VRAM is weights (parameters x bits per weight / 8) plus KV cache plus about 0.3 GB of framework overhead. When architecture data is unavailable, the fallback is a flat 10% overhead. "Fits" is judged with room reserved for a 16K-token context, because a model that loads with a 2K cache is not usable for a real conversation.

The arithmetic for an individual model is printed on its page, for example gpt-oss-20b. Measurements from llama-bench that disagree with an estimate by more than the stated error are welcome, since this is how the constants are corrected.