Part 1 demonstrated that simply averaging benchmarks can push a model tested on just three easy boards higher than flagships evaluated across eighteen. Part 2 swapped this average for an IRT model instead. This final section takes a wider view. Many groups release one score to answer "how good is this model," yet they arrive there through very different methods.
An easy way to see that the approach matters: consider the 137 models with both an LMArena rating and an llmrun Composite Score. Across all of them, the two rankings align closely, showing a rank correlation of 0.91. Among the 20 strongest models, however, they disagree completely: -0.02.
Neither figure is flawed. They address different questions, and the gap appears exactly where people examine it most closely.
Three ways to get one number
Average a fixed set of evals. The publisher runs every model on identical tests and calculates a weighted mean. The Artificial Analysis Intelligence Index uses 10 evaluations split into four weighted categories. The now-retired Hugging Face Open LLM Leaderboard relied on six benchmarks, removing the random-guess baseline from each before scoring. This approach stays clean and controlled, yet it covers only what the publisher can afford to run; plus, swapping out for a new set of evals creates a fresh version that no longer compares fairly with previous results.
Let people vote. LMArena presents users with two anonymous answers and fits a Bradley-Terry model to the votes, which is essentially an Elo system. No benchmark grid is required; only head-to-head battles suffice. The catch? It measures which answer audiences prefer, not what a model can actually do. Length emerged as the strongest style effect. LMArena's style control adjusts for length and markdown, yet a vote remains just a vote. An independent audit, The Leaderboard Illusion, also revealed that closed models get sampled into more battles than open ones.
Infer a hidden ability. Item response theory treats each benchmark result as evidence about a model's ability, assigning its own difficulty to every test. The Epoch Capabilities Index applies this method across more than 50 benchmarks, just like our score over 40. A major benefit is handling sparse data: two models that never shared a benchmark can still be compared via intermediate ones that overlap both. However, an estimate is only as strong as the boards supporting it, and refitting may shift old values unless the scale remains frozen.
At a glance
- Artificial Analysis Intelligence Index. This is a weighted average of 10 evals it runs itself, lacking any interval for the index. It works best when comparing current models within a single harness.
- LMArena. Uses Bradley-Terry to analyze human votes, complete with confidence intervals. Ideal for comparing chat preferences.
- Epoch Capabilities Index. IRT across more than 50 benchmarks, using a 90% bootstrap interval. Ideal for tracking progress along the frontier over time.
- llmrun Composite Score. It applies IRT across 40 boards using a fixed scale and sets a 90% spread for each model. This is ideal for open-weight models; check it alongside your hardware compatibility.
- LiveBench. It runs its own evaluations using rotated questions and includes 95% bootstrap intervals. This suite works best when you need to minimize data contamination.
Methods as each index documents them, checked on 2026-10-22.
Reading one score without fooling yourself
Here are a few rules of thumb, using figures from our leaderboard dated 2026-10-22.
- If two ranges overlap, call it a tie. MiMo V2.6 Pro RL scores 133.4 while Kimi K3 comes in at 132.8. MiMo operates on a 90% range from 130.6 to 136.2, based on 8 boards; Kimi's sits between 131.2 and 134.4 based on 30. Given the current data, they appear evenly matched, though Kimi's figure feels more reliable.
- If you care about one skill, ignore the overall rank. DeepSeek V4.1 Flash ranks fifth among open models with a score of 129.5; however, its coding ability sits at 122.2, roughly 12 points behind Kimi K3's 134.4. The model shines in reasoning, scoring 134.0.
- If it needs to run on your machine, check memory first. Kimi K3 requires around 1.7 TB of memory even at 4-bit precision. None of the top open models fit on a single 24 GB card; the 24 GB page ranks the ones that do.
Which one to read
- Choosing a model for local use. Begin with per-skill scores for open-weight options, then narrow down based on what your GPU can handle. That's exactly where our scoring system applies.
- Following the frontier. ECI or the Artificial Analysis index.
- How a chat model feels. LMArena, preferably with style controls engaged.
- Worried the model saw the test? LiveBench swaps out roughly one-sixth of its questions with every update.
- Any of them. Examine the interval and determine what it actually measures. When a narrow range comes from multiple runs, it reveals little about how those specific benchmarks were chosen.
Where ours fits
ECI counts as its nearest relative, with distinctions lying mainly in scope:
- We don't run any models; everything stems from published benchmarks like Epoch's hub, LiveBench, the SWE-bench leaderboards, and MMLU-Pro.
- Open-weight models are the focus here, with a look at how each one matches your hardware.
- The scale remains locked to a 2026-Q4 reference, meaning scores update solely when fresh results for a given model come in.
- A board stays active only as long as it has results from the past year. This solves the issue mentioned in Part 1.
Known gaps: most benchmarks focus on English, with no vision or multilingual domain yet; we rely entirely on what upstream sources publish. You can find the full method and changelog on the methodology page, while the current ranking is on the leaderboard.
Top comments (1)
A check on the -0.02: part of a drop from 0.91 to near zero is just range restriction, so I simulated how much. 137 models, two noisy measures with a rank correlation of 0.91 overall, take the top 20 on one of them and rank-correlate them with the other (5,000 runs, Gaussian). The median comes out around 0.63, with a 90% range of 0.32 to 0.83. At an overall 0.95 it is still 0.73 (0.47 to 0.88).
So losing some correlation at the top is expected, but -0.02 sits below even the low end of that range, which supports your point that the two indices measure different things up there (preference vs ability) and not only that the top is crowded. The Gaussian assumption is a simplification, since real score distributions have a fat top tail, so I would treat it as a rough baseline.
One more way to check it: bootstrap each model's 90% interval and see how often the top-20 order flips by chance alone. If your intervals are ±2-3 points and the top 20 span ~10, a lot of the within-top-20 order is noise on both sides. Do you have the top-20 correlation with ties between overlapping intervals collapsed?