MMLU, HumanEval, ChatBot Arena — the tests we use to rank AI models are deeply flawed. Here’s exactly how, and what actually matters instead.
In 2023, a model scored 90% on MMLU — the benchmark most commonly used to compare AI intelligence. Researchers celebrated. Headlines followed.
Three months later, someone noticed the model had been trained on data that included MMLU test questions.
It hadn’t learned to reason. It had learned to recognise answers it had seen before. The benchmark measured memorisation and called it intelligence.
This isn’t an isolated incident. It’s a systemic problem — and understanding it matters not just for researchers but for anyone making decisions based on AI capability claims.
What benchmarks are supposed to do
A benchmark is meant to measure something real about a model’s capabilities. Ideally it samples from a broad distribution of tasks the model hasn’t seen, tests generalisation rather than memorisation, and produces a number that meaningfully predicts real-world performance.
That’s the theory. The practice is messier.
The problem is structural: the moment a benchmark becomes widely used, it becomes a target. Labs optimise their models to perform well on it. Training data gets curated to include benchmark-adjacent content. Prompting strategies get tuned to benchmark formats. The benchmark stops measuring generalisation and starts measuring adaptation to the benchmark itself.
Goodhart’s Law — “when a measure becomes a target, it ceases to be a good measure” — was coined in economics. It describes AI benchmarking almost perfectly.
Press enter or click to view image in full size
MMLU — the benchmark everyone cites and nobody fully trusts
MMLU — Massive Multitask Language Understanding — tests models across 57 subjects from elementary mathematics to professional law. A model that scores well presumably has broad knowledge and reasoning ability.
The problems:
First, contamination. MMLU questions are public. They’ve been on the internet for years. Any model trained on a large web crawl has almost certainly seen some of them. The benchmark measures something between knowledge and memorisation, and there’s no clean way to separate the two.
Second, format sensitivity. MMLU is multiple choice. Models can score well by learning patterns in how multiple choice answers are structured — without understanding the underlying content. A model that’s good at elimination strategies on multiple choice tests looks smart on MMLU even if it fails at open-ended reasoning.
Third, saturation. The best models now score above 90% on MMLU. When every frontier model clusters near the ceiling, the benchmark no longer differentiates between them. You can’t use a ruler that only measures up to 90cm to compare things that are all around 90cm.
HumanEval — where code benchmarks break down
HumanEval is OpenAI’s benchmark for code generation — 164 programming problems, each with a function signature and docstring. The model writes the function body. Tests check correctness.
It sounds rigorous. The problems are real coding tasks. The evaluation is automated. What could go wrong?
The problems are too simple for frontier models. The top models now score above 90% on HumanEval. The benchmark was designed when models were far less capable — it no longer tests anything interesting about the best systems.
The tasks don’t reflect real engineering. Writing a function that reverses a string is not what software engineers do. Real code involves understanding large codebases, managing dependencies, debugging across files, handling edge cases that aren’t specified in a docstring. HumanEval measures none of this.
Pass@k gaming. The standard metric is pass@1 — does the model get it right on the first try? But models can generate multiple attempts and pick the best. Labs optimise for pass@k in ways that look better on the leaderboard without improving real-world usefulness.
Chatbot Arena — better, but still broken
Chatbot Arena — run by LMSYS — takes a different approach. Instead of fixed test sets, it shows users two anonymous model responses to the same prompt and asks which is better. Human preference drives the ranking.
This is genuinely better than static benchmarks for some things. It’s hard to game because you can’t train specifically on the test set — the prompts are real user queries that change constantly. And it measures something users actually care about: which response do I prefer?
The problems:
Selection bias. The people voting on Chatbot Arena are predominantly English-speaking, technically literate, and interested in AI. Their preferences don’t represent the global user population. A model that writes eloquent English prose ranks higher than one that’s genuinely useful across languages.
Preference ≠ accuracy. Humans prefer responses that sound confident and well-structured. A model that hallucinates eloquently can beat a model that hedges accurately. Arena ranking correlates with fluency and presentation, not with factual correctness.
Prompt distribution skew. The prompts that get submitted to Arena skew toward creative writing, coding questions, and general knowledge — not the specialist domains where model capability differences actually matter most.
What I learned building a benchmark for MOSAIC
When I built the evaluation harness for MOSAIC — my Linux userspace scheduler for disaster-response edge nodes — I ran into every one of these problems at the system level.
Choosing metrics is hard. P99 tail latency, average latency, throughput, energy efficiency, task starvation rate — these tell different stories. A scheduler that looks great on average latency can look terrible on P99. A scheduler that’s energy-efficient can have higher starvation rates. No single number captures what “good” means.
Workload distribution matters as much as the metric. MOSAIC’s benchmark harness generates 5 different arrival patterns including a disaster-scenario generator specifically because the “right” scheduler changes depending on the workload mix. A benchmark that only tests steady-state traffic would make every scheduler look similar.
Reproducibility is undervalued. I shipped 63 unit tests and a full Docker/Kubernetes deployment specifically so results could be reproduced independently. Most AI benchmarks don’t meet this standard — results depend on prompting strategies, sampling parameters, and evaluation code that isn’t always publicly released.
The lesson: a benchmark is only as good as the assumptions baked into it. Understanding those assumptions is more important than reading the final number.
What actually matters instead
This isn’t an argument that benchmarks are useless. It’s an argument that they’re systematically misread.
What to actually look for:
Task-specific evaluation. If you’re deploying an AI system for legal document review, the relevant benchmark is performance on legal document review — not MMLU. General benchmarks are a starting point, not a conclusion.
Held-out test sets with verified non-contamination. The gold standard is evaluation on data that provably wasn’t in the training set. Some newer benchmarks — like BIG-Bench Hard and MATH — are harder to contaminate because the problems require genuine multi-step reasoning rather than recall.
Behavioural evaluation over capability evaluation. Does the model do what you need in your specific context? That’s harder to measure than a leaderboard score but it’s the only question that matters for deployment.
Disagreement between benchmarks as a signal. When a model ranks first on one benchmark and tenth on another, that tells you something important about what each benchmark is actually measuring. The disagreement is information, not noise.
The uncomfortable truth
The AI industry has a publication incentive problem. Labs release benchmark scores alongside model announcements because strong numbers generate coverage and trust. The incentive to optimise for benchmark performance — even when that doesn’t reflect real capability — is structural, not malicious.
The solution isn’t to distrust all benchmarks. It’s to read them the way a researcher reads a paper — with awareness of methodology, limitations, and what the authors were incentivised to show.
The number on the leaderboard is a starting point for a question, not an answer to one.
This is part of an ongoing series on LLM fundamentals and ML systems. Previous posts covered attention, tokenization, positional encoding, hallucination, KV cache, and Mixture of Experts.
Drop a comment or connect on LinkedIn — always open to feedback and pushback.
Top comments (0)