Every new AI model launches as "state of the art," topping a chart of impressive-looking scores. They can't all be the best — and the benchmarks they're waving around are quietly broken in ways that should make you deeply skeptical of the number.
Benchmarks are how the field claims progress. They're also increasingly gamed, contaminated, and misleading. Here's what the leaderboard isn't telling you.
Contamination: the test answers are in the textbook
The deepest problem is simple. Benchmarks are public datasets of questions and answers. Models train on enormous scrapes of the internet. So the benchmark's questions — and answers — often end up in the training data.
Now the model isn't reasoning through the test; it may be partly recalling answers it already saw. It's the equivalent of a student who memorized the answer key. The score looks like intelligence and is partly just memorization, and from the outside you often can't tell how much. This is benchmark contamination, and as models train on ever more of the web, it gets harder to avoid.
Teaching to the test
When a benchmark becomes the thing everyone competes on, it stops measuring general ability and starts measuring "ability to score on this benchmark." Effort pours into optimizing for it — sometimes subtly, sometimes by training on very similar data. The number climbs; the real-world capability it was supposed to represent doesn't climb nearly as much. Any metric that becomes a target stops being a good measure — an old rule that applies to AI leaderboards with a vengeance.
Saturation and the narrow slice
Many benchmarks are also nearly "solved" — top models cluster at 90-something percent, where the remaining gap is noise and quirks, not meaningful capability differences. And each benchmark tests a narrow slice under clean conditions that look nothing like your messy real use case. A high score on a tidy academic set tells you little about whether the model helps with your actual problem, on your actual data.
What actually tells you something
The move that separates people who ship reliable AI from people who chase leaderboards: stop trusting generic benchmarks and evaluate on your own task. Build a set of examples that look like your real usage, including the hard and weird cases, and measure the model against that. Your eval set can't be gamed by a vendor and can't be contaminated by training scrapes you don't control — it measures the only thing that matters, which is whether the thing works for you. That discipline of honest, purpose-built measurement is something I care about deeply and write about across my work.
The takeaway
Treat "state of the art on benchmark X" as marketing, not truth. The real questions are: was the test data in the training set, is everyone optimizing for this exact number, and does the benchmark resemble anything you actually do? Until you've measured a model on your own representative task, you don't know if it's good — you only know it's good at the test. More on evaluating honestly at www.divyakush.com.
Related reading
- Evaluating LLMs: why 'it looks good' isn't a metric — how to build evaluation you can actually trust.
Top comments (0)