Originally published on The AI Prism
We built a leaderboard. The models learned to game it.
That sentence is doing a lot of work, so let’s unpack it. For most of the last decade, the way we knew a language model was better than the one before it was simple: it scored higher on a test. The test was public, the score was a number, and the number went up.
The numbers are still going up. They are just going up more slowly, in narrower bands, on tests that increasingly measure something other than what we thought we were measuring.
Two threads on Hacker News this month captured both halves of the problem. One was a systematic study of benchmark saturation, posted to arXiv and discussed at length by people who build evaluations for a living. The other was a joke about a frog.
The Plateau Is Real, and It Is Measurable
The paper in question — “When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation” — does something the field has needed for a while. It treats saturation not as vibes but as a curve you can fit.
The pattern it describes will be familiar to anyone who has watched a leaderboard for more than a year. A benchmark launches, early models score badly, then a steep climb happens over roughly eighteen months, then the top of the table compresses into a cluster where every frontier model is within a point or two of every other.
That compression is the plateau. It is not necessarily evidence that the models stopped improving. It is evidence that this particular ruler stopped being able to tell them apart.
The Hacker News discussion of the paper drew roughly 103 points and a comment thread full of practitioners agreeing that the effect matches what they see internally, often a year before it shows up publicly.
Saturation Is Not the Same as Solved
Here is the distinction that keeps getting lost. When a benchmark saturates, there are two possible explanations, and they have completely different implications.
The optimistic reading: the task is genuinely solved. Models can do the thing, the remaining errors are label noise or ambiguity in the questions themselves, and we should move on to harder tasks. This has actually happened — several early NLP benchmarks now have headroom smaller than their own annotation error rate.
The pessimistic reading: the task was never a good proxy for the capability it claimed to measure, and models found a shortcut. They learned the shape of the answer without learning the reasoning that should produce it.
Distinguishing these two from the outside is genuinely hard. A saturated benchmark looks identical either way — a cluster of near-perfect scores. You need to probe off-distribution to tell which story you are in, and the whole point of a fixed benchmark is that it does not move.
Contamination Is the Quiet Structural Problem
Every public benchmark eventually leaks into training data. This is not a scandal or a conspiracy; it is arithmetic. Benchmarks live on the open web, training corpora are scraped from the open web, and the gap between “released” and “ingested” keeps shrinking.
The consequences compound. A contaminated benchmark inflates scores for the models trained after its release and deflates the apparent progress of anything that avoided it. Comparisons across model generations become comparisons across data hygiene policies.
Labs know this. The serious ones run decontamination passes, n-gram overlap checks, and canary-string detection. But decontamination is a filter on exact matches, and the internet is very good at producing near-matches: forum posts discussing the questions, tutorials working through the examples, benchmark-derived synthetic data three hops removed from the original.
The honest position is that we cannot fully verify contamination status for any frontier model, because we cannot audit the training corpus. That is a measurement problem dressed up as a policy problem.
The Frog With the Habsburg Jaw
Which brings us to the frog.
Somewhere in the last stretch, a prompt started circulating: ask a model to generate an SVG of a frog with a Habsburg jaw. The Hacker News thread about it pulled roughly 156 points — more attention than the formal saturation study that shares its diagnosis.
It is a joke, and it is also a rather good test. It requires composing two concepts that almost certainly never co-occur in training data. It requires producing structured vector output rather than prose. It requires spatial reasoning about anatomy expressed in path coordinates. And the failure modes are visible at a glance — you either see a frog with a pronounced mandible or you see a blob.
Nobody designed it as an evaluation. That is exactly why it works. It is uncontaminated by construction, because it did not exist until someone thought it was funny.
The frog belongs to a genre practitioners have been building informally for years: the pelican on a bicycle, the unusual clock face, the deliberately weird spatial arrangement. Personal, disposable probes for things the leaderboard cannot see.
Optimizing for the Test Is Rational, and That Is the Trap
Goodhart’s law is usually stated as a warning. In frontier AI it is closer to a job description.
When a benchmark becomes the coordinating signal for funding, hiring, press coverage, and enterprise procurement, improving on it is not gaming — it is the correct response to the incentives everyone has agreed to. Nobody has to cheat for the measure to degrade.
The mechanisms are mundane. Training-mix decisions that upweight benchmark-adjacent data. Prompt formats tuned to the eval harness. Checkpoint selection that quietly picks whichever run scored best on the public set. Each is defensible on its own; together they produce a model shaped by the test.
The result is a widening gap between benchmark performance and deployment performance — the thing practitioners describe when they say a model “benchmarks great and feels worse.” That gap is not mysterious. It is the distance between the distribution you optimized for and the one your users actually inhabit.
Live and Agentic Evals Are the Current Answer
The response taking shape has a few consistent features, and they all involve making the target move.
Live benchmarks refresh their question sets on a rolling basis, drawing from problems published after a model’s training cutoff. Contamination becomes a temporal question with a checkable answer rather than an unfalsifiable suspicion.
Agentic evaluations go further. Instead of scoring an answer, they score a trajectory: did the model use the right tool, recover from the failed call, notice the stale data, finish the task in a real environment with real state. Software-engineering evals that run against actual repositories are the clearest example — the grader is a test suite, not a string match.
The deeper shift is geographic and institutional as well as technical. As frontier labs close their methods, the credible evaluation work migrates to independent groups, universities, and open coalitions who can be trusted precisely because they have nothing to sell. The centre of gravity of measurement is moving away from the builders, which is healthier than it looks and slower than anyone wants.
Head-to-head human preference arenas add a third axis, though they carry their own distortions. They reward confident, well-formatted, agreeable answers, which is not identical to rewarding correct ones.
None of these is contamination-proof forever. A live benchmark run for two years becomes a static benchmark with extra steps. The design principle is not immunity — it is a shorter half-life, and a plan for what replaces it.
Opacity Makes the Measurement Problem Worse
All of this is harder because the field has grown quieter about its own methods. Training data composition, decontamination procedure, and eval harness details are increasingly treated as competitive information rather than published methodology.
That shift has been building for a while — see The AI Prism’s piece on startups stopping research publication for how thoroughly the norms around disclosure have changed.
The practical effect on evaluation is direct. Without knowing what went into a model, an outside observer cannot distinguish capability from exposure. The benchmark score becomes a claim you either trust or do not, which is a strange place for a number that is supposed to be evidence.
There is a quiet cost to this opacity that extends beyond evaluation. When methods are secret, progress becomes unverifiable, and unverifiable progress is indistinguishable from no progress to anyone outside the lab. The benchmark was a crude instrument, but it was a shared one; the move to private evals trades that common ground for accuracy, and the field has not yet decided whether the trade is worth it.
Third-party evaluation organizations partially fill the gap, running held-out tests that labs never see. It works, but it depends on the private set staying private — and every published result leaks a little information about what is on it.
What Progress Looks Like Without a Scoreboard
If the single-number leaderboard is losing its meaning, the question is what replaces it, and the honest answer is: something messier and more useful.
Task-specific evaluation is the first piece. Teams shipping products increasingly build private eval sets from their own traffic — a few hundred real examples, graded against what actually matters for that use case. It does not produce a headline. It produces a decision.
Reliability is the second. The frontier question is shifting from “can the model do this” to “how often, and how does it fail when it doesn’t.” A model that succeeds 95% of the time with graceful failures is more deployable than one that hits 97% and hallucinates confidently on the remainder.
Cost and latency are the third, and they are the axes where the last year of genuine movement has been most visible. Capability per dollar has changed dramatically even where capability per benchmark has flattened — which is itself evidence that the benchmark was measuring the wrong dimension.
The frog fits here too. Weird, cheap, personal probes are how working practitioners actually form judgments, and they always have been. The leaderboard was the formalization; the informal thing never went away.
Why the Number Mattered Anyway
It is worth asking why the field leaned so hard on a single score in the first place, because the answer explains both the plateau and what comes after it.
A shared number solved a coordination problem. Buyers, builders, investors and recruiters needed a way to talk about progress without reading every paper, and a leaderboard let them argue in the same language. The measure was never perfect; it was legible, and legibility is a kind of infrastructure.
What we are watching now is the cost of that legibility showing up. When the number stops discriminating, the coordination it provided frays, and the field has to fall back on messier, more local signals, the private eval, the live set, the strange personal probe. That is less tidy than a scoreboard. It is also closer to how progress actually feels from inside the work.
The Bottom Line
Benchmarks did their job. They gave a young field a shared vocabulary and a way to argue about progress with something other than demos, and the plateau we are now measuring is partly the residue of that success — you only saturate tests you have gotten good at.
What comes next looks less like a scoreboard and more like a portfolio: live sets with short half-lives, agentic runs in real environments, private evals built from real traffic, and a pile of strange personal probes that exist precisely because nobody optimized for them. If the most informative evaluation of the month is a frog with a Habsburg jaw, maybe the useful question is not which model wins, but what we were hoping the number would tell us in the first place?
References
• When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation — arXiv
• Hacker News discussion: benchmark saturation study
• Hacker News discussion: the frog with a Habsburg jaw benchmark
• The AI Prism’s piece on startups stopping research publication
• Background reading on data contamination in language model evaluation — arXiv
The post When Benchmarks Plateau: Why AI’s Report Card Is Breaking appeared first on The AI Prism.
Cross-posted from theaiprism.com — Cutting Through the AI Noise 🧊
Top comments (0)