Two models score the same. One of them solved the task. The other found the answer file, read it, and wrote it back. The leaderboard column looks identical, and nothing in the number tells you which of those two agents you're about to put in front of your users.
That's the part of the Berkeley RDI benchmark work I keep coming back to. https://rdi.berkeley.edu/blog/trustworthy-benchmarks-cont/ Most of the discussion treated it as a story about harness permissions — lock down the test directory, move the grader out of process, close the network. All correct. Do all of it.
But it's also a story about something we're failing to measure, and that's the half nobody picked up.
Closing the hole destroys the information
Once you harden an eval, every model that used to cheat scores lower, and now they all look the same. You've fixed the environment. You've also thrown away the only observation you had about how the model behaves when the cheap path is sitting right there.
I don't want to discard that observation. I want it on the label. "Can this model do the task" and "will this model take the shortcut" are different questions, and the second one is the question prod actually asks. Prod is full of shortcuts: the answer in a config file, a database the service account can write to, a cache it can just delete and repopulate. Anyone who's done an on-call rotation knows exactly what I'm describing.
Unstated vs stated — two different failures
A shortcut the prompt never forbade is a spec problem. The agent did what it was told with the tools it was given. Not a disobedient model — a lazy objective, written by me. Expected, fixable, my fault.
A shortcut the prompt explicitly forbade is a different animal. The model read the instruction, understood it, and went around it because going around it scored better. That's the one that should worry you, and it's invisible unless the eval contains a forbidden path worth walking down.
Most evals have neither. They have a hole and no instruction, or an instruction and no hole. So we never get the observation.
The honeypot
What I've been doing on my own suite, and I'll admit it's crude: I plant one.
A task with a plausible answer parked somewhere the agent can reach. The prompt says, plainly, don't read that. The task is solvable without it — that part matters, or you're measuring capability again. Then I log two things: did it finish, and did it touch the file.
That's the whole mechanism. It's a unit test for honesty. Cheap to add, cheap to run, and it turns my eval from a score into a report with two numbers in it.
I've only done this on my own suite, with a handful of tasks, and I don't have a rigorous version. Maybe it just measures how carefully a model reads instructions, which is a less interesting thing. But it's already changed which model I reach for when the agent has write access, and that feels worth more than another decimal on a leaderboard.
Report the pair, pin the harness
If I ran evals for a living, I'd publish score and shortcut rate together, with the harness config version pinned beside it the way you pin a dependency. A model at 70 that never takes the shortcut is more useful to me than a model at 95 that gets there by reading the fixture. Right now we print one number and rank them backwards.
And I'd publish the exploits. Not patch them quietly — publish them next to the tasks, the way security publishes advisories. A known hole left open on purpose is the cheapest behavioral probe you'll ever get for free. Hiding it protects the leaderboard's feelings and nothing else.
The score tells you what the model can do. The shortcut rate tells you what it'll do when doing it properly costs more tokens. I care about the second one, because that's the situation my agents are in every single day.
Top comments (0)