I Tested 300+ Models. Then I Killed the Benchmark.
Let It Break — part 1
Tags: #ai #llm #benchmark #postmortem
In May I ran a series called Agent Autopsy. Agents failing — broken packages, forgotten context, cron jobs dying silently — while I was learning what I didn't know I didn't know. Every failure got a post-mortem, because every failure felt like it deserved one.
That series didn't end with a finale. It ended when I stopped caring. Somewhere around part eight, broken things stopped feeling like emergencies. Call it tolerance. Call it resilience. Maybe I just got tired. Probably all three.
This is what came after: a series about what I kill on purpose, what I leave broken on purpose, and what that buys me. Nothing in this post failed. The benchmark worked perfectly, right up until the evening I killed it.
Back then I also wrote about binning an SPFx agent harness nobody asked for. In that post I held up the benchmark as the thing people actually wanted. The thing worth keeping.
This month I killed the benchmark too.
Same lesson. Longer fuse.
What it was
Ten real-world agent coding tasks — file operations, shell commands, error recovery, data parsing, SQL queries. Every model I could reach through OpenRouter, plus everything I could fit on my own hardware. Max tokens 400, temperature 0.1, pattern-matching scoring, pre-flight verification so a flaky endpoint couldn't fake a zero.
The last batch I published put the public dataset at 168 models. The real count — OpenRouter plus local, including batches I never wrote up — passed 300.
A 10-model batch cost about $0.10. The 200-call efficiency study cost $0.56. The whole dataset cost less than a takeaway.
Cheap to run. Expensive to keep honest. I didn't understand the difference until I was hundreds of models deep.
The benchmark wrote its own obituary
Read my own headlines in order.
May: the best local model scored 50%. Days later: five brand new families debuted, none below 75%. Then: two models hit 90%, one for less than a penny.
Fifty percent, to a 75% floor, to 90% at sub-penny prices. In weeks.
When the punchline of every batch is "they're mostly fine and they're all cheap," the leaderboard has answered its own question. The last finding that actually helped anyone wasn't about model quality at all — it was that the free tier is a gamble, 81.6% and 10% in the same batch.
The interesting question stopped being "which model can do this." Most of them can. It became "how little scaffolding do I need." A leaderboard can't answer that.
The signal I ignored
Nobody was using it.
The posts got reads. A few good comments. But I couldn't find one person making one decision off my numbers. I told myself the audience would arrive once the dataset was big enough. The dataset got big. The audience was still me.
Here's the tell I only see in retrospect: the published count stopped at 168, but I kept testing past 300. I was running batches I didn't even bother writing up anymore. A leaderboard nobody read, fed by tests nobody saw — including, eventually, me barely looking at them either.
Traction you have to talk yourself into is not traction. Supporting Liverpool teaches you to sit loyally through seasons that are going nowhere. I gave a leaderboard the same loyalty. The leaderboard had not earned it.
Cause of death
Staleness, and it was structural. Models land weekly. Every batch was a photograph of a moving train — accurate for about as long as it took to write the post. Keeping the leaderboard honest meant re-running everything, forever, on my own time, for an audience of one.
There's a personal irony here. I wrote earlier this year about running 20 agents on €4.57 a month of infrastructure. That efficiency is what kept the benchmark alive past its expiry date. When a zombie project costs pennies to feed and runs itself on cron, killing it requires noticing, not budgeting. Cheap automation doesn't just scale the good ideas.
And the harness had compromises I'd stopped seeing. The 400-token cap punished verbose-but-correct models. Pattern-matching scored format as much as competence. Fixing that meant more harness. The models were getting better faster than the harness could get fairer.
I built scaffolding to measure models. The models outgrew the scaffolding. That's not an engineering failure — that's the ecosystem working as advertised, and me billing myself weekly for refusing to notice.
What survives
The habit. I still test every model that interests me the day it drops, against tasks I actually care about. That reflex came from the benchmark and outlived it.
The findings. I know from data, not vibes: code quality does not equal agent capability, "write efficient code" prompts do nothing for most models, and free tiers charge you in debugging time.
The data, archived. If anyone ever genuinely needs it, it can be served through MCP in an afternoon — a query interface, not a leaderboard. No maintenance, no weekly re-runs, value on demand. The thin version of the same idea.
The harness does not survive. That's the right way around.
The rule
The weekend harness died on the whiteboard, where bad ideas are cheap. The benchmark died past 300 models, where they're not. Same disease, later diagnosis: "is anyone looking for this?" isn't a question you ask once at the start. You ask it every time you're about to maintain something.
Killing it cost one evening. Keeping it cost every week. I did that math embarrassingly late.
Next time I kill it at 50 models, not 300.
Tolerance, resilience, or just tired? This one was tired — and late. The next posts in this series cover the things I've left broken on purpose. Those are harder to defend. That's why they're worth writing.
Top comments (1)
"Killing it cost one evening. Keeping it cost every week." — I felt this one in my bones.
I'm not a developer (physical therapist who ended up building internal tools for my hospital with AI), and somewhere past tool #30 I learned that shipping things is only half the skill. The other half is noticing which ones are quietly dead. I once built an image viewer to replace a discontinued freeware our staff loved. Weeks of work, genuinely proud of it — and then a signed open-source app turned out to do the job better. Retiring my own thing stung for about a day. Maintaining it out of pride would have stung every month.
The test I use now, since internal tools don't have analytics dashboards: if it broke tomorrow, who would complain? If I can't name a person, the tool is already archived — I just haven't admitted it yet. Your line "traction you have to talk yourself into is not traction" is the same test wearing better clothes.
Also, publishing the postmortem probably helps more people than the leaderboard did. This kind of honesty is rarer than benchmarks.