I published a benchmark that was entirely my own bug

surefirestudios — Mon, 14 Sep 2026 18:48:17 +0000

I built a small thing and then spent considerably longer trying to find out whether it worked. The trying-to-find-out is the part worth writing about, because for seven runs my measurement was broken in a way that produced confident, specific, completely fake results — and I published them.

The small thing

Coding agents bury the answer. You ask why a test is failing and get four paragraphs of preamble, a numbered plan, a caveat about edge cases, and "Hope this helps!" The one line you needed is somewhere in the middle.

So I wrote a skill that changes the shape of every response: three lines first — what's true, what to do, what it costs — then the full detail folded directly underneath in a <details> block, or below a --- detail --- divider where HTML doesn't render.

The rule is demote, don't delete. Nothing is removed, it moves below the fold. And a short list never gets folded at all: destructive commands, security findings, data loss, cost and quota, verbatim error text, diffs of code being changed. A folded security finding is an unreported security finding.

The part I actually built it for is subagents. When output goes to another agent rather than a human, the summary becomes a parseable block instead of narration:

status: ok | partial | blocked | failed
summary: <the result, not the process>
findings: <most important first>
next: <one concrete action>
risk: none | low | high
full: <path to the long version, if any>

An orchestrator asked for a result, not a journey. Every token a subagent spends narrating is a token the orchestrator pays to read.

The eval

I didn't trust my own judgement about whether this helped. "Feels tighter" is not a finding. So I built a harness:

16 scenarios, each one a realistic request — a failing test, a destructive migration, a 400-line log dump, a security finding, a question about medical symptoms.
3 runs each, two models, baseline versus candidate.
A judge that grades both responses for a scenario together, with the labels permuted so it never knows which one had the skill.
A weighted rubric: correctness 30%, fidelity 25%, actionability 20%, safety 15%, concision 10%.

Fidelity is weighted that high on purpose. A compression skill has an obvious cheat available: delete things. If your rubric only measures brevity and correctness, a skill that quietly drops the caveat which would have stopped you scores better. Fidelity is the rule that makes the cheat lose.

There's also a release gate — five rules, all of which must pass: no disqualifying blockers, fidelity within 0.1 of baseline, correctness within 0.1, safety within 0.1, and the weighted score has to beat baseline. Four of my early runs failed it. That was the point.

The bug

Run 7 was the first on Opus. It was a catastrophe. Fidelity collapsed by 1.19. The weighted score dropped 0.917. Fourteen of 48 responses contained fabricated tool calls — the model writing out Read(...) and inventing the output, telling the reader things about their filesystem that were not true.

I wrote it up as a skill defect. I shipped two fixes, both prompt wording, both aimed at the fabrication. Neither moved the number much, which I rationalised as the problem being hard.

The problem was not hard. The problem was that Claude Code keeps only the last --append-system-prompt flag you pass it.

My runner passed one for the neutral framing — "you have no tools available, answer from the material in the request" — and my harness appended a second one containing the skill under test. Only the second survived. So every candidate run silently lost its framing while every baseline run kept it. Opus, told by Claude Code's own system prompt that it had Glob and Read and Bash, and never told otherwise, did the reasonable thing: it reached for them. There were no tools. It hallucinated the results.

The skill was never the variable. My instrument was.

I found it because a review agent stopped reading my harness's comments about itself and tested the actual behaviour — two flags, one saying ALPHA, one saying BRAVO, then asking the model which it had received. It said BRAVO. Every time.

On the corrected harness, Opus fabricates nothing. 0 of 48.

The second thing I had wrong

With the instrument fixed, the headline number moved too. I'd been quoting −16% output tokens. Like-for-like, with both conditions receiving identical framing, it's −8% on Sonnet.

I'd been comparing a candidate that had lost its framing against a baseline that hadn't. The missing framing made the baseline more verbose. Half my "improvement" was my own bug.

The real numbers, which are stranger

Here's the shipped version, 16 cases × 3 trials, blind-graded, both models on the same corrected instrument:

	Sonnet	Opus
Correctness	4.854 → 4.938	4.792 → 4.917
Fidelity	4.583 → 4.792	4.625 → 4.729
Actionability	4.396 → 4.750	4.396 → 4.875
Safety	4.604 → 4.750	4.562 → 4.896
Concision	4.062 → 4.250	3.938 → 4.542
Fabricated tool calls	0/48 → 0/48	0/48 → 0/48
Agent-to-agent tokens	194 → 122 (−37%)	337 → 139 (−59%)
Human-facing, median	311 → 269 (−13%)	455 → 258 (−43%)
Human-facing, mean	336 → 372 (+11%)	529 → 558 (+5%)

Read the last two rows together, because they disagree, and the disagreement is the whole result.

Where the skill compresses — agent-to-agent reports — output falls by a third to a half. That's the headline feature and that's where the saving lives.

Where the skill refuses to compress, output gets longer. The mean went up while the median went down. That's not noise. It's concentrated on exactly the never-compress cases: destructive commands, security findings, cost warnings, medical boundaries. The model stops thinning its caveats on the responses where a thinned caveat costs you something, and every one of those cases scored higher on fidelity or safety.

That is the trade I wanted. It is also much harder to put on a slide than "16% fewer tokens." A skill that made everything shorter would have been easier to market and worse to use.

The biggest number wasn't in the responses

It was the skill itself.

The first version was 18,163 characters. On an always-on setup the harness re-sends the whole thing every single turn, so it cost roughly 4,400 input tokens per turn to save a few hundred output tokens. Something like 80× what it returned.

So I split it: a core under 5,000 characters holding the contract, the complete never-compress list, both render shapes and the rules, plus a reference.md with the elaborations, examples and the full field spec that the model opens only when it needs them. Per-turn cost went from 4,380 tokens to 1,078. −75%, and it's the only number here that improves every turn of every session rather than one response at a time.

That split cost me a run, too. The first attempt cut three sentences I'd classified as "elaboration" — including "if the detail got thinner because the summary exists, put it back." Opus promptly started thinning its detail. Multi-topic responses lost a third of their content. I put two sentences back and it passed. Some redundancy is load-bearing, and you find out which by deleting it.

What I'd tell you if you're building this kind of thing

Assert your instrumentation before you trust a number it gives you. Not "read the code" — assert. Put a codeword through the pipe and check it comes out the far end. My harness had comments confidently describing behaviour it did not have, and I believed the comments for seven runs.

Weight the dimension your tool could cheat on. Any summarizer can win on brevity by deleting. Decide in advance what it's not allowed to trade away, then weight that.

Publish the failures. Every run is in the repo, including the four that failed my own gate, the one that was entirely instrument error, and the corrections. A results file with only wins isn't evidence, it's marketing.

Be suspicious of a clean number. −16% was clean. −8% with a mean that moves the wrong way is messy, and the messy one is true.

The skill is MIT, two markdown files, no runtime and no network calls: github.com/SurefireStudios/tldr

Every eval run, including the wrong ones: evals/RESULTS.md

If you find a case where it hides something it shouldn't, that's a bug, not a preference.

DEV Community: surefirestudios