DEV Community: Driftproofhq

We Checked 162 AI Benchmark Gaps. Only 20 Separate Cleanly. The Bigger Problem Was the Missing Data.

Driftproofhq — Fri, 09 Oct 2026 03:43:41 +0000

Six frontier AI launch posts and nine public leaderboards gave us 162 model-vs-model gaps to check.

20 separate cleanly.

That was not the result that bothered us most.

The bigger problem was how often the published data did not let us answer the question at all.

Of the 44 gaps quoted in the six launch posts, 11 separate, 5 do not, and 28 cannot be checked honestly from what was published.

Of 118 adjacent leaderboard pairs, 9 separate.

And across the broader 617-row source audit, 483 comparisons cannot be given an honest repeat-aware interval from the public information.

That is the finding.

Not “benchmark scores are fake.”

Not “these models are secretly equal.”

Something more basic:

AI benchmark charts often look more precise than the public evidence lets us verify.

24 launch-post comparisons we could not test

Gemini 4 Argon’s launch material contains 15 relevant comparisons.

None supports an honest interval from the published data.

Mistral Large 4 contributes another 9.

Again, none.

The reasons vary: scores averaged across repeated runs without publishing the repeat count, different harnesses or effort settings on the two sides, or insufficient per-task results to reconstruct a paired comparison.

We did not manufacture precision to fill those gaps.

They are marked not checkable.

That distinction matters.

Not checkable does not mean wrong.

It means the evidence required to decide was not published.

A 6.4-point lead can still be unresolved

SWE-bench Multilingual makes the problem easy to see.

The top two entries in our snapshot are:

72.7% vs 66.3%.

Looking at the leaderboard, a 6.4-point lead feels substantial.

But the benchmark contains 300 tasks.

Around the statistically least precise part of the scale, its conservative resolution is roughly 24 tasks. That is a benchmark-level reference point, not a universal cutoff: the exact interval depends on where the two scores sit.

For the actual 72.7% vs 66.3% pair, the applicable interval still includes zero.

The benchmark does not establish separation between them.

In fact, six positions on that leaderboard cannot be distinguished from first place under the applicable comparison.

That does not mean those models are equal.

It means something narrower:

this benchmark, at this size, with these published results, cannot separate them.

Our audit removed five of our own results

This is the part that made me trust the final result more.

Our frozen first pass produced 21 separated gaps.

Then we audited every comparison against the benchmark’s actual score type and any uncertainty information published by the source.

Five of our original “separated” verdicts did not survive that audit.

Four gaps that had looked unresolved under the simpler treatment did separate when the source’s own intervals were used.

Final count:

20.

The correction went both ways.

We did not overwrite the frozen results. The original verdict and the audited verdict sit next to each other in the dataset, with the reason for the change recorded.

Why there are 162 comparisons but 617 rows

There are two denominators in the data, so this is worth making explicit.

The 162 are the headline comparisons in this analysis:

44 launch-post gaps
118 adjacent leaderboard pairs

The broader source audit contains 617 comparison rows, including an appendix of effort-curve contrasts that never entered the 162 headline count.

Of those 617 rows, 483 do not support an honest repeat-aware interval from the published information.

That result surprised us more than the 20/162 headline.

The replication problem often starts before statistical significance.

It starts with whether enough information was published to calculate uncertainty at all.

What we actually did

The analysis method was frozen and hashed before the benchmark scores were inspected.

For interpretable binary benchmark rates with a known number of questions, we use Wilson 95% intervals and a Newcombe hybrid Wilson interval for the difference.

Where valid per-task paired data exist, paired evidence takes precedence.

Where a source publishes suitable repeated-run uncertainty, that evidence takes precedence over pretending repeated runs are extra independent benchmark questions.

If the required information is missing, the answer is not guessed.

It is marked:

not calculable from published data.

The repository preserves the frozen method, source hashes, per-comparison results and the later score-type audit.

What this does not prove

It does not prove that two models with overlapping intervals have equal ability.

It does not prove that every comparison without a separated verdict is false.

It does not prove that benchmark sampling noise captures every source of uncertainty in an AI evaluation.

And it definitely does not tell you which model will work better for your workload.

It answers one deliberately narrow question:

When someone puts two benchmark scores next to each other and calls one higher, does the published evidence establish that the gap is larger than the uncertainty we can actually calculate?

For these 162 comparisons, the answer was yes 20 times.

Reproduce it

Full write-up and tables:

https://driftproofhq.com/leaderboard-noise/

All 617 rows, frozen method, provenance and hashes:

https://github.com/driftproofhq/ranknoise

And if you want to ignore our interpretation entirely and check a gap yourself:

https://driftproofhq.com/benchmark-gap/

Put in two scores and the number of benchmark questions. It returns the interval and expresses the difference in questions rather than only percentage points.

I maintain Driftproof, the open-source evaluation instrument this methodology grew out of.

Claude Code was used to build and run the analysis pipeline, including implementation of the interval calculations, source extraction, audit checks and reproducible result generation against the frozen method.

The data, calculator and methodology are public.

One question for people who actually use these benchmark tables to choose models:

What is the smallest benchmark lead you would personally accept as evidence that Model A is better than Model B without seeing repeated runs or per-task results?

I ran the same Claude Code skill on Haiku 5.5 twice. Run 1: +55 points. Run 2: nothing. Driftproofhq

Driftproofhq — Thu, 08 Oct 2026 05:15:26 +0000

Claude Haiku 5.5 came out on 7 October. Within a day I ran three popular Claude Code skills on it, and on Haiku 4.5 next to it. Three runs each, same task, same grader, same Claude Code version.

Here's the result that made me laugh.

The git workflow skill on Haiku 5.5, scored 0 to 1:

Run	Without the skill	With the skill	Gap
1	0.300	0.849	+0.549
2	0.849	0.839	-0.010
3	0.613	0.848	+0.235

Run 1 is the screenshot that goes viral. "This one skill makes Haiku 5.5 55 points better."

Run 2 says the skill does nothing.

Same setup both times. And look at where the swing actually is: the skill arm sits at about 0.84 every time. It's Haiku 5.5 without the skill that jumped from 0.300 to 0.849 between runs. So the "+55 points" was mostly one bad baseline run.

Run it once and you would have published a fake win, with a real number attached.

The rest of it

No skill clearly helped Haiku 5.5 in two or more of its three runs.
The documentation skill clearly helped Haiku 4.5 in two of three runs (+0.14, +0.19, then +0.13 that was too noisy to call). On Haiku 5.5, all three runs had too few answers to tell either way. So does it still help on the new model? I can't tell you. Neither can anyone who ran it once.
The code review skill on Haiku 4.5: run 1 said +0.34, clearly helped. Runs 2 and 3 said +0.09 and +0.02, too few answers to tell.
Four of the six model and skill pairs changed their verdict between repeats.

That last line is the whole story. A lot of "I tried skill X and Claude got way better" posts are one run. Some popular skill-eval tools run a single attempt by default. Any one-run test would have reported run 1 above as a big win.

What this does not show

One task per skill, and only 3 to 10 answers per side in each run, so "too few answers to tell" means these runs couldn't see a 0.05 gap. It does not mean the skill is useless. The Haiku 5.5 calls also ran with Claude Code's per-turn effort setting and the Haiku 4.5 calls didn't, so the two models differ in more than the model. Every answer was scored three times by Claude Opus 5. This is not a model ranking.

The skills are from Addy Osmani's agent-skills pack, and I'm not knocking them. The point is that one run can't tell you whether any skill works, good or bad.

Full report, every number and every receipt: driftproofhq.com/reports/014

So, honest question: how many times do you run a skill before you decide it works?

*I maintain Driftproof, the open-source Claude Code plugin that ran this.

A well-formed number is not a measurement: 13 defects across 7 eval tools

Driftproofhq — Tue, 06 Oct 2026 06:39:25 +0000

On 6 October, MLflow merged a fix of mine for a model-validation bug in MetricThreshold (mlflow/mlflow#26252). It is a small bug, and a good example of a broader failure mode in evaluation software: a tool can return a perfectly well-formed number or verdict that is not supported by the evidence it claims to measure.

One example

MetricThreshold can gate model promotion on relative improvement over a baseline. The implementation divided the change by the signed baseline value. When that baseline was negative, as R² can be, the sign of the relative-change calculation reversed. A worse candidate could pass while a better one failed.

Nothing crashed. No malformed value appeared. The gate produced a clean answer to the wrong measurement.

The merged fix measures change against the magnitude of the baseline and adds 26 regression cases across positive and negative baselines and both metric directions.

Where it sits

Between 14 and 26 September I read the scoring and gating code of seven open-source evaluation tools: the agent-skills eval harness and a reference gate script, NVIDIA SkillEvaluator, DSPy, LangSmith, DeepEval, Harbor and MLflow. I reproduced 13 defects, 12 of them offline, and filed each upstream with a public reproduction. Twelve carried a proposed fix. At the paper's 30 September cutoff, five fixes had merged and another had been approved. Since then, the MLflow fix has merged as well.

In every case the pattern was the same: the tool returned a result its evidence did not support. The 13 fall into six kinds.

Absent evidence scored as a measurement
A result bound to the wrong unit
A proxy credited as the act
Stale state read as current
A sign error in a threshold
A check aimed at a target the artefact never claimed

The write-up is on Zenodo: Reported, Not Measured: An Empirical Study of Measurement Defects in LLM and Agent Evaluation Tools (CC BY 4.0). It lists each case, the reproduction, the upstream link and its status at the cutoff.

The same standard applied to my own tool

I maintain an evaluation tool myself, so the paper classifies defects from Driftproof's public history under the same six kinds: a parser that converted a non-numeric judge response into a score, a badge that verified generation text but not judge text, and six false-pass cases fixed in 0.12.0. They sit in the catalogue with the same weight as the others.

Why this is not a statistics problem

A noisy measurement can be qualified: more samples, confidence intervals, an effect floor. A well-formed number that was never a valid measurement cannot be repaired that way. More runs of the MLflow gate with a negative baseline give you the same wrong answer with more confidence.

This matters because these outputs sit behind decisions: whether a model is promoted, whether a regression is accepted, whether an agent passes CI, or whether one system is reported as better than another.

The paper proposes nine fields an evaluation record should carry, each paired with the check that would read it, and maps each of the 13 defects to the field and check that could have exposed it. That mapping is a design argument. It has not been tested, and the paper says so.

Try the idea on one of your own skills

Driftproof 0.14.0 also shipped today. If you already have a Claude Code skill, /driftproof:start drafts test cases from the skill with you, writes them only after you approve, and runs a short smoke test. The smoke run is deliberately non-conclusive: its receipt says on its face that it cannot produce a verdict. A full run is required for a measured result.

Claude Code: /driftproof:start
CLI: npx driftproof@0.14.0
Release notes: v0.14.0

What this is not

The 13 cases are a catalogue of mechanisms found where I looked. Seven tools, chosen by my judgement, over twelve days. It is a lower bound on what exists, not a prevalence estimate, and it says nothing about tools I did not read.

The point is not that evaluation tools are uniquely unreliable. It is that evaluation software deserves the same suspicion we apply to every other measuring instrument: before asking how precise a number is, first ask whether the code measured what the number says it measured.

Disclosure: I build and maintain Driftproof (open source, Apache 2.0).

The same git skill added almost nothing to Sonnet 5.5 and took Opus 5.5 from about 0.5 to 0.86

Driftproofhq — Tue, 29 Sep 2026 16:59:57 +0000

Claude Sonnet 5.5 came out on 28 September. I started testing it at 03:53 UTC the next day: the same three agent skills and tasks I used on Opus 5.5 last week, with fresh Sonnet 5 and Opus 5.5 runs next to it, and the whole set repeated three times.

Two things came out of it. The same skill can matter a lot on one model and barely at all on another. And one run on its own would often have told a different story.

What I ran

Three public skills from addyosmani/agent-skills: code review (code-review-and-quality), git workflow (git-workflow-and-versioning) and writing docs (documentation-and-adrs). One task per skill, the same tasks as my earlier reports.

Each model answered each task several times with the skill's SKILL.md in the prompt, and several times without it. Claude Opus 5 graded every answer on a 0 to 1 scale, three times per answer. All of it ran on Claude Code 2.1.284, and I repeated the full set three times back to back with the same script and settings. That gives 27 result files (Driftproof calls them receipts), all published with the report.

What the skill added

Score with the skill minus score without it, lowest to highest across the three runs:

Skill	Sonnet 5	Sonnet 5.5	Opus 5.5
Code review	+0.085 to +0.302	+0.009 to +0.051	+0.038 to +0.190
Git workflow	-0.004 to +0.020	+0.000 to +0.013	+0.276 to +0.371
Docs	-0.085 to -0.006	+0.208 to +0.272	+0.116 to +0.322

The git row is the one that surprised me. Without the skill, Sonnet 5.5 already scored 0.85 to 0.86, and Opus 5.5 scored 0.49 to 0.58. With it, both landed at about 0.86. So the skill did almost nothing for one model and a lot for the other, and they ended up in the same place.

Docs goes the other way round: Sonnet 5.5 gained 0.21 to 0.27 in every run, while Sonnet 5 came out slightly lower with the skill than without it in every run.

Sonnet 5.5 against Opus 5.5

With the skill loaded, I couldn't find a clear difference between Sonnet 5.5 and Opus 5.5 on any of the three skills, in any of the three runs.

That isn't the same as "they're equal". Driftproof only calls a difference when the spreads of the two models' scores don't overlap and their averages are at least 0.05 apart. In 5 of those 9 comparisons there weren't enough answers to spot a 0.05 gap even if one was there.

A reader also made a fair point about the git task: Sonnet 5.5 (with or without the skill) and Opus 5.5 with it all land at 0.85 to 0.86. That looks like it could be the most this task's grading gives out. If so, a tie there means both models hit the top, not that they're equally good.

Against Sonnet 5, Sonnet 5.5 with the skill scored clearly higher on docs in 2 of 3 runs, and showed no clear difference on the other two skills.

Why three runs

Because the runs didn't agree with each other:

The code-review skill added +0.076 to Opus 5.5 in run 1, +0.038 in run 2 and +0.190 in run 3.
Opus 5.5 without the skill scored 0.831, 0.857 and then 0.713 on code review.
Sonnet 5 with the code-review skill scored 0.880, 0.881, and then 0.657 in run 3.
Sonnet 5.5 pulled clearly ahead of Sonnet 5 on docs in runs 1 and 3, but not in run 2.

Counting it up, 5 of the 9 skill-and-comparison pairs didn't get the same verdict in all three runs (here "no clear difference" and "too few answers to tell" count as different verdicts). Any one of these runs on its own would have made a confident-looking blog post.

Effort settings

I didn't set effort or thinking for any model, same as my earlier reports, so each ran on its Claude Code default. Claude Code applied a per-turn effort on every Sonnet 5.5 and Opus 5.5 call and on none of the Sonnet 5 calls. That's what you get out of the box, but it means this isn't a pinned-effort head-to-head.

What it cost

The runs went through a Claude subscription, so nothing was metered. At API list prices, the answers themselves came to an estimated $2.30 for Sonnet 5.5 across all three runs, $3.30 for Sonnet 5 and $5.74 for Opus 5.5. Grading was most of the bill: about $54 of the $65.48 total.

What this doesn't show

One task per skill, and 3 to 9 answers per model per run. It says nothing about how these models code in general, or how these skills will do on your own work. The grader is Claude Opus 5, as in my earlier reports, so these numbers compare with those and nothing else.

Full report, all 27 receipts and the raw evidence: https://driftproofhq.com/reports/013/

Disclosure: I build and maintain Driftproof (open source, Apache 2.0).

Opus 5.5 shipped at 16:31 UTC. Within 15 hours, we had receipts on whether three agent skills still help.

Driftproofhq — Thu, 24 Sep 2026 19:45:44 +0000

Anthropic released Claude Opus 5.5 at 16:31 UTC on 22 September. By 07:08 UTC the next morning, Driftproof had six sealed receipts answering the question every skill author has on a release day: does my skill still help on the new model?

What we ran

Three public skills from addyosmani/agent-skills (code-review-and-quality, git-workflow-and-versioning, documentation-and-adrs), one case each, the same cases as our Report 009. Each skill ran with and without its SKILL.md, on Opus 5.5 and on a fresh Opus 5 arm beside it. Same runner, same judge, same grading template, and the same SKILL.md bytes and suites, pinned by hash in every receipt.

The first surprise: the old Claude Code could not call the new model

Report 009 ran on Claude Code 2.1.272. That version stopped at its first call to Opus 5.5 with an error asking for 2.1.280 or newer. So on release day, a model upgrade was also a runtime upgrade. We ran both models on 2.1.280, which keeps the runtime fixed when comparing models, and published Opus 5 on the old and new versions as a separate table.

What it found

With the skill, the two models did not separate on any of the three skills: +0.009, -0.023 and -0.143 on a 0 to 1 scale. On two of the three, these draws could not have told either way. For git-workflow-and-versioning, the runner puts the draws needed at 16 per arm, against the 3 we drew. For documentation-and-adrs, the spread between draws is wide enough that no number of draws at that spread would resolve it. The report says which is which, because "no separation" and "nothing changed" are different claims.

What moved instead

Without the skill, Opus 5.5 scored well below Opus 5 on two tasks. Its commit messages dropped the conventional fix: prefix (0.300 against 0.765), and in two of its four code reviews, some findings sat under category headings with no severity label (0.680 against 0.862). So on those two skills, the skill's lift grew on the new model; on documentation-and-adrs it shrank. With one case per skill, that is an observation to test, not a finding.

Is the judge biased toward its own model?

Every draw was judged by Opus 5, the same family as one of the two models. So we re-graded every answer with Opus 5.5 as the judge, published as Amendment 1 on the report. No verdict changed. 50 of 51 answers stayed on the same side of the pass line, with a mean absolute difference of 0.029 from the original grades, against 0.013 when Opus 5 graded the same answers a second time. On average Opus 5.5 scored its own model's answers 0.020 lower than the control did, and Opus 5's 0.012 lower; skill by skill, the differences are small and point both ways.

What it cost

204 model calls, 0.00 dollars metered (Claude Code on a subscription), about 10 dollars at API prices.

What this is not

Three cases and few draws is a release-day check, not a verdict on Opus 5.5. Every figure in the report is read from a receipt you can validate yourself with npx driftproof validate <file>.

Full report, receipts and raw evidence: https://driftproofhq.com/reports/011/

Disclosure: I build and maintain Driftproof (open source, Apache 2.0).

The biggest improvement in my skill evaluation came from a skill that was never invoked

Driftproofhq — Sun, 20 Sep 2026 17:59:55 +0000

Last week I measured three agent skills twice over: once with Claude Code's built-in claude plugin eval, and once with a runner I maintain. Same skill text, same task prompts, same rubrics.

I expected the interesting part to be which tool scored higher. It wasn't. The interesting part was a result that looked excellent and turned out to be unattributable.

The setup

Three skills from the public addyosmani/agent-skills pack: code review and quality, git workflow and versioning, documentation and ADRs. One case each, every case run twice, once with the skill and once without.

The two tools do not do the same thing, and that matters for everything below.

The built-in eval installs the skill as a real plugin. The model has to discover it and call it through the Skill tool, the task runs with tools in a workspace, and each run is graded pass or fail. Three runs per arm.

My runner puts the SKILL.md text straight into context, so exposure is guaranteed, and scores the answer continuously from 0 to 1 across several draws per arm. It does not test discovery at all.

So one measures discovery plus application plus execution, and the other measures application given guaranteed exposure. Target model and judge were claude-opus-5 in both.

The result that started this

Documentation and ADRs produced the largest apparent benefit in the whole run. Three of three tasks passed with the plugin, one of three without it.

Then I read the traces. The Skill tool was not called in any of those three runs. I added a supplementary run to be sure. It was not called there either. Zero of three, and zero of one.

Something differed between the two arms. The evidence does not say the skill is what differed. With three runs, two failures in the baseline arm are enough to produce a result that reads like a large effect, and nothing in the record ties the passing arm's behaviour to the skill firing.

That is not the built-in eval being wrong. Its output simply cannot support the causal reading, which is a different problem and a more common one.

The opposite failure, one skill down

Code review and quality showed no benefit at all: three of three passing in both arms, a delta of exactly zero.

Scored continuously on the same inputs, the same skill read 0.918 with it against 0.783 without.

Both arms had cleared the pass threshold, so pass/fail had nothing left to report. A zero delta there means the test could not see a difference, not that there wasn't one.

A third way to pass while being wrong

Separately I ran the whole plugin over three tool-using tasks: review a working tree change, make atomic commits, write an ADR in the repository's own convention. Three runs per arm.

Every session passed. Eighteen of eighteen against the native graders, nine with the plugin and nine without, and eighteen of eighteen against post-session verification.

One of the ADRs passed every structural check while asserting a history the fixture never supplied. The file existed, the format was right, the tools had been used, every deterministic check was green, and part of the content was invented.

What this does not prove

One case per skill, so these are effects on those cases and not on the skills in general.

Three runs per arm in the native tool, so a single run moves a pass rate by 33 points.

The two tools measure different things, and I do not compare their numbers with each other.

The ± figures from my runner are the sample standard deviation across draws, a descriptive spread with no coverage probability. Nothing here is a significance claim.

The same model generated and judged, which I did to match the native setup and which carries an obvious self-preference risk.

None of that makes the findings useless. It makes them exploratory, which is what one case per skill can support.

What I check now

Two things, before reading any with-and-without number.

Was the skill invoked at all in the with arm? If activation is not recorded anywhere, the difference is not attributable to the skill, whatever its size.

Did both arms sit at the ceiling? If everything passed either way, the task was not hard enough to answer the question being asked.

Neither check needs my tooling. Both are readable from what the built-in eval already produces.

Someone else hit this from the other side

While I was writing this up, the Superpowers plugin shipped a rebuild of its executing-plans skill after measuring the old one as, in the maintainer's words, "indistinguishable from running with no plugin at all": same tokens, same duration, same shipped bug, every repetition. The measurement is in the pull request. A maintainer publishing a negative result about their own skill is rarer than it should be.

OpenAI's developer post on rethinking skills and prompts for GPT-6 Astra makes a related point from the model side: guidance written for an older model can cost you rather than help you, so a release is a reason to recheck what you already shipped.

The artefacts

Both tools' raw output, every CLI call and the receipt-to-output hash audit are published with the write-up: driftproofhq.com/reports/009

Happy to answer methodology questions.

Disclosure: the continuous runner is Driftproof, which I build and maintain (open source, Apache 2.0). The built-in eval is Anthropic's.