Anthropic released Claude Opus 5.5 at 16:31 UTC on 22 September. By 07:08 UTC the next morning, Driftproof had six sealed receipts answering the question every skill author has on a release day: does my skill still help on the new model?
What we ran
Three public skills from addyosmani/agent-skills (code-review-and-quality, git-workflow-and-versioning, documentation-and-adrs), one case each, the same cases as our Report 009. Each skill ran with and without its SKILL.md, on Opus 5.5 and on a fresh Opus 5 arm beside it. Same runner, same judge, same grading template, and the same SKILL.md bytes and suites, pinned by hash in every receipt.
The first surprise: the old Claude Code could not call the new model
Report 009 ran on Claude Code 2.1.272. That version stopped at its first call to Opus 5.5 with an error asking for 2.1.280 or newer. So on release day, a model upgrade was also a runtime upgrade. We ran both models on 2.1.280, which keeps the runtime fixed when comparing models, and published Opus 5 on the old and new versions as a separate table.
What it found
With the skill, the two models did not separate on any of the three skills: +0.009, -0.023 and -0.143 on a 0 to 1 scale. On two of the three, these draws could not have told either way. For git-workflow-and-versioning, the runner puts the draws needed at 16 per arm, against the 3 we drew. For documentation-and-adrs, the spread between draws is wide enough that no number of draws at that spread would resolve it. The report says which is which, because "no separation" and "nothing changed" are different claims.
What moved instead
Without the skill, Opus 5.5 scored well below Opus 5 on two tasks. Its commit messages dropped the conventional fix: prefix (0.300 against 0.765), and in two of its four code reviews, some findings sat under category headings with no severity label (0.680 against 0.862). So on those two skills, the skill's lift grew on the new model; on documentation-and-adrs it shrank. With one case per skill, that is an observation to test, not a finding.
Is the judge biased toward its own model?
Every draw was judged by Opus 5, the same family as one of the two models. We have re-graded every answer with Opus 5.5 as the judge. The results go up as an amendment to the report with our next release, and I will add them here when they do.
What it cost
204 model calls, 0.00 dollars metered (Claude Code on a subscription), about 10 dollars at API prices.
What this is not
Three cases and few draws is a release-day check, not a verdict on Opus 5.5. Every figure in the report is read from a receipt you can validate yourself with npx driftproof validate <file>.
Full report, receipts and raw evidence: https://driftproofhq.com/reports/011/
Disclosure: I build and maintain Driftproof (open source, Apache 2.0).
Top comments (0)