If you ship any automated score — evals, rankings, quality gates — this is the number you should know about your own pipeline. We measured ours and it's worse than we'd like.
Every score we publish comes from a model reading a SKILL.md against a fixed rubric. We had eleven skills that got evaluated twice: identical input file, same model, same rubric, two hours apart. The second run was for an unrelated reason (a retry-logic fix), not an experiment — which is exactly what makes it useful. Nobody was watching for variance.
Seven of the eleven came back different.
The data
| Skill | First run | Second run | Δ |
|---|---|---|---|
understand-explain |
8.3 | 9.0 | +0.7 |
presentation-styling |
7.3 | 8.0 | +0.7 |
understand-figma |
8.3 | 8.7 | +0.4 |
understand-onboard |
8.0 | 8.3 | +0.3 |
understand-knowledge |
8.3 | 8.0 | −0.3 |
understand-domain |
8.3 | 8.0 | −0.3 |
presentation-structure |
8.3 | 7.3 | −1.0 |
understand, understand-diff, understand-chat, vibe-to-agentic-framework
|
— | — | 0 |
The largest swing was a full point, and it went in both directions.
What I'd conclude from this
The useful resolution is about half a point. Not 0.1. If you're comparing an 8.3 against an 8.7, you are reading noise — regardless of who produced those numbers. What survives re-runs is the band: "in the 9s" versus "in the 7s" means something. The decimal doesn't.
Publish the rationale, not just the number. A bare score gives a reader nothing to argue with. Every one of our scores ships with the written reasoning that produced it, so if you think presentation-structure deserved better than 7.3 you can read why and say so. That's the difference between a measurement and a vibe with a number attached.
Design your publication around movement, not precision. If a re-run can move a score by a point, then your UI, your API, and your headlines all need to be able to absorb that. Use bands. Never put a 0.1-point gap in a title.
The caveat that makes this not a clean experiment
Between the two runs we also changed the generation budget — the first run occasionally hit its token limit before answering. So some part of that movement could be the harness rather than the model's judgment.
I'm telling you that instead of quietly dropping it, because it's the same reason this blog post exists at all: an automated score that can't be audited invites exactly the wrong kind of trust. The variance is real — we've seen it in other re-runs — but this particular measurement is a natural experiment, not a controlled one. Treat it as a floor.
If you run evals
Three things I'd check in your own pipeline this week:
- Re-run 10–20 already-scored items with identical input. Nobody does this, and the answer is always surprising.
- Measure your own resolution floor and put it in your docs. "Scores are accurate to ±0.5" is a hundred times more honest than an unqualified number, and it costs you nothing.
- Keep the reasoning. If your eval only emits a number, you can't debug it, can't appeal it, and can't explain it when someone disagrees.
The rubric we use is public, with the weights and what each dimension means.
Top comments (0)