I published a new walkthrough yesterday: https://www.youtube.com/shorts/7swT71hR-A4 — and it exists because "89%" keeps getting quoted like a verdict when it's really a measurement with a very specific setup behind it.
The reason I filmed it: I kept seeing the number passed around as proof that a model "makes things up constantly," and that framing breaks down the moment you look at the eval. So the Short walks through exactly what produced the 89% — the prompt set, the sampling, and the fact that the score counts confident fabrication, not hesitation or hedging. Two runs on slightly different seeds moved the number by several points, which is the part nobody tweets.
One concrete detail of how it works: each response gets scored by a rubric that separates "unsupported claim" from "wrong claim" — a model can be factually correct and still score a hit if it asserts something the eval can't verify. That distinction is why it still ties Grok on this particular benchmark despite the headline rate. The tie isn't a fluke; it's the rubric rewarding different failure modes equally.
What it costs the operator: each full eval run is about 40 minutes of compute and a few hundred thousand tokens of judging, plus the manual pass I do on a sample of scored responses to sanity-check the rubric itself. That audit step is the expensive part — it's human-hours, not API calls, and it's the only thing standing between a number and a narrative.
Top comments (0)