I ran a fresh eval last week and the headline number came back 88.64 — ahead of whatever sat on the board at the time. The temptation was to post the number and walk away. Instead I filmed a short walkthrough, because the catch lives in the same sentence as the score: it's a 5-shot greedy-decoding run on a fixed subset with a locked seed, so the 88.64 is repeatable but not generalizable.
The walkthrough exists because a bare number invites the wrong conclusion. Concretely, the harness replays the same five exemplars in the same order for every sample, decodes with temperature 0, and scores exact-match after stripping whitespace — which means one lucky formatting quirk can move the score by a point. I show the exact prompt template and the seed in the video so anyone can reproduce my number and see where it frays.
The thing it costs the operator is time, not compute: three hours of careful re-runs to confirm the score is stable across two harness versions, plus the editing pass to make the caveat as loud as the number. That's the honest tax on any benchmark claim — the video is the receipt. It's up here: https://www.youtube.com/shorts/Oda_CkwtN-M
Top comments (0)