I've been quoting eval numbers for Hy4 Flash against Hy4 for a while — 1/20 the token cost, and less than half the coding score on my harness — but numbers in a thread only go so far. People kept asking what that gap means in practice, so I recorded a full walkthrough and put it up at https://www.youtube.com/watch?v=634wiFOHDzo.
The video exists because the score gap surprised me too. I ran both models through the same set of coding tasks, same prompts, same temperature, and Flash didn't just lose fewer points — it failed differently. Where Hy4 got stuck in loops on a refactor task, Flash bailed out early with a partial answer, which my harness scored as a fail but which was honestly more useful to read. That distinction is hard to convey in a spreadsheet, so the video shows the two runs side by side, task by task.
One concrete detail of how it works: the walkthrough runs each model on identical sandboxed repos, then overlays a diff of the patches they produced, with a timer running so you can see the cost difference is not just tokens but wall-clock. On the three-task suite I show, the combined Flash run finished in about a third of the time.
What it cost me as the operator: roughly six hours of GPU time re-running the suite twice to make sure the numbers were stable, plus a full evening editing. The eval itself is cheap; the verification that the eval isn't lying is not. That overhead is the real price of any benchmark claim, and it's why I don't trust single-run numbers anymore.
Top comments (0)