I put up a new video walkthrough on Hy4: it fabricates answers 89% of the time on our adversarial set — and it still ties Grok on the composite leaderboard. The whole point of the video is that these two facts are not a contradiction, they're the same story told from two angles.
Why it exists: I kept getting asked "is Hy4 good or bad?" and the honest answer was "it depends which number you read." Raw accuracy hides that Hy4 is confident-wrong in a very structured way — it commits to an answer fast and defends it. The video walks through the runs side by side so you can see the failure mode instead of just the score.
One concrete detail of how we measured it: we didn't score per-run, we scored per-claim. Every answer gets split into atomic factual claims, each graded against a retrieved source, and the 89% is the share of claims that fail verification. That distinction matters — per-run scoring would have hidden that Hy4's errors cluster in long, confident paragraphs rather than one wrong token in an otherwise careful answer.
What it cost me: re-running the full eval harness twice. Once with the harness as shipped, once after I fixed a bug where long answers were being truncated before grading and silently counted as correct. The fix moved Hy4's numbers by four points and took a weekend of GPU time I'd rather have spent elsewhere.
The walkthrough is here if you want the full run-by-run breakdown: https://www.youtube.com/watch?v=pM_Q0WXFgrA
Top comments (0)