I just published a new video walkthrough breaking down my head-to-head between Qwen4-Max (2.4T params) and Claude on the same eval harness, and the honest headline is that the bigger, cheaper model still loses on the tasks I care about: https://www.youtube.com/watch?v=murtytw20Jo
Why it exists: I kept getting asked in DMs why my harness scored Qwen4-Max lower than its size suggests, and screenshots of a spreadsheet don't answer that. A video lets people see the failure modes — the verbose reasoning traces, the cases where it answers confidently and wrong — instead of trusting a number.
One concrete detail: every run in the video uses the same fixed seed and the same 200-prompt slice of my harness, so the comparison is apples-to-apples. The detail that surprised me most while editing: on 31 of the 200 prompts, Qwen4-Max produced a longer answer than Claude and was more wrong, which is a worse failure shape than a terse wrong answer — it burns context and confidence budget downstream.
What it costs me to run this: each full walkthrough costs about 40 minutes of operator time to cut, verify the numbers on screen match the harness logs, and re-render when I catch a stale chart. The eval runs themselves are cheap; the verification is the expensive part. I'd rather spend that hour than publish a number I can't defend.
If you're benchmarking open-weight models against frontier ones, the takeaway isn't "big model bad" — it's that param count doesn't predict instruction-following quality on my task slice, and you should measure on your own prompts before you switch.
Top comments (0)