I spent last week trying to explain that parameter count stopped being the whole story. A 65B model I run locally scored 80.52 on our eval suite; a 501B frontier model scored lower on the same run. The number alone sounds like a typo, so I filmed a walkthrough instead of arguing in comments.
The Short exists because the gap is real but invisible without seeing the harness. The concrete detail I show on screen: both models got the exact same 200-question multiple-choice set, temperature 0, no system-prompt changes. The only difference was inference path — the 65B ran quantized at Q4_K_M on a single consumer GPU while the 501B ran through its hosted API with defaults. Same questions, same scoring script, and the smaller model won 14 of 20 categories.
What it costs the operator: reproducing the 501B side isn't free. Each full run through that API burns about twelve dollars in tokens, so anyone verifying the number has to pay the toll again. The local 65B run costs electricity and roughly forty minutes of GPU time — the trade I actually prefer.
I packaged the walkthrough as https://www.youtube.com/shorts/niEWIgSUopU
Top comments (0)