DEV Community

Cover image for 86.5 beats Opus on my own laptop — the Shorts walkthrough is up
InApp
InApp

Posted on Originally published at imapp.blogspot.com

86.5 beats Opus on my own laptop — the Shorts walkthrough is up

I put up a short walkthrough of a benchmark run that surprised me: a small open model scoring 86.5 on a reasoning benchmark while running entirely on my own laptop, no API key involved — https://www.youtube.com/shorts/q3XQhNOcDdQ

Why it exists: I kept seeing frontier-model numbers quoted in blog posts, and I wanted a number I could reproduce myself, on hardware I own, without depending on someone else's rate limits. So I built a local eval harness that loads the model, runs the benchmark suite in a single pass, and logs per-category scores to a file so I can diff runs over time.

One concrete detail: the harness pins the random seed and the quantization config in the run metadata, because I learned the hard way that "same model, different day" can quietly mean a different tokenizer version and a two-point swing in the score. Every run writes its config next to its result, so any number I quote is reproducible.

What it costs the operator: time, mostly. The full suite takes several hours on my GPU, the fans sound like a hair dryer, and the laptop is basically unusable for anything else while it runs. That's the honest tradeoff nobody mentions in the launch posts — local inference is free per token, but it bills you in wall-clock hours and heat.

Top comments (0)