DEV Community

Chris Gamer
Chris Gamer

Posted on

Why I built a local GPU benchmark server for 4B to 35B open weight LLMs

Whenever I looked at existing LLM leaderboards I was missing practical numbers for running quantized open weight models on affordable local hardware. Most tables only copy vendor cloud scores or run tiny subsets of a benchmark, and none of them tell you how long a model actually sits there calculating before it finishes.

Because nothing like that existed, I set up a dedicated Linux test server with two RTX 5060 Ti cards giving 32 GB of VRAM in total, running vLLM in Docker. Over the past weeks I ran 12 current local models from 4B up to 35B through complete benchmark datasets without any subsampling, including IFEval, IFBench, ARC AGI 1, SimpleQA, SimpleQA Verified, SciCode and vLLM bench serve. The lineup includes Qwen 3.5, Qwen 3.6 and Qwen 3.8 in 4B, 9B, 27B and 35B A3B, Google Gemma 4 in E4B, 26B A4B and 31B, plus GLM 4.7 Flash in 4 bit quantizations like NVFP4 and AWQ.

Every single model is tested twice across all benchmarks, once with thinking turned off and once with thinking turned on, using the official sampling parameters. While the tests run, my system tracks the exact wall clock duration and throughput in tokens per second to calculate a time efficiency score from 0 to 100 for every single cell. That turned out to be the most interesting part of the whole project. You can immediately see which models deliver strong scores in 20 or 30 minutes and which ones burn GPU power for 10 hours straight because they get stuck reasoning in circles on simple questions. Especially the MoE models like Qwen 3.6 35B A3B and Gemma 4 26B A4B surprised me by finishing in a fraction of the time while beating or matching the dense 27B and 31B models.

I put all results into a clean, sortable comparison table on a new subpage of my site so anyone with a 16 GB or 32 GB setup can check the numbers before downloading 20 GB of weights:

The site is a personal project with no ads, no tracking and no affiliate links. Right now the server is working through 6 additional benchmarks for the current 12 models, which will take about 3 to 4 weeks of nonstop 24/7 GPU time before the next batch of models starts. Feel free to take a look and let me know which local models you would like to see tested next.

Top comments (5)

Collapse
 
sikamikanikobg profile image
Arsen Apostolov •

the wall-clock dimension is what i was missing from the usual leaderboards. a 35b moe that matches a dense 31b on quality in a third of the time is a different machine to own, and the 0-100 time-efficiency score is the right way to make that visible instead of hiding it in tokens/sec. running it on two 5060 ti cards is also the right reference hardware — that is the 32gb most people actually have, not the 80gb the vendor tables assume. the "10 hours of reasoning in circles on simple questions" failure mode is the one i would put at the top of any local model shortlist: it is the kind of cost you only discover after the download, which is exactly what a pre-download table is for.

Collapse
 
llmbenchmarks profile image
Chris Gamer •

Thanks Arsen, you summed up the whole reason this project exists better than I did! Raw tokens per second really hides the biggest trap with current reasoning models. A model running at 60 tok/s that writes 8,000 tokens of circular self doubt on a simple factual question is way slower in real life than a model at 45 tok/s that answers cleanly after 400 tokens. Seeing how Qwen 3.6 35B A3B and Gemma 4 26B A4B behave on a realistic 32 GB consumer setup versus the dense 27B and 31B models was a real eye opener for me too.

Collapse
 
paul_irolla_6b5ae261d0224 profile image
Paul Irolla •

Do you record tokens per run as well as speed? Two models can rank differently on those two numbers.

Collapse
 
llmbenchmarks profile image
Chris Gamer •

Yes, the benchmark runner logs the exact input tokens, output tokens, thinking tokens and max token truncations for every single run alongside wall clock duration.
You are spot on that generation speed in tok/s and total tokens generated per run can rank models completely differently. That is actually why I included both the dedicated vLLM bench serve column (which measures pure tok/s throughput under controlled load) and the actual wall clock time plus the 0 to 100 time efficiency score in every benchmark cell.
A great real example from my logs is SimpleQA Verified (1,000 questions) with thinking turned on: Qwen 3.8 27B NVFP4 and Qwen 3.5 27B AWQ have very similar raw tok/s speed, but Qwen 3.8 27B generated 6.94 million output tokens over 8.5 hours because it kept second guessing itself in long reasoning loops, while Qwen 3.5 27B generated 4.91 million tokens in 4.5 hours. Meanwhile Gemma 4 26B A4B finished the same 1,000 questions in about 41 minutes because it combines high MoE tok/s throughput with much shorter reasoning chains. Right now the web table shows the score, duration and efficiency rating to keep the cells readable, and I am planning to expose the raw token counts in the detailed CSV export as well.

Collapse
 
paul_irolla_6b5ae261d0224 profile image
Paul Irolla •

@llmbenchmarks Our benchmark ranks models by cost per solved task, not by tok/s. Each run logs total cost, the token breakdown and num_turns, and the method is in the public THOL README.