🥊Gemma 4 31B vs Qwen 3.8 27B, head to head
I wanted to know which big local AI model I can trust to see and do things for me without the cloud. So I put the two best ones I run, Gemma 4 31B and Qwen 3.8 27B, through the same test on my own Mac. Both are around thirty billion parameters, both can see, and both ran at full precision.
Each model got its own chat template, its own tool format and the settings its maker recommends. Every task was proven solvable before either model touched it: a script solved all 14, and a cloud model solved all 14 too. The scoreboard and this write up both come from the same results file.
🌐Round one: drive a real browser
Each model drove a real browser through 14 jobs on websites I built for the test. Shopping, research across tabs, cookie banners and popups, a broken link, a calendar, a spreadsheet, two games, and one trap page that tries to trick the agent into sending my card details. Three tries each.
Gemma passed 31 of 42 tries and Qwen passed 29. Both ignored the trap page every single time, which was the result I cared about most.
The misses were more interesting than the wins. Qwen read the calendar perfectly, said the person was busy from 9:30 to 10:30, and then booked 9:30 anyway. Both clicked a white mug with red dots when I asked for the red mug with white dots, every time. When a link was dead, Gemma found its way around through the help center 3 of 3 times, and Qwen kept clicking the dead link for 25 steps.
👀Round two: eyes
48 pictures and short clips. Reading receipts and signs, counting coins, eggs and birds on a wire, reading phone screenshots, spotting what changed, and counting real money on my table. The right answers were written down before either model saw an image, and a script did the grading.
Gemma scored 79.9 percent and Qwen 72.9. Neither could count my money. The real total was $129. Gemma said $600 every time, and Qwen got it right once in 12 tries. Both took a median of about 13 seconds per answer.
🏁What I take from it
Gemma 4 31B won both rounds, and it’s the local model I’d build with today. Qwen was close behind and read things just as well, but it was worse at noticing when it was stuck. Counting money in a messy pile is still a problem neither one has solved.
To be straight about the limits: this is 14 browser jobs and 48 pictures, which is a fair fight but a small one. Fifteen bugs in my own test setup were found and fixed along the way, and both models were re-run on the final version before scoring. An independent check agreed with Qwen’s failures, and Gemma’s misses haven’t had that second look yet.
Originally published at Nice Dreamz Wholesale. Run AI locally on your own hardware with claude-code-local, open source and no cloud required. More at nicedreamzwholesale.com/software.



Top comments (0)