I Benchmarked Two Local LLMs on Real Dev Work — Qwopus 27B vs Muse Glimmer 30B

Two open-weight models, one 20 GB GPU, two real development tasks, and a third model as the referee. Here is what actually happened when I made Qwopus 3.6 27B and Meta's Muse Glimmer 30B implement a bug fix and then a full feature in my own project.

The setup

Both models ran fully local on an AMD Radeon RX 7900 XT (20 GB VRAM) via a llama.cpp multi-model router (one OpenAI-compatible endpoint, GGUF models, load-mode=dio — more on why below). Each model was driven by the pi CLI in non-interactive mode with --thinking high. A third model — Codex, through a disciplined stdin wrapper — reviewed both outputs and gave the verdict.

The fairness method was simple but strict:

1. One task, described in a markdown spec, copied byte-identical into two isolated git clones of my project.
2. Each model worked in its own clone, its own branch, never seeing the other's work.
3. Objective verification by script: existing test suite + new tests + production build.
4. Cross-review by Codex, examining both branches against the same criteria.

The test project: Jeu de Cochons (a "Pass the Pigs" dice game, vanilla JS PWA on Vite + Vitest) — real code, real tests, no toy repo.

	Qwopus 3.6 27B	Muse Glimmer 30B
Source	Community fine-tune of Qwen 3.6	Meta (distilled from Muse Spark)
Size	27B	29.6B
Quant	IQ4_XS (~15 GB)	UD-Q4_K_XL (~14.8 GB)

Round 1 — fixing a regression (short task)

The project had a broken PWA: a commit that added a /jeu-de-cochons/ base path for GitHub Pages had broken 3 service-worker tests (manifest, precache, offline navigation fallback). Task: fix the regression without touching the tests, keep the other 84 green.

	Qwopus	Muse
PWA tests (11)	11/11 ✅	11/11 ✅
Full suite (87)	87/87 ✅	87/87 ✅
Files touched	2	2
Diff size	+4/−4	+4/−4
Wall time	~8.5 min	~21 min
Leftover artifacts	none	one .bak file

The remarkable result: both models produced a byte-identical diff. Same diagnosis (a lost capture group in the asset-discovery regex that crashed precache), same manifest fix, same minimal solution. Codex called it a perfect tie on code — but rejected both on a latent conflict neither model noticed: reverting the manifest to / breaks the /jeu-de-cochons/ production deployment (404 icons). The tests hardcode the root path; the deployment lives under a subpath. That contradiction was visible in the task itself, and only the reviewer caught it.

Lesson #1: on a short, well-bounded task, the two models are interchangeable. Muse thought 2.5× longer and produced the same bytes. You need a richer task to separate them.

Round 2 — a full feature (solo mode vs AI)

Add a single-player mode against an AI opponent: menu selection, bot identity in game state, a thoughtful decision strategy (a pure function choosing to re-roll or bank based on turn score and game situation), automatic bot turns with human controls disabled, unit + integration tests, no regressions, build passing.

	Qwopus	Muse
Existing tests	84 ✅	84 ✅
New tests written	18 (12 unit + 6 integration)	7
Total green	102/105	91/94
Build	✅	✅
Wall time	~32 min	~26 min
Leftover artifacts	none	uncommitted dist/

The AI strategies they designed:

Qwopus — adaptive threshold:

• Bank at 25 pts by default (justified by ~78% rewarding rolls)
• Bank at 15 pts when leading (play it safe, protect the advantage)
• Bank immediately when the turn wins the game
• Bot identity: explicit isBot flag on the player

Muse — fixed threshold:

• Constant 15 pts, bank immediately when winning is reachable
• Cleanly handles ROLLING/DECIDING phases and eliminated players
• Bot identity: inferred from the display name ("Ordinateur")

Codex verdict:

Criterion	Qwopus	Muse
AI strategy (primary)	8/10	5/10
Architecture	8/10	6/10
Robustness	7/10	5/10
Minimalism	6/10	7/10
Tests	8/10	4/10
Verdict	CONDITIONAL	CONDITIONAL

Winner: Qwopus.

What actually separated them:

1. The strategy. Qwopus exploited the game situation (adaptive threshold, conservative when ahead). Muse used a fixed, barely-justified threshold — even though the spec explicitly asked for a thoughtful strategy.
2. Bot modeling. Qwopus passed an explicit flag. Muse derived isBot from the localized name — meaning a human player named "Ordinateur" in multiplayer would silently become an AI. A real bug, confirmed in the code.
3. The tests. Qwopus covered both sides of the thresholds, victory, leading, trailing, determinism (18 tests). Muse wrote 7 — and one of them, titled "returns null for non-bot player", actually asserts the opposite (not.toBeNull()), with a comment documenting the contradiction.
4. Robustness. Muse's deferred callback doesn't re-validate game state, game-over, or the current player before acting (risk of an exception after returning to the menu). Qwopus re-validates everything when the timer fires.

Qwopus's own flaws, to be fair: its solo menu requires filling two human-name fields before swapping the second for the bot (blocking if you only fill one), the "~78%" claim had no math behind it, and its bot timer isn't cleared on menu exit (guarded, but not cleaned up).

What I learned

	Round 1 (short fix)	Round 2 (feature)
Winner	Tie	Qwopus
Deciding factor	none (identical diffs)	strategy + tests + robustness
Time	Qwopus 2.5× faster	Qwopus slightly slower (but more code)

1. On short tasks, these models are interchangeable. Raw "reasoning power" doesn't show in a well-bounded bug fix.
2. On a full feature, the difference lands exactly where you'd hope: domain understanding (the strategy), modeling rigor (explicit flag vs name-matching), and test discipline.
3. The reviewer model earns its place. Codex caught the latent manifest/base-path conflict both models missed in round 1, and the real bugs in round 2 (name-derived bot, inverted test, non-defensive callback).
4. Thinking longer doesn't predict quality. Muse spent 2.5× more time in round 1 for identical output, and 20% less time in round 2 for a worse result.
5. The harness matters as much as the models. Isolated clones, identical specs, existing tests as a non-negotiable contract, and a third-party reviewer are what make the comparison honest and reproducible.

Practical notes from the trenches

• A speculative-decoding drafter added ~80 s to every model load in my multi-model setup — for a router that swaps models, dropping it was strictly better.
• Large GGUFs froze on load with the ROCm backend (known issue #19482, an amdgpu SVM bug). The fix that worked: --load-mode dio (direct I/O), which both avoids the bug and loads 3–8× faster.

Reproduce it

• Project: github.com/chpomob/jeu-de-cochons (Pass the Pigs PWA, Vite + Vitest)
• Serve: llama.cpp in router mode (one OpenAI-compatible endpoint, multiple GGUF models)
• Agents: pi CLI (pi -p --provider llama-router --model <model> --thinking high "<task>")
• Reviewer: Codex via a disciplined stdin wrapper
• Method: 2 git clones → identical spec → each implements → verification script (tests + build) → cross-review

Honesty clauses: this is a single run per model per task, not a statistical benchmark. Both tasks came from one small project, and the Codex review is one reviewer's judgment. The numbers are real and were verified against the artifacts, but treat them as a data point — not a ranking of these models in general.