DEV Community: chpomob

I Benchmarked Two Local LLMs on Real Dev Work — Qwopus 27B vs Muse Glimmer 30B

chpomob — Tue, 11 Aug 2026 15:22:01 +0000

I Benchmarked Two Local LLMs on Real Dev Work — Qwopus 27B vs Muse Glimmer 30B

Two open-weight models, one 20 GB GPU, two real development tasks, and a third model as the referee. Here is what actually happened when I made Qwopus 3.6 27B and Meta's Muse Glimmer 30B implement a bug fix and then a full feature in my own project.

The setup

Both models ran fully local on an AMD Radeon RX 7900 XT (20 GB VRAM) via a llama.cpp multi-model router (one OpenAI-compatible endpoint, GGUF models, load-mode=dio — more on why below). Each model was driven by the pi CLI in non-interactive mode with --thinking high. A third model — Codex, through a disciplined stdin wrapper — reviewed both outputs and gave the verdict.

The fairness method was simple but strict:

One task, described in a markdown spec, copied byte-identical into two isolated git clones of my project.
Each model worked in its own clone, its own branch, never seeing the other's work.
Objective verification by script: existing test suite + new tests + production build.
Cross-review by Codex, examining both branches against the same criteria.

The test project: Jeu de Cochons (a "Pass the Pigs" dice game, vanilla JS PWA on Vite + Vitest) — real code, real tests, no toy repo.

	Qwopus 3.6 27B	Muse Glimmer 30B
Source	Community fine-tune of Qwen 3.6	Meta (distilled from Muse Spark)
Size	27B	29.6B
Quant	IQ4_XS (~15 GB)	UD-Q4_K_XL (~14.8 GB)

Round 1 — fixing a regression (short task)

The project had a broken PWA: a commit that added a /jeu-de-cochons/ base path for GitHub Pages had broken 3 service-worker tests (manifest, precache, offline navigation fallback). Task: fix the regression without touching the tests, keep the other 84 green.

	Qwopus	Muse
PWA tests (11)	11/11 ✅	11/11 ✅
Full suite (87)	87/87 ✅	87/87 ✅
Files touched	2	2
Diff size	+4/−4	+4/−4
Wall time	~8.5 min	~21 min
Leftover artifacts	none	one `.bak` file

The remarkable result: both models produced a byte-identical diff. Same diagnosis (a lost capture group in the asset-discovery regex that crashed precache), same manifest fix, same minimal solution. Codex called it a perfect tie on code — but rejected both on a latent conflict neither model noticed: reverting the manifest to / breaks the /jeu-de-cochons/ production deployment (404 icons). The tests hardcode the root path; the deployment lives under a subpath. That contradiction was visible in the task itself, and only the reviewer caught it.

Lesson #1: on a short, well-bounded task, the two models are interchangeable. Muse thought 2.5× longer and produced the same bytes. You need a richer task to separate them.

Round 2 — a full feature (solo mode vs AI)

Add a single-player mode against an AI opponent: menu selection, bot identity in game state, a thoughtful decision strategy (a pure function choosing to re-roll or bank based on turn score and game situation), automatic bot turns with human controls disabled, unit + integration tests, no regressions, build passing.

	Qwopus	Muse
Existing tests	84 ✅	84 ✅
New tests written	18 (12 unit + 6 integration)	7
Total green	102/105	91/94
Build	✅	✅
Wall time	~32 min	~26 min
Leftover artifacts	none	uncommitted `dist/`

The AI strategies they designed:

Qwopus — adaptive threshold:

Bank at 25 pts by default (justified by ~78% rewarding rolls)
Bank at 15 pts when leading (play it safe, protect the advantage)
Bank immediately when the turn wins the game
Bot identity: explicit isBot flag on the player

Muse — fixed threshold:

Constant 15 pts, bank immediately when winning is reachable
Cleanly handles ROLLING/DECIDING phases and eliminated players
Bot identity: inferred from the display name ("Ordinateur")

Codex verdict:

Criterion	Qwopus	Muse
AI strategy (primary)	8/10	5/10
Architecture	8/10	6/10
Robustness	7/10	5/10
Minimalism	6/10	7/10
Tests	8/10	4/10
Verdict	CONDITIONAL	CONDITIONAL

Winner: Qwopus.

What actually separated them:

The strategy. Qwopus exploited the game situation (adaptive threshold, conservative when ahead). Muse used a fixed, barely-justified threshold — even though the spec explicitly asked for a thoughtful strategy.
Bot modeling. Qwopus passed an explicit flag. Muse derived isBot from the localized name — meaning a human player named "Ordinateur" in multiplayer would silently become an AI. A real bug, confirmed in the code.
The tests. Qwopus covered both sides of the thresholds, victory, leading, trailing, determinism (18 tests). Muse wrote 7 — and one of them, titled "returns null for non-bot player", actually asserts the opposite (not.toBeNull()), with a comment documenting the contradiction.
Robustness. Muse's deferred callback doesn't re-validate game state, game-over, or the current player before acting (risk of an exception after returning to the menu). Qwopus re-validates everything when the timer fires.

Qwopus's own flaws, to be fair: its solo menu requires filling two human-name fields before swapping the second for the bot (blocking if you only fill one), the "~78%" claim had no math behind it, and its bot timer isn't cleared on menu exit (guarded, but not cleaned up).

What I learned

	Round 1 (short fix)	Round 2 (feature)
Winner	Tie	Qwopus
Deciding factor	none (identical diffs)	strategy + tests + robustness
Time	Qwopus 2.5× faster	Qwopus slightly slower (but more code)

On short tasks, these models are interchangeable. Raw "reasoning power" doesn't show in a well-bounded bug fix.
On a full feature, the difference lands exactly where you'd hope: domain understanding (the strategy), modeling rigor (explicit flag vs name-matching), and test discipline.
The reviewer model earns its place. Codex caught the latent manifest/base-path conflict both models missed in round 1, and the real bugs in round 2 (name-derived bot, inverted test, non-defensive callback).
Thinking longer doesn't predict quality. Muse spent 2.5× more time in round 1 for identical output, and 20% less time in round 2 for a worse result.
The harness matters as much as the models. Isolated clones, identical specs, existing tests as a non-negotiable contract, and a third-party reviewer are what make the comparison honest and reproducible.

Practical notes from the trenches

A speculative-decoding drafter added ~80 s to every model load in my multi-model setup — for a router that swaps models, dropping it was strictly better.
Large GGUFs froze on load with the ROCm backend (known issue #19482, an amdgpu SVM bug). The fix that worked: --load-mode dio (direct I/O), which both avoids the bug and loads 3–8× faster.

Reproduce it

Project: github.com/chpomob/jeu-de-cochons (Pass the Pigs PWA, Vite + Vitest)
Serve: llama.cpp in router mode (one OpenAI-compatible endpoint, multiple GGUF models)
Agents: pi CLI (pi -p --provider llama-router --model <model> --thinking high "<task>")
Reviewer: Codex via a disciplined stdin wrapper
Method: 2 git clones → identical spec → each implements → verification script (tests + build) → cross-review

Honesty clauses: this is a single run per model per task, not a statistical benchmark. Both tasks came from one small project, and the Codex review is one reviewer's judgment. The numbers are real and were verified against the artifacts, but treat them as a data point — not a ranking of these models in general.

A friend dared me to build Pass the Pigs during the apéro. Two LLMs reviewed my plan.

chpomob — Mon, 03 Aug 2026 19:47:02 +0000

A friend dared me to build Pass the Pigs during the apéro. Two LLMs reviewed my plan.

Or: a non-technical friend wanted to see what AI could do. He watched it plan — and watched two AIs shred the plan.

The challenge

It happened at an apéritif, in front of a non-technical friend who wanted to see what AI could do. We were playing le jeu de cochons (Pass the Pigs) — the apéro game where you throw two tiny pigs, they land in one of seven positions, and you push your luck against the scoreboard. He picked it up, turned to me, and dared: "go on — make an app out of it. Right now. Show me what AI can do."

The desktop wakes up. Challenge accepted. It looks trivial. It is not.

My concept was "one-line to code": give an LLM a short prompt, let it plan, let it build, ship it. A modern party trick.

Honesty clause #1: it wasn't literally one line. It took a few short exchanges — still almost no hand-holding from me, but I won't pretend it was a single prompt. The pigs are simple, right?

The prompts (zero engineering)

The prompts were apéro-grade: casual, naive, written in French over drinks — no templates, no role-play framing, no few-shot examples, no chain-of-thought coaxing. Translated from the originals:

The spec ask: "I'd like you to analyze this game — https://fr.wikipedia.org/wiki/Jeu_de_cochons — and think about how to turn it into a game. Don't code anything: just analyze the rules and define what would need to be done, in product-definition mode." — the entire game description was one Wikipedia link; the model read the rules itself.
The plan ask: "I like approach B, detail it, prepare an initial dev plan to get a playable game, bug-free, tested and robust, that reproduces the base game's experience as faithfully as possible… give me that plan so I can have it reviewed. If the review is bad, I'll have a bad image of deepseek pro." — my entire incentive system, in one sentence.
The review ask: "Do an adversarial review of this plan, with Codex as the lead and Claude as secondary, with 2 loops."

That's the whole prompt-engineering budget: zero. No magic incantations — the quality came from the process: two independent models, a gate at every stage, disagreement as a signal. Anyone can type these three prompts.

The catch

Pass the Pigs has a famously ambiguous rulebook, and the game is really about probabilities — when to bank, when to roll again. The scoring semantics (Bon Jambon, Cochon à Cheval, Pig Out, the "Somme" rules) are exactly where a happy few-lines plan goes to die.

So instead of trusting the plan those few prompts produced, I ran it through an adversarial review pipeline, orchestrated by Hermes Agent — the framework where these skills live. My Hermes runs on DeepSeek Flash for orchestration: the cheap part of the loop. The real work is done by the two specialist models — the plan was written by Codex as the architect, then reviewed independently by Claude Fable 5 as the inspector; a synthesis pass merged, categorized, and ranked their findings.

The cost story is deliberately mundane: the whole loop ran on basic consumer plans — a €20/month Claude subscription and a €20/month Codex subscription, each model's turn scheduled inside its quota window. No API credits, no enterprise accounts. The pipeline is designed to fit inside the limits a hobbyist already pays for.

Neither model is special, either: the pipeline accepts any LLM CLI on either side — Claude, Codex, Gemini, GLM, or fully local models served by llama.cpp (my Hermes already routes to local models). The only real requirement is two different model families: two copies of the same model share the same blind spots, and the whole point is a second opinion that isn't a copy of your first one.

The numbers

Reviewer	Findings
Codex alone (architect)	7
Claude Fable 5 alone (inspector)	11
Both, merged	16 unique findings

3 findings were found independently by both models — the highest-confidence class. All three are major bugs in the probability model.
6 more reached consensus after discussion; 4 were partial (agreement on the issue, disagreement on severity); 3 were genuinely disputed.

The duo found +5 findings over the best single model (16 vs 11), and — more important than the count — the two models independently converged on exactly the three most dangerous bugs.

The timeline (one apéro)

Everything started at the apéritif and finished the next morning — the session log tells the story: dare at 21:12, plan and adversarial review that night, then a quota wall (one of the two subscriptions ran out mid-loop), and I went to bed while the pipeline kept running. A playable build was committed at 01:38; the real finish came in the morning, when the GitHub Pages URL needed one more fix at 10:32 (the classic Vite base-path 404) — so the first people to try the link that morning saw nothing. Five commits across an evening, a night, and a morning. The commit for the playable version reads, in part:

Moteur de règles complet (push-your-luck, probabilités calibrées) … 73 tests (unitaires + calibration 1M tirages + Playwright E2E) — Build Vite ~16KB gzip. Développé via adversarial dev loop (Codex DEV + Claude Fable 5 REVIEW)

An apéritif game that fits in 16KB gzipped, tested 73 ways, calibrated against a million simulated rolls — and the project's own commit history credits the adversarial loop. The full review trail (647-line plan, both models' findings, synthesis) is committed right next to the code.

The full chain (spec → plan → code)

The plan review was one gate in a chain. The session opened with a spec pass — the product-definition analysis of the game's rules (scoring table, the three possible approaches, the four traps of going digital) — and everything downstream ran on it. After the plan was corrected, the code itself went through the same two-model loop, phase by phase (that's what adversarial-code-loop is for). The last phase's review caught two real major bugs the test suite had missed — in fact, the tests were asserting the buggy behavior: player names were HTML-escaped at the wrong layer (a name like A&B <Bob> was stored and displayed as A&B <Bob>), and the localStorage high-score loader trusted unvalidated names. Both fixed, the tests corrected, all 73 green.

Spec → plan → code: every stage is a gate where two independent models disagree in your place. That's the chaining the skills encode — adversarial-spec drafts the stage, adversarial-plan gates it, adversarial-code-loop keeps the code honest.

The three findings both models caught

The flank probabilities don't add up. The plan assigned 12.25% to each of two flank outcomes — but the two outcomes together total 12.25%, not 24.5%. A player's whole luck-push strategy was built on a doubled probability.
The override rule silently kills a result. Sequential checks for Bon Jambon / Cochon à Cheval overrides change the effective distribution — Cochon à Cheval ends up at 0.78% unless modeled as a disjoint probability partition.
The Jambon rule contradicts itself. One section of the plan treats Bon Jambon as a turn-ending catastrophe; another treats it as a negative score clamped with Math.max(0, score). The game's most exciting rule was unimplementable as specified.

A single model can flag these. A single model cannot confirm them. Two models converging on the same bug is the closest thing to a second opinion you can get from an LLM — and it cost nothing but one extra review pass.

Honesty clause #2: the review made the probabilities internally consistent — it did not make them true. The corrected values still come from the models' reasoning, not from rolling actual pigs. The tests do run a million-roll calibration, but those are simulated rolls: they stress-test the code, not the physics of real pigs. I am genuinely not sure the produced probabilities are right; they need calibration against real throwing data before I'd trust the strategy advice the game displays.

The part nobody talks about: the disputes

The adversarial setup does something a solo review can't: it produces disagreement. Three findings ended up disputed — e.g., "can nextPlayer() loop forever if everyone is eliminated?" — where the reviewer called it a real bug and the architect showed it's unreachable under valid transitions.

That dispute signal is precious: it tells you exactly which parts of your plan are genuinely ambiguous and need a human decision, instead of silently shipping your assumptions.

Honesty clause #3

More findings ≠ better, if they're noise. The 16 findings were categorized precisely because of that: 3 cross-validated, 6 consensus, 4 partial, 3 disputed. The 13 that survived discussion were real, and the plan (v1.1) was corrected around them — probability model reworked, scoring made explicit, state transitions centralized — before a single line of game code was written.

The apéritif game went from dare to playable build by the end of the night — built from a plan that two models shredded, argued about, and rebuilt. That's the "few prompts to code" I actually want: a couple of short exchanges in, a reviewed, correct plan out.

Tools: adversarial-code-loop (build → review → fix pipeline, model-agnostic, one-line installer), adversarial-plan, adversarial-spec — all orchestrated by Hermes Agent. The game — play it, read the code — plan v1.1 after review, full review artifacts in review/.