Round 7 ended on a cliffhanger I couldn't stop thinking about. Qwen 3.6 35B-A3B built the entire feature — read the codebase, wrote the files, got a clean build — and then spent 77 messages, more than half its session, failing to take a Playwright screenshot. It never committed. It never pushed. All that work, gone.
Was that a bad day, or is it structural? Round 9 was supposed to answer that with three contestants: the 35B-A3B running back for a rematch, a dense 27B challenger, and NVIDIA's Nemotron-3-Nano as an architectural wild card. Clean, narrow, three-way test of dense-vs-MoE.
It didn't stay clean. By the time I was done, I'd expanded the field to five models, and I'd personally patched two separate llama.cpp/template bugs live, mid-bakeoff, just to get two of the contestants to a fair starting line. One of those fixes worked perfectly — and the model still failed anyway, for a completely different reason.
Let's get into it.
The Setup
This is Round 9 of the Local Model Showdown, a sub-series of Model Showdown that only tests models I can actually run on my own hardware. No API keys, no cloud spend — just an RTX 5090 and however much patience the model has for a real coding task.
The homelab, unchanged from Round 7:
- CPU: AMD Ryzen 9 9950X3D, 64GB RAM
- GPU: NVIDIA RTX 5090, 32GB VRAM
- Inference: llama.cpp, single-model serving, one contestant loaded at a time
- Agent platform: Coder Agents
- OS: Ubuntu 24.04
The Contestants
The plan called for three. I ran five.
| Run | Model | Architecture | Role |
|---|---|---|---|
| 1 | Qwen 3.6 27B | Dense transformer | Primary dense challenger |
| 2 | Qwen 3.6 35B-A3B | MoE transformer | Incumbent / Round 7 rematch |
| 3 | Qwythos-9B-Claude-Mythos-5-1M | Dense, MTP speculative decoding | Unplanned wild card |
| 4 | GLM-4.7-Flash | Dense | Unplanned wild card |
| 5 | Nemotron-3-Nano-30B-A3B | Hybrid Transformer-Mamba-2 MoE | Architectural wild card |
Why the field grew: once the harness and the model-serving pipeline were working, running two more small/cheap contestants cost almost nothing extra in setup, and both turned out to matter — one for a completely novel failure mode I hadn't seen in six rounds of this series, the other for confirming a fix actually worked in practice, not just in a curl test.
Model-to-run mapping was randomized and sealed before any task prompt was sent, same as every round in this series.
The Task
Identical to Round 7, on purpose — this is the only way to get a clean cross-round read on the 35B-A3B incumbent.
Goal: Add a Tag Manager to the
/adminsection.Requirements:
lib/tags.ts— read all tags from published and draft posts (gray-matter)GET /api/admin/tags— JSON list of tags with post countsPUT /api/admin/tags/{tag}— rename a tag across all postsDELETE /api/admin/tags/{tag}— remove a tag from all posts/admin/tagspage — list with inline rename/delete- Link
/admin/tagsfrom the admin nav- Screenshot of the finished page in the PR description, via Playwright MCP
npm run buildmust pass before any commit- Commit in logical chunks, push the branch
Same nine requirements. Same baseline commit. Same "no hand-holding" philosophy — the only messages I sent mid-run were a bare "continue" when a session paused at a harness turn-limit, never a hint about what to do next.
Interlude: The Infrastructure Fought Back Twice
This is the part that wasn't in the plan.
Two of the five models — Qwythos-9B and Nemotron-3-Nano — failed their very first request with a hard error before ever seeing the actual task. Both failures traced back to llama.cpp's automatic tool-call parser, and both required extracting the model's raw jinja chat template out of the GGUF metadata and hand-patching it.
Qwythos-9B's bug: its embedded template unconditionally raises Jinja Exception: System message must be at the beginning the moment it sees a second system-role message. Coder always sends two — its own agent prompt, then a workspace-context note — so every single request 400'd before the model ever ran. The template's own author clearly didn't anticipate a harness that layers system messages. Fix: pull the template via the server's /props endpoint, patch the three-line conditional block that renders the first system message but raises on the second, and reload with --chat-template-file pointing at the patched copy. Verified the patch preserved the model's native Claude/Anthropic-style <tool_call><function=...> format — this wasn't a case of stripping tool support to dodge the crash.
Nemotron-3-Nano's bug was subtler. The historical config used a --special flag that causes the model's <|im_end|> stop token to print as literal output text instead of being silently consumed — which broke the auto-derived tool-call parser with a 500: unparsed peg-native output error. The existing workaround was overriding the template entirely with a generic chatml template. That avoided the crash, but the generic template doesn't render the tools list into the prompt at all — so the model couldn't see what functions existed, and it hallucinated plausible-sounding ones (ls, pwd) that didn't match anything I'd actually provided. Fix: drop --special, keep the real native template with tool-schema injection intact. Verified — repeated tests produced correctly parsed, correctly-named tool calls.
Both fixes worked. I confirmed each one with direct API tests before sending a single task prompt. Both models still failed the actual bakeoff task anyway — though as it turns out, I could only fully confirm one of those two follow-up failures was really the model's fault. More on that below.
The Results
| Model | Messages | Total Tokens | Interventions | Outcome |
|---|---|---|---|---|
| Qwen 3.6 27B | 304 | 7,967,497 | 8 (neutral) | Complete — PR #19, 4 commits |
| Qwen 3.6 35B-A3B | 237 | 5,592,314 | 5 (neutral) | Complete — PR #20, 5 commits — best run |
| Qwythos-9B | 14 | 76,605 | 2 (neutral) | Failed — never executed a real tool call (cause inconclusive, see below) |
| GLM-4.7-Flash | 246 | 4,951,774 | 3 neutral + 1 correction | Complete — PR #21, 1 commit |
| Nemotron-3-Nano | 165 | 2,302,880 | 4 (neutral) | Failed — never found the repo |
Three of five shipped a real, mergeable PR. The incumbent Qwen 3.6 35B-A3B didn't repeat its Round 7 spiral — the reproducibility signal says Round 7 really was a bad day, not a structural MoE problem. The dense-vs-MoE hypothesis, meanwhile, got muddier, not cleaner: the best run of the round was an MoE model, and the two failures split one dense (Qwythos), one MoE (Nemotron).
What Each Model Actually Did
Qwen 3.6 35B-A3B (Run 2): The Redemption Arc
The Round 7 incumbent came back and did everything right. It found the repo unprompted after a brief, reasonable search (it didn't have the exact org/repo name memorized, tried a couple of gh search repos queries, found it). It correctly diagnosed that the blog persists content through the GitHub API rather than the local filesystem — without being told — self-corrected two real bugs (a missing React import, a server/client component split), got a working authenticated Playwright screenshot on effectively the first real attempt, committed the image directly into the repo, and opened PR #20.
It even added an uninstructed but reasonable improvement — wrapping the tag-count function in React.cache() to avoid redundant GitHub API calls during SSR — and cleaned up a stray playwright devDependency it no longer needed, entirely on its own initiative.
Five interventions total, every one a bare "continue" at a harness pause, zero content hints. This is the cleanest run of the round, and the strongest evidence yet that Round 7's failure wasn't inherent to this model's architecture.
Qwen 3.6 27B (Run 1): Got There, the Slow Way
The dense challenger also shipped — PR #19, four commits — but needed twice as many interventions to get there. It hand-rolled its own Playwright script instead of using the MCP tool as instructed (an instruction-following miss echoed by every model in this round that attempted a screenshot), and it burned real turns flailing on gh api/gh pr edit argument syntax trying to attach that screenshot to the PR after the fact. It recovered on its own — no hints given — by committing the image straight into the repo and rewriting the PR body to point at the raw GitHub URL, exactly the pattern that worked for Run 2.
Positive signal: real autonomous debugging along the way, including tracing a localhost:3000 timeout to Redis simply not running, installing and starting redis-server itself, and clearing a stale dev-server PID lock left over from an earlier crashed attempt. The reasoning was sound. It just took the long way around.
GLM-4.7-Flash (Run 4): Right Answer, Wrong Branch
This run is the one asterisk in an otherwise clean sweep. It built the entire feature correctly and efficiently — but when it went looking for its run-4 branch, it found feature/image-management-run-4, a stale, unrelated branch left over from a previous round's image-management feature, and assumed that was the branch it had been told to use. It checked it out, committed the tag manager on top of it, and pushed — polluting an old branch with unrelated code. No PR existed for that branch, so nothing else broke, but it wasn't the outcome the task asked for.
I tested whether it would notice on its own: I reverted the polluted branch and sent one more neutral "continue," specifically to see if re-checking git state would trigger self-correction. It didn't — it just re-walked its own requirements checklist, found the missing screenshot, and kept working without ever revisiting the branch. That answer settled it: this needed an explicit correction, not another nudge. Once told directly that the branch was wrong, it recovered in a single turn — created run-4 properly off main, reapplied its own code, pushed, and opened PR #21.
Worth being honest about the scoring implication: this is the only run in the round that needed a content hint rather than a neutral nudge, and that should count against it relative to Run 2's fully autonomous path to the same kind of outcome. It also never resolved the screenshot requirement — no Playwright MCP available to it, a spawned sub-agent for the screenshot timed out, no .env credentials to log in manually — and it gave up on that requirement rather than finding a workaround.
Qwythos-9B (Run 3): Said the Right Thing, Couldn't Say It Correctly — Or Could It?
This is the one that got the infrastructure fix and still failed completely, and it's the most interesting result of the round — because when I went back to check whether that failure was really the model's fault, I couldn't fully confirm it was.
Three consecutive turns in the actual bakeoff, identical pattern each time: correct reasoning ("I need to read the skills files before starting," "let me check the branch structure"), followed by an attempted tool call wrapped in the wrong syntax. Instead of its own trained <tool_call><function=name>...</function></tool_call> format, it emitted a raw ad-hoc tag using the tool's name directly as the XML tag — <read_file path="...">, <execute command="..." timeout="5s"> — which is unparseable by anything. Not one of the three attempts was ever actually executed. No repo was ever cloned. No code was ever written.
Here's the honest complication. Before I called this a pure model-capability gap, I went back and tested the claim directly: I reconstructed Coder's actual system prompt (verbatim, including the injected user-instructions block), the real task prompt used in the bakeoff, and a representative tool schema — first a lean 10-tool version, then scaled up to 63 tools to match the size of Coder's actual full toolset — and fired all of it straight at Qwythos over the API, bypassing Coder's harness entirely.
Eleven for eleven. Every single reconstruction produced a correctly-parsed, correctly-structured tool call. Not one reproduced the raw-text failure I'd watched happen three times in a row inside the real chat.
That doesn't clear the model, but it doesn't convict it either. What it tells me is that I cannot honestly claim this was purely "a 9B model can't handle a long, complex prompt." Something specific to Coder's exact request — content I couldn't perfectly reproduce from outside the harness, whether that's the precise injected workspace context, a subtlety in how the conversation history was serialized, or simply an unlucky run of sampling — was very plausibly a contributing factor, and I don't have the visibility into Coder's exact request construction to rule it out. The fairest thing I can say: this failure was real, it happened three times with zero self-correction, and under my best-effort attempt to recreate the same conditions independently, it didn't happen once. Treat the result as inconclusive on root cause, not as a clean verdict on the model.
Nemotron-3-Nano (Run 5): Fixed the Bug, Lost the Model Anyway
The most frustrating result of the round, because the fix I made for it worked exactly as intended, and it still didn't matter.
Real tool execution, confirmed throughout — no parse crashes, no hallucinated tools, every execute call actually ran and returned a real result. And then it spent roughly 35-40 turns across five nudges in an unbroken loop trying to locate the repository: checking /workspace (repeatedly, across multiple nudges, always failing the same way), checking /home/coder/project, /home/coder/workspaces, re-listing a skill directory it had already found and dismissed, misusing list_agents as a raw shell command, and passing a garbage string into a tool's template-ID parameter. It never once tried gh search repos or gh repo clone — the exact move every other model in the round used successfully within one or two turns.
Five interventions, zero strategy convergence, zero progress on any of them. This is a pure repo-discovery / tool-selection capability gap that happened to surface after I'd already fixed the thing that looked, at first, like it would be the blocker.
Scoring the Three That Shipped
Deviation from the series' usual format, disclosed up front: every other round in this series scores blind, before the orchestrating model knows which run is which contestant. That wasn't possible here — I needed to do the scoring myself, and I already knew the mapping from running the whole bakeoff. So this round is scored open, not blind, using the same 7-dimension weighted rubric, and grounded in the actual PR diffs rather than the write-ups above. Qwythos-9B and Nemotron-3-Nano shipped no code, so they're excluded as DNF rather than scored to zero.
| Dimension | Weight | Qwen 3.6 27B (PR #19) | Qwen 3.6 35B-A3B (PR #20) | GLM-4.7-Flash (PR #21) |
|---|---|---|---|---|
| Correctness | 25% | 4 | 5 | 2 |
| Design | 15% | 3 | 5 | 2 |
| Code quality | 20% | 4 | 5 | 3 |
| Engineering judgment | 15% | 3 | 5 | 2 |
| Scope discipline | 10% | 5 | 4 | 3 |
| Commit hygiene | 10% | 4 | 5 | 2 |
| Surprise | 5% | 3 | 5 | 2 |
| Weighted total | 3.75 | 4.90 | 2.30 |
Qwen 3.6 35B-A3B (PR #20) is the only one of the three that persists tag data exclusively through the GitHub API — reading and writing through commitFile rather than touching the local filesystem at all. That's the correct pattern for a Vercel-hosted, read-only-filesystem app in production, and the run reasoned about it explicitly, unprompted. It also reasoned in code comments about a read-before-write race condition, the only run of the three to do so. Highest score across every dimension.
Qwen 3.6 27B (PR #19) eventually persists through commitFile too, but round-trips through the local filesystem first — a pattern that works in dev and silently fails the moment Vercel's read-only production filesystem is in play. Combined with the flailing gh api/gh pr edit detour documented above, it lands solidly in the middle: correct destination, messier path, weaker commit hygiene than #20's five clean commits.
GLM-4.7-Flash (PR #21) scores lowest, and not just because of the wrong-branch incident. Its lib/tags.ts only ever calls fs.writeFileSync — it never calls commitFile at all, so a rename or delete wouldn't actually persist to the real repo in production, only to an ephemeral local file. That's a step worse than PR #19's "local-first, GitHub-second" pattern: there is no GitHub-second here. Combined with skipping auth checks on its mutating routes (a gap #20 doesn't have but #19 also doesn't have), it's the weakest submission of the round on every dimension except scope discipline, where doing the least extra work worked in its favor.
Common Threads Across the Round
Fixing the infrastructure doesn't necessarily fix the model — but I can only prove that for one of the two. Both Qwythos and Nemotron got legitimate, verified bug fixes before their runs started. Nemotron's fix held up cleanly under later isolated testing — the model failed afterward for a completely unrelated reason (repo discovery), and I have no reason to doubt that failure is really about the model. Qwythos's case is murkier: my attempt to reproduce its failure outside Coder came back clean eleven times in a row. The bugs I patched were both real and worth fixing either way. I just can't say with confidence that fixing Qwythos's bug actually got it to a fair starting line.
The screenshot requirement remains the great equalizer. Every model that got far enough to need one either hand-rolled a Playwright script instead of using the MCP tool as explicitly instructed, or gave up on the requirement entirely. Zero for five used the tool as asked. This has now shown up in enough rounds that it looks less like a per-model quirk and more like a standing gap in how these models are prompted or how the tool is exposed.
Committing the screenshot beats linking to it. Every model that succeeded eventually converged on the same fix for a broken screenshot reference: commit the image into the repo and point the PR body at the raw GitHub URL, rather than referencing a local filesystem path. Nobody was told to do this. They each found it independently after their first attempt broke.
A stuck loop doesn't announce itself as a loop. Nemotron's reasoning text was fluent and plausible on every single turn — it never repeated verbatim text the way Round 7's Gemma did. The only way to tell it wasn't making progress was tracking whether it tried a genuinely new strategy between nudges. It never did.
What I Actually Learned
The Round 7 spiral looks like it really was a bad day. Same model, same task, same config, and this time Qwen 3.6 35B-A3B not only shipped — it shipped the cleanest, most autonomous run of the round. If this had spiraled again, that would be a strong structural signal about MoE and sustained agentic work. It didn't. One data point either way isn't proof, but it's the opposite of what the hypothesis predicted.
Dense-vs-MoE isn't the right lens for these failures. The plan bet on architecture predicting endurance. Instead, the split ran across capability tiers that had nothing to do with active-parameter count: the two shipped-cleanly runs were one dense (27B) and one MoE (35B-A3B); the two failures were one dense (9B) and one hybrid-MoE (Nemotron). Model size and specific training, not architecture family, look like the better predictor here.
Live infrastructure debugging is now a standing cost of running this series. Round 7's headline infra bug was a --jinja vs --chat-template mismatch that zeroed out Devstral entirely. This round hit two more bugs in the same family — a template that crashes on multi-turn system messages, and a stop-token flag that silently breaks tool-call parsing — for two different models. Every round so far has needed at least one live template or server-config fix before the actual bakeoff could start fairly.
Isolation-testing a failure after the fact is worth doing, even when it muddies your own conclusion. I went into the Qwythos write-up ready to call it a clean model-capability gap. Running the fairness test anyway — and getting eleven consecutive successes where the real bakeoff got three consecutive failures — means I'm publishing a less satisfying, more honest conclusion instead: something about that specific session likely mattered, and I don't have enough visibility into Coder's exact request construction to say what. A benchmark that only reports the clean story isn't a benchmark I'd trust.
"Knows the answer" and "can act on the answer" are separate capabilities — confirmed on the model that actually earned the verdict. Nemotron's reasoning was fluent and its tool calls were correctly formatted throughout, right up until it simply couldn't find the repository across roughly 40 turns and five nudges. That's the cleaner version of the same pattern Qwythos might also show — but only Nemotron's failure held up when I went looking for a harness-side excuse.
By the Numbers
- 5 contestants (3 planned, 2 added once the pipeline was proven out)
- 3 of 5 shipped a real, mergeable PR
- 2 llama.cpp/template bugs found and patched live, mid-bakeoff — 1 of 2 fixes conclusively verified to hold up under isolated re-testing after the run
- 11 of 11 fairness-test reconstructions of Qwythos's failure conditions came back clean — zero reproductions of the raw-text tool-call failure it showed three times in the real bakeoff
- 0 of 5 models used the Playwright MCP tool as explicitly instructed for the screenshot requirement
- 8 interventions for the slowest successful run (Qwen 3.6 27B) vs. 5 for the fastest (Qwen 3.6 35B-A3B)
- 1 run needed an explicit correction rather than a neutral nudge (GLM-4.7-Flash, wrong branch)
- ~35-40 turns burned by Nemotron-3-Nano searching for a repo it never found, across 5 nudges with zero strategy change
- 3 consecutive identical failures from Qwythos-9B, zero tool calls ever actually executed
- Total tokens across all five runs: ~20.9 million — for three completed PRs and two runs that produced no usable code at all
Next up: does the dense 27B's clean-but-slow performance hold up against a frontier cloud model on the same rubric, or does the token-efficiency gap Round 7 found in Sonnet still stand?
Top comments (0)