DEV Community: Dhanush G

Qwen 3.5 vs Ornith 1.0 9B Models, Same Hardware, Same Quant as Coding Agents

Dhanush G — Wed, 01 Jul 2026 15:00:00 +0000

I ran Qwen 3.5 9B and Ornith 1.0 9B, both at Q8, on the same 16GB Mac, through the same multi-step agent tests. Neither is agent-ready. But they're not ready in interesting, different ways — and the most surprising result is that the native tool-calling API made both of them worse on easy tasks than plain prompting did.

If you run local models, you've seen the recommendation threads:
"Qwen 3.5 9B is great for its size," "Ornith 1.0 is impressive." Both are true in a chat window. The question I care about is different: can either one actually run a coding agent — a multi-step loop that calls tools, reads results, and keeps going — on normal hardware?

So I tested both the same way. Same machine (16GB Apple Silicon, Mainstream class). Same quant (Q8). Same task batteries, run through a real agent loop, each task repeated multiple times, counted as passed only if it succeeds every run (pass^k). Same backend (llama.cpp).

Here's what came back.

The headline: neither clears even the Easy tier

Both models got the same top-line verdict: NOT READY — does not clear Easy, the easiest tier tested.Cleared tier: NONE.

That's worth sitting with. These are capable chat models. In a single back-and-forth they answer fine. But "answers one question well" and "survives a tool-calling loop reliably" are nearly unrelated skills, and the loop is where both fall down.

The surprise: native tool-calling was worse than prompting

This is the result I didn't expect, and it showed up for both models.

On the Easy tier, the native tool-calling path scored worse than just describing the tools in the prompt and parsing the model's text:

Model (Q8, Easy tier) Native tool-calling Prompt-based
Qwen 3.5 9B60% (15/25) 100% (25/25)
Ornith 1.0 9B60% (15/25) 100% (25/25)

Same pattern, same numbers. In both cases the native path failed the exact tasks the prompt path passed — things like running a failing test or linting and reporting. The failure wasn't "the model is dumb." Looking at the failure breakdown, 100% of the easy-tier failures were "reported in prose" — the model did the work correctly but answered in plain text instead of emitting a proper tool call. Right answer, wrong channel.

Everyone's default advice is "use the native function-calling API." For these two 9B models, on easy agent tasks, that advice was backwards.

Where they differ: how they break on harder tasks

On the Medium tier the two models start to diverge in their failure modes, which is more useful than the raw score.

Qwen 3.5 9B (Medium): native 40%, prompt-based 0%. The prompt path collapsed completely — top error "NO OUTPUT," it just stopped producing anything usable. Native limped to 40% but its top error was FAKE DONE — claiming the task was complete when it wasn't.

Ornith 1.0 9B (Medium): native 40%, prompt-based 0% — nearly identical top-line. Its Medium failures broke down as 67% hallucinated completions ("claimed done / called methods outside the schema") and 33% infinite loops ("failed to resolve hidden prerequisites; repeated actions").

So as the tasks get harder, both shift from a harmless formatting problem (prose instead of a tool call) into the two genuinely dangerous agent failures: falsely claiming completion, and looping without progress. A model that hallucinates "done" is the worst case for a coding agent, because in a real pipeline it looks like success.

The hidden killer: the cold-start tax

There's one more finding that doesn't show up in any pass rate, and it's arguably the most practical.

Both models are ~8.9GB at Q8. With KV cache and overhead, a run pushes ~15.6GB on a 16GB machine — right at the ceiling. The result: after each response the model gets evicted from memory, and the next call has to reload it from disk before it can even start. That's a ~20-second time-to-first-token, every single call.

In a chat window you barely notice — one reload, then you talk. In an agent loop it's fatal. A 20-step task pays that ~20s reload twenty times — roughly 7 minutes of pure cold-start tax before counting a single token of actual reasoning. Generation speed itself was fine (~9.8 tok/s once loaded). The model isn't slow to think; it's slow because it can't stay in memory.

This is a hardware-ceiling problem, not a model problem — and it points at a fix: leave headroom. A smaller quant that fits with room to spare would stay resident and skip the per-call reload entirely. "Largest quant that fits" is the wrong rule when fitting exactly means re-loading every step.

So which one?

Honestly: for a coding agent on a 16GB Mac, neither, at Q8. Both are NOT READY, both fail the easiest tier, both pay the cold-start tax. As chat models or for single-shot help they're fine — this isn't a knock on either.

But the comparison taught me three things worth keeping:

Native tool-calling isn't automatically the right path. Both models scored higher with plain prompting on easy tasks. Test both; don't assume the native API wins.
The failure mode matters more than the score. Two models can tie and break completely differently — prose-formatting vs hallucinated-done vs looping — and you debug each one differently.
Fit isn't the same as headroom. A model that just fits cold-starts every call. For agent work, leave memory to spare.

The honest caveats

This is 9B at Q8 on a 16GB Mac, one backend (llama.cpp). Different hardware, quant, or harness may land differently — which is exactly why testing your own combo beats trusting a thread.
Both models are well-regarded for good reason; "not agent-ready here" is a narrow, scoped claim about multi-step tool-calling reliability on constrained hardware, not a verdict on the models overall.
Small sample at Medium. The failure-mode split is a strong signal, not a proof.

I measured all of this with QuantaMind, the open-source tool I'm building to test local models in a real agent loop on your own hardware GitHub: — free, fully offline, with the failure broken down by type instead of hidden behind a single score.

Question for you: have you seen native tool-calling underperform plain prompting on a small model? Or is this a 9B-specific thing that disappears at 27B+? I'm collecting these and would genuinely like to know where the line is.

Qwen 2.5 Coder 7B Q4 vs Q8 scored the same on my agent test, then I read how they failed

Dhanush G — Tue, 23 Jun 2026 21:51:24 +0000

I ran Qwen2.5-Coder-7B at Q8 and Q4 through the same multi-step agent test. Same pass rate at every tier. But on the hardest tier they failed in two completely different ways — and that difference says more than the score does.

If you run local models for agents, you've made this trade a hundred times: do I keep the big Q8, or drop to Q4 to save VRAM and get more speed?
The usual way to decide is the benchmark score. Run both, compare the numbers, pick the one that scores higher. If they score the same, shrug and take the smaller one.
I did exactly that this week with Qwen2.5-Coder-7B. And the scores told me almost nothing. The failures told me everything.

Same model, two quants: q8_0 (7.6 GB) and q4_k_m (4.6 GB). Same machine (16 GB Apple Silicon). Same tasks. I didn't test one chat prompt — I ran each one through a real agent loop: call a tool, read the result, decide the next step, repeat, with traps and failures thrown in. Each task runs many times, and it only counts as passed if it passes every run.

Here's what came back.

Tier Q8(7.6 GB) Q4 (4.6 GB)
Easy Pass (5/5) Pass (5/5)
Medium Pass (40/40) Pass (40/40)
Hard 1 of 4 tasks 1 of 4 tasks

On the score alone, these two are twins. Easy and Medium: both clean. Hard: both fall apart, passing only 1 of 4 tasks. If you stopped at the number, you'd say "no real difference, take the Q4, it's smaller and twice as fast."

That conclusion would be wrong. Here's why.

Same score, two different failures

On the Hard tier, the two quants broke in opposite ways.

Q8's top failure was a forbidden call. One of the hard tasks plants a trap: a tool the model is explicitly told not to use on a decision boundary — think "delete the record" or "deploy to prod," an action it's supposed to gate, not take. Q8 walked right up to the problem, acted decisively… and pulled the trigger on the thing it wasn't allowed to touch. The moment it did, the run failed.

Q4's top failure was a loop cap. Same kind of task. But Q4 never got far enough to trip a trap. It got stuck — repeating the same call, never resolving the task, spinning until it ran out of steps. It didn't do the wrong thing. It couldn't do anything.

Read those two side by side:

Q8 — competent but reckless. It can plan and act. It just blew past a guardrail.
Q4 — can't hold the plan together. It looped in place and made no progress.

That's not the same model with a slightly lower score. That's a different kind of model.

Why the failure mode matters more than the number

Looping is the tell. When a model repeats actions and never advances, it's usually because it's lost the thread of the multi-step plan — it can't keep track of what it's done and what's left. And that planning/state-tracking ability is exactly what lower-bit quantization tends to erode first on long, hard tasks.

So dropping Q8 → Q4 didn't just shave a few points off a score (both landed at 1 of 4). It changed how the model fails — from "acts, but trips a guardrail" to "can't make progress at all." One of those is a discipline problem you might fix with better prompting or tighter tool permissions. The other is a capability problem you mostly can't prompt your way out of.

If you only looked at the pass rate, you'd never see this. Two models tied at 1/4 look identical on a leaderboard. In production they'd fail your users in two completely different ways — and you'd debug them completely differently.

The honest caveats

I'm not going to oversell this, because the whole point is measuring honestly:

Neither quant is "ready" for a real coding agent here. Both failed the context-cliff probe right at the baseline — a tool-call failure, not a context-length limit. The headline is the failure-mode difference at Hard, not "Q8 is production-ready."
This is one model on one machine. Qwen2.5-Coder-7B at 16 GB. Your model, your hardware, your tasks may land differently — which is the entire reason to test your own combo instead of trusting someone else's.
Small sample at Hard (4 tasks). The divergence is a strong signal, not a statistical proof.

The takeaway

"Which quant should I use" is the wrong question if you only check the score. Two quants can tie and still be different models underneath. The number tells you whether it failed. The failure mode tells you what's actually broken — and that's what you need to know before you ship.
I measured this with QuantaMind, the open-source tool I'm building to test exactly this — local models, in a real agent loop, on your own hardware, with the failure broken down by type instead of hidden behind a score. It's free and fully offline.

DEV Community: Dhanush G

Qwen 3.5 vs Ornith 1.0 9B Models, Same Hardware, Same Quant as Coding Agents

Qwen 2.5 Coder 7B Q4 vs Q8 scored the same on my agent test, then I read *how* they failed

Qwen 2.5 Coder 7B Q4 vs Q8 scored the same on my agent test, then I read how they failed