Why We Kept Ternary Bonsai 2 27B After Testing the Faster 9B

October 16, 2026

Our production classifier runs on a single RTX 2080 Ti with 11 GB of VRAM. It is often busy enough that requests spill over to a hosted model. When PrismML released Ternary Bonsai 2 9B on October 15, we had a practical reason to try it: its GGUF file is 2.25 GiB, against 6.70 GiB for the 27B we already use.

We replayed 1,015 production decisions. The 9B was 2.4–3.6× faster in our laptop benchmarks, but it downgraded 15 of 65 escalations that a second model had confirmed. On another task, 43% of its answers contradicted an application rule.

That was enough to rule out a straight swap. Here is what we tested and where it broke.

What we needed to improve

The 27B runs in llama.cpp with PQ2_0 weights, two parallel slots and a shared 32k context. Reasoning is disabled; responses use strict json_schema decoding at temperature 0.

Requests have an 8-second client deadline. When the gateway cannot offer a slot within that budget, it rejects the request and the application uses a hosted fallback. Over one 20-hour window, we logged 9,081 requests:

Outcome	Share
Served by the local model	82%
Rejected fast (deadline could not be met)	13%
Cancelled by the client	4.6%

At the busiest hour, about 30% overflowed to the hosted model. Served requests took a median of 4.3 s and a p95 of 8.1 s of model time. A faster model could reduce both waiting and fallback costs.

The application makes three judgements about informal, non-English posts. Each request includes the current post and up to 11 earlier posts in the thread. Median prompt lengths range from about 0.7k to 2.3k tokens; responses are capped at 64–96 tokens of JSON.

	Output	Why it is hard
Task A: escalation triage	allow / clarify / escalate, plus a severity subtype and a confidence score	High stakes. A missed escalate is the costly error. Each escalate is re-checked by a larger hosted model.
Task B: intervention assessment	A boolean "intervention needed" plus a suggested action, one of which is "continue as normal"	Two fields must stay consistent.
Task C: multi-flag detector	Six booleans, mapped to none / flagged / urgent	Runs only on one category of threads, so volume is low.

Task A decides whether we can make the switch. We call an escalation “confirmed” when the hosted model agrees with the 27B. This gives us a useful reference, though it is still model agreement rather than expert ground truth.

How we tested

We sampled logged decisions from the current prompt versions, keeping rare outcomes well represented:

Task	Cases replayed
A	All 107 escalations (65 confirmed, 42 rejected), 200 clarify, 250 allow
B	175 "intervention needed", 175 "continue"
C	All 108 logged cases (103 flagged or urgent, 5 none)

To estimate production rates, we weighted each case by its sampling and logging probabilities:

weight = (rows in stratum / rows sampled) × (1 / logging probability)

The Task A sample represents about 68k production checks.

We rebuilt requests with the application's own prompt builders, using stored posts and thread windows, the same schemas, token limits and temperature. Prompt token counts matched production in all 275 cases we checked.

The replay ran on an M1 Max under llama.cpp with the production server flags. The 27B used our production build, prism-b10709; the 9B required prism-b10777.

Since we already had the 27B's logged answers, we reran it on only 275 cases: a random control subset and every Task A and C disagreement. It reproduced 273 of 275 production decisions. The two borderline differences may reflect Metal/CUDA numerical differences. This check gave us confidence in the replay, without requiring a second full run.

Task A: fewer clear decisions, missed escalations

Production 27B decision	n	9B agreed	Where the 9B differed
escalate, confirmed by the hosted model	65	50 (76.9%)	15 → clarify
escalate, rejected by the hosted model	42	34 (81.0%)	8 → clarify
clarify	200	184 (92.0%)	12 → allow, 4 → escalate
allow	250	141 (56.4%)	109 → clarify

All outputs were valid. The problem was the decisions inside them.

The 15 downgraded confirmed escalations mattered most. The 27B reproduced all 15 as escalate in the local rerun. Twelve had its most severe, explicit subtype; three had the figurative-language subtype. The 9B assigned all 15 a low-severity subtype with self-reported confidence of 0.85–0.9. A confidence threshold would not have caught them.

We also read those cases ourselves. About 11–12 looked clearly positive, while 3–4 were borderline. Some stated the critical signal directly; others used metaphor or idiom. This was an informal review, not expert annotation.

The broader shift was toward clarify. Reweighted to production volume:

Share of checks	27B	9B
allow	86.4%	49.5%
clarify	13.4%	50.1%
escalate	0.2%	0.4%

Weighted agreement was 61.2%. The 9B never moved an escalation straight to allow, but it moved many cases from both ends into the middle. Since clarification has a cost in our application, that would affect a large share of users.

It also doubled the weighted escalation rate despite missing confirmed cases. New escalations from the clarify stratum alone would add roughly 180 hosted-model reviews over the logged period.

Task B: valid JSON, contradictory answers

Production 27B decision	n	9B agreed	9B invalid	9B opposite
intervention needed	175	23 (13.1%)	128	24
continue	175	148 (84.6%)	23	4

In 151 of 350 answers (43%), the 9B returned “intervention needed: true” alongside “continue as normal.” The application rejects that combination.

Our JSON schema permits it; the prompt tells the model not to use it. The 27B followed that instruction in our control runs. The 9B frequently did not. Constrained decoding kept the JSON parseable, but could not enforce a relationship we had left out of the schema.

Even among valid answers on positive cases, agreement was only 23 of 47. After reweighting, about 17% of all Task B checks would fail validation and go to the hosted fallback, adding to the load we wanted to reduce.

Task C: better agreement, limited evidence

The 9B agreed on 91 of 103 cases the 27B had flagged or marked urgent (88.3%). Disagreements included five flagged cases dropped to none and three urgent cases dropped to flagged.

We had only five none cases, too few to estimate false positives. This task gave us less reason for concern, but also much less evidence.

How much faster?

We ran llama-bench on the M1 Max with flash attention enabled. We also tested the 27B on the newer runtime to check whether the build itself explained the gain.

	pp512, tok/s	tg128, tok/s
27B, prism-b10709	134.1	19.2
27B, prism-b10777	142.9	19.1
9B, prism-b10777	484.5	45.6

Against our pinned 27B build, the 9B was 3.6× faster on prefill and 2.4× on generation. Updating the runtime alone made little difference.

The production 2080 Ti measures 749 tok/s for prefill and 42.6 tok/s for generation with the 27B. Applying the laptop speed ratios suggests checks might take about 1.5–2 s instead of 4.3 s. We have not measured the 9B on that GPU, so this remains an estimate.

What we would try next

This test answers whether the 9B can replace our 27B with the existing prompts. It does not tell us the best the smaller model can do. We tuned those prompts for the 27B, disabled reasoning on both models and tested one language in one domain. We also did not run full-precision Qwen3.5 9B, so we cannot separate model-size effects from ternary quantisation.

The first change worth testing is Task B's output schema. Merging the boolean and action into one enum could make the contradictory combination impossible. We would then tune the prompts for the 9B and replay the same cases. A small reasoning budget might help too, at the cost of some speed.

Another possibility is to use the 9B as a first pass for Task A, sending its non-allow answers to the 27B. On this sample, that would skip the larger model for about half of production checks, letting through roughly 6% of the 27B's clarify cases and none of its escalations. We have not tested this as a pipeline. Keeping both models on an 11 GB card would be tight, and the second hop would add latency.

For now, the 27B stays. Replaying production traffic made that decision straightforward: the speed gain was real, but so were the missed escalations. Any replacement gets the same test, with no misses on confirmed positives and no increase in invalid answers.