A 27B reasoning model in a 5.95 GB file sounds like a download problem solved. Mostly it moves the problem into configuration. We served Ternary Bonsai 2 27B, Prism ML's build of Alibaba's Qwen3.8-27B, on one RTX 4090 with 24 GB. Serving means running the model behind an API that answers requests. It wrote about 104 tokens a second for one user (tokens are the word pieces a model reads and writes, about four for every three words) and peaked at about 12 GB of the card's memory. Four traps stood in the way, none of them about model quality.
The full command and every benchmark figure are in our write-up of Ternary Bonsai 2 27B on one RTX 4090.
Why the file is so small
A model is mostly a long list of learned numbers called weights. Prism ML stores almost every one as -1, 0 or +1, plus one ordinary 16-bit scale for each group of 128. That costs under 2 bits per weight instead of 16, so the file is about nine times smaller than the roughly 54 GB original. Before that conversion, the weights are rotated in a fixed pattern. At run time, the program serving the model must apply the same rotation to the values flowing between its layers.
Trap one: your usual runtime cannot read it
llama.cpp is the open-source program most people use to run model files like this locally. Prism ML's model card is explicit: "Stock llama.cpp will not run these files." Both Bonsai formats fail as unknown, and the known issues page says the popular desktop apps fail too. Older llama.cpp builds could load a related Bonsai file and then reply with gibberish, since they never apply the rotation. We did not try the stock program ourselves.
What you need is Prism ML's fork, its own modified copy of llama.cpp, which ships prebuilt downloads. On Linux, the known issues page says to take the build made with CUDA 12.8 (NVIDIA's GPU toolkit), because the newest version crashes on some machines. That advice predates the fork's 14 October release, so the crash may be fixed. We used 12.8 and saw none. Also, the Linux build needs two NVIDIA libraries it does not ship, so we added them from NVIDIA's Python packages.
Trap two: thinking eats your token budget
The model reasons in hidden text before it answers, and those thinking tokens count against max_tokens, the cap on how much it may write per reply. Set a small cap and the reply can come back empty. The card recommends 16,384.
{
"max_tokens": 16384,
"reasoning_effort": "medium"
}
A low client default is how a working model looks broken. With that cap, all 17 of our timed requests finished cleanly: no empty replies, no loops, nothing cut short.
Trap three: pick the thinking level on purpose
The chat template, the file that formats a conversation for the model, accepts low, medium and xhigh. Per the known issues page, high returns an error, so our custom backend (the small adapter that starts the model on InferenceKey) maps it to xhigh. Prism ML's whitepaper measures the headline 98.2%, Bonsai's score as a share of the original's, with maximum thinking. At medium, Bonsai scores 79.3 against 83.9 at maximum, and the original scores 82.6 at medium.
The card says medium gives shorter answers. Our two samples disagree: the seating puzzle took 1,447 tokens at medium and 670 at maximum. Measure on your own prompts before picking a default.
Trap four: the second user waits in line
On InferenceKey, a model pinned to a single machine handles one request at a time by default. Four questions sent together finished in 22.4 s, the last one stuck in the queue. After raising one setting (max_in_flight to 8), the same four finished in 9.6 s at about 159 tokens a second in total. The same default bit us in our AstaBrief-8B test, so check it wherever you serve.
On the server, the context (how much text the model holds in view) is 65,536 tokens. With kv_unified, that is one pool shared by four slots, one per simultaneous conversation, not 65,536 tokens each.
What to measure, and what the numbers do not say
Memory is the number people get wrong. Serving took about 12 GB, roughly double the file, because the conversation memory and the image reader live on the card too. The per-request timings and memory readings are in the article.
Quality is the number people test wrong. A conventional 2-bit shrink of similar size still scores 88.93 on an easy knowledge quiz, yet drops to 57.5 on a hard math test where Bonsai scores 95.83. On a benchmark of real bug fixes in real repositories, Bonsai scores 60.8 against 80.6 for the original. The known issues page also reports malformed or looping tool calls (the model asking your code to run a function), which we did not test.
Every quality figure here is Prism ML's. Ours came from roughly 20 requests, each question tried once or twice. Unsloth's standard 4-bit build (17.6 GB) also fits a 24 GB card and keeps a little more quality, 98.7%. The case for the 6 GB file is the spare memory it leaves, not the fit.
The Qwen team trained the base model. Prism ML designed the format, borrowed the rotation from Meta's SpinQuant work, built the fork and documented its issues in detail. Mobius Labs' compression method made the image reader smaller. Unsloth published the 4-bit build we compared against. The llama.cpp project is the foundation of the fork. An AI Search video is where we first saw the release. Our part was the custom backend, the SDK example and the measurements.
The launch command, the per-GPU speed table and the platform issues we hit are in the complete Bonsai 2 27B deployment notes.
Top comments (0)