PrismML has released Bonsai 2 27B, a ternary-quantized version of Qwen3.8 27B that compresses a 53.8GB FP16 model down to 5.9GB while retaining 98.2% of the original's benchmark score. The catch: the release only works with a patched llama.cpp fork carrying custom ternary hybrid-attention kernels — stock llama.cpp either refuses to load the files or, for one of the two quant formats, loads them silently and produces garbage output.
What changed
PrismML has shipped Bonsai 2 27B, a ternary-quantized derivative of Qwen3.8 27B that stores weights as {-1, 0, +1} trits with FP16 group-wise scaling instead of standard integer quantization. The company describes the format as averaging 1.76 effective bits per weight across the two GGUF packings it publishes: PTQ1_0, a dense-trit packing at 5.95GB (1.75 bits/weight), and PQ2_0, a 2-bit-slot packing at 7.21GB (2.13 bits/weight). Both derive from the same underlying "ternary g128" format; a reference FP16 checkpoint (53.8GB) is also published for comparison. That puts the compressed model at roughly a ninth of the original's footprint.
The model keeps Qwen3.8 27B's 262K-token context window and multimodal input (text plus image), and supports coding, reasoning, vision, and agentic tasks; the vision path adds a ~0.63GB Q8_0 component loaded only when an image is present. Across benchmarks spanning MMLU-Redux, HumanEval+, CharXiv, and others, PrismML reports an overall score of 83.9 against the FP16 baseline — 98.2% retention — with coding at 99.3% retention, math at 99.5%, and vision the weakest category at 96.3%. On inference speed, the company cites up to 143 tokens/second on an RTX 5090 and 46.8 tokens/second on an Apple M5 Max, with RTX 4090 energy use of 0.714 mWh/token, a figure PrismML says is 40% more efficient per token than a full-precision 8B model.
None of this runs on stock llama.cpp. The ternary hybrid-attention kernels for CUDA and Metal live only in a PrismML-maintained llama.cpp fork; the vendor's own documentation warns that unmodified llama.cpp either refuses the custom quantization types outright or, for PQ2_0 specifically, loads the file without warning and produces garbage output. Ollama, LM Studio, vLLM, and Jan integrations are documented, but all route through that same fork rather than upstream code.
Who this affects
Bonsai 2 27B matters most to people running large open-weight models on a single consumer GPU or Apple silicon with limited memory, since 5.9GB puts a 27B-class model's quality within reach of hardware that could never hold the 53.8GB FP16 checkpoint. Developers already comfortable building llama.cpp from source, or willing to pull PrismML's prebuilt binaries or Docker image, get the compression without much extra friction.
It matters less — or is actively risky — for anyone relying on a stock, unmodified llama.cpp install or a downstream tool that assumes standard GGUF quantization behavior. The PQ2_0 packing's silent failure mode on unpatched llama.cpp (loading without error and returning garbage) is the kind of bug that would be hard to notice in an automated pipeline. Teams that need predictable behavior across a heterogeneous set of inference backends should treat this as a single-fork dependency, not a drop-in GGUF file.
Verdict
Worth trying, not worth adopting blindly. The compression numbers are real and unusually well-documented for a quantization release: 98.2% overall retention at roughly a ninth of the file size is a genuinely strong trade, and the RTX 5090 and M5 Max throughput figures make single-GPU or Mac-only deployment of a 27B model plausible for the first time at this quality bar.
But the dependency on a PrismML-maintained llama.cpp fork — with a documented silent-corruption failure mode on stock builds — is the kind of detail that belongs in a pilot, not a production rollout. Readers who already run custom llama.cpp builds or who are comfortable pinning to a vendor fork should pull it down and benchmark against their current setup. Readers who need mainline llama.cpp, Ollama, or LM Studio compatibility without extra build steps should wait for either upstream support or a second quantization vendor to validate the approach.
Tracked daily from official release feeds and vendor changelogs. Full archive: https://media.patentllm.org
Top comments (0)