How Prism ML Squeezed a 27B Model Into 3.9 GB

Most of us assume that useful AI needs a data center. The bigger the model, the better the reasoning — and the larger the bill. Prism ML's Bonsai 27B is a quiet counterexample: a 27B-class model distilled into binary and ternary weights, small enough to run on a laptop or even a phone, while keeping 89.5% of its full-precision reasoning.

I wrote a full technical walkthrough on my blog, but here is the short version of how the compression pipeline works and why it matters for developers.

This post is a summary. The full article lives at https://fernando-nog.netlify.app/bonsai-distillation-explained-from-qwen36-27b-to-a-phone-friendly-39-gb-model/.

Start with the same architecture, represent weights differently

Bonsai 27B begins as Qwen3.6-27B, a 27.3B parameter hybrid-attention model. Prism ML did not redesign the transformer. Instead, they changed how the weights are stored.

Two formats ship today:

• 1-bit: each weight is either -1 or +1, plus a shared scale per 128 weights. Effective bit width: ~1.125. File size: ~3.9 GB.
• Ternary: each weight is -1, 0, or +1, with the same shared-scale trick. Effective bit width: ~1.71. File size: ~7.2 GB.

The ternary variant is the sweet spot for laptops; the 1-bit variant is the one that fits a phone.

The key trick: train the student while it is already quantized

Conventional quantization takes a trained model and rounds its weights afterward. That works at 4 bits. Below 4 bits, reasoning quietly collapses.

Bonsai does the opposite. The student model is trained with weights already constrained to the binary or ternary alphabet. The loss function sees the rounding error from the start, so the model learns weight patterns that remain useful after extreme compression. That is how math stays at 91.66% of full precision and coding at 81.88% in the 1-bit variant, with the ternary version even closer.

Why this matters for developers

A 3.9 GB 27B-class model changes where capable AI can run:

• Locally, without sending code to a third-party API.
• On a laptop with no external GPU.
• On a phone via MLX Swift on Apple Silicon.
• On a single 24 GB GPU with room for context and batches.

It is not a replacement for frontier cloud models on every task. Long-horizon agentic coding and the hardest reasoning categories still favor full precision. But for routine coding, refactoring, explanation, and long-document analysis, local models just became a lot more credible.

What I am watching next

The most interesting line is agentic coding. Prism ML has said a Bonsai variant tuned for multi-file, run-test-and-repair workflows is next. If a local model can handle that class of work, the boundary between "cloud-only" and "local-first" coding assistants moves again.

If you want the full pipeline — architecture, kernels, DSpark speculative decoding, and the benchmark breakdown — the complete post is here:

Bonsai Distillation Explained: From Qwen3.6-27B to a Phone-Friendly 3.9 GB Model

---

I write about AI, backend development, and practical engineering on my technical blog.