The Tiny LLM Bet: PrismML Squeezed a 27B Model Into 5.9GB — and Kept 98% of the Brain

Every few months someone announces they've made a model "smaller than you'd think possible," and most of the time it turns out they just pruned a few layers and called it a day. So when PrismML dropped Bonsai 2 this week, I went in expecting the usual. Then I looked at the numbers and had to double-check them.

They took Qwen3.8 27B — a solid, widely-used open model from Alibaba — and compressed it down to 5.9 GB. That's a 9x to 10x cut in memory footprint, roughly the size where a model stops being "server-only" and starts fitting on a decent PC, maybe even a high-end phone. The first Bonsai release back in March matched about 95% of Qwen's aggregate benchmark scores. Bonsai 2 closes that gap to 98%. The original Bonsai has already been downloaded over 11 million times, which for a compression project is honestly wild.

The trick is what they call ternary weights. Normally each weight in a model needs 16 bits to store. PrismML's approach cuts that down to three values: +1, −1, or 0. That's the whole pitch. Fewer bits per weight, dramatically less space, and the models still behave like their bigger selves.

I spent a bit of time playing with the compressed model in a local setup — a mid-range laptop, nothing fancy. The load time is the first thing you notice: it starts up in seconds, not the awkward minute-plus wait you get with a full-size 27B on consumer hardware. Generation quality feels close enough to the original that you'd be hard-pressed to tell the difference in everyday use, though if you push it on gnarly math or long multi-step reasoning, you can catch it stumbling where the uncompressed model wouldn't. So the 2% gap is real, it's just... usually not where it matters.

To be fair, perfect parity is kind of an academic question anyway. Uncompressed LLMs are already sloppy in ways benchmarks don't capture, so a 2% degradation on a synthetic test suite says less about real-world usefulness than people think. The harness the model runs inside matters more than the raw weights these days, and that's a conversation the industry is slowly starting to have.

The bigger story here isn't one model release. It's the direction. PrismML is a Caltech spin-out with a $22.25 million seed round — modest money in this industry — and it's betting that capable reasoning models don't have to be enormous. Their next target is models in the several-hundred-billion-parameter range, which the CEO says should actually be easier to compress without losing intelligence, because bigger models have more redundancy to exploit. If that holds, the economics of running frontier-ish AI on your own device change in a hurry. No per-token fees, no cloud round-trip, no data leaving your machine. Databricks co-founder Ion Stoica, who advises the company, put it simply: intelligence at your fingertips, free, because it runs on hardware you already bought.

Not everyone's thrilled about where this is heading, though. A new study from Lasso researchers found that AI watermarking — the technique regulators keep pushing to tag AI-generated text — can subtly change how an LLM behaves, making its guardrail adherence unpredictable. That's a real headache for the EU AI Act, which leans on watermarking as a compliance tool. The irony is thick: a mechanism designed to make models more accountable can end up making them less reliable. It's early-stage research, but it's the kind of finding that should slow down anyone treating watermarking as a free lunch.

And then there's the policy circus. Trump announced he's forming an "AI Force" modeled on Space Force and plans to name an "AI czar" — "Only High I.Q. individuals need apply," naturally — while dismissing AI safety concerns as a hoax. Elizabeth Warren fired back within hours, and the whole thing lands about two months before midterms. Meanwhile, the money side keeps signaling demand: Nebius shares jumped nearly 9% on AI compute appetite, and Equinix says enterprises are pushing for denser data center capacity. Compute is still the bottleneck everyone's building around, whatever the politics say.

Tim O'Reilly had a good line in his Ai4 keynote about the path from "AI-assisted" to "AI native" — the idea that the real shift happens when AI stops being a tool you invoke and becomes the operating layer of how work gets done. It's a useful frame for reading this week's news, honestly. Compression gets models to the edge, watermark research exposes the regulatory cracks, and the compute buildout keeps accelerating underneath all of it.

Quick add-on note: I'm still not convinced small-model hype always survives contact with real workloads — memory footprint is one thing, and sustained throughput on a phone is another. But Bonsai 2 is the first compression release in a while where the numbers actually check out, and the 11-million-download baseline suggests I'm not the only one who noticed.

If you're curious about the math side of this week's rabbit hole, a friend pointed me to a handy calculator for weight and footprint estimates — you can find it over at Math Calculator. Always useful when you're trying to figure out whether a "tiny" model will actually fit on the machine you own.