I did a mad science thing recently and I need to tell you about it. This one didn't touch the whole home lab, just one small piece of it: the master node from the swarm cluster I described back in my first post (the box I called the swarm host there), running just two p150a cards linked together with a cable was the entire lab for this experiment. Here's what I was chasing, and what actually happened.
Quick disclosure, same as always: I work at Tenstorrent and I'll flag what didn't work just as readily as what did.
The Big Question
The question I set out to answer was easy to ask and hard to answer: can you run a genuinely frontier sized model, in this case GLM-5.2, a 744 billion parameter mixture of experts model, on a small two card box? Not a shrunk down version. The real thing.
The trick with a mixture of experts model is that not all of those 744 billion parameters do anything for any given word you generate. The model routes each token to a small handful of "expert" sub networks and ignores the rest. For GLM-5.2, something like 40 billion parameters are actually active per token. The other 700-plus billion just sit there, waiting to be needed for a different token later.
I didn't come up with this idea on my own. Credit goes to a project called Colibri, a CPU and GPU engine that already proved you could run GLM-5.2 in about 25 GB of memory by keeping the common stuff resident and streaming the rest of the experts in from RAM or an SSD as needed, using a lookahead prefetch system they call PILOT along with cache-aware routing and speculative decoding. Colibri runs on CPU, CUDA, and Metal. Nobody had ported this approach to Tenstorrent hardware yet, and that gap is exactly what got me curious.
Early testing also gave me a number worth keeping in mind for the rest of this post: grabbing an expert that's already sitting in a card's memory costs something like 13 times less than streaming one in from RAM or disk. So the entire game becomes maximizing how often the token you're generating already has what it needs close by.
The Setup
For hardware, I used the same box I called the swarm host in my first post. Two Blackhole p150a cards, 32 GB each, linked together with the 800G cable on the back. The host itself is a Ryzen 9900X with about 256 GB of RAM. One card talks to the host over a full Gen5 x8 link at 28 GB/s. The other is stuck on a x4 link at 14 GB/s, so there's already an asymmetry to work around before you get to anything clever. The cards can talk to each other fast over their own fabric connection, NVMe storage on the box does about 7 GB/s, and each card's own memory reads at around 312 GB/s once data is actually sitting on it.
Think of it like a kitchen. A card's own memory is the counter right in front of you, fast to grab from. RAM is the pantry down the hall. NVMe is the grocery store across town. The whole game here is minimizing trips to the grocery store.
Getting a Baseline
Before optimizing anything, I needed to know what slow actually looked like. Running Colibri's own GLM-5.2 setup on CPU gave me 0.35 tokens a second, with about 35 seconds before the first word even showed up. That was the number to beat.
Building It Up
From there it was a lot of careful, unglamorous verification work before I worried about speed at all. Every piece of the model, meaning the normalization step, the routing logic, the expert layers themselves, and the attention mechanism, had to be checked against Hugging Face's own reference implementation using a correlation score called PCC (think of it as a similarity score between two sets of numbers, where 1.0 means identical). Everything I built came back above 0.999, and the full decoder layer landed at 0.999984 against real weights. My math wasn't just close. It was basically indistinguishable from the reference.
Then came the moment of truth. I ran the full 78 layer model, with real 4 bit quantized weights (the same low quantization I said I wanted back in my first post), fed it "The capital of France is," and it came back with " Paris." First real end to end answer, correct, running on Tenstorrent hardware. That's the moment this stopped being a math exercise and started being an actual model.
From there it was all about speed:
- Built a host to device streaming system using a standard least recently used cache strategy, verified to produce results bit-identical to a fully resident run, so none of the speed work that followed came at the cost of correctness.
- Fusing the mixture-of-experts math together with the attention calculation and a couple of smaller operations pushed the theoretical compute ceiling from 3.2 tokens a second up to 7.7.
- Getting an honest, live, unified streaming version working, not a benchmark that cheats by pre-staging everything ahead of time, took me from 0.26 up to 0.88 tokens a second.
- Pinning the "generalist" experts, the ones that keep getting reused no matter what you're asking about, into a card's own memory instead of letting them get evicted took my on device hit rate from 25 percent to 42 percent, and it stopped hitting the NVMe drive almost entirely.
- Running the whole thing correctly across both cards together got me to 0.93 tokens a second.
What Didn't Work
With all these wins I still didn't have everything pan out as planned:
- Prefetching experts ahead of time based on the router's own lookahead helps if you're processing a big prompt, what's called prefill, but it does basically nothing for single-user, one token at a time generation, which is most of what an actual conversation looks like.
- I tried concatenating multiple experts together to process in one pass. It was slower, not faster, 0.79 tokens a second, because stitching nine slots together costs about as much as just running the 27 separate matrix multiplications would have.
- I fixed a fused version of the expert feedforward pass. It ran correctly. The speed didn't move at all.
- Expert pinning raised my hit rate nicely, but tokens per second stayed completely flat. That last one turned out to be the most important negative result of the whole project, because it's what told me the bottleneck wasn't where I thought it was.
The Real Wall: It's Not Memory, It's Math
If hit rate goes up and speed doesn't move, the bottleneck isn't how fast you can fetch experts anymore. It's something else entirely.
Here's the something else. These chips process work in fixed size batches, up to 32 rows at a time, and a batch costs about the same amount of compute whether it's full or nearly empty. Chatting with a model one token at a time, which is the normal way anyone actually uses these things, means you're only ever filling 1 of those 32 slots. It's like renting a 32 seat charter bus to drive one person to work every morning. The bus burns about the same amount of gas whether it's full or almost empty.
That's the wall. It isn't a streaming problem. It's a batch of one problem, and it's specific to how these chips are built.
The fully optimized, resident version of this pipeline tops out at 7.7 tokens a second, and that's a ceiling you literally cannot reach while streaming, because a real generator has to fetch routed experts fresh for every single token. A benchmark that pre-stages everything ahead of time can hit that ceiling. A live conversation can't.
So what would it actually take to get to something like 8 or 9 tokens a second for a single user? You'd need enough card memory to keep every expert resident all the time, no streaming at all. For GLM-5.2 at this quantization that's something like 417 GB, which divided by 32 GB a card works out to about 14 cards. That's not a two card home lab experiment anymore. That's Galaxy territory, the same rack scale system I mentioned when I reviewed the QuietBox 2.
How This Stacks Up
For context, here's how this compares to Colibri's own numbers on other hardware:
Setup, Single-user, tok/s
Colibri, Xeon CPU, 0.35
Colibri, 6x RTX 5090, partial 0.12
Colibri, 6x RTX 5090, full resident 6.8
This project, 1 Blackhole card, streaming 0.88
This project, 2 Blackhole cards, streaming 0.93
A couple things jump out. My two card streaming setup already beats Colibri's CPU baseline and its partial six GPU configuration. It's nowhere near the 6.8 tokens a second Colibri hits when it goes full resident across six top end GPUs, but that's not surprising once you understand the math above. 6.8 tokens a second is exactly what you get when nothing needs to stream, because everything's already sitting resident. Same lesson, different hardware.
Colibri’s creator and I agreed on quite a bit. Hit rate, not prefetch overlap, is the real lever to unlock performance. The "generalist" experts that keep getting reused are worth caching. We both measured that the top 20 percent of experts account for close to 58 percent of all routing traffic. And full residency on card really is where the speed comes from. I reused Colibri's int4 weights, its routing trace instrumentation, and some of its pinning and speculative decode thinking, and I want to be clear that credit belongs there.
What's Left on the Table
There's plenty I haven't done yet, roughly in order of effort:
Wire the overlap runtime I built earlier into the actual live generator. It's already built and verified, it's just never been connected. This is a genuinely easy win I left sitting there.
Cut out a small per layer sync step between the host and the cards. Worth maybe 3 percent, but easy.
Borrow Colibri's cross layer prefetching idea for a higher hit rate.
Speculative decoding, guessing a few tokens ahead and checking them in bulk. Could be a 1.5 to 2x win if it pans out, but it's a real engineering lift and the payoff isn't guaranteed.
More cards, to get closer to full residency.
Batching multiple users together, which sidesteps the whole batch of one problem by definition. Worth something like 8x in aggregate throughput.
A sparse attention indexer needed to correctly support longer context windows.
Tracing the decode process at a lower level to remove dispatch overhead entirely. The biggest structural change on this list.
The two biggest levers by far are more cards, which is the only real path to that 8 to 9 tokens a second single user target, and batching multiple users at once, which sidesteps the batch of one problem by definition instead of fighting it.
Bottom Line
A few weeks ago, running GLM-5.2 on these cards would have gotten you a shrug. Today I've got a full 744 billion parameter model streaming correctly, verified piece by piece against the reference implementation, running end to end on two Blackhole cards at just under one token a second. That's about two and a half times faster than the CPU baseline I started from.
The core idea holds up. You genuinely can run a model this size on hardware this small, as long as you accept single user speeds that reflect the fact that you're constantly re-fetching most of what you need. Getting faster than that isn't really a software problem anymore, it's a hardware one: more cards, more residency, or batching enough users together that the math stops caring about a batch of one. To be clear if I can figure out how to implement a fix that can utilize all 32 rows that would speed things up tremendously. It very well might be possible but was just out of scope for the time I had.
This was a fun project and a really cool proof point that raises more interesting questions then it clarifies. I have to put it down for now but this is something I intend to revisit down the road.
Top comments (0)