I did a mad science thing recently and I need to tell you about it. This one didn't touch the whole home lab, just one small piece of it: the master node from the swarm cluster I described back in my first post (the box I called the swarm host there), running just two P150A cards linked together with a cable. That was the entire lab for this experiment. Here's what I was chasing, and what actually happened. Quick disclosure, same as always: I work at Tenstorrent, this was done on my own hardware on my own time, and I'll flag what didn't work just as readily as what did.
The Big Question
The question I set out to answer was easy to ask and hard to answer: can you run a genuinely frontier sized model, in this case GLM-5.2, a 744 billion parameter mixture of experts model, on a small two card box? Not a shrunk down version. The real thing.
The trick with a mixture of experts model is that not all of those 744 billion parameters do anything for any given word you generate. The model routes each token to a small handful of "expert" sub networks and ignores the rest. For GLM-5.2, something like 40 billion parameters are actually active per token. The other 700-plus billion just sit there, waiting to be needed for a different token later.
I didn't come up with this idea on my own. Credit goes to a project called Colibri, a CPU and GPU engine that already proved you could run GLM-5.2 in about 25 GB of memory by keeping the common stuff resident and streaming the rest of the experts in from RAM or an SSD as needed, using a lookahead prefetch system they call PILOT along with cache aware routing and speculative decoding. Colibri runs on CPU, CUDA, and Metal. Nobody had ported this approach to Tenstorrent hardware yet, and that gap is exactly what got me curious.
There was a second project in the mix too, a fast Rust based tokenizer called gigatoken. I looked at it and shelved it pretty quickly. Tokenization takes microseconds per request. It was never going to be the bottleneck here. Good tool, wrong problem.
Early testing also gave me a number worth keeping in mind for the rest of this post: grabbing an expert that's already sitting in a card's memory costs something like 13 times less than streaming one in from RAM or disk. So the entire game becomes maximizing how often the token you're generating already has what it needs close by.
The Setup
For hardware, I used the same box I called the swarm host in my first post. Two Blackhole P150A cards, 32 GB each, linked together with the cable on the back. The host itself is a Ryzen 9900X with about 238 GB of RAM. One card talks to the host over a full Gen5 x8 link at 28 GB/s. The other is stuck on a x4 link at 14 GB/s, so there's already an asymmetry to work around before you get to anything clever. The cards can talk to each other fast over their own fabric connection, NVMe storage on the box does about 7 GB/s, and each card's own memory reads at around 312 GB/s once data is actually sitting on it.
Think of it like a kitchen. A card's own memory is the counter right in front of you, fast to grab from. RAM is the pantry down the hall. NVMe is the grocery store across town. The whole game here is minimizing trips to the grocery store.
Getting a Baseline
Before optimizing anything, I needed to know what slow actually looked like. Running Colibri's own GLM-5.2 setup on CPU gave me 0.35 tokens a second, with about 35 seconds before the first word even showed up. That was the number to beat.
Building It Up
From there it was a lot of careful, unglamorous verification work before I worried about speed at all. Every piece of the model, meaning the normalization step, the routing logic, the expert layers themselves, and the attention mechanism, had to be checked against Hugging Face's own reference implementation using a correlation score called PCC (think of it as a similarity score between two sets of numbers, where 1.0 means identical). Everything I built came back above 0.999, and the full decoder layer landed at 0.999984 against real weights. My math wasn't just close. It was basically indistinguishable from the reference.
Then came the moment of truth. I ran the full 78 layer model, with real 4 bit quantized weights (the same low quantization I said I wanted back in my first post), fed it "The capital of France is," and it came back with " Paris." First real end to end answer, correct, running on Tenstorrent hardware. That's the moment this stopped being a math exercise and started being an actual model.
From there it was all about speed:
Built a host to device streaming system using a standard least recently used cache strategy, verified to produce results bit-identical to a fully resident run, so none of the speed work that followed came at the cost of correctness.
Fusing the mixture-of-experts math together with the attention calculation and a couple of smaller operations pushed the theoretical compute ceiling from 3.2 tokens a second up to 7.7.
Getting an honest, live, unified streaming version working, not a benchmark that cheats by pre-staging everything ahead of time, took me from 0.26 up to 0.88 tokens a second.
Pinning the "generalist" experts, the ones that keep getting reused no matter what you're asking about, into a card's own memory instead of letting them get evicted took my on device hit rate from 25 percent to 42 percent, and it stopped hitting the NVMe drive almost entirely.
Running the whole thing correctly across both cards together got me to 0.93 tokens a second.
What Didn't Work
With all these wins I still didn't have everything pan out as planned:
Prefetching experts ahead of time based on the router's own lookahead helps if you're processing a big prompt, what's called prefill, but it does basically nothing for single-user, one token at a time generation, which is most of what an actual conversation looks like.
I tried concatenating multiple experts together to process in one pass. It was slower, not faster, 0.79 tokens a second, because stitching nine slots together costs about as much as just running the 27 separate matrix multiplications would have.
I fixed a fused version of the expert feedforward pass. It ran correctly. The speed didn't move at all.
Expert pinning raised my hit rate nicely, but tokens per second stayed completely flat.
That last one turned out to be the most important negative result of the whole project, because it's what told me the bottleneck wasn't where I thought it was.
The Real Wall is Bandwidth
If hit rate goes up and speed doesn't move, the miss penalty isn't the whole story. It's tempting to call that "compute bound," but that's not quite right, and the numbers make a cleaner case for what's actually happening.
GLM-5.2's active weights per token come out to somewhere around 20 to 24 GB (routed experts at roughly 11 GB plus attention and dense layers at roughly 13 GB, at this quantization). Device memory reads at about 312 GB/s once data is already sitting on the card. That puts just reading already-resident weights at 64 to 77 milliseconds, before a single matmul runs. The measured resident ceiling is 7.7 tokens a second, or about 130 milliseconds a token. So somewhere between half and sixty percent of even the best-case time budget is spent reading weights, not computing on them. That's a memory bandwidth cost, and it doesn't care how good your hit rate is, because a hit still has to be read, just from somewhere faster. This is the normal rule for single-token decode on basically any accelerator, and MoE makes it worse, not better, since the reads are scattered across experts instead of one dense block.
There's a second, separate cost stacked on top of that. These chips run in fixed 32-row tiles, and a tile costs about the same whether 1 row or 32 rows are doing real work. Chatting with a model one token at a time means you're only ever filling 1 of those 32 slots. It's like renting a 32 seat charter bus to drive one person to work every morning. The bus burns about the same amount of gas whether it's full or almost empty. That's a parallelism and utilization problem, not evidence the math engine itself is maxed out.
Pinning experts fixed a third, separate cost: the additive penalty of fetching whatever's missing from RAM or NVMe, without touching either the bandwidth floor or the utilization floor above. That's exactly why hit rate went up, 25 to 42 percent, while tokens a second stayed flat. Two of the three costs stacked up here don't care about hit rate at all.
Here's the part that surprised me most once I added it all up. That 130 millisecond number is the idealized floor: resident weights, full tile, zero streaming. The live streaming generator actually runs at around 1,100 milliseconds a token. Almost all of that extra gap is dispatch overhead, hundreds of small host-to-device copy launches and an eager Python re-staging path for every single token, none of which the fused, offline benchmark had to pay for. So the honest streaming number is really overhead bound, stacked on top of bandwidth, stacked on top of utilization. Three different costs, and only one of them, the fetch penalty, is what pinning can actually touch.
That reshuffles what I'd chase next. More cards to reach full residency still matters for closing the fetch-cost gap long term (the math there hasn't changed: roughly 417 GB of active weights divided by 32 GB a card, so about 14 cards, Galaxy territory). But given that overhead alone accounts for most of the gap between 130 milliseconds and 1,100, the nearer term, and probably bigger, win is applying the same fusion work that got the offline benchmark from 3.2 to 7.7 tokens a second to the live generator's actual per-token path. I haven't done that yet. It's now at the top of my list.
How This Stacks Up
For context, here's how this compares to Colibri's own numbers on other hardware:
| Setup | Single-user tok/s |
|---|---|
| Colibri, Xeon CPU | 0.35 |
| Colibri, 6x RTX 5090, partial | 0.12 |
| Colibri, 6x RTX 5090, full resident | 6.8 |
| This project, 1 Blackhole card, streaming | 0.88 |
| This project, 2 Blackhole cards, streaming | 0.93 |
A couple things jump out. My two card streaming setup already beats Colibri's CPU baseline and its partial six GPU configuration. It's nowhere near the 6.8 tokens a second Colibri hits when it goes full resident across six top end GPUs, but that's not surprising once you understand the math above. 6.8 tokens a second is exactly what you get when nothing needs to stream, because everything's already sitting resident. Same lesson, different hardware.
Colibri's creator and I agreed on quite a bit. Hit rate, not prefetch overlap, is the real lever to unlock performance. The "generalist" experts that keep getting reused are worth caching. We both measured that the top 20 percent of experts account for close to 58 percent of all routing traffic. And full residency on card really is where the speed comes from. I reused Colibri's int4 weights, its routing trace instrumentation, and some of its pinning and speculative decode thinking, and I want to be clear that credit belongs there.
What's Left on the Table
There's plenty I haven't done yet, roughly in order of effort:
- Wire the overlap runtime I built earlier into the actual live generator. It's already built and verified, it's just never been connected. This is a genuinely easy win I left sitting there.
- Cut out a small per layer sync step between the host and the cards. Worth maybe 3 percent, but easy.
- Borrow Colibri's cross layer prefetching idea for a higher hit rate.
- Speculative decoding, guessing a few tokens ahead and checking them in bulk. Could be a 1.5 to 2x win if it pans out, but it's a real engineering lift and the payoff isn't guaranteed.
- More cards, to get closer to full residency.
- Batching multiple users together, which sidesteps the whole batch of one problem by definition. Worth something like 8x in aggregate throughput.
- A sparse attention indexer needed to correctly support longer context windows.
- Tracing the decode process at a lower level and applying the same fusion work from the offline benchmark to the live generator's actual per-token path, to close the overhead gap between the idealized floor and the real generator. This looks like the single biggest lever on this entire list.
- Given where the time actually goes, more cards and batching are still real levers for the long run, but closing that overhead gap is probably the cheapest, nearest term win of everything on this list.
Bottom Line
A few weeks ago, running GLM-5.2 on these cards would have gotten you a shrug. Today I've got a full 744 billion parameter model streaming correctly, verified piece by piece against the reference implementation, running end to end on two Blackhole cards at just under one token a second. That's about two and a half times faster than the CPU baseline I started from.
The core idea holds up. You genuinely can run a model this size on hardware this small, as long as you accept single user speeds that reflect the fact that you're constantly re-fetching most of what you need. Getting faster from here is mostly a software problem right now, not a hardware one. The gap between the idealized 130 millisecond floor and the roughly 1,100 milliseconds a token the live generator actually takes is dominated by dispatch overhead, hundreds of small copy launches and an eager Python re-staging path, which is about as pure a software problem as it gets. More cards and more residency still matter for the long run target, but closing that overhead gap first is probably the bigger, cheaper win, and it's one I can chase without buying anything. To be clear, if I can also figure out how to utilize all 32 rows, that would speed things up tremendously on top of that. It very well might be possible, it was just out of scope for the time I had.
This was a fun project and a really cool proof point that raises more interesting questions than it clarifies. I have to put it down for now, but this is something I intend to revisit down the road.
Top comments (0)