DEV Community: Eric Zietlow

Squeezing a 744B Model Onto Two Tenstorrent Cards... Kinda

Eric Zietlow — Mon, 10 Aug 2026 15:29:44 +0000

I did a mad science thing recently and I need to tell you about it. This one didn't touch the whole home lab, just one small piece of it: the master node from the swarm cluster I described back in my first post (the box I called the swarm host there), running just two p150a cards linked together with a cable was the entire lab for this experiment. Here's what I was chasing, and what actually happened.
Quick disclosure, same as always: I work at Tenstorrent and I'll flag what didn't work just as readily as what did.

The Big Question

The question I set out to answer was easy to ask and hard to answer: can you run a genuinely frontier sized model, in this case GLM-5.2, a 744 billion parameter mixture of experts model, on a small two card box? Not a shrunk down version. The real thing.
The trick with a mixture of experts model is that not all of those 744 billion parameters do anything for any given word you generate. The model routes each token to a small handful of "expert" sub networks and ignores the rest. For GLM-5.2, something like 40 billion parameters are actually active per token. The other 700-plus billion just sit there, waiting to be needed for a different token later.
I didn't come up with this idea on my own. Credit goes to a project called Colibri, a CPU and GPU engine that already proved you could run GLM-5.2 in about 25 GB of memory by keeping the common stuff resident and streaming the rest of the experts in from RAM or an SSD as needed, using a lookahead prefetch system they call PILOT along with cache-aware routing and speculative decoding. Colibri runs on CPU, CUDA, and Metal. Nobody had ported this approach to Tenstorrent hardware yet, and that gap is exactly what got me curious.
Early testing also gave me a number worth keeping in mind for the rest of this post: grabbing an expert that's already sitting in a card's memory costs something like 13 times less than streaming one in from RAM or disk. So the entire game becomes maximizing how often the token you're generating already has what it needs close by.

The Setup

For hardware, I used the same box I called the swarm host in my first post. Two Blackhole p150a cards, 32 GB each, linked together with the 800G cable on the back. The host itself is a Ryzen 9900X with about 256 GB of RAM. One card talks to the host over a full Gen5 x8 link at 28 GB/s. The other is stuck on a x4 link at 14 GB/s, so there's already an asymmetry to work around before you get to anything clever. The cards can talk to each other fast over their own fabric connection, NVMe storage on the box does about 7 GB/s, and each card's own memory reads at around 312 GB/s once data is actually sitting on it.
Think of it like a kitchen. A card's own memory is the counter right in front of you, fast to grab from. RAM is the pantry down the hall. NVMe is the grocery store across town. The whole game here is minimizing trips to the grocery store.

Getting a Baseline

Before optimizing anything, I needed to know what slow actually looked like. Running Colibri's own GLM-5.2 setup on CPU gave me 0.35 tokens a second, with about 35 seconds before the first word even showed up. That was the number to beat.

Building It Up

From there it was a lot of careful, unglamorous verification work before I worried about speed at all. Every piece of the model, meaning the normalization step, the routing logic, the expert layers themselves, and the attention mechanism, had to be checked against Hugging Face's own reference implementation using a correlation score called PCC (think of it as a similarity score between two sets of numbers, where 1.0 means identical). Everything I built came back above 0.999, and the full decoder layer landed at 0.999984 against real weights. My math wasn't just close. It was basically indistinguishable from the reference.

Then came the moment of truth. I ran the full 78 layer model, with real 4 bit quantized weights (the same low quantization I said I wanted back in my first post), fed it "The capital of France is," and it came back with " Paris." First real end to end answer, correct, running on Tenstorrent hardware. That's the moment this stopped being a math exercise and started being an actual model.

From there it was all about speed:

Built a host to device streaming system using a standard least recently used cache strategy, verified to produce results bit-identical to a fully resident run, so none of the speed work that followed came at the cost of correctness.
Fusing the mixture-of-experts math together with the attention calculation and a couple of smaller operations pushed the theoretical compute ceiling from 3.2 tokens a second up to 7.7.
Getting an honest, live, unified streaming version working, not a benchmark that cheats by pre-staging everything ahead of time, took me from 0.26 up to 0.88 tokens a second.
Pinning the "generalist" experts, the ones that keep getting reused no matter what you're asking about, into a card's own memory instead of letting them get evicted took my on device hit rate from 25 percent to 42 percent, and it stopped hitting the NVMe drive almost entirely.
Running the whole thing correctly across both cards together got me to 0.93 tokens a second.

What Didn't Work

With all these wins I still didn't have everything pan out as planned:

Prefetching experts ahead of time based on the router's own lookahead helps if you're processing a big prompt, what's called prefill, but it does basically nothing for single-user, one token at a time generation, which is most of what an actual conversation looks like.
I tried concatenating multiple experts together to process in one pass. It was slower, not faster, 0.79 tokens a second, because stitching nine slots together costs about as much as just running the 27 separate matrix multiplications would have.
I fixed a fused version of the expert feedforward pass. It ran correctly. The speed didn't move at all.
Expert pinning raised my hit rate nicely, but tokens per second stayed completely flat. That last one turned out to be the most important negative result of the whole project, because it's what told me the bottleneck wasn't where I thought it was.

The Real Wall: It's Not Memory, It's Math

If hit rate goes up and speed doesn't move, the bottleneck isn't how fast you can fetch experts anymore. It's something else entirely.
Here's the something else. These chips process work in fixed size batches, up to 32 rows at a time, and a batch costs about the same amount of compute whether it's full or nearly empty. Chatting with a model one token at a time, which is the normal way anyone actually uses these things, means you're only ever filling 1 of those 32 slots. It's like renting a 32 seat charter bus to drive one person to work every morning. The bus burns about the same amount of gas whether it's full or almost empty.

That's the wall. It isn't a streaming problem. It's a batch of one problem, and it's specific to how these chips are built.
The fully optimized, resident version of this pipeline tops out at 7.7 tokens a second, and that's a ceiling you literally cannot reach while streaming, because a real generator has to fetch routed experts fresh for every single token. A benchmark that pre-stages everything ahead of time can hit that ceiling. A live conversation can't.
So what would it actually take to get to something like 8 or 9 tokens a second for a single user? You'd need enough card memory to keep every expert resident all the time, no streaming at all. For GLM-5.2 at this quantization that's something like 417 GB, which divided by 32 GB a card works out to about 14 cards. That's not a two card home lab experiment anymore. That's Galaxy territory, the same rack scale system I mentioned when I reviewed the QuietBox 2.

How This Stacks Up

For context, here's how this compares to Colibri's own numbers on other hardware:
Setup, Single-user, tok/s
Colibri, Xeon CPU, 0.35
Colibri, 6x RTX 5090, partial 0.12
Colibri, 6x RTX 5090, full resident 6.8
This project, 1 Blackhole card, streaming 0.88
This project, 2 Blackhole cards, streaming 0.93

A couple things jump out. My two card streaming setup already beats Colibri's CPU baseline and its partial six GPU configuration. It's nowhere near the 6.8 tokens a second Colibri hits when it goes full resident across six top end GPUs, but that's not surprising once you understand the math above. 6.8 tokens a second is exactly what you get when nothing needs to stream, because everything's already sitting resident. Same lesson, different hardware.
Colibri’s creator and I agreed on quite a bit. Hit rate, not prefetch overlap, is the real lever to unlock performance. The "generalist" experts that keep getting reused are worth caching. We both measured that the top 20 percent of experts account for close to 58 percent of all routing traffic. And full residency on card really is where the speed comes from. I reused Colibri's int4 weights, its routing trace instrumentation, and some of its pinning and speculative decode thinking, and I want to be clear that credit belongs there.

What's Left on the Table

There's plenty I haven't done yet, roughly in order of effort:
Wire the overlap runtime I built earlier into the actual live generator. It's already built and verified, it's just never been connected. This is a genuinely easy win I left sitting there.
Cut out a small per layer sync step between the host and the cards. Worth maybe 3 percent, but easy.

Borrow Colibri's cross layer prefetching idea for a higher hit rate.
Speculative decoding, guessing a few tokens ahead and checking them in bulk. Could be a 1.5 to 2x win if it pans out, but it's a real engineering lift and the payoff isn't guaranteed.

More cards, to get closer to full residency.
Batching multiple users together, which sidesteps the whole batch of one problem by definition. Worth something like 8x in aggregate throughput.

A sparse attention indexer needed to correctly support longer context windows.

Tracing the decode process at a lower level to remove dispatch overhead entirely. The biggest structural change on this list.
The two biggest levers by far are more cards, which is the only real path to that 8 to 9 tokens a second single user target, and batching multiple users at once, which sidesteps the batch of one problem by definition instead of fighting it.

Bottom Line

A few weeks ago, running GLM-5.2 on these cards would have gotten you a shrug. Today I've got a full 744 billion parameter model streaming correctly, verified piece by piece against the reference implementation, running end to end on two Blackhole cards at just under one token a second. That's about two and a half times faster than the CPU baseline I started from.
The core idea holds up. You genuinely can run a model this size on hardware this small, as long as you accept single user speeds that reflect the fact that you're constantly re-fetching most of what you need. Getting faster than that isn't really a software problem anymore, it's a hardware one: more cards, more residency, or batching enough users together that the math stops caring about a batch of one. To be clear if I can figure out how to implement a fix that can utilize all 32 rows that would speed things up tremendously. It very well might be possible but was just out of scope for the time I had.

This was a fun project and a really cool proof point that raises more interesting questions then it clarifies. I have to put it down for now but this is something I intend to revisit down the road.

Hands-On With the Tenstorrent QuietBox 2

Eric Zietlow — Mon, 03 Aug 2026 17:10:41 +0000

Quick Disclosure: I want to disclose that I work for the company that built this box. I'm going to do my best to give an unbiased opinion here: the good, the bad, and the ugly.

It Begins

My first hands-on experience with the QuietBox 2 was pretty cool. It arrived later in the day, so I didn't have to wait until I got off work to start playing with it.

The first thing I noticed was the heft of the box itself. This felt a lot more like server equipment than a simple PC tower. Upon opening it, I was greeted by an obviously custom case with two gigantic fans and some pretty cool cosmetic touches. The box itself isn't obtrusive or gaudy. It just looks like it would fit neatly on someone's desk, and it would be just as comfortable pretending to be a gaming rig as it would be the sleeper AI Ferrari that it actually is. The liquid cooling channels on the front were a nice touch. Even before I booted it up, it looked pretty amazing.

Diving a little deeper, the box came with a speaker and some decals, which I set aside for later. I can totally see the intention behind this computer. It's clearly meant to be a household-appliance-style AI server.

More Than Just an Accelerator

The box itself is an incredibly powerful unit, and I don't just mean as an inference server. As a usable desktop workstation, or even a medium to heavy load server, it's quite capable. 256 GB of RAM, a high-end consumer CPU, and enough storage for pretty much any task make it clear that this box isn't the equivalent of just buying a GPU accelerator. It's more than that. It's trying to be a full solution.

As someone who has been building out a home lab for years, this is intriguing, because it means I can treat the box as a truly self-contained appliance. Something like a home assistant could theoretically be built out exclusively on this box, with all the different parts and complex pieces running locally instead of spread out across boxes on a network.

The liquid cooling and the big radiator fans make this an almost illegitimately quiet box for a config with zero hardware trade-offs. If I were building a system like this myself, I'd have to spend quite a lot of money to get anything remotely close to this noise profile at this power level. That said, if you really wanted to, I'm sure you could pull it off.

The Accelerator Cards

Let's talk about the accelerator cards for a second, because these are unique to this system. When you first boot up and look at what's attached, the cards show up as four, even though the box itself only has two physical cards installed. This comes down to the dual-chip design of each P300C card. The end result is something close to four P150A cards' worth of compute in a single box, while only taking up two PCIe slots.

There are a few reasons for this. Anyone who has tried to design a four-card system with full-size, dual-width, high-power-draw cards can tell you it presents a unique challenge. You're pretty much off consumer hardware from the get-go and onto server-grade EPYC or Threadripper CPUs before you even start, and the problem only gets harder from there. That's why these two dual-chip cards make sense in this build. They let the price stay well below what a true four-card system would cost, while still providing the power and bandwidth to handle what's actually running.

The two cards are linked together in what's called a 1x4 fabric configuration. That means each chip functions like an individual card on its own, but they're networked together into a mesh so the resources get pooled when needed. The end result is 128 GB of usable memory (GDDR6) across the accelerator setup, which lets you load models much bigger than you'd manage on a single GPU.

Built Like a Tank

Let me change gears for a moment and talk about the case. This thing feels tough, and I don't mean tough like rigid plastic. It feels like it was built out of armor. It weighs a little over 50 lb, and moving it by the carry handles feels more like carrying an upright server than a desktop tower. You won't find any flimsy aluminum paneling here, just raw, no-nonsense, compact form factor power, contained in a chassis that feels tough enough to shield you from flying debris in a natural disaster.

The case is pretty cool beyond that too. It uses a modified clamshell design where both side panels open to reveal what's essentially a bifurcated physical layout. Each of the P300C cards sits on either side of the motherboard, upright. Both are monsters, coming in at about 7 lb each and taking up three slots, even with the more compact liquid cooling. When the box really gets going, it gives off a ton of heat, but again, nothing a high-end gaming rig wouldn't already do.

Another cool feature is the liquid reservoir built into the front of the case. You can actually see the fluid moving through it while the computer is running. This is definitely one of the more interesting liquid cooling solutions I've seen, and it gives the whole thing a unique vibe.

A Week In

In the week since I started using it, I've been pleasantly surprised. It's quickly become a workhorse in my home lab. I've thrown a lot at it and haven't come close to taxing it fully. I've got a number of projects that I'll hopefully be able to blog about soon, but until then, you'll have to make do with this teaser.
So What Is This Thing, Really?
I'd like to point out what this box isn't. It very clearly isn't what you should buy if you want a scalable system. There's no path to link these together like there would be with Tenstorrent's Galaxy system, and if you only need one or two cards, buying them separately and building your own box will be cheaper.

This box doesn't really feel like it's trying to compete with either of those scenarios though. It's something unique, more like an appliance that can handle everything, and it wouldn't surprise me if this type of system eventually becomes as common in a house as a refrigerator, washer, or dryer.

Just the brain of the house. It's what you get if you don't want to pay a subscription to a major LLM provider, and beyond that, it can run all your home automation and anything else you need. Various privacy movements, like the de-Googleify movement, would make great use of hardware like this.

Now for the Bad (and the Ugly)

The software stack's maturity definitely isn't where GPUs have been for a while now. That means while the box can run very large models, it also has some limitations around bottlenecks, and the kernels just aren't as refined as what you'd get on a lot of GPUs. Throughput isn't always quite where you'd want it to be.

That said, concurrency is this box's strong suit, and it can outpace any other setup I've had the privilege of testing, many times over. Running a single model with 30 open agents, all batching their work against the accelerator cards, is the kind of power you need to run a bunch of different systems on a single box. That, more than anything, is where this box shines.

Set your model up and go to town. This is a powerhouse and a workhorse. It's not trying to be anything else, and it doesn't need to be.

Final Verdict

So where does the QuietBox 2 actually land? If you need something that scales out into a real cluster, this isn't your box, and it was never trying to be. What it is, is one of the more interesting all-in-one AI appliances I've gotten my hands on. It's quiet, it's built like a tank, and it has more than enough headroom to run as a genuine home brain instead of just another box tucked in a rack.

The software stack still shows its sharp edges in places, and that's worth going in with eyes open. But the concurrency numbers make up for a lot, and this ecosystem is moving fast enough that today's rough edges probably won't stick around for long.

Between this and the P150A cluster I've been building out, I've now got two very different answers to the same question: how do you put real AI compute in your house without handing your data, or your wallet, over to someone else every month. One approach is fully DIY. The other shows up ready to go out of the box. I'll keep living with both and report back on how they stack up.

That's a wrap on my first week with the QuietBox 2. Thanks for reading, and stay tuned.

My Tenstorrent Journey, Part 1: Why I'm Building This Thing

Eric Zietlow — Mon, 20 Jul 2026 13:28:09 +0000

I want to take you all on a journey with me through the Tenstorrent ecosystem. Before I get going, I'd love to share a little bit about myself.

I've been homelabbing for the better part of a decade, mostly on whatever hardware I can cobble together from used gaming rigs and the odd piece here or there. When the AI boom hit a few years back, I jumped on the train and was an early user of Ollama, among many other tools. Eventually I upgraded from an RTX 3080 to a RTX 3090 that I'm pretty sure had spent years running in some crypto mining rig. The card still ran well, and it had enough VRAM to run significantly bigger models.

That setup worked fine for a number of years, as long as I stuck to light tasks and workloads that didn't put much strain on it. I could generate images or chat with an LLM, but doing more than one of those things at once was always the sticking point. It became one of the biggest pain points in my whole setup.

I have always wanted to build Jarvis type system like we see in the Iron Man movies. But every time I sat down to design it, I ended up making so many trade-offs because of hardware constraints that the end result always fell short.

That's where Tenstorrent came in.

A Quick Disclosure

Before I get too deep into talking about them, I feel it's important to disclose that I work for Tenstorrent. My interest in their work predates my employment, though. In other words, I was an enthusiast who managed to land a dream job at one of my favorite companies.

Now that's out of the way, let's get down to it.
Not Your Grandma's GPU
Tenstorrent hardware is a very different architecture, built with a different mindset and a different set of strengths and weaknesses. A GPU is very fast and powerful at single task inference, but it struggles with parallel workloads beyond a certain point. You hit limits fast.

Tenstorrent is pretty much the opposite. Individual single card inference is typically slower, but the benefits as you scale are far greater, especially for batched workloads. What that means for me is that it's more practical to run agent swarms across a multi-workload system on a Tenstorrent Blackhole card cluster than it would be on, say, a fleet of RTX 5090s. And that's before you even factor in that 5090 pricing puts a fleet of them almost completely out of reach for a home lab.

To be fair, the build I'm about to walk through isn't cheap either. But the methodology behind it can be scaled down. I'll try to explain not just what parts I chose, but why I chose them, so people on any budget can replicate what I'm doing.
What "Scale" Actually Means Here
I used the word "scale" a couple times up top, so let's define it.

In the GPU world, if you want to scale up, you typically buy a bigger card. There are some solutions for linking cards together, and in more extreme DIY builds you'll see people soldering on new VRAM modules to double or quadruple a card's memory. All of these options are either cost prohibitive or pretty limiting. A single A100 has between 40 and 80 GB of RAM and can cost close to as much as a used car.

For contrast, a single Blackhole card is about $1,300 and comes with 32 GB of RAM that can be linked together with other cards for a much larger memory pool. That scalability is extremely attractive to me, since I can buy three or four of these cards for the price of one new RTX 5090.

One of the more interesting things about Tenstorrent's P150A cards is that they connect to each other through a link cable on the back of the card. Each card has four ports, so you can wire them together in different topologies depending on what you're trying to do. In the simplest case, running one port to one port pools two cards' resources into something the computer sees as basically a single resource pool. Technically it's still two cards, but it can function much like one.

There's also a setup where you can spread multiple small models across a larger number of cards without linking them together at all. I might try that down the road, but for this early test it's out of scope. My actual goal right now is to get high performance out of a model like Qwen 3.6, with all the extra features like vision, up and running. I'll have a lot of memory to work with, and I want the largest context window possible at the lowest quantization I can manage while still getting acceptable tokens per second. If this works, I'm going to swap it in as the main brain behind my Jarvis system.

A Bit About the Home Automation System

The system runs on a small Intel PC appliance and connects to remote Bluetooth speakers around the house. The appliance basically acts as a process server, orchestrating tasks and hitting AI via API endpoints. So if I ask it to generate a picture, it might route that request to a different server than if I'm just having a conversation.

That also means I can potentially scale multiple models across a number of different cards and let the best model for the job take over, either through some kind of intelligent mapping, or maybe even a very light LLM handling the routing logic. What we are building today are the model endpoints that this little appliance will consume.

The Hardware, In Detail

I've got the small Intel NUC-style appliance running Ubuntu 24 (all my boxes run the same Ubuntu version). There's the old gaming rig with the RTX 3090 that I'll be transitioning away from, and three separate servers each with their own Blackhole cards, each serving a different purpose:

Box 1: Another old gaming computer with a single P150A, handling smaller workloads. This card isn’t networked into the others and is a stand alone resource. Basically anything that isn't a heavy lift goes here, so I'm not tying up resources on the workhorse machines. If you wanted to do this yourself you could skip this box and save some cash.
Box 2 (the swarm host): A Ryzen AMD box with dual P150As. This one is specced way up: 256 GB of RAM, 4 TB of storage, a Ryzen 9 9000 series processor, and a 1200W PSU to handle the power draw of the cards.
Box 3: Also a Ryzen box with dual P150As, but built very differently. It only has 16 GB of RAM, a Ryzen 5 9000 series processor, and half a terabyte of storage.

The idea is that Box 2 handles all the heavy data transfer into the card cluster when running large models, while Box 3 acts mostly as a lightweight host for its P150A cards, just enough system spec to handle other workloads. I’ve also built Box 3 in a way that should let us try adding a third P150A down the road, just to see if we can get that working. That's going to be quite the experiment, so stay tuned for it.

Putting It All Together

After the parts arrived courtesy of my friendly neighborhood delivery driver, I assembled both systems. One decision that was probably less than ideal: I went with Asus motherboards. It's not that they don't work, it's just that they need some extra setup to get Linux running properly on them. ASRock or Gigabyte have some decent offerings that probably would have saved me a headache or two.

After wrestling through driver issues, I networked the boxes, locked them down, turned on firewalls, and we were ready to go.
The Tenstorrent Software Ecosystem
The first thing you'll notice diving into the Tenstorrent stack is just how open it is. The whole mindset seems to be built around open source, open access, and sharing knowledge. Anyone who knows me knows I'm an open source guy to the max, so that's a huge win in my book.

The second thing you'll notice is that the product is younger in its lifecycle than most GPU offerings. Plug in a GPU, install something like Ollama to handle inference, and you can typically be online within hours, if not minutes. For a lot of people, the most time-consuming part is physically mounting the card and cable managing the power connectors.

But these cards aren’t GPUs. My experience with the Blackhole cards was a different animal. Model support is currently limited, so you end up bringing things up by hand, which takes time. The upside is that the stack is completely open and the community is active, so asking how to do something usually gets you a handful of answers, and a working solution is never far off.

GPUs, by design, have models built to work with their stack from the hardware up through the software. That means tools like Transformers, Diffusers, and llama.cpp are just native to that ecosystem, and there are more guides out there than you could read in a lifetime for setting them up. For Tenstorrent, it's a bit more wild west. For my first run I will need some approximation of the Tenstorrent stack.

First things first, you need to detect the cards, which is where TT-SMI comes in. Second, you need to actually run things on the cards, which requires TT-Metal. Think of TT-Metal as the underlying layer that runs things on the card itself. It includes TT-NN, the actual kernels, along with TT-Transformers and a lot of the other basic tooling you need to get going. On top of that, I'm installing TT-Lang so I can modify things and write my own fused kernels.

If that sounds big and scary, don't worry, we'll break it down as we go. Did I mention this is going to be a fairly long series? The payoff at the end is worth it.

Up Next

That's going to do it for today. In my next article, I'll start breaking down the fundamentals of how all this actually works and how to get started. There are a lot of different paths through this, and my journey is by no means the only one.