I want to take you all on a journey with me through the Tenstorrent ecosystem. Before I get going, I'd love to share a little bit about myself.
I've been homelabbing for the better part of a decade, mostly on whatever hardware I can cobble together from used gaming rigs and the odd piece here or there. When the AI boom hit a few years back, I jumped on the train and was an early user of Ollama, among many other tools. Eventually I upgraded from an RTX 3080 to a RTX 3090 that I'm pretty sure had spent years running in some crypto mining rig. The card still ran well, and it had enough VRAM to run significantly bigger models.
That setup worked fine for a number of years, as long as I stuck to light tasks and workloads that didn't put much strain on it. I could generate images or chat with an LLM, but doing more than one of those things at once was always the sticking point. It became one of the biggest pain points in my whole setup.
I have always wanted to build Jarvis type system like we see in the Iron Man movies. But every time I sat down to design it, I ended up making so many trade-offs because of hardware constraints that the end result always fell short.
That's where Tenstorrent came in.
A Quick Disclosure
Before I get too deep into talking about them, I feel it's important to disclose that I work for Tenstorrent. My interest in their work predates my employment, though. In other words, I was an enthusiast who managed to land a dream job at one of my favorite companies.
Now that's out of the way, let's get down to it.
Not Your Grandma's GPU
Tenstorrent hardware is a very different architecture, built with a different mindset and a different set of strengths and weaknesses. A GPU is very fast and powerful at single task inference, but it struggles with parallel workloads beyond a certain point. You hit limits fast.
Tenstorrent is pretty much the opposite. Individual single card inference is typically slower, but the benefits as you scale are far greater, especially for batched workloads. What that means for me is that it's more practical to run agent swarms across a multi-workload system on a Tenstorrent Blackhole card cluster than it would be on, say, a fleet of RTX 5090s. And that's before you even factor in that 5090 pricing puts a fleet of them almost completely out of reach for a home lab.
To be fair, the build I'm about to walk through isn't cheap either. But the methodology behind it can be scaled down. I'll try to explain not just what parts I chose, but why I chose them, so people on any budget can replicate what I'm doing.
What "Scale" Actually Means Here
I used the word "scale" a couple times up top, so let's define it.
In the GPU world, if you want to scale up, you typically buy a bigger card. There are some solutions for linking cards together, and in more extreme DIY builds you'll see people soldering on new VRAM modules to double or quadruple a card's memory. All of these options are either cost prohibitive or pretty limiting. A single A100 has between 40 and 80 GB of RAM and can cost close to as much as a used car.
For contrast, a single Blackhole card is about $1,300 and comes with 32 GB of RAM that can be linked together with other cards for a much larger memory pool. That scalability is extremely attractive to me, since I can buy three or four of these cards for the price of one new RTX 5090.
One of the more interesting things about Tenstorrent's P150A cards is that they connect to each other through a link cable on the back of the card. Each card has four ports, so you can wire them together in different topologies depending on what you're trying to do. In the simplest case, running one port to one port pools two cards' resources into something the computer sees as basically a single resource pool. Technically it's still two cards, but it can function much like one.
There's also a setup where you can spread multiple small models across a larger number of cards without linking them together at all. I might try that down the road, but for this early test it's out of scope. My actual goal right now is to get high performance out of a model like Qwen 3.6, with all the extra features like vision, up and running. I'll have a lot of memory to work with, and I want the largest context window possible at the lowest quantization I can manage while still getting acceptable tokens per second. If this works, I'm going to swap it in as the main brain behind my Jarvis system.
A Bit About the Home Automation System
The system runs on a small Intel PC appliance and connects to remote Bluetooth speakers around the house. The appliance basically acts as a process server, orchestrating tasks and hitting AI via API endpoints. So if I ask it to generate a picture, it might route that request to a different server than if I'm just having a conversation.
That also means I can potentially scale multiple models across a number of different cards and let the best model for the job take over, either through some kind of intelligent mapping, or maybe even a very light LLM handling the routing logic. What we are building today are the model endpoints that this little appliance will consume.
The Hardware, In Detail
I've got the small Intel NUC-style appliance running Ubuntu 24 (all my boxes run the same Ubuntu version). There's the old gaming rig with the RTX 3090 that I'll be transitioning away from, and three separate servers each with their own Blackhole cards, each serving a different purpose:
Box 1: Another old gaming computer with a single P150A, handling smaller workloads. This card isn’t networked into the others and is a stand alone resource. Basically anything that isn't a heavy lift goes here, so I'm not tying up resources on the workhorse machines. If you wanted to do this yourself you could skip this box and save some cash.
Box 2 (the swarm host): A Ryzen AMD box with dual P150As. This one is specced way up: 256 GB of RAM, 4 TB of storage, a Ryzen 9 9000 series processor, and a 1200W PSU to handle the power draw of the cards.
Box 3: Also a Ryzen box with dual P150As, but built very differently. It only has 16 GB of RAM, a Ryzen 5 9000 series processor, and half a terabyte of storage.
The idea is that Box 2 handles all the heavy data transfer into the card cluster when running large models, while Box 3 acts mostly as a lightweight host for its P150A cards, just enough system spec to handle other workloads. I’ve also built Box 3 in a way that should let us try adding a third P150A down the road, just to see if we can get that working. That's going to be quite the experiment, so stay tuned for it.
Putting It All Together
After the parts arrived courtesy of my friendly neighborhood delivery driver, I assembled both systems. One decision that was probably less than ideal: I went with Asus motherboards. It's not that they don't work, it's just that they need some extra setup to get Linux running properly on them. ASRock or Gigabyte have some decent offerings that probably would have saved me a headache or two.
After wrestling through driver issues, I networked the boxes, locked them down, turned on firewalls, and we were ready to go.
The Tenstorrent Software Ecosystem
The first thing you'll notice diving into the Tenstorrent stack is just how open it is. The whole mindset seems to be built around open source, open access, and sharing knowledge. Anyone who knows me knows I'm an open source guy to the max, so that's a huge win in my book.
The second thing you'll notice is that the product is younger in its lifecycle than most GPU offerings. Plug in a GPU, install something like Ollama to handle inference, and you can typically be online within hours, if not minutes. For a lot of people, the most time-consuming part is physically mounting the card and cable managing the power connectors.
But these cards aren’t GPUs. My experience with the Blackhole cards was a different animal. Model support is currently limited, so you end up bringing things up by hand, which takes time. The upside is that the stack is completely open and the community is active, so asking how to do something usually gets you a handful of answers, and a working solution is never far off.
GPUs, by design, have models built to work with their stack from the hardware up through the software. That means tools like Transformers, Diffusers, and llama.cpp are just native to that ecosystem, and there are more guides out there than you could read in a lifetime for setting them up. For Tenstorrent, it's a bit more wild west. For my first run I will need some approximation of the Tenstorrent stack.
First things first, you need to detect the cards, which is where TT-SMI comes in. Second, you need to actually run things on the cards, which requires TT-Metal. Think of TT-Metal as the underlying layer that runs things on the card itself. It includes TT-NN, the actual kernels, along with TT-Transformers and a lot of the other basic tooling you need to get going. On top of that, I'm installing TT-Lang so I can modify things and write my own fused kernels.
If that sounds big and scary, don't worry, we'll break it down as we go. Did I mention this is going to be a fairly long series? The payoff at the end is worth it.
Up Next
That's going to do it for today. In my next article, I'll start breaking down the fundamentals of how all this actually works and how to get started. There are a lot of different paths through this, and my journey is by no means the only one.
Top comments (0)