DEV Community: Eric Zietlow

Coding with My QuietBox

Eric Zietlow — Mon, 21 Sep 2026 12:58:52 +0000

Quick Disclosure: same as always, I work for Tenstorrent. That said I'll call out the rough edges as readily as the wins.

I've spent the last few months getting real AI compute into my house. Big models, weird streaming tricks, a QuietBox 2 sitting in my lab humming away. Most of that has been benchmarking work. At some point you have to stop measuring the thing and start using it.

So here's the using it. I pointed OpenCode, a terminal coding agent, at my own inference server, and told it to build me a Tetris clone. The whole thing stayed on my LAN. Setup took about ninety seconds, generation took a minute nine, and it worked.

Why Coding Agents

Coding agents are a good stress test for a local setup. They need tool calling that actually works, since the agent has to read and write files on its own. They need a long context window, because it's pulling your project into the prompt. And they need to hold a plan together across several steps.

That's a lot of surface area to get right at once. When it all lines up, you get a pretty strong signal that the rest of your local tooling will work too.

On the QB2 Side

This part is a single command. It will pull the model with its weights, then serve it.

tt serve mando2222/qwen3.8-27b-dflash2-p300x2-q4kv

That's Qwen3.8-27B, quantized to 8 bit with a Q4 KV cache, sharded across the two P300 cards in the QuietBox 2. If you read my QuietBox post, this is the 1x4 fabric config doing its thing: four chips across two physical cards, pooling their memory so a 27B model with a big context window fits comfortably.

The serve command gives you an OpenAI compatible endpoint. That compatibility is the whole ballgame for what comes next. It means tools built for the OpenAI API can point at your box with a config change and no code change.

Where That Model Came From

Those two commands hide a few months of work, and a good chunk of it isn't mine.

The model string qwen3.8-27b-dflash2-p300x2-q4kv looks like a mouthful, but every segment of it is a decision somebody had to make. Which attention implementation to use. How the weights get split across the cards. What precision the KV cache runs at, which is the difference between a comfortable 256k context window and running out of memory halfway through a long file. I didn't work any of that out from a blank page.

tt-model has a growing set of published model configs, and people have been putting real effort into them. I started by pulling down what was already there and reading it carefully. Different configs were good at different things. One had the attention path sorted out. Another had a smarter approach to how the shards were laid out across chips. A third had done the legwork on quantization quality, which is the part where it's easy to save memory and quietly lose a chunk of the model's ability to follow instructions.

Most of what I did was integration work. Take the win from one config, take the win from another, get them to coexist, then measure whether the combination actually held up instead of assuming it would. That last step is where most of the time went. A lot of runs, a lot of tweaking one variable at a time, a lot of results that looked promising and didn't survive a second look.

None of that would have happened on the timeline it did if the starting points weren't already out there in public. I want to be clear about that, same as I was with Colibri in my earlier post. Credit belongs upstream. The reason I published this config back to tt-model instead of keeping it in a folder on my box is that it's how the whole thing keeps working. Somebody's going to pull this down, find the three things I got wrong, and publish something better.

On the Mac Side

With OpenCode installed and your inference server running, connecting them is four steps.

1. Get the model ID from your server

curl -fsS http://YOURIP:20000/v1/models

Replace the address with your server's IP and port. Use the exact model ID that comes back. In my case that's Qwen/Qwen3.8-27B. This matters more than it looks like it should. If this string doesn't match exactly, OpenCode will hand you an error that doesn't obviously point at the real problem.

2. Create the OpenCode configuration

mkdir -p ~/.config/opencode
vi ~/.config/opencode/opencode.jsonc

Then drop this in, adjusting the server address and model settings for your deployment:

{
  "$schema": "https://opencode.ai/config.json",
  "model": "qb2/qwen-qb2",
  "provider": {
    "qb2": {
      "name": "QB2 node6",
      "npm": "@ai-sdk/openai-compatible",
      "options": {
        "baseURL": "http://YOURIP:20000/v1"
      },
      "models": {
        "qwen-qb2": {
          "id": "Qwen/Qwen3.8-27B",
          "name": "Qwen3.8-27B",
          "tool_call": true,
          "reasoning": true,
          "modalities": {
            "input": ["text"],
            "output": ["text"]
          },
          "interleaved": {
            "field": "reasoning"
          },
          "limit": {
            "context": 262144,
            "output": 16384
          }
        }
      }
    }
  }
}

A few things worth understanding rather than just copying:
qb2 is your provider alias and qwen-qb2 is your model alias. Those are names you pick, and they're what you'll type on the command line. The id field is the one that has to match your server's model ID exactly, because it's what actually gets sent over the wire.

Keep the /v1 on the base URL. The @ai-sdk/openai-compatible package expects the OpenAI route layout underneath it, and leaving it off is the kind of mistake that costs you twenty minutes.

The tool_call and reasoning flags tell OpenCode what this model can do. tool_call is what lets the agent create files and run commands, so it needs to be on for any of this to work. The interleaved block tells OpenCode where to find the model's thinking in the response stream, which is why you see those "Thought: 338ms" lines in the output later.

The token limits should match your server. And this example assumes no authentication on the endpoint, which is fine for a box on your own LAN. Lock it down before exposing it anywhere else.

3. Verify the connection

opencode models qb2

You should see:

qb2/qwen-qb2

Then send a test prompt that isolates the connection from everything else:

opencode run -m qb2/qwen-qb2 \
  "Reply with exactly QB2_OK. Do not use any tools."

I like this test because it fails in a useful way. If QB2_OK comes back, your networking, your model ID, and your response parsing are all correct. Anything else tells you the problem sits upstream of any agent behavior, so you can stop debugging your config and go look at the server.

4. Start coding

From your project directory:

opencode

That's it. You're now running a coding agent against hardware you own.

So How Did I Test

I made a fresh directory, fired up OpenCode, and typed a standard prompt I use to get a feel of any new coding model:

make me a tetris clone

Tetris works well as a test because it's specific. There's a defined set of seven tetrominoes. Rotation has to work, including wall kicks when you rotate against the edge of the board. Collision detection has to be right in four directions. Line clears have to shift everything above them down. Scoring has a known formula. Any of those being subtly wrong gives you something that looks right in a screenshot and falls apart the second you play it.

It's also complex enough that the agent has to actually reason through the state management.

What Came Back

One minute and nine seconds later, a single index.html.

The part I didn't expect was the middle of that run. Before declaring victory, the model wrote itself a headless test harness. It stubbed out the grid, spawned a piece, dropped it to the bottom and checked it landed. It filled a row except for the gap where the piece was going, dropped the piece, and verified exactly one line cleared. It rotated a T piece and confirmed the cell layout actually changed. It shoved an I piece off the left wall and confirmed the collision check caught it.

It also ran four tests on the code it generated. Piece landed at bottom, line clear on full row, rotation changes shape, and wall collision detected all passed.

Nobody asked it to do that. I said "make me a tetris clone." It decided on its own that the way to be confident in a Tetris clone is to simulate one without a browser and check the invariants.

The game itself is genuinely good. All seven tetrominoes with distinct colors. A ghost piece showing where the current block will land. Next-piece preview. Scoring at 100/300/500/800 for one through four lines, scaled by level. Level up every ten lines with the fall speed increasing. Wall kicks on rotation. Pause and hard drop. Arrow keys to move and rotate, down for soft drop, space to slam it, P to pause.

I played it. It plays like Tetris.

There is Still Room To Grow

A 27B model at 4 bit is a capable mid-size model. It handled a self-contained problem with a clear spec beautifully. I'd expect it to struggle more on a large existing codebase where it has to infer the rules from context, and I'll find out when I try that.

Single-user token generation is also where this hardware is weakest, for the reasons I dug into in my last post. You're filling one row of a 32-row tile and paying for the whole bus either way. Concurrency is where the QuietBox actually stretches its legs, and this run used almost none of it. The same server can have thirty agents batching against it at once.

Still, a minute nine for a working Tetris clone is past the threshold where I'd reach for the tool during real work, which is the bar I care about.

There's also the part that's easy to miss from the output alone. Everything about that run stayed on my network. The prompt, the reasoning, the generated code. Nothing metered, nothing rate limited, nothing logged somewhere I can't see. I can run that loop a thousand times tonight and the marginal cost is electricity.

Bottom Line

The config file above is about thirty lines and takes a couple of minutes to write. That's the whole distance between having a local model and having your coding agent run on hardware you own.

What makes that possible is the OpenAI-compatible endpoint quietly becoming a universal adapter. The ecosystem of tools built on top of it will point at your hardware without knowing or caring where it's running. The tooling caught up, the hardware got small enough to sit on a desk, and the models got good enough to be useful at sizes that fit.

Next up I want to push this harder: multiple agents hitting the same server at once, which is the workload this box was built for, and a real project instead of a toy. I'll report back.

Thanks for reading, and stay tuned.

Update I have since learned that the tt-cli tool has been updated and can now run opencode with your model with just tt launch opencode once you have the model.

I wanted to run any model on Tenstorrent hardware

Eric Zietlow — Mon, 21 Sep 2026 12:44:56 +0000

I kept running into the same problem, so I built a tool. I have run into the same general issue over and over for the last few months. I bring a model up on a Tenstorrent card. I get it running under vLLM, tune the mesh, nudge the batch size, and eventually land on the one exact combo of ttnn, a vLLM plugin, and my own hacked-up tt-metal tree that actually spits out coherent tokens. I have a working state but when anyone says "cool, send it over," I spend the next afternoon typing out the answer into a wiki page: which branch of what, which env vars, which mesh, the exact serve flags, the weights repo. Half of it is stale by the time they read it, and the rest is tribal knowledge I didn't even realize I was relying on until I had to write it down.

If you read my previous post about wedging a 744b parameter model onto two cards, you saw a taste of this already. A real chunk of that project wasn't the model at all. It was me keeping track, by hand, of which build I'd compiled against and which set of flags actually worked. The thing that "works" only works on my box, in my head. That's not something you can copy into someone else's setup.

Quick disclosure, same as last time. I work at Tenstorrent. This is in Beta with a small crack team of engineers supporting it. Read everything below as a snapshot of where it stands today, not a promise about next month.

What I wanted was a way to say, once, "here is everything this model needs," and have that description travel with the model instead of living in my head. That's the whole tool. It's called tt-model (the repo is tt-model-manager). It publishes self-contained model bundles into ordinary Hugging Face repos and serves them on a Tenstorrent card through the vLLM plugin.

The weights are not in the bundle

To make this work cleanly, weights stay out of it. They're big, they're shared, and they already live on HuggingFace. So a bundle just carries a pointer, the HF repo ID of the weights, and fetches them at pull time or on first load, into your own cache under your own token. A couple-gigabyte bundle can front a 50 GB model.

What is in the bundle is the runtime. There are two sizes depending on how much you want to ship:

v5.1, "container." The whole serving stack, the built tt-metal tree plus vLLM plus the plugin plus your model code, ships as a standard container image. The user only needs Docker and a card. No external Python, no external venv, nothing to line up manually.
v6, "thin." (under development) Same idea but instead of building a container, the bundle carries a pinned pip spec and builds the per-model environment from it with uv. This ends up being much smaller on disk. Still a draft in development but open to public PRs and issue submissions.

The thing tying these together is the same in both of them: reproducibility stops depending on you happening to have the right build sitting on disk somewhere. It comes from the bundle carrying its own pinned runtime. You state the dependencies once, out in the open, and the model is portable because it brought its own everything.

Kernels still compile on the box

One thing the tool deliberately does not do: precompile kernels and ship a giant frozen blob of them. Kernels JIT-compile into the serving host's own cache the first time you actually run the model. Which means a cold-cache first boot is slow, and the container path will warn you it can be as much as a ten-minute compile up front. Every boot after that is fast because the cache is warm. A bundle ships the recipe for the build (the sources, the pins), and the box does the actual cooking once and then it's cached.

What you actually write

For a container package, the entire authoring interface is one YAML file you commit right next to your model, tt-model.yaml:

schema: "5.1"
repo: you/my-model
name: my-model
weights: org/Weights-7B          # a pointer, never baked in; pin a revision to freeze it
kind: vllm-plugin
arch: blackhole

source:
  tt_metal: /path/to/your/tt-metal   # exactly the tree you validated (or {repo, ref} to clone)
  code:                              # an allowlist of what ships
    - models/common
    - models/autoports/my_model
  ubuntu: "22.04"
  python: "3.12"

runtime:
  vllm:   {version: "0.24.0"}                 # or a wheel / a local source tree
  plugin: {path: /path/to/vllm-tt-plugin}     # your checkout, staged into the image
  extra_models_dir: models/autoports/my_model/vllm_bundle

serve:
  block_size: 64                    # the TT backend rejects vLLM's default
  max_num_seqs: 32
  capabilities:
    tool_parser: hermes
    reasoning_parser: deepseek_r1
  env:
    ARCH_NAME: blackhole

There are no version ranges to negotiate, because there is nothing on the far end to negotiate with. The image gets built once, by the person who brought it up, from exactly the trees they validated by, and every floating reference (a branch name, a plugin checkout, a weights revision) gets pinned down to a concrete commit at build time. What they authored as main goes out as a sha, so nothing quietly moves underneath a model that was already working.

The fat and thin bundles are captured with CLI flags instead of a YAML file, but the principle is identical: you record what worked on your box, once, and lock it there.

Publishing and pulling

There's no special registry to stand up. A bundle is just a Hugging Face model repo with an extra manifest bolted on or a container that can be run. Publishing is a push; pulling is a download.

tt-model login                                # reuses huggingface_hub's token store

# Author + push a self-contained bundle (weights stay a pointer)
tt-model package you/my-model \
  --from-metal ./tt-metal-community \
  --ttnn-wheel dist/ttnn-*.whl \
  --arch-name LlamaForCausalLM \
  --main-class models.tt_transformers.tt.generator_vllm:LlamaForCausalLM \
  --weights unsloth/Llama-3.2-3B-Instruct

# Or build a container package straight from the YAML file
tt-model package --container tt-model.yaml
tt-model push    build/my-model --private

Everything is private by default, because a bundle can point at proprietary weights and nothing should go public by accident. --public shares it by link. --publish is an opt-in tag in huggingface so it can be pulled like a community catalog. It never stores your actual content.

The other end:

tt-model pull  you/my-model        # download and install into its own env (or load the image)
tt-model serve you/my-model        # install if needed, then launch the OpenAI-compatible server
tt-model curl  "write me a haiku"  # composes the chat-completions request for you

tt-model serve is the one-command path. It installs if it has to and then starts up; run it again and it jumps straight to serving. There's a list for what's installed locally, an rm to nuke one, and a search for published bundles (tt-model search gemma, optionally narrowed with --catalog or --arch blackhole).

Still experimental

Everything here is in Beta, and I'm not going to promise correctness or stability. If you build on top of it, expect to re-publish when the format changes, because it will change. I'm writing this up because of a specific idea: make a model state its own dependencies so it becomes portable and reproducible on its own terms. I'm not writing it because the implementation underneath is settled yet. Being open source now is the time for people who know more than I do to contribute and make this into the thing that will carry the TT story into its next chapter.

If you want to poke at it or tell me where it's wrong, it's all here: https://github.com/tenstorrent/tt-model-manager

Squeezing a 744B Model Onto Two Tenstorrent Cards... Kinda

Eric Zietlow — Mon, 10 Aug 2026 15:29:44 +0000

I did a mad science thing recently and I need to tell you about it. This one didn't touch the whole home lab, just one small piece of it: the master node from the swarm cluster I described back in my first post (the box I called the swarm host there), running just two P150A cards linked together with a cable. That was the entire lab for this experiment. Here's what I was chasing, and what actually happened. Quick disclosure, same as always: I work at Tenstorrent, this was done on my own hardware on my own time, and I'll flag what didn't work just as readily as what did.

The Big Question

The question I set out to answer was easy to ask and hard to answer: can you run a genuinely frontier sized model, in this case GLM-5.2, a 744 billion parameter mixture of experts model, on a small two card box? Not a shrunk down version. The real thing.

The trick with a mixture of experts model is that not all of those 744 billion parameters do anything for any given word you generate. The model routes each token to a small handful of "expert" sub networks and ignores the rest. For GLM-5.2, something like 40 billion parameters are actually active per token. The other 700-plus billion just sit there, waiting to be needed for a different token later.

I didn't come up with this idea on my own. Credit goes to a project called Colibri, a CPU and GPU engine that already proved you could run GLM-5.2 in about 25 GB of memory by keeping the common stuff resident and streaming the rest of the experts in from RAM or an SSD as needed, using a lookahead prefetch system they call PILOT along with cache aware routing and speculative decoding. Colibri runs on CPU, CUDA, and Metal. Nobody had ported this approach to Tenstorrent hardware yet, and that gap is exactly what got me curious.

There was a second project in the mix too, a fast Rust based tokenizer called gigatoken. I looked at it and shelved it pretty quickly. Tokenization takes microseconds per request. It was never going to be the bottleneck here. Good tool, wrong problem.

Early testing also gave me a number worth keeping in mind for the rest of this post: grabbing an expert that's already sitting in a card's memory costs something like 13 times less than streaming one in from RAM or disk. So the entire game becomes maximizing how often the token you're generating already has what it needs close by.

The Setup

For hardware, I used the same box I called the swarm host in my first post. Two Blackhole P150A cards, 32 GB each, linked together with the cable on the back. The host itself is a Ryzen 9900X with about 238 GB of RAM. One card talks to the host over a full Gen5 x8 link at 28 GB/s. The other is stuck on a x4 link at 14 GB/s, so there's already an asymmetry to work around before you get to anything clever. The cards can talk to each other fast over their own fabric connection, NVMe storage on the box does about 7 GB/s, and each card's own memory reads at around 312 GB/s once data is actually sitting on it.

Think of it like a kitchen. A card's own memory is the counter right in front of you, fast to grab from. RAM is the pantry down the hall. NVMe is the grocery store across town. The whole game here is minimizing trips to the grocery store.

Getting a Baseline

Before optimizing anything, I needed to know what slow actually looked like. Running Colibri's own GLM-5.2 setup on CPU gave me 0.35 tokens a second, with about 35 seconds before the first word even showed up. That was the number to beat.

Building It Up

From there it was a lot of careful, unglamorous verification work before I worried about speed at all. Every piece of the model, meaning the normalization step, the routing logic, the expert layers themselves, and the attention mechanism, had to be checked against Hugging Face's own reference implementation using a correlation score called PCC (think of it as a similarity score between two sets of numbers, where 1.0 means identical). Everything I built came back above 0.999, and the full decoder layer landed at 0.999984 against real weights. My math wasn't just close. It was basically indistinguishable from the reference.

Then came the moment of truth. I ran the full 78 layer model, with real 4 bit quantized weights (the same low quantization I said I wanted back in my first post), fed it "The capital of France is," and it came back with " Paris." First real end to end answer, correct, running on Tenstorrent hardware. That's the moment this stopped being a math exercise and started being an actual model.

From there it was all about speed:

Built a host to device streaming system using a standard least recently used cache strategy, verified to produce results bit-identical to a fully resident run, so none of the speed work that followed came at the cost of correctness.
Fusing the mixture-of-experts math together with the attention calculation and a couple of smaller operations pushed the theoretical compute ceiling from 3.2 tokens a second up to 7.7.
Getting an honest, live, unified streaming version working, not a benchmark that cheats by pre-staging everything ahead of time, took me from 0.26 up to 0.88 tokens a second.
Pinning the "generalist" experts, the ones that keep getting reused no matter what you're asking about, into a card's own memory instead of letting them get evicted took my on device hit rate from 25 percent to 42 percent, and it stopped hitting the NVMe drive almost entirely.
Running the whole thing correctly across both cards together got me to 0.93 tokens a second.

What Didn't Work

With all these wins I still didn't have everything pan out as planned:

Prefetching experts ahead of time based on the router's own lookahead helps if you're processing a big prompt, what's called prefill, but it does basically nothing for single-user, one token at a time generation, which is most of what an actual conversation looks like.
I tried concatenating multiple experts together to process in one pass. It was slower, not faster, 0.79 tokens a second, because stitching nine slots together costs about as much as just running the 27 separate matrix multiplications would have.
I fixed a fused version of the expert feedforward pass. It ran correctly. The speed didn't move at all.
Expert pinning raised my hit rate nicely, but tokens per second stayed completely flat.

That last one turned out to be the most important negative result of the whole project, because it's what told me the bottleneck wasn't where I thought it was.

The Real Wall is Bandwidth

If hit rate goes up and speed doesn't move, the miss penalty isn't the whole story. It's tempting to call that "compute bound," but that's not quite right, and the numbers make a cleaner case for what's actually happening.

GLM-5.2's active weights per token come out to somewhere around 20 to 24 GB (routed experts at roughly 11 GB plus attention and dense layers at roughly 13 GB, at this quantization). Device memory reads at about 312 GB/s once data is already sitting on the card. That puts just reading already-resident weights at 64 to 77 milliseconds, before a single matmul runs. The measured resident ceiling is 7.7 tokens a second, or about 130 milliseconds a token. So somewhere between half and sixty percent of even the best-case time budget is spent reading weights, not computing on them. That's a memory bandwidth cost, and it doesn't care how good your hit rate is, because a hit still has to be read, just from somewhere faster. This is the normal rule for single-token decode on basically any accelerator, and MoE makes it worse, not better, since the reads are scattered across experts instead of one dense block.

There's a second, separate cost stacked on top of that. These chips run in fixed 32-row tiles, and a tile costs about the same whether 1 row or 32 rows are doing real work. Chatting with a model one token at a time means you're only ever filling 1 of those 32 slots. It's like renting a 32 seat charter bus to drive one person to work every morning. The bus burns about the same amount of gas whether it's full or almost empty. That's a parallelism and utilization problem, not evidence the math engine itself is maxed out.

Pinning experts fixed a third, separate cost: the additive penalty of fetching whatever's missing from RAM or NVMe, without touching either the bandwidth floor or the utilization floor above. That's exactly why hit rate went up, 25 to 42 percent, while tokens a second stayed flat. Two of the three costs stacked up here don't care about hit rate at all.

Here's the part that surprised me most once I added it all up. That 130 millisecond number is the idealized floor: resident weights, full tile, zero streaming. The live streaming generator actually runs at around 1,100 milliseconds a token. Almost all of that extra gap is dispatch overhead, hundreds of small host-to-device copy launches and an eager Python re-staging path for every single token, none of which the fused, offline benchmark had to pay for. So the honest streaming number is really overhead bound, stacked on top of bandwidth, stacked on top of utilization. Three different costs, and only one of them, the fetch penalty, is what pinning can actually touch.

That reshuffles what I'd chase next. More cards to reach full residency still matters for closing the fetch-cost gap long term (the math there hasn't changed: roughly 417 GB of active weights divided by 32 GB a card, so about 14 cards, Galaxy territory). But given that overhead alone accounts for most of the gap between 130 milliseconds and 1,100, the nearer term, and probably bigger, win is applying the same fusion work that got the offline benchmark from 3.2 to 7.7 tokens a second to the live generator's actual per-token path. I haven't done that yet. It's now at the top of my list.

How This Stacks Up

For context, here's how this compares to Colibri's own numbers on other hardware:

Setup	Single-user tok/s
Colibri, Xeon CPU	0.35
Colibri, 6x RTX 5090, partial	0.12
Colibri, 6x RTX 5090, full resident	6.8
This project, 1 Blackhole card, streaming	0.88
This project, 2 Blackhole cards, streaming	0.93

A couple things jump out. My two card streaming setup already beats Colibri's CPU baseline and its partial six GPU configuration. It's nowhere near the 6.8 tokens a second Colibri hits when it goes full resident across six top end GPUs, but that's not surprising once you understand the math above. 6.8 tokens a second is exactly what you get when nothing needs to stream, because everything's already sitting resident. Same lesson, different hardware.

Colibri's creator and I agreed on quite a bit. Hit rate, not prefetch overlap, is the real lever to unlock performance. The "generalist" experts that keep getting reused are worth caching. We both measured that the top 20 percent of experts account for close to 58 percent of all routing traffic. And full residency on card really is where the speed comes from. I reused Colibri's int4 weights, its routing trace instrumentation, and some of its pinning and speculative decode thinking, and I want to be clear that credit belongs there.

What's Left on the Table

There's plenty I haven't done yet, roughly in order of effort:

Wire the overlap runtime I built earlier into the actual live generator. It's already built and verified, it's just never been connected. This is a genuinely easy win I left sitting there.
Cut out a small per layer sync step between the host and the cards. Worth maybe 3 percent, but easy.
Borrow Colibri's cross layer prefetching idea for a higher hit rate.
Speculative decoding, guessing a few tokens ahead and checking them in bulk. Could be a 1.5 to 2x win if it pans out, but it's a real engineering lift and the payoff isn't guaranteed.
More cards, to get closer to full residency.
Batching multiple users together, which sidesteps the whole batch of one problem by definition. Worth something like 8x in aggregate throughput.
A sparse attention indexer needed to correctly support longer context windows.
Tracing the decode process at a lower level and applying the same fusion work from the offline benchmark to the live generator's actual per-token path, to close the overhead gap between the idealized floor and the real generator. This looks like the single biggest lever on this entire list.
Given where the time actually goes, more cards and batching are still real levers for the long run, but closing that overhead gap is probably the cheapest, nearest term win of everything on this list.

Bottom Line

A few weeks ago, running GLM-5.2 on these cards would have gotten you a shrug. Today I've got a full 744 billion parameter model streaming correctly, verified piece by piece against the reference implementation, running end to end on two Blackhole cards at just under one token a second. That's about two and a half times faster than the CPU baseline I started from.

The core idea holds up. You genuinely can run a model this size on hardware this small, as long as you accept single user speeds that reflect the fact that you're constantly re-fetching most of what you need. Getting faster from here is mostly a software problem right now, not a hardware one. The gap between the idealized 130 millisecond floor and the roughly 1,100 milliseconds a token the live generator actually takes is dominated by dispatch overhead, hundreds of small copy launches and an eager Python re-staging path, which is about as pure a software problem as it gets. More cards and more residency still matter for the long run target, but closing that overhead gap first is probably the bigger, cheaper win, and it's one I can chase without buying anything. To be clear, if I can also figure out how to utilize all 32 rows, that would speed things up tremendously on top of that. It very well might be possible, it was just out of scope for the time I had.

This was a fun project and a really cool proof point that raises more interesting questions than it clarifies. I have to put it down for now, but this is something I intend to revisit down the road.

Hands-On With the Tenstorrent QuietBox 2

Eric Zietlow — Mon, 03 Aug 2026 17:10:41 +0000

Quick Disclosure: I want to disclose that I work for the company that built this box. I'm going to do my best to give an unbiased opinion here: the good, the bad, and the ugly.

It Begins

My first hands-on experience with the QuietBox 2 was pretty cool. It arrived later in the day, so I didn't have to wait until I got off work to start playing with it.

The first thing I noticed was the heft of the box itself. This felt a lot more like server equipment than a simple PC tower. Upon opening it, I was greeted by an obviously custom case with two gigantic fans and some pretty cool cosmetic touches. The box itself isn't obtrusive or gaudy. It just looks like it would fit neatly on someone's desk, and it would be just as comfortable pretending to be a gaming rig as it would be the sleeper AI Ferrari that it actually is. The liquid cooling channels on the front were a nice touch. Even before I booted it up, it looked pretty amazing.

Diving a little deeper, the box came with a speaker and some decals, which I set aside for later. I can totally see the intention behind this computer. It's clearly meant to be a household-appliance-style AI server.

More Than Just an Accelerator

The box itself is an incredibly powerful unit, and I don't just mean as an inference server. As a usable desktop workstation, or even a medium to heavy load server, it's quite capable. 256 GB of RAM, a high-end consumer CPU, and enough storage for pretty much any task make it clear that this box isn't the equivalent of just buying a GPU accelerator. It's more than that. It's trying to be a full solution.

As someone who has been building out a home lab for years, this is intriguing, because it means I can treat the box as a truly self-contained appliance. Something like a home assistant could theoretically be built out exclusively on this box, with all the different parts and complex pieces running locally instead of spread out across boxes on a network.

The liquid cooling and the big radiator fans make this an almost illegitimately quiet box for a config with zero hardware trade-offs. If I were building a system like this myself, I'd have to spend quite a lot of money to get anything remotely close to this noise profile at this power level. That said, if you really wanted to, I'm sure you could pull it off.

The Accelerator Cards

Let's talk about the accelerator cards for a second, because these are unique to this system. When you first boot up and look at what's attached, the cards show up as four, even though the box itself only has two physical cards installed. This comes down to the dual-chip design of each P300C card. The end result is something close to four P150A cards' worth of compute in a single box, while only taking up two PCIe slots.

There are a few reasons for this. Anyone who has tried to design a four-card system with full-size, dual-width, high-power-draw cards can tell you it presents a unique challenge. You're pretty much off consumer hardware from the get-go and onto server-grade EPYC or Threadripper CPUs before you even start, and the problem only gets harder from there. That's why these two dual-chip cards make sense in this build. They let the price stay well below what a true four-card system would cost, while still providing the power and bandwidth to handle what's actually running.

The two cards are linked together in what's called a 1x4 fabric configuration. That means each chip functions like an individual card on its own, but they're networked together into a mesh so the resources get pooled when needed. The end result is 128 GB of usable memory (GDDR6) across the accelerator setup, which lets you load models much bigger than you'd manage on a single GPU.

Built Like a Tank

Let me change gears for a moment and talk about the case. This thing feels tough, and I don't mean tough like rigid plastic. It feels like it was built out of armor. It weighs a little over 50 lb, and moving it by the carry handles feels more like carrying an upright server than a desktop tower. You won't find any flimsy aluminum paneling here, just raw, no-nonsense, compact form factor power, contained in a chassis that feels tough enough to shield you from flying debris in a natural disaster.

The case is pretty cool beyond that too. It uses a modified clamshell design where both side panels open to reveal what's essentially a bifurcated physical layout. Each of the P300C cards sits on either side of the motherboard, upright. Both are monsters, coming in at about 7 lb each and taking up three slots, even with the more compact liquid cooling. When the box really gets going, it gives off a ton of heat, but again, nothing a high-end gaming rig wouldn't already do.

Another cool feature is the liquid reservoir built into the front of the case. You can actually see the fluid moving through it while the computer is running. This is definitely one of the more interesting liquid cooling solutions I've seen, and it gives the whole thing a unique vibe.

A Week In

In the week since I started using it, I've been pleasantly surprised. It's quickly become a workhorse in my home lab. I've thrown a lot at it and haven't come close to taxing it fully. I've got a number of projects that I'll hopefully be able to blog about soon, but until then, you'll have to make do with this teaser.
So What Is This Thing, Really?
I'd like to point out what this box isn't. It very clearly isn't what you should buy if you want a scalable system. There's no path to link these together like there would be with Tenstorrent's Galaxy system, and if you only need one or two cards, buying them separately and building your own box will be cheaper.

This box doesn't really feel like it's trying to compete with either of those scenarios though. It's something unique, more like an appliance that can handle everything, and it wouldn't surprise me if this type of system eventually becomes as common in a house as a refrigerator, washer, or dryer.

Just the brain of the house. It's what you get if you don't want to pay a subscription to a major LLM provider, and beyond that, it can run all your home automation and anything else you need. Various privacy movements, like the de-Googleify movement, would make great use of hardware like this.

Now for the Bad (and the Ugly)

The software stack's maturity definitely isn't where GPUs have been for a while now. That means while the box can run very large models, it also has some limitations around bottlenecks, and the kernels just aren't as refined as what you'd get on a lot of GPUs. Throughput isn't always quite where you'd want it to be.

That said, concurrency is this box's strong suit, and it can outpace any other setup I've had the privilege of testing, many times over. Running a single model with 30 open agents, all batching their work against the accelerator cards, is the kind of power you need to run a bunch of different systems on a single box. That, more than anything, is where this box shines.

Set your model up and go to town. This is a powerhouse and a workhorse. It's not trying to be anything else, and it doesn't need to be.

Final Verdict

So where does the QuietBox 2 actually land? If you need something that scales out into a real cluster, this isn't your box, and it was never trying to be. What it is, is one of the more interesting all-in-one AI appliances I've gotten my hands on. It's quiet, it's built like a tank, and it has more than enough headroom to run as a genuine home brain instead of just another box tucked in a rack.

The software stack still shows its sharp edges in places, and that's worth going in with eyes open. But the concurrency numbers make up for a lot, and this ecosystem is moving fast enough that today's rough edges probably won't stick around for long.

Between this and the P150A cluster I've been building out, I've now got two very different answers to the same question: how do you put real AI compute in your house without handing your data, or your wallet, over to someone else every month. One approach is fully DIY. The other shows up ready to go out of the box. I'll keep living with both and report back on how they stack up.

That's a wrap on my first week with the QuietBox 2. Thanks for reading, and stay tuned.

My Tenstorrent Journey, Part 1: Why I'm Building This Thing

Eric Zietlow — Mon, 20 Jul 2026 13:28:09 +0000

I want to take you all on a journey with me through the Tenstorrent ecosystem. Before I get going, I'd love to share a little bit about myself.

I've been homelabbing for the better part of a decade, mostly on whatever hardware I can cobble together from used gaming rigs and the odd piece here or there. When the AI boom hit a few years back, I jumped on the train and was an early user of Ollama, among many other tools. Eventually I upgraded from an RTX 3080 to a RTX 3090 that I'm pretty sure had spent years running in some crypto mining rig. The card still ran well, and it had enough VRAM to run significantly bigger models.

That setup worked fine for a number of years, as long as I stuck to light tasks and workloads that didn't put much strain on it. I could generate images or chat with an LLM, but doing more than one of those things at once was always the sticking point. It became one of the biggest pain points in my whole setup.

I have always wanted to build Jarvis type system like we see in the Iron Man movies. But every time I sat down to design it, I ended up making so many trade-offs because of hardware constraints that the end result always fell short.

That's where Tenstorrent came in.

A Quick Disclosure

Before I get too deep into talking about them, I feel it's important to disclose that I work for Tenstorrent. My interest in their work predates my employment, though. In other words, I was an enthusiast who managed to land a dream job at one of my favorite companies.

Now that's out of the way, let's get down to it.
Not Your Grandma's GPU
Tenstorrent hardware is a very different architecture, built with a different mindset and a different set of strengths and weaknesses. A GPU is very fast and powerful at single task inference, but it struggles with parallel workloads beyond a certain point. You hit limits fast.

Tenstorrent is pretty much the opposite. Individual single card inference is typically slower, but the benefits as you scale are far greater, especially for batched workloads. What that means for me is that it's more practical to run agent swarms across a multi-workload system on a Tenstorrent Blackhole card cluster than it would be on, say, a fleet of RTX 5090s. And that's before you even factor in that 5090 pricing puts a fleet of them almost completely out of reach for a home lab.

To be fair, the build I'm about to walk through isn't cheap either. But the methodology behind it can be scaled down. I'll try to explain not just what parts I chose, but why I chose them, so people on any budget can replicate what I'm doing.
What "Scale" Actually Means Here
I used the word "scale" a couple times up top, so let's define it.

In the GPU world, if you want to scale up, you typically buy a bigger card. There are some solutions for linking cards together, and in more extreme DIY builds you'll see people soldering on new VRAM modules to double or quadruple a card's memory. All of these options are either cost prohibitive or pretty limiting. A single A100 has between 40 and 80 GB of RAM and can cost close to as much as a used car.

For contrast, a single Blackhole card is about $1,300 and comes with 32 GB of RAM that can be linked together with other cards for a much larger memory pool. That scalability is extremely attractive to me, since I can buy three or four of these cards for the price of one new RTX 5090.

One of the more interesting things about Tenstorrent's P150A cards is that they connect to each other through a link cable on the back of the card. Each card has four ports, so you can wire them together in different topologies depending on what you're trying to do. In the simplest case, running one port to one port pools two cards' resources into something the computer sees as basically a single resource pool. Technically it's still two cards, but it can function much like one.

There's also a setup where you can spread multiple small models across a larger number of cards without linking them together at all. I might try that down the road, but for this early test it's out of scope. My actual goal right now is to get high performance out of a model like Qwen 3.6, with all the extra features like vision, up and running. I'll have a lot of memory to work with, and I want the largest context window possible at the lowest quantization I can manage while still getting acceptable tokens per second. If this works, I'm going to swap it in as the main brain behind my Jarvis system.

A Bit About the Home Automation System

The system runs on a small Intel PC appliance and connects to remote Bluetooth speakers around the house. The appliance basically acts as a process server, orchestrating tasks and hitting AI via API endpoints. So if I ask it to generate a picture, it might route that request to a different server than if I'm just having a conversation.

That also means I can potentially scale multiple models across a number of different cards and let the best model for the job take over, either through some kind of intelligent mapping, or maybe even a very light LLM handling the routing logic. What we are building today are the model endpoints that this little appliance will consume.

The Hardware, In Detail

I've got the small Intel NUC-style appliance running Ubuntu 24 (all my boxes run the same Ubuntu version). There's the old gaming rig with the RTX 3090 that I'll be transitioning away from, and three separate servers each with their own Blackhole cards, each serving a different purpose:

Box 1: Another old gaming computer with a single P150A, handling smaller workloads. This card isn’t networked into the others and is a stand alone resource. Basically anything that isn't a heavy lift goes here, so I'm not tying up resources on the workhorse machines. If you wanted to do this yourself you could skip this box and save some cash.
Box 2 (the swarm host): A Ryzen AMD box with dual P150As. This one is specced way up: 256 GB of RAM, 4 TB of storage, a Ryzen 9 9000 series processor, and a 1200W PSU to handle the power draw of the cards.
Box 3: Also a Ryzen box with dual P150As, but built very differently. It only has 16 GB of RAM, a Ryzen 5 9000 series processor, and half a terabyte of storage.

The idea is that Box 2 handles all the heavy data transfer into the card cluster when running large models, while Box 3 acts mostly as a lightweight host for its P150A cards, just enough system spec to handle other workloads. I’ve also built Box 3 in a way that should let us try adding a third P150A down the road, just to see if we can get that working. That's going to be quite the experiment, so stay tuned for it.

Putting It All Together

After the parts arrived courtesy of my friendly neighborhood delivery driver, I assembled both systems. One decision that was probably less than ideal: I went with Asus motherboards. It's not that they don't work, it's just that they need some extra setup to get Linux running properly on them. ASRock or Gigabyte have some decent offerings that probably would have saved me a headache or two.

After wrestling through driver issues, I networked the boxes, locked them down, turned on firewalls, and we were ready to go.
The Tenstorrent Software Ecosystem
The first thing you'll notice diving into the Tenstorrent stack is just how open it is. The whole mindset seems to be built around open source, open access, and sharing knowledge. Anyone who knows me knows I'm an open source guy to the max, so that's a huge win in my book.

The second thing you'll notice is that the product is younger in its lifecycle than most GPU offerings. Plug in a GPU, install something like Ollama to handle inference, and you can typically be online within hours, if not minutes. For a lot of people, the most time-consuming part is physically mounting the card and cable managing the power connectors.

But these cards aren’t GPUs. My experience with the Blackhole cards was a different animal. Model support is currently limited, so you end up bringing things up by hand, which takes time. The upside is that the stack is completely open and the community is active, so asking how to do something usually gets you a handful of answers, and a working solution is never far off.

GPUs, by design, have models built to work with their stack from the hardware up through the software. That means tools like Transformers, Diffusers, and llama.cpp are just native to that ecosystem, and there are more guides out there than you could read in a lifetime for setting them up. For Tenstorrent, it's a bit more wild west. For my first run I will need some approximation of the Tenstorrent stack.

First things first, you need to detect the cards, which is where TT-SMI comes in. Second, you need to actually run things on the cards, which requires TT-Metal. Think of TT-Metal as the underlying layer that runs things on the card itself. It includes TT-NN, the actual kernels, along with TT-Transformers and a lot of the other basic tooling you need to get going. On top of that, I'm installing TT-Lang so I can modify things and write my own fused kernels.

If that sounds big and scary, don't worry, we'll break it down as we go. Did I mention this is going to be a fairly long series? The payoff at the end is worth it.

Up Next

That's going to do it for today. In my next article, I'll start breaking down the fundamentals of how all this actually works and how to get started. There are a lot of different paths through this, and my journey is by no means the only one.