I wanted to run any model on Tenstorrent hardware

I kept running into the same problem, so I built a tool. I have run into the same general issue over and over for the last few months. I bring a model up on a Tenstorrent card. I get it running under vLLM, tune the mesh, nudge the batch size, and eventually land on the one exact combo of ttnn, a vLLM plugin, and my own hacked-up tt-metal tree that actually spits out coherent tokens. I have a working state but when anyone says "cool, send it over," I spend the next afternoon typing out the answer into a wiki page: which branch of what, which env vars, which mesh, the exact serve flags, the weights repo. Half of it is stale by the time they read it, and the rest is tribal knowledge I didn't even realize I was relying on until I had to write it down.

If you read my previous post about wedging a 744b parameter model onto two cards, you saw a taste of this already. A real chunk of that project wasn't the model at all. It was me keeping track, by hand, of which build I'd compiled against and which set of flags actually worked. The thing that "works" only works on my box, in my head. That's not something you can copy into someone else's setup.

Quick disclosure, same as last time. I work at Tenstorrent. This is in Beta with a small crack team of engineers supporting it. Read everything below as a snapshot of where it stands today, not a promise about next month.

What I wanted was a way to say, once, "here is everything this model needs," and have that description travel with the model instead of living in my head. That's the whole tool. It's called tt-model (the repo is tt-model-manager). It publishes self-contained model bundles into ordinary Hugging Face repos and serves them on a Tenstorrent card through the vLLM plugin.

The weights are not in the bundle

To make this work cleanly, weights stay out of it. They're big, they're shared, and they already live on HuggingFace. So a bundle just carries a pointer, the HF repo ID of the weights, and fetches them at pull time or on first load, into your own cache under your own token. A couple-gigabyte bundle can front a 50 GB model.

What is in the bundle is the runtime. There are two sizes depending on how much you want to ship:

• v5.1, "container." The whole serving stack, the built tt-metal tree plus vLLM plus the plugin plus your model code, ships as a standard container image. The user only needs Docker and a card. No external Python, no external venv, nothing to line up manually.
• v6, "thin." (under development) Same idea but instead of building a container, the bundle carries a pinned pip spec and builds the per-model environment from it with uv. This ends up being much smaller on disk. Still a draft in development but open to public PRs and issue submissions.

The thing tying these together is the same in both of them: reproducibility stops depending on you happening to have the right build sitting on disk somewhere. It comes from the bundle carrying its own pinned runtime. You state the dependencies once, out in the open, and the model is portable because it brought its own everything.

Kernels still compile on the box

One thing the tool deliberately does not do: precompile kernels and ship a giant frozen blob of them. Kernels JIT-compile into the serving host's own cache the first time you actually run the model. Which means a cold-cache first boot is slow, and the container path will warn you it can be as much as a ten-minute compile up front. Every boot after that is fast because the cache is warm. A bundle ships the recipe for the build (the sources, the pins), and the box does the actual cooking once and then it's cached.

What you actually write

For a container package, the entire authoring interface is one YAML file you commit right next to your model, tt-model.yaml:

schema: "5.1"
repo: you/my-model
name: my-model
weights: org/Weights-7B          # a pointer, never baked in; pin a revision to freeze it
kind: vllm-plugin
arch: blackhole

source:
  tt_metal: /path/to/your/tt-metal   # exactly the tree you validated (or {repo, ref} to clone)
  code:                              # an allowlist of what ships
    - models/common
    - models/autoports/my_model
  ubuntu: "22.04"
  python: "3.12"

runtime:
  vllm:   {version: "0.24.0"}                 # or a wheel / a local source tree
  plugin: {path: /path/to/vllm-tt-plugin}     # your checkout, staged into the image
  extra_models_dir: models/autoports/my_model/vllm_bundle

serve:
  block_size: 64                    # the TT backend rejects vLLM's default
  max_num_seqs: 32
  capabilities:
    tool_parser: hermes
    reasoning_parser: deepseek_r1
  env:
    ARCH_NAME: blackhole

There are no version ranges to negotiate, because there is nothing on the far end to negotiate with. The image gets built once, by the person who brought it up, from exactly the trees they validated by, and every floating reference (a branch name, a plugin checkout, a weights revision) gets pinned down to a concrete commit at build time. What they authored as main goes out as a sha, so nothing quietly moves underneath a model that was already working.

The fat and thin bundles are captured with CLI flags instead of a YAML file, but the principle is identical: you record what worked on your box, once, and lock it there.

Publishing and pulling

There's no special registry to stand up. A bundle is just a Hugging Face model repo with an extra manifest bolted on or a container that can be run. Publishing is a push; pulling is a download.

tt-model login                                # reuses huggingface_hub's token store

# Author + push a self-contained bundle (weights stay a pointer)
tt-model package you/my-model \
  --from-metal ./tt-metal-community \
  --ttnn-wheel dist/ttnn-*.whl \
  --arch-name LlamaForCausalLM \
  --main-class models.tt_transformers.tt.generator_vllm:LlamaForCausalLM \
  --weights unsloth/Llama-3.2-3B-Instruct

# Or build a container package straight from the YAML file
tt-model package --container tt-model.yaml
tt-model push    build/my-model --private

Everything is private by default, because a bundle can point at proprietary weights and nothing should go public by accident. --public shares it by link. --publish is an opt-in tag in huggingface so it can be pulled like a community catalog. It never stores your actual content.

The other end:

tt-model pull  you/my-model        # download and install into its own env (or load the image)
tt-model serve you/my-model        # install if needed, then launch the OpenAI-compatible server
tt-model curl  "write me a haiku"  # composes the chat-completions request for you

tt-model serve is the one-command path. It installs if it has to and then starts up; run it again and it jumps straight to serving. There's a list for what's installed locally, an rm to nuke one, and a search for published bundles (tt-model search gemma, optionally narrowed with --catalog or --arch blackhole).

Still experimental

Everything here is in Beta, and I'm not going to promise correctness or stability. If you build on top of it, expect to re-publish when the format changes, because it will change. I'm writing this up because of a specific idea: make a model state its own dependencies so it becomes portable and reproducible on its own terms. I'm not writing it because the implementation underneath is settled yet. Being open source now is the time for people who know more than I do to contribute and make this into the thing that will carry the TT story into its next chapter.

If you want to poke at it or tell me where it's wrong, it's all here: https://github.com/tenstorrent/tt-model-manager