TokenSpeed: A New LLM Inference Engine Built for Coding Agents.

If you self-host large language models, you have probably settled into one of two camps. You use vLLM or SGLang because they are easy to run, or you use TensorRT-LLM because it squeezes the most out of NVIDIA hardware and you are willing to pay for that in setup pain.

TokenSpeed wants to end that trade-off. It is an open-source (MIT) inference engine from the LightSeek Foundation that targets TensorRT-LLM-level performance with vLLM-level usability, and it is tuned specifically for agentic workloads like coding assistants.

In this post we will cover what it is, why "agentic" changes the problem, how to get a server running in a few minutes, and whether it deserves a spot in your stack.

What problem is it solving?

Most inference benchmarks look like chat: a short prompt, a short answer, done. Coding agents look nothing like that.

A typical agent session has:

• Very long contexts. Prompts regularly pass 50K tokens once you include files, tool output, and history.
• Many turns. The same conversation comes back dozens of times, each time slightly longer.
• A hard speed floor. If tokens stream slowly, the agent feels broken. The TokenSpeed team targets around 70 tokens per second per user as a minimum, sometimes 200+.

So the goal is not just "maximum throughput." It is maximum throughput per GPU while every user still gets fast generation. That framing drives most of TokenSpeed's design choices.

How it works (the short version)

You do not need to understand the internals to use it, but four pieces are worth knowing about:

1. Modeling layer. Instead of hand-writing tensor-parallel communication code for every model, developers annotate where data should live at module boundaries, and a small static compiler generates the collective operations. Less custom parallelism code means fewer bugs when adding new models.

2. Scheduler. The control plane is written in C++ as a finite-state machine. Request lifecycle and KV cache ownership are modeled explicitly, and the type system catches unsafe KV cache reuse at compile time instead of at 3 a.m. in production. The execution side stays in Python so it is still easy to hack on.

3. Kernels. Kernels live in their own pluggable package with a registry, so NVIDIA and AMD backends can sit side by side. Their MLA (Multi-head Latent Attention) kernel for Blackwell is fast enough that vLLM has adopted it.

4. Entrypoint. Requests come in through an OpenAI-compatible server built on their SMG project, keeping CPU-side overhead low.

Getting started

What you need

• An NVIDIA GPU host (CUDA 13 for the nightly wheels), or an AMD host with ROCm 7.2
• Python 3.11 to 3.13
• Docker with GPU support if you want the easy path

Option 1: Docker (recommended)

The project ships a runner image with everything preinstalled:

docker pull lightseekorg/tokenspeed-runner:latest

docker run -itd \
  --shm-size 32g \
  --gpus all \
  --ipc=host \
  --network=host \
  -v ~/.cache:/home/runner/.cache \
  --name tokenspeed \
  lightseekorg/tokenspeed-runner:latest \
  /bin/bash

Mounting your cache directory means model weights survive container restarts.

Option 2: pip (nightly wheels)

On a CUDA 13 machine:

python -m pip install --upgrade tokenspeed \
  --extra-index-url https://lightseek.org/whl/nightly

Nightlies are versioned by date, so you can pin one for reproducibility:

python -m pip install "tokenspeed==0.1.0.post20260930" \
  --extra-index-url https://lightseek.org/whl/nightly

AMD users should follow the ROCm instructions in the getting started guide, since you need to install ROCm PyTorch first and use a separate wheel index.

Check that everything is in place:

tokenspeed env
tokenspeed serve --help

Launch your first model

GPT-OSS 20B fits on a single GPU and is a good smoke test:

tokenspeed serve openai/gpt-oss-20b \
  --served-model-name gpt-oss-20b \
  --host 0.0.0.0 \
  --port 8000 \
  --tensor-parallel-size 1

Call it like OpenAI

Because the server speaks the OpenAI API, your existing client code works with only a base_url change:

from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")

response = client.chat.completions.create(
    model="gpt-oss-20b",
    messages=[{"role": "user", "content": "Explain KV caching in two sentences."}],
    max_tokens=200,
)
print(response.choices[0].message.content)

Or with curl:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-oss-20b",
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 100
  }'

That is the whole loop. Anything that already talks to OpenAI, including most agent frameworks, can point at this server.

Going bigger

For large MoE models the flags will look familiar if you have used vLLM. Here is the shape of a Kimi K2.5 launch on four Blackwell GPUs:

tokenspeed serve nvidia/Kimi-K2.5-NVFP4 \
  --served-model-name kimi-k2.5 \
  --trust-remote-code \
  --max-model-len 262144 \
  --kv-cache-dtype fp8 \
  --quantization nvfp4 \
  --tensor-parallel-size 4 \
  --enable-expert-parallel \
  --reasoning-parser kimi_k25 \
  --tool-call-parser kimik2 \
  --host 0.0.0.0 --port 8000

The model recipes page has tested launch commands for DeepSeek V4, Qwen3 and Qwen3.8, GLM 5.x, MiniMax M3, Kimi K3, Nemotron-3 Super, and more. Start from a recipe rather than tuning flags from scratch.

What about performance?

The team's headline numbers, all self-reported:

• On Kimi K2.5 running on NVIDIA B200 with a real coding-agent trace (SWE-smith, evaluated with EvalScope), TokenSpeed beat TensorRT-LLM across the whole latency/throughput curve. That works out to about 9% faster at batch size 1 and about 11% more throughput at around 100 tokens per second per user.
• Their MLA decode kernel cut latency roughly in half versus TensorRT-LLM's on typical speculative-decoding workloads.
• They reported up to 580 tokens per second on Qwen3.5-397B-A17B for agentic workloads.

These are vendor benchmarks, so treat them as a reason to test, not as a guarantee. The good news is the benchmark harness is in the repo under test/agentic_benchmark, so you can reproduce it on your own traffic.

Is it worth a try?

Yes, if:

• You serve large MoE models (DeepSeek, Kimi, Qwen, GLM, MiniMax) for coding agents or other long-context, multi-turn workloads.
• You run on recent NVIDIA hardware (Hopper, and especially Blackwell) or AMD MI350-class GPUs, and you care about cost per token.
• You have been eyeing TensorRT-LLM for speed but do not want its build and setup overhead.
• You want day-zero support for new frontier open models. TokenSpeed shipped launch-day support for Kimi K3 and Thinking Machines' Inkling.

Links

• GitHub: lightseekorg/tokenspeed

Comments

[Max Quimby]

The "throughput per GPU while every user still gets fast generation" framing is the part most benchmarks quietly skip. We've watched a cluster post great aggregate tokens/sec and still feel broken in a session, because one long-context agent turn starved everyone else's stream — the floor is where the UX lives, not the ceiling.

The design choice I find most interesting is modeling KV cache ownership in the scheduler's type system so unsafe reuse is a compile error. Most of the nastiest agent-serving bugs we've hit are exactly that: a cache block reused across requests under load, surfacing as a one-in-ten-thousand wrong token that's impossible to repro. Moving it left to compile time is the right instinct.

One question — how does it behave with heavily heterogeneous turn lengths in the same batch, where one request is a 60K-token re-read and three are short tool-result continuations? That mix is where schedulers usually start making someone wait.