The complete Kimi K3 checkpoint - 2.8 trillion parameters, every expert present - is now running across sixteen GB10 mini-workstations in one person's rack, producing roughly 21 to 25 tokens per second for a single conversation. Operator Ciprian Veg published the configuration and benchmarks on NVIDIA's developer forum: sixteen ASUS Ascent GX10 boxes, a MikroTik 400G switch, four breakout cables, about 2.3 kilowatts, and a hardware floor that prices out at roughly $57,200 before tax and cabling.
Key facts
- Sixteen GB10 nodes at 128GB coherent unified memory each, for 2TB pooled, connected through a MikroTik CRS804-4DDQ with 400G-to-4x100G breakouts.
- Measured decode: 21.71 tokens/second average at 4k of existing context, 25.39 at 16k, peaks of 37-38. Prompt ingestion ran 654-758 tokens/second. Current serving limit: 250,000 tokens of context.
- Power draw about 2.3kW under load - roughly 55 kilowatt-hours a day, or $8-$11 daily at typical US rates, before cooling.
- Primary source: the operator's benchmark thread on the NVIDIA developer forums.
Six weeks ago this site covered the other end of the same question: K3 running in 8 gigabytes of RAM at 33 seconds per token. That was proof of access. This is proof of use.
What makes it possible
Three things, and none of them is brute force.
First, K3 is a mixture-of-experts model. It has 2.8 trillion parameters but activates only about 104 billion per token - 16 of 896 routed experts. The weights all have to be somewhere, but only a sliver runs on each step. Two terabytes of pooled memory is enough to hold the somewhere.
Second, Moonshot's own release is already quantisation-aware-trained at 4-bit weights with 8-bit activations. "Full model" here means complete, not uncompressed. If you were expecting a 16-bit checkpoint, this is not that, and it was never going to be - a 16-bit 2.8T model is roughly 5.6 terabytes of weights alone.
Third, and most underappreciated: the operator runs speculative decoding using Inferact's Kimi-K3-DSpark draft model, which proposes seven tokens at a time for K3 to verify and accept or discard. Its draft backbone shares K3's attention-cache layout, avoiding a second cache format. Because no no-DSpark baseline has been published, some unknown share of those 21-25 tokens per second belongs to the drafting trick rather than the hardware or the fabric.
K3's long-context design helps too. Most of its layers use a recurrent attention scheme rather than a key-value cache that grows token by token, with periodic full-attention layers preserving global recall. That makes a million-token limit architecturally plausible. It does not make a million-token session free.
Reading the numbers honestly
Decode speed is what you watch while an answer types itself. Prefill is the model reading your prompt - 750 tokens per second there is not answer-writing speed. A peak is a best instantaneous step, not a sustained rate. And a single-request benchmark says nothing about what happens when two people ask at once.
The community response in the thread got this right, immediately asking about 100G fabric utilisation, power, alternative parallelism splits and maximum context. The operator's answer on parallelism was that it is on his test list. That is the correct posture for what this is: a striking demonstration, not a finished reference architecture.
Why it matters
The open-weight release cadence of the past two months - DeepSeek V4 Flash, GLM 5.2, MiniMax M3, then K3 - has been forcing infrastructure work into public view. OpenRouter's first-party traffic data shows Chinese-authored models passing US models in its token share by early June, with DeepSeek climbing from 9% to 18% of tokens between January and June. That is real adoption evidence for OpenRouter's traffic specifically, not a verdict on the whole field.
What this cluster adds is tangibility. The abstract claim "the weights are public" becomes a concrete one: a determined owner can bring near-frontier intelligence home for the price of a car. Not cheaply - but the cost is now visible and itemisable, which it never is for an API.
There is also a licence wrinkle worth knowing. K3's weights are downloadable and modifiable, but Moonshot's licence requires a separate agreement to run a model-as-a-service business above a stated revenue threshold. Local and internal use is explicitly carved out.
The honest caveat
Sixteen boxes at conversational speed for one person is a genuine engineering result and a poor economic one. An API gives far better throughput with no capital outlay. What it does not give is control - over the weights, the data, or whether the model you used last month still exists. Between the $60,000 rack and the 8GB proof-of-access build there is no single right answer, only a choice about which constraint you would rather live with.
Originally published on Ground Truth, where every claim is checked against the primary source.
Top comments (0)