NVIDIA's Nemotron 3.5 Bolt 74B-A9B 24GB is a 74B agent model that reads text and images. Only about 9B of its parameters (the numbers a model learns in training) work on each token, and a token is roughly a piece of a word. Most are stored at about 2 bits each, so the file is 22.0 GB.
We served it on one A40, a 48 GB data-center card on our cloud, with llama.cpp, the open-source program most people use to run models on one machine. The image reader was on, and so was the drafter, a small helper model that speeds up writing. It used about 27 GB. For one user it wrote 136 tokens a second with the drafter and 70 without.
Three choices matter more than the model: what the 24 GB covers, which llama.cpp you compile, and when the drafter pays off. Every figure sits in our Nemotron 3.5 Bolt deployment write-up with the full measurements.
The 24 GB is a text-only budget for one user
NVIDIA's sum comes to 23.83 GB. That is 22.01 GB for the parameters, 0.87 GB of working memory for roughly 128,000 tokens of conversation, and 0.96 GB for the drafter. The image reader, another 1.68 GB, is deliberately outside the sum.
Two things eat into that margin. First, the unit: a card sold as 24 GB has 24 GiB. Read NVIDIA's figure as decimal and roughly 1.9 GB stays free; read it as GiB and only about 0.2 does. Second, the drafter took close to 2.4 GB on our A40, not 0.96. So a fit on a real 24 GB card stays unproven: we never put Bolt on one.
The recipes NVIDIA publishes for llama.cpp include the image reader, and the smallest of them asks for a 32 GB card. Our 27 GB (25.2 GiB), with the quick-start 98,304 tokens of conversation, counts the image reader too, so it does not check NVIDIA's text-only number. Measure your own server with the drafter on and off, in the unit your card is sold in.
Build the right llama.cpp, and build it in parallel
Bolt needs llama.cpp code from release day, 15 October. Kartik Gulia, an NVIDIA engineer, added the model and its image reader to llama.cpp, and ggerganov merged the change in llama.cpp pull request #30520 that same day. Older builds refuse the file.
We hit exactly that. Our worker (the agent that sets up each GPU machine for InferenceKey) pins a llama.cpp release 1,920 commits older than the merge, so we set v0.7.0 by hand.
Then compiling ate the clock. A fresh machine took 764.8 seconds from request to ready, and 705 of those went into building llama.cpp on 9 CPU cores. Our worker calls cmake --build with no parallel jobs unless CMAKE_BUILD_PARALLEL_LEVEL is set. Even with 12 jobs set by hand, the build was 92% of the wait. On a 31-core machine the same build finished in 2 minutes 41 seconds. A second copy on an already built machine was ready in 15.6 seconds. If first starts matter, count CPU cores, not only the GPU.
The drafter is a server flag, not a request option
Bolt comes with the LiLiCorr drafter, about 1.1B parameters in a 1.0 GB file. The drafter proposes several tokens ahead, Bolt verifies the whole batch in a single step, accepts the run up to its first mismatch and writes that token itself. You turn it on when the server starts:
--spec-draft-hf nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr-GGUF \
--spec-type draft-dflash --spec-draft-n-max 6 --spec-draft-ngl 999
On short and medium prompts it about doubled the speed: over 25 requests the median was 136 tokens a second against 70, with a floor of 110, and 70% of the drafted tokens were accepted. The per-prompt speed and acceptance table shows where it gains most.
Long documents flip the result: the drafter slows prompt reading by about a quarter. A question over a 25,000-token document took 17.5 seconds without the drafter and 21.5 seconds with it. And v0.7.0 has the per-request drafter control compiled out, so the choice is made once per server. Mixed traffic? Benchmark both shapes first.
Two traps outside the model
-
A card can report fine and still be useless. Our first two machines had RTX 5090 cards.
nvidia-smi, NVIDIA's status tool, answered, but CUDA, the layer programs use to reach the GPU, failed to start. Our platform still marked the model ready, so we have no 32 GB numbers yet. - Hidden reasoning burns time. Bolt thinks before it replies, out of your sight. A one-sentence answer cost 374 tokens of writing time. NVIDIA's model page documents a switch to turn thinking off, and InferenceKey does not forward it yet.
What this does and does not show
The quality numbers are NVIDIA's: Bolt beats Qwen3.6-35B-A3B on most agent benchmarks (Terminal-Bench 2.1: 50.98 against 44.38) and loses on factual recall. NVIDIA's speed table puts it behind Qwen3.6 and Gemma 4. Our side was about 50 requests, one GPU, one request at a time; on a two-tool refund task it called both tools correctly in all 5 runs. We did not test several requests sharing the card, where drafters tend to help less, or vLLM, the data-center server software that is the other route.
NVIDIA built, compressed and released Bolt and its drafter under OpenMDW-1.1, which allows commercial use. Kartik Gulia and ggerganov made llama.cpp able to run it. AI Search's news video pointed us to the family. Our part was wiring it end to end and measuring it. The exact command, per-request timings and worker settings are in the complete Nemotron 3.5 Bolt article on inferencekey.com.
Top comments (2)
705s of a 764s cold start going into the llama.cpp build is the part i'd never have guessed. the drafter doubling 70 to 136 tps on short prompts is nice, but the 25k-token doc going from 17.5s to 21.5s with it on makes "is the drafter on" a per-workload question, not a global one. from running a similar 74b-class moe at home, the "nvidia-smi answers but cuda fails" card is not a one-off either — i've seen it twice on used 5090s, and the only reliable check was a tiny cuda context from a second process, not the vendor tool. the 24gb vs 24gib unit note is the kind of thing that quietly breaks a "it fits" claim.
Thanks for the tiny-CUDA-context-from-a-second-process check, that is the piece we were missing. On our two 5090s nvidia-smi looked happy while llama.cpp's CUDA init died with "unknown error" and torch.cuda.is_available() said False, and our platform still marked the model ready, because nothing checks that the GPU actually works. Your check is basically the soundcheck we skipped.
Agreed on the drafter being per workload. On that llama.cpp build it is per server anyway, so we ran a second copy without it, which came up in 15.6 seconds once llama.cpp was already built.