NVIDIA released Nemotron 3.5 Lightning on 11 August 2026, a 30-billion-parameter mixture-of-experts model with just 3 billion parameters active per token, built specifically for the execution layer of always-on agents rather than for reasoning or planning. It ships with weights, training data and recipes under a permissive licence, and alongside a routing library that decides which model in a system should handle each task. NVIDIA claims up to four times the output speed of similarly sized models.
Key facts
- 30B total parameters, 3B active per token, with NVFP4 and BF16 checkpoints, released 11 August 2026.
- NVIDIA claims up to 4x output speed compared with similar-sized models on agentic tasks.
- Weights, training data and recipes released under OpenMDW-1.1, described in the post as "as permissively as possible."
- Primary source: NVIDIA's technical blog post by Chris Alexiuk and Chintan Patel, with the model card on Hugging Face.
The observation behind the model
Chris Alexiuk and Chintan Patel open with a claim that will be obvious to anyone who has watched an agent work and invisible to everyone else: "Long-running AI agents spend most of their time on high-volume execution: tool calls, result validation, and subagent delegation." The interesting cognitive work -- deciding what to do -- is a small fraction of the wall-clock. The rest is plumbing.
Paying frontier-model rates and frontier-model latency for every git pull and every check of whether a file write succeeded is, on this view, a design error rather than an unavoidable cost. So NVIDIA built a model for the plumbing. The blog puts it in one line: "While frontier models may win the headlines, models like Nemotron 3.5 Lightning earn their medals in the trenches."
How it gets fast
Three mechanisms stack. The first is the mixture-of-experts design, which NVIDIA explains cleanly: "a router sends each token to just a few of its many experts, so only a fraction of the model's parameters run per token. This provides the capacity of a larger dense model at the compute cost of a small one." Thirty billion parameters of knowledge, three billion parameters of arithmetic per token.
The second is speculative decoding. Multi-token prediction was included during training rather than bolted on afterwards, so the model can draft several tokens ahead and verify them in one pass. It ships with NVIDIA's DFlash and DSpark inference paths for different serving scenarios. Related concept: multi-token prediction.
The third is the least glamorous and possibly the most consequential: harness-optimised training. The model was trained against the popular agent frameworks it will actually run inside, so its tool calls land in the right format on the first attempt. Every malformed call is a wasted round trip, and at the volumes an always-on agent generates, format accuracy is a latency feature.
Think of a busy restaurant kitchen. The head chef designs the menu and solves the problems, but most of the labour is prep -- chopping, portioning, plating. Nobody staffs a kitchen entirely with head chefs. Lightning is the prep line, and NVIDIA is arguing your agent needs one.
Routing becomes a shipped component
The release includes NeMo Switchyard, a library that routes each task to the appropriate model. This matters more than the model itself. NVIDIA states the architecture plainly: "Frontier reasoning models such as Nemotron 3 Ultra handle orchestration and complex planning while smaller, more efficient models handle the high-volume execution layer."
Model routing has been an optimisation people talk about for two years. Shipping it as a supported library from the company that sells the hardware turns it into a default assumption. See model routing and cascades for how the technique works, and our earlier report that Nemotron Puzzle compresses a 120B model to 75B for the family's other efficiency line.
Why it matters
The openness is unusual in degree. NVIDIA released not just weights but training data and recipes, including Nemotron-RL Agentic Terminal Pivot, an open agentic reinforcement learning dataset used to train some of the coding capability. Small models fine-tune faster, cheaper and on far more modest hardware, and NVIDIA is explicitly courting that: the post claims deployment anywhere from a desktop DGX Spark to a data centre. For teams running agents continuously, the economics of a 3B-active execution model against a frontier model on every step are not close. Background on why the size matters: why LLM inference is memory-bound and quantization.
The honest caveat
Two things to hold. NVIDIA benchmarks NVIDIA, and the "up to 4x output speed" figure comes with no independent replication and the usual latitude that "up to" provides. And "harness-optimised" is a double-edged property: a model tuned to the agent frameworks that are popular in August 2026 is a model whose advantage decays as those frameworks change. The general capability is real; the specific edge is dated the day it ships.
Originally published on Ground Truth, where every claim is checked against the primary source.
Top comments (0)