On August 11, Nvidia put out a press release with two announcements bolted together: a new open-weight language model called Nemotron 3.5 Lightning, and an open-source routing library called NeMo Switchyard. Most of the coverage that followed did what press releases are built to make you do — it led with the model, quoted the benchmark table, and moved on.
That's backwards. Nemotron 3.5 Lightning is a solid, unremarkable 30B model that will be obsolete in the way every LLM is obsolete within two quarters. NeMo Switchyard is a bet about how agentic applications will actually be built going forward, and it comes with a number attached that should get more attention than it has: one team's early benchmark showed it cutting agent costs by 74% by keeping 93% of calls away from a frontier model entirely. If that number holds up outside a vendor's own demo, it's a bigger deal for how you'll architect an agent stack than any single model release this month.
What actually shipped
Two separate things landed on the same day, from the same team, clearly designed to be read together.
Nemotron 3.5 Lightning is a 30-billion-parameter mixture-of-experts model with 3 billion parameters active per token. The architecture interleaves Mamba-2 state-space layers with MoE blocks and a smaller number of attention layers — a hybrid design Nvidia has been iterating on since earlier Nemotron releases, chosen specifically because pure-attention transformers get expensive fast at long context. It ships with a 1M-token context window, runs on a single Blackwell or Hopper GPU (a lone DGX Spark or H100 is enough), and is licensed under OpenMDW-1.1 — permissive enough for commercial use without asking Nvidia or paying a license fee. Pretraining ran past 20 trillion tokens, with a training-data cutoff of September 2025 and post-training data through May 2026.
NeMo Switchyard is a Rust proxy and library, released under Apache 2.0, that sits between an agent and a pool of models. It accepts requests in OpenAI Chat Completions, OpenAI Responses, or Anthropic Messages format, and can forward each request to a different backend in that backend's native format — meaning a client speaking the Anthropic Messages API can be served by a model that only speaks OpenAI's format, transparently. It ships four routing strategies out of the box — an LLM-as-classifier router, a signal-driven "stage router" that reads tool results and error states already in the conversation, an escalation router that runs the cheap model first and lets a judge decide whether to retry on the expensive one, and a random router for A/B splits — plus a library path (switchyard-libsy) for embedding the routing logic directly in your own Rust service.
Individually, neither piece is dramatic. Together, they describe a specific worldview: that the unit of "your AI stack" is no longer one model, it's a pool of models with a dispatcher in front of them, and Nvidia wants to own the dispatcher.
The benchmark gap nobody should skip past
Nvidia's own PinchBench numbers for Lightning claim up to 4x faster output speed than comparably sized models. Read one paragraph further into the same release and the number changes: on an agentic task run, Lightning finished at 85–86% accuracy while completing the work 30% faster than Alibaba's Qwen 3.6 35B at similar accuracy. Both figures are true. They're measuring different things — 4x is raw token throughput in a lab; 30% is wall-clock time on a job that also involves tool calls, retries, and orchestration overhead. Any time a vendor leads with a throughput multiple, it's worth asking what happens to that number once the model has to actually do something, and here Nvidia answered the question for you in its own release notes, which is more candor than most model announcements bother with.
The rest of the benchmark table is honest in the same unglamorous way. 81.94 on MMLU Pro, 75.44 on GPQA Diamond, 51.56 on SWE-bench Verified, 24.58 on Terminal-Bench 2.1. These are respectable mid-tier numbers, not frontier ones, and Nvidia isn't pretending otherwise — the entire pitch for Lightning is that it's a workhorse for narrow, high-volume tasks inside a larger system, not a model you point a hard reasoning problem at directly.
How the router actually decides
The interesting engineering is in how Switchyard picks a model, because "smart routing" is a phrase every AI gateway vendor uses and almost none of them explain. Nvidia's own writeup lays out three signal sources a router can draw on: the request itself (classify the topic or estimated difficulty before dispatch), the model's internal state during generation (logprobs, cascades, attention patterns — for routers with access to that), and system-level signals (price, latency, current load, recent error rate). The stage router is the pragmatic default: instead of running a separate classifier model to guess difficulty up front, it reads signals already present in the conversation — a tool call that just failed, an agent that's three retries deep, a step that historically needs a stronger model — and routes on that.
This matters because it's the same problem ngrok's AI Gateway and half a dozen LLM proxy startups are also trying to solve, and Nvidia is entering with a genuinely different angle: it isn't selling routing as a product, it's giving it away to make its own free model more attractive to route to. NeMo Switchyard's default quick-start actually routes through OpenRouter using an OPENROUTER_API_KEY — which is a small irony worth sitting with: a tool marketed as freeing you from single-vendor lock-in ships, out of the box, defaulting to yet another intermediary you now depend on. You can point it at raw vLLM, Nvidia NIM, or Ollama endpoints instead, and the docs make that path clear, but it's not the one-line default.
What the config actually looks like
The abstraction is thinner than "smart routing" branding usually implies, which is a compliment. A minimal classifier-based deployment is one TOML file: declare an LLM client, declare a couple of named targets, declare a route that ties them together with a threshold.
schema_version = 1
[llm_clients.openrouter]
format = "openai_chat"
base_url = "https://openrouter.ai/api/v1"
api_key_env = "OPENROUTER_API_KEY"
[targets.weak]
id = "openai/gpt-4o-mini"
llm_client = "openrouter"
[targets.strong]
id = "openai/gpt-4o"
llm_client = "openrouter"
[routes.smart]
id = "switchyard"
type = "llm_classifier"
mode = "capability"
classifier_target = "weak"
strong_target = "strong"
weak_target = "weak"
base_threshold = 0.5
Swap type = "llm_classifier" for stage_router and the same file starts reading conversation signals instead of running a separate classification call before every request — cheaper per-request, but it needs signals worth reading, which means it works better once your agent already surfaces tool errors and retry counts in a structured way rather than as free text. api_key_env rather than an inline key is a small but correct detail: the secret lives in the environment, not the config file that inevitably ends up committed somewhere. Point base_url at a local vLLM or NIM endpoint instead of OpenRouter and the "vendor lock-in" concern mostly evaporates — the OpenRouter default in the quick-start is a convenience for the five-minute demo, not a structural dependency.
The number that should make labs nervous
The most concrete evidence for whether any of this works comes from outside Nvidia's own marketing. LangChain ran Switchyard across 145 tasks — figures compiled by MarkTechPost — routing between Lightning and Claude Opus 4.8. Only 7% of calls went to Opus. The routed system cut costs by 74% against sending everything to Opus alone, at a cost of roughly six accuracy points. Separately, Boomi evaluated Switchyard across five routing capabilities and got 100% domain-routing accuracy, sending 59% of traffic to a model roughly five times faster than its default.
Sit with what that 7%/74% split actually implies. It's not a claim that Lightning is competitive with Opus 4.8 — it explicitly isn't, on the hardest 7% of tasks. It's a claim that on a real agentic workload, the overwhelming majority of steps don't need frontier intelligence at all: tool calls, formatting, routine classification, straightforward code edits. If that ratio generalizes even roughly, it's a direct commoditization argument aimed at every lab currently pricing frontier models as if every token needs frontier reasoning behind it. Nvidia has no reason to inflate that number in the labs' favor, and every reason to publish it, because Nvidia doesn't care which model wins the routing decision — it cares that more inference happens overall, on its GPUs, regardless of whose weights are running.
Beyond the two published benchmarks, Nvidia lists a set of companies customizing Lightning for domain-specific agentic work: CrowdStrike for cybersecurity alert triage, Harvey with Trajectory for legal workflows, CodeRabbit with Baseten for code review, plus Lila Sciences and Fastino Labs for scientific and vertical-industry tasks. These read as early-adopter case studies rather than independent validation — they're Nvidia's list, in Nvidia's release — but they do establish that the "specialized sub-agent" pitch is landing with real infrastructure teams, not just showing up in a slide deck.
What the release omits
The Switchyard repository's own README carries a maturity warning that the press coverage skipped entirely: "Switchyard is pre-alpha software that is evolving rapidly. The API and algorithms are expected to change significantly before we reach v1.0," followed by an explicit "Experimental software. Not for production use." That's not a nitpick — it's Nvidia's own engineering team telling you, in the repository you'd actually deploy from, that the thing generating the 74%-cost-reduction headline number is not the thing you should point production traffic at yet. Any team excited by the LangChain and Boomi figures needs to read that as "directionally promising, structurally unfinished," not "ready to ship."
Two other gaps are worth flagging. First, the accuracy cost isn't free — six points on the LangChain benchmark is real, and whether that's acceptable depends entirely on what the other 93% of calls are doing; a support bot answering billing questions can absorb that, a system drafting legal filings might not. Second, routing quality is only as good as the signals feeding it, and stage-router-style approaches that read tool results and error states will only get better with tuning specific to your own agent's failure modes — this isn't a drop-in that works identically across every workload, despite the "no rewrites needed" framing in Nvidia's blog post.
Competitive and strategic context
Nvidia is a chip company giving away a language model and a routing framework, and the motive isn't subtle: cheaper, freely available models increase the total volume of inference running, and inference runs on GPUs. Commoditizing the software layer expands the market for the hardware underneath it. Nvidia signed an open-weights advocacy letter to Washington earlier this year, and Jensen Huang used his first-ever post on X to promote it — notably, OpenAI, Anthropic, and Google did not sign. Giving away Lightning and Switchyard is that same strategy expressed as a product release rather than a policy letter.
Switchyard also isn't entering an empty field. LiteLLM has offered provider-agnostic routing and a unified API surface for two years; academic work on learned routers (RouteLLM and its successors) established the cost/accuracy tradeoff curve Switchyard is now productizing; and commercial gateways sell the same pitch with hosted infrastructure attached. What Switchyard adds isn't a novel routing algorithm — LLM-as-classifier and signal-based routing are both well-trodden ideas — it's that Nvidia is willing to give the whole thing away under Apache 2.0, embeddable as a Rust library with no hosted-service dependency, specifically to make routing traffic toward its own open model the path of least resistance. That's a different competitive move than any router-as-a-service startup can make, because none of them also sell the GPUs the traffic ultimately runs on.
It's also, by Nvidia's own comparison, a catch-up move rather than a leadership claim. The competitive benchmark Nvidia chose for Lightning was Qwen 3.6, not any American open model — because at this size class, Qwen is what you have to beat. DeepSeek has spent the year cutting prices on open models to the point where running something less capable stopped making economic sense, and Moonshot AI's Kimi K3 currently holds the record as the largest open-weight model ever released. Reporting from The Information indicates Nvidia is already building Nemotron 4, a trillion-parameter-plus successor to the 550B Nemotron 3 Ultra, possibly landing as early as this autumn — and even at that size, it would still be smaller than the largest Chinese open models. American open-weight releases have mostly functioned as a policy argument this year rather than a shipping product line; Nemotron 3.5 Lightning is real and genuinely open, but it's Nvidia showing up to a competition China has been running and winning for months.
Practical read: who should actually use this
If you're running an agent that makes a high volume of narrow, repeatable calls — tool invocation, classification, routine code review, ticket triage — inside a larger system already anchored by a frontier model, Switchyard's stage-router pattern is worth prototyping now, specifically because it's designed to slot in without rewriting your application against a new API. Point it at your existing OpenAI- or Anthropic-shaped client code and let it split traffic behind the scenes.
If you're evaluating this for anything customer-facing or accuracy-critical in production this quarter, the repository's own "not for production use" warning should be taken at face value — treat this as an integration you spike on, not one you ship on, until the API stabilizes toward v1.0. And if your workload doesn't actually have a cheap/expensive task split — if every call genuinely needs frontier reasoning — none of this buys you anything; routing only pays off when a meaningful share of your traffic is over-served by whatever model you're currently defaulting to.
The more durable signal from this release isn't Nemotron 3.5 Lightning's benchmark table. It's that Nvidia — a company with zero incentive to care which model wins — just published a benchmark suggesting most of what agents do doesn't need a frontier model at all. That's a more uncomfortable number for OpenAI and Anthropic's pricing than anything in the model card.
Where does the 74% cost cut actually break down in your own agent workloads — do you have a similar cheap/expensive split, or is your traffic more uniformly hard than the LangChain and Boomi benchmarks assume?
Sources:
- NVIDIA Nemotron 3.5 Lightning and NeMo Switchyard Deliver Faster, Smarter, More Efficient Agentic AI
- Route AI Agent Workloads Across Models with NVIDIA NeMo Switchyard
- NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 model card
- NVIDIA-NeMo/Switchyard repository
- Nvidia is building a trillion-parameter open model, and it would still be smaller than China's — The Next Web
Top comments (0)