NVIDIA Just Open-Sourced Model Routing: Switchyard Moves Into the Agent Loop, and "One Model Per Step" Becomes the New Default

#ai #apigateway #wrouter

Opening: Routing Is Sinking Down the Stack

A week ago, routing was still a selling point of managed gateway products: on August 4, Google Cloud API Gateway's model routing entered public preview, and Databricks Unity AI Gateway hit GA the same day. Then on August 11, NVIDIA took a different route entirely — it open-sourced the router. NeMo Switchyard is a model routing library that embeds directly inside agent frameworks, released alongside Nemotron 3.5 Lightning, a 30B MoE open model purpose-built for high-frequency workflow steps. One day later, on August 12, Tetrate shipped its Agent Router as a VS Code extension: one API key turns 160+ models into an editor-level shared resource.
Three headlines, one trend: routing is no longer exclusive to gateway products. It is sinking fast — into open-source libraries, into the agent loop, into the editor itself.

Technical Deep Dive: From Per-Request to Per-Step

The Granularity Shift: Every Agent Step Deserves Its Own Model
Traditional gateway routing decides at the granularity of a request: a request arrives, a classifier estimates difficulty, and traffic goes to the appropriate model. Agent workflows look nothing like that. A single user task fans out into dozens of model calls — planning, tool use, code generation, testing, review, correction. The difficulty of these steps varies wildly. Run them all on one model and you either pay frontier prices for trivial steps or degrade output on the hard ones.
NeMo Switchyard pushes the routing decision inside the agent loop: it automatically selects the most suitable model for each step of a workflow, with routing algorithms tunable along quality, latency, and cost, spanning whatever mix of open, proprietary, and NVIDIA models a developer runs — no application rewrite required. NVIDIA's internal benchmarks: frontier-level accuracy maintained while task completion cost drops to roughly one-third of running Opus 4.8 alone. Partner numbers from Boomi are more concrete: 100% domain-routing accuracy, with 59% of traffic dispatched to a model that is 5x faster.
The Routing Destination: Nemotron 3.5 Lightning
A router needs targets worth routing to. Nemotron 3.5 Lightning is a 30-billion-parameter MoE open model with a clear job description: the high-frequency workstation in a system of models. In NVIDIA's architecture, a frontier reasoning model (Nemotron 3 Ultra or GPT-5.6 class) plans and orchestrates, while Lightning handles high-volume specialized steps — code review, tool use, security alert monitoring, billing Q&A. Official figures claim up to 4x faster output and 30% faster agentic task completion versus its class. Weights are live on Hugging Face, ModelScope, and OpenRouter, along with Nemotron-RL-Agentic-Terminal-Pivot, the RL dataset used to post-train its coding-agent skills.
The "one frontier orchestrator plus a crew of lightweight specialists" architecture is moving from paper language to factory default.
The Other End: Editors and Control Planes
Tetrate's August 12 VS Code extension takes a different sinking path. Instead of BYOK custom endpoints (which only serve the chat view), it registers as a Language Model Chat Provider (an API stable since VS Code 1.104), making models an editor-level shared resource: chat, agent mode, and every extension in the window that calls vscode.lm share one key, with runtime discovery of the 164 models currently reachable. Meanwhile, Cloudflare merged Workers AI and AI Gateway into a unified control plane on August 7 and previewed model-first routing: you declare the model you want, and the gateway decides which provider serves it.
The Scarce Resource Has Moved
Put the three stories together and the conclusion is sharp: routing algorithms themselves are commoditizing. Open-source libraries you can self-host, gateway products with routing built in, native editor support. But commoditized routing exposes two new bottlenecks:

On day one of self-hosting a Switchyard router you discover: eight models in your routing table means eight vendor accounts, eight keys, eight invoices, eight rate-limit policies, and eight availability curves. The smarter the router, the more painful the fragmentation of the supply layer.

In Practice: The Router Decides "Which Model"; a Unified Supply Layer Ensures "All Reachable"

This is exactly where a model relay service sits. wrouter.ai provides one OpenAI-compatible entry point: a complete model catalog (frontier and mainstream open models on a single list), stable service (no per-vendor rate-limit handling or failover logic), and unified billing (one account showing what every step spent on which model). Your routing logic — Switchyard or hand-rolled rules — only needs to output a model name; everything else goes through the same base_url:

from openai import OpenAI

client = OpenAI(
    base_url="https://wrouter.ai/v1",
    api_key="YOUR_WROUTER_KEY",
)

# The router picks a model per workflow step; the supply layer never changes
STEP_MODEL = {
    "plan":      "claude-opus-5",          # planning: frontier model
    "code":      "deepseek-v4",            # codegen: price-performance tier
    "review":    "nemotron-3.5-lightning", # high-volume review: lightweight specialist
}

def run_step(step: str, messages: list):
    return client.chat.completions.create(
        model=STEP_MODEL[step],
        messages=messages,
    )

Swapping a model is a one-line change to a mapping table, not a new vendor account; month-end reconciliation is one bill, not eight CSV exports.

Closing

As routing turns from a product feature into an open-source component, the center of competition shifts from "who can split traffic" to "whose model catalog behind the router is more complete, more stable, and easier to account for." If you are building per-step routing for agent workflows, start by consolidating your supply layer into one entry point — head to wrouter.ai and point your routing table at it.

Sources