Jensen Huang walked on stage at Computex 2026 in Taipei on June 1 and announced what NVIDIA calls the most intelligent open-weights AI model built in the United States: Nemotron 3 Ultra, a 550-billion-parameter mixture-of-experts model that delivers over 300 output tokens per second and cuts complex agentic task costs by 30 percent. The weights ship to Hugging Face on June 4, 2026. Here is everything developers need to understand the architecture, run the benchmarks, and deploy it.
The announcement lands at a significant inflection point. The two models currently at the top of the frontier — Claude Opus 4.8 and GPT-5.5 — are proprietary, API-only, and priced accordingly. DeepSeek V4 Pro, the only open-weights competitor anywhere near frontier performance, requires roughly 862GB of VRAM to run — effectively a dedicated GPU cluster. Nemotron 3 Ultra is NVIDIA's answer to both constraints: intelligence approaching the frontier, open weights, and an architecture engineered for throughput rather than just accuracy.
The Numbers: What You Are Actually Getting
The headline specs: 550B total parameters, 55B active per forward pass via mixture-of-experts routing, a 1-million-token context window, and native support for multi-token prediction. The model was trained in 4-bit NVFP4 precision on NVIDIA's Blackwell architecture — the same hardware on which it runs most efficiently in production.
On the Artificial Analysis Intelligence Index — a composite benchmark aggregating 10 evaluations spanning reasoning, coding, general knowledge, and agentic performance — Nemotron 3 Ultra scores 48.0. That places it as the top US open-weight model by a significant margin. For reference: Claude Opus 4.8 scores 61.4 and GPT-5.5 scores 60.2 on the same index. Nemotron 3 Ultra sits roughly 12–13 points behind the closed-source frontier — but at zero per-token API cost when self-hosted.
The throughput story is more compelling than the intelligence index alone. Serving at over 300 output tokens per second on optimized hardware, Nemotron 3 Ultra runs approximately 5× faster than a comparable dense model at the same accuracy level. NVIDIA attributes this to the LatentMoE architecture and Mamba-2 layers that provide linear-time complexity for long-context inference, in contrast to the quadratic attention cost that makes standard Transformers expensive at 1M-token context lengths.
The LatentMoE Architecture, Explained
Nemotron 3 Ultra introduces a new expert routing mechanism called LatentMoE that is worth understanding for developers planning to fine-tune or build on top of it.
In a standard MoE model, routing selects a small subset of expert networks per token and sends the full token embedding through each selected expert. LatentMoE projects the token from the model's hidden dimension into a smaller latent dimension before routing and expert computation. This compression achieves three things simultaneously:
Reduces the VRAM footprint of routed expert parameters by approximately 4×
Allows the same inference budget to activate 4× more experts per token
Improves accuracy per byte because more specialists contribute to each prediction without a proportional cost increase
The net effect: Nemotron 3 Ultra activates more specialized computation per token than a standard MoE at the same memory cost. The 10% activation ratio (55B of 550B) is already class-leading efficiency, but LatentMoE compounds this by making each activated expert more representationally compressed and therefore faster to route through.
The hybrid Mamba-Transformer design adds a second efficiency layer. Mamba-2 layers handle long-range sequence dependencies with linear time complexity, replacing a subset of the attention layers that would otherwise make 1M-token context inference prohibitively expensive. The result is a model that can process one million tokens in context without the memory and compute explosion that standard Transformer attention would require at that scale.
Multi-Token Prediction (MTP) predicts multiple future tokens in a single forward pass rather than autoregressively generating one token at a time. This technique, popularized by DeepSeek V4, reduces the effective number of forward passes required per output token. Combined with the Mamba-2 linear complexity, it produces the 300+ token/second throughput NVIDIA is advertising — a figure independently corroborated by Artificial Analysis benchmarks.
Benchmark Position: An Honest Assessment
Intelligence Index rankings are aggregate composites, and the gap between Nemotron 3 Ultra (48.0) and frontier closed models (60+) compresses significantly on specific workloads. The categories where the gap matters least for developers:
Code generation and debugging: NVIDIA's own benchmarks show the Ultra model performing within 8–10% of GPT-5.5 on HumanEval and LiveCodeBench. For engineering automation tasks, that margin is often within practical noise.
Long-context RAG: With a 1M token context window and linear-time Mamba layers, Nemotron 3 Ultra has a structural advantage over models limited to 200K tokens. Tasks like codebase-wide refactoring, legal document analysis, and multi-document research synthesis play to its architectural strengths.
High-throughput batch processing: At 300+ tokens/second, a self-hosted Nemotron 3 Ultra node can process multiple document summarization jobs simultaneously that would take 5× longer on a comparable dense model. The economics shift quickly at scale.
The categories where the gap matters most:
Multi-step agentic reasoning: Claude Opus 4.8's Dynamic Workflows and GDPval-AA score (1,890 Elo) reflect a capacity for sustained autonomous reasoning that no current open model fully matches. For mission-critical agent pipelines where reasoning depth directly maps to business outcomes, the closed-model advantage is real.
Instruction following on ambiguous tasks: Frontier models have accumulated years of RLHF refinement that produces better calibration on edge cases. Open-weights models at this scale are still catching up on instruction-following reliability in adversarial production scenarios.
The honest framing: Nemotron 3 Ultra is not GPT-5.5 or Claude Opus 4.8. It is the best open-weights model available for teams that need data sovereignty, cost control, or cannot route enterprise data through third-party APIs. Those constraints cover a substantial fraction of production deployments in finance, healthcare, legal, and defense.
How to Access Nemotron 3 Ultra
NVIDIA is distributing the model through four primary channels, each suited to different deployment contexts.
Option 1: Hugging Face (Self-Hosted)
The weights are published at nvidia/NVIDIA-Nemotron-3-Ultra-550B on Hugging Face. Unlike many nominally "open" models that release only inference weights, NVIDIA is also publishing training recipes, a 2.5-trillion-token pre-training dataset, and specialized code and math datasets through the official NVIDIA-NeMo/Nemotron GitHub repository. This means Nemotron 3 Ultra is genuinely fine-tunable, not just deployable.
Minimum hardware for full BF16: 8× H100s (640GB VRAM). The FP8 quantized variant fits on a 4× H100 configuration (320GB VRAM). NVFP4, the training-native precision, requires Blackwell (H200 or GB200) and reduces VRAM requirements further — NVIDIA has not published the exact NVFP4 memory footprint at time of writing, but early reports suggest it fits a single DGX Spark.
Option 2: NVIDIA NIM Microservice (Managed)
NVIDIA's NIM (NVIDIA Inference Microservices) wraps the model as an OpenAI-compatible REST endpoint with automatic batching, KV cache management, and observability included. Available at build.nvidia.com with an NVIDIA AI Enterprise license for production use. NIM is the fastest path from zero to a compliant, auditable API endpoint — particularly relevant for enterprises subject to data residency requirements where self-hosting is mandatory but engineering overhead must be minimized.
from openai import OpenAI
client = OpenAI(
base_url="https://integrate.api.nvidia.com/v1",
api_key="YOUR_NVIDIA_API_KEY"
)
response = client.chat.completions.create(
model="nvidia/nemotron-3-ultra-550b",
messages=[
{"role": "system", "content": "You are a senior software engineer."},
{"role": "user", "content": "Review this codebase and identify security vulnerabilities."}
],
max_tokens=8192,
temperature=0.1
)
print(response.choices[0].message.content)
Option 3: OpenRouter (Instant API Access)
OpenRouter exposes Nemotron 3 Ultra as a standard OpenAI-compatible API endpoint. This is the fastest path for developers who want to evaluate the model without provisioning GPU infrastructure. No NVIDIA account required. Use the model identifier nvidia/nemotron-3-ultra-550b in OpenRouter's API, billed per token at OpenRouter's published rates.
Option 4: Self-Hosted with vLLM or SGLang
NVIDIA has published official vLLM cookbooks in the NVIDIA-NeMo/Nemotron GitHub repository under usage-cookbook/Nemotron-3-Ultra-Base. For sustained production workloads, NVIDIA also supports TensorRT-LLM, which delivers higher throughput than vLLM at the cost of more complex initial configuration. SGLang is worth evaluating: on H100 hardware, SGLang leads vLLM by approximately 29% throughput on standard workloads and up to 6× on prefix-heavy RAG pipelines where KV cache reuse is significant.
# Quick start with vLLM
pip install vllm
python -m vllm.entrypoints.openai.api_server --model nvidia/NVIDIA-Nemotron-3-Ultra-550B-FP8 --dtype float8 --tensor-parallel-size 4 --max-model-len 131072 --port 8000
Practical Use Cases for 2026
Enterprise Agentic Pipelines With Data Sovereignty
The strongest case for Nemotron 3 Ultra is enterprises running agentic workflows over sensitive data: financial modeling, legal document review, healthcare records analysis, internal code audit. At 48 AA Intelligence Index and 1M token context, it handles the complexity of real enterprise tasks. At open weights with NIM deployment, the data never leaves your infrastructure. This combination — near-frontier intelligence, verified data control, predictable compute costs — is what closes the gap between a proof-of-concept agent and a compliance-approved production system.
High-Volume Code Generation Pipelines
At 300+ tokens/second, a single GPU node running Nemotron 3 Ultra can serve multiple simultaneous code generation sessions with lower latency than a throttled external API endpoint. Teams running CI/CD automation that generates test suites, migration scripts, or documentation should benchmark the cost-per-output-token carefully. At scale, the savings over frontier API pricing can be substantial even after accounting for GPU infrastructure costs. A rough calculation: at 300 tokens/second and $4/GPU-hour on H100 cloud, you are generating roughly 270,000 tokens per dollar of compute — compare that to frontier API pricing in the range of $15–30 per million output tokens.
Long-Context Document and Codebase Workflows
The 1M token context window is functional, not theoretical. The Mamba-2 architecture ensures that processing the full context does not incur quadratic compute cost as you scale toward the context limit. Teams currently chunking large documents due to context limits can run them whole. A 1M-token window fits approximately 750,000 words of text — equivalent to processing a complete enterprise codebase, a full legal agreement package, or several years of customer support transcripts in a single inference call.
The Open-Weights Moment
Nemotron 3 Ultra is not the first large open-weights model — but it may be the most strategically significant one since Llama 4 Scout. The combination of near-frontier intelligence, published training data and recipes, a hardware-aware architecture optimized for Blackwell, and four distinct deployment options represents a clear thesis: NVIDIA believes the long-run value in the AI stack accrues to hardware and infrastructure, not model weights. Publishing the weights is therefore commercially strategic, not charitable. Every team that builds a production pipeline on Nemotron 3 Ultra is a future NVIDIA GPU customer.
For developers, the implication is practical: a high-quality, genuinely open model now exists that can be fine-tuned on proprietary data, audited by compliance teams, deployed on private infrastructure, and modified without a licensing agreement with a closed-model API provider. The intelligence gap with Opus 4.8 and GPT-5.5 is real but narrowing. If the Nemotron 3 Super (120B) trajectory is any precedent, the Ultra will receive ongoing training and post-training refinement updates through 2026.
What to Do Right Now
Evaluate on OpenRouter or build.nvidia.com today. The model is available as an API endpoint with no GPU provisioning required. Run your standard benchmark prompts before the June 4 Hugging Face weights release so you have a baseline.
Pull the NVIDIA-NeMo/Nemotron GitHub repository. The vLLM cookbook, training recipes, and dataset documentation are already live. Reviewing them now will accelerate your deployment decision.
Benchmark against your actual workload, not aggregate indices. If your primary use case is long-context RAG or high-volume batch processing, Nemotron 3 Ultra may outperform or match frontier models on your specific task even with a lower aggregate index score.
Model your hardware economics. The FP8 variant needs 4× H100. NVFP4 on Blackwell requires fewer resources. Compare a dedicated H100 node cost against your current frontier API bill at projected token volume — the crossover point is lower than most teams expect.
Evaluate fine-tuning eligibility. The published training dataset and training recipes are a significant differentiator over every closed model. If your application benefits from domain-specific adaptation — legal reasoning, scientific literature, financial modeling — Nemotron 3 Ultra is currently the only near-frontier option that permits and provides the infrastructure for it.
Conclusion
Nemotron 3 Ultra is the clearest signal yet that NVIDIA is serious about the software layer of AI infrastructure, not just the hardware. A 550B open-weights model with LatentMoE, 1M token context, 300+ token/second throughput, and published training data is not a research release — it is a production bet. It will not replace Claude Opus 4.8 for teams that need the highest reasoning quality and are comfortable routing data through Anthropic's API. It will replace frontier closed models for a meaningful fraction of production workloads where data sovereignty, cost predictability, and customizability outweigh the 12-point intelligence index gap. June 4 is the date to watch.
Originally published at wowhow.cloud
Top comments (0)