NVIDIA Cosmos 3 Nano for Local AI in 2026: 16B Omnimodel, BF16-Only, and Whether Your Consumer RTX Can Actually Run It

This article was originally published on runaihome.com

TL;DR: Cosmos 3 Nano is genuinely open (OpenMDW-1.1, commercial use allowed) and "only" 16B parameters, which makes it sound like a 24GB-card model. It isn't. NVIDIA ships it BF16-only — no FP8, no FP4, no GGUF — so the weights alone are ~32GB, and the recommended card is a 96GB RTX PRO 6000. Your RTX 3090 can't load it; an RTX 5090 holds the weights with almost nothing left for the work.

	Used RTX 3090 24GB	RTX 5090 32GB	RTX PRO 6000 96GB
Runs Cosmos 3 Nano?	No — weights don't fit	Technically, with no headroom	Yes — NVIDIA's recommended card
VRAM vs ~32GB BF16 weights	24GB (short by 8GB)	32GB (exactly the weights)	96GB (room for context + video frames)
Price (Jun 2026)	~$1,070 used	~$4,000+	~$8,000–$13,250
Best for	LLMs, not Cosmos	Edge case only	The intended use case

Honest take: Don't buy hardware for Cosmos 3 Nano right now. Rent a workstation GPU by the hour to evaluate it, keep your local rig on Qwen3.6 and Gemma 4 for everyday LLM work, and wait for community quantization before committing $8,000+ to a card.

What Cosmos 3 Nano actually is

Cosmos 3 is NVIDIA's first fully open "omnimodel" for Physical AI — a single model that reasons about the physical world and generates the pixels, audio, and robot actions that follow from that reasoning. It was released on May 31, 2026 and announced at GTC Taipei on June 1, 2026, with the full technical report following on June 22, 2026.

"Omnimodel" is doing real work in that sentence. Most local models you run today take text in and emit text out. Cosmos 3 natively understands and generates text, images, video, ambient audio, and action sequences in one set of weights. That's the whole point: instead of stitching a vision-language model to a separate video diffuser to a separate policy network, you get one model that reasons before it acts.

The architecture behind this is a two-tower Mixture-of-Transformers (MoT). Each Cosmos 3 model is split into two equally-sized transformers that work together:

• A Reasoner that understands the scene, plans, and emits the structured representation that guides what comes next.
• A Generator that turns that plan into actual pixels, waveforms, or action tokens.

For Cosmos 3 Nano, that's 8B reasoner + 8B generator = 16B total. The larger sibling, Cosmos 3 Super, is 64B (32B + 32B) and is built for datacenter Hopper/Blackwell deployment, not home labs. There are also task-specialized checkpoints — Super Text2Image, Super Image2Video, and a Nano-Policy-DROID world-action model fine-tuned on the DROID robotics dataset.

So "Nano" here means "the small one in a family aimed at robots and autonomous machines," not "the one that fits a gaming GPU." That distinction is the entire hardware story.

The number that breaks the consumer-GPU dream: BF16-only

Here's where the "16B, so it fits 24GB" intuition falls apart.

When you run a 16B language model locally, you almost always run it quantized. A 16B LLM at Q4_K_M is roughly 9–10GB and drops comfortably onto a 12GB card. That math is so ingrained that the obvious assumption — and the one the headline begs you to make — is that Cosmos 3 Nano lands on a 24GB RTX 3090 with room to spare.

It doesn't, because Cosmos 3 Nano is tested and supported in BF16 only. The maintained vLLM deployment path explicitly notes that other precisions — FP4, FP8, even FP16 — are not officially supported for the model. There is no official GGUF, no Q4_K_M, no llama.cpp or Ollama path. As of late June 2026, the only sanctioned way to run it is the full BF16 checkpoint through vllm-omni.

BF16 means 2 bytes per parameter. 16 billion parameters × 2 bytes ≈ 32GB just for the weights — before you add the KV cache, before activation memory, and before the substantial buffers an omnimodel needs to actually generate video frames and audio. That's why NVIDIA's own recommendation for single-GPU Nano inference is the RTX PRO 6000 Blackwell with 96GB of VRAM — not because the weights need 96GB, but because generating world-model output on top of a 32GB weight footprint needs serious headroom.

The official test hardware listed for Cosmos 3 is GB200 and H100, with Ampere, Hopper, and Blackwell as the supported microarchitecture families. Notice what's missing: no mention of GeForce cards as a target.

Card-by-card: what can and can't run it

Used RTX 3090 / 3090 Ti / 4090 (24GB) — Can't load it. The BF16 weights are ~32GB and there's no quantized path to shrink them. This is the painful one, because the used RTX 3090 is still the value king for local LLMs at around $1,070 — but Cosmos 3 Nano is the rare 2026 model where 24GB simply isn't enough and you can't quantize your way out.

RTX 5090 (32GB) — Technically holds the weights, practically marginal. 32GB of GDDR7 matches the ~32GB BF16 footprint almost exactly, which leaves essentially nothing for the KV cache, activation memory, or the frame buffers an omnimodel needs to generate video. You might coax a short text-or-image reasoning request through it; sustained world-generation or action rollout is where you'll hit out-of-memory. At ~$4,000+ street price in June 2026, it's an expensive way to be perpetually one frame from a CUDA OOM. (If you're weighing a 5090 for general AI work, our RTX 5090 vs RTX 4090 breakdown covers the everyday-LLM case, which is a much better fit for that card.)

RTX PRO 6000 Blackwell (96GB) — The intended card. 96GB of GDDR7 gives the ~32GB of weights plenty of room for context and generation buffers, and NVIDIA explicitly recommends it for single-GPU Nano deployment. The catch is the price: it launched around $8,565, and NVIDIA's own listing has since climbed to $13,250 — a 60%+ increase driven by the same GDDR7 supply crunch hitting the whole stack. We went deep on this card in our RTX PRO 6000 Blackwell local-AI guide; for Cosmos 3 Nano specifically, it's the only single consumer-adjacent GPU that runs the model the way NVIDIA tested it.

Cosmos 3 Super (64B) — Out of scope for home labs entirely. It's a datacenter model: the maintained recipe runs it with --tensor-parallel-size 4 --enable-layerwise-offload across four GPUs. If you're asking whether your tower can run Super, the answer is no.

How you'd actually deploy Nano

The maintained path is a single Docker image. NVIDIA ships vllm/vllm-omni:cosmos3, and a single-GPU Nano launch looks like this:

docker run --runtime nvidia --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v "$(pwd):/workspace" \
  -p 8000:8000 --ipc=host \
  vllm/vllm-omni:cosmos3 \
  vllm serve nvidia/Cosmos3-Nano \
    --omni \
    --model-class-name Cosmos3OmniDiffusersPipeline \
    --allowed-local-media-path / \
    --port 8000 \
    --init-timeout 1800

A few things worth flagging before you copy-paste:

• --tensor-parallel-size 1 is correct for Nano — it's a single-GPU model and needs no model parallelism. That's only true because the card is big enough; on Super you'd set it to 4.
• --init-timeout 1800 (30 minutes) isn't paranoia. The Nano download plus CUDA dependencies run into tens of GiB, and first-load initialization is slow. Budget the disk and the wait.
• No --enable-layerwise-offload for Nano. That flag exists for Super on memory-constrained datacenter GPUs. If you find yourself reaching for offload flags to squeeze Nano onto a smaller card, you're fighting the model — there's no supported low-VRAM config today.
• It's vllm, not Ollama or llama.cpp. Because there's no GGUF, the friendly local-runner ecosystem doesn't apply yet. You're running a vLLM server and talking to it over an OpenAI-compatible API on port 8000.

If you don't already have a self-hosting workflow for containerized model s