DEV Community: Nikhil

Gemma-4 31B + vLLM + RTX 6000 PRO : 1168 tokens/sec and still asking for more...

Nikhil — Tue, 30 Jun 2026 12:35:13 +0000

We pushed Gemma-4 31B to 24 concurrent requests on a single RTX 6000 PRO Blackwell. The queue never filled. ~1.17k tokens/sec, and it still had headroom.

Most LLM "benchmarks" show you one request at a time. That tells you almost nothing about production.

So we ran Gemma-4 31B (FP8) on vLLM under a real ShareGPT workload, ramping concurrency 12 → 16 → 20 → 24, and watched what actually happens.

The numbers that mattered:

→ Peak throughput: ~1,168 tokens/sec total (~548 tok/s output)

→ Median time-to-first-token: ~0.7s — snappy even under load

→ Queue depth: averaged 0.41, peaked at just 3 while 14–21 requests ran concurrently

→ Server stayed unsaturated across the entire sweep

The one thing to watch:

Tail TTFT.
Median first-token stays fast, but p99 climbs to ~19s at the heaviest concurrency. That's the first metric to flex as you push higher — not throughput, not the queue.

Setup:

1× RTX 6000 PRO Blackwell (96GB)
Gemma-4 31B-it, FP8 checkpoint
vLLM 0.20 — prefix caching + chunked prefill on
ShareGPT workload, 1024 max output tokens, streaming ON
Max model length (context) : 4096

Verdict:

A single Blackwell card runs a 31B model at 24-way concurrency without breaking a sweat. The high end-to-end latency is just long generations, not queuing — and there's clearly room to climb past 24.

Token Throughput chart:

E2E Latency Chart

Full writeup — configs, charts, and per-concurrency breakdown — in the comments. ↓

Your AI product is the LLM's next feature — unless you own the stack.

Nikhil — Thu, 25 Jun 2026 13:14:49 +0000

Are you using LLM APIs to code your product?

To serve your customers?

To classify your customer data?

To route tickets, summarize documents, score leads, generate copy, power your "chat with X" feature, or quietly run the part of your app that you'd be embarrassed to admit is "just a prompt"?

If yes — welcome. You're in the majority.

Calling a frontier model API is the single highest-leverage thing a small team can do right now. One fetch, and you've got capability that used to need a research lab.

But here's the thing about leverage: it works in both directions.

You're treating the API as plumbing — a neutral utility you pay for by the token. It isn't. It's a two-way pipe.

You're sending capability requests out, sure. But you're also sending a continuous, high-resolution signal in — straight into the roadmap of the company that can ship your entire product as a checkbox.

And that brings us to the oldest law in engineering.

Anything that can go wrong, will go wrong

Murphy's Law isn't really about pessimism. It's about respecting failure modes you've decided not to look at.

Most builders running on LLM APIs have a mental list of what could go wrong:

The model gets deprecated.
Prices change.
Latency spikes.
Rate limits bite during a launch.
An output hallucinates in front of a customer.

All real. All survivable. None of them are the scary one.

And this isn't just paranoia — look at the pattern. It's already happening. Repeatedly.

A few years ago, this was a startup

A few years ago, "AI astrology app" sounded like a startup idea.

Now there are astrology GPTs inside ChatGPT itself.

A few years ago, "AI stock research assistant" sounded like a serious fintech product.

Now people use general-purpose LLMs to summarize earnings, read market news, compare companies, generate investment memos, and test whether models can pull signal out of financial news — no fintech wrapper required.

A few years ago, "AI shopping assistant" was a startup category.

Now ChatGPT has shopping research built in.

A few years ago, "AI coding assistant" was a separate product.

Now OpenAI has Codex. Anthropic has Claude Code. The model companies aren't handing you autocomplete anymore — they're shipping agents that edit files, run tasks, plug into your IDE, and reach into production workflows.

A few years ago, "Search your company documents" was a wrapper startup.

Now the platforms ship connectors and apps for Drive, GitHub, SharePoint, Gmail, Calendar — plus company knowledge and internal search, out of the box.

See it yet? Are you sure?

The lifecycle nobody wants to say out loud

Here's the trajectory, compressed:

Your AI startup begins as a product.
Then it becomes a use case.
Then it becomes a feature.
Then it becomes a dropdown inside the model provider's interface.

Product → use case → feature → dropdown. That's the funnel.

And the platform doesn't have to be malicious to push you through it. It just has to be paying attention.

This is why builders need to get more paranoid about LLM APIs.

Now, the actual problems

1. OpenAI and Claude APIs are not magic privacy boxes

Yes, the official position from major providers is usually something like:

We do not train on API or business customer data by default.

Good.

That matters.

But "not used for training by default" is not the same as:

zero retention
zero logging
zero metadata
zero policy review
zero abuse monitoring
zero operational visibility
zero future terms risk
zero strategic learning

Your prompts and outputs may not be going directly into training. But your usage still exists inside someone else's infrastructure.

What can go wrong:

Big platforms leak, get caught, and move on. ChatGPT had a 2023 bug that exposed other users' conversation titles; Meta's history runs from Cambridge Analytica to hundreds of millions of records left on public servers to AI chats surfacing in a public feed. The pattern is always the same: a statement about taking privacy seriously, a fix, a few days of headlines.
The cushion that lets them shrug it off is the cushion you don't have. A leak that costs a trillion-dollar platform a week of bad press is the leak that ends your company — because the data in those prompts wasn't theirs, it was your customer's medical records, source code, or financials.

2. OpenRouter and provider-router APIs add another trust layer

A lot of AI apps now work like this:

Your app → OpenRouter → another model provider → (maybe more providers) → response back to your app.

That means one simple API call may actually involve multiple parties, multiple policies, and multiple places where your data can be handled.

What can go wrong:

A fallback routes you somewhere you never vetted. You told customers you have Zero Data Retention, but a fallback silently sent a batch of requests to a provider that logs everything, in a jurisdiction you never checked.
You can't prove your own compliance. You can't even reconstruct which provider handled which request. You weren't lying to your customers — but you can't prove you weren't.

3. "Terms & Conditions" are only as good as their enforcement

Most developers never read the Terms & Conditions — they grab a key and ship. But two assumptions builders lean on are worth less than they look.

They won't train on my data : "Not trained on by default" is a promise about one use, not all uses. They can honor it to the letter and still retain prompts, log metadata, run abuse review, and learn from aggregate patterns. And "by default" is not "never" — defaults change.
They'll be fair to developers : Rules are enforced by leverage, not evenly. The same boundary that's a "business-development conversation" for a big partner is "a violation" for a solo dev. You're not depending on the model — you're depending on someone else's reading of their own rules, with no seat at the table.

What can go wrong:

The carve-out narrows. The guarantee only ever covered training — not retention or aggregate learning — and "default" was a door left open. It shifts, and your customers' data was in a wider gray zone than you told them.
"Fair" gets redefined the moment you matter. The behavior that was fine while you were small gets reinterpreted once your traffic or category makes you visible — and you're cut off with no warning and no appeal.
You can't promise more than you were promised. When an enterprise asks what happens to their data, your answer is bounded by terms you didn't write. When they bend, the liability lands on you, not the provider.

4. Cheap APIs are not charity

One of the biggest traps in AI right now is assuming cheap intelligence is just a gift to developers.

It is not.

Cheap APIs create dependency. Cheap APIs create distribution. Cheap APIs create habits and ecosystem lock-in. Cheap APIs make thousands of builders experiment on the provider's behalf.

Every failed wrapper still teaches the market something.
Every successful wrapper reveals demand.
Every repeated API pattern reveals a feature opportunity.
Every vertical app shows where users are willing to pay.

This is why it is in the interest of LLM companies to keep API access attractive.

They get adoption. They get dependency. They get distribution. They get developer mindshare — and eventually, they move up the stack.

What can go wrong:

The intro price was bait. The price that made your unit economics work was the introductory price — and it changes once you're dependent and re-architecting is expensive.
Your category is the roadmap. The aggregate demand signal from your whole category tells them exactly what to build — and they build it.

Your moat becomes their feature.

5. When it breaks, who's responsible?

When something goes wrong, who is responsible?

Your app? The API router? The model provider? The fallback model? The inference host? The logging layer, moderation system, vector database, plugin?

The answer is usually: "It depends."

That is not comforting.

What can go wrong:

A leak or mishandling incident, and you can't trace it. Your enterprise customer asks: where did our data go, was it retained, was it routed across borders, was Zero Data Retention actually on, which provider touched this exact request — prove it.
Most wrappers have no answer. A clever prompt chain and a Stripe checkout, but zero data-governance answers. The liability and the reputation hit land on you.

So is the answer to stop using LLMs? No.

The serious question was never "Can I build this with an LLM?" It's:

If the next model release ships this feature for free, what is left of my business?

Every problem above traces back to one root cause:

The model runs on infrastructure you don't control, governed by terms you don't write, owned by a company that may compete with your layer of the stack.

So the fix isn't to abandon LLMs.

It's to change where the model runs and who controls the stack — to pull the intelligence onto ground you actually own, without giving up the capability that made it worth using in the first place.

That's where private deployment comes in.

Use the API to find the use case. Use a private deployment to own it.

The API is the right tool for discovery. It's how you find product-market fit without buying a GPU. Prototype on GPT-5.5, Claude, or GLM-5.2's hosted endpoint, ship the MVP, and watch what your users actually do.

But once the use case is proven — once a workflow is repeating, your prompts have stabilized, and you know roughly what context lengths and volumes you're serving — that's the moment to move the proven path onto a dedicated deployment of an open-weight model that you control.

The economics flip in your favor at scale, and every problem above quietly closes.

Three platforms to run a private deployment

The good news: you no longer have to hand-roll vLLM or SGLang on raw GPUs to do this.

Three platforms let you pick a model, choose a GPU, and deploy a dedicated endpoint:

Together AI
Baseten
HexGrid Cloud

All three give you an OpenAI-compatible API, an HTTPS endpoint, key-based auth, and observability. Those are table stakes now, not differentiators.

The real differences show up in two places most people don't look until they're in production:

who controls the serving optimizations, and
what sits in the data path between you and the model.

On Together AI, you get single-tenant dedicated GPUs and Together's own optimizations (speculative decoding, "intelligent" quantization) — but the quantization and serving profile are their choices, and your calls run through Together's platform.

Baseten goes further on privacy: alongside its managed cloud (which fronts deployments with its Frontier Gateway), it offers a genuine self-hosted/VPC option where the model runs in your cloud and data never leaves it. The trade-off is that Baseten's engine still decides quantization, tensor parallelism, and batching for you.

HexGrid Cloud splits the difference in a way the others don't: you declare your request profile — context length, typical request sizing — and HexGrid tunes the quantization and serving stack to it, instead of abstracting that away. You connect through a direct endpoint to your own dedicated GPU, with no shared gateway sitting in the data path between your app and your model server — managed for you, but with nothing in the middle, plus an enterprise self-hosted/VPC option when data must stay in your own cloud.

Capability	Together AI	Baseten	HexGrid Cloud
Pick model + GPU, deploy a dedicated endpoint	✅	✅	✅
OpenAI-compatible API	✅	✅	✅
HTTPS endpoint + key-based auth	✅	✅	✅
Built-in observability	✅	✅	✅
Single-tenant / dedicated GPU	✅	✅	✅
Runs in your own VPC (data stays in your cloud)	❌ (their cloud)	✅ (enterprise self-hosted)	✅ (enterprise self-hosted)
You choose quantization for your workload	❌ (provider-chosen)	❌ (engine-chosen)	✅
Serving tuned to your request profile (context length / sizing)	❌	➖ (white-glove, their engineers)	✅
Direct endpoint to your GPU — no shared gateway in the data path	❌	➖ (VPC only; gateway otherwise)	✅

Back to where we started

A few years ago, "private LLM deployment" sounded like something only an ML platform team at a big company could pull off.

Now it's a model, a GPU, and a one-click deploy button.

The API is still the fastest way to discover what to build. Just don't confuse renting intelligence with owning your business.

Use the hosted API to find the use case — then put the proven workload somewhere no one else sits between you and your model, no terms update can ban it, and no model release can quietly turn it into a dropdown.

That's the difference between building on the platform and building at the mercy of it.

Links / sources / examples

Shopping assistants → ChatGPT Shopping Research OpenAI now has product comparison and buying-guide workflows inside ChatGPT. https://openai.com/index/chatgpt-shopping-research/
Coding assistants → OpenAI Codex OpenAI Codex turns codebase Q&A, bug fixing, and feature work into a first-party agent. https://openai.com/index/introducing-codex/
Coding agents → Claude Code Anthropic’s Claude Code brings agentic coding into terminal, IDE, browser, and Slack workflows. https://claude.com/product/claude-code
Research assistants → ChatGPT Deep Research Deep Research absorbs many “AI research assistant” workflows into ChatGPT. https://openai.com/index/introducing-deep-research/
Presentation tools → ChatGPT for PowerPoint ChatGPT can now create, edit, and polish PowerPoint presentations directly. https://chatgpt.com/apps/powerpoint/
Company knowledge bots → ChatGPT Connectors ChatGPT connectors bring external tools and company knowledge into the model interface. https://help.openai.com/en/articles/11487775-connectors-in-chatgpt
Astrology apps → Astrology GPTs Even niche consumer workflows like astrology are already packaged as custom GPTs. https://chatgpt.com/g/g-690d1c9025f88191b9f30d38e31a20a9-ai-india-astrology-gpt/

Disclosure: I work on HexGrid Cloud. I've tried to keep the comparison to verifiable, table-stakes-vs-real-differences facts — corrections welcome in the comments.

Why NVIDIA Nemotron 3 Nano matters for private open-source inference. And an easy way to deploy it privately.

Nikhil — Sun, 21 Jun 2026 04:58:00 +0000

Nemotron 3 Nano is a 30B-class open model from NVIDIA built for efficient reasoning, coding, chat, agentic workflows, and long-context applications. It uses a hybrid Mixture-of-Experts architecture, activating only a small fraction of its total parameters per token, which makes it especially attractive for teams that want strong reasoning performance without the operational cost of serving much larger dense models.

Why Nemotron 3 Nano matters

NVIDIA describes Nemotron as a family of open models with open weights, training data, and recipes designed for building specialized AI agents.[¹]

Nemotron 3 Nano is the compact, efficiency-focused model in the Nemotron 3 family. It is designed for teams that need a practical balance of reasoning quality, inference throughput, long-context support, and deployment cost.

The model is especially interesting because it combines three important ideas:

Reasoning and non-reasoning in one model

Nemotron 3 Nano can generate reasoning traces before producing its final answer, and this behavior can be controlled through the chat template.[²]
Hybrid Mamba-Transformer MoE architecture

NVIDIA’s model card describes the model as a hybrid architecture with Mamba-2, attention, and Mixture-of-Experts layers. It has 30B-class total parameters while activating only around 3B-class parameters per token.[²]
Long-context support up to 1M tokens

NVIDIA’s Nemotron 3 research page highlights long-context support up to 1M tokens, making the model relevant for large documents, retrieval-heavy workflows, agent memory, codebases, logs, and enterprise knowledge tasks.[³]

This is what makes Nemotron 3 Nano compelling: it is not just a smaller model. It is a model designed to make advanced reasoning more practical to deploy.

Press enter or click to view image in full size

Model Snapshot

Press enter or click to view image in full size

Benchmark highlights

NVIDIA’s published benchmark table compares Nemotron 3 Nano 30B-A3B-BF16 against Qwen3–30B-A3B-Thinking-2507 and GPT-OSS-20B across reasoning, coding, agentic, instruction-following, long-context, and multilingual tasks.

Reasoning and coding performance

Press enter or click to view image in full size

NVIDIA’s benchmark data shows Nemotron 3 Nano performing strongly across math, coding, and formal reasoning tasks, with especially large gains on MiniF2F compared with the listed baselines.

What this means for developers

Benchmarks are useful, but the practical question is simpler:

What can I build with this model?

Nemotron 3 Nano is a strong fit for workloads where teams need a model that can reason, follow instructions, write code, use tools, and handle longer context — without deploying a giant dense model.

Good use cases include:

AI coding assistants
Agentic workflows
RAG over large document collections
Private enterprise chat
Technical Q&A
Math-heavy and logic-heavy workflows
Long-context document analysis
Workflow automation agents
Internal developer tools
Customer support assistants with private data

If your application needs more than lightweight chat but you still care about inference cost, Nemotron 3 Nano is a practical model to evaluate.

Agentic and tool-use performance

Nemotron 3 Nano is competitive across agentic benchmarks, with strong results on SWE-Bench, Terminal Bench, TauBench average, and BFCL v4.

Built for agentic AI

The most important shift in AI applications is that models are no longer just answering questions. They are planning, calling tools, reading long context, generating code, debugging workflows, and acting as sub-agents inside larger systems.

NVIDIA positions Nemotron models as open models for building specialized AI agents.[¹]

That makes Nemotron 3 Nano especially relevant for teams building systems where the model needs to:

break down tasks,
reason through multi-step problems,
work with tools,
operate inside workflows,
analyze long context,
and produce structured outputs for downstream systems.

For many teams, this is the gap between a chatbot demo and a production AI application.

Instruction-following and chat quality

Result: Nemotron 3 Nano shows strong instruction-following and chat performance, including a 67.7 Arena-Hard-V2 average in NVIDIA’s published model-card results.

Long context: built for large inputs

Modern AI applications often need more than a short prompt.

A production model may need to reason over:

long contracts,
research papers,
support histories,
code repositories,
logs,
product documentation,
multi-turn agent memory,
or large retrieved context windows.

NVIDIA’s Nemotron 3 research page states that Nemotron 3 models support context length up to 1M tokens, and highlights Nemotron 3 Nano’s performance on long-context RULER evaluations.[³]

The Hugging Face model card reports the following RULER scores:

Result: NVIDIA’s model-card results show Nemotron 3 Nano maintaining strong RULER scores from 256k to 1M context length

Throughput advantage

Performance is not only about benchmark accuracy. For production inference, throughput matters.

NVIDIA’s Nemotron 3 research page states that on an 8K input / 16K output setting with a single H200, Nemotron 3 Nano provides 3.3x higher inference throughput than Qwen3–30B-A3B and 2.2x higher throughput than GPT-OSS-20B.[³]

That matters because throughput directly affects:

user experience,
queueing delay,
infrastructure cost,
concurrency,
and how many requests your deployment can serve.

For production AI teams, a model that is both capable and efficient can be more valuable than a larger model that is harder to serve.

Why deploy Nemotron 3 Nano on HexGrid.cloud?

You can download open model weights yourself. You can run vLLM or SGLang yourself. You can provision GPUs yourself.

But then you own the entire infrastructure path.

That includes:

GPU provisioning,
runtime configuration,
model loading,
object storage,
HTTPS endpoint setup,
authentication,
logs,
usage tracking,
scaling,
billing visibility,
and production reliability.

HexGrid.cloud removes that operational overhead.

With HexGrid.cloud, Nemotron 3 Nano can be deployed as a private production endpoint on dedicated GPU infrastructure. Your app calls the model through the same OpenAI-compatible interface your existing tools already understand.

A typical request looks like this:

curl https://api.hexgrid.cloud/v1/chat/completions \ -H "Authorization: Bearer $HEXGRID_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "nemotron-3-nano-30b-a3b", "messages": [ { "role": "user", "content": "Explain why hybrid MoE models are useful for efficient reasoning." } ] }'

Same interface.
Private runtime.
Dedicated GPU deployment.

Final thoughts

Nemotron 3 Nano 30B-A3B is one of the most interesting open models for teams that want strong reasoning without giving up deployment efficiency.

It brings together:

30B-class model capacity,
roughly 3B-class active-parameter efficiency,
reasoning and non-reasoning modes,
strong coding and math benchmark results,
competitive agentic performance,
long-context support up to 1M tokens,
and a practical path for production inference.

If you are building agents, RAG systems, coding assistants, enterprise chat, or private AI applications, Nemotron 3 Nano is ready to test on HexGrid.cloud today.

Footnotes

[¹]: NVIDIA Developer — NVIDIA Nemotron: https://developer.nvidia.com/topics/ai/nemotron

[²]: Hugging Face — NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 model card: https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16

[³]: NVIDIA Research — Nemotron 3 Family of Models: https://research.nvidia.com/labs/nemotron/Nemotron-3/

[⁴]: NVIDIA NIM model card — nemotron-3-nano-30b-a3b: https://build.nvidia.com/nvidia/nemotron-3-nano-30b-a3b/modelcard

[⁵]: arXiv — “Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning”: https://arxiv.org/abs/2512.20848

[⁶]: Hugging Face — NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 model card: https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8

[⁷]: HexGrid.cloud homepage: https://hexgrid.cloud/