Gemma-4 31B + vLLM + RTX 6000 PRO : 1168 tokens/sec and still asking for more...

#ai #llm #gemma #nvidia

We pushed Gemma-4 31B to 24 concurrent requests on a single RTX 6000 PRO Blackwell. The queue never filled. ~1.17k tokens/sec, and it still had headroom.

Most LLM "benchmarks" show you one request at a time. That tells you almost nothing about production.

So we ran Gemma-4 31B (FP8) on vLLM under a real ShareGPT workload, ramping concurrency 12 → 16 → 20 → 24, and watched what actually happens.

The numbers that mattered:

→ Peak throughput: ~1,168 tokens/sec total (~548 tok/s output)

→ Median time-to-first-token: ~0.7s — snappy even under load

→ Queue depth: averaged 0.41, peaked at just 3 while 14–21 requests ran concurrently

→ Server stayed unsaturated across the entire sweep

The one thing to watch:

Tail TTFT.
Median first-token stays fast, but p99 climbs to ~19s at the heaviest concurrency. That's the first metric to flex as you push higher — not throughput, not the queue.

Setup:

1× RTX 6000 PRO Blackwell (96GB)
Gemma-4 31B-it, FP8 checkpoint
vLLM 0.20 — prefix caching + chunked prefill on
ShareGPT workload, 1024 max output tokens, streaming ON
Max model length (context) : 4096

Verdict:

A single Blackwell card runs a 31B model at 24-way concurrency without breaking a sweat. The high end-to-end latency is just long generations, not queuing — and there's clearly room to climb past 24.