DEV Community: yyyysa4

Same GPU, Same Model, Different Cost per Token: Benchmark Qwen3.8-27B on Your Own Prompts

yyyysa4 — Mon, 02 Nov 2026 02:14:31 +0000

When I'm deciding whether to rent a GPU for an open model, I work out the cost per million output tokens:

cost per 1M output tokens = hourly price ÷ (output tokens/s × 3,600) × 1,000,000

The hourly price is easy to look up. Tokens per second is the number that's easy to get wrong, because it depends on what you send the model.

What the published numbers show

For Qwen3.8-27B on a single RTX PRO 6000 Blackwell (96 GB), the most detailed public numbers I found are in EcoHash's speculative-decoding benchmark. They served the FP8 checkpoint on vLLM 0.27.1 with the model's built-in multi-token-prediction (MTP) head switched on, which is a form of speculative decoding. Then they swept from 1 to 128 concurrent requests, with 150-token replies.

The headline number isn't the interesting part. What caught my eye is that they ran the same sweep with different prompt sets, and the results don't match.

Their latency budget was a 95th-percentile time to first token of 1.5 s or less, and a 95th-percentile time per output token of 30 ms or less. Here's the highest concurrency that stayed inside it:

Prompt set	Highest concurrency in budget	Output tokens/s there	Cost per 1M output tokens at $1.89/hr
Chat (MT-Bench-style)	32	1,274	$0.41
Math (GSM8K-style)	48	1,862	$0.28
Mixed pool, through their API gateway	64	1,954	$0.27

Line chart of 95th-percentile time per token against concurrency for Qwen3.8-27B on one RTX PRO 6000. Chat prompts cross the 30 ms budget after 32 concurrent requests, math prompts after 48.

The card, the weights and the flags were the same in every run. The prompts alone moved the cost per token by about 1.5×.

The reason is the draft head. MTP guesses a few tokens ahead, and the model checks the guesses in one pass. Predictable output like arithmetic gets more guesses accepted, so each step yields more tokens. Open-ended chat gets fewer. The gap shows up even with a single request: 10.5 ms per token on math against 15.0 ms on chat.

Two more things will pull your numbers away from theirs. Qwen3.8-27B reasons at length, so real replies usually run well past 150 tokens. And long prompts add prefill time.

So before I trust anyone's tokens per second, these included, I'd measure my own.

The plan

Rent one RTX PRO 6000 and confirm it's the right card.
Start vLLM with a known-good config.
Replay your own prompts at increasing concurrency.
Take the highest level inside your latency budget and turn it into a cost per 1M tokens.

1. Rent the card and check it

Any provider with the RTX PRO 6000 Blackwell will do. I priced it with EcoHash's $1.89 per GPU-hour, billed per second. Once the machine boots, run:

nvidia-smi --query-gpu=name,memory.total,compute_cap --format=csv

You want to see "RTX PRO 6000 Blackwell", close to 96 GB of memory, and compute capability 12.0.

2. Start the server

This is the config from EcoHash's deployment guide for this model, and it's the setup behind the numbers above:

vllm serve /path/to/Qwen3.8-27B-FP8 \
  --served-model-name qwen3.8-27b \
  --dtype bfloat16 \
  --tensor-parallel-size 1 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 128 \
  --enable-prefix-caching \
  --enable-prompt-tokens-details \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --safetensors-load-strategy prefetch \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

A few of these flags matter more than they look, and the guide explains why. In short: use bfloat16, not float16. Don't add --quantization, because the checkpoint is already FP8. And use the qwen3_xml tool parser. --max-model-len 32768 is a choice rather than a limit of the model, so raise it if your prompts are longer. Longer requests take more cache each, so expect a lower ceiling. The benchmark will show you how much lower.

3. Collect your prompts

Pull 20 to 50 real prompts from your logs into prompts.jsonl, one per line. Each line can be {"prompt": "..."} or a full {"messages": [...]} conversation that includes your system prompt. Set --max-tokens close to your real reply length. For a reasoning model, that's probably well above 150.

4. Run the script

"""Measure an OpenAI-compatible endpoint at several concurrency levels,
then turn the results into a cost per 1M output tokens.

    python bench.py --base-url http://localhost:8000/v1 --model qwen3.8-27b \
        --prompts prompts.jsonl --levels 1,8,16,32,48,64 --max-tokens 512 --price 1.89
"""
import argparse, asyncio, json, math, os, time
from openai import AsyncOpenAI


def p95(xs):
    xs = sorted(xs)
    return xs[math.ceil(0.95 * len(xs)) - 1] if xs else float("nan")


def has_text(delta):
    # Reasoning models stream their thinking in a separate field before the answer.
    extra = delta.model_extra or {}
    return bool(delta.content or extra.get("reasoning_content") or extra.get("reasoning"))


async def one_request(client, model, messages, max_tokens):
    t0 = time.perf_counter()
    first, tokens = None, 0
    stream = await client.chat.completions.create(
        model=model, messages=messages, max_tokens=max_tokens, temperature=0.5,
        stream=True, stream_options={"include_usage": True},
    )
    async for chunk in stream:
        if first is None and chunk.choices and has_text(chunk.choices[0].delta):
            first = time.perf_counter()
        if chunk.usage:
            tokens = chunk.usage.completion_tokens
    end = time.perf_counter()
    if first is None or tokens < 2:
        return None
    # Divide by tokens, not chunks: with speculative decoding one chunk can carry several.
    return first - t0, (end - first) / (tokens - 1), tokens


async def run_level(client, args, prompts, conc):
    for i in range(args.warmup):
        await one_request(client, args.model, prompts[i % len(prompts)], args.max_tokens)
    queue = asyncio.Queue()
    for i in range(max(24, 3 * conc)):
        queue.put_nowait(prompts[i % len(prompts)])
    results, errors = [], 0

    async def worker():
        nonlocal errors
        while True:
            try:
                messages = queue.get_nowait()
            except asyncio.QueueEmpty:
                return
            try:
                r = await one_request(client, args.model, messages, args.max_tokens)
                if r:
                    results.append(r)
                else:
                    errors += 1
            except Exception as e:
                errors += 1
                print(f"  request failed: {e!r}")

    t0 = time.perf_counter()
    await asyncio.gather(*(worker() for _ in range(conc)))
    wall = time.perf_counter() - t0
    tok_s = sum(r[2] for r in results) / wall
    return {
        "conc": conc, "ok": len(results), "errors": errors, "tok_s": tok_s,
        "ttft_p95_ms": p95([r[0] for r in results]) * 1000,
        "tpot_p95_ms": p95([r[1] for r in results]) * 1000,
        "usd_per_1m": args.price / (tok_s * 3600) * 1e6 if tok_s else float("nan"),
    }


def load_prompts(path):
    prompts = []
    with open(path) as f:
        for line in f:
            if line.strip():
                row = json.loads(line)
                prompts.append(row["messages"] if "messages" in row
                               else [{"role": "user", "content": row["prompt"]}])
    return prompts


async def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--base-url", default="http://localhost:8000/v1")
    ap.add_argument("--model", default="qwen3.8-27b")
    ap.add_argument("--prompts", required=True, help="JSONL: {\"prompt\": ...} or {\"messages\": [...]}")
    ap.add_argument("--levels", default="1,8,16,32,48,64")
    ap.add_argument("--max-tokens", type=int, default=512)
    ap.add_argument("--warmup", type=int, default=2)
    ap.add_argument("--price", type=float, default=1.89, help="GPU price, $/hour")
    ap.add_argument("--ttft-budget", type=float, default=1500, help="p95 ms")
    ap.add_argument("--tpot-budget", type=float, default=30, help="p95 ms per token")
    args = ap.parse_args()

    client = AsyncOpenAI(base_url=args.base_url, api_key=os.environ.get("API_KEY", "none"),
                         timeout=600, max_retries=0)
    prompts = load_prompts(args.prompts)
    rows = []
    for conc in map(int, args.levels.split(",")):
        row = await run_level(client, args, prompts, conc)
        row["fits"] = (row["errors"] == 0 and row["ttft_p95_ms"] <= args.ttft_budget
                       and row["tpot_p95_ms"] <= args.tpot_budget)
        rows.append(row)
        print(f"C={conc}: {row['tok_s']:.0f} tok/s, TPOT p95 {row['tpot_p95_ms']:.1f} ms")

    print("\n| conc | ok | errors | output tok/s | TTFT p95 ms | TPOT p95 ms | $ per 1M out | in budget |")
    print("|---|---|---|---|---|---|---|---|")
    for r in rows:
        print(f"| {r['conc']} | {r['ok']} | {r['errors']} | {r['tok_s']:.0f} | {r['ttft_p95_ms']:.0f} "
              f"| {r['tpot_p95_ms']:.1f} | ${r['usd_per_1m']:.2f} | {'yes' if r['fits'] else 'no'} |")
    best = max((r for r in rows if r["fits"]), key=lambda r: r["tok_s"], default=None)
    if best:
        print(f"\nBest level inside budget: C={best['conc']}, {best['tok_s']:.0f} tok/s, "
              f"${best['usd_per_1m']:.2f} per 1M output tokens at ${args.price}/hr")


if __name__ == "__main__":
    asyncio.run(main())

Run it on the GPU instance itself, against localhost:8000, so you're measuring the card and not your network. If you have to go through a public endpoint, put its key in the API_KEY environment variable.

pip install openai
python bench.py --prompts prompts.jsonl --levels 1,8,16,32,48,64 --max-tokens 1024 --price 1.89

A few choices in the script are deliberate:

Time to first token counts reasoning. Qwen3.8-27B streams its thinking in a separate field before the answer. The clock stops at whichever arrives first, because that's when generation has started.
Time per token divides by tokens, not chunks. With speculative decoding, one streamed chunk can carry several tokens. Counting chunks makes the server look slower than it is, a trap EcoHash's write-up also points out.
Each level sends max(24, 3 × concurrency) requests after two warm-ups. That matches the sampling EcoHash describes for its published run, so your numbers line up with theirs. The last few requests at each level run with fewer in flight, so throughput comes out slightly low, which is the safe direction to be wrong in.

Reading the result

The script prints a markdown table and names the best level inside your budget. If your budget is different, change --tpot-budget and --ttft-budget. For scale, 30 ms per token is about 33 tokens per second for each stream.

Then compare that cost per 1M with what an API would charge you for the same model. EcoHash's own API lists Qwen3.8-27B at $0.60 per 1M output tokens, and the 21 providers on OpenRouter ranged from $1.25 to $3.20 when I checked on November 1. If the card can't beat the API you'd otherwise use at a realistic load, renting it is about control, such as a longer context or your own serving flags, not about price.

Keep in mind that the cost at the best level assumes the card stays that busy every hour you pay for it. Real traffic doesn't. Scale the number by your average load, not your peak.

What it costs to find out

At $1.89 an hour, billed per second, a 30-minute session costs about 95 cents, and EcoHash's $1 signup credit covers roughly that. Terminate the instance when you're done, because the clock keeps running while it sits idle.

Limits

This measures throughput and latency only. It says nothing about answer quality, so test that separately.
The published numbers come from the Server Edition of the card. The workstation editions have more memory bandwidth, so expect somewhat different results.
vLLM changes quickly. Record your version alongside your results.

There Are Four GPUs Called RTX 6000. Here's How I Make Sure I Rent the 96 GB One

yyyysa4 — Tue, 20 Oct 2026 02:48:40 +0000

Search a GPU marketplace for "RTX 6000" and you'll get a mix of four different cards. They share a name and not much else. The one you probably want for bigger open models, the 96 GB Blackwell card, is easy to confuse with the other three.

These days I treat "RTX 6000" in a listing the way I treat "large" on a coffee menu: it's a hint, not a spec. Here's what I check now.

Four cards, one name

Card	Architecture	Year	Memory	Bandwidth
Quadro RTX 6000	Turing	2018	24 GB GDDR6	672 GB/s
RTX A6000	Ampere	2020	48 GB GDDR6	768 GB/s
RTX 6000 Ada Generation	Ada Lovelace	2022	48 GB GDDR6	960 GB/s
RTX PRO 6000 Blackwell	Blackwell	2025	96 GB GDDR7	1,597–1,792 GB/s

The Ada card is the main source of confusion. NVIDIA first sold it as just "RTX 6000" and only later added "Ada Generation", so for a while it had exactly the same name as the 2018 Quadro.

Is "RTX 6000 Blackwell" the same as "RTX PRO 6000"?

Yes. "RTX 6000 Blackwell" is shorthand for the RTX PRO 6000 Blackwell, the 96 GB GDDR7 card NVIDIA announced at GTC in March 2025. If a listing says Blackwell and 96 GB, you're in the right family.

The family has three editions, though, and they aren't identical:

Workstation Edition: 600 W, flow-through cooler, 1,792 GB/s
Max-Q Workstation Edition: 300 W, built for packing several cards into one workstation, also 1,792 GB/s
Server Edition: 400–600 W, passively cooled for multi-GPU servers, 1,597 GB/s

All three have 96 GB, so the memory size won't tell you which edition you're on. The bandwidth difference is worth knowing when you read benchmarks. Token generation for a single user is mostly limited by memory bandwidth, so a review run on the Workstation Edition can come out up to about 12% faster than a Server Edition would. If a provider runs its cards in servers, ask which edition you're getting.

Price is a tell

On the GPU price tracker getdeploying, the median on-demand price this week is about $1.03 per GPU-hour for the RTX 6000 Ada and about $2.24 for the RTX PRO 6000. So if a listing just says "RTX 6000" and costs about a dollar an hour, my first guess is the 48 GB Ada card, not a bargain on the Blackwell one. Cheaper PRO 6000 listings do exist, so check the specs before you assume either way.

What I check before and right after renting

Read the full name and the memory size. 96 GB GDDR7 is the Blackwell card. 48 GB is an A6000 or a 6000 Ada, whatever the listing calls it. 24 GB is the old Quadro.
Ask which edition it is, especially if you plan to compare against published benchmarks.
Run one command as soon as the machine boots:

nvidia-smi --query-gpu=name,memory.total,compute_cap --format=csv

The name should say RTX PRO 6000 Blackwell, the total memory should be close to 96 GB, and the compute capability should be 12.0. An A6000 reports 8.6 and a 6000 Ada reports 8.9, so the output settles any remaining doubt.

That last check costs almost nothing on a provider that bills by the second. One provider that makes this easy is EcoHash. Its RTX Pro 6000 page says plainly that the card isn't an RTX A6000 or RTX 6000 Ada but the Blackwell Server Edition with 96 GB per GPU. It charges $1.89 per GPU-hour, billed per second, and its $1 signup credit covers about half an hour on one card, which is plenty to boot it and run the command above.

The short version

"RTX 6000" on its own doesn't tell you which card you're getting. "RTX PRO 6000 Blackwell, 96 GB" does.
A 48 GB card isn't the Blackwell one, whatever the listing says.
Ask which edition you're on, and trust nvidia-smi over the listing.

Sources: NVIDIA's product pages for the RTX PRO 6000 family, the Max-Q edition and the RTX 6000 Ada; the Server Edition datasheet; GPUperHour for the older cards; and getdeploying's RTX 6000 Ada and RTX PRO 6000 pages, checked October 20, 2026.

RTX PRO 6000 vs H100 for LLM Inference: Which Is More Cost-Effective in 2026?

yyyysa4 — Wed, 07 Oct 2026 03:19:48 +0000

The NVIDIA H100 is the default answer to "what GPU should I serve this model on?" The RTX PRO 6000 Blackwell costs about two-thirds as much per hour, has more memory, and less than half the memory bandwidth. Which of those facts wins depends on one question that most comparisons skip: does your model fit on a single card?

This post works through that question using independent benchmarks, current market prices, and arithmetic you can check. No new measurements were taken; sources and a reproduction script are at the end.

TL;DR

Single-GPU models: the RTX PRO 6000 is cheaper per token — about 30–40% cheaper than an H100 at October 2026 median on-demand prices.
Multi-GPU models: the advantage shrinks and then reverses. Roughly 8% cheaper at 4-way tensor parallelism, roughly 8% more expensive at 8-way. The RTX PRO 6000 has no NVLink.
The H100 has 2.1× the memory bandwidth. The RTX PRO 6000 has 16 GB more memory, which turns into 1.8–2.6× more KV-cache room for 30B–70B models.
Rule of thumb: cost-per-token ratio = price ratio × throughput ratio. Today's price ratio is about 0.63, so the RTX PRO 6000 wins whenever an H100 is less than ~1.6× faster on your workload.

The specs that matter for inference

	RTX PRO 6000 Blackwell Server Edition	H100 SXM
Memory	96 GB GDDR7	80 GB HBM3
Memory bandwidth	1,597 GB/s	3,350 GB/s
Multi-GPU interconnect	PCIe Gen 5 only — no NVLink	NVLink 900 GB/s, plus PCIe Gen 5
Lowest native tensor precision	FP4	FP8
MIG partitions	up to 4	up to 7
Max power	up to 600 W	up to 700 W
On-demand price, median, Oct 2026	$2.20 / GPU-hr (53 providers)	$3.49 / GPU-hr (57 providers)

Two notes on that table.

Peak TFLOPS are left out on purpose. NVIDIA's product pages quote them with different sparsity conventions, so putting them side by side invites a wrong conclusion. For LLM serving, where the decode phase is usually limited by how fast weights and KV cache can be read, memory bandwidth is the more predictive spec.

"RTX PRO 6000" also covers more than one card. The Workstation Edition lists 1,792 GB/s of bandwidth; the Server Edition most clouds rent lists 1,597 GB/s. That difference shows up in the benchmarks below.

What independent benchmarks show

The most useful public data comes from two vLLM benchmark runs published by CloudRift. Both compare the RTX PRO 6000 and the H100 directly, on the same models and settings.

Run A — November 2025. Single GPU, GLM-4.5-Air (AWQ 4-bit), 1,000 input / 1,000 output tokens, 256–512 concurrent requests, RTX PRO 6000 Workstation Edition.

GPU	Total throughput
RTX PRO 6000 (Workstation)	3,140 tok/s
H100 SXM	2,987 tok/s

Run B — January 2026. Google Cloud 8-GPU nodes (G4 with RTX PRO 6000, A3 High with H100), 8,000 input / 8,000 output tokens.

Workload	RTX PRO 6000	H100	H100 advantage
GLM-4.5-Air AWQ, one model per GPU	2,290.69 tok/s	2,556.03 tok/s	+11.6%
Qwen3-Coder-480B-A35B AWQ, 4-GPU tensor parallel	1,602.96 tok/s	2,328.63 tok/s	+45.3%
GLM-4.6 FP8, 8-GPU tensor parallel	1,651.67 tok/s	2,833.77 tok/s	+71.5%

Both machines are 8-GPU nodes running identical configurations, so the ratios hold however the throughput was aggregated across the node.

The two runs disagree on the single-GPU case: the RTX PRO 6000 is 5% ahead in Run A, while the H100 is 12% ahead in Run B. They differ in three ways at once — card edition, sequence length, and host — so neither result is wrong. The most plausible reading is that Run A's Workstation card had about 12% more bandwidth than a typical cloud Server Edition, and Run B's 8K-token sequences make each decoded token read much more KV cache, which favours the H100's bandwidth.

The multi-GPU rows are not ambiguous. As tensor parallelism widens, the H100's NVLink pulls away: +45% at 4-way, +72% at 8-way.

Cost per million tokens

Throughput alone does not answer a cost question. Combining it with price:

cost per token (RTX / H100) = (RTX $/hr ÷ H100 $/hr) × (H100 tok/s ÷ RTX tok/s)

At median on-demand prices ($2.20 vs $3.49, price ratio 0.63):

Scenario	H100 throughput vs RTX PRO 6000	RTX PRO 6000 cost per token	RTX break-even price
1 GPU, 1K/1K (Run A)	0.95×	40% cheaper	$3.67/hr
1 GPU, 8K/8K (Run B)	1.12×	30% cheaper	$3.13/hr
4-GPU tensor parallel (Run B)	1.45×	8% cheaper	$2.40/hr
8-GPU tensor parallel (Run B)	1.72×	8% more expensive	$2.03/hr

The break-even column is the RTX PRO 6000 hourly price at which it matches an H100 at $3.49/hr. Rent one below that price and it is the cheaper option for that workload.

In absolute terms, Run A works out to about $0.195 per million tokens on the RTX PRO 6000 and $0.325 on the H100, counting input and output tokens together.

CloudRift's own cost tables differ from these. Each run used a different price basis — Runpod list prices in November 2025, and an estimated cost of ownership in January 2026. The table above re-prices both runs with one consistent, current source, so the scenarios are comparable with each other.

What the extra 16 GB actually buys

Twenty percent more memory sounds marginal. It is not, because every extra gigabyte lands after the weights are loaded, in the space left for KV cache. For models that nearly fill an 80 GB card, that space roughly doubles.

KV cache per token for a standard attention model:

2 (K and V) × layers × KV heads × head dim × bytes per element

With an FP8 KV cache and 4 GB reserved for the runtime:

Model	Weights	KV per token	Free for KV on H100 80 GB	Free for KV on RTX PRO 6000	32K sequences: H100 → RTX
Qwen3-Coder-30B-A3B (BF16)	56.8 GB	48 KB	19.2 GB	35.2 GB	12 → 23
Qwen3-32B (BF16)	61.1 GB	128 KB	14.9 GB	30.9 GB	3 → 7
Llama-3.3-70B (FP8)	65.8 GB	160 KB	10.2 GB	26.2 GB	2 → 5
Llama-3.3-70B (BF16)	131.5 GB	160 KB	does not fit	does not fit	—

A 70B model at FP8 is the clearest case. On an H100 it fits, but leaves room for only two concurrent 32K-token conversations. On an RTX PRO 6000, five. For long-context, RAG-heavy, or agentic workloads, that headroom decides how much traffic a single card can take.

The last row matters just as much. A 70B model at BF16 needs two cards on either GPU — and once two cards have to talk to each other, the interconnect comes back into play, and with it the H100's advantage.

When the H100 is the better buy

Your model needs four or more GPUs. Once weights plus KV cache outgrow two cards (roughly 160–190 GB), you are into 4-way tensor parallelism or wider, where NVLink beats PCIe by a margin that outweighs the price gap.
Single-stream decode speed is the product. If latency for one user matters more than total throughput, 2.1× the memory bandwidth is hard to argue with.
You are serving very long sequences on a model that barely fits. Run B suggests the bandwidth gap grows with sequence length.
You need many small isolated tenants. MIG gives you up to seven partitions on an H100 versus four on an RTX PRO 6000.

When the RTX PRO 6000 is the better buy

The model and its KV cache fit on one 96 GB card. That covers most 7B–35B models at BF16 and 70B-class models at FP8 or below.
You scale out with replicas rather than tensor parallelism. Data-parallel serving never touches the interconnect, so the PCIe limitation does not apply.
Concurrency on long contexts is the constraint. The KV-cache table above is the whole argument.
You want to serve FP4-quantized models. FP4 tensor cores are a Blackwell feature; Hopper stops at FP8.

A short decision checklist

Add up weights and the KV cache your target concurrency needs, at the precision you plan to serve.
If the total fits in 96 GB minus a few GB of overhead, the RTX PRO 6000 is very likely cheaper per token. Benchmark it against your real prompt lengths.
If you need 2 GPUs, test both — the result could go either way.
If you need 4 or more GPUs, start from the H100 (or H200), unless your RTX PRO 6000 rate is below the break-even price for your workload.
Re-run the cost formula with the prices you are actually quoted. Market medians move every month.

FAQ

Is the RTX PRO 6000 faster than the H100 for LLM inference?
Not usually per card. In independent vLLM tests the H100 was between 5% slower and 12% faster on a single-GPU model, and 45–72% faster once tensor parallelism across 4–8 GPUs was involved. The RTX PRO 6000's case is cost per token, not raw speed.

How much does it cost to rent an RTX PRO 6000 versus an H100?
As of October 7, 2026, the median on-demand price across tracked providers was $2.20 per GPU-hour for the RTX PRO 6000 and $3.49 for the H100, per getdeploying.com.

What is the RTX PRO 6000's memory bandwidth?
1,597 GB/s for the Server Edition and 1,792 GB/s for the Workstation Edition, both with 96 GB of GDDR7. The H100 SXM has 3,350 GB/s of HBM3.

How many tokens per second can an RTX PRO 6000 serve?
It depends heavily on model, precision and concurrency. One public data point: about 3,140 total tokens/s on GLM-4.5-Air (AWQ 4-bit) at 256–512 concurrent requests, 1K input / 1K output, on a Workstation Edition card.

Is the "RTX 6000 Blackwell" the same GPU as the "RTX PRO 6000"?
Yes — both names refer to the 96 GB Blackwell card, sold in Workstation, Max-Q and Server editions. It is a different card from the RTX 6000 Ada (48 GB) and the older Quadro RTX 6000 (24 GB).

Method and caveats

Nothing here was measured by the author. Throughput comes from CloudRift's published benchmarks, prices from getdeploying.com's public listings, specs from NVIDIA's product pages, and model shapes from each model's config.json.
Prices are medians. Your actual price determines the answer, which is why the formula and break-even prices are included.
Throughput ratios are workload-specific. The benchmarks used GLM and Qwen3-Coder models; a different model, quantization, or serving stack will shift them.
Run A likely flatters the RTX PRO 6000 slightly, because it used the Workstation Edition, with about 12% more bandwidth than the Server Edition most clouds rent.
Units: in the VRAM table, GB means GiB (2³⁰ bytes), the unit nvidia-smi reports.

Reproduce the numbers

GiB = 1024**3

# Prices: getdeploying.com on-demand medians, 2026-10-07
RTX_HR, H100_HR = 2.20, 3.49

# (rtx tok/s, h100 tok/s) from CloudRift's published runs
RUNS = {
    "1 GPU, 1K/1K":  (3140.00, 2987.00),
    "1 GPU, 8K/8K":  (2290.69, 2556.03),
    "4-GPU TP":      (1602.96, 2328.63),
    "8-GPU TP":      (1651.67, 2833.77),
}
for name, (rtx, h100) in RUNS.items():
    rel = (RTX_HR / H100_HR) * (h100 / rtx)
    print(f"{name:<14} RTX cost/token = {rel:.0%} of H100 | break-even ${H100_HR / (h100 / rtx):.2f}/hr")

# KV-cache headroom: FP8 KV cache, 4 GiB runtime reserve, 32K-token sequences
MODELS = [  # name, params (B), bytes/param, layers, kv_heads, head_dim
    ("Qwen3-Coder-30B-A3B BF16", 30.5, 2, 48, 4, 128),
    ("Qwen3-32B BF16",           32.8, 2, 64, 8, 128),
    ("Llama-3.3-70B FP8",        70.6, 1, 80, 8, 128),
]
for name, p, wb, L, kvh, hd in MODELS:
    weights = p * 1e9 * wb / GiB
    kv_seq = 2 * L * kvh * hd * 1 * 32_768 / GiB
    fits = {cap: int((cap - weights - 4) / kv_seq) for cap in (80, 96)}
    print(f"{name:<26} H100: {fits[80]:>2} seqs | RTX PRO 6000: {fits[96]:>2} seqs")

Sources

NVIDIA, RTX PRO 6000 Blackwell Server Edition and Workstation Edition datasheet
NVIDIA, H100 Tensor Core GPU
CloudRift, RTX PRO 6000 vs H100, H200, and L40S: LLM Inference (Nov 2025)
CloudRift, Benchmarking LLM Inference on B200, H200, H100, and RTX PRO 6000 (Jan 2026)
getdeploying.com, RTX PRO 6000 rental prices and H100 rental prices (checked 2026-10-07)
Model configs: Qwen3-Coder-30B-A3B-Instruct, Qwen3-32B, Llama-3.3-70B-Instruct

Parameter count is a bad way to pick an open model for one GPU

yyyysa4 — Thu, 24 Sep 2026 08:34:54 +0000

Model selection usually starts with a number: 8B, 27B, 30B. It is the first thing on the model card and the first thing in every comparison thread. It is also close to useless for the question most teams are actually asking, which is how much traffic will this serve on the card I have.

Below is that question worked out for four current open models on a single 96 GB GPU. Everything here is arithmetic on published configs — no benchmarks were run. The script is at the end and takes about a second.

Units: GB throughout means GiB (2³⁰ bytes), which is what nvidia-smi reports.

The four models

Model	Total params	Architecture	Native context	License
Qwen3-8B	8.2B	Dense, 36 layers, GQA 32Q/8KV, head dim 128	32,768 (131K with YaRN)	Apache-2.0
Qwen3.5-9B	~9B	Hybrid: 3 linear-attention layers per 1 full, 32 total, 4 KV heads, head dim 256	262,144 (1.01M with YaRN)	Apache-2.0
Gemma-4-26B-A4B-it	25.2B (3.8B active)	MoE, 128 experts, top-8 + 1 shared, 30 layers, 5 full-attention + 25 sliding (window 1024), 8 KV heads, head dim 256	262,144	Apache-2.0
Qwen3.8-27B	~27B	Hybrid: same 3:1 pattern, 64 layers, 4 KV heads, head dim 256	262,144 (1M with YaRN)	Apache-2.0

Every figure in that table comes from the model card or config.json of the repo it links to.

Weights: where MoE stops helping

Mixture-of-Experts models are usually introduced with their active parameter count, because that is what determines compute per token. Gemma-4-26B-A4B activates 3.8B parameters out of 25.2B.

Memory does not work that way. Routing is per-token and unpredictable, so every expert has to be resident. At BF16:

Qwen3-8B — 15.3 GB
Qwen3.5-9B — 16.8 GB
Gemma-4-26B-A4B — 46.9 GB
Qwen3.8-27B — 50.3 GB

The "4B active" model costs nearly three times the VRAM of the 9B. What you buy for that is compute: it runs at roughly 4B-model speed per token while holding 26B-model knowledge. That is a real and valuable trade. It is just not a memory saving, and model cards that lead with the active count invite exactly that misreading.

KV cache: the term nobody budgets

Weights are static. KV cache grows with every token in flight, and it is where single-GPU deployments actually fall over. For a standard attention layer:

bytes per token = 2 (K and V) × layers × kv_heads × head_dim × dtype_bytes

The variable that matters most is layers — specifically, how many layers keep a cache that grows with sequence length. Three of these four models do something to reduce that count:

Qwen3-8B is conventional. All 36 layers cache. 2 × 36 × 8 × 128 × 2 = 144 KB per token.
Qwen3.5-9B interleaves 3 linear-attention layers for every 1 full-attention layer. Linear attention carries a fixed-size recurrent state instead of a growing cache, so only 8 of 32 layers contribute: 2 × 8 × 4 × 256 × 2 = 32 KB per token.
Qwen3.8-27B uses the same 3:1 pattern over 64 layers, so 16 layers cache: 64 KB per token.
Gemma-4-26B-A4B splits differently: 5 full-attention layers grow with the sequence (40 KB per token), while the other 25 use a 1024-token sliding window and stay flat at about 200 MB per sequence no matter how long the context gets.

So an 8B dense model spends 4.5× more KV memory per token than a 9B hybrid. Parameter count predicted the opposite.

What that buys in concurrency

Take 96 GB, subtract weights, reserve 4 GB for activations and framework overhead, and divide by the KV cost of one 32,768-token sequence:

Model	KV per 32K sequence	Concurrent 32K sequences
Qwen3-8B	4.50 GB	17
Qwen3.5-9B	1.00 GB	75
Gemma-4-26B-A4B	1.45 GB	31
Qwen3.8-27B	2.00 GB	20

The ranking has almost nothing to do with size. Qwen3.5-9B — nominally the second-smallest model here — serves four times the concurrent long-context traffic of Qwen3-8B, and nearly four times that of the 27B. The 26B MoE, despite spending 47 GB on weights, still fits more concurrent sequences than the 8B dense model does.

If your workload is RAG, document processing, or anything else that puts long prompts in flight, this table is a better predictor of cost per request than any leaderboard.

Why there is no quality column here

There is an obvious missing dimension, and it is missing on purpose.

The four model cards report largely disjoint benchmark sets. Gemma-4-26B-A4B-it reports MMLU-Pro 82.6%, GPQA Diamond 82.3%, LiveCodeBench v6 77.1%. Qwen3.5-9B reports MMLU-Pro 82.5 and C-Eval 88.2. Qwen3.8-27B reports GPQA Diamond 89.2 and LiveCodeBench v6 90.3. Qwen3-8B's card reports neither.

Two of those MMLU-Pro numbers are within 0.1 of each other, which is exactly the kind of coincidence that produces a confident and wrong blog post. Vendor-run evaluations differ in prompt format, few-shot count, decoding parameters, and whether reasoning mode was enabled — none of which are consistently disclosed. Putting them in one table implies a comparison that the underlying numbers do not support.

The honest version: use these figures to shortlist, then run your own eval on your own prompts. Memory math transfers between deployments. Benchmark scores often do not.

What this analysis does not tell you

Nothing about throughput. Memory determines what fits. Tokens per second depends on memory bandwidth, kernel quality, batching policy, and the serving stack. A model that fits 75 sequences will not necessarily serve them fast.
Quantization changes every number. FP8 weights roughly halve the weight column; FP8 or INT8 KV cache halves the cache column. The relative ordering mostly survives, the absolute headroom does not.
Linear attention is a trade, not a free win. The architectures that cut KV cost do so by compressing history into a fixed-size state. That is very good for throughput and is not free on tasks that need precise long-range recall. Test it rather than assuming the memory win is costless.
Real allocators are messier. vLLM and SGLang preallocate a KV pool via gpu_memory_utilization; fragmentation, CUDA graphs, and multimodal encoders all take a cut. Treat the concurrency numbers as ceilings.
Parameter counts are approximate. Qwen publishes "9B" and "27B" rather than exact totals, so those weight figures carry a percent or two of slack.

Reproduce it

GiB = 1024**3
BYTES = 2  # BF16 KV cache

# name, params(B), full_attn_layers, sliding_layers, window, kv_heads, head_dim
MODELS = [
    ("Qwen3-8B",         8.2, 36,  0,    0, 8, 128),
    ("Qwen3.5-9B",       9.0,  8,  0,    0, 4, 256),
    ("Gemma-4-26B-A4B", 25.2,  5, 25, 1024, 8, 256),
    ("Qwen3.8-27B",     27.0, 16,  0,    0, 4, 256),
]

def kv_for_seq(full_l, sl_l, win, kvh, hd, seq):
    growing = 2 * full_l * kvh * hd * BYTES * seq
    capped  = 2 * sl_l   * kvh * hd * BYTES * min(seq, win)
    return growing + capped

VRAM, RESERVE, SEQ = 96, 4, 32_768
for name, p, full_l, sl_l, win, kvh, hd in MODELS:
    weights = p * 1e9 * BYTES / GiB
    per_seq = kv_for_seq(full_l, sl_l, win, kvh, hd, SEQ) / GiB
    free    = VRAM - weights - RESERVE
    print(f"{name:<18} weights {weights:5.1f} GB | "
          f"{per_seq:4.2f} GB/seq | {int(free/per_seq):3d} concurrent {SEQ//1024}K seqs")

Change SEQ to your real context length and BYTES to 1 if you serve an FP8 KV cache. The layer counts come straight from each repo's config.json — if a model ships a new revision, re-read it rather than trusting this table.

Best models to run on RTX Pro 6000 96GB

yyyysa4 — Thu, 24 Sep 2026 03:43:33 +0000

The right model depends on the task, not on the GPU. A 96 GB RTX Pro 6000 holds a 20B to 35B open model with room for context and batching, so you get to pick by the job instead of by a memory ceiling. For coding and agents that means Qwen3 Coder 30B; for low-cost chat, gpt-oss 20B; for retrieval, an embedding model with a reranker; for voice, Whisper and Kokoro. All of them run through the same OpenAI-compatible API at https://api.ecohash.com/v1, so you start on the shared API and move to a dedicated endpoint or a workspace later without touching your code.

Pick by task, not by memory. Coding: qwen3-coder-30b-a3b-instruct. Chat: gpt-oss-20b. Retrieval: jina-embeddings-v3 + bge-reranker-v2-m3. Voice: whisper-large-v3-turbo + kokoro-82m.

Model by task

Task	Model	Model ID
Coding, agents, tool use	Qwen3 Coder 30B	`qwen3-coder-30b-a3b-instruct`
General chat, low cost	gpt-oss 20B	`gpt-oss-20b`
General chat, higher capability	GLM 5.2, DeepSeek V4 Flash	`GLM-5.2`, `DeepSeek-V4-Flash`
Embeddings for RAG	Jina v3, Qwen3 Embedding	`jina-embeddings-v3`, `qwen3-embedding-8b`
Reranking for RAG	BGE reranker v2 m3	`bge-reranker-v2-m3`
Speech to text	Whisper Large V3 Turbo	`whisper-large-v3-turbo`
Text to speech	Kokoro 82M	`kokoro-82m`

The higher-capability chat models run through the hosted API, not on a single 96 GB card. Here is what the single-card models measure end-to-end, in July 2026:

Task	Model	Measured
Coding	`qwen3-coder-30b-a3b-instruct`	30 ms TTFT, ~110 tok/s, 32k context
General chat	`gpt-oss-20b`	170 ms TTFT, ~125 tok/s
General chat	`llama-3.1-8b-instruct`	205 ms TTFT, ~143 tok/s
Speech to text	`whisper-large-v3-turbo`	WER 4.37, ~0.5 s per 10 s of audio
Text to speech	`kokoro-82m`	first audio ~120 ms

Against other providers of the same open models, we serve them at full BF16 precision and land competitively on price and speed (llama-3.1-8b-instruct, EcoHash in purple):

Full data and charts: ecohash-benchmarks.

Facts


GPU	NVIDIA RTX Pro 6000 Blackwell Server Edition, 96 GB GDDR7
Single-card sweet spot	20B to 35B open models
API base URL	`https://api.ecohash.com/v1`
Access	Shared API, dedicated endpoint, GPU workspace
Price	RTX Pro 6000 at $1.89 per GPU-hour (pricing)

Who this is for

Developers choosing an open model for a coding, chat, retrieval, or voice feature.
Teams deciding what to run on a 96 GB card versus what to call through a hosted API.

What you can decide

Which model to start with for your task.
Whether it fits on a single RTX Pro 6000 or is better used through the hosted API.
When to move from the shared API to a dedicated endpoint or a workspace.

How to choose

Four questions get you most of the way:

What is the task? Code, chat, retrieval, or speech. Start here, not with the hardware.
Does it fit in 96 GB at your context length and batch size? A 20B to 35B model fits with room to spare, models near 70B need quantization, and 235B and up are better through the hosted API.
What are your latency and throughput needs? Steady, high-volume traffic is the reason to reserve a dedicated endpoint.
Do you need to self-host at all? For most teams the shared API is enough, and the GPU is our problem, not yours.

Coding

Qwen3 Coder 30B (qwen3-coder-30b-a3b-instruct) is the default for code generation, refactoring, and tool-using agents. It is a Mixture-of-Experts model with about 30B total parameters, it fits comfortably on one 96 GB card, and it serves a 32,768-token context at about 110 tokens/sec single-stream with a 30 ms time to first token. See the model page.

For lighter or cheaper high-volume coding, gpt-oss 20B (gpt-oss-20b) is the smaller option. The three deployment paths for Qwen3 Coder are covered in Qwen3 Coder 30B on RTX Pro 6000.

General chat

Start with gpt-oss 20B (gpt-oss-20b) for low-cost chat and assistants. It fits on one card, has a long context, and reaches about 8,900 output tokens/sec under load.

For stronger general capability, we also serve larger models such as GLM 5.2 (GLM-5.2) and DeepSeek V4 Flash (DeepSeek-V4-Flash), and very large Mixture-of-Experts models like Qwen3-235B-A22B (Qwen3-235B-A22B). Those are not single-card deployments. Use them through the shared API or a dedicated endpoint and let us handle the hardware.

Retrieval (RAG)

A retrieval pipeline needs two models: an embedding model to index and search text, and a reranker to reorder the top results.

Embeddings: jina-embeddings-v3, jina-embeddings-v4, or the Qwen3 Embedding family (qwen3-embedding-8b, qwen3-embedding-4b, qwen3-embedding-0.6b).
Reranking: bge-reranker-v2-m3.

All of them run through the same key, so one service can embed, rerank, and generate the answer without stitching vendors together. See the RAG use case.

Voice

A voice agent uses three model types: speech to text, a chat model, and text to speech.

Speech to text: whisper-large-v3-turbo or qwen3-asr-1-7b.
Text to speech: kokoro-82m or qwen3-tts.

These speech models are small and do not need an RTX Pro 6000. They run through the API and move onto GPU capacity only when throughput grows. The full build is in Build a voice agent with Whisper, Kokoro, and an OpenAI-compatible API; see also voice and speech, Kokoro 82M, and Whisper Large V3 Turbo.

Calling these models

The API is OpenAI-compatible. A coding call:

from openai import OpenAI

client = OpenAI(base_url="https://api.ecohash.com/v1", api_key="eco_...")

resp = client.chat.completions.create(
    model="qwen3-coder-30b-a3b-instruct",
    messages=[
        {"role": "user", "content": "Refactor this to be iterative:\n\ndef fib(n):\n    return n if n < 2 else fib(n-1) + fib(n-2)"}
    ],
)
print(resp.choices[0].message.content)

An embeddings call for a RAG index:

emb = client.embeddings.create(
    model="jina-embeddings-v3",
    input=["EcoHash serves open models through an OpenAI-compatible API."],
)
print(len(emb.data[0].embedding))

Shared API, dedicated endpoint, or workspace

Shared API: no GPU to manage, pay per token, good for building and variable traffic. See inference.
Dedicated endpoint: reserved capacity for one model, steadier throughput and more predictable latency. See dedicated inference.
GPU workspace: an hourly RTX Pro 6000 for experiments, fine-tuning, and batch jobs. See GPU compute.

Limitations

Model fit depends on quantization, context length, and batch size. The ranges here are starting points to validate, not guarantees.
The higher-capability chat models are for the hosted API, not single-card serving.
Real performance depends on your setup. Reproduce any figure with its method and date.
Model IDs change as the catalog changes. Check the model catalog before you build.

FAQ

What is the best model to run on an RTX Pro 6000 96GB? It depends on the task. Coding: qwen3-coder-30b-a3b-instruct. General chat: gpt-oss-20b. Retrieval: jina-embeddings-v3 with bge-reranker-v2-m3. Speech: whisper-large-v3-turbo with kokoro-82m.

What is the best coding model for the RTX Pro 6000? Qwen3 Coder 30B (qwen3-coder-30b-a3b-instruct). It fits on one 96 GB card and serves a 32,768-token context.

What is a good low-cost general chat model? gpt-oss 20B (gpt-oss-20b). It fits on one card and has a long context.

Can I run a 235B model on one RTX Pro 6000? No. Use a model like Qwen3-235B-A22B through the hosted API instead.

What models do I need for RAG? An embedding model and a reranker, for example jina-embeddings-v3 and bge-reranker-v2-m3, plus a chat model to write the answer.

Do speech models need an RTX Pro 6000? No. Whisper and Kokoro run through the API, and move onto GPU capacity only when throughput demands it.

How many models can I use with one API key? All of them. Every model shares one key and one base URL, https://api.ecohash.com/v1.

How much does this cost? API usage is billed per token; a GPU workspace runs at $1.89 per GPU-hour. See pricing for current rates.

Next steps

Browse all models: ecohash.com/models
See the RTX Pro 6000 page: ecohash.com/gpu-compute
Reserve capacity: ecohash.com/dedicated-inference