DEV Community: yyyysa4

RTX PRO 6000 vs H100 for LLM Inference: Which Is More Cost-Effective in 2026?

yyyysa4 — Wed, 07 Oct 2026 03:19:48 +0000

The NVIDIA H100 is the default answer to "what GPU should I serve this model on?" The RTX PRO 6000 Blackwell costs about two-thirds as much per hour, has more memory, and less than half the memory bandwidth. Which of those facts wins depends on one question that most comparisons skip: does your model fit on a single card?

This post works through that question using independent benchmarks, current market prices, and arithmetic you can check. No new measurements were taken; sources and a reproduction script are at the end.

TL;DR

Single-GPU models: the RTX PRO 6000 is cheaper per token — about 30–40% cheaper than an H100 at October 2026 median on-demand prices.
Multi-GPU models: the advantage shrinks and then reverses. Roughly 8% cheaper at 4-way tensor parallelism, roughly 8% more expensive at 8-way. The RTX PRO 6000 has no NVLink.
The H100 has 2.1× the memory bandwidth. The RTX PRO 6000 has 16 GB more memory, which turns into 1.8–2.6× more KV-cache room for 30B–70B models.
Rule of thumb: cost-per-token ratio = price ratio × throughput ratio. Today's price ratio is about 0.63, so the RTX PRO 6000 wins whenever an H100 is less than ~1.6× faster on your workload.

The specs that matter for inference

	RTX PRO 6000 Blackwell Server Edition	H100 SXM
Memory	96 GB GDDR7	80 GB HBM3
Memory bandwidth	1,597 GB/s	3,350 GB/s
Multi-GPU interconnect	PCIe Gen 5 only — no NVLink	NVLink 900 GB/s, plus PCIe Gen 5
Lowest native tensor precision	FP4	FP8
MIG partitions	up to 4	up to 7
Max power	up to 600 W	up to 700 W
On-demand price, median, Oct 2026	$2.20 / GPU-hr (53 providers)	$3.49 / GPU-hr (57 providers)

Two notes on that table.

Peak TFLOPS are left out on purpose. NVIDIA's product pages quote them with different sparsity conventions, so putting them side by side invites a wrong conclusion. For LLM serving, where the decode phase is usually limited by how fast weights and KV cache can be read, memory bandwidth is the more predictive spec.

"RTX PRO 6000" also covers more than one card. The Workstation Edition lists 1,792 GB/s of bandwidth; the Server Edition most clouds rent lists 1,597 GB/s. That difference shows up in the benchmarks below.

What independent benchmarks show

The most useful public data comes from two vLLM benchmark runs published by CloudRift. Both compare the RTX PRO 6000 and the H100 directly, on the same models and settings.

Run A — November 2025. Single GPU, GLM-4.5-Air (AWQ 4-bit), 1,000 input / 1,000 output tokens, 256–512 concurrent requests, RTX PRO 6000 Workstation Edition.

GPU	Total throughput
RTX PRO 6000 (Workstation)	3,140 tok/s
H100 SXM	2,987 tok/s

Run B — January 2026. Google Cloud 8-GPU nodes (G4 with RTX PRO 6000, A3 High with H100), 8,000 input / 8,000 output tokens.

Workload	RTX PRO 6000	H100	H100 advantage
GLM-4.5-Air AWQ, one model per GPU	2,290.69 tok/s	2,556.03 tok/s	+11.6%
Qwen3-Coder-480B-A35B AWQ, 4-GPU tensor parallel	1,602.96 tok/s	2,328.63 tok/s	+45.3%
GLM-4.6 FP8, 8-GPU tensor parallel	1,651.67 tok/s	2,833.77 tok/s	+71.5%

Both machines are 8-GPU nodes running identical configurations, so the ratios hold however the throughput was aggregated across the node.

The two runs disagree on the single-GPU case: the RTX PRO 6000 is 5% ahead in Run A, while the H100 is 12% ahead in Run B. They differ in three ways at once — card edition, sequence length, and host — so neither result is wrong. The most plausible reading is that Run A's Workstation card had about 12% more bandwidth than a typical cloud Server Edition, and Run B's 8K-token sequences make each decoded token read much more KV cache, which favours the H100's bandwidth.

The multi-GPU rows are not ambiguous. As tensor parallelism widens, the H100's NVLink pulls away: +45% at 4-way, +72% at 8-way.

Cost per million tokens

Throughput alone does not answer a cost question. Combining it with price:

cost per token (RTX / H100) = (RTX $/hr ÷ H100 $/hr) × (H100 tok/s ÷ RTX tok/s)

At median on-demand prices ($2.20 vs $3.49, price ratio 0.63):

Scenario	H100 throughput vs RTX PRO 6000	RTX PRO 6000 cost per token	RTX break-even price
1 GPU, 1K/1K (Run A)	0.95×	40% cheaper	$3.67/hr
1 GPU, 8K/8K (Run B)	1.12×	30% cheaper	$3.13/hr
4-GPU tensor parallel (Run B)	1.45×	8% cheaper	$2.40/hr
8-GPU tensor parallel (Run B)	1.72×	8% more expensive	$2.03/hr

The break-even column is the RTX PRO 6000 hourly price at which it matches an H100 at $3.49/hr. Rent one below that price and it is the cheaper option for that workload.

In absolute terms, Run A works out to about $0.195 per million tokens on the RTX PRO 6000 and $0.325 on the H100, counting input and output tokens together.

CloudRift's own cost tables differ from these. Each run used a different price basis — Runpod list prices in November 2025, and an estimated cost of ownership in January 2026. The table above re-prices both runs with one consistent, current source, so the scenarios are comparable with each other.

What the extra 16 GB actually buys

Twenty percent more memory sounds marginal. It is not, because every extra gigabyte lands after the weights are loaded, in the space left for KV cache. For models that nearly fill an 80 GB card, that space roughly doubles.

KV cache per token for a standard attention model:

2 (K and V) × layers × KV heads × head dim × bytes per element

With an FP8 KV cache and 4 GB reserved for the runtime:

Model	Weights	KV per token	Free for KV on H100 80 GB	Free for KV on RTX PRO 6000	32K sequences: H100 → RTX
Qwen3-Coder-30B-A3B (BF16)	56.8 GB	48 KB	19.2 GB	35.2 GB	12 → 23
Qwen3-32B (BF16)	61.1 GB	128 KB	14.9 GB	30.9 GB	3 → 7
Llama-3.3-70B (FP8)	65.8 GB	160 KB	10.2 GB	26.2 GB	2 → 5
Llama-3.3-70B (BF16)	131.5 GB	160 KB	does not fit	does not fit	—

A 70B model at FP8 is the clearest case. On an H100 it fits, but leaves room for only two concurrent 32K-token conversations. On an RTX PRO 6000, five. For long-context, RAG-heavy, or agentic workloads, that headroom decides how much traffic a single card can take.

The last row matters just as much. A 70B model at BF16 needs two cards on either GPU — and once two cards have to talk to each other, the interconnect comes back into play, and with it the H100's advantage.

When the H100 is the better buy

Your model needs four or more GPUs. Once weights plus KV cache outgrow two cards (roughly 160–190 GB), you are into 4-way tensor parallelism or wider, where NVLink beats PCIe by a margin that outweighs the price gap.
Single-stream decode speed is the product. If latency for one user matters more than total throughput, 2.1× the memory bandwidth is hard to argue with.
You are serving very long sequences on a model that barely fits. Run B suggests the bandwidth gap grows with sequence length.
You need many small isolated tenants. MIG gives you up to seven partitions on an H100 versus four on an RTX PRO 6000.

When the RTX PRO 6000 is the better buy

The model and its KV cache fit on one 96 GB card. That covers most 7B–35B models at BF16 and 70B-class models at FP8 or below.
You scale out with replicas rather than tensor parallelism. Data-parallel serving never touches the interconnect, so the PCIe limitation does not apply.
Concurrency on long contexts is the constraint. The KV-cache table above is the whole argument.
You want to serve FP4-quantized models. FP4 tensor cores are a Blackwell feature; Hopper stops at FP8.

A short decision checklist

Add up weights and the KV cache your target concurrency needs, at the precision you plan to serve.
If the total fits in 96 GB minus a few GB of overhead, the RTX PRO 6000 is very likely cheaper per token. Benchmark it against your real prompt lengths.
If you need 2 GPUs, test both — the result could go either way.
If you need 4 or more GPUs, start from the H100 (or H200), unless your RTX PRO 6000 rate is below the break-even price for your workload.
Re-run the cost formula with the prices you are actually quoted. Market medians move every month.

FAQ

Is the RTX PRO 6000 faster than the H100 for LLM inference?
Not usually per card. In independent vLLM tests the H100 was between 5% slower and 12% faster on a single-GPU model, and 45–72% faster once tensor parallelism across 4–8 GPUs was involved. The RTX PRO 6000's case is cost per token, not raw speed.

How much does it cost to rent an RTX PRO 6000 versus an H100?
As of October 7, 2026, the median on-demand price across tracked providers was $2.20 per GPU-hour for the RTX PRO 6000 and $3.49 for the H100, per getdeploying.com.

What is the RTX PRO 6000's memory bandwidth?
1,597 GB/s for the Server Edition and 1,792 GB/s for the Workstation Edition, both with 96 GB of GDDR7. The H100 SXM has 3,350 GB/s of HBM3.

How many tokens per second can an RTX PRO 6000 serve?
It depends heavily on model, precision and concurrency. One public data point: about 3,140 total tokens/s on GLM-4.5-Air (AWQ 4-bit) at 256–512 concurrent requests, 1K input / 1K output, on a Workstation Edition card.

Is the "RTX 6000 Blackwell" the same GPU as the "RTX PRO 6000"?
Yes — both names refer to the 96 GB Blackwell card, sold in Workstation, Max-Q and Server editions. It is a different card from the RTX 6000 Ada (48 GB) and the older Quadro RTX 6000 (24 GB).

Method and caveats

Nothing here was measured by the author. Throughput comes from CloudRift's published benchmarks, prices from getdeploying.com's public listings, specs from NVIDIA's product pages, and model shapes from each model's config.json.
Prices are medians. Your actual price determines the answer, which is why the formula and break-even prices are included.
Throughput ratios are workload-specific. The benchmarks used GLM and Qwen3-Coder models; a different model, quantization, or serving stack will shift them.
Run A likely flatters the RTX PRO 6000 slightly, because it used the Workstation Edition, with about 12% more bandwidth than the Server Edition most clouds rent.
Units: in the VRAM table, GB means GiB (2³⁰ bytes), the unit nvidia-smi reports.

Reproduce the numbers

GiB = 1024**3

# Prices: getdeploying.com on-demand medians, 2026-10-07
RTX_HR, H100_HR = 2.20, 3.49

# (rtx tok/s, h100 tok/s) from CloudRift's published runs
RUNS = {
    "1 GPU, 1K/1K":  (3140.00, 2987.00),
    "1 GPU, 8K/8K":  (2290.69, 2556.03),
    "4-GPU TP":      (1602.96, 2328.63),
    "8-GPU TP":      (1651.67, 2833.77),
}
for name, (rtx, h100) in RUNS.items():
    rel = (RTX_HR / H100_HR) * (h100 / rtx)
    print(f"{name:<14} RTX cost/token = {rel:.0%} of H100 | break-even ${H100_HR / (h100 / rtx):.2f}/hr")

# KV-cache headroom: FP8 KV cache, 4 GiB runtime reserve, 32K-token sequences
MODELS = [  # name, params (B), bytes/param, layers, kv_heads, head_dim
    ("Qwen3-Coder-30B-A3B BF16", 30.5, 2, 48, 4, 128),
    ("Qwen3-32B BF16",           32.8, 2, 64, 8, 128),
    ("Llama-3.3-70B FP8",        70.6, 1, 80, 8, 128),
]
for name, p, wb, L, kvh, hd in MODELS:
    weights = p * 1e9 * wb / GiB
    kv_seq = 2 * L * kvh * hd * 1 * 32_768 / GiB
    fits = {cap: int((cap - weights - 4) / kv_seq) for cap in (80, 96)}
    print(f"{name:<26} H100: {fits[80]:>2} seqs | RTX PRO 6000: {fits[96]:>2} seqs")

Sources

NVIDIA, RTX PRO 6000 Blackwell Server Edition and Workstation Edition datasheet
NVIDIA, H100 Tensor Core GPU
CloudRift, RTX PRO 6000 vs H100, H200, and L40S: LLM Inference (Nov 2025)
CloudRift, Benchmarking LLM Inference on B200, H200, H100, and RTX PRO 6000 (Jan 2026)
getdeploying.com, RTX PRO 6000 rental prices and H100 rental prices (checked 2026-10-07)
Model configs: Qwen3-Coder-30B-A3B-Instruct, Qwen3-32B, Llama-3.3-70B-Instruct

Parameter count is a bad way to pick an open model for one GPU

yyyysa4 — Thu, 24 Sep 2026 08:34:54 +0000

Model selection usually starts with a number: 8B, 27B, 30B. It is the first thing on the model card and the first thing in every comparison thread. It is also close to useless for the question most teams are actually asking, which is how much traffic will this serve on the card I have.

Below is that question worked out for four current open models on a single 96 GB GPU. Everything here is arithmetic on published configs — no benchmarks were run. The script is at the end and takes about a second.

Units: GB throughout means GiB (2³⁰ bytes), which is what nvidia-smi reports.

The four models

Model	Total params	Architecture	Native context	License
Qwen3-8B	8.2B	Dense, 36 layers, GQA 32Q/8KV, head dim 128	32,768 (131K with YaRN)	Apache-2.0
Qwen3.5-9B	~9B	Hybrid: 3 linear-attention layers per 1 full, 32 total, 4 KV heads, head dim 256	262,144 (1.01M with YaRN)	Apache-2.0
Gemma-4-26B-A4B-it	25.2B (3.8B active)	MoE, 128 experts, top-8 + 1 shared, 30 layers, 5 full-attention + 25 sliding (window 1024), 8 KV heads, head dim 256	262,144	Apache-2.0
Qwen3.8-27B	~27B	Hybrid: same 3:1 pattern, 64 layers, 4 KV heads, head dim 256	262,144 (1M with YaRN)	Apache-2.0

Every figure in that table comes from the model card or config.json of the repo it links to.

Weights: where MoE stops helping

Mixture-of-Experts models are usually introduced with their active parameter count, because that is what determines compute per token. Gemma-4-26B-A4B activates 3.8B parameters out of 25.2B.

Memory does not work that way. Routing is per-token and unpredictable, so every expert has to be resident. At BF16:

Qwen3-8B — 15.3 GB
Qwen3.5-9B — 16.8 GB
Gemma-4-26B-A4B — 46.9 GB
Qwen3.8-27B — 50.3 GB

The "4B active" model costs nearly three times the VRAM of the 9B. What you buy for that is compute: it runs at roughly 4B-model speed per token while holding 26B-model knowledge. That is a real and valuable trade. It is just not a memory saving, and model cards that lead with the active count invite exactly that misreading.

KV cache: the term nobody budgets

Weights are static. KV cache grows with every token in flight, and it is where single-GPU deployments actually fall over. For a standard attention layer:

bytes per token = 2 (K and V) × layers × kv_heads × head_dim × dtype_bytes

The variable that matters most is layers — specifically, how many layers keep a cache that grows with sequence length. Three of these four models do something to reduce that count:

Qwen3-8B is conventional. All 36 layers cache. 2 × 36 × 8 × 128 × 2 = 144 KB per token.
Qwen3.5-9B interleaves 3 linear-attention layers for every 1 full-attention layer. Linear attention carries a fixed-size recurrent state instead of a growing cache, so only 8 of 32 layers contribute: 2 × 8 × 4 × 256 × 2 = 32 KB per token.
Qwen3.8-27B uses the same 3:1 pattern over 64 layers, so 16 layers cache: 64 KB per token.
Gemma-4-26B-A4B splits differently: 5 full-attention layers grow with the sequence (40 KB per token), while the other 25 use a 1024-token sliding window and stay flat at about 200 MB per sequence no matter how long the context gets.

So an 8B dense model spends 4.5× more KV memory per token than a 9B hybrid. Parameter count predicted the opposite.

What that buys in concurrency

Take 96 GB, subtract weights, reserve 4 GB for activations and framework overhead, and divide by the KV cost of one 32,768-token sequence:

Model	KV per 32K sequence	Concurrent 32K sequences
Qwen3-8B	4.50 GB	17
Qwen3.5-9B	1.00 GB	75
Gemma-4-26B-A4B	1.45 GB	31
Qwen3.8-27B	2.00 GB	20

The ranking has almost nothing to do with size. Qwen3.5-9B — nominally the second-smallest model here — serves four times the concurrent long-context traffic of Qwen3-8B, and nearly four times that of the 27B. The 26B MoE, despite spending 47 GB on weights, still fits more concurrent sequences than the 8B dense model does.

If your workload is RAG, document processing, or anything else that puts long prompts in flight, this table is a better predictor of cost per request than any leaderboard.

Why there is no quality column here

There is an obvious missing dimension, and it is missing on purpose.

The four model cards report largely disjoint benchmark sets. Gemma-4-26B-A4B-it reports MMLU-Pro 82.6%, GPQA Diamond 82.3%, LiveCodeBench v6 77.1%. Qwen3.5-9B reports MMLU-Pro 82.5 and C-Eval 88.2. Qwen3.8-27B reports GPQA Diamond 89.2 and LiveCodeBench v6 90.3. Qwen3-8B's card reports neither.

Two of those MMLU-Pro numbers are within 0.1 of each other, which is exactly the kind of coincidence that produces a confident and wrong blog post. Vendor-run evaluations differ in prompt format, few-shot count, decoding parameters, and whether reasoning mode was enabled — none of which are consistently disclosed. Putting them in one table implies a comparison that the underlying numbers do not support.

The honest version: use these figures to shortlist, then run your own eval on your own prompts. Memory math transfers between deployments. Benchmark scores often do not.

What this analysis does not tell you

Nothing about throughput. Memory determines what fits. Tokens per second depends on memory bandwidth, kernel quality, batching policy, and the serving stack. A model that fits 75 sequences will not necessarily serve them fast.
Quantization changes every number. FP8 weights roughly halve the weight column; FP8 or INT8 KV cache halves the cache column. The relative ordering mostly survives, the absolute headroom does not.
Linear attention is a trade, not a free win. The architectures that cut KV cost do so by compressing history into a fixed-size state. That is very good for throughput and is not free on tasks that need precise long-range recall. Test it rather than assuming the memory win is costless.
Real allocators are messier. vLLM and SGLang preallocate a KV pool via gpu_memory_utilization; fragmentation, CUDA graphs, and multimodal encoders all take a cut. Treat the concurrency numbers as ceilings.
Parameter counts are approximate. Qwen publishes "9B" and "27B" rather than exact totals, so those weight figures carry a percent or two of slack.

Reproduce it

GiB = 1024**3
BYTES = 2  # BF16 KV cache

# name, params(B), full_attn_layers, sliding_layers, window, kv_heads, head_dim
MODELS = [
    ("Qwen3-8B",         8.2, 36,  0,    0, 8, 128),
    ("Qwen3.5-9B",       9.0,  8,  0,    0, 4, 256),
    ("Gemma-4-26B-A4B", 25.2,  5, 25, 1024, 8, 256),
    ("Qwen3.8-27B",     27.0, 16,  0,    0, 4, 256),
]

def kv_for_seq(full_l, sl_l, win, kvh, hd, seq):
    growing = 2 * full_l * kvh * hd * BYTES * seq
    capped  = 2 * sl_l   * kvh * hd * BYTES * min(seq, win)
    return growing + capped

VRAM, RESERVE, SEQ = 96, 4, 32_768
for name, p, full_l, sl_l, win, kvh, hd in MODELS:
    weights = p * 1e9 * BYTES / GiB
    per_seq = kv_for_seq(full_l, sl_l, win, kvh, hd, SEQ) / GiB
    free    = VRAM - weights - RESERVE
    print(f"{name:<18} weights {weights:5.1f} GB | "
          f"{per_seq:4.2f} GB/seq | {int(free/per_seq):3d} concurrent {SEQ//1024}K seqs")

Change SEQ to your real context length and BYTES to 1 if you serve an FP8 KV cache. The layer counts come straight from each repo's config.json — if a model ships a new revision, re-read it rather than trusting this table.

Best models to run on RTX Pro 6000 96GB

yyyysa4 — Thu, 24 Sep 2026 03:43:33 +0000

The right model depends on the task, not on the GPU. A 96 GB RTX Pro 6000 holds a 20B to 35B open model with room for context and batching, so you get to pick by the job instead of by a memory ceiling. For coding and agents that means Qwen3 Coder 30B; for low-cost chat, gpt-oss 20B; for retrieval, an embedding model with a reranker; for voice, Whisper and Kokoro. All of them run through the same OpenAI-compatible API at https://api.ecohash.com/v1, so you start on the shared API and move to a dedicated endpoint or a workspace later without touching your code.

Pick by task, not by memory. Coding: qwen3-coder-30b-a3b-instruct. Chat: gpt-oss-20b. Retrieval: jina-embeddings-v3 + bge-reranker-v2-m3. Voice: whisper-large-v3-turbo + kokoro-82m.

Model by task

Task	Model	Model ID
Coding, agents, tool use	Qwen3 Coder 30B	`qwen3-coder-30b-a3b-instruct`
General chat, low cost	gpt-oss 20B	`gpt-oss-20b`
General chat, higher capability	GLM 5.2, DeepSeek V4 Flash	`GLM-5.2`, `DeepSeek-V4-Flash`
Embeddings for RAG	Jina v3, Qwen3 Embedding	`jina-embeddings-v3`, `qwen3-embedding-8b`
Reranking for RAG	BGE reranker v2 m3	`bge-reranker-v2-m3`
Speech to text	Whisper Large V3 Turbo	`whisper-large-v3-turbo`
Text to speech	Kokoro 82M	`kokoro-82m`

The higher-capability chat models run through the hosted API, not on a single 96 GB card. Here is what the single-card models measure end-to-end, in July 2026:

Task	Model	Measured
Coding	`qwen3-coder-30b-a3b-instruct`	30 ms TTFT, ~110 tok/s, 32k context
General chat	`gpt-oss-20b`	170 ms TTFT, ~125 tok/s
General chat	`llama-3.1-8b-instruct`	205 ms TTFT, ~143 tok/s
Speech to text	`whisper-large-v3-turbo`	WER 4.37, ~0.5 s per 10 s of audio
Text to speech	`kokoro-82m`	first audio ~120 ms

Against other providers of the same open models, we serve them at full BF16 precision and land competitively on price and speed (llama-3.1-8b-instruct, EcoHash in purple):

Full data and charts: ecohash-benchmarks.

Facts


GPU	NVIDIA RTX Pro 6000 Blackwell Server Edition, 96 GB GDDR7
Single-card sweet spot	20B to 35B open models
API base URL	`https://api.ecohash.com/v1`
Access	Shared API, dedicated endpoint, GPU workspace
Price	RTX Pro 6000 at $1.89 per GPU-hour (pricing)

Who this is for

Developers choosing an open model for a coding, chat, retrieval, or voice feature.
Teams deciding what to run on a 96 GB card versus what to call through a hosted API.

What you can decide

Which model to start with for your task.
Whether it fits on a single RTX Pro 6000 or is better used through the hosted API.
When to move from the shared API to a dedicated endpoint or a workspace.

How to choose

Four questions get you most of the way:

What is the task? Code, chat, retrieval, or speech. Start here, not with the hardware.
Does it fit in 96 GB at your context length and batch size? A 20B to 35B model fits with room to spare, models near 70B need quantization, and 235B and up are better through the hosted API.
What are your latency and throughput needs? Steady, high-volume traffic is the reason to reserve a dedicated endpoint.
Do you need to self-host at all? For most teams the shared API is enough, and the GPU is our problem, not yours.

Coding

Qwen3 Coder 30B (qwen3-coder-30b-a3b-instruct) is the default for code generation, refactoring, and tool-using agents. It is a Mixture-of-Experts model with about 30B total parameters, it fits comfortably on one 96 GB card, and it serves a 32,768-token context at about 110 tokens/sec single-stream with a 30 ms time to first token. See the model page.

For lighter or cheaper high-volume coding, gpt-oss 20B (gpt-oss-20b) is the smaller option. The three deployment paths for Qwen3 Coder are covered in Qwen3 Coder 30B on RTX Pro 6000.

General chat

Start with gpt-oss 20B (gpt-oss-20b) for low-cost chat and assistants. It fits on one card, has a long context, and reaches about 8,900 output tokens/sec under load.

For stronger general capability, we also serve larger models such as GLM 5.2 (GLM-5.2) and DeepSeek V4 Flash (DeepSeek-V4-Flash), and very large Mixture-of-Experts models like Qwen3-235B-A22B (Qwen3-235B-A22B). Those are not single-card deployments. Use them through the shared API or a dedicated endpoint and let us handle the hardware.

Retrieval (RAG)

A retrieval pipeline needs two models: an embedding model to index and search text, and a reranker to reorder the top results.

Embeddings: jina-embeddings-v3, jina-embeddings-v4, or the Qwen3 Embedding family (qwen3-embedding-8b, qwen3-embedding-4b, qwen3-embedding-0.6b).
Reranking: bge-reranker-v2-m3.

All of them run through the same key, so one service can embed, rerank, and generate the answer without stitching vendors together. See the RAG use case.

Voice

A voice agent uses three model types: speech to text, a chat model, and text to speech.

Speech to text: whisper-large-v3-turbo or qwen3-asr-1-7b.
Text to speech: kokoro-82m or qwen3-tts.

These speech models are small and do not need an RTX Pro 6000. They run through the API and move onto GPU capacity only when throughput grows. The full build is in Build a voice agent with Whisper, Kokoro, and an OpenAI-compatible API; see also voice and speech, Kokoro 82M, and Whisper Large V3 Turbo.

Calling these models

The API is OpenAI-compatible. A coding call:

from openai import OpenAI

client = OpenAI(base_url="https://api.ecohash.com/v1", api_key="eco_...")

resp = client.chat.completions.create(
    model="qwen3-coder-30b-a3b-instruct",
    messages=[
        {"role": "user", "content": "Refactor this to be iterative:\n\ndef fib(n):\n    return n if n < 2 else fib(n-1) + fib(n-2)"}
    ],
)
print(resp.choices[0].message.content)

An embeddings call for a RAG index:

emb = client.embeddings.create(
    model="jina-embeddings-v3",
    input=["EcoHash serves open models through an OpenAI-compatible API."],
)
print(len(emb.data[0].embedding))

Shared API, dedicated endpoint, or workspace

Shared API: no GPU to manage, pay per token, good for building and variable traffic. See inference.
Dedicated endpoint: reserved capacity for one model, steadier throughput and more predictable latency. See dedicated inference.
GPU workspace: an hourly RTX Pro 6000 for experiments, fine-tuning, and batch jobs. See GPU compute.

Limitations

Model fit depends on quantization, context length, and batch size. The ranges here are starting points to validate, not guarantees.
The higher-capability chat models are for the hosted API, not single-card serving.
Real performance depends on your setup. Reproduce any figure with its method and date.
Model IDs change as the catalog changes. Check the model catalog before you build.

FAQ

What is the best model to run on an RTX Pro 6000 96GB? It depends on the task. Coding: qwen3-coder-30b-a3b-instruct. General chat: gpt-oss-20b. Retrieval: jina-embeddings-v3 with bge-reranker-v2-m3. Speech: whisper-large-v3-turbo with kokoro-82m.

What is the best coding model for the RTX Pro 6000? Qwen3 Coder 30B (qwen3-coder-30b-a3b-instruct). It fits on one 96 GB card and serves a 32,768-token context.

What is a good low-cost general chat model? gpt-oss 20B (gpt-oss-20b). It fits on one card and has a long context.

Can I run a 235B model on one RTX Pro 6000? No. Use a model like Qwen3-235B-A22B through the hosted API instead.

What models do I need for RAG? An embedding model and a reranker, for example jina-embeddings-v3 and bge-reranker-v2-m3, plus a chat model to write the answer.

Do speech models need an RTX Pro 6000? No. Whisper and Kokoro run through the API, and move onto GPU capacity only when throughput demands it.

How many models can I use with one API key? All of them. Every model shares one key and one base URL, https://api.ecohash.com/v1.

How much does this cost? API usage is billed per token; a GPU workspace runs at $1.89 per GPU-hour. See pricing for current rates.

Next steps

Browse all models: ecohash.com/models
See the RTX Pro 6000 page: ecohash.com/gpu-compute
Reserve capacity: ecohash.com/dedicated-inference