Same GPU, Same Model, Different Cost per Token: Benchmark Qwen3.8-27B on Your Own Prompts

When I'm deciding whether to rent a GPU for an open model, I work out the cost per million output tokens:

cost per 1M output tokens = hourly price ÷ (output tokens/s × 3,600) × 1,000,000

The hourly price is easy to look up. Tokens per second is the number that's easy to get wrong, because it depends on what you send the model.

What the published numbers show

For Qwen3.8-27B on a single RTX PRO 6000 Blackwell (96 GB), the most detailed public numbers I found are in EcoHash's speculative-decoding benchmark. They served the FP8 checkpoint on vLLM 0.27.1 with the model's built-in multi-token-prediction (MTP) head switched on, which is a form of speculative decoding. Then they swept from 1 to 128 concurrent requests, with 150-token replies.

The headline number isn't the interesting part. What caught my eye is that they ran the same sweep with different prompt sets, and the results don't match.

Their latency budget was a 95th-percentile time to first token of 1.5 s or less, and a 95th-percentile time per output token of 30 ms or less. Here's the highest concurrency that stayed inside it:

Prompt set	Highest concurrency in budget	Output tokens/s there	Cost per 1M output tokens at $1.89/hr
Chat (MT-Bench-style)	32	1,274	$0.41
Math (GSM8K-style)	48	1,862	$0.28
Mixed pool, through their API gateway	64	1,954	$0.27

Line chart of 95th-percentile time per token against concurrency for Qwen3.8-27B on one RTX PRO 6000. Chat prompts cross the 30 ms budget after 32 concurrent requests, math prompts after 48.

The card, the weights and the flags were the same in every run. The prompts alone moved the cost per token by about 1.5×.

The reason is the draft head. MTP guesses a few tokens ahead, and the model checks the guesses in one pass. Predictable output like arithmetic gets more guesses accepted, so each step yields more tokens. Open-ended chat gets fewer. The gap shows up even with a single request: 10.5 ms per token on math against 15.0 ms on chat.

Two more things will pull your numbers away from theirs. Qwen3.8-27B reasons at length, so real replies usually run well past 150 tokens. And long prompts add prefill time.

So before I trust anyone's tokens per second, these included, I'd measure my own.

The plan

1. Rent one RTX PRO 6000 and confirm it's the right card.
2. Start vLLM with a known-good config.
3. Replay your own prompts at increasing concurrency.
4. Take the highest level inside your latency budget and turn it into a cost per 1M tokens.

1. Rent the card and check it

Any provider with the RTX PRO 6000 Blackwell will do. I priced it with EcoHash's $1.89 per GPU-hour, billed per second. Once the machine boots, run:

nvidia-smi --query-gpu=name,memory.total,compute_cap --format=csv

You want to see "RTX PRO 6000 Blackwell", close to 96 GB of memory, and compute capability 12.0.

2. Start the server

This is the config from EcoHash's deployment guide for this model, and it's the setup behind the numbers above:

vllm serve /path/to/Qwen3.8-27B-FP8 \
  --served-model-name qwen3.8-27b \
  --dtype bfloat16 \
  --tensor-parallel-size 1 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 128 \
  --enable-prefix-caching \
  --enable-prompt-tokens-details \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --safetensors-load-strategy prefetch \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

A few of these flags matter more than they look, and the guide explains why. In short: use bfloat16, not float16. Don't add --quantization, because the checkpoint is already FP8. And use the qwen3_xml tool parser. --max-model-len 32768 is a choice rather than a limit of the model, so raise it if your prompts are longer. Longer requests take more cache each, so expect a lower ceiling. The benchmark will show you how much lower.

3. Collect your prompts

Pull 20 to 50 real prompts from your logs into prompts.jsonl, one per line. Each line can be {"prompt": "..."} or a full {"messages": [...]} conversation that includes your system prompt. Set --max-tokens close to your real reply length. For a reasoning model, that's probably well above 150.

4. Run the script

"""Measure an OpenAI-compatible endpoint at several concurrency levels,
then turn the results into a cost per 1M output tokens.

    python bench.py --base-url http://localhost:8000/v1 --model qwen3.8-27b \
        --prompts prompts.jsonl --levels 1,8,16,32,48,64 --max-tokens 512 --price 1.89
"""
import argparse, asyncio, json, math, os, time
from openai import AsyncOpenAI


def p95(xs):
    xs = sorted(xs)
    return xs[math.ceil(0.95 * len(xs)) - 1] if xs else float("nan")


def has_text(delta):
    # Reasoning models stream their thinking in a separate field before the answer.
    extra = delta.model_extra or {}
    return bool(delta.content or extra.get("reasoning_content") or extra.get("reasoning"))


async def one_request(client, model, messages, max_tokens):
    t0 = time.perf_counter()
    first, tokens = None, 0
    stream = await client.chat.completions.create(
        model=model, messages=messages, max_tokens=max_tokens, temperature=0.5,
        stream=True, stream_options={"include_usage": True},
    )
    async for chunk in stream:
        if first is None and chunk.choices and has_text(chunk.choices[0].delta):
            first = time.perf_counter()
        if chunk.usage:
            tokens = chunk.usage.completion_tokens
    end = time.perf_counter()
    if first is None or tokens < 2:
        return None
    # Divide by tokens, not chunks: with speculative decoding one chunk can carry several.
    return first - t0, (end - first) / (tokens - 1), tokens


async def run_level(client, args, prompts, conc):
    for i in range(args.warmup):
        await one_request(client, args.model, prompts[i % len(prompts)], args.max_tokens)
    queue = asyncio.Queue()
    for i in range(max(24, 3 * conc)):
        queue.put_nowait(prompts[i % len(prompts)])
    results, errors = [], 0

    async def worker():
        nonlocal errors
        while True:
            try:
                messages = queue.get_nowait()
            except asyncio.QueueEmpty:
                return
            try:
                r = await one_request(client, args.model, messages, args.max_tokens)
                if r:
                    results.append(r)
                else:
                    errors += 1
            except Exception as e:
                errors += 1
                print(f"  request failed: {e!r}")

    t0 = time.perf_counter()
    await asyncio.gather(*(worker() for _ in range(conc)))
    wall = time.perf_counter() - t0
    tok_s = sum(r[2] for r in results) / wall
    return {
        "conc": conc, "ok": len(results), "errors": errors, "tok_s": tok_s,
        "ttft_p95_ms": p95([r[0] for r in results]) * 1000,
        "tpot_p95_ms": p95([r[1] for r in results]) * 1000,
        "usd_per_1m": args.price / (tok_s * 3600) * 1e6 if tok_s else float("nan"),
    }


def load_prompts(path):
    prompts = []
    with open(path) as f:
        for line in f:
            if line.strip():
                row = json.loads(line)
                prompts.append(row["messages"] if "messages" in row
                               else [{"role": "user", "content": row["prompt"]}])
    return prompts


async def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--base-url", default="http://localhost:8000/v1")
    ap.add_argument("--model", default="qwen3.8-27b")
    ap.add_argument("--prompts", required=True, help="JSONL: {\"prompt\": ...} or {\"messages\": [...]}")
    ap.add_argument("--levels", default="1,8,16,32,48,64")
    ap.add_argument("--max-tokens", type=int, default=512)
    ap.add_argument("--warmup", type=int, default=2)
    ap.add_argument("--price", type=float, default=1.89, help="GPU price, $/hour")
    ap.add_argument("--ttft-budget", type=float, default=1500, help="p95 ms")
    ap.add_argument("--tpot-budget", type=float, default=30, help="p95 ms per token")
    args = ap.parse_args()

    client = AsyncOpenAI(base_url=args.base_url, api_key=os.environ.get("API_KEY", "none"),
                         timeout=600, max_retries=0)
    prompts = load_prompts(args.prompts)
    rows = []
    for conc in map(int, args.levels.split(",")):
        row = await run_level(client, args, prompts, conc)
        row["fits"] = (row["errors"] == 0 and row["ttft_p95_ms"] <= args.ttft_budget
                       and row["tpot_p95_ms"] <= args.tpot_budget)
        rows.append(row)
        print(f"C={conc}: {row['tok_s']:.0f} tok/s, TPOT p95 {row['tpot_p95_ms']:.1f} ms")

    print("\n| conc | ok | errors | output tok/s | TTFT p95 ms | TPOT p95 ms | $ per 1M out | in budget |")
    print("|---|---|---|---|---|---|---|---|")
    for r in rows:
        print(f"| {r['conc']} | {r['ok']} | {r['errors']} | {r['tok_s']:.0f} | {r['ttft_p95_ms']:.0f} "
              f"| {r['tpot_p95_ms']:.1f} | ${r['usd_per_1m']:.2f} | {'yes' if r['fits'] else 'no'} |")
    best = max((r for r in rows if r["fits"]), key=lambda r: r["tok_s"], default=None)
    if best:
        print(f"\nBest level inside budget: C={best['conc']}, {best['tok_s']:.0f} tok/s, "
              f"${best['usd_per_1m']:.2f} per 1M output tokens at ${args.price}/hr")


if __name__ == "__main__":
    asyncio.run(main())

Run it on the GPU instance itself, against localhost:8000, so you're measuring the card and not your network. If you have to go through a public endpoint, put its key in the API_KEY environment variable.

pip install openai
python bench.py --prompts prompts.jsonl --levels 1,8,16,32,48,64 --max-tokens 1024 --price 1.89

A few choices in the script are deliberate:

• Time to first token counts reasoning. Qwen3.8-27B streams its thinking in a separate field before the answer. The clock stops at whichever arrives first, because that's when generation has started.
• Time per token divides by tokens, not chunks. With speculative decoding, one streamed chunk can carry several tokens. Counting chunks makes the server look slower than it is, a trap EcoHash's write-up also points out.
• Each level sends max(24, 3 × concurrency) requests after two warm-ups. That matches the sampling EcoHash describes for its published run, so your numbers line up with theirs. The last few requests at each level run with fewer in flight, so throughput comes out slightly low, which is the safe direction to be wrong in.

Reading the result

The script prints a markdown table and names the best level inside your budget. If your budget is different, change --tpot-budget and --ttft-budget. For scale, 30 ms per token is about 33 tokens per second for each stream.

Then compare that cost per 1M with what an API would charge you for the same model. EcoHash's own API lists Qwen3.8-27B at $0.60 per 1M output tokens, and the 21 providers on OpenRouter ranged from $1.25 to $3.20 when I checked on November 1. If the card can't beat the API you'd otherwise use at a realistic load, renting it is about control, such as a longer context or your own serving flags, not about price.

Keep in mind that the cost at the best level assumes the card stays that busy every hour you pay for it. Real traffic doesn't. Scale the number by your average load, not your peak.

What it costs to find out

At $1.89 an hour, billed per second, a 30-minute session costs about 95 cents, and EcoHash's $1 signup credit covers roughly that. Terminate the instance when you're done, because the clock keeps running while it sits idle.

Limits

• This measures throughput and latency only. It says nothing about answer quality, so test that separately.
• The published numbers come from the Server Edition of the card. The workstation editions have more memory bandwidth, so expect somewhat different results.
• vLLM changes quickly. Record your version alongside your results.

---