DEV Community: sam ros

Priced per TB/s, the H200 beats the H100 at 6 of 7 providers

sam ros — Mon, 02 Nov 2026 17:39:28 +0000

An H200 has exactly as much compute as an SXM H100 and usually rents for more, so on a FLOPS spreadsheet it never makes sense. The H200 vs H100 cost per token question turns on a different column: the H200 moves 4.8 TB/s out of memory against the H100's 3.35, and decode spends its time waiting on memory. Memory is also what the industry is short of: TechCrunch reports that Samsung used its Q2 earnings call to say the shortage worsens in 2027 and stays tight until at least 2028.

So we priced that column, using our GPU table for 2026-10-30.

Decode waits on memory, even at large batches

The FlashAttention paper defines arithmetic intensity as arithmetic operations per byte of memory touched, and says most Transformer operations are held back by memory accesses. Divide a GPU's dense FLOPS by its bandwidth and you get the intensity it can feed: at FP8, about 591 operations per byte for an SXM H100, 412 for an H200, 562 for a B200. Any workload well below that line runs out of bandwidth first.

Researchers at the Barcelona Supercomputing Center and IBM measured where decode sits, with vLLM on an H100. Attention kernels stayed between about 0.5 and 1 operation per byte from batch 1 to the maximum. At batch 512 on OPT-1.3B, memory traffic reached about 1.51 TB/s of a 1.63 TB/s ceiling, while compute sat near 4% of the paper's own compute line. That line, 25.6 trillion floating-point operations a second, is far below NVIDIA's FP8 figure, and decode sits well under both. In their tests on OPT and Llama 2 models, large batches didn't move decode into compute-bound territory.

The table that follows from that

Same provider, same day, SXM H100 against H200 (Vast, Lambda and Baseten list no H200, so they're out):

Provider	H100 $/h	H200 $/h	H100 $/TB/s-h	H200 $/TB/s-h	B200 $/TB/s-h
Fireworks	8.00	8.00	2.39	1.67	1.62
CoreWeave	6.16	6.31	1.84	1.31	1.07
Nebius	4.50	5.40	1.34	1.13	1.06
Modal	3.95	4.54	1.18	0.95	0.78
DeepInfra	2.90	3.49	0.87	0.73	0.61
RunPod	3.99	5.29	1.19	1.10	1.00
Together AI	3.99	5.99	1.19	1.25	1.02

The break-even is 4.8 / 3.35, so the H200 wins per TB/s whenever it costs under 1.43 times the H100. Together AI, at 1.50 times, is the one provider here where it loses. The B200 wins at all seven, and it's the cheapest row at each.

The cheap-looking row is the trap. RunPod's PCIe H100 lists at $2.89, but NVIDIA's product brief gives that card a peak of 2,000 GB/s, so you pay $1.45 per TB/s-hour: more than RunPod's SXM H100, H200 or B200.

For one model, 70 GB of weights read once per token caps an H100 at 47.9 tokens/s and an H200 at 68.6. At RunPod's prices that ceiling works out to $23.16 per million tokens on the H100 and $21.43 on the H200. Those are batch-1 limits for one stream, so a real stream lands below that speed on both cards. Batching changes the sum: one pass over the weights serves many streams, so batched cost per token can come in far under these prices, and this arithmetic doesn't tell you by how much on either card.

Here's the whole calculation. Put in your own quotes and weights:

"""Rank GPUs for memory-bound decode by dollars per TB/s-hour."""
SPECS = {  # peak memory TB/s and dense FP8 TFLOPS (half NVIDIA's sparse figure)
    "H100 PCIe": (2.0, None),
    "H100 SXM": (3.35, 1979),
    "H200": (4.8, 1979),
    "B200": (8.0, 4500),
}
PRICES = [  # $ per GPU-hour, public on-demand list prices, 2026-10-30
    ("RunPod", "H100 PCIe", 2.89), ("RunPod", "H100 SXM", 3.99),
    ("RunPod", "H200", 5.29), ("RunPod", "B200", 7.99),
    ("CoreWeave", "H100 SXM", 6.16), ("CoreWeave", "H200", 6.31),
    ("CoreWeave", "B200", 8.60),
    ("Nebius", "H100 SXM", 4.50), ("Nebius", "H200", 5.40),
    ("Nebius", "B200", 8.50),
    ("Modal", "H100 SXM", 3.95), ("Modal", "H200", 4.54),
    ("Modal", "B200", 6.25),
    ("Together AI", "H100 SXM", 3.99), ("Together AI", "H200", 5.99),
    ("Together AI", "B200", 8.19),
    # Neither page names the H100's form factor, so SXM is an assumption here.
    ("DeepInfra", "H100 SXM", 2.90), ("DeepInfra", "H200", 3.49),
    ("DeepInfra", "B200", 4.89),
    ("Fireworks", "H100 SXM", 8.00), ("Fireworks", "H200", 8.00),
    ("Fireworks", "B200", 13.00),
]


def per_tb_hour(usd_hr, gpu):
    return usd_hr / SPECS[gpu][0]


def ops_per_byte(gpu):
    return SPECS[gpu][1] / SPECS[gpu][0]  # TFLOPS / (TB/s) = operations per byte


def h200_breakeven():
    """H200 wins per TB/s while it costs less than this many times an SXM H100."""
    return SPECS["H200"][0] / SPECS["H100 SXM"][0]


def max_tokens_per_s(weights_gb, gpu):
    """Ceiling: every weight read once per token, at peak bandwidth, batch 1."""
    return SPECS[gpu][0] * 1000 / weights_gb


def usd_per_m_tokens(usd_hr, weights_gb, gpu):
    return usd_hr / (max_tokens_per_s(weights_gb, gpu) * 3600) * 1e6


if __name__ == "__main__":
    for provider, gpu, usd in sorted(PRICES, key=lambda r: per_tb_hour(r[2], r[1])):
        print(f"{provider:12} {gpu:10} ${usd:5.2f}/h  "
              f"${per_tb_hour(usd, gpu):.2f}/TB/s-h  "
              f"${usd_per_m_tokens(usd, 70, gpu):5.2f}/M tokens at the 70 GB ceiling")

Where this ranking stops being true

Prefill is the other case. Philip Kiely's Inference Engineering says it runs out of FLOPS first, and we didn't measure that. If that's your traffic, an H200 gives you no extra FLOPS over an SXM H100.

Memory size isn't in the table. An H100 holds 80 GB, an H200 141 GB and a B200 180 GB, before any KV cache. If your model doesn't fit, the honest comparison is two cheaper cards against one bigger one, and the per-TB/s number won't settle that.

Peak bandwidth isn't sustained bandwidth. NVIDIA's figures are peaks ("up to 8TB/s" for the B200). That paper's memory ceiling for its card is 1.63 TB/s, under half the SXM figure, though it describes an unusual 64 GB H100, so don't treat that as a discount rate.

More bandwidth doesn't promise proportionally more tokens. The paper found CPU time reaching up to 30% of decode at batch 512 on OPT-1.3B, and a faster memory bus doesn't shrink that.

DeepInfra and Fireworks list their H100 only as 80 GB, so we assumed SXM. If either is a PCIe card the H200 wins by more, so the count holds.

We ran nothing on these GPUs. The tokens-per-second figures are bounds from published specs. Discounts reorder the table too: on Together AI's 91 to 180 day reserved rates ($3.19 and $3.99) the H200 costs 1.25 times the H100 and wins, though its preemptible rates stay at 1.50 times. Some of these rates only come in bulk or as a managed service, such as CoreWeave's 8-GPU instances and DeepInfra's dedicated deployments, so compare within a row, not across providers.

Do this today

Divide your GPU's hourly price by its peak bandwidth, then do the same for the next card up at that provider. If that number is lower and your traffic is mostly decode, benchmark your own model on both before you renew.

We also priced the same seven providers per dense PFLOP-hour, the number that matters for prefill, and the ranking changes; ask in the comments and we'll post the table.

When you pick a GPU for serving, do you compare on bandwidth, memory size or the hourly price?

Prices are InferPrice's public on-demand list prices for 2026-10-30, each from the provider's own pricing page: RunPod, CoreWeave, Nebius, Modal, Together AI, DeepInfra, Fireworks. InferPrice ran the arithmetic and its tests; nothing here was benchmarked on these GPUs.

Sources

NVIDIA H100 specifications and H100 PCIe product brief
NVIDIA H200 specifications
NVIDIA HGX AI Factory components and DGX B200 specifications
Dao et al., FlashAttention (2022)
Recasens et al., Mind the Memory Gap (2025)
TechCrunch on Samsung's Q2 call (2026-07-31)
Further reading and the prefill rule of thumb: Philip Kiely, Inference Engineering (Baseten Books, 2026), section 2.4

Originally published at inferprice.pages.dev.

SGLang's +75% sampler speedup skips Qwen3 chat requests

sam ros — Mon, 26 Oct 2026 15:19:07 +0000

SGLang v0.5.20's release notes credit a nine-line change in the sampler with "Qwen3.5-2B decode +75% at batch 1 and +23% at batch 32". The nine lines are real, they are in the published wheel, and they do less work on the host than the line they replace. What the note leaves out is the predicate they sit behind: if a single request in the decode batch asks for top_k, top_p or min_p, your tokens never reach them. Serve a Qwen3 model through /v1/chat/completions with SGLang's own defaults and none of your traffic does, because the model's generation_config.json sets top_k and top_p for you.

This is the first of our claim checks: one published number, a test plan written before anything runs, hardware anyone can get, and a plain account of which parts hold. Nobody paid for this one.

Nine lines, and exactly one call site

PR #38117 landed on 6 September 2026 and shipped in v0.5.20 (tag commit 94602c9c, 18 September; v0.5.19 is 0bcd8223). Inside sampling_from_probs_torch in python/sglang/srt/layers/sampler.py, v0.5.19 drew the next token like this:

sampled_index = torch.multinomial(probs, num_samples=1)

v0.5.20 keeps that line as a fallback and puts the new rule in front of it, behind an environment switch that defaults to on:

if envs.SGLANG_OPT_USE_GUMBEL_SAMPLE.get():
    q = torch.empty_like(probs, dtype=torch.float32).exponential_(1.0)
    q.clamp_min_(torch.finfo(torch.float32).tiny)
    sampled_index = (probs.float() / q).argmax(dim=-1, keepdim=True)
else:
    sampled_index = torch.multinomial(probs, num_samples=1)

Dividing the probabilities by an Exp(1) draw and taking the argmax is the Gumbel-max trick, and it picks tokens from the same distribution as a categorical draw. The code comment is specific about where the time goes: torch.multinomial keeps CPU-side philox offset bookkeeping and runs a distribution-validity assert, both on the decode critical path. That work is on the host, between kernel launches, which is why the note's number is large at batch 1 and smaller at batch 32, and why your CPU decides how much you get.

SGLANG_OPT_USE_GUMBEL_SAMPLE is declared EnvBool(True) in python/sglang/srt/environ.py and accepts 0, false, no or n. That switch is the honest way to A/B this on your own hardware: same wheel, same weights, one environment variable.

One request with top_p takes the whole batch off the fast path

In v0.5.20 that function has one caller, Sampler._sample_from_probs, under if simple_sampling_case. The flags come from SamplingBatchInfo.from_schedule_batch, with any() over the requests: need_top_p_sampling is any top_p != 1.0, need_top_k_sampling is any top_k != TOP_K_ALL (1 << 30), need_min_p_sampling is any min_p > 0. Before that, is_all_greedy (every request with top_k <= 1) sends the batch to the greedy path, and --enable-deterministic-inference sends it to the seeded path.

So there are four outcomes, and only one of them is the new code:

Batch	Path in v0.5.20
Every request plain temperature sampling	Gumbel-max, the new code
Every request greedy (temperature 0)	greedy argmax, untouched
Any request with top_k, top_p or min_p	the filtered path, untouched
Server with deterministic inference	seeded multinomial, untouched

The third row is the one that bites. A greedy request has top_k = 1 after normalization, and 1 != 1 << 30, so one temperature: 0 request sharing a decode batch moves everyone onto the filtered path.

It also lasts longer than the request that caused it. The flags are computed when requests are prefilled, and merge_batch ORs them into the running decode batch when new requests join. When a request finishes, filter_batch drops its rows and leaves every flag as it was. So one top_p request keeps the whole running batch on the filtered path after it has finished, until the running batch empties completely and the next prefill batch starts it fresh. On a server that's never idle, that can be a long time.

This port of SGLang's own arithmetic tells you which row a batch is in:

TOP_K_ALL = 1 << 30  # sampling_params.py: top_k = -1 ("whole vocabulary") becomes this
SAMPLING_EPS = 1e-6  # sampling_params.py: _SAMPLING_EPS


def normalize(params):
    """SamplingParams.normalize(): temperature below _SAMPLING_EPS means greedy (top_k = 1)."""
    p = dict(params)
    if 0 <= p["temperature"] < SAMPLING_EPS:
        p["temperature"] = 1.0
        p["top_k"] = 1
    if p["top_k"] == -1:
        p["top_k"] = TOP_K_ALL
    return p


def path_for_batch(requests, enable_deterministic=False):
    """The sampler path for one decode batch. `requests` are normalized parameter dicts.

    Returns "greedy", "gumbel" (the v0.5.20 fast path), "deterministic" or "filtered".
    """
    if not requests:
        raise ValueError("a batch needs at least one request")
    is_all_greedy = all(r["top_k"] <= 1 for r in requests)
    need_top_p = any(r["top_p"] != 1.0 for r in requests)
    need_top_k = any(r["top_k"] != TOP_K_ALL for r in requests)
    need_min_p = any(r["min_p"] > 0 for r in requests)
    if is_all_greedy:
        return "greedy"
    if not (need_top_p or need_top_k or need_min_p):  # simple_sampling_case
        return "deterministic" if enable_deterministic else "gumbel"
    return "filtered"

Your model's generation_config decides, and only on the chat endpoint

SGLang's --sampling-defaults defaults to model: the server reads temperature, top_k, top_p, min_p and repetition_penalty out of the model's generation_config.json (ModelConfig.get_default_sampling_params). They're applied in OpenAIServingChat, whose request conversion documents the order as user value, then model generation_config, then OpenAI defaults. /v1/responses is served by a subclass of it and takes the model's temperature and top_p too. /v1/completions and /generate never read the file, so they keep top_p = 1.0 and top_k = -1 and stay on the fast path, unless you started the server with --preferred-sampling-params, which the tokenizer manager merges into every request on every endpoint.

Here is that merge for the chat endpoint, from the same port:

# ChatCompletionRequest._DEFAULT_SAMPLING_PARAMS; /v1/completions and /generate use the same
# numbers from their own defaults, but never read the model's generation_config.
OPENAI_DEFAULTS = {"temperature": 1.0, "top_p": 1.0, "top_k": -1, "min_p": 0.0}
MERGED_KEYS = ("temperature", "top_k", "top_p", "min_p")  # model_config.get_default_sampling_params()


def effective(request, endpoint="chat", generation_config=None, sampling_defaults="model"):
    """The sampling parameters the scheduler will see, before normalization.

    endpoint: "chat" for /v1/chat/completions, "generate" for /generate and /v1/completions.
    sampling_defaults: the server's --sampling-defaults ("model" is the default).
    """
    from_model = {}
    if endpoint == "chat" and sampling_defaults == "model" and generation_config:
        from_model = {k: generation_config[k] for k in MERGED_KEYS if generation_config.get(k) is not None}
    out = {}
    for key in MERGED_KEYS:
        if request.get(key) is not None:
            out[key] = request[key]
        elif key in from_model:
            out[key] = from_model[key]
        else:
            out[key] = OPENAI_DEFAULTS[key]
    return out

Pass a request body's sampling fields (missing ones as None) through effective, then normalize, then put the results in a list for path_for_batch.

Qwen's documentation recommends Temperature 0.6, TopP 0.95, TopK 20 and MinP 0 for thinking mode and calls those "the default setting in generation_config.json". Serve a Qwen3 model that ships that file, send a plain chat completion, and need_top_k_sampling is true before your request reaches the scheduler. The model in the release note is the exception, but only half of one: Qwen/Qwen3.5-2B had no generation_config.json when we checked on 26 October, so plain chat requests to it do land on the new path. Its model card, though, recommends four sampling configurations, and every one of them sets top_k=20. Clients that follow the card are on the filtered path anyway.

To see what your own server merged, grep its startup log for "Using default chat sampling params from model generation config", which OpenAIServingChat logs once with the dictionary. No line, no merge.

What a free T4 can and cannot tell you

The plan, written before the run: time both versions' rules on the same probability tensors at batch 1, 8, 32 and 64 and at vocabulary 32,768 and 151,936, once measuring the host-side call alone and once synchronizing the GPU, 200 calls a cell after 30 warm-ups, as medians with their interquartile spread; download both wheels to confirm the timed lines are the published code; compare the two rules' distributions over 200,000 draws; then install v0.5.20 and run a decode benchmark with the switch on and off.

The session gave us a Tesla T4 (sm_75, 15,360 MiB, driver 580.178.04), four vCPUs and torch 2.11.0+cu128. Both wheels still carry the lines printed above, so these numbers come from published code. The new rule landed 0.0023 from the target distribution in total-variation distance, and SGLang's two spellings of the trick picked the same token in 100% of 1,024 rows on shared randomness. Qwen/Qwen3.5-2B answered 404 for generation_config.json, which checks the paragraph above.

At Qwen3.5-2B's vocabulary of 151,936 tokens, per sampler call, median of 200 calls with every interquartile spread under 25%:

Batch	v0.5.19 multinomial	v0.5.20 Gumbel-max	Saved per call
1, host-side only	143.2 µs	45.1 µs	98.1 µs
1, GPU synchronized	170.6 µs	61.9 µs	108.7 µs
32, host-side only	693.5 µs	46.1 µs	647.4 µs
32, GPU synchronized	725.4 µs	628.1 µs	97.3 µs

Three separate sessions produced the same table within 10%.

The host-side column is the mechanism the patch targets, and it behaves as the code comment says: the new rule's host cost stays near 46 µs at every batch size, while multinomial's climbs. That's what you'd expect if its validity check makes the host wait for the device: at batch 32 its host-side time (693.5 µs) is almost all of its synchronized time (725.4 µs). The synchronized column is what a decode step pays, and there the saving is about 0.1 ms a step at both batch sizes.

That is where the claim and our measurement part company. A 0.109 ms saving becomes +75% decode throughput only if the old step cost about 0.25 ms (0.109 × 1.75 ÷ 0.75), near 4,000 tokens a second on one stream. We measured no step: pip install sglang==0.5.20 would not build in the session, because outlines_core has no wheel for its Python 3.13. So the verdict is narrow on purpose. The mechanism holds and is worth about 0.1 ms a decode step here; the +75% as worded is unverified rather than disproved, and a saving in host work scales with the host, which on Kaggle is four vCPUs and not the authors' machine. Nothing here speaks to Hopper, Blackwell or multi-GPU setups.

Gotchas

Benchmark with temperature 0 and you measure nothing. SGLang's own one-batch benchmark defaults --temperature to 0.0, which normalizes to greedy, which never calls the changed function. Qwen's docs separately tell you not to use greedy decoding with their models.

Comparing wheels is not comparing the patch. Between the two tags sampler.py alone gained 288 insertions across five commits, sampling-mask capture among them. To measure this change, flip SGLANG_OPT_USE_GUMBEL_SAMPLE inside one install.

Expect different tokens, not different quality. The two rules draw from the same distribution but consume randomness differently, so a test that pinned exact token IDs will diff after the upgrade. If you need reproducibility, --enable-deterministic-inference moves you to the seeded path, which also forces the pytorch sampling backend and skips this change.

Do this today

Run the port above against your real request bodies and your model's generation_config.json, and include the other traffic that shares your server: one request with top_k, top_p, min_p or temperature 0 in the running batch is enough. If it prints filtered, this release note isn't about your server.
If it prints gumbel, measure your own gain: start the same build twice, once with SGLANG_OPT_USE_GUMBEL_SAMPLE=0, and compare decode tokens/s.
To get the fast path on the chat endpoint, send top_p: 1.0 and top_k: -1 explicitly, or start the server with --sampling-defaults openai and drop the model author's recommended settings. Then serve greedy and filtered traffic from another instance, because one such request holds the whole running batch off the fast path until it drains.

We also timed both rules at batch 8 and 64, and at a 32,768-token vocabulary next to Qwen3.5-2B's own 151,936, in the same run. Ask in the comments and we will post that table with its spreads.

Which vendor performance claim would you like checked next, and on what hardware?

Checked against SGLang v0.5.19 (0bcd8223), v0.5.20 (94602c9c), the v0.5.20 release notes and PR #38117's commit be00a543. InferPrice ran the sampler rules, the distribution checks and the wheel comparison on a free Kaggle Tesla T4 on 26 October 2026. We ran no SGLang server anywhere, so no end-to-end decode throughput here is ours, and we measured no Hopper, Blackwell or multi-GPU path.

Sources

SGLang v0.5.20 release notes: https://github.com/sgl-project/sglang/releases/tag/v0.5.20
The change itself, commit be00a543 (PR #38117), and python/sglang/srt/layers/sampler.py at tags v0.5.19 and v0.5.20: https://github.com/sgl-project/sglang
SGLANG_OPT_USE_GUMBEL_SAMPLE in python/sglang/srt/environ.py, the predicate in python/sglang/srt/sampling/sampling_batch_info.py, and the defaults in python/sglang/srt/sampling/sampling_params.py, all at v0.5.20
filter_batch and merge_batch in python/sglang/srt/sampling/sampling_batch_info.py, and the running-batch merge in python/sglang/srt/managers/scheduler.py, at v0.5.20
OpenAIServingResponses in python/sglang/srt/entrypoints/openai/serving_responses.py, and --preferred-sampling-params in python/sglang/srt/managers/tokenizer_manager.py, at v0.5.20
--sampling-defaults in python/sglang/srt/arg_groups/fields/serving.py, ModelConfig.get_default_sampling_params in python/sglang/srt/configs/model_config.py, and the merge order in python/sglang/srt/entrypoints/openai/protocol.py
--temperature default in python/sglang/benchmark/one_batch_server.py at v0.5.20
Qwen's recommended sampling parameters: https://qwen.readthedocs.io/en/stable/getting_started/quickstart.html
The Qwen/Qwen3.5-2B model card's recommended sampling parameters: https://huggingface.co/Qwen/Qwen3.5-2B
The Qwen/Qwen3.5-2B file list, checked 26 October 2026: https://huggingface.co/Qwen/Qwen3.5-2B/tree/main

Originally published at inferprice.pages.dev.

llama.cpp's MoE cache nearly doubled 30B decode speed on a free T4

sam ros — Sun, 18 Oct 2026 11:02:14 +0000

If your MoE model is too big for your GPU, llama.cpp fills VRAM with as many whole layers of experts as fit and runs the rest on the CPU, from system RAM. Every token then waits on the slowest part of your machine. Since build b11480 there's a better use for that VRAM: a cache of the experts the model actually keeps picking, from every layer.

We ran it on a free 16 GB Tesla T4 with Qwen3-30B-A3B, an 18.6 GB model that doesn't fit on the card. Decode went from 22.6 tokens/s with llama.cpp's default placement to 43.2 tokens/s with every expert in RAM and a 10,000 MiB cache. The llama.cpp developer who wrote the feature measured 2.2x on an RTX 5090 with a 93.7 GiB model that doesn't fit an 80 GB H100 at all. Holding that in VRAM on one GPU takes a 141 GB H200, listed at $5.29 to $6.31 per GPU-hour on on-demand cloud VMs in InferPrice's October 17 data.

Hits come from VRAM, misses cross PCIe

The cache went into llama.cpp's master branch on October 7 as commit d6cf9ac, squashed from PR #29887 (whose page still said "Open" on October 17), and shipped in build b11480. Multi-GPU support followed a day later in b11507. We checked everything here against b11712. The flag is --moe-cache-mib N (env LLAMA_ARG_MOE_CACHE_MIB), default 0, and llama.h still marks it experimental.

The mechanics, from src/llama-moe-cache.cpp: llama.cpp takes the layers whose experts all live in host memory, groups those whose expert tensors share types and shapes, and gives each group a pool of slots in VRAM. When the router picks experts, hits are read straight from VRAM and only misses are uploaded over PCIe, evicting the least recently used. A pool is shared, so a hot expert in layer 3 can push out a cold one in layer 40. Only the ffn_*_exps tensors are cached, not the router, attention or shared experts.

The catch is batch size. The cache only serves batches of 32 tokens or fewer, and only if the batch's experts fit in the slots at once. Bigger batches, which means prompt processing, take the old offload path, though they copy any experts that happen to be cached instead of uploading them.

1. Measure your experts before you pick a number

You need three things from your model: how many bytes of experts it has, how big one cache slot is, and which layers share a pool. The GGUF header has all three. This stdlib script reads it and repeats llama.cpp's slot arithmetic, so you see what a budget buys before you load anything:

"""Size llama.cpp's --moe-cache-mib from a GGUF model (standard library only).

Usage: python3 moe_cache_plan.py model-00001-of-00002.gguf [model-00002-of-00002.gguf] [--mib N] [--n-cpu-moe N]
Models one CUDA or ROCm GPU with --cpu-moe (or --n-cpu-moe N), as in llama.cpp b11712.
"""
import argparse
import re
import struct

EXPS = re.compile(r"blk\.(\d+)\.ffn_(gate_up|gate|up|down)_exps\.weight$")  # what the cache holds
ALIGN = 128  # ggml's CUDA buffer alignment (ROCm builds use the same code)
SCALAR = {0: "B", 1: "b", 2: "H", 3: "h", 4: "I", 5: "i", 6: "f", 7: "?", 10: "Q", 11: "q", 12: "d"}


def read_gguf(path):
    """Return (metadata, {tensor name: (dims, ggml type, bytes)}) from one GGUF file or split part."""
    with open(path, "rb") as f:
        def rd(fmt):
            return struct.unpack("<" + fmt, f.read(struct.calcsize(fmt)))[0]

        def string():
            return f.read(rd("Q")).decode("utf-8", "replace")

        def value(vtype):
            if vtype == 8:
                return string()
            if vtype == 9:
                etype, n = rd("I"), rd("Q")
                return [value(etype) for _ in range(n)]
            return rd(SCALAR[vtype])

        if f.read(4) != b"GGUF":
            raise ValueError(f"{path}: not a GGUF file")
        _version, n_tensors, n_kv = rd("I"), rd("Q"), rd("Q")
        meta = {}
        for _ in range(n_kv):
            key = string()
            meta[key] = value(rd("I"))
        infos = []
        for _ in range(n_tensors):
            name = string()
            dims = [rd("Q") for _ in range(rd("I"))]
            infos.append((name, dims, rd("I"), rd("Q")))  # sizes come from offsets: no table of quant types to go stale
        align = meta.get("general.alignment", 32)
        data_start = -(-f.tell() // align) * align
        data_len = f.seek(0, 2) - data_start
    ends = sorted(o for *_, o in infos) + [data_len]
    nxt = dict(zip(ends, ends[1:]))
    sizes = {}
    for name, dims, gtype, off in infos:
        size = nxt[off] - off
        rows = dims[1] * dims[2] if len(dims) == 3 else 1
        sizes[name] = (dims, gtype, size - size % rows)  # experts are rows x row bytes; their padding is < rows bytes
    return meta, sizes


def pad(n):
    return -(-n // ALIGN) * ALIGN


def plan(paths, budget=None, n_cpu_moe=None):
    """budget is in bytes; None means 10% of the model's expert bytes, rounded down to whole MiB.
    n_cpu_moe=None models --cpu-moe (every expert in host memory); N models --n-cpu-moe N (blk.0 to blk.N-1)."""
    meta, tensors = {}, {}
    for p in paths:
        m, t = read_gguf(p)
        meta.update(m)
        tensors.update(t)
    arch = meta["general.architecture"]
    n_expert, k = meta[f"{arch}.expert_count"], meta[f"{arch}.expert_used_count"]
    layers = {}  # layer -> [(type, ne0, ne1, bytes per expert)] of its expert tensors
    for name in sorted(tensors):
        m = EXPS.match(name)
        if m:
            dims, gtype, size = tensors[name]
            layers.setdefault(int(m.group(1)), []).append((gtype, dims[0], dims[1], size // dims[2]))
    if not layers:
        raise SystemExit("no MoE expert tensors found: the MoE cache only helps MoE models")
    total = sum(n_expert * sum(t[3] for t in ts) for ts in layers.values())
    groups = {}  # same types and shapes = same cache banks, as in llama.cpp
    for il, ts in sorted(layers.items()):
        if n_cpu_moe is None or il < n_cpu_moe:
            groups.setdefault(tuple(ts), []).append(il)
    if not groups:
        raise SystemExit("no MoE layer keeps its experts in host memory, so llama.cpp disables the cache")
    host = sum(len(ls) * n_expert * sum(t[3] for t in ts) for ts, ls in groups.items())
    budget = total // 10 // 2**20 * 2**20 if budget is None else budget
    out = {"arch": arch, "n_expert": n_expert, "k": k, "layers": len(layers), "expert_gib": total / 2**30,
           "host_gib": host / 2**30, "budget": budget, "groups": []}
    for ts, ls in groups.items():
        sz = [t[3] for t in ts]
        g_host = len(ls) * n_expert * sum(sz)
        share = int(budget * g_host / host)  # split by expert bytes, as in llama.cpp
        slots = 0
        while slots < len(ls) * n_expert and sum(pad(x * (slots + 2)) for x in sz) <= share:
            slots += 1  # each bank keeps one spare slot at the end
        slots = slots if slots >= k else 0
        max_batch = max([n for n in range(1, 33) if min(n * k, n_expert) <= slots], default=0)
        need = -(-sum(pad(x * (k + 1)) for x in sz) * host // g_host)  # smallest budget that gives this group k slots
        out["groups"].append({"layers": len(ls), "slots": slots, "pct": 100 * slots / (len(ls) * n_expert),
                              "max_batch": max_batch, "min_bytes": need})
    return out


if __name__ == "__main__":
    ap = argparse.ArgumentParser(description="Size llama.cpp's --moe-cache-mib from a GGUF model.")
    ap.add_argument("gguf", nargs="+", help="the model file, or every part of a split model")
    ap.add_argument("--mib", type=int, help="the --moe-cache-mib to check (default: 10%% of the experts)")
    ap.add_argument("--n-cpu-moe", type=int, help="model --n-cpu-moe N instead of --cpu-moe")
    a = ap.parse_args()
    r = plan(a.gguf, None if a.mib is None else a.mib * 2**20, a.n_cpu_moe)
    print(f"{r['arch']}: {r['layers']} MoE layers, {r['n_expert']} experts, {r['k']} used per token, "
          f"{r['expert_gib']:.2f} GiB of experts" + ("" if a.n_cpu_moe is None else f", {r['host_gib']:.2f} GiB on the CPU"))
    print(f"--moe-cache-mib {r['budget'] // 2**20}" + ("" if a.mib is not None else "  (10% of the experts)"))
    for i, g in enumerate(r["groups"], 1):
        print(f"  group {i}: {g['layers']} layers, {g['slots']} slots ({g['pct']:.1f}%), "
              f"batches up to {g['max_batch']} tokens use the cache, cached at all from {-(-g['min_bytes'] // 2**20)} MiB")

Run it on your model (pass every split part) to get the 10% starting point, or with --mib to test a budget. Add --n-cpu-moe N if that's how you run, because then only the first N layers are cached:

python3 moe_cache_plan.py Qwen3-30B-A3B-Q4_K_M.gguf --mib 10000

It prints one line per cache group: the slots you'll get, their share of the experts, the largest batch that can use them, and the budget below which the group isn't cached at all. For Qwen3-30B-A3B Q4_K_M that's 16.35 GiB of experts (48 layers, 128 experts, 8 per token) in two groups of 24 layers, because the Q4_K_M mix doesn't quantize every layer's experts the same way. 10,000 MiB buys 1,834 slots per group (59.7%). On the T4, llama-server's startup lines reported exactly those slots and 16,740 MiB of host experts, and they matched again with --n-cpu-moe 32.

The floor matters. Below it, llama.cpp leaves that group uncached with only a warning (MoE cache budget is too small for N layers, they are not cached), and if no group qualifies, llama-server won't start. Our model's floor is 50 MiB, and at 49 MiB the T4 run stopped with "MoE cache is too small to hold the experts of one token".

2. Give the VRAM to the cache, not to whole layers

There are two ways to run it. The easy one keeps llama.cpp's automatic placement (--fit, on by default), which decides how many expert layers stay in VRAM, and it counts the cache: it sizes things with a dry-run context built from your flags, and the cache reports its VRAM into that context's memory breakdown. On our T4, --fit kept 1,670 MiB less of the model in VRAM (11,952 down to 10,283 MiB) to make room for a 1,671 MiB cache.

llama-server -m model-00001-of-00003.gguf --moe-cache-mib 6544 --port 8080

The faster one, on both of the developer's test machines and on our T4, puts every expert in RAM with --cpu-moe (-cmoe for short) and hands the freed VRAM to the cache:

llama-server -m model-00001-of-00003.gguf --cpu-moe --moe-cache-mib 19000 -c 16384 --port 8080

I'd start with the second form: a whole layer in VRAM only helps when the router picks one of its experts, while the cache holds whichever experts are hot, from every layer. On the T4 it nearly doubled decode, where a 10% cache under --fit barely moved it (23.6 against 22.6 tokens/s). But you take over the sizing. --cpu-moe is a tensor override, and --fit won't move layers once one is set: it only checks that your settings fit (shrinking the context if you didn't pass -c), and otherwise gives up with a warning ending in tensor_buft_overrides already set by user, abort. Asking the T4 for 15,000 MiB got that warning, then "failed to allocate the MoE cache buffers". Leave room for the dense weights, the KV cache for your -c and the compute buffers.

Pass --port while you're upgrading. Build b11521 moved llama-server's default port from 8080 to 9931 (our T4 run without --port listened there), and the same commit pinned LLAMA_ARG_PORT=8080 in the official Dockerfiles, so containers keep 8080 while a bare binary or systemd unit moves.

3. You won't see the hit rate without -lv 4

The cache logs its size at startup and its hit rate at shutdown, and llama-server hides both. They're libllama INFO messages, which its logger files under verbosity 4, above the default threshold of 3 (common/log.cpp). Our first T4 run printed none of them. Add -lv 4:

llama-server -m model-00001-of-00003.gguf --cpu-moe --moe-cache-mib 19000 -c 16384 --port 8080 -lv 4

On the T4 that printed this at startup (timestamps cut), followed by one line per group with its layer count, quant type and slots:

I impl:      CUDA0 MoE cache size =  9999.32 MiB for 16740.00 MiB of host experts

And this when the server stopped:

I llama_moe_cache: ubatch <= 8: hits = 234520, misses = 10452, hit rate = 95.73%, uploaded = 28587.06 MiB
I llama_moe_cache: ubatch  > 8: hits = 2668, misses = 1785, hit rate = 59.91%, uploaded = 4862.46 MiB
I llama_moe_cache: large batches: 46881 experts copied from the cache, 42544.79 MiB

The hit rates appear only when the context is torn down, so you read them after stopping the server. ubatch <= 8 counts batches of up to 8 tokens (generation) and ubatch > 8 those of 9 to 32. Bigger batches don't use the slots, but when llama.cpp uploads host experts for them it copies already-cached ones on the GPU instead, and the last line counts those. -lv 4 prints plenty of other trace output, so filter for moe.

What a free T4 showed

Qwen3-30B-A3B Q4_K_M on one T4	Decode tokens/s	Prompt tokens/s	`ubatch <= 8` hit rate
`--fit` (default)	22.6	349
`--fit --moe-cache-mib 1674` (10%)	23.6	357	75.6%
`--cpu-moe`	9.6	150
`--cpu-moe --moe-cache-mib 50` (the floor)	3.8	152	0.0%
`--cpu-moe --moe-cache-mib 10000`	43.2	382	95.7%
the same, four requests at once	37.0 in total		92.3%

Decode is the mean of four 128-token replies at temperature 0 after a warm-up request, from llama-server's own timings, and prompt speed is one 1,436-token prompt. It's one run on one shared Kaggle VM (4 vCPUs, 31 GB of RAM), so don't read much into the decimals. --cpu-moe alone is slow because every expert runs on four CPU cores; with 10,000 MiB of cache it's the fastest, because 95.7% of expert lookups were served from VRAM. The 10% cache under --fit hit 75.6% of the time and still barely moved decode: it only covers the 20 layers --fit left in RAM, and --fit gave up resident layers to pay for it. Prompt processing got faster too, 382 against 150 tokens/s without the cache, and the cache's counter shows why: for those big batches llama.cpp copied 46,881 cached experts on the GPU instead of uploading them.

The developer's rule of thumb: start at 10%

The developer who wrote the feature found "10% of the total expert size is good" as a starting point, and more cache kept paying off on their RTX 4090 and 5090 (the 5090's 67.8 tokens/s came with 19,000 MiB and an 89% hit rate). That's one model, one prompt set and one host. Your gain depends on how often your prompts reuse experts and on PCIe bandwidth, since every miss is an upload, so treat 10% as where to start measuring.

Where it stops helping

Concurrency eats the cache. llama-server decodes all active requests together, one token each, with 4 slots by default. Four users generating at once means four tokens per step, each picking its own experts, so a step needs up to 4 times as many slots and churns the LRU harder. Past 32 tokens in a step the cache is skipped, and a step that also carries a new request's prompt is a large batch too. On the T4, four requests at once made 37.0 tokens/s between them, less than one request managed alone, and the hit rate slipped from 95.7% to 92.3%. This is a feature for one person or a small team on one box, not for a busy endpoint.

Speculative decoding grows the batch the same way, since verifying a draft feeds several tokens per request at once, and the draft model never gets a cache of its own.

On several GPUs, the budget is split like the layers (by --tensor-split, or by free memory), and llama.cpp turns off pipeline parallelism while the cache is on. Tensor-parallel split mode (-sm tensor) refuses to start with it; on our two T4s the error read MoE cache requires a GPU backend.

A budget near the floor is worse than none. At the script's "cached at all from" number, a group holds about one token's experts for one layer, and all its layers share that pool, so the next layer evicts them before they're used again. On the T4, 50 MiB gave 3.8 tokens/s against 9.6 with no cache, and 2 hits in 243,904 lookups.

Don't expect identical replies at temperature 0 either. With the cache, those experts run in CUDA kernels instead of on the CPU, and the results aren't bit-for-bit the same, so a greedy pick can flip and the text diverges from there. Under --cpu-moe, 1 of our 4 replies matched the no-cache run word for word and the others diverged after 15 to 367 characters; under --fit, 2 of 4 matched. Judge quality on your own evals.

Checks before you trust it

Run the script on your GGUF for your --moe-cache-mib. It models --cpu-moe and --n-cpu-moe; under --fit, only the layers left in RAM are cached.
Start llama-server with -lv 4 and confirm the startup lines show the same slots and no "budget is too small" warning.
Replay real prompts with --parallel set to what you'll run, stop the server, and read the ubatch <= 8 hit rate.
Compare decode tokens/s against the same replay with --moe-cache-mib 0.

Do this today

If you run an MoE model on one GPU that can't hold it, update to b11712 or later, run the script, and try --cpu-moe with most of your free VRAM as cache, plus -lv 4 so you can see what it hits. On a T4 that was one flag and a restart for nearly twice the decode speed.

What hit rate do you get on your own prompts, and does it hold up once a second user starts generating?

InferPrice ran the script and these commands on a free Kaggle T4 (16 GB, CUDA 12.8) on October 17, 2026, with llama.cpp b11712 built from source and Qwen3-30B-A3B Q4_K_M from Qwen's Hugging Face repo, and checked the rest against the b11712 source. The RTX 4090 and 5090 figures were measured by the llama.cpp developer who wrote the feature, in pull request #29887. The script is tested against GGUF files written by llama.cpp's own gguf-py and cross-checked against a line-by-line port of llama-moe-cache.cpp.

Sources

Originally published at inferprice.pages.dev.

vLLM 0.32 stops letting any website call your model server

sam ros — Wed, 14 Oct 2026 13:42:01 +0000

Run vLLM 0.31 with default flags and it tells every browser that any website may call it. A page on some unrelated site can have its JavaScript send chat requests to the vLLM server on your laptop or your office network, and read the answers, because the server replies Access-Control-Allow-Origin: *. vLLM 0.32 turns that off.

The change landed on October 11 in a commit titled "Restrict default CORS origins" and shipped in the 0.32.0rc0 release candidate on October 12. The current stable release, 0.31.0, still has the wildcard default, and so did releases as far back as 0.6.

The default flips from everyone to no one

The --allowed-origins flag now defaults to [] instead of ["*"], in both the Python server and the Rust frontend. vLLM hands the value straight to Starlette's CORSMiddleware, together with --allowed-methods, --allowed-headers and --allow-credentials, whose defaults (["*"], ["*"] and false) haven't changed. With an empty list, a browser's preflight gets a 400 with no Access-Control-Allow-Origin header, so the browser never sends the JSON POST that would follow.

Clients that aren't browsers don't do CORS. curl, the OpenAI SDK, your gateway and any agent running on a server keep working after the upgrade exactly as before.

Who is exposed today, and who breaks on upgrade

You're exposed on 0.31 if a vLLM server runs without --api-key anywhere a browser can reach it: localhost on a developer laptop, a GPU box on the office network, a cluster service reachable over the VPN. Any page that person opens can spend your GPU time and read the output. Setting --api-key helps, because the page can't supply the key, but the key only guards the /v1, /v2, /inference and /cohere prefixes. vLLM's security guide lists other inference routes, /invocations, /score and /pooling among them, that stay open even with a key.

You break on upgrade if a web page served from a different origin calls vLLM straight from the browser: an internal chat UI, a playground on another port, a notebook front end. Those calls start failing with a CORS error in the browser console. Origins have to match exactly on scheme, host and port, so http://localhost:3000 and http://127.0.0.1:3000 count as two different origins. The Python server passes no origin pattern to Starlette, so there's no *.example.com shortcut: list every origin.

Set the flag on 0.31 now

Don't wait for 0.32. Set the origins yourself today and the upgrade becomes a no-op. The value is a JSON list:

# Only these browser origins may call the server; everyone else is denied
vllm serve Qwen/Qwen3-8B \
  --api-key "$VLLM_API_KEY" \
  --allowed-origins '["https://chat.internal.example"]'

If no browser should ever talk to vLLM directly, which is the normal setup behind a gateway, pass --allowed-origins '[]' and you get the 0.32 behavior on 0.31.

Then check from the outside, the way a browser would. This preflight asks whether a page on another site may send an authenticated JSON POST:

curl -si -X OPTIONS http://vllm.internal:8000/v1/chat/completions \
  -H 'Origin: https://some-other-site.example' \
  -H 'Access-Control-Request-Method: POST' \
  -H 'Access-Control-Request-Headers: authorization,content-type' \
  | grep -i -e '^HTTP' -e 'access-control-allow-origin'

A locked-down server answers 400 with no access-control-allow-origin line. If you see *, or the origin you sent echoed back, any website can use that server.

CORS is not access control

CORS only decides what a browser lets a page do. Requests that don't need a preflight, such as a plain GET, still reach vLLM and run. The browser just hides the response from the page. In the test for this article, GET /v1/models from a foreign origin still returns 200 under the new default, only without the header. So keep --api-key, keep the server off the public internet, and put a reverse proxy in front that exposes only the routes you use, as vLLM's security guide recommends.

Never pair ["*"] with --allow-credentials. Starlette then echoes back whichever origin asked, which is the wildcard again, now with cookies allowed. List the origins instead.

Your ingress may be answering for vLLM. If NGINX, Envoy or an API gateway in front of the server adds its own CORS headers or handles OPTIONS itself, vLLM's default never comes into play. Run the curl check against the URL your users' browsers actually hit, not against the pod.

Do this today

Run that preflight against every vLLM endpoint you own, using the hostname a browser would see. Wherever it comes back with *, add --allowed-origins with the exact origins of your UIs, or '[]' if you have none, and roll the deployment.

Checked against the vLLM v0.32.0rc0, v0.31.0 and v0.6.0 source (cli_args.py, middleware/register.py) and the security guide in the repo. The CORS behavior was reproduced with Starlette 1.6.0 and uvicorn 0.53.0 in a stand-in app that wires CORSMiddleware with vLLM's arguments; the code and tests are in articles/examples/vllm-cors in this site's repo. InferPrice didn't run vLLM itself or test it on GPU hardware.

Sources

Originally published at inferprice.pages.dev.

Your MCP server probably accepts tokens that were never meant for it

sam ros — Tue, 13 Oct 2026 15:59:16 +0000

Your MCP server probably accepts tokens that were never meant for it. In the official Python SDK, that's what happens when you leave one setting unset. And the tool someone adds six months from now will probably skip your scope check, unless a test stops it.

This matters now because Atlassian's rebuilt MCP server is generally available, and it shows the pattern that's becoming the norm. It uses OAuth 2.1 with PKCE so the agent only gets the user's own permissions, it sorts tools into read, write and destructive, and it adds admin scopes and audit logging. You can build the same controls into your own server with the official Python SDK and the current MCP specification (2026-07-28). The example is a small "tickets" server, and the code below runs in CI against mcp 2.3.0, the current release.

Reject tokens issued for someone else

Your MCP server is an OAuth 2.1 resource server. It never logs anyone in. It validates tokens issued by your authorization server (Okta, Entra ID, Keycloak and the like). The spec makes two checks mandatory: servers MUST validate that a token was issued for them as the audience (RFC 8707), and MUST NOT accept or pass on any other tokens.

from pydantic import AnyHttpUrl
from mcp.server import MCPServer
from mcp.server.auth.provider import AccessToken, TokenVerifier
from mcp.server.auth.settings import AuthSettings

RESOURCE = "https://mcp.example.com/mcp"  # canonical URI of this server, no trailing slash

class JwtVerifier(TokenVerifier):
    async def verify_token(self, token: str) -> AccessToken | None:
        claims = validate_jwt(token)  # YOUR code: JWKS signature, exp, iss checks
        if claims is None:
            return None
        return AccessToken(
            token=token,
            client_id=claims["client_id"],
            scopes=claims.get("scope", "").split(),
            expires_at=claims["exp"],
            resource=claims["aud"],   # RFC 8707 resource indicator / audience
            subject=claims["sub"],
        )

mcp = MCPServer(
    "tickets",
    token_verifier=JwtVerifier(),
    auth=AuthSettings(
        issuer_url=AnyHttpUrl("https://auth.example.com"),
        resource_server_url=AnyHttpUrl(RESOURCE),
        required_scopes=["tickets:read"],
        validate_token_resource=True,
    ),
)

Set validate_token_resource=True explicitly. In mcp 2.3.0, leaving it unset while resource_server_url is set logs a warning and behaves as False, so tokens minted for other services get accepted. The SDK says 3.0 will flip the default. If your verifier already checks the audience, set it to False on purpose rather than leaving it unset. If your IdP issues multi-audience tokens, the SDK's docstring says to set resource to the entry that equals resource_server_url.

resource_server_url also drives discovery. The SDK uses it as the base route for OAuth Protected Resource Metadata (RFC 9728), which the spec requires so clients can find your authorization server from the WWW-Authenticate header on a 401.

Check scopes inside every tool

required_scopes gates the whole server. Real systems need more: read for search, write for updates, a separate scope for deletes. The SDK exposes the current token inside a tool:

from mcp.server.auth.middleware.auth_context import get_access_token
from mcp.server.mcpserver.exceptions import ToolError

def require_scope(scope: str) -> AccessToken:
    token = get_access_token()
    if token is None or scope not in token.scopes:  # no token fails closed
        raise ToolError(f"This action needs the '{scope}' scope. Re-authorize to grant it.")
    return token

Raise ToolError, not a generic exception. The SDK turns ToolError into an isError: true result the model can read and act on. Anything else is treated as a crash, and the model sees only Error executing tool <name>, which gives the agent nothing to recover from.

Keep the token is None branch. Writing the check as if token and scope not in token.scopes reads almost the same, but it lets every call without a token straight through.

At the HTTP level, the spec describes the full step-up flow: respond 403 with WWW-Authenticate: Bearer error="insufficient_scope", scope="...", resource_metadata="...", and list all the scopes the operation needs in one challenge, not one per retry. If a gateway in front of the server can see which tool is being called, issue the challenge there. The in-tool check is the minimum.

Annotations are hints, not security

from mcp.types import ToolAnnotations

@mcp.tool(title="Search tickets",
          annotations=ToolAnnotations(read_only_hint=True, open_world_hint=False))
def search_tickets(query: str) -> list[dict]:
    require_scope("tickets:read")
    ...

@mcp.tool(title="Close ticket",
          annotations=ToolAnnotations(read_only_hint=False, destructive_hint=False,
                                      idempotent_hint=True))
def close_ticket(ticket_id: str) -> str:
    require_scope("tickets:write")
    ...

@mcp.tool(title="Delete ticket",
          annotations=ToolAnnotations(read_only_hint=False, destructive_hint=True))
def delete_ticket(ticket_id: str) -> str:
    require_scope("tickets:admin")
    ...

Annotations let a client ask for confirmation before delete_ticket runs, the same read/write/destructive split Atlassian describes. But destructive_hint=False on close_ticket is a claim the client has to take on trust, and the spec tells clients to treat annotations from untrusted servers as untrusted. The SDK docs say not to rely on clients honouring them. Your scope check is the control. destructive_hint and idempotent_hint only mean something on tools that write, which is why the read-only tool leaves them out.

Hints aren't purely decorative, which is exactly why they can't be trusted. VS Code skips its confirmation prompt for tools marked readOnlyHint and asks for every other tool, so a write tool mislabelled as read-only runs with no prompt at all. Most clients do far less: in March the MCP project's own blog noted that no client surfaces annotations in its approval prompts. The strongest version of this rule isn't a hint. If a deployment doesn't need write tools, don't register them, and turn them on only through server config. A tool that doesn't exist can't be called.

Make the scope check impossible to forget

A scope check that lives in each tool body is only as good as the next person who adds a tool. So test the rule, not the tools you have today, and don't let annotations decide what gets tested. Declare the scope each tool needs next to its registration:

# tickets_server.py, next to the tools
REQUIRED_SCOPE = {"search_tickets": "tickets:read", "close_ticket": "tickets:write", "delete_ticket": "tickets:admin"}

Then enumerate the registry, check that the matrix covers exactly the registered tools, and call each tool with every scope except its own:

import asyncio
import pytest
from mcp.server import MCPServer
from mcp.server.auth.middleware.auth_context import auth_context_var
from mcp.server.auth.middleware.bearer_auth import AuthenticatedUser
from mcp.server.auth.provider import AccessToken
from mcp.server.mcpserver.exceptions import ToolError

from tickets_server import REQUIRED_SCOPE, TICKETS, mcp


async def as_caller(scopes, fn, *args):
    user = AuthenticatedUser(AccessToken(token="t", client_id="test", scopes=scopes))
    reset = auth_context_var.set(user)  # what the auth middleware does per request
    try:
        return await fn(*args)
    finally:
        auth_context_var.reset(reset)


def assert_every_tool_needs_its_scope(server, required, store):
    # enumerate the registry itself: a scope-filtered tools/list override can't hide a tool from this
    tools = asyncio.run(MCPServer.list_tools(server))
    assert {t.name for t in tools} == set(required), "every registered tool needs an entry in REQUIRED_SCOPE"
    all_scopes = sorted(set(required.values()))
    for tool in tools:
        need = required[tool.name]
        if tool.annotations and tool.annotations.read_only_hint:
            assert need == "tickets:read", f"{tool.name} is marked read-only but needs {need}"
        args = {p: "T-1" for p in tool.input_schema.get("required", [])}
        before = dict(store)
        with pytest.raises(ToolError, match="scope"):  # every scope except its own
            asyncio.run(as_caller([s for s in all_scopes if s != need], server.call_tool, tool.name, args))
        assert store == before, f"{tool.name} changed data without its scope"
        asyncio.run(as_caller([need], server.call_tool, tool.name, args))  # and its own scope is enough


def test_every_tool_needs_its_declared_scope():
    assert_every_tool_needs_its_scope(mcp, REQUIRED_SCOPE, TICKETS)

When someone adds reopen_ticket next year and forgets both require_scope and the matrix entry, the coverage line goes red before the tool ships. If they add the entry but not the check, the call with the wrong scopes doesn't raise and the test fails. If they mark it read_only_hint=True while it needs tickets:write, the annotation line fails. TICKETS is the example's stand-in for the ticketing backend, and comparing it before and after catches a tool that writes first and checks the scope second.

This version replaces an earlier test that picked its targets by annotation: it called every tool not marked read-only with a tickets:read token. A reader showed the hole. Mark a write tool read_only_hint=True, skip require_scope, and that test never calls it, so it passes while a read token changes data. The annotation decided what got tested, which is backwards when annotations are the thing you can't trust. The new test also calls MCPServer.list_tools(server) directly, so a scope-filtered tools/list override (see below) can't hide a tool from it, and it builds the scope list from the matrix rather than from a hard-coded list that goes stale when you add a scope. The example repo has a mutation test for each of these cases. You could instead scan the registry for tools without require_scope in their source, which is cheaper. But that only proves the call appears somewhere in the function, not that it runs before the side effect. The behavioral test is the one to keep.

Background work keeps a stale token

get_access_token() reads a context variable that the SDK's auth middleware sets for each request and resets when the request ends. What your code sees depends on how it starts extra work:

Started with	Sees the token?
A sync tool (`def`), which the SDK runs via `anyio.to_thread.run_sync`	Yes, AnyIO copies the context to the worker thread
`asyncio.create_task()`	Yes, and it keeps seeing it after the request ends
`asyncio.to_thread()`	Yes
`loop.run_in_executor()`	No, `None`
`threading.Thread`	No, `None`

The None cases are the safe ones: require_scope fails closed and denies. The dangerous case is the task that outlives its request. It authorizes with a snapshot of the token, even after that token has expired or been revoked. So check scopes in the handler before you spawn anything, pass the values the background work needs explicitly, and never call require_scope from a task that can outlive the request.

Spec details that cost you tokens

The spec lets tools/list vary by the authorization on the request, so you can list only the tools the caller's scopes allow. Be careful what you hide. The spec has clients start with the minimal scopes_supported set and add scopes through step-up when a call fails with insufficient_scope. A tool the model never sees is never called, so the challenge never fires and a user who would happily grant tickets:write is never asked. Hide only the tools behind scopes this caller can't get at all, such as delete_ticket for users who will never hold tickets:admin. That still saves prompt tokens, since every tool definition the client loads costs context on every turn, and it doesn't break the upgrade path for the rest.

Keep tool order stable. The spec says servers SHOULD return tools in the same order when the set hasn't changed, because it improves prompt-cache hit rates when tools sit in the model's context. A shuffled list quietly defeats prefix caching.

Never forward the user's token downstream. If close_ticket calls your ticketing API, use the server's own credentials or a token from a proper exchange. Passing the incoming token through breaks the spec's "MUST NOT ... transit any other tokens" rule.

Do this today

Add the scope-matrix test above to your server's CI. It takes a few minutes, it catches the mistakes annotations can't, and it keeps protecting you as the tool list grows. Then check your server for the four basics: no token gets a 401 with resource_metadata in WWW-Authenticate; the protected resource metadata is served at that URL; a token for a different audience gets a 401; and a tickets:read token calling close_ticket gets an isError: true result that names the missing scope.

Thanks to Arhan Canli, whose comment prompted the scope test and the background-task section, and to Edward Izgorodin and Moein, whose comments corrected the advice on hiding tools and hardened the test, and to tgapps, who showed that a write tool mislabelled as read-only slipped past the old test, which led to the scope matrix. Updated October 26, 2026. The code is tested in CI against mcp 2.3.0 on Python 3.13 (articles/examples/mcp-scopes in this site's repo). validate_jwt stands in for your identity provider's JWT validation. Pin your SDK version, because names like MCPServer replaced earlier ones.

Sources

Originally published at inferprice.pages.dev.