DEV Community: AI Tech News

Prefill/Decode Disaggregation in vLLM: The KV Transfer Math That Decides If It Pays Off

AI Tech News — Mon, 19 Oct 2026 14:03:51 +0000

TL;DR

Prefill/decode (P/D) disaggregation puts the two phases of LLM inference on separate engine instances. vLLM's own docs are blunt about the trade: it lets you tune time to first token (TTFT) and inter-token latency (ITL) independently and controls tail ITL, but "disaggregated prefill does not improve throughput."
The price you pay is moving the KV cache across a network for every request. That cost is a ratio you can compute before you buy a single NIC.
For a 70B-class model with 8 KV heads, the KV cache is 320 KiB per token in BF16. Against an illustrative prefill rate, shipping it over a 400 Gb link costs about 7% of prefill time, over 100 GbE about 30%. FP8 KV halves both.
Because both transfer and prefill scale linearly with prompt length, the ratio does not change with prompt length. What changes is how badly colocated prefill stalls your decode batch, and that grows with every long prompt.
Decision rule at the end, plus a script to measure the thing that actually matters: p99 ITL under mixed load.

All throughput and latency figures below come from arithmetic on public model configs and stated assumptions. They are marked illustrative. Measure on your own hardware before you act.

What problem does disaggregated prefill actually solve?

LLM inference has two phases with opposite hardware personalities.

Prefill processes the whole prompt in one forward pass. It is compute bound: large matrix multiplies, high arithmetic intensity, GPU tensor cores busy.

Decode generates one token per step per sequence. It is memory bandwidth bound: every step reads the full weights and the growing KV cache to produce a single token per request.

When both run on the same engine, they interfere. A long prompt arriving mid-flight injects a compute-heavy step into a batch of sequences that are all waiting for their next token. Every one of those sequences sees an ITL spike equal to roughly the duration of the prefill step. Users experience this as a stream that stutters whenever someone else submits a big document.

Chunked prefill is the usual first mitigation: split the prompt into chunks and interleave them with decode steps, so each stall is bounded by the chunk size. P/D disaggregation is the stronger one: prefill runs on its own instances, decode instances never execute a prefill step, and the decode step cadence becomes steady. The vLLM documentation states that this reduces tail ITL more reliably than chunked prefilling.

Two other benefits follow from separation. You can give each phase a different parallelism strategy (for example, more tensor parallelism on the prefill side to cut TTFT), and you can scale the two pools independently when your traffic mix shifts.

What does it cost? The KV transfer arithmetic

Once prefill happens elsewhere, the resulting KV cache has to reach the decode instance before the first token can be produced. The bytes to move per token are:

kv_bytes_per_token = 2 * layers * kv_heads * head_dim * dtype_bytes

The factor 2 is for keys and values. I pulled the shape values from the public config.json files of two open models:

Model	Layers	KV heads	Head dim	BF16 per token	FP8 per token
Llama-3.1-8B	32	8	128	128 KiB	64 KiB
Llama-3.1-70B	80	8	128	320 KiB	160 KiB

Grouped-query attention is why these numbers are tolerable. A 70B model with full multi-head attention (64 KV heads) would carry 8 times more.

An 8,192-token prompt on the 70B model therefore produces 2.68 GB of KV cache in BF16 (8,192 x 327,680 bytes). That is the payload for one request.

How long does the transfer take compared with prefill?

Here is the calculator I used. It is deliberately small so you can swap in your own model shape, link speed, and measured prefill throughput.

def kv_bytes_per_token(layers, kv_heads, head_dim, dtype_bytes):
    return 2 * layers * kv_heads * head_dim * dtype_bytes

def prefill_seconds(params_b, tokens, gpus, eff_tflops):
    # dense-model approximation: 2 * params FLOPs per token, attention term ignored
    return 2 * params_b * 1e9 * tokens / (gpus * eff_tflops * 1e12)

LINKS_GBPS = {"100GbE": 12.5, "200Gb": 25.0, "400Gb": 50.0}  # GB/s at line rate

tokens = 8192
pf = prefill_seconds(70, tokens, gpus=4, eff_tflops=400)      # assumption, replace with a measurement
size = tokens * kv_bytes_per_token(80, 8, 128, 2)             # BF16 KV
for name, gbs in LINKS_GBPS.items():
    t = size / (gbs * 1e9)
    print(f"{name}: {t*1000:.0f} ms = {t/pf*100:.0f}% of prefill ({pf*1000:.0f} ms)")

The assumptions are the weak point, so I will state them plainly. Illustrative inputs: a 70B dense model, four GPUs sharing the prefill, 400 TFLOPS of effective throughput per GPU, link speeds at line rate with no protocol overhead. Real prefill throughput depends on your kernels, tensor parallel efficiency, and sequence length. Replace eff_tflops with your own measurement.

With those inputs:

Prompt tokens	KV (BF16)	Prefill	100GbE	200Gb	400Gb
2,048	0.67 GB	179 ms	54 ms (30%)	27 ms (15%)	13 ms (7%)
8,192	2.68 GB	717 ms	215 ms (30%)	107 ms (15%)	54 ms (7%)
32,768	10.74 GB	2,867 ms	859 ms (30%)	429 ms (15%)	215 ms (7%)

The percentage column is identical in every row. That is not a coincidence. Both quantities are linear in prompt length, so the ratio collapses to:

transfer / prefill = (kv_bytes_per_token / link_bandwidth) / (2 * params / (gpus * flops))

Prompt length cancels. The practical consequences:

You can decide on hardware alone. Model shape, link bandwidth, and prefill throughput set the ratio. Traffic mix does not.
Long prompts are, if anything, better for disaggregation than this table suggests. Real prefill includes an attention term that grows faster than linearly, so long prompts take relatively longer to compute than to ship.
Tensor parallelism helps the transfer side too. KV heads are sharded across the prefill GPUs, so if each GPU has its own NIC path, the aggregate bandwidth scales with the group. The 50 GB/s figure above is a single 400 Gb link. Per-GPU links change the picture a lot, and it is worth checking your topology rather than assuming.
FP8 KV cache pays twice. It halves the memory for concurrent sequences and halves the bytes on the wire. If you already quantize the KV cache, the transfer ratio gets cheaper for free. I covered how to measure that quantization fairly in an earlier post (linked below).

A caveat on overlap: transfer is not necessarily serialized after prefill. Implementations can start moving layers or blocks while later ones are still being computed, which hides part of the cost. The numbers here are the worst case where everything waits. The upside from overlap is real but depends on the connector and your network.

Why does colocated prefill hurt decode so much?

The other side of the ledger is what you are paying to avoid. A decode step on a loaded engine takes tens of milliseconds. A prefill step of 8,192 tokens, under the same illustrative assumptions, takes 717 ms. During that time no sequence in the decode batch advances.

Chunking shrinks the stall (179 ms at 2,048 tokens, 45 ms at 512) but never removes it, and small chunks have their own cost: more steps per prompt, lower prefill efficiency, and a longer TTFT for the prompt being chunked. Disaggregation removes the interference entirely on the decode side. What it adds is the transfer step, which in the table above is 4% to 30% of a prefill that no longer blocks anyone.

So the real comparison is not "transfer cost versus nothing." It is transfer cost versus the p99 ITL damage of colocated prefill at your actual request mix. If your workload is short chat turns, the damage is small and the transfer is pure overhead. If it is long documents, long RAG contexts, or agent traces with big tool outputs, the damage can dominate your tail latency.

How do you set it up in vLLM?

The feature is configured with --kv-transfer-config, a JSON string that selects a connector. The vLLM documentation lists several connectors, including NixlConnector, MooncakeConnector, LMCacheConnectorV1, and OffloadingConnector, and marks the feature as experimental and subject to change. A documented NixlConnector example looks like this:

vllm serve <model> \
  --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both","kv_buffer_device":"cuda","kv_connector_extra_config":{"backends":["UCX","GDS"]}}'

Check the docs for your installed version for the exact role values per instance and for the proxy or router that sends the prompt to a prefill instance and then the decode instance. The interface has been moving.

It is moving in an interesting direction. In a recent post on the vLLM blog (October 14, 2026), the project describes a push-based NIXL connector. In the traditional pull model the decode instance requests KV from the prefill instance. In the push model the prefill instance writes KV directly into decode memory over RDMA. The stated goal is to remove three steps from the TTFT critical path: the router relay, decode-side allocation, and transfer setup. That matters for this article's math because those fixed costs are not in the bandwidth ratio above. At small prompt sizes, per-request setup can dominate the pure byte-transfer time, so the ratio table is a floor on overhead, not a ceiling.

How do you measure whether it helped?

Do not decide from average latency or from throughput. Disaggregation is a tail-latency tool, and the vLLM docs say it does not improve throughput, so a throughput benchmark will mislead you in both directions.

Measure p50 and p99 ITL while a background stream of long prompts is arriving. This client uses the OpenAI-compatible streaming endpoint and records the gap between consecutive chunks of each response:

import asyncio, json, time, httpx

async def stream_itl(client, url, model, prompt, max_tokens=256):
    gaps, last = [], None
    body = {"model": model, "prompt": prompt, "max_tokens": max_tokens,
            "stream": True, "temperature": 0}
    async with client.stream("POST", url, json=body, timeout=300) as r:
        async for line in r.aiter_lines():
            if not line.startswith("data: ") or line.endswith("[DONE]"):
                continue
            now = time.perf_counter()
            if last is not None:
                gaps.append(now - last)
            last = now
    return gaps

def pct(xs, q):
    xs = sorted(xs)
    return xs[min(len(xs) - 1, int(q * len(xs)))]

Run it with a fixed pool of "victim" short requests while a separate task injects long prompts at a constant rate, once against the colocated deployment and once against the disaggregated one. Report p50 and p99 ITL for the victims, plus TTFT for the long prompts.

Rules that keep the comparison fair, in the spirit of the benchmark traps I wrote about before:

Same total GPUs on both sides. Splitting 8 GPUs into 4 prefill and 4 decode must be compared with 8 colocated, not with 4.
Same arrival process and same prompt set, replayed from a file, not regenerated.
Warm the prefix cache identically, or disable it, before every run. A reader pointed this out under my KV cache quantization post, and it applies here with more force. A warm shared cache flatters whichever configuration runs second, and in a P/D setup it also changes how much KV actually needs to be transferred.
Dump the resolved engine config for both arms and diff them. Identical launch flags are not identical conditions.
Run long enough to hit steady state, and report the offered load, not just the latency.

When should you use it, and when should you skip it?

Situation	Verdict
Strict p99 ITL target, long or bursty prompts	Strong candidate
Different best parallelism for prefill and decode	Strong candidate
Fast interconnect between pools (RDMA, 200 Gb or more per path)	Favorable math
Short prompts, low concurrency	Skip, chunked prefill is enough
Offline batch jobs where only total throughput matters	Skip, no throughput gain
100GbE or slower between pools and no FP8 KV	Check the ratio first, 30% overhead is hard to hide
Small cluster where splitting pools leaves each too small to batch well	Skip, utilization loss can exceed the gain

The last row deserves emphasis. A decode pool needs enough concurrent sequences to fill its batches, and a prefill pool needs enough arrivals to stay busy. On a small cluster, splitting can leave both under-utilized. That cost does not show up in the transfer ratio at all.

FAQ

Does prefill/decode disaggregation improve throughput?
Not by itself. The vLLM documentation states that disaggregated prefill does not improve throughput. Its value is independent tuning of TTFT and ITL and tighter tail ITL control.

How big is the KV cache I have to transfer?
Use 2 * layers * kv_heads * head_dim * dtype_bytes per token. For Llama-3.1-70B in BF16 that is 320 KiB per token, about 2.68 GB for an 8,192-token prompt. FP8 KV halves it.

Is chunked prefill enough instead?
Often, for short prompts and moderate load. Chunking bounds the decode stall by chunk size but does not eliminate it. The vLLM docs say disaggregation controls tail ITL more reliably than chunked prefilling, at the cost of the KV transfer and a more complex deployment.

Which vLLM connector should I use?
The docs list several, including NixlConnector, MooncakeConnector, LMCacheConnectorV1, and OffloadingConnector. Choice depends on your interconnect and stack, and the feature is marked experimental, so pin your version and re-test on upgrade.

What is the difference between pull-based and push-based KV transfer?
In pull-based transfer the decode instance requests KV from the prefill instance. In push-based transfer the prefill instance writes KV directly into decode memory over RDMA, which the vLLM blog says removes the router relay, decode-side allocation, and transfer setup from the TTFT critical path.

On-Device LLM Inference: The Tradeoffs Nobody Puts on the Spec Sheet

AI Tech News — Mon, 12 Oct 2026 10:36:47 +0000

TL;DR

Running a language model on a device, a tablet, a handset, an edge box, is not just the cloud without the network. It is a different budget. Memory is fixed, precision is a lever, latency includes a cold start, and there is no autoscaler to hide your mistakes. This post walks through the tradeoffs that decide whether on-device inference is a demo or a product.

Why on-device at all

Three reasons push inference onto the device: data that cannot leave, environments with no reliable network, and latency that must not depend on a round trip. A factory floor, a secure facility, or a field device often has all three at once. When any of them holds, the cloud is not an option, and the question becomes how much capability you can fit locally.

Memory is the hard wall

In the cloud you rent more memory. On a device you do not. The model weights, the key-value cache for context, and the runtime all have to fit in what the device has, at the same time.

This makes context length a budget decision, not a free parameter. A longer prompt grows the cache, which competes with the weights for the same fixed memory. The practical move is to size context to the device, not to the model's theoretical maximum.

Precision is the main lever

Lower numeric precision shrinks the model and speeds it up, at some cost to quality. The art is choosing a precision that fits memory and still holds the behavior you need.

Two cautions. First, the same model at a lower precision can behave differently, so benchmark the precision you will actually ship, not a higher one you tested in the lab. Second, quality loss from quantization is not uniform across tasks, so test on your task, not a generic one.

Latency includes a cold start

Cloud latency is usually measured warm. On a device, the model often has to load into memory first, and that cold start can dominate the first user experience. A request that arrives before the model is ready will fail or wait.

The fix is to treat loading as a visible state. Show that the model is warming up, retry the first request with backoff, and do not report the warm-up error as a crash. A 503 during load is a normal signal, not a fault.

No autoscaler, no hiding

In the cloud a bad capacity estimate gets papered over by scaling. On a device there is nothing to scale into. That raises the bar on three things: measure real memory headroom before you ship, estimate throughput with the mean rather than the median so one slow case does not get lost, and watch which resource is actually saturating under load rather than trusting a ratio.

A short checklist

Does the model, cache, and runtime fit in device memory at your target context length?
Did you benchmark the exact precision you will ship, on your task?
Is cold start a visible, retried state rather than an error?
Did you size context to the device, not the model maximum?
Did you measure headroom and throughput under real load, not at idle?

FAQ

Is on-device always worth it? No. It is worth it when data cannot leave, the network is unreliable, or latency cannot depend on a round trip. Otherwise the cloud is simpler.

How low can precision go? As low as your task tolerates. That is an empirical question you answer by testing on your task, not a fixed number.

Why does cold start matter so much? Because it often dominates the first request, which is the one that shapes the user's impression.

What breaks most often? Memory. Context length quietly grows the cache until it collides with the weights.

KV Cache Quantization in LLM Serving: FP8 and INT8 Tradeoffs, the Silent config.json Trap, and How to Measure It Fairly

AI Tech News — Tue, 06 Oct 2026 19:10:48 +0000

TL;DR

The KV cache is often the real memory ceiling in LLM serving, not the weights. Quantizing it to FP8 or INT8 roughly halves KV memory, which lets you serve longer contexts or more concurrent requests on the same GPU. The catch: the quality impact is real but small when measured correctly, and it is easy to measure it wrong. A single kv_cache_dtype flag, or a scaling factor baked into a model's config.json, can quietly change your outputs while every dashboard still shows green. This post covers where the savings come from, what actually degrades, the silent-config trap, and a fair measurement protocol.

Why is the KV cache the thing you should quantize?

When a transformer generates tokens autoregressively, it stores the key and value tensors for every past token so it does not recompute them. That store is the KV cache, and it grows linearly with sequence length and batch size.

A rough per-token KV size:

bytes_per_token = 2 (K and V)
               * num_layers
               * num_kv_heads
               * head_dim
               * dtype_bytes

For a mid-size model with 32 layers, 8 KV heads (grouped-query attention), and head_dim 128, in FP16 that is:

2 * 32 * 8 * 128 * 2 = 131,072 bytes  (128 KB per token)

At 8K context that is 1 GB per sequence, before you batch anything. Weights are a fixed cost you pay once; the KV cache is a per-request cost that scales with your traffic. That is why it is the highest-leverage place to spend a quantization budget. Drop the dtype from 2 bytes to 1 byte and you halve that 128 KB to 64 KB per token, which directly buys you longer contexts or a bigger batch.

FP8 vs INT8 for the KV cache: what is the real tradeoff?

Both formats store each cached value in 8 bits, so the memory savings are effectively identical, about 2x versus FP16/BF16. The difference is in how they represent numbers and what that does to accuracy.

FP8 (typically e4m3 or e5m2) keeps an exponent field, so it covers a wide dynamic range with graceful precision falloff. The vLLM docs note that FP8 KV cache is supported on CUDA 11.8 and later and on ROCm, and that e4m3 is the common default on NVIDIA hardware. Because attention keys and values can have occasional large-magnitude outliers, the exponent bits help: an outlier does not blow out the whole tensor's scale.

INT8 uses a uniform grid with a per-tensor or per-channel scale. When the distribution is well behaved, INT8 can actually be more precise than FP8 inside its range because it does not spend bits on an exponent. The risk is outliers: one large value forces a coarse scale on everything else. INT8 KV usually wants calibration to pick good scales.

Practical reading:

FP8 is the lower-friction default. On recent NVIDIA and AMD GPUs it often needs no calibration and tends to be robust to outliers.
INT8 can match or beat FP8 quality with good per-channel scales, and it is useful where FP8 hardware paths are missing, but it asks more of you up front.

How much quality do you actually lose?

Less than people fear, if you measure the right thing. KV quantization only touches the cache, not the weights or the activations on the compute path, so the error is confined to attention's read of past tokens. In practice:

Short-context, factual tasks: differences are usually within benchmark noise.
Long-context retrieval and exact-copy tasks: this is where FP8/INT8 KV shows up first, because tiny per-token errors accumulate across thousands of cached positions.
Reasoning chains: a small per-step perturbation can flip a final answer even when average token accuracy barely moves, so pass@1 on reasoning sets is more sensitive than perplexity.

The headline number ("FP8 KV costs X percent") is meaningless without naming the task, the context length, and whether you measured a distribution or a single greedy run.

The silent config.json trap

Here is the failure that burns teams. You benchmark model A and model B with what looks like the identical serving command, same flags, same prompts, and you get different accuracy. The flags were identical. The conditions were not.

Two things commonly hide inside a model directory rather than on your command line:

A checkpoint-level KV cache dtype. Some published checkpoints ship a quantization config in config.json (for example a quantization_config or a KV-specific field) that sets the cache to FP8 even when your launch command says nothing about it. Your serving engine reads it and honors it. Your benchmark harness never sees it.
Baked-in KV scaling factors. FP8 KV can use static per-layer scales stored in the checkpoint. If one checkpoint has calibrated scales and another falls back to default scales, their attention precision differs even though both report "fp8 KV" in the logs. vLLM exposes --calculate-kv-scales to compute scales at runtime for the e4m3 path precisely so you are not silently depending on whatever shipped in the file.

The lesson that keeps recurring: identical flags are not identical conditions. The authoritative record of what ran is the resolved config inside the model folder plus the engine's startup log, not the command you typed.

A one-line guard before trusting any KV benchmark:

# Dump the resolved KV settings the engine actually used
python - <<'PY'
import json, glob
for f in glob.glob("models/**/config.json", recursive=True):
    c = json.load(open(f))
    qc = c.get("quantization_config", {})
    print(f, "| kv:", c.get("kv_cache_dtype"), "| qc_kv:", qc.get("kv_cache_dtype"))
PY

Then confirm against the server's own report:

vllm serve <model> --kv-cache-dtype fp8_e4m3 --calculate-kv-scales 2>&1 \
  | grep -iE "kv.cache|kv_scale|quantiz"

If the two disagree, the file wins, and your benchmark was comparing two different things.

How do you measure KV quantization fairly?

Treat it as an A/B where the only thing allowed to change is the KV dtype. Everything else gets pinned.

# Fair KV-quant comparison: pin everything except kv_cache_dtype
from vllm import LLM, SamplingParams

COMMON = dict(
    model="your/model",
    max_model_len=8192,
    gpu_memory_utilization=0.90,
    seed=1234,            # pin the seed
    enforce_eager=True,   # avoid graph-capture variance while measuring
)

sp = SamplingParams(temperature=0.0, max_tokens=512)  # greedy, fixed budget

baseline = LLM(**COMMON, kv_cache_dtype="auto")     # FP16/BF16 KV
fp8      = LLM(**COMMON, kv_cache_dtype="fp8_e4m3", calculate_kv_scales=True)

Rules that make the result trustworthy:

Pin the seed and the decoding params. Compare greedy to greedy, or run N samples per prompt and compare distributions. Never compare one greedy run to one sampled run.
Give reasoning models room. A max_tokens that is too small truncates the chain and manufactures a fake quality gap that has nothing to do with the KV cache.
Disable prefix caching, or warm both runs identically. A shared prefix cache can make the second configuration look faster or different for reasons unrelated to quantization.
Measure where it bites. Include a long-context retrieval task and a reasoning set, not just short QA. If your eval cannot tell FP16 from FP8, your eval is too easy, not your quantization too good.
Report two standard errors, not a point. KV-quant effects are small. A 0.4 point difference with an SE of 0.6 is noise, not a regression.
Separate memory from quality. Show the KV memory you saved and the accuracy you paid in the same table, because the whole point is the tradeoff.

A minimal results table makes the decision obvious:

Config	KV bytes/token	Max batch at 8K	Long-ctx retrieval	Reasoning pass@1
FP16 KV	128 KB	1x	94.1 +/- 0.5	71.2 +/- 0.6
FP8 e4m3 KV	64 KB	~2x	93.8 +/- 0.5	70.9 +/- 0.6

(Numbers above are illustrative placeholders; fill them from your own run.)

If the accuracy columns overlap within two standard errors and the batch column doubles, FP8 KV is a free win for that workload. If the long-context column drops outside the error bars, you have found the ceiling for that task and should keep FP16 KV there.

FAQ

Does KV cache quantization quantize the model weights too?
No. It only changes how cached keys and values are stored. Weights and the active compute path are untouched unless you separately apply weight quantization. That is why the quality impact is usually smaller than full weight quantization.

FP8 or INT8, which should I pick first?
Start with FP8 e4m3 on recent NVIDIA or AMD GPUs. It is robust to attention outliers and often needs no calibration. Reach for INT8 when FP8 hardware paths are unavailable or when you have invested in good per-channel calibration.

Why did two checkpoints give different accuracy with the same command?
Almost always a setting inside the model folder. A config.json KV dtype or baked-in KV scales override your intent silently. Dump the resolved config and the engine startup log before trusting the comparison.

What does --calculate-kv-scales do?
For the FP8 e4m3 path it computes the KV scaling factors at runtime instead of relying on whatever scales shipped in the checkpoint, which keeps your comparison honest and avoids depending on an unknown baked-in value.

Will KV quantization make serving faster, or just smaller?
Primarily smaller, which indirectly raises throughput: a smaller KV cache means a larger batch or longer context fits, and higher batch occupancy is where serving throughput actually comes from. Do not expect a large per-token latency drop by itself.

How do I know my eval is sensitive enough to detect a regression?
Include a task you know FP16 passes and a weak baseline fails. If FP8 KV and FP16 KV score identically on everything, verify the eval can discriminate at all before concluding there is no loss.

What Does It Actually Cost to Self-Host an LLM? The Batching Math Nobody Shows You

AI Tech News — Tue, 06 Oct 2026 00:43:30 +0000

TL;DR

Self-hosting an open-weight LLM is rarely expensive because the GPU is expensive. It is expensive because most people run the GPU at single-digit utilization. The headline rental price of an accelerator is fixed per hour, so your true cost per million tokens is set almost entirely by how many tokens you push through that hour. Continuous batching, a realistic input/output token mix, and honest utilization numbers move the price by more than 10x. This post gives you a back-of-the-envelope model you can plug your own numbers into, plus the three server settings that actually change the result.

Why is the per-hour GPU price the wrong number to quote?

When someone says "an H100 costs about 2 to 3 dollars per hour," they have quoted an input to the problem, not the answer. What you sell or consume is tokens, not GPU-seconds. The conversion factor between them is throughput, and throughput is not a property of the GPU alone. It is a property of the GPU plus the model plus the request pattern plus your server configuration.

The single formula that matters:

cost_per_1M_tokens = (gpu_hourly_cost / tokens_per_hour) * 1_000_000
tokens_per_hour    = throughput_tokens_per_sec * 3600 * utilization

Two deployments renting the identical GPU at the identical hourly price can differ by an order of magnitude in cost_per_1M_tokens, purely because one achieves 800 tokens per second at 70 percent utilization and the other achieves 90 tokens per second at 15 percent utilization. The hardware invoice is the same. The unit economics are not.

How much does continuous batching really change the math?

A naive server processes one request, waits for it to finish generating, then starts the next. The GPU spends most of its time idle between the compute-heavy prefill and the memory-bound decode steps. Continuous batching, popularized by the vLLM project, instead keeps a running set of in-flight sequences and injects new requests into the batch as soon as slots free up, token by token. The vLLM docs describe this as iteration-level scheduling, and it is the main reason a well-tuned open server reaches throughput that feels implausible if you have only ever benchmarked batch size 1.

Here is the intuition in a tiny simulator. It is not a GPU model, it just shows why aggregate throughput climbs with concurrency until something saturates:

def tokens_per_hour(single_stream_tps, max_concurrency, scaling_efficiency):
    # scaling_efficiency < 1 captures memory-bandwidth and KV-cache limits
    effective = single_stream_tps * max_concurrency * scaling_efficiency
    return effective * 3600

for c in (1, 8, 32, 64):
    tph = tokens_per_hour(42, c, scaling_efficiency=0.55)
    cost = (2.5 / tph) * 1_000_000   # $2.5/hr GPU, illustrative
    print(f"concurrency={c:>3}  tok/hr={tph/1e6:5.2f}M  $/1M={cost:6.2f}")

The numbers above are illustrative, not measured, but the shape is real: moving from one in-flight request to a few dozen can turn a 20-dollar price per million tokens into something close to 1 dollar. The cover chart shows the same relationship. The curve flattens once you hit a bottleneck, which is usually KV-cache memory rather than raw compute.

What caps the batch size before compute does?

The quiet ceiling is the KV cache. Every active sequence stores keys and values for all previous tokens, and that memory grows with context length and batch size. When the cache fills, the scheduler either queues new requests or preempts running ones, and your effective concurrency stops climbing.

A rough KV-cache size estimate for a transformer:

kv_bytes_per_token = 2 * num_layers * num_kv_heads * head_dim * bytes_per_elem
# factor of 2 is for keys AND values

Two levers shrink this and therefore raise the concurrency ceiling:

Grouped-query attention (fewer KV heads than query heads) directly reduces num_kv_heads, which is why many recent open models adopt it.
KV-cache quantization (storing the cache in FP8 instead of FP16) halves bytes_per_elem. vLLM exposes this through kv_cache_dtype.

If your p99 latency is fine but throughput plateaus early, you are almost certainly cache-bound, not compute-bound. Measure gpu_cache_usage in the server metrics before you reach for a bigger GPU.

Does quantizing the weights lower cost, or just fit a bigger model?

Weight quantization (for example 4-bit GGUF for llama.cpp, or AWQ and GPTQ for GPU serving) does two separate things, and people conflate them.

It reduces the memory the weights occupy, which frees room for a larger KV cache and therefore higher concurrency. This lowers cost per token.
It reduces memory-bandwidth pressure during the decode phase, which is bandwidth-bound. This can raise single-stream throughput.

What it does not reliably do is cut cost on a GPU you were already underutilizing. If you serve one user at a time, a 4-bit model on an idle GPU still produces an embarrassing price per million tokens, because the denominator (tokens per hour) is still tiny. Quantization pays off when you combine it with batching so the freed memory becomes extra concurrency.

GGUF on CPU is a related but different trade. llama.cpp can serve quantized models with no GPU at all, which is attractive for low-traffic internal tools where a GPU would sit idle. The crossover point is traffic volume: below some requests-per-minute threshold, a CPU box you already own beats a rented GPU you barely use. Above it, the GPU wins decisively because its tokens-per-hour ceiling is so much higher.

How do I estimate my own number in ten minutes?

You need four inputs, three of which you can measure and one you must assume:

gpu_hourly = 2.5          # your actual rental or amortized price
measured_tps = 650        # steady-state tokens/sec under realistic load
utilization = 0.60        # fraction of the hour the GPU is actually busy
overhead = 1.15           # networking, retries, idle head start, replicas

tph = measured_tps * 3600 * utilization
cost_per_1M = (gpu_hourly / tph) * 1_000_000 * overhead
print(round(cost_per_1M, 2), "USD per 1M tokens")

The one input people fake is measured_tps. Do not copy a vendor benchmark run at batch 256 with 128-token outputs if your workload is batch 4 with 1,000-token outputs. Output-heavy workloads spend far more time in the bandwidth-bound decode phase, so their tokens-per-hour is lower and their cost is higher. Benchmark with your own input/output length distribution or the number is fiction.

Which three settings move the result the most?

After tuning many open-model deployments, the settings with the largest effect on cost per token are consistently these:

max_num_seqs (or the equivalent max concurrency). Set it too low and you leave throughput on the table. Set it too high and you thrash the KV cache and trigger preemption. Sweep it against your real traffic.
gpu_memory_utilization. vLLM pre-allocates the KV cache from this fraction. Nudging it from 0.80 to 0.90 can meaningfully raise the concurrency ceiling, as long as you keep headroom for activation spikes.
kv_cache_dtype set to fp8. This is often the single cheapest way to raise concurrency on memory-bound models, with a quality impact that is small for many workloads but must be validated per model.

Everything else (speculative decoding, chunked prefill, tensor parallel degree) matters, but these three are where the first large wins live.

FAQ

Is self-hosting cheaper than a hosted API?
It depends entirely on utilization. A hosted API amortizes one GPU across thousands of tenants, so it runs at high utilization you would struggle to match. Self-hosting wins on cost only when your sustained traffic keeps your own GPU busy, or when data residency and latency, not price, are the reason.

Why is my cost per token so much higher than the blog posts I read?
Almost always because those posts report peak throughput at large batch sizes with short outputs, while your workload has long outputs and modest concurrency. Re-run the benchmark with your own token distribution.

Does a bigger GPU always lower cost per token?
No. A bigger GPU only helps if you can fill it. If you are already at low utilization, a larger accelerator raises your hourly cost while the tokens-per-hour barely moves, making the unit cost worse.

Where do continuous batching and quantization overlap?
Quantization frees memory, continuous batching turns that freed memory into extra concurrent sequences. Used together they compound. Used alone on an idle GPU, neither fixes the underlying utilization problem.

How do I know if I am compute-bound or memory-bound?
Watch KV-cache usage and GPU compute utilization together. If cache usage hits the ceiling while compute sits below 100 percent, you are memory-bound and should attack the cache (GQA models, FP8 cache, shorter contexts) before buying compute.

What is a realistic utilization target?
For bursty interactive traffic, sustained 50 to 70 percent is a reasonable and honest target. Claiming 95 percent usually means you are either batching offline jobs or quoting a benchmark, not a production SLA.

Five Ways Your LLM Serving Benchmark Is Lying to You (and How to Catch Each One)

AI Tech News — Mon, 05 Oct 2026 08:55:22 +0000

TL;DR

In my benchmarking work on vLLM serving stacks, every one of the following produced a confident, wrong conclusion before anyone noticed:

Prefix caching inflated an A/B test unequally. The harness reused the same prompt on every call. Cache hit rate was 67% on one arm and 45% on the other, and paired throughput came out 2.0 to 2.8x higher than a sweep run on the identical server.
The harness counted SSE chunks, not tokens. Under speculative decoding one chunk carries several tokens, so measured output was cut nearly in half (698 vs 362). On a 1:1 chat workload that error is about 20% of total throughput: enough to report a +49% win as a loss.
Identical CLI flags, different runtime. A config.json inside one model folder declared an 8-bit KV cache scheme, and vLLM silently enabled FP8 KV. KV capacity was 1.775x larger on that arm (940,736 vs 529,856 tokens).
Five knobs changed at once. Result: -16% to -22% on long-context reads, with no way to tell which knob did it.
A "stability-first" auto-selection gate picked the slow config. It was the only candidate with run-to-run spread under 3% because it was 20% slower everywhere.

Bonus: Restart=always turned a configuration bug into a silent crash loop that systemctl is-active reported as healthy.

Each section below gives the symptom, the mechanism, and a code-level fix.

Why does prefix caching inflate vLLM benchmarks, and why unequally?

vLLM enables Automatic Prefix Caching by default in recent V1 releases (docs). If two requests share a prefix, the second one skips prefill for the cached blocks. That is great in production and poison in a synthetic benchmark.

Here is what happened. A team I worked with ran a paired A/B of two serving configs and wanted one shot per configuration, so they used --reps 1 --warm 0. The harness derived the prompt seed from the batch index b. With one rep and no warmup, b was always 0. Every call sent the exact same prompt.

Measured prefix cache hit rate:

The clue was a ratio that made no sense: on the same server with the same settings, paired-comparison throughput was 2.0 to 2.8x the sweep throughput. At concurrency 32 on a long-context workload, the paired run reported 61,347 tok/s against 22,159 tok/s in the sweep.

The part most people miss is that the contamination is not symmetric. Cache capacity is whatever KV memory is left over, so the arm with more free KV blocks keeps cached prefixes alive longer and gets a larger bonus. You stop measuring "which config is faster" and start measuring "which config retains cache better". If the two arms have different memory headroom, that alone biases the result.

Fix: seed every call independently and log the hit rate

import random

LARGE_PRIME = 1_000_003

def make_prompt(seed: int, batch: int, idx: int, n_words: int) -> str:
    # Separate axes for run seed, batch and request index, so
    # no two calls in any phase or concurrency level share a prefix.
    rng = random.Random(seed * LARGE_PRIME + batch * 10_000 + idx)
    # Put the randomness at the START of the prompt; a shared system
    # prompt in front would still produce prefix hits.
    nonce = f"[req {rng.getrandbits(64):016x}] "
    words = [rng.choice(VOCAB) for _ in range(n_words)]
    return nonce + " ".join(words)

Then prove it. Scrape the server's Prometheus endpoint before and after each run and record the delta as part of the result artifact:

import re, requests

def prefix_cache_counters(base_url: str) -> dict:
    text = requests.get(f"{base_url}/metrics", timeout=5).text
    out = {}
    for name in ("vllm:prefix_cache_queries_total", "vllm:prefix_cache_hits_total"):
        m = re.search(rf"^{re.escape(name)}(?:{{[^}}]*}})?\s+([0-9.e+]+)$", text, re.M)
        out[name] = float(m.group(1)) if m else float("nan")
    return out
# hit_rate = d(hits) / d(queries) over the run window

(Metric names vary across vLLM versions; check your /metrics output.) If you want a cache-free number, start the server with --no-enable-prefix-caching, but still log the counters: "we disabled it" is a claim, the counter is evidence.

One more lesson from that week: before you discount a competitor's published numbers because "they use prefix caching and we don't", print your own hit rate. In this case the team had made exactly that argument while running at 67%.

Why does my tokens-per-second drop when I enable speculative decoding?

Sometimes it doesn't drop. Your counter does.

A common streaming harness pattern:

async for chunk in stream:          # OpenAI-compatible SSE stream
    if chunk.choices[0].delta.content:
        ntok += 1                   # BUG: counts chunks, not tokens

With normal decoding, one SSE event roughly equals one token, so this works by accident. Speculative decoding (EAGLE, MTP, n-gram drafting; see the vLLM speculative decoding docs) accepts several tokens per step, and the server emits them in a single chunk. The harness undercounts only the accelerated arm.

In the case I saw, measured output dropped from 698 to 362 tok/s for the speculative arm while total throughput moved only 6%. Those two numbers cannot both be true: you cannot halve output and keep total flat. That contradiction was the entrance to the bug.

Why didn't it show up sooner? The test was long-context with a 32:1 input:output ratio, so output tokens were a tiny share of the total and the error on total was about 1%. On a 1:1 chat-shaped workload, the same bug costs about 20% of total throughput. The real speculative result there was +49%; a 20% haircut on one arm is big enough to flip the headline.

Fix: count tokens from the source of truth

payload = {
    "model": MODEL,
    "prompt": prompt,
    "max_tokens": OUT_LEN,
    "ignore_eos": True,             # vLLM extension: fixes output length
    "stream": True,
    "stream_options": {"include_usage": True},
}
# The final SSE event carries usage.completion_tokens. Use it.
# Cross-check: with ignore_eos, completion_tokens must equal OUT_LEN.

Because ignore_eos=True plus max_tokens=N pins the output length, you can often recompute historical results without rerunning: true_total = in_tok_per_s * (in_len + out_len) / in_len, provided input token counts were exact. Apply the correction to every arm with the same formula. Correcting only the arm you suspect is a new contamination.

General rule: any counter that answers "how much" by counting "how many times" (chunks, callbacks, events) breaks under acceleration techniques, because those techniques work precisely by packing more into each event. Batching, speculative decoding and stream coalescing all do this, and each one breaks a different counter in a different direction. Log at least two quantities that must agree with each other.

Can identical vLLM command-line flags produce different runtime configs?

Yes, and command-line diffs will never show it.

Two quantized checkpoints of the same base model were served with scripts identical except for model path, port and GPU index. The comparison was presented as a clean weights-only A/B. Then someone diffed the startup logs:

Arm	KV cache dtype (from log)	KV cache capacity (tokens)
A	`kv_cache_dtype=auto` (16-bit)	529,856
B	`Using standard fp8 KV cache`	940,736 (1.775x)

The cause: arm B's config.json carried a quantization block with a kv_cache_scheme of 8 bits. vLLM reads quantization metadata from the checkpoint and enabled FP8 KV on its own (quantized KV cache docs). Arm A had no such entry. Same flags, different conditions.

The same report had a second confound: one row was "weights B + default settings", another was "weights A + tuned settings", and the conclusion said "tuning lost". Two axes moved at once; that sentence was never proven.

Fix: diff the realized config, not the intended one

# Run after both servers are up. Compare what the engine actually did.
for f in serverA.log serverB.log; do
  echo "== $f"
  grep -Ei "kv_cache_dtype|fp8 KV|GPU KV cache size|Maximum concurrency|attention backend|quantization=|tensor_parallel_size|max_num_seqs|enable_prefix_caching" "$f"
done

# And check the checkpoint itself before you start:
jq '.quantization_config | {quant_method, kv_cache_scheme}' modelB/config.json

If you want to force parity, pass --kv-cache-dtype explicitly on both arms rather than relying on auto. And label every row of a results table with both (weights, config), never just one.

A useful habit: when a difference appears, check whether it is a number you already know. That 1.775x matched an earlier experiment where increasing KV capacity by about 78% did not raise the concurrency ceiling at all. So "B wins because it has more KV" was ruled out in minutes.

Why should you tune vLLM settings one at a time?

An "apply the recommended profile" change modified five settings together: FP8 KV cache, --gpu-memory-utilization 0.95, lower --max-model-len, higher --max-num-seqs, and async scheduling. Result: -16% to -22% across every long-context point. Nobody could say which knob caused it, and each server restart cost about 10 minutes, so bisecting was expensive.

Serving knobs trade against each other (memory you gain, compute you may lose), so their effects have mixed signs and do not add up neatly.

Two things made the failure useful anyway:

The failure named the bottleneck. Growing KV capacity by 78% (523,264 to 933,120 tokens) left the concurrency-64 throughput ceiling unchanged. Therefore the bottleneck was compute, not memory. A success would never have told us that.
The same setting flips sign by workload. The identical profile was -22% on prefill-dominated long-context reads and +6.8% on a balanced read/write workload.

Fix: one axis per run, and ask the code before restarting

# Baseline, then vary exactly one flag per run.
BASE="--max-model-len 32768 --max-num-seqs 128 --gpu-memory-utilization 0.90"
for VARIANT in "" "--kv-cache-dtype fp8" "--max-num-seqs 256" "--async-scheduling"; do
  vllm serve "$MODEL" $BASE $VARIANT --port 8000 > "log_${VARIANT// /_}.txt" 2>&1 &
  wait_for_ready && run_bench --shape longctx && run_bench --shape chat
  kill %1; wait
done

Flag names drift across releases, so check vllm serve --help for your version. For expensive restarts (for example, choosing between attention backends) try to answer feasibility in-process before launching a server: in one case, calling the backend's support-check function directly classified four candidates in about a minute, after two full restarts had been wasted trying them one by one.

Always benchmark at least two workload shapes (prefill-heavy and decode-heavy). A single-shape result is half a result.

Why do "stability-first" auto-tuning gates pick the slowest config?

An auto-selection script chose among three batch-limit candidates with this rule:

Among candidates whose run-to-run spread is under 3% of peak, pick the one with the highest peak.

Only one candidate passed (spread 0.1%). The other two had spreads around 20%. The winner was stable because it was 20% slower at every point. Throttling a system removes variance along with throughput. With no performance floor, "do nothing" is always the most stable option.

The resulting combined profile lost to the best single change on four of five measured points: -13.0%, +2.0%, -3.1%, -3.3%, -13.9%.

The same gate had a second hole: "adopt if it wins on both shapes". Actual margins were +0.0% to +0.5%, all inside the noise, and they were adopted because the sign was positive.

This is the same mistake at a smaller scale that I have seen in model evaluation: a probe at 89.5% vs a baseline at 89.1% was marked "pass" because A > B. On 276 questions the standard error was about 1.9 points, the bootstrap 95% interval for the difference was [-4.3, +5.1] points, and McNemar z was 0.00. Not a win.

Fix: gates need a floor and an interval, written in code

import numpy as np

def bootstrap_ci(a, b, n=10_000, seed=0):
    """Paired bootstrap of mean(a - b); a, b are per-run (or per-item) values."""
    rng = np.random.default_rng(seed)
    d = np.asarray(a) - np.asarray(b)
    idx = rng.integers(0, len(d), size=(n, len(d)))
    means = d[idx].mean(axis=1)
    return np.percentile(means, [2.5, 97.5])

def accept(candidate_runs, baseline_runs, baseline_peak):
    peak = max(candidate_runs)
    spread = (max(candidate_runs) - min(candidate_runs)) / peak
    lo, _ = bootstrap_ci(candidate_runs, baseline_runs)
    return (
        spread < 0.03
        and peak >= 0.95 * baseline_peak   # performance floor
        and lo > 0                          # interval, not sign
    )

Two more rules: always measure the auto-selected combination next to the best single change (combos can be worse than their parts), and do not assume more knobs means more speed. Of the five knobs tested that day, one helped; the rest were neutral or harmful.

Can systemd Restart=always hide a broken benchmark worker?

Yes. Restart=always is medicine for transient failures. A configuration error is permanent, and the restart policy disguises it as a retry.

In one fleet, an automatic rebalancer assigned zero work slots to three workers. The generated unit line ended up with an empty argument, which the shell dropped entirely. The worker script ran with set -u, hit $4: unbound variable, and exited immediately. systemd plus a watchdog revived it every 15 seconds, forever. Load average went to 67 with every GPU at 0%, and systemctl is-active said active.

A related trap: a watchdog that "restores the service when the queue is idle" cannot tell idle from "I deliberately stopped things to run a benchmark". In one case it revived a 50 GB service onto the GPU under test, and the control server got roughly one tenth of its normal KV capacity (89,536 vs 919,232 tokens). Left alone, that would have produced a headline claiming speculative decoding was 10x faster.

Fix

[Unit]
StartLimitIntervalSec=300
StartLimitBurst=5          # after 5 failures in 5 min, stay failed and alert

[Service]
Restart=on-failure
RestartSec=15
ExecStartPre=/usr/local/bin/validate-args.sh %i   # fail loudly on empty args

Validate generated argument lists at generation time (raise on empty values), disable watchdogs and auto-restorers before a measurement window, and record nvidia-smi memory per GPU in the benchmark artifact so a stray tenant shows up in the data.

Pre-flight checklist for LLM serving benchmarks

[ ] Every request has a unique prompt prefix; seeds use separate axes for run, batch and request index.
[ ] Prefix cache hit rate is scraped from /metrics and stored with the results.
[ ] Output tokens come from usage.completion_tokens, not chunk counts; ignore_eos plus max_tokens pins length.
[ ] At least two quantities in each run must agree (output tok/s, total tok/s, completion_tokens x requests / wall time).
[ ] Startup logs of both arms are diffed for KV dtype, KV capacity, attention backend, parallelism and max sequences.
[ ] Checkpoint config.json is inspected for quantization_config and KV cache schemes.
[ ] Results table rows state both weights and config.
[ ] One knob per run; at least one prefill-heavy and one decode-heavy shape.
[ ] Acceptance gates use a confidence interval lower bound above zero, plus a performance floor relative to baseline.
[ ] Auto-selected combos are benchmarked against the best single change.
[ ] Watchdogs off during measurement; per-GPU memory snapshot recorded; restart policies have burst limits.

FAQ

Should I disable prefix caching for every vLLM benchmark?

Not necessarily. If production traffic shares system prompts, a cache-on benchmark is more realistic. What matters is that both arms see the same cache conditions and that you report the measured hit rate. Unique prompts with caching left on, plus a logged hit rate near zero, is a clean way to measure raw engine speed.

How do I count tokens correctly with streaming responses?

Request stream_options: {"include_usage": true} and read usage.completion_tokens from the final event. If you need per-token timing, tokenize the accumulated text with the model's tokenizer instead of counting events, and validate against the usage field.

How can I tell if vLLM enabled FP8 KV cache on its own?

Grep the startup log for the KV cache dtype and the reported KV cache size in tokens, and inspect quantization_config in the checkpoint's config.json. Passing --kv-cache-dtype explicitly on every arm removes the ambiguity.

How many repetitions do I need before declaring a winner?

Enough that the confidence interval of the difference excludes zero. Start with at least five runs per arm, compute a paired bootstrap interval, and treat any gap smaller than the observed run-to-run spread as a tie.

Is a lower-variance config always better for production?

No. Low variance is easy to buy by throttling throughput. Require a performance floor (for example, at least 95% of the best peak) before stability becomes a tiebreaker.

Why did my tuning help chat traffic but hurt long documents?

Prefill-dominated and decode-dominated workloads hit different bottlenecks. In the case above the same profile measured -22% on long-context reads and +6.8% on a balanced workload. Benchmark both shapes and report them separately.

A 4B Model Just Beat 8B — We Tested 18 Small LLMs and the Results Are Wild

AI Tech News — Tue, 10 Mar 2026 14:59:27 +0000

The "bigger is better" assumption is wrong.

We spent weeks evaluating 18 small language models from 12 different makers on 125 questions across 7 languages — and the results seriously challenge conventional wisdom about model scaling.

Here's what the data actually shows:

A 4B model outperforms an 8B model — using 36% of the RAM
A 1.5GB MoE model matches dense models that need 8.5GB
A 1.7B model beats three separate 7B–14B models
A 1.3B model fabricates fake content 80% of the time

These aren't theoretical predictions. These are measured results.

🤔 Why we built yet another benchmark

Here's the thing — MMLU, GPQA, and HumanEval weren't built for edge AI.

They give the same test to a 0.5B model and a 500B model. That's fine if you only care about "how smart is it?" But if you're deploying on a phone, a Raspberry Pi, or an 8GB laptop, you need to know:

Does it fit? → How much RAM does it actually need?
Does it lie? → How often does it fabricate fake information?
Is it fast enough? → How many tokens per second?
Is it worth the cost? → What's the performance per GB of RAM?

Existing benchmarks answer none of these. So we built one that answers all of them.

🏟️ Introducing SHIFT — 5 axes, not 1

Size · Honesty · Intelligence · Fast · Thrift

Axis	What it measures	How
S	How big is the model?	Parameter count, active params for MoE
H	Does it resist hallucination?	40 questions — traps, calibration, refusal, self-correction
I	How smart is it?	85 questions — reasoning, math, coding, 7 languages
F	How fast does it run?	tok/s measured via HF Inference API
T	How much resource does it need?	Peak RAM at Q4 quantization

All 125 questions require JSON-structured output with verifiable fields. No keyword matching. 75 questions are fully automatic — zero human grading needed.

📊 The ranking formula: WCS

The tricky part — how do you rank models when you're measuring both quality and efficiency?

SHIFT alone? Then 14B always beats 1.7B. Boring. Expected.

PIR (efficiency) alone? Then a terrible 1.3B model becomes #1 because it's tiny. Misleading.

Our solution: WorldCup Score (WCS) — the geometric mean of both:

WCS = √( SHIFT × PIR_norm )

Where:
  SHIFT    = H × 0.4 + I × 0.6       → quality
  PIR      = (I × H × F) ÷ (S × T)   → efficiency
  PIR_norm = log₁₀(PIR) / log₁₀(max) × 100

Why geometric mean? Because √(A × B) requires both to be high. Smart but huge? Low WCS. Tiny but dumb? Also low WCS. You need both quality and efficiency to rank well.

🏆 The results

Here are the top 5 — and they're not what you'd expect:

#  Model              WCS    SHIFT  RAM     League
🏆 GPT-OSS-20B       82.6   76.9   1.5GB   🥅 Raspberry Pi tier
🥈 Gemma-3n-E4B      81.8   77.3   2.0GB   ⚽ Smartphone tier
🥉 Llama-4-Scout     79.3   74.2   10GB    🏆 Desktop (but 240 tok/s!)
4  Qwen3-4B          76.6   76.8   2.8GB   ⚽ Smartphone tier
5  Qwen3-1.7B        76.1   66.8   1.2GB   🥅 IoT tier

The WCS champion runs on a Raspberry Pi. Let that sink in.

🔬 5 findings that surprised us

Finding 1: 4B = 8B (at 36% of the RAM)

Gemma-3n-E4B  → SHIFT 77.3  (4B,  2.0GB)  ← #1 quality!
Qwen3-8B      → SHIFT 76.9  (8B,  5.5GB)
                              Gap: 0.4 points
                              RAM: 2.75× more

Google's PLE architecture and Qwen3's training pipeline have made 4B models functionally equivalent to 8B on structured evaluation tasks. The extra 3.5GB of RAM buys you almost nothing.

Finding 2: MoE is the cheat code for edge AI

GPT-OSS-20B   → 21B total, 3.6B active, 1.5GB RAM → SHIFT 76.9
Gemma-3-12B   → 12B total, 12B active,  8.5GB RAM → SHIFT 75.7

Same quality. 5.7× less RAM. MoE models activate only a fraction of their parameters at inference time, giving you big-model knowledge with small-model resources.

Finding 3: Thinking models have a dark side

Models with <think> reasoning tokens (DeepSeek-R1, Nemotron-Nano) face a double penalty:

Quality hit — <think> tags break JSON structured output:

Qwen3-8B (non-thinking)    → SHIFT 76.9
DeepSeek-R1-7B (thinking)  → SHIFT 68.2  (−8.7 points!)

Speed hit — internal reasoning = 2–6× more tokens generated:

Qwen3-8B        → 186.8 tok/s
DeepSeek-R1-7B  →  69.2 tok/s  (2.7× slower)
Nemotron-Nano   →  29.8 tok/s  (6.3× slower)

Thinking helps for complex math (DeepSeek-R1-14B's reasoning score is the highest we measured), but for real-time structured tasks, non-thinking models win.

Finding 4: The hallucination gap is enormous

Our H1 test presents fake people, papers, and products. Models must refuse to fabricate.

The score range? 20 to 100. That's an 80-point spread — the widest of any metric.

H1 = 100: Qwen3-4B, Qwen3-8B, GPT-OSS-20B, GLM-4.7-Flash
H1 = 90:  Gemma-3n-E4B, Llama-4-Scout
H1 = 60:  Qwen3-1.7B, DeepSeek-R1-14B
H1 = 20:  Llama-3.2-1B  ← fabricates 80% of the time

The Qwen3 family is remarkably consistent at hallucination resistance across all sizes. Meanwhile, the smallest model (1.3B) will confidently tell you about a nonexistent professor's nonexistent research paper, complete with fake citations.

Finding 5: 1.7B beats 14B

Qwen3-1.7B    (1.2GB)  → SHIFT 66.8
Mistral-7B    (5.0GB)  → SHIFT 60.6  ← 4.2× bigger, 6.2 points worse
Llama-3.1-8B  (5.5GB)  → SHIFT 61.0  ← 4.7× bigger, 5.8 points worse
DeepSeek-R1-14B (9.5GB) → SHIFT 59.8  ← 8.7× bigger, 7.0 points worse

Architecture generation matters more than parameter count. A 2025 model at 1.7B outperforms three 2024 models at 7–14B.

🏅 vs SOTA: How do small models compare to Claude and GPT-5?

We give the same 19 questions to both our small models and frontier giants:

Claude Sonnet 4.6  → 69.9  (ceiling)
Claude Opus 4.6    → 69.3
GPT-5.4            → 62.4
Qwen3.5-397B       → 57.1
────────────────────────────
Gemma-3-12B        → 57.1  (82% of Claude!)
GPT-OSS-20B        → 54.2  (78% of Claude)
Gemma-3n-E4B       → 47.4  (68% of Claude)

A 12B model matches a 397B model on identical questions. The gap between small and large is narrower than most people think.

⚡ Speed: Provider matters more than model size

Llama-4-Scout (Groq)       → 240.5 tok/s
Llama-3.1-8B (Cerebras)    → 187.7 tok/s
Qwen3-8B (Fireworks)       → 186.8 tok/s
...
Gemma-3-12B (Featherless)  →  18.7 tok/s
Mistral-7B (Featherless)   →  17.8 tok/s

The fastest model is 13× faster than the slowest — and it's a bigger model. The difference? Groq's inference chip vs. generic GPU hosting. Infrastructure choice dominates model size in determining real-world speed.

🗓️ Anti-contamination: Season system

One concern with any public benchmark: models will eventually train on the questions.

Our defense:

30 anchor questions stay fixed across seasons (for IRT calibration)
95 questions rotate (70%+ replaced each season)
Union Eval questions are never published
Season 2 planned for 2026 Q3

🤝 Built with the community

This benchmark was developed in collaboration with the FINAL Bench research team. The Union Eval cross-benchmark design draws on their evaluation methodology.

It also integrates with the ALL Bench Leaderboard — so you can see where your small model ranks among small models (Smol WorldCup) and against the full landscape including GPT-5 and Claude (ALL Bench).

Try it yourself

The dataset is open under Apache 2.0. We welcome new model submissions.

from datasets import load_dataset

ds = load_dataset("ginigen-ai/smol-worldcup")
print(f"Total: {len(ds['train'])} questions")

# Filter by axis
honesty = ds['train'].filter(lambda x: x['shift_axis'] == 'H')

# Filter by language
korean = ds['train'].filter(lambda x: x['category'] == 'multilingual_ko')

🏟️ Live Leaderboard
📊 Dataset on HuggingFace
🏅 ALL Bench Leaderboard

Developed by Ginigen.ai · Small but Mighty AI

MARL: Runtime Middleware That Reduces LLM Hallucination Without Fine-Tuning

AI Tech News — Mon, 09 Mar 2026 08:45:56 +0000

Your LLM is confidently wrong, and it can't stop itself.

Ask GPT about a historical date, and it answers with full confidence — right or wrong. Ask Claude to analyze a contract, and it commits to its first interpretation without ever reconsidering. This is hallucination, and in 2026, it remains the #1 blocker for production AI.

The root cause is structural. LLMs are autoregressive: each token is conditioned on previous tokens. Once generation starts, the model cannot stop mid-stream and say "wait, I was wrong." If the initial framing is flawed, it rides that trajectory to the end.

We built MARL to fix this.

pip install marl-middleware

What the Data Says

We released FINAL Bench — the world's first benchmark measuring AI metacognition (the ability to know what you know and what you don't). We tested 9 SOTA models including GPT-5.2, Claude Opus 4.6, and Gemini 3 Pro across 1,800 assessments:

Metric	What It Measures	Score
MA (Metacognitive Accuracy)	Can it say "I might be wrong"?	0.694
ER (Error Recovery)	Can it actually find and fix errors?	0.302
Gap	The chasm between knowing and doing	0.392

AI models sense they could be wrong. But they can't fix what's broken. A 39.2 percentage-point gap between awareness and action.

How MARL Works

MARL (Model-Agnostic Runtime Middleware for LLMs) decomposes a single LLM call into a 5-stage expert pipeline:

User Query
    │
    ▼
S1: Hypothesis  → Designs the optimal approach
    │
    ▼
S2: Solver      → Performs deep reasoning
    │
    ▼
S3: Auditor     → Audits for gaps and contradictions
    │
    ▼
S4: Verifier    → Adversarial cross-validation
    │
    ▼
S5: Synthesizer → Integrates ALL feedback,
                   generates entirely new final response
    │
    ▼
Clean Answer (user sees only the refined result)

Two mechanisms run simultaneously:

Cooperative Reinforcement — knowledge compounds across S1→S2→S3
Adversarial Cross-Validation — S4 deliberately attacks S2's conclusions

The Synthesizer (S5) doesn't patch the original. It writes a completely new response informed by every correction. This transforms "answer in one shot" into "think, doubt, correct, and rewrite."

In our FINAL Bench tests, this metacognitive scaffolding improved performance on the hardest tasks by over 70%, with 94.8% of the gain coming from error recovery.

Not Fine-Tuning. Not RAG. A Third Way.

	Fine-Tuning	RAG	MARL
Changes	Model weights	External knowledge	Reasoning structure
Cost	$10K+ GPU	Vector DB setup	1 line of code
Time	Weeks	Days	Instant
Lock-in	Yes	No	No
Fixes	Domain gaps	Knowledge gaps	Reasoning errors

MARL never touches weights. Switch from GPT-5.4 to Claude to Llama — the MARL layer stays. No vendor lock-in.

Integration: One Line

from openai import OpenAI

# Just change base_url. That's it.
client = OpenAI(
    api_key="sk-...",
    base_url="http://localhost:8080/v1"  # ← MARL server
)

# Everything else stays exactly the same
response = client.chat.completions.create(
    model="gpt-5.4",
    messages=[{"role": "user", "content": "Your prompt here"}]
)

Every call now flows through the 5-stage pipeline automatically.

9 Domain-Specific Emergence Engines

Beyond default reasoning enhancement, MARL ships with 9 specialized engines — activated by appending ::mode to the model name:

model="gpt-5.4::pharma"     # 💊 Drug discovery (172 items)
model="gpt-5.4::invent"     # 🔬 Invention & patents (4,275 items)
model="gpt-5.4::genomics"   # 🧬 Genomics & bio (104 items)
model="gpt-5.4::chemistry"  # 🧪 Chemistry & materials (135 items)
model="gpt-5.4::ecology"    # 🌍 Ecology & environment (105 items)
model="gpt-5.4::law"        # ⚖️ Legal & regulatory (59 items)
model="gpt-5.4::create"     # 🎨 General creative (493 seeds)
model="gpt-5.4::doc"        # 📝 Document generation
model="gpt-5.4::recipe"     # 🍳 Culinary fusion

5,538 expert data items cross-combined across multiple layers. Each engine has 5 emergence rules and 10 cross-layer bonus pairs. Works with any LLM model name — not just OpenAI.

Open Core: Protected Engine, Transparent Reasoning

The core engine (pipeline logic, attention matrix, agent prompts) ships as a compiled binary — proprietary tech stays protected.

Everything else is open: installation, API integration, A/B test demos, and most importantly — the full reasoning trace. Every stage is logged transparently. You can see exactly where an error was caught and how it was corrected.

If LLMs are black boxes, MARL is a glass box.

Available Everywhere

We shipped MARL simultaneously across four platforms:

# PyPI
pip install marl-middleware

# Docker
docker pull vidraft/marl:latest

# ClawHub (OpenClaw — 260K+ developers, 3,200+ AI skills)
clawhub install marl-middleware

# GitHub
git clone https://github.com/Vidraft/MARL.git

On ClawHub, MARL is the first middleware in the Reasoning Enhancement category. One command gives your AI agent a metacognition upgrade — it thinks before it acts.

Try It Now

📝 Technical deep dive: huggingface.co/blog/FINAL-Bench/marl-middleware

🤗 Live A/B test (Raw LLM vs MARL): huggingface.co/spaces/VIDraft/MARL

📦 PyPI: pypi.org/project/marl-middleware

🐙 GitHub: github.com/Vidraft/MARL

🦀 ClawHub: clawhub.ai/Cutechicken99/marl-middleware

Built by VIDRAFT — the team behind FINAL Bench (HF Dataset Global #5), FACTS Grounding Medical AI World #2 (CNRS-verified), and HuggingFace STAR AI TOP 12 (2024). 2M monthly active users, 1,500+ public AI models.

DEV Community: AI Tech News

Prefill/Decode Disaggregation in vLLM: The KV Transfer Math That Decides If It Pays Off

TL;DR

What problem does disaggregated prefill actually solve?

What does it cost? The KV transfer arithmetic

How long does the transfer take compared with prefill?

Why does colocated prefill hurt decode so much?

How do you set it up in vLLM?

How do you measure whether it helped?

When should you use it, and when should you skip it?

FAQ

Further reading

On-Device LLM Inference: The Tradeoffs Nobody Puts on the Spec Sheet

TL;DR

Why on-device at all

Memory is the hard wall

Precision is the main lever

Latency includes a cold start

No autoscaler, no hiding

A short checklist

FAQ

Further reading

KV Cache Quantization in LLM Serving: FP8 and INT8 Tradeoffs, the Silent config.json Trap, and How to Measure It Fairly

TL;DR

Why is the KV cache the thing you should quantize?

FP8 vs INT8 for the KV cache: what is the real tradeoff?

How much quality do you actually lose?

The silent config.json trap

How do you measure KV quantization fairly?

FAQ

Further reading

What Does It Actually Cost to Self-Host an LLM? The Batching Math Nobody Shows You

TL;DR

Why is the per-hour GPU price the wrong number to quote?

How much does continuous batching really change the math?

What caps the batch size before compute does?

Does quantizing the weights lower cost, or just fit a bigger model?

How do I estimate my own number in ten minutes?

Which three settings move the result the most?

FAQ

Further reading

Five Ways Your LLM Serving Benchmark Is Lying to You (and How to Catch Each One)

TL;DR

Why does prefix caching inflate vLLM benchmarks, and why unequally?

Fix: seed every call independently and log the hit rate

Why does my tokens-per-second drop when I enable speculative decoding?

Fix: count tokens from the source of truth

Can identical vLLM command-line flags produce different runtime configs?

Fix: diff the realized config, not the intended one

Why should you tune vLLM settings one at a time?

Fix: one axis per run, and ask the code before restarting

Why do "stability-first" auto-tuning gates pick the slowest config?

Fix: gates need a floor and an interval, written in code

Can systemd Restart=always hide a broken benchmark worker?

Fix

Pre-flight checklist for LLM serving benchmarks

FAQ

Should I disable prefix caching for every vLLM benchmark?

How do I count tokens correctly with streaming responses?

How can I tell if vLLM enabled FP8 KV cache on its own?

How many repetitions do I need before declaring a winner?

Is a lower-variance config always better for production?

Why did my tuning help chat traffic but hurt long documents?

A 4B Model Just Beat 8B — We Tested 18 Small LLMs and the Results Are Wild

The "bigger is better" assumption is wrong.

🤔 Why we built yet another benchmark

🏟️ Introducing SHIFT — 5 axes, not 1

📊 The ranking formula: WCS

🏆 The results

🔬 5 findings that surprised us

Finding 1: 4B = 8B (at 36% of the RAM)

Finding 2: MoE is the cheat code for edge AI

Finding 3: Thinking models have a dark side

Finding 4: The hallucination gap is enormous

Finding 5: 1.7B beats 14B

🏅 vs SOTA: How do small models compare to Claude and GPT-5?

⚡ Speed: Provider matters more than model size

🗓️ Anti-contamination: Season system

🤝 Built with the community

Try it yourself