DEV Community: Alex Chen

SGLang vs vLLM: Your Agent Is Slow Because You Picked the Wrong Cache

Alex Chen — Fri, 23 Oct 2026 01:58:02 +0000

If you serve LLM agents in production, you have probably seen this week's numbers: in a multi-turn agent workload with an 8,000-token system context, SGLang's RadixAttention cut time-to-first-token (TTFT) from 440ms to 38ms — an 11.57x speedup — while vLLM v0.6 with prefix caching managed 480ms down to 92ms, a 5.21x improvement (Qwen2.5-Coder-32B-Instruct-AWQ), as benchmarked by NextByteTech in their Dev.to post on ollama vs vLLM vs SGLang in 2026.

The data is real, and the conclusion is mostly right: for long-context, multi-turn agent traffic, SGLang currently wins. But "mostly" hides the interesting part. Those speedups only materialize when your workload has the right shape. Run the same two engines against different traffic and the gap collapses — or flips. This post is the decision framework the benchmark post didn't give you: when RadixAttention genuinely pulls ahead, and when both engines tie.

Why RadixAttention Wins in Agent Workloads

Both vLLM's prefix caching and SGLang's RadixAttention reuse previously computed KV cache for repeated prompt prefixes. The difference is the data structure.

vLLM's automatic prefix caching (APC) historically hashes whole blocks of the prompt and matches them against cached blocks. It's effective when requests share long, identical prefixes — the classic system-prompt-plus-few-shot-template shape — but it degrades when prefixes diverge mid-sequence or when the overlap doesn't align to block boundaries.

SGLang's RadixAttention stores the KV cache in a radix tree (a compressed trie) over token sequences. Repeated prefixes become shared tree paths, and the cache can be reused at the granularity of any common prefix, not just aligned blocks. Multi-turn agent conversations are exactly this: the same 8,000-token system context plus growing conversation history, turn after turn. Each new turn shares a massive common prefix with the previous one. The tree reuses nearly all of it, which is why the reported TTFT dropped to 38ms — almost the entire prefill was skipped.

The second Dev.to comparison (ifnodoraemon's SGLang vs vLLM architecture showdown) makes the mechanism visible in the metrics: at a 4,096-token prompt with 75% prefix overlap and concurrency 64, SGLang hit a 78.6% KV cache hit rate vs vLLM's 41.2%, with median TTFT of 85ms vs 380ms (4.47x), P99 TTFT of 280ms vs 1,250ms, and throughput of 5,430 vs 3,120 tok/s. The hit-rate gap is the performance gap. When your workload is prefix-overlap-heavy, the radix tree's finer-grained reuse shows up directly in every latency percentile.

But Sometimes They Tie

Here is where the story gets useful. Three published counter-cases show where the advantage evaporates.

1. Unique-prompt traffic: the gap nearly disappears

bex.co's September 2026 report (SGLang Crossed 400,000 Production GPUs) benchmarked Llama 3.3 70B FP8 on Spheron H100s at 50 concurrent requests with unique prompts — traffic with no shared prefix to reuse. The result: SGLang at 1,920 tok/s vs vLLM at 1,850 tok/s, with TTFT within 20ms of each other. No shared prefix, no cache to share; the radix tree has nothing to work with, and the two engines are functionally tied.

2. Raw throughput on short prompts: vLLM can win

The same bex.co report includes a May 2026 result on a single H100 with Qwen2.5-7B: vLLM delivered 23,500 tok/s of raw throughput vs SGLang's 16,800 tok/s — a clear vLLM win on aggregate tokens per second. SGLang still held a 4x+ TTFT advantage (1.8s vs 7.8s), so it wasn't a loss on every axis, but for a throughput-first, latency-tolerant batch workload, the "SGLang is faster" headline would have been wrong.

3. High concurrency: the crossover is real

bex.co's DGX Spark concurrency sweep found SGLang winning at 1–4 concurrent requests, with vLLM winning at 6+ concurrent requests. And RunPod's blog on the SGLang vs vLLM KV cache comparison (2x H100, DeepSeek-R1-Distill-Llama-70B, 7k context) concluded that a cache hit yields roughly +20% speed — and that on fresh context the two engines are comparable. If your traffic is many short, independent prompts at high concurrency — the classic chatbot-serving shape — the radix tree overhead buys you nothing and vLLM's throughput engine takes the round.

Workload shape	Winner (per published results)
Multi-turn agent, long shared system prompt (8k tokens), moderate concurrency	SGLang, large margin (11.57x vs 5.21x TTFT speedup)
4k prompts, 75% prefix overlap, concurrency 64	SGLang (4.47x median TTFT, 2x KV hit rate)
Unique prompts, no shared prefix, 50 concurrent	Tie (1,920 vs 1,850 tok/s, TTFT within 20ms)
Short prompts, throughput-first batch, 1x H100	vLLM (23,500 vs 16,800 tok/s raw throughput)
Short prompts, 6+ concurrency (DGX Spark)	vLLM
Fresh context, no cache reuse (RunPod)	Tie

The Decision Checklist

Before you switch engines, measure your workload against these four questions:

How much prefix overlap do you actually have? If most of your requests share a long system prompt or conversation history (agents, RAG with fixed instruction blocks, few-shot templates), RadixAttention's finer-grained reuse pays off. If every prompt is unique, the tree is dead weight.
What's your concurrency? Published sweeps show a crossover: SGLang ahead at low concurrency, vLLM ahead as concurrency climbs (around 6+ on DGX Spark in one report — verify against your own hardware). Size for your actual traffic, not the benchmark's.
Are you prefill-heavy or decode-heavy? Prefix caching only accelerates the prefill phase. If your outputs are long and your prompts are short, most of your time is in decode, and the cache advantage shrinks to a rounding error. RunPod's +20%-on-cache-hit figure is a good mental anchor.
Which latency percentile matters to you? SGLang's biggest reported edge is in TTFT and tail latency (P99 280ms vs 1,250ms in the 75%-overlap scenario). If your SLO is on end-to-end tokens-per-second for batch jobs, that edge may not buy you anything — vLLM's raw throughput win shows the other side of the trade.

Conclusion

The benchmark post's headline number — 38ms vs 92ms TTFT in the multi-turn agent scenario — is correct for that workload, and RadixAttention's radix-tree design is genuinely the better mechanism for shared-prefix traffic. But engines don't run benchmarks; they run your traffic. Unique-prompt serving, short-prompt batch throughput, and high-concurrency chatbot shapes all show the two engines tying or vLLM winning, per bex.co, RunPod, and the second Dev.to comparison. Measure your prefix overlap and concurrency first, then pick the cache that fits.

We write about production LLM infra and agent systems for builders shipping real products. Follow along here:
https://t.me/qisuancloud
Or try our free GPU Advisor: https://qisuanai.com/advisor

Your 128K Context Fits on a 4090 — If You Quantize the KV Cache

Alex Chen — Mon, 19 Oct 2026 06:10:55 +0000

Researched October 2026. Sizing math from public model cards and the vLLM docs; not my own benchmarks.

In my last post I showed that a 128K context window can push a 7B model from 17 GB to 33 GB of VRAM — and that you have three moves, in order of cost-effectiveness: quantize the cache, pick a GQA-heavy architecture, or rent the bigger card. This post is about move #1, the cheapest one. It halves the KV cache with a single flag.

The One Flag

In vLLM:

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --kv-cache-dtype fp8

That's it. The weights stay in FP16/BF16 — untouched. Only the KV cache is stored as 1 byte per value instead of 2. Per the vLLM docs, fp8 maps to E4M3, and there are three calibration options: no calibration (scales default to 1.0), random-token warmup, or dataset calibration via llm-compressor (recommended for max accuracy).

Why FP8 Works for KV Cache

Weight quantization gets all the attention, but KV values are arguably the easier target: within a single attention head, their values live in a narrow, well-behaved range. E4M3's tighter range and 3-bit mantissa match that distribution well; E5M2's wider range is mostly wasted on values that never appear, and its 2-bit mantissa visibly degrades attention scores that compound across layers. In other words, the format choice matters — fp8 (E4M3) is the right default, not fp8_e5m2.

Where the Math Flips

Llama 3.1 8B, FP16 weights (~16 GB), 2 GB overhead:

Context	FP16 cache → total	FP8 cache → total
32K	4 GB → ~22 GB ✅	2 GB → ~20 GB ✅
64K	8 GB → ~26 GB ❌	4 GB → ~22 GB ✅ fits 4090
128K	16 GB → ~34 GB ❌	8 GB → ~26 GB ❌ still over

Honest note: at 128K, FP8 cache alone doesn't save Llama 3.1 8B on a 24 GB card — 26 GB is still 26 GB. Your options there: pair it with FP8 weights (8 + 8 + 2 ≈ 18 GB ✅), or pick the GQA-heavy architecture from last post — Qwen2.5 7B at 128K needs ~7 GB of FP16 cache, ~3.5 GB in FP8, for a total around 19 GB ✅. The flag is powerful, not magic; the arithmetic still rules.

Three Caveats Before You Ship It

1. Hardware matters. The biggest wins are on Hopper GPUs with native FP8 support. On older cards, check your vLLM version's kernel support — the memory savings are real, but the compute path may not be as optimized.

2. Calibrate if quality matters. Default scales of 1.0 are fine to start, but for production, dataset calibration (llm-compressor) exists for a reason. Per-head quantization — finer-grained than per-tensor — needs the FlashAttention backend plus that calibration path.

3. Measure retrieval, not just perplexity. Standard benchmarks show small degradation, but long-context tasks with exact-recall demands (needle-in-a-haystack style document QA) are where you'd notice it first. Run your own eval before and after.

What This Costs in Practice

The whole point: quantization keeps you on the cheap card. A 64K-context 7B workload that needed an 80 GB card at ~$1.50+/hr in FP16 fits on a 24 GB 4090 at ~$0.35/hr with FP8 cache — same model, same context, roughly a quarter of the hourly cost, for one CLI flag.

I built a free GPU Advisor that walks through these tradeoffs (model size, context, quantization tolerance) and recommends the cheapest fitting setup. No signup.

Have you tried FP8 KV cache in production — and did the quality hold up on your workload? Curious where it breaks in practice.

Stop Overpaying for GPUs: Spot vs On-Demand for LLM Inference (With Real Numbers)

Alex Chen — Fri, 16 Oct 2026 07:03:05 +0000

Stop Overpaying for GPUs: Spot vs On-Demand for LLM Inference (With Real Numbers)

If you're running LLM inference on rented GPUs, you're probably overpaying. Not by a little — by 2-3x. The culprit isn't which cloud you picked. It's how you're paying for it.

I spend my days comparing GPU cloud pricing (it's literally what I do), and the spot vs on-demand decision is where most people leave the most money on the table. Here's the practical breakdown.

The Price Gap Is Bigger Than You Think

Take an RTX 4090 (24GB) — the workhorse for 7B-32B model inference:

On-demand: ~$0.34–$0.74/hour depending on the cloud
Spot/preemptible: ~$0.15–$0.30/hour for the same card

That's a 50-70% discount for the exact same silicon. For an A100 80GB, the gap is similar: ~$1.20/hr on-demand vs ~$0.50/hr spot.

But cheap isn't free. Here's what you're actually trading.

What "Spot" Really Means in Practice

Spot instances can be reclaimed by the provider with little warning (typically 30 seconds to 2 minutes). For LLM inference, that means:

Model reload time: A 70B model takes 3-5 minutes to load into VRAM. If you get preempted twice an hour, you've lost more than you saved.
Request interruption: In-flight requests die. You need retry logic or a queue.
Availability variance: During peak hours, spot capacity dries up. Your "cheap" GPU might simply not exist when you need it.

The Decision Framework

Use spot when:

You're doing batch inference (embeddings, bulk classification, offline evals)
Your workload is checkpoint-friendly or stateless
You can tolerate 5-10 minute interruptions
You're running experiments and cost matters more than latency

Use on-demand when:

You're serving real users with latency SLAs
Your model takes >2 minutes to load (large models)
It's peak hours in US/EU and spot markets are thin
The cost of an interruption exceeds the savings (do the math!)

The hybrid most people should use:
Run your baseline traffic on 1-2 on-demand instances, burst with spot. Most inference workloads have a predictable floor and a spiky ceiling. Pay full price for the floor, spot price for the spikes.

The Math People Skip

Don't just compare hourly rates. Calculate effective cost per 1K tokens:

effective_hourly = (hourly_rate × hours_run + reload_cost) / hours_run

Where reload_cost = preemptions × reload_time_hours × hourly_rate.

Example: $0.20/hr spot, preempted 3x/hour, 4-min reloads on a 70B model:

Reload overhead: 3 × (4/60) × $0.20 = $0.04/hr extra
Effective rate: $0.24/hr — still way cheaper than $0.74 on-demand

But with 10 preemptions/hour? Effective rate hits $0.33/hr plus the latency pain. At that point, on-demand wins.

One More Thing: Egress Fees

The sneakiest cost isn't compute — it's data transfer out. Some clouds charge $0.05-0.09/GB for egress. If you're doing RAG with large context windows or serving multimodal models, egress can exceed your GPU bill. Always check the egress pricing before committing.

I track GPU cloud prices weekly and post updates when prices move significantly. If you want the short version without reading pricing pages, I share them on my Telegram channel: https://t.me/qisuancloud

And if you want to skip the spreadsheet entirely, our free GPU Advisor picks the cheapest option for your workload in about 30 seconds: https://qisuanai.com/advisor

Prices referenced are from public pricing pages, October 2026. Always verify before you rent — cloud pricing changes fast.

Your 7B Model Fits on a 4090 — Until You Open a 128K Context Window

Alex Chen — Sun, 11 Oct 2026 02:41:11 +0000

Researched October 2026. Architecture specs from public model cards; all numbers below are arithmetic, not my own benchmarks.

In my last post I showed that a 7B model's weights need ~16 GB in FP16 — a comfortable fit on a 24 GB RTX 4090. Several readers asked the obvious follow-up: then why does my 7B inference server OOM the moment I enable long context?

Because weights are only half the story. The other half is the KV cache, and it scales in a way that surprises almost everyone.

The One Formula

For each token in context, the model must store a key and a value vector per layer:

KV cache per token = 2 × layers × kv_heads × head_dim × bytes_per_value

Take Llama 3.1 8B (32 layers, 8 KV heads, 128 head dim, FP16):

2 × 32 × 8 × 128 × 2 bytes = 131,072 bytes ≈ 128 KiB per token

Now multiply by context length:

Context	KV cache (Llama 3.1 8B, FP16)	+ 16 GB weights → total
8K	~1 GB	~17 GB fits 4090
32K	~4 GB	~21 GB fits 4090
128K	~16 GB	~33 GB needs 48 GB+

Same model, same weights — the context window alone decides whether you need a $0.35/hr card or a $1.50+/hr one. (Prices from my provider comparison.)

Not All 7B Models Are Equal

KV head count varies wildly between architectures:

Model	Layers	KV heads	KiB/token (FP16)	128K context cache
Llama 3.1 8B	32	8	128	~16 GB
Mistral 7B v0.3	32	8	128	~16 GB
Qwen2.5 7B	28	4	56	~7 GB

Qwen2.5 7B at 128K needs roughly half the cache of Llama 3.1 8B. Two "7B models," very different GPU bills.

The 70B Reality Check

Llama 3.1 70B (80 layers, 8 KV heads): 320 KiB/token → 128K context = ~41 GB of KV cache. Add 140 GB of FP16 weights and you're at ~181 GB — that's three 80 GB cards. Even in FP8 (70 GB weights), you're at ~111 GB: still two cards minimum. Long context is where 70B deployments quietly become multi-GPU projects.

Three Things Most Guides Skip

1. Batch size is a silent multiplier. KV cache scales with batch × sequence length. Serving 8 concurrent users at 8K context costs roughly the same cache as 1 user at 64K. Your "fits on a 4090" math breaks the moment traffic grows.

2. PagedAttention doesn't shrink the cache. vLLM's PagedAttention eliminates fragmentation waste, but the bytes are the bytes — it can't make 33 GB fit in 24 GB.

3. Quantize the cache too. KV cache in FP8 halves these numbers with minimal quality loss on most workloads. It's the same lever as weight quantization, applied to the forgotten half of VRAM.

A 30-Second Sizing Rule

context_budget = (VRAM − weights − 2 GB overhead) ÷ KiB_per_token ÷ batch_size

If your planned context exceeds the budget, you have three moves: quantize the cache, pick a GQA-heavy architecture, or rent the bigger card — in that order of cost-effectiveness.

I built a free GPU Advisor that walks through these tradeoffs with four questions and recommends the cheapest fitting setup. No signup.

What's the longest context you're actually serving in production — and did the KV cache math surprise you the first time? Curious what caught people off guard.

Your 7B Model Doesn't Need an H100: A Practical GPU Sizing Guide for LLM Inference

Alex Chen — Fri, 09 Oct 2026 10:00:34 +0000

Your 7B Model Doesn't Need an H100: A Practical GPU Sizing Guide for LLM Inference

Researched October 2026. Sizing rules from public docs and community practice, not my own benchmarks.

The most expensive mistake in LLM inference isn't picking the wrong cloud — it's renting too much GPU. I keep seeing people spin up H100s to serve 7B chatbots. Here's a practical guide to right-sizing your GPU.

The One Formula That Matters

Weights in FP16 take roughly 2 bytes per parameter:

Model size	FP16 weights	Minimum sensible GPU
7–8B	~14–16 GB	RTX 4090 / L4 (24 GB)
13B	~26 GB	A100 40GB
34B	~68 GB	2× A100 40GB or 1× H100 80GB
70B	~140 GB	2× A100/H100 80GB

Then add ~20–30% headroom for KV cache (grows with context length and batch size). Long-context workloads can double your memory needs — that's the part most guides skip.

Quantization Changes Everything

FP8 / AWQ / GPTQ roughly halve the memory requirement:

A 70B model in FP8 (~70 GB) can fit on a single H100 80GB
A 13B AWQ model runs comfortably on a 24 GB card

If your accuracy budget allows it, quantization is the single biggest cost lever you have — bigger than switching cloud providers.

Throughput vs. Latency: Pick Your Bottleneck

Chatbot (low concurrency): you need enough VRAM, that's it. A 4090 serving a 7B model at 60+ tok/s is overkill for 5 concurrent users — but cheap overkill.
Batch/API workloads: memory bandwidth is king. This is where A100/H100 earn their premium.

Don't pay for bandwidth you won't use.

A Decision Checklist

What's the largest model you'll serve in the next 6 months?
FP16 or can you quantize?
Max context length × max batch size → KV cache estimate
Latency SLA? (real-time chat vs. background jobs)
Spot-tolerant? (see my previous post on spot pricing)

What This Costs in Practice

Combining sizing with cloud prices (researched Oct 2026): a 7B model on a Vast.ai spot 4090 (~$0.35/hr) vs. an H100 on-demand (~$2.50+/hr) is a 7x cost difference for a workload that might not even notice.

I'm building a free tool that asks about your use case and ranks GPU options: https://qisuanai.com/advisor

What are you serving, and on what? Curious where these rules break down in practice.

I Compared 5 GPU Clouds for LLM Inference in 2026 — Here's What I Found

Alex Chen — Tue, 06 Oct 2026 09:41:57 +0000

I Compared 5 GPU Clouds for LLM Inference in 2026 — Here's What I Found

Researched October 2026. Prices from public pricing pages and third-party trackers.

Running LLMs in production gets expensive fast. I spent a week reviewing public pricing data from GPU cloud providers to find the cheapest way to serve inference workloads. Here are the results.

The Contenders

I looked at five providers that offer on-demand GPU instances suitable for LLM inference:

RunPod — community favorite, wide GPU selection
Lambda Labs — straightforward pricing, good availability
Vast.ai — marketplace model, often cheapest
AWS — the default choice, premium pricing
Google Cloud — strong TPUs, GPU pricing varies

Price Comparison (A100 80GB, per hour)

Provider	A100 80GB/hr	Notes
Vast.ai	~$1.20	Marketplace, prices fluctuate
RunPod	~$1.89	Secure Cloud pricing
Lambda Labs	~$1.50	On-demand
AWS (p4d)	~$3.20	On-demand, us-east-1
GCP (a2-highgpu)	~$2.90	On-demand

Prices are approximate, researched October 2026. Always check current pricing.

What About RTX 4090?

For smaller models (7B-13B), a 4090 is often enough and much cheaper:

Provider	RTX 4090/hr
Vast.ai	~$0.35
RunPod	~$0.69

A 4090 can handle Llama 3 8B inference comfortably. If your model fits, don't overpay for an A100.

My Takeaways

Marketplace > fixed pricing — Vast.ai is consistently 30-50% cheaper, but you trade reliability
Match GPU to model size — Don't rent an H100 for a 7B model
Spot/preemptible instances — Can cut costs another 50-70% if your workload tolerates interruptions
Watch egress fees — Some providers charge heavily for data transfer out

What's Next

I'm building a free tool that recommends the cheapest GPU for your specific workload. Drop a comment with your use case and I'll share early access.

What's your go-to GPU cloud? Am I missing a cheaper option?

Originally published on the QisuanCloud blog — or skip the reading: our free GPU Advisor ranks GPU clouds for your workload in about 30 seconds. No signup.