DEV Community

Serving at Scale Series' Articles

Back to AI Tech News's Series
What Does It Actually Cost to Self-Host an LLM? The Batching Math Nobody Shows You
Cover image for What Does It Actually Cost to Self-Host an LLM? The Batching Math Nobody Shows You

What Does It Actually Cost to Self-Host an LLM? The Batching Math Nobody Shows You

Comments
6 min read
KV Cache Quantization in LLM Serving: FP8 and INT8 Tradeoffs, the Silent config.json Trap, and How to Measure It Fairly

KV Cache Quantization in LLM Serving: FP8 and INT8 Tradeoffs, the Silent config.json Trap, and How to Measure It Fairly

1
Comments 1
7 min read
On-Device LLM Inference: The Tradeoffs Nobody Puts on the Spec Sheet

On-Device LLM Inference: The Tradeoffs Nobody Puts on the Spec Sheet

Comments
3 min read
Prefill/Decode Disaggregation in vLLM: The KV Transfer Math That Decides If It Pays Off
Cover image for Prefill/Decode Disaggregation in vLLM: The KV Transfer Math That Decides If It Pays Off

Prefill/Decode Disaggregation in vLLM: The KV Transfer Math That Decides If It Pays Off

Comments
11 min read
Speculative Decoding at High Batch Size: Where the 3x Speedup Turns Into a Tax
Cover image for Speculative Decoding at High Batch Size: Where the 3x Speedup Turns Into a Tax

Speculative Decoding at High Batch Size: Where the 3x Speedup Turns Into a Tax

Comments
12 min read
KV Cache Offloading to NVMe: The Break-Even Bandwidth Math and Where It Does Not Pay Off
Cover image for KV Cache Offloading to NVMe: The Break-Even Bandwidth Math and Where It Does Not Pay Off

KV Cache Offloading to NVMe: The Break-Even Bandwidth Math and Where It Does Not Pay Off

Comments
14 min read
One Fleet for Serving and Training? What a 325M Pretraining Run on an Inference NPU Means for AI Infrastructure
Cover image for One Fleet for Serving and Training? What a 325M Pretraining Run on an Inference NPU Means for AI Infrastructure

One Fleet for Serving and Training? What a 325M Pretraining Run on an Inference NPU Means for AI Infrastructure

Comments
2 min read
Your LLM Serving Engine Ships Every 13 Days: Reading vLLM's Breaking-Change Lists as Upgrade Risk Data
Cover image for Your LLM Serving Engine Ships Every 13 Days: Reading vLLM's Breaking-Change Lists as Upgrade Risk Data

Your LLM Serving Engine Ships Every 13 Days: Reading vLLM's Breaking-Change Lists as Upgrade Risk Data

Comments 1
13 min read
When Is a KV Cache Slot Actually Free? Lessons From the SGLang and Cloudflare Cross-Request Exposure
Cover image for When Is a KV Cache Slot Actually Free? Lessons From the SGLang and Cloudflare Cross-Request Exposure

When Is a KV Cache Slot Actually Free? Lessons From the SGLang and Cloudflare Cross-Request Exposure

Comments
15 min read
When Should an LLM Router Stop Being Sticky? llm-d 1.0 Cache-Affinity Routing, Break-Even Math, and a Simulation
Cover image for When Should an LLM Router Stop Being Sticky? llm-d 1.0 Cache-Affinity Routing, Break-Even Math, and a Simulation

When Should an LLM Router Stop Being Sticky? llm-d 1.0 Cache-Affinity Routing, Break-Even Math, and a Simulation

Comments
14 min read
vLLM Fast Start Explained: What Preload and Snapshots Save, What They Cannot, and the Fingerprint Gap
Cover image for vLLM Fast Start Explained: What Preload and Snapshots Save, What They Cannot, and the Fingerprint Gap

vLLM Fast Start Explained: What Preload and Snapshots Save, What They Cannot, and the Fingerprint Gap

Comments
14 min read
vLLM 0.33 Adds a Lossy Speculative Verifier: Why Single-Run Accuracy Tables Cannot Tell You It Is Safe
Cover image for vLLM 0.33 Adds a Lossy Speculative Verifier: Why Single-Run Accuracy Tables Cannot Tell You It Is Safe

vLLM 0.33 Adds a Lossy Speculative Verifier: Why Single-Run Accuracy Tables Cannot Tell You It Is Safe

Comments
15 min read
Running an MoE Model Larger Than Your VRAM: SGLang Paged Experts, the PCIe Math, and the 4-Request Cliff
Cover image for Running an MoE Model Larger Than Your VRAM: SGLang Paged Experts, the PCIe Math, and the 4-Request Cliff

Running an MoE Model Larger Than Your VRAM: SGLang Paged Experts, the PCIe Math, and the 4-Request Cliff

Comments 1
14 min read
vLLM 0.33 Ships an ABI-Stable Wheel, but PyPI Still Pins torch==2.14.0: What You Can Actually Swap
Cover image for vLLM 0.33 Ships an ABI-Stable Wheel, but PyPI Still Pins torch==2.14.0: What You Can Actually Swap

vLLM 0.33 Ships an ABI-Stable Wheel, but PyPI Still Pins torch==2.14.0: What You Can Actually Swap

Comments 1
13 min read
Day-0 Model Support Is a Fork: What 26 NVIDIA Dynamo Snapshot Builds Say About Pinning Your LLM Serving Stack
Cover image for Day-0 Model Support Is a Fork: What 26 NVIDIA Dynamo Snapshot Builds Say About Pinning Your LLM Serving Stack

Day-0 Model Support Is a Fork: What 26 NVIDIA Dynamo Snapshot Builds Say About Pinning Your LLM Serving Stack

1
Comments 1
15 min read
TensorRT-LLM 1.4 Makes trtllm-serve Run 8 HTTP Frontends by Default: RAM Cost, Connection Balance, and What Still Differs
Cover image for TensorRT-LLM 1.4 Makes trtllm-serve Run 8 HTTP Frontends by Default: RAM Cost, Connection Balance, and What Still Differs

TensorRT-LLM 1.4 Makes trtllm-serve Run 8 HTTP Frontends by Default: RAM Cost, Connection Balance, and What Still Differs

Comments
13 min read