Skip to content
Navigation menu
Search
Powered by Algolia
Search
Log in
Create account
DEV Community
Close
Serving at Scale Series' Articles
Back to AI Tech News's Series
What Does It Actually Cost to Self-Host an LLM? The Batching Math Nobody Shows You
AI Tech News
AI Tech News
AI Tech News
Follow
Oct 6
What Does It Actually Cost to Self-Host an LLM? The Batching Math Nobody Shows You
#
llm
#
vllm
#
performance
#
devops
Comments
Add Comment
6 min read
Moderation
KV Cache Quantization in LLM Serving: FP8 and INT8 Tradeoffs, the Silent config.json Trap, and How to Measure It Fairly
AI Tech News
AI Tech News
AI Tech News
Follow
Oct 6
KV Cache Quantization in LLM Serving: FP8 and INT8 Tradeoffs, the Silent config.json Trap, and How to Measure It Fairly
#
llm
#
vllm
#
performance
#
devops
1
reaction
Comments
1
comment
7 min read
Moderation
On-Device LLM Inference: The Tradeoffs Nobody Puts on the Spec Sheet
AI Tech News
AI Tech News
AI Tech News
Follow
Oct 12
On-Device LLM Inference: The Tradeoffs Nobody Puts on the Spec Sheet
#
ai
#
machinelearning
#
llm
#
edgeai
Comments
Add Comment
3 min read
Moderation
Prefill/Decode Disaggregation in vLLM: The KV Transfer Math That Decides If It Pays Off
AI Tech News
AI Tech News
AI Tech News
Follow
Oct 19
Prefill/Decode Disaggregation in vLLM: The KV Transfer Math That Decides If It Pays Off
#
ai
#
machinelearning
#
llm
#
vllm
Comments
Add Comment
11 min read
Moderation
Speculative Decoding at High Batch Size: Where the 3x Speedup Turns Into a Tax
AI Tech News
AI Tech News
AI Tech News
Follow
Oct 20
Speculative Decoding at High Batch Size: Where the 3x Speedup Turns Into a Tax
#
ai
#
machinelearning
#
vllm
#
performance
Comments
Add Comment
12 min read
Moderation
KV Cache Offloading to NVMe: The Break-Even Bandwidth Math and Where It Does Not Pay Off
AI Tech News
AI Tech News
AI Tech News
Follow
Oct 21
KV Cache Offloading to NVMe: The Break-Even Bandwidth Math and Where It Does Not Pay Off
#
ai
#
machinelearning
#
vllm
#
llm
Comments
Add Comment
14 min read
Moderation
One Fleet for Serving and Training? What a 325M Pretraining Run on an Inference NPU Means for AI Infrastructure
AI Tech News
AI Tech News
AI Tech News
Follow
Oct 26
One Fleet for Serving and Training? What a 325M Pretraining Run on an Inference NPU Means for AI Infrastructure
#
ai
#
machinelearning
#
infrastructure
#
hardware
Comments
Add Comment
2 min read
Moderation
Your LLM Serving Engine Ships Every 13 Days: Reading vLLM's Breaking-Change Lists as Upgrade Risk Data
AI Tech News
AI Tech News
AI Tech News
Follow
Oct 31
Your LLM Serving Engine Ships Every 13 Days: Reading vLLM's Breaking-Change Lists as Upgrade Risk Data
#
ai
#
machinelearning
#
vllm
#
llm
Comments
1
comment
13 min read
Moderation
When Is a KV Cache Slot Actually Free? Lessons From the SGLang and Cloudflare Cross-Request Exposure
AI Tech News
AI Tech News
AI Tech News
Follow
Nov 1
When Is a KV Cache Slot Actually Free? Lessons From the SGLang and Cloudflare Cross-Request Exposure
#
ai
#
machinelearning
#
llm
#
security
Comments
Add Comment
15 min read
Moderation
When Should an LLM Router Stop Being Sticky? llm-d 1.0 Cache-Affinity Routing, Break-Even Math, and a Simulation
AI Tech News
AI Tech News
AI Tech News
Follow
Nov 2
When Should an LLM Router Stop Being Sticky? llm-d 1.0 Cache-Affinity Routing, Break-Even Math, and a Simulation
#
ai
#
machinelearning
#
llm
#
kubernetes
Comments
Add Comment
14 min read
Moderation
vLLM Fast Start Explained: What Preload and Snapshots Save, What They Cannot, and the Fingerprint Gap
AI Tech News
AI Tech News
AI Tech News
Follow
Nov 3
vLLM Fast Start Explained: What Preload and Snapshots Save, What They Cannot, and the Fingerprint Gap
#
ai
#
machinelearning
#
llm
#
vllm
Comments
Add Comment
14 min read
Moderation
vLLM 0.33 Adds a Lossy Speculative Verifier: Why Single-Run Accuracy Tables Cannot Tell You It Is Safe
AI Tech News
AI Tech News
AI Tech News
Follow
Nov 5
vLLM 0.33 Adds a Lossy Speculative Verifier: Why Single-Run Accuracy Tables Cannot Tell You It Is Safe
#
ai
#
machinelearning
#
llm
#
vllm
Comments
Add Comment
15 min read
Moderation
Running an MoE Model Larger Than Your VRAM: SGLang Paged Experts, the PCIe Math, and the 4-Request Cliff
AI Tech News
AI Tech News
AI Tech News
Follow
Nov 7
Running an MoE Model Larger Than Your VRAM: SGLang Paged Experts, the PCIe Math, and the 4-Request Cliff
#
ai
#
machinelearning
#
llm
#
sglang
Comments
1
comment
14 min read
Moderation
vLLM 0.33 Ships an ABI-Stable Wheel, but PyPI Still Pins torch==2.14.0: What You Can Actually Swap
AI Tech News
AI Tech News
AI Tech News
Follow
Nov 8
vLLM 0.33 Ships an ABI-Stable Wheel, but PyPI Still Pins torch==2.14.0: What You Can Actually Swap
#
ai
#
python
#
llm
#
vllm
Comments
1
comment
13 min read
Moderation
Day-0 Model Support Is a Fork: What 26 NVIDIA Dynamo Snapshot Builds Say About Pinning Your LLM Serving Stack
AI Tech News
AI Tech News
AI Tech News
Follow
Nov 9
Day-0 Model Support Is a Fork: What 26 NVIDIA Dynamo Snapshot Builds Say About Pinning Your LLM Serving Stack
#
ai
#
opensource
#
llm
#
vllm
1
reaction
Comments
1
comment
15 min read
Moderation
TensorRT-LLM 1.4 Makes trtllm-serve Run 8 HTTP Frontends by Default: RAM Cost, Connection Balance, and What Still Differs
AI Tech News
AI Tech News
AI Tech News
Follow
Nov 11
TensorRT-LLM 1.4 Makes trtllm-serve Run 8 HTTP Frontends by Default: RAM Cost, Connection Balance, and What Still Differs
#
ai
#
opensource
#
llm
#
devops
Comments
Add Comment
13 min read
Moderation
We're a place where coders share, stay up-to-date and grow their careers.
Log in
Create account