DEV Community

Cover image for SGLang vs vLLM: Install, Serve, and Benchmark on Bare Metal
Jakson Tate
Jakson Tate

Posted on • Originally published at servermo.com

SGLang vs vLLM: Install, Serve, and Benchmark on Bare Metal

#ai #machinelearning #devops #python

Two open-source engines currently dominate self-hosted LLM inference: vLLM and SGLang. Both promise the exact same thing—feed them a Hugging Face safetensors model, and they will spin up an ultra-fast, OpenAI-compatible API endpoint.

However, standard benchmarks comparing SGLang vs vLLM suffer from a glaring problem: amateurs benchmark these enterprise engines on a single, rented consumer GPU (like an RTX 4090). To understand real-world TTFT (Time To First Token) and TPOT (Time Per Output Token) metrics, you must analyze how these engines orchestrate memory and concurrency on bare metal GPU hosting architectures.

Phase 1: Escaping the VRAM & Compilation Traps

Before diving into performance numbers, you must address the catastrophic installation failures that plague both frameworks on Ubuntu 24.04.

⚠️ CRITICAL WARNING: The 0.9 VRAM Death Trap

Most official documentation tells you to set --gpu-memory-utilization 0.9 (vLLM) or --mem-fraction-static 0.9 (SGLang). If you are running an 80GB H100, this allocates 72GB. During CUDA Graph compilation, the engine requires temporary System RAM proportional to the GPU allocation. This immediately exhausts your host machine's RAM and triggers an OS-level OOM (Out-Of-Memory) kill. Always scale this parameter down to 0.8 or 0.85.

💡 SRE HIDDEN GEM: The ninja-build & PyTorch Hell

SGLang utilizes FlashInfer to compile highly optimized CUDA kernels on first launch. If your Linux server lacks the ninja-build package, the server will crash instantly with a FileNotFoundError. Furthermore, standard pip installations often trigger PyTorch version conflicts. Bypass this dependency hell by pre-installing ninja and fetching the latest FlashInfer wheel directly from their release index based on your specific CUDA version.

# 1. Install critical build tools to prevent FlashInfer compilation crashes
sudo apt update && sudo apt install -y python3-venv python3-pip git ninja-build build-essential

# 2. Create isolated environments to prevent Python global contamination
python3 -m venv /opt/llm_engine
source /opt/llm_engine/bin/activate

# 3. Safely install SGLang bypassing dependency hell
pip install --upgrade pip
pip install "sglang[all]"
pip install flashinfer -i [https://flashinfer.ai/whl/cu124/torch2.4/](https://flashinfer.ai/whl/cu124/torch2.4/)

# 4. Safely install vLLM
pip install vllm
Enter fullscreen mode Exit fullscreen mode

Phase 2: Analyzing TTFT vs TPOT Performance (The Truth)

When comparing SGLang vs vLLM, you must understand their architectural philosophies. vLLM uses PagedAttention, which treats the KV Cache like OS virtual memory to eliminate fragmentation. SGLang uses RadixAttention, which treats the KV cache like a compressed tree structure to maximize prefix sharing.

  • Pure Batch Throughput (vLLM Wins): If you are processing 10,000 completely unique prompts (no shared context), vLLM's highly optimized C++ PagedAttention queue handles continuous batching flawlessly.
  • Multi-Turn Chat & Agents (SGLang Annihilates): While vLLM offers an --enable-prefix-caching flag, its block-level storage struggles with complex conversational branching. In agentic workflows, multiple users share the exact same System Prompt. SGLang calculates it exactly once via Radix trees and its modern Rust-based router, delivering 5x faster TTFT and saving up to 80% VRAM.

Phase 3: Securing the 0.0.0.0 Exposure Vulnerability

The most dangerous mistake engineers make is copying default launch commands from GitHub documentation directly into a production server.

🚨 CRITICAL SECURITY ALERT: Unauthenticated Exposure

Running vllm serve --host 0.0.0.0 or sglang.launch_server --host 0.0.0.0 binds your LLM engine directly to the public internet. These frameworks do not have built-in API-key authentication or rate limiting. Attackers will scan your IP, steal your GPU compute, and execute malicious Prompt Injections to hijack your agents. Never bind to 0.0.0.0 without a Reverse Proxy!

Always bind the engine strictly to 127.0.0.1 (localhost) and place a secure web server (like Caddy or Nginx) in front of it:

# SAFE DEPLOYMENT: Bind strictly to localhost (127.0.0.1) with 0.8 memory fraction

# SGLang Example:
python -m sglang.launch_server \
  --model-path Qwen/Qwen2.5-7B-Instruct \
  --host 127.0.0.1 --port 30000 \
  --mem-fraction-static 0.8

# vLLM Example:
vllm serve Qwen/Qwen2.5-7B-Instruct \
  --host 127.0.0.1 --port 8000 \
  --gpu-memory-utilization 0.8
Enter fullscreen mode Exit fullscreen mode

Phase 4: Multi-GPU & The ServerMO Bare Metal Advantage

To serve large models (like Llama 70B or DeepSeek) efficiently, you must split model weights across multiple GPUs using Tensor Parallelism (--tp 2 or --tp 8).

However, if you run these workloads in Docker containers on multi-GPU setups without the --ipc=host flag, the NVIDIA Collective Communications Library (NCCL) cannot utilize shared memory. This results in silent, catastrophic performance degradation.

Deploying LLM inference on shared Cloud VMs introduces hypervisor latency and "noisy neighbor" I/O contention. To achieve true microsecond TTFT and exploit full NVLink speeds required by vLLM and SGLang, deploy on ServerMO USA Dedicated Bare Metal Servers. Our infrastructure bypasses virtualization completely, offering dedicated PCIe Gen5 lanes and unmetered network bandwidth to ensure your inference engine operates at absolute peak theoretical throughput.

💬 SGLang & vLLM Inference FAQ

Which is better for Multi-Turn AI Agents: SGLang or vLLM?

SGLang is vastly superior for Multi-Turn AI Agents. While vLLM offers --enable-prefix-caching, its block-level storage struggles with complex branching. SGLang's Radix tree architecture handles multi-turn agents and dynamic context natively, delivering 5x faster TTFT and saving up to 80% VRAM.

Why does vLLM crash with OutOfMemoryError on an 80GB H100 GPU?

The crash is often caused by setting --gpu-memory-utilization to 0.9 or 0.95. During CUDA Graph capture, the engine allocates temporary System RAM proportional to the GPU memory. This exhausts the host machine's RAM, triggering an OS-level OOM kill. Always scale this down to 0.8 for stable compilation.

How do I fix FlashInfer compilation errors in SGLang on Ubuntu 24.04?

SGLang relies on FlashInfer to compile CUDA kernels on first launch. If you lack the ninja-build OS package, it will throw a FileNotFoundError. Install it via sudo apt install ninja-build. Also, ensure you fetch the latest FlashInfer wheel matching your CUDA environment.

Does SGLang support Multi-GPU Tensor Parallelism like vLLM?

Yes, SGLang fully supports Tensor Parallelism (e.g., --tp 2 or --tp 8). However, when running via Docker, you must include the --ipc=host flag. Without it, the NVIDIA Collective Communications Library (NCCL) cannot use shared memory for inter-GPU communication.

Why should I run LLM Inference on Bare Metal instead of Cloud VMs?

Cloud VMs introduce hypervisor latency and "noisy neighbor" I/O contention, which severely degrades Time-Per-Output-Token (TPOT). Bare Metal GPU servers provide unthrottled, direct access to PCIe Gen5 lanes and NVLink interconnects, extracting 100% of the hardware's theoretical throughput.

👉 Read the full benchmark on ServerMO:

SGLang vs vLLM: Install, Serve, and Benchmark on Bare Metal | ServerMO

Top comments (0)