DEV Community

Cover image for How to Deploy GLM-4.5-Air with SGLang on a Dedicated GPU Server
Thea Lauren
Thea Lauren

Posted on Edited on Originally published at leoservers.com

How to Deploy GLM-4.5-Air with SGLang on a Dedicated GPU Server

Running a large language model on dedicated hardware gives you full control over the serving environment, model configuration and API access. In this guide, we deploy GLM-4.5-Air-FP8 from Z.ai on an NVIDIA GPU server using SGLang, exposing an OpenAI-compatible API.

This is a condensed version. The complete tutorial is on Leo Servers.

About the model

GLM-4.5-Air is a mixture-of-experts (MoE) model built for reasoning, coding and agent tasks. It has about 106B total parameters and about 12B active during inference. We use the FP8 checkpoint:

zai-org/GLM-4.5-Air-FP8
Enter fullscreen mode Exit fullscreen mode

Hardware

Z.ai documents these NVIDIA configurations for GLM-4.5-Air-FP8:

Use case GPUs
Standard FP8 inference 2x H100
Full 128K context 4x H100

The full-featured conditions also call for more than 1TB of system RAM, a batch size of eight or less, and no CPU offloading. Treat these as a reference, not a universal minimum.

Prepare Ubuntu and verify the GPUs

sudo apt update && sudo apt upgrade -y
sudo apt install -y curl git wget build-essential
ubuntu-drivers devices
sudo ubuntu-drivers install
sudo reboot
Enter fullscreen mode Exit fullscreen mode

After the reboot:

nvidia-smi
nvidia-smi -L
nvidia-smi topo -m
Enter fullscreen mode Exit fullscreen mode

Install SGLang

curl -LsSf https://astral.sh/uv/install.sh | sh
source "$HOME/.local/bin/env"
mkdir -p ~/glm-deployment && cd ~/glm-deployment
uv venv --python 3.12
source .venv/bin/activate
uv pip install --prerelease=allow sglang
Enter fullscreen mode Exit fullscreen mode

Point the Hugging Face cache at a large volume:

sudo install -d -o "$USER" -g "$(id -gn)" /data/huggingface
echo 'export HF_HOME=/data/huggingface' >> ~/.bashrc
source ~/.bashrc
Enter fullscreen mode Exit fullscreen mode

Launch the model

python -m sglang.launch_server \
    --model-path zai-org/GLM-4.5-Air-FP8 \
    --served-model-name glm-4.5-air-fp8 \
    --host 127.0.0.1 \
    --port 30000
Enter fullscreen mode Exit fullscreen mode

Binding to 127.0.0.1 keeps the API off the public interface.

Test the API

curl http://127.0.0.1:30000/v1/models
Enter fullscreen mode Exit fullscreen mode
from openai import OpenAI

client = OpenAI(api_key="local-development", base_url="http://127.0.0.1:30000/v1")

response = client.chat.completions.create(
    model="glm-4.5-air-fp8",
    messages=[{"role": "user", "content": "Explain the role of GPU memory in LLM inference."}],
    max_tokens=300,
)
print(response.choices[0].message.content)
Enter fullscreen mode Exit fullscreen mode

Run it with systemd and secure it

A systemd unit keeps SGLang running and restarts it on failure. For remote access, use an authenticated HTTPS reverse proxy, firewall rules and private networking. Never expose an unauthenticated inference endpoint to the internet.

Read the full guide

The full tutorial includes the complete systemd unit file, monitoring tips and a troubleshooting table:

👉 How to Deploy Zhipu AI GLM-4.5-Air with SGLang on a Dedicated GPU Server

Top comments (0)