Running a large language model on dedicated hardware gives you full control over the serving environment, model configuration and API access. In this guide, we deploy GLM-4.5-Air-FP8 from Z.ai on an NVIDIA GPU server using SGLang, exposing an OpenAI-compatible API.
This is a condensed version. The complete tutorial is on Leo Servers.
About the model
GLM-4.5-Air is a mixture-of-experts (MoE) model built for reasoning, coding and agent tasks. It has about 106B total parameters and about 12B active during inference. We use the FP8 checkpoint:
zai-org/GLM-4.5-Air-FP8
Hardware
Z.ai documents these NVIDIA configurations for GLM-4.5-Air-FP8:
| Use case | GPUs |
|---|---|
| Standard FP8 inference | 2x H100 |
| Full 128K context | 4x H100 |
The full-featured conditions also call for more than 1TB of system RAM, a batch size of eight or less, and no CPU offloading. Treat these as a reference, not a universal minimum.
Prepare Ubuntu and verify the GPUs
sudo apt update && sudo apt upgrade -y
sudo apt install -y curl git wget build-essential
ubuntu-drivers devices
sudo ubuntu-drivers install
sudo reboot
After the reboot:
nvidia-smi
nvidia-smi -L
nvidia-smi topo -m
Install SGLang
curl -LsSf https://astral.sh/uv/install.sh | sh
source "$HOME/.local/bin/env"
mkdir -p ~/glm-deployment && cd ~/glm-deployment
uv venv --python 3.12
source .venv/bin/activate
uv pip install --prerelease=allow sglang
Point the Hugging Face cache at a large volume:
sudo install -d -o "$USER" -g "$(id -gn)" /data/huggingface
echo 'export HF_HOME=/data/huggingface' >> ~/.bashrc
source ~/.bashrc
Launch the model
python -m sglang.launch_server \
--model-path zai-org/GLM-4.5-Air-FP8 \
--served-model-name glm-4.5-air-fp8 \
--host 127.0.0.1 \
--port 30000
Binding to 127.0.0.1 keeps the API off the public interface.
Test the API
curl http://127.0.0.1:30000/v1/models
from openai import OpenAI
client = OpenAI(api_key="local-development", base_url="http://127.0.0.1:30000/v1")
response = client.chat.completions.create(
model="glm-4.5-air-fp8",
messages=[{"role": "user", "content": "Explain the role of GPU memory in LLM inference."}],
max_tokens=300,
)
print(response.choices[0].message.content)
Run it with systemd and secure it
A systemd unit keeps SGLang running and restarts it on failure. For remote access, use an authenticated HTTPS reverse proxy, firewall rules and private networking. Never expose an unauthenticated inference endpoint to the internet.
Read the full guide
The full tutorial includes the complete systemd unit file, monitoring tips and a troubleshooting table:
👉 How to Deploy Zhipu AI GLM-4.5-Air with SGLang on a Dedicated GPU Server
Top comments (0)