How to Serve an LLM with an OpenAI-Compatible API on a Dedicated GPU Server

#ai #llm #devops #python

Running a large language model on your own hardware gives you control over deployment, inference resources, and API access. With an OpenAI-compatible API, your applications can talk to a self-hosted model using the same client libraries and request formats you may already use.

This post summarizes how to do it on a dedicated GPU server with vLLM, an open-source inference and serving engine. The full tutorial, with all commands and tables, is linked at the end.

What "OpenAI-compatible" means

vLLM exposes familiar endpoints:

/v1/chat/completions for conversational requests
/v1/completions for text generation from a prompt
/v1/models to list available models

One caveat: compatibility doesn't mean every OpenAI feature is available. Supported parameters depend on the inference engine and the model you deploy.

Hardware: VRAM comes first

GPU memory is the key constraint. You need room for the model weights, runtime overhead, and the KV cache. A 16-bit model needs roughly two bytes per parameter for raw weights alone, and real usage is higher. Size your GPU from the model's documented requirements and your expected workload, not parameter count alone.

Install vLLM

On Ubuntu 22.04 or 24.04, with a working NVIDIA driver (check with nvidia-smi), create an isolated environment and install vLLM:

curl -LsSf https://astral.sh/uv/install.sh | sh

mkdir ~/llm-server && cd ~/llm-server
uv venv --python 3.12 --seed
source .venv/bin/activate

uv pip install vllm --torch-backend=auto

Start the API server

The tutorial uses Qwen/Qwen2.5-7B-Instruct as an example. Check that your GPU has enough VRAM before choosing a model.

export MODEL_ID="Qwen/Qwen2.5-7B-Instruct"
export VLLM_API_KEY="replace-with-a-long-random-secret"

vllm serve "$MODEL_ID" \
    --host 127.0.0.1 \
    --port 8000 \
    --dtype auto \
    --max-model-len 4096 \
    --api-key "$VLLM_API_KEY"

Binding to 127.0.0.1 keeps the API off the public network, which is a safer starting point.

Test it with the OpenAI Python SDK

Because the API is compatible, you only change the base_url:

import os
from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8000/v1",
    api_key=os.environ["VLLM_API_KEY"]
)

response = client.chat.completions.create(
    model="Qwen/Qwen2.5-7B-Instruct",
    messages=[
        {"role": "system", "content": "You are a helpful technical assistant."},
        {"role": "user", "content": "Explain the benefits of dedicated GPU servers."}
    ],
    temperature=0.7,
    max_tokens=200
)

print(response.choices[0].message.content)

Security: don't rely on the API key alone

To serve remote applications, put Nginx in front of vLLM, enable HTTPS, and restrict access with firewall rules. The --api-key option doesn't cover every endpoint on the HTTP server (the /invocations endpoint needs extra protection), so never expose an unrestricted inference endpoint to the public internet.

Monitor and troubleshoot

Watch GPU utilization, memory, temperature, power, request latency, and tokens per second with watch -n 2 nvidia-smi. If memory runs short, reduce context length, choose a smaller model, or use a supported quantization method.

Read the full tutorial

The complete walkthrough, including parameter tables and a troubleshooting section, is on the Leo Servers tutorial page:

👉 How to Serve an LLM with an OpenAI-Compatible API on a GPU Server

Have you self-hosted an LLM before? Which inference engine did you use? Let me know in the comments.

DEV Community