Running large language models in production requires more than a GPU. Kubernetes has become the default orchestration layer for teams that need autoscaling, multi-tenancy, and hardware abstraction. This guide walks through deploying an open-source LLM on a Kubernetes cluster using vLLM, from node provisioning to autoscaling. If you would rather skip cluster management, Oxlo.ai offers a fully OpenAI-compatible inference API with flat per-request pricing and no cold starts on popular models.
Prerequisites
You will need a Kubernetes cluster with GPU support. Most cloud providers offer managed Kubernetes with NVIDIA GPU node pools. Before you start, install the following:
- NVIDIA GPU Operator to manage drivers, device plugins, and the Container Toolkit across nodes.
- kubectl and Helm configured for your cluster.
- Hugging Face access token if you are pulling gated models.
- A StorageClass that supports ReadWriteOnce volumes for caching model weights.
Choose an Inference Runtime
vLLM is a high-throughput serving engine with OpenAI-compatible endpoints. It supports continuous batching, PagedAttention, and tensor parallelism, which makes it a practical choice for Llama, Qwen, and DeepSeek families. Alternative runtimes like TGI or TensorRT-LLM are also valid, but this guide uses vLLM because its API shape mirrors what managed providers expose. In fact, Oxlo.ai exposes the same /v1/chat/completions schema, so you can migrate client code later without rewriting prompts or parsing logic.
Configure GPU Nodes and Scheduling
Label your GPU nodes so the scheduler places inference pods correctly. For example, if you are using NVIDIA L4 GPUs for inference:
kubectl label nodes <node-name> node-type=gpu-l4
Top comments (0)