Self-Managed LLM Inference on Kubernetes — Part I: Deploying a Highly Available Inference Server
Background
In a world where LLMs have reshaped multiple industries, many enterprises are quickly realizing that this has come to stay. Closed models from frontier labs have driven innovation to the edge of possibility, and AI adoption has come at a pace some might consider unprecedented.
This development, however, has created its own set of new challenges that need to be solved — one of which is data privacy and sovereignty: how do you prevent handing your data over to frontier model providers to improve their models further and sell it back to you at a higher/lower token output cost?
At the same time, significant progress has been made in self-managed ML pipelines, from model training to inference. A few years ago (centuries ago in AI time), training models were thought to be the obvious challenge, with inference taking a back seat — maybe understandably so. But now, with the plethora of very capable closed and open models, inference became the million (maybe billion) dollar problem, and the market has reacted accordingly.
Enter Kubernetes: developed by Google initially (Borg), Kubernetes (k8s) has morphed to become perhaps the backbone of modern application infrastructure. The maintainers, with their foresight, have made AI objects first-class citizens. Given the massive and rapid improvements made over the years, ML workflows and specifically inference has become easier to democratize — well, as long as you can afford the GPUs to serve requests.
Series Overview
In this multi-part article I will be demonstrating and condensing some insight in:
- Deploying inference workloads in a Highly Available (HA) setting
- Enhancing models (specialists) using LoRA adapters and the challenges surrounding scaling, observability, and governance, some of which include:
- Respecting regulations
- Curating data
- Creating fact sheets
All of which are of extreme importance in highly regulated industries, as quoted from an IBM Expert:
Source attribution is the new explainability.
Why Run Your Own Inference?
Why would I run my own inference, and not just use a frontier model, you might ask? Well, good question. Here are just some reasons:
- Your data belongs to you
- Air-gapped environments — your infra can be run in air-gapped environments used by governments and offshore environments
- Easier compliance — makes compliance and regulations easier to carry out
- Secret containment — your users/employees can paste secrets/proprietary data (yes, they still do, and yes, it's still not advised), and you will be at peace knowing that your company secrets are not being used to train a general model somewhere in the cloud
- Better performance — when fine-tuned
- Faster response time — in pre-fill and decode stages
- Cost-effectiveness — for the right use case, it can actually be more cost-effective
Rather than having a single model that tries to solve problems for all industries — thereby making you pay for that overhead — you can have your own model fine-tuned for your use case. Yes, it will not be Fable or Sol level, but it will be just as good and even better for you.
Fine-tuning will be covered in a future article. But how do you run the foundation model in the first place?
What You'll Learn in Part I
By the end of Part I, you will see how to:
- Take a raw model and make it accept prompts, from both raw API requests and from the UI
- Create a Grafana Dashboard to monitor model performance
- Load test using Grafana K6s
Prerequisites
| Category | Component |
|---|---|
| Infrastructure | Kubernetes cluster (GKE Autopilot), NVIDIA L4 GPU |
| Inference Server | vLLM |
| Model | google/gemma-2-2b-it |
| Observability |
kube-prometheus-operator Helm chart |
| Security | Hugging Face API token (for pulling model weights) |
This is not a step-by-step guide — it shows roughly all the components needed to stand up an inference server.
Setup
Step 1 — Spin Up a Cluster
Use the Google Cloud Console or an IaC tool like Terraform to bootstrap a Kubernetes cluster. This is where your model will be deployed.
Step 2 — Create Your Model Deployment
Create a deployment using the vLLM inference server image, passing your model in the args. On the Google Cloud Console, you can directly deploy your model, which automatically generates the manifest with the right annotations. You can then pull this and modify it as you see fit.
Note: Make sure to create a Kubernetes Secret containing your Hugging Face API token if you're pulling the model from Hugging Face.
Example Model Deployment YAML
apiVersion: apps/v1
kind: Deployment
metadata:
name: inference-with-lora
namespace: default
labels:
app: gemma-2-2b-it-vllm-inference-server
examples.ai.gke.io/applied-by: gke-aiml-ui
examples.ai.gke.io/deployment-id: inference-XgDD8
examples.ai.gke.io/deployment-path: aire
recommender.ai.gke.io/generated: "true"
recommender.ai.gke.io/inference-server: vllm
spec:
replicas: 1
selector:
matchLabels:
app: gemma-2-2b-it-vllm-inference-server
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 25%
maxSurge: 25%
template:
metadata:
labels:
ai.gke.io/inference-server: vllm
ai.gke.io/model: gemma-2-2b-it
app: gemma-2-2b-it-vllm-inference-server
examples.ai.gke.io/applied-by: gke-aiml-ui
examples.ai.gke.io/deployment-id: inference-XgDD8
examples.ai.gke.io/deployment-path: aire
examples.ai.gke.io/source: blueprints
recommender.ai.gke.io/generated: "true"
recommender.ai.gke.io/inference-server: vllm
spec:
# ------------------------------------------------------------------ #
# Init container: pre-pull LoRA adapters from HF Hub into shared vol #
# so vLLM can load them from disk rather than fetching at request time #
# ------------------------------------------------------------------ #
initContainers:
- name: pull-lora-adapters
image: python:3.11-slim
command:
- /bin/sh
- -c
- |
pip install -q huggingface_hub && \
python3 - <<'EOF'
from huggingface_hub import snapshot_download
import os
token = os.environ["HUGGING_FACE_HUB_TOKEN"]
# Medical adapter — swap in your preferred adapter here
snapshot_download(
repo_id="azizdeniz890/gemma-medical-qa-lora",
local_dir="/adapters/medical",
token=token,
)
# Add more adapters below as needed, e.g.:
# snapshot_download(
# repo_id="your-org/gemma-2-legal-lora",
# local_dir="/adapters/legal",
# token=token,
# )
EOF
env:
- name: HUGGING_FACE_HUB_TOKEN
valueFrom:
secretKeyRef:
name: inference-secret
key: hf_api_token
volumeMounts:
- name: adapters
mountPath: /adapters
containers:
- name: inference-server
# Bumped from v0.7.2 — v0.8.x+ has stable LoRA hub resolver +
# runtime updating support. Pin to a specific version in prod.
image: vllm/vllm-openai:v0.8.5
ports:
- name: metrics
containerPort: 8000
protocol: TCP
command:
- python3
- -m
- vllm.entrypoints.openai.api_server
args:
- --model=$(MODEL_ID)
- --disable-log-requests
- --tensor-parallel-size=1
- --max-num-seq=1024
- --num-scheduler-steps=8
# ---- LoRA flags -------------------------------------------- #
- --enable-lora
# Must be >= the rank of any adapter you load.
# azizdeniz890/gemma-medical-qa-lora uses rank 16.
- --max-lora-rank=64
# How many adapters to keep hot in GPU memory simultaneously.
# Increase if you add more adapters; each costs ~rank * hidden_dim MB.
- --max-loras=4
# Keeps adapter weights in CPU memory between requests so they
# don't need to be re-fetched from disk every time.
- --max-cpu-loras=8
# ------------------------------------------------------------ #
env:
- name: MODEL_ID
value: google/gemma-2-2b-it
- name: HUGGING_FACE_HUB_TOKEN
valueFrom:
secretKeyRef:
name: inference-secret
key: hf_api_token
# ---- LoRA runtime env vars ---------------------------------- #
# Allow adapters to be loaded/swapped without restarting vLLM
- name: VLLM_ALLOW_RUNTIME_LORA_UPDATING
value: "true"
# Tell vLLM to look in the mounted /adapters directory first
- name: VLLM_PLUGINS
value: lora_filesystem_resolver
- name: VLLM_LORA_RESOLVER_CACHE_DIR
value: /adapters
# ------------------------------------------------------------ #
resources:
limits:
nvidia.com/gpu: "1"
requests:
nvidia.com/gpu: "1"
readinessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 60 # was 0 — give vLLM time to load model
periodSeconds: 10
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 600
volumeMounts:
- name: dshm
mountPath: /dev/shm
- name: adapters
mountPath: /adapters
readOnly: true
volumes:
- name: dshm
emptyDir:
medium: Memory
# Shared volume between init container (writer) and main container (reader)
- name: adapters
emptyDir: {}
nodeSelector:
cloud.google.com/gke-accelerator: nvidia-l4
cloud.google.com/gke-accelerator-count: "1"
Important: The
annotationsandnodeSelectorfields regarding accelerator selection are critical — the pod will not be scheduled on GKE nodes without them.
This deployment has LoRA enabled with a built-in adapter; it will be used for the fine-tuned demo in the next part of the series.
Create a k8s service that proxies traffic for the deployment, and port-forward it for quick testing.
kubectl port-forward svc/inference-service 8000:8080 -n default
Send a curl request to the /v1/completions endpoint to quickly send a test prompt
curl -X POST http://localhost:8000/v1/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "google/gemma-2-2b-it",
"prompt": "Why is the plant green?",
"max_tokens": 512
}' | jq
Observability
For observability, you can use the readily available kube-prometheus-operator community chart, which comes with predefined dashboards. However, many of those dashboards will not be relevant as they measure CPU workload metrics — the GPU is primarily being used to serve the model — so you will have to create your own dashboards.
A Bit on the Metrics
So you have deployed your model, and you send requests to your server getting your prompt response; now how do you actually measure how well your deployment works and compare it to others? Some of the relevant metrics, specific to the two stages of inference — prefill and decode — are:
| Metric | Full Name | Description |
|---|---|---|
| TTFT | Time to First Token | How long it takes to get the first token response |
| Tokens/s | Tokens per Second | How many tokens your model can output per second |
| E2E Request Latency | End-to-End Request Latency | How long it takes from sending your prompt to the last token being returned by the model |
| Avg Queue Wait Time | Average Queue Wait Time | How long requests are being queued before being processed |
Coming up Next
Now you are up and running, but you want to run your fine-tuned model alongside the base model; this is what we will discuss in the next part.
Thank You
If there's any topic you will want me to discuss next, add it to the comments and I will take a look
Screenshot dump
Top comments (0)