Serve your own ChatGPT. How to Serve Inference from an Ops Perspective

Ebube Okeke — Thu, 23 Jul 2026 00:36:43 +0000

Self-Managed LLM Inference on Kubernetes — Part I: Deploying a Highly Available Inference Server

Background

In a world where LLMs have reshaped multiple industries, many enterprises are quickly realizing that this has come to stay. Closed models from frontier labs have driven innovation to the edge of possibility, and AI adoption has come at a pace some might consider unprecedented.

This development, however, has created its own set of new challenges that need to be solved — one of which is data privacy and sovereignty: how do you prevent handing your data over to frontier model providers to improve their models further and sell it back to you at a higher/lower token output cost?

At the same time, significant progress has been made in self-managed ML pipelines, from model training to inference. A few years ago (centuries ago in AI time), training models were thought to be the obvious challenge, with inference taking a back seat — maybe understandably so. But now, with the plethora of very capable closed and open models, inference became the million (maybe billion) dollar problem, and the market has reacted accordingly.

Enter Kubernetes: developed by Google initially (Borg), Kubernetes (k8s) has morphed to become perhaps the backbone of modern application infrastructure. The maintainers, with their foresight, have made AI objects first-class citizens. Given the massive and rapid improvements made over the years, ML workflows and specifically inference has become easier to democratize — well, as long as you can afford the GPUs to serve requests.

Series Overview

In this multi-part article I will be demonstrating and condensing some insight in:

Deploying inference workloads in a Highly Available (HA) setting
Enhancing models (specialists) using LoRA adapters and the challenges surrounding scaling, observability, and governance, some of which include:
- Respecting regulations
- Curating data
- Creating fact sheets

All of which are of extreme importance in highly regulated industries, as quoted from an IBM Expert:

Source attribution is the new explainability.

Why Run Your Own Inference?

Why would I run my own inference, and not just use a frontier model, you might ask? Well, good question. Here are just some reasons:

Your data belongs to you
Air-gapped environments — your infra can be run in air-gapped environments used by governments and offshore environments
Easier compliance — makes compliance and regulations easier to carry out
Secret containment — your users/employees can paste secrets/proprietary data (yes, they still do, and yes, it's still not advised), and you will be at peace knowing that your company secrets are not being used to train a general model somewhere in the cloud
Better performance — when fine-tuned
Faster response time — in pre-fill and decode stages
Cost-effectiveness — for the right use case, it can actually be more cost-effective

Rather than having a single model that tries to solve problems for all industries — thereby making you pay for that overhead — you can have your own model fine-tuned for your use case. Yes, it will not be Fable or Sol level, but it will be just as good and even better for you.

Fine-tuning will be covered in a future article. But how do you run the foundation model in the first place?

What You'll Learn in Part I

By the end of Part I, you will see how to:

Take a raw model and make it accept prompts, from both raw API requests and from the UI
Create a Grafana Dashboard to monitor model performance
Load test using Grafana K6s

Prerequisites

Category	Component
Infrastructure	Kubernetes cluster (GKE Autopilot), NVIDIA L4 GPU
Inference Server	vLLM
Model	`google/gemma-2-2b-it`
Observability	`kube-prometheus-operator` Helm chart
Security	Hugging Face API token (for pulling model weights)

This is not a step-by-step guide — it shows roughly all the components needed to stand up an inference server.

Setup

Step 1 — Spin Up a Cluster

Use the Google Cloud Console or an IaC tool like Terraform to bootstrap a Kubernetes cluster. This is where your model will be deployed.

Step 2 — Create Your Model Deployment

Create a deployment using the vLLM inference server image, passing your model in the args. On the Google Cloud Console, you can directly deploy your model, which automatically generates the manifest with the right annotations. You can then pull this and modify it as you see fit.

Note: Make sure to create a Kubernetes Secret containing your Hugging Face API token if you're pulling the model from Hugging Face.

Example Model Deployment YAML

apiVersion: apps/v1
kind: Deployment
metadata:
  name: inference-with-lora
  namespace: default
  labels:
    app: gemma-2-2b-it-vllm-inference-server
    examples.ai.gke.io/applied-by: gke-aiml-ui
    examples.ai.gke.io/deployment-id: inference-XgDD8
    examples.ai.gke.io/deployment-path: aire
    recommender.ai.gke.io/generated: "true"
    recommender.ai.gke.io/inference-server: vllm
spec:
  replicas: 1
  selector:
    matchLabels:
      app: gemma-2-2b-it-vllm-inference-server
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 25%
      maxSurge: 25%
  template:
    metadata:
      labels:
        ai.gke.io/inference-server: vllm
        ai.gke.io/model: gemma-2-2b-it
        app: gemma-2-2b-it-vllm-inference-server
        examples.ai.gke.io/applied-by: gke-aiml-ui
        examples.ai.gke.io/deployment-id: inference-XgDD8
        examples.ai.gke.io/deployment-path: aire
        examples.ai.gke.io/source: blueprints
        recommender.ai.gke.io/generated: "true"
        recommender.ai.gke.io/inference-server: vllm
    spec:
      # ------------------------------------------------------------------ #
      # Init container: pre-pull LoRA adapters from HF Hub into shared vol  #
      # so vLLM can load them from disk rather than fetching at request time #
      # ------------------------------------------------------------------ #
      initContainers:
        - name: pull-lora-adapters
          image: python:3.11-slim
          command:
            - /bin/sh
            - -c
            - |
              pip install -q huggingface_hub && \
              python3 - <<'EOF'
              from huggingface_hub import snapshot_download
              import os
              token = os.environ["HUGGING_FACE_HUB_TOKEN"]

              # Medical adapter — swap in your preferred adapter here
              snapshot_download(
                repo_id="azizdeniz890/gemma-medical-qa-lora",
                local_dir="/adapters/medical",
                token=token,
              )

              # Add more adapters below as needed, e.g.:
              # snapshot_download(
              #   repo_id="your-org/gemma-2-legal-lora",
              #   local_dir="/adapters/legal",
              #   token=token,
              # )
              EOF
          env:
            - name: HUGGING_FACE_HUB_TOKEN
              valueFrom:
                secretKeyRef:
                  name: inference-secret
                  key: hf_api_token
          volumeMounts:
            - name: adapters
              mountPath: /adapters

      containers:
        - name: inference-server
          # Bumped from v0.7.2 — v0.8.x+ has stable LoRA hub resolver +
          # runtime updating support. Pin to a specific version in prod.
          image: vllm/vllm-openai:v0.8.5
          ports:
            - name: metrics
              containerPort: 8000
              protocol: TCP
          command:
            - python3
            - -m
            - vllm.entrypoints.openai.api_server
          args:
            - --model=$(MODEL_ID)
            - --disable-log-requests
            - --tensor-parallel-size=1
            - --max-num-seq=1024
            - --num-scheduler-steps=8
            # ---- LoRA flags -------------------------------------------- #
            - --enable-lora
            # Must be >= the rank of any adapter you load.
            # azizdeniz890/gemma-medical-qa-lora uses rank 16.
            - --max-lora-rank=64
            # How many adapters to keep hot in GPU memory simultaneously.
            # Increase if you add more adapters; each costs ~rank * hidden_dim MB.
            - --max-loras=4
            # Keeps adapter weights in CPU memory between requests so they
            # don't need to be re-fetched from disk every time.
            - --max-cpu-loras=8
            # ------------------------------------------------------------ #
          env:
            - name: MODEL_ID
              value: google/gemma-2-2b-it
            - name: HUGGING_FACE_HUB_TOKEN
              valueFrom:
                secretKeyRef:
                  name: inference-secret
                  key: hf_api_token
            # ---- LoRA runtime env vars ---------------------------------- #
            # Allow adapters to be loaded/swapped without restarting vLLM
            - name: VLLM_ALLOW_RUNTIME_LORA_UPDATING
              value: "true"
            # Tell vLLM to look in the mounted /adapters directory first
            - name: VLLM_PLUGINS
              value: lora_filesystem_resolver
            - name: VLLM_LORA_RESOLVER_CACHE_DIR
              value: /adapters
            # ------------------------------------------------------------ #
          resources:
            limits:
              nvidia.com/gpu: "1"
            requests:
              nvidia.com/gpu: "1"
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 60   # was 0 — give vLLM time to load model
            periodSeconds: 10
            timeoutSeconds: 5
            successThreshold: 1
            failureThreshold: 600
          volumeMounts:
            - name: dshm
              mountPath: /dev/shm
            - name: adapters
              mountPath: /adapters
              readOnly: true

      volumes:
        - name: dshm
          emptyDir:
            medium: Memory
        # Shared volume between init container (writer) and main container (reader)
        - name: adapters
          emptyDir: {}

      nodeSelector:
        cloud.google.com/gke-accelerator: nvidia-l4
        cloud.google.com/gke-accelerator-count: "1"

Important: The annotations and nodeSelector fields regarding accelerator selection are critical — the pod will not be scheduled on GKE nodes without them.

This deployment has LoRA enabled with a built-in adapter; it will be used for the fine-tuned demo in the next part of the series.

Create a k8s service that proxies traffic for the deployment, and port-forward it for quick testing.

kubectl port-forward svc/inference-service 8000:8080 -n default

Send a curl request to the /v1/completions endpoint to quickly send a test prompt

curl -X POST http://localhost:8000/v1/completions \
    -H 'Content-Type: application/json' \
    -d '{
        "model": "google/gemma-2-2b-it",
        "prompt": "Why is the plant green?",
        "max_tokens": 512
    }' | jq

Observability

For observability, you can use the readily available kube-prometheus-operator community chart, which comes with predefined dashboards. However, many of those dashboards will not be relevant as they measure CPU workload metrics — the GPU is primarily being used to serve the model — so you will have to create your own dashboards.

A Bit on the Metrics

So you have deployed your model, and you send requests to your server getting your prompt response; now how do you actually measure how well your deployment works and compare it to others? Some of the relevant metrics, specific to the two stages of inference — prefill and decode — are:

Metric	Full Name	Description
TTFT	Time to First Token	How long it takes to get the first token response
Tokens/s	Tokens per Second	How many tokens your model can output per second
E2E Request Latency	End-to-End Request Latency	How long it takes from sending your prompt to the last token being returned by the model
Avg Queue Wait Time	Average Queue Wait Time	How long requests are being queued before being processed

Coming up Next

Now you are up and running, but you want to run your fine-tuned model alongside the base model; this is what we will discuss in the next part.

Thank You

If there's any topic you will want me to discuss next, add it to the comments and I will take a look

DEV Community: Ebube Okeke