How to setup a secure Inference Platform using LiteLLM and vLLM

Introduction

In my last post I showed how to run an air-gapped LLM using Ollama on a single machine. That works great for one person, but as soon as a whole team, or even multiple teams, want to use the models, you need something more robust: concurrent requests, API keys per user, cost tracking and a single, secured entry point.

This article will show how I set up exactly that on Kubernetes, using three building blocks:

• vLLM as the inference server, which actually runs the model on the GPU
• LiteLLM as the gateway in front of it, which handles API keys, routing and spend tracking
• An Ingress controller as the single entry point, which terminates TLS

The request flow looks like this:

Client ──HTTPS──► Ingress ──► LiteLLM (API key check) ──► vLLM (GPU)

As an example model I'm using Qwen3.8-27B in NVFP4, which fits nicely on a single 40 GB GPU.

Info: I'm running this on a VMware Tanzu (VKS) cluster with NVIDIA vGPUs, but everything shown here works on any Kubernetes flavour with GPU nodes, be it AKS, EKS, GKE or a bare metal cluster.

Why not just use Ollama?

Fair question, since Ollama is what I used last time. The short answer is concurrency. Ollama is great for a single user, but it effectively processes requests one after another. vLLM uses continuous batching and PagedAttention, so it can serve many requests at once on the same GPU. On my hardware, with the very same model, I measured this:

Concurrent requests	Ollama (tok/s total)	vLLM (tok/s total)
1	44.8	52.4
5	45.9	230.7
10	47.6	427.1
15	47.8	608.8

As soon as more than one person uses the platform, vLLM wins by a lot.

Prerequisites: The GPU Operator

Before any pod can request a GPU, your nodes need a driver, the NVIDIA container toolkit and the device plugin. The easiest way to get all of that is the NVIDIA GPU Operator. I won't go into detail here, since NVIDIA documents it really well:

• GPU Operator - Getting Started
• GPU Operator with NVIDIA vGPU, if you are running on vSphere like me

Info: If you are on vGPU, the open drivers won't bind to your GPU. You need the licensed GRID guest driver, and its version must match the vGPU Manager on your ESXi host. In my case I installed the driver directly on the nodes and run the operator with driver.enabled: false.

Once the operator is up, you can check that your nodes advertise their GPUs:

kubectl get nodes -o custom-columns='NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu'

The operator also creates the nvidia RuntimeClass, which we'll need in a moment.

Setting up vLLM

The namespace

I start with a dedicated namespace for vLLM. Since vLLM can run fully rootless, I enforce the strictest Pod Security Standard on it:

apiVersion: v1
kind: Namespace
metadata:
  name: vllm
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

Storage for the model weights

The model weights live on a PersistentVolumeClaim, so they don't have to be downloaded again on every pod restart:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: vllm-models-pvc
  namespace: vllm
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: <your-rwx-storage-class>
  resources:
    requests:
      storage: 100Gi

Info: I'm using ReadWriteMany (NFS) here on purpose. The vLLM namespace has no internet access at all (more on that below), so a separate, internet-capable "loader" namespace mounts the same NFS export and downloads the weights from Hugging Face onto it. This way, the pod that actually processes your data never needs to talk to the outside world. It's the same idea as exporting the volume in my Ollama post, just on Kubernetes.

The vLLM Deployment

Now to the interesting part. vLLM ships an official image with an OpenAI-compatible server, so all I have to do is pass the right flags:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-qwen38
  namespace: vllm
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: vllm
      instance: vllm-qwen38
  template:
    metadata:
      labels:
        app: vllm
        instance: vllm-qwen38
    spec:
      runtimeClassName: nvidia
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
      enableServiceLinks: false
      securityContext:
        runAsNonRoot: true
        runAsUser: 2000
        runAsGroup: 2000
        fsGroup: 2000
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: vllm
          image: vllm/vllm-openai:v0.28.0
          args:
            - "--host"
            - "0.0.0.0"
            - "--port"
            - "8000"
            - "--model"
            - "Inferact/Qwen3.8-27B-NVFP4"
            - "--served-model-name"
            - "qwen3.8-27b-vllm"
            - "--gpu-memory-utilization"
            - "0.85"
            - "--max-model-len"
            - "32768"
            - "--kv-cache-dtype"
            - "fp8"
            - "--enable-prefix-caching"
            - "--enable-auto-tool-choice"
            - "--tool-call-parser"
            - "qwen3_coder"
            - "--reasoning-parser"
            - "qwen3"
          env:
            - name: HF_HOME
              value: /models/hf
            - name: VLLM_CACHE_ROOT
              value: /models/hf/vllm-cache
            - name: HF_HUB_OFFLINE
              value: "1"
            - name: HOME
              value: /home/vllm
          resources:
            limits:
              nvidia.com/gpu: "1"
              memory: "48Gi"
            requests:
              cpu: "8"
              memory: "16Gi"
          ports:
            - name: http
              containerPort: 8000
          startupProbe:
            httpGet:
              path: /health
              port: 8000
            periodSeconds: 15
            failureThreshold: 80
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            periodSeconds: 15
          volumeMounts:
            - name: models
              mountPath: /models
            - name: home
              mountPath: /home/vllm
            - name: dshm
              mountPath: /dev/shm
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
      volumes:
        - name: models
          persistentVolumeClaim:
            claimName: vllm-models-pvc
        - name: home
          emptyDir: {}
        - name: dshm
          emptyDir:
            medium: Memory
            sizeLimit: 8Gi

There are a few things in here I'd like to point out:

• --served-model-name is the name the model is addressed by. We'll need this exact name again in the LiteLLM config.
• HF_HUB_OFFLINE=1 makes vLLM fail fast if a weight file is missing, instead of trying to reach huggingface.co.
• --kv-cache-dtype fp8 halves the memory needed per token in the KV cache. This is what lets you fit a long context and many parallel requests on one GPU.
• strategy: Recreate prevents a rolling update from starting a second pod, which would just sit in Pending because the only GPU is still taken.
• /dev/shm as a memory-backed volume. The container default of 64 MiB is too small for PyTorch and leads to really obscure crashes.
• The generous startupProbe. A cold start (loading the model plus torch.compile) can easily take 5 minutes. Since VLLM_CACHE_ROOT lives on the PVC, the next restart is way faster.

If you are on vGPU: never set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, even though PyTorch's own out-of-memory message recommends it. vGPU doesn't implement the needed CUDA APIs and vLLM will crash right at startup with operation not supported. Also keep in mind that the vGPU reserves around 3 GB of memory for itself, so --gpu-memory-utilization values above ~0.91 won't work on a 40 GB profile.

The Service

vLLM gets a plain ClusterIP Service. It must not be reachable from outside the cluster, since vLLM itself has no authentication at all:

apiVersion: v1
kind: Service
metadata:
  name: vllm-qwen38
  namespace: vllm
spec:
  type: ClusterIP
  selector:
    app: vllm
    instance: vllm-qwen38
  ports:
    - name: http
      port: 8000
      targetPort: 8000

Locking down the network

Now the part that makes this setup actually secure. I add two NetworkPolicies to the vLLM namespace. The first one only allows LiteLLM to talk to vLLM:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: vllm-allow-litellm-only
  namespace: vllm
spec:
  podSelector:
    matchLabels:
      app: vllm
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: litellm
          podSelector:
            matchLabels:
              app: litellm
      ports:
        - protocol: TCP
          port: 8000

The second one blocks all egress to the internet. Only DNS and cluster-internal (RFC1918) traffic is allowed:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: vllm-egress-internal-only
  namespace: vllm
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53
    - to:
        - ipBlock:
            cidr: 10.0.0.0/8
        - ipBlock:
            cidr: 172.16.0.0/12
        - ipBlock:
            cidr: 192.168.0.0/16

Info: Images are pulled by the kubelet on the node, so they are not affected by this policy. The pod itself however can't reach anything outside of your network anymore. That is basically the Kubernetes version of the iptables DROP rule from my Ollama post.

After applying everything, it's time to wait for the model to load:

kubectl -n vllm rollout status deployment/vllm-qwen38 --timeout=30m

Setting up LiteLLM

vLLM gives us a fast, OpenAI-compatible API, but without any users, keys or limits. That's where LiteLLM comes in. It sits in front of one or more inference servers and adds:

• Virtual API keys per user or team, with model access, budgets and rate limits
• Spend and usage tracking
• Load balancing across multiple instances of the same model
• An Admin UI to manage all of the above

Postgres and Secrets

For virtual keys and spend tracking, LiteLLM needs a Postgres database. I run a simple single-replica Postgres in the litellm namespace with a ClusterIP Service named postgres. Nothing special, so I'll skip the manifest here.

The secrets for LiteLLM are created straight from the command line, so they never end up in a YAML file in git:

kubectl create namespace litellm

kubectl create secret generic litellm-secrets -n litellm \
  --from-literal=LITELLM_MASTER_KEY="sk-$(openssl rand -hex 24)" \
  --from-literal=LITELLM_SALT_KEY="$(openssl rand -hex 32)" \
  --from-literal=DATABASE_URL="postgresql://<user>:<password>@postgres.litellm.svc.cluster.local:5432/litellm"

Make sure to keep the LITELLM_SALT_KEY safe and never change it after the first start. It's used to encrypt the credentials stored in the database. If it changes, LiteLLM can't decrypt them anymore.

Hooking up LiteLLM with vLLM

This is where LiteLLM gets connected to vLLM. The config lives in a ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: litellm-config
  namespace: litellm
data:
  config.yaml: |
    model_list:
      - model_name: qwen3.8-27b
        litellm_params:
          model: openai/qwen3.8-27b-vllm
          api_base: http://vllm-qwen38.vllm.svc.cluster.local:8000/v1
          api_key: "none"
        model_info:
          max_input_tokens: 32768
          max_output_tokens: 32768

    router_settings:
      routing_strategy: usage-based-routing-v2

    general_settings:
      master_key: os.environ/LITELLM_MASTER_KEY
      store_model_in_db: true

What is happening here:

• model_name is the public name your users will call the model by.
• model: openai/qwen3.8-27b-vllm: since vLLM speaks the OpenAI API, I use the openai/ provider. The part after the slash must match vLLM's --served-model-name.
• api_base is the cluster-internal DNS name of the vLLM Service: <service>.<namespace>.svc.cluster.local.
• api_key: "none": vLLM has no auth. The security comes from the NetworkPolicy, which only lets LiteLLM in.

Info: If you later want to scale out, just deploy a second vLLM instance with its own Service and add a second entry under the same model_name. LiteLLM then balances the requests across both. I prefer this over a Deployment with replicas: 2, since Kubernetes Services balance per connection and not per request, which doesn't work well with keep-alive HTTP clients.

The LiteLLM Deployment

I use the litellm-database image, since it ships the Prisma binaries and runs the database migrations automatically:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: litellm
  namespace: litellm
spec:
  replicas: 1
  selector:
    matchLabels:
      app: litellm
  template:
    metadata:
      labels:
        app: litellm
    spec:
      containers:
        - name: litellm
          image: ghcr.io/berriai/litellm-database:main-stable
          args: ["--config", "/app/proxy_config.yaml"]
          envFrom:
            - secretRef:
                name: litellm-secrets
          ports:
            - name: http
              containerPort: 4000
          readinessProbe:
            httpGet:
              path: /health/readiness
              port: 4000
            initialDelaySeconds: 60
            periodSeconds: 15
          livenessProbe:
            httpGet:
              path: /health/liveliness
              port: 4000
            initialDelaySeconds: 120
            periodSeconds: 15
          volumeMounts:
            - name: config
              mountPath: /app/proxy_config.yaml
              subPath: config.yaml
              readOnly: true
      volumes:
        - name: config
          configMap:
            name: litellm-config
---
apiVersion: v1
kind: Service
metadata:
  name: litellm-service
  namespace: litellm
spec:
  type: ClusterIP
  selector:
    app: litellm
  ports:
    - name: http
      port: 4000
      targetPort: 4000

Again, I'm using a ClusterIP Service here. In my first iteration, LiteLLM had its own LoadBalancer IP. That turned out to be a plaintext back door that skipped TLS completely, so I removed it. The ingress should be the only way in.

Same as for vLLM, I add an egress policy to the litellm namespace that blocks the internet. LiteLLM doesn't need it: the optional fetch of the model cost map falls back to a bundled copy. I also add an ingress policy so that only the ingress controller can reach LiteLLM:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: litellm-allow-ingress-only
  namespace: litellm
spec:
  podSelector:
    matchLabels:
      app: litellm
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: ingress-nginx
      ports:
        - protocol: TCP
          port: 4000

Setting up the Ingress

The last piece is the single entry point. I'm using ingress-nginx behind one LoadBalancer Service with a pinned IP, so my DNS record stays valid even if the Service gets recreated.

Heads up: the ingress-nginx project was retired in March 2026 and no longer gets updates or security fixes. I'm still showing it here because that's what my cluster runs today, but for a new setup I'd recommend a Gateway API implementation. The concepts (TLS termination, streaming-friendly timeouts, a single entry point) are exactly the same.

Installing the controller

These are the Helm values I use:

controller:
  replicaCount: 2
  service:
    type: LoadBalancer
    loadBalancerIP: "<your-pinned-ip>"
  ingressClassResource:
    name: nginx
    enabled: true
    default: true
  config:
    proxy-body-size: "0"
    proxy-read-timeout: "3600"
    proxy-send-timeout: "3600"

The config block is important for LLM traffic. Prompts can be large, and the responses are streamed as Server-Sent Events, sometimes for minutes. With the nginx defaults (1 MB body limit, 60 second timeout), long generations would simply get cut off.

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm upgrade --install ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx --create-namespace \
  -f ingress-nginx-values.yaml

Hooking up the Ingress with LiteLLM

Now the Ingress resource, which connects the outside world to LiteLLM:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: litellm-ingress
  namespace: litellm
  annotations:
    cert-manager.io/cluster-issuer: <your-cluster-issuer>
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/proxy-buffering: "off"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - llm.example.com
      secretName: llm-example-com-tls
  rules:
    - host: llm.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: litellm-service
                port:
                  number: 4000

• cert-manager.io/cluster-issuer tells cert-manager to issue a certificate for the host and store it in the Secret from the tls block. No manual certificate handling needed.
• proxy-buffering: "off" is the one people forget most often. Without it, nginx buffers the streamed response and your users see the whole answer at once, after a long wait, instead of token by token.
• ingress-nginx redirects plain HTTP to HTTPS by default, so nothing goes over the wire unencrypted.

Finally, point a DNS A record for llm.example.com at the pinned IP of the ingress controller.

Testing the whole chain

First, a quick health check through the ingress:

curl https://llm.example.com/health/liveliness

If LiteLLM answers with "I'm alive!", the Ingress → LiteLLM part works. Next, I create a virtual key for a user, which only has access to our Qwen model:

curl -X POST https://llm.example.com/key/generate \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{"models": ["qwen3.8-27b"], "key_alias": "florian-test", "max_budget": 10}'

And with the returned key, I can now throw my first prompt at the model:

curl https://llm.example.com/v1/chat/completions \
  -H "Authorization: Bearer <virtual-key>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3.8-27b",
    "messages": [{"role": "user", "content": "Explain Kubernetes NetworkPolicies in one sentence."}]
  }'

If an answer comes back, the full chain Ingress → LiteLLM → vLLM is working. And since the whole thing speaks the OpenAI API, you can point basically any tool at it, be it a chat frontend, a VS Code extension or your own app using the OpenAI SDK, just by changing the base URL and the API key.

Where to go from here

The setup above already gives you a solid baseline. In my cluster I went a few steps further, and I might cover these in future posts:

• A service mesh (Linkerd) for mTLS on every hop inside the cluster, so even the traffic from the ingress to LiteLLM and from LiteLLM to vLLM is encrypted and mutually authenticated.
• Monitoring with Prometheus and Grafana: GPU utilization via DCGM, time to first token, and usage per team, using LiteLLM's built-in Prometheus metrics.
• Multiple models and instances behind the same LiteLLM gateway, including embedding models for RAG.

Conclusion

With vLLM serving the model, LiteLLM handling keys and routing, and a single TLS-secured ingress in front, I now have my own OpenAI-compatible inference platform. My data never leaves my network: the GPU workload can't reach the internet, vLLM is only reachable by LiteLLM, and LiteLLM is only reachable through HTTPS with a valid key. That's a great basis for a secure coding assistant, an internal chatbot, or any other Gen AI use case that has to work with restricted data.