DEV Community

Cover image for Self-Host an LLM Gateway on Kubernetes: A Weekend Guide
Kuldeep Paul
Kuldeep Paul

Posted on

Self-Host an LLM Gateway on Kubernetes: A Weekend Guide

#devops #kubernetes #ai #opensource

Self-Host an LLM Gateway on Kubernetes: A Weekend Guide

TL;DR

  • Self-hosting an LLM gateway on Kubernetes brings external model routing, credential security, token budgeting, and caching entirely within your cluster boundary.
  • Bifrost, an open-source AI gateway written in Go by Maxim AI, adds 11 microseconds of routing overhead at 5,000 requests per second.
  • Deploying the gateway with Helm provides built-in high availability, horizontal autoscaling, Kubernetes pod discovery, and native Prometheus telemetry.
  • Teams can complete a production-grade Kubernetes deployment over a single weekend by provisioning PostgreSQL storage, injecting encrypted API keys, and configuring Ingress controllers.

Production AI applications running across multiple commercial and local model providers fail when upstream APIs return rate-limit errors, network timeouts, or sudden outages. Rather than hardcoding provider credentials and retry loops into separate microservices, platform engineers route inference calls through a dedicated ingress point. To self-host an LLM gateway on Kubernetes means operating a centralized proxy inside your own cloud boundary to control model routing, credentials, and telemetry without sending prompt logs to third-party services. Bifrost is an open-source AI gateway designed for high-concurrency environments, supporting drop-in SDK replacements across supported providers. This weekend guide walks through the architectural choices, Helm configurations, ingress setups, and observability pipelines required to run a scalable LLM gateway on Kubernetes.

Why Self-Host an LLM Gateway on Kubernetes?

A self-hosted LLM gateway centralizes AI routing, key rotation, and request governance within an organization's existing Kubernetes infrastructure to eliminate security blind spots and vendor lock-in. Instead of exposing raw provider keys to every developer pod, the gateway accepts requests via virtual credentials, checks policy boundaries, and distributes traffic across providers.

+-----------------------------------------------------------------------+
|                         Kubernetes Cluster                            |
|                                                                       |
|  +--------------------+       +------------------------------------+  |
|  | Application Pods   | ----> | Bifrost Gateway Service            |  |
|  | (Client SDKs)      |       | (ClusterIP / Ingress)              |  |
|  +--------------------+       +-----------------+------------------+  |
|                                                 |                     |
|                   +-----------------------------+-----------+         |
|                   |                             |           |         |
|                   v                             v           v         |
|         +-------------------+         +-------------+  +------------+ |
|         | Virtual Keys      |         | Semantic    |  | Prometheus | |
|         | & Rate Limits     |         | Caching     |  | Metrics    | |
|         +-------------------+         +-------------+  +------------+ |
|                   |                             |                     |
+-------------------|-----------------------------|---------------------+
                    v                             v
       +-------------------------+   +-------------------------+
       | Upstream Cloud APIs     |   | In-Cluster Inference    |
       | (OpenAI, Bedrock, etc.) |   | (vLLM / Triton / Ollama)|
       +-------------------------+   +-------------------------+
Enter fullscreen mode Exit fullscreen mode

When engineering teams scale generative AI features, five infrastructure challenges arise almost immediately:

  • Credential dispersion: Distributing proprietary API keys across dozens of Kubernetes namespaces, secrets, and developer repositories increases the risk of secret leaks.
  • Provider instability: Upstream provider downtime or rate limits (HTTP 429 status codes) break application workflows unless complex fallback logic is re-implemented across every service.
  • Cost opacity: Without centralized attribution, platform teams cannot trace which microservices or user groups generate token spikes.
  • Compliance boundaries: Regulated environments subject to HIPAA, SOC 2, or GDPR require strict control over prompt residency, preventing data from flowing through intermediate SaaS proxies.
  • Latency penalties: Adding proxy layers can degrade the user experience of streaming text tokens if the proxy architecture introduces millisecond-level bottlenecks.

Operating the gateway inside Kubernetes resolves these operational issues. The cluster scheduler manages pod health, native services route internal RPCs over private subnets, and in-VPC deployments ensure that unencrypted prompts never touch external infrastructure before routing to authorized model providers.

Architecture of a Self-Hosted LLM Gateway on Kubernetes

A production LLM gateway architecture on Kubernetes decouples the stateless routing data plane from stateful storage layers such as PostgreSQL and Redis. The gateway pods handle HTTP and gRPC traffic, stream Server-Sent Events (SSE) back to client pods, and synchronize cluster state across nodes.

A sleek modern networking switch in an infrastructure rack with fiber-optic strands splitting cleanly into multiple illu

When designing the deployment, platform engineers must choose between a lightweight single-replica setup for development or a multi-replica clustered topology for mission-critical workloads. Bifrost supports both operating models out of the box through its official Helm chart.

Deployment Model Compute Pattern Storage Backend State Synchronization Target Workload
Sandbox / Staging StatefulSet (1 Replica) Local SQLite (PersistentVolumeClaim) Single-process memory Local developer clusters, integration testing
High Availability (HA) Deployment (3+ Replicas) Managed PostgreSQL (RDS/Cloud SQL) Kubernetes API Pod Discovery Production microservices, internal enterprise APIs
Distributed Multi-Zone Multi-AZ Deployment High-Availability PostgreSQL + Redis Gossip-based peer discovery Cross-region failover, mission-critical production

In production setups, stateful components should remain decoupled from gateway pods. SQLite on a PersistentVolumeClaim works well for sandboxes, but shared persistent storage introduces locking issues when scaling out to multiple replicas. A stateless Deployment backed by a managed PostgreSQL instance allows the Horizontal Pod Autoscaler (HPA) to scale pods dynamically based on CPU and concurrent request metrics.

Prerequisites and Pre-Flight Cluster Planning

Deploying an enterprise-grade LLM gateway requires a Kubernetes cluster running version 1.19 or higher, administrative access via kubectl, and Helm 3.2.0 or newer. Additionally, verify that your cluster has a default StorageClass configured if persistent volumes are required for local caching.

Before applying manifests, complete the following pre-flight preparations:

  1. Namespace Isolation: Dedicate a distinct namespace (such as bifrost) to isolate network policies, role bindings, and secrets.
  2. PostgreSQL Database: Provision a dedicated PostgreSQL database instance (version 14 or later) with UTF-8 encoding. Ensure network connectivity from the Kubernetes worker node subnets to the database port.
  3. Master Encryption Key: Generate a cryptographically secure 32-byte key using OpenSSL to encrypt provider tokens and virtual keys at rest.
  4. Provider Credentials: Gather API keys for OpenAI, Anthropic, AWS Bedrock, Google Vertex AI, or local in-cluster endpoints like vLLM.
  5. Ingress and DNS: Ensure an Ingress controller (such as ingress-nginx) is operational if external or cross-cluster access is required.

According to official Kubernetes networking documentation, using explicit namespaces and standard Ingress controllers simplifies network policy management and TLS termination across microservice boundaries.

Step 1: Configuring Storage, Secrets, and the Helm Repository

Begin the setup on Saturday morning by initializing the Bifrost Helm repository and creating Kubernetes secrets to hold sensitive credentials. Never store API keys or database passwords in plaintext Helm values files.

First, add and update the official Bifrost Helm repository:

helm repo add bifrost https://maximhq.github.io/bifrost/helm-charts
helm repo update
Enter fullscreen mode Exit fullscreen mode

Next, create the dedicated namespace:

kubectl create namespace bifrost
Enter fullscreen mode Exit fullscreen mode

Generate the master encryption key and store it inside a native Kubernetes Secret. This key secures sensitive credentials in the database:

# Generate a 32-byte base64-encoded key
ENCRYPTION_KEY=$(openssl rand -base64 32)

kubectl create secret generic bifrost-encryption-key \
  --namespace bifrost \
  --from-literal=encryption-key="${ENCRYPTION_KEY}"
Enter fullscreen mode Exit fullscreen mode

Create a second secret containing upstream provider credentials and PostgreSQL connection parameters:

kubectl create secret generic bifrost-core-secrets \
  --namespace bifrost \
  --from-literal=postgres-password="YourSecureDbPasswordHere" \
  --from-literal=openai-api-key="sk-proj-yourOpenAiKey" \
  --from-literal=anthropic-api-key="sk-ant-yourAnthropicKey"
Enter fullscreen mode Exit fullscreen mode

Managing secrets through native Kubernetes mechanisms allows integration with external secret stores such as HashiCorp Vault, AWS Secrets Manager, or Google Secret Manager via the Kubernetes External Secrets Operator.

Step 2: Deploying Bifrost with Helm and High Availability

Deploying Bifrost in high availability mode ensures that incoming inference requests continue processing even if a worker node fails or undergoes rolling updates. This step uses a structured values.yaml file to configure multi-replica scaling, database connectivity, and resource reservations.

Create a local deployment configuration file named bifrost-production-values.yaml:

# bifrost-production-values.yaml
image:
  repository: maximhq/bifrost
  tag: v1.4.11
  pullPolicy: IfNotPresent

replicaCount: 3

bifrost:
  encryptionKeySecret:
    name: bifrost-encryption-key
    key: encryption-key

  database:
    type: postgres
    host: "postgres-service.database.svc.cluster.local"
    port: 5432
    name: "bifrost_db"
    user: "bifrost_user"
    sslmode: "require"
    passwordSecret:
      name: bifrost-core-secrets
      key: postgres-password

  clustering:
    enabled: true
    discoveryMethod: "kubernetes"
    k8sLabelSelector: "app.kubernetes.io/name=bifrost"

  providers:
    openai:
      keys:
        - name: "primary-openai"
          value: "env.OPENAI_API_KEY"
          weight: 1
    anthropic:
      keys:
        - name: "primary-anthropic"
          value: "env.ANTHROPIC_API_KEY"
          weight: 1

providerSecrets:
  openai:
    existingSecret: bifrost-core-secrets
    key: openai-api-key
    envVar: OPENAI_API_KEY
  anthropic:
    existingSecret: bifrost-core-secrets
    key: anthropic-api-key
    envVar: ANTHROPIC_API_KEY

resources:
  requests:
    cpu: "250m"
    memory: "512Mi"
  limits:
    cpu: "2000m"
    memory: "2Gi"

autoscaling:
  enabled: true
  minReplicas: 3
  maxReplicas: 10
  targetCPUUtilizationPercentage: 70

affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchExpressions:
              - key: app.kubernetes.io/name
                operator: In
                values:
                  - bifrost
          topologyKey: "kubernetes.io/hostname"
Enter fullscreen mode Exit fullscreen mode

Apply the Helm release using the custom configuration file:

helm install bifrost bifrost/bifrost \
  --namespace bifrost \
  -f bifrost-production-values.yaml
Enter fullscreen mode Exit fullscreen mode

The configuration sets pod anti-affinity rules, instructing Kubernetes to schedule gateway replicas across distinct physical worker nodes. Clustering uses native Kubernetes API discovery to locate peer pods and sync state across the cluster.

Verify that all pods have reached the Running state:

kubectl get pods -n bifrost -l app.kubernetes.io/name=bifrost
Enter fullscreen mode Exit fullscreen mode

Step 3: Configuring Ingress, TLS, and Health Probes

Exposing the LLM gateway to client microservices inside or outside the cluster requires reliable networking rules and continuous health checks. Streaming LLM tokens rely on long-lived HTTP connections, meaning ingress proxies must be configured to prevent connection drops mid-generation.

Bifrost serves HTTP traffic on port 8080 and exposes a dedicated /health endpoint on the same port. Configure readiness and liveness probes in your Helm values or deployment manifests:

livenessProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 15
  periodSeconds: 10
  timeoutSeconds: 3
  failureThreshold: 3

readinessProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 5
  timeoutSeconds: 2
  successThreshold: 1
  failureThreshold: 2
Enter fullscreen mode Exit fullscreen mode

For ingress routing via ingress-nginx, define an Ingress manifest that disables proxy response buffering. Buffering breaks Server-Sent Events, increasing time-to-first-token (TTFT) for user-facing applications.

# bifrost-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: bifrost-ingress
  namespace: bifrost
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-buffering: "off"
    nginx.ingress.kubernetes.io/proxy-body-size: "50m"
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
  tls:
    - hosts:
        - llm-gateway.internal.example.com
      secretName: bifrost-tls-cert
  rules:
    - host: llm-gateway.internal.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: bifrost
                port:
                  number: 8080
Enter fullscreen mode Exit fullscreen mode

Apply the ingress manifest:

kubectl apply -f bifrost-ingress.yaml
Enter fullscreen mode Exit fullscreen mode

Once applied, applications can call the gateway using its fully qualified domain name (FQDN). Bifrost serves as a drop-in replacement for OpenAI-compatible SDKs, requiring changes only to the base URL parameter in application code.

Step 4: Setting Up Model Routing, Fallbacks, and Virtual Keys

Configuring intelligent routing, fallback chains, and virtual keys ensures resilient operation under provider outages while enforcing departmental spend controls. The gateway handles model translation transparently, allowing client pods to request a standard model alias while the gateway routes to available backends.

Through Bifrost's configuration engine, administrators configure automatic fallbacks. If a primary provider returns an HTTP 500 or 429 status code, traffic diverts instantly to an alternative model or provider without throwing an exception to the client pod.

Incoming Request: "claude-3-5-sonnet"
                  |
                  v
       +-----------------------+
       | Bifrost Proxy Engine  |
       +-----------+-----------+
                   |
       +-----------+-----------+
       |                       |
(Primary: Healthy)     (Primary Fails: 429 / 5xx)
       |                       |
       v                       v
+---------------+      +-----------------------+
| Anthropic API |      | AWS Bedrock Fallback  |
| Direct        |      | (claude-3-5-sonnet)   |
+---------------+      +-----------------------+
Enter fullscreen mode Exit fullscreen mode

To govern access across teams, configure virtual keys. A virtual key acts as a virtual token issued to specific application deployments or teams, enforcing budgets, rate limits, and allowed model lists.

{
  "name": "data-science-team-key",
  "budget": {
    "amount": 500.0,
    "currency": "USD",
    "reset_period": "monthly"
  },
  "rate_limits": {
    "requests_per_minute": 120,
    "tokens_per_minute": 150000
  },
  "allowed_models": [
    "openai/gpt-4o",
    "anthropic/claude-3-5-sonnet",
    "meta-llama/llama-3.1-70b"
  ]
}
Enter fullscreen mode Exit fullscreen mode

In addition to routing, enabling semantic caching reduces inference spend and response latency by serving semantically similar requests directly from an in-cluster vector cache without invoking upstream APIs. Review the comprehensive LLM Gateway Buyer's Guide to evaluate how caching mechanisms lower overall token overhead across large engineering organizations.

Step 5: Enabling Observability with Prometheus and OpenTelemetry

Observability is essential when managing production LLM traffic. Platform engineers must monitor token consumption, time-to-first-token latency, error rates, and cache hit ratios across all services. Bifrost provides native Prometheus metrics and distributed tracing via OpenTelemetry without requiring external daemon sidecars.

The gateway exposes Prometheus metrics natively. If your cluster operates the Prometheus Operator, configure a ServiceMonitor resource to scrape metrics on port 8080:

# bifrost-servicemonitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: bifrost-metrics
  namespace: bifrost
  labels:
    release: prometheus-stack
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: bifrost
  endpoints:
    - port: http
      path: /metrics
      interval: 15s
      scrapeTimeout: 10s
Enter fullscreen mode Exit fullscreen mode

Apply the ServiceMonitor configuration:

kubectl apply -f bifrost-servicemonitor.yaml
Enter fullscreen mode Exit fullscreen mode

The Prometheus metrics include request counts, token counters split by prompt and completion, error statuses, and histogram latencies.

For distributed tracing across microservices, Bifrost exports trace telemetry using the OpenTelemetry (OTLP) protocol. Traces capture the full request lifecycle, including request parsing, semantic cache lookup, upstream provider latency, and token streaming durations. Standardizing on OpenTelemetry standards enables platform teams to visualize request flows in Grafana Tempo, Jaeger, or Honeycomb.

In sustained tests published on the official benchmarks page, Bifrost introduced only 11 microseconds of overhead per request under sustained loads of 5,000 requests per second. This negligible latency makes it suitable for high-throughput microservices where intermediate network hops must not impact user experience.

Step 6: Extending Cluster Governance to Developer Workstations with Bifrost Edge

While securing server-side microservices within Kubernetes is a major milestone, ungoverned AI access on developer laptops creates an organizational blind spot known as shadow AI. Beyond routing, Bifrost applies governance and security controls (virtual keys, budgets, guardrails, audit logs) centrally, and Bifrost Edge extends that same governance and security to AI traffic on employee machines, with endpoint enforcement on each device.

A clean workstation desk featuring a sleek laptop connected by a subtle luminous thread back toward a central rack-mount

Bifrost operates as the centralized control plane and policy engine deployed on your infrastructure, while Bifrost Edge, currently in alpha, acts as the endpoint extension running on macOS, Windows, and Linux machines. It routes desktop AI tools, browser interfaces, and terminal coding assistants through the central Kubernetes gateway without requiring developers to change their local configuration files.

Key capabilities delivered by combining the Kubernetes gateway with Bifrost Edge include:

  • Unified policy enforcement: Developers running desktop apps like Claude Desktop, Cursor, or terminal tools like Claude Code automatically inherit the virtual keys, budgets, and rate limits defined on the Kubernetes gateway.
  • Application governance: Security administrators use app governance to approve or block specific desktop AI clients across employee machines.
  • MCP server visibility: As developers wire Model Context Protocol (MCP) servers into their local environments, Edge inventories and controls tool permissions fleet-wide.
  • Enterprise rollout: Systems teams deploy the Edge client silently across fleets via MDM deployment tools like Jamf, Microsoft Intune, and Kandji.

By extending the same governance policies enforced inside your Kubernetes cluster directly onto engineer workstations, platform teams maintain complete visibility over AI operations across both cloud services and local endpoints.

Operational Maintenance and Day-2 Guide

Operating an LLM gateway in production requires structured maintenance workflows for schema updates, provider key rotation, and disaster recovery. The following routine maintenance procedures ensure uninterrupted service.

Zero-Downtime Key Rotation

When rotating upstream provider keys (such as an OpenAI or Anthropic corporate credential), update the underlying Kubernetes secret:

kubectl create secret generic bifrost-core-secrets \
  --namespace bifrost \
  --from-literal=openai-api-key="sk-proj-newKeyHere" \
  --dry-run=client -o yaml | kubectl apply -f -
Enter fullscreen mode Exit fullscreen mode

Because Bifrost loads environment variables and watches configuration updates, performing a rollout restart propagates the updated credentials across all running pods without dropping active sessions:

kubectl rollout restart deployment/bifrost -n bifrost
Enter fullscreen mode Exit fullscreen mode

Database Schema Migrations

Bifrost executes necessary schema migrations automatically upon startup. When upgrading gateway container versions, deploy an updated image tag through Helm. Multi-replica deployments coordinate startup sequences safely across peer nodes, ensuring that new migrations complete before incoming HTTP traffic switches to upgraded pods.

According to the official Helm documentation, utilizing declarative values files with strict version pinning (--set image.tag=v1.4.11) prevents accidental upgrades during automated CI/CD pipeline runs.

Frequently Asked Questions

What hardware resources are required to run an LLM gateway on Kubernetes?

A standard production deployment of Bifrost running three replicas requires minimal compute: approximately 250m CPU and 512Mi memory requests per pod, with limits set to 2 CPU and 2Gi memory. Because the gateway is compiled in Go and functions as a high-throughput network proxy rather than an inference engine, it runs efficiently on standard commodity CPU nodes without requiring expensive GPU accelerators.

Can a self-hosted LLM gateway route to local models running in the same cluster?

Yes. Bifrost routes requests to any OpenAI-compatible HTTP endpoint. This includes self-hosted open-weight model runtimes running inside the same Kubernetes cluster, such as vLLM, KServe, Triton Inference Server, or Ollama. Internal cluster services can be targeted directly using standard Kubernetes DNS names like http://vllm-service.inference.svc.cluster.local:8000/v1.

How does the gateway handle streaming responses via Server-Sent Events?

Bifrost streams Server-Sent Events (SSE) natively using chunked transfer encoding. When deploying behind Kubernetes Ingress controllers (such as ingress-nginx or Traefik), ensure that proxy response buffering is explicitly disabled. Disabling buffering ensures that generated tokens stream directly from upstream providers to client pods with zero intermediate buffering delay.

Does self-hosting an LLM gateway ensure complete data privacy?

Self-hosting the gateway ensures that all prompt logs, API keys, virtual tokens, routing rules, and internal metrics remain strictly within your private network boundary. However, if you route requests to external commercial model APIs (such as OpenAI or Anthropic), the prompt payload will still leave your network to reach that specific vendor unless you route exclusively to local in-cluster model runtimes.

How do virtual keys differ from native provider API keys?

Native provider API keys grant unrestricted access to a specific provider account and are difficult to rotate safely across distributed applications. Virtual keys are internal tokens generated and validated by the gateway. They allow administrators to assign fine-grained budgets, rate limits, and model access permissions to individual teams without ever revealing the master provider keys to application developers.

Is SQLite recommended for production Kubernetes deployments?

No. SQLite requires a PersistentVolumeClaim and locks the database file during concurrent write operations, which prevents scaling the gateway beyond a single pod. For production environments requiring high availability and autoscaling, always configure an external PostgreSQL database with UTF-8 encoding.

Getting Started and Next Steps

Self-hosting an LLM gateway on Kubernetes provides engineering teams with complete control over AI traffic, data privacy, model routing, and cost attribution. Over a single weekend, you can transition your infrastructure from scattered API tokens and brittle client-side retries to a centralized, highly available control plane.

Once your gateway is operational, test failure scenarios by intentionally revoking a primary provider key to verify that fallback chains route traffic smoothly to secondary models. Teams evaluating enterprise AI gateways can request a Bifrost demo or review the open-source repository to inspect the codebase and join the community.

Sources

Top comments (0)