Nerav Doshi

Posted on Jul 12 • Originally published at pipelineandprompts.com

Swapping LLM Providers Without Rewriting Your Stack

#openshift #platformengineering #litellm #aiinthestack

🤖 AI in the Stack #5

Pipeline & Prompts | Byte size guides on DevOps, Cloud and AI

⚡ Byte Size Summary

Deploy a LiteLLM proxy on OpenShift to decouple your applications from individual LLM providers — switch or failover between models by changing a ConfigMap, not your code

Inject cluster-specific context (API versions, cloud annotations, golden templates) before every prompt to prevent models from generating "plausible averages" that look correct but fail on your platform

Validate every generated manifest with oc apply --dry-run=server before applying — the cluster's API server catches structural errors that no amount of prompt engineering will prevent

The Story

I was building a GenAI data poisoning demo for a proof of concept on ROSA (Red Hat OpenShift on AWS). The demo needed OpenShift-native manifests — Routes with TLS termination, proper SecurityContextConstraints, current API versions. I built it using Claude, and it worked. The model understood OpenShift. The manifests deployed cleanly. The demo ran on the cluster.

Then I ran out of free credits.

I was under a deadline, so I switched to OpenAI. The first manifest it generated used a Kubernetes Ingress instead of an OpenShift Route. The API versions referenced documentation that was two releases behind our cluster version. The YAML was syntactically valid. It was functionally wrong for OpenShift.

I tried to fix it by researching the correct documentation through NotebookLM, then feeding corrections back into the model. When that didn't converge, I switched to Gemini. Gemini gave me enough directional guidance that I could manually correct the manifests — using my own OpenShift knowledge, not the model's.

Two days later, I had a working demo again and ten-plus markdown files across my project. I couldn't tell which model had generated which file, which version was current, or which one had actually deployed successfully on the cluster. The project state had diverged so badly across three providers that rolling back to any previous working state was impossible — even if the credits had come back.

My model selection strategy during those two days was trial and error. I switched between Sonnet and Opus without understanding the difference between them. If a model's output deployed on the cluster, I stuck with it for that session. Every new session was a fresh gamble.

The ROSA cluster was my only validation oracle. If oc apply succeeded, the manifest stayed. If it errored, I had the model strip the broken section out and try again.

The failure wasn't switching providers. The failure was having no system in place that could make any provider produce correct output — no context injection, no validation gate, no abstraction layer. The fix is designing the control plane first — so models can be wrong without causing damage.

The Problem

LLMs generate statistically averaged infrastructure artifacts. They don't know your cluster — they know all clusters. The output is a blend of Kubernetes, OpenShift, ROSA, ARO (Azure Red Hat OpenShift), EKS (Amazon Elastic Kubernetes Service), and whatever else was in the training data. It looks right because it's an average of right things. An average of all correct configurations is correct for none of them.

This manifests in three ways:

Provider coupling. Your prompts are implicitly tuned to a specific model's training data and behavior. Switch providers — because of cost, credits, rate limits, or compliance — and the same prompt produces different output. Not always wrong output. Different output that may or may not be valid for your specific environment. You don't find out until you try to deploy it.

Silent knowledge gaps. A model that knows Kubernetes well may not know OpenShift. It generates a networking.k8s.io/v1 Ingress where route.openshift.io/v1 is what your cluster expects. It uses AWS load balancer annotations for a manifest targeting an ARO cluster. The output is valid YAML, valid Kubernetes — and wrong for your platform. There's no error message. The model doesn't know what it doesn't know.

No validation boundary. When a model generates a manifest, nothing in the default workflow checks whether it's correct for your cluster version, your cloud provider, or your installed operators. The human is the only validation layer — and under deadline pressure, the human trusts the output that looks right.

Why Existing Approaches Fall Short

"Just switch models." This was my first instinct. Claude ran out of credits, try OpenAI. OpenAI broke things, try Gemini. Each switch introduced different knowledge gaps without solving the underlying problem: no model has complete, current knowledge of your specific platform. Switching providers is moving the problem, not fixing it.

"Write better prompts." Prompt engineering helps with reasoning tasks — structuring output, defining roles, specifying constraints. It doesn't fix knowledge gaps. Telling a model "generate OpenShift-native manifests" doesn't inject the knowledge it needs about your cluster's API versions or your cloud provider's annotation requirements. If the information isn't in the model's training data, no prompt will extract it.

"Run it locally." I run Ollama on a 16GB Mac with models up to 14B parameters — Gemma 3 12B, DeepSeek-R1 14B, Qwen 2.5 Coder 14B. At that size, the models don't have meaningful OpenShift-specific knowledge. Loading a 9GB model on a machine with 16GB total — while running an IDE, terminal, and browser — means switching between models causes memory exhaustion. Local inference is a privacy-preserving fallback, not a primary working tier, for most practitioners on standard-issue hardware.

"Pick one provider and stay." Until you can't. Free credits expire mid-demo. Rate limits hit during a customer presentation. Your compliance team flags a data residency concern. A premium reasoning model handles many of the same tasks a cheap utility model could — paying for capability you don't need on every request adds up fast. The question isn't whether you'll need to switch providers — it's whether your system survives the switch when it happens.

The Architecture

The architecture has three layers, each preventing a different failure mode:

Layer 1 — Context Injection (Pre-processing). Before any prompt reaches a model, platform-specific context is injected: your cluster's API versions, your cloud provider's annotation requirements, your organization's golden templates. This separates "what the model knows" from "what your environment requires" — and fills the gap with verified context, not model guessing.

Layer 2 — LiteLLM Gateway (Routing). A LiteLLM proxy running in the ai-gateway namespace provides a single OpenAI-compatible API endpoint. Applications call the proxy, not individual providers. Model selection, failover, and tiering happen in the proxy configuration — not in application code. Three tiers:

Tier	Purpose	Characteristics
Utility	YAML generation, CRDs, JSON extraction, glue code	Cheap, fast, deterministic
Reasoning	Security analysis, architecture decisions, multi-hop logic	Slower, more expensive
Escape hatch	Edge cases, novel problems	Manual or explicit opt-in

Layer 3 — Post-Generation Validation (Post-processing). Every generated manifest runs through oc apply --dry-run=server before being applied. The cluster's API server validates structure, API versions, and field correctness against the actual cluster state — not against documentation that may be stale.

The key design insight: if a task is hard because of logic, route to a stronger model. If a task is hard because of domain knowledge, inject the missing context and keep it on a cheap model. Knowledge gaps are a context problem, not a model problem.

New to evaluating AI tooling on OpenShift? AI in the Stack #1 — AI Tooling on OpenShift: A Practitioner's Evaluation Framework covers the governance layer this gateway builds on.

How It Works: Step by Step

Prerequisites

OpenShift 4.14+ cluster (ROSA, ARO, or self-managed)
oc CLI 4.14+
At least one LLM provider API key (OpenAI, Anthropic, or both)
Namespace creation privileges on the cluster

Step 1 — Configure Model Tiers

The LiteLLM proxy configuration defines your model tiers and failover behavior:

# config.yaml — LiteLLM proxy configuration
model_list:
  - model_name: utility
    litellm_params:
      model: gpt-4.1-mini
      api_key: os.environ/OPENAI_API_KEY

  - model_name: utility-fallback
    litellm_params:
      model: claude-haiku-4-5-20251001
      api_key: os.environ/ANTHROPIC_API_KEY

  - model_name: reasoning
    litellm_params:
      model: claude-sonnet-5
      api_key: os.environ/ANTHROPIC_API_KEY

litellm_settings:
  fallbacks:
    - utility: [utility-fallback]
  num_retries: 2
  request_timeout: 30

router_settings:
  routing_strategy: simple-shuffle

The fallbacks setting means if the utility model fails — rate limit, timeout, provider outage — LiteLLM automatically retries with the fallback. Your application code doesn't change. It still sends requests to model: "utility" and the proxy handles the routing.

Step 2 — Build the Context Layer

Store platform-specific context in a ConfigMap. This is the knowledge your models lack — injected before every prompt, regardless of which provider handles the request:

# manifests/context-templates.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: openshift-context-templates
  namespace: ai-gateway
data:
  openshift-base.md: |
    You are generating manifests for an OpenShift 4.14+ cluster.

    Required API versions:
    - Deployments: apps/v1
    - Services: v1
    - Routes: route.openshift.io/v1
    - NetworkPolicies: networking.k8s.io/v1

    Rules:
    - Use OpenShift Routes, NOT Kubernetes Ingress
    - Never use deprecated APIs (extensions/v1beta1, apps/v1beta1)
    - Include resource requests and limits on all containers
    - Use securityContext with runAsNonRoot: true

  rosa-annotations.md: |
    Target platform: ROSA (Red Hat OpenShift on AWS)

    For Services of type LoadBalancer:
    - service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
    - service.beta.kubernetes.io/aws-load-balancer-scheme: "internal"

    Do NOT use Azure-specific annotations on this cluster.

  aro-annotations.md: |
    Target platform: ARO (Azure Red Hat OpenShift)

    For Services of type LoadBalancer:
    - service.beta.kubernetes.io/azure-load-balancer-internal: "true"

    Do NOT use AWS-specific annotations on this cluster.

Your application selects the appropriate context file based on the target cluster. On ROSA, prepend openshift-base.md and rosa-annotations.md to the system message. On ARO, swap in aro-annotations.md. The model never has to guess which cloud it's targeting.

Step 3 — Deploy the Gateway on OpenShift

oc new-project ai-gateway

Create the API key Secret — populate values from your external secret management system, never hardcode keys in manifests committed to Git:

# manifests/secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: llm-api-keys
  namespace: ai-gateway
type: Opaque
stringData:
  OPENAI_API_KEY: ""
  ANTHROPIC_API_KEY: ""

Deploy the proxy, service, and route:

# manifests/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: litellm-proxy
  namespace: ai-gateway
spec:
  replicas: 2
  selector:
    matchLabels:
      app: litellm-proxy
  template:
    metadata:
      labels:
        app: litellm-proxy
    spec:
      serviceAccountName: litellm-proxy
      containers:
        - name: litellm
          image: ghcr.io/berriai/litellm:main-latest
          ports:
            - containerPort: 4000
          envFrom:
            - secretRef:
                name: llm-api-keys
          volumeMounts:
            - name: config
              mountPath: /app/config.yaml
              subPath: config.yaml
              readOnly: true
          args: ["--config", "/app/config.yaml", "--port", "4000"]
          resources:
            requests:
              cpu: 250m
              memory: 512Mi
            limits:
              cpu: 500m
              memory: 1Gi
          livenessProbe:
            httpGet:
              path: /health
              port: 4000
            initialDelaySeconds: 15
            periodSeconds: 30
          readinessProbe:
            httpGet:
              path: /health
              port: 4000
            initialDelaySeconds: 5
            periodSeconds: 10
          securityContext:
            runAsNonRoot: true
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
      volumes:
        - name: config
          configMap:
            name: litellm-config
---
apiVersion: v1
kind: Service
metadata:
  name: litellm-proxy
  namespace: ai-gateway
spec:
  selector:
    app: litellm-proxy
  ports:
    - port: 4000
      targetPort: 4000
---
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: litellm-proxy
  namespace: ai-gateway
spec:
  to:
    kind: Service
    name: litellm-proxy
  port:
    targetPort: 4000
  tls:
    termination: edge

Load the configuration and deploy:

oc create configmap litellm-config \
  --from-file=config.yaml \
  -n ai-gateway

oc apply -f manifests/context-templates.yaml
oc apply -f manifests/secret.yaml
oc apply -f manifests/deployment.yaml

Verify the proxy is running:

oc get pods -n ai-gateway -l app=litellm-proxy
oc get route litellm-proxy -n ai-gateway -o jsonpath='{.spec.host}'

Step 4 — Wire Post-Generation Validation

The minimum viable safety net — run this after every manifest generation, before oc apply:

#!/bin/bash
# scripts/validate-manifest.sh
set -euo pipefail

MANIFEST="${1:?Usage: validate-manifest.sh <manifest.yaml>}"

echo "Validating against cluster API server..."
if oc apply --dry-run=server -f "$MANIFEST" 2>&1; then
  echo "PASS: Manifest accepted by $(oc whoami --show-server)"
else
  echo "FAIL: Manifest rejected — check API versions and resource types"
  exit 1
fi

What --dry-run=server catches:

API versions that don't exist on this cluster
Resource types not installed (missing CRDs — Custom Resource Definitions — or operators)
Required fields that are missing
Field type mismatches

What it does NOT catch:

Wrong cloud-provider annotations (annotations are unvalidated metadata)
Runtime behavior (a valid Deployment can still crash-loop)
Cross-resource dependency issues (Service referencing a non-existent Deployment)

For annotation validation on ROSA, add a targeted check:

# scripts/check-cloud-annotations.sh
if grep -q "azure-load-balancer" "$MANIFEST"; then
  echo "WARN: Azure LB annotation found — target cluster is ROSA"
  exit 1
fi

Step 5 — Lock Down Access

# manifests/rbac-and-network.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: litellm-proxy
  namespace: ai-gateway
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: litellm-proxy-access
  namespace: ai-gateway
spec:
  podSelector:
    matchLabels:
      app: litellm-proxy
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              ai-gateway-access: "true"
      ports:
        - port: 4000
          protocol: TCP

Only namespaces with the ai-gateway-access: "true" label can reach the proxy. Label the namespaces that need access:

oc label namespace my-ai-workloads ai-gateway-access=true

Security and Operational Considerations

API keys belong in Secrets, not ConfigMaps. The llm-api-keys Secret is referenced via envFrom — keys are injected as environment variables, never mounted as files and never committed to Git. In production, populate this Secret from your external secret management system (Vault, AWS Secrets Manager, Azure Key Vault) using an operator like External Secrets.

Operational IDs leak infrastructure details. When troubleshooting with an LLM, you can scrub cluster names, VPC IDs, and IAM roles from the prompt context. You can't easily scrub operational IDs — error codes, correlation IDs, operation request IDs — because they're the thing you need the model to analyze. Those IDs can reveal internal infrastructure patterns, timing, and region information depending on the error format. For sensitive troubleshooting, route to local models and accept the latency cost. For everything else, the gateway's routing tier makes this a configuration choice, not a code change.

Infrastructure topology is the quieter data leak. Even without credentials, the manifests you send to external providers reveal your namespace naming conventions, RBAC (role-based access control) structures, network policy architecture, and CRD layouts. The context templates compound this — they describe your exact cluster configuration. Ensure context injection happens in-cluster (before the request leaves your network) so the platform knowledge stays on your infrastructure, even when the LLM call goes external.

The Route should use TLS termination appropriate to your environment. The example uses tls.termination: edge. For environments where the payload between the Route and the proxy must also be encrypted, use reencrypt and configure a serving certificate on the proxy container.

Restrict egress from the gateway namespace. The proxy needs outbound access to LLM provider APIs. It does not need access to your internal services. Add an egress NetworkPolicy that allows HTTPS to external endpoints and blocks everything else — this prevents the proxy from being used as an unintended hop into your internal network.

What Breaks at Scale

Cloud-specific annotations are the silent failure. In a multi-cluster deployment spanning ROSA and ARO, I've seen models confidently generate AWS Network Load Balancer annotations for manifests targeting an ARO cluster. The manifest passes --dry-run=server — annotations are unvalidated metadata. The failure surfaces only at runtime when traffic doesn't route correctly. The context templates per cluster are the mitigation, but they require discipline: every cluster needs its own context maintained, and the mapping from target cluster to correct context file must be automated, not manual.

Cluster versions diverge. Your staging cluster is on OpenShift 4.15. Production is still on 4.14 because the upgrade window hasn't opened. A manifest generated with 4.15 API assumptions may reference resources or fields that don't exist on 4.14. The --dry-run=server validation catches this — but only if you run it against the target cluster. Validating against staging and deploying to production defeats the purpose.

Knowledge fragments across teams. Five engineers, three different LLM providers, each with their own prompt patterns and context assumptions. The LiteLLM gateway standardizes the API layer. It doesn't standardize the context layer. Shared context templates in a ConfigMap help — but they require the same governance discipline as any other shared configuration artifact. Someone has to own the templates, review changes, and ensure they stay current when operators are upgraded or CRDs change.

LLMs smooth over differences instead of flagging them. This is the hardest failure mode to catch. A model generating manifests for a mixed ROSA/ARO environment doesn't flag the differences between the platforms — it generates something generic that works on neither. The output looks correct because it's a plausible blend of both platforms. The only defense is platform-specific context injection and a practitioner who knows enough to recognize when the model is giving a plausible average instead of a specific answer.

Quick Recap

Design the control plane before the first prompt — context injection, gateway routing, and post-generation validation catch errors that trial-and-error model switching cannot
oc apply --dry-run=server is your minimum viable safety net — it collapses a multi-day debugging spiral into a five-minute feedback loop, at zero infrastructure cost
Knowledge gaps are a context problem, not a model problem — inject verified platform context into cheap models instead of paying for expensive models that still guess

What's Next

AI in the Stack #6 — Building n8n Workflows for Platform Engineering Automation

This article kept the LLM on a short leash — every call went through a gateway, every output through a validation gate before it touched the cluster. The next one hands an LLM agent a longer leash inside an n8n workflow, wired to an MCP server and a RAG pipeline for automated incident triage. The tradeoff: the moment an agent node can decide how many tool calls to make, the workflow stops being deterministic. That article covers the execution limits you need before you trust it to run unattended.

Browse the rest of the AI in the Stack series index for the full run.

Written by Pipeline & Prompts | Byte size guides on DevOps, Cloud and AI

DEV Community