In February 2024, a major financial services firm evaluating an AIOps platform for incident response discovered that the vendor's root cause analysis feature was sending structured incident telemetry — including service names, error messages, environment identifiers, and deployment metadata — to an external LLM API for processing. The telemetry did not contain customer data. It did not contain financial transactions. But it contained the operational topology of the firm's production infrastructure: which services existed, how they were connected, and how they failed.
The firm's Chief Information Security Officer terminated the evaluation the same day. Not because the data was classified. But because the operational topology of a systemically important financial institution is precisely the intelligence that a sophisticated adversary needs to plan a targeted attack — and sending it to an external service was a risk that the firm's threat model did not permit, regardless of the vendor's contractual data handling commitments.
This is the data sovereignty problem in AI-assisted SRE operations. It is not hypothetical in regulated environments. It is the default constraint that determines whether external LLM services are even in scope for evaluation. In healthcare environments subject to HIPAA, production telemetry may contain PHI in log messages. In financial services environments subject to PCI-DSS, incident data may reference cardholder systems. In energy sector environments subject to NERC CIP, operational topology data may reveal critical infrastructure attack surfaces.
The on-premises LLM gateway architecture is the engineering response to this constraint. It preserves the operational value of AI-assisted incident response while maintaining data sovereignty over the telemetry that enables it.
Architecture Overview: The Gateway Pattern
The gateway pattern centralises all LLM access behind a single proxy layer that enforces data classification, applies model routing logic, maintains the audit trail, and provides the fallback chain that keeps AI-assisted operations available when any individual model backend is degraded.
The critical design decision is the placement of the data classification gate before the LiteLLM Proxy routing decision. Classification cannot be delegated to the proxy itself because the proxy does not have the context to understand whether a Kubernetes event log message contains PHI, PCI-relevant data, or NERC CIP-sensitive operational topology. The classification logic must run in the HolmesGPT context gathering layer, where the telemetry content is still structured and inspectable.
Data Classification Gate
The data classification gate is the mechanism that prevents regulated data from being routed to external model endpoints. It operates on the structured context package that HolmesGPT assembles before making an LLM request.
# HolmesGPT Data Classification Configuration
# Defines the rules that determine which model backend receives each request
apiVersion: v1
kind: ConfigMap
metadata:
name: holmesgpt-data-classification
namespace: holmesgpt
annotations:
sre.internal/policy-version: "v1.2"
sre.internal/approved-by: "ciso,sre-lead"
data:
classification_rules.yaml: |
# REGULATED: must route to on-premises Ollama only
regulated_indicators:
namespace_patterns:
- "pci-zone"
- "hipaa-zone"
- "nerc-cip-zone"
- "tier-0-clinical"
- "tier-1-clinical"
label_patterns:
- "compliance.internal/regulated=true"
- "data-classification=restricted"
log_content_patterns:
# PHI indicators in log messages
- "patient_id"
- "mrn"
- "ssn"
- "account_number"
# PCI indicators
- "card_number"
- "cvv"
- "pan"
# NERC CIP indicators
- "substation"
- "scada"
- "ems_topology"
incident_service_patterns:
- "payments-core"
- "cardholder-data"
- "clinical-decision"
# STANDARD: may route to external models
# (telemetry contains no regulated data)
standard_indicators:
default_when_no_regulated_match: true
external_model_allowed: true
routing:
regulated: "ollama-primary"
standard: "github-models-primary"
fallback: "ollama-primary"
# Data Classification Gate — Implementation (HolmesGPT pre-routing hook)
# Classifies assembled context before LiteLLM routing decision
import re
from typing import Literal
DataClass = Literal["REGULATED", "STANDARD"]
def classify_context(context: dict) -> DataClass:
"""
Classify assembled incident context before LLM routing.
Returns REGULATED if any indicator matches; STANDARD otherwise.
"""
rules = load_classification_rules()
# Check namespace of affected services
for service in context.get("affected_services", []):
namespace = service.get("namespace", "")
for pattern in rules["regulated_indicators"]["namespace_patterns"]:
if pattern in namespace:
return "REGULATED"
# Check labels on affected resources
for resource in context.get("kubernetes_resources", []):
labels = resource.get("labels", {})
for label_pattern in rules["regulated_indicators"]["label_patterns"]:
key, value = label_pattern.split("=")
if labels.get(key) == value:
return "REGULATED"
# Check log content for regulated data indicators
log_content = " ".join(context.get("log_snippets", []))
for pattern in rules["regulated_indicators"]["log_content_patterns"]:
if re.search(pattern, log_content, re.IGNORECASE):
return "REGULATED"
return "STANDARD"
LiteLLM Proxy Configuration
The LiteLLM Proxy is the unified gateway layer. It receives requests from HolmesGPT with a data_classification header set by the classification gate, applies the routing logic, manages fallback chains, enforces rate limits, and writes to the audit log.
# LiteLLM Proxy — Full Production Configuration
# Deployed on Shared Services TKGs cluster (ai-ops namespace)
# All AI workloads route through this gateway
model_list:
# ── ON-PREMISES MODELS (Ollama) ────────────────────────────────────────
# Primary for REGULATED data; fallback for all workloads
- model_name: ollama-primary
litellm_params:
model: ollama/qwen2.5:14b
api_base: http://ollama.ai-ops.svc.cluster.local:11434
timeout: 120
max_tokens: 8192
# qwen2.5:14b selection rationale:
# - 14B parameters: sufficient for multi-service incident correlation
# - Strong multilingual capability for diverse log formats
# - Quantised to Q4_K_M: fits in 10GB VRAM on GPU node
# - Benchmark: 78% accuracy on SRE incident diagnosis test set
- model_name: ollama-large
litellm_params:
model: ollama/qwen2.5:72b
api_base: http://ollama-large.ai-ops.svc.cluster.local:11434
timeout: 300
max_tokens: 16384
# 72B for complex multi-cluster incidents requiring larger context
# GPU requirement: 48GB VRAM (A100 or equivalent)
# ── EXTERNAL MODELS (GitHub Models) ─────────────────────────────────
# Primary for STANDARD data; higher capability for complex cases
- model_name: github-models-primary
litellm_params:
model: openai/gpt-4.1
api_base: https://models.inference.ai.azure.com
api_key: os.environ/GITHUB_MODELS_PAT
timeout: 60
max_tokens: 16384
# gpt-4.1 for production RCA: highest accuracy on complex incidents
- model_name: github-models-mini
litellm_params:
model: openai/gpt-4.1-mini
api_base: https://models.inference.ai.azure.com
api_key: os.environ/GITHUB_MODELS_PAT
timeout: 30
max_tokens: 4096
# gpt-4.1-mini for high-volume alert triage: lower latency, lower cost
router_settings:
routing_strategy: custom
model_group_alias:
"regulated-routing":
- ollama-primary
- ollama-large
"standard-routing":
- github-models-primary
- ollama-primary # Fallback if GitHub Models unavailable
"triage-routing":
- github-models-mini
- ollama-primary
# Custom routing function based on data_classification header
routing_logic: |
classification = request.headers.get("X-Data-Classification", "STANDARD")
urgency = request.headers.get("X-Incident-Urgency", "standard")
if classification == "REGULATED":
return "ollama-primary" # Never leaves network boundary
if urgency == "triage":
return "github-models-mini" # Fast triage; lower capability
return "github-models-primary" # Full capability for RCA
fallback_model: "ollama-primary" # Always falls back to on-premises
fallback_on_status_codes: [429, 500, 502, 503, 504]
num_retries: 2
request_timeout: 180
# ── RATE LIMITING ──────────────────────────────────────────────────────
litellm_settings:
max_budget: 500.0 # Monthly GitHub Models budget cap (USD)
budget_duration: "1mo"
max_parallel_requests: 10
# Per-model rate limits
model_max_parallel_requests:
ollama-primary: 5 # Constrained by Ollama throughput
github-models-primary: 10
github-models-mini: 20
# ── AUDIT LOGGING ──────────────────────────────────────────────────────
success_callback: ["splunk"]
failure_callback: ["splunk"]
splunk_host: "https://splunk.internal:8088"
splunk_token: os.environ/SPLUNK_HEC_TOKEN
splunk_index: "aiops_gateway"
# Logged fields per request:
# request_id, timestamp, model_used, routing_decision,
# data_classification, incident_id, tokens_used, latency_ms,
# success/failure, fallback_triggered
Multi-Cluster Deployment via Argo CD ApplicationSet
HolmesGPT is deployed per production cluster — each cluster gets its own HolmesGPT instance that analyses incidents in that cluster's context. All instances route through the centralised LiteLLM Proxy on the shared services cluster.
# Argo CD ApplicationSet — HolmesGPT per TKGs Cluster
# Deploys HolmesGPT to every cluster registered in Argo CD
# Each instance connects to the shared LiteLLM Proxy
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: holmesgpt-per-cluster
namespace: argocd
spec:
generators:
- clusters:
selector:
matchLabels:
sre.internal/holmesgpt-enabled: "true"
template:
metadata:
name: "holmesgpt-{{name}}"
annotations:
notifications.argoproj.io/subscribe.on-sync-failed.slack: "sre-aiops-alerts"
spec:
project: sre-platform
source:
repoURL: https://git.internal/sre/holmesgpt-config
targetRevision: main
path: clusters/holmesgpt
helm:
values: |
cluster:
name: "{{name}}"
environment: "{{metadata.labels.environment}}"
litellmProxy:
url: "http://litellm-proxy.ai-ops.sre-shared.svc.cluster.local:4000"
# Cross-cluster routing: HolmesGPT on prod cluster routes to
# LiteLLM Proxy on shared services cluster via Istio ServiceEntry
holmesgpt:
escalationPolicyConfigMap: "holmesgpt-escalation-policy"
dataClassificationConfigMap: "holmesgpt-data-classification"
auditSplunkIndex: "holmesgpt_{{name}}"
actionAuthority: "none" # Override per-cluster based on trust level
destination:
server: "{{server}}"
namespace: holmesgpt
syncPolicy:
automated:
prune: true
selfHeal: true
# Istio ServiceEntry — Cross-Cluster LiteLLM Proxy Access
# Allows HolmesGPT on production clusters to reach LiteLLM Proxy
# on shared services cluster via mTLS-secured cross-cluster communication
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
name: litellm-proxy-shared-services
namespace: holmesgpt
spec:
hosts:
- litellm-proxy.ai-ops.sre-shared.svc.cluster.local
ports:
- number: 4000
name: http
protocol: HTTP
location: MESH_EXTERNAL
resolution: DNS
endpoints:
- address: litellm-proxy.shared-services.internal
ports:
http: 4000
Kyverno Governance for the AI Ops Layer
The AI gateway is a production service with production governance requirements. Kyverno policies enforce the operational standards that prevent AI-ops components from becoming a governance gap in the platform.
# Kyverno: AI-Ops Governance Policies
# Policy 1: All AI-ops deployments must have data classification annotation
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: ai-ops-data-classification-required
spec:
validationFailureAction: Enforce
rules:
- name: require-data-classification-annotation
match:
any:
- resources:
kinds: [Deployment]
namespaces: [holmesgpt, ai-ops, litellm]
validate:
message: >
AI-ops deployments must declare data classification policy.
Add annotation: sre.internal/data-classification-policy=<configmap-name>
pattern:
metadata:
annotations:
sre.internal/data-classification-policy: "?*"
---
# Policy 2: LiteLLM Proxy must not allow external routing without
# an active data classification gate (enforces gateway pattern)
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: litellm-external-routing-gate-required
spec:
validationFailureAction: Enforce
rules:
- name: require-classification-gate-for-external-models
match:
any:
- resources:
kinds: [ConfigMap]
names: ["litellm-proxy-config"]
namespaces: [ai-ops]
validate:
message: >
LiteLLM Proxy config must declare data classification gate.
External model routing without classification gate is prohibited.
pattern:
data:
config.yaml: "*X-Data-Classification*"
---
# Policy 3: Ollama models must be on the approved model list
# Prevents unauthorised model deployment that bypasses governance
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: ollama-approved-models-only
spec:
validationFailureAction: Enforce
rules:
- name: check-ollama-model-tag
match:
any:
- resources:
kinds: [Pod]
namespaces: [ai-ops]
selector:
matchLabels:
app: ollama
validate:
message: >
Only approved Ollama models permitted. Approved:
qwen2.5:14b, qwen2.5:72b, llama3.1:8b, llama3.1:70b
Submit model approval request to sre-aiops-governance.
deny:
conditions:
all:
- key: "{{ request.object.spec.containers[].env[?name=='OLLAMA_MODEL'].value | [0] }}"
operator: AnyNotIn
value:
- "qwen2.5:14b"
- "qwen2.5:72b"
- "llama3.1:8b"
- "llama3.1:70b"
SLO Design for the AI Gateway
The LiteLLM Proxy is a production dependency of the incident response workflow. It requires the same SLO treatment as any other production dependency: defined SLIs, error budgets, and degradation posture.
# Prometheus: LiteLLM Proxy SLIs
groups:
- name: litellm.slo
rules:
# SLI 1: Request availability
# Fraction of LLM requests returning a response within timeout
- record: sli:litellm_availability:ratio_rate5m
expr: |
sum(rate(litellm_request_total{status="success"}[5m]))
/
sum(rate(litellm_request_total[5m]))
# SLI 2: On-premises fallback reliability
# When external models fail, Ollama must respond within 120s
- record: sli:ollama_fallback_success:ratio_rate1h
expr: |
sum(rate(litellm_request_total{
model=~"ollama.*",
status="success",
triggered_by_fallback="true"
}[1h]))
/
sum(rate(litellm_request_total{
triggered_by_fallback="true"
}[1h]))
# SLI 3: Routing compliance
# Fraction of REGULATED requests correctly routed to Ollama (not external)
- record: sli:regulated_routing_compliance:ratio_rate1h
expr: |
sum(rate(litellm_request_total{
data_classification="REGULATED",
model=~"ollama.*"
}[1h]))
/
sum(rate(litellm_request_total{
data_classification="REGULATED"
}[1h]))
# Alert: Routing compliance breach — REGULATED data sent to external model
- alert: LiteLLM_RegulatedDataRoutingBreach
expr: sli:regulated_routing_compliance:ratio_rate1h < 1.0
for: 0s # Immediate — no tolerance for compliance breach
labels:
severity: critical
compliance: "true"
regulatory_notification: "assess"
annotations:
summary: >
CRITICAL: REGULATED data routed to external model.
Data classification gate may have failed.
Immediate investigation required.
runbook: "https://wiki.internal/sre/runbooks/regulated-data-routing-breach"
# SLO error budget: 99.5% availability for AI gateway
# (lower than application services — AI is enhancing, not critical path)
- record: slo:litellm_budget_remaining:ratio
expr: |
1 - (
(1 - sli:litellm_availability:ratio_rate5m)
/ (1 - 0.995)
)
Operational Considerations for Ollama in Production
Running local LLMs in production Kubernetes introduces infrastructure requirements that typical enterprise Kubernetes platforms are not provisioned for.
────────────────────────────────────────────────────────────────────────────
OLLAMA PRODUCTION INFRASTRUCTURE REQUIREMENTS
GPU NODE REQUIREMENTS:
qwen2.5:14b (Q4_K_M quantisation):
VRAM required: ~10 GB
Suitable GPUs: NVIDIA A10G, RTX 3090, T4 (marginal)
Inference throughput: ~30–50 tokens/second
Concurrent requests: 1–2 (GPU memory limited)
qwen2.5:72b (Q4_K_M quantisation):
VRAM required: ~45 GB
Suitable GPUs: NVIDIA A100 80GB (single), 2× A100 40GB (NVLink)
Inference throughput: ~15–25 tokens/second
Concurrent requests: 1 (memory constrained)
KUBERNETES NODE CONFIGURATION:
Node label: sre.internal/gpu-class=ollama
Taint: nvidia.com/gpu=present:NoSchedule
GPU time-slicing: NOT recommended for LLM inference
(VRAM is not sliceable; time-slicing adds context switch overhead)
OLLAMA KUBERNETES DEPLOYMENT:
apiVersion: apps/v1
kind: Deployment
metadata:
name: ollama
namespace: ai-ops
spec:
replicas: 1 # Single replica; GPU memory constraint
template:
spec:
nodeSelector:
sre.internal/gpu-class: "ollama"
tolerations:
- key: "nvidia.com/gpu"
operator: "Exists"
effect: "NoSchedule"
containers:
- name: ollama
image: ollama/ollama:latest
resources:
limits:
nvidia.com/gpu: "1"
memory: "32Gi"
requests:
nvidia.com/gpu: "1"
memory: "16Gi"
volumeMounts:
- name: ollama-models
mountPath: /root/.ollama
env:
- name: OLLAMA_MODELS
value: "/root/.ollama/models"
- name: OLLAMA_NUM_PARALLEL
value: "2" # Allow 2 concurrent requests (small models only)
- name: OLLAMA_MAX_LOADED_MODELS
value: "1" # Keep only one model in VRAM at a time
volumes:
- name: ollama-models
persistentVolumeClaim:
claimName: ollama-models-pvc # Models pre-loaded; not pulled at runtime
MODEL PRE-LOADING STRATEGY:
Models must be pulled and cached before production deployment.
Pulling at runtime introduces unacceptable cold-start latency.
Solution: InitContainer or pre-population Job that pulls models
into the PVC before Ollama deployment proceeds.
────────────────────────────────────────────────────────────────────────────
Common Antipatterns
The Classification-After-Routing antipattern → Applying data classification logic inside the LiteLLM Proxy routing function rather than in the HolmesGPT context layer. The proxy cannot inspect log content for PHI indicators — it sees the assembled request, not the source telemetry. Classification must happen before the request is constructed, not after.
The Fallback-to-External antipattern → Configuring external models as the fallback when Ollama is unavailable. The fallback chain must maintain data sovereignty: if the primary on-premises model fails, the fallback must also be on-premises. An external fallback for a REGULATED request is a compliance failure even if the primary model was correctly on-premises.
The Model-at-Runtime-Pull antipattern → Pulling Ollama models at container startup rather than pre-loading them into a persistent volume. Model pull times for 14B models are 5–15 minutes depending on network speed. An Ollama pod restart during an active incident creates a 15-minute AI assistance outage. Pre-load models into a PVC; treat model updates as planned changes with a change record.
The Single-Cluster Gateway antipattern → Deploying LiteLLM Proxy on a production cluster rather than on a dedicated shared services cluster. A LiteLLM Proxy running on the same cluster whose incidents it is analysing creates an operational dependency loop: when the production cluster is degraded (the scenario where AI assistance is most valuable), the AI gateway may also be degraded. Shared services cluster isolation is mandatory.
The Ungoverned Model antipattern → Allowing any Ollama model to be pulled and run without an approval process. Models have different capability levels, different licensing terms (some open-weights models have commercial use restrictions), and different privacy and security properties. The Kyverno approved-models policy enforces the approved list; the policy review process is the human governance layer that maintains it.
Maturity Progression
────────────────────────────────────────────────────────────────────────────
STAGE AI GATEWAY MATURITY NORTH STAR SIGNAL
────────────────────────────────────────────────────────────────────────────
Reactive No AI-assisted operations. All incident investigation
External LLM APIs used is manual. Or: external
without data classification LLM used without
gate (compliance risk). compliance review.
Defined Gateway architecture LiteLLM Proxy deployed.
documented. Data Data classification gate
classification rules implemented. Ollama
defined. Model list running on GPU node.
approved.
Measured All four SLIs instrumented. Routing compliance SLI
Multi-cluster ApplicationSet at 100%. Fallback
deploying HolmesGPT. reliability measured.
Kyverno policies active. No REGULATED data
reaching external models.
Optimised Shadow mode evaluation PoC gates passed.
complete (see AIOps HolmesGPT operating
Trustworthy post). at Level 1–2 autonomy.
Model routing calibrated. AI assistance reducing
GPU utilisation optimised. investigation time.
Generative On-premises model fine-tuned Investigation accuracy
on internal incident corpus. above 70% precision.
Multi-model routing by Regulated workloads
incident type operational. fully served by
AI gateway SLO stable. on-premises models.
────────────────────────────────────────────────────────────────────────────
Five Action Items for This Week
Audit every LLM API call currently made from your production environment and classify the data being sent. For each call: what telemetry is included in the prompt? Does that telemetry touch a regulated namespace, contain potential PHI indicators, or reference operational topology that your threat model considers sensitive? The audit output is your data classification rule set.
Deploy LiteLLM Proxy on your shared services cluster with Ollama as the only backend. Do not configure external models yet. Establish the on-premises routing baseline first — verify that Ollama is reachable, the SLI recording rules are working, and the audit log is flowing to Splunk. Add external model routing only after on-premises baseline is stable.
Pre-load your chosen Ollama model into a PVC before any production deployment. Pull the model in a test environment, measure the pull time, and design the pre-loading Job that populates the PVC before the Ollama deployment runs. Document the model version as a deployment dependency — the same way you version application images.
Implement the Kyverno regulated-routing compliance policy and run it in audit mode for two weeks before enforcing. Audit mode surfaces all requests that would be blocked by the policy without actually blocking them. Two weeks of audit mode data reveals whether the classification gate is correctly implemented before enforcement creates operational disruption.
Define the fallback chain explicitly: on-premises primary → on-premises secondary → human escalation. There is no "external model as fallback for regulated data" option. If both on-premises models are unavailable, the fallback is human incident investigation without AI assistance — which is the pre-AI baseline, not a failure. Document this explicitly in the escalation policy so that operators know what to expect when the AI gateway is degraded.
"The question of whether to use an external LLM for incident response in a regulated environment is not a question about model capability — it is a question about data sovereignty. The model that produces the best diagnosis is not useful if sending your operational telemetry to it violates your threat model or your regulatory obligations. The on-premises gateway is not a compromise on AI capability; it is the architecture that makes AI capability permissible in the environments that need it most."
Top comments (0)