Validate AI-Generated Kubernetes Manifests Before They Hit Git

#kubernetes #devops

Originally published on kuryzhev.cloud

We started letting an internal tool generate boilerplate Deployments and Services from a short intent spec, and within two weeks a teammate almost merged a manifest with policy/v1beta1 — an apiVersion removed since Kubernetes 1.25. That's the moment we realized validating AI-generated Kubernetes manifests isn't optional review theater, it's a hard gate that has to sit before the commit, not just before the deploy. Here's what we've locked into our pipeline since, tip by tip.

Pin the model version and temperature — stop manifest drift between runs

If you're generating YAML from an LLM and re-running the same prompt gives you a different resources: block twice in a row, that's not a fluke — it's temperature. Set temperature: 0 (or the provider's closest equivalent) for anything manifest-related; you want deterministic output, not creative writing.

Also pin the exact model string. gpt-4o as an alias will silently shift behavior when the provider updates "latest" mid-sprint — use gpt-4o-2024-08-06 instead. Log the model string plus a hash of the prompt alongside the generated commit so you can actually reproduce a bad manifest six weeks later when someone asks "why does this Deployment have three replicas."

Validate against your cluster's real API surface, not generic schemas

LLMs are trained on a snapshot of the internet, which means they happily emit deprecated kinds like extensions/v1beta1 or CRD versions that don't exist in your live cluster. Run kubeconform with -kubernetes-version 1.29.0 against your actual target version, not whatever ships as the default schema set.

For CRDs — ArgoCD's Application, Cilium policies, whatever you run — pull the live OpenAPI schema straight from the cluster instead of trusting a public catalog:

# export the cluster's real OpenAPI schema for CRD-aware validation
kubectl get --raw /openapi/v2 > schema.json

Watch out for this error: no matches for kind "Application" in version "argoproj.io/v1alpha1". Nine times out of ten that's not a real problem with the manifest — it's a stale CRD schema cache on your validation side, and it'll waste an hour if you assume the LLM is wrong.

Enforce a pre-commit hook, not just a CI check

We used to only validate in CI, which meant a rejected PR still left a "clean-looking" diff sitting in git history. Reviewers would open manifests that were never actually valid Kubernetes objects and waste time reasoning about them. Shift the check left — block the commit itself.

#!/usr/bin/env bash
# .git/hooks/pre-commit (or wired via .pre-commit-config.yaml local hook)
# Generates a manifest via LLM, validates it, blocks commit on failure.

set -euo pipefail

SPEC_FILE="deploy/intent.yaml"          # human-authored high-level spec
OUT_FILE="deploy/manifests/deployment.yaml"
CACHE_FILE=".ai-gitops-cache.json"
CLUSTER_VERSION="1.29.0"

# Skip LLM call if spec hasn't changed since last validated run
CURRENT_HASH=$(sha256sum "$SPEC_FILE" | awk '{print $1}')
CACHED_HASH=$(jq -r '.hash // ""' "$CACHE_FILE" 2>/dev/null || echo "")

if [[ "$CURRENT_HASH" == "$CACHED_HASH" ]]; then
  echo "[ai-gitops] No spec change detected, skipping regeneration."
  exit 0
fi

echo "[ai-gitops] Spec changed, calling LLM to (re)generate manifest..."
python3 scripts/generate_manifest.py \
  --spec "$SPEC_FILE" \
  --model "gpt-4o-2024-08-06" \
  --temperature 0 \
  --out "$OUT_FILE"

echo "[ai-gitops] Validating against schema for k8s ${CLUSTER_VERSION}..."
if ! kubeconform -strict -kubernetes-version "${CLUSTER_VERSION}" "$OUT_FILE"; then
  echo "[ai-gitops] Schema validation FAILED. Commit blocked."
  exit 1
fi

echo "[ai-gitops] Checking resource/security policy with Kyverno..."
if ! kyverno apply policies/resource-limits.yaml --resource "$OUT_FILE" | grep -q "pass"; then
  echo "[ai-gitops] Policy validation FAILED. Commit blocked."
  exit 1
fi

echo "[ai-gitops] Building through kustomize to catch merge/indentation issues..."
kustomize build deploy/overlays/staging | kubeconform -strict -kubernetes-version "${CLUSTER_VERSION}" -

# All good — update cache and stage the generated file
jq -n --arg h "$CURRENT_HASH" '{hash: $h}' > "$CACHE_FILE"
git add "$OUT_FILE" "$CACHE_FILE"

echo "[ai-gitops] Manifest generated and validated. Proceeding with commit."

If you use kustomize overlays, add a post-build validation step too — patches from an LLM can silently merge wrong due to indentation, and pre-build validation won't catch that.

Never trust the LLM's resource requests/limits — enforce with policy

Left unchecked, LLMs either omit the resources: block entirely or copy suspiciously round numbers — cpu: 1000m, memory: 1Gi — regardless of what the workload actually needs. We've seen teams over-provision nodes by 2-3x within a month just from trusting these guesses.

Run a policy engine as a second, independent pass. We landed on Kyverno 1.11+ for this because kyverno apply --resource gives you a fast local dry-run before anything reaches the cluster:

# policies/resource-limits.yaml — Kyverno policy that overrides LLM guesses
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-resource-limits
spec:
  validationFailureAction: Enforce
  rules:
    - name: check-container-resources
      match:
        any:
          - resources:
              kinds:
                - Deployment
                - StatefulSet
      validate:
        message: >-
          LLM-generated manifest is missing resource requests/limits,
          or exceeds max allowed values (cpu: 2, memory: 4Gi).
        pattern:
          spec:
            template:
              spec:
                containers:
                  - resources:
                      requests:
                        cpu: "?*"
                        memory: "?*"
                      limits:
                        cpu: "<=2"
                        memory: "<=4Gi"

# Example failure output when the LLM omits resources entirely:
# policy require-resource-limits/check-container-resources fail:
#   validation error: LLM-generated manifest is missing resource
#   requests/limits, or exceeds max allowed values (cpu: 2, memory: 4Gi)

OPA/Conftest works just as well if that's already in your stack — the point is that schema validation and cost/security policy are two separate concerns and neither one substitutes for the other.

Strip secrets and internal context from prompts before sending to hosted models

Treat every prompt as an egress channel. The most common accidental data leak we've seen is someone pasting a real Secret value, an internal hostname, or a VPC CIDR into a prompt like "generate a manifest based on our existing setup" — and that text now lives on a third-party API log.

Put a redaction preprocessor in front of anything going to a hosted model — a regex or a small Python filter that masks env values and internal domains before the request goes out. If your compliance policy is strict about production namespace names, route that generation through a self-hosted model like Ollama running CodeLlama instead. I stopped sending anything touching prod-adjacent context to hosted APIs after one near-miss with a staging hostname that mapped 1:1 to an internal DNS record — not worth the risk for a boilerplate Deployment.

Force schema-constrained output instead of free-form YAML

Free-form generation invites hallucinated fields — we've seen spec.replicaCount show up instead of the real spec.replicas, which parses fine as YAML and fails silently downstream. Use JSON mode or function-calling (OpenAI's response_format: json_schema) or a grammar-constrained approach via guidance/outlines so the model literally can't invent a field that doesn't exist in the schema.

This doesn't replace kubeconform, though. Schema-constrained output reduces structural hallucination but says nothing about semantic correctness — a field can be perfectly valid and still hold the wrong value. "It parsed as YAML" and "it's a valid Kubernetes object" are two separate validation layers, and conflating them is the mistake we see most often on teams new to this. Check the Kubernetes API concepts docs if you want to understand exactly why schema and semantics diverge.

Cache prompts and diffs to control LLM API cost on repeated pushes

Regenerating the full manifest tree on every single commit burns tokens for no reason. Hash the source intent file and only call the LLM when that hash actually changes — and when it does, send a diff-style prompt ("here's the current manifest, apply this change") instead of full regeneration. That alone cut our token usage roughly 40-60% on iterative PRs.

Use a cheaper model like gpt-4o-mini for the validation/fix-suggestion loop and reserve the larger model for initial generation only — there's no reason to pay premium-model prices to check whether a field name is right. We keep the last-validated manifest and its input hash in .ai-gitops-cache.json, which is exactly what lets the pre-commit hook above skip regeneration entirely when nothing meaningful has changed. If you're building out a broader GitOps pipeline around this pattern, we cover more of the plumbing over at kuryzhev.cloud's GitOps posts.

None of this is exotic tooling — kubeconform, Kyverno, and a pre-commit hook are things most teams already have lying around. The only real shift is treating AI-generated Kubernetes manifests as untrusted input by default, the same way you'd treat a PR from someone who's never seen your cluster before.