DEV Community

Cover image for Why valid Kubernetes YAML still breaks in prod, and how to catch it before deploy
Andrii Zhukovskyi
Andrii Zhukovskyi

Posted on Originally published at zhukovskyi.space

Why valid Kubernetes YAML still breaks in prod, and how to catch it before deploy

In short. Linting a manifest checks the file, not the cluster it's going to. To catch deploy-time failures you also need the cluster's context (quotas, node sizes, LimitRanges, the Secrets and CRDs it references) and a diff against what is running now. k8s-guardian does both, fixes only what is safe to fix, and exports the same rules as admission policies so CI and the cluster agree. Running it on 30 popular Helm charts found five bugs in the tool itself, all fixed.

At work I defined deployment and configuration standards across more than ten environments. Standards help, but nothing enforced them, so every few weeks something drifted: a Helm value, a resource limit, a forgotten feature flag. Worse were the changes that passed every check and then failed at deploy time. I wrote the first version of this post in July as a plan. This is what the plan turned into: k8s-guardian, now at v0.7.

Why does a valid manifest fail at deploy time?

Because "valid" is a property of the file, and most failures are a property of the cluster. The questions I kept answering by hand were these:

Question at deploy time What actually goes wrong
Will it fit in prod? ResourceQuota has no headroom, no node is big enough, a LimitRange rejects the pod
Does everything it needs exist? A referenced ConfigMap, Secret or CRD is missing in that namespace
What breaks if I apply it over what's running? An immutable selector changed, an API was removed, a port changed under a Service
Will the cluster enforce what CI checks? CI and admission policies drift apart and disagree

What k8s-guardian checks

  • 21 built-in rules: requests and limits, non-root, privilege escalation, capabilities, probes, :latest tags, host namespaces and ports, seccomp, literal secrets in env, Service wiring, PodDisruptionBudgets, removed APIs and RBAC wildcards. Custom rules are a small YAML file, no Rego.
  • Helm, Kustomize and Argo Rollouts, rendered with your own values file.
  • Live context with --live: quotas, node capacity, LimitRanges, missing references and CRDs.
  • A breaking-change diff against what is deployed now.
  • A baseline for an existing repo, so only new findings fail the build.
k8s-guardian check -f k8s/                   # files, charts, Kustomize
k8s-guardian check -f k8s/ --fix             # apply the safe fixes
k8s-guardian check -f k8s/ --live -n prod    # will it fit the cluster?
k8s-guardian diff -f k8s/ -n prod            # what breaks if I apply it?
k8s-guardian export vap > guardrails.yaml    # same rules at admission
Enter fullscreen mode Exit fullscreen mode

What running it on 30 Helm charts taught me

Hand-written examples flatter a tool. So I ran k8s-guardian against 30 widely used Helm charts, among them ingress-nginx, kube-prometheus-stack, Argo CD, cert-manager, Vault and Longhorn: 652 Kubernetes objects in total. It found five bugs in k8s-guardian itself.

Bug Where it showed up
A numeric targetPort flagged when the container declares no ports Harbor, Vault
Services flagged when an operator, not the chart, creates the pods kube-prometheus-stack, Longhorn
Probes required on run-once pods such as Helm test hooks 28 findings in 8 charts
--fix rewrote every document; in Loki it changed a ConfigMap's content All charts, up to 85,000 lines in Kyverno
--fix produced manifests that don't run Longhorn, Falco, node-exporter

All five are fixed, each with a regression test, and a weekly workflow repeats the run on pinned chart versions so they can't come back.

Which fixes are safe to apply automatically?

Only the ones that don't change how a workload runs. The last bug in the table was the important one: an auto-fix set privileged: false on Longhorn's manager, which the API server then rejects, and removed hostNetwork from node-exporter, which breaks it. Now --fix applies only safe fixes and writes unchanged documents back byte for byte, comments included. Anything that can change behaviour, such as forcing non-root, a read-only filesystem or resource values, needs --fix --unsafe-fixes and a review. Privileged containers and host namespaces are reported but never changed.

How to make CI and the cluster enforce the same rules

Export them. export vap writes the rules as ValidatingAdmissionPolicies, and export kyverno as Kyverno policies, tested to make the same decisions as the CLI. The same binary also runs as a GitHub Action with PR annotations and SARIF, a pre-commit hook, a kubectl plugin and an MCP server for AI coding assistants.

Source and docs: github.com/andronaft/k8s-guardian (Apache 2.0) ยท the full chart run: real-world test

Which deploy-time failures do you hit that a check like this would miss? I'd like to add them.

Originally published at zhukovskyi.space.

Top comments (0)