A pod gets evicted at 3am because a node ran out of memory. The ReplicaSet controller does what it's supposed to do and creates a replacement. The replacement never gets admitted, because six weeks earlier someone applied a disallow-latest-tag policy and that Deployment still references :latest. The workload had been running fine the entire time. Now it's gone, and it isn't coming back on its own.
That's the shape of the problem. A security policy that was working perfectly, right up until the cluster tried to heal itself.
What people expect a latest-tag policy to do
The mental model most people have is straightforward: apply the policy, and anything using :latest stops working. You'd see breakage immediately, fix the offending manifests, and move on. Fail loudly, fail fast, done in an afternoon.
Kyverno's actual policy is more surgical than that. The canonical version from the policy library has two rules:
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: disallow-latest-tag
spec:
rules:
- name: require-image-tag
match:
any:
- resources:
kinds: [Pod]
validate:
failureAction: Enforce # spec.validationFailureAction on older versions
message: "An image tag is required."
pattern:
spec:
containers:
- image: "*:*"
- name: validate-image-tag
match:
any:
- resources:
kinds: [Pod]
validate:
failureAction: Enforce
message: "Using a mutable image tag e.g. 'latest' is not allowed."
pattern:
spec:
containers:
- image: "!*:latest"
Note that the field moved. spec.validationFailureAction was the home for Enforce/Audit for years; Kyverno 1.13 introduced per-rule validate.failureAction and deprecated the spec-level field. If you copy a policy from a blog post written against 1.9 and apply it to a newer install, check which one your version actually honors before you assume enforcement is on.
Where the model breaks
Admission control is an event, not a state. Kyverno's webhook fires on CREATE and UPDATE of matching resources. It does not walk your cluster and terminate things that were already there.
So when you apply that policy on a Tuesday afternoon, nothing happens. Every existing :latest pod keeps running. Kyverno's background scanner will generate PolicyReports flagging them, but reports don't stop workloads. You get a clean kubectl apply, no alerts, and the strong impression that your cluster is now compliant.
It isn't. You've created a cluster with two populations: workloads that satisfy the policy, and workloads that only survive as long as their pod object is never recreated.
The second population is a landmine field with no map.
Container restarts don't help you here
This is the detail that makes the failure so confusing when you hit it. There are two very different things people call "a restart," and only one of them goes through admission.
When a container crashes and the kubelet restarts it in place, the Pod object never changes. No CREATE, no UPDATE, no webhook call. A pod in CrashLoopBackOff with :latest will loop forever under an Enforce policy without ever tripping it. That's why the violation stays hidden for so long.
Anything that produces a new Pod object goes through the webhook:
- Node drain or
kubectl delete pod - Eviction from memory pressure or a disruption budget
kubectl rollout restart- Node reboot with a
Recreate-style workload - A StatefulSet pod being rescheduled after a volume detach
All of those hand the ReplicaSet or StatefulSet controller the job of creating a fresh Pod. The webhook evaluates it fresh, sees :latest, and denies it.
The observability gap
Here's what makes this expensive to debug: the error is nowhere near where you're looking.
kubectl get deploy shows 0/1 ready. kubectl describe deploy shows the ReplicaSet scaled up and nothing else interesting. There are no pods, so kubectl logs and kubectl describe pod have nothing to say. It looks like a scheduling problem.
The actual message is on the ReplicaSet:
kubectl describe rs -l app=my-app
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedCreate 2m (x8 over 14m) replicaset-controller Error creating: admission webhook
"validate.kyverno.svc-fail" denied the request: policy Pod/media/my-app-7d9f4c8b6-
for resource violation: disallow-latest-tag: validate-image-tag: 'Using a mutable
image tag e.g. "latest" is not allowed.'
The ReplicaSet controller backs off exponentially on repeated create failures, so as time passes the retries get further apart and the events get staler. If you come to it an hour in, the last FailedCreate might be 15 minutes old and easy to dismiss as historical noise.
I wrote a whole post on why Kyverno is worth running, and I still think that. But a policy that blocks a pod while presenting the operator with an empty kubectl get pods is lying by omission, and that costs real minutes during an incident.
The compounding version: fixing one problem triggers another
The nastiest version of this isn't the policy on its own. It's the policy sitting downstream of an unrelated infrastructure fault.
Say a Longhorn volume goes read-only after a replica rebuild hiccup. The container is running, the pod is Ready, and the readiness probe passes because it only checks that the HTTP port answers. The application is writing errors into a filesystem it can't write to. I covered that class of problem in the gap between "Healthy" and actually working.
The standard remediation is to delete the pod so it remounts cleanly. So you delete it. And now you have two failures stacked on top of each other: the storage issue is resolved, and the workload is permanently down because the replacement pod can't be admitted. You've converted a recoverable glitch into an outage, using the recovery procedure.
GitOps makes this worse rather than better. ArgoCD reconciles the Deployment, the Deployment object applies cleanly (Kyverno's autogen rules validate the pod template, but if the Deployment already existed with :latest and you're not touching the image field, nothing changes about the violation state). Sync status reports fine. The health check may report Progressing or Degraded, but the reason lives three objects away. If you run App-of-Apps across a lot of applications, one Degraded app in a wall of green is easy to lose.
allowExistingViolations doesn't save you
Kyverno 1.13 added allowExistingViolations to validate rules, defaulting to true. Reasonable people read the name and assume it grandfathers in their pre-existing :latest workloads.
It doesn't do that. The field governs updates to resources that already exist and already violate. A Pod created by a ReplicaSet controller is a brand-new object performing a CREATE. There's no prior version for Kyverno to compare against, so there's nothing to grandfather. The rule evaluates and denies.
It's a useful field for letting people patch an annotation on a non-compliant Deployment without being blocked. It is not a safety net for pod recreation.
The fix
Three layers, in the order you'd actually do them.
Unblock right now. Flip the rule to Audit, or scope it away from the affected namespace. Validate rules are mutable, so a patch works:
kubectl patch clusterpolicy disallow-latest-tag --type=json \
-p='[{"op":"replace","path":"/spec/rules/1/validate/failureAction","value":"Audit"}]'
One caveat if your policy bundle includes generate rules: those are immutable in Kyverno. Editing a generate rule's target or data requires deleting and recreating the ClusterPolicy, which will briefly remove enforcement. Plan that for a moment when you aren't already mid-incident.
Kyverno also supports PolicyException as the proper escape hatch, but it has to be enabled at install time (--enablePolicyException=true) and confined to a namespace you control. If it isn't already on, an incident is a bad time to discover that.
Fix the manifest properly. Pin the digest. This is the part people skip because it feels like a workaround, and it's actually the correct answer:
spec:
containers:
- name: app
# tag for humans, digest for the runtime
image: ghcr.io/example/app:v1.8.2@sha256:9f2a1c4e7b03d5a68e1f4c92b7d0a3e5f81c6d4b2a9e07f3c5d1b8a6e4f2c0d9
imagePullPolicy: IfNotPresent
When both are present, the container runtime resolves by digest and ignores the tag. You get a human-readable version in kubectl get pod -o wide, byte-for-byte reproducibility, and a string that doesn't match *:latest.
The digest-only form (ghcr.io/example/app@sha256:...) also passes both rules, though it's worth understanding why: the require-image-tag rule matches *:*, and sha256:9f2a... happens to contain a colon. The policy is doing string pattern matching, not image reference parsing. That should give you a healthy skepticism about how airtight these policies are in general.
Stop it from reaching the cluster. Image tag validation belongs in CI, not in a webhook that fires during a node drain. Kyverno ships kyverno apply for exactly this:
# fail the PR, not the 3am pod recreation
kyverno apply ./policies/ --resource ./manifests/ --detailed-results
Wire that into the same job that runs your schema validation. I went through the setup for that in Kubernetes manifest validation in CI. The admission webhook then becomes a backstop against things that bypassed the pipeline, which is what it should have been all along.
Rolling out policy without setting a trap
The general rule: an Enforce policy is only safe once you've proven the existing fleet complies. Audit mode plus the background scanner gives you that proof.
# every currently non-compliant resource in the cluster
kubectl get policyreport -A -o json \
| jq -r '.items[].results[]
| select(.result=="fail" and .policy=="disallow-latest-tag")
| "\(.resources[0].namespace)/\(.resources[0].name)"' \
| sort -u
Drive that list to zero, then switch to Enforce. Not the other way around.
Two more things worth setting before you turn enforcement on:
-
Check your webhook failure policy. Kyverno's validating webhooks default to
failurePolicy: Fail. If Kyverno itself is unavailable during a control-plane restart, all matching pod creation stops cluster-wide. That's a much bigger version of the same self-inflicted outage. For policies that aren't strictly security-critical,Ignoreis the safer tradeoff. -
Exclude the namespaces that have to come back first.
kube-system, your CNI, your storage system, your ingress controller. If those can't recreate pods during a node failure, the policy has stopped being a security control and started being a single point of failure.
The general lesson
Admission control validates transitions, not states. That distinction sounds academic until a policy you wrote in April silently marks a dozen workloads as "runs fine, never restarts," and then a node goes down.
The tell is that Kubernetes stops self-healing without telling you why in the obvious place. Whenever a Deployment sits at zero pods and there's nothing to describe, walk down to the ReplicaSet before you go anywhere else. kubectl get events -A --field-selector reason=FailedCreate will find it across the whole cluster in one shot, and it should probably be a Prometheus alert rather than something you remember to type.
Policy that makes your cluster less able to recover isn't security, it's a reliability liability wearing a security badge. Getting that boundary right (what belongs in CI, what belongs in a webhook, and what should never block a pod creation) is most of the work. If you're sorting out where those lines go in your own infrastructure, that's the kind of thing I help teams with.
Top comments (0)