DEV Community

jai sandesh
jai sandesh

Posted on

Kyverno Policy Conflicts: Why “Last-Writer-Wins” Is a Production Bug

#kubernetes #policyascode #kyverno #devops

▶️Introduction

Kyverno has become the go-to policy engine for Kubernetes because it feels simple:

  • Policies are YAML,
  • They run at admission time,
  • No custom controllers required.

That simplicity is also why teams run into policy conflicts in production.

Most Kyverno incidents are not bugs in Kyverno.
They are caused by incorrect assumptions about how policies are executed.

🤔The Assumption Almost Everyone Makes

Most teams implicitly assume at least one of the following:

  • Policies run in the order they are applied
  • Cluster-scoped policies are evaluated before namespace-scoped ones
  • Kyverno resolves conflicts deterministically

⚠️ None of these assumptions are true.

Kyverno provides:

  • no policy priority,
  • no guaranteed ordering across policies,
  • no conflict detection between mutations.

🧠How Kyverno Actually Processes an Admission Request

For a single admission request, Kyverno evaluates policies in phases:

  • All matching mutate rules are executed
  • All matching validate rules are evaluated
  • All matching generate rules are processed

This order is guaranteed.

What is not guaranteed:

  • The execution order between two different mutate policies
  • The execution order between mutate rules across different policies

Inside a single policy, rules are processed top-to-bottom.Across multiple policies, order is explicitly undefined.

📌Concrete Failure Modes

🧪Example1: Mutate vs Mutate: Where Things Quietly Break

⚠️The examples are written on kyverno version 1.17 and policies are created at the same time

Consider two mutate policies that both match the same Pod.
Policy A

apiVersion: policies.kyverno.io/v1
kind: MutatingPolicy
metadata:
  name: enforce-non-root
spec:
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["pods"]
  mutations:
    - patchType: ApplyConfiguration
      applyConfiguration:
        expression: >
          Object{
            spec: Object.spec{
              securityContext: Object.spec.securityContext{
                runAsNonRoot: true
              }
            }
          }
Enter fullscreen mode Exit fullscreen mode

Policy B

apiVersion: policies.kyverno.io/v1
kind: MutatingPolicy
metadata:
  name: force-run-as-user-0
spec:
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["pods"]
  mutations:
    - patchType: ApplyConfiguration
      applyConfiguration:
        expression: >
          Object{
            spec: Object.spec{
              securityContext: Object.spec.securityContext{
                runAsUser: 0
              }
            }
          }
Enter fullscreen mode Exit fullscreen mode

What Kyverno Sees

  • Both policies match the same Pod
  • Both apply mutations
  • Kyverno applies both patches
  • Execution order across policies is not guaranteed

🧪Example 2: Mutate vs Validate — Self-Inflicted Admission Failure

Scenario
One policy mutates a resource to enforce a default.Another policy validates the same field assuming a different expected state.
Both policies are correct in isolation.

MutatingPolicy — Add default label

apiVersion: policies.kyverno.io/v1
kind: MutatingPolicy
metadata:
  name: add-default-app-label
spec:
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["pods"]
  mutations:
    - patchType: ApplyConfiguration
      applyConfiguration:
        expression: >
          Object{
            metadata: Object.metadata{
              labels: Object.metadata.labels{
                app: "default"
              }
            }
          }
Enter fullscreen mode Exit fullscreen mode

ValidatingPolicy — Require a specific label value

apiVersion: policies.kyverno.io/v1
kind: ValidatingPolicy
metadata:
  name: require-app-frontend
spec:
  validationActions:
    - Deny
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["pods"]
  validations:
    - message: "Label app=frontend is required"
      expression: "object.metadata.?labels.app.orValue('') == 'frontend'"
Enter fullscreen mode Exit fullscreen mode

What Actually Happens

  • Mutation runs first and sets app=default
  • Validation runs after mutation
  • Validation fails because app=frontend is required

❗ Admission denied.
❗ Even if the Pod originally satisfied the validation rule.

📉Why This Gets Worse at Scale

With just one or two policies, these interactions are easy to reason about.

At scale:

  • multiple teams mutate the same fields,
  • validation rules encode different assumptions,
  • scopes overlap silently.

Failures:

  • don’t always happen immediately,
  • often appear only for specific workloads,
  • are hard to trace back to policy interaction, not policy logic.

Kyverno evaluates each policy independently.
It does not infer intent or resolve conflicts.

💡Key Insight

Kyverno evaluates policies independently; it does not infer or reconcile intent across them. When intent is fragmented across multiple policies, Kyverno will enforce all matching rules, even if their combined effects are contradictory. Part II presents design patterns to prevent this class of conflicts.

Top comments (0)