DEV Community: jai sandesh

Designing Conflict-Free Kyverno Policies: Patterns for Ownership, Scope, and Determinism

jai sandesh — Mon, 09 Feb 2026 08:15:53 +0000

▶️Introduction

In Kyverno Policy Conflicts, we saw a pattern of structural failures:

Multiple policies mutating the same fields
Validation rules assuming a state that mutations later invalidate
Policy intent fragmented across independent rules
Even when policies are “correct,” nothing prevents someone from introducing a conflicting policy later

This is not a blog about fixing YAML syntax or tweaking individual rules.

It’s about designing policies so contradictions cannot exist in the first place.

🗺️The Strategy (High Level)

We prevent future contradictions using four layers, all enforceable by Kyverno:

RBAC - Very few having access to creates, update,and delete the policy.
Policy ownership – only one policy may own a given field.
Intent isolation – policies for different intents must never overlap.
Guardrail policies – deny creation of conflicting policies.

👥RBAC

RBAC can provide a basic guard against policy tampering or the introduction of contradictory Kyverno policies by unknown or unauthorized actors. However, RBAC alone is not sufficient—it only controls who can act, not what they are allowed to change or why.

There are two complementary ways to address this:

Restrict access via RBAC
Define strict RoleBindings / ClusterRoleBindings so that only approved users or groups are allowed to create, update, or delete Kyverno policies.
Enforce ownership with a Kyverno validating policy
Add a validating policy that denies any modification to Kyverno policies unless the request originates from approved service accounts—typically the ones responsible for policy reconciliation, such as Argo CD or CI/CD systems.

Example policy:

apiVersion: policies.kyverno.io/v1
kind: ValidatingPolicy
metadata:
  name: only-argocd-can-manage-kyverno-policies
spec:
  validationActions:
    - Deny
  matchConstraints:
    resourceRules:
      - apiGroups: ["policies.kyverno.io"]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE", "DELETE"]
        resources:
          - mutatingpolicies
          - validatingpolicies
          - generatingpolicies
          - deletingpolicies
          - imagevalidatingpolicies
          - policyexceptions
  variables:
    - name: sa
      expression: parseServiceAccount(request.userInfo.username)
  validations:
    - message: "Only Argo CD is allowed to create/update/delete Kyverno policies."
      expression: >
        variables.sa.Namespace == "argocd" &&
        variables.sa.Name == "argocd-application-controller"

🛡️Ownership + Guardrails

The core idea is simple: every mutable JSON path must have exactly one owning policy, and the cluster must enforce it.
Ownership means nothing if it isn’t enforced.Guardrails mean nothing without clear ownership.These two have to exist together.

Examples:
This policy owns the field entirely, encodes a complete and coherent intent, does not rely on execution order.

apiVersion: policies.kyverno.io/v1
kind: MutatingPolicy
metadata:
  name: canonical-security-context
spec:
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["pods"]
  mutations:
    - patchType: ApplyConfiguration
      applyConfiguration:
        expression: >
          Object{
            spec: Object.spec{
              securityContext: Object.spec.securityContext{
                runAsNonRoot: true,
                runAsUser: 1000
              }
            }
          }

Once a single policy is defined as the owner of securityContext, no other policy should be allowed to change it.
From that point on, the cluster must reject any new policy that tries to modify securityContext fields.

The YAML below enforces this by denying the creation of such policies.

apiVersion: policies.kyverno.io/v1
kind: ValidatingPolicy
metadata:
  name: guardrail-security-context-ownership
spec:
  validationActions:
    - Deny
  matchConstraints:
    resourceRules:
      - apiGroups: ["policies.kyverno.io"]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources:
          - mutatingpolicies
  validations:
    - message: >
        spec.securityContext is owned by the canonical-security-context policy.
        Do not mutate it elsewhere.
      expression: >
        !object.spec.mutations.exists(m,
          m.applyConfiguration.expression.contains("securityContext")
        )

🏷️Intent Isolation: Prevent Accidental Interaction

The rule is simple policies of the same kind but with different intentions must be clearly separated.
This separation must be enforced through conditions in the policy spec. One of the most common and reliable ways to do this is by using labels to scope policy execution.

Intent via labels:

metadata:
  labels:
    policy.intent: security

Policies match only that intent:

matchConstraints:
  resourceRules:
    - apiGroups: [""]
      apiVersions: ["v1"]
      resources: ["pods"]
      selector:
        matchLabels:
          policy.intent: security

📌Conculsion

Preventing policy conflicts isn’t about writing more rules it’s about setting clear boundaries.
When ownership is explicit, intent is isolated, and guardrails are enforced, Kyverno policies stop interacting in surprising ways. Exceptions stay intentional, and future mistakes are blocked early.

Kyverno Policy Conflicts: Why “Last-Writer-Wins” Is a Production Bug

jai sandesh — Mon, 09 Feb 2026 08:14:19 +0000

▶️Introduction

Kyverno has become the go-to policy engine for Kubernetes because it feels simple:

Policies are YAML,
They run at admission time,
No custom controllers required.

That simplicity is also why teams run into policy conflicts in production.

Most Kyverno incidents are not bugs in Kyverno.
They are caused by incorrect assumptions about how policies are executed.

🤔The Assumption Almost Everyone Makes

Most teams implicitly assume at least one of the following:

Policies run in the order they are applied
Cluster-scoped policies are evaluated before namespace-scoped ones
Kyverno resolves conflicts deterministically

⚠️ None of these assumptions are true.

Kyverno provides:

no policy priority,
no guaranteed ordering across policies,
no conflict detection between mutations.

🧠How Kyverno Actually Processes an Admission Request

For a single admission request, Kyverno evaluates policies in phases:

All matching mutate rules are executed
All matching validate rules are evaluated
All matching generate rules are processed

This order is guaranteed.

What is not guaranteed:

The execution order between two different mutate policies
The execution order between mutate rules across different policies

Inside a single policy, rules are processed top-to-bottom.Across multiple policies, order is explicitly undefined.

📌Concrete Failure Modes

🧪Example1: Mutate vs Mutate: Where Things Quietly Break

⚠️The examples are written on kyverno version 1.17 and policies are created at the same time

Consider two mutate policies that both match the same Pod.
Policy A

apiVersion: policies.kyverno.io/v1
kind: MutatingPolicy
metadata:
  name: enforce-non-root
spec:
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["pods"]
  mutations:
    - patchType: ApplyConfiguration
      applyConfiguration:
        expression: >
          Object{
            spec: Object.spec{
              securityContext: Object.spec.securityContext{
                runAsNonRoot: true
              }
            }
          }

Policy B

apiVersion: policies.kyverno.io/v1
kind: MutatingPolicy
metadata:
  name: force-run-as-user-0
spec:
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["pods"]
  mutations:
    - patchType: ApplyConfiguration
      applyConfiguration:
        expression: >
          Object{
            spec: Object.spec{
              securityContext: Object.spec.securityContext{
                runAsUser: 0
              }
            }
          }

What Kyverno Sees

Both policies match the same Pod
Both apply mutations
Kyverno applies both patches
Execution order across policies is not guaranteed

🧪Example 2: Mutate vs Validate — Self-Inflicted Admission Failure

Scenario
One policy mutates a resource to enforce a default.Another policy validates the same field assuming a different expected state.
Both policies are correct in isolation.

MutatingPolicy — Add default label

apiVersion: policies.kyverno.io/v1
kind: MutatingPolicy
metadata:
  name: add-default-app-label
spec:
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["pods"]
  mutations:
    - patchType: ApplyConfiguration
      applyConfiguration:
        expression: >
          Object{
            metadata: Object.metadata{
              labels: Object.metadata.labels{
                app: "default"
              }
            }
          }

ValidatingPolicy — Require a specific label value

apiVersion: policies.kyverno.io/v1
kind: ValidatingPolicy
metadata:
  name: require-app-frontend
spec:
  validationActions:
    - Deny
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["pods"]
  validations:
    - message: "Label app=frontend is required"
      expression: "object.metadata.?labels.app.orValue('') == 'frontend'"

What Actually Happens

Mutation runs first and sets app=default
Validation runs after mutation
Validation fails because app=frontend is required

❗ Admission denied.
❗ Even if the Pod originally satisfied the validation rule.

📉Why This Gets Worse at Scale

With just one or two policies, these interactions are easy to reason about.

At scale:

multiple teams mutate the same fields,
validation rules encode different assumptions,
scopes overlap silently.

Failures:

don’t always happen immediately,
often appear only for specific workloads,
are hard to trace back to policy interaction, not policy logic.

Kyverno evaluates each policy independently.
It does not infer intent or resolve conflicts.

💡Key Insight

Kyverno evaluates policies independently; it does not infer or reconcile intent across them. When intent is fragmented across multiple policies, Kyverno will enforce all matching rules, even if their combined effects are contradictory. Part II presents design patterns to prevent this class of conflicts.

Zero-Trust in Practice: Enforcing Time-Bound RoleBindings with Kyverno

jai sandesh — Sun, 26 Oct 2025 17:32:41 +0000

👨‍💻Intro

Temporary admin access in Kubernetes is messy.You grant a RoleBinding for debugging and forget to remove it congratulations, you now have a hidden cluster admin forever.
I wanted to fix that with Kyverno no CronJobs, no cleanup scripts.

This guide explains how to enforce temporary RoleBindings both at the cluster level and the namespace level, using Kyverno policies that clean up automatically.

⚙️ Why Kyverno?

Manual cleanup jobs are fragile.
They rely on human discipline, timing, and extra components like scripts or CronJobs.Kyverno solves this at the policy level by using its inbuilt features
It watches resources in real time and can automatically delete expired RoleBindings based on their labels.
No sidecars, no containers, no scripting just YAML that enforces itself.

🏗️ High-Level Overview

Give Kyverno permission to delete RoleBindings and ClusterRoleBindings.
Use Kyverno’s built-in cleanup capability (cleanup.kyverno.io/ttl) to define expiration times.
Kyverno will automatically remove the temporary elevated access when the TTL expires.

🔧 Implementation

Tested on:
Kyverno: v1.15.2
Kubernetes: v1.34

Step1: Install Kyverno

helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update
helm install kyverno kyverno/kyverno -n kyverno --create-namespace

Step2: Grant Cleanup Permissions
Kyverno’s cleanup controller requires permission to delete RoleBindings and ClusterRoleBindings.
Create a ClusterRole and bind it to Kyverno’s service account (kyverno-cleanup-controller) in the kyverno namespace.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kyverno-cleanup-access
rules:
- apiGroups: ["rbac.authorization.k8s.io"]
  resources:
    - rolebindings
  verbs:
    - delete
    - get
    - list
    - watch
- apiGroups: ["rbac.authorization.k8s.io"]
  resources:
    - clusterrolebindings
  verbs:
    - delete
    - get
    - list
    - watch

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kyverno-cleanup-access-binding
subjects:
- kind: ServiceAccount
  name: kyverno-cleanup-controller
  namespace: kyverno
roleRef:
  kind: ClusterRole
  name: kyverno-cleanup-temp-access
  apiGroup: rbac.authorization.k8s.io

Step3: Create Temporary RoleBindings
I’ll create a RoleBinding (and assume the Role already exists). For elevated access, I’ll use the built-in edit Role as the example.

kubectl create ns test-ns
kubectl create sa test

creating roles and rolebindings

kubectl create rolebinding edit-access-binding --clusterrole=edit --serviceaccount=test-ns:test --namespace=test-ns

kubectl create clusterrolebinding edit-access-clusterbinding --clusterrole=edit --serviceaccount=test-ns:test

Step4: Add TTL Labels
Label these RoleBindings with Kyverno’s built-in TTL label.
This determines how long they should exist before automatic cleanup.

kubectl label rolebinding edit-access-binding cleanup.kyverno.io/ttl=60m -n test-ns

kubectl label clusterrolebinding edit-access-clusterbinding cleanup.kyverno.io/ttl=60m

After 60 minutes, Kyverno will automatically delete both bindings — no scripts required.

🧾 Summary:

Kyverno natively supports automatic cleanup using the cleanup.kyverno.io/ttl label.
By granting it proper permissions, you can enforce expiration policies for RoleBindings, ClusterRoleBindings, or any other resources that need timed deletion.

This approach:

Eliminates manual cleanup scripts
Prevents forgotten privileged bindings
Keeps your cluster secure and self-governing

No external scheduler. No drift. Just policy.