Enforcing Kubernetes Probes with a Custom Admission Webhook

#kubernetes #sre #devops

Introduction

Container health checks via liveness and readiness probes are critical for resilient workloads in Kubernetes. However, it’s easy for teams to forget adding them, leading to issues that are only discovered in production.

In this post, I’ll walk through building a custom Kubernetes Admission Controller that enforces the presence of liveness and readiness probes on all pods. We’ll cover everything: why, how, code, testing, and deployment.

Why Enforce Probes?

Resilience: Probes help Kubernetes automatically restart failing pods and avoid routing traffic to unhealthy ones.

Standardization: Ensures organizational best practices are followed without relying on manual checks or code reviews.

Automation: Moves policy enforcement to the cluster level, catching issues before they reach production.

Design Overview

Our Validating Admission Controller runs as a K8s deployment. It intercepts all pod creation requests in it's cluster and:

Checks for startup, readinessProbe and livenessProbe on every container.
Rejects pods missing atleast one required probe, with a clear error message. You can configure which probes you want to enforce by setting the environment variable ENFORCE_PROBES (default: liveness,readiness). You can also exclude some namespaces using the environment variable EXCLUDE_NAMESPACES (default: kube-system,kube-public)

The flow:

User creates/updates a pod.
K8s API calls our webhook.
Webhook inspects the pod spec.

If probes are missing, request is denied. The pod creation fails with an error from our custom admission controller.

Demo

Pod in `default` namespace with no probes

First, let's try to create a pod with no probes defined in the default namespace:

Looks like our admission webhook intercepted it and blocked the pod creation. Let's confirm that from the logs:

Pod in `kube-system` namespace with no probes defined

Now, we try to create a pod with no probes defined in the kube-system namespace:

Looks like the pod creation succeeded this time, which is the expected behaviour. This is because, we have excluded the kube-system namespace from the probe enforcer. Let's verify from the logs:

Pod in `default` namespace with both liveness and readiness probes defined

The pod creation succeeded. Let's verify from the logs again:

Installation

Pre-requisites

A Kubernetes cluster with cert-manager installed. https://cert-manager.io/docs/installation/#default-static-install

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.18.0/cert-manager.yaml

Now, apply the manifests required for installing the admission controller:

kubectl kustomize https://github.com/ashiqursuperfly/k8s-probe-enforcer-ac/k8s | kubectl apply -f -

Once all the manifests are applied, our custom admission webhook is ready to intercept all pod create/update requests. Try creating a pod with a liveness/readiness probe missing and see the magic!

Implementation

The complete source code is hosted on Github: https://github.com/ashiqursuperfly/k8s-probe-enforcer-ac
Here's the core logic that validates an incoming pod create/update request:

if cfg.ExcludeNamespaces[pod.Namespace] {
    log.Printf("Namespace '%s' is excluded from validation. Allowing pod '%s'.", pod.Namespace, pod.Name)
    respondAdmission(w, review, true, "")
    return
}

containersMissingProbes := map[string][]string{}
for _, c := range pod.Spec.Containers {
    for _, probe := range cfg.EnforceProbes {
        switch probe {
        case "readiness":
            if c.ReadinessProbe == nil {
                containersMissingProbes[c.Name] = append(containersMissingProbes[c.Name], "readinessProbe")
            }
        case "liveness":
            if c.LivenessProbe == nil {
                containersMissingProbes[c.Name] = append(containersMissingProbes[c.Name], "livenessProbe")
            }
        case "startup":
            if c.StartupProbe == nil {
                containersMissingProbes[c.Name] = append(containersMissingProbes[c.Name], "startupProbe")
            }
        }
    }
}

if len(containersMissingProbes) == 0 {
    log.Printf("Pod '%s' passed validation. All required probes are present.", pod.Name)
    respondAdmission(w, review, true, "")
    return
}

// Prepare detailed error message
var failMsg string
for container, probes := range containersMissingProbes {
    failMsg += fmt.Sprintf("Container '%s' missing: %v; ", container, probes)
}
log.Printf("Pod '%s' rejected. %s", pod.Name, failMsg)

respondAdmission(w, review, false, failMsg)