Writing a very basic kubernetes mutating admission webhook

My findings when attempting to write a (very) simple kubernetes mutating admission webhook.

Okay.. so, webhooks…?

Admission webhooks help you do some really cool stuff, there are two kinds of webhooks, validating and mutating. We will concentrate on the mutating admission webhook in this post.

Mutating admission webhooks allow you to “modify” a (e.g.) Pod (or any kubernetes resource) request. E.g. you can modify a Pod to use a particular scheduler, add / inject sidecar containers (think LinkerD sidecar), or even reject it if it doesn’t meet some security requirements, etc. etc. — all without having to write a full fledged “micro” service to do this. The webhook can live anywhere, in practice, k8s just needs to know where anywhere is.

Setup

The setup is easy, but important, all you really need to make sure is that the MutatingAdminssionController is enabled in the k8s api-server. To check if your k8s cluster has this enabled, you can use

kubectl api-versions | grep admissionregistration

For development, I can recommend using Kubernetes-In-Docker (KinD), all you need is Docker and KinD. KinD doesn’t auto-enable these, you can use this KinD configuration (kind.yaml)

---
kind: Cluster
apiVersion: kind.sigs.k8s.io/v1alpha3
kubeadmConfigPatches:
- |
 apiVersion: kubeadm.k8s.io/v1beta2
 kind: ClusterConfiguration
 metadata:
 name: config
 apiServer:
 extraArgs:
 "enable-admission-plugins": "NamespaceLifecycle,LimitRanger,ServiceAccount,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,PersistentVolumeClaimResize,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota"
nodes:
- role: control-plane

Spin up the KinD cluster with

kind create cluster --conifg kind.yaml

I trust you’ll be able to configure kubectl etc. to now use this cluster going forward.

This is it, the setup is done. Well done! :)

Deployment

I’d like to start here, as this is the easiest part of this “tutorial” (if you will). Deploying an admission webhook is in practice the same as deploying any other service onto a k8s cluster.

All we need is

• a Service
• a Deployment
• a MutatingWebhookConfiguration

The first two are simple, so I won’t go into those deeper, you can see what I’ve used on Github.

Before you just go ahead and deploy this to prod ...

Please do think about a couple _what-if_s before using webhooks in a production environment:

• what if the request to your webhook fails (conn drop, conn reset, no dns, etc.)
• what if the deployment failed and the pod is stuck in a crash-loop
• what if the webhook has become mission critical and must be functional 100% of the time
• what if you create a circular dependency (my favourites!)
• what if you have a 3214 node cluster with many 10s of thousand resources & requests all depending on this webhook, which is scaled to 1 pod ;)

Note: Webhooks are only called over SSL/TLS so your webhook must have a valid signed certificate (we do this further down the line)

The MutatingWebhookConfiguration is where we tell k8s which resource requests should be sent to our webhook. The configuration consists of the following properties:

• apiVersion (at the time it is: admissionregistration.k8s.io/v1beta1)
• kind (must be: MutatingWebhookConfiguration)
• metadata (the usual: name, annotations, labels)
• webhooks (a list of type webhook)

The webhook (type) consists of these properties:

• name
• clientConfig
• - caBundle (we will get this from the k8s cluster itself)
• - service to send the AdmissionReview requests to
• rules ( a list of rules that define which resource operations should be matched, these rules make sure that k8s resource requests are sent to your webhook )
• namespaceSelector (the usual: matchLabels: {“label_name”: “label_value”}
• …

there are many more that could be used, for a simple non-production webhook to play with, the above will suffice.

The full list of properties can be seen here.

A rule consists of the following:

• operations (a list of [operations](https://godoc.org/k8s.io/api/admissionregistration/v1beta1#OperationType) to match, in our case ["CREATE"])
• apiGroups (in our case, empty [""])
• apiVersions (in our case, this is ["v1"])
• resources (in our case, this is ["pods"])

apiGroups, apiVersions and resources are all (kind of) dependent on each other, in this example it’s quite easy as Pod is part of the core api group so it doesn’t need specifying, the empty [""] is matching the core api group.

Here is an example:

---
apiVersion: admissionregistration.k8s.io/v1beta1
kind: MutatingWebhookConfiguration
metadata:
 name: mutateme
 labels:
 app: mutateme
webhooks:
 - name: mutateme.default.svc.cluster.local
 clientConfig:
 caBundle: ${CA_BUNDLE}
 service:
 name: mutateme
 namespace: default
 path: "/mutate"
 rules:
 - operations: ["CREATE"]
 apiGroups: [""]
 apiVersions: ["v1"]
 resources: ["pods"]
 namespaceSelector:
 matchLabels:
 mutateme: enabled

the ${CA_BUNDLE} above refers to the actual CA bundle retrieved from the k8s API, replace it with your own; you can get your cluster’s CA bundle with

kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.certificate-authority-data}'

As webhooks can only be called over HTTPS (SSL/TLS) you will need to generate a new ssl key & certificate for it. Doing this is somewhat involved, so probably best to take a look here:

and/or

• https://github.com/morvencao/kube-mutating-webhook-tutorial/blob/master/deployment/webhook-create-signed-cert.sh

The webhook

Pasting the entire code here is probably not very useful, so I’ll concentrate on the essentials that helped me understand a few things.

Conceptually, what a webhook has to do is relatively easy, it receives a AdmissionReview, and responds with an AdmissionReview :) — most blog posts I’ve found concentrate on the Request and Response objects only, which can be a bit confusing.

In essence, this is what needs to happen ..

• the K8S api server will send a AdmissionReview “request” and expects a AdmissionReview “response” ; this can get confusing, but i.e. it is something like this

{
 "kind": "AdmissionReview",
 "apiVersion": "admission.k8s.io/v1beta1",
 "request": {...}
}

• the AdmissionReview consists of AdmissionRequest and AdmissionResponse objects
• the webhook needs to “unmarshal” the AdmissionReview from JSON format into some kind of object so it can read the AdmissionRequest and modify the AdmissionResponse object within it
• the webhook i.e. creates its own AdmissionResponse object, copies the UID from the AdmissionRequest object and replaces the AdmissionResponse object within the AdmissionReview with its own (overwrites it)
• responds with a AdmissionReview object in JSON

{
 "kind": "AdmissionReview",
 "apiVersion": "admission.k8s.io/v1beta1",
 "request":  { ... ORIGINAL REQUEST ... },
 "response": { ... OUR RESPONSE ... }
}

Where to start?

We will start with a basic web server, that supports SSL/TLS, and can read and respond in JSON format.

In practice, you can use whatever programming language you’d like for this, I have used Go, but this can easily be done in Python or any other compiled or interpreted language. Ideally though, use a language that already has K8S libraries so you don’t have to create our own objects/types; Go (naturally) has these, but there are also at least the Python libraries you could use.

Here’s what I did to accept requests on port 8443 with SSL/TLS set up to use a key & cert.

The above example creates a server s with 2 handlers, one for / and one for /mutate which is the endpoint that will be called by k8s (which is what we specified in the MutatingWebhookConfiguration); it listens on port :8443 and we use the ListenAndServeTLS method to serve requests over SSL/TLS.

I split up the logic of http request/response from the actual processing of the AdmissionReview request as I find it’ll be easier to test the function/s independently; so the /mutate handler really only does

• take the JSON http request and read in the BODY
• send the BODY to the Mutate function in the mutate package (m) — the unmarshalling from JSON into the appropriate object structure, modification and marshalling back to JSON is done here
• respond with a JSON message, either one that describes an Error or a AdmissionReview including our AdmissionResponse

So far so good, the potentially more challenging part is next.

pkg/mutate/mutate.go

The mutate package does the actual processing of AdmissionReview requests by …

• unmarshalling the received JSON payload into a AdmissionReview object
• using the AdmissionRequest object to decide what we should do
• creating the JSONPatch
• creating a new AdmissionResponse
• updating the AdmissionReview with our new AdmissionResponse
• marshal the final AdmissionReview back into JSON and return it

API Resources, Types, etc.

Something that I’m struggling the most with is finding the correct repository, version and even file/s that I need to do things with k8s or its supported resources.

Here’s what I’ve found out so far ..

• Pod and other core resource types (which are used to unmarshal JSON into) can be found in kubernetes/api/blob/master/core/v1/types.go
• Metadata objects (which usually are part of main object like Pod), seem to be kept in the apimachinery repo, it doesn’t have a description so 🤷🏻‍♂️apimachinery/blob/master/pkg/apis/meta/v1/types.go
• Other resources and their definitions seem to reside in kubernetes/api/{extensions,node,autoscaling,..}; the admission types are located here: kubernetes/api/blob/master/admission/v1beta1/types.go (for now)

However, that’s not how they’re imported (at least not in Go); for our example, we need these imports:

import (
 v1beta1 "k8s.io/api/admission/v1beta1"
 corev1  "k8s.io/api/core/v1"
 metav1  "k8s.io/apimachinery/pkg/apis/meta/v1"
)

IIRC they must be k8s.io/ imports and not github.com/ imports, even though the repos are hosted on Github.

Marshal & Unmarshal

These terms always get me confused, but I’ll try to explain

• marshal: when you convert a object into a JSON (or other, e.g. protobuf) equivalent representation
• unmarshal: when you convert JSON (or other) into your own object’s representation

For the webhook we need to

• unmarshal the JSON AdmissionReview into a Go AdmissionReview object, I called it ar
• check if there’s a embedded raw object and if so, again, unmarshal it into the object type that we expect, in our case a Pod, which I called simply pod; if that fails, we should not try to continue and return an error
• marshal the JSONPatch map into a valid JSONPatch, which, needs to be valid JSON
• marshal the final (Go) AdmissionReview object back into a JSON AdmissionReview

this is why we needed to find those imports earlier, so we can create objects of the correct types and unmarshal JSON into them or marshal them into JSON.

The types we use/need are

• AdmissionReview (v1beta1.AdmissionReview)
• AdmissionRequest (v1beta1.AdmissionRequest)
• Pod (corev1.Pod)
• PatchTypeJSONPatch (v1beta1.PatchTypeJSONPatch)
• AdmissionResponse (v1beta1.AdmissionResponse)
• Status (metav1.Status) to set the AdmissionResponse.Result

Why do we create a JSONPatch?

This is because the mutating webhook does not modify the k8s resource, but responds with a patch, telling k8s how to modify the object for us. I found this rather counter intuitive, since we’re creating a "Mutating Admission Webhook" but don’t actually mutate anything.

More about JSONPatch expressions and how they work can be found here, but essentially they consist of an operation (op), a path and a value, e.g.:

{
 "op": "replace",
 "path": "/spec/containers/0/image",
 "value": "debian"
}

In our case, it instructs how and what operations it should apply to the resource that was requested. You can have more than one operation/instruction, e.g. a list of operations [{"op": ..},{"op": ..},{"op": ..}] and I believe they’ll be executed in the same order. Something for you to try and find out ;).

I used the following to create a JSONPatch list for the webhook, it replaces any container image to use the Debian docker image instead of the originally requested .. FWIW - it's a list as a Pod may have more than 1 container.

// resp is the AdmissionReview.Response 
p := []map[string]string{}

for i := range pod.Spec.Containers {
  patch := map[string]string{
    "op": "replace",
    "path": fmt.Sprintf("/spec/containers/%d/image", i), 
    "value": "debian",
  }
  p = append(p, patch)
}

// parse the []map into JSON
resp.Patch, err = json.Marshal(p)

This is it!

You can find the all the code on Github, I’d encourage you to try it without looking first :) … except for the details on the SSL ca, key, cert signing. I’ve also added links to resource that I believe will be helpful to understand what’s going on.

Kudos to the IBM-Cloud blog on Medium (link below) that I used as inspiration for this post and also as part reference when I got stuck. It’s more complete but also more complicated (at least at the time it seemed like it); if you're intending to create a proper version of this, you should check it out.

alex-leonhardt/k8s-mutate-webhook

Tip: Use a IDE / Editor that can help with code completion, dependencies, etc. — I used VSCode, but there are also good Vim plugins.

Resources

• https://godoc.org/k8s.io/api/admissionregistration/v1beta1#MutatingWebhook
• https://godoc.org/k8s.io/api/admission/v1beta1
• https://godoc.org/k8s.io/kubernetes/pkg/apis/admission
• https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.15/#mutatingwebhookconfiguration-v1beta1-admissionregistration-k8s-io
• https://github.com/morvencao/kube-mutating-webhook-tutorial

Kudos

https://medium.com/ibm-cloud/diving-into-kubernetes-mutatingadmissionwebhook-6ef3c5695f74

---

Comments

[Robert Nemeti]

Hey Alex, thanks for the great article. quick question, if you let the api server sign your tls certificate, why do u need to load the apiserver's CA into the mutatingwebhook? shouldn't the apiserver trust the certificate signed by itself automatically?

[Alex Leonhardt]

Hi Robert, I'm not sure where you're seeing that? It is however loading the signed Cert and the Key, here:

log.Fatal(s.ListenAndServeTLS("./ssl/mutateme.pem", "./ssl/mutateme.key"))

[Robert Nemeti]

Hi Alex, thanks for the reply. What I meant is that in you ssl.sh script you create the tsl csr, upload it into the apiserver, and you sign it ( basically with the CA of the apiserver )

kubectl certificate approve ${CSR_NAME}
``

then you get the CA fron the apiserver and put it into the `mutatingwebhookconfiguration` resource:

• caBundle (we will get this from the k8s cluster itself)

So basically you sign the certificate with the apiserver's CA and you load the CA into apiserver with the `caBundle` field. Shouldn't the apiserver already trust it's own CA?

[Alex Leonhardt]

Ooohhhh - you're right! Shouldn't be needed if you sign it with the K8S cluster's CA, it's only needed when you use your own CA.

For reference: godoc.org/k8s.io/api/admissionregi...

// `caBundle` is a PEM encoded CA bundle which will be used to validate the webhook's server certificate.
    // If unspecified, system trust roots on the apiserver are used.
    // +optional
    CABundle []byte `json:"caBundle,omitempty" protobuf:"bytes,2,opt,name=caBundle"`

[Robert Nemeti]

I tried without caBundle, but it doesn't work, it is complaining about unknown certificate. I thought maybe you know why ....

[Alex Leonhardt]

Hmm.. potentially something to do with the api server "client" portion not trusting its own (k8s) CA - just like curl, I'm pretty sure it'll use whatever system CAs are installed by default (ca-certs package?);

I've not further looked into this so cannot really help too much, but I'd check if the API servers own CAs are actually configured to be trusted when the api server is "the client".

Sorry if I cannot be more of help, but short of knowing what your setup is and how things are configured, I don't think I can help much more here.