Platform Engineering On Kubernetes Part 1: Operators

#kubernetes #devops #docker #programming

How can engineers take an in-depth and sometimes complex platform like Kubernetes and morph it into the platform of their choosing? How can engineers take Kubernetes and turn it into exactly what they envision for their team and organization?

Luckily, that exact question is why Kubernetes exists. The whole idea of it is to be as extendable and customizable as possible.

In a recent blog post which you can find here, you’ll read about a few key ways to think about how Platform Engineering falls into the conversation of Kubernetes. In this blog post and series, you’ll start diving deeper into each of the sections inside of that blog post to get a better grasp of how Platform Engineering and Kubernetes tie so well together.

Part 1 is all about Kubernetes Operators.

Prerequisites

This blog post contains both theoretical and hands-on breakdowns. If you would like to follow along with the hands-on section, you will need the following:

Go (golang) installed, which you can find here.

Why Operators?

In the opening of this blog post, “extendable and customizable” was mentioned in the context of why Kubernetes exists in the first place, and it’s very much true. The whole original idea of Kubernetes was to give engineers a platform that they could build how they wanted with a simple baseline.

Think about it like a house. When a house first gets built, there are some outside walls, wood inside, perhaps some cutouts where rooms/light fixtures will be, and if you’re lucky, a little insulation. The rest is up to you. Where the bathroom will be, which room will be the main bedroom, what type of lights you’ll use, and what colors the walls will be.

Kubernetes is the same. It gives you a base and then you use things like Plugins (CRI, CNI, CSI, etc.) to customize it in a way that’s unique to you. The question then becomes - what if the exact type of implementation you’re looking for doesn’t exist?

Luckily, Kubernetes gives you a way to extend the API, as in, build your own API and run it on Kubernetes.

That’s where Operators come into play.

Operator Breakdown

When extending the Kubernetes API, it’ll come down to three pieces:

Custom Resource Definitions (CRD)
Controllers
Operators

A Custom Resource Definition gives you the ability to extend the API with or without an Operator (which you’ll learn about coming up) using the CustomResourceDefinition Object/kind in the apiextensions.k8s.io/v1 API. It’s an API out of the box that has one purpose; to give you the ability to extend the Kubernetes API. The CustomResourceDefinition object exists to give you the ability to create your own API.

A Controller gives you the ability to ensure that the current state of a Kubernetes object is the desired state. Let’s think about an example with Replicas. If you’re deploying Kubernetes Pods and within your Deployment you specify you want three replicas, there has to be a way to ensure that those three replicas are always running. If they aren’t, they have to be. This is where Controllers come in. The ReplicaSet Controller will run through a reconciliation loop to constantly check and ensure that the Kubernetes Deployment has three Pods running because that’s what was specified in the Kubernetes Manifest. It’s ensuring that the current state (three replicas) is the desired state (specified in the Manifest as three replicas).

A Kubernetes Operator brings the two together. It gives you the ability to create CRD’s and Controllers for the APIs you’re creating. You create the spec/object, ensure that a CRD is generated for it, and then create the reconciliation steps for ensuring that the current state of the deployment is the desired state.

Please note that you don’t need to use an Operator when creating a CRD, but it’s typically the safest option for production because you want the capabilities of a Controller.

Getting Started With Kubebuilder

You’ll typically see two ways to work with Kubernetes Operators:

Kubebuilder
The Operator Framework

Both are popular, but it does appear that Kubebuilder is used a bit more. For example, Kubebuilder is the backend for Cluster API.

The thing is, Kubebuilder is a huge topic. In fact, there’s an entire book on it: https://book.kubebuilder.io/quick-start.html. Because of that, this section will go over what you’ll need to get started with Kubebuilder. Otherwise, it’ll end up being 40+ pages long.

First, install Kubebuilder.

curl -L -o kubebuilder "https://go.kubebuilder.io/dl/latest/$(go env GOOS)/$(go env GOARCH)"
chmod +x kubebuilder && mv kubebuilder /usr/local/bin/

Next, initialize Kubebuilder. This will initialize the project and give you template code to work with so you don’t have to start from scratch. In this case, the domain and repo are specified, but you can use your own.

kubebuilder init --domain simplyanengineer.com --repo github.com/adminturneddevops/mikesapi

Once the project is initialized, it’s time to create your group and API/object. Much like how the Deployment object is part of the apps/v1 Core API Group, you’re using the below command (kubebuilder create api to create your own group and object/kind.

kubebuilder create api --group simplyanengineer.com --version v1 --kind MikesAPI

After that, you’ll see the new project created along with the templates that you need to use Kubebuilder for your API purposes.

Open up api/v1/filename_types.go. Once you do that, you’ll see a type called YourAPINameSpec that’s defined as a Struct. This is where you begin to specify the maps/lists that you’re passing into the Kubernetes object that you’re creating.

For example, in a Kubernetes Deployment Manifest spec, there’s a section for image to specify your container image for the Kubernetes Pod. The below section is where that image map would be specified.

Under the controllers directory is where you’ll find the core template to start your reconciliation steps. Reconciliation is the loop that’s constantly iterated over to confirm that the current state of your Kubernetes object is the desired state.