DEV Community

Cover image for Build a Kubernetes Operator
Austin Cunningham
Austin Cunningham

Posted on • Edited on

Build a Kubernetes Operator

#kubernetes #openshift #operatorsdk #go

Prerequisites :

  • Kubernetes cluster (as I am going to be adding some Openshift native resources I will be using CRC which is a local dev kubernetes cluster you can install on your pc)
  • Cluster admin access to that cluster

Using the Operator-sdk to bootstrap your project

First what is an Kubernetes operator? Operators allow you to extend the Kubernetes API by adding you own custom resources to a cluster. This is the most basic operator I can make, It will create a pod for a micro service and create a route for the micro service and allow you to specify the amount of replicas. I will outline all the steps I have take. First I use the generate command to scaffold the operator project.

mkdir pod-route
cd pod-route
# --domain example.com is used in the operator-sdk quickstart guide this is used to create the api group, think of package in java. 
# This was my first gotcha following my misreading of the docs. You need to be careful when choosing domain name as is difficult to revert after its generated.I will continue with quay.io for now.  
# --repo is your git repo where you operator code will live 
operator-sdk init --domain quay.io --repo github.com/austincunningham/pod-route
# Add a controller
# --version I use v1alpha1 (this is a Kubernetes API version for early candidates)
# --kind name of Custom Resource
operator-sdk create api --version v1alpha1 --kind Podroute --resource --controller
# build and push the operator image make docker-build docker-push IMG="quay.io/austincunningham/pod-route:v0.0.1"
Enter fullscreen mode Exit fullscreen mode

Your files should look like this and the container repo is pushed
Next I edit my api/v1alpha1/podroute_types.go file spec PodrouteSpec. The spec is basically what I want to be managed by the operator.

// PodrouteSpec defines the desired state of Podroute
type PodrouteSpec struct {
    // Image container image string e.g. "quay.io/austincunningham/always200:latest"
    // Replicas number of containers to spin up
    Image string `json:"image,omitempty"`
    Replicas int32 `json:"replicas,omitempty"`
}
Enter fullscreen mode Exit fullscreen mode

After changing the types file we need to update the files in the operator run the following commands

make generate
make manifests
Enter fullscreen mode Exit fullscreen mode

Add your controller logic

Now I can start looking at my reconcile logic in controllers/podroute_controller.go we add some RBAC rules for pods and deployments and do a client get on the cluster to find the Custom Resource(CR). The rest of this is generated code.

package controllers

import (
    "context"

    "k8s.io/apimachinery/pkg/runtime"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/client"
    "sigs.k8s.io/controller-runtime/pkg/log"

    quayiov1alpha1 "github.com/austincunningham/pod-route/api/v1alpha1"
)

// PodrouteReconciler reconciles a Podroute object
type PodrouteReconciler struct {
    client.Client
    Scheme *runtime.Scheme
}

//+kubebuilder:rbac:groups=quay.io,resources=podroutes,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=quay.io,resources=podroutes/status,verbs=get;update;patch
//+kubebuilder:rbac:groups=quay.io,resources=podroutes/finalizers,verbs=update
//+kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=core,resources=pods,verbs=get;list;

func (r *PodrouteReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    _ = log.FromContext(ctx)

    // your logic here
    // Create a Custom Resource object for Podroute, quayio part of the name is due to my earlier mistake
    cr := &quayiov1alpha1.Podroute{}
    // do a kubernetes client get to check if the CR is on the Cluster
    err := r.Client.Get(ctx, req.NamespacedName, cr)
    if err != nil {
        return ctrl.Result{}, err
    }

    return ctrl.Result{}, nil
}

// SetupWithManager sets up the controller with the Manager.
func (r *PodrouteReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&quayiov1alpha1.Podroute{}).
        Complete(r)
}
Enter fullscreen mode Exit fullscreen mode

So the first thing I need to do is check for a existing deployment and if it doesn't exist create it.
In the Reconcile function before the last return ctrl.Result{}, nil add a call to createDeployment function like so

    deployment, err := r.createDeployment(cr, r.podRouteDeployment(cr))
    if err != nil {
        return reconcile.Result{}, err
    }
    // just logging here to keep Go happy will use later
    log.Log.Info("deployment", deployment)
Enter fullscreen mode Exit fullscreen mode

I create a labels function as will be using this for all resources

func labels(cr *quayiov1alpha1.Podroute, tier string) map[string]string {
    // Fetches and sets labels

    return map[string]string{
        "app":         "PodRoute",
        "podroute_cr": cr.Name,
        "tier":        tier,
    }
}
Enter fullscreen mode Exit fullscreen mode

I create a deployment object

// This is the equivalent of creating a deployment yaml and returning it
// It doesn't create anything on cluster
func (r *PodrouteReconciler) podRouteDeployment(cr *quayiov1alpha1.Podroute) *appsv1.Deployment {
    // Build a Deployment
    labels := labels(cr, "backend-podroute")
    size := cr.Spec.Replicas
    podRouteDeployment := &appsv1.Deployment{
        ObjectMeta: metav1.ObjectMeta{
            Name:      "pod-route",
            Namespace: cr.Namespace,
        },
        Spec: appsv1.DeploymentSpec{
            Replicas: &size,
            Selector: &metav1.LabelSelector{
                MatchLabels: labels,
            },
            Template: corev1.PodTemplateSpec{
                ObjectMeta: metav1.ObjectMeta{
                    Labels: labels,
                },
                Spec: corev1.PodSpec{
                    Containers: []corev1.Container{{
                        Image:           cr.Spec.Image,
                        ImagePullPolicy: corev1.PullAlways,
                        Name:            "podroute-pod",
                        Ports: []corev1.ContainerPort{{
                            ContainerPort: 8080,
                            Name:          "podroute",
                        }},
                    }},
                },
            },
        },
    }

    // sets the this controller as owner
    controllerutil.SetControllerReference(cr, podRouteDeployment, r.Scheme)
    return podRouteDeployment
}
Enter fullscreen mode Exit fullscreen mode

I check the cluster using Client.Get for an existing deployment if not then create one using the deployment object created above.

// check for a deployment if it doesn't exist it creates one on cluster using the deployment created in deployment
func (r PodrouteReconciler) createDeployment(cr *quayiov1alpha1.Podroute, deployment *appsv1.Deployment) (*appsv1.Deployment, error) {
    // check for a deployment in the namespace
    found := &appsv1.Deployment{}
    err := r.Client.Get(context.TODO(), types.NamespacedName{Name: deployment.Name, Namespace: cr.Namespace}, found)
    if err != nil {
        log.Log.Info("Creating Deployment")
        err = r.Client.Create(context.TODO(), deployment)
        if err != nil {
            log.Log.Error(err, "Failed to create deployment")
            return found, err
        }
    }
    return found, nil
}
Enter fullscreen mode Exit fullscreen mode

Next I check if the deployment replicas match the number in the CR(Custom Resource) in the Reconcile function remove the comment log.Log.Info("deployment", deployment) and replace it with

    // If the spec.Replicas in the CR changes, update the deployment number of replicas
    if deployment.Spec.Replicas != &cr.Spec.Replicas {
        controllerutil.CreateOrUpdate(context.TODO(), r.Client, deployment, func() error {
            deployment.Spec.Replicas = &cr.Spec.Replicas
            return nil
        })
    }
Enter fullscreen mode Exit fullscreen mode

So what have we done so far we have a CR that takes in a image(container) and number of replicas and creates a deployment for it. Next we will create the Service and the route , These will have a similar pattern to the deployment i.e. create an route/service object and check if it exists if not create. We will start with the service. In the reconcile function before the last return add a createService function call return ctrl.Result{}, nil

    err = r.createService(cr, r.podRouteService(cr))
    if err != nil {
        return reconcile.Result{}, err
    }
Enter fullscreen mode Exit fullscreen mode

Use this function to create the service object

// This is the equivalent of creating a service yaml and returning it
// It doesnt create anything on cluster
func (r PodrouteReconciler) podRouteService(cr *quayiov1alpha1.Podroute) *corev1.Service {
    labels := labels(cr, "backend-podroute")

    podRouteService := &corev1.Service{
        ObjectMeta: metav1.ObjectMeta{
            Name:      "podroute-service",
            Namespace: cr.Namespace,
        },
        Spec: corev1.ServiceSpec{
            Selector: labels,
            Ports: []corev1.ServicePort{{
                Protocol:   corev1.ProtocolTCP,
                Port:       8080,
                TargetPort: intstr.FromInt(8080),
            }},
        },
    }

    controllerutil.SetControllerReference(cr, podRouteService, r.Scheme)
    return podRouteService
}
Enter fullscreen mode Exit fullscreen mode

Add a function to create the service from the service object above

// check for a service if it doesn't exist it creates one on cluster using the service created in podRouteService
func (r PodrouteReconciler) createService(cr *quayiov1alpha1.Podroute, podRouteServcie *corev1.Service) error {
    // check for a service in the namespace
    found := &corev1.Service{}
    err := r.Client.Get(context.TODO(), types.NamespacedName{Name: podRouteServcie.Name, Namespace: cr.Namespace}, found)
    if err != nil {
        log.Log.Info("Creating Service")
        err = r.Client.Create(context.TODO(), podRouteServcie)
        if err != nil {
            log.Log.Error(err, "Failed to create Service")
            return err
        }
    }
    return nil
}
Enter fullscreen mode Exit fullscreen mode

And finally the Route add createRoute function call in the Reconcile before the last return ctrl.Result{}, nil

    err = r.createRoute(cr, r.podRouteRoute(cr))
    if err != nil{
        return reconcile.Result{}, err
    }
Enter fullscreen mode Exit fullscreen mode

Create a function for the route object

// This is the equivalent of creating a route yaml file and returning it
// It doesn't create anything on cluster
func (r PodrouteReconciler) podRouteRoute(cr *quayiov1alpha1.Podroute) *routev1.Route {
    labels := labels(cr, "backend-podroute")

    podRouteRoute := &routev1.Route{
        ObjectMeta: metav1.ObjectMeta{
            Name:      "podroute-route",
            Namespace: cr.Namespace,
            Labels:    labels,
        },
        Spec: routev1.RouteSpec{
            To: routev1.RouteTargetReference{
                Kind: "Service",
                Name: "podroute-service",
            },
            Port: &routev1.RoutePort{
                TargetPort: intstr.FromInt(8080),
            },
        },
    }
    controllerutil.SetControllerReference(cr, podRouteRoute, r.Scheme)
    return podRouteRoute
}
Enter fullscreen mode Exit fullscreen mode

Add a function to create the route from the route object above

// check for a route if it doesn't exist it creates one on cluster using the route created in podRouteRoute
func (r PodrouteReconciler) createRoute(cr *quayiov1alpha1.Podroute, podRouteRoute *routev1.Route) error {
    // check for a route in the namespace
    found := &routev1.Route{}
    err := r.Client.Get(context.TODO(), types.NamespacedName{Name: podRouteRoute.Name, Namespace: cr.Namespace}, found)
    if err != nil {
        log.Log.Info("Creating Route")
        err = r.Client.Create(context.TODO(), podRouteRoute)
        if err != nil {
            log.Log.Error(err, "Failed to create Route")
            return err
        }
    }
    return nil
}
Enter fullscreen mode Exit fullscreen mode

NOTE: imports did change with these code changes

import (
    "context"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/apimachinery/pkg/types"
    "k8s.io/apimachinery/pkg/util/intstr"
    "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
    "sigs.k8s.io/controller-runtime/pkg/reconcile"

    routev1 "github.com/openshift/api/route/v1"
    appsv1 "k8s.io/api/apps/v1"
    corev1 "k8s.io/api/core/v1"
    "k8s.io/apimachinery/pkg/runtime"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/client"
    "sigs.k8s.io/controller-runtime/pkg/log"

    quayiov1alpha1 "github.com/austincunningham/pod-route/api/v1alpha1"
)
Enter fullscreen mode Exit fullscreen mode

Also because route is an openshift thing and not kubernetes native I had to add it to the scheme in the main.go file

    if err := routev1.AddToScheme(mgr.GetScheme()); err != nil {
        setupLog.Error(err, "failed to add routev1 to scheme")
        os.Exit(1)
    }
Enter fullscreen mode Exit fullscreen mode

Testing your Operator

Start up CRC (code ready containers) with crc start

crc start
INFO Adding crc-admin and crc-developer contexts to kubeconfig... 
Started the OpenShift cluster.

The server is accessible via web console at:
  https://console-openshift-console.apps-crc.testing

Log in as administrator:
  Username: kubeadmin
  Password: KUBEADMIN_PASSWORD

Log in as user:
  Username: developer
  Password: developer

Use the 'oc' command line interface:
  $ eval $(crc oc-env)
  $ oc login -u developer https://api.crc.testing:6443
Enter fullscreen mode Exit fullscreen mode

Login to the CRC cluster as kubeadmin

oc login -u kubeadmin -p KUBEADMIN_PASSWORD https://api.crc.testing:6443
Enter fullscreen mode Exit fullscreen mode

Create a project, the Makefile has a a lot of commands generated by the opeator-sdk which we can use

oc new-project podroute
# Installs the custom resource definitions onto the cluster
make install
# Create the CR on cluster
oc apply -f - <<EOF
---
apiVersion: quay.io/v1alpha1
kind: Podroute
metadata:
  name: test-podroute
  namespace: podroute
spec:
  image: quay.io/austincunningham/always200:latest
  replicas: 3
EOF
# We can then run the operator locally
make run
# Should see something like
2022-06-10T14:41:28.854+0100    INFO    Creating Deployment
2022-06-10T14:41:28.980+0100    INFO    Creating Service
2022-06-10T14:41:29.114+0100    INFO    Creating Route
Enter fullscreen mode Exit fullscreen mode

You can confirm everything is up

# get the servic
oc get service
NAME               TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)    AGE
podroute-service   ClusterIP   10.217.5.14   <none>        8080/TCP   4m38s
# get the route
oc get route
NAME             HOST/PORT                                  PATH   SERVICES           PORT   TERMINATION   WILDCARD
podroute-route   podroute-route-podroute.apps-crc.testing          podroute-service   8080                 None
# should be 3 pod replicas 
oc get pods
NAME                        READY   STATUS    RESTARTS   AGE
pod-route-96b87c455-6sw2h   1/1     Running   0          4m12s
pod-route-96b87c455-ghdm8   1/1     Running   0          4m12s
pod-route-96b87c455-md426   1/1     Running   0          4m12s
# the get route should be alive and return ok
curl http://podroute-route-podroute.apps-crc.testing/get
OK%
Enter fullscreen mode Exit fullscreen mode

Reference:
Operator-sdk quickstart guide
Operator-sdk Golang tutorial
Git repo

NOTE: built with operator-sdk v1.15.0

Top comments (2)

Collapse
 
souvikhaldar profile image
Souvik Haldar •

For the operator-sdk init command what shall I use in domain flag if I want to publish it in docker hub?

Collapse
 
austincunningham profile image
Austin Cunningham •

Best to use a domain you own to make your extension to the Kubernetes API unique. My use of quay.io has no bearing about where you publish. They use example.com in the official docs

operator-sdk init --domain example.com --repo github.com/example/memcached-operator

Read next

sf_fc profile image

Use KEDA Scaling Modifiers to Manage AI Infrastructure

Sam Farid -

lucasdeataides profile image

Why Clean Architecture Struggles in Golang and What Works Better

Lucas de Ataides -

awedis profile image

Smart File Organizer Using AWS Lambda & S3 - (Let's Build 🏗️ Series)

awedis -

pareekplatform profile image

How to Set up Disk and Bandwidth Limits in Docker

Shubham -