Building My First Kubernetes Custom Resource

In my previous post, I wrote about how I started looking at Kubernetes not only as something to deploy applications on, but also as something we can use to build platforms.

The next thing I wanted to understand was how Kubernetes can be extended.

Normally, we work with resources like:

Pod
Deployment
Service
ConfigMap
Secret

But what if I want to create my own Kubernetes resource?

Something like:

apiVersion: platform.shubforge.dev/v1alpha1
kind: Greeting

metadata:
  name: hello

spec:
  message: "Hello from Platform Lab"

Kubernetes does not know what a Greeting is by default.

This is where Custom Resource Definitions, or CRDs, come in.

What is a CRD?

A Custom Resource Definition allows us to extend the Kubernetes API with our own resource types.

For example, after creating a Greeting CRD, I can run:

kubectl get greetings

just like I normally run:

kubectl get pods

This is what I found interesting.

We are not creating a completely separate API.

We are extending the Kubernetes API itself.

Creating the Greeting CRD

For my first experiment, I wanted to keep the resource very simple.

A Greeting will only contain a message.

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition

metadata:
  name: greetings.platform.shubforge.dev

spec:
  group: platform.shubforge.dev

  scope: Namespaced

  names:
    plural: greetings
    singular: greeting
    kind: Greeting
    shortNames:
      - greet

  versions:
    - name: v1alpha1
      served: true
      storage: true

      schema:
        openAPIV3Schema:
          type: object

          properties:
            spec:
              type: object

              properties:
                message:
                  type: string

              required:
                - message

There are a few important things here.

The resource is called:

Greeting

Its API group is:

platform.shubforge.dev

and the first API version is:

v1alpha1

I am using v1alpha1 because this API is still experimental and will probably change while I continue learning and building the project.

The resource is also Namespaced, which means a Greeting belongs to a Kubernetes namespace, just like resources such as Deployments and ConfigMaps.

Installing the CRD

For this project, I am running Kubernetes locally using Kind.

Once the cluster is running, the CRD can be installed using:

kubectl apply -f k8s/crds/greetings.yaml

After installing it, I can check the available CRDs:

kubectl get crds

Now Kubernetes knows about:

greetings.platform.shubforge.dev

At this point, Greeting has become a valid Kubernetes resource type.

I can also check it directly:

kubectl get crd greetings.platform.shubforge.dev

Creating My First Greeting

Now I can create an actual Greeting.

apiVersion: platform.shubforge.dev/v1alpha1
kind: Greeting

metadata:
  name: hello

spec:
  message: "Hello from Platform Lab"

I have this sample resource under:

k8s/samples/greeting.yaml

After applying it:

kubectl apply -f k8s/samples/greeting.yaml

I can run:

kubectl get greetings

and get something similar to:

NAME    AGE
hello   10s

This now feels similar to working with any other Kubernetes resource.

I can also inspect it using:

kubectl describe greeting hello

or:

kubectl get greeting hello -o yaml

Because I added a short name in the CRD:

shortNames:
  - greet

I can also use:

kubectl get greet

Kubernetes Can Also Validate It

Another useful thing about CRDs is that we can define the structure of our custom resource.

For the Greeting resource, I defined:

message:
  type: string

and made it required:

required:
  - message

So if I try to create a Greeting without a message, Kubernetes will reject it.

For example, this is not valid:

apiVersion: platform.shubforge.dev/v1alpha1
kind: Greeting

metadata:
  name: invalid

There is no:

spec:
  message:

but our CRD says that message is required.

This helped me understand that a CRD is not only about giving Kubernetes a new resource name.

We can also define the structure and validation rules for our API.

Using a Taskfile

While working on this project, I also wanted to avoid repeatedly typing the same commands.

So I added a Taskfile.yml at the root of the repository.

For example, instead of remembering commands like:

kind create cluster ...
kubectl apply -f k8s/crds/
kubectl apply -f k8s/samples/greeting.yaml
kubectl get greetings

I can use simpler project commands:

task cluster:create
task crd:install
task greeting:create
task greeting:get

The Taskfile is only a convenience around tools like kind and kubectl.

It gives me one common way to run the project locally.

As the project grows, this should also help avoid having setup instructions spread across multiple scripts and commands.

I can see all the available tasks using:

task --list

Running the Example

The current example can be started with a few commands.

First, create the local Kind cluster:

task cluster:create

Install the CRD:

task crd:install

Create the sample Greeting:

task greeting:create

Check the created resource:

task greeting:get

To inspect it in more detail:

task greeting:describe

The flow currently looks like:

task cluster:create
        |
        v
Kind Kubernetes Cluster
        |
        v
task crd:install
        |
        v
Greeting CRD
        |
        v
task greeting:create
        |
        v
Greeting Resource

When I am finished testing, I can clean everything up using:

task greeting:delete
task crd:delete
task cluster:delete

But What Does Greeting Actually Do?

Right now, nothing.

And this was another important thing for me to understand.

We currently have:

Greeting
    |
    v
Kubernetes API
    |
    v
Stored in Kubernetes

Kubernetes now understands the Greeting resource.

It can:

• validate it
• store it
• retrieve it
• update it
• delete it
• allow something else to watch it

But creating this:

spec:
  message: "Hello from Platform Lab"

does not create anything else.

There is no Pod.

There is no ConfigMap.

There is no Deployment.

A CRD gives us the API.

We still need something that provides the behavior behind that API.

For that, we need a controller.

That is what I want to build next.

The next step will look something like:

Greeting
    |
    v
Greeting Controller
    |
    v
ConfigMap

The controller will watch Greeting resources and create a ConfigMap containing the greeting message.

For example:

apiVersion: platform.shubforge.dev/v1alpha1
kind: Greeting

metadata:
  name: hello

spec:
  message: "Hello from Platform Lab"

could eventually result in something like:

Greeting: hello
       |
       v
Greeting Controller
       |
       v
ConfigMap: hello
       |
       v
message=Hello from Platform Lab

I also want to see what happens if I manually delete that ConfigMap.

If the controller creates it again, that will take me into another important Kubernetes concept: reconciliation.

Source Code

I am keeping the complete setup and code in my Platform Lab GitHub repository.

Repository: Platform Lab

The repository contains the complete setup, including:

• local Kind cluster configuration
• SDKMAN Java configuration
• Taskfile commands
• Kubernetes CRD definitions
• sample resources
• project documentation

The specific changes covered in this post are available in the Greeting CRD pull request:

Pull Request: Add Greeting CRD

I plan to continue using the same repository while I learn and build more around Kubernetes operators and platform engineering.

What's Next?

So far, I have only created the API.

Greeting
    |
    v
Kubernetes API

The next step is to add the behavior.

Greeting
    |
    v
Greeting Controller
    |
    v
ConfigMap

I will build a simple controller that watches Greeting resources and creates a ConfigMap from them.

That should help me understand how Kubernetes controllers actually work and why reconciliation is such an important part of Kubernetes.