DEV Community

shubham goel
shubham goel

Posted on

Building the First Real API for My Kubernetes Platform

#kubernetes #devops #java #platformengineering

Over the last few posts, I have been building a small Greeting Kubernetes operator.

It started with a simple CRD:

Greeting
    |
    v
Controller
    |
    v
ConfigMap
Enter fullscreen mode Exit fullscreen mode

That small example helped me understand a lot of Kubernetes operator concepts:

Custom Resources
Controllers
Reconciliation
Dependent Resources
Status
Conditions
RBAC
Failures
Retries
Events
Manual Reconciliation
Owner References
Garbage Collection
Enter fullscreen mode Exit fullscreen mode

At this point, I felt that continuing to add more concepts to the Greeting operator would mostly make the example more complicated.

So I decided to keep it as a learning/reference operator and start working on the actual idea behind Platform Lab.

The first real platform API is:

Application
Enter fullscreen mode Exit fullscreen mode

What I Want the Platform to Do

The original idea behind Platform Lab is simple.

As an application developer, I do not want to write all of this every time:

Deployment
Service
ConfigMap
Secret
ServiceAccount
NetworkPolicy
AuthorizationPolicy
Gateway configuration
Observability configuration
Enter fullscreen mode Exit fullscreen mode

Instead, I want the developer-facing API to be much smaller.

Something like:

apiVersion: platform.shubforge.dev/v1alpha1
kind: Application

metadata:
  name: greeting-service

spec:
  image: greeting-service:1.0.0

  replicas: 1

  port:
    containerPort: 8080
Enter fullscreen mode Exit fullscreen mode

Eventually, the platform operator should translate that into:

Application
     |
     v
Platform Operator
     |
     +---- Deployment
     |
     +---- Service
Enter fullscreen mode Exit fullscreen mode

And later, the same platform API can grow into other capabilities.

Application
    |
    +---- Deployment
    +---- Service
    +---- Configuration
    +---- Secrets
    +---- Identity
    +---- Networking
    +---- Authorization
    +---- Observability
Enter fullscreen mode Exit fullscreen mode

But I do not want to build everything at once.

The first step is only defining the API.

Starting With the Application CRD

I created a new Custom Resource Definition:

applications.platform.shubforge.dev
Enter fullscreen mode Exit fullscreen mode

The API is:

Group:   platform.shubforge.dev
Version: v1alpha1
Kind:    Application
Enter fullscreen mode Exit fullscreen mode

The resource is namespace scoped.

For now, the specification is intentionally small:

spec:

  image: greeting-service:1.0.0

  replicas: 1

  port:
    containerPort: 8080
Enter fullscreen mode Exit fullscreen mode

That gives us only three concepts:

image
replicas
container port
Enter fullscreen mode Exit fullscreen mode

I deliberately did not add things like:

CPU
memory
environment variables
Secrets
ConfigMaps
health checks
volumes
routes
Istio
authorization
API dependencies
Enter fullscreen mode Exit fullscreen mode

yet.

Those can be introduced when the platform actually starts supporting them.

The CRD

The CRD currently looks roughly like this:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition

metadata:
  name: applications.platform.shubforge.dev

spec:
  group: platform.shubforge.dev

  scope: Namespaced

  names:
    plural: applications
    singular: application
    kind: Application

    shortNames:
      - papp

  versions:
    - name: v1alpha1
      served: true
      storage: true

      schema:
        openAPIV3Schema:
          type: object

          properties:

            spec:
              type: object

              required:
                - image
                - port

              properties:

                image:
                  type: string
                  minLength: 1

                replicas:
                  type: integer
                  minimum: 1
                  default: 1

                port:
                  type: object

                  required:
                    - containerPort

                  properties:

                    containerPort:
                      type: integer
                      minimum: 1
                      maximum: 65535
Enter fullscreen mode Exit fullscreen mode

The important thing here is that some validation belongs directly in the API.

For example:

replicas >= 1
Enter fullscreen mode Exit fullscreen mode

and:

1 <= containerPort <= 65535
Enter fullscreen mode Exit fullscreen mode

can be validated by Kubernetes before the controller even receives the resource.

Why port Is an Object

I could have used:

spec:
  port: 8080
Enter fullscreen mode Exit fullscreen mode

But I chose:

spec:

  port:
    containerPort: 8080
Enter fullscreen mode Exit fullscreen mode

because it gives the API some room to grow.

Later it could become:

port:
  containerPort: 8080
  protocol: TCP
Enter fullscreen mode Exit fullscreen mode

or eventually:

ports:
  - name: http
    containerPort: 8080
Enter fullscreen mode Exit fullscreen mode

For now, I still want to keep it to one port.

This is v1alpha1, so the API can evolve as I understand the platform better.

Defining Status Early

There is no Application controller yet.

Still, I added the basic status shape I expect the platform operator to eventually populate.

Something like:

status:

  observedGeneration: 1

  deploymentName: greeting-service

  serviceName: greeting-service

  readyReplicas: 1

  conditions:
    - type: Ready
      status: "True"
Enter fullscreen mode Exit fullscreen mode

This gives the API an initial contract between:

desired state
Enter fullscreen mode Exit fullscreen mode

and:

observed state
Enter fullscreen mode Exit fullscreen mode

At the moment, the READY field remains empty because there is no controller.

That is expected.

Creating the First Application

I added a sample resource:

apiVersion: platform.shubforge.dev/v1alpha1
kind: Application

metadata:
  name: greeting-service

spec:
  image: greeting-service:1.0.0

  replicas: 1

  port:
    containerPort: 8080
Enter fullscreen mode Exit fullscreen mode

Then installed the CRD:

kubectl apply -f k8s/crds/applications.yaml
Enter fullscreen mode Exit fullscreen mode

and created the Application:

kubectl apply -f k8s/samples/application.yaml
Enter fullscreen mode Exit fullscreen mode

Now Kubernetes understands:

Application/greeting-service
Enter fullscreen mode Exit fullscreen mode

I can check it using:

kubectl get applications
Enter fullscreen mode Exit fullscreen mode

or the short name:

kubectl get papp
Enter fullscreen mode Exit fullscreen mode

The output includes information such as:

NAME               READY   IMAGE                    REPLICAS
greeting-service           greeting-service:1.0.0   1
Enter fullscreen mode Exit fullscreen mode

READY is still empty because the platform operator does not exist yet.

That will be the next step.

Introducing Tests From the Beginning

One thing I want to do differently from the Greeting operator is testing.

I built several Greeting features first and only thought about automated tests later.

For the actual platform APIs, I want the flow to be:

feature
   +
test
   +
documentation
Enter fullscreen mode Exit fullscreen mode

from the beginning.

There is no Java controller yet, so introducing JUnit, Awaitility, or operator test frameworks would be premature.

But the API itself can already be tested.

Testing the Application API

I added:

scripts/tests/application-api-test.sh
Enter fullscreen mode Exit fullscreen mode

The test talks to the real Kubernetes API server.

It first installs the CRD and then uses:

kubectl apply --dry-run=server
Enter fullscreen mode Exit fullscreen mode

to validate different Application manifests.

The important part is:

--dry-run=server
Enter fullscreen mode Exit fullscreen mode

The resource is sent to the Kubernetes API server and validated there, but it is not persisted.

So the test is not implementing Kubernetes validation itself.

It is asking Kubernetes:

Would you accept this Application?

Testing a Valid Application

The first test checks the normal sample:

spec:
  image: greeting-service:1.0.0

  replicas: 1

  port:
    containerPort: 8080
Enter fullscreen mode Exit fullscreen mode

Kubernetes should accept it.

Conceptually:

valid Application
        |
        v
Kubernetes API
        |
        v
accepted
Enter fullscreen mode Exit fullscreen mode

Testing an Invalid Port

Next I test:

port:
  containerPort: 70000
Enter fullscreen mode Exit fullscreen mode

But the CRD defines:

maximum = 65535
Enter fullscreen mode Exit fullscreen mode

So Kubernetes should reject it.

containerPort = 70000
        |
        v
CRD validation
        |
        v
rejected
Enter fullscreen mode Exit fullscreen mode

Testing a Missing Image

The Application API requires:

image:
Enter fullscreen mode Exit fullscreen mode

So this resource should fail:

spec:

  replicas: 1

  port:
    containerPort: 8080
Enter fullscreen mode Exit fullscreen mode

Again, Kubernetes handles the validation.

Testing Invalid Replicas

The API currently defines:

replicas >= 1
Enter fullscreen mode Exit fullscreen mode

So this:

replicas: 0
Enter fullscreen mode Exit fullscreen mode

should be rejected.

The test now verifies several parts of the Application contract:

valid resource
        → accepted

invalid port
        → rejected

missing image
        → rejected

replicas = 0
        → rejected
Enter fullscreen mode Exit fullscreen mode

Running the API Tests

I added a Taskfile command:

task application:test:api
Enter fullscreen mode Exit fullscreen mode

which runs:

scripts/tests/application-api-test.sh
Enter fullscreen mode Exit fullscreen mode

The test output is roughly:

Testing Application CRD...

1. Installing Application CRD
✓ Application CRD installed

2. Testing valid Application
✓ Valid Application accepted

3. Testing invalid container port
✓ Invalid port rejected

4. Testing missing image
✓ Missing image rejected

5. Testing replicas below minimum
✓ Invalid replica count rejected

Application API tests passed.
Enter fullscreen mode Exit fullscreen mode

This is simple, but it gives me the first automated test for the actual Platform Lab APIs.

Why I Am Not Using JUnit Yet

There is currently no Java code involved in the Application API.

So using something like:

JUnit
Awaitility
Java Operator SDK test utilities
Testcontainers
Enter fullscreen mode Exit fullscreen mode

would not add much value yet.

Right now the thing I want to test is:

Does Kubernetes correctly understand and validate my platform API?
Enter fullscreen mode Exit fullscreen mode

The real Kubernetes API server is the best place to answer that.

Once I build:

ApplicationReconciler
Enter fullscreen mode Exit fullscreen mode

the test requirements will change.

Then I will want to verify things like:

Application created
        |
        v
Deployment created

Application created
        |
        v
Service created
Enter fullscreen mode Exit fullscreen mode

That is where Java integration tests will start making sense.

Taskfile Commands

I also added Application-related commands to the Taskfile.

Install the CRD:

task application:crd:install
Enter fullscreen mode Exit fullscreen mode

Create the sample Application:

task application:create
Enter fullscreen mode Exit fullscreen mode

List Applications:

task application:get
Enter fullscreen mode Exit fullscreen mode

Describe the sample:

task application:describe
Enter fullscreen mode Exit fullscreen mode

Delete it:

task application:delete
Enter fullscreen mode Exit fullscreen mode

Run API tests:

task application:test:api
Enter fullscreen mode Exit fullscreen mode

The Taskfile continues to be the main developer interface for Platform Lab.

Where the Platform Is Now

The repository has moved from:

Greeting
    |
    v
learning Kubernetes operators
Enter fullscreen mode Exit fullscreen mode

to:

Application
    |
    v
building the actual platform
Enter fullscreen mode Exit fullscreen mode

Right now:

Application CRD
      |
      v
Kubernetes API
Enter fullscreen mode Exit fullscreen mode

is all that exists.

The next step is:

Application
      |
      v
ApplicationReconciler
      |
      +---- Deployment
      |
      +---- Service
Enter fullscreen mode Exit fullscreen mode

That will be the first actual platform behavior.

Source Code

The complete project is available in my Platform Lab repository.

Repository: Platform Lab

The changes from this post are available in:

Pull Request: Add Platform Application API

The main additions are:

Application CRD
Application sample
CRD validation
Application Taskfile commands
Application API tests
Enter fullscreen mode Exit fullscreen mode

What I Learned

One thing I am trying to change as I move into the actual platform work is when I introduce complexity.

With the Greeting operator, I explored many Kubernetes concepts because the purpose was learning.

For the platform itself, I want features to have a reason to exist.

For example, I am not adding:

finalizers
advanced reconciliation
external API handling
Secrets
Istio configuration
Enter fullscreen mode Exit fullscreen mode

until a real platform feature needs them.

The same applies to testing.

For this API, a small server-side Kubernetes validation test is enough.

When the controller arrives, the testing strategy can grow with it.

What's Next?

The next step is much more interesting.

I want this:

apiVersion: platform.shubforge.dev/v1alpha1
kind: Application

metadata:
  name: greeting-service

spec:
  image: greeting-service:1.0.0

  replicas: 1

  port:
    containerPort: 8080
Enter fullscreen mode Exit fullscreen mode

to produce:

Application/greeting-service
            |
            v
      Platform Operator
        /          \
       v            v
Deployment       Service
Enter fullscreen mode Exit fullscreen mode

That means the next step is building the actual:

platform-operator
Enter fullscreen mode Exit fullscreen mode

and its first controller:

ApplicationReconciler
Enter fullscreen mode Exit fullscreen mode

This time, the controller and its integration tests will be built together.

Top comments (0)