DEV Community

shubham goel
shubham goel

Posted on

Adding Status and Conditions to My Kubernetes Operator

In the previous posts, I created a custom Kubernetes resource called Greeting, built a Java controller for it, and then deployed that controller inside Kubernetes.

The current flow looks like this:

Greeting
    |
    v
Java Operator
    |
    v
ConfigMap
Enter fullscreen mode Exit fullscreen mode

For example, I can create:

apiVersion: platform.shubforge.dev/v1alpha1
kind: Greeting

metadata:
  name: hello

spec:
  message: "Hello from Platform Lab"
Enter fullscreen mode Exit fullscreen mode

and the operator creates:

hello-greeting
Enter fullscreen mode Exit fullscreen mode

as a ConfigMap.

This works.

But there is still one problem.

If I run:

kubectl get greeting hello -o yaml
Enter fullscreen mode Exit fullscreen mode

the Greeting resource itself does not tell me much about what the operator has done.

I can check the ConfigMap.

I can check operator logs.

But ideally, the custom resource itself should tell me its current state.

That is where the Kubernetes status subresource comes in.


What I Want to Achieve

I want my Greeting to eventually look something like this:

apiVersion: platform.shubforge.dev/v1alpha1
kind: Greeting

metadata:
  name: hello
  generation: 1

spec:
  message: "Hello from Platform Lab"

status:
  observedGeneration: 1
  configMapName: hello-greeting

  conditions:
    - type: Ready
      status: "True"
      reason: ConfigMapReady
      message: Managed ConfigMap hello-greeting is in the desired state
Enter fullscreen mode Exit fullscreen mode

Now someone looking at the resource can understand:

Greeting requested
       |
       v
Operator processed it
       |
       v
ConfigMap exists
       |
       v
Ready = True
Enter fullscreen mode Exit fullscreen mode

without needing to inspect the operator logs.


Spec vs Status

One thing that became clearer while working on this was the difference between:

spec
Enter fullscreen mode Exit fullscreen mode

and:

status
Enter fullscreen mode Exit fullscreen mode

I think about them like this:

spec
=
What the user wants
Enter fullscreen mode Exit fullscreen mode

while:

status
=
What the operator observed or achieved
Enter fullscreen mode Exit fullscreen mode

For example:

spec:
  message: "Hello from Platform Lab"
Enter fullscreen mode Exit fullscreen mode

is provided by the user.

But:

status:
  configMapName: hello-greeting
Enter fullscreen mode Exit fullscreen mode

is written by the operator.

So conceptually:

User
 |
 v
spec

Operator
 |
 v
status
Enter fullscreen mode Exit fullscreen mode

This separation becomes important once the custom resource starts doing more than storing configuration.


Enabling the Status Subresource

The first change was in the Greeting CRD.

Under the API version, I added:

subresources:
  status: {}
Enter fullscreen mode Exit fullscreen mode

So part of the CRD now looks like:

versions:
  - name: v1alpha1
    served: true
    storage: true

    subresources:
      status: {}
Enter fullscreen mode Exit fullscreen mode

This tells Kubernetes that status should be treated as a separate subresource.

The resource still has:

spec:
Enter fullscreen mode Exit fullscreen mode

for desired configuration.

But the operator can now separately update:

status:
Enter fullscreen mode Exit fullscreen mode

to report the current state.


Defining the Status Schema

I also added the status structure to the CRD schema.

For now, I want to expose:

observedGeneration
configMapName
conditions
Enter fullscreen mode Exit fullscreen mode

So the status schema contains something like:

status:
  type: object

  properties:

    observedGeneration:
      type: integer
      format: int64

    configMapName:
      type: string

    conditions:
      type: array

      x-kubernetes-list-type: map

      x-kubernetes-list-map-keys:
        - type

      items:
        type: object

        properties:

          type:
            type: string

          status:
            type: string
            enum:
              - "True"
              - "False"
              - "Unknown"

          observedGeneration:
            type: integer
            format: int64

          lastTransitionTime:
            type: string
            format: date-time

          reason:
            type: string

          message:
            type: string

        required:
          - type
          - status
Enter fullscreen mode Exit fullscreen mode

This gives the operator a structured place to report its state.


Representing Status in Java

I already had an empty GreetingStatus class.

Previously it looked like:

public class GreetingStatus {
}
Enter fullscreen mode Exit fullscreen mode

Now it contains the fields I want to expose:

public class GreetingStatus {

    private Long observedGeneration;

    private String configMapName;

    private List<Condition> conditions = new ArrayList<>();

    public Long getObservedGeneration() {
        return observedGeneration;
    }

    public void setObservedGeneration(Long observedGeneration) {
        this.observedGeneration = observedGeneration;
    }

    public String getConfigMapName() {
        return configMapName;
    }

    public void setConfigMapName(String configMapName) {
        this.configMapName = configMapName;
    }

    public List<Condition> getConditions() {
        return conditions;
    }

    public void setConditions(List<Condition> conditions) {
        this.conditions = conditions;
    }
}
Enter fullscreen mode Exit fullscreen mode

I am using Kubernetes/Fabric8's Condition model instead of creating my own condition structure.

The status now looks conceptually like:

GreetingStatus
    |
    +-- observedGeneration
    |
    +-- configMapName
    |
    +-- conditions
            |
            +-- Ready
Enter fullscreen mode Exit fullscreen mode

What is observedGeneration?

This was another useful Kubernetes concept to understand.

Kubernetes maintains:

metadata:
  generation:
Enter fullscreen mode Exit fullscreen mode

for resources.

If I change the desired configuration:

spec:
  message: "Hello"
Enter fullscreen mode Exit fullscreen mode

to:

spec:
  message: "Hello Updated"
Enter fullscreen mode Exit fullscreen mode

the resource generation increases.

For example:

metadata:
  generation: 2
Enter fullscreen mode Exit fullscreen mode

The operator can then report:

status:
  observedGeneration: 2
Enter fullscreen mode Exit fullscreen mode

This tells us:

The operator has processed generation 2 of this resource.

So if I see:

generation          = 3
observedGeneration  = 2
Enter fullscreen mode Exit fullscreen mode

I know the resource has changed, but the status still represents the previous generation.

After successful reconciliation:

generation          = 3
observedGeneration  = 3
Enter fullscreen mode Exit fullscreen mode

The flow becomes:

User updates spec
      |
      v
generation increases
      |
      v
Operator reconciles
      |
      v
observedGeneration updated
Enter fullscreen mode Exit fullscreen mode

This becomes much more useful when operators start doing slower or more complicated work.


Adding the Ready Condition

I also wanted the resource to expose a simple health state.

For now, I added one condition:

Ready
Enter fullscreen mode Exit fullscreen mode

When the ConfigMap has been reconciled successfully, I set:

conditions:

  - type: Ready

    status: "True"

    reason: ConfigMapReady

    message: Managed ConfigMap hello-greeting is in the desired state
Enter fullscreen mode Exit fullscreen mode

This gives users a simple answer to:

Is my Greeting ready?
Enter fullscreen mode Exit fullscreen mode

For the current implementation:

Ready=True
Enter fullscreen mode Exit fullscreen mode

means the managed ConfigMap exists in the desired state.

Later, I can also introduce:

Ready=False
Enter fullscreen mode Exit fullscreen mode

with failure reasons.

For example:

type: Ready
status: "False"
reason: ReconciliationFailed
message: Failed to create ConfigMap
Enter fullscreen mode Exit fullscreen mode

But I am keeping failure handling for a later step.


Updating Status from the Reconciler

Previously my reconciler finished with:

return UpdateControl.noUpdate();
Enter fullscreen mode Exit fullscreen mode

The dependent resource was reconciled, but the Greeting itself was not updated.

Now the reconciler also builds a GreetingStatus.

Conceptually:

var status = new GreetingStatus();

status.setObservedGeneration(
        greeting.getMetadata().getGeneration()
);

status.setConfigMapName(
        configMap.getMetadata().getName()
);

status.setConditions(
        List.of(readyCondition)
);

greeting.setStatus(status);
Enter fullscreen mode Exit fullscreen mode

Then instead of:

UpdateControl.noUpdate()
Enter fullscreen mode Exit fullscreen mode

I return:

UpdateControl.patchStatus(greeting)
Enter fullscreen mode Exit fullscreen mode

So the reconciliation flow has become:

Greeting
    |
    v
Reconcile dependent resources
    |
    v
ConfigMap ready
    |
    v
Update Greeting status
Enter fullscreen mode Exit fullscreen mode

The operator is now managing both:

resources
Enter fullscreen mode Exit fullscreen mode

and:

observable state
Enter fullscreen mode Exit fullscreen mode

Reading the Managed ConfigMap

Because the ConfigMap is a managed dependent resource, the reconciler can access it through the reconciliation context.

Conceptually:

var configMap = context
        .getSecondaryResource(ConfigMap.class)
        .orElseThrow();
Enter fullscreen mode Exit fullscreen mode

The controller can then use the actual ConfigMap to populate:

status.configMapName
Enter fullscreen mode Exit fullscreen mode

This is useful because I am not simply guessing what was created.

The status is based on the resource managed by the reconciliation workflow.


lastTransitionTime

Conditions also contain:

lastTransitionTime:
Enter fullscreen mode Exit fullscreen mode

Initially, I thought this could simply be updated every time the controller runs.

But that is not what it means.

lastTransitionTime represents when the condition actually changed state.

For example:

Ready=False
     |
     v
Ready=True
Enter fullscreen mode Exit fullscreen mode

is a transition.

But:

Ready=True
     |
     v
reconciliation runs again
     |
     v
Ready=True
Enter fullscreen mode Exit fullscreen mode

is not a transition.

So if the condition remains:

Ready=True
Enter fullscreen mode Exit fullscreen mode

I keep the existing lastTransitionTime.

That makes the field much more meaningful.


Updating RBAC

Because status is a separate Kubernetes subresource, the operator also needs permission to update it.

The RBAC configuration includes:

- apiGroups:
    - platform.shubforge.dev

  resources:
    - greetings/status

  verbs:
    - get
    - patch
    - update
Enter fullscreen mode Exit fullscreen mode

This is another interesting detail.

Permission to work with:

greetings
Enter fullscreen mode Exit fullscreen mode

does not automatically mean the operator can update:

greetings/status
Enter fullscreen mode Exit fullscreen mode

The status subresource has its own RBAC permissions.


Better kubectl Output

Now that the Greeting has useful status information, I also wanted:

kubectl get greetings
Enter fullscreen mode Exit fullscreen mode

to show more than just:

NAME    AGE
hello   2m
Enter fullscreen mode Exit fullscreen mode

So I added additional printer columns to the CRD.

additionalPrinterColumns:

  - name: Ready
    type: string
    jsonPath: .status.conditions[?(@.type=="Ready")].status

  - name: ConfigMap
    type: string
    jsonPath: .status.configMapName

  - name: Age
    type: date
    jsonPath: .metadata.creationTimestamp
Enter fullscreen mode Exit fullscreen mode

Now:

kubectl get greetings
Enter fullscreen mode Exit fullscreen mode

can show something like:

NAME    READY   CONFIGMAP        AGE
hello   True    hello-greeting   2m
Enter fullscreen mode Exit fullscreen mode

I really like this improvement because the custom resource starts feeling more like a normal Kubernetes API.

Instead of checking:

kubectl logs ...
Enter fullscreen mode Exit fullscreen mode

I can simply run:

kubectl get greetings
Enter fullscreen mode Exit fullscreen mode

and immediately see the state.


Running the Updated Operator

Because the CRD changed, I first reinstall it:

task crd:install
Enter fullscreen mode Exit fullscreen mode

Then rebuild the operator image:

task operator:image:build
Enter fullscreen mode Exit fullscreen mode

Load it into Kind:

task operator:image:load
Enter fullscreen mode Exit fullscreen mode

Deploy it:

task operator:deploy
Enter fullscreen mode Exit fullscreen mode

Since I am currently using the same local development image tag, I restart the operator Deployment:

kubectl rollout restart \
  deployment/greeting-operator \
  -n platform-system
Enter fullscreen mode Exit fullscreen mode

and wait for it:

kubectl rollout status \
  deployment/greeting-operator \
  -n platform-system
Enter fullscreen mode Exit fullscreen mode

Then I create the Greeting:

task greeting:create
Enter fullscreen mode Exit fullscreen mode

Checking the Status

Now I can run:

kubectl get greeting hello -o yaml
Enter fullscreen mode Exit fullscreen mode

and see:

apiVersion: platform.shubforge.dev/v1alpha1
kind: Greeting

metadata:
  generation: 1
  name: hello

spec:
  message: Hello from Platform Lab

status:
  configMapName: hello-greeting

  observedGeneration: 1

  conditions:
    - type: Ready
      status: "True"
      reason: ConfigMapReady
      message: Managed ConfigMap hello-greeting is in the desired state
      observedGeneration: 1
      lastTransitionTime: "..."
Enter fullscreen mode Exit fullscreen mode

Now the resource itself tells me what happened.


Testing a Spec Update

Next, I change:

spec:
  message: "Hello from Platform Lab"
Enter fullscreen mode Exit fullscreen mode

to:

spec:
  message: "Greeting updated"
Enter fullscreen mode Exit fullscreen mode

and apply it again:

task greeting:create
Enter fullscreen mode Exit fullscreen mode

Kubernetes increments:

metadata:
  generation:
Enter fullscreen mode Exit fullscreen mode

The operator reconciles the ConfigMap and then updates:

status:
  observedGeneration:
Enter fullscreen mode Exit fullscreen mode

So I eventually get:

generation          = 2
observedGeneration  = 2
Enter fullscreen mode Exit fullscreen mode

The complete flow becomes:

User changes spec
       |
       v
generation = 2
       |
       v
Operator reconciles
       |
       v
ConfigMap updated
       |
       v
status updated
       |
       v
observedGeneration = 2
Enter fullscreen mode Exit fullscreen mode

Why Status Matters

Before adding status, the operator worked correctly.

But understanding its state required looking elsewhere:

Greeting
    |
    +-- check ConfigMap
    |
    +-- check operator logs
Enter fullscreen mode Exit fullscreen mode

Now the resource itself reports its current state:

Greeting
    |
    +-- spec
    |
    +-- status
          |
          +-- Ready
          +-- ConfigMap
          +-- observedGeneration
Enter fullscreen mode Exit fullscreen mode

This becomes even more important when the controller starts managing multiple resources or talking to external systems.

A custom resource should not only accept desired configuration.

It should also give useful feedback about what is actually happening.


Current Architecture

The project now looks like:

Greeting
    |
    | spec
    v
Kubernetes API
    |
    v
Greeting Operator
    |
    +--------> ConfigMap
    |
    v
Greeting Status
    |
    +-- Ready
    +-- configMapName
    +-- observedGeneration
Enter fullscreen mode Exit fullscreen mode

And the operator itself continues to run inside Kubernetes:

Operator Pod
     |
     v
ServiceAccount
     |
     v
ClusterRoleBinding
     |
     v
ClusterRole
     |
     v
Kubernetes API
Enter fullscreen mode Exit fullscreen mode

Taskfile

I also added a simple command for checking the Greeting status:

task greeting:status
Enter fullscreen mode Exit fullscreen mode

which runs something similar to:

kubectl get greeting hello -o yaml
Enter fullscreen mode Exit fullscreen mode

The full development flow remains:

task cluster:create
task crd:install
task operator:image:build
task operator:image:load
task operator:deploy
task greeting:create
task greeting:status
Enter fullscreen mode Exit fullscreen mode

The Taskfile continues to act as the simple developer interface for the project.


Source Code

The complete implementation is available in my Platform Lab repository.

Repository: Platform Lab

The changes covered in this post are available separately in:

Pull Request: Add Greeting Status and Conditions

The repository contains:

  • Greeting CRD
  • Java Greeting operator
  • managed ConfigMap
  • Docker packaging
  • Kubernetes deployment
  • ServiceAccount and RBAC
  • status subresource
  • Ready condition
  • observed generation
  • kubectl printer columns

What I Learned

The main thing I learned from this step is that reconciliation is not only about making resources exist.

An operator should also communicate what it observed.

The user provides:

spec
Enter fullscreen mode Exit fullscreen mode

The operator works toward that desired state and reports:

status
Enter fullscreen mode Exit fullscreen mode

So the flow is now:

Desired State
     |
     v
spec
     |
     v
Operator
     |
     v
Actual Resources
     |
     v
status
Enter fullscreen mode Exit fullscreen mode

That makes the custom resource much more useful.


What's Next?

Right now, the happy path works:

Ready=True
Enter fullscreen mode Exit fullscreen mode

The next thing I want to explore is what happens when reconciliation fails.

That means looking at things like:

Ready=False
reason
message
retry behavior
Kubernetes Events
Enter fullscreen mode Exit fullscreen mode

I also want to understand how errors should be reported to users without requiring them to inspect operator logs.

After that, I can start looking at:

finalizers
integration tests
GitHub Actions
Enter fullscreen mode Exit fullscreen mode

For now, the Greeting resource can finally tell us both:

what we want
Enter fullscreen mode Exit fullscreen mode

and:

what the operator actually achieved
Enter fullscreen mode Exit fullscreen mode

Top comments (0)