DEV Community: shubham goel

From Provisioned to Ready: Tracking Application Health in My Kubernetes Platform

shubham goel — Sat, 03 Oct 2026 15:33:21 +0000

In the previous posts, I built the first real platform flow:

Application
     |
     v
Platform Operator
     |
     +------ Deployment
     |
     +------ Service

The platform could also handle updates.

If I changed:

spec:
  image: greeting-service:2.0.0
  replicas: 3

  port:
    containerPort: 9090

the existing Deployment and Service were reconciled to the new desired state.

That worked well.

But there was still one important problem.

The platform could tell me:

I created your Deployment.

It could not tell me:

Your application is actually running.

Those are not the same thing.

That is what I wanted to solve next.

Provisioned Does Not Mean Ready

Suppose the Platform Operator successfully creates:

Deployment/greeting-service

and:

Service/greeting-service

From the controller's perspective, provisioning succeeded.

But the Pods might still be:

Pending
ImagePullBackOff
CrashLoopBackOff

So this:

Deployment exists

does not mean:

Application Ready=True

That was the main difference compared with my earlier Greeting operator.

For Greeting, the managed resource was a ConfigMap.

Greeting
    |
    v
ConfigMap

If the ConfigMap was successfully reconciled, that was basically enough to consider the Greeting ready.

For an Application:

Application
    |
    v
Deployment
    |
    v
Pods

there is another runtime layer.

Where Should Readiness Come From?

The Platform Operator does not need to manage Pods directly.

Kubernetes already has the Deployment controller for that.

The responsibilities are:

Platform Operator
=
Application → Deployment + Service

while:

Kubernetes Deployment Controller
=
Deployment → ReplicaSets + Pods

So instead of watching Pods directly, I decided to use:

Deployment.status

as the first source of Application readiness.

The flow becomes:

Application
      |
      v
Platform Operator
      |
      v
Deployment
      |
      v
Kubernetes Deployment Controller
      |
      v
Pods
      |
      v
Deployment.status
      |
      v
Platform Operator
      |
      v
Application.status

What Deployment Status Gives Me

For the first readiness implementation, I use values such as:

spec.replicas
status.readyReplicas
status.availableReplicas
status.updatedReplicas
status.observedGeneration

Suppose the Application wants:

3 replicas

and the Deployment reports:

readyReplicas     = 3
availableReplicas = 3
updatedReplicas   = 3

and the latest Deployment generation has been observed.

Then the Application can be considered ready.

Adding Readiness to Application Status

The Application status now includes:

status:
  observedGeneration: 2

  deploymentName: greeting-service
  serviceName: greeting-service

  readyReplicas: 3

  conditions:
    - type: Ready
      status: "True"
      observedGeneration: 2
      reason: DeploymentReady
      message: Deployment has 3/3 ready replicas
      lastTransitionTime: "2026-10-03T14:20:00Z"

The new parts are:

readyReplicas
Ready condition

So the Application API is starting to expose workload state instead of only provisioning information.

Ready=False

Suppose the Deployment wants:

3 replicas

but only one is currently ready.

Then Application status may look like:

status:
  readyReplicas: 1

  conditions:
    - type: Ready
      status: "False"
      reason: DeploymentProgressing
      message: Deployment has 1/3 ready replicas

The platform is saying:

The Kubernetes resources exist, but the workload is not fully available yet.

Ready=True

Once all replicas become available:

desired replicas   = 3
updated replicas   = 3
ready replicas     = 3
available replicas = 3

the Application can report:

conditions:
  - type: Ready
    status: "True"
    reason: DeploymentReady
    message: Deployment has 3/3 ready replicas

So the lifecycle becomes:

Application created
        |
        v
Deployment created
        |
        v
Pods starting
        |
        v
Ready=False
        |
        v
Pods become available
        |
        v
Ready=True

Separating Readiness Logic

I did not want to put all the readiness calculations directly inside:

ApplicationReconciler

So I created:

ApplicationReadinessEvaluator.java

and a small result model:

ApplicationReadiness

The evaluator returns:

readyReplicas
ready
reason
message

Conceptually:

Deployment
     |
     v
ApplicationReadinessEvaluator
     |
     v
ApplicationReadiness
     |
     v
Application.status

This keeps the controller focused on reconciliation while the evaluator contains the actual platform rule.

The Basic Readiness Rule

For now, the rule is intentionally simple.

The Application is ready when:

latest Deployment generation observed

AND

updatedReplicas == desiredReplicas

AND

readyReplicas == desiredReplicas

AND

availableReplicas == desiredReplicas

Then:

Ready=True

Otherwise:

Ready=False

I expect this rule to evolve later, but it is enough for the first version.

Why Deployment Generation Matters

There are actually two different generations involved now.

The Application has:

Application.metadata.generation

and the Deployment has:

Deployment.metadata.generation

The Platform Operator tracks the Application generation using:

Application.status.observedGeneration

But for Deployment readiness, I also need to check:

Deployment.status.observedGeneration

For example:

Deployment generation         = 3
Deployment observedGeneration = 2

means the Kubernetes Deployment controller has not processed the latest Deployment specification yet.

Even if some replica values look okay, I should not assume that the latest desired Deployment state is ready.

So the Application stays:

Ready=False

until the Deployment controller catches up.

Continuous Readiness

One thing I wanted to understand was whether readiness should only be calculated during deployment.

It should not.

Application health can change later.

Suppose everything is healthy:

Ready=True
3/3 replicas ready

Then one Pod becomes unhealthy.

The flow is:

Pod becomes unhealthy
        |
        v
Deployment Controller notices
        |
        v
Deployment.status changes
        |
        v
Platform Operator receives update
        |
        v
Application reconciled again
        |
        v
readyReplicas decreases
        |
        v
Ready=False

Then Kubernetes tries to recover the Deployment.

Once the replacement Pod becomes ready:

new Pod ready
      |
      v
Deployment.status changes
      |
      v
Application reconciled
      |
      v
Ready=True

So Application status can move like:

Ready=True
3/3
    |
    v
Ready=False
2/3
    |
    v
Ready=True
3/3

That is much more useful than setting Ready=True once and never checking again.

The Platform Operator Does Not Heal Pods

This was another important distinction.

If a Pod goes down, the Platform Operator should not do this:

Pod failed
    |
    v
Platform Operator creates another Pod

That is already Kubernetes responsibility.

Instead:

Pod failed
      |
      v
Deployment Controller
      |
      v
replacement Pod

The Platform Operator only observes:

Deployment.status

and reflects that into:

Application.status

So each controller has a clear responsibility.

Why I Am Not Watching Pods Yet

I could watch individual Pods and inspect things like:

CrashLoopBackOff
ImagePullBackOff
Pending

But I decided not to add that yet.

For basic readiness, Deployment status already tells me enough:

how many replicas are desired
how many are updated
how many are ready
how many are available

So the current flow stays:

Application
     |
     v
Deployment
     |
     v
Deployment.status
     |
     v
Application.status

Later, Pod-level observation can be added if the platform needs better failure explanations.

Ready Condition

The Application currently uses one condition:

Ready

Some reasons are:

DeploymentPending
DeploymentProgressing
DeploymentReady

For example:

conditions:
  - type: Ready
    status: "False"
    reason: DeploymentProgressing
    message: Deployment has 1/3 ready replicas

and later:

conditions:
  - type: Ready
    status: "True"
    reason: DeploymentReady
    message: Deployment has 3/3 ready replicas

lastTransitionTime

The condition also contains:

lastTransitionTime

I wanted this to represent an actual condition transition.

For example:

Ready=False
0/3

then:

Ready=False
1/3

then:

Ready=False
2/3

does not mean the Ready condition changed.

It is still:

False

So I preserve the existing lastTransitionTime.

But:

Ready=False
      |
      v
Ready=True

is a real transition.

That gets a new timestamp.

kubectl Output

I also added a ReadyReplicas printer column to the Application CRD.

Now:

kubectl get papp

can show something like:

NAME               READY   READYREPLICAS   IMAGE                    REPLICAS
greeting-service   True    3               greeting-service:2.0.0   3

If one replica becomes unavailable:

NAME               READY   READYREPLICAS   IMAGE                    REPLICAS
greeting-service   False   2               greeting-service:2.0.0   3

That is closer to the developer experience I want.

A developer should not always need to run:

kubectl get deployment
kubectl get pods

just to understand the basic state of their Application.

Adding Java Unit Tests

This feature also gave me a good reason to introduce Java unit tests into the real Platform Operator.

Until now, most tests were using the Kubernetes API directly.

But readiness evaluation is platform logic.

So I added tests for:

ApplicationReadinessEvaluator

For example:

desired = 3
ready = 3
available = 3
updated = 3
latest generation observed

→ Ready=True

Another test covers:

desired = 3
ready = 1

→ Ready=False

And another checks:

Deployment generation = 3
observedGeneration = 2

→ Ready=False

Now the test layers look like:

API tests
      |
      v
CRD validation


Java unit tests
      |
      v
platform logic


controller integration tests
      |
      v
real Kubernetes reconciliation

This feels much more natural than introducing all the testing infrastructure before there was real platform logic to test.

Running the Tests

The API tests are still:

task application:test:api

The Java tests are:

task platform-operator:test

And the controller integration tests are:

task application:test:controller

So the testing strategy is growing together with the platform.

Current Platform Architecture

At this point the architecture looks like:

                           Application
                                |
                                v
                      ApplicationReconciler
                                |
                 +--------------+--------------+
                 |                             |
                 v                             v
            Deployment                      Service
                 |
                 v
       Kubernetes Deployment
             Controller
                 |
                 v
                Pods
                 |
                 v
         Deployment.status
                 |
                 v
         Platform Operator
                 |
                 v
         Application.status
                 |
        +--------+--------+
        |                 |
        v                 v
 readyReplicas       Ready Condition

This is the first time the platform is not only creating infrastructure but also reporting workload state.

What This Still Does Not Tell Me

The current Application status can say:

Ready=False
0/3 replicas ready

but it cannot yet tell me exactly why.

For example, it does not currently distinguish between:

ImagePullBackOff

CrashLoopBackOff

failed readiness probe

Pod unschedulable

That would require deeper workload inspection.

For now, I want to keep the first readiness model simple.

Source Code

The complete project is available in my Platform Lab repository.

Repository: Platform Lab

The changes covered in this post are available in:

Pull Request: Add Application Readiness

The main changes include:

readyReplicas in Application status

Ready condition

ApplicationReadinessEvaluator

Deployment status observation

continuous readiness updates

Java unit tests

ReadyReplicas kubectl printer column

What I Learned

The biggest lesson from this step was the difference between:

resource provisioning

and:

workload readiness

Creating a Deployment successfully only tells me that Kubernetes accepted the desired workload configuration.

It does not tell me that the workload is actually running.

The Deployment controller already knows how to manage Pods and recover failed replicas.

So instead of duplicating that responsibility, the Platform Operator can observe Deployment status and expose a simpler platform-level view.

That gives me:

Application
=
developer desired state
+
platform workload status

which is much closer to what I want the Platform API to eventually become.

What's Next?

Right now my sample Application still uses placeholder images.

So the next useful step is to build a small real application image and run it through Platform Lab.

That will let me observe a real lifecycle:

Application created
        |
        v
Ready=False
        |
        v
Pod starts
        |
        v
Deployment becomes available
        |
        v
Application Ready=True

Then I also want to deliberately make the workload unhealthy and observe:

Ready=True
      |
      v
Ready=False
      |
      v
Kubernetes recovery
      |
      v
Ready=True

That should give me a real end-to-end test of the readiness model before moving into configuration and Secrets.

Making My Platform API Reconcile Application Updates

shubham goel — Sat, 03 Oct 2026 13:05:45 +0000

In the previous post, I built the first real behavior for Platform Lab.

A developer could create:

apiVersion: platform.shubforge.dev/v1alpha1
kind: Application

metadata:
  name: greeting-service

spec:
  image: greeting-service:1.0.0
  replicas: 1

  port:
    containerPort: 8080

and the Platform Operator would create:

Application
     |
     v
Platform Operator
     |
     +------ Deployment
     |
     +------ Service

That was the first point where the Application API actually resulted in real Kubernetes resources.

But there was another important question.

What happens when the Application changes?

Creating resources once is useful, but a Kubernetes API should represent desired state.

If I change:

image: greeting-service:1.0.0

to:

image: greeting-service:2.0.0

I expect the platform to update the existing Deployment.

The same should happen for replicas and ports.

That is what I explored next.

Application Is Desired State

Initially, my Application looked like:

spec:
  image: greeting-service:1.0.0
  replicas: 1

  port:
    containerPort: 8080

The Platform Operator created:

Deployment
  image = 1.0.0
  replicas = 1
  port = 8080

Service
  port = 8080
  targetPort = 8080

Now I changed the Application to:

spec:
  image: greeting-service:2.0.0
  replicas: 3

  port:
    containerPort: 9090

What I wanted was:

same Deployment
    |
    +---- image = 2.0.0
    +---- replicas = 3
    +---- port = 9090

same Service
    |
    +---- port = 9090
    +---- targetPort = 9090

The important word here is:

same

I did not want the platform to delete everything and recreate it.

Updating the Application

For the experiment, I used:

kubectl patch application greeting-service \
  --type=merge \
  -p '{
    "spec": {
      "image": "greeting-service:2.0.0",
      "replicas": 3,
      "port": {
        "containerPort": 9090
      }
    }
  }'

I also added a Taskfile helper:

task application:update

so I can repeat the same experiment more easily.

After the update, Kubernetes increments:

metadata.generation

For example:

generation = 1

becomes:

generation = 2

That tells the controller that the desired Application specification changed.

The Interesting Part: Almost No New Controller Logic

I initially expected that I might need code like:

if (imageChanged) {
    updateDeployment();
}

if (replicasChanged) {
    updateDeployment();
}

if (portChanged) {
    updateDeployment();
    updateService();
}

But that is not how I had structured the controller.

The Deployment dependent resource already builds its desired state from:

application.getSpec()

The Service dependent resource does the same.

So when the Application changes, the flow is simply:

Application spec changes
        |
        v
controller receives update
        |
        v
desired Deployment recalculated
        |
        v
desired Service recalculated
        |
        v
actual resources reconciled

That means creation and updates use the same model.

Creation and Update Are the Same Reconciliation Problem

For creation:

desired Deployment exists
actual Deployment does not exist
        |
        v
create Deployment

For updates:

desired Deployment changed
actual Deployment exists
but differs
        |
        v
update Deployment

The same applies to the Service.

This is one of the things I like about the managed dependent resource approach.

Instead of thinking in terms of:

CREATE
UPDATE
DELETE

I can think in terms of:

What should the resource look like now?

and let reconciliation handle the difference.

Verifying the Deployment Update

After changing the Application, I checked:

kubectl get deployment greeting-service \
  -o jsonpath='{.spec.replicas}'

and got:

Then:

kubectl get deployment greeting-service \
  -o jsonpath='{.spec.template.spec.containers[0].image}'

returned:

greeting-service:2.0.0

And:

kubectl get deployment greeting-service \
  -o jsonpath='{.spec.template.spec.containers[0].ports[0].containerPort}'

returned:

So the Deployment had moved to the new desired state.

Verifying the Service Update

The port change also affected the Service.

I checked:

kubectl get service greeting-service \
  -o jsonpath='{.spec.ports[0].port}'

and got:

Then:

kubectl get service greeting-service \
  -o jsonpath='{.spec.ports[0].targetPort}'

also returned:

So one higher-level platform change:

port:
  containerPort: 9090

updated both Kubernetes resources that depend on that value.

Application.port
      |
      +------ Deployment containerPort
      |
      +------ Service port
      |
      +------ Service targetPort

That is exactly the kind of translation I want the platform to handle.

Were the Resources Recreated?

I also wanted to make sure the operator was updating the existing resources rather than deleting and recreating them.

Before the Application update, I captured the Deployment UID:

kubectl get deployment greeting-service \
  -o jsonpath='{.metadata.uid}'

and the Service UID:

kubectl get service greeting-service \
  -o jsonpath='{.metadata.uid}'

Then I updated the Application and checked them again.

The UIDs stayed the same.

Conceptually:

Before

Deployment UID = A
Service UID    = B


After

Deployment UID = A
Service UID    = B

So the resources were updated in place.

Checking the Service Identity Too

I also checked:

kubectl get service greeting-service \
  -o jsonpath='{.spec.clusterIP}'

before and after the update.

The ClusterIP remained unchanged.

That gave me another useful signal that the Service was being reconciled rather than replaced.

The behavior I wanted was:

Application update
        |
        v
existing Service updated

not:

delete Service
      |
      v
create another Service

observedGeneration

The Application status already contains:

status:
  observedGeneration:

This becomes more useful once updates exist.

After changing the Application:

metadata.generation = 2

the controller eventually reports:

status.observedGeneration = 2

So:

metadata.generation
=
latest desired state

while:

status.observedGeneration
=
latest state processed by the controller

When they match:

generation          = 2
observedGeneration  = 2

I know the controller has processed the current Application specification.

Extending the Integration Test

I already had a black-box controller test that checked the initial provisioning flow:

create Application
      |
      v
Deployment created
      |
      v
Service created
      |
      v
status updated

For this feature, I extended the same test.

The new flow is:

create Application
        |
        v
verify Deployment + Service
        |
        v
capture resource UIDs
        |
        v
update Application
        |
        v
wait for reconciliation
        |
        v
verify Deployment changes
        |
        v
verify Service changes
        |
        v
verify UIDs unchanged
        |
        v
verify observedGeneration

This made the test much more representative of an actual controller lifecycle.

What the Test Verifies

The test starts with:

spec:
  image: example/test:1.0.0
  replicas: 2

  port:
    containerPort: 8080

and later updates it to:

spec:
  image: example/test:2.0.0
  replicas: 3

  port:
    containerPort: 9090

Then it verifies:

Deployment replicas = 3
Deployment image = example/test:2.0.0
Deployment port = 9090

Service port = 9090
Service targetPort = 9090

It also verifies that:

Deployment UID did not change
Service UID did not change
Service ClusterIP did not change

Finally, it waits until:

metadata.generation
=
status.observedGeneration

Why the Test Waits

One important thing with controller tests is that reconciliation is asynchronous.

After running:

kubectl patch ...

I cannot immediately assume the Deployment has already changed.

The real flow is:

Application updated
        |
        v
Kubernetes stores update
        |
        v
watch event
        |
        v
controller receives event
        |
        v
reconciliation happens
        |
        v
Deployment / Service updated

So the test waits until the actual resources match the expected state.

This is much better than assuming everything happens immediately.

The Current Platform Lifecycle

The platform now supports:

Application created
        |
        v
Deployment + Service created

Application updated
        |
        v
same Deployment + Service updated

So Application is starting to behave like a proper desired-state API.

The controller is not performing one-time provisioning.

It keeps asking:

Does the current Kubernetes state match the Application?

Desired State vs Provisioning Request

This distinction became clearer during this step.

A provisioning API might behave like:

POST /applications
        |
        v
create resources
        |
        v
done

But the Kubernetes controller model is different:

Application exists
        |
        v
continuously describe desired state
        |
        v
reconcile actual state toward it

That means updates naturally become part of the same model.

There is no separate:

update application workflow

inside the controller.

There is only:

reconciliation

Current Architecture

The current platform now looks like:

                        Developer
                            |
                            v
                       Application
                            |
                            v
                    Platform Operator
                            |
              +-------------+-------------+
              |                           |
              v                           v
         Deployment                    Service
              |                           |
              v                           |
             Pods <-----------------------+

When the Application changes:

Application v1
      |
      v
Deployment + Service

      |
      | spec changes
      v

Application v2
      |
      v
same Deployment + Service
reconciled to new state

What This Still Does Not Mean

Even though the Deployment has been updated successfully, I am still not setting:

Ready=True

on the Application.

That is because:

Deployment spec updated

does not mean:

application rollout completed

For example:

Application updated
      |
      v
Deployment updated
      |
      v
new ReplicaSet created
      |
      v
Pods starting

Those Pods could still end up in:

ImagePullBackOff
CrashLoopBackOff
Pending

So there is another layer we still need to understand:

provisioned

vs:

ready

That will be the next feature.

Source Code

The full implementation is available in my Platform Lab repository.

Repository: Platform Lab

The changes covered in this post are available in:

Pull Request: Reconcile Application Updates

The main changes are:

Application update Taskfile command
extended controller integration test
in-place resource update verification
observedGeneration verification

Interestingly, there was very little controller implementation change.

Most of this feature was about proving that the existing reconciliation model worked the way I expected.

What I Learned

The main thing I learned from this step is that a controller should not think only in terms of operations.

Instead of:

create
update
delete

the more useful question is:

What should the resource look like right now?

The Application spec defines that desired state.

The dependent resources calculate their desired Kubernetes representation.

The operator then reconciles the actual resources toward that state.

Because of that, adding Application updates did not require building a completely separate update path.

The same reconciliation logic handled both creation and modification.

That made the platform design feel much more aligned with Kubernetes.

What's Next?

The Platform Operator can now create and update resources.

But it still cannot answer:

Is my application actually ready?

That is the next problem I want to work on.

The Deployment exposes information such as:

desired replicas
ready replicas
available replicas
updated replicas

The platform can use that information to start building meaningful Application status.

Eventually I want something like:

status:
  observedGeneration: 2
  readyReplicas: 3

  conditions:
    - type: Ready
      status: "True"
      reason: DeploymentReady

But first I want to understand what Ready should actually mean.

So the next step is:

Application
      |
      v
Deployment
      |
      v
Deployment status
      |
      v
Application status

That will move the platform from:

I created your Kubernetes resources

toward:

I know whether your application is actually running.

Turning My Platform API Into Real Kubernetes Resources

shubham goel — Sat, 03 Oct 2026 09:01:19 +0000

In the previous post, I introduced the first real API for Platform Lab:

Application

The idea was to let an application developer define something simple like:

apiVersion: platform.shubforge.dev/v1alpha1
kind: Application

metadata:
  name: greeting-service

spec:
  image: greeting-service:1.0.0

  replicas: 1

  port:
    containerPort: 8080

At that point, Kubernetes could understand and validate the resource.

But nothing actually happened after creating it.

There was no Deployment.

There was no Service.

There was no platform behavior yet.

So the next step was to build the real Platform Operator.

What I Wanted to Build

The first platform flow is intentionally small.

Application
     |
     v
Platform Operator
     |
     +------ Deployment
     |
     +------ Service

The application developer should only define:

image
replicas
container port

The platform should handle the Kubernetes-specific resources.

For now, I deliberately avoided adding things like:

ConfigMaps
Secrets
health checks
resource requests
Istio
authorization
routes
observability

Those can come later when the platform actually needs them.

Creating the Platform Operator

Until now, I had a separate:

greeting-operator

which I used mainly to learn Kubernetes controller concepts.

For the actual platform work, I created a new module:

operators/platform-operator/

The idea is that this becomes the main operator for Platform Lab.

Eventually it can contain controllers for resources such as:

Application
ApplicationRelease
ApiDependency
AccessGrant

So I did not call it:

application-operator

because Application is only the first platform API.

Mapping the Application CRD to Java

The custom resource is represented in Java using:

Application.java
ApplicationSpec.java
ApplicationPort.java
ApplicationStatus.java

The main resource looks roughly like:

@Group("platform.shubforge.dev")
@Version("v1alpha1")
@Kind("Application")
@Plural("applications")
public class Application
        extends CustomResource<ApplicationSpec, ApplicationStatus>
        implements Namespaced {
}

This maps directly to:

apiVersion: platform.shubforge.dev/v1alpha1
kind: Application

The specification contains:

image
replicas
port

which matches:

spec:
  image: greeting-service:1.0.0

  replicas: 1

  port:
    containerPort: 8080

The First Real Translation

This is where the platform starts becoming useful.

The user writes:

spec:
  image: greeting-service:1.0.0
  replicas: 1

  port:
    containerPort: 8080

The Platform Operator translates that into:

Deployment
+
Service

So instead of thinking only in Kubernetes objects:

Deployment YAML
Service YAML
selectors
pod template
container ports
target ports

the developer works with a higher-level platform API.

Creating the Deployment

The Deployment is implemented as a managed dependent resource:

ApplicationDeploymentDependentResource.java

The operator takes the Application specification and builds a desired Deployment.

Conceptually:

Application
    |
    v
image
replicas
port
    |
    v
Deployment

The generated Deployment is roughly equivalent to:

apiVersion: apps/v1
kind: Deployment

metadata:
  name: greeting-service

spec:
  replicas: 1

  selector:
    matchLabels:
      platform.shubforge.dev/application: greeting-service

  template:

    metadata:
      labels:
        platform.shubforge.dev/application: greeting-service

    spec:

      containers:
        - name: application
          image: greeting-service:1.0.0

          ports:
            - containerPort: 8080

The application developer does not need to define any of this directly.

Creating the Service

The Service is another managed dependent resource:

ApplicationServiceDependentResource.java

The generated Service uses the same application label:

platform.shubforge.dev/application: greeting-service

The flow becomes:

Application
      |
      v
Deployment
      |
      v
Pods
      ^
      |
Service selector

The generated Service is roughly:

apiVersion: v1
kind: Service

metadata:
  name: greeting-service

spec:

  selector:
    platform.shubforge.dev/application: greeting-service

  ports:
    - name: http
      port: 8080
      targetPort: 8080
      protocol: TCP

This is the first place where the platform is hiding real Kubernetes wiring from the developer.

Why I Used Managed Dependent Resources

I could have written the controller manually like:

if Deployment does not exist
    create it

if Deployment exists
    update it

if Service does not exist
    create it

if Service exists
    update it

But that quickly becomes a lot of imperative logic.

Instead, I describe:

desired Deployment

and:

desired Service

and let the operator framework reconcile the actual Kubernetes resources toward that desired state.

So the mental model becomes:

Application spec
      |
      v
desired Deployment
      |
      v
actual Deployment

and:

Application spec
      |
      v
desired Service
      |
      v
actual Service

That feels much closer to the Kubernetes reconciliation model.

The Application Reconciler

The main reconciler is:

ApplicationReconciler.java

Its workflow includes both dependent resources:

Application
     |
     v
ApplicationReconciler
     |
     +----------------+
     |                |
     v                v
Deployment         Service

The dependent resources handle provisioning.

The main reconciler currently focuses mainly on status.

Initial Application Status

For the first version, I kept status simple.

The controller writes:

status:
  observedGeneration: 1
  deploymentName: greeting-service
  serviceName: greeting-service

This tells me:

which Application generation was processed

which Deployment belongs to it

which Service belongs to it

I intentionally did not set:

Ready=True

yet.

Why Ready Is Not Implemented Yet

Creating a Deployment does not mean the application is actually healthy.

For example, the Deployment may exist while Pods are failing with:

ImagePullBackOff
CrashLoopBackOff
failed readiness probe
scheduling failure

So:

resource created

is not the same thing as:

application ready

I want to implement readiness separately by actually observing Deployment status.

For this version, the operator only reports what it has provisioned.

Running the Platform Operator in Kubernetes

The operator itself is packaged as:

platform-operator:dev

and runs inside the local Kind cluster.

The Kubernetes deployment includes:

Namespace
ServiceAccount
ClusterRole
ClusterRoleBinding
Deployment

The operator runs as:

platform-system/platform-operator

RBAC

The first version only needs permissions for:

Applications
Application status
Deployments
Services

I did not copy every permission from the old Greeting operator.

The rule I want to follow is:

Add permissions only when the platform actually needs them.

So RBAC should grow with platform capabilities.

Creating the First Application

Once the operator was deployed, I created:

apiVersion: platform.shubforge.dev/v1alpha1
kind: Application

metadata:
  name: greeting-service

spec:
  image: greeting-service:1.0.0

  replicas: 1

  port:
    containerPort: 8080

Then checked:

kubectl get deployment greeting-service

and:

kubectl get service greeting-service

The platform had created both resources.

So the complete flow was finally working:

Application/greeting-service
            |
            v
      Platform Operator
        /          \
       v            v
Deployment       Service

This was the first point where Platform Lab started feeling like an actual platform rather than only an operator-learning project.

Testing the Controller

I also wanted testing to grow alongside the platform.

For the Application API itself, I already had:

application-api-test.sh

which checks CRD validation.

For the controller, I added:

application-controller-test.sh

This test uses the real local Kubernetes cluster and the real Platform Operator.

The flow is:

test script
     |
     v
temporary namespace
     |
     v
Application
     |
     v
Platform Operator
     |
     +------ Deployment
     |
     +------ Service

The test verifies that the platform really provisions the expected resources.

What the Controller Test Checks

The test creates an Application like:

spec:
  image: example/test:1.0.0

  replicas: 2

  port:
    containerPort: 8080

Then it waits for:

Deployment
Service

to appear.

After that it checks the Deployment.

For example:

replicas = 2

image = example/test:1.0.0

containerPort = 8080

Then it checks the Service:

port = 8080
targetPort = 8080

Finally, it verifies that Application status contains:

deploymentName
serviceName

So the test is proving the entire first platform flow.

Why I Started With a Shell Integration Test

I originally considered starting immediately with:

JUnit
Awaitility
Java Operator SDK test extensions

But for this stage, I wanted something easier to understand.

The current test is very explicit:

create Application
        |
        v
wait for Deployment
        |
        v
verify Deployment
        |
        v
wait for Service
        |
        v
verify Service

It talks to the same Kubernetes API I use manually.

That makes it easy to understand exactly what the test is proving.

Later I can still add more focused Java tests where they make sense.

Test Isolation

The controller test creates a temporary namespace.

Something like:

platform-test-12345

All test resources live there:

Application
Deployment
Service

At the end, the namespace is deleted.

So the test leaves the cluster clean.

The lifecycle is:

create temporary namespace
        |
        v
run test
        |
        v
delete namespace

Running the Tests

The Application API test is:

task application:test:api

The controller integration test is:

task application:test:controller

So testing is now becoming part of the normal platform development workflow.

My goal from this point is:

feature
   +
test
   +
documentation

rather than adding all tests later.

Current Platform Architecture

The current architecture now looks like:

                        Developer
                            |
                            v
                       Application
                            |
                            v
                    Platform Operator
                            |
              +-------------+-------------+
              |                           |
              v                           v
         Deployment                    Service
              |                           |
              v                           |
             Pods <-----------------------+

The Application resource is the source of desired state.

The Platform Operator converts that desired state into Kubernetes implementation details.

What This Version Does Not Handle Yet

There are still many things missing.

For example:

workload readiness
Application conditions
update behavior tests
ConfigMaps
Secrets
health checks
resource requests
routes
authorization
failure handling

I am intentionally leaving them out.

The current milestone is only:

Application
→ Deployment
→ Service

and proving that behavior works.

Source Code

The complete implementation is available in my Platform Lab repository.

Repository: Platform Lab

The changes covered in this post are available in:

Pull Request: Add Application Controller

The main additions include:

Platform Operator module
Application Java models
ApplicationReconciler
Deployment dependent resource
Service dependent resource
Platform Operator Kubernetes deployment
RBAC
Application controller integration test

What I Learned

The biggest shift in this step was moving from:

Kubernetes operator concepts

to:

platform abstraction

With the Greeting operator, my main question was:

How does a controller work?

With the Application controller, the question is becoming:

What Kubernetes complexity should the platform hide from an application developer?

The developer writes:

spec:
  image: ...
  replicas: ...
  port:
    containerPort: ...

and the platform generates:

Deployment
Service
selectors
Pod template
container port
service port

That translation is the real beginning of Platform Lab.

What's Next?

The next thing I want to test is whether the platform maintains desired state when the Application changes.

For example:

spec:
  image: greeting-service:2.0.0
  replicas: 3

should update the existing Deployment.

The flow should be:

Application generation 1
        |
        v
Deployment
image = 1.0.0
replicas = 1

        |
        | Application updated
        v

Application generation 2
        |
        v
same Deployment
image = 2.0.0
replicas = 3

That will let me explore Application updates and reconciliation as part of the real platform instead of only through the learning operator.

After that, I want to move toward:

Application readiness
configuration
Secrets
ApplicationRelease
ApiDependency
AccessGrant
networking
authorization

one capability at a time.

Building the First Real API for My Kubernetes Platform

shubham goel — Sat, 03 Oct 2026 07:33:10 +0000

Over the last few posts, I have been building a small Greeting Kubernetes operator.

It started with a simple CRD:

Greeting
    |
    v
Controller
    |
    v
ConfigMap

That small example helped me understand a lot of Kubernetes operator concepts:

Custom Resources
Controllers
Reconciliation
Dependent Resources
Status
Conditions
RBAC
Failures
Retries
Events
Manual Reconciliation
Owner References
Garbage Collection

At this point, I felt that continuing to add more concepts to the Greeting operator would mostly make the example more complicated.

So I decided to keep it as a learning/reference operator and start working on the actual idea behind Platform Lab.

The first real platform API is:

Application

What I Want the Platform to Do

The original idea behind Platform Lab is simple.

As an application developer, I do not want to write all of this every time:

Deployment
Service
ConfigMap
Secret
ServiceAccount
NetworkPolicy
AuthorizationPolicy
Gateway configuration
Observability configuration

Instead, I want the developer-facing API to be much smaller.

Something like:

apiVersion: platform.shubforge.dev/v1alpha1
kind: Application

metadata:
  name: greeting-service

spec:
  image: greeting-service:1.0.0

  replicas: 1

  port:
    containerPort: 8080

Eventually, the platform operator should translate that into:

Application
     |
     v
Platform Operator
     |
     +---- Deployment
     |
     +---- Service

And later, the same platform API can grow into other capabilities.

Application
    |
    +---- Deployment
    +---- Service
    +---- Configuration
    +---- Secrets
    +---- Identity
    +---- Networking
    +---- Authorization
    +---- Observability

But I do not want to build everything at once.

The first step is only defining the API.

Starting With the Application CRD

I created a new Custom Resource Definition:

applications.platform.shubforge.dev

The API is:

Group:   platform.shubforge.dev
Version: v1alpha1
Kind:    Application

The resource is namespace scoped.

For now, the specification is intentionally small:

spec:

  image: greeting-service:1.0.0

  replicas: 1

  port:
    containerPort: 8080

That gives us only three concepts:

image
replicas
container port

I deliberately did not add things like:

CPU
memory
environment variables
Secrets
ConfigMaps
health checks
volumes
routes
Istio
authorization
API dependencies

yet.

Those can be introduced when the platform actually starts supporting them.

The CRD

The CRD currently looks roughly like this:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition

metadata:
  name: applications.platform.shubforge.dev

spec:
  group: platform.shubforge.dev

  scope: Namespaced

  names:
    plural: applications
    singular: application
    kind: Application

    shortNames:
      - papp

  versions:
    - name: v1alpha1
      served: true
      storage: true

      schema:
        openAPIV3Schema:
          type: object

          properties:

            spec:
              type: object

              required:
                - image
                - port

              properties:

                image:
                  type: string
                  minLength: 1

                replicas:
                  type: integer
                  minimum: 1
                  default: 1

                port:
                  type: object

                  required:
                    - containerPort

                  properties:

                    containerPort:
                      type: integer
                      minimum: 1
                      maximum: 65535

The important thing here is that some validation belongs directly in the API.

For example:

replicas >= 1

and:

1 <= containerPort <= 65535

can be validated by Kubernetes before the controller even receives the resource.

Why `port` Is an Object

I could have used:

spec:
  port: 8080

But I chose:

spec:

  port:
    containerPort: 8080

because it gives the API some room to grow.

Later it could become:

port:
  containerPort: 8080
  protocol: TCP

or eventually:

ports:
  - name: http
    containerPort: 8080

For now, I still want to keep it to one port.

This is v1alpha1, so the API can evolve as I understand the platform better.

Defining Status Early

There is no Application controller yet.

Still, I added the basic status shape I expect the platform operator to eventually populate.

Something like:

status:

  observedGeneration: 1

  deploymentName: greeting-service

  serviceName: greeting-service

  readyReplicas: 1

  conditions:
    - type: Ready
      status: "True"

This gives the API an initial contract between:

desired state

and:

observed state

At the moment, the READY field remains empty because there is no controller.

That is expected.

Creating the First Application

I added a sample resource:

apiVersion: platform.shubforge.dev/v1alpha1
kind: Application

metadata:
  name: greeting-service

spec:
  image: greeting-service:1.0.0

  replicas: 1

  port:
    containerPort: 8080

Then installed the CRD:

kubectl apply -f k8s/crds/applications.yaml

and created the Application:

kubectl apply -f k8s/samples/application.yaml

Now Kubernetes understands:

Application/greeting-service

I can check it using:

kubectl get applications

or the short name:

kubectl get papp

The output includes information such as:

NAME               READY   IMAGE                    REPLICAS
greeting-service           greeting-service:1.0.0   1

READY is still empty because the platform operator does not exist yet.

That will be the next step.

Introducing Tests From the Beginning

One thing I want to do differently from the Greeting operator is testing.

I built several Greeting features first and only thought about automated tests later.

For the actual platform APIs, I want the flow to be:

feature
   +
test
   +
documentation

from the beginning.

There is no Java controller yet, so introducing JUnit, Awaitility, or operator test frameworks would be premature.

But the API itself can already be tested.

Testing the Application API

I added:

scripts/tests/application-api-test.sh

The test talks to the real Kubernetes API server.

It first installs the CRD and then uses:

kubectl apply --dry-run=server

to validate different Application manifests.

The important part is:

--dry-run=server

The resource is sent to the Kubernetes API server and validated there, but it is not persisted.

So the test is not implementing Kubernetes validation itself.

It is asking Kubernetes:

Would you accept this Application?

Testing a Valid Application

The first test checks the normal sample:

spec:
  image: greeting-service:1.0.0

  replicas: 1

  port:
    containerPort: 8080

Kubernetes should accept it.

Conceptually:

valid Application
        |
        v
Kubernetes API
        |
        v
accepted

Testing an Invalid Port

Next I test:

port:
  containerPort: 70000

But the CRD defines:

maximum = 65535

So Kubernetes should reject it.

containerPort = 70000
        |
        v
CRD validation
        |
        v
rejected

Testing a Missing Image

The Application API requires:

image:

So this resource should fail:

spec:

  replicas: 1

  port:
    containerPort: 8080

Again, Kubernetes handles the validation.

Testing Invalid Replicas

The API currently defines:

replicas >= 1

So this:

replicas: 0

should be rejected.

The test now verifies several parts of the Application contract:

valid resource
        → accepted

invalid port
        → rejected

missing image
        → rejected

replicas = 0
        → rejected

Running the API Tests

I added a Taskfile command:

task application:test:api

which runs:

scripts/tests/application-api-test.sh

The test output is roughly:

Testing Application CRD...

1. Installing Application CRD
✓ Application CRD installed

2. Testing valid Application
✓ Valid Application accepted

3. Testing invalid container port
✓ Invalid port rejected

4. Testing missing image
✓ Missing image rejected

5. Testing replicas below minimum
✓ Invalid replica count rejected

Application API tests passed.

This is simple, but it gives me the first automated test for the actual Platform Lab APIs.

Why I Am Not Using JUnit Yet

There is currently no Java code involved in the Application API.

So using something like:

JUnit
Awaitility
Java Operator SDK test utilities
Testcontainers

would not add much value yet.

Right now the thing I want to test is:

Does Kubernetes correctly understand and validate my platform API?

The real Kubernetes API server is the best place to answer that.

Once I build:

ApplicationReconciler

the test requirements will change.

Then I will want to verify things like:

Application created
        |
        v
Deployment created

Application created
        |
        v
Service created

That is where Java integration tests will start making sense.

Taskfile Commands

I also added Application-related commands to the Taskfile.

Install the CRD:

task application:crd:install

Create the sample Application:

task application:create

List Applications:

task application:get

Describe the sample:

task application:describe

Delete it:

task application:delete

Run API tests:

task application:test:api

The Taskfile continues to be the main developer interface for Platform Lab.

Where the Platform Is Now

The repository has moved from:

Greeting
    |
    v
learning Kubernetes operators

to:

Application
    |
    v
building the actual platform

Right now:

Application CRD
      |
      v
Kubernetes API

is all that exists.

The next step is:

Application
      |
      v
ApplicationReconciler
      |
      +---- Deployment
      |
      +---- Service

That will be the first actual platform behavior.

Source Code

The complete project is available in my Platform Lab repository.

Repository: Platform Lab

The changes from this post are available in:

Pull Request: Add Platform Application API

The main additions are:

Application CRD
Application sample
CRD validation
Application Taskfile commands
Application API tests

What I Learned

One thing I am trying to change as I move into the actual platform work is when I introduce complexity.

With the Greeting operator, I explored many Kubernetes concepts because the purpose was learning.

For the platform itself, I want features to have a reason to exist.

For example, I am not adding:

finalizers
advanced reconciliation
external API handling
Secrets
Istio configuration

until a real platform feature needs them.

The same applies to testing.

For this API, a small server-side Kubernetes validation test is enough.

When the controller arrives, the testing strategy can grow with it.

What's Next?

The next step is much more interesting.

I want this:

apiVersion: platform.shubforge.dev/v1alpha1
kind: Application

metadata:
  name: greeting-service

spec:
  image: greeting-service:1.0.0

  replicas: 1

  port:
    containerPort: 8080

to produce:

Application/greeting-service
            |
            v
      Platform Operator
        /          \
       v            v
Deployment       Service

That means the next step is building the actual:

platform-operator

and its first controller:

ApplicationReconciler

This time, the controller and its integration tests will be built together.

Who Deletes Resources Created by a Kubernetes Operator?

shubham goel — Sat, 03 Oct 2026 06:34:06 +0000

In the previous posts, I built a small Kubernetes operator around a custom Greeting resource.

The operator currently does a few things:

Greeting
    |
    v
Greeting Operator
    |
    v
ConfigMap

It watches Greeting resources, creates a ConfigMap, updates status, records Events, retries failures, and also supports manual reconciliation.

So far, most of my focus was on:

create
update
reconcile
retry

But there was one lifecycle question I had not really explored yet:

What happens to the ConfigMap when the Greeting is deleted?

That led me into Kubernetes owner references and garbage collection.

The Current Resource Relationship

A Greeting looks like this:

apiVersion: platform.shubforge.dev/v1alpha1
kind: Greeting

metadata:
  name: hello

spec:
  message: "Hello from Platform Lab"

The operator creates:

apiVersion: v1
kind: ConfigMap

metadata:
  name: hello-greeting

data:
  message: "Hello from Platform Lab"

So conceptually:

Greeting/hello
      |
      v
ConfigMap/hello-greeting

Until now, I was mainly thinking about this as:

controller creates another resource

But Kubernetes also needs to understand the lifecycle relationship between those two resources.

Looking at the Generated ConfigMap

I started by inspecting the ConfigMap created by the operator:

kubectl get configmap hello-greeting -o yaml

I got:

apiVersion: v1

data:
  message: Hello from the updated Greeting

kind: ConfigMap

metadata:
  labels:
    app.kubernetes.io/managed-by: greeting-operator

  name: hello-greeting
  namespace: default

  ownerReferences:
    - apiVersion: platform.shubforge.dev/v1alpha1
      blockOwnerDeletion: false
      controller: false
      kind: Greeting
      name: hello
      uid: 4c749fca-e2a3-49f7-a330-1374b72c3bf8

The interesting part was:

ownerReferences:

The ConfigMap had an owner reference pointing back to:

Greeting/hello

So the relationship was not only something my Java code understood.

Kubernetes itself also knew:

Greeting/hello
       |
       | owns
       v
ConfigMap/hello-greeting

What Is an Owner Reference?

An owner reference is metadata stored on the dependent resource.

In my case, the dependent is:

ConfigMap/hello-greeting

and the owner is:

Greeting/hello

The ConfigMap contains:

ownerReferences:
  - apiVersion: platform.shubforge.dev/v1alpha1
    kind: Greeting
    name: hello
    uid: ...

This tells Kubernetes that the lifecycle of the ConfigMap is related to that specific Greeting.

The uid is especially interesting.

Kubernetes does not only store:

Greeting
+
hello

It also stores the unique identity of that exact resource.

So the relationship is closer to:

ConfigMap
    |
    v
specific Greeting UID

That matters because a resource can later be recreated with the same name but a different UID.

controller and blockOwnerDeletion

My generated owner reference also contained:

controller: false
blockOwnerDeletion: false

At first I expected something like:

controller: true

because the ConfigMap is created by my controller.

But the controller field has a more specific meaning in Kubernetes owner references.

What mattered for this experiment was that the ownership relationship itself existed:

Greeting
    |
    v
ConfigMap

Similarly:

blockOwnerDeletion: false

does not mean the ConfigMap is unrelated to the Greeting.

It controls a more specific deletion behavior.

For this simple experiment, I mainly wanted to understand whether Kubernetes could use the owner reference for dependent cleanup.

First Experiment: Delete the ConfigMap

I first tested what happens if I delete the dependent resource.

Both resources existed:

kubectl get greeting hello

and:

kubectl get configmap hello-greeting

Then I deleted only the ConfigMap:

kubectl delete configmap hello-greeting

At that point:

Greeting still exists
ConfigMap does not

But the desired state represented by the Greeting still says:

this Greeting should have a ConfigMap

So the controller reconciled again and recreated it.

The flow looked like:

ConfigMap deleted
       |
       v
Greeting still exists
       |
       v
desired state still requires ConfigMap
       |
       v
controller reconciles
       |
       v
ConfigMap recreated

This is reconciliation doing its job.

The resource disappeared, but the owner still existed, so the operator restored the desired state.

Second Experiment: Delete the Greeting

Then I tested the opposite case.

I recreated the Greeting and confirmed both resources existed again:

task greeting:create

Then:

kubectl get greeting hello

and:

kubectl get configmap hello-greeting

This time I deleted the owner:

kubectl delete greeting hello

Then I checked:

kubectl get greeting hello

and:

kubectl get configmap hello-greeting

The Greeting disappeared.

The ConfigMap also disappeared.

This time the operator did not recreate it.

That made sense because the source of the desired state was gone.

The flow was:

Greeting deleted
      |
      v
owner no longer exists
      |
      v
Kubernetes sees dependent ownerReference
      |
      v
ConfigMap removed

This was Kubernetes garbage collection rather than normal desired-state reconciliation.

Reconciliation vs Garbage Collection

This experiment helped me separate two mechanisms that looked similar at first.

When the dependent disappears

ConfigMap deleted
      |
      v
Greeting still exists
      |
      v
operator reconciles
      |
      v
ConfigMap recreated

This is:

reconciliation

The controller is maintaining desired state.

When the owner disappears

Greeting deleted
      |
      v
ownerReference
      |
      v
Kubernetes garbage collection
      |
      v
ConfigMap removed

This is:

garbage collection

The desired state itself has disappeared.

So I now think about the two like this:

Reconciliation
=
keep resources in the desired state while the owner exists

and:

Garbage Collection
=
remove dependents when their owner disappears

Watching It Live

One useful way to see this behavior is to watch the ConfigMap.

In one terminal:

kubectl get configmap hello-greeting -w

Then in another terminal:

kubectl delete greeting hello

That makes it easier to see the dependent disappear.

You can also watch both resources:

kubectl get greeting,configmap -w

Then delete:

kubectl delete greeting hello

This gives a much clearer view of the lifecycle than running one command at a time.

The Operator Did Not Need deleteConfigMap()

One part I found interesting was that I did not need to write something like:

deleteConfigMap();

inside the controller.

The ConfigMap lives inside Kubernetes and already has an owner reference.

That means Kubernetes has enough information to manage the dependent lifecycle.

So instead of:

Greeting deleted
      |
      v
operator manually finds ConfigMap
      |
      v
operator deletes ConfigMap

I get:

Greeting deleted
      |
      v
Kubernetes ownership information
      |
      v
garbage collector
      |
      v
ConfigMap deleted

That is much more Kubernetes-native.

Same Namespace

Greeting is a namespaced custom resource.

The managed ConfigMap is created in the same namespace.

For example:

default/hello
      |
      v
default/hello-greeting

If I create the Greeting in another namespace:

demo/hello

the ConfigMap should also be:

demo/hello-greeting

The operator uses the Greeting namespace when building the dependent resource.

Conceptually:

var namespace =
        greeting.getMetadata().getNamespace();

and then:

.withNamespace(namespace)

So the lifecycle stays scoped together.

Owner Reference Does Not Mean Finalizer

Another thing I wanted to keep separate was:

owner references

and:

finalizers

They are related to deletion, but they solve different problems.

An owner reference says:

This Kubernetes resource belongs to another Kubernetes resource.

For example:

Greeting
    |
    v
ConfigMap

Kubernetes can clean up that ConfigMap when the Greeting disappears.

A finalizer is different.

A finalizer says something closer to:

Before this resource is completely deleted,
the controller still has cleanup work to do.

That becomes important when the cleanup is not something Kubernetes can automatically handle.

A Future Example

Imagine the operator eventually created something through an external API:

Custom Resource
      |
      v
Operator
      |
      v
External API
      |
      v
External Resource

If I delete the custom resource:

kubectl delete ...

Kubernetes does not know anything about that external system.

An owner reference cannot help here because the external object is not a Kubernetes resource.

That is where finalizers become interesting.

The flow would be more like:

delete custom resource
        |
        v
deletionTimestamp set
        |
        v
controller sees deletion
        |
        v
external cleanup
        |
        v
finalizer removed
        |
        v
resource deletion completes

That is what I want to explore next.

Current Operator Lifecycle

At this point, the Greeting Operator has several different paths.

Desired state changes

Greeting.spec changes
        |
        v
generation changes
        |
        v
reconcile

Manual reconciliation

reconcile-at changes
        |
        v
update filter
        |
        v
reconcile

Failure

reconciliation fails
        |
        v
Ready=False
        |
        v
retry

Dependent deletion

ConfigMap deleted
        |
        v
Greeting still exists
        |
        v
reconcile
        |
        v
ConfigMap recreated

Owner deletion

Greeting deleted
        |
        v
ownerReference
        |
        v
Kubernetes garbage collection
        |
        v
ConfigMap deleted

Seeing all of these together made the operator lifecycle much clearer for me.

Current Architecture

The current flow looks roughly like this:

                         Greeting
                            |
              +-------------+-------------+
              |                           |
          spec change              reconcile-at change
              |                           |
              +-------------+-------------+
                            |
                            v
                       reconcile()
                            |
                            v
                        ConfigMap
                            |
             +--------------+--------------+
             |                             |
      ConfigMap deleted             Greeting deleted
             |                             |
             v                             v
        reconcile                   garbage collection
             |                             |
             v                             v
      ConfigMap recreated            ConfigMap removed

So the operator is no longer only about creating resources.

It is also about understanding resource lifecycle.

Source Code

The complete implementation is available in my Platform Lab repository.

Repository: Platform Lab

The changes and documentation covered in this post are available in:

Pull Request: Explore Greeting Resource Ownership

The current project includes:

Greeting CRD
Java Operator SDK controller
managed ConfigMap dependent resource
status and conditions
failure handling
automatic retries
Kubernetes Events
manual reconciliation
update filtering
owner references
Kubernetes garbage collection

What I Learned

The main thing I learned from this step is that deleting a resource is not always the controller's responsibility.

For Kubernetes resources, ownership metadata can allow Kubernetes itself to manage dependent cleanup.

The two cases are different:

dependent deleted
=
restore desired state

while:

owner deleted
=
desired state is gone

So:

delete ConfigMap
→ operator recreates it

but:

delete Greeting
→ ConfigMap disappears with it

That distinction made the relationship between reconciliation and garbage collection much clearer.

What's Next?

The ConfigMap in this example exists completely inside Kubernetes.

That means owner references and garbage collection work well for cleanup.

But what if the operator needs to clean up something Kubernetes does not control?

That leads to the next topic:

Finalizers

I want to explore:

what deletionTimestamp means

how a controller gets a chance to perform cleanup

how finalizers delay deletion

what happens when cleanup fails

why stuck finalizers can leave resources in Terminating

That should complete another important part of the operator lifecycle.

Manually Triggering Reconciliation in My Kubernetes Operator

shubham goel — Wed, 30 Sep 2026 03:08:04 +0000

In the previous post, I explored what happens when my Kubernetes operator fails.

I created a real failure by removing ConfigMap creation permission from the operator.

The controller failed, retried several times, and eventually stopped retrying.

Then I restored the RBAC permissions.

But something unexpected happened.

Nothing happened.

The operator now had permission to create the ConfigMap again, but the failed Greeting was not immediately reconciled.

That led me to another question:

How can I manually ask the controller to reconcile a resource again without changing its actual specification?

That is what I wanted to solve next.

The Problem

The failure flow looked like this:

Greeting created
      |
      v
ConfigMap creation fails
      |
      v
Ready=False
      |
      v
automatic retries
      |
      v
retry limit reached

Then I fixed the external problem:

RBAC fixed

But fixing the ClusterRole did not produce a new event for the Greeting controller.

So the resource remained in its failed state.

Previously I recovered by changing:

spec:
  message:

That worked because changing the spec caused another reconciliation.

But changing business configuration only to make the controller run again did not feel right.

I wanted something more explicit.

A Manual Reconciliation Annotation

I decided to use a Kubernetes annotation:

platform.shubforge.dev/reconcile-at

The idea is simple.

If I run:

kubectl annotate greeting hello \
  platform.shubforge.dev/reconcile-at="$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  --overwrite

the annotation gets a new timestamp.

For example:

metadata:

  annotations:
    platform.shubforge.dev/reconcile-at: "2026-09-29T07:00:00Z"

Every time I want another reconciliation, I update this value.

Conceptually:

reconcile-at changed
        |
        v
Greeting updated
        |
        v
Controller
        |
        v
reconcile()

This means I can request another reconciliation without changing:

spec:

But Annotation Changes Were Being Ignored

There was one more problem.

The Java Operator SDK normally avoids triggering reconciliation for every metadata-only update.

For example:

status changed
resourceVersion changed
annotation changed

should not necessarily cause the controller to run again.

For normal desired-state changes, Kubernetes updates:

metadata.generation

So the controller can react when the actual spec changes.

But my manual reconciliation only changes an annotation.

That means:

generation

does not change.

So I needed a custom update filter.

Creating a Greeting Update Filter

I added:

GreetingUpdateFilter.java

The goal is simple.

Reconcile when either:

spec generation changed

or:

reconcile-at annotation changed

Conceptually:

generation changed?
       |
       yes
       |
       v
   reconcile


reconcile-at changed?
       |
       yes
       |
       v
   reconcile


only status changed?
       |
       no
       |
       v
     ignore

The filter looks roughly like this:

public class GreetingUpdateFilter
        implements OnUpdateFilter<Greeting> {

    public static final String RECONCILE_AT_ANNOTATION =
            "platform.shubforge.dev/reconcile-at";

    @Override
    public boolean accept(
            Greeting newResource,
            Greeting oldResource) {

        return generationChanged(newResource, oldResource)
                || reconcileAnnotationChanged(
                        newResource,
                        oldResource
                );
    }

    private boolean generationChanged(
            Greeting newResource,
            Greeting oldResource) {

        return !Objects.equals(
                newResource.getMetadata().getGeneration(),
                oldResource.getMetadata().getGeneration()
        );
    }

    private boolean reconcileAnnotationChanged(
            Greeting newResource,
            Greeting oldResource) {

        return !Objects.equals(
                annotationValue(newResource),
                annotationValue(oldResource)
        );
    }

    private String annotationValue(
            Greeting greeting) {

        var annotations =
                greeting.getMetadata().getAnnotations();

        if (annotations == null) {
            return null;
        }

        return annotations.get(
                RECONCILE_AT_ANNOTATION
        );
    }
}

The important part is:

generationChanged(...)
        ||
reconcileAnnotationChanged(...)

This keeps the controller focused on meaningful updates.

Why Not Reconcile on Every Update?

One option would have been to simply accept every resource update.

Something like:

return true;

But that would be too broad.

The operator itself updates:

status:

and status updates also change the Kubernetes resource.

If every update triggered another reconciliation, it could create unnecessary reconciliation cycles.

So I want this:

spec changed
     |
     v
reconcile

and this:

manual reconcile annotation changed
     |
     v
reconcile

but not:

status changed
     |
     v
reconcile again
     |
     v
status changed
     |
     v
reconcile again

The custom filter lets me control that.

Configuring the Controller

I then configured the GreetingReconciler to use the custom update filter.

Conceptually:

@ControllerConfiguration(
    defaultFilters = false,
    informer = @Informer(
        onUpdateFilter = GreetingUpdateFilter.class
    )
)

The controller already had the managed ConfigMap workflow, so the overall structure becomes:

Greeting
    |
    v
Update Filter
    |
    +------ generation changed
    |
    +------ reconcile-at changed
    |
    v
GreetingReconciler
    |
    v
ConfigMap

Adding a Taskfile Command

Since I am already using Task as the developer interface for the project, I added:

task greeting:reconcile

The task updates the annotation with the current timestamp.

Conceptually:

greeting:reconcile:

  desc: Manually trigger Greeting reconciliation

  cmds:
    - |
      kubectl annotate greeting {{.GREETING_NAME}} \
        platform.shubforge.dev/reconcile-at="$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
        --overwrite

By default:

task greeting:reconcile

reconciles:

hello

For another Greeting:

task greeting:reconcile \
  GREETING_NAME=failure-test

This makes manual reconciliation much easier to use during development.

Testing It

I first tested it on the healthy hello Greeting.

I ran:

task greeting:reconcile

Then checked the annotation:

kubectl get greeting hello \
  -o jsonpath='{.metadata.annotations.platform\.shubforge\.dev/reconcile-at}'

The annotation contained the new timestamp.

Then I checked the Greeting Events:

kubectl describe greeting hello

and saw:

Events:
  Type    Reason      Age                 From                Message
  ----    ------      ----                ----                -------
  Normal  Reconciled  5s (x2 over 2m12s) greetingreconciler  Managed ConfigMap hello-greeting is in the desired state

The interesting part was:

x2

Kubernetes had aggregated two equivalent Reconciled Events.

One came from the normal reconciliation.

The second came from my manual reconciliation request.

So the annotation was doing exactly what I wanted.

Why Observed Generation Stayed at 1

Something else was interesting.

After manual reconciliation, I still had:

Observed Generation: 1

At first this might look strange.

But it is actually correct.

The manual reconciliation changed:

metadata:
  annotations:

It did not change:

spec:

So Kubernetes did not increment:

metadata.generation

Before manual reconciliation:

generation          = 1
observedGeneration  = 1

After manual reconciliation:

generation          = 1
observedGeneration  = 1

The operator simply reconciled generation 1 again.

This gave me a useful distinction.

generation
=
version of the desired spec

while:

reconcile-at
=
request to re-evaluate
the current desired spec

So:

spec changed
     |
     v
new generation
     |
     v
reconcile

is different from:

reconcile-at changed
     |
     v
same generation
     |
     v
reconcile again

Testing the Real Recovery Scenario

The healthy-resource test proved that the annotation worked.

But the real reason I added this feature was failure recovery.

So I repeated the RBAC failure experiment.

First, I edited the operator's ClusterRole:

kubectl edit clusterrole greeting-operator

The normal ConfigMap permissions looked like:

verbs:
  - get
  - list
  - watch
  - create
  - update
  - patch
  - delete

For the experiment I temporarily changed them to:

verbs:
  - get
  - list
  - watch

Then I verified:

kubectl auth can-i \
  create configmaps \
  --namespace default \
  --as=system:serviceaccount:platform-system:greeting-operator

and got:

no

Creating a Test Greeting

I created another Greeting directly:

kubectl apply -f - <<'EOF'
apiVersion: platform.shubforge.dev/v1alpha1
kind: Greeting

metadata:
  name: manual-reconcile-test

spec:
  message: "Manual reconciliation test"
EOF

The operator attempted to create:

manual-reconcile-test-greeting

but received:

403 Forbidden

The Greeting eventually moved to:

Ready=False

and automatic retries started.

After the retry limit was reached, it remained failed.

Fixing the External Problem

I restored the repository RBAC configuration:

task operator:deploy

Then verified:

kubectl auth can-i \
  create configmaps \
  --namespace default \
  --as=system:serviceaccount:platform-system:greeting-operator

This time:

yes

Previously, at this point I had changed:

spec:

to trigger another reconciliation.

This time I did not touch the specification.

Instead I ran:

task greeting:reconcile \
  GREETING_NAME=manual-reconcile-test

The annotation changed.

The update filter accepted the update.

The controller reconciled the same desired state again.

And this time ConfigMap creation succeeded.

Recovery Flow

The new recovery flow became:

ConfigMap creation fails
        |
        v
Ready=False
        |
        v
automatic retries
        |
        v
retry limit reached
        |
        v
external issue fixed
        |
        v
manual reconcile requested
        |
        v
reconcile-at annotation changes
        |
        v
update filter accepts event
        |
        v
reconcile()
        |
        v
ConfigMap created
        |
        v
Ready=True

This feels much cleaner than modifying application configuration only to wake the controller up.

Different Ways Reconciliation Happens

At this point my controller can be reconciled for several different reasons.

Spec Change

Greeting.spec changes
        |
        v
generation changes
        |
        v
reconcile

Managed Resource Change

ConfigMap changes or disappears
        |
        v
dependent resource event
        |
        v
reconcile

Failure Retry

reconciliation fails
        |
        v
automatic retry

Manual Reconciliation

reconcile-at changes
        |
        v
update filter
        |
        v
reconcile

All of them eventually call the same controller logic.

That also reinforces another important idea:

Reconciliation should be safe to run multiple times.

The controller should keep trying to move actual state toward desired state rather than assuming it runs only once.

Current Flow

The operator now looks roughly like:

                         Greeting
                            |
              +-------------+-------------+
              |                           |
          spec change              reconcile-at change
              |                           |
              v                           v
      generation change             update filter
              |                           |
              +-------------+-------------+
                            |
                            v
                       reconcile()
                            |
                     +------+------+
                     |             |
                  success        failure
                     |             |
                     v             v
                 ConfigMap     Ready=False
                     |             |
                     v             v
                 Ready=True    Warning Event
                     |             |
                     v             v
               Normal Event       Retry

Source Code

The complete implementation is available in my Platform Lab repository.

Repository: Platform Lab

The changes covered in this post are available in:

Pull Request: Add Manual Greeting Reconciliation

The main changes include:

GreetingUpdateFilter
reconcile-at annotation support
manual reconciliation Taskfile command
filtering spec changes and manual reconciliation separately

What I Learned

The main thing I learned from this step is that:

desired state changed

and:

reconcile the current desired state again

are two different operations.

Changing spec means:

I want something different.

Manual reconciliation means:

I still want the same thing.
Please check it again.

That is why keeping the manual trigger in metadata rather than the resource specification feels like a better fit.

It also helped me understand why:

generation = 1

can remain unchanged even though the controller ran multiple times.

The generation represents the desired configuration, not the number of reconciliation attempts.

What's Next?

There is still more I want to understand around reconciliation itself.

Some areas I want to explore are:

idempotency
periodic reconciliation
rescheduling
external dependency handling
event filtering
reconciliation best practices

After that, I want to move into another important part of the operator lifecycle:

owner references
resource deletion
finalizers

For now, the Greeting operator supports both automatic reconciliation and an explicit way to ask it to evaluate the current desired state again.

Understanding Retries and Failures in a Kubernetes Operator

shubham goel — Sat, 26 Sep 2026 12:42:53 +0000

In the previous post, I added status and conditions to my Greeting custom resource.

That gave me something like:

status:
  observedGeneration: 1
  configMapName: hello-greeting

  conditions:
    - type: Ready
      status: "True"
      reason: ConfigMapReady
      message: Managed ConfigMap hello-greeting is in the desired state

This was useful for the successful path.

But until now, everything was mostly working.

So the next question was:

What happens when reconciliation fails?

For example, what if the operator cannot create the ConfigMap?

That led me into failure handling, retry behavior, and Kubernetes Events.

Current Flow

The normal flow looks like this:

Greeting
    |
    v
Greeting Operator
    |
    v
ConfigMap
    |
    v
Ready=True

But I also want the failed path to be visible:

Greeting
    |
    v
Greeting Operator
    |
    v
Failure
    |
    +----> Ready=False
    |
    +----> Warning Event
    |
    +----> Retry

The goal is not only to log the exception.

The custom resource itself should explain that something went wrong.

Adding Failure Status

When reconciliation fails, I update the Greeting status to:

status:

  conditions:
    - type: Ready
      status: "False"
      reason: ReconciliationFailed
      message: ...

Conceptually:

reconcile()
    |
    v
exception
    |
    v
update status
    |
    v
Ready=False

The full exception still goes to the operator logs.

But the custom resource gets a shorter and more useful message.

I think about it like this:

Operator logs
=
developer debugging

while:

status.message
=
resource-level information

Error Status Handling

The reconciler now handles failures using the Java Operator SDK error status mechanism.

The flow is roughly:

@Override
public ErrorStatusUpdateControl<Greeting> updateErrorStatus(
        Greeting greeting,
        Context<Greeting> context,
        Exception exception) {

    var status = greeting.getStatus();

    if (status == null) {
        status = new GreetingStatus();
    }

    status.setObservedGeneration(
            greeting.getMetadata().getGeneration()
    );

    status.setConditions(
            List.of(
                    createCondition(
                            status,
                            greeting.getMetadata().getGeneration(),
                            "False",
                            "ReconciliationFailed",
                            errorMessage(exception)
                    )
            )
    );

    greeting.setStatus(status);

    return ErrorStatusUpdateControl.patchStatus(greeting);
}

So now the resource can move between:

Ready=True

and:

Ready=False

depending on the reconciliation result.

Recording Kubernetes Events

I also wanted failures to be visible through normal Kubernetes tooling.

So the operator now records Kubernetes Events.

For a successful reconciliation:

Normal  Reconciled

For a failure:

Warning  ReconciliationFailed

This means I can run:

kubectl describe greeting hello

and see what happened directly on the resource.

For example:

Events:
  Type     Reason                  Message
  ----     ------                  -------
  Normal   Reconciled              Managed ConfigMap hello-greeting is in the desired state

During a failure:

Events:
  Type     Reason                  Message
  ----     ------                  -------
  Warning  ReconciliationFailed    ...

This is much nicer than requiring someone to immediately search operator logs.

RBAC for Events

Because the operator now creates Kubernetes Events, it also needs permission to do that.

I added:

- apiGroups:
    - ""

  resources:
    - events

  verbs:
    - get
    - create
    - patch

So the operator permissions now include:

watch Greetings
update Greeting status
manage ConfigMaps
record Kubernetes Events

Creating a Real Failure

Instead of adding something artificial like:

spec:
  fail: true

I wanted to create a real Kubernetes authorization failure.

The Greeting operator normally has permission to create ConfigMaps through its ClusterRole.

I opened the ClusterRole directly:

kubectl edit clusterrole greeting-operator

The ConfigMap permissions normally look something like:

- apiGroups:
    - ""

  resources:
    - configmaps

  verbs:
    - get
    - list
    - watch
    - create
    - update
    - patch
    - delete

For this experiment, I temporarily removed the write permissions:

- apiGroups:
    - ""

  resources:
    - configmaps

  verbs:
    - get
    - list
    - watch

Now the operator could still read and watch ConfigMaps, but it could no longer create one.

I verified this using:

kubectl auth can-i \
  create configmaps \
  --namespace default \
  --as=system:serviceaccount:platform-system:greeting-operator

The expected result was:

no

Creating a New Greeting Directly

Now I wanted the operator to attempt creation of a brand new ConfigMap.

Instead of modifying my existing sample YAML, I created a temporary Greeting directly from the terminal:

kubectl apply -f - <<'EOF'
apiVersion: platform.shubforge.dev/v1alpha1
kind: Greeting

metadata:
  name: failure-test

spec:
  message: "This should fail temporarily"
EOF

Here:

kubectl apply -f -

means that kubectl reads the manifest from standard input instead of a file.

This is useful for small temporary experiments because I do not need to add another YAML file to the repository.

I checked the new resource:

kubectl get greeting failure-test

and then its full YAML:

kubectl get greeting failure-test -o yaml

The operator tried to create:

failure-test-greeting

but Kubernetes rejected the request because the ServiceAccount no longer had permission to create ConfigMaps.

The failure was a real:

403 Forbidden

The Greeting Moved to Ready=False

The resource status eventually showed:

status:

  conditions:
    - type: Ready
      status: "False"
      reason: ReconciliationFailed

So the flow became:

Greeting
    |
    v
Operator
    |
    v
create ConfigMap
    |
    v
403 Forbidden
    |
    v
Ready=False

I could also inspect the Kubernetes Events:

kubectl describe greeting failure-test

and see something like:

Events:
  Type     Reason                  Message
  ----     ------                  -------
  Warning  ReconciliationFailed   ...

The Operator Was Retrying

One thing I initially misunderstood was what happened after the failure.

The Events output showed:

Warning  ReconciliationFailed  5m12s (x6 over 5m31s)

The important part was:

x6

That meant the operator had not failed only once.

It had retried several times.

The actual flow was closer to:

Attempt 1
   |
   v
403 Forbidden
   |
   v
retry

Attempt 2
   |
   v
403 Forbidden
   |
   v
retry

Attempt 3
   |
   v
...

retry limit reached

So automatic retry was already working.

The operator simply does not retry forever.

Retry vs Reconciliation Trigger

This experiment helped me understand an important difference.

A retry means:

The previous reconciliation failed,
so try it again.

A reconciliation trigger means:

Something relevant changed,
so evaluate the resource again.

These are not the same thing.

The automatic retries eventually stopped after the retry limit was reached.

At that point, the Greeting remained:

Ready=False

until another relevant event happened.

Restoring the Permissions

After observing the failure and retries, I restored the correct RBAC permissions.

Instead of manually editing the ClusterRole back again, I reapplied the operator manifests from the repository:

task operator:deploy

This restored the ConfigMap permissions defined in:

k8s/operator/rbac.yaml

I verified that the operator could create ConfigMaps again:

kubectl auth can-i \
  create configmaps \
  --namespace default \
  --as=system:serviceaccount:platform-system:greeting-operator

This time the result was:

yes

But something interesting happened.

The ConfigMap was still not created immediately.

Why Fixing RBAC Did Not Trigger Reconciliation

The retries had already been exhausted.

Changing:

ClusterRole

does not produce an event for the Greeting controller.

The controller is mainly watching resources relevant to its reconciliation flow, such as:

Greeting
ConfigMap

It is not watching:

ClusterRole

So this happened:

ConfigMap creation fails
        |
        v
retries exhausted
        |
        v
RBAC fixed
        |
        v
ClusterRole changed
        |
        v
no Greeting event
        |
        v
no immediate reconciliation

The permission problem was fixed, but nothing told the controller to evaluate failure-test again.

Triggering Another Reconciliation

For this experiment, I changed the Greeting.spec.

I ran:

kubectl patch greeting failure-test \
  --type=merge \
  -p '{"spec":{"message":"Recovery test"}}'

This changed the desired resource.

Kubernetes increased the resource generation, which produced a new event for the controller.

Now the flow was:

RBAC fixed
    |
    v
Greeting spec changed
    |
    v
generation increased
    |
    v
new reconciliation
    |
    v
ConfigMap creation retried
    |
    v
success

This time the operator had permission to create the ConfigMap.

Checking the Recovery

I verified the ConfigMap:

kubectl get configmap failure-test-greeting

and then inspected the Greeting:

kubectl get greeting failure-test -o yaml

The status had moved back to:

conditions:
  - type: Ready
    status: "True"
    reason: ConfigMapReady

The Events now showed both parts of the story:

kubectl describe greeting failure-test

with output similar to:

Warning  ReconciliationFailed  5m12s (x6 over 5m31s)
Normal   Reconciled            8s     Managed ConfigMap failure-test-greeting is in the desired state

That was probably the most useful part of this experiment.

The event history showed:

failure
    |
    v
automatic retries
    |
    v
retry limit reached
    |
    v
external problem fixed
    |
    v
new reconciliation
    |
    v
recovery

Failure and Recovery Lifecycle

The complete experiment looked like this:

Greeting created
      |
      v
ConfigMap creation fails
      |
      v
Ready=False
      |
      v
Warning Event
      |
      v
automatic retries
      |
      v
retry limit reached
      |
      v
RBAC fixed
      |
      v
no reconciliation yet
      |
      v
Greeting spec updated
      |
      v
new reconciliation
      |
      v
ConfigMap created
      |
      v
Ready=True
      |
      v
Normal Event

This gave me a much clearer picture of how operator failure handling works.

Status, Events and Logs

At this point, the operator exposes information in three different places.

Logs

Useful for debugging the operator itself.

stack traces
internal errors
implementation details

Status

Useful for understanding the current state of the custom resource.

type: Ready
status: "False"
reason: ReconciliationFailed

Events

Useful for understanding important things that happened to the resource.

Warning  ReconciliationFailed
Normal   Reconciled

So I now think about them like this:

Logs
    |
    v
developer debugging

Status
    |
    v
current resource state

Events
    |
    v
important resource history

Each solves a different problem.

Successful Flow

The normal path is now:

Greeting
    |
    v
Operator
    |
    v
ConfigMap
    |
    +----> Ready=True
    |
    +----> Normal Reconciled Event

Example:

status:

  configMapName: hello-greeting

  observedGeneration: 1

  conditions:
    - type: Ready
      status: "True"
      reason: ConfigMapReady
      message: Managed ConfigMap hello-greeting is in the desired state

Failure Flow

The failed path looks like:

Greeting
    |
    v
Operator
    |
    v
Failure
    |
    +----> Ready=False
    |
    +----> Warning Event
    |
    +----> Retry

If retries keep failing:

failure
    |
    v
retry
    |
    v
failure
    |
    v
retry
    |
    v
retry limit reached

The resource then waits for another reconciliation trigger.

Taskfile Commands

I added a command for checking Greeting Events:

task greeting:events

which is basically:

kubectl describe greeting hello

The useful Greeting commands now include:

task greeting:create
task greeting:get
task greeting:status
task greeting:events

For the operator:

task operator:image:build
task operator:image:load
task operator:deploy
task operator:restart
task operator:status
task operator:logs

The Taskfile continues to be the main developer interface for the project.

Current Architecture

The current operator flow now looks like:

                     Greeting
                         |
                         v
                  Greeting Operator
                         |
                  +------+------+
                  |             |
               success        failure
                  |             |
                  v             v
              ConfigMap     Ready=False
                  |             |
                  v             v
              Ready=True    Warning Event
                  |             |
                  v             v
            Normal Event       Retry

The operator is now handling more than resource creation.

It is also exposing:

state
failures
retry behavior
events
recovery

through Kubernetes-native mechanisms.

Source Code

The complete implementation is available in my Platform Lab repository.

Repository: Platform Lab

The changes covered in this post are available in:

Pull Request: Add Failure Handling and Kubernetes Events

The project currently includes:

Greeting CRD
Java Operator SDK controller
ConfigMap dependent resource
status and conditions
Docker packaging
Kubernetes deployment
ServiceAccount and RBAC
failure handling
automatic retries
Kubernetes Events

What I Learned

The biggest thing I learned from this step was that:

retry

and:

reconciliation trigger

are different things.

The operator automatically retried the failed reconciliation.

But those retries were limited.

Once they were exhausted, fixing an unrelated external dependency like RBAC did not automatically create a new Greeting event.

A new relevant event, such as changing the resource specification, caused reconciliation to run again.

That gave me a much clearer picture of how event-driven controllers actually behave.

What's Next?

There is still more to understand around reconciliation itself.

Some questions I want to explore separately are:

Can reconciliation be triggered manually?
Should there be periodic reconciliation?
When should we reschedule?
How should external dependency changes be handled?
What are good reconciliation practices?

I want to keep that as a separate topic.

After that, I also want to explore:

owner references
resource deletion
finalizers
integration tests
GitHub Actions

For now, the Greeting Operator can report both success and failure using Kubernetes-native mechanisms and automatically retry transient failures.

Adding Status and Conditions to My Kubernetes Operator

shubham goel — Fri, 25 Sep 2026 20:14:28 +0000

In the previous posts, I created a custom Kubernetes resource called Greeting, built a Java controller for it, and then deployed that controller inside Kubernetes.

The current flow looks like this:

Greeting
    |
    v
Java Operator
    |
    v
ConfigMap

For example, I can create:

apiVersion: platform.shubforge.dev/v1alpha1
kind: Greeting

metadata:
  name: hello

spec:
  message: "Hello from Platform Lab"

and the operator creates:

hello-greeting

as a ConfigMap.

This works.

But there is still one problem.

If I run:

kubectl get greeting hello -o yaml

the Greeting resource itself does not tell me much about what the operator has done.

I can check the ConfigMap.

I can check operator logs.

But ideally, the custom resource itself should tell me its current state.

That is where the Kubernetes status subresource comes in.

What I Want to Achieve

I want my Greeting to eventually look something like this:

apiVersion: platform.shubforge.dev/v1alpha1
kind: Greeting

metadata:
  name: hello
  generation: 1

spec:
  message: "Hello from Platform Lab"

status:
  observedGeneration: 1
  configMapName: hello-greeting

  conditions:
    - type: Ready
      status: "True"
      reason: ConfigMapReady
      message: Managed ConfigMap hello-greeting is in the desired state

Now someone looking at the resource can understand:

Greeting requested
       |
       v
Operator processed it
       |
       v
ConfigMap exists
       |
       v
Ready = True

without needing to inspect the operator logs.

Spec vs Status

One thing that became clearer while working on this was the difference between:

spec

and:

status

I think about them like this:

spec
=
What the user wants

while:

status
=
What the operator observed or achieved

For example:

spec:
  message: "Hello from Platform Lab"

is provided by the user.

But:

status:
  configMapName: hello-greeting

is written by the operator.

So conceptually:

User
 |
 v
spec

Operator
 |
 v
status

This separation becomes important once the custom resource starts doing more than storing configuration.

Enabling the Status Subresource

The first change was in the Greeting CRD.

Under the API version, I added:

subresources:
  status: {}

So part of the CRD now looks like:

versions:
  - name: v1alpha1
    served: true
    storage: true

    subresources:
      status: {}

This tells Kubernetes that status should be treated as a separate subresource.

The resource still has:

spec:

for desired configuration.

But the operator can now separately update:

status:

to report the current state.

Defining the Status Schema

I also added the status structure to the CRD schema.

For now, I want to expose:

observedGeneration
configMapName
conditions

So the status schema contains something like:

status:
  type: object

  properties:

    observedGeneration:
      type: integer
      format: int64

    configMapName:
      type: string

    conditions:
      type: array

      x-kubernetes-list-type: map

      x-kubernetes-list-map-keys:
        - type

      items:
        type: object

        properties:

          type:
            type: string

          status:
            type: string
            enum:
              - "True"
              - "False"
              - "Unknown"

          observedGeneration:
            type: integer
            format: int64

          lastTransitionTime:
            type: string
            format: date-time

          reason:
            type: string

          message:
            type: string

        required:
          - type
          - status

This gives the operator a structured place to report its state.

Representing Status in Java

I already had an empty GreetingStatus class.

Previously it looked like:

public class GreetingStatus {
}

Now it contains the fields I want to expose:

public class GreetingStatus {

    private Long observedGeneration;

    private String configMapName;

    private List<Condition> conditions = new ArrayList<>();

    public Long getObservedGeneration() {
        return observedGeneration;
    }

    public void setObservedGeneration(Long observedGeneration) {
        this.observedGeneration = observedGeneration;
    }

    public String getConfigMapName() {
        return configMapName;
    }

    public void setConfigMapName(String configMapName) {
        this.configMapName = configMapName;
    }

    public List<Condition> getConditions() {
        return conditions;
    }

    public void setConditions(List<Condition> conditions) {
        this.conditions = conditions;
    }
}

I am using Kubernetes/Fabric8's Condition model instead of creating my own condition structure.

The status now looks conceptually like:

GreetingStatus
    |
    +-- observedGeneration
    |
    +-- configMapName
    |
    +-- conditions
            |
            +-- Ready

What is observedGeneration?

This was another useful Kubernetes concept to understand.

Kubernetes maintains:

metadata:
  generation:

for resources.

If I change the desired configuration:

spec:
  message: "Hello"

to:

spec:
  message: "Hello Updated"

the resource generation increases.

For example:

metadata:
  generation: 2

The operator can then report:

status:
  observedGeneration: 2

This tells us:

The operator has processed generation 2 of this resource.

So if I see:

generation          = 3
observedGeneration  = 2

I know the resource has changed, but the status still represents the previous generation.

After successful reconciliation:

generation          = 3
observedGeneration  = 3

The flow becomes:

User updates spec
      |
      v
generation increases
      |
      v
Operator reconciles
      |
      v
observedGeneration updated

This becomes much more useful when operators start doing slower or more complicated work.

Adding the Ready Condition

I also wanted the resource to expose a simple health state.

For now, I added one condition:

Ready

When the ConfigMap has been reconciled successfully, I set:

conditions:

  - type: Ready

    status: "True"

    reason: ConfigMapReady

    message: Managed ConfigMap hello-greeting is in the desired state

This gives users a simple answer to:

Is my Greeting ready?

For the current implementation:

Ready=True

means the managed ConfigMap exists in the desired state.

Later, I can also introduce:

Ready=False

with failure reasons.

For example:

type: Ready
status: "False"
reason: ReconciliationFailed
message: Failed to create ConfigMap

But I am keeping failure handling for a later step.

Updating Status from the Reconciler

Previously my reconciler finished with:

return UpdateControl.noUpdate();

The dependent resource was reconciled, but the Greeting itself was not updated.

Now the reconciler also builds a GreetingStatus.

Conceptually:

var status = new GreetingStatus();

status.setObservedGeneration(
        greeting.getMetadata().getGeneration()
);

status.setConfigMapName(
        configMap.getMetadata().getName()
);

status.setConditions(
        List.of(readyCondition)
);

greeting.setStatus(status);

Then instead of:

UpdateControl.noUpdate()

I return:

UpdateControl.patchStatus(greeting)

So the reconciliation flow has become:

Greeting
    |
    v
Reconcile dependent resources
    |
    v
ConfigMap ready
    |
    v
Update Greeting status

The operator is now managing both:

resources

and:

observable state

Reading the Managed ConfigMap

Because the ConfigMap is a managed dependent resource, the reconciler can access it through the reconciliation context.

Conceptually:

var configMap = context
        .getSecondaryResource(ConfigMap.class)
        .orElseThrow();

The controller can then use the actual ConfigMap to populate:

status.configMapName

This is useful because I am not simply guessing what was created.

The status is based on the resource managed by the reconciliation workflow.

lastTransitionTime

Conditions also contain:

lastTransitionTime:

Initially, I thought this could simply be updated every time the controller runs.

But that is not what it means.

lastTransitionTime represents when the condition actually changed state.

For example:

Ready=False
     |
     v
Ready=True

is a transition.

But:

Ready=True
     |
     v
reconciliation runs again
     |
     v
Ready=True

is not a transition.

So if the condition remains:

Ready=True

I keep the existing lastTransitionTime.

That makes the field much more meaningful.

Updating RBAC

Because status is a separate Kubernetes subresource, the operator also needs permission to update it.

The RBAC configuration includes:

- apiGroups:
    - platform.shubforge.dev

  resources:
    - greetings/status

  verbs:
    - get
    - patch
    - update

This is another interesting detail.

Permission to work with:

greetings

does not automatically mean the operator can update:

greetings/status

The status subresource has its own RBAC permissions.

Better kubectl Output

Now that the Greeting has useful status information, I also wanted:

kubectl get greetings

to show more than just:

NAME    AGE
hello   2m

So I added additional printer columns to the CRD.

additionalPrinterColumns:

  - name: Ready
    type: string
    jsonPath: .status.conditions[?(@.type=="Ready")].status

  - name: ConfigMap
    type: string
    jsonPath: .status.configMapName

  - name: Age
    type: date
    jsonPath: .metadata.creationTimestamp

Now:

kubectl get greetings

can show something like:

NAME    READY   CONFIGMAP        AGE
hello   True    hello-greeting   2m

I really like this improvement because the custom resource starts feeling more like a normal Kubernetes API.

Instead of checking:

kubectl logs ...

I can simply run:

kubectl get greetings

and immediately see the state.

Running the Updated Operator

Because the CRD changed, I first reinstall it:

task crd:install

Then rebuild the operator image:

task operator:image:build

Load it into Kind:

task operator:image:load

Deploy it:

task operator:deploy

Since I am currently using the same local development image tag, I restart the operator Deployment:

kubectl rollout restart \
  deployment/greeting-operator \
  -n platform-system

and wait for it:

kubectl rollout status \
  deployment/greeting-operator \
  -n platform-system

Then I create the Greeting:

task greeting:create

Checking the Status

Now I can run:

kubectl get greeting hello -o yaml

and see:

apiVersion: platform.shubforge.dev/v1alpha1
kind: Greeting

metadata:
  generation: 1
  name: hello

spec:
  message: Hello from Platform Lab

status:
  configMapName: hello-greeting

  observedGeneration: 1

  conditions:
    - type: Ready
      status: "True"
      reason: ConfigMapReady
      message: Managed ConfigMap hello-greeting is in the desired state
      observedGeneration: 1
      lastTransitionTime: "..."

Now the resource itself tells me what happened.

Testing a Spec Update

Next, I change:

spec:
  message: "Hello from Platform Lab"

to:

spec:
  message: "Greeting updated"

and apply it again:

task greeting:create

Kubernetes increments:

metadata:
  generation:

The operator reconciles the ConfigMap and then updates:

status:
  observedGeneration:

So I eventually get:

generation          = 2
observedGeneration  = 2

The complete flow becomes:

User changes spec
       |
       v
generation = 2
       |
       v
Operator reconciles
       |
       v
ConfigMap updated
       |
       v
status updated
       |
       v
observedGeneration = 2

Why Status Matters

Before adding status, the operator worked correctly.

But understanding its state required looking elsewhere:

Greeting
    |
    +-- check ConfigMap
    |
    +-- check operator logs

Now the resource itself reports its current state:

Greeting
    |
    +-- spec
    |
    +-- status
          |
          +-- Ready
          +-- ConfigMap
          +-- observedGeneration

This becomes even more important when the controller starts managing multiple resources or talking to external systems.

A custom resource should not only accept desired configuration.

It should also give useful feedback about what is actually happening.

Current Architecture

The project now looks like:

Greeting
    |
    | spec
    v
Kubernetes API
    |
    v
Greeting Operator
    |
    +--------> ConfigMap
    |
    v
Greeting Status
    |
    +-- Ready
    +-- configMapName
    +-- observedGeneration

And the operator itself continues to run inside Kubernetes:

Operator Pod
     |
     v
ServiceAccount
     |
     v
ClusterRoleBinding
     |
     v
ClusterRole
     |
     v
Kubernetes API

Taskfile

I also added a simple command for checking the Greeting status:

task greeting:status

which runs something similar to:

kubectl get greeting hello -o yaml

The full development flow remains:

task cluster:create
task crd:install
task operator:image:build
task operator:image:load
task operator:deploy
task greeting:create
task greeting:status

The Taskfile continues to act as the simple developer interface for the project.

Source Code

The complete implementation is available in my Platform Lab repository.

Repository: Platform Lab

The changes covered in this post are available separately in:

Pull Request: Add Greeting Status and Conditions

The repository contains:

Greeting CRD
Java Greeting operator
managed ConfigMap
Docker packaging
Kubernetes deployment
ServiceAccount and RBAC
status subresource
Ready condition
observed generation
kubectl printer columns

What I Learned

The main thing I learned from this step is that reconciliation is not only about making resources exist.

An operator should also communicate what it observed.

The user provides:

spec

The operator works toward that desired state and reports:

status

So the flow is now:

Desired State
     |
     v
spec
     |
     v
Operator
     |
     v
Actual Resources
     |
     v
status

That makes the custom resource much more useful.

What's Next?

Right now, the happy path works:

Ready=True

The next thing I want to explore is what happens when reconciliation fails.

That means looking at things like:

Ready=False
reason
message
retry behavior
Kubernetes Events

I also want to understand how errors should be reported to users without requiring them to inspect operator logs.

After that, I can start looking at:

finalizers
integration tests
GitHub Actions

For now, the Greeting resource can finally tell us both:

what we want

and:

what the operator actually achieved

Running My Java Kubernetes Operator Inside the Cluster

shubham goel — Fri, 25 Sep 2026 19:21:28 +0000

In the previous post, I built a simple Kubernetes controller in Java.

The flow was:

Greeting
    |
    v
Java Controller
    |
    v
ConfigMap

The controller was working, but there was one important limitation.

It was still running on my laptop.

My Laptop
    |
    | Java Process
    |
    v
Greeting Controller
    |
    | kubeconfig
    |
    v
Kind Kubernetes Cluster

That was useful while developing the controller, but eventually an operator should run inside Kubernetes itself.

So the next step was to package the Java application into a container and deploy it into the cluster.

The new setup looks like:

Kind Kubernetes Cluster

Greeting
    |
    v
Greeting Operator Pod
    |
    v
ConfigMap

This introduced a few new things for me:

building an executable JAR
creating a Docker image
loading the image into Kind
creating a ServiceAccount
understanding Kubernetes RBAC
using a ClusterRole
using a ClusterRoleBinding
deploying the operator as a Kubernetes Deployment

Building an Executable JAR

Previously, I was starting the controller locally using Maven:

task operator:run

Internally, this was running something similar to:

./mvnw \
  -f operators/greeting-operator/pom.xml \
  compile exec:java

For a container, I wanted something simpler:

java -jar greeting-operator.jar

So I configured Maven to create an executable JAR containing the dependencies required by the operator.

Now:

task operator:build

creates the application JAR under:

operators/greeting-operator/target/

This gives me a Java application that can be started independently from Maven.

Creating the Docker Image

The operator Dockerfile is kept under:

operators/greeting-operator/Dockerfile

The basic idea is simple:

FROM eclipse-temurin:26-jre

WORKDIR /app

COPY target/greeting-operator-0.1.0-SNAPSHOT.jar app.jar

USER 10001

ENTRYPOINT ["java", "-jar", "/app/app.jar"]

The build flow is now:

Java Source
    |
    v
Maven Build
    |
    v
Executable JAR
    |
    v
Docker Build
    |
    v
greeting-operator:dev

I wrapped this inside the Taskfile as well.

Build the image:

task operator:image:build

Since I am using Kind, the image also needs to be made available to the Kind cluster.

task operator:image:load

This loads the local image into the Kind nodes.

Deploying the Operator

I created a separate directory for the operator Kubernetes resources:

k8s/operator/
├── namespace.yaml
├── service-account.yaml
├── rbac.yaml
└── deployment.yaml

The operator itself runs inside:

platform-system

I created a separate namespace because I want platform components to be separate from application workloads.

apiVersion: v1
kind: Namespace

metadata:
  name: platform-system

Why Does the Operator Need a ServiceAccount?

When I was running the operator on my laptop, the Java process could access Kubernetes using my local kubeconfig.

Conceptually:

Java Controller
      |
      v
~/.kube/config
      |
      v
Kubernetes API

Once the application is running inside Kubernetes, I do not want to copy my personal kubeconfig into the container.

Instead, Kubernetes provides a workload identity using a ServiceAccount.

apiVersion: v1
kind: ServiceAccount

metadata:
  name: greeting-operator
  namespace: platform-system

Now the operator Pod runs using:

ServiceAccount: greeting-operator

The Deployment connects the Pod to that identity:

spec:
  template:
    spec:
      serviceAccountName: greeting-operator

So now the identity chain looks like:

Greeting Operator Pod
        |
        v
ServiceAccount
greeting-operator

But a ServiceAccount by itself does not automatically have permission to watch Greetings or create ConfigMaps.

That is where RBAC comes in.

Understanding Kubernetes RBAC

This part was initially a little confusing because Kubernetes has a few similarly named resources:

Role
ClusterRole
RoleBinding
ClusterRoleBinding

The easiest way I found to understand them is to separate two questions.

Question 1

What operations are allowed?

That is defined by:

Role
or
ClusterRole

Question 2

Who gets those permissions?

That is defined by:

RoleBinding
or
ClusterRoleBinding

So:

Role / ClusterRole
        =
permissions

RoleBinding / ClusterRoleBinding
        =
attach those permissions to somebody

The "somebody" could be:

User
Group
ServiceAccount

In this project, it is the operator's ServiceAccount.

Role vs ClusterRole

A Role belongs to one namespace.

For example, a Role created in:

namespace: default

can give permissions to resources in the default namespace.

Conceptually:

Role
namespace: default

can:
  get ConfigMaps
  create ConfigMaps

But only inside:

default

A ClusterRole, on the other hand, is not tied to one namespace.

For the Greeting operator, I created:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole

metadata:
  name: greeting-operator

The rules include permissions for our custom resource:

- apiGroups:
    - platform.shubforge.dev
  resources:
    - greetings
  verbs:
    - get
    - list
    - watch
    - patch
    - update

and permissions for ConfigMaps:

- apiGroups:
    - ""
  resources:
    - configmaps
  verbs:
    - get
    - list
    - watch
    - create
    - update
    - patch
    - delete

This defines what the operator is allowed to do.

But defining a ClusterRole alone does not give anybody those permissions.

We still need to bind it.

What Does ClusterRoleBinding Do?

This is the important part.

The operator runs using this ServiceAccount:

platform-system/greeting-operator

The ClusterRoleBinding connects that ServiceAccount to the ClusterRole.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding

metadata:
  name: greeting-operator

subjects:
  - kind: ServiceAccount
    name: greeting-operator
    namespace: platform-system

roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: greeting-operator

The relationship is:

Pod
 |
 v
ServiceAccount
platform-system/greeting-operator
 |
 v
ClusterRoleBinding
 |
 v
ClusterRole
 |
 +---- watch Greetings
 |
 +---- create ConfigMaps
 |
 +---- update ConfigMaps
 |
 +---- delete ConfigMaps

The ClusterRoleBinding does not contain the actual permissions.

It only says:

Give this subject the permissions defined by this ClusterRole.

How Does It Get Access to All Namespaces?

This was one of the things I specifically wanted to understand.

The important combination is:

ClusterRole
      +
ClusterRoleBinding

Our Greeting resource is namespaced.

For example, we could have:

default/hello

and:

team-a/hello

and:

team-b/hello

The ClusterRole says the operator may work with:

greetings
configmaps

And because that ClusterRole is attached using a ClusterRoleBinding, those permissions apply across namespaces.

Conceptually:

ClusterRoleBinding
        |
        v
ClusterRole
        |
        +--------------------------+
        |                          |
        v                          v
namespace: default          namespace: team-a
Greeting                    Greeting
ConfigMap                   ConfigMap
        |
        v
namespace: team-b
Greeting
ConfigMap

The ServiceAccount still belongs to:

platform-system

but that does not restrict where it can operate.

This is an important distinction.

The namespace of a ServiceAccount tells Kubernetes where the ServiceAccount object lives.

ServiceAccount
platform-system/greeting-operator

It does not mean:

This ServiceAccount can only access resources inside platform-system.

Its actual permissions depend on the RBAC bindings attached to it.

ClusterRoleBinding vs RoleBinding

Here is another useful comparison.

Suppose I have the same ClusterRole:

ClusterRole: greeting-operator

If I bind it using a ClusterRoleBinding:

ClusterRole
      |
      v
ClusterRoleBinding

the permissions apply across namespaces.

But Kubernetes also allows a RoleBinding to reference a ClusterRole.

For example:

kind: RoleBinding

metadata:
  name: greeting-operator
  namespace: team-a

with:

roleRef:
  kind: ClusterRole
  name: greeting-operator

In that case, the same ClusterRole permissions are restricted to:

team-a

So:

ClusterRole
   +
ClusterRoleBinding

= cluster-wide access

while:

ClusterRole
   +
RoleBinding in team-a

= permissions only in team-a

This was probably the easiest way for me to understand the difference.

A Simple RBAC Mental Model

I now think about it like this:

Role
    permissions inside one namespace

ClusterRole
    reusable permission definition
    that can work cluster-wide

RoleBinding
    give permissions inside one namespace

ClusterRoleBinding
    give permissions cluster-wide

For our operator:

ServiceAccount
platform-system/greeting-operator

        |
        v

ClusterRoleBinding

        |
        v

ClusterRole
greeting-operator

        |
        +---- Greetings
        |
        +---- ConfigMaps
        |
        v

all namespaces

One Important Detail: RBAC vs Watch Scope

RBAC controls what the operator is allowed to access.

It does not necessarily decide what the controller chooses to watch.

These are two separate ideas.

RBAC
=
What am I allowed to access?

Controller configuration
=
What do I actually watch?

For example, the ServiceAccount may have cluster-wide permission, but the controller could still be configured to only watch:

team-a

Similarly:

RBAC: all namespaces

Controller:
watch namespace = team-a

would mean the operator has more permission than it actually uses.

Ideally, permissions and controller scope should stay as close as possible.

For this learning project, I want the Greeting operator to work across namespaces, so using a ClusterRole with a ClusterRoleBinding keeps the setup simple.

For a production platform, I would think more carefully about least privilege and whether cluster-wide access is really needed.

Deploying the Operator

The operator itself runs as a regular Kubernetes Deployment.

apiVersion: apps/v1
kind: Deployment

metadata:
  name: greeting-operator
  namespace: platform-system

spec:
  replicas: 1

  selector:
    matchLabels:
      app: greeting-operator

  template:

    metadata:
      labels:
        app: greeting-operator

    spec:

      serviceAccountName: greeting-operator

      containers:

        - name: greeting-operator

          image: greeting-operator:dev

          imagePullPolicy: IfNotPresent

The important part from the RBAC perspective is:

serviceAccountName: greeting-operator

That connects the Pod to the ServiceAccount.

The complete authorization flow is now:

Greeting Operator Pod
        |
        v
ServiceAccount
        |
        v
ClusterRoleBinding
        |
        v
ClusterRole
        |
        v
Kubernetes API

Why Is There No Kubernetes Service?

I did not create a Kubernetes Service for the operator.

That is because the operator is not exposing an HTTP endpoint that other applications need to call.

The operator mainly communicates outward to the Kubernetes API.

Operator
   |
   v
Kubernetes API

It watches resources and reacts to events.

There is currently no need for:

Service
Ingress
Gateway

That may change later if the operator exposes metrics, health endpoints, admission webhooks, or some other network API.

But for this simple controller, a Deployment is enough.

Running Everything

The full flow now looks like this.

Create the cluster:

task cluster:create

Install the CRD:

task crd:install

Build the operator image:

task operator:image:build

Load it into Kind:

task operator:image:load

Deploy the operator:

task operator:deploy

Check its status:

task operator:status

The operator Pod should now be running inside:

platform-system

For example:

kubectl get pods -n platform-system

Now I can create the Greeting:

task greeting:create

and check:

kubectl get configmaps

The controller should create:

hello-greeting

Testing the RBAC Setup

I can also verify the ServiceAccount permissions directly using kubectl auth can-i.

For example:

kubectl auth can-i \
  list greetings.platform.shubforge.dev \
  --as=system:serviceaccount:platform-system:greeting-operator \
  --all-namespaces

Expected:

yes

For ConfigMaps:

kubectl auth can-i \
  create configmaps \
  --as=system:serviceaccount:platform-system:greeting-operator \
  --all-namespaces

Again:

yes

This is a useful way to understand what RBAC is actually allowing instead of only reading the YAML.

I can also test one particular namespace:

kubectl auth can-i \
  create configmaps \
  --namespace default \
  --as=system:serviceaccount:platform-system:greeting-operator

Testing Across Namespaces

Since the operator now has cluster-wide permissions, I can create another namespace:

kubectl create namespace demo

Then create a Greeting inside it:

apiVersion: platform.shubforge.dev/v1alpha1
kind: Greeting

metadata:
  name: hello
  namespace: demo

spec:
  message: "Hello from demo namespace"

After applying it, the controller should be able to create:

demo/hello-greeting

So the operator can remain in:

platform-system

while managing resources in:

default
demo
team-a
team-b
...

as long as its RBAC permissions and controller watch configuration allow it.

The Architecture Now

The project has now moved from:

Laptop

Java Controller
      |
      v
Kind Cluster

to:

Kubernetes Cluster

+------------------------------------+
|                                    |
| platform-system                    |
|                                    |
|  greeting-operator Pod             |
|        |                           |
|        v                           |
|  ServiceAccount                    |
|        |                           |
+--------|---------------------------+
         |
         v
  ClusterRoleBinding
         |
         v
    ClusterRole
         |
         v
+------------------------------------+
|                                    |
| Kubernetes API                     |
|                                    |
| Greeting ------> ConfigMap         |
|                                    |
+------------------------------------+

The controller is now a Kubernetes workload itself.

That makes the setup feel much closer to a real operator.

Source Code

The complete implementation is available in my Platform Lab repository.

Repository: Platform Lab

The changes for this step are available in:

Pull Request: Deploy Greeting Operator to Kubernetes

The repository contains:

Java Greeting controller
Greeting CRD
Dockerfile
Kind setup
ServiceAccount
ClusterRole
ClusterRoleBinding
operator Deployment
Taskfile commands

What's Next?

The operator is now running inside Kubernetes and has the permissions required to manage resources.

The current flow is:

Greeting
    |
    v
Java Operator
    |
    v
ConfigMap

and the operator itself is now running as:

Deployment
    |
    v
Pod
    |
    v
ServiceAccount
    |
    v
RBAC

The next things I want to explore are what makes an operator more production-like:

status updates
conditions
reconciliation behavior
error handling
retry behavior
owner references
finalizers
automated operator tests
GitHub Actions

For now, moving the controller from my laptop into Kubernetes helped me understand an important part of operators:

The controller is just another application, but Kubernetes gives it an identity and permissions to interact with the Kubernetes API.

Building My First Kubernetes Controller in Java

shubham goel — Fri, 25 Sep 2026 16:14:28 +0000

In the previous post, I created my first Kubernetes Custom Resource called Greeting.

It looked like this:

apiVersion: platform.shubforge.dev/v1alpha1
kind: Greeting

metadata:
  name: hello

spec:
  message: "Hello from Platform Lab"

After installing the CRD, Kubernetes understood the new resource.

I could run:

kubectl get greetings

and Kubernetes could store, validate, update, and delete the resource.

But there was one important thing missing.

Nothing actually happened when I created a Greeting.

That is what I wanted to solve next.

What I Want to Build

For now, I want to keep the controller very simple.

Whenever I create:

apiVersion: platform.shubforge.dev/v1alpha1
kind: Greeting

metadata:
  name: hello

spec:
  message: "Hello from Platform Lab"

I want the controller to create a ConfigMap like:

apiVersion: v1
kind: ConfigMap

metadata:
  name: hello-greeting

data:
  message: "Hello from Platform Lab"

So the flow becomes:

Greeting
    |
    v
Greeting Controller
    |
    v
ConfigMap

This is a very small example, but it gives me a way to understand how Kubernetes controllers actually work.

Creating the Java Operator

For the controller, I am using Java and the Java Operator SDK.

I created a separate module inside the project:

operators/
└── greeting-operator/
    ├── pom.xml
    └── src/
        └── main/
            └── java/

The Maven wrapper stays at the root of the repository:

platform-lab/
├── .mvn/
├── mvnw
├── Taskfile.yml
│
└── operators/
    └── greeting-operator/
        └── pom.xml

This allows me to keep one Maven wrapper for the repository.

The operator can be built using:

./mvnw \
  -f operators/greeting-operator/pom.xml \
  clean package

I have also wrapped this inside the Taskfile, so normally I just run:

task operator:build

Representing the Greeting in Java

The first thing the controller needs is a Java representation of the Kubernetes resource.

The CRD already defines:

Group:   platform.shubforge.dev
Version: v1alpha1
Kind:    Greeting

I represent the same thing in Java:

@Group("platform.shubforge.dev")
@Version("v1alpha1")
@Kind("Greeting")
@Plural("greetings")
public class Greeting
        extends CustomResource<GreetingSpec, GreetingStatus>
        implements Namespaced {
}

The important part here is that this Java class represents the custom Kubernetes resource.

The spec is represented by another class:

public class GreetingSpec {

    private String message;

    public String getMessage() {
        return message;
    }

    public void setMessage(String message) {
        this.message = message;
    }
}

So this YAML:

spec:
  message: "Hello from Platform Lab"

becomes:

greeting.getSpec().getMessage()

inside the controller.

For now, I also created an empty status class:

public class GreetingStatus {
}

I will come back to status handling later.

Describing the Desired ConfigMap

The next step is defining what should exist for a Greeting.

I created:

GreetingConfigMapDependentResource.java

The main part looks like this:

@KubernetesDependent
public class GreetingConfigMapDependentResource
        extends CRUDKubernetesDependentResource<ConfigMap, Greeting> {

    public GreetingConfigMapDependentResource() {
        super(ConfigMap.class);
    }

    @Override
    protected ConfigMap desired(
            Greeting greeting,
            Context<Greeting> context) {

        var name = greeting.getMetadata().getName();
        var namespace = greeting.getMetadata().getNamespace();

        return new ConfigMapBuilder()
                .withNewMetadata()
                    .withName(name + "-greeting")
                    .withNamespace(namespace)
                    .addToLabels(
                        "app.kubernetes.io/managed-by",
                        "greeting-operator")
                .endMetadata()
                .addToData(
                    "message",
                    greeting.getSpec().getMessage())
                .build();
    }
}

I found the desired() method interesting.

Instead of thinking:

Does ConfigMap exist?

If not:
    create it

If it exists:
    compare it

If message changed:
    update it

I describe what I want the ConfigMap to look like.

Greeting
    |
    v
desired()
    |
    v
ConfigMap

The controller framework then works toward keeping the actual state aligned with this desired state.

This is where reconciliation starts becoming easier to understand.

Creating the Reconciler

Next, I created the actual reconciler.

@Workflow(
    dependents = {
        @Dependent(
            type = GreetingConfigMapDependentResource.class
        )
    }
)
@ControllerConfiguration
public class GreetingReconciler
        implements Reconciler<Greeting> {

    private static final Logger log =
            LoggerFactory.getLogger(GreetingReconciler.class);

    @Override
    public UpdateControl<Greeting> reconcile(
            Greeting greeting,
            Context<Greeting> context) {

        log.info(
            "Reconciled Greeting {}/{}",
            greeting.getMetadata().getNamespace(),
            greeting.getMetadata().getName()
        );

        return UpdateControl.noUpdate();
    }
}

At first this looked slightly strange to me.

There is no code here saying:

createConfigMap();

That work is defined by the dependent resource.

The relationship is basically:

GreetingReconciler
        |
        v
GreetingConfigMapDependentResource
        |
        v
ConfigMap

The reconciler handles the Greeting, while the dependent resource describes the ConfigMap that should exist for it.

Starting the Operator

I then created a simple Java application:

public class GreetingOperatorApplication {

    public static void main(String[] args) {

        Operator operator = new Operator();

        operator.register(
            new GreetingReconciler()
        );

        operator.installShutdownHook();
        operator.start();
    }
}

For now, I am running the controller locally on my machine.

So the setup currently looks like:

My Machine
    |
    | Java Process
    |
    v
Greeting Controller
    |
    | kubeconfig
    |
    v
Kind Kubernetes Cluster

The controller uses my local Kubernetes configuration to communicate with the Kind cluster.

The operator itself is not running inside Kubernetes yet.

I want to keep that as a separate step.

Running the Controller

I added another command to the Taskfile:

task operator:run

Internally it runs:

./mvnw \
  -f operators/greeting-operator/pom.xml \
  compile exec:java

So the complete setup is now very small.

First create the cluster:

task cluster:create

Install the CRD:

task crd:install

Build the operator:

task operator:build

Run it:

task operator:run

I keep this terminal running.

Then from another terminal I create the Greeting:

task greeting:create

and verify it:

task greeting:get

The First Result

After creating the Greeting:

apiVersion: platform.shubforge.dev/v1alpha1
kind: Greeting

metadata:
  name: hello

spec:
  message: "Hello from Platform Lab"

I can check the ConfigMaps:

kubectl get configmaps

and now I see:

NAME             DATA   AGE
hello-greeting   1      10s

I can inspect it:

kubectl get configmap hello-greeting -o yaml

and see:

data:
  message: Hello from Platform Lab

So now the complete flow is working:

Greeting YAML
      |
      v
Kubernetes API
      |
      v
Greeting Resource
      |
      v
Java Controller
      |
      v
ConfigMap

This was the point where the idea of a Kubernetes controller started becoming much clearer to me.

What Happens If I Change the Greeting?

Now I can change the resource:

spec:
  message: "Hello from the updated Greeting"

and apply it again:

task greeting:create

The desired state has changed.

The controller sees the new Greeting and updates the ConfigMap.

Now:

kubectl get configmap hello-greeting -o yaml

contains:

data:
  message: Hello from the updated Greeting

So we now have:

Greeting changed
      |
      v
Controller reconciles
      |
      v
ConfigMap changed

What If I Delete the ConfigMap?

This is the part I found even more interesting.

I manually delete the ConfigMap:

kubectl delete configmap hello-greeting

At this point the actual state no longer matches what the controller expects.

The Greeting still exists:

Greeting: hello

and according to our controller, this means this ConfigMap should exist:

ConfigMap: hello-greeting

The controller notices that the managed resource is missing and creates it again.

So instead of:

Create ConfigMap once

the behavior is closer to:

Desired State
     |
     v
Compare with Actual State
     |
     v
Make them match

This is reconciliation.

CRD vs Controller

This experiment also helped me understand the difference between a CRD and a controller.

The CRD gives Kubernetes the API:

Greeting

The controller gives that API behavior:

Greeting
    |
    v
Controller
    |
    v
ConfigMap

Without the controller:

Greeting
    |
    v
Stored in Kubernetes

With the controller:

Greeting
    |
    v
Something actually happens

That distinction was probably the most useful part of this experiment for me.

Why I Am Using a Taskfile

As I add more steps, the number of commands also starts increasing.

Without the Taskfile, I would need to remember things like:

kind create cluster ...
kubectl apply -f ...
./mvnw -f operators/greeting-operator/pom.xml ...
kubectl get ...

Instead, I now have commands like:

task cluster:create
task crd:install
task operator:build
task operator:run
task greeting:create
task greeting:get

I can see everything available using:

task --list

The Taskfile is not doing anything special to Kubernetes.

It is just giving the repository a simple and consistent developer interface.

As the project grows, I want to keep most common operations available through:

task <something>

instead of maintaining setup commands separately.

Source Code

I am keeping the complete implementation in my Platform Lab repository.

Repository: Platform Lab

The repository contains the complete project setup, including:

Kind cluster configuration
SDKMAN Java setup
Taskfile
Greeting CRD
sample Greeting
Java controller
ConfigMap dependent resource

The changes for this post can be seen separately in:

Pull Request: Add Greeting Controller

This makes it easier to see exactly what changed at this stage without looking through later changes in the repository.

What's Next?

Right now, the controller runs as a Java process on my machine:

Laptop
   |
   v
Java Controller
   |
   v
Kind Cluster

The next thing I want to do is run the controller inside Kubernetes itself.

That will introduce a few new pieces:

Greeting Operator
      |
      v
Docker Image
      |
      v
Kubernetes Deployment
      |
      v
ServiceAccount
      |
      v
RBAC

I also want to understand why the controller needs permissions to watch Greeting resources and create ConfigMaps.

After that, I can start looking more deeply into reconciliation, status, error handling, and testing.

For now, the small flow is working:

Greeting
    |
    v
Java Controller
    |
    v
ConfigMap

And that already makes Kubernetes operators much easier for me to understand.

Building My First Kubernetes Custom Resource

shubham goel — Fri, 25 Sep 2026 13:31:27 +0000

In my previous post, I wrote about how I started looking at Kubernetes not only as something to deploy applications on, but also as something we can use to build platforms.

The next thing I wanted to understand was how Kubernetes can be extended.

Normally, we work with resources like:

Pod
Deployment
Service
ConfigMap
Secret

But what if I want to create my own Kubernetes resource?

Something like:

apiVersion: platform.shubforge.dev/v1alpha1
kind: Greeting

metadata:
  name: hello

spec:
  message: "Hello from Platform Lab"

Kubernetes does not know what a Greeting is by default.

This is where Custom Resource Definitions, or CRDs, come in.

What is a CRD?

A Custom Resource Definition allows us to extend the Kubernetes API with our own resource types.

For example, after creating a Greeting CRD, I can run:

kubectl get greetings

just like I normally run:

kubectl get pods

This is what I found interesting.

We are not creating a completely separate API.

We are extending the Kubernetes API itself.

Creating the Greeting CRD

For my first experiment, I wanted to keep the resource very simple.

A Greeting will only contain a message.

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition

metadata:
  name: greetings.platform.shubforge.dev

spec:
  group: platform.shubforge.dev

  scope: Namespaced

  names:
    plural: greetings
    singular: greeting
    kind: Greeting
    shortNames:
      - greet

  versions:
    - name: v1alpha1
      served: true
      storage: true

      schema:
        openAPIV3Schema:
          type: object

          properties:
            spec:
              type: object

              properties:
                message:
                  type: string

              required:
                - message

There are a few important things here.

The resource is called:

Greeting

Its API group is:

platform.shubforge.dev

and the first API version is:

v1alpha1

I am using v1alpha1 because this API is still experimental and will probably change while I continue learning and building the project.

The resource is also Namespaced, which means a Greeting belongs to a Kubernetes namespace, just like resources such as Deployments and ConfigMaps.

Installing the CRD

For this project, I am running Kubernetes locally using Kind.

Once the cluster is running, the CRD can be installed using:

kubectl apply -f k8s/crds/greetings.yaml

After installing it, I can check the available CRDs:

kubectl get crds

Now Kubernetes knows about:

greetings.platform.shubforge.dev

At this point, Greeting has become a valid Kubernetes resource type.

I can also check it directly:

kubectl get crd greetings.platform.shubforge.dev

Creating My First Greeting

Now I can create an actual Greeting.

apiVersion: platform.shubforge.dev/v1alpha1
kind: Greeting

metadata:
  name: hello

spec:
  message: "Hello from Platform Lab"

I have this sample resource under:

k8s/samples/greeting.yaml

After applying it:

kubectl apply -f k8s/samples/greeting.yaml

I can run:

kubectl get greetings

and get something similar to:

NAME    AGE
hello   10s

This now feels similar to working with any other Kubernetes resource.

I can also inspect it using:

kubectl describe greeting hello

or:

kubectl get greeting hello -o yaml

Because I added a short name in the CRD:

shortNames:
  - greet

I can also use:

kubectl get greet

Kubernetes Can Also Validate It

Another useful thing about CRDs is that we can define the structure of our custom resource.

For the Greeting resource, I defined:

message:
  type: string

and made it required:

required:
  - message

So if I try to create a Greeting without a message, Kubernetes will reject it.

For example, this is not valid:

apiVersion: platform.shubforge.dev/v1alpha1
kind: Greeting

metadata:
  name: invalid

There is no:

spec:
  message:

but our CRD says that message is required.

This helped me understand that a CRD is not only about giving Kubernetes a new resource name.

We can also define the structure and validation rules for our API.

Using a Taskfile

While working on this project, I also wanted to avoid repeatedly typing the same commands.

So I added a Taskfile.yml at the root of the repository.

For example, instead of remembering commands like:

kind create cluster ...
kubectl apply -f k8s/crds/
kubectl apply -f k8s/samples/greeting.yaml
kubectl get greetings

I can use simpler project commands:

task cluster:create
task crd:install
task greeting:create
task greeting:get

The Taskfile is only a convenience around tools like kind and kubectl.

It gives me one common way to run the project locally.

As the project grows, this should also help avoid having setup instructions spread across multiple scripts and commands.

I can see all the available tasks using:

task --list

Running the Example

The current example can be started with a few commands.

First, create the local Kind cluster:

task cluster:create

Install the CRD:

task crd:install

Create the sample Greeting:

task greeting:create

Check the created resource:

task greeting:get

To inspect it in more detail:

task greeting:describe

The flow currently looks like:

task cluster:create
        |
        v
Kind Kubernetes Cluster
        |
        v
task crd:install
        |
        v
Greeting CRD
        |
        v
task greeting:create
        |
        v
Greeting Resource

When I am finished testing, I can clean everything up using:

task greeting:delete
task crd:delete
task cluster:delete

But What Does Greeting Actually Do?

Right now, nothing.

And this was another important thing for me to understand.

We currently have:

Greeting
    |
    v
Kubernetes API
    |
    v
Stored in Kubernetes

Kubernetes now understands the Greeting resource.

It can:

validate it
store it
retrieve it
update it
delete it
allow something else to watch it

But creating this:

spec:
  message: "Hello from Platform Lab"

does not create anything else.

There is no Pod.

There is no ConfigMap.

There is no Deployment.

A CRD gives us the API.

We still need something that provides the behavior behind that API.

For that, we need a controller.

That is what I want to build next.

The next step will look something like:

Greeting
    |
    v
Greeting Controller
    |
    v
ConfigMap

The controller will watch Greeting resources and create a ConfigMap containing the greeting message.

For example:

apiVersion: platform.shubforge.dev/v1alpha1
kind: Greeting

metadata:
  name: hello

spec:
  message: "Hello from Platform Lab"

could eventually result in something like:

Greeting: hello
       |
       v
Greeting Controller
       |
       v
ConfigMap: hello
       |
       v
message=Hello from Platform Lab

I also want to see what happens if I manually delete that ConfigMap.

If the controller creates it again, that will take me into another important Kubernetes concept: reconciliation.

Source Code

I am keeping the complete setup and code in my Platform Lab GitHub repository.

Repository: Platform Lab

The repository contains the complete setup, including:

local Kind cluster configuration
SDKMAN Java configuration
Taskfile commands
Kubernetes CRD definitions
sample resources
project documentation

The specific changes covered in this post are available in the Greeting CRD pull request:

Pull Request: Add Greeting CRD

I plan to continue using the same repository while I learn and build more around Kubernetes operators and platform engineering.

What's Next?

So far, I have only created the API.

Greeting
    |
    v
Kubernetes API

The next step is to add the behavior.

Greeting
    |
    v
Greeting Controller
    |
    v
ConfigMap

I will build a simple controller that watches Greeting resources and creates a ConfigMap from them.

That should help me understand how Kubernetes controllers actually work and why reconciliation is such an important part of Kubernetes.

Understanding Kubernetes Beyond Pods and Deployments

shubham goel — Wed, 23 Sep 2026 13:16:46 +0000

I have been learning more about Kubernetes recently, and one thing that helped me understand it better is that Kubernetes is not just about running containers.

At first, I was mostly thinking like this:

Application -> Docker image -> Pod -> Deployment

But Kubernetes is more about defining what state you want your system to be in.

For example, we can say:

I want 3 instances of my application
I want this application to be accessible through a service
I want some configs and secrets available to it
I want Kubernetes to restart it if something goes wrong

Kubernetes controllers keep checking the current state and try to make it match the state we defined.

This also makes Kubernetes very extensible.

Using CRDs and operators, we can create our own resources and build higher level abstractions on top of Kubernetes.

For example, instead of developers creating deployments, services, configmaps and other resources separately, we can define something like:

ApplicationRelease: 1.2.3

and let the controller create and manage everything needed for that application.

I am currently exploring more around this idea of using Kubernetes not just for deploying applications, but also for building platforms on top of it.

Will keep sharing what I learn around Kubernetes, Java, Spring Boot and distributed systems.

DEV Community: shubham goel

From Provisioned to Ready: Tracking Application Health in My Kubernetes Platform

Provisioned Does Not Mean Ready

Where Should Readiness Come From?

What Deployment Status Gives Me

Adding Readiness to Application Status

Ready=False

Ready=True

Separating Readiness Logic

The Basic Readiness Rule

Why Deployment Generation Matters

Continuous Readiness

The Platform Operator Does Not Heal Pods

Why I Am Not Watching Pods Yet

Ready Condition

lastTransitionTime

kubectl Output

Adding Java Unit Tests

Running the Tests

Current Platform Architecture

What This Still Does Not Tell Me

Source Code

What I Learned

What's Next?

Making My Platform API Reconcile Application Updates

Application Is Desired State

Updating the Application

The Interesting Part: Almost No New Controller Logic

Creation and Update Are the Same Reconciliation Problem

Verifying the Deployment Update

Verifying the Service Update

Were the Resources Recreated?

Checking the Service Identity Too

observedGeneration

Extending the Integration Test

What the Test Verifies

Why the Test Waits

The Current Platform Lifecycle

Desired State vs Provisioning Request

Current Architecture

What This Still Does Not Mean

Source Code

What I Learned

What's Next?

Turning My Platform API Into Real Kubernetes Resources

What I Wanted to Build

Creating the Platform Operator

Mapping the Application CRD to Java

The First Real Translation

Creating the Deployment

Creating the Service

Why I Used Managed Dependent Resources

The Application Reconciler

Initial Application Status

Why Ready Is Not Implemented Yet

Running the Platform Operator in Kubernetes

RBAC

Creating the First Application

Testing the Controller

What the Controller Test Checks

Why I Started With a Shell Integration Test

Test Isolation

Running the Tests

Current Platform Architecture

What This Version Does Not Handle Yet

Source Code

What I Learned

What's Next?

Building the First Real API for My Kubernetes Platform

What I Want the Platform to Do

Starting With the Application CRD

The CRD

Why port Is an Object

Defining Status Early

Creating the First Application

Introducing Tests From the Beginning

Testing the Application API

Testing a Valid Application

Testing an Invalid Port

Testing a Missing Image

Why `port` Is an Object