In the previous post, I explored what happens when my Kubernetes operator fails.
I created a real failure by removing ConfigMap creation permission from the operator.
The controller failed, retried several times, and eventually stopped retrying.
Then I restored the RBAC permissions.
But something unexpected happened.
Nothing happened.
The operator now had permission to create the ConfigMap again, but the failed Greeting was not immediately reconciled.
That led me to another question:
How can I manually ask the controller to reconcile a resource again without changing its actual specification?
That is what I wanted to solve next.
The Problem
The failure flow looked like this:
Greeting created
|
v
ConfigMap creation fails
|
v
Ready=False
|
v
automatic retries
|
v
retry limit reached
Then I fixed the external problem:
RBAC fixed
But fixing the ClusterRole did not produce a new event for the Greeting controller.
So the resource remained in its failed state.
Previously I recovered by changing:
spec:
message:
That worked because changing the spec caused another reconciliation.
But changing business configuration only to make the controller run again did not feel right.
I wanted something more explicit.
A Manual Reconciliation Annotation
I decided to use a Kubernetes annotation:
platform.shubforge.dev/reconcile-at
The idea is simple.
If I run:
kubectl annotate greeting hello \
platform.shubforge.dev/reconcile-at="$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
--overwrite
the annotation gets a new timestamp.
For example:
metadata:
annotations:
platform.shubforge.dev/reconcile-at: "2026-09-29T07:00:00Z"
Every time I want another reconciliation, I update this value.
Conceptually:
reconcile-at changed
|
v
Greeting updated
|
v
Controller
|
v
reconcile()
This means I can request another reconciliation without changing:
spec:
But Annotation Changes Were Being Ignored
There was one more problem.
The Java Operator SDK normally avoids triggering reconciliation for every metadata-only update.
For example:
status changed
resourceVersion changed
annotation changed
should not necessarily cause the controller to run again.
For normal desired-state changes, Kubernetes updates:
metadata.generation
So the controller can react when the actual spec changes.
But my manual reconciliation only changes an annotation.
That means:
generation
does not change.
So I needed a custom update filter.
Creating a Greeting Update Filter
I added:
GreetingUpdateFilter.java
The goal is simple.
Reconcile when either:
spec generation changed
or:
reconcile-at annotation changed
Conceptually:
generation changed?
|
yes
|
v
reconcile
reconcile-at changed?
|
yes
|
v
reconcile
only status changed?
|
no
|
v
ignore
The filter looks roughly like this:
public class GreetingUpdateFilter
implements OnUpdateFilter<Greeting> {
public static final String RECONCILE_AT_ANNOTATION =
"platform.shubforge.dev/reconcile-at";
@Override
public boolean accept(
Greeting newResource,
Greeting oldResource) {
return generationChanged(newResource, oldResource)
|| reconcileAnnotationChanged(
newResource,
oldResource
);
}
private boolean generationChanged(
Greeting newResource,
Greeting oldResource) {
return !Objects.equals(
newResource.getMetadata().getGeneration(),
oldResource.getMetadata().getGeneration()
);
}
private boolean reconcileAnnotationChanged(
Greeting newResource,
Greeting oldResource) {
return !Objects.equals(
annotationValue(newResource),
annotationValue(oldResource)
);
}
private String annotationValue(
Greeting greeting) {
var annotations =
greeting.getMetadata().getAnnotations();
if (annotations == null) {
return null;
}
return annotations.get(
RECONCILE_AT_ANNOTATION
);
}
}
The important part is:
generationChanged(...)
||
reconcileAnnotationChanged(...)
This keeps the controller focused on meaningful updates.
Why Not Reconcile on Every Update?
One option would have been to simply accept every resource update.
Something like:
return true;
But that would be too broad.
The operator itself updates:
status:
and status updates also change the Kubernetes resource.
If every update triggered another reconciliation, it could create unnecessary reconciliation cycles.
So I want this:
spec changed
|
v
reconcile
and this:
manual reconcile annotation changed
|
v
reconcile
but not:
status changed
|
v
reconcile again
|
v
status changed
|
v
reconcile again
The custom filter lets me control that.
Configuring the Controller
I then configured the GreetingReconciler to use the custom update filter.
Conceptually:
@ControllerConfiguration(
defaultFilters = false,
informer = @Informer(
onUpdateFilter = GreetingUpdateFilter.class
)
)
The controller already had the managed ConfigMap workflow, so the overall structure becomes:
Greeting
|
v
Update Filter
|
+------ generation changed
|
+------ reconcile-at changed
|
v
GreetingReconciler
|
v
ConfigMap
Adding a Taskfile Command
Since I am already using Task as the developer interface for the project, I added:
task greeting:reconcile
The task updates the annotation with the current timestamp.
Conceptually:
greeting:reconcile:
desc: Manually trigger Greeting reconciliation
cmds:
- |
kubectl annotate greeting {{.GREETING_NAME}} \
platform.shubforge.dev/reconcile-at="$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
--overwrite
By default:
task greeting:reconcile
reconciles:
hello
For another Greeting:
task greeting:reconcile \
GREETING_NAME=failure-test
This makes manual reconciliation much easier to use during development.
Testing It
I first tested it on the healthy hello Greeting.
I ran:
task greeting:reconcile
Then checked the annotation:
kubectl get greeting hello \
-o jsonpath='{.metadata.annotations.platform\.shubforge\.dev/reconcile-at}'
The annotation contained the new timestamp.
Then I checked the Greeting Events:
kubectl describe greeting hello
and saw:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Reconciled 5s (x2 over 2m12s) greetingreconciler Managed ConfigMap hello-greeting is in the desired state
The interesting part was:
x2
Kubernetes had aggregated two equivalent Reconciled Events.
One came from the normal reconciliation.
The second came from my manual reconciliation request.
So the annotation was doing exactly what I wanted.
Why Observed Generation Stayed at 1
Something else was interesting.
After manual reconciliation, I still had:
Observed Generation: 1
At first this might look strange.
But it is actually correct.
The manual reconciliation changed:
metadata:
annotations:
It did not change:
spec:
So Kubernetes did not increment:
metadata.generation
Before manual reconciliation:
generation = 1
observedGeneration = 1
After manual reconciliation:
generation = 1
observedGeneration = 1
The operator simply reconciled generation 1 again.
This gave me a useful distinction.
generation
=
version of the desired spec
while:
reconcile-at
=
request to re-evaluate
the current desired spec
So:
spec changed
|
v
new generation
|
v
reconcile
is different from:
reconcile-at changed
|
v
same generation
|
v
reconcile again
Testing the Real Recovery Scenario
The healthy-resource test proved that the annotation worked.
But the real reason I added this feature was failure recovery.
So I repeated the RBAC failure experiment.
First, I edited the operator's ClusterRole:
kubectl edit clusterrole greeting-operator
The normal ConfigMap permissions looked like:
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
For the experiment I temporarily changed them to:
verbs:
- get
- list
- watch
Then I verified:
kubectl auth can-i \
create configmaps \
--namespace default \
--as=system:serviceaccount:platform-system:greeting-operator
and got:
no
Creating a Test Greeting
I created another Greeting directly:
kubectl apply -f - <<'EOF'
apiVersion: platform.shubforge.dev/v1alpha1
kind: Greeting
metadata:
name: manual-reconcile-test
spec:
message: "Manual reconciliation test"
EOF
The operator attempted to create:
manual-reconcile-test-greeting
but received:
403 Forbidden
The Greeting eventually moved to:
Ready=False
and automatic retries started.
After the retry limit was reached, it remained failed.
Fixing the External Problem
I restored the repository RBAC configuration:
task operator:deploy
Then verified:
kubectl auth can-i \
create configmaps \
--namespace default \
--as=system:serviceaccount:platform-system:greeting-operator
This time:
yes
Previously, at this point I had changed:
spec:
to trigger another reconciliation.
This time I did not touch the specification.
Instead I ran:
task greeting:reconcile \
GREETING_NAME=manual-reconcile-test
The annotation changed.
The update filter accepted the update.
The controller reconciled the same desired state again.
And this time ConfigMap creation succeeded.
Recovery Flow
The new recovery flow became:
ConfigMap creation fails
|
v
Ready=False
|
v
automatic retries
|
v
retry limit reached
|
v
external issue fixed
|
v
manual reconcile requested
|
v
reconcile-at annotation changes
|
v
update filter accepts event
|
v
reconcile()
|
v
ConfigMap created
|
v
Ready=True
This feels much cleaner than modifying application configuration only to wake the controller up.
Different Ways Reconciliation Happens
At this point my controller can be reconciled for several different reasons.
Spec Change
Greeting.spec changes
|
v
generation changes
|
v
reconcile
Managed Resource Change
ConfigMap changes or disappears
|
v
dependent resource event
|
v
reconcile
Failure Retry
reconciliation fails
|
v
automatic retry
Manual Reconciliation
reconcile-at changes
|
v
update filter
|
v
reconcile
All of them eventually call the same controller logic.
That also reinforces another important idea:
Reconciliation should be safe to run multiple times.
The controller should keep trying to move actual state toward desired state rather than assuming it runs only once.
Current Flow
The operator now looks roughly like:
Greeting
|
+-------------+-------------+
| |
spec change reconcile-at change
| |
v v
generation change update filter
| |
+-------------+-------------+
|
v
reconcile()
|
+------+------+
| |
success failure
| |
v v
ConfigMap Ready=False
| |
v v
Ready=True Warning Event
| |
v v
Normal Event Retry
Source Code
The complete implementation is available in my Platform Lab repository.
Repository: Platform Lab
The changes covered in this post are available in:
Pull Request: Add Manual Greeting Reconciliation
The main changes include:
GreetingUpdateFilter-
reconcile-atannotation support - manual reconciliation Taskfile command
- filtering spec changes and manual reconciliation separately
What I Learned
The main thing I learned from this step is that:
desired state changed
and:
reconcile the current desired state again
are two different operations.
Changing spec means:
I want something different.
Manual reconciliation means:
I still want the same thing.
Please check it again.
That is why keeping the manual trigger in metadata rather than the resource specification feels like a better fit.
It also helped me understand why:
generation = 1
can remain unchanged even though the controller ran multiple times.
The generation represents the desired configuration, not the number of reconciliation attempts.
What's Next?
There is still more I want to understand around reconciliation itself.
Some areas I want to explore are:
idempotency
periodic reconciliation
rescheduling
external dependency handling
event filtering
reconciliation best practices
After that, I want to move into another important part of the operator lifecycle:
owner references
resource deletion
finalizers
For now, the Greeting operator supports both automatic reconciliation and an explicit way to ask it to evaluate the current desired state again.
Top comments (0)