In the previous post, I added status and conditions to my Greeting custom resource.
That gave me something like:
status:
observedGeneration: 1
configMapName: hello-greeting
conditions:
- type: Ready
status: "True"
reason: ConfigMapReady
message: Managed ConfigMap hello-greeting is in the desired state
This was useful for the successful path.
But until now, everything was mostly working.
So the next question was:
What happens when reconciliation fails?
For example, what if the operator cannot create the ConfigMap?
That led me into failure handling, retry behavior, and Kubernetes Events.
Current Flow
The normal flow looks like this:
Greeting
|
v
Greeting Operator
|
v
ConfigMap
|
v
Ready=True
But I also want the failed path to be visible:
Greeting
|
v
Greeting Operator
|
v
Failure
|
+----> Ready=False
|
+----> Warning Event
|
+----> Retry
The goal is not only to log the exception.
The custom resource itself should explain that something went wrong.
Adding Failure Status
When reconciliation fails, I update the Greeting status to:
status:
conditions:
- type: Ready
status: "False"
reason: ReconciliationFailed
message: ...
Conceptually:
reconcile()
|
v
exception
|
v
update status
|
v
Ready=False
The full exception still goes to the operator logs.
But the custom resource gets a shorter and more useful message.
I think about it like this:
Operator logs
=
developer debugging
while:
status.message
=
resource-level information
Error Status Handling
The reconciler now handles failures using the Java Operator SDK error status mechanism.
The flow is roughly:
@Override
public ErrorStatusUpdateControl<Greeting> updateErrorStatus(
Greeting greeting,
Context<Greeting> context,
Exception exception) {
var status = greeting.getStatus();
if (status == null) {
status = new GreetingStatus();
}
status.setObservedGeneration(
greeting.getMetadata().getGeneration()
);
status.setConditions(
List.of(
createCondition(
status,
greeting.getMetadata().getGeneration(),
"False",
"ReconciliationFailed",
errorMessage(exception)
)
)
);
greeting.setStatus(status);
return ErrorStatusUpdateControl.patchStatus(greeting);
}
So now the resource can move between:
Ready=True
and:
Ready=False
depending on the reconciliation result.
Recording Kubernetes Events
I also wanted failures to be visible through normal Kubernetes tooling.
So the operator now records Kubernetes Events.
For a successful reconciliation:
Normal Reconciled
For a failure:
Warning ReconciliationFailed
This means I can run:
kubectl describe greeting hello
and see what happened directly on the resource.
For example:
Events:
Type Reason Message
---- ------ -------
Normal Reconciled Managed ConfigMap hello-greeting is in the desired state
During a failure:
Events:
Type Reason Message
---- ------ -------
Warning ReconciliationFailed ...
This is much nicer than requiring someone to immediately search operator logs.
RBAC for Events
Because the operator now creates Kubernetes Events, it also needs permission to do that.
I added:
- apiGroups:
- ""
resources:
- events
verbs:
- get
- create
- patch
So the operator permissions now include:
watch Greetings
update Greeting status
manage ConfigMaps
record Kubernetes Events
Creating a Real Failure
Instead of adding something artificial like:
spec:
fail: true
I wanted to create a real Kubernetes authorization failure.
The Greeting operator normally has permission to create ConfigMaps through its ClusterRole.
I opened the ClusterRole directly:
kubectl edit clusterrole greeting-operator
The ConfigMap permissions normally look something like:
- apiGroups:
- ""
resources:
- configmaps
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
For this experiment, I temporarily removed the write permissions:
- apiGroups:
- ""
resources:
- configmaps
verbs:
- get
- list
- watch
Now the operator could still read and watch ConfigMaps, but it could no longer create one.
I verified this using:
kubectl auth can-i \
create configmaps \
--namespace default \
--as=system:serviceaccount:platform-system:greeting-operator
The expected result was:
no
Creating a New Greeting Directly
Now I wanted the operator to attempt creation of a brand new ConfigMap.
Instead of modifying my existing sample YAML, I created a temporary Greeting directly from the terminal:
kubectl apply -f - <<'EOF'
apiVersion: platform.shubforge.dev/v1alpha1
kind: Greeting
metadata:
name: failure-test
spec:
message: "This should fail temporarily"
EOF
Here:
kubectl apply -f -
means that kubectl reads the manifest from standard input instead of a file.
This is useful for small temporary experiments because I do not need to add another YAML file to the repository.
I checked the new resource:
kubectl get greeting failure-test
and then its full YAML:
kubectl get greeting failure-test -o yaml
The operator tried to create:
failure-test-greeting
but Kubernetes rejected the request because the ServiceAccount no longer had permission to create ConfigMaps.
The failure was a real:
403 Forbidden
The Greeting Moved to Ready=False
The resource status eventually showed:
status:
conditions:
- type: Ready
status: "False"
reason: ReconciliationFailed
So the flow became:
Greeting
|
v
Operator
|
v
create ConfigMap
|
v
403 Forbidden
|
v
Ready=False
I could also inspect the Kubernetes Events:
kubectl describe greeting failure-test
and see something like:
Events:
Type Reason Message
---- ------ -------
Warning ReconciliationFailed ...
The Operator Was Retrying
One thing I initially misunderstood was what happened after the failure.
The Events output showed:
Warning ReconciliationFailed 5m12s (x6 over 5m31s)
The important part was:
x6
That meant the operator had not failed only once.
It had retried several times.
The actual flow was closer to:
Attempt 1
|
v
403 Forbidden
|
v
retry
Attempt 2
|
v
403 Forbidden
|
v
retry
Attempt 3
|
v
...
retry limit reached
So automatic retry was already working.
The operator simply does not retry forever.
Retry vs Reconciliation Trigger
This experiment helped me understand an important difference.
A retry means:
The previous reconciliation failed,
so try it again.
A reconciliation trigger means:
Something relevant changed,
so evaluate the resource again.
These are not the same thing.
The automatic retries eventually stopped after the retry limit was reached.
At that point, the Greeting remained:
Ready=False
until another relevant event happened.
Restoring the Permissions
After observing the failure and retries, I restored the correct RBAC permissions.
Instead of manually editing the ClusterRole back again, I reapplied the operator manifests from the repository:
task operator:deploy
This restored the ConfigMap permissions defined in:
k8s/operator/rbac.yaml
I verified that the operator could create ConfigMaps again:
kubectl auth can-i \
create configmaps \
--namespace default \
--as=system:serviceaccount:platform-system:greeting-operator
This time the result was:
yes
But something interesting happened.
The ConfigMap was still not created immediately.
Why Fixing RBAC Did Not Trigger Reconciliation
The retries had already been exhausted.
Changing:
ClusterRole
does not produce an event for the Greeting controller.
The controller is mainly watching resources relevant to its reconciliation flow, such as:
Greeting
ConfigMap
It is not watching:
ClusterRole
So this happened:
ConfigMap creation fails
|
v
retries exhausted
|
v
RBAC fixed
|
v
ClusterRole changed
|
v
no Greeting event
|
v
no immediate reconciliation
The permission problem was fixed, but nothing told the controller to evaluate failure-test again.
Triggering Another Reconciliation
For this experiment, I changed the Greeting.spec.
I ran:
kubectl patch greeting failure-test \
--type=merge \
-p '{"spec":{"message":"Recovery test"}}'
This changed the desired resource.
Kubernetes increased the resource generation, which produced a new event for the controller.
Now the flow was:
RBAC fixed
|
v
Greeting spec changed
|
v
generation increased
|
v
new reconciliation
|
v
ConfigMap creation retried
|
v
success
This time the operator had permission to create the ConfigMap.
Checking the Recovery
I verified the ConfigMap:
kubectl get configmap failure-test-greeting
and then inspected the Greeting:
kubectl get greeting failure-test -o yaml
The status had moved back to:
conditions:
- type: Ready
status: "True"
reason: ConfigMapReady
The Events now showed both parts of the story:
kubectl describe greeting failure-test
with output similar to:
Warning ReconciliationFailed 5m12s (x6 over 5m31s)
Normal Reconciled 8s Managed ConfigMap failure-test-greeting is in the desired state
That was probably the most useful part of this experiment.
The event history showed:
failure
|
v
automatic retries
|
v
retry limit reached
|
v
external problem fixed
|
v
new reconciliation
|
v
recovery
Failure and Recovery Lifecycle
The complete experiment looked like this:
Greeting created
|
v
ConfigMap creation fails
|
v
Ready=False
|
v
Warning Event
|
v
automatic retries
|
v
retry limit reached
|
v
RBAC fixed
|
v
no reconciliation yet
|
v
Greeting spec updated
|
v
new reconciliation
|
v
ConfigMap created
|
v
Ready=True
|
v
Normal Event
This gave me a much clearer picture of how operator failure handling works.
Status, Events and Logs
At this point, the operator exposes information in three different places.
Logs
Useful for debugging the operator itself.
stack traces
internal errors
implementation details
Status
Useful for understanding the current state of the custom resource.
type: Ready
status: "False"
reason: ReconciliationFailed
Events
Useful for understanding important things that happened to the resource.
Warning ReconciliationFailed
Normal Reconciled
So I now think about them like this:
Logs
|
v
developer debugging
Status
|
v
current resource state
Events
|
v
important resource history
Each solves a different problem.
Successful Flow
The normal path is now:
Greeting
|
v
Operator
|
v
ConfigMap
|
+----> Ready=True
|
+----> Normal Reconciled Event
Example:
status:
configMapName: hello-greeting
observedGeneration: 1
conditions:
- type: Ready
status: "True"
reason: ConfigMapReady
message: Managed ConfigMap hello-greeting is in the desired state
Failure Flow
The failed path looks like:
Greeting
|
v
Operator
|
v
Failure
|
+----> Ready=False
|
+----> Warning Event
|
+----> Retry
If retries keep failing:
failure
|
v
retry
|
v
failure
|
v
retry
|
v
retry limit reached
The resource then waits for another reconciliation trigger.
Taskfile Commands
I added a command for checking Greeting Events:
task greeting:events
which is basically:
kubectl describe greeting hello
The useful Greeting commands now include:
task greeting:create
task greeting:get
task greeting:status
task greeting:events
For the operator:
task operator:image:build
task operator:image:load
task operator:deploy
task operator:restart
task operator:status
task operator:logs
The Taskfile continues to be the main developer interface for the project.
Current Architecture
The current operator flow now looks like:
Greeting
|
v
Greeting Operator
|
+------+------+
| |
success failure
| |
v v
ConfigMap Ready=False
| |
v v
Ready=True Warning Event
| |
v v
Normal Event Retry
The operator is now handling more than resource creation.
It is also exposing:
state
failures
retry behavior
events
recovery
through Kubernetes-native mechanisms.
Source Code
The complete implementation is available in my Platform Lab repository.
Repository: Platform Lab
The changes covered in this post are available in:
Pull Request: Add Failure Handling and Kubernetes Events
The project currently includes:
- Greeting CRD
- Java Operator SDK controller
- ConfigMap dependent resource
- status and conditions
- Docker packaging
- Kubernetes deployment
- ServiceAccount and RBAC
- failure handling
- automatic retries
- Kubernetes Events
What I Learned
The biggest thing I learned from this step was that:
retry
and:
reconciliation trigger
are different things.
The operator automatically retried the failed reconciliation.
But those retries were limited.
Once they were exhausted, fixing an unrelated external dependency like RBAC did not automatically create a new Greeting event.
A new relevant event, such as changing the resource specification, caused reconciliation to run again.
That gave me a much clearer picture of how event-driven controllers actually behave.
What's Next?
There is still more to understand around reconciliation itself.
Some questions I want to explore separately are:
Can reconciliation be triggered manually?
Should there be periodic reconciliation?
When should we reschedule?
How should external dependency changes be handled?
What are good reconciliation practices?
I want to keep that as a separate topic.
After that, I also want to explore:
owner references
resource deletion
finalizers
integration tests
GitHub Actions
For now, the Greeting Operator can report both success and failure using Kubernetes-native mechanisms and automatically retry transient failures.
Top comments (0)