In the previous posts, I built a small Kubernetes operator around a custom Greeting resource.
The operator currently does a few things:
Greeting
|
v
Greeting Operator
|
v
ConfigMap
It watches Greeting resources, creates a ConfigMap, updates status, records Events, retries failures, and also supports manual reconciliation.
So far, most of my focus was on:
create
update
reconcile
retry
But there was one lifecycle question I had not really explored yet:
What happens to the ConfigMap when the Greeting is deleted?
That led me into Kubernetes owner references and garbage collection.
The Current Resource Relationship
A Greeting looks like this:
apiVersion: platform.shubforge.dev/v1alpha1
kind: Greeting
metadata:
name: hello
spec:
message: "Hello from Platform Lab"
The operator creates:
apiVersion: v1
kind: ConfigMap
metadata:
name: hello-greeting
data:
message: "Hello from Platform Lab"
So conceptually:
Greeting/hello
|
v
ConfigMap/hello-greeting
Until now, I was mainly thinking about this as:
controller creates another resource
But Kubernetes also needs to understand the lifecycle relationship between those two resources.
Looking at the Generated ConfigMap
I started by inspecting the ConfigMap created by the operator:
kubectl get configmap hello-greeting -o yaml
I got:
apiVersion: v1
data:
message: Hello from the updated Greeting
kind: ConfigMap
metadata:
labels:
app.kubernetes.io/managed-by: greeting-operator
name: hello-greeting
namespace: default
ownerReferences:
- apiVersion: platform.shubforge.dev/v1alpha1
blockOwnerDeletion: false
controller: false
kind: Greeting
name: hello
uid: 4c749fca-e2a3-49f7-a330-1374b72c3bf8
The interesting part was:
ownerReferences:
The ConfigMap had an owner reference pointing back to:
Greeting/hello
So the relationship was not only something my Java code understood.
Kubernetes itself also knew:
Greeting/hello
|
| owns
v
ConfigMap/hello-greeting
What Is an Owner Reference?
An owner reference is metadata stored on the dependent resource.
In my case, the dependent is:
ConfigMap/hello-greeting
and the owner is:
Greeting/hello
The ConfigMap contains:
ownerReferences:
- apiVersion: platform.shubforge.dev/v1alpha1
kind: Greeting
name: hello
uid: ...
This tells Kubernetes that the lifecycle of the ConfigMap is related to that specific Greeting.
The uid is especially interesting.
Kubernetes does not only store:
Greeting
+
hello
It also stores the unique identity of that exact resource.
So the relationship is closer to:
ConfigMap
|
v
specific Greeting UID
That matters because a resource can later be recreated with the same name but a different UID.
controller and blockOwnerDeletion
My generated owner reference also contained:
controller: false
blockOwnerDeletion: false
At first I expected something like:
controller: true
because the ConfigMap is created by my controller.
But the controller field has a more specific meaning in Kubernetes owner references.
What mattered for this experiment was that the ownership relationship itself existed:
Greeting
|
v
ConfigMap
Similarly:
blockOwnerDeletion: false
does not mean the ConfigMap is unrelated to the Greeting.
It controls a more specific deletion behavior.
For this simple experiment, I mainly wanted to understand whether Kubernetes could use the owner reference for dependent cleanup.
First Experiment: Delete the ConfigMap
I first tested what happens if I delete the dependent resource.
Both resources existed:
kubectl get greeting hello
and:
kubectl get configmap hello-greeting
Then I deleted only the ConfigMap:
kubectl delete configmap hello-greeting
At that point:
Greeting still exists
ConfigMap does not
But the desired state represented by the Greeting still says:
this Greeting should have a ConfigMap
So the controller reconciled again and recreated it.
The flow looked like:
ConfigMap deleted
|
v
Greeting still exists
|
v
desired state still requires ConfigMap
|
v
controller reconciles
|
v
ConfigMap recreated
This is reconciliation doing its job.
The resource disappeared, but the owner still existed, so the operator restored the desired state.
Second Experiment: Delete the Greeting
Then I tested the opposite case.
I recreated the Greeting and confirmed both resources existed again:
task greeting:create
Then:
kubectl get greeting hello
and:
kubectl get configmap hello-greeting
This time I deleted the owner:
kubectl delete greeting hello
Then I checked:
kubectl get greeting hello
and:
kubectl get configmap hello-greeting
The Greeting disappeared.
The ConfigMap also disappeared.
This time the operator did not recreate it.
That made sense because the source of the desired state was gone.
The flow was:
Greeting deleted
|
v
owner no longer exists
|
v
Kubernetes sees dependent ownerReference
|
v
ConfigMap removed
This was Kubernetes garbage collection rather than normal desired-state reconciliation.
Reconciliation vs Garbage Collection
This experiment helped me separate two mechanisms that looked similar at first.
When the dependent disappears
ConfigMap deleted
|
v
Greeting still exists
|
v
operator reconciles
|
v
ConfigMap recreated
This is:
reconciliation
The controller is maintaining desired state.
When the owner disappears
Greeting deleted
|
v
ownerReference
|
v
Kubernetes garbage collection
|
v
ConfigMap removed
This is:
garbage collection
The desired state itself has disappeared.
So I now think about the two like this:
Reconciliation
=
keep resources in the desired state while the owner exists
and:
Garbage Collection
=
remove dependents when their owner disappears
Watching It Live
One useful way to see this behavior is to watch the ConfigMap.
In one terminal:
kubectl get configmap hello-greeting -w
Then in another terminal:
kubectl delete greeting hello
That makes it easier to see the dependent disappear.
You can also watch both resources:
kubectl get greeting,configmap -w
Then delete:
kubectl delete greeting hello
This gives a much clearer view of the lifecycle than running one command at a time.
The Operator Did Not Need deleteConfigMap()
One part I found interesting was that I did not need to write something like:
deleteConfigMap();
inside the controller.
The ConfigMap lives inside Kubernetes and already has an owner reference.
That means Kubernetes has enough information to manage the dependent lifecycle.
So instead of:
Greeting deleted
|
v
operator manually finds ConfigMap
|
v
operator deletes ConfigMap
I get:
Greeting deleted
|
v
Kubernetes ownership information
|
v
garbage collector
|
v
ConfigMap deleted
That is much more Kubernetes-native.
Same Namespace
Greeting is a namespaced custom resource.
The managed ConfigMap is created in the same namespace.
For example:
default/hello
|
v
default/hello-greeting
If I create the Greeting in another namespace:
demo/hello
the ConfigMap should also be:
demo/hello-greeting
The operator uses the Greeting namespace when building the dependent resource.
Conceptually:
var namespace =
greeting.getMetadata().getNamespace();
and then:
.withNamespace(namespace)
So the lifecycle stays scoped together.
Owner Reference Does Not Mean Finalizer
Another thing I wanted to keep separate was:
owner references
and:
finalizers
They are related to deletion, but they solve different problems.
An owner reference says:
This Kubernetes resource belongs to another Kubernetes resource.
For example:
Greeting
|
v
ConfigMap
Kubernetes can clean up that ConfigMap when the Greeting disappears.
A finalizer is different.
A finalizer says something closer to:
Before this resource is completely deleted,
the controller still has cleanup work to do.
That becomes important when the cleanup is not something Kubernetes can automatically handle.
A Future Example
Imagine the operator eventually created something through an external API:
Custom Resource
|
v
Operator
|
v
External API
|
v
External Resource
If I delete the custom resource:
kubectl delete ...
Kubernetes does not know anything about that external system.
An owner reference cannot help here because the external object is not a Kubernetes resource.
That is where finalizers become interesting.
The flow would be more like:
delete custom resource
|
v
deletionTimestamp set
|
v
controller sees deletion
|
v
external cleanup
|
v
finalizer removed
|
v
resource deletion completes
That is what I want to explore next.
Current Operator Lifecycle
At this point, the Greeting Operator has several different paths.
Desired state changes
Greeting.spec changes
|
v
generation changes
|
v
reconcile
Manual reconciliation
reconcile-at changes
|
v
update filter
|
v
reconcile
Failure
reconciliation fails
|
v
Ready=False
|
v
retry
Dependent deletion
ConfigMap deleted
|
v
Greeting still exists
|
v
reconcile
|
v
ConfigMap recreated
Owner deletion
Greeting deleted
|
v
ownerReference
|
v
Kubernetes garbage collection
|
v
ConfigMap deleted
Seeing all of these together made the operator lifecycle much clearer for me.
Current Architecture
The current flow looks roughly like this:
Greeting
|
+-------------+-------------+
| |
spec change reconcile-at change
| |
+-------------+-------------+
|
v
reconcile()
|
v
ConfigMap
|
+--------------+--------------+
| |
ConfigMap deleted Greeting deleted
| |
v v
reconcile garbage collection
| |
v v
ConfigMap recreated ConfigMap removed
So the operator is no longer only about creating resources.
It is also about understanding resource lifecycle.
Source Code
The complete implementation is available in my Platform Lab repository.
Repository: Platform Lab
The changes and documentation covered in this post are available in:
Pull Request: Explore Greeting Resource Ownership
The current project includes:
- Greeting CRD
- Java Operator SDK controller
- managed ConfigMap dependent resource
- status and conditions
- failure handling
- automatic retries
- Kubernetes Events
- manual reconciliation
- update filtering
- owner references
- Kubernetes garbage collection
What I Learned
The main thing I learned from this step is that deleting a resource is not always the controller's responsibility.
For Kubernetes resources, ownership metadata can allow Kubernetes itself to manage dependent cleanup.
The two cases are different:
dependent deleted
=
restore desired state
while:
owner deleted
=
desired state is gone
So:
delete ConfigMap
→ operator recreates it
but:
delete Greeting
→ ConfigMap disappears with it
That distinction made the relationship between reconciliation and garbage collection much clearer.
What's Next?
The ConfigMap in this example exists completely inside Kubernetes.
That means owner references and garbage collection work well for cleanup.
But what if the operator needs to clean up something Kubernetes does not control?
That leads to the next topic:
Finalizers
I want to explore:
what deletionTimestamp means
how a controller gets a chance to perform cleanup
how finalizers delay deletion
what happens when cleanup fails
why stuck finalizers can leave resources in Terminating
That should complete another important part of the operator lifecycle.
Top comments (1)
The UID point is the one that matters most, and it's worth extending: it's why deleting and recreating a resource with the same name doesn't resurrect its old dependents. Name is an address, UID is an identity, and garbage collection keys on identity.
Three gotchas worth adding to the series.
Cross-namespace owner references aren't allowed. A namespaced dependent must have an owner in the same namespace, and a cluster-scoped dependent can only have a cluster-scoped owner. This bites people building operators that create resources into user namespaces.
blockOwnerDeletion: true requires delete permission on the owner's finalizers subresource, not just on the owner. An operator with an otherwise-correct RBAC role will fail to set it and usually won't say why clearly.
And garbage collection is eventually consistent. Deleting an owner doesn't remove the dependent instantly , the collector runs on a resync, so there's a window where the dependent still exists with a dangling owner reference. Tests asserting immediate cleanup go flaky for reasons that have nothing to do with the operator.
The controller: false observation is a good catch. Only one owner can hold that flag, and it decides which controller gets to adopt the object.