A Kubernetes controller is deceptively simple: watch objects, call Reconcile, make the world match the spec. Then you ship it, and three weeks later someone reports that deleting a namespace hangs forever, your API server audit log shows 40 000 PATCH requests per minute from a single controller pod, and an orphaned bucket is still billing you in production.
All three failures come from the same three concepts: idempotency, finalizers, and status write discipline. This article builds a minimal but correct controller with controller-runtime, scaffolded by kubebuilder, that manages a CRD representing an external resource. We focus on the parts that actually break in production.
Context: why the reconcile loop is harder than it looks
Reconcile is level-triggered, not edge-triggered. You never receive "the user changed spec.size from 10 to 20". You receive "something about default/my-bucket may have changed; go look". The event may be a resync, a status update your own controller just wrote, a cache re-list after an API server restart, or a genuine user edit. It may arrive twice for the same change, or once for three changes.
Three consequences that drive the entire design:
- Reconcile must be idempotent. Running it 100 times must produce the same cluster state as running it once, and must not create 100 external resources.
- Reconcile must be restartable. The process can die between "create the external bucket" and "record the bucket ID in status". The next pass must recover without leaking.
-
Garbage collection only covers in-cluster objects.
ownerReferenceswill clean up aSecretyou created. Nothing in Kubernetes knows about an S3 bucket, a DNS record, or a row in an external IPAM. That is exactly what finalizers are for.
Scaffolding the project
mkdir bucket-operator && cd bucket-operator
kubebuilder init \
--domain example.com \
--repo github.com/acme/bucket-operator \
--plugins go/v4
kubebuilder create api \
--group storage \
--version v1alpha1 \
--kind BucketClaim \
--resource --controller
The API type. Note observedGeneration and the conditions slice — both are load-bearing, not decoration.
// api/v1alpha1/bucketclaim_types.go
type BucketClaimSpec struct {
// +kubebuilder:validation:MinLength=3
// +kubebuilder:validation:MaxLength=63
Region string `json:"region"`
// +kubebuilder:validation:Enum=Retain;Delete
// +kubebuilder:default=Delete
ReclaimPolicy string `json:"reclaimPolicy,omitempty"`
}
type BucketClaimStatus struct {
// BucketID is the opaque identifier returned by the storage backend.
BucketID string `json:"bucketID,omitempty"`
// ObservedGeneration is the .metadata.generation this status reflects.
ObservedGeneration int64 `json:"observedGeneration,omitempty"`
// +listType=map
// +listMapKey=type
Conditions []metav1.Condition `json:"conditions,omitempty"`
}
// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:printcolumn:name="Region",type=string,JSONPath=`.spec.region`
// +kubebuilder:printcolumn:name="Ready",type=string,JSONPath=`.status.conditions[?(@.type=="Ready")].status`
type BucketClaim struct { /* ... */ }
The status subresource is non-negotiable: without it, every Update on status bumps metadata.generation and re-triggers your own controller. That is infinite loop number one.
The generated CRD can carry CEL validation so you reject bad specs at admission time instead of in Go:
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: bucketclaims.storage.example.com
spec:
group: storage.example.com
scope: Namespaced
names:
kind: BucketClaim
listKind: BucketClaimList
plural: bucketclaims
singular: bucketclaim
versions:
- name: v1alpha1
served: true
storage: true
subresources:
status: {}
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
required: [region]
x-kubernetes-validations:
- rule: "self.region == oldSelf.region"
message: "region is immutable"
properties:
region:
type: string
reclaimPolicy:
type: string
enum: [Retain, Delete]
default: Delete
status:
type: object
properties:
bucketID: { type: string }
observedGeneration: { type: integer, format: int64 }
conditions:
type: array
items:
type: object
required: [type, status, lastTransitionTime, reason]
properties:
type: { type: string }
status: { type: string }
reason: { type: string }
message: { type: string }
observedGeneration: { type: integer, format: int64 }
lastTransitionTime: { type: string, format: date-time }
Immutability enforced by CEL removes an entire class of reconcile branches ("what do I do when the user moves the bucket to another region?").
The reconcile loop, step by step
const finalizerName = "storage.example.com/bucket-cleanup"
func (r *BucketClaimReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
log := logf.FromContext(ctx)
var bc storagev1alpha1.BucketClaim
if err := r.Get(ctx, req.NamespacedName, &bc); err != nil {
// NotFound means the object is gone and fully finalized. Nothing to do.
return ctrl.Result{}, client.IgnoreNotFound(err)
}
if !bc.DeletionTimestamp.IsZero() {
return r.reconcileDelete(ctx, &bc)
}
// 1. Finalizer BEFORE any external side effect.
if controllerutil.AddFinalizer(&bc, finalizerName) {
if err := r.Update(ctx, &bc); err != nil {
return ctrl.Result{}, err
}
// This Update generates a new event; stop here, the next pass continues.
return ctrl.Result{}, nil
}
// 2. Idempotent provisioning, keyed on a deterministic name.
name := bucketName(&bc) // e.g. fmt.Sprintf("%s-%s-%s", bc.Namespace, bc.Name, bc.UID[:8])
id, err := r.Storage.EnsureBucket(ctx, name, bc.Spec.Region)
if err != nil {
meta.SetStatusCondition(&bc.Status.Conditions, metav1.Condition{
Type: "Ready", Status: metav1.ConditionFalse,
Reason: "BackendError", Message: err.Error(),
ObservedGeneration: bc.Generation,
})
_ = r.patchStatus(ctx, &bc)
return ctrl.Result{}, err // exponential backoff from the workqueue
}
// 3. Owned in-cluster object: GC handles deletion, no finalizer needed.
if err := r.ensureCredentialsSecret(ctx, &bc, id); err != nil {
return ctrl.Result{}, err
}
// 4. Status last, and only status.
bc.Status.BucketID = id
bc.Status.ObservedGeneration = bc.Generation
meta.SetStatusCondition(&bc.Status.Conditions, metav1.Condition{
Type: "Ready", Status: metav1.ConditionTrue,
Reason: "Provisioned", ObservedGeneration: bc.Generation,
})
if err := r.patchStatus(ctx, &bc); err != nil {
return ctrl.Result{}, err
}
// Re-check the backend periodically: no watch exists on external systems.
return ctrl.Result{RequeueAfter: 10 * time.Minute}, nil
}
Why the finalizer goes first
If you create the bucket and then add the finalizer, there is a window where the controller can crash, the user deletes the CR, the API server removes it immediately, and the bucket leaks. Adding the finalizer is a cheap, idempotent metadata write. Do it before you touch anything outside the cluster, always.
Why provisioning must be keyed on a deterministic name
EnsureBucket must be "create if absent, return existing otherwise" and derive its key from immutable CR data (namespace, name, UID). Never key on status.BucketID alone: if the process dies between the backend call and the status write, the next pass must still find the existing bucket instead of creating a second one. Reading the status you just wrote back from the cache is also unreliable — the informer cache is eventually consistent.
Patch, don't update, for status
func (r *BucketClaimReconciler) patchStatus(ctx context.Context, bc *storagev1alpha1.BucketClaim) error {
// base is the object as it was read; MergeFrom computes a minimal patch.
return r.Status().Patch(ctx, bc, client.MergeFrom(bc.DeepCopy()))
}
In practice, capture the base object right after Get (base := bc.DeepCopy()), mutate bc, then r.Status().Patch(ctx, bc, client.MergeFrom(base)). A merge patch that computes to an empty diff is a no-op on the API server: no write, no event, no loop. A blind Status().Update() on every reconcile rewrites resourceVersion every time and is the second classic infinite loop — made worse by lastTransitionTime if you rebuild conditions by hand instead of using meta.SetStatusCondition, which only touches the timestamp when the status actually flips.
Deletion: the finalizer path
func (r *BucketClaimReconciler) reconcileDelete(ctx context.Context, bc *storagev1alpha1.BucketClaim) (ctrl.Result, error) {
if !controllerutil.ContainsFinalizer(bc, finalizerName) {
return ctrl.Result{}, nil // someone else's finalizer, not our turn
}
if bc.Spec.ReclaimPolicy == "Delete" {
err := r.Storage.DeleteBucket(ctx, bucketName(bc))
switch {
case err == nil, errors.Is(err, backend.ErrNotFound):
// Idempotent: already gone is success.
default:
r.Recorder.Eventf(bc, corev1.EventTypeWarning, "CleanupFailed",
"bucket cleanup failed: %v", err)
return ctrl.Result{}, err
}
}
controllerutil.RemoveFinalizer(bc, finalizerName)
if err := r.Update(ctx, bc); err != nil {
return ctrl.Result{}, client.IgnoreNotFound(err)
}
return ctrl.Result{}, nil
}
Four rules for finalizers:
- Idempotent cleanup. "Resource not found in the backend" is a success, not an error. Otherwise an already-deleted bucket blocks the CR forever.
- Remove the finalizer only after cleanup succeeded. The order in the code above is the whole point.
-
Never block forever. A finalizer that always errors makes the CR undeletable and, worse, makes its namespace stuck in
Terminating. Expose the failure as an Event and a condition so an operator can see why, and support an escape hatch (for example, honourstorage.example.com/skip-cleanup: "true"as an annotation). -
Check it is your finalizer. Other controllers (and the
kubernetesnamespace finalizer) add their own. Only ever touch your own string.
Emergency unblocking, for the runbook:
# Inspect what is holding the object
oc get bucketclaim my-bucket -o jsonpath='{.metadata.finalizers}{"\n"}'
# Last resort: strip the finalizer manually (this LEAKS the external resource)
oc patch bucketclaim my-bucket --type=merge -p '{"metadata":{"finalizers":null}}'
Wiring the controller and filtering events
func (r *BucketClaimReconciler) SetupWithManager(mgr ctrl.Manager) error {
return ctrl.NewControllerManagedBy(mgr).
For(&storagev1alpha1.BucketClaim{},
builder.WithPredicates(predicate.Or(
predicate.GenerationChangedPredicate{},
predicate.AnnotationChangedPredicate{},
))).
Owns(&corev1.Secret{}).
WithOptions(controller.Options{MaxConcurrentReconciles: 4}).
Named("bucketclaim").
Complete(r)
}
GenerationChangedPredicate drops update events where metadata.generation is unchanged — that includes your own status writes. It does not drop deletions: when the API server marks an object for deletion while finalizers are pending, it bumps generation, so the delete-intent event still reaches you. The trade-off: it also drops periodic resync events, so never rely on resync for drift detection — use the explicit RequeueAfter in the success path instead.
MaxConcurrentReconciles > 1 means the same controller processes different objects in parallel, but the workqueue guarantees a given key is never reconciled concurrently with itself. Your Reconcile must still be safe against two different CRs touching the same backend.
Returning the right thing
| Situation | Return |
|---|---|
| Done, nothing more to do | ctrl.Result{}, nil |
| Transient failure |
ctrl.Result{}, err — exponential backoff, logged, metered |
| Waiting on something with no watch | ctrl.Result{RequeueAfter: d}, nil |
| Permanent, user-caused error |
ctrl.Result{}, nil + a Ready=False condition |
Never return an error for a user mistake like an invalid region string: the workqueue will retry it forever, spam your logs, and inflate controller_runtime_reconcile_errors_total. Surface it in status.conditions and stop. Also, do not hand-roll time.Sleep inside Reconcile — you are blocking a worker goroutine; use RequeueAfter.
Conflicts (apierrors.IsConflict) are normal under optimistic concurrency. Just return the error and let the backoff re-read a fresh object; do not wrap your whole reconcile in retry.RetryOnConflict.
Testing against a real API server
envtest runs a real kube-apiserver and etcd binary, which is the only way to validate finalizer semantics and the status subresource.
make test # kubebuilder wires setup-envtest + ginkgo
make manifests generate
make install # applies config/crd into the cluster
make run # runs the manager out-of-cluster against your kubeconfig
Two tests worth writing on day one: delete the CR while the fake backend returns an error, and assert the object still exists with its finalizer; then make the backend succeed and assert the object disappears.
Security best practices
Least-privilege RBAC. Kubebuilder markers generate the role, so keep them narrow — no groups=* or resources=*:
// +kubebuilder:rbac:groups=storage.example.com,resources=bucketclaims,verbs=get;list;watch;update;patch
// +kubebuilder:rbac:groups=storage.example.com,resources=bucketclaims/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=storage.example.com,resources=bucketclaims/finalizers,verbs=update
// +kubebuilder:rbac:groups=core,resources=secrets,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=core,resources=events,verbs=create;patch
Note the dedicated /finalizers subresource permission: it is required as soon as OwnerReferencesPermissionEnforcement is enabled, which is the default on OpenShift.
Scope the cache. A cluster-wide watch on Secrets pulls every secret in the cluster into controller memory. Restrict it:
mgr, err := ctrl.NewManager(cfg, ctrl.Options{
Cache: cache.Options{
ByObject: map[client.Object]cache.ByObject{
&corev1.Secret{}: {
Label: labels.SelectorFromSet(labels.Set{"app.kubernetes.io/managed-by": "bucket-operator"}),
},
},
},
})
Harden the pod. On OpenShift, target the restricted-v2 SCC: do not pin runAsUser, let the namespace range assign it.
apiVersion: apps/v1
kind: Deployment
metadata:
name: bucket-operator-controller-manager
namespace: bucket-operator-system
spec:
replicas: 2
selector:
matchLabels: { control-plane: controller-manager }
template:
metadata:
labels: { control-plane: controller-manager }
spec:
serviceAccountName: bucket-operator-controller-manager
automountServiceAccountToken: true
securityContext:
runAsNonRoot: true
seccompProfile: { type: RuntimeDefault }
containers:
- name: manager
image: quay.io/acme/bucket-operator:v0.1.0
args: ["--leader-elect", "--health-probe-bind-address=:8081"]
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities: { drop: ["ALL"] }
resources:
requests: { cpu: 10m, memory: 64Mi }
limits: { memory: 256Mi }
livenessProbe:
httpGet: { path: /healthz, port: 8081 }
readinessProbe:
httpGet: { path: /readyz, port: 8081 }
Leader election (--leader-elect with replicas: 2) is a correctness control, not just HA: two active managers racing on the same external backend will double-provision. It needs leases in coordination.k8s.io, namespace-scoped.
Never log secret material. logf.FromContext(ctx) is structured — log the object key and the backend resource ID, never credentials or full object dumps. Credentials belong in a Secret with an ownerReference pointing at the CR, so they are garbage-collected automatically and never written to status.
Validate at admission, not in Go. CEL rules in the CRD (and a validating webhook with a service-ca managed certificate on OpenShift) block bad input before it ever reaches a worker. A NetworkPolicy restricting egress to your backend endpoint closes the loop.
Key takeaways
-
Reconcileis level-triggered: make it idempotent, restartable, and keyed on deterministic names derived from immutable fields. - Add the finalizer before the first external side effect; remove it only after cleanup succeeds, and treat "already gone" as success.
- Use the status subresource with
Status().Patch(client.MergeFrom(base))andmeta.SetStatusCondition— blindUpdatecalls are the most common infinite loop. - Filter events with
GenerationChangedPredicate, but keep an explicitRequeueAfterto detect drift in systems you cannot watch. - Return an error only for transient failures; report user errors as conditions so the workqueue does not retry them forever.
- A finalizer that can never complete is a namespace that can never terminate: always expose the failure and provide an escape hatch.
- Lock down RBAC, scope the informer cache, and run the manager as a non-root, read-only, capability-dropped pod.
Top comments (1)
One point worth adding to the escape hatch is to treat the skip-cleanup annotation as a privileged action. If any user who can edit the claim can set it, cleanup can be bypassed by mistake and leave an external bucket running with no owner. Restricting who can set it, and recording each use in an event or audit log, keeps the hatch useful for a genuine emergency without quietly creating orphaned resources.