⚡ TL;DR
Four things about controller mechanics are widely half-understood: what Reconcile receives, where its work comes from, what a periodic resync is, and what a predicate turns off. I built an operator, broke it five ways, and measured each mechanism directly. The reconcile function runs in 2.71ms mean, 77/77 under 25ms, a short resync period costs zero additional API requests, and GenerationChangedPredicate cut steady-state reconciles by 48.5% without touching live repair at all.
That last combination is the one that matters at scale. Repo, raw data, and harness: kirPoNik/k8s-drift-operator.
🧩 The four barriers
Everyone who runs Kubernetes knows the platform repairs itself. Delete a pod, it comes back. Scale a Deployment by accident, something puts it back.
Almost nobody who relies on that property can say how it works, and the gaps are specific and consequential. I keep meeting the same four:
People think a controller is told what changed. It is not, and the reason it is not is the single most important design decision in Kubernetes.
People think a controller polls the API server. It does not, and knowing what it does instead tells you where your API load actually comes from.
People think a
resyncis a re-check against the cluster. It is not, which is why a shortresyncperiod is nearly free — and why the number that is expensive sits somewhere else entirely.People treat a predicate as a pure optimisation. It is a filter with a silent cost, and the cost is not the one the documentation warns you about first.
So I built the smallest system that has the self-healing property, broke it on purpose ten times per failure mode, and instrumented each of those four mechanisms until I could state what it does rather than what it is said to do.
What I built. One CRD called Echo, holding an image, a replica count, and a greeting. A controller keeps three child objects in sync with it — a Deployment, a Service, and a ConfigMap holding the greeting — with owner references on all three so Kubernetes garbage-collects them when the Echo goes away. Observed state written back to Echo.status. Plus a Python harness that injects drift and measures repair.
Environment, because every number depends on it: macOS on Apple Silicon, Go 1.27.0, kind v0.32.0 with node image kindest/node:v1.36.1 (Kubernetes 1.36.1), kubebuilder v4.15.0, controller-runtime v0.24.1, demo image hashicorp/http-echo:1.0.0.
🎯 Barrier 1: Reconcile has no idea what changed
Here is the whole thing, and it explains more of Kubernetes than any diagram:
func (r *EchoReconciler) Reconcile(
ctx context.Context,
req ctrl.Request,
) (ctrl.Result, error) {
// function body
}
req is a namespace and a name. That is the entire input. Not a diff. Not "replicas changed from 1 to 3." Not a copy of the object.
Every invocation re-reads the desired spec from scratch, recomputes what all three children should look like, and hands that to controllerutil.CreateOrUpdate to diff against whatever exists right now. Identical work whether the trigger was a kubectl command a millisecond ago, a timer firing with nothing wrong, or a process that just booted after eight hours down.
This is level-triggered control: act on the current state of the world, not on the transition that produced it. Edge-triggered is the opposite — act on the delta the event carries. Edge-triggered is the more intuitive model coming from event-driven systems, and it looks cheaper. Why rebuild an entire Deployment spec when you already know exactly what changed?
Because the cheaper model requires the event stream to be perfect, and it never is.
Kubernetes settled this early and wrote it down. The architecture principles in the design proposals archive state that functionality must be level-based: correct behaviour given desired state and observed state, regardless of how many intermediate updates were missed. Edge-triggered behaviour, in their words, "must be just an optimization."
The consequence is the entire reason the model holds up. A missed watch event is not a lost repair. It is a delayed one. There is no catch-up code path you could have forgotten to write, because there is no catch-up code path. There is one path, and it runs constantly.
This is also why the API surface looks the way it does. Reconciling from a name rather than a diff is what lets a Deployment controller, a ReplicaSet controller, and your own operator all act on the same objects without coordinating, and lets any of them be restarted, upgraded, or briefly killed without a recovery procedure.
🧵 Barrier 2: where the work comes from, and what it costs
A controller does not poll. Four pieces do the work, and keeping them separate in your head is what makes the rest of this legible — because the interesting failures live in the gaps between them.
Watch. A long-lived HTTP connection to the API server that streams changes for one resource type. Established once at startup and held open. Not a request per check.
Informer cache. A local in-memory copy of every watched object of that type, kept current by the watch. This is the piece with the largest practical consequence: reads inside Reconcile are served from local memory and cost no API request. A reconciler that reads ten objects and writes nothing generates zero API traffic. Writes always go to the API server — the cache is a read path only.
Work queue. Holds keys, not events: namespace/name strings. It deduplicates. If the same key is enqueued five times before a worker picks it up, one Reconcile runs, not five. This is why the reconciler never learns which watch woke it, and why it does not need to.
Predicates. Filters that sit in front of the queue and decide which events are allowed to enqueue a key at all. Hold onto that position — barrier 4 is entirely about it.
Wiring, four watches into one queue:
return ctrl.NewControllerManagedBy(mgr).
For(&labv1alpha1.Echo{}).
Owns(&appsv1.Deployment{}).
Owns(&corev1.Service{}).
Owns(&corev1.ConfigMap{}).
Named("echo").
Complete(r)
For watches the custom resource. Each Owns watches a child type and maps any event on it back to the owner's key through the owner reference. Four event sources, one key, one reconciler.
One reconciler box, not four handlers. That is the point of the diagram.
⏱️ Barrier 3: what "fast" means, and how to measure it without measuring yourself
Five drift types, ten runs each. Timing starts when the drift command returns and stops when the affected object matches desired state again — not when the pod serves traffic, since pod startup is the kubelet's problem, not this loop's.
| Drift | n | converged | p50 | max | stdev |
|---|---|---|---|---|---|
| D1 — delete the Deployment | 10 | 10/10 | 0.0576s | 0.0630s | 0.0028s |
| D2 — scale replicas to 0 | 10 | 10/10 | 0.0587s | 0.0738s | 0.0062s |
| D3 — hand-edit the ConfigMap | 10 | 10/10 | 0.0544s | 0.0626s | 0.0031s |
| D4 — drift while the controller is down, then restart | 10 | 10/10 | 0.3673s | 0.3737s | 0.0038s |
D5 — D2 with GenerationChangedPredicate on the Deployment watch |
10 | 10/10 | 0.0555s | 0.0751s | 0.0080s |
Those four sub-100ms figures are upper bounds set by my measurement harness, not durations of the reconcile loop. Here is how I know, because the reasoning generalises to any convergence measurement you build from the outside.
The poller is four lines, and the ordering is the whole story:
def wait_until(check, timeout):
start = now()
while True:
if check(): # runs immediately, before any sleep
return now() - start
if now() - start > timeout:
return None
time.sleep(0.1)
check() shells out to kubectl get deployment -o json: fork, exec, TLS, API round trip, JSON parse. Roughly 50 to 60 milliseconds.
A run needing a second look would therefore report at least 0.055 + 0.100 + 0.055 ≈ 0.21s. The largest value anywhere in my data — all 60 timed runs, including the in-cluster comparison below — is 0.1091s. Nothing approaches 0.21s.
Every run resolved on its first observation. The sleep(0.1) never executed once, in any configuration. Each figure in that table is the wall time of one kubectl get, and the repair had finished before that kubectl returned. The stdev column describes how consistently my laptop spawns a subprocess.
I caught this because a different number looked impossible first. My original D4 script started its timer after the manager subprocess had booted, which excluded the exact cost D4 exists to measure. It reported 0.054s — a cold process booting, connecting, listing and reconciling in 54 milliseconds, indistinguishable from a warm watch-triggered repair. That implausibility is the only reason a wrong number is not in the table above. Re-measured correctly, D4 is 0.367s. Run one sample and sanity-check it against physics before committing to ten.
📊 The instrument that can see it
controller-runtime already exposes a histogram of Reconcile call duration: controller_runtime_reconcile_time_seconds. No polling in the measurement path at all. So I stopped timing from outside and read the histogram's delta across exactly ten D1 injections.
A Prometheus histogram counts observations into cumulative buckets — le="0.005" holds every reconcile at or under 5ms, le="0.01" at or under 10ms. Cumulative buckets give you real bounds instead of a mean that hides its own tail, which is exactly what you want when the question is "how bad does this get."
| Metric | Value |
|---|---|
| Reconciles observed | 77 |
| Total time | 0.208939s |
| Mean | 2.71ms |
| ≤ 5ms | 64/77 (83%) |
| ≤ 10ms | 71/77 (92%) |
| ≤ 25ms | 77/77 (100%) |
The claim that carries weight is 77/77 under 25ms. The 77 reconciles are a mixed population — ten repairs plus the settling and status-write passes each triggers — and no-op passes are cheaper than repairs, so the mean is diluted downward by design. The histogram measures time inside the function, including its own write round-trips, excluding watch delivery and queue wait.
So: bounded above by the polling harness at ~55ms, resolved below by the histogram at single-digit milliseconds. Both instruments correct, measuring different things.
🏠 The controller-down case is a different shape entirely
D4 is the only row where the harness resolved a real duration: 0.3673s, stdev 0.0038s. Six to seven times the live cases, and far more consistent.
That shape is a fixed cost. The delay is process boot plus informer cache warm-up, and the tight stdev is what a fixed cost looks like next to variable watch and queue timing.
The mechanism is worth internalising, because it is the direct payoff of barrier 1. The controller does not "catch up." On startup, the manager lists everything that exists and reconciles all of it through the same code path as every other trigger. Down for five minutes or five hours produces an identical repair, because no separate downtime-recovery mechanism exists to degrade.
I also repeated D1 with the controller deployed in-cluster instead of running on the host:
| Config | n | p50 | max | stdev |
|---|---|---|---|---|
| D1, outside-cluster | 10 | 0.0576s | 0.0630s | 0.0028s |
| D1, in-cluster | 10 | 0.0599s | 0.1091s | 0.0166s |
Both rows sit at the harness floor, so nearly identical p50s establish that both configurations are faster than this instrument can resolve. If one were 2ms and the other 8ms, this table would look the same. The tail is real — max doubles, stdev goes up roughly sixfold — and it is the tail of the observation. In-cluster, the controller Pod shares a node with the API server, etcd, kube-proxy and CoreDNS, and a loaded node's API server answers any client's GET more slowly, mine included. That accounts for a slower kubectl get. Attaching it to reconcile latency requires the histogram in-cluster, which is a redeploy I have not run.
🕳️ Barrier 4: what a predicate actually turns off
There is a known trap in controller-runtime: add GenerationChangedPredicate to an owned-type watch and lose your self-healing safety net, silently, with no error anywhere. I set out to demonstrate it — predicate on the Deployment watch, repeat the scale-to-zero drift, watch it fail to converge.
It converged. Ten out of ten, same speed as without.
That result is the barrier, and resolving it requires two Kubernetes fields that are constantly conflated:
resourceVersionchanges on every write to an object, including status writes.generationchanges only when.specchanges. It is the API server saying "someone asked for something different."
GenerationChangedPredicate compares .metadata.generation on the old and new object and drops the event when they match. A kubectl scale is a genuine spec write, so generation bumps, so the predicate passes the event and the reconciler heals it exactly as if no predicate existed. The test was aimed at the wrong class of event.
The class it actually drops is a resync: a local timer inside each informer that periodically re-emits every object it holds as a synthetic update event, where old and new are literally the same cached object. Same generation, by construction, because nothing happened.
Not inference — the Options.SyncPeriod doc comment in the exact version pinned here, v0.24.1, states it. A resync locally triggers an artificial update event with the same object as both old and new, and predicates expecting those to differ (it names GenerationChangedPredicate) "will filter out this event." It also states that a resync does not sync between the local cache and the server, which is barrier 3's answer and the next section's foundation.
🔍 Isolating it
So I measured the thing the predicate actually suppresses: steady-state reconcile rate with no drift at all. Resync period 10 seconds, predicate applied to two of the four watches (Echo itself and the owned Deployment), Service and ConfigMap left unguarded.
| Config | window | reconciles | reconciles/min |
|---|---|---|---|
| predicate off | 180s | 66 | 22.0 |
| predicate on (2 of 4 watches) | 180s | 34 | 11.3 |
A 48.5% drop. No error, no log line, no externally visible difference. The only way to detect it is to notice a rate that should exist and does not.
That drop is well clear of measurement noise: a separate, identically configured 10-second window elsewhere in my data logged 62 reconciles where this one logged 66, about 6% spread. The clean near-50% is a coincidence rather than a law — the queue deduplicates, so suppressing half the event sources should not linearly halve the output.
The precise claim is stronger than the one I set out to prove. It is not that this predicate breaks self-healing. It is that the predicate leaves live repair fully intact and silently removes the periodic net that catches drift no watch ever reported. That is a harder failure to find, because every test that injects real drift passes with the predicate on.
💸 Barrier 3, resolved: where API cost actually lives
Three resync periods, 180-second steady-state window each, measuring how often the controller re-checked and how many API requests it issued.
| SyncPeriod | window | reconciles | reconciles/min | additional API requests |
|---|---|---|---|---|
| default (unset → 10h) | 180s | 0 | 0.0 | 0 |
| 1 minute | 180s | 5 | 1.67 | 0 |
| 10 seconds | 180s | 62 | 20.67 | 0 |
One 180-second window per row. Reconcile rate climbs as expected. API load does not move at all, even at twenty re-checks a minute.
The precise version of that zero: rest_client_requests_total is not zero in absolute terms. At window-open it reads 11 — six GETs (the four informers' initial lists, plus discovery and RESTMapper overhead), three POSTs (one-time creates of ConfigMap, Deployment and Service), two PUTs (status-subresource writes as the Echo settles to Ready). The claim is zero additional requests after cache sync, at every setting.
That 11 is the interesting number, not the zero. The API cost of this controller is almost entirely a startup cost: list once per watched type, then hold watches open. Steady-state re-checking is free because it reads local memory. The SyncPeriod doc comment's own rationale for jittering informers is to stop controllers issuing list requests simultaneously — the library's designers put the jitter where the cost is, and the cost is at boot.
Two conditions have to hold for that zero, and only one came free.
The free one is architectural: defaultSyncPeriod = 10 * time.Hour in this version, with a per-informer jitter factor from [0.9, 1.1), and a resync that re-emits from cache rather than listing the server.
The one I had to earn was idempotency, and I broke it first. My initial Deployment builder replaced the entire Containers slice on every reconcile:
// Mutate the "echo" container's fields in place rather than replacing the
// whole slice: a wholesale replace would wipe out fields the API server
// fills in with defaults after creation (ImagePullPolicy,
// TerminationMessagePath, ...), making every future reconcile see a "diff"
// against those defaults forever and never settle to a no-op pass.
container := findOrAppendContainer(&dep.Spec.Template.Spec.Containers, "echo")
container.Image = echo.Spec.Image
The comment is the postmortem. On create, the API server defaults fields on the container. My mutate function then overwrote the whole slice with a literal holding Go zero values for those fields. CreateOrUpdate saw a real diff, issued an Update, the server re-defaulted the stripped fields, and the next reconcile repeated it. Every reconcile logged "deployment": "updated" and never once settled to "unchanged".
Unfixed, that table would show API requests climbing in lockstep with reconcile rate. The flat zero column is therefore also a regression check on that bug.
📐 Reading the resync counts precisely
The counts are not (4 watched types) × (window / period). At one minute, four independent per-type resyncs over three cycles gives a naive ~12; I measured 5. At ten seconds, 4 × 18 = 72 naive; I measured 62.
The jitter does not explain this, and the arithmetic rules it out. A jitter factor with mean 1.0 leaves the expected number of cycles in a fixed window unchanged — it shifts each informer's phase, not its average frequency. De-phasing four timers makes simultaneous, coalescible arrivals less likely, which pushes the count up toward the naive figure, not down.
What remains is the work queue's per-key deduplication, which is the correct category of explanation and which I have not isolated with a measurement. The bound I will defend: at a 10-second period, four watched types produced 62 reconciles in 180 seconds, and the naive multiplication overestimates. Design against the measured number, not the arithmetic one.
🧭 My take: what this means on a large estate
Everything above was measured on one object in one namespace on a laptop. The mechanisms are what generalise, and their consequences change character as the estate grows. The following is reasoning from the measured mechanism, not from a measurement at scale.
Steady-state reconciliation is free; startup is where you pay. A quiescent, correctly idempotent controller reads local memory and issues nothing. The measured cost was 11 requests at boot and 0 thereafter, regardless of resync period. On a large cluster the shape holds but the magnitude moves: those initial lists scale with the number of objects per watched type, not with the number of controllers doing nothing. Which means the API server event you should plan for is not "controllers running" — it is many controllers starting at once, after a control-plane upgrade, a node drain, or a rollout of your own operator fleet. Cache warm-up is the cost, WaitForCacheSync is where it shows, and D4's fixed-cost profile (0.3673s, stdev 0.0038s, for a trivial cache) is the same curve you will walk up with object count.
Cache memory is the constraint people hit before API load. Every watched type means a full in-memory copy of every object of that type the controller is allowed to see. On a large cluster, watching Pods cluster-wide is a very different proposition from watching your own CRD. Narrow the cache before you tune anything else: label and field selectors, namespace scoping, and stripping managed fields all reduce the copy rather than the traffic.
Predicates are a genuine scaling lever, and they cost a safety net. 48.5% fewer reconciles for one predicate on two watches is a real reduction, and on an estate with thousands of objects and a busy controller it is the kind of lever you will reach for. Reach for it with the trade priced: you are trading a periodic net that catches drift no watch reported for a lower reconcile rate. Two rules follow. Never put GenerationChangedPredicate on a watch whose job is detecting external mutation — that is the exact case it filters. And when you do add one, test the resync path specifically, because every drift test you already have will pass.
Idempotency bugs are write amplification against shared infrastructure. My container-slice bug produced no error, no failed test, and no user-visible symptom. It produced an unconditional Update per reconcile. One object on a laptop, that is invisible. A thousand objects with a short resync period, on a control plane shared by every other team, and it becomes someone else's incident with your controller's name on it. The check is cheap and belongs in CI: reconcile twice, assert the second pass writes nothing.
Level-triggered is the right default, and the tax is where you should spend review time. The reason this design survives contact with large infrastructure is that there is one code path. No catch-up logic, no downtime-recovery mode, no ordering assumptions to violate. Missed events become latency, not corruption. That property is worth the price, and the price is that every reconciler must be provably idempotent — which is not the kind of correctness a type checker or an integration test finds for you.
One operational lesson from a bug unrelated to any of this. During the in-cluster run, make docker-build failed with package cmd/main.go is not in std, which names nothing relevant. Cause: docker buildx was not installed, so docker build silently fell back to the legacy builder, which does not recurse into a **-excluded directory to honour a later !**/*.go re-inclusion — only BuildKit does. Result: an empty build context for every Go source directory, with COPY . . reporting success. Found by committing the failed intermediate container and looking inside. Verify BuildKit is actually in use before trusting a Dockerfile that leans on ARG TARGETOS auto-population or .dockerignore re-inclusion patterns. On a build fleet, that failure mode is per-runner and reproduces only on the hosts missing the plugin.
📚 Resources
kirPoNik/k8s-drift-operator— the repo.docs/measurements.mdhas every table with the command that produced it,docs/raw/the unedited CSVs and manager logs,experiment/the harness includingreconcile_time.py.docs/artifact-adr.md— the architecture decision record underneath this piece: level-triggered versus edge-triggered, with the predicate trap as evidence.docs/build-log.md— the unedited build log, dead ends included.controller-runtimev0.24.1 —pkg/cache— theOptions.SyncPerioddoc comment is the primary source for resync semantics and theGenerationChangedPredicateinteraction. Read that comment; it is the densest paragraph in the project.Kubernetes design proposals archive —
architecture/principles.md— where level-based control is stated as a requirement rather than a convention.
Top comments (0)