DEV Community

Cover image for Your Crossplane may be sleeping
JOOJO DONTOH
JOOJO DONTOH

Posted on

Your Crossplane may be sleeping

#devops #infrastructure #kubernetes

TL;DR

I came up with a two-cluster Crossplane lab to test experiment that hopes to show what the communication crossplane has it resources it manages both in a native and remote cluster.

Summary of the findings.

  • Native composition cannot reach another cluster. This is not an error, there is just a silent absence. I saw that there is no field for it. You need provider-kubernetes Objects for this mechanism.
  • Readiness reports green for a workload that cannot start. The default policy means "applied successfully" and not that it "works". There are 4 policy options and nothing tells you which one is active.
  • Drift goes uncorrected for ten minutes. The watch field that sounds like it fixes this is alpha, gated, and silently does nothing unless you also pass a flag to the provider pod.
  • Both of those are fixable, with a CEL readiness query and a DeploymentRuntimeConfig. Once configured, drift is caught in about a second and reported honestly.
  • The credential one is not fixable. I rotated a Postgres password directly on the target. Eight green rows across two traces, and FATAL: password authentication failed from inside the application container. Understandable behaviour because a control plane isnt supposed to touch the data plan. But this is worth knowing and understanding

The pattern I realised is that the capability exists, but the defaults dont work as you would assume. The status output for good or bad is identical which makes it hard to tell whether you control plane is watching or asleep.

Repo: github.com/Joojo7/crossplane-lab

Intro

Hello my people its me again. A while back I wrote about BYOC in platform engineering, and the argument I kept coming back to was the difference between a control plane that reconciles infrastructure in your account and one that merely references it. I ended that piece with a hands-on path for anyone who wanted to go further, and Crossplane was on it, mostly because it is the most legible open implementation of the reconciling side of that line.
As part of an encouragement for people to get people to try out crossplane, this article is me trying to go through a series of experiments with you which will hopefully get you to understand some of the basics of crossplane

I put together a small repository and worked through this experiment Joojo7/crossplane-lab. It is built to be followed step by step alongside this article, and I intentionally made it small (for now lol).

The problem

Using a control plane to manage things especially across boundaries is not an easy job. Based on my experiments it turns out that most of the things you would assume are not on by default. The control plane seems to need extra and specific configurations to get some of these things up. Please feel free to comment if you have or know ways to make some of the stuff below available by default.

Also because most tutorials only go with happy paths, you dont get to see times where things are intentionally broken. This article is trying to do something different. I intend to break things and see what sticks.

The problem stated above leads to further questions

  1. Can a control plane reach a cluster it does not own?
  2. Does it notice when something drifts?
  3. Does it know whether the thing it built actually works?

The approach

To answer those three questions we needed something to observe, so this lab experiment builds a very small scale platform. We will define two custom abstractions from scratch to work with.

The first is a Workload. I chose this name because it feels familiar. It stands in for any long-running application, the sort of thing that would normally arrive as a Deployment and a Service.
The second is a DataService, which covers stateful storage. In the lab it resolves to Postgres, but the shape would be much the same for Redis or anything else that holds data and hands out a connection string with some auth stuff. These two primitives should be enough to have a dependency between them, we will find some interesting stuff about the dependency as we go.

We will compose each of these abstractions twice. Once natively in the control plane's cluster and once across cluster boundaries, where the control plane cluster creates them in another cluster, this is usually called a target cluster. We will do this so that running it one after the other will make the differences clear.

After that we will break things and we will go back and read what the control plane is reporting, and compare it against what was actually true in the target/native cluster.

This kind of classifies this article as a tutorial and an audit (Most os of these will be in the docs anyway but its good to get hands-on).

Setup

Enough of the plenty talk. Lets get started

1. Install the CLI

The CLI is a separate install from the controller. Having Crossplane healthy in your cluster gives you no binary at all.

brew install crossplane/tap/crossplane
Enter fullscreen mode Exit fullscreen mode

Or without Homebrew:

curl -sL https://raw.githubusercontent.com/crossplane/crossplane/main/install.sh | sh
sudo mv crossplane /usr/local/bin
Enter fullscreen mode Exit fullscreen mode

Check it:

crossplane version
Enter fullscreen mode Exit fullscreen mode

If it reports a client version and then unable to get crossplane version: Crossplane version or image tag not found, that is not a broken CLI. It means it cannot find an installation in the cluster yet, which at this point is correct.

2. Two clusters

crossplane in one and resources in the other.

kind create cluster --name crossplane-lab
kind create cluster --name workload-target

kubectl config use-context kind-crossplane-lab
Enter fullscreen mode Exit fullscreen mode

Remember to switch the context so that you dont apply in the wrong cluster

3. Install Crossplane in the control plane 

helm repo add crossplane-stable https://charts.crossplane.io/stable
helm repo update
helm upgrade --install crossplane crossplane-stable/crossplane \
  --namespace crossplane-system --create-namespace --wait
Enter fullscreen mode Exit fullscreen mode

4. Install provider-kubernetes

if you check the repo you will see it in this location providers/kubernetes/provider.yaml

apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-kubernetes
spec:
  package: xpkg.crossplane.io/crossplane-contrib/provider-kubernetes:v1.1.0
Enter fullscreen mode Exit fullscreen mode

Now apply the providers

kubectl apply -f providers/kubernetes/provider.yaml
kubectl get providers -w
Enter fullscreen mode Exit fullscreen mode

Wait for INSTALLED and HEALTHY.

5. The kubeconfig trick

This is the one non-obvious piece. A normal kind kubeconfig points at
127.0.0.1, which is meaningless from inside a pod in another cluster.

kind get kubeconfig --name workload-target --internal > /tmp/target.kubeconfig
grep server: /tmp/target.kubeconfig
Enter fullscreen mode Exit fullscreen mode

You want a hostname, not a loopback address:
server: https://workload-target-control-plane:6443

--internal rewrites the address to the target's name on the Docker network.
Both kind clusters sit on that network, so the control plane can reach it.

Load it into a secret:

kubectl create secret generic target-cluster \
  -n crossplane-system \
  --from-file=kubeconfig=/tmp/target.kubeconfig
Enter fullscreen mode Exit fullscreen mode

That kubeconfig grants cluster-admin on the target. Don't do this in prod please. Only for demonstration

Building the Workload primitive

The idea is that you want to reduce and streamline the work your user will do when they want to ship something so instead of them writing four different files, they just write one resource type. We decided to call this a workload earlier. It will say which image, which port, how much CPU and how much memory. Using this workload, the
platform produces a Deployment, a Service, a ConfigMap and an autoscaler. I am assuming you know these already to keep this article short

There are 3 components that make this work. Check them below

The XRD is the contract

A CompositeResourceDefinition popularly known as XRD is a schema. It declares which fields exist, which are required, and what the defaults are. If you have used Zod or Yup, it is the same idea: You can define a shape, validate input against it and reject what does not fit.

Two differences from a Zod or Yup schema. It registers a real API endpoint in the cluster, so kubectl get workloads starts working. Thats cool right? And validation runs server-side at admission, so a malformed Workload will be rejected before anything downstream sees it, no matter who submitted it or how. Check the definition.yaml files in the repo

apiVersion: apiextensions.crossplane.io/v2
kind: CompositeResourceDefinition
metadata:
  name: workloads.e01.lab.example.org
spec:
  scope: Namespaced
  group: e01.lab.example.org
  names:
    kind: Workload
    plural: workloads
  versions:
  - name: v1alpha1
    served: true
    referenceable: true
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            properties:
              image:
                type: object
                # ...
Enter fullscreen mode Exit fullscreen mode

The XRD is the only thing a creator of a workload interacts with.

An instance of that type is a Composite Resource, or XR. That is the thing the user or creator makes.

The composition is the recipe

So the XRD sits at the door to tell the user what is allowed, the Composition says what comes out. So given one Workload, produce these four resources, with these values computed from those fields.

apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: workload-local
spec:
  compositeTypeRef:
    apiVersion: e01.lab.example.org/v1alpha1
    kind: Workload
  mode: Pipeline
  pipeline:
  - step: render
    functionRef:
      name: function-go-templating
    input:
      # the template that emits the four resources
  - step: ready
    functionRef:
      name: function-auto-ready
Enter fullscreen mode Exit fullscreen mode

You may think this is the same as Helm but Helm renders once and applies. Crossplane renders on every reconcile and keeps reality matching the output. That continuous correction is the value proposition of crossplane.

Functions do the work

In v2 a composition cannot produce anything on its own. mode: Pipeline with at least one functionRef is the only mechanism there is. I think and at least at the time of writing this

Functions are packages, roughly the way an npm module or a Go package is. Think of them as a reusable unit someone else wrote that you pull in to do a job. The difference is that installing one runs a pod in your cluster. Crossplane calls it over gRPC on every reconcile, passing in observed state and taking back desired state.

Two are enough to start. One renders resources from Go templates, one computes
readiness:

apiVersion: pkg.crossplane.io/v1
kind: Function
metadata:
  name: function-go-templating
spec:
  package: xpkg.crossplane.io/crossplane-contrib/function-go-templating:v0.12.2
---
apiVersion: pkg.crossplane.io/v1
kind: Function
metadata:
  name: function-auto-ready
spec:
  package: xpkg.crossplane.io/crossplane-contrib/function-auto-ready:v0.5.0
Enter fullscreen mode Exit fullscreen mode

They live in a functions file and go in with kubectl like anything else:

kubectl apply -f functions/functions.yaml
kubectl get functions
Enter fullscreen mode Exit fullscreen mode

Wait for INSTALLED and HEALTHY on both before going further. Note that v2
dropped the default registry, so a fully qualified package URL is now
mandatory rather than good practice.

Applying it 

kubectl apply -f experiments/01-native-vs-remote/definition.yaml
kubectl api-resources | grep lab.example.org
Enter fullscreen mode Exit fullscreen mode

workloads e01.lab.example.org/v1alpha1 true Workload

This is a nice milestone moment. Your abstraction is now a
first-class API in the cluster, indistinguishable to a user from a built-in type.

Note NAMESPACED: true in that third column. That is scope: Namespaced
doing its job, and it is what lets two teams each have a hello Workload without colliding.

Lesson: referenceable is still required

The first apply failed: The CompositeResourceDefinition "workloads.e01.lab.example.org" is invalid: spec.versions[0].referenceable: Required value

That field exists to tell Crossplane which version a claim refers to. Claims were removed in v2.

Not a big problem tbh. The error names the field clearly and the fix is one line. But it is the first small sign of something that shows up repeatedly in this lab: the surface has moved faster than the parts underneath it, and the gaps are only visible when you try things.

Crossing the boundary

Crossplane v2 advertises that it can compose any Kubernetes resource
directly. That is true, and it works well. Now lets test, what happens when the resources needs to land somewhere else.

First, the local case

The first version composes into the control plane's own cluster. One
Workload in, four resources out, no credentials and no networking involved.

Note: the example ships pinned to compositionRef.name: workload-remote.
Set it to workload-local first for this section, then back for the remote case.

kubectl apply -f experiments/01-native-vs-remote/examples/workload.yaml
kubectl get deploy,svc,cm,hpa -n default
Enter fullscreen mode Exit fullscreen mode 
NAME                         READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/orders-api   1/1     1            1           30s

NAME                 TYPE        CLUSTER-IP      PORT(S)   AGE
service/orders-api   ClusterIP   10.96.135.216   80/TCP    30s
Enter fullscreen mode Exit fullscreen mode

Most of it works first time but two tweaks are needed.

Crossplane is not permitted to create ordinary Kubernetes resources out of the box, so the composition renders perfectly and then creates nothing. You have to give it an explicit grant:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: crossplane-compose-workloads
  labels:
    rbac.crossplane.io/aggregate-to-crossplane: "true"
rules:
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["*"]
- apiGroups: [""]
  resources: ["services", "configmaps"]
  verbs: ["*"]
- apiGroups: ["autoscaling"]
  resources: ["horizontalpodautoscalers"]
  verbs: ["*"]
Enter fullscreen mode Exit fullscreen mode

Now the dual cluster case

Now try to send that Deployment to another cluster.

You cannot. Not because it errors. Because there is no field anywhere in the composition where you could express it. Resources land in the control plane's own cluster because that is the only place they can land.

This is worth dwelling on, because it is a different kind of discovery from a failure. An error tells you what you did wrong but an absence tells you nothing at all, and you only notice it if you go looking for the field and find it is not there.

Object and ProviderConfig: cargo plus address

The mechanism for the above is provider-kubernetes, and it splits the problem in two.

A target cluster is, in this design, just Kubernetes. Nothing special
installed. So the control plane needs two separate things: something that says what should exist there, and something that says how to reach it.

An Object is the cargo. The object can have a kind of HorizontalPodAutoscaler, ConfigMap, Deployment, Service, etc. It wraps an ordinary manifest and adds an address:

apiVersion: kubernetes.m.crossplane.io/v1alpha1
kind: Object
spec:
  providerConfigRef:
    kind: ProviderConfig
    name: workload-target      # where it goes
  forProvider:
    manifest:                  # what should exist there
      apiVersion: apps/v1
      kind: Deployment
      # ...
Enter fullscreen mode Exit fullscreen mode

The manifest inside is a plain Deployment. Nothing about it is
Crossplane-specific. What Object adds is delivery instructions, and a reconcile loop that keeps checking.

A ProviderConfig is the address book entry. It gives a name to a set of credentials:

apiVersion: kubernetes.m.crossplane.io/v1alpha1
kind: ProviderConfig
metadata:
  name: workload-target
  namespace: default
spec:
  credentials:
    source: Secret
    secretRef:
      namespace: crossplane-system
      name: target-cluster
      key: kubeconfig
Enter fullscreen mode Exit fullscreen mode

The indirection matters. Compositions reference a ProviderConfig by name and know nothing about how to authenticate. When you rotate the kubeconfig, update the secret, every Object using that name picks it up. if you add a second customer cluster, add a second ProviderConfig, The composition will only need to change by one string. This is close to the Open close principle. The composition can be extended without modifying it.

Bringing it all together 

kubectl --context kind-workload-target get deploy,svc,cm,hpa -n default
Enter fullscreen mode Exit fullscreen mode 
NAME                         READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/orders-api   1/1     1            1           37s

NAME                 TYPE        CLUSTER-IP    PORT(S)   AGE
service/orders-api   ClusterIP   10.96.5.197   80/TCP    37s
Enter fullscreen mode Exit fullscreen mode

Four resources running in a cluster that has never heard of Crossplane:

kubectl --context kind-workload-target get crds
Enter fullscreen mode Exit fullscreen mode 
No resources found
Enter fullscreen mode Exit fullscreen mode

That is the control plane and data plane split made literal. Everything Crossplane knows lives in one cluster. Everything the user's workload actually is lives in another, with no agent, no operator, and no CRDs.

Lesson: the wrapper is the mechanism

It is tempting to read Object as legacy baggage. The v2 release notes say the new version removes the need for "awkward abstractions like claims and provider-kubernetes Objects", and in the common case that is true: if you are composing into your own cluster, native composition is cleaner.

But that is not the BYOC case. The moment the target is a cluster you do not own, the wrapper stops being awkward and becomes the only route across. It carries the one thing native composition has no way to express which is a destination.

The interesting thing here is that it is not about a boundary being about AWS versus GCP or one provider versus another. It is local composition versus remote composition, and everything that gets harder once the thing you are managing is somewhere you cannot simply log into and look. This ties to my BYOC article from earlier

Break it: readiness

Ok now lets move to understanding what readiness means across 2 clusters means. So now that we have stood up all the workloads, crossplan will say everything is up. But this claim in a control plane is a claim about a cluster you can or cannot see. In the case of BYOC you cannot always see this, so the only way to find out what crossplane's report means is to break something and check whether it will notices.

Point it at an image that does not exist 

kubectl patch workload.e01.lab.example.org orders-api --type=merge \
  -p '{"spec":{"image":{"versionSet":{"dev":{"path":"nginx:does-not-exist"}}}}}'
Enter fullscreen mode Exit fullscreen mode

The target cluster reacts immediately:

kubectl --context kind-workload-target get pods -n default
Enter fullscreen mode Exit fullscreen mode 
NAME                          READY   STATUS             RESTARTS   AGE
orders-api-74bf7fd79-mf7lz    1/1     Running            0          2m39s
orders-api-d54b4d5d6-wkj5g    0/1     ImagePullBackOff   0          42s
Enter fullscreen mode Exit fullscreen mode

The rollout is stuck. The new pod cannot start. Now the control plane:

crossplane resource trace workload.e01.lab.example.org orders-api
Enter fullscreen mode Exit fullscreen mode 
NAME                                        SYNCED   READY   STATUS
Workload/orders-api (default)               True     True    Available
├─ Object/orders-api-configmap (default)    True     True    Available
├─ Object/orders-api-deployment (default)   True     True    Available
├─ Object/orders-api-hpa (default)          True     True    Available
└─ Object/orders-api-service (default)      True     True    Available
Enter fullscreen mode Exit fullscreen mode

This is the same response for a healthy output in crossplane. Five rows of green over a workload that cannot run. This is what I saw with the config I have

Why: the default is the weakest of four

kubectl explain is where the answer is, and it is worth reading the exact wording:

kubectl explain object.spec.readiness --api-version=kubernetes.m.crossplane.io/v1alpha1
Enter fullscreen mode Exit fullscreen mode 
readiness <Object>
  Readiness defines how the object's readiness condition should be computed,
  if not specified it will be considered ready as soon as the underlying
  external resource is considered up-to-date.

FIELDS:
  policy <string>
  enum: SuccessfulCreate, DeriveFromObject, AllTrue, DeriveFromCelQuery
  celQuery <string>
Enter fullscreen mode Exit fullscreen mode

There is more info here

So without even looking carefully is easy to understand that up-to-date is not the same as healthy. The default is doing precisely what it says. The problem is mostly with the naming. The column heading says READY, and nothing anywhere tells you which of the four policies is in play (SuccessfulCreate, DeriveFromObject, AllTrue, DeriveFromCelQuery).

Note also, from the same output on spec, that watch is alpha and "not honored unless the watches feature gate is enabled". I had watch: true set on all four Objects from the start and assumed it was doing something it was not, and nothing said so. More to come in the drift section.

Fix, per kind

The instinct is to apply one policy everywhere. That is wrong, and it fails in the opposite direction.

A Deployment publishes an Available condition. A ConfigMap and a Service publish no conditions at all, so a query looking for Available on them would be false forever, and the XR would sit permanently red.

So the mapping is a per-kind decision, and that decision is a product choice rather than a technical detail. Mostly depending on what you want to surface to your users

Deployment. Derive from the condition Kubernetes already publishes:

spec:
  readiness:
    policy: DeriveFromCelQuery
    celQuery: object.status.conditions.exists(c, c.type == "Available" && c.status == "True")
Enter fullscreen mode Exit fullscreen mode

ConfigMap and Service. Existing is working:

spec:
  readiness:
    policy: SuccessfulCreate
Enter fullscreen mode Exit fullscreen mode

HPA. Left on SuccessfulCreate for now. It publishes AbleToScale, ScalingActive and ScalingLimited rather than Available, and with no metrics-server in kind, ScalingActive would be false for reasons that have nothing to do with the workload.

After

The image is still broken from the patch above, so this is the first run after applying the new policies:

NAME                                        SYNCED   READY   STATUS
Workload/orders-api (default)               True     False   Creating: Unready resources: deployment
├─ Object/orders-api-configmap (default)    True     True    Available
├─ Object/orders-api-deployment (default)   True     False   Unavailable
├─ Object/orders-api-hpa (default)          True     True    Available
└─ Object/orders-api-service (default)      True     True    Available
Enter fullscreen mode Exit fullscreen mode

The Deployment is red, the other three are green, and the XR names exactly what is wrong. Restore the image and it goes back:

kubectl patch workload.e01.lab.example.org orders-api --type=merge \
  -p '{"spec":{"image":{"versionSet":{"dev":{"path":"nginx:1.27"}}}}}'
Enter fullscreen mode Exit fullscreen mode 
Workload/orders-api (default)               True     True    Available
Enter fullscreen mode Exit fullscreen mode

Red to green on recovery matters as much as green to red. It proves the signal is derived rather than stuck.

Lesson: dashes were more honest than ticks

Before the Object wrapper, when the same four resources were composed
natively into the control plane's own cluster, the trace looked like this:

Workload/orders-api (default)               True     False   Creating: Unready resources: ...
├─ Deployment/orders-api (default)          -        -
├─ Service/orders-api (default)             -        -
Enter fullscreen mode Exit fullscreen mode

Blank without an opinion. Crossplane had no readiness convention for a plain Kubernetes resource and said so by saying nothing.

This is both ok and bad. Bad because it was like this regardless of the actual truth. But it was ok because it forces you to go and check why it is blank. It does not claim anything.

A green tick does claim something. It tells you the thing is fine, and because it is confident and specific and sits in a column labelled READY, it encourages you not to check. The wrapper's status output is more polished than the native path's.

Break it: drift

Readiness was about whether the control plane can see health. Drift is the other half which about realising the different between current and desired.

Delete it from the target cluster 

kubectl --context kind-workload-target delete deploy orders-api -n default
Enter fullscreen mode Exit fullscreen mode 
deployment.apps "orders-api" deleted
Enter fullscreen mode Exit fullscreen mode

Wait three minutes, which felt like more than enough:

sleep 180 && kubectl --context kind-workload-target get deploy -n default
Enter fullscreen mode Exit fullscreen mode 
No resources found in default namespace.
Enter fullscreen mode Exit fullscreen mode

Still gone. And the control plane:

kubectl get objects.kubernetes.m.crossplane.io -n default
Enter fullscreen mode Exit fullscreen mode 
NAME                    KIND         PROVIDERCONFIG    SYNCED   READY   AGE
orders-api-deployment   Deployment   workload-target   True     True    5m12s
Enter fullscreen mode Exit fullscreen mode

It says synced and ready but the Deployment it is describing does not exist.

Poke it and it comes straight back

Change something, anything, on the control-plane side:

kubectl annotate object.kubernetes.m.crossplane.io orders-api-deployment \
  -n default poke=1 --overwrite
Enter fullscreen mode Exit fullscreen mode 
sleep 15 && kubectl --context kind-workload-target get deploy -n default
Enter fullscreen mode Exit fullscreen mode 
NAME         READY   UP-TO-DATE   AVAILABLE   AGE
orders-api   0/1     1            0           15s
Enter fullscreen mode Exit fullscreen mode

Fifteen seconds, from an annotation that means nothing to anyone.

This was the find I did not expect. The reconciliation machinery works perfectly. It just is not triggered by the thing you would assume triggers it. Drift on the target is invisible; drift on the control plane is caught immediately. The loop watches its own side of the boundary.

Which is worth sitting with, because the mental model most people carry, mine included, is that the control plane is watching the target. It is not. It is watching its own resources and pushing outward on change.

watch: true was set the whole time

Every Object in the composition had it:

spec:
  watch: true
Enter fullscreen mode Exit fullscreen mode

I had put it there from the first version because the name says exactly what I wanted so I thought my job was done

kubectl explain object.spec --api-version=kubernetes.m.crossplane.io/v1alpha1
Enter fullscreen mode Exit fullscreen mode 
watch <boolean>
Watch enables watching the referenced or managed kubernetes resources.

THIS IS AN ALPHA FIELD. Do not use it in production. It is not honored unless "watches" feature gate is enabled, and may be changed or removed without notice.
Enter fullscreen mode Exit fullscreen mode

Accepted by the schema. Persisted in the object. Visible in every
kubectl get -o yaml. Doing nothing, with no warning, no condition and no event.

The gate belongs to the provider

This is the part that cost the most time. The feature gate is not a
Crossplane flag, so reinstalling Crossplane with extra arguments would have achieved nothing. It belongs to provider-kubernetes, and the way to pass flags to a provider pod is a DeploymentRuntimeConfig.

First, what the provider was running with:

kubectl -n crossplane-system get deploy \
  -o "custom-columns=NAME:.metadata.name,ARGS:.spec.template.spec.containers[0].args"
Enter fullscreen mode Exit fullscreen mode 
NAME                                  ARGS
crossplane                            [core start]
crossplane-rbac-manager               [rbac start --provider-clusterrole=...]
provider-kubernetes-0dc934180f0d      <none>
Enter fullscreen mode Exit fullscreen mode

No arguments at all.

apiVersion: pkg.crossplane.io/v1beta1
kind: DeploymentRuntimeConfig
metadata:
  name: provider-kubernetes-watches
spec:
  deploymentTemplate:
    spec:
      selector: {}
      template:
        spec:
          containers:
          - name: package-runtime
            args:
            - --enable-watches
            - --debug
Enter fullscreen mode Exit fullscreen mode

The container must be named package-runtime, and selector: {} is required to get past schema validation despite doing nothing. Then reference it from the Provider:

apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-kubernetes
spec:
  package: xpkg.crossplane.io/crossplane-contrib/provider-kubernetes:v1.1.0
  runtimeConfigRef:
    apiVersion: pkg.crossplane.io/v1beta1
    kind: DeploymentRuntimeConfig
    name: provider-kubernetes-watches
Enter fullscreen mode Exit fullscreen mode 
NAME                                  ARGS
provider-kubernetes-eb133ed78b8c      [--enable-watches --debug]
Enter fullscreen mode Exit fullscreen mode

With the gate on

Same test:

kubectl --context kind-workload-target delete deploy orders-api -n default
sleep 30 && kubectl --context kind-workload-target get deploy -n default
Enter fullscreen mode Exit fullscreen mode 
NAME         READY   UP-TO-DATE   AVAILABLE   AGE
orders-api   1/1     1            1           30s
Enter fullscreen mode Exit fullscreen mode

Deleted and already back. Sampling the Object's status every three seconds through the same window shows it was reported, not just silently fixed:

orders-api-deployment   Deployment   workload-target   True   False   3h47m
orders-api-deployment   Deployment   workload-target   True   True    3h47m
orders-api-deployment   Deployment   workload-target   True   True    3h47m
Enter fullscreen mode Exit fullscreen mode

Red at t+0, green by t+3. Detected in about a second, reported honestly, corrected. The provider's debug logs confirm what changed:

Running garbage collection for resource informers
  {"controller": "managed/object.kubernetes.m.crossplane.io", "count": 4}
Enter fullscreen mode Exit fullscreen mode

Four informers, one per Object, watching the target cluster. That machinery did not exist before, and the spec field alone could not create it.

A correction

Those same logs contain something that softens the original claim:

External resource is up to date
  {"requeue-after": "2026-09-06T10:38:50.100Z"}
Enter fullscreen mode Exit fullscreen mode

Ten minutes out. So reconciliation is not absent without the gate. It is on a roughly ten minute poll, and the 180 second test was simply too short to see it. Three tiers, then:

default:                ~10 min poll, status green throughout the gap
watch: true, gate off:  identical to default, field silently inert
watch: true, gate on:   informers on the remote cluster, seconds
Enter fullscreen mode Exit fullscreen mode

I want to state this plainly because it is the difference between "this does not work" and "this works, on a timescale nobody told you about, unless you turn on two things nobody told you about".

Lesson: accepted, inert, silent

The readiness finding was a bad default. This one is kind of worse, because there is a field whose name and description say exactly what you want, and setting it does nothing.

Nothing rejected it. Nothing warned. No condition appeared on the Object saying "watch requested but gate disabled". The only place the truth exists is one line of kubectl explain, and you would only go looking after something has already failed to happen.

For ten minutes at a time, a control plane in the default configuration will tell you a deleted resource is Synced and Ready. That is not a long window on a lab. On a fleet, it is the difference between an incident you caught and an incident a customer reported.

Connect two abstractions

One primitive on its own proves the mechanism. Two primitives that reference each other is where a platform starts to look like a platform, and it is where the interesting failure lives.

The shape is that a DataService composes Postgres onto the target as a StatefulSet with a PVC, a Service, and a Secret holding generated credentials. A Workload optionally names one, and if it does, gets the connection wired in.

The DataService publishes a contract

The composition writes back to its own XR's status:

---
apiVersion: {{ .observed.composite.resource.apiVersion }}
kind: {{ .observed.composite.resource.kind }}
status:
  host: {{ $name }}.{{ $ns }}.svc.cluster.local
  port: 5432
  secretName: {{ $secretName }}
Enter fullscreen mode Exit fullscreen mode 
kubectl get dataservice.e02.lab.example.org orders-db -o jsonpath='{.status}' | jq
Enter fullscreen mode Exit fullscreen mode 
{
  "host": "orders-db.default.svc.cluster.local",
  "port": 5432,
  "secretName": "orders-db-credentials"
}
Enter fullscreen mode Exit fullscreen mode

That status block is the interface between the two abstractions. Not the composition, not the XRD: the three fields one primitive publishes for another to consume.

The Workload resolves it by label

function-extra-resources which is a function looks up the referenced DataService at composition time:

  - step: fetch-dataservice
    functionRef:
      name: function-extra-resources
    input:
      apiVersion: extra-resources.fn.crossplane.io/v1beta1
      kind: Input
      spec:
        extraResources:
        - kind: DataService
          into: dataservice
          apiVersion: e02.lab.example.org/v1alpha1
          type: Selector
          selector:
            minMatch: 0
            maxMatch: 1
            matchLabels:
            - key: lab.example.org/dataservice
              type: FromCompositeFieldPath
              valueFromFieldPath: spec.dataServiceRef
Enter fullscreen mode Exit fullscreen mode

The template then reads the fetched resource out of the pipeline context and branches on whether it found anything:

{{- if $connected }}
DB_HOST: "{{ $ds.status.host }}"
DB_PORT: "{{ $ds.status.port }}"
DB_NAME: "{{ $ds.spec.database }}"
{{- end }}
Enter fullscreen mode Exit fullscreen mode

and mounts the DataService's own Secret rather than copying it:

                        envFrom:
                        - configMapRef:
                            name: {{ $name }}
                        {{- if $connected }}
                        - secretRef:
                            name: {{ $ds.status.secretName }}
                        {{- end }}
Enter fullscreen mode Exit fullscreen mode

That choice matters. The control plane learns the secret's name. It never sees the value. Which is the production pattern, and, as the next section shows, the reason it cannot tell you when the value stops working.

The password is generated once

Postgres would be unusable if the credential churned on every reconcile, so the composition reads the existing value out of observed state and only generates when there is nothing there:

{{- $password := "" }}
{{- if .observed.resources }}
{{- if index .observed.resources "credentials" }}
{{- $password = dig "resource" "spec" "forProvider" "manifest" "data" "password" "" (index .observed.resources "credentials") }}
{{- end }}
{{- end }}
{{- if eq $password "" }}
{{- $password = randAlphaNum 24 | b64enc }}
{{- end }}
Enter fullscreen mode Exit fullscreen mode

Two things that bit here. On the first reconcile .observed.resources is nil rather than an empty map, and Go templates evaluate arguments before default can help, so the guard has to wrap the whole lookup. And the reassignment inside the if needs = rather than :=, or it creates a block-scoped
variable, the outer one stays empty, and you get a fresh password every time without any obvious sign.

Verified by sampling the secret 45 seconds apart:

kubectl --context kind-workload-target get secret orders-db-credentials \
  -n default -o jsonpath='{.data.password}'
Enter fullscreen mode Exit fullscreen mode

Same value both times, and Postgres accepted it.

Note that crossplane render cannot test this. It has no observed state, so it generates a new password every run, which also rules out rending diffing for this composition.

Connected and standalone, side by side

Two Workloads from one composition. orders-api names a DataService,
search-api does not.

kubectl --context kind-workload-target exec -n default deploy/orders-api -- env \
  | grep -E "DB_|username|password|database" | sort
Enter fullscreen mode Exit fullscreen mode 
database=orders
DB_HOST=orders-db.default.svc.cluster.local
DB_NAME=orders
DB_PORT=5432
password=wbLttJ2Qcev0rKEuag7ogHc6-not-real
username=orders
Enter fullscreen mode Exit fullscreen mode 
kubectl --context kind-workload-target exec -n default deploy/search-api -- env \
  | grep -E "DB_|username|password|database" | sort
Enter fullscreen mode Exit fullscreen mode 
(nothing)
Enter fullscreen mode Exit fullscreen mode

And the same thing from the control plane's point of view:

kubectl get workload.e02.lab.example.org \
  -o custom-columns="NAME:.metadata.name,CONNECTED:.status.connected"
Enter fullscreen mode Exit fullscreen mode 
NAME         CONNECTED
orders-api   true
search-api   false
Enter fullscreen mode Exit fullscreen mode

One composition, one field of difference, and a working database connection in a cluster that has nothing to do with Crossplane.

Lesson: an optional dependency cannot be optional

search-api did not work first time. With dataServiceRef simply absent, the pipeline failed outright:

pipeline step "fetch-dataservice" returned a fatal result:
could not build extra resource requirements:
cannot get value from field path "spec.dataServiceRef": no such field
Enter fullscreen mode Exit fullscreen mode

minMatch: 0 does not help. It governs how many matches are acceptable, not whether the field exists. The path is resolved before any matching happens, so an absent field is fatal regardless of what the selector would have done with it.

The workaround is dataServiceRef: "". The empty string resolves, matches nothing, and the pipeline proceeds. An XRD default: "" handles it for real users, though not for crossplane render, which applies no defaults at all.

So an optional dependency requires the field to always be present. "Optional" becomes a matter of convention rather than something the schema can express, and every consumer of the API has to know to write an empty string rather than omitting the field.

Lesson: API groups do not isolate composed resource names

Experiment 01's Workload lives in e01.lab.example.org. Experiment 02's lives in e02.lab.example.org. Different API groups, deliberately, so both could be installed at once.

Both XRs were called orders-api. Both compositions name their Objects from spec.name. So both produced orders-api-deployment in the same namespace, and the API server refused the write:

cannot apply composed resource "deployment":
Object.kubernetes.m.crossplane.io "orders-api-deployment" is invalid:
metadata.ownerReferences: Only one reference can have Controller set to true. Found "true" in references for Workload/orders-api and Workload/orders-api
Enter fullscreen mode Exit fullscreen mode

Two owners, both claiming control, both named the same, distinguishable only by their apiVersion. It is one of the better error messages in this whole exercise: not silent, not last-write-wins, and it names both claimants.

The lesson generalises past a lab collision. Composed resource names are a flat namespace. Nothing scopes them to the XRD that produced them, so two teams shipping two abstractions onto one target will collide on any name either of them generates. Which is why the production composition I started from suffixes every Object name with its environment namespace.

Break it: credentials

Readiness was basically saying "can the control plane see health". Drift was "does it notice when the target changes". Credentials is the third face: does it notice when the value it delivered stops working.

The setup we're using

For this one the Workload has to do something with the credential rather than
just receive it. examples/db-checker.yaml swaps nginx for a small Express service (database-checker:local) that exposes /check, connects to Postgres using the injected env vars and runs SELECT version().

The service source lives at
experiments/02-connected-resources/database-checker/. Build the image and side-load it into the target cluster (there is no registry involved), then apply the example:

make e02-image
kubectl apply -f experiments/02-connected-resources/examples/db-checker.yaml
Enter fullscreen mode Exit fullscreen mode

Baseline: nine green rows and a connection 

kubectl --context kind-workload-target exec deployment/db-checker -- \
  wget -qO- http://localhost:3000/check
Enter fullscreen mode Exit fullscreen mode 
{"status":"connected","details":{"version":"PostgreSQL 16.15 (Debian 16.15-1.pgdg13+2) on aarch64-unknown-linux-gnu, compiled by gcc (Debian 14.2.0-19) 14.2.0, 64-bit"}}
Enter fullscreen mode Exit fullscreen mode 
crossplane resource trace dataservice.e02.lab.example.org/checker-db -n default
crossplane resource trace workload.e02.lab.example.org/db-checker -n default
Enter fullscreen mode Exit fullscreen mode 
NAME                                         SYNCED   READY   STATUS
DataService/checker-db (default)             True     True    Available
├─ Object/checker-db-credentials (default)   True     True    Available
├─ Object/checker-db-service (default)       True     True    Available
└─ Object/checker-db-statefulset (default)   True     True    Available

NAME                                        SYNCED   READY   STATUS
Workload/db-checker (default)               True     True    Available
├─ Object/db-checker-configmap (default)    True     True    Available
├─ Object/db-checker-deployment (default)   True     True    Available
├─ Object/db-checker-hpa (default)          True     True    Available
└─ Object/db-checker-service (default)      True     True    Available
Enter fullscreen mode Exit fullscreen mode

Nine green rows across two traces. The application confirms it can connect.

Rotate the password on the target

Do it directly on Postgres, bypassing the control plane. This is the kind of thing a DBA might do during an incident, or an out-of-band rotation script, or a manual reset.

kubectl --context kind-workload-target exec statefulset/checker-db -c postgres -- \
  psql -U checker -d checker \
    -c "ALTER USER checker WITH PASSWORD 'rotated-out-of-band';"
Enter fullscreen mode Exit fullscreen mode 
ALTER ROLE
Enter fullscreen mode Exit fullscreen mode

After: nine green rows and a failing app 

crossplane resource trace dataservice.e02.lab.example.org/checker-db -n default
crossplane resource trace workload.e02.lab.example.org/db-checker -n default
Enter fullscreen mode Exit fullscreen mode 
NAME                                         SYNCED   READY   STATUS
DataService/checker-db (default)             True     True    Available
├─ Object/checker-db-credentials (default)   True     True    Available
├─ Object/checker-db-service (default)       True     True    Available
└─ Object/checker-db-statefulset (default)   True     True    Available

NAME                                        SYNCED   READY   STATUS
Workload/db-checker (default)               True     True    Available
├─ Object/db-checker-configmap (default)    True     True    Available
├─ Object/db-checker-deployment (default)   True     True    Available
├─ Object/db-checker-hpa (default)          True     True    Available
└─ Object/db-checker-service (default)      True     True    Available
Enter fullscreen mode Exit fullscreen mode

Nine identical green rows. And the app:

kubectl --context kind-workload-target exec deployment/db-checker -- \
  node -e "require('http').get('http://localhost:3000/check', r => { let d=''; r.on('data', c=>d+=c); r.on('end', () => { console.log('HTTP', r.statusCode); console.log(d); }); });"
Enter fullscreen mode Exit fullscreen mode 
HTTP 500
{"status":"error","error":"password authentication failed for user \"checker\""}
Enter fullscreen mode Exit fullscreen mode

So you can see that for a broken image, readiness went red, drift went red once the gate was on, however broken connection from secret rotation this stays green.

Why there is nothing to notice

The mechanism from the "Connect two abstractions" section is the reason. The Workload does not copy the credential onto the control plane; it holds the name of the Secret on the target and mounts it via envFrom.secretRef. Object/checker-db-credentials still contains the same Secret manifest it always did, and applying that manifest still succeeds. From the control plane's side of the boundary, nothing has changed.

Lesson: reference-and-report is not verify

If you remeber my last artcle about the clear boundaries of a control plane especially in BYOC context, the credential design is right (referring beats copying, keeps the secret off the control plane). Many teams have different ways to manage this gap with a checker sidecar, a scheduled Operation, an external prober writing results back somewhere the control plane can read etc.

Conclusion

Reconcile versus reference is not a property of the tool. Crossplane will do either, and by default it does the weaker one.

Past a certain point it stops being a configuration choice and becomes a structural limit. Anything whose health lives outside the Kubernetes API is something the control plane can create, reference and report on, but sometimes cannot verify.

So the questions worth asking a BYOC vendor or your infra team are not about architecture diagrams. They are:

  • Is drift detected, and on what timescale?
  • Does a green status mean the resource was applied, or that it works?
  • What happens to that status when something changes outside your control plane?
  • How would I know, from your dashboard, that you had stopped watching?

A vendor/team who has thought about this will answer specifically. A vendor/team who has not will show you a screen full of ticks.

The repo is at github.com/Joojo7/crossplane-lab.

Top comments (0)