Crossplane v2.3 Deep Dive — High-Fidelity Render Engine, Provider Deletion Protection, Reconciliation Annotations, and CLI Separation

Crossplane v2.3 Deep Dive — High-Fidelity Render Engine, Provider Deletion Protection, Reconciliation Annotations, and CLI Separation Redefining the 2026 Kubernetes Control Plane

On May 21, 2026, the Crossplane maintainers shipped v2.3.0, the quarterly release that — for the first time in the v2 series — turned the "production-grade control plane" pitch into measurable operational evidence. v2.0 brought the big architectural earthquake (namespaced XRs and MRs, composing any resource, Operations), v2.1/v2.2 made it run, and v2.3 closes the long-standing day-2 gaps that platform teams have been quietly working around for years.

Six changes carry the release:

1. High-Fidelity Render Engine — crossplane render now drives the real in-cluster composite reconciler via a hidden crossplane internal render subcommand, instead of a parallel reimplementation.
2. Alpha Provider Deletion Protection — Crossplane auto-creates ClusterUsage resources that block Provider deletion through the existing Usage webhook while managed resources of that Provider's kinds still exist.
3. Two new reconciliation annotations — crossplane.io/poll-interval overrides the controller-level poll interval per-resource, and crossplane.io/reconcile-requested-at triggers an immediate reconcile whenever the value changes.
4. XR Circuit Breaker reset — when an XR is deleted, its circuit-breaker state is now discarded so a same-named replacement starts clean.
5. No-op status update skip for CompositionRevision and composite reconcilers, behind the alpha gate --enable-no-op-status-update-skip.
6. Crossplane CLI repository split — the CLI moves to its own repository (crossplane/crossplane-cli) with an independent release cadence.

This article unpacks each of the six changes at the level of code paths and alpha gate flags, and lays out the staged upgrade/observation/rollback workflow we used at ManoIT across three control planes (prod/stage/dev).

Disclosure: cross-posted from the ManoIT tech blog. Original (Korean) published 2026-05-27. AI-assisted authoring with editorial review.

1. Why May 21, 2026 Is a Crossplane Inflection Point

Crossplane was open-sourced by Upbound in December 2018, joined the CNCF as a Sandbox project in 2020, became Incubating in 2021, and was promoted to CNCF Graduated on October 28, 2025. The identity drift between v1 ("Kubernetes-native IaC, a Terraform alternative") and v2 ("a control-plane SDK on top of the Kubernetes API") matters because it changes how you should read the v2.3 release notes.

Date	Event	Operational meaning
2018-12	Upbound open-sources Crossplane	"IaC on Kubernetes" vision begins
2020-09	CNCF Sandbox accepted	Community governance, stage 1
2021-09	CNCF Incubating	Production-use accumulation phase
2024-05	v1.17 — native patch & transform deprecated	Composition Function era declared
2025-07	v2.0 — namespaced XR/MR, Operations alpha, compose any resource	Identity shift to "control-plane SDK"
2025-10-28	CNCF Graduated	Enterprise adoption guidance formalized
2025-11	v2.1 — namespaced MR stable, MRD alpha	Selective resource activation for large Providers
2026-03	v2.2 — Pipeline Inspector, RequiredSchemas, ImageConfig, XRD CEL validation	Composition Function debugging/validation gaps closed
2026-05-21	v2.3 — Render Engine unification, Provider deletion protection, reconcile annotations ×2, CLI split	Local ↔ cluster gap removed; operational safety net hardened
2027-02	v2.3 EOL planned	Quarterly release + 9-month support window maintained

Two operational headlines:

• The six-year chronic pain of "render locally, fail in cluster" is structurally gone — the maintainers retired the parallel reconciler used by crossplane render and now expose the real composite reconciler as the hidden crossplane internal render subcommand, which crossplane render (and downstream tools like crossplane-diff) calls.
• Two new alpha gates (--enable-provider-deletion-protection, --enable-no-op-status-update-skip) close the operational safety-net gap immediately — accidental Provider deletion is the #1 way Crossplane operators have orphaned MRs, and the no-op status skip cuts ETCD PUT pressure that scales linearly with cluster size.

2. High-Fidelity Render Engine — Removing the Local Render ↔ Cluster Reconcile Gap

The biggest maintainer-side work in v2.3 happened where you couldn't see it. crossplane render has, since the v1 days, been the way to preview how an XR resolves through a Composition Function pipeline. The catch: the reconciler that render used was a structurally separate reimplementation of what the in-cluster composite controller actually ran.

2.1 The Two-Reconciler Gap Through v2.2

Axis	Local render (v2.2)	In-cluster controller (v2.2)
Reconciler implementation	Render-only reimplementation	Official composite package implementation
Pipeline step context	Partial propagation	Full propagation
Required Resources/Schemas	Partial in local	Full RequiredSchemas since v2.2
Managed metadata (labels, owner refs)	Some missing post-render	Attached as actually applied
Downstream tools (crossplane-diff)	Inherited the gap through render output	—
"Works locally, breaks in cluster" issues	Frequent	—

2.2 The v2.3 Fix — Share Code via crossplane internal render

v2.3 exposes the in-cluster composite reconciler as a callable subcommand. The new name is crossplane internal render — the "internal" prefix is deliberate: it's a backend for other tools, not a user-facing command. crossplane render now calls this backend, so local output goes through the exact same code path as the cluster.

# v2.3: crossplane render now invokes the same composite reconciler internally
crossplane render \
  examples/xr.yaml \
  examples/composition.yaml \
  examples/functions.yaml \
  --include-full-xr \
  --include-context

# You can now 1:1 compare the local output to what the controller produces in cluster
kubectl get app my-app -o yaml > cluster.yaml
crossplane render examples/xr.yaml examples/composition.yaml examples/functions.yaml > local.yaml
diff cluster.yaml local.yaml   # Note: post-v2.3, substantive diff is 0 modulo metadata/status

The downstream impact is large. crossplane-diff, crossplane-test, and every CI workflow that validates compositions on PRs were all subject to the same gap, so v2.3 removes one whole class of "PR was green, merge broke prod" incident.

2.3 Operational Application — Render-vs-Cluster Diff Gate in CI

# .github/workflows/crossplane-composition-check.yml
# Note: pre-v2.3, this comparison is meaningless because of the render gap
name: Crossplane Composition Drift Gate
on:
  pull_request:
    paths: ["compositions/**", "xrs/**", "functions/**"]

jobs:
  drift-gate:
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@v5
      - name: Install Crossplane CLI v2.3.0
        run: |
          curl -sSL "https://releases.crossplane.io/stable/v2.3.0/bin/linux_amd64/crossplane" -o crossplane
          install -m 0755 crossplane /usr/local/bin/crossplane
          crossplane version --client   # client: 2.3.0

      - name: Render with Composition Function pipeline
        run: |
          # High-Fidelity Render — same code path as in-cluster controller
          crossplane render \
            xrs/${{ matrix.xr }}.yaml \
            compositions/${{ matrix.composition }}.yaml \
            functions/index.yaml \
            --include-full-xr \
            --include-context \
            -o yaml > rendered.yaml

      - name: Fetch live cluster state (prod read-only)
        run: |
          kubectl --context prod-ro get $(yq '.kind' rendered.yaml) \
            $(yq '.metadata.name' rendered.yaml) -o yaml \
            | yq 'del(.metadata.managedFields, .metadata.resourceVersion, .metadata.uid, .status)' \
            > cluster.yaml

      - name: Diff and fail on unexpected drift
        run: |
          diff -u cluster.yaml rendered.yaml || {
            echo "::error::Composition output diverges from cluster — review before merge"
            exit 1
          }

3. Alpha Provider Deletion Protection — Auto ClusterUsage + Usage Webhook

The second change targets the highest-frequency operator incident: "I accidentally deleted a Provider and every MR of its kinds was orphaned." Every Crossplane operator has either lived this or heard the story.

3.1 The Existing Usage Webhook's Limitation — XR/MR Level Only

Crossplane has shipped the Usage resource since v1 to express "while resource A is in use, refuse to delete resource B." A ValidatingAdmissionWebhook intercepts DELETE requests and rejects them if a Usage still names a live dependency. The problem: Provider packages themselves had no equivalent guard. One kubectl delete provider provider-aws would strand every MR of every kind that Provider defined.

3.2 v2.3 — Auto-Create ClusterUsage to Block Provider DELETE

v2.3 introduces the alpha gate --enable-provider-deletion-protection. When on, Crossplane automatically:

Step	Action	Implementation
1	On Provider install, create a ClusterUsage	Provider controller creates kind: ClusterUsage at bootstrap, spec.of points to the Provider itself
2	While MRs of that Provider's kinds exist, mark ClusterUsage Active	spec.by selector auto-maps to the Provider's CRD labels
3	Provider DELETE intercepted by Usage webhook	Reuses the existing Usage webhook code path
4	DELETE allowed only when active MRs = 0	Otherwise: HTTP 422 + human-readable message
5	Provider tear-down requires explicit opt-out	Operator clears all MRs, then kubectl delete clusterusage protect-provider-aws-...

3.3 Turning It On — Helm Values + Gate Flag

# helm/crossplane-values.yaml
# Note: alpha feature — verify for 1 week in staging before production
crossplane:
  args:
    - --debug
    - --enable-environment-configs
    - --enable-operations
    - --enable-provider-deletion-protection   # v2.3 alpha gate
    - --enable-no-op-status-update-skip       # v2.3 alpha — cut ETCD writes
  resourcesCrossplane:
    limits:
      cpu: "500m"
      memory: "1Gi"
    requests:
      cpu: "100m"
      memory: "256Mi"
  metrics:
    enabled: true   # Prometheus scraping recommended

# Verify after enable
helm upgrade crossplane crossplane-stable/crossplane \
  --version 2.3.0 \
  --namespace crossplane-system \
  -f helm/crossplane-values.yaml

# ClusterUsage is auto-created on Provider install
kubectl get clusterusage
# NAME                              OF                       BY                 AGE
# protect-provider-aws-12fa3        provider-aws             provider-aws-mrs   1m

# Intentional delete attempt — refused
kubectl delete provider provider-aws
# Error from server (Forbidden): admission webhook "no-usages.apiextensions.crossplane.io" denied the request:
# this provider is in-use by 247 managed resources of 12 kinds: cannot delete

3.4 Standard Tear-Down Workflow

Even with the alpha gate on, you still need a clean tear-down procedure. Our ManoIT standard:

# Step 1: inventory every MR of the Provider
kubectl get $(kubectl api-resources --api-group=aws.upbound.io -o name | paste -sd, -) -A \
  -o jsonpath='{range .items[*]}{.kind}{"\t"}{.metadata.namespace}{"/"}{.metadata.name}{"\n"}{end}' \
  > aws-mrs-inventory.tsv

# Step 2: decide deletionPolicy=Orphan or proper delete, apply in bulk
xargs -a aws-mrs-inventory.tsv -I{} kubectl patch {} \
  --type=merge -p '{"spec":{"deletionPolicy":"Orphan"}}'

# Step 3: delete all MRs — ClusterUsage auto-transitions to Inactive
xargs -a aws-mrs-inventory.tsv -I{} kubectl delete {}
kubectl get clusterusage protect-provider-aws-12fa3 -o yaml | yq '.status.conditions[0].reason'
# Inactive

# Step 4: remove ClusterUsage → Provider delete now allowed
kubectl delete clusterusage protect-provider-aws-12fa3
kubectl delete provider provider-aws

4. Two Reconciliation Annotations — Per-Resource Polling and Immediate Trigger

The third change resolves a six-year-old operator ask: per-resource reconcile cadence control, via two annotations.

Annotation	Meaning	Example	Use case
crossplane.io/poll-interval	Override controller-level poll interval for this resource	"24h", "30m", "5m"	Low-volatility IAM, baseline infra
crossplane.io/reconcile-requested-at	Trigger an immediate reconcile whenever the value changes	RFC3339 timestamp ("2026-05-27T08:15:00Z")	Post-external-change sync, debugging, operational force-refresh

4.1 Per-Resource Poll Override — End of Global-Only Cadence

Through v2.2 the poll interval was a single controller-startup flag (--poll-interval). The result: an IAM Role that almost never changes was polled at the same cadence as an RDS Instance, inflating cloud-API call cost and controller load.

# IAM Role — barely changes, 24h polling is enough
apiVersion: iam.aws.m.upbound.io/v1beta1
kind: Role
metadata:
  namespace: platform
  name: eks-node-role
  annotations:
    crossplane.io/poll-interval: "24h"   # v2.3 new
spec:
  forProvider:
    assumeRolePolicy: |
      {"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"ec2.amazonaws.com"},"Action":"sts:AssumeRole"}]}
---
# RDS Instance — fast sync for backup/snapshot state, 1m polling
apiVersion: rds.aws.m.upbound.io/v1beta1
kind: Instance
metadata:
  namespace: marketing
  name: marketing-pg
  annotations:
    crossplane.io/poll-interval: "1m"
spec:
  forProvider:
    region: ap-northeast-2
    engine: postgres
    engineVersion: "18.4"

4.2 Immediate Trigger — Post-External-Change Sync

The second annotation, reconcile-requested-at, re-enqueues the resource immediately whenever its value changes. Two operational examples:

# Scenario 1: rotate the RDS master password out-of-band, force immediate sync
aws rds modify-db-instance --db-instance-identifier marketing-pg \
  --master-user-password "$(openssl rand -base64 32)" --apply-immediately
kubectl annotate -n marketing instance.rds.aws.m.upbound.io marketing-pg \
  crossplane.io/reconcile-requested-at="$(date -u +%FT%TZ)" --overwrite

# Scenario 2: debug — a new Composition Function just merged, force re-evaluate every XR
kubectl get app -A -o name | while read xr; do
  ns=$(echo $xr | awk -F/ '{print $1}')
  name=$(echo $xr | awk -F/ '{print $2}')
  kubectl annotate -n $ns $xr \
    crossplane.io/reconcile-requested-at="$(date -u +%FT%TZ)" --overwrite
done

5. XR Circuit Breaker Reset — Same-Named Replacements Start Clean

Through v2.2, if an XR tripped its circuit breaker (Crossplane's protection against reconcile thrashing) and you deleted the XR, the breaker state was inherited by any same-named replacement. The natural recovery instinct — "delete it, recreate it, it'll work" — didn't actually work, and operators ended up restarting controller pods to flush state.

v2.3 discards the circuit-breaker state the moment the XR is deleted. A same-named replacement starts from the same clean state as a brand-new resource.

Scenario	v2.2 (before)	v2.3 (after)
XR reconcile is throttled by thrashing	Circuit open	Circuit open
Operator deletes the XR	Circuit state cached/retained	Circuit state discarded immediately
Same-named XR recreated	Inherits open circuit → no reconcile after recreate	Starts clean → reconciles immediately
Recovery procedure	Restart controller pod or use a different name	Same name + recreate is sufficient
Operational cognitive cost	Tribal-knowledge accretion	Reduced to a 1-step standard procedure

6. No-op Status Update Skip on CompositionRevision/Composite Reconcilers

The fifth change is ETCD write-load optimization. Through v2.2, the CompositionRevision controller and the composite reconciler issued a status update PUT every reconcile loop, even when nothing in the status had actually changed. At cluster scale this "no-change status PUT" was a measurable fraction of ETCD traffic.

v2.3 compares the previous and new status and skips the PUT when they're identical. Enable with the alpha gate --enable-no-op-status-update-skip. On our staging cluster (~4,200 MRs) we measured ETCD PUT call volume down ~31%, apiserver CPU down ~18% in steady state. The effect scales with cluster size.

# Prometheus queries — before/after the alpha gate
# (1) ETCD PUT call volume
sum(rate(etcd_request_duration_seconds_count{operation="put"}[5m]))

# (2) apiserver CPU
sum(rate(container_cpu_usage_seconds_total{namespace="kube-system",pod=~"kube-apiserver-.*"}[5m]))

# (3) Crossplane controller's own reconcile rate (side-effect watch)
sum(rate(controller_runtime_reconcile_total{controller="composite"}[5m])) by (result)

7. Crossplane CLI Repository Split and Independent Release Cycle

The sixth change touches even non-coder operators. The CLI (formerly called crank) leaves the core repo with v2.3.0. Its new home is github.com/crossplane/crossplane-cli, and from here on out the version numbers and release cadences are independent.

Axis	Pre-v2.3	Post-v2.3
Repository	crossplane/crossplane (single)	crossplane/crossplane (core) + crossplane/crossplane-cli (CLI)
Version sync	Always identical	Independent — CLI can move faster
Release cadence	Quarterly (3 months)	Core quarterly, CLI as needed
Install command	curl ... /bin/linux_amd64/crank	curl ... /bin/linux_amd64/crossplane (unified name)
Version compatibility	1:1	CLI guarantees backwards compat to core N-2
New command	crossplane beta trace (table-only)	crossplane beta trace -o yaml (YAML output added)

7.1 YAML Trace Output — GitOps Friendliness

# v2.3 new: take trace output as YAML and pipe to other tools
crossplane beta trace -o yaml app/my-app -n marketing > trace.yaml

# Combine with kubectl-tree to visualize control-plane topology
yq '.children[].name' trace.yaml | while read child; do
  kubectl tree $child -n marketing
done

# Drift detection — store trace output in Git, surface diffs as PRs
cp trace.yaml history/trace-$(date +%Y%m%d).yaml
git add history/trace-*.yaml && git commit -m "chore: nightly trace snapshot"

8. Upgrade Workflow — v2.2 → v2.3 (Non-Disruptive Standard)

v2.3 is a minor upgrade inside the v2.x series, so API compatibility is preserved. Alpha gates must still be staged. Our standard four-step procedure:

# Step 1: precondition check — Provider/Function packages are fully qualified URLs
# v2 rejects short names, so this needs verification just before v2.2 → v2.3
kubectl get pkg -A -o jsonpath='{range .items[*]}{.kind}{"\t"}{.metadata.name}{"\t"}{.spec.package}{"\n"}{end}' \
  | awk -F'\t' '$3 !~ /\// {print "❌ NOT FQ:", $0}'
# (must be empty to pass)

# Step 2: dev cluster upgrade — alpha gates OFF
helm upgrade crossplane crossplane-stable/crossplane \
  --version 2.3.0 \
  --namespace crossplane-system \
  --reuse-values \
  --wait
kubectl -n crossplane-system get deploy crossplane -o jsonpath='{.spec.template.spec.containers[0].image}'
# crossplane/crossplane:v2.3.0

# Step 3: regression — render-diff every Composition against golden output
for f in compositions/*.yaml; do
  comp=$(yq '.metadata.name' $f)
  crossplane render xrs/test-${comp}.yaml $f functions/index.yaml > /tmp/render-$comp.yaml
  diff /tmp/render-$comp.yaml golden/render-$comp.golden.yaml \
    || { echo "❌ regression in $comp"; exit 1; }
done

# Step 4: staged staging → prod rollout. Alpha gates: ON in staging first, then prod
helm upgrade crossplane crossplane-stable/crossplane \
  --version 2.3.0 \
  --namespace crossplane-system \
  --set 'args={--debug,--enable-environment-configs,--enable-operations,--enable-provider-deletion-protection,--enable-no-op-status-update-skip}' \
  --wait

9. ManoIT In-House Adoption Checklist — Three Control Planes × Sixteen Steps

ManoIT runs three control planes (prod/stage/dev), so we stage the alpha gates: one week in staging, one more week in prod, then enable. The full checklist:

#	Item	Owner	Done when
1	Inventory Crossplane/Provider/Function versions on all three control planes	Platform	Merged spreadsheet
2	Audit fully qualified package URLs — flag any remaining short names	Platform	kubectl get pkg shows 0 NOT-FQ
3	Upgrade dev to v2.3.0 (alpha gates OFF)	Platform	crossplane version --server = v2.3.0
4	Composition regression — render vs. golden	Service owners	All diffs = 0
5	Staging: v2.3.0 + --enable-no-op-status-update-skip	Platform	1-week soak, ETCD PUT delta report
6	Staging: add --enable-provider-deletion-protection	Platform	ClusterUsage auto-created, refused-delete smoke test passes
7	Apply crossplane.io/poll-interval to low-volatility MRs (IAM/VPC)	Service owners	≥30% drop in cloud API calls post-apply
8	Standardize force-refresh procedure on reconcile-requested-at — runbook update	SRE	Runbook merged, ≥1 incident application
9	Upgrade prod to v2.3.0 (alpha gates OFF)	Platform	crossplane version --server = v2.3.0
10	Prod: --enable-no-op-status-update-skip	Platform	1-week soak, ETCD PUT + apiserver CPU report
11	Prod: --enable-provider-deletion-protection	Platform	ClusterUsage created; tear-down doc updated to "delete ClusterUsage → delete Provider"
12	CI gate added — High-Fidelity Render diff on PRs	Platform	Workflow merged, ≥1 regression PR blocked in practice
13	Internal asdf/Mise picks Crossplane CLI from new repo	Platform	asdf install crossplane 2.3.0 works
14	crossplane beta trace -o yaml snapshotted daily to Git	Observability	Nightly Cron + PR automation
15	Prometheus alerts: XR circuit breaker open, MR poll-interval=24h+ ratio	Observability	Alert PR merged, fire/resolve test passed
16	RFC: alpha gate enablement policy (dev immediate, staging 1w, prod +1w)	Platform	RFC merged, in quarterly security/release review

10. Conclusion — The Inflection Point of "Operationally Trustworthy Control Plane"

The six v2.3 changes share one sentence: "the v2 series, for the first time, demonstrates its reliability claim with operational metrics." The High-Fidelity Render Engine structurally resolves the six-year "local diverges from cluster" pain. Provider Deletion Protection blocks the top-incident scenario with one alpha gate. The two reconcile annotations finally hand operators the per-resource cadence control they've been asking for. The XR circuit-breaker reset simplifies post-incident recovery. The no-op status update skip removes ETCD write pressure. The CLI repo split opens a faster lane for CLI-side evolution.

Three things to keep in mind going into adoption:

1. Stage the alpha gates. dev OFF for regression, staging ON for 1 week, prod for another 1 week before flipping.
2. Re-audit fully qualified package URLs immediately before upgrading. v2 rejects short names. Leftover short-name packages will fail controller boot even on a v2.2 → v2.3 minor upgrade.
3. The High-Fidelity Render payoff shows up in CI, not at the CLI. A single crossplane render call won't feel different. Wire render-diff into PR gates and you'll catch composition regressions before merge.

Shortest single-line recommendation: "Upgrade dev to v2.3 today, enable both alpha gates in staging this week."

---

Cross-posted from ManoIT Tech Blog. Authored by the ManoIT Platform Team with AI-assisted drafting (Claude Opus 4.6) on May 27, 2026. All operational figures cited are from internal staging measurements and are reproducible on any cluster of comparable size.

---

Originally published at ManoIT Tech Blog.