DEV Community

daniel jeong
daniel jeong

Posted on • Originally published at manoit.co.kr

Crossplane v2.2 Deep Dive — Pipeline Inspector, CEL Validation, and Production Control Planes

#crossplane #kubernetes #platformengineering #devops

Crossplane v2.2 Deep Dive — Pipeline Inspector, CEL Metadata Validation, and Building Production Control Planes

In March 2026, Crossplane shipped v2.2 with long-awaited features for platform engineering teams: pipeline debugging, metadata-level validation, unified runtime configuration, and schema-aware composition functions. Unlike Terraform or Pulumi, Crossplane is a Kubernetes-native control plane framework that leverages the Kubernetes API directly, integrating seamlessly with GitOps workflows. This article provides an in-depth analysis of every new feature with practical examples, plus a migration strategy for v1 users.

1. Crossplane v2 Architecture — What Changed from v1

Before diving into v2.2 specifics, let's examine the fundamental architectural shifts introduced in v2.

Aspect Crossplane v1 Crossplane v2.x
Resource Scope Cluster-scoped (Claims for namespace access) Namespaced by default (Cluster scope optional)
Claims Required (XRC → XR mapping) Removed — XRs exist directly in namespaces
Composition Mode Patch & Transform (P&T) default Pipeline mode only (P&T removed)
Composable Resources Crossplane Managed Resources only Any Kubernetes resource (Deployments, CRDs, etc.)
Operations None Operation / CronOperation / WatchOperation
Package Registry Default registry auto-applied Fully Qualified Image URLs required
Metadata Structure Mixed in spec Operational fields separated to spec.crossplane

The most significant change is the complete removal of Claims. In v1, developers used Claims for namespace-level infrastructure requests while platform teams managed XRs at the cluster level. In v2, XRs exist directly in namespaces, enabling Kubernetes RBAC for access control.

# Crossplane v2 — XRD (CompositeResourceDefinition) Example
apiVersion: apiextensions.crossplane.io/v2
kind: CompositeResourceDefinition
metadata:
  name: databases.platform.example.com
spec:
  group: platform.example.com
  names:
    kind: Database
    plural: databases
  # v2: Namespaced by default — Cluster scope also available
  scope: Namespaced
  versions:
    - name: v1alpha1
      served: true
      referenceable: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                engine:
                  type: string
                  enum: ["postgres", "mysql", "aurora-postgres"]
                  description: "Database engine type"
                size:
                  type: string
                  enum: ["small", "medium", "large"]
                  default: "small"
                region:
                  type: string
                  default: "ap-northeast-2"
              required: ["engine"]
Enter fullscreen mode Exit fullscreen mode

2. Pipeline Inspector — Revolutionizing Composition Debugging

The headline feature of Crossplane v2.2 is the Pipeline Inspector. Previously, Composition Functions were essentially black boxes — when a pipeline error occurred, tracing which function failed on which input was extremely difficult. Pipeline Inspector solves this fundamentally.

2.1 How It Works

When enabled, the Crossplane core controller intercepts all function requests and responses and forwards them via Unix socket to a sidecar container. The sidecar exposes a gRPC endpoint that can dump to stdout in development or forward to audit/observability systems in production.

# Helm values: Enable Pipeline Inspector
# values-pipeline-inspector.yaml
args:
  - --enable-pipeline-inspector       # Enable alpha feature
  - --pipeline-inspector-socket=/var/run/pipeline-inspector/socket

extraVolumes:
  - name: inspector-socket
    emptyDir: {}

extraVolumeMounts:
  - name: inspector-socket
    mountPath: /var/run/pipeline-inspector

sidecars:
  - name: inspector
    image: xpkg.crossplane.io/crossplane/inspector-sidecar:v0.0.3
    resources:
      requests:
        cpu: 10m
        memory: 64Mi
      limits:
        cpu: 100m
        memory: 128Mi
    volumeMounts:
      - name: inspector-socket
        mountPath: /var/run/pipeline-inspector
Enter fullscreen mode Exit fullscreen mode

2.2 Deploying the Inspector 

# Install Crossplane v2.2 with Pipeline Inspector
helm repo add crossplane-stable https://charts.crossplane.io/stable
helm repo update

helm upgrade --install crossplane crossplane-stable/crossplane \
  --namespace crossplane-system \
  --create-namespace \
  --version 2.2.0 \
  -f values-pipeline-inspector.yaml

# Verify sidecar is running
kubectl get pods -n crossplane-system -o wide
kubectl logs -n crossplane-system deployment/crossplane -c inspector --tail=50
Enter fullscreen mode Exit fullscreen mode

With the Inspector active, creating an XR shows each function call's request/response payload in real-time:

# Real-time Inspector log monitoring
kubectl logs -n crossplane-system deployment/crossplane -c inspector -f

# Output example:
# [2026-03-31T09:15:23Z] Pipeline: databases.platform.example.com/my-db
# [2026-03-31T09:15:23Z] Step 1/3: function-patch-and-transform
# [2026-03-31T09:15:23Z]   Request:  desired=2 resources, observed=0 resources
# [2026-03-31T09:15:23Z]   Response: desired=2 resources, results=[Normal: "patched RDS instance"]
# [2026-03-31T09:15:23Z] Step 2/3: function-auto-ready
# [2026-03-31T09:15:23Z]   Request:  desired=2 resources, observed=2 resources
# [2026-03-31T09:15:23Z]   Response: desired=2 resources, conditions=[Ready: True]
Enter fullscreen mode Exit fullscreen mode

Warning: Pipeline Inspector is currently an alpha feature. In production, monitor sidecar resource usage and socket communication overhead carefully. The socket path (/var/run/pipeline-inspector/socket) is configurable via the --pipeline-inspector-socket flag.

3. CEL Metadata Validation — XRD-Level Policy Enforcement

Before v2.2, x-kubernetes-validations in XRDs could only be applied within spec. This meant metadata-level validation (names, labels) was impossible. v2.2 lifts this restriction, enabling CEL rules for metadata validation.

# XRD with metadata validation: enforcing resource naming conventions
apiVersion: apiextensions.crossplane.io/v2
kind: CompositeResourceDefinition
metadata:
  name: databases.platform.example.com
spec:
  group: platform.example.com
  names:
    kind: Database
    plural: databases
  scope: Namespaced
  versions:
    - name: v1alpha1
      served: true
      referenceable: true
      schema:
        openAPIV3Schema:
          type: object
          # v2.2 NEW: x-kubernetes-validations outside spec
          x-kubernetes-validations:
            # Resource name must start with 'db-' prefix
            - rule: "self.metadata.name.startsWith('db-')"
              message: "Database resource names must start with 'db-'."
              fieldPath: ".metadata.name"
            # Required label validation
            - rule: "has(self.metadata.labels) && 'team' in self.metadata.labels"
              message: "'team' label is required in metadata.labels."
              fieldPath: ".metadata.labels"
Enter fullscreen mode Exit fullscreen mode

This validation executes at the Kubernetes API server level, immediately rejecting non-compliant resources on kubectl apply. No need for external admission controllers like OPA/Gatekeeper or Kyverno for simple XR-level policies.

4. ImageConfig Runtime Settings — Unified Dependency Management

The ImageConfig feature in v2.2 solves the common production issue where dependency packages don't inherit parent package runtime configurations.

# ImageConfig: Apply Workload Identity across all Azure family providers
apiVersion: pkg.crossplane.io/v1beta1
kind: ImageConfig
metadata:
  name: azure-workload-identity
spec:
  matchImages:
    # Image prefix matching: applies to all Azure family packages
    - prefix: "xpkg.crossplane.io/crossplane-contrib/provider-upjet-azure*"
  runtime:
    configRef:
      name: azure-wi-runtime
---
apiVersion: pkg.crossplane.io/v1beta1
kind: DeploymentRuntimeConfig
metadata:
  name: azure-wi-runtime
spec:
  deploymentTemplate:
    spec:
      template:
        spec:
          serviceAccountName: crossplane-azure-sa
          containers:
            - name: package-runtime
              env:
                - name: AZURE_CLIENT_ID
                  valueFrom:
                    secretKeyRef:
                      name: azure-creds
                      key: client-id
Enter fullscreen mode Exit fullscreen mode

Key Point: When matchImages specifies an image prefix, the runtime configuration applies to all matching packages — both explicitly installed and auto-installed dependencies. When both runtimeConfigRef and ImageConfig exist, ImageConfig takes precedence.

5. Enhanced Trace Command — Large-Scale Cluster Debugging

The crossplane beta trace command received major improvements in v2.2. Previously limited to single resource instances, it now supports tracing all resources of a given Kind and live monitoring with --watch.

# Previous v2.1: Single resource only
crossplane beta trace database.platform.example.com/db-team-alpha-prod

# v2.2 NEW: Trace entire Kind
crossplane beta trace database.platform.example.com

# v2.2 NEW: Namespace filtering
crossplane beta trace database.platform.example.com -n team-alpha

# v2.2 NEW: Watch mode — real-time updates
crossplane beta trace database.platform.example.com --watch

# Output:
# NAME                              SYNCED   READY   STATUS
# Database/db-team-alpha-prod       True     True    Available
# |- Instance/db-alpha-rds-xxxxx    True     True    Available
# +- SecurityGroup/db-alpha-sg      True     True    Available
# Database/db-team-beta-staging     True     False   Creating
# |- Instance/db-beta-rds-yyyyy    True     False   Creating (3m12s)
# +- SecurityGroup/db-beta-sg       True     True    Available
Enter fullscreen mode Exit fullscreen mode

6. Operations — Task-Based Execution Model

Introduced in v2, Operations provide a task-based execution model distinct from continuous reconciliation. Three types are available: one-shot Operations, scheduled CronOperations, and event-driven WatchOperations.

# CronOperation: Daily DB backup snapshot at 2 AM
apiVersion: apiextensions.crossplane.io/v2
kind: CronOperation
metadata:
  name: nightly-db-backup
  namespace: team-alpha
spec:
  schedule: "0 2 * * *"    # Daily 02:00 UTC
  compositeTypeRef:
    apiVersion: platform.example.com/v1alpha1
    kind: Database
  pipeline:
    - step: create-snapshot
      functionRef:
        name: function-go-templating
Enter fullscreen mode Exit fullscreen mode

7. Server-Side Apply and Reliability Improvements

In v2.2, the MRD controller now uses Server-Side Apply for CRD updates, directly impacting production reliability:

Aspect Client-Side Apply (Previous) Server-Side Apply (v2.2)
Conflict Detection Last-applied-configuration based Field manager based
Concurrent Updates Race condition risk Safe concurrent updates
Annotation Overhead Growing last-applied-configuration No annotation needed
Large CRD Updates Full replace Partial field updates only

8. Troubleshooting Common Issues

Pipeline Inspector Socket Connection Failure 

# Symptom: "dial unix: connection refused" in inspector sidecar logs
# Root cause: Volume mount or socket path mismatch

# Diagnosis
kubectl exec -n crossplane-system deployment/crossplane -c crossplane -- \
  ls -la /var/run/pipeline-inspector/

# Fix: Verify extraVolumes and sidecars volumeMounts paths match in Helm values
Enter fullscreen mode Exit fullscreen mode

ImageConfig Matching Failure 

# Symptom: Runtime configuration not applied to dependency packages
# Root cause: matchImages prefix doesn't match actual image path

# Diagnosis: Check actual package images
kubectl get providers.pkg.crossplane.io -o yaml | grep "image:"

# Fix: Expand prefix matching range
# prefix: "xpkg.crossplane.io/crossplane-contrib/provider-upjet-azure"
Enter fullscreen mode Exit fullscreen mode

CEL Validation Rule Debugging 

# Symptom: CEL rules not behaving as expected
# Root cause: CEL expression syntax or field path mismatch

# Diagnosis: Check XRD status
kubectl get xrd databases.platform.example.com -o yaml | grep -A 5 "conditions"

# Test with dry-run
kubectl apply -f test-database.yaml --dry-run=server
Enter fullscreen mode Exit fullscreen mode

9. Architecture Diagram

Crossplane v2.2 composition pipeline architecture with XRD definitions, composition functions, pipeline inspector sidecar for debugging, and multi-cloud provider resources

The diagram above illustrates the complete Crossplane v2.2 architecture flow: from platform team XRD definitions and Composition pipelines, through the new Pipeline Inspector interception layer, down to the provisioned cloud provider resources.

10. Conclusion — Crossplane Is Becoming a True Control Plane

Crossplane v2.2 is not just an incremental update. Pipeline Inspector, CEL metadata validation, ImageConfig, and schema-aware functions are all features that production platform teams actually needed.

The investment in debugging and observability is particularly noteworthy. Pipeline Inspector and the enhanced trace command directly address the most common complaint: "I don't know why my Composition Function failed." As platform engineering maturity grows, transparency in infrastructure abstraction layers becomes increasingly critical.

Looking Ahead: v2.3 (expected May 2026) is likely to promote Pipeline Inspector to beta status with richer observability integrations (native OpenTelemetry support). Crossplane is establishing itself not as a "Terraform alternative" but as a Kubernetes-native control plane framework — a core tool for platform teams building Internal Developer Platforms (IDPs).

Practical Recommendations:

  • New projects: Start directly with v2.2. Designing with the v2 architecture from scratch is far more efficient than migrating later.
  • Existing v1 users: Stay stable on v1.20 LTS while validating v2.2 in non-production environments, then migrate incrementally.
  • Pipeline Inspector: Enable immediately in dev/staging environments to dramatically improve Composition debugging.
  • CEL validation: Doesn't fully replace OPA/Kyverno, but handles simple XR-level policies directly in XRDs, reducing external dependencies.

Originally published at ManoIT Tech Blog.

Top comments (0)