Testing Crossplane Compositions with Chainsaw: kind Without Credentials, Then a Prod-Like Environment

TL;DR. On top of part 2's offline render tests I added two Kyverno Chainsaw layers. One applies every composition to a throwaway kind cluster with no cloud credentials on every pull request, so nothing real gets created. The other applies the same use cases with real credentials to a prod-like environment, which mirrors production but can run a branch, and waits for Ready and verified cloud state. A third job upgrades existing composite resources (XRs) to the PR's compositions. I chose Chainsaw over Uptest, and the new layers found real bugs that every render test had passed.

This is an opinionated write-up of what worked for me as a solo DevOps engineer, in one company's setup. Treat it as experience to weigh, not as the one right answer.

This is part 3 of a series on Crossplane:

1. Crossplane Black Boxes for Developer Self-Service
2. Writing and Testing Crossplane Compositions in KCL
3. This article

---

Why

Part 2 ended with render tests: every composition is rendered offline against JSON fixtures and diffed against recorded expected output. That catches KCL logic errors in milliseconds. It cannot catch anything that needs a reconcile loop: one XR waiting for another to resolve, a provider writing a Secret, a composition that only finishes on its second pass, or deletion leaving nothing behind.

Before this work, the only live check was a post-sync job waiting on permanent "canary" XRs. It ran after merge, and only after a successful sync, so a failed sync left that revision untested and nobody was told.

I had put Chainsaw aside once, because my only clusters were shared, the wrong place for test XRDs. That is specific to my case: a dedicated test cluster is a fine answer in general, and with many contributors a dedicated cluster plus Uptest is the safer choice. Reviewing part 2, a reader from the Crossplane community suggested a cheaper middle layer: a local kind cluster with no cloud credentials. Applying an XR there creates nothing real, so the isolation and cost problems disappear.

What

Layer	Tool	Where	Proves
1. Render	KCL + fixtures	no cluster, CI on every PR	the composition renders the expected resources
2. Applied to a cluster	Chainsaw	throwaway kind cluster, no cloud credentials, CI on every PR	XRs apply, gates hold across reconciles, composed resources are right, deletion is clean
3. For real	Chainsaw	prod-like environment, with real credentials, before merge	the cloud accepts it: IAM, real IDs, Ready, verified state, deletion

The permanent canaries stay. They are the only check of each environment's own project and IAM, of XRs that live through provider and composition upgrades, and of drift no PR caused. The post-sync job is gone, replaced by an alert that fires when a canary stays not Ready, or Ready but unverified, for 30 minutes.

One set of use cases

There is one XR manifest per kind in a use-cases/ directory, and both Chainsaw layers apply them, so they can't drift. Negative cases stay inline in the kind tests, because the prod-like layer applies everything in that directory.

Which layer owns a check

The rule I settled on: put each check in the cheapest layer that can see it.

Owner	What goes there
Render	every field value, every variant, every assert (one failing scenario per assert), verified status from mocked observations
kind	only what needs a cluster: references resolving across XRs, gates over several reconciles, a provider writing a Secret, an assert surfacing as Synced=False, deletion leaving no managed resource behind
Prod-like	what only the cloud decides: IAM, API acceptance, real IDs, Ready, real verified state

Every composition has at least one render fixture and one kind test. A field asserted in both layers is a second place to update, so I don't add those.

How

kind without credentials

The kind job installs the same Crossplane version, providers, functions and compositions as the real clusters, from the same directories. Cloud providers can't authenticate, so reconciling a managed resource (MR) fails at the cloud API, leaving no status.atProvider and no Ready, while provider-kubernetes works normally.

That breaks every composition that finishes on its second pass, such as a grant waiting for its identity. A small script writes status.atProvider on the cloud MR, in the real provider's ID format; a provider whose observe always fails never overwrites it. The mocks live in the tests and never in the compositions, where a test-only branch would ship to every environment.

A trimmed test looks like this:

apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
  name: bucket
spec:
  steps:
    - name: apply
      try:
        - apply:
            file: ../../use-cases/bucket.yaml
        - assert:
            resource:
              apiVersion: platform.example.com/v1alpha1
              kind: Bucket
              metadata:
                name: example-bucket
              status:
                (conditions[?type == 'Synced'] | [0].status): 'True'
    - name: observed
      try:
        - script:
            content: ../../lib/mock-observed.sh Bucket example-bucket
        - assert:
            file: assert-verified.yaml

A run takes about five minutes, and its largest phase is setup: all providers becoming healthy takes roughly 100 to 135 seconds. Tests run in parallel, so the run takes as long as its slowest test. Draft PRs skip the job, and marking the PR ready starts the first run.

The Crossplane install and the compositions are separate GitOps applications, so a PR changing both can roll out in either order. Such a PR also runs both skewed pairs: new compositions on the old install, and the reverse.

Prod-like environment, from a developer's machine

The prod-like layer applies the same use cases, waits for Ready and verified state, then deletes everything. Three decisions shaped it.

It runs before merge. The compositions sync to every environment automatically, so a test after merge reports a break already in production. Branch-testing there already runs the PR's compositions.

It runs from a developer's machine, not CI. Running in CI would need that environment pinned to the PR first, and that means a workflow running PR code with rights to repoint a GitOps application. I didn't want CI to hold that right, so the script refuses to run unless three things hold:

• the cluster context provisions into that environment's own project;
• the branch tracks its own remote branch, has nothing uncommitted and is pushed;
• the environment's application is synced to exactly that commit.

Every cloud-facing name is unique to the run. The compositions use the XR's name as the cloud resource name. With fixed names, two runs would fight over one resource, and a run could adopt an existing resource with the same name and then delete it during cleanup. A render step copies the use cases with a per-run suffix on every name and reference.

Fourteen kinds run in dependency chains, and a consumer is applied only after what it references is Ready and verified. Kinds configured once per database instance are excluded, because a test copy would fight the real one and its delete would remove it. The alert kind's test rule is rendered so it can never fire. Every gateway the route kind accepts is shared with other services, so instead of throwaway routes it gets the branch test of a real route.

Why Chainsaw and not Uptest

Uptest is Upbound's tool for exactly this job, and it uses Chainsaw internally. I read its own templates before choosing.

On kind, Uptest waits for the wrong thing. Without credentials nothing becomes Ready, which is what Uptest waits on. The kind layer proves composed fields, absences and negative cases, which Chainsaw asserts declaratively and Uptest would need as hand-written shell hooks.

On a shared cluster, Uptest's lifecycle is unsafe for me. In the version I read:

• every apply downloads and runs a helper script from the project's main branch, unpinned;
• its import step scales every provider's runtime config to zero replicas and back, restarting those providers for the whole cluster;
• delete, by default, waits for every managed resource on the cluster to go away.

With import and update turned off, what is left is apply, wait for Ready, delete. That is a few Chainsaw steps over the same use cases.

What the new layers found

Three composition bugs that every render test had passed:

1. A service account composition reported allVerified: true before anything was observed. The list of resources to verify was empty on the first reconcile, and all([]) is true.
2. An alert composition with no log-based metric composed nothing to verify, so it reported verified with nothing checked.
3. A readiness gate observed the secret's name instead of the referenced bundle's name. When the two differed, the gate watched an object that doesn't exist, and the dependent resource was never created.

Each one became a render scenario first, then a fix.

A hole in the render harness. A failing scenario passed on any non-zero exit, so a scenario that tripped a different assert, or failed on a broken fixture, still passed. The harness now records the error type and assert message for each failing scenario and diffs them in CI.

Gotchas

• A provider is Installed before its CRDs exist. The revision installs managed resource definitions, and a controller creates each CRD afterwards. Gating setup on Installed failed a run with no matches for kind. Wait for Healthy.
• A mocked status must satisfy the CRD schema. The status subresource accepts an invalid patch, but every later full-object update is then rejected. That includes the garbage collector removing its own finalizer, so cleanup hangs. The mock script now validates the whole object with a server-side dry run first.
• Delete with Foreground propagation. With Background, the namespace is deleted alongside the composed objects. Each object needs a ProviderConfigUsage in that namespace to finalize, so the namespace deadlocks.
• Foreground is still not enough. The garbage collector blocks the owner only on dependents already in its cache, so an MR can outlive its XR. Each test's first step has a cleanup that waits for every managed kind to go; cleanup runs in reverse, so it runs last.
• Assert every composed resource before the test ends. Otherwise cleanup can delete the XR while a reconcile is still applying, and the resource created in that window outlives it. This only appeared when I raised parallelism, because load widens the window.
• Interrupted runs leave real resources. Chainsaw documents cleanup after a failed step, not after a killed process, and laptops get interrupted. The script reports earlier leftovers before starting and prints its run ID on Ctrl-C, and a daily orphan-detection job covers cloud resources whose XR is gone.

Upgrade testing

Everything above tests a change on fresh XRs. Many breaks hit XRs that already exist: a new composition revision re-renders them, a renamed composed resource is deleted and recreated, a removed field disappears. So when the compositions change, the kind workflow runs one more job:

1. Install the main branch's tree and run its tests without cleanup, so their XRs stay.
2. Install the PR's tree over them.
3. Fail if any XR or composed resource was recreated (UIDs compared), or an XR is no longer synced and verified.

An intended recreation goes in a reviewed list file. As a negative control, renaming one composed resource failed exactly that kind's XRs. The job adds about eight minutes.

When a changed field makes the cloud provider replace the external resource, the MR keeps its UID, so that case is covered by the prod-like branch test and the permanent canaries instead.

Checklist

• [ ] One use-case manifest per kind, shared by the kind and prod-like layers.
• [ ] Each check lives in the cheapest layer that can see it.
• [ ] Mocks only in tests, in the real provider's ID format, validated against the CRD schema.
• [ ] Foreground deletion, plus an explicit wait for every managed kind before namespace delete.
• [ ] Every composed resource asserted before the test ends.
• [ ] The prod-like layer runs before merge, on unique names, only against the PR's exact commit.
• [ ] Upgrade job: main's tree first, then the PR's, with no unexpected recreations.

References

• Kyverno Chainsaw: https://kyverno.github.io/chainsaw/
• Upbound Uptest: https://github.com/crossplane/uptest
• kind: https://kind.sigs.k8s.io/
• Kubernetes foreground deletion: https://kubernetes.io/docs/concepts/architecture/garbage-collection/#foreground-deletion

---

I wrote this article with the help of Claude, an AI assistant. I reviewed and edited it by hand, and it is based on my own hands-on experience.