DEV Community

Ilya Gorban
Ilya Gorban

Posted on

Writing and Testing Crossplane Compositions in KCL

#devops #kubernetes #crossplane #testing

TL;DR. I write every Crossplane Composition in KCL via function-kcl. Shared logic lives in one helper module mounted from a ConfigMap. Editing happens in a real .k file with IDE support, then the code goes back into the YAML. Every composition is rendered in CI against JSON fixtures and diffed against recorded expected output, and that check also verifies two things kcl run cannot see: XRD status-schema conformance and multi-version compatibility.

This is an opinionated write-up of what worked for me as a solo DevOps engineer, in one company's setup. Treat it as experience to weigh, not as the one right answer.

This is part 2 of a two-part series on Crossplane. Part 1: Crossplane Black Boxes for Developer Self-Service.

Why

I started with a mix of function-patch-and-transform and KCL, then moved everything to KCL. KCL covers all our needs: loops, conditionals, functions, asserts. An audit of our 12 compositions found the runtime was fine. The real cost was maintenance:

Duplicated block Compositions carrying it
Resolve a GCP SA email by observing a K8s ServiceAccount annotation 7 of 12
Header boilerplate (oxr / ocds / name / namespace / spec) 12 of 12
Content-key sanitiser .replace("/","-").replace(".","-") ~5

Fixing a bug in the SA-resolve block meant editing seven files. And KCL embedded as a YAML block scalar has no highlighting or completion, and indentation mistakes show up only when the function pod fails.

Everything below was added later, mostly after a specific escape.

What 

flowchart LR
  subgraph repo[GitOps repo]
    H[kcl-helpers/ module<br/>+ kcl test]
    C[composition-*.yaml<br/>inline KCL source]
    F[tests/*.json fixtures]
    E[tests/expected/&lt;comp&gt;/&lt;scenario&gt;.yaml]
  end
  H -->|ConfigMap, mounted at /kcl-helpers| FK[function-kcl pod]
  C --> FK
  H & C & F --> R[run.sh: kcl run per scenario] -->|diff| E
  • Pipeline: each Composition is mode: Pipeline with two steps. A function-kcl render step does all the logic, and function-auto-ready handles readiness.
  • Shared module helpers: KSA-email resolve, an observe-Object scaffold, the content-key sanitiser, an ownerRef helper and a status helper. Compositions use import helpers as h.
  • Delivery choice: I evaluated an OCI package against a ConfigMap. I picked the ConfigMap: no registry, no publish step, versioned by git history. The trade-off is that the ConfigMap must sync before the compositions that use it (see Gotchas).
  • Blast radius kept small on purpose: only the grant compositions import the module. The simple leaf compositions stay self-contained.

How

1. Editing: extract, edit in an IDE, put it back

The KCL lives at spec.pipeline[N].input.spec.source. I extract it to a sibling .k file, edit it with the KCL VS Code extension, and write it back:

# extract
import yaml
doc = yaml.safe_load(open("composition-x.yaml"))
src = next(s["input"]["spec"]["source"] for s in doc["spec"]["pipeline"]
           if s.get("functionRef", {}).get("name") == "function-kcl")
open("composition-x.k", "w").write(src)
Enter fullscreen mode Exit fullscreen mode

Reintegrating is the part people get wrong. Never yaml.dump() the document back, because it destroys comments and key order. Instead I string-replace the block. I detect the indent of source: | from the file (never hard-code 12 spaces), re-indent the new source by that amount plus 2, and substitute only that block. Then I verify the round trip: re-parse the YAML and assert the extracted source is byte-equal to the .k file. An off-by-one indent silently breaks KCL parsing, and only the round-trip check catches it before the cluster does.

For completion on option("params"), drop a minimal kcl.mod next to the .k file. If the composition imports the helpers, that kcl.mod needs a [dependencies] entry pointing at the module directory. Without it, nothing runs locally (CannotFindModule).

2. Fixtures: one JSON per composition, one key per scenario

Each top-level key is a scenario. Its value is exactly what option("params") returns at runtime:

{
  "a_creating":       { "oxr": { "apiVersion": "platform.example.io/v1alpha1", "spec": {"...": "..."} } },
  "b_email_resolved": { "oxr": {"...": "..."}, "ocds": {
      "my-sa-ksa-observe": { "Resource": { "status": { "atProvider": { "manifest": {
        "metadata": { "annotations": { "iam.gke.io/gcp-service-account": "<gcp-sa-email>" }}}}}}}}},
  "err_missing_project": { "oxr": {"...": "..."} }
}
Enter fullscreen mode Exit fullscreen mode

Coverage rules, enforced in review:

  • At least one scenario with no ocds (first reconcile) and one with ocds populated (second and later reconciles). Most of my bugs lived in the gap between those two passes.
  • One err_* scenario per assert. It must fail with that assert's message, not merely exit non-zero. A missing module, a syntax error or a malformed fixture also exits non-zero and would otherwise "pass".
  • If the composition reads extraResources, cover both "fetched" and "not fetched yet".
  • Reordered-twin scenario for every composition that iterates a list: same items, reversed order, identical set of composed-resource names (see content-keyed naming below).
  • Every served XRD version needs at least one scenario whose oxr.apiVersion is that version.
  • One scenario mirrors a live XR exactly. A rarely set optional field broke the render only on real input (the ** trap below).
  • Replacing a specialised composition? Feed the new one the old XR's live inputs and require identical rule uids.

3. run.sh: render, diff against expected output, fail on drift 

./run.sh            # CI mode: render every scenario, diff against tests/expected/
./run.sh --update   # re-record expected output after an INTENTIONAL change
Enter fullscreen mode Exit fullscreen mode

For each composition with a fixture, it extracts the source with yq. It writes a temp kcl.mod whose [dependencies] points at the repo's helper directory, which is the same files the in-cluster ConfigMap is built from, so local and cluster resolve the same code. Then it runs kcl run per scenario and diff -us the result against tests/expected/<composition>/<scenario>.yaml.

The same script runs two extra checks that a render alone can't:

  • Status declared in the XRD. Crossplane XRD schemas are structural. If the composition writes dxr.status.foo and the XRD doesn't declare foo, the composite's status apply fails on every reconcile, forever. The composed MRs are still created, so everything looks green except a reconcile-error alert. This reached production once. The check walks each rendered composite's .status keys against the XRD's status schema.
  • Multi-version invariant. The composition only ever sees the XR at the referenceable version, so that version's spec must be a superset of every other served version's fields. Otherwise a field exposed on a newer version is invisible to the KCL.

The helper module has its own unit tests (kcl test). They catch invariants no fixture happens to exercise. My example: siblings that differ only in a name suffix must not collide after sanitising.

4. CI

A path-filtered GitHub Actions workflow runs on PRs touching the compositions, the helper module, the workflow itself, or the repo's tool-version file:

  1. Read the KCL CLI version from the repo's tool-version file; I once had three KCL pins in three places.
  2. Assert it equals the helper module's kcl.mod edition, and fail with a pointer to the docs if not.
  3. Install that KCL version, run kcl test on the helpers, then run.sh.

5. Beyond unit tests: live e2e and canaries

  • Live e2e: a small Helm chart keeps one permanent instance of every XR kind on real clusters, plus a PostSync Job that kubectl waits each one for Ready and asserts its connection secret. The chart has helm unittest tests too. Its Role must list every XR kind it waits on. Add a new kind without updating RBAC and the test fails with Forbidden.
  • Canary for non-additive changes: every XR floats on compositionUpdatePolicy: Automatic by default, so a merged Composition change rolls out to every XR of that type on the next reconcile. For anything that renames composed resources, touches immutable fields or migrates grants, I use channel labels. A label on the Composition alone protects nothing: an Automatic XR with no selector always takes the latest revision. So consumers first select channel: stable via spec.crossplane.compositionRevisionSelector.matchLabels, the changed Composition is labelled channel: canary, one test XR selects canary, and I promote (relabel to stable) only after it reconciles cleanly.

KCL language traps (hit in practice)

Trap Do this instead
d.get("k") or {} always gives {}, because get returns Undefined, which is falsy d["k"] if "k" in d else {}
A bare ternary dict entry (k = a if c else b) once dropped the entry after it; simple cases don't reproduce, so this is defensive wrap it: k = (a if c else b)
{**base, key: v} or {**base, "key": v} raises conflicting values if base already has key (: unifies, it doesn't override) override with =: {**base, key = v}, or union: `base \
{% raw %}any([...]) is a parse error, because any is a quantifier keyword len([x for x in xs if cond]) > 0
ocds[name].resource returns Undefined capital R: ocds[name].Resource (Go struct, no json tag)
KCLRun spec.dependencies with your own [dependencies] header gives a duplicate TOML key, and every importing XR goes Synced=False give only the entry line: helpers = { path = "/kcl-helpers" }

Content-keyed names: the rule that prevents an outage

When a composition emits one resource per list item, name it by a sanitised content key (sa-iam-roles-cloudsql-client), never by enumerate index (sa-iam-0). Composed resources have immutable key fields (ProjectIAMMember.role, a Grant's role). With index names, inserting an item at the top re-maps every existing MR onto a different role, and the provider refuses: cannot change the value of the argument "role". Switching an existing composition to content keys renames every live MR once, which is a brief delete/recreate per binding. Roll that out env by env.

Pros / cons

Pros Cons
Real language: functions, asserts, loops One more language for the team to learn (it is not Python)
Render tests need no cluster and run in seconds Fixtures must model ocds shapes by hand
Expected-output diff proves a refactor is byte-identical --update can become a rubber stamp
Shared module removes 7-way duplication Helpers and compositions ship from two ArgoCD apps

When not to bother: with two or three trivial compositions, patch-and-transform plus a staging smoke test is enough. The harness pays off once compositions branch on observed state.

Gotchas / lessons learned

  • run.sh does not test function-kcl's dependencies parsing. It builds its own kcl.mod. A malformed dependencies field in the Composition is only caught by a staging branch-test, so always branch-test a shared-module change.
  • Helpers and compositions are a multi-app change. The helper ConfigMap ships with the app that installs Crossplane, and the compositions with the app that defines the XRDs. Pin and sync only one in a branch test, and every XR goes Degraded with failed to run kcl function pipelines. The ConfigMap name is stable, so once it syncs the mount refreshes within about 30 to 60 seconds and XRs recover without a restart.
  • The helper mount is configured twice. function-kcl has its own DeploymentRuntimeConfig (it needs much more memory) next to the default one used by the other functions. Change the mount in both.
  • --update keeps the stale file on a render error. A scenario that hits an evaluation error is not re-recorded, so git status shows zero changes and the edit looks like a no-op. Run plain ./run.sh first and look for render error.
  • Never run --update twice concurrently. Contention on the KCL package cache prints waiting for package-cache lock... to stdout, and it gets baked into expected files for compositions you never touched.
  • Read the re-recorded expected output before committing. Check the new resource is present with the right fields, and absent from scenarios that shouldn't have it. --update re-records everything, so an unexpected file in git status means you changed another composition's render.
  • Gate dependent MRs on the observation, not just on other inputs. One composition fell back to the XR's name as the bucket name before the referenced bucket was observed, emitted a ForceNew field with the wrong value, and wedged the MR.
  • Composition metadata patches don't land; status does. dxr.metadata.labels never reaches the composite. For sibling discovery, fetch all composites with an empty matchLabels and filter in KCL on a declared status field.
  • Name fixtures after the XRD kind (project-grant), not the cloud service. A file named after the service hid a composition from grep <Kind>, and that led to a wrong conclusion that a provider was unused.

Checklist

  • [ ] KCL edited in a .k file, reintegrated by string replace, round trip verified.
  • [ ] Fixture covers first pass, resolved pass, every assert, reordered twin, every served version.
  • [ ] ./run.sh green with no --update. Then --update once, then read the diff.
  • [ ] Status fields declared in the XRD (the harness checks).
  • [ ] KCL version single-sourced. CI asserts CLI equals module edition.
  • [ ] Shared-module or non-additive change: staging branch-test of both apps, canary channel.

References

I wrote this article with the help of Claude, an AI assistant. I reviewed and edited it by hand, and it is based on my own hands-on experience.

Top comments (0)