DEV Community

Ilya Gorban
Ilya Gorban

Posted on

Crossplane Black Boxes for Developer Self-Service

#devops #kubernetes #crossplane #platformengineering

TL;DR. Developers create cloud resources and IAM grants themselves by committing a small namespaced YAML next to their app. Each kind is a Crossplane XRD (a "black box"): the schema is the whole contract, the composition behind it belongs to the platform team, and the controls that matter (allowed roles, which project, who can be granted, what happens on delete) are enforced by the API server rather than by review. The same docs that teach humans also let a coding agent turn "my worker needs to read the uploads bucket" into the correct manifests.

This is an opinionated write-up of what worked for me as a solo DevOps engineer, in one company's setup. Treat it as experience to weigh, not as the one right answer.

This is part 1 of a two-part series on Crossplane. Part 2: Writing and Testing Crossplane Compositions in KCL.

Why

Without self-service, every bucket, service account or role binding for an app is a ticket to the platform team and a Terraform change in the IaC repo. That has three costs:

  • Latency. A one-line IAM grant waits for someone on the platform team.
  • Drift in ownership. App config lives in the GitOps manifests repo, but its cloud dependencies live in Terraform, reviewed by different people at different times.
  • Over-broad fallbacks. An "unrestricted" fallback tier tends to end up editor-level; a restricted tier covering every real need removes the reason for one.

The goal is self-service that cannot be used to do the wrong thing, rather than self-service that someone has to police.

What 

flowchart LR
  Dev[App PR: crossplane.yaml] --> Argo[ArgoCD app<br/>AppProject allows only platform.* kinds]
  Argo --> XR[Namespaced XR<br/>schema = contract]
  XR --> Comp[Composition<br/>function-kcl]
  Comp --> MR[Managed resources<br/>cloud providers]
  MR --> GCP[(GCP project<br/>via scoped provider identity)]
  Comp --> Secret[Connection Secret<br/>in the app namespace]

Two layers, never mixed

Layer Kinds (generic names) Rule
1: provision Identity, Bucket, Topic, Subscription, Dataset, SecretBundle Manages the resource only. Never its IAM.
2: grant ProjectGrant, BucketGrant, TopicGrant, DatasetGrant, DatabaseGrant Additive IAM only (*IAMMember), one grant target per XR, granted to an in-cluster Identity XR via identityRef.

DatabaseGrant is the one deliberate exception: it creates the database IAM user and the SQL grants, because those two are inseparable.

The split exists for a concrete reason. If a layer-1 resource manages an authoritative IAM field (for example access[] on a BigQuery dataset), it fights the additive layer-2 grants and the dataset's IAM oscillates. I saw a grant flap in and out for over a day (172 UpdatedExternalResource events in 28h) until I stopped the dataset MR from late-initialising access.

Claimless v2

I use Crossplane v2 namespaced XRs, not v1 claims: one object in the app's namespace, with its connection Secret beside it, removes all cross-namespace secret plumbing.

How: the guardrails, layer by layer

1. Developers can only create the black boxes

The ArgoCD AppProject for app teams whitelists the XR API group (platform.example.io, kind *) and not the provider groups (*.gcp.m.upbound.io, *.gcp.upbound.io). A developer cannot sneak a raw ProjectIAMMember into their kustomization: ArgoCD refuses to sync it.

2. The schema is the policy 

# xrd: ProjectGrant (excerpt)
spec:
  required: [identityRef, project, roles]
  properties:
    identityRef: {type: object, required: [name], properties: {name: {type: string}}}
    project: {type: string}
    roles:
      type: array
      items:
        type: string
        enum:
          - roles/storage.objectViewer
          - roles/pubsub.viewer
          - roles/logging.viewer
          - roles/monitoring.viewer
          # ... only what a real request needed, never *.admin
Enter fullscreen mode Exit fullscreen mode
  • Role allow-list as an enum. Start with read-only roles and add a write role only when a real request needs it. No *.admin, nothing system-only. Adding a role is additive and safe. Removing one breaks every XR that uses it, so the list starts small and only grows on request.
  • identityRef only, no raw member. A composition only ever grants to an identity that Crossplane itself manages in that cluster. Groups, humans and external or cross-project identities go through Terraform, where a reviewer sees them.
  • Patterns at admission. Connection-secret names are constrained by a pattern (for example "^[a-z][a-z0-9-]*-conn$"), and identity names have their own limits.
  • Per-env schema narrowing. An env overlay can patch the XRD itself: a narrower enum on non-prod clusters keeps them out of shared prod-grade projects entirely.
  • Safe deletion defaults. Resource-creating MRs default to orphan on delete (buckets, datasets, topics and database users survive an accidental kubectl delete). Grants and IAM members default to delete, so removing a grant XR really does revoke access. A copy-pasted orphan policy on IAM-member MRs means deleting the XR silently leaves the cloud grant in place. Check this field in review. (v2 namespaced MRs have no deletionPolicy: orphaning means managementPolicies without Delete.)

3. Env isolation lives in the provider layer, not in good intentions

  • One provider identity per cluster, bound through Workload Identity, holding admin only over the resource types the compositions create, IAM-member management on the project, and custom roles scoped down where a predefined role grants more than needed (for example, managing secret containers without reading secret versions). Nothing editor-level, anywhere.
  • Cross-project creation through an enum of ProviderConfigs on layer-1 kinds. The prod-targeting ProviderConfig exists only on prod-tier clusters. On a non-prod cluster the value passes the enum but resolves to nothing, so the XR never provisions. Non-prod cannot reach prod through self-service, by construction.
  • A layer-2 grant XR references its target through a *Ref to a layer-1 XR and inherits that XR's provider. It never re-declares a project. To grant on a bucket owned elsewhere, you first declare an ObserveOnly Bucket for it and reference that. Every resource has exactly one owner, and everyone else observes it.

4. Identity-only service accounts, no default roles

The Identity XR creates the cloud service account, the Workload Identity binding and the Kubernetes service account, and nothing else. Workload Identity already is the credential, so a workload needs no baseline role just to "be" its identity. Log or metric writer roles only if the workload ships through the cloud's own services; token creation only if the code signs URLs or mints ID tokens.

Defaults can be moved out safely in three separately merged phases:

  1. Preserve: add explicit grant XRs re-granting exactly what the defaults gave.
  2. Deprecate: stop the composition emitting them.
  3. Trim: grep the code for each capability and drop what nothing uses.

This cuts over without any flapping, because ProjectIAMMember is keyed by project/role/member: the old and new MRs point at the same binding.

5. Green in ArgoCD means correct

ArgoCD can show an XR green while the composite behind it fails. Have shared helpers record each cloud resource in the XR's status with a verified flag (observed, right name, intended project) and roll them up into allVerified. Then write one custom health check for the whole XR API group that follows the same convention: reconcile error → Degraded, paused → Suspended, Ready and allVerified → Healthy, else Progressing, with every False condition in the message. Green means confirmed in the cloud; otherwise the developer sees why.

The same black box for alerts

The pattern also fits platform-owned resources. An Alert kind can be one Grafana alert rule group whose rules may own the log-based counter metric they threshold. Its schema holds the rules that are easy to break by hand:

  • Derived identity. A rule's uid is <xr>-<rule>-<env>, since a silence binds to the uid; a new uid detaches every silence.
  • Names that never change. Log-based metrics aren't retroactive: a rename restarts the counter at zero.
  • Loud failure. A query error maps to Error, not OK, so a metric never created (missing IAM) can't pass as "no events".
  • One XR for every env. The filter takes a {project} placeholder; overlays patch one field.

Evolving XRDs and compositions

Removing or changing a field in a served XRD version goes against Crossplane's own guidance: a version's schema can't change existing fields, and breaking changes belong in a new XRD. What the docs don't describe well is the actual path: how to move live XRs, their composed resources and their connection secrets from the old shape to the new one.

What worked for me needed no separate clean cluster. Every XRD or composition change went through the staging environment first, with the staging cluster pointed at the branch, and to prod only after it reconciled cleanly there. It works, but isn't fully automated: promotion and the checks between steps are manual.

Letting a coding agent generate the right resources

There is no magic "intent to YAML" skill. What works is a docs chain that a coding agent follows the same way a new engineer would:

  1. The repo's agent instructions say: v2 claimless XRs only, never raw managed resources, and point to one file.
  2. A single authoritative self-service guide next to the XRDs: per kind, the required spec, a copy-paste example, output secret keys and gotchas. Crucially, it maps use case to field ("role → when you need it"), so "my worker reads the uploads bucket" resolves to a BucketGrant with a viewer role, not a guess.
  3. A commented scaffold, so the agent copies structure rather than inventing it.
  4. XRD description: fields written for the reader, so kubectl explain doubles as documentation.

Then the important part: validation that does not trust the agent. kustomize build passes an out-of-enum role happily, because it doesn't know the XRD schema. Only a server-side diff (kustomize build … | kubectl diff --server-side) or the ArgoCD sync validates against the live structural schema. That is the mandatory pre-merge gate for any manifest touching an XRD.

Pros / cons

Pros Cons
Developer PR = cloud change. No ticket, no static keys Two to five minute convergence for multi-reconcile kinds (DatabaseGrant)
Controls enforced at admission, not in review XRD schema evolution is hard: removing a field means migrating every XR
One owner per resource, one place to look Platform team owns a KCL codebase of compositions and their tests
Env isolation structural (provider absent off prod-tier) Some principals (groups, external identities) still need Terraform

When not to do this: if you have fewer than a handful of teams, or your cloud footprint changes rarely, Terraform modules with CODEOWNERS are cheaper. The black boxes earn their keep when request volume is high and the set of allowed shapes is small.

Gotchas I actually hit

  • A *Ref never crosses clusters. The reference is resolved by observing an object on the local API server. Point it at an XR that lives on another cluster and you get Synced=True, Ready=False, and nothing emitted: the cloud is never called. Check which cluster an overlay actually deploys to, not what the overlay is named.
  • Synced=True is not "the cloud has it". After a merge, the ArgoCD app, the XR and the MR can all be green while the cloud still holds the old config, because the provider has one more reconcile to do (about 2 minutes observed). The verification of record is a gcloud … describe.
  • Sync waves: put all XRs in one negative wave, and leave the workload on the default wave 0. A chain of negative waves deadlocked a first install (an existing namespace can't show it): grants waited on an identity whose wave hadn't run. Compositions already sequence XRs among themselves (a grant waits for its identity), so waves add nothing. Only the app that defines the XRDs must do the opposite (XRs after definitions).
  • Remove XRs before their XRD, in two syncs. A foreground prune deletes the XRD's RBAC roles first, so Crossplane can't delete its own XR and the XRD sticks in Terminating.
  • Adopting an existing resource writes your spec onto it. Replicate the live config (location, lifecycle, CORS) before adopting, or "adoption" becomes a mutation.

Checklist

  • [ ] AppProject whitelists your XR group, not provider groups.
  • [ ] Every role/location/provider choice is an enum, read-only roles first. No free-text principals.
  • [ ] Prod-targeting ProviderConfig exists only on prod-tier clusters.
  • [ ] Data resources orphan on delete. Grants revoke on delete. Review that field.
  • [ ] Layer 1 never manages IAM. Layer 2 is additive only.
  • [ ] One authoritative guide, with a use-case-to-field table, linked from the agent instructions.
  • [ ] Server-side diff as the pre-merge gate. kustomize build is not schema validation.

References

I wrote this article with the help of Claude, an AI assistant. I reviewed and edited it by hand, and it is based on my own hands-on experience.

Top comments (0)