DEV Community: Ilya Gorban

An AI-Native Dev Lifecycle Across R&D: The Plan

Ilya Gorban — Mon, 26 Oct 2026 12:24:17 +0000

TL;DR. AI coding agents now open pull requests faster than people can review them. Instead of reviewing faster, I plan to decide which PRs need a human at all. A PR review agent puts each PR in one of three risk tiers by written rules: merge on green, AI review with a human after merge, or human review before merge. It starts in dry run: people decide as today, and every disagreement is a finding. After merge, an alert-triage agent links alerts to recent merges and proposes reverts, never merging them. Part 2 will report what survived real PRs.

This is an opinionated write-up of what I am planning as a solo DevOps engineer for a whole R&D organization, in one company's setup. It is a plan, not a result. Treat it as reasoning to weigh, not as the one right answer.

This is part 1 of a series on an AI-native dev lifecycle:

This article
What was adopted, and the first measured results (after implementation)

Why

Most of our pull requests are now opened by AI coding agents. That changed the shape of the review problem.

Waiting for a review is not the bottleneck. Attention is. Every PR that asks for a human competes with every other one, and an agent can open a dozen in the time it takes to read one carefully.

The obvious answers both fail:

Review everything. It does not scale, and a reviewer reading their thirtieth agent PR of the week is approving, not reviewing.
Trust the agent. Then nothing stands between a confident, wrong change and production.

Instead: spend human attention where a change can hurt production, and prove with data that the rest is safe without it.

What

Three risk tiers

Tier	Gate	Typical changes
A: merge on green	Required checks only	Human-facing docs, non-prod overlays, pin-only bumps
B: AI review, human after merge	Required checks plus an AI review with no open findings; the owner is notified on merge and can ask for a revert	Application code, workflow logic, tests, tooling, runtime markdown, agent skills and knowledge
C: human before merge	Required checks plus one approval from the owning team	Prod-grade config, shared CI libraries, infrastructure as code, secrets and permissions, every database change

Four rules make the tiers hold:

The riskiest file decides. A docs edit bundled with a prod overlay is tier C.
A tier can be raised, never lowered, and nothing an agent writes can lower it.
Default-deny. A path that matches no tier list fails a deterministic check, so a new directory blocks until classified instead of silently becoming tier A.
Database changes are always C. Queries, schema and permissions. In the history I mined, nine in ten database-touching PRs sat in ordinary application paths that a path rule would put in tier B.

The rule seed is the set of paths that already require approval from people with elevated permissions, such as production or infrastructure approvers. It is the closest thing to a written statement of "this path needs these people".

Two levels of knowledge

A reviewer, human or AI, needs two kinds of knowledge:

The application. The shared conventions every app follows (how code, comments, docs, tests and infrastructure are written, and how a PR is made), plus each app's own contract: its interfaces, where its database contract lives, and which of its files are runtime.
The change. What in this diff needs human eyes, and what can merge on green.

The first level exists because of one finding. Path-only tiering treats *.md as documentation, wrongly in two places. In one service, every "docs" PR in four weeks edited LLM prompt assets that ship with the service. In another repo, runbooks and agent knowledge files are rendered into a live agent's knowledge base. Both are behaviour changes that look like documentation. Only the application knows which files it reads at runtime, so every app has to declare them before any path rule can be trusted.

Agent knowledge is the same trap. An edit to a skill that does something important can break it. An in-repo knowledge base that is wordy or wrong bloats every later agent's context, or teaches it the wrong thing. Those files are reviewed for correctness and brevity, not waved through as docs.

Ground truth from history

A dry run needs the right answer for each PR. History gives a start:

reverts;
a follow-up fix PR that references this PR within a week;
changes requested by a reviewer, or review threads followed by new commits;
rubber-stamp approvals: no thread, no change, no follow-up.

They are candidates, not labels. On a spot check, a "follow-up fix" was sometimes a real regression and sometimes a sibling change or ordinary iteration. A shared changed file was mostly noise. So people confirm a seed set by hand, one row at a time: should the agent have said "merge" or "human" here, and why.

And the measure has to cover both directions:

false automerge: the agent would have merged something a human needed to see;
false human: the agent would have made a person wait for nothing;
noise: findings nobody acts on, because noisy AI review loses trust faster than no review.

How

Before merge: a read-only classifier

My proposed first version runs Claude Code headless in a GitHub Action:

A workflow runs claude -p on each PR, with read-only tools, the repository's rules and the diff.
--json-schema forces the shape of the verdict: tier, findings with cited evidence, the rules hit, a confidence.
A small script, not the model, validates that JSON and posts it as a GitHub check. During the dry run the check is neutral.

The design is not tied to it: any agent that runs unattended with schema-shaped output fits, and agents can be compared on the same evaluation set.

Reviewing a PR that is still changing burns tokens. So the agent reviews only PRs marked ready, after the cheap required checks pass. The shared knowledge base tells every agent session to open PRs as drafts. The platform's own draft state beats a custom label: the ready-for-review event is the trigger, with no extra convention to forget.

The diff, PR text and comments come from whoever opened the PR: untrusted input. The agent that reads them holds no write token and no cluster access, and the merge decision is enforced by the repository ruleset, outside the model.

Alternatives I ranked:

Writing my own agent loop on an SDK. Full control, but then I own a harness, which is a commodity; what is ours is the knowledge and the evaluation set.
Hosted review bots. They produce comments rather than a verdict a script can enforce, and our per-application rules do not plug in easily.

After merge: link alerts to merges

Live tier A and B changes still need watching. The planned first version reuses an alert-triage agent I already run, built on the open-source HolmesGPT, which investigates alerts with runbooks and read-only access. It will gain one step: did this application deploy a merge shortly before the alert? If so, it names the PR, opens a revert PR and pages the owner. None of this is built yet, and the first thing to confirm is that the agent's toolsets can read which revision is actually deployed.

Three constraints:

It never merges its own revert, and never changes the cluster. A revert is a git change like any other.
Merge is not deploy. It attributes an alert to a merge only once the GitOps controller's synced revision includes the merge commit. A merge still awaiting sync caused nothing yet.
Different trust profile, different agent. The pre-merge reviewer reads untrusted diffs and must hold no authority. The post-merge one reads live system state and needs cluster read access. Putting both in one agent would join untrusted input to authority.

Feedback: teach the PR agent from outcomes

A feedback run, started by the agent's maintainer by hand or on a schedule, sets the agent's verdicts over a period against what happened next: human overrides, reverts, follow-up fixes, linked alerts. It proposes rule and knowledge changes as a tier C PR, because changing the PR agent's rules changes what merges unseen. Each confirmed miss joins the evaluation set.

The compliance gate

If you hold a security certification, your change-management control almost certainly requires review before a change ships. Automerge under that text and your audit evidence no longer matches what happens.

So the agent does not leave dry run until the change-management control has been read and either already covers AI review, or has been updated with the auditor's agreement. Check your control text before you automerge anything. My proposed shape is risk-tiered change management: human approval kept for tier C, and for A and B, required automated checks, the AI verdict recorded on the PR, owner notification and a revert window. Whether that satisfies an auditor is the auditor's call.

What I took from published ADLCs, and where I disagree

None of three outside sources targets my problem, more PRs than reviewers; most move human attention earlier, into specs and plans.

"Inside the ADLC" (Towards AI) describes a spec-driven pipeline with role-tiered agents. I took three ideas:

review as several read-only reviewers by dimension, each returning structured findings;
"a reviewer that can change code has a conflict of interest";
autonomy as an explicit list of stops, which I invert: list what forces human review, and everything else merges on green. That list is also the audit artefact.

Where I disagree: no outcome data, it reviews only its own code, and its wrap-up step merges unattended despite saying merging needs approval.

The PDD kit keeps one product model per application in Markdown, updated by the agents before code. It is the closest existing shape for my level-1 contract; my history adds runtime files and where the database contract lives. Its evidence is a launch post.

"The Agentic Awakening" (a venture investor's playbook) argues that AI review in CI is the cheapest autonomy step, that you should own your context and evaluations and buy the harness, and that a human gate stops prompt injection only when the decision is enforced outside the model. I took all three. I discount its evidence: about twenty interviews, mostly from one investor's portfolio, no control group, and lines of code called a vanity metric, then used as its headline.

What part 2 will report

Part 2 will report, against this plan:

Dry-run accuracy against the confirmed seed set: false automerge, false human and noise, separately, per tier.
Whether path rules plus per-app knowledge caught the runtime markdown and database changes that path rules alone missed.
Tier mix and reviewer load: how many tier C requests each reviewer got per week, before and after.
Stability, using the signals the DORA AI Capabilities Model ties to AI adoption: PR size, reverts of A and B merges, incidents traced to them, rework rate (follow-up fixes within a week) and change failure rate.
The post-merge link: first, whether the agent can read the deployed revision at all; then how many alerts it tied to a recent merge, and how many of those it got right.
What I dropped or changed, and why.

The success bar is fixed now so it cannot drift: fewer tier C requests per reviewer than today's total review requests, with no production incident traced to a tier A or B merge.

References

Inside the ADLC (Towards AI): https://pub.towardsai.net/inside-the-adlc-how-an-ai-native-development-lifecycle-works-8ccd1a52e8ba
PDD kit (launch post): https://www.linkedin.com/posts/itaig_a-week-after-i-released-pdd-kit-microsoft-share-7518954454730043392-GQmM/
The Agentic Awakening: https://theagenticawakening.com
Claude Code headless mode: https://code.claude.com/docs/en/headless
GitHub rulesets, available rules: https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-rulesets/available-rules-for-rulesets
HolmesGPT: https://github.com/robusta-dev/holmesgpt
DORA AI Capabilities Model: https://cloud.google.com/blog/products/ai-machine-learning/introducing-doras-inaugural-ai-capabilities-model

I wrote this article with the help of Claude, an AI assistant. I reviewed and edited it by hand, and it is based on my own hands-on experience.

Testing Crossplane Compositions with Chainsaw: kind Without Credentials, Then a Prod-Like Environment

Ilya Gorban — Thu, 22 Oct 2026 15:55:15 +0000

TL;DR. On top of part 2's offline render tests I added two Kyverno Chainsaw layers. One applies every composition to a throwaway kind cluster with no cloud credentials on every pull request, so nothing real gets created. The other applies the same use cases with real credentials to a prod-like environment, which mirrors production but can run a branch, and waits for Ready and verified cloud state. A third job upgrades existing composite resources (XRs) to the PR's compositions. I chose Chainsaw over Uptest, and the new layers found real bugs that every render test had passed.

This is an opinionated write-up of what worked for me as a solo DevOps engineer, in one company's setup. Treat it as experience to weigh, not as the one right answer.

This is part 3 of a series on Crossplane:

Why

Part 2 ended with render tests: every composition is rendered offline against JSON fixtures and diffed against recorded expected output. That catches KCL logic errors in milliseconds. It cannot catch anything that needs a reconcile loop: one XR waiting for another to resolve, a provider writing a Secret, a composition that only finishes on its second pass, or deletion leaving nothing behind.

Before this work, the only live check was a post-sync job waiting on permanent "canary" XRs. It ran after merge, and only after a successful sync, so a failed sync left that revision untested and nobody was told.

I had put Chainsaw aside once, because my only clusters were shared, the wrong place for test XRDs. That is specific to my case: a dedicated test cluster is a fine answer in general, and with many contributors a dedicated cluster plus Uptest is the safer choice. Reviewing part 2, a reader from the Crossplane community suggested a cheaper middle layer: a local kind cluster with no cloud credentials. Applying an XR there creates nothing real, so the isolation and cost problems disappear.

What

Layer	Tool	Where	Proves
1. Render	KCL + fixtures	no cluster, CI on every PR	the composition renders the expected resources
2. Applied to a cluster	Chainsaw	throwaway `kind` cluster, no cloud credentials, CI on every PR	XRs apply, gates hold across reconciles, composed resources are right, deletion is clean
3. For real	Chainsaw	prod-like environment, with real credentials, before merge	the cloud accepts it: IAM, real IDs, `Ready`, verified state, deletion

The permanent canaries stay. They are the only check of each environment's own project and IAM, of XRs that live through provider and composition upgrades, and of drift no PR caused. The post-sync job is gone, replaced by an alert that fires when a canary stays not Ready, or Ready but unverified, for 30 minutes.

One set of use cases

There is one XR manifest per kind in a use-cases/ directory, and both Chainsaw layers apply them, so they can't drift. Negative cases stay inline in the kind tests, because the prod-like layer applies everything in that directory.

Which layer owns a check

The rule I settled on: put each check in the cheapest layer that can see it.

Owner	What goes there
Render	every field value, every variant, every `assert` (one failing scenario per assert), verified status from mocked observations
`kind`	only what needs a cluster: references resolving across XRs, gates over several reconciles, a provider writing a Secret, an `assert` surfacing as `Synced=False`, deletion leaving no managed resource behind
Prod-like	what only the cloud decides: IAM, API acceptance, real IDs, `Ready`, real verified state

Every composition has at least one render fixture and one kind test. A field asserted in both layers is a second place to update, so I don't add those.

How

`kind` without credentials

The kind job installs the same Crossplane version, providers, functions and compositions as the real clusters, from the same directories. Cloud providers can't authenticate, so reconciling a managed resource (MR) fails at the cloud API, leaving no status.atProvider and no Ready, while provider-kubernetes works normally.

That breaks every composition that finishes on its second pass, such as a grant waiting for its identity. A small script writes status.atProvider on the cloud MR, in the real provider's ID format; a provider whose observe always fails never overwrites it. The mocks live in the tests and never in the compositions, where a test-only branch would ship to every environment.

A trimmed test looks like this:

apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
  name: bucket
spec:
  steps:
    - name: apply
      try:
        - apply:
            file: ../../use-cases/bucket.yaml
        - assert:
            resource:
              apiVersion: platform.example.com/v1alpha1
              kind: Bucket
              metadata:
                name: example-bucket
              status:
                (conditions[?type == 'Synced'] | [0].status): 'True'
    - name: observed
      try:
        - script:
            content: ../../lib/mock-observed.sh Bucket example-bucket
        - assert:
            file: assert-verified.yaml

A run takes about five minutes, and its largest phase is setup: all providers becoming healthy takes roughly 100 to 135 seconds. Tests run in parallel, so the run takes as long as its slowest test. Draft PRs skip the job, and marking the PR ready starts the first run.

The Crossplane install and the compositions are separate GitOps applications, so a PR changing both can roll out in either order. Such a PR also runs both skewed pairs: new compositions on the old install, and the reverse.

Prod-like environment, from a developer's machine

The prod-like layer applies the same use cases, waits for Ready and verified state, then deletes everything. Three decisions shaped it.

It runs before merge. The compositions sync to every environment automatically, so a test after merge reports a break already in production. Branch-testing there already runs the PR's compositions.

It runs from a developer's machine, not CI. Running in CI would need that environment pinned to the PR first, and that means a workflow running PR code with rights to repoint a GitOps application. I didn't want CI to hold that right, so the script refuses to run unless three things hold:

the cluster context provisions into that environment's own project;
the branch tracks its own remote branch, has nothing uncommitted and is pushed;
the environment's application is synced to exactly that commit.

Every cloud-facing name is unique to the run. The compositions use the XR's name as the cloud resource name. With fixed names, two runs would fight over one resource, and a run could adopt an existing resource with the same name and then delete it during cleanup. A render step copies the use cases with a per-run suffix on every name and reference.

Fourteen kinds run in dependency chains, and a consumer is applied only after what it references is Ready and verified. Kinds configured once per database instance are excluded, because a test copy would fight the real one and its delete would remove it. The alert kind's test rule is rendered so it can never fire. Every gateway the route kind accepts is shared with other services, so instead of throwaway routes it gets the branch test of a real route.

Why Chainsaw and not Uptest

Uptest is Upbound's tool for exactly this job, and it uses Chainsaw internally. I read its own templates before choosing.

On kind, Uptest waits for the wrong thing. Without credentials nothing becomes Ready, which is what Uptest waits on. The kind layer proves composed fields, absences and negative cases, which Chainsaw asserts declaratively and Uptest would need as hand-written shell hooks.

On a shared cluster, Uptest's lifecycle is unsafe for me. In the version I read:

every apply downloads and runs a helper script from the project's main branch, unpinned;
its import step scales every provider's runtime config to zero replicas and back, restarting those providers for the whole cluster;
delete, by default, waits for every managed resource on the cluster to go away.

With import and update turned off, what is left is apply, wait for Ready, delete. That is a few Chainsaw steps over the same use cases.

What the new layers found

Three composition bugs that every render test had passed:

A service account composition reported allVerified: true before anything was observed. The list of resources to verify was empty on the first reconcile, and all([]) is true.
An alert composition with no log-based metric composed nothing to verify, so it reported verified with nothing checked.
A readiness gate observed the secret's name instead of the referenced bundle's name. When the two differed, the gate watched an object that doesn't exist, and the dependent resource was never created.

Each one became a render scenario first, then a fix.

A hole in the render harness. A failing scenario passed on any non-zero exit, so a scenario that tripped a different assert, or failed on a broken fixture, still passed. The harness now records the error type and assert message for each failing scenario and diffs them in CI.

Gotchas

A provider is Installed before its CRDs exist. The revision installs managed resource definitions, and a controller creates each CRD afterwards. Gating setup on Installed failed a run with no matches for kind. Wait for Healthy.
A mocked status must satisfy the CRD schema. The status subresource accepts an invalid patch, but every later full-object update is then rejected. That includes the garbage collector removing its own finalizer, so cleanup hangs. The mock script now validates the whole object with a server-side dry run first.
Delete with Foreground propagation. With Background, the namespace is deleted alongside the composed objects. Each object needs a ProviderConfigUsage in that namespace to finalize, so the namespace deadlocks.
Foreground is still not enough. The garbage collector blocks the owner only on dependents already in its cache, so an MR can outlive its XR. Each test's first step has a cleanup that waits for every managed kind to go; cleanup runs in reverse, so it runs last.
Assert every composed resource before the test ends. Otherwise cleanup can delete the XR while a reconcile is still applying, and the resource created in that window outlives it. This only appeared when I raised parallelism, because load widens the window.
Interrupted runs leave real resources. Chainsaw documents cleanup after a failed step, not after a killed process, and laptops get interrupted. The script reports earlier leftovers before starting and prints its run ID on Ctrl-C, and a daily orphan-detection job covers cloud resources whose XR is gone.

Upgrade testing

Everything above tests a change on fresh XRs. Many breaks hit XRs that already exist: a new composition revision re-renders them, a renamed composed resource is deleted and recreated, a removed field disappears. So when the compositions change, the kind workflow runs one more job:

Install the main branch's tree and run its tests without cleanup, so their XRs stay.
Install the PR's tree over them.
Fail if any XR or composed resource was recreated (UIDs compared), or an XR is no longer synced and verified.

An intended recreation goes in a reviewed list file. As a negative control, renaming one composed resource failed exactly that kind's XRs. The job adds about eight minutes.

When a changed field makes the cloud provider replace the external resource, the MR keeps its UID, so that case is covered by the prod-like branch test and the permanent canaries instead.

Checklist

[ ] One use-case manifest per kind, shared by the kind and prod-like layers.
[ ] Each check lives in the cheapest layer that can see it.
[ ] Mocks only in tests, in the real provider's ID format, validated against the CRD schema.
[ ] Foreground deletion, plus an explicit wait for every managed kind before namespace delete.
[ ] Every composed resource asserted before the test ends.
[ ] The prod-like layer runs before merge, on unique names, only against the PR's exact commit.
[ ] Upgrade job: main's tree first, then the PR's, with no unexpected recreations.

References

Kyverno Chainsaw: https://kyverno.github.io/chainsaw/
Upbound Uptest: https://github.com/crossplane/uptest
kind: https://kind.sigs.k8s.io/
Kubernetes foreground deletion: https://kubernetes.io/docs/concepts/architecture/garbage-collection/#foreground-deletion

I wrote this article with the help of Claude, an AI assistant. I reviewed and edited it by hand, and it is based on my own hands-on experience.

Writing and Testing Crossplane Compositions in KCL

Ilya Gorban — Tue, 13 Oct 2026 10:19:35 +0000

TL;DR. I write every Crossplane Composition in KCL via function-kcl. Shared logic lives in one helper module mounted from a ConfigMap. Editing happens in a real .k file with IDE support, then the code goes back into the YAML. Every composition is rendered in CI against JSON fixtures and diffed against recorded expected output, and that check also verifies two things kcl run cannot see: XRD status-schema conformance and multi-version compatibility.

This is an opinionated write-up of what worked for me as a solo DevOps engineer, in one company's setup. Treat it as experience to weigh, not as the one right answer.

This is part 2 of a series on Crossplane:

Why

I started with a mix of function-patch-and-transform and KCL, then moved everything to KCL. KCL covers all our needs: loops, conditionals, functions, asserts. An audit of our 12 compositions found the runtime was fine. The real cost was maintenance:

Duplicated block	Compositions carrying it
Resolve a GCP SA email by observing a K8s ServiceAccount annotation	7 of 12
Header boilerplate (`oxr` / `ocds` / name / namespace / spec)	12 of 12
Content-key sanitiser `.replace("/","-").replace(".","-")`	~5

Fixing a bug in the SA-resolve block meant editing seven files. And KCL embedded as a YAML block scalar has no highlighting or completion, and indentation mistakes show up only when the function pod fails.

Everything below was added later, mostly after a specific escape.

What

flowchart LR
  subgraph repo[GitOps repo]
    H[kcl-helpers/ module<br/>+ kcl test]
    C[composition-*.yaml<br/>inline KCL source]
    F[tests/*.json fixtures]
    E[tests/expected/&lt;comp&gt;/&lt;scenario&gt;.yaml]
  end
  H -->|ConfigMap, mounted at /kcl-helpers| FK[function-kcl pod]
  C --> FK
  H & C & F --> R[run.sh: kcl run per scenario] -->|diff| E

Pipeline: each Composition is mode: Pipeline with two steps. A function-kcl render step does all the logic, and function-auto-ready handles readiness.
Shared module helpers: KSA-email resolve, an observe-Object scaffold, the content-key sanitiser, an ownerRef helper and a status helper. Compositions use import helpers as h.
Delivery choice: I evaluated an OCI package against a ConfigMap. I picked the ConfigMap: no registry, no publish step, versioned by git history. The trade-off is that the ConfigMap must sync before the compositions that use it (see Gotchas).
Blast radius kept small on purpose: only the grant compositions import the module. The simple leaf compositions stay self-contained.

How

1. Editing: extract, edit in an IDE, put it back

The KCL lives at spec.pipeline[N].input.spec.source. I extract it to a sibling .k file, edit it with the KCL VS Code extension, and write it back:

# extract
import yaml
doc = yaml.safe_load(open("composition-x.yaml"))
src = next(s["input"]["spec"]["source"] for s in doc["spec"]["pipeline"]
           if s.get("functionRef", {}).get("name") == "function-kcl")
open("composition-x.k", "w").write(src)

Reintegrating is the part people get wrong. Never yaml.dump() the document back, because it destroys comments and key order. Instead I string-replace the block. I detect the indent of source: | from the file (never hard-code 12 spaces), re-indent the new source by that amount plus 2, and substitute only that block. Then I verify the round trip: re-parse the YAML and assert the extracted source is byte-equal to the .k file. An off-by-one indent silently breaks KCL parsing, and only the round-trip check catches it before the cluster does.

For completion on option("params"), drop a minimal kcl.mod next to the .k file. If the composition imports the helpers, that kcl.mod needs a [dependencies] entry pointing at the module directory. Without it, nothing runs locally (CannotFindModule).

2. Fixtures: one JSON per composition, one key per scenario

Each top-level key is a scenario. Its value is exactly what option("params") returns at runtime:

{
  "a_creating":       { "oxr": { "apiVersion": "platform.example.io/v1alpha1", "spec": {"...": "..."} } },
  "b_email_resolved": { "oxr": {"...": "..."}, "ocds": {
      "my-sa-ksa-observe": { "Resource": { "status": { "atProvider": { "manifest": {
        "metadata": { "annotations": { "iam.gke.io/gcp-service-account": "<gcp-sa-email>" }}}}}}}}},
  "err_missing_project": { "oxr": {"...": "..."} }
}

Coverage rules, enforced in review:

At least one scenario with no ocds (first reconcile) and one with ocds populated (second and later reconciles). Most of my bugs lived in the gap between those two passes.
One err_* scenario per assert. It must fail with that assert's message, not merely exit non-zero. A missing module, a syntax error or a malformed fixture also exits non-zero and would otherwise "pass".
If the composition reads extraResources, cover both "fetched" and "not fetched yet".
Reordered-twin scenario for every composition that iterates a list: same items, reversed order, identical set of composed-resource names (see content-keyed naming below).
Every served XRD version needs at least one scenario whose oxr.apiVersion is that version.
One scenario mirrors a live XR exactly. A rarely set optional field broke the render only on real input (the ** trap below).
Replacing a specialised composition? Feed the new one the old XR's live inputs and require identical rule uids.

3. `run.sh`: render, diff against expected output, fail on drift

./run.sh            # CI mode: render every scenario, diff against tests/expected/
./run.sh --update   # re-record expected output after an INTENTIONAL change

For each composition with a fixture, it extracts the source with yq. It writes a temp kcl.mod whose [dependencies] points at the repo's helper directory, which is the same files the in-cluster ConfigMap is built from, so local and cluster resolve the same code. Then it runs kcl run per scenario and diff -us the result against tests/expected/<composition>/<scenario>.yaml.

The same script runs two extra checks that a render alone can't:

Status declared in the XRD. Crossplane XRD schemas are structural. If the composition writes dxr.status.foo and the XRD doesn't declare foo, the composite's status apply fails on every reconcile, forever. The composed MRs are still created, so everything looks green except a reconcile-error alert. This reached production once. The check walks each rendered composite's .status keys against the XRD's status schema.
Multi-version invariant. The composition only ever sees the XR at the referenceable version, so that version's spec must be a superset of every other served version's fields. Otherwise a field exposed on a newer version is invisible to the KCL.

The helper module has its own unit tests (kcl test). They catch invariants no fixture happens to exercise. My example: siblings that differ only in a name suffix must not collide after sanitising.

4. CI

A path-filtered GitHub Actions workflow runs on PRs touching the compositions, the helper module, the workflow itself, or the repo's tool-version file:

Read the KCL CLI version from the repo's tool-version file; I once had three KCL pins in three places.
Assert it equals the helper module's kcl.mod edition, and fail with a pointer to the docs if not.
Install that KCL version, run kcl test on the helpers, then run.sh.

5. Beyond unit tests: live e2e and canaries

Live e2e: a small Helm chart keeps one permanent instance of every XR kind on real clusters, plus a PostSync Job that kubectl waits each one for Ready and asserts its connection secret. The chart has helm unittest tests too. Its Role must list every XR kind it waits on. Add a new kind without updating RBAC and the test fails with Forbidden.
Canary for non-additive changes: every XR floats on compositionUpdatePolicy: Automatic by default, so a merged Composition change rolls out to every XR of that type on the next reconcile. For anything that renames composed resources, touches immutable fields or migrates grants, I use channel labels. A label on the Composition alone protects nothing: an Automatic XR with no selector always takes the latest revision. So consumers first select channel: stable via spec.crossplane.compositionRevisionSelector.matchLabels, the changed Composition is labelled channel: canary, one test XR selects canary, and I promote (relabel to stable) only after it reconciles cleanly.

KCL language traps (hit in practice)

Trap	Do this instead
`d.get("k") or {}` always gives `{}`, because `get` returns `Undefined`, which is falsy	`d["k"] if "k" in d else {}`
A bare ternary dict entry (`k = a if c else b`) once dropped the entry after it; simple cases don't reproduce, so this is defensive	wrap it: `k = (a if c else b)`
`{base, key: v}` or `{base, "key": v}` raises `conflicting values` if `base` already has `key` (`:` unifies, it doesn't override)	override with `=`: `{**base, key = v}`, or union: `base \
{% raw %}`any([...])` is a parse error, because `any` is a quantifier keyword	`len([x for x in xs if cond]) > 0`
`ocds[name].resource` returns `Undefined`	capital R: `ocds[name].Resource` (Go struct, no json tag)
KCLRun `spec.dependencies` with your own `[dependencies]` header gives a duplicate TOML key, and every importing XR goes `Synced=False`	give only the entry line: `helpers = { path = "/kcl-helpers" }`

Content-keyed names: the rule that prevents an outage

When a composition emits one resource per list item, name it by a sanitised content key (sa-iam-roles-cloudsql-client), never by enumerate index (sa-iam-0). Composed resources have immutable key fields (ProjectIAMMember.role, a Grant's role). With index names, inserting an item at the top re-maps every existing MR onto a different role, and the provider refuses: cannot change the value of the argument "role". Switching an existing composition to content keys renames every live MR once, which is a brief delete/recreate per binding. Roll that out env by env.

Pros / cons

Pros	Cons
Real language: functions, asserts, loops	One more language for the team to learn (it is not Python)
Render tests need no cluster and run in seconds	Fixtures must model `ocds` shapes by hand
Expected-output diff proves a refactor is byte-identical	`--update` can become a rubber stamp
Shared module removes 7-way duplication	Helpers and compositions ship from two ArgoCD apps

When not to bother: with two or three trivial compositions, patch-and-transform plus a smoke test in a prod-like environment is enough. The harness pays off once compositions branch on observed state.

Gotchas / lessons learned

run.sh does not test function-kcl's dependencies parsing. It builds its own kcl.mod. A malformed dependencies field in the Composition is only caught by a branch-test in a prod-like environment, so always branch-test a shared-module change.
Helpers and compositions are a multi-app change. The helper ConfigMap ships with the app that installs Crossplane, and the compositions with the app that defines the XRDs. Pin and sync only one in a branch test, and every XR goes Degraded with failed to run kcl function pipelines. The ConfigMap name is stable, so once it syncs the mount refreshes within about 30 to 60 seconds and XRs recover without a restart.
The helper mount is configured twice. function-kcl has its own DeploymentRuntimeConfig (it needs much more memory) next to the default one used by the other functions. Change the mount in both.
--update keeps the stale file on a render error. A scenario that hits an evaluation error is not re-recorded, so git status shows zero changes and the edit looks like a no-op. Run plain ./run.sh first and look for render error.
Never run --update twice concurrently. Contention on the KCL package cache prints waiting for package-cache lock... to stdout, and it gets baked into expected files for compositions you never touched.
Read the re-recorded expected output before committing. Check the new resource is present with the right fields, and absent from scenarios that shouldn't have it. --update re-records everything, so an unexpected file in git status means you changed another composition's render.
Gate dependent MRs on the observation, not just on other inputs. One composition fell back to the XR's name as the bucket name before the referenced bucket was observed, emitted a ForceNew field with the wrong value, and wedged the MR.
Composition metadata patches don't land; status does. dxr.metadata.labels never reaches the composite. For sibling discovery, fetch all composites with an empty matchLabels and filter in KCL on a declared status field.
Name fixtures after the XRD kind (project-grant), not the cloud service. A file named after the service hid a composition from grep <Kind>, and that led to a wrong conclusion that a provider was unused.

Checklist

[ ] KCL edited in a .k file, reintegrated by string replace, round trip verified.
[ ] Fixture covers first pass, resolved pass, every assert, reordered twin, every served version.
[ ] ./run.sh green with no --update. Then --update once, then read the diff.
[ ] Status fields declared in the XRD (the harness checks).
[ ] KCL version single-sourced. CI asserts CLI equals module edition.
[ ] Shared-module or non-additive change: prod-like branch-test of both apps, canary channel.

References

function-kcl: https://github.com/crossplane-contrib/function-kcl
KCL language tour: https://kcl-lang.io/docs/reference/lang/tour
Crossplane composition revisions: https://docs.crossplane.io/latest/composition/composition-revisions/
Crossplane XRD status fields: https://docs.crossplane.io/latest/composition/composite-resource-definitions/

Update, 14 October 2026

Thanks to a reader from the Crossplane community who reviewed this article and answered part of my open questions about Chainsaw and Uptest. The piece I was missing: run the cluster layer on a local kind cluster with no cloud credentials, so applying an XR creates no real resources. That gives a fast, isolated loop between the offline render tests and a final "for real" run with credentials in CI, and it removes most of the reasons I had for putting those tools aside.

I'm going to try it. A follow-up article will describe either the revised testing setup, or the problems I couldn't get past and why I stopped there.

Follow-up: the revised setup is in part 3.

I wrote this article with the help of Claude, an AI assistant. I reviewed and edited it by hand, and it is based on my own hands-on experience.

Crossplane Black Boxes for Developer Self-Service

Ilya Gorban — Tue, 13 Oct 2026 10:19:18 +0000

TL;DR. Developers create cloud resources and IAM grants themselves by committing a small namespaced YAML next to their app. Each kind is a Crossplane XRD (a "black box"): the schema is the whole contract, the composition behind it belongs to the platform team, and the controls that matter (allowed roles, which project, who can be granted, what happens on delete) are enforced by the API server rather than by review. The same docs that teach humans also let a coding agent turn "my worker needs to read the uploads bucket" into the correct manifests.

This is an opinionated write-up of what worked for me as a solo DevOps engineer, in one company's setup. Treat it as experience to weigh, not as the one right answer.

This is part 1 of a series on Crossplane:

Why

Without self-service, every bucket, service account or role binding for an app is a ticket to the platform team and a Terraform change in the IaC repo. That has three costs:

Latency. A one-line IAM grant waits for someone on the platform team.
Drift in ownership. App config lives in the GitOps manifests repo, but its cloud dependencies live in Terraform, reviewed by different people at different times.
Over-broad fallbacks. An "unrestricted" fallback tier tends to end up editor-level; a restricted tier covering every real need removes the reason for one.

The goal is self-service that cannot be used to do the wrong thing, rather than self-service that someone has to police.

What

flowchart LR
  Dev[App PR: crossplane.yaml] --> Argo[ArgoCD app<br/>AppProject allows only platform.* kinds]
  Argo --> XR[Namespaced XR<br/>schema = contract]
  XR --> Comp[Composition<br/>function-kcl]
  Comp --> MR[Managed resources<br/>cloud providers]
  MR --> GCP[(GCP project<br/>via scoped provider identity)]
  Comp --> Secret[Connection Secret<br/>in the app namespace]

Two layers, never mixed

Layer	Kinds (generic names)	Rule
1: provision	`Identity`, `Bucket`, `Topic`, `Subscription`, `Dataset`, `SecretBundle`	Manages the resource only. Never its IAM.
2: grant	`ProjectGrant`, `BucketGrant`, `TopicGrant`, `DatasetGrant`, `DatabaseGrant`	Additive IAM only (`*IAMMember`), one grant target per XR, granted to an in-cluster `Identity` XR via `identityRef`.

DatabaseGrant is the one deliberate exception: it creates the database IAM user and the SQL grants, because those two are inseparable.

The split exists for a concrete reason. If a layer-1 resource manages an authoritative IAM field (for example access[] on a BigQuery dataset), it fights the additive layer-2 grants and the dataset's IAM oscillates. I saw a grant flap in and out for over a day (172 UpdatedExternalResource events in 28h) until I stopped the dataset MR from late-initialising access.

Claimless v2

I use Crossplane v2 namespaced XRs, not v1 claims: one object in the app's namespace, with its connection Secret beside it, removes all cross-namespace secret plumbing.

How: the guardrails, layer by layer

1. Developers can only create the black boxes

The ArgoCD AppProject for app teams whitelists the XR API group (platform.example.io, kind *) and not the provider groups (*.gcp.m.upbound.io, *.gcp.upbound.io). A developer cannot sneak a raw ProjectIAMMember into their kustomization: ArgoCD refuses to sync it.

2. The schema is the policy

# xrd: ProjectGrant (excerpt)
spec:
  required: [identityRef, project, roles]
  properties:
    identityRef: {type: object, required: [name], properties: {name: {type: string}}}
    project: {type: string}
    roles:
      type: array
      items:
        type: string
        enum:
          - roles/storage.objectViewer
          - roles/pubsub.viewer
          - roles/logging.viewer
          - roles/monitoring.viewer
          # ... only what a real request needed, never *.admin

Role allow-list as an enum. Start with read-only roles and add a write role only when a real request needs it. No *.admin, nothing system-only. Adding a role is additive and safe. Removing one breaks every XR that uses it, so the list starts small and only grows on request.
identityRef only, no raw member. A composition only ever grants to an identity that Crossplane itself manages in that cluster. Groups, humans and external or cross-project identities go through Terraform, where a reviewer sees them.
Patterns at admission. Connection-secret names are constrained by a pattern (for example "^[a-z][a-z0-9-]*-conn$"), and identity names have their own limits.
Per-env schema narrowing. An env overlay can patch the XRD itself: a narrower enum on non-prod clusters keeps them out of shared prod-grade projects entirely.
Safe deletion defaults. Resource-creating MRs default to orphan on delete (buckets, datasets, topics and database users survive an accidental kubectl delete). Grants and IAM members default to delete, so removing a grant XR really does revoke access. A copy-pasted orphan policy on IAM-member MRs means deleting the XR silently leaves the cloud grant in place. Check this field in review. (v2 namespaced MRs have no deletionPolicy: orphaning means managementPolicies without Delete.)

3. Env isolation lives in the provider layer, not in good intentions

One provider identity per cluster, bound through Workload Identity, holding admin only over the resource types the compositions create, IAM-member management on the project, and custom roles scoped down where a predefined role grants more than needed (for example, managing secret containers without reading secret versions). Nothing editor-level, anywhere.
Cross-project creation through an enum of ProviderConfigs on layer-1 kinds. The prod-targeting ProviderConfig exists only on prod-tier clusters. On a non-prod cluster the value passes the enum but resolves to nothing, so the XR never provisions. Non-prod cannot reach prod through self-service, by construction.
A layer-2 grant XR references its target through a *Ref to a layer-1 XR and inherits that XR's provider. It never re-declares a project. To grant on a bucket owned elsewhere, you first declare an ObserveOnly Bucket for it and reference that. Every resource has exactly one owner, and everyone else observes it.

4. Identity-only service accounts, no default roles

The Identity XR creates the cloud service account, the Workload Identity binding and the Kubernetes service account, and nothing else. Workload Identity already is the credential, so a workload needs no baseline role just to "be" its identity. Log or metric writer roles only if the workload ships through the cloud's own services; token creation only if the code signs URLs or mints ID tokens.

Defaults can be moved out safely in three separately merged phases:

Preserve: add explicit grant XRs re-granting exactly what the defaults gave.
Deprecate: stop the composition emitting them.
Trim: grep the code for each capability and drop what nothing uses.

This cuts over without any flapping, because ProjectIAMMember is keyed by project/role/member: the old and new MRs point at the same binding.

5. Green in ArgoCD means correct

ArgoCD can show an XR green while the composite behind it fails. Have shared helpers record each cloud resource in the XR's status with a verified flag (observed, right name, intended project) and roll them up into allVerified. Then write one custom health check for the whole XR API group that follows the same convention: reconcile error → Degraded, paused → Suspended, Ready and allVerified → Healthy, else Progressing, with every False condition in the message. Green means confirmed in the cloud; otherwise the developer sees why.

The same black box for alerts

The pattern also fits platform-owned resources. An Alert kind can be one Grafana alert rule group whose rules may own the log-based counter metric they threshold. Its schema holds the rules that are easy to break by hand:

Derived identity. A rule's uid is <xr>-<rule>-<env>, since a silence binds to the uid; a new uid detaches every silence.
Names that never change. Log-based metrics aren't retroactive: a rename restarts the counter at zero.
Loud failure. A query error maps to Error, not OK, so a metric never created (missing IAM) can't pass as "no events".
One XR for every env. The filter takes a {project} placeholder; overlays patch one field.

Evolving XRDs and compositions

Removing or changing a field in a served XRD version goes against Crossplane's own guidance: a version's schema can't change existing fields, and breaking changes belong in a new XRD. What the docs don't describe well is the actual path: how to move live XRs, their composed resources and their connection secrets from the old shape to the new one.

What worked for me needed no separate clean cluster. Every XRD or composition change went through a prod-like environment first, with its cluster pointed at the branch, and to prod only after it reconciled cleanly there. It works, but isn't fully automated: promotion and the checks between steps are manual.

Letting a coding agent generate the right resources

There is no magic "intent to YAML" skill. What works is a docs chain that a coding agent follows the same way a new engineer would:

The repo's agent instructions say: v2 claimless XRs only, never raw managed resources, and point to one file.
A single authoritative self-service guide next to the XRDs: per kind, the required spec, a copy-paste example, output secret keys and gotchas. Crucially, it maps use case to field ("role → when you need it"), so "my worker reads the uploads bucket" resolves to a BucketGrant with a viewer role, not a guess.
A commented scaffold, so the agent copies structure rather than inventing it.
XRD description: fields written for the reader, so kubectl explain doubles as documentation.

Then the important part: validation that does not trust the agent. kustomize build passes an out-of-enum role happily, because it doesn't know the XRD schema. Only a server-side diff (kustomize build … | kubectl diff --server-side) or the ArgoCD sync validates against the live structural schema. That is the mandatory pre-merge gate for any manifest touching an XRD.

Pros / cons

Pros	Cons
Developer PR = cloud change. No ticket, no static keys	Two to five minute convergence for multi-reconcile kinds (`DatabaseGrant`)
Controls enforced at admission, not in review	XRD schema evolution is hard: removing a field means migrating every XR
One owner per resource, one place to look	Platform team owns a KCL codebase of compositions and their tests
Env isolation structural (provider absent off prod-tier)	Some principals (groups, external identities) still need Terraform

When not to do this: if you have fewer than a handful of teams, or your cloud footprint changes rarely, Terraform modules with CODEOWNERS are cheaper. The black boxes earn their keep when request volume is high and the set of allowed shapes is small.

Gotchas I actually hit

A *Ref never crosses clusters. The reference is resolved by observing an object on the local API server. Point it at an XR that lives on another cluster and you get Synced=True, Ready=False, and nothing emitted: the cloud is never called. Check which cluster an overlay actually deploys to, not what the overlay is named.
Synced=True is not "the cloud has it". It only says the last reconcile succeeded; readiness is what Ready reports. Even with everything green after a merge, the cloud can still hold the old config until one more provider reconcile (about 2 minutes observed). The verification of record is a gcloud … describe.
Sync waves: put all XRs in one negative wave, and leave the workload on the default wave 0. A chain of negative waves deadlocked a first install (an existing namespace can't show it): grants waited on an identity whose wave hadn't run. Compositions already sequence XRs among themselves (a grant waits for its identity), so waves add nothing. Only the app that defines the XRDs must do the opposite (XRs after definitions).
Remove XRs before their XRD, in two syncs. A foreground prune deletes the XRD's RBAC roles first, so Crossplane can't delete its own XR and the XRD sticks in Terminating.
Adopting an existing resource writes your spec onto it. Replicate the live config (location, lifecycle, CORS) before adopting, or "adoption" becomes a mutation.

Checklist

[ ] AppProject whitelists your XR group, not provider groups.
[ ] Every role/location/provider choice is an enum, read-only roles first. No free-text principals.
[ ] Prod-targeting ProviderConfig exists only on prod-tier clusters.
[ ] Data resources orphan on delete. Grants revoke on delete. Review that field.
[ ] Layer 1 never manages IAM. Layer 2 is additive only.
[ ] One authoritative guide, with a use-case-to-field table, linked from the agent instructions.
[ ] Server-side diff as the pre-merge gate. kustomize build is not schema validation.

References

Crossplane composite resource definitions: https://docs.crossplane.io/latest/composition/composite-resource-definitions/
Crossplane composition revisions: https://docs.crossplane.io/latest/composition/composition-revisions/
ArgoCD projects (resource whitelists): https://argo-cd.readthedocs.io/en/stable/user-guide/projects/
GKE Workload Identity Federation: https://cloud.google.com/kubernetes-engine/docs/concepts/workload-identity

I wrote this article with the help of Claude, an AI assistant. I reviewed and edited it by hand, and it is based on my own hands-on experience.

Running 15 Claude Code Sessions at Once

Ilya Gorban — Thu, 08 Oct 2026 20:15:21 +0000

TL;DR: Ten or more parallel Claude Code sessions work if you treat them as a small distributed system: ownership by function, one writer (a governance session) for shared rules and hooks, approval never relayed, repeat mistakes turned into guard hooks. Assumptions are verified before a human is asked, and no Claude-generated edit ships unreviewed, with GitOps as the record.

This is an opinionated write-up of what worked for me as a solo DevOps engineer, in one company's setup. Treat it as experience to weigh, not as the one right answer.

Why: what goes wrong with an "octopus"

I run 10–15 sessions at once in cmux, one per task. I'm the only DevOps engineer, so all are mine. Failures:

File collisions. Two sessions edit one rule file at once. git add -A swept another session's edits into the wrong commit; staging explicit paths doesn't help, since a plain git commit takes the whole index.
Relayed "approvals". A tells B "the user approved this"; B can't verify it, A can't transfer it.
Worktree sprawl, stale state. About 18 worktrees across two repos; a sweep couldn't tell "abandoned" from "mid-task". A peer's "it's merged" can precede the merge.
Context loss on compaction. A brief lives only in the sender's transcript, so it vanishes when both sides compact.
Hook fixes racing. One session's regex fix undid another's; the suite stayed green, each fix tested only on its own case.

What: the setup

flowchart LR
  subgraph cmux["cmux window"]
    S1["Session: task A<br/>worktree A"]
    S2["Session: task B<br/>worktree B"]
    S3["Session: task C<br/>worktree C"]
    G["Governance delegate<br/>(owns global rules + hooks)"]
  end
  S1 -- "SendMessage: brief" --> G
  S2 -- "SendMessage: blocked by guard X" --> G
  G -- "result + 'you can close your side'" --> S1
  S3 <-- "collision notice" --> S2
  U(("Human")) -. "approves in EACH session" .-> S1 & S2 & S3 & G

Piece	Rule
cmux workspaces	one per session; worktrees named `<session>--<desc>`, so `git worktree list` shows owners.
One session = one task = one worktree/branch/PR	launch at the parent directory of all repos, never inside a worktree: it breaks on removal and can't switch repos.
Local task list, `T<n>` ids	created before investigating. `T<n>`, never `#<n>` (a GitHub reference; a hook enforces it).
Governance delegate	one long-lived session owns all shared-config writes, found by role, never by name.
Single owners	hook repair → the delegate, wherever the hook fired; machine-wide worktree removal → one sweep owner; a repo's `.claude/`, PRs and tickets → the session working it. SendMessage crosses ownership boundaries; it doesn't offload your own work.

From about 57 real cross-session messages: finding a defect doesn't make it yours to fix, and being handed a fix doesn't make it yours to approve.

How

1. One dedicated session owns the global rules and hooks

The global CLAUDE.md, pattern docs, hooks, tests and settings are shared; two writers on one rule is a silent race. So one dedicated session writes there and takes no feature work. Others send it a brief for any wrong rule, hook defect or missing guard, create a local task, and never edit inline.

The boundary is mechanical: global config is one directory symlinked into ~/.claude/, so readlink -f tells you (plan files exempt).

2. Delegation moves the work, never the approval

Every brief says: "Nothing below is pre-approved; route every approval through the user in your own session." A harness timeout isn't approval. A peer's word counts when it restricts you ("these worktrees are still mine"), never when it widens what you may do.

3. A brief is a hypothesis, and the receiver files its own ticket

A good brief states: kind (request, finding, FYI); approval status; evidence (exact command or file:line); what was ruled out; what is deliberately not proposed; constraints the receiver can't see; disk state.

The receiver creates the local task and ticket, and the ticket id goes inside the brief; a message is not a tracking system. The receiver verifies conclusion and mechanism separately: a peer once rightly called a worktree safe to remove, for a reason that was impossible.

4. Many sessions on one repo: each knows its co-workers by name

Several sessions on one repo is fine, each on its own worktree, if each gets, at start, the list of its peers by name and what each owns. Then a session that finds work belonging to a peer's piece delegates it there instead of duplicating it, and interconnected changes land in the right order. Before touching a shared path, announce your workstream, ticket, the exact meeting path, and the peer's likely surprise.

5. Order the delegate's queue

Hooks, then rules, then other substantive work, worktree cleanup last. A broken hook hits every session; a broken rule only its readers.

6. When a guard blocks you with no legitimate way through

Send the delegate the command, the guard, its exact message, and what you tried. Then stop: "blocked" is a complete answer. Repackaging the command into a script, reading the refusal's hint as permission, and narrowing the guard yourself are ruled out.

7. No Claude-generated edit goes unreviewed by a human

Nothing a session writes reaches a shared branch, cluster or cloud project until a human has read it.

Sessions stop at "ready for review". They open the PR; I read the diff and merge. A session never merges, even if its own option said so.
No hand changes to live systems. Mutating kubectl, console edits and Terraform applies are blocked or handed to me as a command; clusters change only via a reviewed commit.
GitOps is the second check. Git records every change; ArgoCD shows drift from git. To check what slipped past review, I compare the two, not a session's summary.

The cost: review is the tiring bottleneck; small PRs and summaries of what was tested, not edited, help. A session 95% right ships its 5% wrong unnoticed.

8. Verify before you ask: a non-negotiable global rule

Asking about a flag that doesn't exist turns approval into fact-checking. So the global "verify, don't claim" rule makes asking the gate: verify every load-bearing assumption before putting a change or an option list to me, attach the evidence, and mark anything unsettled as unverified. Three sources, strongest first:

The real environment. Read-only commands (describe, get, --dry-run=server, a live metrics or logs query); a live counter-example beats any document.
Authoritative documentation or source, at the version actually deployed. Docs via a lookup tool, not memory; upstream source at the release tag you run, not main; --help on the exact subcommand.
Online research, as corroboration when the first two are silent.

A proposal claims a blast radius; a conclusion like "impossible" needs a search for a working instance. A silent authoritative source is my decision, never a gap to guess.

A hook enforces the query part. The version clause came from fact-checking this series: a checker proposed a migration flag that upstream main has but the target release lacks, and that isn't equivalent to the documented one. Only per-tag source showed that.

The payoff: fewer, evidence-backed questions and fewer false results in PRs and docs.

9. Sessions have priorities: visit the urgent ones more often

An incident outranks a doc tidy-up. A waiting session is stalled, so a rarely visited one runs at my response rate. I visit urgent sessions far more often and answer questions as they arrive.

Having ADHD really helps here: jumping between many short threads is how my attention works anyway. Every session also writes for it: action first, open questions restated in full, one next step.

Rule layering: where knowledge goes

Layer	Holds	Test
Global `CLAUDE.md`	Short non-negotiables pointing to detail docs	Loaded every session; characters cost tokens
Global pattern/workflow/troubleshooting docs	Examples, command lists	Default home for anything over one line
In-repo `.claude/` + `CLAUDE.md`	Team-visible conventions	Would a teammate's agent act wrongly without it?
Per-user memory	Facts about this human	Agent-behaviour lessons are not personal

Copy, never point. A team repo never references a personal KB file teammates lack; copy the content (a hook rejects personal-KB paths).
If you broke an existing rule, extend its doc, never memory: a silent rule lets the next session repeat the breach.

Guard hooks as enforcement

About 49 hook files, 57 test files:

Class	Example
Block mutating `kubectl`	reads, `diff`, `--dry-run=server`, ArgoCD sync patches allowed.
Block commits/edits on `main`	team repos need a worktree; personal exempt.
Require full PR URLs	a bare `#123` can't be opened from a terminal.
Prevent scratch files in `/tmp`	it vanishes mid-session.
Require query verification	denies PromQL/LogQL/SQL in a doc unless that exact query ran live this session: a documented join had never matched a row.
Stop-time push check	blocks ending a turn with unpushed commits or a PR-less branch, in every worktree the session itself edited or committed to (checking only the current repo once missed one).

The enforcement ladder, weakest first: prose in a doc → a hook that blocks the call → removing the option entirely. A bare-name permissions.deny entry removes a whole tool from Claude's context; a scoped rule like Bash(rm *) only blocks matching calls.

Use PreToolUse (exit 2 blocks; PostToolUse cannot): a post-edit CLAUDE.md size warning only watched it grow; denying the projected size in PreToolUse fixed it.

Hooks are code: test-first, every time

A broken hook blocks work everywhere or silently stops guarding, so hooks get strict TDD:

Write the failing test first. Every hook has a suite in hooktests/; each new case (an input to deny, a near-miss to allow) goes in first.
Prove it goes RED. Run the new cases against a copy of the hook from before the change; a test passing against the old hook may not exercise the branch.
Make it GREEN, then run the whole suite. One runner executes every suite, a syntax sweep, and a check that every registered hook path resolves (a parsing hook can be dead if unregistered); per-fix spot checks caused the racing failure.
A hook change without a test change is denied by a commit guard, so time pressure can't skip it.

Fail open on unparseable input or a missing jq; an over-broad Bash guard also removes file search. The exception is a guard on an irreversible action (see the update below).

Team rules (in-repo)

Branches <ticket>/<type>/<desc>, never commit to main, Conventional Commits without scopes (on squash-merge repos the PR title's type is the release bump).
Each repo has a KB index (.claude/README.md) and its own hooks in .claude/settings.json, so the team runs them too. Quote SKILL.md descriptions: unquoted ones hid 9 of 14 skills from a stricter-YAML agent.

Cost: model routing

Opus primary plans, synthesises and reviews; cheaper worker subagents execute in parallel. Routine work: Sonnet primary, with an Opus oracle for hard decisions. Never downgrade reasoning on risky or irreversible decisions.
Context isolation saves the most: one misconfigured autocompact threshold (a 1M-context session compacting only at about 800k tokens) outweighed everything else.

Pros / cons

Pros	Cons
Real parallelism, full context each	Coordination overhead: briefs, notices, a queue
One writer removes write races	Delegate is a bottleneck; you still approve everything
Hooks catch repeat mistakes everywhere	Hooks need tests; a bad hook blocks every session
Review plus GitOps catches what guards miss	Reviewing every PR is the real, tiring bottleneck

When not to do this: one or two sessions, or no shared global config.

Gotchas

After removing a worktree, re-list and tell the owning session, or its next command fails.
A session's readable name needs a cmux call; the environment holds only UUIDs.
cmux's default next/previous-workspace shortcut crosses groups; since 0.64.23, bind Next/Previous Workspace in Group.

Takeaways

One dedicated session writes global rules and hooks; approval is never relayed; the receiver files the ticket.
Copy team knowledge into the repo; memory holds only facts about the human.
Climb the ladder: doc, PreToolUse deny, remove the option; hooks are test-first with a proven RED.
A human reviews every change; git plus ArgoCD's diff shows nothing slipped past.
Verify before asking: live environment, docs or source, then the web.
Name each session's co-workers; visit urgent sessions often.

References

Claude Code hooks reference (exit codes): https://code.claude.com/docs/en/hooks
Claude Code permissions (permissions.deny): https://code.claude.com/docs/en/permissions
cmux changelog (0.64.23): https://github.com/manaflow-ai/cmux/blob/main/CHANGELOG.md

Update, 12 October 2026: what changed after publishing

Readers' comments on this article led straight to fixes. Thank you to everyone who took the time to write.

Prompted by readers:

I replayed my real commands against the guards. One reader measured which risky commands their agents actually ran, instead of the ones they feared. I did the same: about 30,000 shell commands from my sessions held 161 destructive git commands, and none of my guards blocked any of them. Most named explicit files, a session throwing away its own edit. Seven were checkout or restore with no real path (git checkout -- ., git restore .), which in a shared checkout also drops every other session's uncommitted work, with no undo. A new guard blocks only that form; explicit paths stay allowed. Covering all 161 would have fired constantly and been muted within a day.
Guards now count their own runs and blocks. Another reader keeps per-lesson counters to see whether a lesson holds. I adopted the run-and-block half: every guard logs each run and each block, so whether it still earns its place is decided from counts, not impressions. I did not adopt retiring a guard after a quiet stretch. A quiet guard may simply be working.
"Have you retired a guard yet?" Once. A nudge hook fired on 20% of 3,583 replayed commands, so it was retired for being wrong too often, not for being quiet. Its own tests had all passed, because every case was one where it should fire.

Other corrections since publishing:

Guards on irreversible actions now fail closed. Claude Code added onFailure: "block" to hook registrations, so a guard that can't run, times out or crashes denies the action instead of letting it through. I set it only where a miss can't be undone: mutating kubectl, Terraform applies, secret access, mutating SQL, and pointing prod at a branch. Never on Edit or Write: a failing guard there would block its own repair. Everything else still fails open, because stopping every session over a recoverable mistake is the wrong trade.
The stop-time push check now covers only worktrees the session itself changed. It used to check any branch mentioned in the transcript, which flagged other sessions' work.

I wrote this article with the help of Claude, an AI assistant. I reviewed and edited it by hand, and it is based on my own hands-on experience.

DEV Community: Ilya Gorban

An AI-Native Dev Lifecycle Across R&D: The Plan

Why

What

Three risk tiers

Two levels of knowledge

Ground truth from history

How

Before merge: a read-only classifier

After merge: link alerts to merges

Feedback: teach the PR agent from outcomes

The compliance gate

What I took from published ADLCs, and where I disagree

What part 2 will report

References

Testing Crossplane Compositions with Chainsaw: kind Without Credentials, Then a Prod-Like Environment

Why

What

One set of use cases

Which layer owns a check

How

kind without credentials

Prod-like environment, from a developer's machine

Why Chainsaw and not Uptest

What the new layers found

Gotchas

Upgrade testing

Checklist

References

Writing and Testing Crossplane Compositions in KCL

Why

What

How

1. Editing: extract, edit in an IDE, put it back

2. Fixtures: one JSON per composition, one key per scenario

3. run.sh: render, diff against expected output, fail on drift

4. CI

5. Beyond unit tests: live e2e and canaries

KCL language traps (hit in practice)

Content-keyed names: the rule that prevents an outage

Pros / cons

Gotchas / lessons learned

Checklist

References

Update, 14 October 2026

Crossplane Black Boxes for Developer Self-Service

Why

What

Two layers, never mixed

Claimless v2

How: the guardrails, layer by layer

1. Developers can only create the black boxes

2. The schema is the policy

3. Env isolation lives in the provider layer, not in good intentions

4. Identity-only service accounts, no default roles

5. Green in ArgoCD means correct

The same black box for alerts

Evolving XRDs and compositions

Letting a coding agent generate the right resources

Pros / cons

Gotchas I actually hit

Checklist

References

Running 15 Claude Code Sessions at Once

Why: what goes wrong with an "octopus"

What: the setup

How

1. One dedicated session owns the global rules and hooks

2. Delegation moves the work, never the approval

3. A brief is a hypothesis, and the receiver files its own ticket

4. Many sessions on one repo: each knows its co-workers by name

5. Order the delegate's queue

6. When a guard blocks you with no legitimate way through

7. No Claude-generated edit goes unreviewed by a human

8. Verify before you ask: a non-negotiable global rule

9. Sessions have priorities: visit the urgent ones more often

Rule layering: where knowledge goes

Guard hooks as enforcement

Hooks are code: test-first, every time

Team rules (in-repo)

`kind` without credentials

3. `run.sh`: render, diff against expected output, fail on drift