DEV Community: Moep90

Kapitan generator releases, 19 to 25 October 2026

Moep90 — Sat, 07 Nov 2026 21:05:40 +0000

Versions published on kadet-hub from 19 to 25 October 2026, each through the gate and signed. The generator page shows each version's output diff against the version before it, its SBOM and the command that verifies it.

Generator	Versions	Changes
cloudnative-pg	1.1.0, 1.1.1	CHANGELOG
external-secrets	1.0.5	CHANGELOG
keycloak	1.1.0	CHANGELOG
kubernetes	1.4.0, 1.5.0, 1.6.0, 1.7.0, 1.8.0, 1.9.0, 1.9.1	CHANGELOG
kubevirt	1.1.2, 1.2.0, 1.3.0, 1.3.1, 1.3.2	CHANGELOG
mutate (new)	0.1.0	CHANGELOG
rabbitmq	1.1.0	CHANGELOG
strimzi	1.1.0	CHANGELOG

cloudnative-pg

cloudnative-pg 1.1.1

__version__ now reports the release version, which still read 1.0.1. Rendered output is unchanged.

cloudnative-pg 1.1.0

name on any entry names its resource and its output file, which before always took the entry's key. A recipe that may write only keys of its own name can then keep a resource name, such as the Cluster app-db whose claims and Secrets carry its name. Two entries of a section with the same name in one namespace now fail the compile; before, the later one replaced the earlier output file. Output is unchanged for inventories without name and without such a collision.

external-secrets

external-secrets 1.0.5

Built for External Secrets Operator 2.13.0. The upstream CRDs in config/crds/bases are unchanged against 2.12.0, so the shipped schemas, CRDs and rendered output are unchanged.

keycloak

keycloak 1.1.0

name on an entry of instances, realm_imports, oidc_clients or saml_clients names its resource and its output file, which before always took the entry's key. The defaults realm.realm and keycloakCRName follow it. A recipe that may write only keys of its own name can then keep a Keycloak name, such as sso, whose StatefulSet, Services and Secrets carry it. Two entries of a section with the same name in one namespace now fail the compile. Output is unchanged for inventories without name and without such a collision.

kubernetes

kubernetes 1.9.1

__version__ now reports the release version, which still read 1.1.1. Rendered output is unchanged.

kubernetes 1.9.0

webhook_configuration_name and validating_webhook_configuration_name on a component name its MutatingWebhookConfiguration and ValidatingWebhookConfiguration. Without them both stay <namespace>-<component>. The CloudNativePG operator writes its CA only into configurations of fixed names.
A volume_mounts entry takes volume, the volume it mounts. Its key then names only the mount, so one volume can be mounted at several paths with their own sub_path, as Nextcloud's chart mounts its claim six times. Without volume the key names the volume, as before.

Output is unchanged for inventories without the two name keys and without volume in volume_mounts.

kubernetes 1.8.0

generators.kubernetes.routes.<name> renders one route attached to Gateways outside the inventory, named in parent_refs by name and optionally namespace, section_name and port. It takes the route keys of gateways.<gw>.routes and renders no Gateway. Before, a route was rendered only together with its Gateway, so a route to the cluster's shared Gateway needed a manifests entry. The file is <namespace>/<name>-route.
A redirect rule takes replace_full_path or replace_prefix_match, which render the redirect's path as ReplaceFullPath or ReplacePrefixMatch, on the routes of gateways and of routes. Both together fail the compile.
webhook_ca_injected: true on a component renders its webhooks with failurePolicy: Fail and no caBundle, for a server that writes the CA into its webhook configurations itself, as the CloudNativePG operator does. Without it such a webhook still fails the compile.

Output is unchanged for inventories without routes, without the two redirect keys and without webhook_ca_injected.

kubernetes 1.7.0

args takes a mapping next to the list. Its values render in the mapping's order: a string as one argument, a list as several, null as none. A target can override or drop one argument a class sets by its key, which a list did not allow because the backends append or merge lists. A mapping value fails the compile.
service: {} renders the component's Service with the API defaults. Before, an empty mapping rendered no Service and type: ClusterIP had to be written. api_service accepts it as well. A StatefulSet with service: null no longer fails the compile.
README: component labels go on the workload, its Services, HPA and VPA, not on the pod template. Service, PodDisruptionBudget and NetworkPolicy selectors take all pod template labels. The README also states the label-value rule for names, the sorted container ports and namespaceSelector: {} on a webhook.

Output is unchanged for inventories without an empty service mapping and without an args mapping.

Full changelog

kubevirt

kubevirt 1.3.2

__version__ now reports the release version, which still read 1.1.1. Rendered output is unchanged.

kubevirt 1.3.1

The KubeVirt CR renders spec: {} when the inventory sets no spec or spec: null. Before, it rendered without spec, which the CRD requires, so the API server rejected it with spec: Required value. Targets that set spec render the same output.

kubevirt 1.3.0

csi.image is optional and defaults to ghcr.io/kadet-hub/mirror/kubevirt-csi-driver:3a1a099330045754011803e3d53cd8276ff21723@sha256:44c98353e6137455d6a393ad13e04f60e10ee49fdcd00cf078f7f7084714f409, kadet-hub's unmodified copy of the quay.io latest build of 3a1a099. Targets that set image render the same output. The README's Driver image section links the image's upstream sources.

kubevirt 1.2.0

Adds csi.snapshot_controller for the tenant role. true renders the snapshot-controller of external-snapshotter v8.7.0 with its RBAC and the VolumeSnapshotClass, VolumeSnapshotContent and VolumeSnapshot CRDs, so a tenant VolumeSnapshotClass works without a separate install. The controller runs in the CSI namespace with a restricted security context. The README's CSI defaults list where it differs from upstream's manifest. On the infra role the key fails the compile. Output without the key stays the same.

The snapshot.storage.k8s.io schema group now names external-snapshotter 8.7.0 (client/config/crd) as its source instead of the KubeVirt CSI driver's copy, and adds the VolumeSnapshotContent and VolumeSnapshot CRDs. All three files are upstream's byte for byte, and the output renders them from there.

kubevirt 1.1.2

Built for the KubeVirt CSI driver at main@e13603a. Upstream now deploys what csi already rendered. Since kubevirt/csi-driver#392 its controller runs with the restricted security context, with UID and GID 1001 from the plain-kubernetes component, and the node plugin's csi-driver container as UID 0. Since kubevirt/csi-driver#396 it pins csi-snapshotter v8.7.0. The rendered output stays the same, and the README's CSI defaults keep only the remaining differences. The VolumeSnapshotClass CRD and schema move to e13603a, whose file is external-snapshotter's v8.7.0 CRD and differs from the shipped one only in descriptions.

The driver image is still the latest build of 3a1a099. Upstream removed its image workflow (kubevirt/csi-driver#393), so the controller fix of kubevirt/csi-driver#397 is in no published image.

mutate

mutate 0.1.0

First release. Patches the YAML files an earlier compile entry of the same output_path wrote, such as Kapitan's helm input, in place: mutations keyed by id, applied in id order with deletes last, match objects exactly on apiVersion, kind, name, namespace and labels, and merge (JSON Merge Patch, with merge_lists for keyed list merge), json6902, delete or images. expects fails the compile when a mutation matches no object, or not the given count. Only changed files are written; a file whose objects are all deleted becomes a comment line. The compile entry needs output_type: plain.

rabbitmq

rabbitmq 1.1.0

name on an entry of a custom resource section names its resource, its output file and, for the kinds with spec.name, the default of spec.name, which before always took the entry's key. A recipe that may write only keys of its own name can then keep a resource name, such as the RabbitmqCluster production whose StatefulSet, claims and Secrets carry its name. Two entries of a section with the same name in one namespace now fail the compile; before, the later one replaced the earlier output file. Output is unchanged for inventories without name on those sections and without such a collision.

strimzi

strimzi 1.1.0

name on a Kafka, node pool, topic or user entry names its resource and its output key, which before always came from the entry's key. A recipe that may write only keys of its own name can then keep a cluster's name, such as the Kafka events with the node pool dual whose pods, claims, Services and Secrets carry both names. cluster and strimzi.io/cluster name the Kafka's resource. A topic or user whose cluster names Kafkas in two namespaces fails the compile unless it sets namespace. Output is unchanged for inventories without name.

Originally published at kadet-hub.org.

Docker tag and digest: which one Kubernetes runs

Moep90 — Sat, 07 Nov 2026 13:43:39 +0000

An image reference can carry a tag and a digest at
once, as in busybox:1.37.0@sha256:73aaf090…. Kubernetes runs the image
the digest names. The tag is removed before the pull and never compared
with the digest. If the two disagree, the pod runs the digest's image while
the spec shows the tag's version.

The recipes in kadet-hub/recipes could produce exactly that
mismatch. They kept the tag in one file and the digest in another, until
kadet-hub/recipes#50 put
both into one literal line.

Everything below ran on 2026-11-02 with kind v0.33.0 and its default node
image kindest/node:v1.37.0@sha256:a1ed56cf… (Kubernetes v1.37.0,
containerd v2.3.4), kubectl v1.37.2 and oras 1.3.6.

A tag that disagrees with its digest

The two busybox tags resolved to these digests:

$ oras manifest fetch --descriptor docker.io/library/busybox:1.36.1
{"mediaType":"application/vnd.oci.image.index.v1+json","digest":"sha256:73aaf090f3d85aa34ee199857f03fa3a95c8ede2ffd4cc2cdb5b94e566b11662","size":9535}
$ oras manifest fetch --descriptor docker.io/library/busybox:1.37.0
{"mediaType":"application/vnd.oci.image.index.v1+json","digest":"sha256:a7dbc319b1438115a207e11930dc4bb3e643dcb3ec0bdd694bb36cd8fad289ec","size":9535}

Each is the digest of an OCI image index, the list of per-platform images
behind the tag. Tags can move, so a later run may resolve other digests.

The pod takes the tag 1.37.0 and the digest of 1.36.1. That is the state a
version bump leaves behind when the digest stays:

apiVersion: v1
kind: Pod
metadata:
  name: tag-and-digest
spec:
  restartPolicy: Never
  containers:
  - name: busybox
    image: busybox:1.37.0@sha256:73aaf090f3d85aa34ee199857f03fa3a95c8ede2ffd4cc2cdb5b94e566b11662
    command: [sh, -c, "busybox | head -1"]

The API server accepts it, and the kubelet pulls and starts it:

$ kubectl get events --field-selector involvedObject.name=tag-and-digest -o jsonpath='{range .items[*]}{.reason}{": "}{.message}{"\n"}{end}'
Scheduled: Successfully assigned default/tag-and-digest to post-digest-24c921-control-plane
Pulling: Pulling image "busybox:1.37.0@sha256:73aaf090f3d85aa34ee199857f03fa3a95c8ede2ffd4cc2cdb5b94e566b11662"
Pulled: Successfully pulled image "busybox:1.37.0@sha256:73aaf090f3d85aa34ee199857f03fa3a95c8ede2ffd4cc2cdb5b94e566b11662" in 2.15s (2.15s including waiting). Image size: 2217006 bytes.
Created: Container created
Started: Container started

The container prints the version of the binary it runs:

$ kubectl logs tag-and-digest
BusyBox v1.36.1 (2023-05-18 22:34:17 UTC) multi-call binary.

The pod's image fields:

$ kubectl get pod tag-and-digest -o jsonpath='{.spec.containers[0].image}{"\n"}{.spec.containers[0].imagePullPolicy}{"\n"}{.status.containerStatuses[0].image}{"\n"}{.status.containerStatuses[0].imageID}{"\n"}'
busybox:1.37.0@sha256:73aaf090f3d85aa34ee199857f03fa3a95c8ede2ffd4cc2cdb5b94e566b11662
IfNotPresent
sha256:b116e155074440ffd9e449559433feb4cd2341eb3554b1da1c638c976e56451d
docker.io/library/busybox@sha256:73aaf090f3d85aa34ee199857f03fa3a95c8ede2ffd4cc2cdb5b94e566b11662

.spec.containers[].image keeps the string as written, tag included.
.status.containerStatuses[].imageID holds the repository and the digest
without a tag. It is the field that tells you which image runs. Its digest
is the one oras resolved for 1.36.1.

.status.containerStatuses[].image carries no tag either. containerd
reports the image's first tag there, and the node-local image ID when the
image has none
(container_status.go).
That ID, sha256:b116e155…, is the digest of the image's config. The node
stored no tag for the image it pulled by digest:

$ docker exec post-digest-24c921-control-plane crictl images --digests | grep -E 'IMAGE|busybox'
IMAGE                                           TAG                  DIGEST              IMAGE ID            SIZE
docker.io/library/busybox                       <none>               73aaf090f3d85       b116e15507444       2.22MB

A second pod with busybox:1.37.0 and no digest prints BusyBox v1.37.0.
Its imageID ends in a7dbc319…, the digest of 1.37.0. Its
.status.containerStatuses[].image is docker.io/library/busybox:1.37.0.

The tag is never looked up

The tag no-such-tag does not exist:

$ oras manifest fetch --descriptor docker.io/library/busybox:no-such-tag
Error response from registry: failed to find "docker.io/library/busybox:no-such-tag": docker.io/library/busybox:no-such-tag: not found

A third pod uses it in front of the 1.36.1 digest, with
imagePullPolicy: Always so that the kubelet asks for a pull:

apiVersion: v1
kind: Pod
metadata:
  name: no-such-tag
spec:
  restartPolicy: Never
  containers:
  - name: busybox
    image: busybox:no-such-tag@sha256:73aaf090f3d85aa34ee199857f03fa3a95c8ede2ffd4cc2cdb5b94e566b11662
    imagePullPolicy: Always
    command: [sh, -c, "busybox | head -1"]

Its events report
Successfully pulled image "busybox:no-such-tag@sha256:73aaf090…", and it
runs 1.36.1:

$ kubectl logs no-such-tag
BusyBox v1.36.1 (2023-05-18 22:34:17 UTC) multi-call binary.

The kubelet hands the reference to the container runtime as written. Its
applyDefaultImageTag appends :latest only when a reference has neither
a tag nor a digest
(image_manager.go,
Kubernetes v1.37.0).

containerd's CRI plugin normalizes the reference with ParseDockerRef
before it pulls
(image_pull.go).
The function comes from distribution/reference v0.6.0, the version
containerd v2.3.4 depends on. Its documentation says what happens to the
tag:

For references containing both a tag and a digest, it returns a digested reference.

Its example turns docker.io/library/busybox:latest@sha256:7cc4b5ae…
into docker.io/library/busybox@sha256:7cc4b5ae…
(normalize.go).
containerd also resolves a reference against the images already on the
node through ParseDockerRef
(service.go).
The presence check behind IfNotPresent therefore ignores the tag as well.

The Kubernetes documentation states the outcome in its list of image name
examples. It describes registry.k8s.io/pause:3.5@sha256:1ff6c18f… as
"Image name with tag and digest. Only the digest will be used for
pulling."
(Images).

The tag still sets the pull policy

When imagePullPolicy is omitted, the API server defaults it from the tag.
SetDefaults_Container sets Always for the tag latest and
IfNotPresent for any other
(defaults.go).
The tag comes from ParseImageName, which returns it even when a digest is
present
(parsers.go).
A server-side dry run shows both cases:

$ kubectl run latest-and-digest --restart=Never --image=busybox:latest@sha256:73aaf090f3d85aa34ee199857f03fa3a95c8ede2ffd4cc2cdb5b94e566b11662 --dry-run=server -o jsonpath='{.spec.containers[0].imagePullPolicy}{"\n"}'
Always
$ kubectl run digest-only --restart=Never --image=busybox@sha256:73aaf090f3d85aa34ee199857f03fa3a95c8ede2ffd4cc2cdb5b94e566b11662 --dry-run=server -o jsonpath='{.spec.containers[0].imagePullPolicy}{"\n"}'
IfNotPresent

The documentation's list of defaults says: "if you omit the
imagePullPolicy field, and you specify the digest for the container
image, the imagePullPolicy is automatically set to IfNotPresent"
(default image pull policy).
With :latest in front of the digest, the tag decides and the result is
Always. Under Always the runtime contacts the registry at every
container start, although the digest already fixes the image
(image pull policy).

Two files, one image

The recipes are Kapitan inventory classes. Until
kadet-hub/recipes#50, each kept an image's tag in
applications.<name>.version in its init.yml, and the digest in the
image line of another class file. In the miniflux recipe, init.yml set
version: 2.3.4 and miniflux.yml had this line:

image: miniflux/miniflux:${applications.miniflux.version}@sha256:72e49f513a63104506aab98ba8cfc3a409d4fde23550a583dec07ec941f1e94e

The ${...} is an interpolation that Kapitan
fills in from the parameter. A target that raised the version without the
digest compiled to manifests with the new tag and the old digest. That is
the first pod above. The spec and every diff of the compiled manifests show
the new version, and the kubelet runs the old image.

The fix writes each image as one literal,
<registry>/<repository>:<tag>@sha256:<digest>:

image: miniflux/miniflux:2.3.4@sha256:72e49f513a63104506aab98ba8cfc3a409d4fde23550a583dec07ec941f1e94e

applications.<name>.version is gone wherever it only fed image tags.
check lint, the linter the recipes run in CI, now fails on a line of a
class file that holds both @sha256: and ${:

for n, line in enumerate(text.splitlines(), 1):
    if "@sha256:" in line and "${" in line:
        errors.append(f"{f}:{n}: image reference takes its tag from an interpolation (RCP-66)")

A reference without a digest does not match this check, and the recipes'
specification leaves those to review
(RCP-66).

Renovate reads the tag from the same line

Renovate writes references in this form. Its Docker documentation says
that "When pinning a digest, Renovate retains the Docker tag in the FROM
line for readability", with FROM node:14.15.1@sha256:d938c176… as the
example
(Digest Pinning).
The pinDigests option pins images "by means of their SHA256 digest and
not only by tag"
(pinDigests).

Renovate's updates start from the tag it finds in the file
(How It Works):

If the image tag in use "looks" like a version (e.g. myimage:1, myimage:1.1, myimage:1.1.0, myimage:1-onbuild) then Renovate checks the Docker registry for upgrades

The same page describes pull requests for a new digest behind an unchanged
tag
(Digest Updating).

In kadet-hub/recipes, Renovate's regex managers capture the tag as
currentValue and the digest as currentDigest from the same image line.
Those are the capture groups the
regex manager documentation
asks for. The old miniflux line had ${applications.miniflux.version}
where that version belongs.

Originally published at kadet-hub.org.

Renovate automerge with required reviews on GitHub

Moep90 — Sat, 07 Nov 2026 13:43:30 +0000

Renovate's automerge stops at a branch that requires an approving review.
Its documentation says so: "If you have mandatory Pull Request reviews then
it means Renovate can't automerge its own PR until such a review has
happened"
(Renovate docs).
The main ruleset of kadet-hub/registry requires one approving review and
a code-owner review. Since 2 November 2026 Renovate merges its own minor,
patch and digest updates there without either. The Renovate GitHub App has
a ruleset bypass that works on pull requests only. GitHub's auto-merge
stays off, so Renovate merges once every check on the pull request has
passed, including the checks the ruleset does not require.

The change is
kadet-hub/registry#441.
The first pull request merged this way was
#439, a digest update of
the distroless debug image that the gate's sandbox
image copies its busybox from. The GitHub docs quoted below were read on
2 November 2026, the Renovate docs and source at Renovate 44.201.0.

Why an approval bot does not help

Renovate's docs name two ways past a required review on GitHub. One is a
bypass of the review requirement. The other is the helper apps
renovate-approve and renovate-approve-2, which "will mark all automerging
Pull Requests by Renovate as approved"
(Renovate docs).

An approval from an app does not satisfy a code-owner review. With
code-owner review required, "any pull request that modifies content with a
code owner must be approved by that code owner"
(GitHub docs).
A CODEOWNERS file lists owners as "GitHub usernames or team names"
(GitHub docs).
A workflow that approves with its GITHUB_TOKEN approves as an app too,
because "The GITHUB_TOKEN secret is a GitHub App installation access
token"
(GitHub docs).
Renovate's docs put it this way: "having a CODEOWNERS file could block
automerging, because it means a code owner must review the PR"
(Renovate docs).

kadet-hub/registry has a code owner for .github/, gate/, policy/,
scan/, sandbox/ and .clusterfuzzlite/. Every file Renovate updates in
that repository sits under one of them, apart from the registry entries.

439 changed `sandbox/Dockerfile`.

GitHub's auto-merge is the other half of that setup, and Renovate uses it
by default. "Auto-merge merges a pull request automatically after all
required reviews and status checks pass"
(GitHub docs).
It does not wait for a check the branch does not require. The registry's
ruleset requires two checks, gate and renders.

Several checks that exercise a dependency update in this repository come
from path-filtered workflows. selftest runs the gate's self tests when
sandbox/, gate/, scan/, policy/ or a few other paths change.
sandbox-advisories
fails on an advisory that has a fix for the sandbox image. GitHub's
troubleshooting page says that the checks of a workflow skipped by path
filtering "stay in a "Pending" state and block merging", and advises:
"Avoid requiring workflows that can be skipped"
(GitHub docs).
So these checks are not required, and auto-merge would not wait for them.

A bypass for pull requests only

441 added one entry to the bypass list of the `main` ruleset. The

repository keeps a copy of the ruleset in .github/ruleset.json:

-  "bypass_actors": [{"actor_id": 5, "actor_type": "RepositoryRole", "bypass_mode": "pull_request"}],
+  "bypass_actors": [
+    {"actor_id": 5, "actor_type": "RepositoryRole", "bypass_mode": "pull_request"},
+    {"actor_id": 2740, "actor_type": "Integration", "bypass_mode": "pull_request"}
+  ],

Integration is the actor type of a GitHub App. 2740 is the ID that
gh api /apps/renovate returns for the Renovate app. The live ruleset
carries the same entry:

$ gh api repos/kadet-hub/registry/rulesets/24720369 --jq '.bypass_actors[] | select(.actor_type == "Integration")'
{"actor_id":2740,"actor_type":"Integration","bypass_mode":"pull_request"}

The REST API accepts always, pull_request and exempt for
bypass_mode, and its reference says: "pull_request means that an actor
can only bypass rules on pull requests"
(GitHub REST docs).
The default is always. An entry that leaves bypass_mode out lets the
app push to main directly.

In the ruleset editor, pull_request is the option "For pull requests
only", which GitHub offers "to grant bypass to an actor without allowing
them to push directly to a repository". The same page goes on:

The selected actor is now required to open a pull request to make
changes to a repository, creating a clear trail of their changes in the
pull request and audit log. The actor can then choose to bypass any
branch protections and merge that pull request.

(GitHub docs)

The bypass covers the rules of the whole ruleset, the required status
checks among them. Once the app is on the list, GitHub no longer stops its
merge on a failed check. That job moves to Renovate.

What Renovate waits for

The top of renovate.json leaves automerge off by default and switches
GitHub's auto-merge off:

  "automerge": false,
  "platformAutomerge": false,
  "minimumReleaseAge": "3 days",
  "minimumReleaseAgeBehaviour": "timestamp-required",

With platformAutomerge at its default true, Renovate hands the merge to
the platform, which merges "after all (if any) branch policies have been
met"
(Renovate docs).
With false, Renovate merges with its own API call. For the default
automergeType of pr, Renovate "automerges the PR on a subsequent run
once it detects the PR's status checks are 'green'", and "Renovate only
automerges branches which are up-to-date and green"
(Renovate docs).
The entry for ignoreTests states the default: "Renovate's default
behavior is to only automerge if every status check has succeeded"
(Renovate docs).

What green means on GitHub is decided in getBranchStatus
(Renovate 44.201.0 source).
It reads the commit statuses and the check runs of the branch's head
commit. One check run with the conclusion failure makes the branch red.
The branch is green only when every check run has the conclusion
success, neutral or skipped and the remaining commit statuses are
successful. A check run that is queued or still running has no conclusion
yet, so the branch stays yellow and Renovate does not merge. Successful
statuses that Renovate sets itself, such as renovate/stability-days, are
filtered out first, so they cannot turn a branch green on their own. GitHub
counts the same three conclusions as a passing required check
(GitHub docs).

Renovate sees only the checks that exist on the commit when it runs. A
workflow that has not created its check run yet cannot hold the merge.

Which updates merge

The last three packageRules of renovate.json decide what Renovate may
merge:

    {
      "description": "REV-5: Renovate merges its own pull request once every check passed",
      "matchUpdateTypes": [
        "minor",
        "patch",
        "digest",
        "pin",
        "pinDigest"
      ],
      "automerge": true
    },
    {
      "description": "REV-5: an entry bump publishes a new version, which a maintainer approves (DEC-3)",
      "matchFileNames": [
        "generators/*.yaml",
        "recipes/*.yaml"
      ],
      "automerge": false
    },
    {
      "description": "REV-5: a digest update of an action under the same version is a moved tag",
      "matchManagers": [
        "github-actions"
      ],
      "matchUpdateTypes": [
        "digest"
      ],
      "automerge": false
    }

The descriptions cite kadet-hub's specs. REV-5 is the automerge requirement
in gate.md,
and DEC-3 is the decision in
marketplace.md
that every new version needs a maintainer's approval.

Renovate applies packageRules in order, and "later rules may override
configuration options from earlier ones, if they both specify the same
option"
(Renovate docs).
The first rule turns automerge on for five update types. A major update
matches none of the three and keeps the top-level "automerge": false.

The second rule covers the registry entries. Each file under generators/
and recipes/ pins a generator or a recipe to a tag and commit in its
author's repository. Merging a bump of that pin publishes the new version
in the catalog, so these bumps stay with a maintainer whatever their update
type.

The third rule is about GitHub Actions. kadet-hub pins every action to a
commit SHA with the version as a comment, through the
helpers:pinGitHubActionDigests preset. A new release of an action changes
the version in that comment. A digest update under the same version
means that the tag now points at another commit. That is how
tj-actions/changed-files was compromised in March 2025: "Attackers
retroactively modified multiple version tags to reference a malicious
commit"
(GHSA-mrrh-fwg8-r2c3).
The three days of minimumReleaseAge do not hold such an update back: "If
an existing tag is force-pushed to new commits, the digest update ages
against the original release date, so it may pass Minimum Release Age
immediately"
(Renovate docs).

Everything else still waits three days after its release when the
datasource reports a release timestamp. kadet-hub's config treats git tags
and registries other than Docker Hub as reporting none and lets those
updates through without one (minimumReleaseAgeBehaviour: timestamp-optional). Such an update merges as soon as its checks pass.

kadet-hub's threat model in marketplace.md records what this gives up. A
malicious minor, patch or image digest release now reaches main without
a maintainer once the release age allows it and every check passes. Before

441 it needed one approval by the single maintainer.

kadet-hub/recipes#56 made
the same change in kadet-hub/recipes and keeps more updates manual. Lock
file maintenance is not on its automerge list, because "the updated
packages have no release timestamp for Renovate to check"
(Renovate docs).
The pin of the next tag of kadet-hub's Kapitan image stays manual, since
that image is rebuilt up to every 15 minutes and its digest updates have no
release timestamp. A recipe's Source line names the chart or image the
recipe is built from, and updates to it wait for a reviewer. The main ruleset
there requires one approval without a code-owner review and has the same
pull_request bypass for the Renovate app.

The first merge

439 has no review, and GitHub still reports a review as required:

$ gh pr view 439 -R kadet-hub/registry --json mergedBy,mergedAt,reviewDecision,reviews
{"mergedAt":"2026-11-02T10:11:50Z","mergedBy":{"is_bot":true,"login":"app/renovate"},"reviewDecision":"REVIEW_REQUIRED","reviews":[]}

The check runs on its last commit, ordered by completion time in UTC:

$ gh pr view 439 -R kadet-hub/registry --json statusCheckRollup \
    --jq '.statusCheckRollup | map(select(.__typename == "CheckRun")) | sort_by(.completedAt)[] | [.completedAt, .conclusion, .name] | @tsv'
2026-11-02T09:25:20Z    SUCCESS workflow-lint
2026-11-02T09:25:24Z    SUCCESS checksum-pins
2026-11-02T09:25:31Z    SUCCESS sandbox-advisories
2026-11-02T09:25:49Z    SUCCESS gate
2026-11-02T09:25:49Z    SKIPPED renders
2026-11-02T09:25:59Z    SUCCESS comment
2026-11-02T09:26:06Z    SUCCESS consumer-selftest
2026-11-02T09:26:26Z    SUCCESS CodeQL
2026-11-02T09:26:31Z    SUCCESS analyze
2026-11-02T10:10:36Z    SUCCESS gate-selftest

The same rollup holds one commit status, renovate/stability-days, with
the state SUCCESS.

gate and renders are the required checks. renders checks a recipe on
kind and runs only when the gate compiled one, so on #439 it was skipped.
A skipped job "will not prevent a pull request from merging, even if it is
a required check"
(GitHub docs).
Both required checks were done at 09:25:49. gate-selftest finished at
10:10:36, and Renovate merged at 10:11:50.

GitHub keeps the rule evaluations of the merge as a rule suite, which the
REST API returns
(GitHub REST docs):

$ gh api repos/kadet-hub/registry/rulesets/rule-suites/4853993537 \
    --jq '{actor_name, ref, result, rules: [.rule_evaluations[] | {rule_type, result}]}'
{"actor_name":"renovate[bot]","ref":"refs/heads/main","result":"bypass","rules":[{"result":"fail","rule_type":"pull_request"},{"result":"pass","rule_type":"required_status_checks"},{"result":"pass","rule_type":"non_fast_forward"},{"result":"pass","rule_type":"deletion"}]}

The pull_request rule failed, and its details name the missing
code-owner review. The required status checks passed on their own. The
merge went through as bypass.

Originally published at kadet-hub.org.

Actions Runner Controller dind image from your own registry

Moep90 — Tue, 27 Oct 2026 21:28:06 +0000

With containerMode.type: dind, the gha-runner-scale-set chart of
actions-runner-controller (ARC) adds a Docker daemon to every runner pod.
The chart fixes that container's image as docker:dind, which the kubelet
pulls from Docker Hub, and the current release, 0.15.0 from 1 October 2026,
has no value to change it. This post replaces the image on the rendered
manifests, first with Kustomize over helm template, then with
Kapitan's mutate
generator, for a pull-through mirror and for a
registry inside an air-gapped network.

Each patch was rendered from the 0.15.0 chart with Helm 4.3.1, Kustomize
5.8.3, and Kapitan 0.36.3 with mutate 0.1.0, and applied to a kind cluster
on Kubernetes 1.37.0. The guide Patch a Helm chart
compares Helm post-renderers, Kustomize, Flux and Argo CD as places for such
a patch.

The dind image in gha-runner-scale-set {#arc-dind-image}

actions/actions-runner-controller#3997,
opened in March 2025 and at 28 thumbs up, asks for a way to pull the dind
image from your own registry instead of Docker Hub, whose pull limits for
unauthenticated users it cites. Pull request #3851, which made the image
configurable, was closed unmerged in August 2025. Closing the related #4219,
the ARC maintainer nikola-jokic
wrote
that "we don't want to create a patch that will simply modify the dind
image" and pointed to the way the chart's values.yaml documents: leave
containerMode empty and write the whole runner pod spec into template
yourself. That spec is two init containers, the runner container and three
volumes, which you then keep in step with the chart.

In 0.15.0 the image is a literal in
_helpers.tpl:

{{- define "gha-runner-scale-set.dind-container" -}}
image: docker:dind

The chart gha-runner-scale-set-experimental, published next to it as
0.15.0, has the value runner.dind.container.image, and renders a mirror
image with it. Its release notes say experimental charts "are not supported
for production workloads and may be modified or republished without
notice."

Where the chart puts dind

autoscalingrunnerset.yaml
tests ge (.Capabilities.KubeVersion.Minor | int) 29. From Kubernetes 1.29,
which turned on sidecar containers
by default, dind is the second init container, with restartPolicy: Always
and a startup probe that runs docker info, so the runner container starts
once dockerd answers. Below 1.29 dind is the second entry in containers.

helm template has no cluster to read the version from and uses its
built-in default, v1.37.0 for Helm 4.3.1. For an older cluster, pass
--kube-version. A patch has to look for dind in the list the chart
rendered for that version.

Kustomize over helm template

The values select dind mode and pin the runner image by tag and
digest; the chart uses the image of the container named
runner for the init container that copies the runner's externals, too.
controllerServiceAccount names the ServiceAccount of a controller
installed as release arc in arc-systems. Without it the chart looks up
the controller Deployment in the cluster, and helm template, which has
none, fails with No gha-rs-controller deployment found.

githubConfigUrl: https://github.com/example-org
githubConfigSecret: arc-github
controllerServiceAccount:
  namespace: arc-systems
  name: arc-gha-rs-controller
containerMode:
  type: dind
template:
  spec:
    containers:
      - name: runner
        image: ghcr.io/actions/actions-runner:2.338.0@sha256:4ffadc0002b2581327e06101fc8c06cd189232baf79fe561fac9caeb76f5e807
        command: ["/home/runner/run.sh"]

Render the chart and build the patch over the output:

helm template arc-runners oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set \
  --version 0.15.0 --namespace arc-runners -f values.yaml --kube-version 1.37.0 > helm.yaml
kustomize build .

with this kustomization.yaml next to helm.yaml:

resources:
  - helm.yaml
images:
  - name: docker
    newName: mirror.gcr.io/library/docker
    newTag: 29.9.1-dind
    digest: sha256:3e5756e2b6dad6a5f1f5deacf0a19f9eaa2581b76ae092b2d4a118162e1d4d4e

Kustomize's images transformer looks for containers and
initContainers at any depth of any object, so it finds dind in the
AutoscalingRunnerSet, a custom resource, in both layouts. The test
rendered the chart for 1.37.0 and for 1.28.0, and in both outputs the
dind image was the only field of the AutoscalingRunnerSet that changed. A strategic merge patch is no
alternative here: Kustomize has no schema for the AutoscalingRunnerSet and
replaces the whole initContainers list, so a patch that names only dind
and its image leaves a list with that one entry, without
init-dind-externals and without dind's arguments and privileged.

Kapitan's mutate

A class pins the chart by version and by the digest of
its OCI artifact, renders it with the same values file and passes the
Kubernetes version to Helm:

parameters:
  kube_version: 1.37.0
  kapitan:
    dependencies:
      - type: helm
        source: oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set@sha256:bdfdad17cf8e45ae064c73ed6764f0c75d4bf71a6e2912e6363a43d6dfc787de
        chart_name: gha-runner-scale-set
        version: 0.15.0
        output_path: charts/gha-runner-scale-set
    compile:
      - input_type: helm
        input_paths: [charts/gha-runner-scale-set]
        output_path: gha-runner-scale-set
        helm_values_files: [values.yaml]
        helm_params:
          name: arc-runners
          namespace: arc-runners
          kube_version: ${kube_version}

mutate's images action covers Pods and the Pod templates of workload
kinds, not custom resources, so the mutation is a JSON Patch:

parameters:
  mutations:
    dind-image:
      match: {apiVersion: actions.github.com/v1alpha1, kind: AutoscalingRunnerSet, name: arc-runners}
      json6902:
        - {op: test, path: /spec/template/spec/initContainers/1/name, value: dind}
        - op: replace
          path: /spec/template/spec/initContainers/1/image
          value: mirror.gcr.io/library/docker:29.9.1-dind@sha256:3e5756e2b6dad6a5f1f5deacf0a19f9eaa2581b76ae092b2d4a118162e1d4d4e

test fails the compile unless the second init container is dind, and
replace sets its image. The target adds mutate as a
second compile entry on the chart's output_path, as the guide shows. For a
cluster below 1.29 the class below sets the Kubernetes version and patches
containers instead:

parameters:
  kube_version: 1.28.0
  mutations:
    dind-image:
      match: {apiVersion: actions.github.com/v1alpha1, kind: AutoscalingRunnerSet, name: arc-runners}
      json6902:
        - {op: test, path: /spec/template/spec/containers/1/name, value: dind}
        - op: replace
          path: /spec/template/spec/containers/1/image
          value: mirror.gcr.io/library/docker:29.9.1-dind@sha256:3e5756e2b6dad6a5f1f5deacf0a19f9eaa2581b76ae092b2d4a118162e1d4d4e

Both mutated renders produced the same AutoscalingRunnerSet as Kustomize.
The first mutation on a render for 1.28.0, where dind is not an init container,
stops the compile:

mutate: mutation dind-image: gha-runner-scale-set/templates/autoscalingrunnerset.yaml: json6902 path '/spec/template/spec/initContainers/1/name' does not exist

Kapitan's helm compile entry also accepts a kube_version key of its own,
not the one under helm_params. Kapitan 0.36.3
passes it to Helm
as --api-versions, which leaves .Capabilities.KubeVersion at Helm's
default: with kube_version: 1.28.0 on the entry, dind was still rendered
as an init container.

On kind

The kind cluster got the four CRDs of the controller chart
gha-runner-scale-set-controller 0.15.0 and the mutated output of the
scale-set chart. The API server accepted both Kustomize outputs and the
mutated 1.28.0 output in a server-side dry run, and stored the
AutoscalingRunnerSet with these init containers (name, image,
restartPolicy):

init-dind-externals ghcr.io/actions/actions-runner:2.338.0@sha256:4ffadc0002b2581327e06101fc8c06cd189232baf79fe561fac9caeb76f5e807
dind    mirror.gcr.io/library/docker:29.9.1-dind@sha256:3e5756e2b6dad6a5f1f5deacf0a19f9eaa2581b76ae092b2d4a118162e1d4d4e    Always

The controller creates runner pods only for a scale set registered with
GitHub, so the test built a Pod from the stored spec.template and
replaced the runner's command with docker version followed by a sleep.
The Pod became Ready. The kubelet reported the dind image as pulled from
the mirror, with the digest of the patch:

dind    mirror.gcr.io/library/docker@sha256:3e5756e2b6dad6a5f1f5deacf0a19f9eaa2581b76ae092b2d4a118162e1d4d4e    started=true

and the runner container, which started after dind's startup probe, printed
the daemon's version:

server 29.9.1

Air-gapped registry

mirror.gcr.io serves Google's cached copies of Docker Hub images, and for
this tag it returned the same index digest as Docker Hub. In a network
without that access, copy the image into your registry by digest:

oras cp -r mirror.gcr.io/library/docker:29.9.1-dind@sha256:3e5756e2b6dad6a5f1f5deacf0a19f9eaa2581b76ae092b2d4a118162e1d4d4e \
  registry.example.internal/library/docker:29.9.1-dind

oras cp copies the image index with every platform, so the copy has the
same digest, sha256:3e5756e2…, and the patch needs only the new registry
in newName, or in the mutation's value. In the test, oras 1.3.5 copied
the index to a local registry, which then served it with that digest and
the platforms linux/amd64, linux/arm64v8, linux/armv6 and
linux/armv7.

Originally published at kadet-hub.org.

Bitnami PostgreSQL chart alternatives: CloudNativePG and Kapitan

Moep90 — Tue, 27 Oct 2026 10:52:38 +0000

If you installed PostgreSQL, Redis or nginx from a Bitnami chart, the
packaged chart you pinned and the images it pulls no longer receive
updates, and the legacy images are meant only to buy time for a migration.
For PostgreSQL the move with the least friction is an operator its own
project maintains, CloudNativePG, together with images from that project.
Its custom resources can come from the upstream cluster chart or be
rendered in your own repository, for example with the
cloudnative-pg generator. Both work. Neither
replaces Bitnami one to one, because a chart or a generator only produces
manifests; the images still need a source you trust. For the other
Bitnami charts, Bitnami chart alternatives
lists the replacement chart by chart.

What Broadcom changed, and when

Bitnami announced the change in
bitnami/containers#83267,
opened on 16 July 2025 under the title "Upcoming changes to the Bitnami
catalog (effective August 28th, 2025)". The issue states that "since August
28th, we have not published new Bitnami container images or Helm charts to
Docker Hub in OCI format". The deletion of the public catalog
docker.io/bitnami was postponed to 29 September 2025, with three 24-hour
brownouts on 28 August, 2 September and 17 September.

Images and charts are affected differently:

Versioned image tags moved to docker.io/bitnamilegacy. That catalog "will receive no further updates or support and should only be used for temporary migration purposes". Tags based on debian-8, debian-9, debian-10, centos-7 and ol-7 were not copied.
docker.io/bitnami keeps a limited set of hardened images that "are intended for development and are only available on the 'latest' tag". Versioned, supported images are part of the paid Bitnami Secure Images offering.
Packaged charts "will remain available at docker.io/bitnamicharts as OCI artifacts, but they will no longer receive updates". The FAQ adds that deploying them "will not work out-of-the-box unless you override the bundled images with valid ones".
The chart source at bitnami/charts stays available under Apache 2.0.

The documented stopgap is to keep the chart version and point every image
repository at bitnamilegacy, for PostgreSQL with
--set image.repository=bitnamilegacy/postgresql and the same for
os-shell and postgres-exporter, plus
global.security.allowInsecureImages=true. That keeps a release running. It
does not get you a patched PostgreSQL.

On 24 September 2025 the CNCF
stated
that "these changes do not impact the Helm project itself" and that
"Bitnami Helm Charts are a specific vendor-maintained implementation". It
advised users to "begin exploring migration or mirroring strategies". Helm
itself stays a CNCF graduated project under Apache 2.0.

What the PostgreSQL chart looks like today

On 17 October 2026, the source of bitnami/postgresql in bitnami/charts is
at chart version 17.1.2, last changed on 8 September 2026. Its default image
is docker.io/bitnami/postgresql:17.6.0-debian-12-r4, and that tag does not
resolve. The same tag does resolve in docker.io/bitnamilegacy. The OCI
chart docker.io/bitnamicharts/postgresql:17.1.2 does not exist, while
16.7.0, the version from the announcement's example, still does. The source
keeps moving, but nothing published matches it, so consuming it means
packaging the chart and choosing images yourself.

The options

Option	Who maintains the manifests	Images	Cost of staying
Bitnami chart with `bitnamilegacy`	nobody, frozen	frozen, no fixes	grows with every CVE
Bitnami Secure Images	Broadcom	Broadcom	subscription
Upstream CloudNativePG charts	CloudNativePG project	`ghcr.io/cloudnative-pg/postgresql`	chart values to track
Operator plus rendered manifests	you, from a generator	`ghcr.io/cloudnative-pg/postgresql`	an inventory and a compile step

The last two share the operator. CloudNativePG 1.30.1 was released on 23
September 2026 as cnpg-1.30.1.yaml and as the chart cloudnative-pg
0.29.1. Moving to it is a platform change, not a chart swap. The Bitnami
chart renders StatefulSets and Services directly; a CloudNativePG Cluster
is a resource the operator reconciles into instances, services and
failover. Existing data
moves with a logical import; the operator's
import documentation
describes it as pg_dump over the network followed by pg_restore.

For the Cluster itself the upstream project ships a cluster chart.
helm template of cluster 0.9.0 with instances=3 and
storage.size=10Gi produced the Cluster, a ConfigMap of monitoring queries and a helm test
Job. The image came out as ghcr.io/cloudnative-pg/postgresql:16, because
the chart's version.postgresql defaults to "16". If you go this way,
pin cluster.imageName.

Rendering the Cluster with a generator

The other way is to keep the manifests in your repository and render them
from an inventory. The example
does this with the cloudnative-pg generator 1.0.1, which is built for
CloudNativePG 1.30.1 and the Barman Cloud Plugin 0.15.1 (see
compatibility). The target maps the Bitnami defaults of
one database and one owner onto a three-instance cluster:

parameters:
  kapitan:
    dependencies:
      - type: oci
        source: ghcr.io/kadet-hub/cloudnative-pg:1.0.1@sha256:281417a421b0053070fbdc552f9032a3f8d564c0ca2047ec500842a0d060a8f7
        output_path: lib/cloudnative-pg
    compile:
      - output_path: manifests/
        input_type: kadet
        output_type: yml
        input_paths: [lib/cloudnative-pg]
  namespace: databases
  generators:
    cloudnative-pg:
      clusters:
        app-db:
          image_name: ghcr.io/cloudnative-pg/postgresql:18.6-system-trixie@sha256:5efb66993894055f0aa50bd94337c6c4ff2e2129f4d433c527151aebbf283fb0
          storage: 10Gi
          bootstrap:
            initdb: {database: app, owner: app}

The generator fills instances: 3 and turns storage: 10Gi into
storage.size. The operator creates the app database, the app role and
a Secret app-db-app with the credentials and a connection URI, which
replaces the password Secret the Bitnami chart rendered.

Kapitan 0.36.3 checks neither digests nor attestations when it fetches, so
kapitan compile --fetch gives you no integrity check. It also needs the
kapitan[oci] extra for OCI sources; without it the compile stops with
"The 'oras' package is required for OCI dependencies". The example
therefore verifies first and fetches itself:

id=https://github.com/kadet-hub/registry/.github/workflows/release.yml@refs/heads/main
gh attestation verify "oci://ghcr.io/kadet-hub/index@$(oras resolve ghcr.io/kadet-hub/index:latest)" \
  --owner kadet-hub --cert-identity "$id"
gh attestation verify oci://ghcr.io/kadet-hub/cloudnative-pg@sha256:281417a4... \
  --owner kadet-hub --cert-identity "$id"
oras pull -o .oci ghcr.io/kadet-hub/cloudnative-pg@sha256:281417a4...
tar -xf .oci/cloudnative-pg.tar -C lib/cloudnative-pg
uvx --python 3.13 --from 'kapitan[omegaconf,reclass-rs]==0.36.3' \
  --with omegaconf==2.4.0.dev4 kapitan compile --inventory-backend omegaconf

Between the two verify calls, run.sh checks that the attested index
lists that digest as version 1.0.1 and not yanked. The omegaconf pin is not
optional: without it the resolver installed a newer omegaconf and the
compile failed with cannot import name 'ListMergeMode'.

The compile produced one file, compiled/app-db/manifests/databases/app-db-cluster.yml.
Plain kubeconform -strict reports "could not find schema for Cluster",
so the generator's shipped CRD schemas go in as the schema location:

kubeconform -strict -summary \
  -schema-location 'lib/cloudnative-pg/schemas/{{.Group}}/{{.ResourceKind}}_{{.ResourceAPIVersion}}.json' \
  compiled/
# Summary: 1 resource found in 1 file - Valid: 1, Invalid: 0, Errors: 0, Skipped: 0

The schemas are closed, so a value carried over from Bitnami habits fails.
Adding replicaCount: 2 under spec gives
"additional properties 'replicaCount' not allowed". A schema check is not
the operator's admission webhook; Validate Kubernetes custom resources
against their CRDs covers what it
misses.

Trade-offs

A generator renders manifests, not images. In this example the images come
from the CloudNativePG project, pinned by digest, and patching them is now
your job: bump the digest, recompile, review the diff. Renovate can propose
the generator updates with the
kadet-hub preset, but not
the PostgreSQL image inside your inventory.

The generator does not install the operator or the Barman Cloud Plugin, and
it is built for one CloudNativePG line. When you upgrade the operator, pick
a generator version built for it.

Kapitan is one more tool to learn. For a single database the upstream
cluster chart with a pinned image is less to set up. Rendering pays off
when the same pattern repeats across clusters and environments, and when
you want the rendered YAML in review instead of in a release. The
generator, Helm chart or jinja2 guide
compares the inputs.

A local compile runs the generator's Python with your environment and
credentials. The consumer workflow described in
Kapitan generators as supply-chain artifacts
runs the same compile in a sandbox without network.

Originally published at kadet-hub.org.

cdk8s vs Tanka vs Kapitan for Kubernetes config

Moep90 — Tue, 27 Oct 2026 10:52:28 +0000

Pick cdk8s if your team writes TypeScript and wants the compiler to reject a
bad value before anything renders. Pick Tanka if you want a small, fast,
Kubernetes-only toolchain and accept Jsonnet as the language. Pick Kapitan if
one inventory has to drive Kubernetes and Terraform and you would rather keep
values in YAML than maintain template code. All three rendered the same app to
identical manifests and passed the derived-value test that Helm and Kustomize
failed in the companion post.

The example
builds the app with each tool, and ./check exits 0 only when the normalized
manifests match per environment. Versions, commands and raw output behind
every number are in its NOTES.md.

The app and the check

The spec is the companion's: namespace shop, apps web and api with a
Deployment, Service and ConfigMap each, restricted Pod Security, dev and prod,
PodDisruptionBudgets in prod only, and an api ConfigMap entry web_url built
from the web Service port.

Tanka 0.39.4, jsonnet-bundler 0.6.0, k8s-libsonnet 1.36.
cdk8s 2.70.111 in TypeScript 5.9.3, using the generated low-level Kube* classes for Kubernetes 1.36.
Kapitan 0.36.3 with the kadet-hub kubernetes generator 1.2.0 pinned by digest, as in the companion minus endpoints.json.

check sorts objects and keys and drops one label,
app.kubernetes.io/managed-by, which only the Kapitan generator sets and
which names the tool. Nothing else is ignored. Changing one replica count in a
copy makes it exit 1.

$ ./check
dev: tanka = kapitan (6 objects)
dev: cdk8s = kapitan (6 objects)
prod: tanka = kapitan (8 objects)
prod: cdk8s = kapitan (8 objects)

How much you write

Hand-written files only; lock files, fetched libraries and generated code are
listed separately.

	web only	both apps	fetched or generated
Tanka	8 files, 138 lines	8 files, 167 lines	k8s-libsonnet, 30037 lines
cdk8s	5 files, 148 lines	5 files, 159 lines	`imports/k8s.ts`, 41214 lines, committed
Kapitan	5 files, 65 lines	6 files, 101 lines	generator, 4166 lines Python

For the Helm vs Kustomize vs Jsonnet comparison: Helm needed 156 lines and
Kustomize 299 for both apps in the companion.

The gap is about who owns the template. Tanka and cdk8s ship building blocks,
not an opinion on what a Deployment looks like, so the example carries its
own: lib/shop/app.libsonnet (58 lines) and app.ts (87 lines). Both spell
out fields Kubernetes would default, such as probe thresholds and
defaultMode: 420, to match the generator. In Kapitan the template is the
generator, maintained elsewhere and pinned by digest.

cdk8s has an opinionated layer too, cdk8s+.
An 11-line cdk8s-plus-36 program for the web Deployment produced a
cdk8s.io/metadata.addr selector, 1000m/512Mi requests, 1500m/2048Mi limits,
a read-only root filesystem and a TCP startup probe. Reasonable defaults, but
not the spec's, so the example stays on the low-level classes.

Adding the second app

	files changed	lines added	lines removed
Tanka	3	29	0
cdk8s	1	16	5
Kapitan	3	38	2

All three reuse their template, so api is configuration only. cdk8s is
smallest partly through layout: its per-environment values are one object
literal per line, the Jsonnet and YAML files use one line per field. Helm needed 25 lines here and Kustomize 149.

A value derived from another

All three compute web_url from the web Service port. In a copy I changed
only that port to 8081 in each tool. check still passed, and all three
rendered web_url: http://web.shop.svc:8081. In the same test Helm and
Kustomize rendered the stale :80 without an error. For Tanka vs Helm and
cdk8s vs Helm this is the main difference: a language lets one value reference
another, separate Helm values files cannot.

The mechanisms differ. Jsonnet binds $._config late, so an environment that
overrides the web port also changes the api URL. In cdk8s it is an ordinary
variable, so the dependency follows your code structure. Kapitan interpolates
${...} across the whole inventory, so any class or target can reference any
value in another file.

Typos and type safety

I misspelled pdb_min_available as pdb_min_availabe in the prod web values
of each tool:

	result
Tanka	renders, exit 0, web PodDisruptionBudget missing
cdk8s	synth fails: `error TS2561: ... 'pdbMinAvailabe' does not exist in type 'Sizing'. Did you mean to write 'pdbMinAvailable'?`
Kapitan	compiles, exit 0, `web-scaling.yaml` missing

cdk8s catches it because the example declares an interface for its values,
and the generated imports type every Kubernetes field for the editor. Jsonnet
has no types: reading a missing field fails, adding a misspelled one does not,
unless you write asserts. jsonnet-language-server
adds completion. Kapitan's inventory is YAML, and neither Kapitan nor the
generator rejected the unknown key. An output diff in CI, like check, caught
it in all three.

Toolchain and speed

Install is one run with cold caches and depends on the network. Node.js 26 and
Python 3.13 were already present and are not counted. Render covers both
environments, five warm runs.

	install	on disk	render
Tanka	12.1 s for `tk` and `jb`, 3.4 s `jb install`	29 MB + 7.3 MB	0.10 to 0.11 s
cdk8s	4.2 s `npm ci`, 173 packages	95 MB	3.46 to 3.59 s
Kapitan	63.1 s `uv run`, 89 packages; 13.3 s generator verify and pull	285 MB + 6.2 MB	1.61 to 1.67 s

Tanka is two static Go binaries and by far the fastest. Much of the cdk8s
time is ts-node type-checking the 41214-line import file: transpile-only, the
same synth took 2.27 to 2.36 s. Kapitan reports 0.68 s for the compile, the
rest is uv run and Python startup. Its install is large because the base
package depends on cloud SDKs for its secret backends:
google-api-python-client alone is 112 of the 267 MB installed.

Reuse and packages

Tanka libraries are Git repositories that
jsonnet-bundler pulls
and locks by commit, with no central registry. jsonnet-bundler had no commits in the
last 12 months and its latest release, v0.6.0, dates from August 2024.

cdk8s uses npm or the package manager of its other languages, and
Construct Hub indexes published constructs.

The kadet-hub catalog lists 36 Kapitan generators, each an OCI artifact
with a build attestation. The registry is young: created on 2026-10-08, one
human committer so far.

All three consume Helm charts: Tanka through
tanka-util, cdk8s through cdk8s import helm:
and its Helm construct, Kapitan through its helm input type.

Kapitan and Jsonnet

Kapitan has a jsonnet input type, so the choice is less exclusive than it
looks. I pointed a Kapitan target at the unchanged Tanka library and the
vendored k8s-libsonnet, with an 11-line main.jsonnet that reads the prod
values from the inventory. After the same normalization the output matched
tk show environments/prod, all 8 objects. A team with a Jsonnet codebase
can keep it and add inventory, refs and non-Kubernetes outputs, giving up
tk diff and tk apply against a cluster.

Output beyond Kubernetes

Tanka and cdk8s render Kubernetes objects. Plain Jsonnet can emit
Terraform's JSON syntax,
outside Tanka. CDKTF, the Terraform counterpart of cdk8s, was
archived on 10 December 2025
with no further updates. Kapitan compiles any output from one inventory, and
kadet-hub lists Terraform generators for AWS, AzureRM, Google Cloud and Open
Telekom Cloud, all at 0.1.x.

GitOps

Argo CD renders Helm, Kustomize and plain
Jsonnet
directories natively, Flux renders Helm and Kustomize. None of the three tools
is native to either. A Tanka environment does not fit Argo CD's Jsonnet support:
tk eval returns nested objects (apps.web.deployment) that Tanka flattens,
and the namespace comes from spec.json.

The usual routes are an Argo CD
config management plugin
that runs the tool in the repo server, or rendering in CI and committing the
YAML or pushing it with
flux push artifact. Tanka
has tk export for that, and Kapitan always writes files. cdk8s has one more
option: cdk8s synth --format helm writes a chart with Chart.yaml and
templates, which both Argo CD and Flux deploy natively.

Project activity

Read-only GitHub API on 2026-10-17, window from 2025-10-17. Human commits
exclude bot and automation accounts.

repository	stars	commits	human commits (authors)	releases	latest
grafana/tanka	2690	500	41 (14)	17	v0.39.4, 2026-10-08
cdk8s-team/cdk8s	4859	499	7 (4)	0	umbrella repo
cdk8s-team/cdk8s-core	87	578	0	90	v2.70.111, 2026-10-12
cdk8s-team/cdk8s-cli	41	479	8 (4)	125	v2.207.67, 2026-10-13
cdk8s-team/cdk8s-plus	163	635	8 (6)	330	cdk8s-plus-37 v2.0.0, 2026-10-16
kapicorp/kapitan	1931	135	93 (11)	6	v0.36.3, 2026-07-03
kadet-hub/registry	1	716	711 (1)	5	index-47, 2026-10-16

441 of Tanka's 500 commits are Renovate updates. All 578 cdk8s-core commits
were automated dependency upgrades; its frequent releases carry those.
Kapitan releases less often and has the most human commits of the three tools.

Running it

You need tk and jb, Node.js with npm, and uv, plus oras and a logged-in
gh for the first generator fetch:

$ git clone https://github.com/kadet-hub/examples
$ cd examples/config-tools
$ ./check

The first run installs dependencies from the lock files and verifies the
generator's attestation before pulling it by digest; the
companion post covers what that fetch does not check.

Originally published at kadet-hub.org.

cosign verify-attestation vs gh attestation verify

Moep90 — Mon, 26 Oct 2026 21:50:37 +0000

To verify an OCI artifact that carries GitHub artifact attestations, run
gh attestation verify or cosign verify-attestation with the exact
workflow identity. gh reads the Sigstore bundles from GitHub's attestation
API, which has them for every kadet-hub generator version. cosign reads
them from the registry, where the release workflow has pushed them since
17 October 2026. This post walks through both on a real generator, shows
the failures you want to see, and ends with a script that does it in one
call. Why a generator needs this at all is covered in
supply-chain security for Kapitan generators.

Everything below ran on 2026-10-19 with gh 2.102.0, cosign v3.1.3 (sha256
4629c757…7f71 against the release's cosign_checksums.txt), jq 1.8.2 and
oras 1.3.4.

What kadet-hub attests

The release workflow pushes each generator
as an OCI artifact to ghcr.io/kadet-hub/<name> and attests its digest
up to three times: SLSA provenance (https://slsa.dev/provenance/v1), an SPDX 2.3
SBOM (https://spdx.dev/Document/v2.3) and the diff of its fixture output
against the previous version (https://kadet-hub.org/output-diff/v1). It
also attests the index that lists every version with its digest, and
publishes that index as a GitHub release index-<serial> with a Sigstore
bundle next to it.

Each attestation of a digest also goes to ghcr.io as an OCI referrer of
that digest, which is where cosign looks.

All of them are signed by one identity:

https://github.com/kadet-hub/registry/.github/workflows/release.yml@refs/heads/main

A workflow on another branch of the same repository can still get a valid
signature, but its certificate names its own file and ref. Matching this
string exactly is what rejects it.

The worked example is mutate 0.1.0 from index 48, the first version
published with referrers. It is mutate's first version, so there is
nothing to diff against: the index records "base": null with
"error": "no earlier listed version", and the release attested
provenance and SBOM only.

Start from the signed index

The digest you pin should come from a verified index, not from a tag:

ID=https://github.com/kadet-hub/registry/.github/workflows/release.yml@refs/heads/main
gh release download index-48 --repo kadet-hub/registry --pattern 'index.json*'
gh attestation verify index.json --bundle index.json.sigstore.json \
  --owner kadet-hub --cert-identity "$ID"
jq -r '.serial, .generators.mutate["0.1.0"].digest' index.json

48
sha256:428e548bb10c6dda0105220c3a9f135d3a964a419303e30849a06f6d9e194a15

When stdout is not a terminal, gh attestation verify prints nothing on
success: 0 bytes on stdout and stderr, exit code 0. In CI the exit code is
your only signal unless you ask for JSON. On a TTY it prints a summary
that starts with the policy it enforced:

The following policy criteria will be enforced:
- Predicate type must match:............. https://slsa.dev/provenance/v1
- Source Repository Owner URI must match: https://github.com/kadet-hub
- Subject Alternative Name must match:... https://github.com/kadet-hub/registry/.github/workflows/release.yml@refs/heads/main
- OIDC Issuer must match:................ https://token.actions.githubusercontent.com

Also check yanked is null for the version, and that serial is not lower
than the last one you accepted. The serial is the rollback protection.

Verify the artifact with gh

D=sha256:428e548bb10c6dda0105220c3a9f135d3a964a419303e30849a06f6d9e194a15
gh attestation verify "oci://ghcr.io/kadet-hub/mutate@$D" \
  --owner kadet-hub --cert-identity "$ID" --format json \
  | jq -r '.[].verificationResult.statement.predicateType'

https://slsa.dev/provenance/v1

Only provenance, the predicate type of the policy above, unless you pass
--predicate-type (manual).
With the SPDX URI it also exits 0. With the output-diff URI it fails with
an HTTP 404 from the attestations API, since mutate 0.1.0 has none.
--bundle-from-oci reads the bundles from ghcr.io instead of the API and
prints the same result. To see every attestation for a digest, download
them:

gh attestation download "oci://ghcr.io/kadet-hub/mutate@$D" --owner kadet-hub
jq -r '.dsseEnvelope.payload | @base64d | fromjson | .predicateType' "$D.jsonl"

https://spdx.dev/Document/v2.3
https://slsa.dev/provenance/v1

The file holds one Sigstore bundle (application/vnd.dev.sigstore.bundle.v0.3+json)
per line, each with a Rekor entry.

The same with cosign

The registry lists the same two bundles as referrers of the digest:

$ oras discover "ghcr.io/kadet-hub/mutate@$D"
ghcr.io/kadet-hub/mutate@sha256:428e548bb10c6dda0105220c3a9f135d3a964a419303e30849a06f6d9e194a15
└── application/vnd.dev.sigstore.bundle.v0.3+json
    ├── sha256:d197b25d45961fe128b5294f3d3741f941b77395195109b20e89d948a3db1afa
    │   └── [annotations]
    │       ├── org.opencontainers.image.created: "2026-10-19T16:59:12.130Z"
    │       ├── dev.sigstore.bundle.content: dsse-envelope
    │       └── dev.sigstore.bundle.predicateType: https://slsa.dev/provenance/v1
    └── sha256:b928bfd9c69cb04e9675212a21f6ef0bcff2b4a9420a8721a19a281f23957ae6
        └── [annotations]
            ├── org.opencontainers.image.created: "2026-10-19T16:59:18.390Z"
            ├── dev.sigstore.bundle.content: dsse-envelope
            └── dev.sigstore.bundle.predicateType: https://spdx.dev/Document/v2.3

cosign verifies them in place:

cosign verify-attestation --type slsaprovenance1 \
  --certificate-identity "$ID" \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  "ghcr.io/kadet-hub/mutate@$D" > provenance.json

Verification for ghcr.io/kadet-hub/mutate@sha256:428e548bb10c6dda0105220c3a9f135d3a964a419303e30849a06f6d9e194a15 --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - The code-signing certificate was verified using trusted certificate authority certificates

That summary goes to stderr. stdout gets the verified DSSE envelope:

jq -r '.payload | @base64d | fromjson | .predicateType, .subject[0].name' provenance.json

https://slsa.dev/provenance/v1
ghcr.io/kadet-hub/mutate

slsaprovenance1 is cosign's short name for the provenance URI. For the
SBOM, pass the URI itself, --type https://spdx.dev/Document/v2.3. The
short name spdxjson fails here, and the error lists what the digest
carries:

Error: none of the attestations matched the predicate type: spdxjson, found: https://slsa.dev/provenance/v1,https://spdx.dev/Document/v2.3

cosign verify with the same identity and issuer also exits 0. kadet-hub signs nothing
with cosign sign; cosign v3 accepts the two attestation bundles as
signatures and prints one entry per bundle with its predicate type as
critical.type. It checks no predicate type, so use verify-attestation
with --type.

Versions published before 17 October 2026 have no referrers: on
kubernetes 1.2.0, oras discover lists none and cosign verify-attestation
fails with no matching attestations:. For those, cosign verifies the
bundles that gh downloads, one bundle per file. The approach is the one
from the Sigstore blog
(cosign v2.4.0), with two changes v3.1.3 needed here. On mutate's
downloaded bundles it gives the same answer as the registry:

sed -n 2p "$D.jsonl" > provenance.sigstore.json   # the line with slsa.dev/provenance/v1
cosign verify-blob-attestation --bundle provenance.sigstore.json \
  --certificate-identity "$ID" \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  --type https://slsa.dev/provenance/v1 \
  --digest "${D#sha256:}" --digestAlg sha256

Verified OK

Passing the whole two-bundle .jsonl fails with invalid character '{' after top-level value. --type must match the predicate URI: the default
custom fails with invalid predicate type, expected custom got https://slsa.dev/provenance/v1. Instead of --digest, you can pass the
manifest itself as the blob, since its sha256 is the digest:
oras manifest fetch "ghcr.io/kadet-hub/mutate@$D" > manifest.json.
The index works directly with its release bundle:
cosign verify-blob-attestation --bundle index.json.sigstore.json … --type slsaprovenance1 index.json.
The index.json.intoto.jsonl asset does not, because it carries no
certificate.

cosign fetched the Sigstore public good trusted root through TUF.
gh attestation trusted-root prints two roots: public good, and GitHub's
own instance, which GitHub uses for private repositories
(docs).
Passing the first as --trusted-root to cosign verify-attestation
verifies. The second fails with not enough verified log entries from transparency log: 0 < 1, since it has no Rekor log.

cosign also checks individual GitHub claims, for example
--certificate-github-workflow-repository kadet-hub/registry --certificate-github-workflow-ref refs/heads/main. Do not pin
--certificate-github-workflow-trigger push: kadet-hub's release workflow
also runs on a schedule and on manual dispatch.

Failures you want to see

Case	gh	cosign
identity `@refs/heads/dev`	`verifying with issuer "sigstore.dev"`	`expected SAN value ".../release.yml@refs/heads/dev", got ".../release.yml@refs/heads/main"`
index with `serial` changed	`verifying with issuer "sigstore.dev"`	`provided artifact digests do not match digests in statement`
another generator's digest under `ghcr.io/kadet-hub/mutate`	`MANIFEST_UNKNOWN` from ghcr.io	`no matching attestations:`
repository `kadet-hub/generators`, gh with `--bundle-from-oci`	`expected SourceRepositoryURI to be https://github.com/kadet-hub/generators, got https://github.com/kadet-hub/registry`	`expected GithubWorkflowRepository to be "kadet-hub/generators", got "kadet-hub/registry"`
issuer `https://accounts.google.com`	`expected Issuer to be https://accounts.google.com, got https://token.actions.githubusercontent.com`	`expected issuer value "https://accounts.google.com", got "https://token.actions.githubusercontent.com"`

All exit 1. gh's message for a wrong identity or a tampered file is the
same, so it tells you that verification failed, not why. Without
--bundle-from-oci, --repo kadet-hub/generators fails earlier with an
HTTP 404 from the attestations API. cosign finds no referrers for another
generator's digest; with mutate's downloaded bundle and that digest,
verify-blob-attestation reports provided artifact digest does not match any digest in statement.

One case passes that should worry you: --certificate-identity-regexp '^https://github.com/kadet-hub/' verifies and exits 0. That pattern
accepts any workflow on any branch in the organization. Use the exact
identity.

Read the provenance

Verification proves who signed. The predicate tells you what ran.
Abridged, from jq '.payload | @base64d | fromjson | .predicate' provenance.json:

"externalParameters": {"workflow": {"ref": "refs/heads/main",
  "repository": "https://github.com/kadet-hub/registry", "path": ".github/workflows/release.yml"}},
"internalParameters": {"github": {"event_name": "push", "runner_environment": "github-hosted"}},
"resolvedDependencies": [{"uri": "git+https://github.com/kadet-hub/registry@refs/heads/main",
  "digest": {"gitCommit": "c043c5ab78befe28cfab9a0698b5735cb39799fe"}}],
"runDetails": {"builder": {"id": "https://github.com/kadet-hub/registry/.github/workflows/release.yml@refs/heads/main"},
  "metadata": {"invocationId": "https://github.com/kadet-hub/registry/actions/runs/39288368802/attempts/1"}}

Field	Check
`externalParameters.workflow`	path, ref and repository match the identity you trust
`resolvedDependencies[].digest.gitCommit`	the registry commit; read the workflow at that commit if you audit it
`internalParameters.github.event_name`	what started the run (`push` here); the release workflow also runs on `schedule` and `workflow_dispatch`, so record it rather than require one value
`runner_environment`	`github-hosted`; a self-hosted runner is a different trust boundary
`invocationId`	the run, with its logs; the index attestation of serial 48 names the same run

The commit here is the registry's, the merge that listed mutate, not the
generator's. The generator source (925525b… in kadet-hub/generators)
appears in the index and in the SBOM's downloadLocation,
git+https://github.com/kadet-hub/generators.git@925525b2fabb1c2fb33c00f1b00b0d2495edce06#mutate.
Those are statements by the same signed workflow, not something Sigstore
checks.

One script

verify-oci/verify
needs gh, jq and sha256sum. It downloads the latest index release,
verifies it, enforces a minimum serial, rejects yanked versions, verifies
provenance, SBOM and output diff for the digest, checks the diff hash
against the index, and confirms that a tampered index, another branch,
another repository and another workflow are rejected. Any failure exits
non-zero.

$ ./verify mutate 0.1.0 48
index.json: https://slsa.dev/provenance/v1
serial: 48
mutate 0.1.0: sha256:428e548bb10c6dda0105220c3a9f135d3a964a419303e30849a06f6d9e194a15
https://slsa.dev/provenance/v1: https://github.com/kadet-hub/registry/actions/runs/39288368802/attempts/1
https://spdx.dev/Document/v2.3: https://github.com/kadet-hub/registry/actions/runs/39288368802/attempts/1
rejected as expected: tampered index
rejected as expected: branch other than main
rejected as expected: wrong source repository
rejected as expected: workflow other than release.yml
OK: mutate 0.1.0 verified against index 48

The output diff is checked only when the index records one, which is why
mutate 0.1.0 stops at two predicates. Versions published before the SBOM
existed, such as keda 1.0.0, are checked for provenance only.

Originally published at kadet-hub.org.

gVisor in CI: sandboxing untrusted code on GitHub Actions

Moep90 — Mon, 26 Oct 2026 21:50:28 +0000

If your CI executes code that contributors send you, a plain container is not
a boundary. Run it under gVisor with the network off and a read-only root
filesystem, keep a syscall trace, and treat everything else (timeouts, what
you mount in, your tokens) as separate problems. This is what the
kadet-hub gate does to compile contributor-supplied Kapitan generators,
and the pieces transfer to any project that runs plugins, formatters, test
code or build scripts from a pull request.

The problem

A pull request that adds a generator ships Python that the gate has to run to
see what it produces. The same shape shows up anywhere CI runs contributor
code: a linter plugin, a pytest suite, a Terraform provider, a code
formatter. Once that code runs, in-language guardrails do not hold, because
the attacker writes the code and picks the API.

Three defenses that look like confinement are not. Each of these runs with
host Python 3.13 (pinned through uv), python -I:

A Python audit hook that blocks subprocess:

blocked: subprocess blocked by audit hook
via fork_exec

The hook rejects subprocess.run, then _posixsubprocess.fork_exec spawns
the process directly. fork_exec never raises the subprocess.Popen audit
event, so the hook does not see it. CPython's own subprocess is the only
thing that raises that event.

Blocking DNS, against a raw IP (192.0.2.1 is TEST-NET-1):

name lookup blocked: DNS blocked
raw-IP connect reached network, result: TimeoutError timed out

getaddrinfo is patched to refuse, and the connect() to a literal address
leaves the host anyway. Name resolution was never the exfiltration path.

An rlimit and a private HOME:

HOME = $HOME
RLIMIT_NOFILE = (1048576, 1048576)
read /etc/passwd -> root:x:0:0:Super User:/root:/bin/bash

Pointing HOME at a scratch directory and capping file descriptors does
nothing to stop a read of a path outside HOME. Resource limits bound
resource use, not what the filesystem exposes.

What gVisor changes

gVisor (runsc) is a userspace kernel. The sandboxed process makes syscalls
to gVisor, not to the host kernel, so the host attack surface the process can
reach is small. Two options close the holes above. The runs below use the
same scripts inside runsc --rootless --network=none run, in an OCI bundle
whose root filesystem is python:3.13-slim exported read-only, with no host
mounts.

With --network=none the sandbox has no interfaces. In this rootless
bundle setup the network namespace from runsc spec already cuts it off,
so the flag matters most where the namespace is shared:

name lookup blocked: DNS blocked
raw-IP connect reached network, result: OSError [Errno 101] Network is unreachable

The raw-IP connect now fails at once rather than reaching the network. For a
compile that needs no network, this is the single highest-value flag.

A read-only minimal rootfs with no host mounts:

HOME = /root
RLIMIT_NOFILE = (1024, 1024)
read /etc/passwd -> root:x:0:0:root:/root:/bin/bash

The read resolves inside the container's own filesystem (note the root
GECOS field, where the host showed Super User). Host paths are gone: a probe
in the same bundle gets FileNotFoundError for the host home and for
host scratch directories, while a file bind-mounted at /mnt/secret is still
readable. That last point matters: gVisor confines the filesystem to what you
give it, so anything you mount in is in scope.

This is the kadet-hub sandbox/run shape. The container runs
--network none --read-only --user 10001:10001 --cap-drop ALL --security-opt no-new-privileges with /tmp and /out as noexec tmpfs,
the project mounted read-only, and no environment inherited from the runner.

The one gVisor does not stop, and the trace that does

gVisor is not an audit-hook sandbox. The fork_exec bypass runs inside it
unchanged: the process is allowed to start a child. What catches it is the
syscall trace. Run the same script under runsc --strace --strace-syscalls=execve,... --debug-log=... and the spawn is in the log:

python3 E execve(... /bin/echo, [... "/bin/echo", "via", "fork_exec"], ...)

The execve the audit hook never observed is right there, because gVisor is the
kernel the process talks to and writes the trace outside the sandbox, where
the code being judged cannot disable or forge it, and it follows child
processes. The gate installs the runsc-trace runtime and fails a compile
whose trace shows any execve other than the interpreter, Kapitan's known
probes and the declared binaries, any outbound connect, sendto or
sendmsg to an AF_INET address, a read of a decoy file, and a few other
syscalls. That rule set is SEC-10 in the gate spec. The design reasoning is
in how kadet-hub works.

What gVisor does not cover

gVisor is a kernel boundary, not a policy. Four things stay your job.

CPU and wall-clock exhaustion. A sandboxed process can spin or allocate
forever. gVisor does not time it out; you set a job timeout and a memory
bound. The gate runs each compile under a step timeout and a tmpfs size cap.

Data you mount in. Everything inside the rootfs and every bind mount is
reachable. Mount the contributor's input read-only, mount nothing else, and
never mount a checkout that carries secrets or a writable host path.

Your CI token. gVisor does not hide GITHUB_TOKEN or other secrets that the
workflow puts in the environment or in a mounted file. The sandbox run should
inherit no environment from the runner, which is why sandbox/run uses an
explicit --env-file and the gate passes KUBECONFIG=/dev/null and an empty
HELM_PLUGINS.

The workflow trigger. Run untrusted pull requests under pull_request, never
pull_request_target. pull_request_target runs with the base repository's
secrets and a writable token, so a sandbox around the code does not help if
the workflow itself hands it the keys. GitHub caps the permissions key for
fork pull requests under pull_request; set permissions: {} anyway and add
back only what a step needs.

Cost

Measured on one workstation, five runs each, same slim rootfs. A trivial
python -c pass takes about 0.02 s native and about 0.12 s under gVisor; a
bare sandbox boot (runsc do /bin/true) is about 0.07 s. So gVisor adds
roughly 0.07 to 0.10 s of startup per run plus per-syscall overhead. For a CI
compile that already takes seconds, the sandbox is not the bottleneck.

gVisor vs a plain Docker container (runc) {#gvisor-vs-docker}

The runs so far are unconfined and under rootless runsc. To put gVisor
directly against a plain container, the same three scripts also run under
docker run with Docker's default runtime (runc) and under docker run --runtime=runsc, every other option equal, so only the runtime changes.
The script is piped in on stdin and nothing else is mounted. These were
measured on the example repository's CI, one ubuntu-24.04 runner, in
run 39571252377.

Swapping runc for runsc changes nothing the three scripts can show. The
fork_exec child runs under both, because neither runtime is an audit-hook
sandbox. The raw-IP connect reaches the network and times out at
192.0.2.1 under both, because the default Docker network routes out either
way; gVisor blocks it only with --network=none, as above. The read of
/etc/passwd returns the image's own file under both, because the image,
not the runtime, is the filesystem the process sees.

What differs is the kernel the process talks to:

runc:  Linux 6.17.0-1022-azure
runsc: Linux 4.19.0-gvisor

Under runc the sandboxed code's syscalls reach the runner's own kernel, so
a bug in that kernel is reachable from the code being judged. Under runsc
they reach gVisor's userspace kernel, and the host kernel surface the code
can touch is small. That is what a plain container does not give you: the
network and the spawn are bounded by flags and the trace, which you set on
runc too, while the runtime is what moves the host kernel out of reach.

Cost of the swap on the same runner, five runs each: a python -c pass in
the container is about 0.20 s under runc and about 0.27 s under runsc, so
runsc adds roughly 0.07 s per run. The absolute figures are higher than the
rootless runsc ... do numbers above because each docker run starts a
container through the daemon, which dominates both runtimes.

This measures gVisor against Docker's default runtime. It does not answer
"gVisor vs Docker sandbox", Docker's Sandboxes product, or gVisor against
microVM runtimes such as Firecracker or Kata, which would need microVM
measurements this example does not take.

Decoy credentials

Detection beats hope for the untargeted credential stealer. The gate seeds the
sandbox HOME with decoy ~/.aws/credentials, ~/.kube/config,
~/.ssh/id_ed25519 and a ~/.gnupg key, plus decoy AWS_SECRET_ACCESS_KEY,
GITHUB_TOKEN and VAULT_TOKEN in the environment, each random per run. Two
checks fire: the trace fails the run on any open of a decoy file, and the
output is scanned for any decoy value in plain, base64 or hex form. A stealer
that scrapes HOME and writes what it found into the generated manifests
trips both. Code written specifically against this gate will avoid the decoys;
the point is the untargeted collector, which is most of them.

A minimal GitHub Actions job

This installs runsc with a pinned release and a SHA-512 check (the digest
and release are what kadet-hub pins), then runs the example rootless. Actions
are pinned by commit SHA, permissions is empty, and there are no caches.

name: sandboxed-compile
on: pull_request
permissions: {}
jobs:
  run:
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          persist-credentials: false
      - name: Install gVisor
        env:
          GVISOR_RELEASE: "20261005"
          GVISOR_SHA512: 27fed731b432d6a39d908956623fd273979fda66b3ccf4b9c12413ea417c516e18a9dc3cced403fb28efe495d64dce4251f51cf8c4d9f4a72f9833bd96b6bb35
        run: |
          curl -fsSLo runsc.tar.zstd \
            "https://storage.googleapis.com/gvisor/releases/release/${GVISOR_RELEASE}/x86_64/gvisor.tar.zstd"
          echo "${GVISOR_SHA512}  runsc.tar.zstd" | sha512sum -c -
          # runsc starts its sidecars from gvisor-bin/ next to it: unpack everything
          sudo tar --zstd -xf runsc.tar.zstd -C /usr/local/bin
      - name: Allow unprivileged user namespaces
        # ubuntu-24.04 restricts them with AppArmor; rootless runsc fails
        # with "fork/exec /proc/self/exe: permission denied" otherwise
        run: sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0
      - name: Run contributor code sandboxed
        run: runsc --rootless --network=none --ignore-cgroups do python3 -I pr-code.py

The example repository runs this setup weekly on ubuntu-24.04. Its first
run failed without the sysctl step: Ubuntu 24.04 lets AppArmor restrict
unprivileged user namespaces, and rootless runsc needs them. The step only
changes the throwaway runner.

The gate itself goes further, running under Docker with the runsc-trace
runtime so it can judge the syscall trace, which needs daemon configuration
and root on the runner. The rootless runsc ... do form above needs neither
and is enough to get the network and filesystem boundary in a plain job.

The escape scripts and a run script that reproduces every output above are
in the example.
For how this fits the rest of the registry, see
kadet-hub generators as supply-chain artifacts.

Originally published at kadet-hub.org.

Helm vs Kustomize vs Kapitan on one Kubernetes app

Moep90 — Mon, 26 Oct 2026 11:28:52 +0000

Comparisons between Helm, Kustomize and Kapitan usually compare features.
This one compares output. The
example renders
the same small app with all three tools, and ./check fails unless the
manifests match for each environment. Every number below comes from that
directory.

The app

Two services in namespace shop: web and api. Each has a Deployment, a
ClusterIP Service and a ConfigMap mounted into the pod. Pods run with the
restricted Pod Security profile. The environments differ in these values:

	dev	prod
web image tag	1.5.0-rc.1	1.4.2
web replicas, requests	1, 50m/64Mi	3, 250m/256Mi
api image tag	2.0.0-rc.3	1.9.1
api replicas, requests	1, 50m/64Mi	2, 500m/512Mi
log level	debug	info
PodDisruptionBudget	none	web and api

The api reads the web URL from its ConfigMap, web_url: http://web.shop.svc:80. That value matters later.

The Helm side is one generic chart rendered once per app with a base and an
environment values file. The Kustomize side is a base per app and an overlay
per environment. The Kapitan side is an inventory that feeds the
kubernetes generator from kadet-hub, pinned by
digest:

dependencies:
  - type: oci
    source: ghcr.io/kadet-hub/kubernetes:1.2.0@sha256:52017626e77dd5bf7293ce465ff0c3dc7a61dae1ae8353a56a53ffebe1bfb67c
    output_path: lib/kubernetes

check normalizes all three outputs (one document per object, sorted, keys
sorted) and diffs them. It drops one label, app.kubernetes.io/managed-by,
because it names the tool. Everything else must match.

$ ./check
dev: helm = kapitan (6 objects)
dev: kustomize = kapitan (6 objects)
prod: helm = kapitan (8 objects)
prod: kustomize = kapitan (8 objects)

One app

I built web alone first and counted the files and lines each tool needs,
excluding rendered output and the fetched generator:

	files	lines
Helm	9	131
Kustomize	9	152
Kapitan	5	65

The numbers need context. The generator sets fields explicitly that
Kubernetes would default anyway: the rolling update strategy, every probe
threshold, defaultMode: 420, sessionAffinity: None. The Helm templates
and the Kustomize base spell them out so the output matches. A chart written
from scratch would be shorter.

The bigger point is who owns the template. Of Helm's 131 lines, 108 are the
chart. You write and maintain it. Kapitan's 65 lines are inventory only. The
templating lives in the generator, 4166 lines of Python that someone else
maintains and that you pin, verify and run. It is the same trade as pulling a
generic chart from a chart repository, with the difference that the generator
reads the whole inventory instead of one values tree.

The one-app prod target in Kapitan, inventory/targets/prod.yml:

classes: [common, apps.web]
parameters:
  web_tag: 1.4.2
  log_level: info
  components:
    web:
      replicas: 3
      resources:
        requests: {cpu: 250m, memory: 256Mi}
        limits: {memory: 512Mi}
      pdb_min_available: 2

Adding a second app

Then I added api with the same pattern and diffed against the one-app
version:

	files changed	lines added	lines removed
Helm	3	25	0
Kustomize	9	149	2
Kapitan	5	47	2

Helm wins this round. The chart is generic, so a new app is three values
files. Kapitan's 47 lines include 9 for the JSON output described below; the
api itself is 38 lines, mostly the component block in
classes/apps/api.yml. Kustomize has no parameters, so the usual move is
copying the base and renaming. You can share one base through namePrefix
and replacements, but labels, selectors, container names and mount paths
then become patches of their own.

A value derived from another

The api needs the web Service URL. In Kapitan it is an interpolation, so it
follows the web definition:

web_url:
  value: http://web.${namespace}.svc:${components:web:ports:http:service_port}

Helm releases cannot read each other's values, so the Helm side repeats the
literal. The Kustomize side does too:

$ grep -rn web_url helm kustomize kapitan/inventory
helm/values/api-prod.yaml:6:  web_url: http://web.shop.svc:80
helm/values/api-dev.yaml:6:  web_url: http://web.shop.svc:80
kustomize/overlays/prod/api.yaml:7:  web_url: http://web.shop.svc:80
kustomize/overlays/dev/api.yaml:7:  web_url: http://web.shop.svc:80
kapitan/inventory/classes/apps/api.yml:20:            web_url:

To show what that costs, I changed only the web Service port to 8081 in each
tool, in a copy of the example, and ran check:

--- /tmp/tmp.TpmPDYldGS/prod.kapitan
+++ /tmp/tmp.TpmPDYldGS/prod.helm
@@ -1,7 +1,7 @@
 apiVersion: v1
 data:
   log_level: info
-  web_url: http://web.shop.svc:8081
+  web_url: http://web.shop.svc:80

The same diff appears for Kustomize and for dev, four mismatches in total,
and check exits 1. Kapitan followed the change. In Helm and Kustomize the
stale URL renders without an error.

Both tools have a workaround. An umbrella chart with global values can
share an input between subcharts, but you restructure the values around it.
Kustomize replacements can copy the Service port into part of the string
with delimiter: ":" and index: 2. I tested that in a copy of the prod
overlay: six lines, and the URL followed. It works per overlay and per field,
and it breaks when the string format changes. The example keeps the
literal.

Output that is not Kubernetes

The same inventory also renders endpoints.json per environment, through a
six-line jinja2 template and a three-line compile entry:

{
"api": {"url": "http://api.shop.svc:80", "image": "ghcr.io/example/api:1.9.1"},
"web": {"url": "http://web.shop.svc:80", "image": "ghcr.io/example/web:1.4.2"}
}

That file can feed smoke tests, monitoring or docs. With the port change above
it says 8081 as well. Helm and Kustomize render Kubernetes objects only. The
equivalent there is a separate script that parses values files or rendered
YAML, and it is one more place to keep in sync. The same mechanism covers
Terraform: kadet-hub lists generators for AWS, AzureRM, Google Cloud and Open
Telekom Cloud that read the same inventory.

What Helm and Kustomize do better

Helm's ecosystem is the main reason to stay. Most vendors publish a chart,
Artifact Hub makes them searchable, and helm install, upgrade and rollback
keep release history in the cluster. Kapitan has no release concept. It
writes files, and you apply them with kubectl, Argo CD or Flux. For
third-party software, a vendor chart maintained by the vendor beats a
generator maintained by someone else. Kapitan can render charts through its
helm input type, so you do not have to give them up.

Kustomize ships inside kubectl. In this example kubectl kustomize and the
standalone binary produced identical output. Nothing to install, and patching
vendor YAML you do not control is where Kustomize is strongest. Kapitan has a
kustomize input type for that case too.

Both have a gentler start. A values file and a patch are plain YAML.
Kapitan asks you to learn classes, targets, interpolation and the generator's
schema before the first manifest, and it needs a Python toolchain. Argo CD and Flux render Helm and Kustomize natively. With Kapitan you
commit compiled output or add a compile step in CI.

Fetching generators honestly

Kapitan 0.36.3 checks neither attestations nor digests when it fetches
dependencies, so kapitan compile --fetch gives no integrity check. It also
runs generator code with your environment and credentials. check does not
use --fetch. On first run it verifies the generator's attestation with
gh attestation verify against the kadet-hub release workflow, pulls by
digest with oras and unpacks into kapitan/lib/kubernetes.

That is not the full consumer flow. It does not check the attested index
for yanked versions, and it compiles on your machine without a sandbox. In
CI, use the consumer
workflow,
which does both and compiles in gVisor without network. The
supply-chain guide
explains why.

Running it

You need helm, kustomize and uv, plus oras and a logged-in gh for the first fetch:

$ git clone https://github.com/kadet-hub/examples
$ cd examples/same-app
$ ./check

check runs Kapitan through uv run with
kapitan[omegaconf,reclass-rs]==0.36.3, the version the kadet-hub sandbox
pins. Versions and every command behind the numbers above are in NOTES.md.

If your stack is one or two apps built from vendor charts, Helm and
Kustomize are the better fit. Kapitan pays off once values start depending
on each other across apps, or the same facts need to reach Kubernetes,
Terraform and docs. To start, see Generator, Helm chart or
jinja2? and the
catalog.

Originally published at kadet-hub.org.

kube-prometheus-stack default Grafana dashboards, one by one

Moep90 — Mon, 26 Oct 2026 11:28:42 +0000

kube-prometheus-stack 92.11.0, the release of 20 October 2026, renders 29
Grafana dashboards by default, each as a ConfigMap that Grafana's sidecar
loads. grafana.defaultDashboardsEnabled turns all of them off at once,
and the switches for scraped components remove some of them along with
the scraping. No value turns off a single dashboard. This post shows which
dashboard depends on which switch and removes single dashboards from the
rendered manifests, first with Kustomize over helm template, then with
Kapitan's mutate
generator.

Each patch was rendered from the 92.11.0 chart with Helm 4.3.1, Kustomize
5.8.3, and Kapitan 0.36.3 with mutate 0.1.0, and the result was installed
on a kind cluster on Kubernetes 1.37.0. The guide
Patch a Helm chart compares the places such a
patch can live.

Disable single default dashboards {#disable-default-dashboards}

prometheus-community/helm-charts#3763,
opened in September 2023 and at 23 thumbs up, asks for a switch per default
dashboard. The reporter's workaround, overwriting the ConfigMaps by hand,
lasts until the next deploy. The one maintainer reply, from jkroepke in
October 2025, is "just open a PR to solve this." Pull request #7015, opened
in June 2026, adds a map grafana.defaultDashboardsDisabled. It has no
review and conflicts with the current chart.

Every dashboard template starts with the same condition, here from
grafana-overview.yaml:

{{- if and (or .Values.grafana.enabled .Values.grafana.forceDeployDashboards) (semverCompare ">=1.14.0-0" $kubeTargetVersion) (semverCompare "<9.9.9-9" $kubeTargetVersion) .Values.grafana.defaultDashboardsEnabled }}
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: {{ template "kube-prometheus-stack-grafana.namespace" . }}
  name: {{ printf "%s-%s" (include "kube-prometheus-stack.fullname" $) "grafana-overview" | trunc 63 | trimSuffix "-" }}

Some templates add a component switch to that condition. Sixteen add
nothing, so defaultDashboardsEnabled is their only switch:
cluster-total, grafana-overview, the eight k8s-resources dashboards from
cluster to workloads-namespace, namespace-by-pod, namespace-by-workload,
persistentvolumesusage, pod-total, prometheus and workload-total.

Which dashboard hangs on which switch

Setting a component's enabled to false removes its dashboard and also
what the chart renders to monitor that component. Rendered with each
switch set to false in turn, compared with the default render:

Switch set to false	Dashboard ConfigMaps removed	Also removed
`kubeApiServer.enabled`	apiserver	ServiceMonitor, 4 PrometheusRules
`kubeControllerManager.enabled`	controller-manager	Service, ServiceMonitor, 1 PrometheusRule
`kubeScheduler.enabled`	scheduler	Service, ServiceMonitor, 2 PrometheusRules
`kubeEtcd.enabled`	etcd	Service, ServiceMonitor, 1 PrometheusRule
`kubeProxy.enabled`	proxy	Service, ServiceMonitor, 1 PrometheusRule
`coreDns.enabled`	k8s-coredns	Service, ServiceMonitor
`kubelet.enabled`	kubelet	ServiceMonitor, PrometheusRule kubelet.rules
`nodeExporter.enabled`	node-cluster-rsrc-use, node-rsrc-use, nodes, nodes-aix, nodes-darwin	the node-exporter DaemonSet, Service, ServiceAccount and ServiceMonitor
`alertmanager.enabled`	alertmanager-overview	Alertmanager, its Secret, Service, ServiceAccount and ServiceMonitor

Where a component is not scraped at all, as the control plane of many
managed clusters, the switch removes the dashboard together with the
monitoring objects that have nothing to scrape. To keep scraping and
alerting for a component and drop only its dashboard, or to drop one of
the sixteen dashboards without a switch, delete the ConfigMap from the
rendered output.
nodeExporter.forceDeployDashboards and
alertmanager.forceDeployDashboards keep the dashboards of a disabled
node-exporter or Alertmanager, and nodeExporter.operatingSystems.aix.enabled
and .darwin.enabled remove only nodes-aix and nodes-darwin.

Dashboard names

A dashboard ConfigMap is named <fullname>-<dashboard>, cut to 63
characters. The chart's kube-prometheus-stack.fullname starts from the
release name when that contains kube-prometheus-stack, and from
<release>-kube-prometheus-stack otherwise, and cuts either to 26
characters. Release kube-prometheus-stack therefore names a dashboard
kube-prometheus-stack-grafana-overview, and release monitoring, whose
monitoring-kube-prometheus-stack is cut to monitoring-kube-prometheus,
names it monitoring-kube-prometheus-grafana-overview. The 63-character
cut never applies to the default dashboards: the longest name, for a
fullname of the full 26 characters, has 60.

Kustomize over helm template

For release monitoring in namespace monitoring:

helm template monitoring oci://ghcr.io/prometheus-community/charts/kube-prometheus-stack \
  --version 92.11.0 --namespace monitoring --skip-tests > helm.yaml
kustomize build .

--skip-tests leaves out the Grafana subchart's test Pod, as Kapitan's
helm input does by default.

with this kustomization.yaml next to helm.yaml, which deletes the
dashboards Grafana Overview and Kubernetes / Compute Resources /
Multi-Cluster:

resources:
  - helm.yaml
patches:
  - patch: |-
      $patch: delete
      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: monitoring-kube-prometheus-grafana-overview
        namespace: monitoring
  - patch: |-
      $patch: delete
      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: monitoring-kube-prometheus-k8s-resources-multicluster
        namespace: monitoring

A strategic merge patch with $patch: delete removes the object it names.
Kustomize's output kept every other object of the render; it only dropped
metadata.annotations where the chart renders it empty or null. Over a
render for release kube-prometheus-stack, where the names differ, the
build fails instead of passing without effect:

Error: no resource matches strategic merge patch "ConfigMap.v1.[noGrp]/monitoring-kube-prometheus-grafana-overview.monitoring": no matches for Id ConfigMap.v1.[noGrp]/monitoring-kube-prometheus-grafana-overview.monitoring; failed to find unique target for patch ConfigMap.v1.[noGrp]/monitoring-kube-prometheus-grafana-overview.monitoring

Kapitan's mutate

The class pins the chart by version and by the
digest of its OCI artifact, and renders it as release
monitoring:

parameters:
  kapitan:
    dependencies:
      - type: helm
        source: oci://ghcr.io/prometheus-community/charts/kube-prometheus-stack@sha256:5b9b0215085f00f61c0fb2aa01ec402609d91c0a6a91bc84ca85cce2ed69e635
        chart_name: kube-prometheus-stack
        version: 92.11.0
        output_path: charts/kube-prometheus-stack
    compile:
      - input_type: helm
        input_paths: [charts/kube-prometheus-stack]
        output_path: kube-prometheus-stack
        helm_params:
          name: monitoring
          namespace: monitoring

The mutations delete the same two ConfigMaps:

parameters:
  mutations:
    no-grafana-overview-dashboard:
      match: {apiVersion: v1, kind: ConfigMap, name: monitoring-kube-prometheus-grafana-overview}
      delete: true
    no-multicluster-dashboard:
      match: {apiVersion: v1, kind: ConfigMap, name: monitoring-kube-prometheus-k8s-resources-multicluster}
      delete: true

The target adds mutate as a second compile entry on
the chart's output_path, as the guide shows. Each dashboard template is
a file of its own, so both files become a single comment line:

kube-prometheus-stack/templates/grafana/dashboards-1.14/grafana-overview.yaml: # all objects removed by mutation no-grafana-overview-dashboard
kube-prometheus-stack/templates/grafana/dashboards-1.14/k8s-resources-multicluster.yaml: # all objects removed by mutation no-multicluster-dashboard

The mutated render lost exactly the two ConfigMaps compared with the plain
one, and kept the same 27 dashboards as the Kustomize output. With the
class changed to release kube-prometheus-stack, the compile stops:

mutate: mutation no-grafana-overview-dashboard: expects 1+, matched 0

On kind

The mutated output went to a kind cluster on Kubernetes 1.37.0, the CRDs
first and everything with a server-side apply. Grafana, the operator,
kube-state-metrics, Prometheus, Alertmanager and node-exporter all became
ready, and the Kustomize output passed a server-side dry run against the
same cluster. The cluster held 27 dashboard ConfigMaps. Grafana's search
API listed 27 dashboards, with the same titles as those ConfigMaps:

Grafana lists 27 dashboards, the mutated render has 27
neither removed dashboard is listed: ["Grafana Overview","Kubernetes / Compute Resources /  Multi-Cluster"]

The ServiceMonitor monitoring-kube-prometheus-kube-controller-manager
stayed in place, so the controller manager is still scraped, without its
dashboard.

With the Grafana Operator

With grafana.operator.dashboardsConfigMapRefEnabled: true and
grafana.operator.matchLabels set, the chart also renders a
GrafanaDashboard per dashboard, with the same name as its ConfigMap and
a configMapRef to it. The patches above delete only the ConfigMaps and
would leave two GrafanaDashboards that reference ConfigMaps that no longer
exist. With the operator, the Kustomize patch deletes both kinds:

resources:
  - helm.yaml
patches:
  - patch: |-
      $patch: delete
      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: monitoring-kube-prometheus-grafana-overview
        namespace: monitoring
  - patch: |-
      $patch: delete
      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: monitoring-kube-prometheus-k8s-resources-multicluster
        namespace: monitoring
  - patch: |-
      $patch: delete
      apiVersion: grafana.integreatly.org/v1beta1
      kind: GrafanaDashboard
      metadata:
        name: monitoring-kube-prometheus-grafana-overview
        namespace: monitoring
  - patch: |-
      $patch: delete
      apiVersion: grafana.integreatly.org/v1beta1
      kind: GrafanaDashboard
      metadata:
        name: monitoring-kube-prometheus-k8s-resources-multicluster
        namespace: monitoring

and each mutation matches every kind by name and expects two objects:

parameters:
  mutations:
    no-grafana-overview-dashboard:
      match: {kind: "*", name: monitoring-kube-prometheus-grafana-overview}
      delete: true
      expects: 2
    no-multicluster-dashboard:
      match: {kind: "*", name: monitoring-kube-prometheus-k8s-resources-multicluster}
      delete: true
      expects: 2

Rendered with those two values, both removed exactly the two ConfigMaps
and the two GrafanaDashboards, and no remaining GrafanaDashboard refers to
a missing ConfigMap. The Grafana Operator itself was not installed, so
this variant is tested on the rendered output only.

Originally published at kadet-hub.org.

Kustomize delete resource: Kyverno PDBs and ESO webhooks

Moep90 — Sun, 25 Oct 2026 19:55:55 +0000

Some Helm charts render an object you need to own yourself and offer no
value to leave it out. This post covers two such cases from open issues:
the Kyverno chart creates a PodDisruptionBudget for every controller that
runs more than one replica, and the External Secrets chart renders its two
ValidatingWebhookConfigurations whenever it renders the webhook. In both,
the object is deleted from the rendered manifests, first with a Kustomize
patch over helm template, then with Kapitan's
mutate generator.

Each patch was rendered from the chart version named in its section with
Helm 4.3.1, Kustomize 5.8.3, and Kapitan 0.36.3 with mutate 0.1.0, and the
result was installed on a kind cluster on Kubernetes 1.37.0. The guide
Patch a Helm chart compares the places such a
patch can live.

Kyverno Helm chart: PodDisruptionBudgets with more than one replica {#kyverno-pdb}

kyverno/kyverno#11469,
opened in October 2024, reports that the chart's PodDisruptionBudgets cannot
be turned off. The reporter wants to supply PDBs of their own instead. The
maintainer realshuting
asked
the authors of the change that introduced the behaviour whether there was a
reason to keep PDBs on for more than one replica, and the question is
unanswered. In chart 3.9.2, published to the chart repository and to
ghcr.io on 21 October 2026, the PDB template of each of the four
controllers has the same condition, here from
admission-controller/poddisruptionbudget.yaml:

{{- if or .Values.admissionController.podDisruptionBudget.enabled (gt (int .Values.admissionController.replicas) 1) -}}

podDisruptionBudget.enabled: false is the default, and with more than one
replica the or ignores it. The chart's values.yaml says so: "Will
always be enabled if replicas > 1. This non-declarative behavior should
ideally be avoided, but changing it now would be breaking." With these
values the chart rendered four PDBs, and the same four with
podDisruptionBudget.enabled=false set for every controller:

admissionController:
  replicas: 3
backgroundController:
  replicas: 2
cleanupController:
  replicas: 2
reportsController:
  replicas: 2

All four carry the label app.kubernetes.io/part-of: kyverno. Its value is
the chart's fullname, so it follows the release name: release policy
gives policy-kyverno.

Kustomize over helm template

helm template kyverno oci://ghcr.io/kyverno/charts/kyverno --version 3.9.2 \
  --namespace kyverno -f kyverno-values.yaml > helm.yaml
kustomize build .

with this kustomization.yaml next to helm.yaml:

resources:
  - helm.yaml
patches:
  - target:
      kind: PodDisruptionBudget
      labelSelector: app.kubernetes.io/part-of=kyverno
    patch: |-
      $patch: delete
      apiVersion: policy/v1
      kind: PodDisruptionBudget
      metadata:
        name: ignored

With a target, Kustomize applies the patch to every object the selector
matches, here the four PDBs, and the metadata.name in the patch is
required but not used. The output was the render without the four PDBs and
otherwise unchanged. A selector that matches nothing is not an error: over
a render for release policy, the build passed and kept all four PDBs.

Kapitan's mutate

The class pins the chart by version and by the
digest of its OCI artifact, and reads the same values
file:

parameters:
  kapitan:
    dependencies:
      - type: helm
        source: oci://ghcr.io/kyverno/charts/kyverno@sha256:83054f30148c3cfc001087c1f5772eda7c7224e9b74b7f0e70c39c9cafe009d1
        chart_name: kyverno
        version: 3.9.2
        output_path: charts/kyverno
    compile:
      - input_type: helm
        input_paths: [charts/kyverno]
        output_path: kyverno
        helm_values_files: [kyverno-values.yaml]
        helm_params:
          name: kyverno
          namespace: kyverno

The mutation deletes the PDBs by label and states how many it expects:

parameters:
  mutations:
    no-pdbs:
      match: {apiVersion: policy/v1, kind: PodDisruptionBudget, labels: {app.kubernetes.io/part-of: kyverno}}
      delete: true
      expects: 4

The target adds mutate as a second compile entry on
the chart's output_path, as the guide shows. Each PDB file became a
single comment line, and the rest of the render was unchanged. With the
class changed to release policy, the compile stops:

mutate: mutation no-pdbs: expects 4, matched 0

Helm hooks in the rendered output

helm template also renders the chart's hooks: a post-upgrade Job
kyverno-migrate-resources with its ServiceAccount, ClusterRole and
ClusterRoleBinding, and the pre-delete Jobs kyverno-scale-to-zero and
kyverno-rm-webhooks. helm install runs each of them only at its hook.
kubectl apply of the whole output runs them at once, and on kind the
Job kyverno-scale-to-zero left every controller at zero replicas:

kyverno-admission-controller    0/0     0            0           5s
kyverno-background-controller   0/0     0            0           5s

The test therefore applied every object except those with a
helm.sh/hook annotation, as helm install would. A tool that maps Helm
hooks to its own, as Argo CD does, needs no such filter.

On kind

The four controllers, with three admission controller replicas and two of
each other, all reported Available, and the namespace held no
PodDisruptionBudget. The Kustomize output, without hooks, passed a
server-side dry run. To check that Kyverno still enforces policies, the
test added a ValidatingPolicy that requires the label team on ConfigMaps
in the namespace demo. A ConfigMap without it was refused:

error: failed to create configmap: admission webhook "vpol.validate.kyverno.svc-fail" denied the request: Policy require-team-label failed: label team is required

External Secrets Helm chart: the ValidatingWebhookConfigurations {#external-secrets-webhooks}

external-secrets/external-secrets#6961,
opened in September 2026, asks for a value that leaves the two
ValidatingWebhookConfigurations out of the chart while it still deploys the
webhook. The reporter applies cluster-scoped objects from a separate chart
and wants the configurations to come from there, with the
external-secrets chart used as published. The maintainers asked for the
use case. alekc
called it
"a very niche usecase which would be best suited by a custom helm chart for
your org" and wrote that such a toggle "would mean allowing users to deploy
a broken (unless they take separate steps outside of the helm chart)
deployment of eso". evrardj-roche closed the pull request with the toggle
"temporarily until we have clarified the multi-tenant story" and pointed
to the multi-namespace design in #7023.

In chart 2.13.0, the release of 16 October 2026,
validatingwebhook.yaml
renders secretstore-validate and externalsecret-validate under the same
condition as the webhook itself:

{{- if .Values.webhook.create }}
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: secretstore-validate

webhook.create=false removes the configurations and also the webhook's
Deployment, Service, Secret and ServiceAccount. The configurations have
failurePolicy: Fail and no caBundle. cert-controller adds it, and its
ClusterRole
allows it to update only these two names:

  - apiGroups:
    - "admissionregistration.k8s.io"
    resources:
    - "validatingwebhookconfigurations"
    resourceNames:
    - "secretstore-validate"
    - "externalsecret-validate"
    verbs:
    - "update"
    - "patch"

Kustomize over helm template

helm template external-secrets oci://ghcr.io/external-secrets/charts/external-secrets \
  --version 2.13.0 --namespace external-secrets > helm.yaml
kustomize build .

with this kustomization.yaml:

resources:
  - helm.yaml
patches:
  - target:
      kind: ValidatingWebhookConfiguration
      labelSelector: external-secrets.io/component=webhook
    patch: |-
      $patch: delete
      apiVersion: admissionregistration.k8s.io/v1
      kind: ValidatingWebhookConfiguration
      metadata:
        name: ignored

The label external-secrets.io/component: webhook is also on the webhook's
Service and Secret, so the target names the kind as well. The output
was the render without the two configurations and otherwise unchanged.

Kapitan's mutate

The class pins chart 2.13.0 by OCI digest like the Kyverno one, and the
mutation expects exactly the two configurations:

parameters:
  mutations:
    no-validating-webhooks:
      match: {apiVersion: admissionregistration.k8s.io/v1, kind: ValidatingWebhookConfiguration, labels: {external-secrets.io/component: webhook}}
      delete: true
      expects: 2

On kind

Without the configurations, the controller and the webhook Deployments
became Available, and cert-controller did not become Ready. Its readiness
probe kept failing with HTTP status 500:

external-secrets-cert-controller-56d7b45fc5-btm8q ready=false restarts=0

With no webhook registered, a SecretStore whose fake
provider has an empty key was accepted. The test then applied the two
configurations as the plain render has them, names and labels included,
standing in for the separate chart. cert-controller injected the CA and
became Ready:

secretstore-validate caBundle bytes: 1196
externalsecret-validate caBundle bytes: 1196

A valid SecretStore passed the webhook, which added its warning about the
fake provider, and the one with the empty key was now refused:

Error from server (Forbidden): error when creating "eso-store-invalid.yaml": admission webhook "validate.secretstore.external-secrets.io" denied the request: key must be set in data 0

The deletion is half of the change the issue asks for. The other chart has
to apply both configurations with these names, or cert-controller cannot
update them and stays unready.

Originally published at kadet-hub.org.

Kustomize JSON patch add: fields a Helm chart does not template

Moep90 — Sun, 25 Oct 2026 19:55:45 +0000

A chart can only set the fields its templates write. When a Kubernetes
field you need is missing from them, the value you add to values.yaml
is silently ignored, unless the chart has a values schema that rejects it.
This post sets nine such fields from open issues on the rendered manifests,
first with a Kustomize patch over helm template, then with
Kapitan's mutate
generator. Most of them are RFC 6902 JSON Patch
operations, and mutate's json6902 takes the same operations as
Kustomize, so the two versions of each patch read almost the same.

Each patch was rendered from the chart version named in its section with
Helm 4.3.1, Kustomize 5.8.3, and Kapitan 0.36.3 with mutate 0.1.0. In every
case the Kustomize output and the mutate output set the same value and
changed nothing else in the render. The mutate output was installed on
kind clusters on Kubernetes 1.37.0 with three worker nodes, and each
Kustomize output's patched object passed a server-side dry run there.
Charts from HTTP
repositories matched the sha256 their index.yaml lists. The guide
Patch a Helm chart compares the places such a
patch can live.

The class for each chart pins its version, like this
one for Vault, and the target adds mutate as a second
compile entry on the chart's output_path, as the guide shows:

parameters:
  kapitan:
    dependencies:
      - type: helm
        source: https://helm.releases.hashicorp.com
        chart_name: vault
        version: 0.34.1
        output_path: charts/vault
    compile:
      - input_type: helm
        input_paths: [charts/vault]
        output_path: vault
        helm_values_files: [values/vault.yaml]
        helm_params:
          name: vault
          namespace: vault

Vault Helm chart: Service appProtocol {#vault-service-appprotocol}

hashicorp/vault-helm#1108,
opened in April 2025 and at 9 thumbs up, asks for appProtocol: https on
the Vault Service. Without it, Gateway API implementations such as GKE's
Gateway controller talk plain HTTP to a Vault that serves HTTPS, according
to the issue and the GKE documentation it links. No maintainer has
replied. Chart 0.34.1, released on 13 August 2026, writes the ports in
server-service.yaml
without it:

  ports:
    - name: {{ include "vault.scheme" . }}
      port: {{ .Values.server.service.port }}
      targetPort: {{ .Values.server.service.targetPort }}
      {{- if and (.Values.server.service.nodePort) (eq (.Values.server.service.type | toString) "NodePort") }}
      nodePort: {{ .Values.server.service.nodePort }}
      {{- end }}
    - name: https-internal
      port: 8201
      targetPort: 8201

The port takes its name from the scheme, https once global.tlsDisable
is false, which this values file sets:

global:
  tlsDisable: false

resources:
  - helm.yaml
patches:
  - target:
      kind: Service
      name: vault
    patch: |-
      - op: test
        path: /spec/ports/0/name
        value: https
      - op: add
        path: /spec/ports/0/appProtocol
        value: https

parameters:
  mutations:
    vault-app-protocol:
      match: {apiVersion: v1, kind: Service, name: vault}
      json6902:
        - {op: test, path: /spec/ports/0/name, value: https}
        - {op: add, path: /spec/ports/0/appProtocol, value: https}

test makes both patches refuse to mark a plain HTTP port as https. With
the chart's default tlsDisable: true the first port is named http, and
both stopped:

Error: testing value /spec/ports/0/name failed: test failed
mutate: mutation vault-app-protocol: vault/templates/server-service.yaml: json6902 test of '/spec/ports/0/name' failed

The Kustomize target name: vault matched the Service vault only, not
vault-internal or vault-agent-injector-svc. The test applied the
chart's Services to kind, without the Vault server, and the stored Service
carries the field:

https 8200 appProtocol=https

CloudNativePG cluster chart: topologySpreadConstraints {#cloudnative-pg-topology-spread-constraints}

cloudnative-pg/charts#817,
opened in February 2026, asks for topologySpreadConstraints on the
Cluster that the cluster chart renders. The CloudNativePG operator
supports the field, but the chart's
cluster.yaml
only templates affinity. Pull request #818, opened the same day, adds
it. It has had no review and now conflicts with the chart's main branch.
In its thread, a user pointed out that the label cnpg.io/cluster alone
also matches pooler and Job pods, and suggested adding
cnpg.io/podRole: instance to the selector.

On node-local storage that selector has a catch. CloudNativePG creates
each instance's volume through an initdb or join Job, whose pod gets the
instance's constraints but not the label cnpg.io/podRole, so the
constraint does not count it. kind's local-path volumes are bound to the
node of the first pod that mounts them. In a run with that selector, the
join Job of the third instance landed on the node of the second, the
volume followed it, and the third instance stayed Pending:

0/4 nodes are available: 1 node(s) didn't match pod topology spread constraints, 1 node(s) had untolerated taint(s), 2 node(s) didn't match PersistentVolume's node affinity. preemption: 0/4 nodes are available: 1 No preemption victims found for incoming pod, 3 Preemption is not helpful for scheduling.

A selector that counts the Job pods too and leaves out only poolers spread
all three instances. Until a chart release has the value, chart 0.9.0
takes this patch:

resources:
  - helm.yaml
patches:
  - target:
      group: postgresql.cnpg.io
      kind: Cluster
      name: pg-cluster
    patch: |-
      - op: add
        path: /spec/topologySpreadConstraints
        value:
          - maxSkew: 1
            topologyKey: kubernetes.io/hostname
            whenUnsatisfiable: DoNotSchedule
            labelSelector:
              matchLabels:
                cnpg.io/cluster: pg-cluster
              matchExpressions:
                - key: cnpg.io/podRole
                  operator: NotIn
                  values: [pooler]

parameters:
  mutations:
    cluster-topology-spread:
      match: {apiVersion: postgresql.cnpg.io/v1, kind: Cluster, name: pg-cluster}
      json6902:
        - op: add
          path: /spec/topologySpreadConstraints
          value:
            - maxSkew: 1
              topologyKey: kubernetes.io/hostname
              whenUnsatisfiable: DoNotSchedule
              labelSelector:
                matchLabels:
                  cnpg.io/cluster: pg-cluster
                matchExpressions:
                  - key: cnpg.io/podRole
                    operator: NotIn
                    values: [pooler]

With operator chart 0.29.1 on the cluster and the release named pg, the
Cluster pg-cluster became Ready on a fresh cluster. Its three instance
pods carry the constraint and run on three different workers:

pg-cluster-1 node=set-fields-worker2 ready=true spread=kubernetes.io/hostname
pg-cluster-2 node=set-fields-worker ready=true spread=kubernetes.io/hostname
pg-cluster-3 node=set-fields-worker3 ready=true spread=kubernetes.io/hostname

Harbor Helm chart: annotations on the core Deployment {#harbor-core-deployment-annotations}

goharbor/harbor-helm#2221,
opened in August 2025, asks for annotations and labels on the components'
Deployments, beyond the pod annotations the chart takes. One commenter
needs an annotation on harbor-core for Reloader. A collaborator
suggested a Helm hook, which the reporter cannot use with Argo CD. Three
pull requests for the values were closed unmerged by the stale bot. In chart
1.19.4, released on 16 October 2026,
core-dpl.yaml
writes no annotations into the Deployment's metadata, and
core.podAnnotations, whose comment in values.yaml reads "Additional
deployment annotations", goes to the pod template only:

metadata:
  name: {{ template "harbor.core" . }}
  namespace: {{ .Release.Namespace | quote }}
  labels:
{{ include "harbor.labels" . | indent 4 }}
    component: core
    app.kubernetes.io/component: core

An annotation is a key in a map, which a strategic merge patch and
mutate's merge add without touching other keys:

resources:
  - helm.yaml
patches:
  - patch: |-
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: harbor-core
        namespace: harbor
        annotations:
          reloader.stakater.com/auto: "true"

parameters:
  mutations:
    harbor-core-annotations:
      match: {apiVersion: apps/v1, kind: Deployment, name: harbor-core}
      merge:
        metadata:
          annotations:
            reloader.stakater.com/auto: "true"

Harbor generates secrets on every render, so two renders differ in their
Secrets and in the Deployments' checksum annotations. The comparison of
the renders left those out. On kind, every Harbor Deployment became
Available, and the stored harbor-core carries the annotation:

harbor-core annotations: {"deployment.kubernetes.io/revision":"1","reloader.stakater.com/auto":"true"}

ECK operator chart: ServiceMonitor labels and interval {#eck-operator-servicemonitor}

elastic/cloud-on-k8s#7717,
opened in April 2024, asks for labels and the interval on the operator's
ServiceMonitor, because a Prometheus from kube-prometheus-stack selects
ServiceMonitors by its release label. A commenter confirmed in August
2026 that 3.5.0 ignores serviceMonitor.labels and keeps a second
ServiceMonitor by hand. The chart has no values schema, so the key is
silently dropped. In 3.5.0, released on 4 August 2026,
service-monitor.yaml
renders the ServiceMonitor only with metrics in secure mode, with the
chart's own labels and a fixed interval:

{{- if and .Values.config.metrics.secureMode.enabled .Values.serviceMonitor.enabled }}
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: {{ include "eck-operator.fullname" . }}
  namespace: {{ ternary .Values.serviceMonitor.namespace .Release.Namespace (not (and (.Values.serviceMonitor) (empty .Values.serviceMonitor.namespace))) }}
  labels: {{- include "eck-operator.labels" . | nindent 4 }}
spec:
  namespaceSelector:
    matchNames:
      - {{ .Release.Namespace }}
  selector:
    matchLabels:
      app.kubernetes.io/name: {{ include "eck-operator.name" . }}-metrics-service
      app.kubernetes.io/instance: {{ .Release.Name }}
  endpoints:
  - port: https
    path: /metrics
    scheme: https
    interval: 30s

These values turn the metrics endpoint and secure mode on:

config:
  metrics:
    port: "8080"
    secureMode:
      enabled: true

resources:
  - helm.yaml
patches:
  - target:
      group: monitoring.coreos.com
      kind: ServiceMonitor
      name: elastic-operator
    patch: |-
      - op: add
        path: /metadata/labels/release
        value: kube-prometheus-stack
      - op: test
        path: /spec/endpoints/0/interval
        value: 30s
      - op: replace
        path: /spec/endpoints/0/interval
        value: 60s

parameters:
  mutations:
    eck-servicemonitor:
      match: {apiVersion: monitoring.coreos.com/v1, kind: ServiceMonitor, name: elastic-operator}
      json6902:
        - {op: add, path: /metadata/labels/release, value: kube-prometheus-stack}
        - {op: test, path: /spec/endpoints/0/interval, value: 30s}
        - {op: replace, path: /spec/endpoints/0/interval, value: 60s}

add on /metadata/labels/release adds one key to the existing labels.
The test of 30s stops the patch if a release changes the interval. On
kind, with the ServiceMonitor CRD of kube-prometheus-stack 92.11.0, the
operator became ready and the stored ServiceMonitor has both values:

release=kube-prometheus-stack interval=60s

Temporal Helm chart: Service trafficDistribution {#temporal-service-trafficdistribution}

temporalio/helm-charts#897,
opened in April 2026, asks for spec.trafficDistribution on the server
Services, at least the frontend. Its author sets the field today with a
separate server-side apply after each install. No maintainer has replied.
Chart 1.7.0, released on 12 September 2026, writes type, ports and
selector in
server-service.yaml
and has no values schema, so server.frontend.service.trafficDistribution
is ignored. The chart needs a database to render, and this values file
names one that the test never contacted:

server:
  config:
    persistence:
      datastores:
        default:
          sql:
            pluginName: postgres12
            databaseName: temporal
            connectAddr: db.example.internal:5432
        visibility:
          sql:
            pluginName: postgres12
            databaseName: temporal_visibility
            connectAddr: db.example.internal:5432

resources:
  - helm.yaml
patches:
  - target:
      kind: Service
      name: temporal-frontend
    patch: |-
      - op: add
        path: /spec/trafficDistribution
        value: PreferSameZone

parameters:
  mutations:
    temporal-traffic-distribution:
      match: {apiVersion: v1, kind: Service, name: temporal-frontend}
      json6902:
        - {op: add, path: /spec/trafficDistribution, value: PreferSameZone}

PreferClose, the value in the issue, is the older name for
PreferSameZone, and the Kubernetes documentation
lists it as deprecated.
The field is generally available since Kubernetes 1.33. The value
PreferSameZone is enabled by default from 1.34, so a 1.33 cluster needs
PreferClose. Without a database the Temporal servers cannot start, so
the test applied the chart's Services only, and the API server stored the
field:

temporal-frontend trafficDistribution: PreferSameZone

MetalLB Helm chart: the speaker without host networking {#metallb-speaker-hostnetwork}

metallb/metallb#3005,
opened in April 2026, asks for hostNetwork: false on the speaker
DaemonSet. The reporter runs FRR on every node next to MetalLB, and FRR
does not accept iBGP peering with an address in its own network
namespace, so the speaker has to run in the pod network and peer with the
node's FRR. Another user peers the speakers with Calico the same way. The
maintainers asked for the use case. Pull request #3099 adds a
speaker.hostNetwork value, and on it the maintainer fedepaol
wrote
"I think it makes sense to document this, mentioning that l2 won't be
able to work." It is not merged. Chart 0.16.1 hard-codes the field in
speaker.yaml:

      hostNetwork: true

By default the chart runs BGP in a separate frr-k8s DaemonSet, which keeps
host networking whatever the speaker does. The test therefore uses
MetalLB's native mode, in which the speaker itself speaks BGP:

frrk8s:
  enabled: false
speaker:
  frr:
    enabled: false

Flipping hostNetwork alone is not enough in 0.16.1. The speaker serves
its health endpoint on 127.0.0.1 inside its own network namespace
(speaker/main.go),
and the chart's probes ask for 127.0.0.1, which the kubelet resolves in
the node's network namespace. With only the field changed, no speaker
became ready on kind, and every probe was refused:

Readiness probe failed: Get "http://127.0.0.1:17472/readyz": dial tcp 127.0.0.1:17472: connect: connection refused
Liveness probe failed: Get "http://127.0.0.1:17472/healthz": dial tcp 127.0.0.1:17472: connect: connection refused

Pull request #3099 changes the field only, so it would leave the same
probes in place. The patch below also removes both probes. The speaker
then has no health check, and the kubelet restarts it only when the
process exits:

resources:
  - helm.yaml
patches:
  - target:
      kind: DaemonSet
      name: metallb-speaker
    patch: |-
      - op: test
        path: /spec/template/spec/hostNetwork
        value: true
      - op: replace
        path: /spec/template/spec/hostNetwork
        value: false
      - op: test
        path: /spec/template/spec/containers/0/name
        value: speaker
      - op: remove
        path: /spec/template/spec/containers/0/livenessProbe
      - op: remove
        path: /spec/template/spec/containers/0/readinessProbe

parameters:
  mutations:
    metallb-speaker-pod-network:
      match: {apiVersion: apps/v1, kind: DaemonSet, name: metallb-speaker}
      json6902:
        - {op: test, path: /spec/template/spec/hostNetwork, value: true}
        - {op: replace, path: /spec/template/spec/hostNetwork, value: false}
        - {op: test, path: /spec/template/spec/containers/0/name, value: speaker}
        - {op: remove, path: /spec/template/spec/containers/0/livenessProbe}
        - {op: remove, path: /spec/template/spec/containers/0/readinessProbe}

On kind, a speaker ran on each of the four nodes with its own pod IP.
Without a readiness probe, ready means that the container runs:

metallb-speaker-6nsjd hostNetwork=false pod IP is the node IP: no ready=true restarts=0
metallb-speaker-6rr28 hostNetwork=false pod IP is the node IP: no ready=true restarts=0
metallb-speaker-8z7rr hostNetwork=false pod IP is the node IP: no ready=true restarts=0
metallb-speaker-hc622 hostNetwork=false pod IP is the node IP: no ready=true restarts=0

L2 mode answers ARP requests for the service address on the node's
interface, which a speaker in the pod network does not have. With an L2
pool from the kind network and an nginx LoadBalancer Service, the test
compared both settings from the host:

speaker in the pod network: the LoadBalancer address from the host: not reachable
speaker in the host network (plain chart): the LoadBalancer address from the host: reachable

The test did not set up a BGP peer, so it shows that the speakers run in
the pod network, not that they announce routes over BGP from there.

Consul Helm chart: tolerations on the hook Jobs {#consul-hook-job-tolerations}

hashicorp/consul-k8s#4469,
opened in January 2025 and at 7 thumbs up, asks for tolerations and a
nodeSelector on the Job that generates the gossip encryption key. When
every node carries a taint, that Job and tls-init stay Pending, and the
thread names a third Job, tls-init-cleanup. The workaround offered there is
to helm pull the chart and edit its templates, which a commenter who
installs from HashiCorp's repository cannot do. Pull request #4625, opened
in June 2025, adds the values and has had no review. In chart 2.0.4,
released on 11 September 2026, none of the three Jobs sets either field in
its pod spec, and 2.1.0-rc2 does not either. All three are Helm hooks, as
tls-init-job.yaml
shows for tls-init:

  annotations:
    "helm.sh/hook": pre-install,pre-upgrade
    "helm.sh/hook-weight": "1"
    "helm.sh/hook-delete-policy":  hook-succeeded,before-hook-creation

The gossip key Job runs at the same hooks, tls-init-cleanup at
pre-delete. These values turn on TLS and the gossip key, and
global.name gives the Jobs the names from the issue:

global:
  name: consul
  tls:
    enabled: true
  gossipEncryption:
    autoGenerate: true

The test tainted and labelled all three workers with
example.com/purpose=infra and set the chart's own tolerations and
nodeSelector values for the server, the connect injector and the webhook
cert manager. global.acls.tolerations reaches the
consul-gateway-resources Job, which a commenter also mentions, and the
two gateway cleanup Jobs, even with ACLs off. Those three have no
nodeSelector value. With the plain chart, both pre-install Jobs stayed
Pending:

consul-tls-init-fh7nh: 0/4 nodes are available: 4 node(s) had untolerated taint(s). preemption: 0/4 nodes are available: 4 Preemption is not helpful for scheduling.

The three Jobs carry app: consul whatever the release is called, and
component with their own name, so one Kustomize target with a set-based
selector matches all three:

resources:
  - helm.yaml
patches:
  - target:
      group: batch
      kind: Job
      labelSelector: app=consul,component in (tls-init,tls-init-cleanup,gossip-encryption-autogenerate)
    patch: |-
      - op: add
        path: /spec/template/spec/tolerations
        value:
          - key: example.com/purpose
            operator: Equal
            value: infra
            effect: NoSchedule
      - op: add
        path: /spec/template/spec/nodeSelector
        value:
          example.com/purpose: infra

mutate's match takes one value per key, so each Job gets a mutation of
its own, and the three share the operations through an inventory key:

parameters:
  hook_scheduling:
    - op: add
      path: /spec/template/spec/tolerations
      value:
        - {key: example.com/purpose, operator: Equal, value: infra, effect: NoSchedule}
    - op: add
      path: /spec/template/spec/nodeSelector
      value:
        example.com/purpose: infra
  mutations:
    consul-tls-init:
      match: {apiVersion: batch/v1, kind: Job, labels: {app: consul, component: tls-init}}
      json6902: ${hook_scheduling}
    consul-tls-init-cleanup:
      match: {apiVersion: batch/v1, kind: Job, labels: {app: consul, component: tls-init-cleanup}}
      json6902: ${hook_scheduling}
    consul-gossip-encryption-autogenerate:
      match: {apiVersion: batch/v1, kind: Job, labels: {app: consul, component: gossip-encryption-autogenerate}}
      json6902: ${hook_scheduling}

Without a cluster to ask, helm template renders the chart's two
PodDisruptionBudgets as policy/v1beta1, which Kubernetes 1.25 removed,
and the first server-side dry run refused them. Both renders therefore
pass --api-versions policy/v1/PodDisruptionBudget, which Kapitan takes as
api_versions in helm_params.

Helm hooks with the patched Jobs

kubectl apply of the whole render would also start tls-init-cleanup,
which deletes the Secrets consul-ca-cert, consul-ca-key and
consul-server-cert. The test applied the mutate output in the order of
helm install: the pre-install hooks, the templates, then the
post-install Job. The server and the connect injector became ready. The
pre-delete hooks, applied last as helm uninstall would run them, removed
the three Secrets. Every Job completed on a tainted worker:

consul-tls-init node=cac-consul-worker2 taint=example.com/purpose=infra:NoSchedule complete=True
consul-gossip-encryption-autogenerate node=cac-consul-worker taint=example.com/purpose=infra:NoSchedule complete=True
consul-tls-init-cleanup node=cac-consul-worker2 taint=example.com/purpose=infra:NoSchedule complete=True

For helm install itself, Helm 4 passes hooks to the post-renderer
together with the templates. Helm 3 never did. With Helm 3.22.1 and the
same Kustomize patch as a post-renderer script, helm template printed
the three Jobs without the fields. A Helm 4 post-renderer is a plugin, and
this one runs the Kustomize patch from the directory it is given:

name: kustomize-pr
version: 0.1.0
type: postrenderer/v1
apiVersion: v1
runtime: subprocess
runtimeConfig:
  platformCommand:
    - command: "${HELM_PLUGIN_DIR}/kustomize.sh"

#!/usr/bin/env bash
# stdin: Helm's render, hooks included. $1: the directory with kustomization.yaml.
set -euo pipefail
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT
cp "$1/kustomization.yaml" "$tmp/"
cat > "$tmp/helm.yaml"
kustomize build "$tmp"

helm plugin install ./kustomize-pr
helm install consul consul --repo https://helm.releases.hashicorp.com --version 2.0.4 \
  -n consul -f consul-values.yaml --post-renderer kustomize-pr --post-renderer-args "$PWD" --wait

The plugin's output equalled the Kustomize build. With Helm 4.3.1 on kind,
the release installed, and helm uninstall ran tls-init-cleanup, each Job
on a tainted worker:

Successfully assigned consul/consul-gossip-encryption-autogenerate-jdm6k to cac-consul-worker
Successfully assigned consul/consul-tls-init-qt7ql to cac-consul-worker3
Successfully assigned consul/consul-tls-init-cleanup-wkt87 to cac-consul-worker

APISIX ingress controller chart: pod labels {#apisix-ingress-controller-pod-labels}

apache/apisix-helm-chart#978,
opened in June 2026 and at 11 thumbs up, asks for a podLabels value in
the apisix-ingress-controller chart, so that the controller's pods can
carry labels for observability, scheduling or policies. Nobody has
commented. The apisix chart in the same repository got podLabels through
pull request #965, merged on 30 September 2026. The controller chart takes
deployment.podAnnotations, but in 1.4.0, released on 17 September 2026,
deployment.yaml
gives the pod template the selector labels only:

      labels:
        {{- include "apisix-ingress-controller-manager.selectorLabels" . | nindent 8 }}

The chart's labelsOverride replaces these selector labels, so a label
set there also lands in the Deployment's spec.selector and in the
selectors of both Services. Over the installed chart, the API server
refused the change:

The Deployment "apisix-ingress-controller" is invalid: spec.selector: Invalid value: {"matchLabels":{"app.kubernetes.io/instance":"apisix-ingress-controller","app.kubernetes.io/name":"apisix-ingress-controller","team":"platform"}}: field is immutable

add on a path below the pod template's labels sets one label there and
leaves the selector alone:

resources:
  - helm.yaml
patches:
  - target:
      group: apps
      kind: Deployment
      name: apisix-ingress-controller
    patch: |-
      - op: add
        path: /spec/template/metadata/labels/team
        value: platform

parameters:
  mutations:
    apisix-pod-labels:
      match: {apiVersion: apps/v1, kind: Deployment, name: apisix-ingress-controller}
      json6902:
        - {op: add, path: /spec/template/metadata/labels/team, value: platform}

A key with a slash such as example.com/team goes into the path as
example.com~1team, and both tools then set the label example.com/team.
The chart generates its webhook certificate on every render, so the
comparison of the renders left out that Secret and the webhook
configuration's caBundle. helm template renders the chart's CRDs only
with --include-crds, which Kapitan's helm input passes by default. On
kind, the controller became ready, and its pod carries the label:

apisix-ingress-controller-85b5bb8fb4-9m4kc labels: {"app.kubernetes.io/instance":"apisix-ingress-controller","app.kubernetes.io/name":"apisix-ingress-controller","pod-template-hash":"85b5bb8fb4","team":"platform"}

Cilium Helm chart: ipFamilyPolicy on cilium-ingress {#cilium-ingress-ipfamilypolicy}

cilium/cilium#27003,
opened in July 2023, asks for ipFamilyPolicy and ipFamilies on the
Service of Cilium's ingress controller. Without them the Service gets the
cluster's primary family only, so on a dual-stack cluster the Ingresses
answer on IPv4 or on IPv6, not on both. The maintainer youngnick wrote in
July 2023 that LB Service customisation was planned for that cycle, and
pinned the issue in December 2023. Chart 1.20.3, released on 15 October
2026,
renders the Service
with its ports, type, loadBalancerClass,
allocateLoadBalancerNodePorts, loadBalancerIP and
externalTrafficPolicy, and 1.21.0-pre.3 renders the same. The chart has
this Service in shared mode only. In dedicated mode the operator creates a
Service per Ingress, which a patch of the chart does not reach.

The ingress controller needs Cilium's kube-proxy replacement, so the test
ran Cilium as the CNI of a dual-stack kind cluster without kube-proxy:

kubeProxyReplacement: true
k8sServiceHost: cac-cilium-control-plane
k8sServicePort: 6443
ipam:
  mode: kubernetes
ipv6:
  enabled: true
ingressController:
  enabled: true
  loadbalancerMode: shared

resources:
  - helm.yaml
patches:
  - target:
      kind: Service
      name: cilium-ingress
    patch: |-
      - op: add
        path: /spec/ipFamilyPolicy
        value: RequireDualStack
      - op: add
        path: /spec/ipFamilies
        value: [IPv4, IPv6]

parameters:
  mutations:
    cilium-ingress-dual-stack:
      match: {apiVersion: v1, kind: Service, name: cilium-ingress}
      json6902:
        - {op: add, path: /spec/ipFamilyPolicy, value: RequireDualStack}
        - {op: add, path: /spec/ipFamilies, value: [IPv4, IPv6]}

The chart generates the Hubble certificates on every render, so the
comparison of the renders left out those Secrets. The plain chart gave the
Service one IPv4 cluster IP. The mutate output, applied over it, changed
the existing Service in place, and 60 seconds later the operator had not
reverted it:

ipFamilyPolicy=SingleStack ipFamilies=["IPv4"] clusterIPs=["10.96.179.251"] loadBalancer=198.51.100.0
ipFamilyPolicy=RequireDualStack ipFamilies=["IPv4","IPv6"] clusterIPs=["10.96.179.251","fd00:10:96::b12c"] loadBalancer=198.51.100.0 2001:db8::

LB-IPAM had a pool with one block of each family from the documentation
ranges and gave the Service an address of each. An Ingress for an echo
backend then answered on all four addresses from a node, with curl in
the node's network namespace:

10.96.179.251 from node: 200, from pod: 000
fd00:10:96::b12c from node: 200, from pod: 200
198.51.100.0 from node: 200, from pod: 000
2001:db8:: from node: 200, from pod: 200

From a pod, requests over IPv4 timed out on this test host, with the plain
chart as well and on an IPv4-only kind cluster too. The patch does not
cause it, and the test does not cover that path.

ipFamilies with both families needs a dual-stack cluster, whatever the
policy. On an IPv4-only kind cluster the server-side dry run refused the
patched Service:

* spec.ipFamilyPolicy: Invalid value: "RequireDualStack": this cluster is not configured for dual-stack services
* spec.ipFamilies[1]: Invalid value: "IPv6": not configured on this cluster

With PreferDualStack it still refused ipFamilies[1]. PreferDualStack
without ipFamilies passed there and got IPv4 only.

Originally published at kadet-hub.org.

DEV Community: Moep90

Kapitan generator releases, 19 to 25 October 2026

cloudnative-pg

cloudnative-pg 1.1.1

cloudnative-pg 1.1.0

external-secrets

external-secrets 1.0.5

keycloak

keycloak 1.1.0

kubernetes

kubernetes 1.9.1

kubernetes 1.9.0

kubernetes 1.8.0

kubernetes 1.7.0

kubevirt

kubevirt 1.3.2

kubevirt 1.3.1

kubevirt 1.3.0

kubevirt 1.2.0

kubevirt 1.1.2

mutate

mutate 0.1.0

rabbitmq

rabbitmq 1.1.0

strimzi

strimzi 1.1.0

Docker tag and digest: which one Kubernetes runs

A tag that disagrees with its digest

The tag is never looked up

The tag still sets the pull policy

Two files, one image

Renovate reads the tag from the same line

Renovate automerge with required reviews on GitHub

Why an approval bot does not help

439 changed sandbox/Dockerfile.

A bypass for pull requests only

441 added one entry to the bypass list of the main ruleset. The

What Renovate waits for

Which updates merge

441 it needed one approval by the single maintainer.

The first merge

439 has no review, and GitHub still reports a review as required:

Actions Runner Controller dind image from your own registry

The dind image in gha-runner-scale-set {#arc-dind-image}

Where the chart puts dind

Kustomize over helm template

Kapitan's mutate

On kind

Air-gapped registry

Bitnami PostgreSQL chart alternatives: CloudNativePG and Kapitan

What Broadcom changed, and when

What the PostgreSQL chart looks like today

The options

Rendering the Cluster with a generator

Trade-offs

cdk8s vs Tanka vs Kapitan for Kubernetes config

The app and the check

How much you write

Adding the second app

A value derived from another

Typos and type safety

Toolchain and speed

Reuse and packages

Kapitan and Jsonnet

Output beyond Kubernetes

GitOps

Project activity

Running it

cosign verify-attestation vs gh attestation verify

What kadet-hub attests

Start from the signed index

Verify the artifact with gh

The same with cosign

Failures you want to see

Read the provenance

One script

gVisor in CI: sandboxing untrusted code on GitHub Actions

The problem

What gVisor changes

The one gVisor does not stop, and the trace that does

439 changed `sandbox/Dockerfile`.

441 added one entry to the bypass list of the `main` ruleset. The