DEV Community: Core Solutions

Helm charts are technical debt: blame the templating model

Billy Walker — Mon, 31 Aug 2026 08:03:53 +0000

The pull request is four lines of template, adding an optional nodeSelector to an internal chart. CI has already done its job properly: helm lint passed, the chart rendered against every environment's values file, kubeconform validated each manifest that came out, and the snapshot diff of the rendered output is sitting in the check run waiting to be read.

And you still cannot say, from the diff, what this change does in staging without going and reading three other templates first.

Nobody files that as technical debt. It goes in the mental folder marked “charts are a bit fiddly”. But if your team maintains a dozen first-party charts, that tax gets levied on every review, and no amount of pipeline pays it off, because the pipeline was never the thing that was broken.

This is not a “Helm was always bad” post. Helm solved a real problem, and most teams running Kubernetes are better off with it than without it. The rot is somewhere more specific: Helm renders Kubernetes manifests by running Go's text/template over YAML as strings, and once you are authoring charts rather than just installing them, that model generates work forever.

That model has survived every major version so far, Helm 4 included. If your chart estate already feels brittle, the next release will not make it less so, and I will come back to why.

Helm won for good reasons

Helm is a CNCF graduated project and the default packaging story in Kubernetes, and it deserves to be:

install, upgrade, rollback and release history in a single tool
a standard way for vendors and open-source projects to ship applications
OCI registry support that slots into existing GitOps workflows
ecosystem gravity, which by now matters nearly as much as technical merit

Nobody sensible is still having the “should we use Helm at all?” argument. If you are installing Grafana, or External Secrets Operator, or almost anything with a healthy community behind it, the chart is the path of least resistance and you should take it.

The trouble is that the same tool that consumes packaged software so well also makes it very easy to start writing your own. One chart for the internal API. One for the batch worker. Then a shared library chart, because three of them had grown the same _helpers.tpl. At some point you stopped being a Helm user and became the maintainer of a templating system. That is a perfectly reasonable thing to be, as long as it was a decision rather than a drift.

The debt is in the rendering model

Helm templates know nothing about Kubernetes objects. They emit text that happens to be YAML.

Here is the shape of it, lifted from roughly every deployment.yaml in every internal chart:

# templates/deployment.yaml
spec:
  template:
    spec:
      {{- with .Values.nodeSelector }}
      nodeSelector:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      containers:
        - name: {{ .Chart.Name }}
          image: {{ .Values.image.repository }}:{{ .Values.image.tag }}

That 8 is load-bearing. nodeSelector: sits at column six, so its keys have to land at column eight, and nindent exists precisely because chart authors need to re-indent embedded blocks constantly. Write nindent 6 by mistake and you have not made a formatting error. You have set nodeSelector to null and promoted its contents to siblings of containers, and the render will print that quite happily, because it is still a well-formed YAML document. Nothing in the YAML layer objects. It takes schema validation downstream, kubeconform -strict in the pipeline or a server-side dry run, before anything flags it. That is a lot of machinery to need in order to catch a two-space mistake.

Two spaces. Helm's own template best practices page walks through indentation, whitespace chomping and generated-output formatting, and concedes the underlying problem in passing: “YAML is a whitespace-oriented language.” Full marks for honesty, but that is a peculiar property for an abstraction layer to have.

All of which is the cheap version of the problem, and it is cheap precisely because a machine can express it. Schema validation catches a wrong shape, and it catches it in seconds. What no pipeline catches is a chart whose right shapes have become impossible to hold in your head.

Once your abstraction works on strings rather than typed objects, a few things follow:

indentation becomes part of correctness, not just tidiness
a branch that renders the wrong shape still looks plausible in review
values become an API surface long before anyone has designed them like one
refactoring is hard, because the behaviour lives across helpers, partials and value conventions rather than in a schema your tooling understands

Disciplined teams are not exempt from this. They often accumulate more chart machinery, because they build helpers and layering and conventions to contain the complexity. That containment is genuinely useful, and it is also the tell: you are building scaffolding around a model that produces complexity faster than you can absorb it.

How it shows up on day two

You rarely spot chart debt by reading a chart. You spot it in the changes that need far more scrutiny than their diff size suggests they should.

`values.yaml` turns into an untyped product surface

values.yaml is the front door to your chart, which is convenient right up until you realise you have shipped a long-lived API with no type system behind it.

A values.schema.json belongs in every first-party chart, and Helm will enforce it on install, upgrade, lint and template. But the schema file is optional, so it is opt-in per chart and easy to leave behind on the one chart nobody has revisited, and it describes structure rather than behaviour. A value can be structurally perfect and still produce nonsense once it has threaded its way through conditionals, helper templates, defaults and subchart overrides.

Then another team starts depending on a value name. Renaming a field stops being a cleanup and becomes a migration, complete with a deprecation window and a note in the release notes that nobody reads.

The source is harder to review than the output

A rendered manifest can be entirely ordinary while the logic that produced it is anything but.

Humans review the chart source. A pull request that flips a condition or threads one more flag through three templates is not asking “is this Deployment valid?”. It is asking “which combinations of values, defaults and includes now produce a different document?”, and that question has no diff you can look at.

Which is why the gates all end up pointed at the rendered output rather than the source. Those gates are worth building, and I will come back to ours. They are also a fair signal about the model: you test the output because nobody can reliably reason about the input.

Subcharts and value plumbing sprawl quietly

Small Helm setups look tidy. Then the platform grows, one chart wraps five subcharts, globals appear, child values get overridden from the parent, and a flag in one place changes behaviour three templates away.

Some of those values exist because users asked for them. Some exist because a subchart expects them. Some exist because somebody needed a workaround two years ago and nobody has ever felt brave enough to delete it. That is the point where a chart stops being a packaging unit and starts being institutional memory encoded as YAML conventions.

Helm 4 improved Helm without changing this

This is where lazy anti-Helm takes get sloppy, so let me be precise about it.

Helm 4.0.0 shipped in November 2025 and it is a genuine improvement: server-side apply, a redesigned plugin system with WebAssembly plugins, resource watching built on kstatus, post-renderers as plugins, reproducible chart archives. If you run Helm heavily, upgrade.

All of that is release machinery, though. Rendering still goes through Go's text templating, values.yaml is still the primary interface, and the combinatorial growth of flags and helpers in your own charts remains entirely your problem. Helm 4 made Helm better at the job Helm is good at. Chart authoring was never on the list.

A version bump on its own will not retire this argument, so here is what would: a release that swaps text templating for a typed object model, or one that makes schema validation mandatory instead of opt-in. If you are reading this a few releases later, that is the thing to go and check. Short of it, the version numbers in this post move and the point does not.

What the alternatives actually change

Different tools replace different parts of Helm, which is where most “Helm alternatives” posts go wrong. There is no swap that hands you better authoring, better packaging, better release management and better ecosystem compatibility all at once.

Tool	What it replaces	What you still need
Kustomize	Templating, for the overlay case	Packaging, versioning, release history
Timoni	The authoring model, with CUE typing	Ecosystem gravity, and patience while it is pre-1.0
cdk8s	Templating, with a real programming language	Packaging, plus a language runtime you now own
Jsonnet / Tanka	Templating, with a compact composition syntax	Types, and a way to stop dynamic behaviour relocating complexity
KCL	Templating, with a typed config language	Ecosystem gravity, and a team willing to learn a new language
Score	The developer-facing values surface	A platform implementation underneath that satisfies the spec

Kustomize is the one I reach for most often outside Helm, largely because it already ships inside kubectl and patches structured YAML instead of templating arbitrary strings. If your real problem is “I have plain manifests and need per-environment overlays”, it removes a surprising amount of cleverness for almost no adoption cost. Expecting it to do packaging too is how people end up disappointed by it.

Timoni is the most interesting answer to “what if we fixed the authoring model?”. CUE, real typing, modules distributed as OCI artifacts: conceptually most of what people are asking for. The caveat is maturity, and it is still on its 0.x line (v0.29.0 as I write this), so go and check where it has got to before committing. A 1.0 would change the calculation considerably more than another point release will.

The code-generation family trades typing for operational surface. cdk8s (joined the CNCF Sandbox in 2020) gives you real programming languages, Jsonnet with Tanka gives you a compact composition language, and KCL (joined in September 2023) gives you an explicitly typed configuration language. In exchange you own a language runtime, a build step, and an onboarding cost for everyone who ever touches a manifest. Good trade for a platform team with a large estate, poor trade for a team with six charts.

Score (joined in July 2024) sits one layer up. It is a workload specification, asking “what does this application need from the platform?” rather than “how do I render manifests for every deployment case?”. If your real complaint is that developers keep being handed an ever-growing values surface to fill in, that is the layer to attack. It still assumes a platform underneath that can satisfy it.

How we actually run this

We stayed on Helm, so it is only fair to say what containing the cost looks like day to day.

Most of our platform components are wrapper charts, and that pattern carries a lot of the weight. Chart.yaml declares a pinned dependency on the upstream chart, Chart.lock freezes what actually resolved, and our own templates/ directory layers on what upstream does not ship: the NetworkPolicies, the PrometheusRules, the RBAC that upstream leaves to you. Renovate raises the upstream bumps, and they do not land until CI is green.

Values sit in one place per chart. values.yaml carries the defaults every environment shares, and a values/ directory holds the environment-specific overrides beside it, one file each. CI then lints the chart once against every one of those files rather than once overall, so a change that only breaks one environment cannot ride in on another's values.

That catches everything Helm can be made to fail on. The quieter failure needs its own gate, because of a footgun worth internalising: Helm deep-merges maps, but it replaces lists outright. An environment file that overrides a list has to restate the entire list, so adding a component in one environment and forgetting another raises no error anywhere. Both files are valid. You just ship a shorter list to the environment nobody updated, and find out when something is missing. Nothing you can put in values.yaml expresses “these lists must stay in step”, so a parity check in CI does it instead, and a red PR replaces copy-paste discipline. That is the scaffolding tax from earlier made concrete: a shell script exists because the type system does not.

None of this makes the templating model typed. It makes the cost visible and bounded, which is the achievable goal rather than the ideal one.

Where I would draw the line

Helm is not technical debt because it exists in your stack, and authoring your own charts is not a mistake. We do both, and given the same constraints we would do both again. A third-party chart you install, version and occasionally override is packaging doing exactly the job it is good at, and a first-party chart is a reasonable way to ship your own software into a cluster where everything else already arrives as one.

So I reach for Helm when:

I am consuming third-party software that already ships a chart and I have no wish to become its packager
I want release history, rollback and the install workflow Helm already gives me
the surrounding ecosystem expects Helm artifacts, GitOps tooling included
I am packaging our own components for a platform where every other component already arrives as a chart

What accrues interest is the estate you build afterwards, and how much of it you let grow without a gate on it. Our largest first-party chart carries a values.yaml north of 800 lines. That is not automatically a failure, but it is a number I would want any team to be able to quote about themselves, because the moment nobody can tell you what a chart's values surface costs, you have stopped managing it. Same instinct as right-sizing the rest of your platform: the tool is fine, and the real question is whether the estate around it still fits the team.

The alternatives are worth knowing even when you stay. We have stayed, because packaging gravity and release history are worth more to us today than typed authoring would be. If that trade flips, if the estate outgrows its gates or Timoni reaches 1.0 with real ecosystem behind it, the honest move is to notice rather than to defend the original choice.

You do not have to become anti-Helm to admit that the charts you maintain can rot. In most Kubernetes estates that is hardly a hot take. It is just what year two looks like.

PodSecurityPolicy replacement: Pod Security Admission

Billy Walker — Mon, 31 Aug 2026 07:57:40 +0000

If you still think of Pod Security Admission as the cut-down replacement for PodSecurityPolicy, that is exactly the mindset that makes rollouts harder than they need to be. It is less powerful. That is also why it is practical. You get three fixed profiles, three modes, a handful of labels, and an admission controller that is already in the API server. No custom policy language, no mutation, no controller to install, and far fewer ways to surprise yourself mid-change.

The trick is to use that simplicity properly. A safe rollout is not “turn on restricted everywhere and hope for the best”. It is version-pinned labels, audit and warn first, then namespace-by-namespace enforcement once you have seen what will break. Done that way, you can move from the old PSP world to a built-in control that teams will actually leave enabled.

What replaced PodSecurityPolicy

PodSecurityPolicy was deprecated in Kubernetes v1.21 and removed in v1.25. Pod Security Admission became generally available in the same release, which matters because there is nothing extra to deploy: if your cluster is current enough, the mechanism is already there.

The model is deliberately small:

Profiles: privileged, baseline, restricted
Modes: enforce, audit, warn
Version pinning: pod-security.kubernetes.io/<mode>-version, pinned to a Kubernetes minor version or latest

Those labels live on namespaces:

# pin policy to the minor version your cluster is running
pss="$(kubectl version -o json | jq -r '.serverVersion.gitVersion' | cut -d. -f1,2)"

kubectl label ns payments \
  pod-security.kubernetes.io/enforce=baseline \
  pod-security.kubernetes.io/enforce-version="$pss"

That simplicity is the point. PSP mixed policy with RBAC, allowed mutation, and had enough moving parts that many teams either avoided it or never quite trusted what a change would do. PSA is narrower. It sets a floor.

The mental model: three profiles, three modes, one version pin

You can think of PSA as a matrix:

Mode	What happens
`enforce`	Rejects a non-compliant pod creation request
`audit`	Allows it, but records an audit annotation
`warn`	Allows it, but shows a warning to the caller

And the profiles are ordered from least to most strict:

Profile	What it is for
`privileged`	System namespaces and workloads that genuinely need host access or privileged behaviour
`baseline`	A sensible minimum floor for ordinary application namespaces
`restricted`	Current pod hardening best practice

For most application namespaces, the safe starting pattern is:

kubectl label --overwrite ns payments \
  pod-security.kubernetes.io/enforce=baseline \
  pod-security.kubernetes.io/enforce-version="$pss" \
  pod-security.kubernetes.io/warn=restricted \
  pod-security.kubernetes.io/warn-version="$pss" \
  pod-security.kubernetes.io/audit=restricted \
  pod-security.kubernetes.io/audit-version="$pss"

That gives you a firm minimum bar in enforce, while warn and audit show you what stands between the namespace and restricted.

The version pin is not optional bureaucracy. If you leave a namespace on latest, any cluster upgrade can quietly tighten policy under your workloads. Pinning to the minor version you are actually running means policy changes happen when you decide to move the label.

What `restricted` actually checks

The easiest way to lose credibility on this topic is to hand-wave the restricted profile and get one detail wrong. Several of the checklists floating around in other posts are simply false — especially around runAsUser and seccomp.

Here is the practical checklist, straight from the Pod Security Standards:

Control	What `restricted` expects
Volume types	Only `configMap`, `csi`, `downwardAPI`, `emptyDir`, `ephemeral`, `persistentVolumeClaim`, `projected`, `secret`
Privilege escalation	`allowPrivilegeEscalation: false` on every container, including init and ephemeral containers
Non-root	`runAsNonRoot: true` at pod level or per container
UID	`runAsUser` does not need to be set, but if it is, it cannot be `0`
Capabilities	Drop `ALL`; the only allowed add is `NET_BIND_SERVICE`
Seccomp	Must be explicitly `RuntimeDefault` or `Localhost`; leaving it unset is a violation under `restricted`

Two of those rows are where the bad checklists go wrong. runAsUser is not required — the policy forbids UID 0, it does not make you hard-code some other UID just to satisfy the admission check. And seccomp cuts the other way: under baseline an unset profile is fine, but under restricted it is a violation in its own right, which is one of the reasons plain demo manifests still bounce when teams first try this.

The rollout that does not break production

The safest PSA rollout starts with a dry run, not an enforcement change — the official migration guide is built around the same idea.

First, test label application server-side across all namespaces:

kubectl label --dry-run=server --overwrite ns --all \
  pod-security.kubernetes.io/enforce=baseline

That does not change anything. It asks the API server what violations it would report if the label were applied.

Then stage audit and warn cluster-wide:

kubectl label --overwrite ns --all \
  pod-security.kubernetes.io/audit=baseline \
  pod-security.kubernetes.io/warn=baseline

Or, if your end state is more ambitious, go straight to the pattern that tends to work well in practice:

enforce=baseline
warn=restricted
audit=restricted

That gives you signal without immediately creating an outage because some forgotten sidecar, root-running image, or hostPath mount was hiding in a namespace nobody had looked at for months.

You can also find namespaces that have no PSA labels yet:

kubectl get ns --selector='!pod-security.kubernetes.io/enforce'

Once the warnings are clean for a namespace, pin the version and turn on enforce there. Namespace by namespace is slower than one giant switch, but slower is exactly what you want when admission control is involved.

While the rollout is in flight, the API server's own metrics — pod_security_evaluations_total, pod_security_errors_total and pod_security_exemptions_total — are worth scraping. They tell you whether evaluation is happening where you think it is, and whether exemptions are starting to sprawl.

What a real rejection looks like

The good news about PSA failures is that the error message is the fix list.

Apply a plain pod into a namespace with enforce=restricted and you get something like this:

Error from server (Forbidden): pods "nginx" is forbidden: violates PodSecurity "restricted:latest":
allowPrivilegeEscalation != false (container "nginx" must set securityContext.allowPrivilegeEscalation=false),
unrestricted capabilities (container "nginx" must set securityContext.capabilities.drop=["ALL"]),
runAsNonRoot != true (pod or container "nginx" must set securityContext.runAsNonRoot=true),
seccompProfile (pod or container "nginx" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")

That message is unusually helpful by Kubernetes standards. Every violation names the exact field to set and the value it wants, so you can work through it like a checklist.

A minimal compliant pod looks like this:

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  securityContext:
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  containers:
    - name: nginx
      image: nginxinc/nginx-unprivileged:stable
      ports:
        - containerPort: 8080
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop:
            - ALL

Two quiet fixes are hiding in that example: the container-level security context, and the image itself. A lot of “hello world” manifests still assume a root-running image on port 80, which is a fine way to discover that restricted is not interested in your tutorial shortcuts.

The trap most teams hit once: Deployments apply, pods do not

This is the operational detail that makes or breaks a rollout plan.

warn and audit evaluate workload resources such as Deployment pod templates. enforce does not. enforce applies only to the resulting pod objects.

That means this can happen:

You apply a Deployment
Kubernetes accepts the Deployment
The ReplicaSet tries to create pods
The pods are rejected by PSA
You now have a deployment that looks “applied” but never becomes healthy

When that happens, the signal is in the ReplicaSet and the events, not in the initial kubectl apply output:

kubectl get rs -n payments
kubectl get events -n payments --sort-by=.metadata.creationTimestamp

If you remember only one rollout lesson from this post, make it this one. Admission on pod creation and admission on workload templates are not symmetrical.

The namespaces that should not be `restricted`

kube-system is the obvious example. CNIs, CSIs, node agents, and static pods routinely need things that baseline or restricted will reject: hostPath, hostNetwork, privileged containers, or other host-level access.

Be explicit about that. Label those namespaces privileged so the exception is documented in the cluster state rather than living as folklore.

Istio adds another common surprise — relevant if you run a mesh for zero-trust networking. With sidecar injection but without Istio CNI, the injected istio-init container needs NET_ADMIN and NET_RAW to set up traffic redirection. That is enough to fail baseline, not just restricted. The usual fix is to enable the CNI node agent and keep istio-system itself privileged.

Why version pinning matters more than it looks

A pinned PSA version is an operational control, not a documentation detail.

Kubernetes v1.34 added a baseline control that blocks the host field in httpGet and tcpSocket probes and lifecycle hooks. The reason is sensible: that field can be abused as an SSRF path through the kubelet. The operational consequence is that a cluster upgrade can start rejecting manifests that were fine the day before — if your labels track latest.

With version pinning, you choose when that tighter rule lands. With $pss still set to the minor version you were running before the upgrade:

kubectl label --overwrite ns payments \
  pod-security.kubernetes.io/enforce-version="$pss" \
  pod-security.kubernetes.io/warn-version="$pss" \
  pod-security.kubernetes.io/audit-version="$pss"

That makes upgrades much less dramatic. Upgrade the cluster first. Move the policy version when you are ready to deal with the findings.

The RBAC hole nobody should ignore

PSA policy lives on namespace labels. That means anyone who can update namespace labels can also weaken or remove enforcement.

Review that RBAC carefully. It is easy to focus on pod-creation rights and forget that update on namespaces is effectively policy-admin access.

There is a more central exemption mechanism through the API server AdmissionConfiguration, where you can exempt namespaces, usernames, or runtime classes. That is useful on self-managed control planes. On managed services such as EKS, GKE, and AKS, you usually do not get to edit that file at all, which means namespace labels are the real policy surface.

Where Pod Security Admission stops

PSA is the built-in floor. That is its job, and it is a good one.

There is no mutation, so no PSP-style defaulting is waiting to save a sloppy manifest. There is no custom rule language, and no way to express fine-grained per-workload exceptions inside one namespace. Once you want those things, you are into policy engines such as Kyverno or OPA Gatekeeper.

That is not a weakness so much as a division of labour. Use PSA to guarantee the cluster-wide minimum. Use a policy engine above it when you need mutation, richer exceptions, or application-specific constraints.

PodSecurityPolicy is gone. The sensible replacement is not to rebuild PSP in another tool and pretend nothing changed. It is to accept the smaller built-in model for what it is: predictable, cheap to operate, and strong enough to become the default floor across a cluster. That is a better trade than a perfect policy system nobody trusts enough to enable.