DEV Community: Core Solutions

GitOps with Helm for small teams: what

Billy Walker — Thu, 24 Sep 2026 17:34:34 +0000

Most small teams arrive at GitOps from the same place. Every service might already have a Helm chart, but production is whatever someone last ran helm upgrade with from their laptop, plus the --set replicaCount=4 somebody added during last month's incident, plus a values file that only exists in one engineer's home directory. Then they go looking for help and find a wall of machinery: app-of-apps layouts, promotion pipelines, multi-cluster generators, progressive delivery. If you are a team of three trying to stop deploys happening from somebody's laptop, that is not a realistic starting point.

For small teams, the part of GitOps that pays off immediately is the reconciliation loop. Most of the machinery built around it can wait.

Strip GitOps back to first principles and it is genuinely useful. The OpenGitOps principles give you four: desired state is declarative, versioned and immutable, pulled automatically, and continuously reconciled. Those get you reviewed change, a real audit trail, drift detection and rollback by git revert, and none of them needs an elaborate delivery platform.

If you already use Helm, most of the pieces are in place. What you need on top is small: charts and values files in Git, one reconciler to apply them, a values layout that keeps environments honest, and a secrets plan that is not reckless. This post takes each in turn, then covers the machinery you can leave alone and the sharp edges you will meet once the loop is running.

The useful bit of GitOps is boring in the best way

Once a tool such as Argo CD or Flux is watching a repository and rendering your charts from it, a few things improve straight away:

Change becomes reviewable. Every deploy is a commit to a chart or a values file, and usually a pull request.
Rollback gets simpler. In the happy path it is a git revert rather than a reconstruction job.
Drift becomes visible. When someone changes a live object out of band, the reconciler notices.
Cluster state becomes reproducible. You stop relying on memory and shell history to explain which values produced what is running.

That rollback point changes one Helm habit. Under GitOps, helm rollback stops being the tool for the job. On Argo CD there is no Helm release to roll back, and on Flux a manual rollback leaves the release out of step with what Git declares, so the controller upgrades it straight back. Revert the commit instead and let the loop do the deploy.

All of this is a meaningful operational upgrade at any size, and it lines up with DORA's research. Its continuous delivery capability lists version control for all production artefacts, and deployment automation, among the practices that drive continuous delivery, which in turn improves delivery performance. I would be careful not to overclaim here. DORA does not measure GitOps as a separate intervention with a tidy uplift figure. What it does back is the discipline GitOps happens to enforce well.

Start with the loop

If I were introducing GitOps to a team that already ships Helm charts, I would start with a setup that feels almost underwhelming:

charts and their values in Git, with environment differences as values files on one branch
one reconciler running in the cluster
pull requests as the only normal path into production
chart versions pinned, so the same commit always renders the same manifests
automated sync switched on, with pruning and self-heal treated as things the team earns

That last point deserves a closer look, because the two main tools handle it differently. Self-healing sounds great until an urgent out-of-band fix gets reverted mid-incident because Git still says otherwise. The controller is doing what you asked, not what you meant five minutes into an outage.

Argo CD leaves both sharp edges off by default. An Application with automated sync and nothing else set behaves like this:

syncPolicy:
  automated:
    prune: false
    selfHeal: false

With those settings, Argo CD syncs when a new commit lands. It does not delete resources that disappear from Git, and it does not revert a live change on its own: the drift shows up as OutOfSync for a human to deal with. Turn on selfHeal and prune once the team trusts the loop and, more to the point, once everyone has stopped fixing production by hand.

Flux splits the same decision across its two controllers, and the split catches Helm users out. A plain Kustomization re-applies what Git says on every interval and corrects any drift it finds, with no switch to turn that off. A HelmRelease is different: drift detection is opt-in, so by default a kubectl edit against a Helm-managed Deployment stays put until the next chart or values change triggers an upgrade. You choose the behaviour explicitly:

spec:
  driftDetection:
    mode: warn # report drift as an event; change to `enabled` to correct it

warn is the Flux equivalent of Argo CD with self-heal off, and a sensible place to start. The section on running the loop comes back to what to do when you need the reconciler to back off.

Argo CD vs Flux: pick by how your team works

People love turning this into theology. It doesn't need to be.

Flux and Argo, the project that includes Argo CD, reached CNCF Graduated status within a week of each other in late 2022. Either can be the right answer, and neither is a toy. The first difference is how people expect to interact with deployments.

Argo CD suits teams that want an app-centric control plane with a strong UI. You declare Application resources and get a visual model of sync status, health and drift, so people can see what is going on without living in the CLI.

Flux suits teams that prefer a composable set of controllers and an experience shaped around Git, the CLI and the Kubernetes API. It feels closer to assembling the controllers you need than to logging into a deployment console.

The second difference is specific to Helm, and it is worth knowing before you pick. Argo CD uses Helm only as a template engine: it runs helm template and applies the rendered manifests itself, so helm list shows nothing and there is no Helm release history in the cluster. Flux's helm-controller performs real Helm installs and upgrades, so releases, history and helm list all behave as they did before GitOps, and failed upgrades can be remediated with Helm's own rollback. Neither approach is wrong. If your team leans on Helm tooling and release history, Flux will feel familiar; if you would rather Helm stayed a rendering step and the Argo CD UI became the source of truth for what is deployed, Argo CD's model is simpler to reason about.

So my heuristic is mostly a social one:

pick Argo CD if people routinely want to look at deploy state and reason about it in one place
pick Flux if your team is happy in Git and the terminal, wants real Helm releases, and would rather compose small controllers than run a dashboard

The secrets section below adds one more technical tie-breaker. If you want to go further with Argo, we have a hands-on post on GitOps for Kubernetes with Argo CD.

One repo per service, chart included

Repository debates get oddly ideological. Keep it plain: each service is its own repo, and its Helm chart lives next to the code it deploys. The chart then versions with the application, so a change that needs a new environment variable and the code that reads it lands in one pull request.

payments-api/
├── src/
├── Dockerfile
└── chart/
    ├── Chart.yaml
    ├── values.yaml          # base: what every environment shares
    ├── values/
    │   ├── staging.yaml     # only what differs in staging
    │   └── production.yaml  # only what differs in production
    └── templates/

The layout mirrors how Helm already merges values: a base that every environment shares, and a thin file per environment that only says what differs. In Argo CD, each environment's Application points at the service repo's chart/ directory and names its environment file. The chart's own values.yaml is always the base, and anything in valueFiles is layered on top of it:

source:
  repoURL: https://github.com/example/payments-api.git
  targetRevision: v1.4.2
  path: chart
  helm:
    valueFiles:
      - values/production.yaml

Flux does the same with a HelmRelease whose chart comes from the service's GitRepository. Its valuesFiles replace the chart's default values and merge in order, with paths relative to the repo root, so list the base first:

spec:
  chart:
    spec:
      chart: ./chart
      sourceRef:
        kind: GitRepository
        name: payments-api
      valuesFiles:
        - ./chart/values.yaml
        - ./chart/values/production.yaml

Those Application and HelmRelease definitions are the only thing that isn't per service: a small deployments repo holding one per service and environment, which is also where the reconciler reads what runs where.

Argo CD's best-practices guide recommends keeping config in a separate repo from application source, and its reasons are worth handling rather than ignoring. A values-only change shouldn't trigger a full CI build, so path-filter the pipeline. CI shouldn't commit image tags back into the repo it builds from, or you get a build loop. And a code commit shouldn't reach production on its own, which is what the pinned targetRevision above is for: staging tracks main, production pins a release, and promotion is a reviewed pull request that bumps the pin.

A few opinions, stated plainly:

Use values files for environments, not branches. Branch-per-environment sounds tidy until promotion turns into cherry-picking and drift archaeology.
Keep environment files thin. If production.yaml is most of a copy of values.yaml, the base isn't doing its job, and the next shared change will land in one file and not the other.
Pin what production runs. Pin the service's targetRevision to a tag or commit, and pin upstream chart versions exactly rather than to a range such as 6.5.*. Let a bot such as Renovate raise the bumps as pull requests. A range means the same commit can render different manifests next week.

One Helm behaviour bites almost everyone who adopts this layout. Helm merges maps deeply, but it replaces lists outright: "scalar values and arrays are replaced, maps are merged". So a production.yaml that adds one environment variable to an env: list doesn't add it; it replaces every entry the base defined, and the base's variables silently vanish from production. Keep list-valued settings in one file, or have the chart accept a map and render the list from it.

Shared infrastructure follows the same rule: cert-manager, ingress and External Secrets are each their own chart, deployed before the services that use them. Your charts will create custom resources, such as a cert-manager Certificate or an ExternalSecret, and those only work once the CRDs exist and the controller behind them is actually running. It is also a Helm limitation: Helm never upgrades or deletes CRDs from a chart's crds/ directory, and its own docs suggest a separate chart for them. Flux handles the ordering with dependsOn, so a service's release waits until the infrastructure it needs reports ready. Argo CD uses sync waves, set with the argocd.argoproj.io/sync-wave annotation, which order resources within a single Application. Ordering whole Applications needs app-of-apps plus a custom health check for the Application kind, and that is one of the few good reasons to adopt app-of-apps early.

Secrets in GitOps: SOPS, Sealed Secrets or External Secrets

Every GitOps conversation eventually arrives at the part the tidy diagrams leave out, and with Helm the first place a secret leaks is a values file. "Just put them in Git" is not a serious answer. Git is durable, replicated, and very good at remembering things you wish it had forgotten.

The usual options:

SOPS encrypts the values in the files you commit, so diffs stay reviewable. Flux decrypts SOPS natively through spec.decryption on a Kustomization, and a HelmRelease can read the decrypted Secret as values through valuesFrom. Argo CD has no built-in support, so you add SOPS to its repo server with a plugin. That moves decryption into manifest generation, which Argo CD's own secret-management guide strongly cautions against, because generated manifests sit in plaintext in its Redis cache.
Sealed Secrets gives you a SealedSecret that is safe to commit and that only the controller in the target cluster can decrypt. It is easy to explain and a fine early move, but it is cluster-bound. By default the controller renews its sealing key every 30 days and keeps the old ones, so disaster recovery means backing up every secret labelled sealedsecrets.bitnami.com/sealed-secrets-key, and refreshing that backup after each renewal. Lose those keys along with the cluster and every SealedSecret in Git is ciphertext nobody can open.
External Secrets Operator keeps references in Git and the values in a real secrets manager such as Vault, AWS Secrets Manager or Azure Key Vault. Your chart templates an ExternalSecret instead of a Secret, and the values file only ever holds the key's path. It is often the cleanest long-term model, and our post on syncing secrets into Kubernetes with External Secrets walks through it. The trade-off is a hard runtime dependency on that external system, and on the operator itself: in 2025 the project paused releases for several weeks over maintainer burnout, until more maintainers joined. Anything on your runtime path deserves a look at who maintains it.

Argo CD's guide recommends populating secrets on the destination cluster, which is what the last two do. Put that together and my bias looks like this:

if you already have a cloud secrets manager, go straight to External Secrets Operator
if you don't and you run Flux, SOPS is the cleanest Git-native option
if you don't and you run Argo CD, start with Sealed Secrets and treat the key backup as part of the install

Picking the "wrong" tool is recoverable. The mistake that lingers is deferring the decision, migrating everything else, and leaving secrets as a loose end to tidy up later. Sort them out alongside the reconciler, because they are part of the same design.

The machinery to skip until it hurts

This is where small teams get talked into complexity they have not earned yet. Plenty of the canonical GitOps add-ons are real solutions, just to later problems:

app-of-apps sprawl solves large estates with many related deployments
ApplicationSet generators solve repetition across fleets of charts and clusters
promotion pipelines solve controlled movement across several environments
progressive delivery, with a tool such as Argo Rollouts, solves high-stakes releases where traffic shaping is worth another controller
multi-cluster GitOps solves hard isolation, geography, tenancy or blast-radius constraints

If those are not your current pressures, adopting them early is overhead you pay every week for a problem you don't have yet.

It is the same argument as our post on right-sizing platform architecture: do not borrow scale machinery before the scale arrives. GitOps has the trap every fashionable platform idea has. The sensible core gets wrapped in an architecture identity, and teams start buying components to prove they are doing it properly. You are doing GitOps properly if Git is the source of truth and a reconciler is continuously working to make the cluster match it.

Running the loop once it's live

The machinery can wait. These can't, because every team meets them within a few weeks of switching the reconciler on, however small the estate.

Pause the loop before you fix production by hand. During an incident the reconciler is on Git's side, not yours. In Flux, flux suspend helmrelease <name> (or flux suspend kustomization <name> for plain manifests) stops new revisions and drift correction, and the matching flux resume turns them back on. In Argo CD, argocd app set <app> --sync-policy none takes one Application off automated sync. There are two catches on the Argo side: an Application generated by an ApplicationSet ignores changes to its own sync policy, and a parent app with self-heal on can put a child's policy straight back. Whatever you change by hand, commit the same change to the values file before you resume, or the loop will quietly undo your fix. Put those commands in the incident runbook now, while things are calm.

Let the autoscaler own replicas. If a HorizontalPodAutoscaler manages a Deployment and your chart also renders spec.replicas, the two fight, and every sync resets the count the HPA chose. The chart that helm create scaffolds already handles this: its Deployment only renders replicas when autoscaling.enabled is false, so the fix is usually one value rather than a template change. Argo CD's best-practices guide and the Flux FAQ give the same advice: leave replicas out of what you apply. If a third-party chart insists on rendering it, Argo CD's ignoreDifferences on /spec/replicas hides it from the diff, but the sync still applies it unless you also set the RespectIgnoreDifferences=true sync option. On Flux, a HelmRelease takes driftDetection.ignore with paths: ["/spec/replicas"], which its docs suggest for exactly this case.

Preview what a pull request will actually render. With Helm, reading a values diff is not the same as knowing what the manifests will look like, because one value can fan out across a dozen templates. The cheapest check needs no cluster at all: run helm template with the base and environment files on your branch and on main, and diff the two outputs. For the live comparison, argocd app diff <app> --revision <branch> renders the chart and compares it with what is running, with one gap: it leaves Secrets out. flux diff kustomization is less useful here, because for a HelmRelease it shows the change to the HelmRelease object rather than to the rendered manifests. Running a live diff in CI means giving CI read access to the cluster, which is a real trade for a pull-based setup. Running it locally before you open the pull request costs nothing.

Make the loop react faster than it polls. By default Argo CD checks Git every two minutes plus up to a minute of jitter, and Flux fetches each GitRepository on whatever interval you give it. A merged pull request that appears to do nothing for a few minutes is usually waiting for the next poll. Both tools accept webhooks: Argo CD's API server takes Git webhook events directly, and Flux uses a Receiver from its notification controller. Keep polling as the fallback, so a missed webhook costs you minutes rather than a deploy.

Where to start

Pick one low-risk chart, ideally a stateless internal service, and move it to the base-plus-environment layout: values.yaml for what is shared, values/production.yaml for the handful of lines that differ. Put it under the reconciler with automated sync on and pruning off, and on Flux set driftDetection.mode: warn. Make a pull request the only way it changes for a fortnight, and keep a note of every drift the reconciler reports, whether that is Argo CD marking the app OutOfSync or Flux emitting a drift event on the HelmRelease. Every one of those is a hand-edit or a --set someone still reaches for, and each becomes either a runbook entry, a value that belongs in Git, or a field handed over to another controller. When a week goes by with nothing to explain, turn on self-heal and pruning, move the next chart across, and leave the rest of the GitOps catalogue on the shelf until a real problem asks for it.

For a small team, most of the effort in this post goes into standing the loop up rather than using it. That is the part we built Kupe Cloud to take away. It is our managed Kubernetes platform, and both halves of the setup are ready as soon as you spin up your first cluster. Argo CD is already running, with a project set up for your tenant, so once you connect a service repo laid out as above, its chart deploys as it is. Secrets work the same way from day one: you create a secret once in the console, it is stored encrypted in the platform's vault, and Kupe syncs it as a Kubernetes Secret into the clusters and namespaces you choose. That is the External Secrets model from earlier, with nothing extra to install or run. Everything else in this post still applies; you just start with the loop already running.

Falco on Kubernetes: install, tune, and route alerts

Billy Walker — Tue, 22 Sep 2026 18:28:51 +0000

You scan images in CI, sign them before deploy, and gate manifests at admission. Every one of those controls answers the same question: should this workload be allowed to start? None of them can tell you what it did at 02:00 once it was running. Someone opening a shell in a production pod, a binary that was never in the image suddenly executing, a process reading /etc/shadow for no good reason. Those are runtime events, and runtime is where Falco lives.

The catch is that Falco tutorials age faster than most. The project has dropped its legacy eBPF probe, its gVisor engine and its gRPC output in recent releases, deprecated the append: true rule syntax in favour of override:, and moved rule distribution onto OCI artifacts pulled by a sidecar. A lot of the snippets still ranking on the first page will not load on a current install. So this walkthrough sticks to what the current chart and docs actually do: install, trigger one stock alert, write one custom rule, and then spend most of the time on tuning, which is the part that decides whether Falco is still switched on in six months.

Where runtime detection fits

A Kubernetes security stack has layers, and each one answers a different question:

image scanning: what went into the artefact
signing and provenance: whether you trust where it came from
admission policy: whether the cluster should accept it
runtime detection: what the workload is doing now

The last layer is the one teams skip until something odd is already running in production. Falco fills it by watching syscalls from the kernel, matching them against a rules engine in userspace, and emitting an alert when a rule fires. If you want the eBPF side of that story in more depth, the Cilium networking post covers how the same machinery is used for the network path.

How Falco collects events, and where it differs from Tetragon

Search for "Falco vs eBPF" and you will find a lot of confused threads. Falco is not an alternative to eBPF. It uses eBPF as one way of collecting kernel events, and there are two supported drivers today:

modern_ebpf, a CO-RE probe embedded in the binary, which is what Falco itself defaults to
kmod, the kernel module, still supported and still the right answer on kernels the modern probe cannot run on

The Helm chart adds a third value, auto, which is its default. It tries the modern probe first and falls back to the kernel module. The older ebpf and gvisor driver kinds are gone: the chart will not even render if you pass either, which is a kinder failure than a pod that starts and silently sees nothing.

Tetragon, from the Cilium project, makes a different trade-off. It leans into in-kernel enforcement: a tracing policy can override a kernel function's return value or send SIGKILL to the offending process before it gets any further. Falco keeps the decision in userspace, with a mature rules engine, a maintained default ruleset, and a lot of flexibility around outputs and tuning. If your first question is "what happened inside that container?", Falco is the more natural place to start. If it is "stop that from ever completing", look at Tetragon.

Install Falco with the Helm chart

helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update
helm install falco falcosecurity/falco \
  --create-namespace \
  --namespace falco \
  --set falcosidekick.enabled=true \
  --set falcosidekick.webui.enabled=true \
  --set collectors.kubernetes.enabled=true

Those three flags are worth understanding rather than copying:

falcosidekick.enabled=true deploys Falcosidekick and switches Falco to JSON output over HTTP, so alerts can be forwarded to systems people already watch
falcosidekick.webui.enabled=true adds the web UI and a Redis instance behind it, which is handy while you are testing and worth turning off once alerts flow somewhere durable
collectors.kubernetes.enabled=true deploys the metadata collector and the k8smeta plugin, which adds the owning Deployment, ReplicaSet and Service to each alert. Without it you still get the pod name, namespace and labels, because Falco reads those from the container runtime directly

The chart also deploys two containers you did not ask for by name: a falcoctl init container that pulls the default rules as an OCI artifact, and a falcoctl sidecar that checks for a newer artifact once a week and swaps it in. That has consequences for tuning, which we will come back to.

If you need to be explicit about the driver, use one of the two current values:

--set driver.kind=modern_ebpf   # or driver.kind=kmod

Then watch the Falco container start. The pod has several containers now, so name the one you want:

kubectl -n falco logs ds/falco -c falco

You are looking for a clean driver load and a Loading rules from file line for each rules file. If the driver is wrong for the kernel underneath, this is where you find out.

Trigger one stock alert first

Before writing anything custom, prove the default path works. Start a throwaway pod:

kubectl run runtime-test --image=busybox --restart=Never -- sleep 3600

Then exec into it with a terminal attached:

kubectl exec -it runtime-test -- sh

That fires the stock Terminal shell in container rule. The -t is doing real work here: the rule checks that the shell has a TTY, so kubectl exec runtime-test -- sh -c 'id' without one will not trigger it. Check the Falco logs or the Falcosidekick UI and you should see the alert with container_id and container_name appended to the end of the output line. Those fields are not in the rule's own output string; the container plugin suggests them and Falco appends suggested fields automatically.

This is also the moment to reset expectations about volume. The stock falco_rules.yaml is a small, conservative file: 25 stable rules at the time of writing, every one enabled, none of them experimental. The noisy reputation comes from the incubating and sandbox rulesets, which the chart does not load by default, and from teams tuning carelessly once they do.

Write one custom rule

The simplest useful rule is one you can explain in a sentence: tell me if a miner-like binary starts inside a container. With the chart, custom rules go in the customRules value, and each key becomes a file under /etc/falco/rules.d/:

customRules:
  miners.yaml: |-
    - rule: Crypto miner process in container
      desc: Detect common miner binaries starting inside a container
      condition: >
        spawned_process and container and
        proc.name = anyof (xmrig, minerd, cryptominer)
      output: >
        Crypto miner process in container | user=%user.name
        process=%proc.name command=%proc.cmdline
      priority: WARNING
      tags: [container, malware, crypto-miner]

Two things about that condition. spawned_process and container are macros from the stock file, so you get exec events inside containers for free. And anyof is a comparator modifier from the current rule language: one field compared against a short list, without a chain of or clauses. It sits after the operator, with a space either side and the values in parentheses.

One trap that is not in the tutorials: proc.name is the kernel's process name, and the kernel truncates it to 15 characters. The stock rules file depends on this, which is why you will find entries like mysql_install_d in its lists. If your binary has a long name, match on proc.exepath or container.image.repository instead.

Load order matters. The default rules_files list reads falco_rules.yaml, then falco_rules.local.yaml, then everything in rules.d, and a file that overrides a stock rule has to load after it. Give the file a .yaml or .yml extension too, because anything else in that directory is ignored.

Before you apply, validate:

falco -V miners.yaml

That parses and compiles the rules file and exits, and the easiest place to run it is the same Falco image tag you deploy, so the engine version matches. It is worth the thirty seconds, because a rules error is fatal. Falco stops at the first file that fails to load, prints the reason, and exits non-zero, which on Kubernetes means the whole DaemonSet goes into CrashLoopBackOff over one bad line in one file. Warnings are logged and tolerated. Errors are not.

Tuning is where most Falco deployments succeed or fail

Install is the easy part. Tuning decides whether people trust the alerts next month, and the stock rules file is built for it in a specific way that most tutorials, including the official exceptions examples, obscure.

The fact the docs bury is that none of the stable stock rules ships an exceptions: block. The docs' canonical example appends values to an exception on Write below binary dir, and that rule lives in the sandbox ruleset, not the one you just installed. Copy it against a stable rule and the loader rejects it, because there is no existing exception to inherit fields from.

What the stable rules give you instead is a set of hooks with names starting user_, each defined as (never_true), plus empty image lists. Take Contact K8S API Server From Container, which fires the first time an operator or controller you run outside kube-system talks to the API server. Its condition ends with and not user_known_contact_k8s_api_server_activities, and that macro is yours to replace:

- macro: user_known_contact_k8s_api_server_activities
  condition: >
    (k8s.ns.name = argocd and
     container.image.repository = quay.io/argoproj/argocd)
  override:
    condition: replace

The same shape works for the other stable rules. Terminal shell in container has user_expected_terminal_shell_in_container_conditions. Read sensitive file untrusted has a read_sensitive_file_images list you can append to:

- list: read_sensitive_file_images
  items: [quay.io/argoproj/argocd]
  override:
    items: append

If you would rather use exceptions, you can. Appending a new named exception to a stable rule is supported as long as you supply fields alongside values:

- rule: Contact K8S API Server From Container
  exceptions:
    - name: argocd_controllers
      fields: [container.image.repository, k8s.ns.name]
      values:
        - [quay.io/argoproj/argocd, argocd]
  override:
    exceptions: append

Falco adds and not (container.image.repository = ... and k8s.ns.name = ...) to the condition for you. Either approach beats disabling the rule, because the rule is still right for every other workload.

When a rule really does not fit your threat model, turn it off explicitly:

- rule: Drop and execute new binary in container
  enabled: false
  override:
    enabled: replace

The override: block is the part older posts get wrong. It states, per key, whether you are appending to or replacing the stock definition. append: true still parses for now but is deprecated, so translate any snippet built around it before you paste.

One more thing the file already did for you: the API-server rule's k8s_containers macro exempts everything in kube-system. If it is firing, the workload is somewhere else.

Route alerts somewhere people already look

An alert that only exists in one DaemonSet's log stream changes nothing. Falcosidekick takes Falco's JSON output and fans it out to Slack, Loki, Elasticsearch, Splunk, Datadog, Alertmanager and a long list beyond that. The chart values map directly to its outputs:

falcosidekick:
  enabled: true
  config:
    slack:
      webhookurl: https://hooks.slack.com/services/XXXX/YYYY/ZZZZ
      minimumpriority: warning
    loki:
      hostport: http://loki-gateway.monitoring:80

There is no global minimum priority in Falcosidekick. Each output has its own minimumpriority, and its severity ladder spells the level informational rather than Falco's info, which catches people out when a filter silently matches nothing.

The pattern that holds up on platform teams is a split by urgency: Slack or Teams for the handful of rules that need a human now, a log backend such as Loki for searchable history, and the rest filtered out at the output. If Grafana is already your front door, the Grafana monitoring post covers getting Grafana itself stood up and the first dashboards in place.

Operating it after week one

The first three sections get Falco running. These are the three things that go wrong once it has been running for a while.

Your rules are floating. The falcoctl sidecar follows a major-version tag of the rules artifact and installs whatever is newest every 168 hours, into an emptyDir mounted over /etc/falco. Two consequences: an edited falco_rules.yaml inside the pod is overwritten on the next restart or the next follow, and a rule you tuned by name can change underneath you on a Tuesday. Keep every override in customRules, and pin falcoctl.config.artifact.install.refs and follow.refs to an exact artifact tag or an @sha256: digest once you are past the exploring stage.

You are dropping events and nobody knows. Falco emits an internal alert named Falco internal: syscall event drop when the kernel ring buffer overflows, but it is a debug-priority alert rate-limited to one message every 30 seconds. Raise Falco's minimum priority to cut noise, or set minimumpriority: warning on your Slack output, and it vanishes. The durable answer is metrics:

--set metrics.enabled=true \
--set serviceMonitor.create=true

That turns on Falco's Prometheus endpoint and a ServiceMonitor for it. The two series to graph against each other are falcosecurity_scap_n_drops_total and falcosecurity_scap_n_evts_total. If drops climb with load, the documented remedies are a bigger ring buffer via engine.modern_ebpf.buf_size_preset, fewer CPUs per buffer via engine.modern_ebpf.cpus_for_each_buffer, and base_syscalls to narrow what the driver captures. Which of those helps depends on your syscall mix, which is also why there is no universal "Falco overhead" figure worth quoting: it tracks what your workloads do, not a percentage.

You want to act, not just alert. Falco itself only detects. Falco Talon is the incubating response engine in the same ecosystem: it receives events from Falcosidekick's talon output and applies rules such as terminating the pod or labelling it for quarantine. Its Kubernetes actions need k8s.pod.name and k8s.ns.name among the event's output fields, which the stock rule outputs do not carry, so expect to append those to any rule you want it to act on. It is receiving commits but has had long gaps between tagged releases, so treat it as something to evaluate rather than a default, and keep the enforcement question in mind when you weigh it against Tetragon.

The failure you will hit first

Almost every first Falco tuning session ends one of two ways. Either the DaemonSet is crash-looping, in which case the Falco container's log has the offending file and line near the top and falco -V would have told you first. Or the pods are healthy and the override seems to do nothing. For the second, check three things before you touch the rule. Confirm the file landed in /etc/falco/rules.d/ with a .yaml extension and appears in the Loading rules from file lines. Check the rule name you are overriding still exists with that exact spelling in the ruleset the falcoctl sidecar just installed. Then check whether the workload is already exempt, the way kube-system is for the API-server rule. Usually the rule was fine and the plumbing was not.

CIS Benchmark for Kubernetes with kube-bench and Kubescape

Billy Walker — Thu, 17 Sep 2026 17:51:08 +0000

Most teams get told to “harden Kubernetes to CIS” long before anyone explains what happens after the first scan. You run the tool, it prints a wall of FAIL and WARN, half the findings turn out to be somebody else’s problem on a managed control plane, and the rest sit in a ticket queue until the next audit comes round. There is a narrower version that works. Use kube-bench for the host and control-plane checks only it can see, use Kubescape for the posture checks you want to keep running from the API side, and treat the CIS Kubernetes Benchmark as a triage tool rather than a moral score.

One thing to know before either tool runs: the scanners trail the benchmark. CIS revises the benchmark as new Kubernetes releases land, and each tool picks the revision up some time later, as a new profile that somebody has to write and merge. On a recently upgraded cluster there is a fair chance your scanner is running the closest older profile and hasn’t mentioned it. The result is still useful, but that caveat belongs in the first line of anything you hand to an auditor, and there is a quick way to check, which we’ll get to.

Two scanners, two views of the cluster

The short version:

kube-bench runs on a node. It reads config files, file permissions and the flags on running processes, then compares them with the CIS checks.
Kubescape talks to the API server. It evaluates the resources in your cluster, or the manifests in your repo, against frameworks such as NSA-CISA, MITRE ATT&CK and CIS.
They overlap on the policy-style checks, and that is about it.

That is why “kube-bench or Kubescape?” is the wrong question. Run only kube-bench and you get a point-in-time audit with no idea what changed last Tuesday. Run only the Kubescape CLI and you miss everything that depends on a file mode or a kubelet flag on a real node.

There is also one deliciously awkward detail to get out early. The standard kube-bench Job sets hostPID: true and mounts host paths such as /etc/kubernetes and /var/lib/kubelet, so the compliance tool needs an exemption from the restricted pod policy it is about to tell you to enforce. A host-level audit has to see the host, so plan for the exemption now rather than discovering it when admission rejects the pod.

What the CIS Kubernetes Benchmark is, and what it is not

The benchmark is a hardening checklist for cluster configuration. Its sections cover the control plane components, etcd, control plane configuration, worker nodes, and policies: RBAC, pod security, network policies and secrets. Every recommendation comes with an audit procedure and a remediation, which is what makes it automatable.

What it leaves out is where people over-read the score. A good CIS result tells you the configuration is in better shape than it was. It does not tell you:

whether workloads behave safely at runtime
whether your images are full of known CVEs
whether your admission policies are broad enough for your actual risk model
whether the control plane you rent from a provider was set up the way you would have done it

So treat the benchmark the way you would treat a good linter: very useful, occasionally annoying, and dangerous only when somebody mistakes it for a complete security strategy. If a score is going in front of leadership, put that caveat on the same slide. “100% CIS” and “secure” are different claims.

Start with kube-bench, on the nodes

kube-bench is an open-source project from Aqua Security, and it does one blunt thing well: read the files and flags that exist on the host and compare them with the benchmark profile it knows about.

The normal entry point is the Job manifest in the root of the repo. On a self-managed cluster, apply it and read the logs:

kubectl apply -f https://raw.githubusercontent.com/aquasecurity/kube-bench/main/job.yaml
kubectl logs job/kube-bench

Every check comes back in one of four states. PASS and FAIL are what they sound like. WARN means kube-bench could not make the call for you, either because the benchmark marks the check as manual or because the check is unscored. INFO is a section header or a check you skipped. The habit to avoid is reading WARN as “fine”. In the vanilla profile nothing in the policies section is scored, so nothing in it can ever show as FAIL, and that section holds the RBAC and pod security checks you probably care about most.

The Job needs hostPID and that stack of read-only host mounts because kube-bench is looking at the node directly rather than asking the API server to summarise it. You can see the same assumption in the container one-liner from the upstream docs:

docker run --pid=host \
  -v /etc:/etc:ro \
  -v /var:/var:ro \
  -t docker.io/aquasec/kube-bench:latest

That is how it catches things an API-only scanner cannot. It is also why the Job belongs in its own clearly named namespace, labelled privileged for Pod Security Admission, rather than squeezed under your strictest workload policy. One more practical note: to audit control-plane nodes the pod has to be scheduled onto them, which means adding a nodeSelector and tolerations to the manifest first.

Benchmark lag is normal, so check which profile ran

The easiest way to confuse yourself is to assume the latest kube-bench means the latest CIS revision. kube-bench maps your Kubernetes version to a benchmark profile using a table in its config, and the platforms page in the docs shows the same mapping. When your cluster is newer than anything in that table, kube-bench walks the minor version down until it finds a match and runs that profile instead. There is no error, and at the default log level there is no warning either.

The JSON output is where you catch it:

kube-bench run --targets node,policies --json

Each block carries a version field, which is the profile that ran, and a detected_version field, which is the Kubernetes version kube-bench found. Compare the pair with the platforms table before you write policy around the result.

Once you know which profile you want, pin it:

kube-bench run --benchmark <profile> --targets node,policies

Here <profile> is a name from the “kube-bench config” column of that table. Pinning can’t give you a newer benchmark than the tool ships. What it buys is a result you can describe accurately, and a scheduled run that won’t switch profile underneath you on the day somebody bumps the image tag.

The same table matters if you are not on vanilla Kubernetes. k3s, RKE2, OpenShift and the managed services each have their own profiles, because the vanilla one looks in paths those platforms do not use and rewards you with a page of false failures.

One finding, end to end

Reports are easy to admire from a distance, so take one finding all the way through. In the current vanilla profiles, check 5.2.3 is “Minimize the admission of containers wishing to share the host process ID namespace”. Four questions get you from the line in the report to a decision:

Which check fired, and in which profile? IDs move between profiles. The same control is 4.2.2 in the EKS profile, which is one more reason to pin.
What did kube-bench run to decide? For this check, a kubectl loop over every pod in every namespace that reads .spec.hostPID. It reports pods that share the host PID namespace today, and never asks whether a policy would stop the next one.
Is this a real gap, an intentional exception, or the platform’s business?
What is the narrowest fix that changes the risk, not just the score?

The first answer matters more than it looks. The vanilla profile treats this check as unscored, so a failure prints as WARN. The EKS profile treats it as automated and scored, so the same pods print as FAIL. The headline changes with the profile; the risk does not.

The benchmark’s remediation text is “Add policies to each namespace in the cluster which has user workloads to restrict the admission of hostPID containers”, and on a current cluster the policy in question is a Pod Security Admission label. The baseline level already forbids host namespaces, so you do not need restricted to clear this finding. Ask the API server what would break before you enforce anything:

kubectl label --dry-run=server --overwrite ns --all \
  pod-security.kubernetes.io/enforce=baseline

A server-side dry run changes nothing, and it prints a warning for every namespace with pods that would violate the level (any baseline violation, so host networking and hostPath volumes show up too). The namespaces it names are your candidate exceptions: usually a node exporter, a logging or security agent, and kube-bench itself. From there the fix is a split model:

keep most namespaces on baseline or restricted
carve out one clearly named namespace for infrastructure that needs host access
write down why that namespace exists and who can deploy into it
re-run only the relevant check, so you know you changed the right thing

kube-bench run --targets policies --check 5.2.3

Expect the check to stay at WARN, because the pods in your exception namespace still have hostPID: true. That is correct. What has changed is that you can now explain every pod behind it, and an explained exception is what an auditor is looking for.

Managed clusters: part of the benchmark is out of reach

The kube-bench docs are direct about this: “It is impossible to inspect the master nodes of managed clusters, e.g. GKE, EKS, AKS and ACK”. That sentence should change how you describe the result. On a managed control plane, you are auditing:

the worker nodes you can reach
the policy and workload checks that are visible from inside the cluster
whatever managed-service profile kube-bench ships for your platform

The project ships separate manifests such as job-eks.yaml and job-gke.yaml for exactly this reason. Running kube-bench on EKS, for example, you cannot schedule a pod onto the control-plane nodes, so the master checks are simply unavailable.

Your compliance story on a managed platform is therefore partly technical and partly contractual. You verify what you can, and for the control plane you lean on the provider’s documentation, attestations and shared-responsibility model. Say so in the report: “out of our reach, covered by the provider’s attestation” is accurate, and easier to defend than a suspiciously complete set of green ticks.

Kubescape for the part that keeps running

Kubescape is a CNCF project and it now covers a lot more than compliance: misconfiguration scanning, image scanning, admission control and runtime detection all live under the same name. The slice that matters here is narrower: the API-side posture layer that sits next to kube-bench.

The CLI is straightforward:

kubescape list frameworks
kubescape scan framework nsa

Run them in that order. The CIS framework IDs have the benchmark revision baked into the name, so an ID copied from an old tutorial may no longer exist in the version you installed. nsa and mitre are stable names; for CIS, list first and use what is actually there.

For CI or a recurring check, the threshold flag is the practical one:

kubescape scan framework nsa --compliance-threshold 80

Below the score you set, the command exits with code 1. That is usually enough to stop “we should probably look at this later” from becoming a permanent state. Pick the number from a real baseline run, though. A threshold the cluster can’t currently meet tends to get switched off rather than fixed.

Kubescape vs kube-bench: which question goes where

Reach for kube-bench when the question is:

are kubelet and control-plane settings aligned with the benchmark?
do file permissions and host-level configuration match the guidance?

Reach for Kubescape when the question is:

what is the posture of the workloads and resources in this cluster right now?
can I run this in CI against manifests, before anything reaches a cluster?

The numbers won’t line up, because the two tools count different things. Where they do overlap, use it as a cross-check: if both are unhappy about the same class of policy gap, that finding goes to the top of the pile.

The in-cluster path is where Kubescape starts paying for itself

The Kubescape operator is the natural next step once you are past one-off CLI scans, and the install is refreshingly ordinary:

helm repo add kubescape https://kubescape.github.io/helm-charts/
helm upgrade --install kubescape kubescape/kubescape-operator \
  --namespace kubescape \
  --create-namespace

Two details in the chart’s values are easy to miss. Out of the box, configuration scans run on a schedule; the change-driven rescans that most people picture when they hear “continuous” sit behind capabilities.continuousScan, which is disabled by default. And the operator’s node agent mounts the host filesystem read-only, so the kubescape namespace needs the same Pod Security exception as kube-bench. Your one documented exceptions namespace becomes two, which is still a list short enough to defend.

A minimal recurring setup, and where to start

If you want the smallest loop that is still worth having, I would keep it to this:

kube-bench on a schedule, as a CronJob with the profile pinned, for the host-level checks only it can see
Kubescape in CI or in-cluster for the API-side posture checks
one threshold that fails something real, instead of a dashboard with no owner
one exception path for justified infrastructure privileges, documented and reviewed

As for where to start: run the kube-bench Job once, and before you fix anything, sort every FAIL and WARN into three piles: real fixes, documented exceptions, and the provider’s side of the line. That sorted list is a more useful deliverable than a score, and it is what the recurring scans then keep current.

Kubernetes is not a platform: it is a toolkit

Billy Walker — Wed, 09 Sep 2026 09:40:15 +0000

A team says it has “built a platform” because it runs Kubernetes. Then a developer tries to ship a service and still has to assemble a Deployment, Service, HorizontalPodAutoscaler, NetworkPolicy, PodDisruptionBudget, ServiceAccount, and half a dozen bits of environment-specific glue by hand. That is the category error in one scene. Kubernetes is the substrate. The platform is still on the backlog.

The Kubernetes project makes the same point in its own overview docs. Under the heading “What Kubernetes is not”, it says Kubernetes is “not a traditional, all-inclusive PaaS” and that it provides “the building blocks for building developer platforms”. Building blocks, from the people who maintain them. So when you hand product teams raw kubectl, raw YAML and a wiki full of tribal knowledge, what you have handed over is a very capable control plane and a pile of homework.

Kubernetes is a very good substrate

Kubernetes earns its place in all this. It gives you strong primitives and a programmable control plane, and that combination is hard to come by anywhere else.

Concretely:

declarative desired state
reconciliation loops
scheduling and placement
service discovery
rollout machinery
extensibility through CRDs and controllers

Which is a serious foundation, and it explains why Kubernetes keeps ending up in the middle of internal platforms.

Foundations and finished products sit in different categories, though. Postgres is a foundation; the finance dashboard someone logs into every morning is the product built on it. A cluster running important workloads sits on the foundation side of that line, however well run it is.

This is where a lot of platform work goes sideways. Infrastructure teams do the hard bit: secure the cluster, patch it, make it observable, get some governance around it. Somewhere in the following six months the organisation starts calling the cluster “the platform”, on the assumption that a working substrate produces a usable product on its own. It never has.

A platform is a product with users

The CNCF Platforms white paper uses much stricter language than most teams do in everyday conversation. It defines a platform as an integrated collection of capabilities defined and presented according to the needs of the platform’s users, and it calls out interfaces such as web portals, project templates, and self-service APIs.

“We run Kubernetes” clears none of that bar. The platform is the opinionated layer above it, the one that turns raw technology into something people can use safely and repeatedly.

Usually that means some combination of:

self-service interfaces so teams can create or change what they need without waiting on tickets
paved roads so the safe path is the easy path
policy and defaults so teams do not need to rediscover every security and reliability lesson themselves
documentation and examples that reflect how the organisation actually expects software to be shipped
support boundaries so people know what is standard, what is allowed, and what will get help at 02:00

The last one does more work than it looks. A platform is partly software and partly promises. An interface where no one can tell you the support status is a hopeful abstraction: teams route around it the first time it surprises them, and now you maintain two paths instead of one.

The category error shows up as cognitive rent

When teams say “Kubernetes is our platform”, they usually mean “Kubernetes is where our applications run”. The second claim is easy to earn; the first one is a product commitment.

There is a test for which one you have: what does a product engineer still need to understand to ship a normal service safely? If the answer runs to “quite a lot of Kubernetes”, you are still handing over the substrate.

That usually means developers still need working knowledge of things like:

how requests and limits interact with scheduling
when an HPA will or will not behave the way they expect
why a missing PodDisruptionBudget only hurts during maintenance windows
how NetworkPolicy defaults work in their cluster
which ingress, gateway, or certificate path is considered standard
which bits of YAML are mandatory because the platform has no stronger abstraction yet

Developers learning infrastructure is fine, and plenty of them should. The trouble starts when shipping an ordinary service requires them to learn the platform team’s entire problem space.

That is the bill for confusing a toolkit with a platform. The complexity stays exactly where it was; it just gets paid for by every team that touches it, in small instalments, every time they deploy.

This is why the Team Topologies framing still holds up. A platform is valuable in proportion to the cognitive load it takes off the teams consuming it. Ask developers to become part-time Kubernetes operators and you have moved that load rather than absorbed it.

A thin platform still counts

One reason teams overstate what Kubernetes gives them is that they imagine the alternative must be some giant internal developer portal project. If they are not ready to build Backstage templates, service catalogs, policy packs, and self-service workflows, they decide the cluster itself must count as the platform. Most of the useful work lives in the gap between those two options.

Team Topologies has a saner idea: the thinnest viable platform. On the Team Topologies TVP page, Matthew Skelton says that this can be “just a wiki page” if that is all you need at your current scale. That line keeps the argument grounded. What qualifies something as a platform is whether it gives your developers a clearer, safer, more repeatable way to get work done. Scale decides how thick it has to be; nothing else does.

Sometimes the thinnest useful platform is:

one golden path for a standard web service
a starter template with the right defaults already wired in
one supported CI path
one documented way to expose a service
one place to request a database or secret with known guardrails

That is still platform work, and at most organisations it is the right platform work to do first.

Starting small is fine. The mistake is skipping the product layer entirely and calling the control plane beneath it finished.

What a real platform adds on top of Kubernetes

If Kubernetes is the substrate, what does the platform layer actually contribute?

At minimum, it should remove repeated decisions that your teams do not benefit from making over and over.

A decent internal platform usually adds:

An opinionated service shape. New services start from a template or scaffold that already includes the organisation’s baseline for probes, resources, identity, networking, and delivery.
A supported path to production. One default route that the platform team actively maintains, and that most services can take without needing a conversation first.
Guardrails with teeth. Policy, admission control, CI checks, and defaults that catch the boring mistakes before they get merged.
Self-service for common needs. Databases, secrets, DNS, certificates, queues, environments, or delivery workflows that do not require a human to hand-hold every request.
A product experience. Good docs, clear ownership, visible support boundaries, and feedback loops so the thing improves instead of calcifying.

Notice what is missing from that list: “owning a cluster”. Owning a cluster is an operational capability, and platform engineering usually needs one. It buys you the right to start.

Use the maturity model as a self-check

The CNCF Platform Engineering Maturity Model is useful here because it stops the conversation from collapsing into slogans.

It describes four levels — Provisional, Operationalized, Scalable, and Optimizing — and looks across five aspects: investment, adoption, interfaces, operations, and measurement.

That works better as a diagnostic than asking whether you “have a platform”. Almost every team is somewhere in the middle of that grid, and the useful question is which cell they are in and which one they want to be in next.

A few blunt checks help:

Interfaces: do teams have a real self-service interface, or are they still stitching together YAML and Slack messages?
Adoption: do product teams voluntarily use the paved road because it is better, or because they were told to?
Operations: is there a supported day-two story behind the abstraction, or only a shiny day-one story?
Measurement: can you show that the platform shortens delivery loops or reduces support load, or is that still mostly assumed?
Investment: is there sustained ownership, or is the “platform” just a side project orbiting the cluster team?

Answer those five properly and the picture resolves fast. A team with a well-run cluster, ad-hoc interfaces and adoption it has never measured is doing valuable infrastructure work. The platform sits a column to the right of where that team is standing.

Where the finish line actually is

The finish line is a state where most teams can ship a service without writing a production-grade manifest from scratch.

You still want depth somewhere. Strong operational understanding in the platform team, and enough shared literacy that developers can reason about what runs underneath their code — that is a real asset, and it is a different thing from requiring it of everyone before they are allowed to deploy.

Developers should be able to use the platform the way you hope customers use any good product — confidently, and without carrying a mental model of its internals around with them.

Calling the substrate the platform has a specific cost. It lets you declare victory early, and after that every improvement gets spent on the cluster rather than on the people using it.

If you want one thing to do this quarter: take the service shape your teams create most often and give it a single supported path — a template, a sane set of defaults, one documented route to production. Then time how long it takes somebody outside the platform team to get from an empty repository to a running service, before and after. If that number holds steady, you built the abstraction for your own convenience.

Helm charts are technical debt: blame the templating model

Billy Walker — Mon, 31 Aug 2026 08:03:53 +0000

The pull request is four lines of template, adding an optional nodeSelector to an internal chart. CI has already done its job properly: helm lint passed, the chart rendered against every environment's values file, kubeconform validated each manifest that came out, and the snapshot diff of the rendered output is sitting in the check run waiting to be read.

And you still cannot say, from the diff, what this change does in staging without going and reading three other templates first.

Nobody files that as technical debt. It goes in the mental folder marked “charts are a bit fiddly”. But if your team maintains a dozen first-party charts, that tax gets levied on every review, and no amount of pipeline pays it off, because the pipeline was never the thing that was broken.

This is not a “Helm was always bad” post. Helm solved a real problem, and most teams running Kubernetes are better off with it than without it. The rot is somewhere more specific: Helm renders Kubernetes manifests by running Go's text/template over YAML as strings, and once you are authoring charts rather than just installing them, that model generates work forever.

That model has survived every major version so far, Helm 4 included. If your chart estate already feels brittle, the next release will not make it less so, and I will come back to why.

Helm won for good reasons

Helm is a CNCF graduated project and the default packaging story in Kubernetes, and it deserves to be:

install, upgrade, rollback and release history in a single tool
a standard way for vendors and open-source projects to ship applications
OCI registry support that slots into existing GitOps workflows
ecosystem gravity, which by now matters nearly as much as technical merit

Nobody sensible is still having the “should we use Helm at all?” argument. If you are installing Grafana, or External Secrets Operator, or almost anything with a healthy community behind it, the chart is the path of least resistance and you should take it.

The trouble is that the same tool that consumes packaged software so well also makes it very easy to start writing your own. One chart for the internal API. One for the batch worker. Then a shared library chart, because three of them had grown the same _helpers.tpl. At some point you stopped being a Helm user and became the maintainer of a templating system. That is a perfectly reasonable thing to be, as long as it was a decision rather than a drift.

The debt is in the rendering model

Helm templates know nothing about Kubernetes objects. They emit text that happens to be YAML.

Here is the shape of it, lifted from roughly every deployment.yaml in every internal chart:

# templates/deployment.yaml
spec:
  template:
    spec:
      {{- with .Values.nodeSelector }}
      nodeSelector:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      containers:
        - name: {{ .Chart.Name }}
          image: {{ .Values.image.repository }}:{{ .Values.image.tag }}

That 8 is load-bearing. nodeSelector: sits at column six, so its keys have to land at column eight, and nindent exists precisely because chart authors need to re-indent embedded blocks constantly. Write nindent 6 by mistake and you have not made a formatting error. You have set nodeSelector to null and promoted its contents to siblings of containers, and the render will print that quite happily, because it is still a well-formed YAML document. Nothing in the YAML layer objects. It takes schema validation downstream, kubeconform -strict in the pipeline or a server-side dry run, before anything flags it. That is a lot of machinery to need in order to catch a two-space mistake.

Two spaces. Helm's own template best practices page walks through indentation, whitespace chomping and generated-output formatting, and concedes the underlying problem in passing: “YAML is a whitespace-oriented language.” Full marks for honesty, but that is a peculiar property for an abstraction layer to have.

All of which is the cheap version of the problem, and it is cheap precisely because a machine can express it. Schema validation catches a wrong shape, and it catches it in seconds. What no pipeline catches is a chart whose right shapes have become impossible to hold in your head.

Once your abstraction works on strings rather than typed objects, a few things follow:

indentation becomes part of correctness, not just tidiness
a branch that renders the wrong shape still looks plausible in review
values become an API surface long before anyone has designed them like one
refactoring is hard, because the behaviour lives across helpers, partials and value conventions rather than in a schema your tooling understands

Disciplined teams are not exempt from this. They often accumulate more chart machinery, because they build helpers and layering and conventions to contain the complexity. That containment is genuinely useful, and it is also the tell: you are building scaffolding around a model that produces complexity faster than you can absorb it.

How it shows up on day two

You rarely spot chart debt by reading a chart. You spot it in the changes that need far more scrutiny than their diff size suggests they should.

`values.yaml` turns into an untyped product surface

values.yaml is the front door to your chart, which is convenient right up until you realise you have shipped a long-lived API with no type system behind it.

A values.schema.json belongs in every first-party chart, and Helm will enforce it on install, upgrade, lint and template. But the schema file is optional, so it is opt-in per chart and easy to leave behind on the one chart nobody has revisited, and it describes structure rather than behaviour. A value can be structurally perfect and still produce nonsense once it has threaded its way through conditionals, helper templates, defaults and subchart overrides.

Then another team starts depending on a value name. Renaming a field stops being a cleanup and becomes a migration, complete with a deprecation window and a note in the release notes that nobody reads.

The source is harder to review than the output

A rendered manifest can be entirely ordinary while the logic that produced it is anything but.

Humans review the chart source. A pull request that flips a condition or threads one more flag through three templates is not asking “is this Deployment valid?”. It is asking “which combinations of values, defaults and includes now produce a different document?”, and that question has no diff you can look at.

Which is why the gates all end up pointed at the rendered output rather than the source. Those gates are worth building, and I will come back to ours. They are also a fair signal about the model: you test the output because nobody can reliably reason about the input.

Subcharts and value plumbing sprawl quietly

Small Helm setups look tidy. Then the platform grows, one chart wraps five subcharts, globals appear, child values get overridden from the parent, and a flag in one place changes behaviour three templates away.

Some of those values exist because users asked for them. Some exist because a subchart expects them. Some exist because somebody needed a workaround two years ago and nobody has ever felt brave enough to delete it. That is the point where a chart stops being a packaging unit and starts being institutional memory encoded as YAML conventions.

Helm 4 improved Helm without changing this

This is where lazy anti-Helm takes get sloppy, so let me be precise about it.

Helm 4.0.0 shipped in November 2025 and it is a genuine improvement: server-side apply, a redesigned plugin system with WebAssembly plugins, resource watching built on kstatus, post-renderers as plugins, reproducible chart archives. If you run Helm heavily, upgrade.

All of that is release machinery, though. Rendering still goes through Go's text templating, values.yaml is still the primary interface, and the combinatorial growth of flags and helpers in your own charts remains entirely your problem. Helm 4 made Helm better at the job Helm is good at. Chart authoring was never on the list.

A version bump on its own will not retire this argument, so here is what would: a release that swaps text templating for a typed object model, or one that makes schema validation mandatory instead of opt-in. If you are reading this a few releases later, that is the thing to go and check. Short of it, the version numbers in this post move and the point does not.

What the alternatives actually change

Different tools replace different parts of Helm, which is where most “Helm alternatives” posts go wrong. There is no swap that hands you better authoring, better packaging, better release management and better ecosystem compatibility all at once.

Tool	What it replaces	What you still need
Kustomize	Templating, for the overlay case	Packaging, versioning, release history
Timoni	The authoring model, with CUE typing	Ecosystem gravity, and patience while it is pre-1.0
cdk8s	Templating, with a real programming language	Packaging, plus a language runtime you now own
Jsonnet / Tanka	Templating, with a compact composition syntax	Types, and a way to stop dynamic behaviour relocating complexity
KCL	Templating, with a typed config language	Ecosystem gravity, and a team willing to learn a new language
Score	The developer-facing values surface	A platform implementation underneath that satisfies the spec

Kustomize is the one I reach for most often outside Helm, largely because it already ships inside kubectl and patches structured YAML instead of templating arbitrary strings. If your real problem is “I have plain manifests and need per-environment overlays”, it removes a surprising amount of cleverness for almost no adoption cost. Expecting it to do packaging too is how people end up disappointed by it.

Timoni is the most interesting answer to “what if we fixed the authoring model?”. CUE, real typing, modules distributed as OCI artifacts: conceptually most of what people are asking for. The caveat is maturity, and it is still on its 0.x line (v0.29.0 as I write this), so go and check where it has got to before committing. A 1.0 would change the calculation considerably more than another point release will.

The code-generation family trades typing for operational surface. cdk8s (joined the CNCF Sandbox in 2020) gives you real programming languages, Jsonnet with Tanka gives you a compact composition language, and KCL (joined in September 2023) gives you an explicitly typed configuration language. In exchange you own a language runtime, a build step, and an onboarding cost for everyone who ever touches a manifest. Good trade for a platform team with a large estate, poor trade for a team with six charts.

Score (joined in July 2024) sits one layer up. It is a workload specification, asking “what does this application need from the platform?” rather than “how do I render manifests for every deployment case?”. If your real complaint is that developers keep being handed an ever-growing values surface to fill in, that is the layer to attack. It still assumes a platform underneath that can satisfy it.

How we actually run this

We stayed on Helm, so it is only fair to say what containing the cost looks like day to day.

Most of our platform components are wrapper charts, and that pattern carries a lot of the weight. Chart.yaml declares a pinned dependency on the upstream chart, Chart.lock freezes what actually resolved, and our own templates/ directory layers on what upstream does not ship: the NetworkPolicies, the PrometheusRules, the RBAC that upstream leaves to you. Renovate raises the upstream bumps, and they do not land until CI is green.

Values sit in one place per chart. values.yaml carries the defaults every environment shares, and a values/ directory holds the environment-specific overrides beside it, one file each. CI then lints the chart once against every one of those files rather than once overall, so a change that only breaks one environment cannot ride in on another's values.

That catches everything Helm can be made to fail on. The quieter failure needs its own gate, because of a footgun worth internalising: Helm deep-merges maps, but it replaces lists outright. An environment file that overrides a list has to restate the entire list, so adding a component in one environment and forgetting another raises no error anywhere. Both files are valid. You just ship a shorter list to the environment nobody updated, and find out when something is missing. Nothing you can put in values.yaml expresses “these lists must stay in step”, so a parity check in CI does it instead, and a red PR replaces copy-paste discipline. That is the scaffolding tax from earlier made concrete: a shell script exists because the type system does not.

None of this makes the templating model typed. It makes the cost visible and bounded, which is the achievable goal rather than the ideal one.

Where I would draw the line

Helm is not technical debt because it exists in your stack, and authoring your own charts is not a mistake. We do both, and given the same constraints we would do both again. A third-party chart you install, version and occasionally override is packaging doing exactly the job it is good at, and a first-party chart is a reasonable way to ship your own software into a cluster where everything else already arrives as one.

So I reach for Helm when:

I am consuming third-party software that already ships a chart and I have no wish to become its packager
I want release history, rollback and the install workflow Helm already gives me
the surrounding ecosystem expects Helm artifacts, GitOps tooling included
I am packaging our own components for a platform where every other component already arrives as a chart

What accrues interest is the estate you build afterwards, and how much of it you let grow without a gate on it. Our largest first-party chart carries a values.yaml north of 800 lines. That is not automatically a failure, but it is a number I would want any team to be able to quote about themselves, because the moment nobody can tell you what a chart's values surface costs, you have stopped managing it. Same instinct as right-sizing the rest of your platform: the tool is fine, and the real question is whether the estate around it still fits the team.

The alternatives are worth knowing even when you stay. We have stayed, because packaging gravity and release history are worth more to us today than typed authoring would be. If that trade flips, if the estate outgrows its gates or Timoni reaches 1.0 with real ecosystem behind it, the honest move is to notice rather than to defend the original choice.

You do not have to become anti-Helm to admit that the charts you maintain can rot. In most Kubernetes estates that is hardly a hot take. It is just what year two looks like.

PodSecurityPolicy replacement: Pod Security Admission

Billy Walker — Mon, 31 Aug 2026 07:57:40 +0000

If you still think of Pod Security Admission as the cut-down replacement for PodSecurityPolicy, that is exactly the mindset that makes rollouts harder than they need to be. It is less powerful. That is also why it is practical. You get three fixed profiles, three modes, a handful of labels, and an admission controller that is already in the API server. No custom policy language, no mutation, no controller to install, and far fewer ways to surprise yourself mid-change.

The trick is to use that simplicity properly. A safe rollout is not “turn on restricted everywhere and hope for the best”. It is version-pinned labels, audit and warn first, then namespace-by-namespace enforcement once you have seen what will break. Done that way, you can move from the old PSP world to a built-in control that teams will actually leave enabled.

What replaced PodSecurityPolicy

PodSecurityPolicy was deprecated in Kubernetes v1.21 and removed in v1.25. Pod Security Admission became generally available in the same release, which matters because there is nothing extra to deploy: if your cluster is current enough, the mechanism is already there.

The model is deliberately small:

Profiles: privileged, baseline, restricted
Modes: enforce, audit, warn
Version pinning: pod-security.kubernetes.io/<mode>-version, pinned to a Kubernetes minor version or latest

Those labels live on namespaces:

# pin policy to the minor version your cluster is running
pss="$(kubectl version -o json | jq -r '.serverVersion.gitVersion' | cut -d. -f1,2)"

kubectl label ns payments \
  pod-security.kubernetes.io/enforce=baseline \
  pod-security.kubernetes.io/enforce-version="$pss"

That simplicity is the point. PSP mixed policy with RBAC, allowed mutation, and had enough moving parts that many teams either avoided it or never quite trusted what a change would do. PSA is narrower. It sets a floor.

The mental model: three profiles, three modes, one version pin

You can think of PSA as a matrix:

Mode	What happens
`enforce`	Rejects a non-compliant pod creation request
`audit`	Allows it, but records an audit annotation
`warn`	Allows it, but shows a warning to the caller

And the profiles are ordered from least to most strict:

Profile	What it is for
`privileged`	System namespaces and workloads that genuinely need host access or privileged behaviour
`baseline`	A sensible minimum floor for ordinary application namespaces
`restricted`	Current pod hardening best practice

For most application namespaces, the safe starting pattern is:

kubectl label --overwrite ns payments \
  pod-security.kubernetes.io/enforce=baseline \
  pod-security.kubernetes.io/enforce-version="$pss" \
  pod-security.kubernetes.io/warn=restricted \
  pod-security.kubernetes.io/warn-version="$pss" \
  pod-security.kubernetes.io/audit=restricted \
  pod-security.kubernetes.io/audit-version="$pss"

That gives you a firm minimum bar in enforce, while warn and audit show you what stands between the namespace and restricted.

The version pin is not optional bureaucracy. If you leave a namespace on latest, any cluster upgrade can quietly tighten policy under your workloads. Pinning to the minor version you are actually running means policy changes happen when you decide to move the label.

What `restricted` actually checks

The easiest way to lose credibility on this topic is to hand-wave the restricted profile and get one detail wrong. Several of the checklists floating around in other posts are simply false — especially around runAsUser and seccomp.

Here is the practical checklist, straight from the Pod Security Standards:

Control	What `restricted` expects
Volume types	Only `configMap`, `csi`, `downwardAPI`, `emptyDir`, `ephemeral`, `persistentVolumeClaim`, `projected`, `secret`
Privilege escalation	`allowPrivilegeEscalation: false` on every container, including init and ephemeral containers
Non-root	`runAsNonRoot: true` at pod level or per container
UID	`runAsUser` does not need to be set, but if it is, it cannot be `0`
Capabilities	Drop `ALL`; the only allowed add is `NET_BIND_SERVICE`
Seccomp	Must be explicitly `RuntimeDefault` or `Localhost`; leaving it unset is a violation under `restricted`

Two of those rows are where the bad checklists go wrong. runAsUser is not required — the policy forbids UID 0, it does not make you hard-code some other UID just to satisfy the admission check. And seccomp cuts the other way: under baseline an unset profile is fine, but under restricted it is a violation in its own right, which is one of the reasons plain demo manifests still bounce when teams first try this.

The rollout that does not break production

The safest PSA rollout starts with a dry run, not an enforcement change — the official migration guide is built around the same idea.

First, test label application server-side across all namespaces:

kubectl label --dry-run=server --overwrite ns --all \
  pod-security.kubernetes.io/enforce=baseline

That does not change anything. It asks the API server what violations it would report if the label were applied.

Then stage audit and warn cluster-wide:

kubectl label --overwrite ns --all \
  pod-security.kubernetes.io/audit=baseline \
  pod-security.kubernetes.io/warn=baseline

Or, if your end state is more ambitious, go straight to the pattern that tends to work well in practice:

enforce=baseline
warn=restricted
audit=restricted

That gives you signal without immediately creating an outage because some forgotten sidecar, root-running image, or hostPath mount was hiding in a namespace nobody had looked at for months.

You can also find namespaces that have no PSA labels yet:

kubectl get ns --selector='!pod-security.kubernetes.io/enforce'

Once the warnings are clean for a namespace, pin the version and turn on enforce there. Namespace by namespace is slower than one giant switch, but slower is exactly what you want when admission control is involved.

While the rollout is in flight, the API server's own metrics — pod_security_evaluations_total, pod_security_errors_total and pod_security_exemptions_total — are worth scraping. They tell you whether evaluation is happening where you think it is, and whether exemptions are starting to sprawl.

What a real rejection looks like

The good news about PSA failures is that the error message is the fix list.

Apply a plain pod into a namespace with enforce=restricted and you get something like this:

Error from server (Forbidden): pods "nginx" is forbidden: violates PodSecurity "restricted:latest":
allowPrivilegeEscalation != false (container "nginx" must set securityContext.allowPrivilegeEscalation=false),
unrestricted capabilities (container "nginx" must set securityContext.capabilities.drop=["ALL"]),
runAsNonRoot != true (pod or container "nginx" must set securityContext.runAsNonRoot=true),
seccompProfile (pod or container "nginx" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")

That message is unusually helpful by Kubernetes standards. Every violation names the exact field to set and the value it wants, so you can work through it like a checklist.

A minimal compliant pod looks like this:

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  securityContext:
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  containers:
    - name: nginx
      image: nginxinc/nginx-unprivileged:stable
      ports:
        - containerPort: 8080
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop:
            - ALL

Two quiet fixes are hiding in that example: the container-level security context, and the image itself. A lot of “hello world” manifests still assume a root-running image on port 80, which is a fine way to discover that restricted is not interested in your tutorial shortcuts.

The trap most teams hit once: Deployments apply, pods do not

This is the operational detail that makes or breaks a rollout plan.

warn and audit evaluate workload resources such as Deployment pod templates. enforce does not. enforce applies only to the resulting pod objects.

That means this can happen:

You apply a Deployment
Kubernetes accepts the Deployment
The ReplicaSet tries to create pods
The pods are rejected by PSA
You now have a deployment that looks “applied” but never becomes healthy

When that happens, the signal is in the ReplicaSet and the events, not in the initial kubectl apply output:

kubectl get rs -n payments
kubectl get events -n payments --sort-by=.metadata.creationTimestamp

If you remember only one rollout lesson from this post, make it this one. Admission on pod creation and admission on workload templates are not symmetrical.

The namespaces that should not be `restricted`

kube-system is the obvious example. CNIs, CSIs, node agents, and static pods routinely need things that baseline or restricted will reject: hostPath, hostNetwork, privileged containers, or other host-level access.

Be explicit about that. Label those namespaces privileged so the exception is documented in the cluster state rather than living as folklore.

Istio adds another common surprise — relevant if you run a mesh for zero-trust networking. With sidecar injection but without Istio CNI, the injected istio-init container needs NET_ADMIN and NET_RAW to set up traffic redirection. That is enough to fail baseline, not just restricted. The usual fix is to enable the CNI node agent and keep istio-system itself privileged.

Why version pinning matters more than it looks

A pinned PSA version is an operational control, not a documentation detail.

Kubernetes v1.34 added a baseline control that blocks the host field in httpGet and tcpSocket probes and lifecycle hooks. The reason is sensible: that field can be abused as an SSRF path through the kubelet. The operational consequence is that a cluster upgrade can start rejecting manifests that were fine the day before — if your labels track latest.

With version pinning, you choose when that tighter rule lands. With $pss still set to the minor version you were running before the upgrade:

kubectl label --overwrite ns payments \
  pod-security.kubernetes.io/enforce-version="$pss" \
  pod-security.kubernetes.io/warn-version="$pss" \
  pod-security.kubernetes.io/audit-version="$pss"

That makes upgrades much less dramatic. Upgrade the cluster first. Move the policy version when you are ready to deal with the findings.

The RBAC hole nobody should ignore

PSA policy lives on namespace labels. That means anyone who can update namespace labels can also weaken or remove enforcement.

Review that RBAC carefully. It is easy to focus on pod-creation rights and forget that update on namespaces is effectively policy-admin access.

There is a more central exemption mechanism through the API server AdmissionConfiguration, where you can exempt namespaces, usernames, or runtime classes. That is useful on self-managed control planes. On managed services such as EKS, GKE, and AKS, you usually do not get to edit that file at all, which means namespace labels are the real policy surface.

Where Pod Security Admission stops

PSA is the built-in floor. That is its job, and it is a good one.

There is no mutation, so no PSP-style defaulting is waiting to save a sloppy manifest. There is no custom rule language, and no way to express fine-grained per-workload exceptions inside one namespace. Once you want those things, you are into policy engines such as Kyverno or OPA Gatekeeper.

That is not a weakness so much as a division of labour. Use PSA to guarantee the cluster-wide minimum. Use a policy engine above it when you need mutation, richer exceptions, or application-specific constraints.

PodSecurityPolicy is gone. The sensible replacement is not to rebuild PSP in another tool and pretend nothing changed. It is to accept the smaller built-in model for what it is: predictable, cheap to operate, and strong enough to become the default floor across a cluster. That is a better trade than a perfect policy system nobody trusts enough to enable.