DEV Community: Billy Walker

GitOps with Helm for small teams: what

Billy Walker — Thu, 24 Sep 2026 17:34:34 +0000

Most small teams arrive at GitOps from the same place. Every service might already have a Helm chart, but production is whatever someone last ran helm upgrade with from their laptop, plus the --set replicaCount=4 somebody added during last month's incident, plus a values file that only exists in one engineer's home directory. Then they go looking for help and find a wall of machinery: app-of-apps layouts, promotion pipelines, multi-cluster generators, progressive delivery. If you are a team of three trying to stop deploys happening from somebody's laptop, that is not a realistic starting point.

For small teams, the part of GitOps that pays off immediately is the reconciliation loop. Most of the machinery built around it can wait.

Strip GitOps back to first principles and it is genuinely useful. The OpenGitOps principles give you four: desired state is declarative, versioned and immutable, pulled automatically, and continuously reconciled. Those get you reviewed change, a real audit trail, drift detection and rollback by git revert, and none of them needs an elaborate delivery platform.

If you already use Helm, most of the pieces are in place. What you need on top is small: charts and values files in Git, one reconciler to apply them, a values layout that keeps environments honest, and a secrets plan that is not reckless. This post takes each in turn, then covers the machinery you can leave alone and the sharp edges you will meet once the loop is running.

The useful bit of GitOps is boring in the best way

Once a tool such as Argo CD or Flux is watching a repository and rendering your charts from it, a few things improve straight away:

Change becomes reviewable. Every deploy is a commit to a chart or a values file, and usually a pull request.
Rollback gets simpler. In the happy path it is a git revert rather than a reconstruction job.
Drift becomes visible. When someone changes a live object out of band, the reconciler notices.
Cluster state becomes reproducible. You stop relying on memory and shell history to explain which values produced what is running.

That rollback point changes one Helm habit. Under GitOps, helm rollback stops being the tool for the job. On Argo CD there is no Helm release to roll back, and on Flux a manual rollback leaves the release out of step with what Git declares, so the controller upgrades it straight back. Revert the commit instead and let the loop do the deploy.

All of this is a meaningful operational upgrade at any size, and it lines up with DORA's research. Its continuous delivery capability lists version control for all production artefacts, and deployment automation, among the practices that drive continuous delivery, which in turn improves delivery performance. I would be careful not to overclaim here. DORA does not measure GitOps as a separate intervention with a tidy uplift figure. What it does back is the discipline GitOps happens to enforce well.

Start with the loop

If I were introducing GitOps to a team that already ships Helm charts, I would start with a setup that feels almost underwhelming:

charts and their values in Git, with environment differences as values files on one branch
one reconciler running in the cluster
pull requests as the only normal path into production
chart versions pinned, so the same commit always renders the same manifests
automated sync switched on, with pruning and self-heal treated as things the team earns

That last point deserves a closer look, because the two main tools handle it differently. Self-healing sounds great until an urgent out-of-band fix gets reverted mid-incident because Git still says otherwise. The controller is doing what you asked, not what you meant five minutes into an outage.

Argo CD leaves both sharp edges off by default. An Application with automated sync and nothing else set behaves like this:

syncPolicy:
  automated:
    prune: false
    selfHeal: false

With those settings, Argo CD syncs when a new commit lands. It does not delete resources that disappear from Git, and it does not revert a live change on its own: the drift shows up as OutOfSync for a human to deal with. Turn on selfHeal and prune once the team trusts the loop and, more to the point, once everyone has stopped fixing production by hand.

Flux splits the same decision across its two controllers, and the split catches Helm users out. A plain Kustomization re-applies what Git says on every interval and corrects any drift it finds, with no switch to turn that off. A HelmRelease is different: drift detection is opt-in, so by default a kubectl edit against a Helm-managed Deployment stays put until the next chart or values change triggers an upgrade. You choose the behaviour explicitly:

spec:
  driftDetection:
    mode: warn # report drift as an event; change to `enabled` to correct it

warn is the Flux equivalent of Argo CD with self-heal off, and a sensible place to start. The section on running the loop comes back to what to do when you need the reconciler to back off.

Argo CD vs Flux: pick by how your team works

People love turning this into theology. It doesn't need to be.

Flux and Argo, the project that includes Argo CD, reached CNCF Graduated status within a week of each other in late 2022. Either can be the right answer, and neither is a toy. The first difference is how people expect to interact with deployments.

Argo CD suits teams that want an app-centric control plane with a strong UI. You declare Application resources and get a visual model of sync status, health and drift, so people can see what is going on without living in the CLI.

Flux suits teams that prefer a composable set of controllers and an experience shaped around Git, the CLI and the Kubernetes API. It feels closer to assembling the controllers you need than to logging into a deployment console.

The second difference is specific to Helm, and it is worth knowing before you pick. Argo CD uses Helm only as a template engine: it runs helm template and applies the rendered manifests itself, so helm list shows nothing and there is no Helm release history in the cluster. Flux's helm-controller performs real Helm installs and upgrades, so releases, history and helm list all behave as they did before GitOps, and failed upgrades can be remediated with Helm's own rollback. Neither approach is wrong. If your team leans on Helm tooling and release history, Flux will feel familiar; if you would rather Helm stayed a rendering step and the Argo CD UI became the source of truth for what is deployed, Argo CD's model is simpler to reason about.

So my heuristic is mostly a social one:

pick Argo CD if people routinely want to look at deploy state and reason about it in one place
pick Flux if your team is happy in Git and the terminal, wants real Helm releases, and would rather compose small controllers than run a dashboard

The secrets section below adds one more technical tie-breaker. If you want to go further with Argo, we have a hands-on post on GitOps for Kubernetes with Argo CD.

One repo per service, chart included

Repository debates get oddly ideological. Keep it plain: each service is its own repo, and its Helm chart lives next to the code it deploys. The chart then versions with the application, so a change that needs a new environment variable and the code that reads it lands in one pull request.

payments-api/
├── src/
├── Dockerfile
└── chart/
    ├── Chart.yaml
    ├── values.yaml          # base: what every environment shares
    ├── values/
    │   ├── staging.yaml     # only what differs in staging
    │   └── production.yaml  # only what differs in production
    └── templates/

The layout mirrors how Helm already merges values: a base that every environment shares, and a thin file per environment that only says what differs. In Argo CD, each environment's Application points at the service repo's chart/ directory and names its environment file. The chart's own values.yaml is always the base, and anything in valueFiles is layered on top of it:

source:
  repoURL: https://github.com/example/payments-api.git
  targetRevision: v1.4.2
  path: chart
  helm:
    valueFiles:
      - values/production.yaml

Flux does the same with a HelmRelease whose chart comes from the service's GitRepository. Its valuesFiles replace the chart's default values and merge in order, with paths relative to the repo root, so list the base first:

spec:
  chart:
    spec:
      chart: ./chart
      sourceRef:
        kind: GitRepository
        name: payments-api
      valuesFiles:
        - ./chart/values.yaml
        - ./chart/values/production.yaml

Those Application and HelmRelease definitions are the only thing that isn't per service: a small deployments repo holding one per service and environment, which is also where the reconciler reads what runs where.

Argo CD's best-practices guide recommends keeping config in a separate repo from application source, and its reasons are worth handling rather than ignoring. A values-only change shouldn't trigger a full CI build, so path-filter the pipeline. CI shouldn't commit image tags back into the repo it builds from, or you get a build loop. And a code commit shouldn't reach production on its own, which is what the pinned targetRevision above is for: staging tracks main, production pins a release, and promotion is a reviewed pull request that bumps the pin.

A few opinions, stated plainly:

Use values files for environments, not branches. Branch-per-environment sounds tidy until promotion turns into cherry-picking and drift archaeology.
Keep environment files thin. If production.yaml is most of a copy of values.yaml, the base isn't doing its job, and the next shared change will land in one file and not the other.
Pin what production runs. Pin the service's targetRevision to a tag or commit, and pin upstream chart versions exactly rather than to a range such as 6.5.*. Let a bot such as Renovate raise the bumps as pull requests. A range means the same commit can render different manifests next week.

One Helm behaviour bites almost everyone who adopts this layout. Helm merges maps deeply, but it replaces lists outright: "scalar values and arrays are replaced, maps are merged". So a production.yaml that adds one environment variable to an env: list doesn't add it; it replaces every entry the base defined, and the base's variables silently vanish from production. Keep list-valued settings in one file, or have the chart accept a map and render the list from it.

Shared infrastructure follows the same rule: cert-manager, ingress and External Secrets are each their own chart, deployed before the services that use them. Your charts will create custom resources, such as a cert-manager Certificate or an ExternalSecret, and those only work once the CRDs exist and the controller behind them is actually running. It is also a Helm limitation: Helm never upgrades or deletes CRDs from a chart's crds/ directory, and its own docs suggest a separate chart for them. Flux handles the ordering with dependsOn, so a service's release waits until the infrastructure it needs reports ready. Argo CD uses sync waves, set with the argocd.argoproj.io/sync-wave annotation, which order resources within a single Application. Ordering whole Applications needs app-of-apps plus a custom health check for the Application kind, and that is one of the few good reasons to adopt app-of-apps early.

Secrets in GitOps: SOPS, Sealed Secrets or External Secrets

Every GitOps conversation eventually arrives at the part the tidy diagrams leave out, and with Helm the first place a secret leaks is a values file. "Just put them in Git" is not a serious answer. Git is durable, replicated, and very good at remembering things you wish it had forgotten.

The usual options:

SOPS encrypts the values in the files you commit, so diffs stay reviewable. Flux decrypts SOPS natively through spec.decryption on a Kustomization, and a HelmRelease can read the decrypted Secret as values through valuesFrom. Argo CD has no built-in support, so you add SOPS to its repo server with a plugin. That moves decryption into manifest generation, which Argo CD's own secret-management guide strongly cautions against, because generated manifests sit in plaintext in its Redis cache.
Sealed Secrets gives you a SealedSecret that is safe to commit and that only the controller in the target cluster can decrypt. It is easy to explain and a fine early move, but it is cluster-bound. By default the controller renews its sealing key every 30 days and keeps the old ones, so disaster recovery means backing up every secret labelled sealedsecrets.bitnami.com/sealed-secrets-key, and refreshing that backup after each renewal. Lose those keys along with the cluster and every SealedSecret in Git is ciphertext nobody can open.
External Secrets Operator keeps references in Git and the values in a real secrets manager such as Vault, AWS Secrets Manager or Azure Key Vault. Your chart templates an ExternalSecret instead of a Secret, and the values file only ever holds the key's path. It is often the cleanest long-term model, and our post on syncing secrets into Kubernetes with External Secrets walks through it. The trade-off is a hard runtime dependency on that external system, and on the operator itself: in 2025 the project paused releases for several weeks over maintainer burnout, until more maintainers joined. Anything on your runtime path deserves a look at who maintains it.

Argo CD's guide recommends populating secrets on the destination cluster, which is what the last two do. Put that together and my bias looks like this:

if you already have a cloud secrets manager, go straight to External Secrets Operator
if you don't and you run Flux, SOPS is the cleanest Git-native option
if you don't and you run Argo CD, start with Sealed Secrets and treat the key backup as part of the install

Picking the "wrong" tool is recoverable. The mistake that lingers is deferring the decision, migrating everything else, and leaving secrets as a loose end to tidy up later. Sort them out alongside the reconciler, because they are part of the same design.

The machinery to skip until it hurts

This is where small teams get talked into complexity they have not earned yet. Plenty of the canonical GitOps add-ons are real solutions, just to later problems:

app-of-apps sprawl solves large estates with many related deployments
ApplicationSet generators solve repetition across fleets of charts and clusters
promotion pipelines solve controlled movement across several environments
progressive delivery, with a tool such as Argo Rollouts, solves high-stakes releases where traffic shaping is worth another controller
multi-cluster GitOps solves hard isolation, geography, tenancy or blast-radius constraints

If those are not your current pressures, adopting them early is overhead you pay every week for a problem you don't have yet.

It is the same argument as our post on right-sizing platform architecture: do not borrow scale machinery before the scale arrives. GitOps has the trap every fashionable platform idea has. The sensible core gets wrapped in an architecture identity, and teams start buying components to prove they are doing it properly. You are doing GitOps properly if Git is the source of truth and a reconciler is continuously working to make the cluster match it.

Running the loop once it's live

The machinery can wait. These can't, because every team meets them within a few weeks of switching the reconciler on, however small the estate.

Pause the loop before you fix production by hand. During an incident the reconciler is on Git's side, not yours. In Flux, flux suspend helmrelease <name> (or flux suspend kustomization <name> for plain manifests) stops new revisions and drift correction, and the matching flux resume turns them back on. In Argo CD, argocd app set <app> --sync-policy none takes one Application off automated sync. There are two catches on the Argo side: an Application generated by an ApplicationSet ignores changes to its own sync policy, and a parent app with self-heal on can put a child's policy straight back. Whatever you change by hand, commit the same change to the values file before you resume, or the loop will quietly undo your fix. Put those commands in the incident runbook now, while things are calm.

Let the autoscaler own replicas. If a HorizontalPodAutoscaler manages a Deployment and your chart also renders spec.replicas, the two fight, and every sync resets the count the HPA chose. The chart that helm create scaffolds already handles this: its Deployment only renders replicas when autoscaling.enabled is false, so the fix is usually one value rather than a template change. Argo CD's best-practices guide and the Flux FAQ give the same advice: leave replicas out of what you apply. If a third-party chart insists on rendering it, Argo CD's ignoreDifferences on /spec/replicas hides it from the diff, but the sync still applies it unless you also set the RespectIgnoreDifferences=true sync option. On Flux, a HelmRelease takes driftDetection.ignore with paths: ["/spec/replicas"], which its docs suggest for exactly this case.

Preview what a pull request will actually render. With Helm, reading a values diff is not the same as knowing what the manifests will look like, because one value can fan out across a dozen templates. The cheapest check needs no cluster at all: run helm template with the base and environment files on your branch and on main, and diff the two outputs. For the live comparison, argocd app diff <app> --revision <branch> renders the chart and compares it with what is running, with one gap: it leaves Secrets out. flux diff kustomization is less useful here, because for a HelmRelease it shows the change to the HelmRelease object rather than to the rendered manifests. Running a live diff in CI means giving CI read access to the cluster, which is a real trade for a pull-based setup. Running it locally before you open the pull request costs nothing.

Make the loop react faster than it polls. By default Argo CD checks Git every two minutes plus up to a minute of jitter, and Flux fetches each GitRepository on whatever interval you give it. A merged pull request that appears to do nothing for a few minutes is usually waiting for the next poll. Both tools accept webhooks: Argo CD's API server takes Git webhook events directly, and Flux uses a Receiver from its notification controller. Keep polling as the fallback, so a missed webhook costs you minutes rather than a deploy.

Where to start

Pick one low-risk chart, ideally a stateless internal service, and move it to the base-plus-environment layout: values.yaml for what is shared, values/production.yaml for the handful of lines that differ. Put it under the reconciler with automated sync on and pruning off, and on Flux set driftDetection.mode: warn. Make a pull request the only way it changes for a fortnight, and keep a note of every drift the reconciler reports, whether that is Argo CD marking the app OutOfSync or Flux emitting a drift event on the HelmRelease. Every one of those is a hand-edit or a --set someone still reaches for, and each becomes either a runbook entry, a value that belongs in Git, or a field handed over to another controller. When a week goes by with nothing to explain, turn on self-heal and pruning, move the next chart across, and leave the rest of the GitOps catalogue on the shelf until a real problem asks for it.

For a small team, most of the effort in this post goes into standing the loop up rather than using it. That is the part we built Kupe Cloud to take away. It is our managed Kubernetes platform, and both halves of the setup are ready as soon as you spin up your first cluster. Argo CD is already running, with a project set up for your tenant, so once you connect a service repo laid out as above, its chart deploys as it is. Secrets work the same way from day one: you create a secret once in the console, it is stored encrypted in the platform's vault, and Kupe syncs it as a Kubernetes Secret into the clusters and namespaces you choose. That is the External Secrets model from earlier, with nothing extra to install or run. Everything else in this post still applies; you just start with the loop already running.

Falco on Kubernetes: install, tune, and route alerts

Billy Walker — Tue, 22 Sep 2026 18:28:51 +0000

You scan images in CI, sign them before deploy, and gate manifests at admission. Every one of those controls answers the same question: should this workload be allowed to start? None of them can tell you what it did at 02:00 once it was running. Someone opening a shell in a production pod, a binary that was never in the image suddenly executing, a process reading /etc/shadow for no good reason. Those are runtime events, and runtime is where Falco lives.

The catch is that Falco tutorials age faster than most. The project has dropped its legacy eBPF probe, its gVisor engine and its gRPC output in recent releases, deprecated the append: true rule syntax in favour of override:, and moved rule distribution onto OCI artifacts pulled by a sidecar. A lot of the snippets still ranking on the first page will not load on a current install. So this walkthrough sticks to what the current chart and docs actually do: install, trigger one stock alert, write one custom rule, and then spend most of the time on tuning, which is the part that decides whether Falco is still switched on in six months.

Where runtime detection fits

A Kubernetes security stack has layers, and each one answers a different question:

image scanning: what went into the artefact
signing and provenance: whether you trust where it came from
admission policy: whether the cluster should accept it
runtime detection: what the workload is doing now

The last layer is the one teams skip until something odd is already running in production. Falco fills it by watching syscalls from the kernel, matching them against a rules engine in userspace, and emitting an alert when a rule fires. If you want the eBPF side of that story in more depth, the Cilium networking post covers how the same machinery is used for the network path.

How Falco collects events, and where it differs from Tetragon

Search for "Falco vs eBPF" and you will find a lot of confused threads. Falco is not an alternative to eBPF. It uses eBPF as one way of collecting kernel events, and there are two supported drivers today:

modern_ebpf, a CO-RE probe embedded in the binary, which is what Falco itself defaults to
kmod, the kernel module, still supported and still the right answer on kernels the modern probe cannot run on

The Helm chart adds a third value, auto, which is its default. It tries the modern probe first and falls back to the kernel module. The older ebpf and gvisor driver kinds are gone: the chart will not even render if you pass either, which is a kinder failure than a pod that starts and silently sees nothing.

Tetragon, from the Cilium project, makes a different trade-off. It leans into in-kernel enforcement: a tracing policy can override a kernel function's return value or send SIGKILL to the offending process before it gets any further. Falco keeps the decision in userspace, with a mature rules engine, a maintained default ruleset, and a lot of flexibility around outputs and tuning. If your first question is "what happened inside that container?", Falco is the more natural place to start. If it is "stop that from ever completing", look at Tetragon.

Install Falco with the Helm chart

helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update
helm install falco falcosecurity/falco \
  --create-namespace \
  --namespace falco \
  --set falcosidekick.enabled=true \
  --set falcosidekick.webui.enabled=true \
  --set collectors.kubernetes.enabled=true

Those three flags are worth understanding rather than copying:

falcosidekick.enabled=true deploys Falcosidekick and switches Falco to JSON output over HTTP, so alerts can be forwarded to systems people already watch
falcosidekick.webui.enabled=true adds the web UI and a Redis instance behind it, which is handy while you are testing and worth turning off once alerts flow somewhere durable
collectors.kubernetes.enabled=true deploys the metadata collector and the k8smeta plugin, which adds the owning Deployment, ReplicaSet and Service to each alert. Without it you still get the pod name, namespace and labels, because Falco reads those from the container runtime directly

The chart also deploys two containers you did not ask for by name: a falcoctl init container that pulls the default rules as an OCI artifact, and a falcoctl sidecar that checks for a newer artifact once a week and swaps it in. That has consequences for tuning, which we will come back to.

If you need to be explicit about the driver, use one of the two current values:

--set driver.kind=modern_ebpf   # or driver.kind=kmod

Then watch the Falco container start. The pod has several containers now, so name the one you want:

kubectl -n falco logs ds/falco -c falco

You are looking for a clean driver load and a Loading rules from file line for each rules file. If the driver is wrong for the kernel underneath, this is where you find out.

Trigger one stock alert first

Before writing anything custom, prove the default path works. Start a throwaway pod:

kubectl run runtime-test --image=busybox --restart=Never -- sleep 3600

Then exec into it with a terminal attached:

kubectl exec -it runtime-test -- sh

That fires the stock Terminal shell in container rule. The -t is doing real work here: the rule checks that the shell has a TTY, so kubectl exec runtime-test -- sh -c 'id' without one will not trigger it. Check the Falco logs or the Falcosidekick UI and you should see the alert with container_id and container_name appended to the end of the output line. Those fields are not in the rule's own output string; the container plugin suggests them and Falco appends suggested fields automatically.

This is also the moment to reset expectations about volume. The stock falco_rules.yaml is a small, conservative file: 25 stable rules at the time of writing, every one enabled, none of them experimental. The noisy reputation comes from the incubating and sandbox rulesets, which the chart does not load by default, and from teams tuning carelessly once they do.

Write one custom rule

The simplest useful rule is one you can explain in a sentence: tell me if a miner-like binary starts inside a container. With the chart, custom rules go in the customRules value, and each key becomes a file under /etc/falco/rules.d/:

customRules:
  miners.yaml: |-
    - rule: Crypto miner process in container
      desc: Detect common miner binaries starting inside a container
      condition: >
        spawned_process and container and
        proc.name = anyof (xmrig, minerd, cryptominer)
      output: >
        Crypto miner process in container | user=%user.name
        process=%proc.name command=%proc.cmdline
      priority: WARNING
      tags: [container, malware, crypto-miner]

Two things about that condition. spawned_process and container are macros from the stock file, so you get exec events inside containers for free. And anyof is a comparator modifier from the current rule language: one field compared against a short list, without a chain of or clauses. It sits after the operator, with a space either side and the values in parentheses.

One trap that is not in the tutorials: proc.name is the kernel's process name, and the kernel truncates it to 15 characters. The stock rules file depends on this, which is why you will find entries like mysql_install_d in its lists. If your binary has a long name, match on proc.exepath or container.image.repository instead.

Load order matters. The default rules_files list reads falco_rules.yaml, then falco_rules.local.yaml, then everything in rules.d, and a file that overrides a stock rule has to load after it. Give the file a .yaml or .yml extension too, because anything else in that directory is ignored.

Before you apply, validate:

falco -V miners.yaml

That parses and compiles the rules file and exits, and the easiest place to run it is the same Falco image tag you deploy, so the engine version matches. It is worth the thirty seconds, because a rules error is fatal. Falco stops at the first file that fails to load, prints the reason, and exits non-zero, which on Kubernetes means the whole DaemonSet goes into CrashLoopBackOff over one bad line in one file. Warnings are logged and tolerated. Errors are not.

Tuning is where most Falco deployments succeed or fail

Install is the easy part. Tuning decides whether people trust the alerts next month, and the stock rules file is built for it in a specific way that most tutorials, including the official exceptions examples, obscure.

The fact the docs bury is that none of the stable stock rules ships an exceptions: block. The docs' canonical example appends values to an exception on Write below binary dir, and that rule lives in the sandbox ruleset, not the one you just installed. Copy it against a stable rule and the loader rejects it, because there is no existing exception to inherit fields from.

What the stable rules give you instead is a set of hooks with names starting user_, each defined as (never_true), plus empty image lists. Take Contact K8S API Server From Container, which fires the first time an operator or controller you run outside kube-system talks to the API server. Its condition ends with and not user_known_contact_k8s_api_server_activities, and that macro is yours to replace:

- macro: user_known_contact_k8s_api_server_activities
  condition: >
    (k8s.ns.name = argocd and
     container.image.repository = quay.io/argoproj/argocd)
  override:
    condition: replace

The same shape works for the other stable rules. Terminal shell in container has user_expected_terminal_shell_in_container_conditions. Read sensitive file untrusted has a read_sensitive_file_images list you can append to:

- list: read_sensitive_file_images
  items: [quay.io/argoproj/argocd]
  override:
    items: append

If you would rather use exceptions, you can. Appending a new named exception to a stable rule is supported as long as you supply fields alongside values:

- rule: Contact K8S API Server From Container
  exceptions:
    - name: argocd_controllers
      fields: [container.image.repository, k8s.ns.name]
      values:
        - [quay.io/argoproj/argocd, argocd]
  override:
    exceptions: append

Falco adds and not (container.image.repository = ... and k8s.ns.name = ...) to the condition for you. Either approach beats disabling the rule, because the rule is still right for every other workload.

When a rule really does not fit your threat model, turn it off explicitly:

- rule: Drop and execute new binary in container
  enabled: false
  override:
    enabled: replace

The override: block is the part older posts get wrong. It states, per key, whether you are appending to or replacing the stock definition. append: true still parses for now but is deprecated, so translate any snippet built around it before you paste.

One more thing the file already did for you: the API-server rule's k8s_containers macro exempts everything in kube-system. If it is firing, the workload is somewhere else.

Route alerts somewhere people already look

An alert that only exists in one DaemonSet's log stream changes nothing. Falcosidekick takes Falco's JSON output and fans it out to Slack, Loki, Elasticsearch, Splunk, Datadog, Alertmanager and a long list beyond that. The chart values map directly to its outputs:

falcosidekick:
  enabled: true
  config:
    slack:
      webhookurl: https://hooks.slack.com/services/XXXX/YYYY/ZZZZ
      minimumpriority: warning
    loki:
      hostport: http://loki-gateway.monitoring:80

There is no global minimum priority in Falcosidekick. Each output has its own minimumpriority, and its severity ladder spells the level informational rather than Falco's info, which catches people out when a filter silently matches nothing.

The pattern that holds up on platform teams is a split by urgency: Slack or Teams for the handful of rules that need a human now, a log backend such as Loki for searchable history, and the rest filtered out at the output. If Grafana is already your front door, the Grafana monitoring post covers getting Grafana itself stood up and the first dashboards in place.

Operating it after week one

The first three sections get Falco running. These are the three things that go wrong once it has been running for a while.

Your rules are floating. The falcoctl sidecar follows a major-version tag of the rules artifact and installs whatever is newest every 168 hours, into an emptyDir mounted over /etc/falco. Two consequences: an edited falco_rules.yaml inside the pod is overwritten on the next restart or the next follow, and a rule you tuned by name can change underneath you on a Tuesday. Keep every override in customRules, and pin falcoctl.config.artifact.install.refs and follow.refs to an exact artifact tag or an @sha256: digest once you are past the exploring stage.

You are dropping events and nobody knows. Falco emits an internal alert named Falco internal: syscall event drop when the kernel ring buffer overflows, but it is a debug-priority alert rate-limited to one message every 30 seconds. Raise Falco's minimum priority to cut noise, or set minimumpriority: warning on your Slack output, and it vanishes. The durable answer is metrics:

--set metrics.enabled=true \
--set serviceMonitor.create=true

That turns on Falco's Prometheus endpoint and a ServiceMonitor for it. The two series to graph against each other are falcosecurity_scap_n_drops_total and falcosecurity_scap_n_evts_total. If drops climb with load, the documented remedies are a bigger ring buffer via engine.modern_ebpf.buf_size_preset, fewer CPUs per buffer via engine.modern_ebpf.cpus_for_each_buffer, and base_syscalls to narrow what the driver captures. Which of those helps depends on your syscall mix, which is also why there is no universal "Falco overhead" figure worth quoting: it tracks what your workloads do, not a percentage.

You want to act, not just alert. Falco itself only detects. Falco Talon is the incubating response engine in the same ecosystem: it receives events from Falcosidekick's talon output and applies rules such as terminating the pod or labelling it for quarantine. Its Kubernetes actions need k8s.pod.name and k8s.ns.name among the event's output fields, which the stock rule outputs do not carry, so expect to append those to any rule you want it to act on. It is receiving commits but has had long gaps between tagged releases, so treat it as something to evaluate rather than a default, and keep the enforcement question in mind when you weigh it against Tetragon.

The failure you will hit first

Almost every first Falco tuning session ends one of two ways. Either the DaemonSet is crash-looping, in which case the Falco container's log has the offending file and line near the top and falco -V would have told you first. Or the pods are healthy and the override seems to do nothing. For the second, check three things before you touch the rule. Confirm the file landed in /etc/falco/rules.d/ with a .yaml extension and appears in the Loading rules from file lines. Check the rule name you are overriding still exists with that exact spelling in the ruleset the falcoctl sidecar just installed. Then check whether the workload is already exempt, the way kube-system is for the API-server rule. Usually the rule was fine and the plumbing was not.

CIS Benchmark for Kubernetes with kube-bench and Kubescape

Billy Walker — Thu, 17 Sep 2026 17:51:08 +0000

Most teams get told to “harden Kubernetes to CIS” long before anyone explains what happens after the first scan. You run the tool, it prints a wall of FAIL and WARN, half the findings turn out to be somebody else’s problem on a managed control plane, and the rest sit in a ticket queue until the next audit comes round. There is a narrower version that works. Use kube-bench for the host and control-plane checks only it can see, use Kubescape for the posture checks you want to keep running from the API side, and treat the CIS Kubernetes Benchmark as a triage tool rather than a moral score.

One thing to know before either tool runs: the scanners trail the benchmark. CIS revises the benchmark as new Kubernetes releases land, and each tool picks the revision up some time later, as a new profile that somebody has to write and merge. On a recently upgraded cluster there is a fair chance your scanner is running the closest older profile and hasn’t mentioned it. The result is still useful, but that caveat belongs in the first line of anything you hand to an auditor, and there is a quick way to check, which we’ll get to.

Two scanners, two views of the cluster

The short version:

kube-bench runs on a node. It reads config files, file permissions and the flags on running processes, then compares them with the CIS checks.
Kubescape talks to the API server. It evaluates the resources in your cluster, or the manifests in your repo, against frameworks such as NSA-CISA, MITRE ATT&CK and CIS.
They overlap on the policy-style checks, and that is about it.

That is why “kube-bench or Kubescape?” is the wrong question. Run only kube-bench and you get a point-in-time audit with no idea what changed last Tuesday. Run only the Kubescape CLI and you miss everything that depends on a file mode or a kubelet flag on a real node.

There is also one deliciously awkward detail to get out early. The standard kube-bench Job sets hostPID: true and mounts host paths such as /etc/kubernetes and /var/lib/kubelet, so the compliance tool needs an exemption from the restricted pod policy it is about to tell you to enforce. A host-level audit has to see the host, so plan for the exemption now rather than discovering it when admission rejects the pod.

What the CIS Kubernetes Benchmark is, and what it is not

The benchmark is a hardening checklist for cluster configuration. Its sections cover the control plane components, etcd, control plane configuration, worker nodes, and policies: RBAC, pod security, network policies and secrets. Every recommendation comes with an audit procedure and a remediation, which is what makes it automatable.

What it leaves out is where people over-read the score. A good CIS result tells you the configuration is in better shape than it was. It does not tell you:

whether workloads behave safely at runtime
whether your images are full of known CVEs
whether your admission policies are broad enough for your actual risk model
whether the control plane you rent from a provider was set up the way you would have done it

So treat the benchmark the way you would treat a good linter: very useful, occasionally annoying, and dangerous only when somebody mistakes it for a complete security strategy. If a score is going in front of leadership, put that caveat on the same slide. “100% CIS” and “secure” are different claims.

Start with kube-bench, on the nodes

kube-bench is an open-source project from Aqua Security, and it does one blunt thing well: read the files and flags that exist on the host and compare them with the benchmark profile it knows about.

The normal entry point is the Job manifest in the root of the repo. On a self-managed cluster, apply it and read the logs:

kubectl apply -f https://raw.githubusercontent.com/aquasecurity/kube-bench/main/job.yaml
kubectl logs job/kube-bench

Every check comes back in one of four states. PASS and FAIL are what they sound like. WARN means kube-bench could not make the call for you, either because the benchmark marks the check as manual or because the check is unscored. INFO is a section header or a check you skipped. The habit to avoid is reading WARN as “fine”. In the vanilla profile nothing in the policies section is scored, so nothing in it can ever show as FAIL, and that section holds the RBAC and pod security checks you probably care about most.

The Job needs hostPID and that stack of read-only host mounts because kube-bench is looking at the node directly rather than asking the API server to summarise it. You can see the same assumption in the container one-liner from the upstream docs:

docker run --pid=host \
  -v /etc:/etc:ro \
  -v /var:/var:ro \
  -t docker.io/aquasec/kube-bench:latest

That is how it catches things an API-only scanner cannot. It is also why the Job belongs in its own clearly named namespace, labelled privileged for Pod Security Admission, rather than squeezed under your strictest workload policy. One more practical note: to audit control-plane nodes the pod has to be scheduled onto them, which means adding a nodeSelector and tolerations to the manifest first.

Benchmark lag is normal, so check which profile ran

The easiest way to confuse yourself is to assume the latest kube-bench means the latest CIS revision. kube-bench maps your Kubernetes version to a benchmark profile using a table in its config, and the platforms page in the docs shows the same mapping. When your cluster is newer than anything in that table, kube-bench walks the minor version down until it finds a match and runs that profile instead. There is no error, and at the default log level there is no warning either.

The JSON output is where you catch it:

kube-bench run --targets node,policies --json

Each block carries a version field, which is the profile that ran, and a detected_version field, which is the Kubernetes version kube-bench found. Compare the pair with the platforms table before you write policy around the result.

Once you know which profile you want, pin it:

kube-bench run --benchmark <profile> --targets node,policies

Here <profile> is a name from the “kube-bench config” column of that table. Pinning can’t give you a newer benchmark than the tool ships. What it buys is a result you can describe accurately, and a scheduled run that won’t switch profile underneath you on the day somebody bumps the image tag.

The same table matters if you are not on vanilla Kubernetes. k3s, RKE2, OpenShift and the managed services each have their own profiles, because the vanilla one looks in paths those platforms do not use and rewards you with a page of false failures.

One finding, end to end

Reports are easy to admire from a distance, so take one finding all the way through. In the current vanilla profiles, check 5.2.3 is “Minimize the admission of containers wishing to share the host process ID namespace”. Four questions get you from the line in the report to a decision:

Which check fired, and in which profile? IDs move between profiles. The same control is 4.2.2 in the EKS profile, which is one more reason to pin.
What did kube-bench run to decide? For this check, a kubectl loop over every pod in every namespace that reads .spec.hostPID. It reports pods that share the host PID namespace today, and never asks whether a policy would stop the next one.
Is this a real gap, an intentional exception, or the platform’s business?
What is the narrowest fix that changes the risk, not just the score?

The first answer matters more than it looks. The vanilla profile treats this check as unscored, so a failure prints as WARN. The EKS profile treats it as automated and scored, so the same pods print as FAIL. The headline changes with the profile; the risk does not.

The benchmark’s remediation text is “Add policies to each namespace in the cluster which has user workloads to restrict the admission of hostPID containers”, and on a current cluster the policy in question is a Pod Security Admission label. The baseline level already forbids host namespaces, so you do not need restricted to clear this finding. Ask the API server what would break before you enforce anything:

kubectl label --dry-run=server --overwrite ns --all \
  pod-security.kubernetes.io/enforce=baseline

A server-side dry run changes nothing, and it prints a warning for every namespace with pods that would violate the level (any baseline violation, so host networking and hostPath volumes show up too). The namespaces it names are your candidate exceptions: usually a node exporter, a logging or security agent, and kube-bench itself. From there the fix is a split model:

keep most namespaces on baseline or restricted
carve out one clearly named namespace for infrastructure that needs host access
write down why that namespace exists and who can deploy into it
re-run only the relevant check, so you know you changed the right thing

kube-bench run --targets policies --check 5.2.3

Expect the check to stay at WARN, because the pods in your exception namespace still have hostPID: true. That is correct. What has changed is that you can now explain every pod behind it, and an explained exception is what an auditor is looking for.

Managed clusters: part of the benchmark is out of reach

The kube-bench docs are direct about this: “It is impossible to inspect the master nodes of managed clusters, e.g. GKE, EKS, AKS and ACK”. That sentence should change how you describe the result. On a managed control plane, you are auditing:

the worker nodes you can reach
the policy and workload checks that are visible from inside the cluster
whatever managed-service profile kube-bench ships for your platform

The project ships separate manifests such as job-eks.yaml and job-gke.yaml for exactly this reason. Running kube-bench on EKS, for example, you cannot schedule a pod onto the control-plane nodes, so the master checks are simply unavailable.

Your compliance story on a managed platform is therefore partly technical and partly contractual. You verify what you can, and for the control plane you lean on the provider’s documentation, attestations and shared-responsibility model. Say so in the report: “out of our reach, covered by the provider’s attestation” is accurate, and easier to defend than a suspiciously complete set of green ticks.

Kubescape for the part that keeps running

Kubescape is a CNCF project and it now covers a lot more than compliance: misconfiguration scanning, image scanning, admission control and runtime detection all live under the same name. The slice that matters here is narrower: the API-side posture layer that sits next to kube-bench.

The CLI is straightforward:

kubescape list frameworks
kubescape scan framework nsa

Run them in that order. The CIS framework IDs have the benchmark revision baked into the name, so an ID copied from an old tutorial may no longer exist in the version you installed. nsa and mitre are stable names; for CIS, list first and use what is actually there.

For CI or a recurring check, the threshold flag is the practical one:

kubescape scan framework nsa --compliance-threshold 80

Below the score you set, the command exits with code 1. That is usually enough to stop “we should probably look at this later” from becoming a permanent state. Pick the number from a real baseline run, though. A threshold the cluster can’t currently meet tends to get switched off rather than fixed.

Kubescape vs kube-bench: which question goes where

Reach for kube-bench when the question is:

are kubelet and control-plane settings aligned with the benchmark?
do file permissions and host-level configuration match the guidance?

Reach for Kubescape when the question is:

what is the posture of the workloads and resources in this cluster right now?
can I run this in CI against manifests, before anything reaches a cluster?

The numbers won’t line up, because the two tools count different things. Where they do overlap, use it as a cross-check: if both are unhappy about the same class of policy gap, that finding goes to the top of the pile.

The in-cluster path is where Kubescape starts paying for itself

The Kubescape operator is the natural next step once you are past one-off CLI scans, and the install is refreshingly ordinary:

helm repo add kubescape https://kubescape.github.io/helm-charts/
helm upgrade --install kubescape kubescape/kubescape-operator \
  --namespace kubescape \
  --create-namespace

Two details in the chart’s values are easy to miss. Out of the box, configuration scans run on a schedule; the change-driven rescans that most people picture when they hear “continuous” sit behind capabilities.continuousScan, which is disabled by default. And the operator’s node agent mounts the host filesystem read-only, so the kubescape namespace needs the same Pod Security exception as kube-bench. Your one documented exceptions namespace becomes two, which is still a list short enough to defend.

A minimal recurring setup, and where to start

If you want the smallest loop that is still worth having, I would keep it to this:

kube-bench on a schedule, as a CronJob with the profile pinned, for the host-level checks only it can see
Kubescape in CI or in-cluster for the API-side posture checks
one threshold that fails something real, instead of a dashboard with no owner
one exception path for justified infrastructure privileges, documented and reviewed

As for where to start: run the kube-bench Job once, and before you fix anything, sort every FAIL and WARN into three piles: real fixes, documented exceptions, and the provider’s side of the line. That sorted list is a more useful deliverable than a score, and it is what the recurring scans then keep current.

Kubernetes is not a platform: it is a toolkit

Billy Walker — Wed, 09 Sep 2026 09:40:15 +0000

A team says it has “built a platform” because it runs Kubernetes. Then a developer tries to ship a service and still has to assemble a Deployment, Service, HorizontalPodAutoscaler, NetworkPolicy, PodDisruptionBudget, ServiceAccount, and half a dozen bits of environment-specific glue by hand. That is the category error in one scene. Kubernetes is the substrate. The platform is still on the backlog.

The Kubernetes project makes the same point in its own overview docs. Under the heading “What Kubernetes is not”, it says Kubernetes is “not a traditional, all-inclusive PaaS” and that it provides “the building blocks for building developer platforms”. Building blocks, from the people who maintain them. So when you hand product teams raw kubectl, raw YAML and a wiki full of tribal knowledge, what you have handed over is a very capable control plane and a pile of homework.

Kubernetes is a very good substrate

Kubernetes earns its place in all this. It gives you strong primitives and a programmable control plane, and that combination is hard to come by anywhere else.

Concretely:

declarative desired state
reconciliation loops
scheduling and placement
service discovery
rollout machinery
extensibility through CRDs and controllers

Which is a serious foundation, and it explains why Kubernetes keeps ending up in the middle of internal platforms.

Foundations and finished products sit in different categories, though. Postgres is a foundation; the finance dashboard someone logs into every morning is the product built on it. A cluster running important workloads sits on the foundation side of that line, however well run it is.

This is where a lot of platform work goes sideways. Infrastructure teams do the hard bit: secure the cluster, patch it, make it observable, get some governance around it. Somewhere in the following six months the organisation starts calling the cluster “the platform”, on the assumption that a working substrate produces a usable product on its own. It never has.

A platform is a product with users

The CNCF Platforms white paper uses much stricter language than most teams do in everyday conversation. It defines a platform as an integrated collection of capabilities defined and presented according to the needs of the platform’s users, and it calls out interfaces such as web portals, project templates, and self-service APIs.

“We run Kubernetes” clears none of that bar. The platform is the opinionated layer above it, the one that turns raw technology into something people can use safely and repeatedly.

Usually that means some combination of:

self-service interfaces so teams can create or change what they need without waiting on tickets
paved roads so the safe path is the easy path
policy and defaults so teams do not need to rediscover every security and reliability lesson themselves
documentation and examples that reflect how the organisation actually expects software to be shipped
support boundaries so people know what is standard, what is allowed, and what will get help at 02:00

The last one does more work than it looks. A platform is partly software and partly promises. An interface where no one can tell you the support status is a hopeful abstraction: teams route around it the first time it surprises them, and now you maintain two paths instead of one.

The category error shows up as cognitive rent

When teams say “Kubernetes is our platform”, they usually mean “Kubernetes is where our applications run”. The second claim is easy to earn; the first one is a product commitment.

There is a test for which one you have: what does a product engineer still need to understand to ship a normal service safely? If the answer runs to “quite a lot of Kubernetes”, you are still handing over the substrate.

That usually means developers still need working knowledge of things like:

how requests and limits interact with scheduling
when an HPA will or will not behave the way they expect
why a missing PodDisruptionBudget only hurts during maintenance windows
how NetworkPolicy defaults work in their cluster
which ingress, gateway, or certificate path is considered standard
which bits of YAML are mandatory because the platform has no stronger abstraction yet

Developers learning infrastructure is fine, and plenty of them should. The trouble starts when shipping an ordinary service requires them to learn the platform team’s entire problem space.

That is the bill for confusing a toolkit with a platform. The complexity stays exactly where it was; it just gets paid for by every team that touches it, in small instalments, every time they deploy.

This is why the Team Topologies framing still holds up. A platform is valuable in proportion to the cognitive load it takes off the teams consuming it. Ask developers to become part-time Kubernetes operators and you have moved that load rather than absorbed it.

A thin platform still counts

One reason teams overstate what Kubernetes gives them is that they imagine the alternative must be some giant internal developer portal project. If they are not ready to build Backstage templates, service catalogs, policy packs, and self-service workflows, they decide the cluster itself must count as the platform. Most of the useful work lives in the gap between those two options.

Team Topologies has a saner idea: the thinnest viable platform. On the Team Topologies TVP page, Matthew Skelton says that this can be “just a wiki page” if that is all you need at your current scale. That line keeps the argument grounded. What qualifies something as a platform is whether it gives your developers a clearer, safer, more repeatable way to get work done. Scale decides how thick it has to be; nothing else does.

Sometimes the thinnest useful platform is:

one golden path for a standard web service
a starter template with the right defaults already wired in
one supported CI path
one documented way to expose a service
one place to request a database or secret with known guardrails

That is still platform work, and at most organisations it is the right platform work to do first.

Starting small is fine. The mistake is skipping the product layer entirely and calling the control plane beneath it finished.

What a real platform adds on top of Kubernetes

If Kubernetes is the substrate, what does the platform layer actually contribute?

At minimum, it should remove repeated decisions that your teams do not benefit from making over and over.

A decent internal platform usually adds:

An opinionated service shape. New services start from a template or scaffold that already includes the organisation’s baseline for probes, resources, identity, networking, and delivery.
A supported path to production. One default route that the platform team actively maintains, and that most services can take without needing a conversation first.
Guardrails with teeth. Policy, admission control, CI checks, and defaults that catch the boring mistakes before they get merged.
Self-service for common needs. Databases, secrets, DNS, certificates, queues, environments, or delivery workflows that do not require a human to hand-hold every request.
A product experience. Good docs, clear ownership, visible support boundaries, and feedback loops so the thing improves instead of calcifying.

Notice what is missing from that list: “owning a cluster”. Owning a cluster is an operational capability, and platform engineering usually needs one. It buys you the right to start.

Use the maturity model as a self-check

The CNCF Platform Engineering Maturity Model is useful here because it stops the conversation from collapsing into slogans.

It describes four levels — Provisional, Operationalized, Scalable, and Optimizing — and looks across five aspects: investment, adoption, interfaces, operations, and measurement.

That works better as a diagnostic than asking whether you “have a platform”. Almost every team is somewhere in the middle of that grid, and the useful question is which cell they are in and which one they want to be in next.

A few blunt checks help:

Interfaces: do teams have a real self-service interface, or are they still stitching together YAML and Slack messages?
Adoption: do product teams voluntarily use the paved road because it is better, or because they were told to?
Operations: is there a supported day-two story behind the abstraction, or only a shiny day-one story?
Measurement: can you show that the platform shortens delivery loops or reduces support load, or is that still mostly assumed?
Investment: is there sustained ownership, or is the “platform” just a side project orbiting the cluster team?

Answer those five properly and the picture resolves fast. A team with a well-run cluster, ad-hoc interfaces and adoption it has never measured is doing valuable infrastructure work. The platform sits a column to the right of where that team is standing.

Where the finish line actually is

The finish line is a state where most teams can ship a service without writing a production-grade manifest from scratch.

You still want depth somewhere. Strong operational understanding in the platform team, and enough shared literacy that developers can reason about what runs underneath their code — that is a real asset, and it is a different thing from requiring it of everyone before they are allowed to deploy.

Developers should be able to use the platform the way you hope customers use any good product — confidently, and without carrying a mental model of its internals around with them.

Calling the substrate the platform has a specific cost. It lets you declare victory early, and after that every improvement gets spent on the cluster rather than on the people using it.

If you want one thing to do this quarter: take the service shape your teams create most often and give it a single supported path — a template, a sane set of defaults, one documented route to production. Then time how long it takes somebody outside the platform team to get from an empty repository to a running service, before and after. If that number holds steady, you built the abstraction for your own convenience.

Helm charts are technical debt: blame the templating model

Billy Walker — Mon, 31 Aug 2026 08:03:53 +0000

The pull request is four lines of template, adding an optional nodeSelector to an internal chart. CI has already done its job properly: helm lint passed, the chart rendered against every environment's values file, kubeconform validated each manifest that came out, and the snapshot diff of the rendered output is sitting in the check run waiting to be read.

And you still cannot say, from the diff, what this change does in staging without going and reading three other templates first.

Nobody files that as technical debt. It goes in the mental folder marked “charts are a bit fiddly”. But if your team maintains a dozen first-party charts, that tax gets levied on every review, and no amount of pipeline pays it off, because the pipeline was never the thing that was broken.

This is not a “Helm was always bad” post. Helm solved a real problem, and most teams running Kubernetes are better off with it than without it. The rot is somewhere more specific: Helm renders Kubernetes manifests by running Go's text/template over YAML as strings, and once you are authoring charts rather than just installing them, that model generates work forever.

That model has survived every major version so far, Helm 4 included. If your chart estate already feels brittle, the next release will not make it less so, and I will come back to why.

Helm won for good reasons

Helm is a CNCF graduated project and the default packaging story in Kubernetes, and it deserves to be:

install, upgrade, rollback and release history in a single tool
a standard way for vendors and open-source projects to ship applications
OCI registry support that slots into existing GitOps workflows
ecosystem gravity, which by now matters nearly as much as technical merit

Nobody sensible is still having the “should we use Helm at all?” argument. If you are installing Grafana, or External Secrets Operator, or almost anything with a healthy community behind it, the chart is the path of least resistance and you should take it.

The trouble is that the same tool that consumes packaged software so well also makes it very easy to start writing your own. One chart for the internal API. One for the batch worker. Then a shared library chart, because three of them had grown the same _helpers.tpl. At some point you stopped being a Helm user and became the maintainer of a templating system. That is a perfectly reasonable thing to be, as long as it was a decision rather than a drift.

The debt is in the rendering model

Helm templates know nothing about Kubernetes objects. They emit text that happens to be YAML.

Here is the shape of it, lifted from roughly every deployment.yaml in every internal chart:

# templates/deployment.yaml
spec:
  template:
    spec:
      {{- with .Values.nodeSelector }}
      nodeSelector:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      containers:
        - name: {{ .Chart.Name }}
          image: {{ .Values.image.repository }}:{{ .Values.image.tag }}

That 8 is load-bearing. nodeSelector: sits at column six, so its keys have to land at column eight, and nindent exists precisely because chart authors need to re-indent embedded blocks constantly. Write nindent 6 by mistake and you have not made a formatting error. You have set nodeSelector to null and promoted its contents to siblings of containers, and the render will print that quite happily, because it is still a well-formed YAML document. Nothing in the YAML layer objects. It takes schema validation downstream, kubeconform -strict in the pipeline or a server-side dry run, before anything flags it. That is a lot of machinery to need in order to catch a two-space mistake.

Two spaces. Helm's own template best practices page walks through indentation, whitespace chomping and generated-output formatting, and concedes the underlying problem in passing: “YAML is a whitespace-oriented language.” Full marks for honesty, but that is a peculiar property for an abstraction layer to have.

All of which is the cheap version of the problem, and it is cheap precisely because a machine can express it. Schema validation catches a wrong shape, and it catches it in seconds. What no pipeline catches is a chart whose right shapes have become impossible to hold in your head.

Once your abstraction works on strings rather than typed objects, a few things follow:

indentation becomes part of correctness, not just tidiness
a branch that renders the wrong shape still looks plausible in review
values become an API surface long before anyone has designed them like one
refactoring is hard, because the behaviour lives across helpers, partials and value conventions rather than in a schema your tooling understands

Disciplined teams are not exempt from this. They often accumulate more chart machinery, because they build helpers and layering and conventions to contain the complexity. That containment is genuinely useful, and it is also the tell: you are building scaffolding around a model that produces complexity faster than you can absorb it.

How it shows up on day two

You rarely spot chart debt by reading a chart. You spot it in the changes that need far more scrutiny than their diff size suggests they should.

`values.yaml` turns into an untyped product surface

values.yaml is the front door to your chart, which is convenient right up until you realise you have shipped a long-lived API with no type system behind it.

A values.schema.json belongs in every first-party chart, and Helm will enforce it on install, upgrade, lint and template. But the schema file is optional, so it is opt-in per chart and easy to leave behind on the one chart nobody has revisited, and it describes structure rather than behaviour. A value can be structurally perfect and still produce nonsense once it has threaded its way through conditionals, helper templates, defaults and subchart overrides.

Then another team starts depending on a value name. Renaming a field stops being a cleanup and becomes a migration, complete with a deprecation window and a note in the release notes that nobody reads.

The source is harder to review than the output

A rendered manifest can be entirely ordinary while the logic that produced it is anything but.

Humans review the chart source. A pull request that flips a condition or threads one more flag through three templates is not asking “is this Deployment valid?”. It is asking “which combinations of values, defaults and includes now produce a different document?”, and that question has no diff you can look at.

Which is why the gates all end up pointed at the rendered output rather than the source. Those gates are worth building, and I will come back to ours. They are also a fair signal about the model: you test the output because nobody can reliably reason about the input.

Subcharts and value plumbing sprawl quietly

Small Helm setups look tidy. Then the platform grows, one chart wraps five subcharts, globals appear, child values get overridden from the parent, and a flag in one place changes behaviour three templates away.

Some of those values exist because users asked for them. Some exist because a subchart expects them. Some exist because somebody needed a workaround two years ago and nobody has ever felt brave enough to delete it. That is the point where a chart stops being a packaging unit and starts being institutional memory encoded as YAML conventions.

Helm 4 improved Helm without changing this

This is where lazy anti-Helm takes get sloppy, so let me be precise about it.

Helm 4.0.0 shipped in November 2025 and it is a genuine improvement: server-side apply, a redesigned plugin system with WebAssembly plugins, resource watching built on kstatus, post-renderers as plugins, reproducible chart archives. If you run Helm heavily, upgrade.

All of that is release machinery, though. Rendering still goes through Go's text templating, values.yaml is still the primary interface, and the combinatorial growth of flags and helpers in your own charts remains entirely your problem. Helm 4 made Helm better at the job Helm is good at. Chart authoring was never on the list.

A version bump on its own will not retire this argument, so here is what would: a release that swaps text templating for a typed object model, or one that makes schema validation mandatory instead of opt-in. If you are reading this a few releases later, that is the thing to go and check. Short of it, the version numbers in this post move and the point does not.

What the alternatives actually change

Different tools replace different parts of Helm, which is where most “Helm alternatives” posts go wrong. There is no swap that hands you better authoring, better packaging, better release management and better ecosystem compatibility all at once.

Tool	What it replaces	What you still need
Kustomize	Templating, for the overlay case	Packaging, versioning, release history
Timoni	The authoring model, with CUE typing	Ecosystem gravity, and patience while it is pre-1.0
cdk8s	Templating, with a real programming language	Packaging, plus a language runtime you now own
Jsonnet / Tanka	Templating, with a compact composition syntax	Types, and a way to stop dynamic behaviour relocating complexity
KCL	Templating, with a typed config language	Ecosystem gravity, and a team willing to learn a new language
Score	The developer-facing values surface	A platform implementation underneath that satisfies the spec

Kustomize is the one I reach for most often outside Helm, largely because it already ships inside kubectl and patches structured YAML instead of templating arbitrary strings. If your real problem is “I have plain manifests and need per-environment overlays”, it removes a surprising amount of cleverness for almost no adoption cost. Expecting it to do packaging too is how people end up disappointed by it.

Timoni is the most interesting answer to “what if we fixed the authoring model?”. CUE, real typing, modules distributed as OCI artifacts: conceptually most of what people are asking for. The caveat is maturity, and it is still on its 0.x line (v0.29.0 as I write this), so go and check where it has got to before committing. A 1.0 would change the calculation considerably more than another point release will.

The code-generation family trades typing for operational surface. cdk8s (joined the CNCF Sandbox in 2020) gives you real programming languages, Jsonnet with Tanka gives you a compact composition language, and KCL (joined in September 2023) gives you an explicitly typed configuration language. In exchange you own a language runtime, a build step, and an onboarding cost for everyone who ever touches a manifest. Good trade for a platform team with a large estate, poor trade for a team with six charts.

Score (joined in July 2024) sits one layer up. It is a workload specification, asking “what does this application need from the platform?” rather than “how do I render manifests for every deployment case?”. If your real complaint is that developers keep being handed an ever-growing values surface to fill in, that is the layer to attack. It still assumes a platform underneath that can satisfy it.

How we actually run this

We stayed on Helm, so it is only fair to say what containing the cost looks like day to day.

Most of our platform components are wrapper charts, and that pattern carries a lot of the weight. Chart.yaml declares a pinned dependency on the upstream chart, Chart.lock freezes what actually resolved, and our own templates/ directory layers on what upstream does not ship: the NetworkPolicies, the PrometheusRules, the RBAC that upstream leaves to you. Renovate raises the upstream bumps, and they do not land until CI is green.

Values sit in one place per chart. values.yaml carries the defaults every environment shares, and a values/ directory holds the environment-specific overrides beside it, one file each. CI then lints the chart once against every one of those files rather than once overall, so a change that only breaks one environment cannot ride in on another's values.

That catches everything Helm can be made to fail on. The quieter failure needs its own gate, because of a footgun worth internalising: Helm deep-merges maps, but it replaces lists outright. An environment file that overrides a list has to restate the entire list, so adding a component in one environment and forgetting another raises no error anywhere. Both files are valid. You just ship a shorter list to the environment nobody updated, and find out when something is missing. Nothing you can put in values.yaml expresses “these lists must stay in step”, so a parity check in CI does it instead, and a red PR replaces copy-paste discipline. That is the scaffolding tax from earlier made concrete: a shell script exists because the type system does not.

None of this makes the templating model typed. It makes the cost visible and bounded, which is the achievable goal rather than the ideal one.

Where I would draw the line

Helm is not technical debt because it exists in your stack, and authoring your own charts is not a mistake. We do both, and given the same constraints we would do both again. A third-party chart you install, version and occasionally override is packaging doing exactly the job it is good at, and a first-party chart is a reasonable way to ship your own software into a cluster where everything else already arrives as one.

So I reach for Helm when:

I am consuming third-party software that already ships a chart and I have no wish to become its packager
I want release history, rollback and the install workflow Helm already gives me
the surrounding ecosystem expects Helm artifacts, GitOps tooling included
I am packaging our own components for a platform where every other component already arrives as a chart

What accrues interest is the estate you build afterwards, and how much of it you let grow without a gate on it. Our largest first-party chart carries a values.yaml north of 800 lines. That is not automatically a failure, but it is a number I would want any team to be able to quote about themselves, because the moment nobody can tell you what a chart's values surface costs, you have stopped managing it. Same instinct as right-sizing the rest of your platform: the tool is fine, and the real question is whether the estate around it still fits the team.

The alternatives are worth knowing even when you stay. We have stayed, because packaging gravity and release history are worth more to us today than typed authoring would be. If that trade flips, if the estate outgrows its gates or Timoni reaches 1.0 with real ecosystem behind it, the honest move is to notice rather than to defend the original choice.

You do not have to become anti-Helm to admit that the charts you maintain can rot. In most Kubernetes estates that is hardly a hot take. It is just what year two looks like.

PodSecurityPolicy replacement: Pod Security Admission

Billy Walker — Mon, 31 Aug 2026 07:57:40 +0000

If you still think of Pod Security Admission as the cut-down replacement for PodSecurityPolicy, that is exactly the mindset that makes rollouts harder than they need to be. It is less powerful. That is also why it is practical. You get three fixed profiles, three modes, a handful of labels, and an admission controller that is already in the API server. No custom policy language, no mutation, no controller to install, and far fewer ways to surprise yourself mid-change.

The trick is to use that simplicity properly. A safe rollout is not “turn on restricted everywhere and hope for the best”. It is version-pinned labels, audit and warn first, then namespace-by-namespace enforcement once you have seen what will break. Done that way, you can move from the old PSP world to a built-in control that teams will actually leave enabled.

What replaced PodSecurityPolicy

PodSecurityPolicy was deprecated in Kubernetes v1.21 and removed in v1.25. Pod Security Admission became generally available in the same release, which matters because there is nothing extra to deploy: if your cluster is current enough, the mechanism is already there.

The model is deliberately small:

Profiles: privileged, baseline, restricted
Modes: enforce, audit, warn
Version pinning: pod-security.kubernetes.io/<mode>-version, pinned to a Kubernetes minor version or latest

Those labels live on namespaces:

# pin policy to the minor version your cluster is running
pss="$(kubectl version -o json | jq -r '.serverVersion.gitVersion' | cut -d. -f1,2)"

kubectl label ns payments \
  pod-security.kubernetes.io/enforce=baseline \
  pod-security.kubernetes.io/enforce-version="$pss"

That simplicity is the point. PSP mixed policy with RBAC, allowed mutation, and had enough moving parts that many teams either avoided it or never quite trusted what a change would do. PSA is narrower. It sets a floor.

The mental model: three profiles, three modes, one version pin

You can think of PSA as a matrix:

Mode	What happens
`enforce`	Rejects a non-compliant pod creation request
`audit`	Allows it, but records an audit annotation
`warn`	Allows it, but shows a warning to the caller

And the profiles are ordered from least to most strict:

Profile	What it is for
`privileged`	System namespaces and workloads that genuinely need host access or privileged behaviour
`baseline`	A sensible minimum floor for ordinary application namespaces
`restricted`	Current pod hardening best practice

For most application namespaces, the safe starting pattern is:

kubectl label --overwrite ns payments \
  pod-security.kubernetes.io/enforce=baseline \
  pod-security.kubernetes.io/enforce-version="$pss" \
  pod-security.kubernetes.io/warn=restricted \
  pod-security.kubernetes.io/warn-version="$pss" \
  pod-security.kubernetes.io/audit=restricted \
  pod-security.kubernetes.io/audit-version="$pss"

That gives you a firm minimum bar in enforce, while warn and audit show you what stands between the namespace and restricted.

The version pin is not optional bureaucracy. If you leave a namespace on latest, any cluster upgrade can quietly tighten policy under your workloads. Pinning to the minor version you are actually running means policy changes happen when you decide to move the label.

What `restricted` actually checks

The easiest way to lose credibility on this topic is to hand-wave the restricted profile and get one detail wrong. Several of the checklists floating around in other posts are simply false — especially around runAsUser and seccomp.

Here is the practical checklist, straight from the Pod Security Standards:

Control	What `restricted` expects
Volume types	Only `configMap`, `csi`, `downwardAPI`, `emptyDir`, `ephemeral`, `persistentVolumeClaim`, `projected`, `secret`
Privilege escalation	`allowPrivilegeEscalation: false` on every container, including init and ephemeral containers
Non-root	`runAsNonRoot: true` at pod level or per container
UID	`runAsUser` does not need to be set, but if it is, it cannot be `0`
Capabilities	Drop `ALL`; the only allowed add is `NET_BIND_SERVICE`
Seccomp	Must be explicitly `RuntimeDefault` or `Localhost`; leaving it unset is a violation under `restricted`

Two of those rows are where the bad checklists go wrong. runAsUser is not required — the policy forbids UID 0, it does not make you hard-code some other UID just to satisfy the admission check. And seccomp cuts the other way: under baseline an unset profile is fine, but under restricted it is a violation in its own right, which is one of the reasons plain demo manifests still bounce when teams first try this.

The rollout that does not break production

The safest PSA rollout starts with a dry run, not an enforcement change — the official migration guide is built around the same idea.

First, test label application server-side across all namespaces:

kubectl label --dry-run=server --overwrite ns --all \
  pod-security.kubernetes.io/enforce=baseline

That does not change anything. It asks the API server what violations it would report if the label were applied.

Then stage audit and warn cluster-wide:

kubectl label --overwrite ns --all \
  pod-security.kubernetes.io/audit=baseline \
  pod-security.kubernetes.io/warn=baseline

Or, if your end state is more ambitious, go straight to the pattern that tends to work well in practice:

enforce=baseline
warn=restricted
audit=restricted

That gives you signal without immediately creating an outage because some forgotten sidecar, root-running image, or hostPath mount was hiding in a namespace nobody had looked at for months.

You can also find namespaces that have no PSA labels yet:

kubectl get ns --selector='!pod-security.kubernetes.io/enforce'

Once the warnings are clean for a namespace, pin the version and turn on enforce there. Namespace by namespace is slower than one giant switch, but slower is exactly what you want when admission control is involved.

While the rollout is in flight, the API server's own metrics — pod_security_evaluations_total, pod_security_errors_total and pod_security_exemptions_total — are worth scraping. They tell you whether evaluation is happening where you think it is, and whether exemptions are starting to sprawl.

What a real rejection looks like

The good news about PSA failures is that the error message is the fix list.

Apply a plain pod into a namespace with enforce=restricted and you get something like this:

Error from server (Forbidden): pods "nginx" is forbidden: violates PodSecurity "restricted:latest":
allowPrivilegeEscalation != false (container "nginx" must set securityContext.allowPrivilegeEscalation=false),
unrestricted capabilities (container "nginx" must set securityContext.capabilities.drop=["ALL"]),
runAsNonRoot != true (pod or container "nginx" must set securityContext.runAsNonRoot=true),
seccompProfile (pod or container "nginx" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")

That message is unusually helpful by Kubernetes standards. Every violation names the exact field to set and the value it wants, so you can work through it like a checklist.

A minimal compliant pod looks like this:

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  securityContext:
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  containers:
    - name: nginx
      image: nginxinc/nginx-unprivileged:stable
      ports:
        - containerPort: 8080
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop:
            - ALL

Two quiet fixes are hiding in that example: the container-level security context, and the image itself. A lot of “hello world” manifests still assume a root-running image on port 80, which is a fine way to discover that restricted is not interested in your tutorial shortcuts.

The trap most teams hit once: Deployments apply, pods do not

This is the operational detail that makes or breaks a rollout plan.

warn and audit evaluate workload resources such as Deployment pod templates. enforce does not. enforce applies only to the resulting pod objects.

That means this can happen:

You apply a Deployment
Kubernetes accepts the Deployment
The ReplicaSet tries to create pods
The pods are rejected by PSA
You now have a deployment that looks “applied” but never becomes healthy

When that happens, the signal is in the ReplicaSet and the events, not in the initial kubectl apply output:

kubectl get rs -n payments
kubectl get events -n payments --sort-by=.metadata.creationTimestamp

If you remember only one rollout lesson from this post, make it this one. Admission on pod creation and admission on workload templates are not symmetrical.

The namespaces that should not be `restricted`

kube-system is the obvious example. CNIs, CSIs, node agents, and static pods routinely need things that baseline or restricted will reject: hostPath, hostNetwork, privileged containers, or other host-level access.

Be explicit about that. Label those namespaces privileged so the exception is documented in the cluster state rather than living as folklore.

Istio adds another common surprise — relevant if you run a mesh for zero-trust networking. With sidecar injection but without Istio CNI, the injected istio-init container needs NET_ADMIN and NET_RAW to set up traffic redirection. That is enough to fail baseline, not just restricted. The usual fix is to enable the CNI node agent and keep istio-system itself privileged.

Why version pinning matters more than it looks

A pinned PSA version is an operational control, not a documentation detail.

Kubernetes v1.34 added a baseline control that blocks the host field in httpGet and tcpSocket probes and lifecycle hooks. The reason is sensible: that field can be abused as an SSRF path through the kubelet. The operational consequence is that a cluster upgrade can start rejecting manifests that were fine the day before — if your labels track latest.

With version pinning, you choose when that tighter rule lands. With $pss still set to the minor version you were running before the upgrade:

kubectl label --overwrite ns payments \
  pod-security.kubernetes.io/enforce-version="$pss" \
  pod-security.kubernetes.io/warn-version="$pss" \
  pod-security.kubernetes.io/audit-version="$pss"

That makes upgrades much less dramatic. Upgrade the cluster first. Move the policy version when you are ready to deal with the findings.

The RBAC hole nobody should ignore

PSA policy lives on namespace labels. That means anyone who can update namespace labels can also weaken or remove enforcement.

Review that RBAC carefully. It is easy to focus on pod-creation rights and forget that update on namespaces is effectively policy-admin access.

There is a more central exemption mechanism through the API server AdmissionConfiguration, where you can exempt namespaces, usernames, or runtime classes. That is useful on self-managed control planes. On managed services such as EKS, GKE, and AKS, you usually do not get to edit that file at all, which means namespace labels are the real policy surface.

Where Pod Security Admission stops

PSA is the built-in floor. That is its job, and it is a good one.

There is no mutation, so no PSP-style defaulting is waiting to save a sloppy manifest. There is no custom rule language, and no way to express fine-grained per-workload exceptions inside one namespace. Once you want those things, you are into policy engines such as Kyverno or OPA Gatekeeper.

That is not a weakness so much as a division of labour. Use PSA to guarantee the cluster-wide minimum. Use a policy engine above it when you need mutation, richer exceptions, or application-specific constraints.

PodSecurityPolicy is gone. The sensible replacement is not to rebuild PSP in another tool and pretend nothing changed. It is to accept the smaller built-in model for what it is: predictable, cheap to operate, and strong enough to become the default floor across a cluster. That is a better trade than a perfect policy system nobody trusts enough to enable.

Kubernetes cost allocation with OpenCost

Billy Walker — Wed, 05 Aug 2026 08:12:08 +0000

You can have a perfectly healthy Kubernetes cluster and still have no idea who is driving the bill. The cloud invoice will tell you what the cluster cost. It will not tell you which namespace, workload, or team consumed that capacity, or how much of the spend was simply idle headroom nobody requested. OpenCost fills that gap. It gives you a vendor-neutral way to allocate Kubernetes cost down to cluster objects, query it over an API, and build showback without buying into a commercial platform first.

As of v1.121.0 (July 2026), the current release, OpenCost is a CNCF Incubating project. The useful mental split is straightforward: OpenCost is the open-source engine and specification; Kubecost is the commercial product line in the same space. The OpenCost repository notes that the project was originally developed and open-sourced by Kubecost, while Kubecost's public site is now branded IBM Kubecost. For this post, we are staying on the OSS side: self-hosted OpenCost, standard Helm install, standard API, standard Prometheus queries.

The allocation model is the part to understand first

If you only remember one detail from OpenCost, make it this one: workload cost is not just raw usage. In the OpenCost specification, workload costs are defined as max(request, usage) for resources with allocation costs such as CPU and GPU. That matters because the bill follows reserved and allocatable capacity rather than whatever a container happened to burn in a five-minute slice.

A pod that requests 2 CPU and uses 200m is still consuming scheduling capacity someone else cannot have. OpenCost treats that as real cost, which is why its numbers are useful for showback and bill reconciliation instead of just efficiency charts.

Idle cost is the second half of the picture. The specification defines cluster idle cost as:

Cluster Idle Cost = Cluster Asset Costs - Workload Costs

That is the spend you are carrying in the cluster but have not allocated to workloads. It is the cost of spare node capacity, overprovisioned requests, and all the other room you are paying for so the platform can absorb change.

OpenCost also treats shared costs as first-class. The specification calls out three common ways to distribute them:

uniformly across tenants
proportionally to a tenant's consumption
by a custom metric such as network egress

That means the honest answer to “what does this namespace cost?” is often “its direct workload cost, plus its share of idle and shared platform cost”. If you skip that second part, your showback report will look cleaner than the bill you actually pay.

Install it with Helm against an existing Prometheus

The current Helm docs assume you already have Prometheus. That is still the normal path.

helm repo add opencost-charts https://opencost.github.io/opencost-helm-chart
helm repo update

Then point OpenCost at the Prometheus service it should read from:

opencost:
  prometheus:
    internal:
      namespaceName: monitoring
      serviceName: prometheus-kube-prometheus-prometheus
      port: 9090
  exporter:
    defaultClusterId: production-eks

And install it:

helm install opencost opencost-charts/opencost \
  --namespace opencost \
  --create-namespace \
  -f values.yaml

That gives you the full OpenCost deployment plus the API and UI. The docs use localhost:9003 as the default API address, so a quick validation path is:

kubectl -n opencost port-forward deployment/opencost 9003 9090

Port 9003 is the API; 9090 is the UI.

There is a lighter path worth knowing about too. The Prometheus integration docs note that you do not need a Prometheus client just to emit cost metrics, and a "Promless" mode has been taking shape across releases since late 2025. Treat that as a fast-moving area rather than a settled default. If you want the most documented path today, use the Helm install against an existing Prometheus stack. If you want the leaner route, pin the version and read the release notes for the exact build you are adopting.

Start with the Allocation API, not a hand-rolled dashboard

The fastest way to get useful showback is the Allocation API. It already applies the OpenCost cost model, which means you are querying the model's own output instead of rebuilding the logic yourself out of raw metrics.

A seven-day namespace breakdown looks like this:

curl 'http://localhost:9003/allocation?window=7d&aggregate=namespace&shareIdle=true'

Two parts of that query earn their keep:

aggregate=namespace gives you a shape platform teams can actually hand to someone. Most organisations do not want “cost by pod” as the first report; they want “what did team-a's namespace cost this week?” Namespace is a good default grain because it is close enough to ownership to start a useful conversation.
shareIdle=true is what keeps the number honest. Without idle-cost sharing, namespace totals can look impressively low while a large chunk of the bill sits outside the report as unallocated capacity — fine for an efficiency drill-down, but weak showback.

If you label workloads by team, you can roll the same data up at the ownership layer instead:

curl 'http://localhost:9003/allocation?window=7d&aggregate=label:team&shareIdle=true'

The Allocation API accepts label-based aggregation, so you are not forced into namespace as the only billing dimension. That is useful when a single team owns several namespaces, or when shared namespaces are unavoidable and labels are the cleaner boundary.

This is the point where OpenCost usually becomes operationally useful. You stop arguing about the cluster bill as one opaque number and start asking better questions: which teams carry the most idle share, which namespaces reserve far more than they use, and which platform defaults are making the bill harder to attribute than it should be.

Prometheus is still useful, but use it for the right layer

OpenCost also exposes metrics for Prometheus, which is what you want for dashboards and trend lines.

For a simple monthly node-cost view, the documentation's example is:

sum(node_total_hourly_cost) * 730

That is deliberately blunt. It tells you what the currently provisioned nodes cost over a rough 730-hour month. It is good for keeping an eye on baseline cluster spend, but it is not showback yet.

For a namespace-level dashboard, you can build a query from OpenCost's allocation and node-cost metrics, for example:

sum by (namespace) (
  container_cpu_allocation * on (node) group_left node_cpu_hourly_cost +
  container_memory_allocation_bytes * on (node) group_left node_ram_hourly_cost / (1024 * 1024 * 1024)
)

That is closer to what most teams expect to see in Grafana: cost broken down by namespace, derived from allocation metrics multiplied by node pricing. It is useful for trends, deltas, and “what changed this week?” views.

The caveat is which layer you treat as authoritative. When you need a number to hand to finance or to another engineering team, start from the Allocation API. When you need a graph that shows whether cost by namespace is climbing, Prometheus is the right tool.

The honesty box: what the numbers are, and what they are not

Two assumptions trip people up:

That the allocation metrics are pure usage metrics. They aren't. In the current code (v1.121.0) the synthetic container_cpu_allocation and container_memory_allocation_bytes metrics are built from max(request, usage), matching the specification — which is exactly why they beat a plain CPU-usage graph for cost allocation.
That pricing is magically exact out of the box. OpenCost's own API docs describe the standard reporting path as on-demand list pricing, with cloud-provider cost-and-usage data available through integrations. The allocation model is useful immediately, but to reflect your committed discounts, private pricing, or billing exports, you have to wire that data in.

Neither of these is a flaw. They are just the boundaries of what the tool can do for you automatically.

Showback first, chargeback if your organisation actually wants it

FinOps discussions often make chargeback sound like the inevitable end state. In practice, most teams should start with showback.

Showback means the spend is visible to the people who influence it. The namespace or team sees a monthly figure, understands what drove it, and can act on it. Chargeback adds the accounting decision on top: that cost is posted back to a budget, cost centre, or P&L.

The FinOps Foundation's allocation guidance frames allocation as the foundation for either route. That is the useful way to think about OpenCost as well. It gives you a defensible allocation model and enough reporting surface to support showback immediately. Whether you turn that into formal chargeback is an organisational decision, not a Kubernetes feature flag.

The practical takeaway

If your current Kubernetes cost story is “the cluster costs £X and nobody can explain why”, OpenCost is a solid place to start. Install it against the Prometheus stack you already run, use the Allocation API for the numbers you want people to trust, include idle cost when you report by namespace or team, and use Prometheus for dashboards rather than trying to rebuild the cost model yourself. That gets you from one undifferentiated cluster bill to a showback report people can actually argue with usefully, which is a much better place to be than arguing with the invoice alone.

Nobody warns you about year two of platform engineering

Billy Walker — Thu, 30 Jul 2026 08:43:31 +0000

Year one of a platform gets a budget, a roadmap, and a launch deck. Year two gets the support queue, the upgrade backlog, and the first awkward realisation that the golden path you shipped six months ago is already drifting away from the estate it was meant to simplify. That is the part nobody budgets for properly: a platform is a product, so shipping it once is the same mistake as shipping any product once.

This is also why so many internal platforms feel promising in the demo and frustrating a year later. The visible work in year one is building: templates, guardrails, self-service workflows, maybe a portal, maybe a paved road people can actually use. The invisible work in year two is keeping the thing alive while the world underneath it keeps moving: Kubernetes releases, Helm chart churn, identity changes, cloud policy updates, framework upgrades, and the steady trickle of support requests from teams whose use case never quite fit the happy path.

Platform engineering does not usually fail because the idea was wrong. It stalls because teams fund the build and underfund the maintenance.

The launch was the easy part

There is a familiar shape to the first year of a platform effort.

A team gets a mandate to improve developer experience, standardise delivery, or reduce ticket-driven toil. It has executive attention. It has a date to hit. It has a story people can understand: we are building the internal platform.

That phase is real work, but it has one advantage: the scope looks finite. You can imagine a first version. You can plan the launch. You can decide what the thinnest viable platform looks like and cut everything else.

Year two is harder because the work stops looking like a project and starts looking like ownership.

Now you are carrying things like:

patching and upgrading the platform's own dependencies
repaving templates and starter repos as upstream defaults change
deciding which one-off exceptions should become product features
pruning workflows nobody uses before they become permanent support burdens
proving that adoption is still growing, rather than assuming it is

None of that produces the same tidy before-and-after slide as the original launch. It is still the work that determines whether the platform becomes part of how the organisation ships software or just another ambitious internal system people route around.

That is why Team Topologies keeps insisting on platform as a product rather than platform as a shared tools pile. The point is not that platforms need prettier language. The point is that products have ongoing discovery, maintenance, and evolution. A platform that is not continuously evolved is just a frozen opinion about how engineering used to work.

Most of the field is only just getting to year two

One reason there is so little useful day-two writing is that the discipline itself is still young.

The 2024 State of Platform Engineering report from Humanitec and Gitpod, based on a vendor-sponsored survey of platform teams, found that 56% of respondent teams were less than two years old, and only 13% had been doing platform engineering for more than five years. That is enough to explain why there is still so little hard-won writing about the maintenance phase: a lot of the field is only just arriving there.

In other words, there is not a deep institutional playbook here. There are patterns. There are good principles. There is some solid research. But there is also a lot of collective improvisation.

That matters because year-two problems do not look like year-one problems.

The first year asks whether you can build a paved road. The second asks whether you can keep repaving it while traffic is already on it.

The first year asks whether you can make self-service possible. The second asks whether the self-service path still fits the organisation as it changes.

The first year asks whether the platform team can ship. The second asks whether it can avoid becoming the bottleneck it was created to remove.

Golden paths do not stay golden by themselves

A platform's main promise is usually straightforward: give teams a safer, faster default path than stitching everything together from scratch.

That is a good promise. It is also one with a permanent maintenance bill attached.

The paved road starts decaying the moment upstream moves. A base chart changes shape. A cluster policy tightens. A framework introduces a new recommended default. A cloud provider deprecates some corner of the identity model you buried in a template months ago. A service team needs one capability the platform path did not anticipate, so it forks the template "just for now" and never really comes back.

This is where platform credibility leaks away quietly. Not in one dramatic outage, but in a long series of tiny moments where engineers learn that the official path is slightly stale, slightly rigid, or slightly behind the reality of the stack. Once that happens often enough, people stop asking whether they should use the platform and start assuming they will need to work around it.

That is also why the advice in Stop copying big-tech platform architecture: right-size to your scale matters even more in year two than in year one. Over-built platform machinery does not just cost more to launch. It costs more to keep current. Every extra control plane, abstraction layer, and bespoke workflow becomes another thing that needs tending after the excitement has worn off.

A golden path without a gardener is just a future migration project.

The upgrade treadmill is not a side quest

If you want one brutally practical example of year-two reality, use Kubernetes itself.

Kubernetes now ships three minor releases a year, and the project maintains patch support for only the most recent three minor versions. Under the yearly support policy that works out to roughly a 14-month window per release: twelve months of standard patches, then two months of maintenance mode before end of life. If your platform sits on Kubernetes, the upgrade tax is not occasional. It is structural. You are budgeting for two to three cluster upgrades a year, forever, before you even count the other components attached to the platform.

And of course it is never just Kubernetes.

There is ingress or Gateway API, cert management, policy tooling, observability agents, CI runners, secret syncing, identity plumbing, image build tooling, and whatever home-grown wrappers your platform created around those things to make them usable. One upstream change can ripple through templates, documentation, tests, and support expectations all at once.

Teams often talk about this as if it were operational overhead adjacent to the roadmap. It is not. For a platform team, the upgrade treadmill is part of the roadmap. If you do not plan for it explicitly, it eats the roadmap anyway.

This is the part that breaks year-two planning. A team thinks it has funded one staff engineer to "keep the lights on" while everyone else moves on to the next strategic thing. Then the calendar turns, the supported version window narrows, and suddenly the strategic thing is an upgrade programme whether anyone likes it or not.

Adoption stalls when nobody is listening

There is a second year-two trap that looks softer but is just as destructive: the team assumes that shipping the platform was the same thing as winning adoption.

It is not.

DORA's 2024 Accelerate State of DevOps report is useful here because it is more honest than most of the hype. It found that internal platforms were associated with better individual productivity and team performance, but also with weaker throughput and change stability where the implementation added complexity or reduced developer independence. DORA describes this as a J-curve: you can see early gains, then hit a dip, and only recover if the platform matures in a user-centred way.

That finding lands because it matches what many teams feel in practice. A platform can absolutely remove toil. It can also create a new layer of process, abstraction, and waiting if the product discipline is weak.

The warning signs are usually familiar:

teams need tickets for routine changes the platform was meant to self-serve
exceptions pile up because the happy path is too narrow
the platform roadmap is driven by internal architecture debates rather than user pain
the team measures launches and features, but not adoption or abandonment

This is why Team Topologies talks about sensing user needs rather than just building services for them. The thinnest viable platform is only a starting point. If you stop there, the platform freezes while the organisation keeps changing around it.

Puppet's 2024 State of DevOps report makes the same point from a different angle: 52% of respondents said a product manager was crucial to a platform team's success, with a further 21% calling one nice to have. That is not a plea to add ceremony. It is a recognition that somebody needs to own adoption, prioritisation, and the discipline of learning what the users of the platform actually need next.

Without that function, year two becomes a familiar kind of drift. The platform team keeps shipping things. The users keep needing different things. Everyone is technically busy, and the gap still widens.

This is how the platform team becomes the bottleneck again

Platform engineering is often sold as the way out of ticket Ops. Build the paved road, standardise the sharp edges, push routine work into self-service, and let product teams move without waiting for a central platform group.

That is the right goal. But an under-staffed year-two platform team can relapse into the exact behaviour it was supposed to replace.

You see it when every unusual request becomes a handoff. You see it when upgrades consume so much attention that user-facing platform work slips for a quarter. You see it when the team starts protecting itself with queues and process because it no longer has the slack to evolve the underlying system.

At that point the platform still exists, but the interaction model has changed. The platform is no longer a capability. It is a department.

And yes, that sounds harsh. It is still a useful test. If the main experience of the platform for most engineers is waiting for another team, something important has gone wrong even if the underlying architecture is perfectly respectable.

What year-two funding actually looks like

If the first-year mistake is treating the platform like a one-off delivery project, the year-two fix is not mysterious. It is just less glamorous than the original launch.

A serious year-two platform budget needs to fund four things explicitly:

maintenance as first-class work: upgrades, repaving, dependency churn, template hygiene, and pruning stale paths
product management: somebody accountable for adoption, prioritisation, and the sensing loop with platform users
operational slack: enough capacity that support and upgrades do not consume every sprint
evidence: metrics for adoption, developer independence, and the support burden the platform is actually removing

That last point matters because year-two platform conversations often become strangely qualitative. People can tell that the platform feels helpful or painful, but the organisation stops short of measuring the signs that matter.

A better year-two review asks questions like:

how many teams are still on the paved path versus a forked one?
which support requests repeat often enough to deserve product work?
how old are the core templates and platform dependencies?
where does a platform user still need a human instead of self-service?
did the last upgrade consume planned capacity, or did it blow up the quarter?

Those are product questions as much as technical ones. Which is the point.

The real work starts after the applause

Platform engineering year two is where the discipline becomes real. The launch matters, but the launch is only the opening move. After that comes the unglamorous part: keeping the path current, paying the upgrade tax, listening hard enough to notice when adoption is flattening, and funding the platform team as owners of a product rather than custodians of a project.

If you get that part right, the J-curve recovers and the platform becomes genuinely compounding infrastructure for the organisation. If you get it wrong, the platform slowly turns back into the queue it was meant to remove.

Nobody warns you about year two because year one is easier to tell as a story. Year two is mostly maintenance, judgement, and sustained ownership. It is also the part that decides whether the platform was ever really a platform at all.

Observability in the age of LLMs: tracing what your AI actually did

Billy Walker — Tue, 28 Jul 2026 08:24:40 +0000

A feature built on an LLM can hit every familiar SLO and still be failing in the way that matters. The API is up. The median latency looks fine. Error rate is flat. Meanwhile the model is taking the scenic route through three tool calls, burning tokens, and returning answers you would not want a customer to trust. The thing you need to see is not just whether the service is healthy. It is what the system actually did.

That is why LLM observability is best thought of as an extension of ordinary tracing, not a fresh religion with its own priesthood. An LLM call is a span. An agent run is a span tree. The useful questions are still familiar: where did the time go, which step failed, what changed, and how much did this request cost? What changes is the shape of the work. Now you also care about token usage, model choices, tool execution, and whether the answer was any good.

If you already run OpenTelemetry, that is good news. You do not need to throw away your tracing model to observe AI features properly. You need to extend it.

Why classic observability stops short

Traditional service telemetry assumes that the interesting failures are mostly about infrastructure or request handling: timeouts, retries, saturation, bad downstreams, deployment regressions. Those problems still exist in LLM-backed systems, but they are no longer the whole story.

A generated answer can be wrong while every system metric stays green. A retrieval step can bring back unhelpful context without producing an obvious exception. An agent can choose a slow tool path, or call the same tool twice, or hand the model a bloated prompt that quietly doubles token spend. Keep the old graphs; just stop expecting them to catch this class of failure.

OpenTelemetry made the same point in its 2026 GenAI observability post: every application call to an LLM hides "a chain of model calls, tool invocations, and token exchanges" behind the scenes. That is the part standard web-service monitoring does not naturally reveal. When a request takes 45 seconds, the question is no longer just "which service was slow?" but "which decision inside the trace made it slow?"

This is the same reason fast feedback loops matter more when LLMs write your code. When generation gets cheaper, validation becomes the scarce resource. In production, observability is one of those validation loops.

The debugging primitive is still the trace

There is a mild industry temptation to treat LLM systems as too exotic for the tooling we already have. That is usually the wrong instinct.

A useful mental model looks more like this:

agent.run
├── llm.chat openai/gpt-5.6
├── tool.retrieval postgres/pgvector
├── llm.chat openai/gpt-5.6
├── tool.http billing-api
└── llm.chat openai/gpt-5.6

That tree is already enough to answer a lot of awkward production questions:

which step dominated latency
whether the model asked for tools more than once
whether retrieval happened before the answer changed course
whether one provider or model variant behaved differently
where token usage and therefore spend started to climb

None of that needs a brand-new theory of observability. It just means treating model calls and tool calls as first-class trace events instead of leaving them hidden inside application logs or framework internals.

That framing matters because it keeps you portable. If the trace is the source of truth, you can ship it to a general-purpose backend, to an AI-focused product, or to both. If the observability story starts and ends inside one vendor's SDK, you have made your debugging model harder to move.

What the OpenTelemetry GenAI conventions give you

The concrete standard here is the OpenTelemetry semantic conventions for Generative AI. As of July 2026 the gen_ai work has moved into its own dedicated repository, semantic-conventions-genai, and every document in it is still marked Development rather than stable. The new repository does not even have a tagged release yet, so there is no version number to pin against; the honest options are pinning a commit, or recording the last main-repo release that carried the gen_ai conventions before the move. That does not make them useless. It means you should adopt them with your eyes open and expect some movement.

You do not need to memorise every field. The useful thing is the shape of the telemetry the conventions are trying to standardise.

For spans and attributes, the useful core includes fields such as:

gen_ai.operation.name for the kind of model operation
gen_ai.provider.name for the provider
gen_ai.request.model and gen_ai.response.model
gen_ai.usage.input_tokens and gen_ai.usage.output_tokens
gen_ai.response.finish_reasons
gen_ai.tool.name for tool execution
gen_ai.agent.name and gen_ai.conversation.id when you need to group larger workflows

For metrics, the current GenAI docs define client-side measurements such as gen_ai.client.token.usage and gen_ai.client.operation.duration.

That is enough to capture the parts of an LLM request you usually end up caring about first: how many tokens went in, how many came out, how long the operation took, which model handled it, and where tool execution sat in the middle.

It is also worth being precise about one thing the conventions do not settle for you: cost. Cost is still best treated as a derived metric rather than a magical standard field. In practice that means joining token counts to your model pricing table and calculating spend from there. If someone promises a universal LLM cost number with no context, raise an eyebrow.

Instrument these first

You can spend a lot of time arguing about evals, guardrails, and which agent framework is fashionable this month. Instrumentation is less glamorous, but it is the bit that keeps the rest honest.

If you are tracing LLM-backed features today, start with these:

One span per model call. Capture provider, model, operation, latency, finish reason, and token counts.
One span per tool call. Retrieval, HTTP calls, database lookups, and internal helpers should sit in the same trace, not in a separate logging universe.
Prompt and response references, not reckless raw dumps. Keep enough data to debug, but be deliberate about privacy, redaction, and storage costs.
A derived cost metric. Token counts are useful; token counts joined to pricing are useful to finance and engineering at the same time.
Quality signals beside the trace. User rating, evaluator score, moderation outcome, or task success should sit close enough to the trace that you can compare bad answers with the path that produced them.

Getting the first span is cheaper than most teams expect. If your stack is Python calling OpenAI, the contrib instrumentation does it with zero code changes:

pip install opentelemetry-distro opentelemetry-instrumentation-openai-v2
OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true opentelemetry-instrument python app.py

Note the environment variable. Prompt and completion content is off by default and opt-in, which is exactly the right default: you get spans, models, token counts, and finish reasons without capturing user text, and you turn on content capture deliberately, with redaction and retention already thought through.

That is the practical core. You can add fancier workflow views later.

Experimental is fine. Invisible is not.

Some teams hesitate because the GenAI conventions are still moving. That is a real caveat, but it is not a good reason to keep model behaviour opaque.

The honest trade-off is this:

if you wait for the conventions to feel perfectly settled, you stay blind longer
if you adopt them now, you may need a migration later

I would take the migration.

A moving standard is still much better than bespoke telemetry that only one SDK understands. The conventions already give you a common vocabulary for spans, token usage, and model metadata. Even if a field changes later, the shape of the problem is now visible in your traces instead of trapped inside ad hoc logs.

The practical discipline is boring, which is usually how you know it is right:

record the commit or snapshot of the conventions you instrumented against, since there is no tagged release to pin yet
note that it is a Development-status schema
keep your instrumentation behind a thin abstraction where you can
re-check the docs before each major upgrade

That is manageable. Debugging a production agent system with no trace of its internal choices is much less manageable.

The AI observability tools are mostly OpenTelemetry underneath

Once you start looking around, the AI observability market can seem louder than the underlying technical differences justify.

There are good tools here. Langfuse's own docs position its OpenTelemetry integration around ingesting OTLP traces for LLM observability. MLflow describes its tracing product as OpenTelemetry-compatible and aimed at LLM and agent observability. OpenInference, from the Arize ecosystem, describes itself more plainly still: OpenTelemetry instrumentation for AI observability.

That is the important pattern.

The credible tools are increasingly building on OpenTelemetry rather than asking you to abandon it. What they compete on is everything around the wire format:

workflow visualisation
evaluator and feedback loops
prompt inspection
dataset and experiment tooling
agent-specific views
hosted versus self-managed trade-offs

Those are meaningful differences. They just are not a reason to skip portable traces.

If you already have an observability stack you trust, start there. Send the traces somewhere you can query them. Add a specialised tool when you have a concrete gap: better prompt inspection, easier annotation workflows, richer experiment tracking, or an agent timeline that your current backend does not give you.

What I would not do is buy an AI observability platform before I had first proved that we were actually capturing the right spans.

Agent observability is where this gets interesting

Plain LLM tracing is useful. Agent tracing is where the debugging payoff gets larger.

The difficult production questions are rarely about one completion in isolation. They are about execution paths:

why did this agent choose that tool
why did it call the same tool twice
why did it stop early
why did one branch of the workflow take ten times longer than the others
which step turned a reasonable user request into an expensive one

That is why the recent push toward agent timelines is worth paying attention to. Honeycomb launched its Agent Observability features on 12 May 2026, with an Agent Timeline view that is basically a more opinionated way to walk the same trace tree. That kind of view earns its keep because agent systems are not just slow or fast. They are branching, recursive, and occasionally a bit absurd.

The frontier here is less about inventing new telemetry categories than about making traces easier for humans to read. Multi-agent systems create deeper trees, more spans, and more "why on earth did it do that?" moments. The backend that wins will be the one that makes that path legible under pressure.

What to do this week

If you are shipping LLM-backed features and do not yet have a plan, the sensible order of work is refreshingly unromantic:

instrument model calls as spans
instrument tool calls in the same trace
capture token counts and duration
derive cost from usage and pricing
attach one quality signal you trust
only then decide whether you need a specialised platform

That sequence gets you out of the worst failure mode, which is running a system that can think, call tools, and spend money without leaving behind an explanation.

LLM observability is not separate from good observability practice. It is what happens when the service starts making choices on your behalf. Start with traces. Make token usage and cost visible. Keep the data portable. Then, if you need a shinier lens on top, add one later.

Stop copying big-tech platform architecture: right-size to your scale

Billy Walker — Sat, 11 Jul 2026 07:57:12 +0000

A lot of platform over-engineering starts with a diagram that looks serious. A small team has a few services, a modest growth curve, and perfectly ordinary operational needs, but the design doc still reaches for the full hyperscaler silhouette: Kubernetes everywhere, a service mesh, Kafka, multi-cluster, and a bespoke internal platform before anyone has proved they need one. The problem is not that those tools are bad. It is that they are answers to constraints most teams do not have. Right-sizing platform architecture means copying the principles behind big-tech systems, not the stack itself.

That becomes expensive quickly because complexity is not free. Every extra control plane becomes another upgrade path, another source of pages, another thing a new engineer has to understand before they can safely ship a change. Big companies take on that cost because, at their scale, it often pays for itself. Small and mid-sized teams usually take it on up front and hope the benefits show up later.

They often do not.

Big-tech architecture is the output of big-tech constraints

It helps to say the quiet part out loud. Google, Amazon, and the rest did not end up with sprawling platform stacks because they enjoy making life difficult. They built those systems under pressures most teams will never see all at once:

traffic and data volumes that make single-node or single-database assumptions collapse
hundreds of teams that need to change software independently
regulatory or blast-radius boundaries that force hard isolation
dedicated platform and reliability teams to operate the machinery

If your team has three product engineers and one person who gets the on-call phone, you are operating under a different set of forces. Copying the architecture without copying the forces is how you end up with enterprise-grade overhead and startup-grade staffing.

That is also the practical reading of Conway’s Law. System design follows communication structure. A microservice estate with separate deployment pipelines, cross-cutting policy layers, and multiple control planes assumes an organisation shaped to own those boundaries. If you do not have that organisation, the elegant diagram turns into one team carrying all of the complexity by hand.

Copy the principles, not the implementation

The useful parts of big-tech engineering are usually the ideas underneath the stack choice.

You can borrow those almost anywhere:

reduce cognitive load so engineers can ship without learning every platform detail
improve feedback loops so changes are cheap to validate and rollback
create clear boundaries between services, teams, and operational ownership
standardise the paved road so the safe path is also the easy path
treat cost as a design constraint, not something to apologise for later

What does not travel as cleanly is the implementation detail. Service mesh, multi-cluster, event streaming everywhere, and a bespoke internal developer platform are all legitimate responses to particular thresholds. They are poor defaults.

This is where Dan McKinley’s Choose Boring Technology still lands so cleanly. His line about companies having roughly three innovation tokens has aged well because the underlying point is still right: novelty is expensive, and you should spend it where it creates advantage rather than where it merely creates architecture. “Boring” in McKinley’s sense is not old or unfashionable. It is technology whose failure modes are already well understood.

For platform teams, that is a strong default. If the thing you are building is not your differentiator, buying more operational complexity than you need is rarely a flex. It is a loan.

“You are not Google” is a useful design rule, not a slogan

Oz Nova’s You Are Not Google remains one of the clearest antidotes to architecture cargo culting because it forces a more honest sequence of questions. What problem are you actually solving? Which candidate solutions exist? What historical pressure produced the famous answer everyone keeps citing? Does that answer fit your team, budget, and operational maturity now?

Those questions are annoyingly effective because they make a lot of platform theatre disappear.

A Kubernetes platform is a good example. Plenty of teams do need Kubernetes. Plenty also adopt it because it feels like the proper destination for anything serious. The real question is not “is Kubernetes modern?” It is “what operational problem does this solve for us today that simpler deployment machinery does not?” If the answer is mostly social proof, that is a warning light.

The same applies higher up the stack. A bespoke internal platform can be the right move when many teams need a common workflow and the cost of hand-held support keeps rising. But Team Topologies’ thinnest viable platform is a better starting point than an immediate platform product. Sometimes the right platform is a template, a few scripts, and good docs. It does not have to begin life as an ecosystem.

The Prime Video story is a correction, not a manifesto

The most abused recent example in this space is Prime Video’s write-up on reducing the cost of one monitoring service by more than 90%. The point of that post was not “Amazon abandoned microservices” and it certainly was not “monoliths always win”. It was narrower, and more useful.

One team had built a distributed serverless design around Step Functions, Lambda, and S3. At the workload they were handling, the orchestration overhead had become the cost problem. They moved to a single process running on ECS and passed data in memory instead of via a chain of external services. The result was a cost reduction above 90% for that tool.

That matters because it shows what right-sizing actually looks like in practice. The original design was valid in the abstract. It was just mis-sized for the workload and economics in front of the team. Adrian Cockcroft’s corrective take on the reaction was useful here too: this was a refactoring decision inside one service, not a grand ideological retreat.

The lesson is not “always build a monolith”. It is “distributed systems premiums only make sense when you are actually collecting the benefit”.

Martin Fowler has been making the same point for years. Monolith First is still one of the sanest defaults in software architecture because a modular monolith lets you prove boundaries before turning them into network calls. You can extract a service later. You cannot cheaply un-distribute a badly fragmented system that nobody really needed.

A right-sizing table for platform teams

If you want one decision aid from this post, make it this: treat big-tech patterns as threshold tools, not identity markers.

Pattern people cargo-cult	Right-sized default for most teams	Real signal the bigger pattern may now pay off
Service mesh	Use ingress, `NetworkPolicy`, and straightforward app-level telemetry first. Add mTLS or traffic shaping only when you can name the requirement.	You need per-service identity, consistent mTLS, or progressive traffic control across many teams and many services.
Multi-cluster Kubernetes	Start with one well-run cluster and strong namespace, RBAC, quota, and environment boundaries.	You have hard isolation needs: data residency, materially different blast radii, or control-plane scale limits.
Kafka for everything	Start with a managed queue, an outbox pattern, or even Postgres when throughput and replay needs are modest.	You genuinely need a durable, replayable event log with multiple independent consumers and sustained high throughput.
Bespoke internal developer platform	Start with a thinnest viable platform: templates, automation, docs, and one paved path that removes toil.	Many teams are repeating the same workflow, support demand is compounding, and a platform product will clearly save more time than it costs to run.
Microservices from day one	Start with a modular monolith and explicit internal boundaries.	Teams need independent deployability, boundaries are already clear, and you have the operational discipline to support distributed systems well.

This is also where the DORA research is often flattened into the wrong slogan. High-performing teams benefit from loose coupling and independent deployability. That does not automatically mean “split everything into microservices”. You can get plenty of that benefit from a well-structured monolith, a clean delivery pipeline, and sensible ownership boundaries.

The label matters less than the change cost.

Cost, team shape, and on-call load should be first-class inputs

Werner Vogels’ Frugal Architect material is useful beyond AWS because it frames architecture as a series of trade-offs rather than a hunt for the most sophisticated-looking answer. That sounds obvious, but teams forget it surprisingly easily when a design starts accumulating fashionable parts.

A few questions cut through the theatre quickly:

What does this add to the pager surface area?
Who upgrades it, patches it, and debugs it at 02:00?
What skill set does it require from every new engineer?
What failure mode becomes easier because we adopted it?
What failure mode becomes harder or more likely?
Could a simpler design buy us another year before this complexity is justified?

If those questions make the design feel awkward, that is usually a useful signal rather than conservatism.

This is especially true in platform engineering because platforms quietly multiply costs across the organisation. A hard-to-understand abstraction does not stay local to the team that built it. Every product engineer pays rent on it.

When the bigger architecture is the right call

None of this is an argument for permanent smallness or for avoiding ambitious systems. Some teams absolutely do cross the threshold where the heavier pattern is worth it.

You are probably getting there when several of these are true at the same time:

multiple teams are blocked on each other because deployments cannot move independently
reliability problems are coming from shared blast radius rather than application bugs
platform support has turned into a queue because manual help does not scale
data residency, compliance, or tenancy requirements demand hard technical boundaries
workload volume has made the simpler option materially more expensive or less reliable

At that point, more machinery can be the cheaper choice overall. But it is still better to adopt it deliberately than to inherit it by imitation.

That is the part people miss when they turn “choose boring technology” into a bumper sticker. McKinley’s innovation tokens exist so you can spend them where they matter. The platform question is whether you are spending one on a real constraint or on architecture envy.

The sophisticated choice is often the smaller one

The cheapest architecture that meets your actual constraints is not the amateur option. Usually it is the disciplined one.

Small teams do not fail because they lacked enough control planes. They usually fail because they made delivery too slow, operations too fragile, and ownership too muddy for their actual size. Big-tech architecture can solve those problems at big-tech scale. At smaller scale, it often creates them.

So copy the habits that travel: clear interfaces, good defaults, fast feedback, cost awareness, and platforms that remove toil rather than advertise sophistication. Leave the rest until the numbers, the team shape, and the pager all agree that you have earned it.

Progressive delivery on Kubernetes with Argo Rollouts

Billy Walker — Thu, 09 Jul 2026 19:05:06 +0000

Most teams meet the limits of a plain Kubernetes Deployment at the worst possible moment: mid-rollout, watching a new version march out to every pod while a graph slowly turns the wrong colour. A Deployment rolls pods out carefully enough — it respects readiness, it surges and drains in order — but it has one stubborn blind spot. Once the new ReplicaSet is Ready, it keeps going unless something stops it. There is no built-in way to say “send 10% of traffic here, wait, check an SLO, then continue”, or “only promote this preview stack if the metrics hold up”.

That missing control loop is exactly what Argo Rollouts adds. It keeps the Deployment mental model you already know and layers canary, blue-green, analysis, and automated rollback on top. For platform teams, that quietly turns rollout safety from a runbook someone has to remember into a controller that does it the same way every time. As of Argo Rollouts v1.9.0, that controller can drive percentage-based canaries, blue-green cutovers, and metric-gated promotion without asking you to rip out the rest of your Kubernetes delivery stack.

The gap in a normal Deployment

A Deployment is genuinely good at one thing: swapping one ReplicaSet for another while keeping enough healthy pods around to serve traffic. Readiness probes, maxUnavailable, and maxSurge all pull their weight here. What none of them give you is a first-class sense of whether the new version is actually any good.

That distinction matters most when “pods are Ready” and “this version is safe” quietly stop meaning the same thing. A service can start up cleanly and still come apart under real load. Error rate can sit flat through the first few requests and then climb once traffic arrives in earnest. Latency can drift at the 95th percentile while the average stays there looking reassuring. And a bad release often hurts just one path through the app — which is cold comfort when a straight rollout has already reached every user before anyone has the evidence to hit stop.

Argo Rollouts lives precisely in that gap. It is not a replacement for the Kubernetes scheduler, and it is not a GitOps controller — it does not want either job. It is a rollout controller that takes over the update strategy for a single workload and adds the parts a Deployment never had: promotion steps, pause points, analysis runs, and traffic-shaping integrations.

The mental model: Deployment behaviour, but with a promotion engine

The easiest way to hold Argo Rollouts in your head is “a Deployment with a state machine bolted on”. The Rollout resource still owns ReplicaSets and still reacts to pod template changes, but on top of that it keeps track of where a release currently sits in its promotion flow.

At a high level, the loop runs like this:

A new pod template lands in Git and is applied to the cluster.
The Rollout controller creates a new ReplicaSet.
The strategy decides what happens next: shift some traffic, create a preview stack, pause, or run analysis.
If the checks pass, the controller promotes the release.
If the checks fail, the controller aborts and keeps the stable version serving traffic.

That separation is really the whole idea. Kubernetes carries on running the pods, exactly as it did before. Argo Rollouts is the part that decides whether a new version has earned more traffic yet.

The pieces that matter

As of v1.9.0, the CRDs worth genuinely understanding are Rollout, AnalysisTemplate, ClusterAnalysisTemplate, AnalysisRun, and Experiment. That list looks longer than it needs to be, though — in day-to-day work most teams spend nearly all their time in the first three.

`Rollout`

The Rollout (apiVersion: argoproj.io/v1alpha1, kind: Rollout) sits at the centre of everything. It is deliberately built to be a drop-in replacement for a Deployment, which is why migrating an existing workload is mostly a matter of changing the apiVersion, the kind, and the strategy — the pod template underneath can stay much as it was.

A Rollout can either carry its own pod template or point at an existing Deployment through workloadRef. That second option is the gentle on-ramp: it lets you bring a live workload under Rollouts gradually, rather than rewriting the whole object in one nervous commit.

`AnalysisTemplate` and `AnalysisRun`

AnalysisTemplate describes the check; AnalysisRun is that check actually running during a rollout. The template might ask Prometheus for a success rate, a latency figure, or some service-specific business metric that only your team would think to measure. The run takes the answer and decides whether the rollout carries on, pauses, or aborts.

This is the part that makes Argo Rollouts something more than “rolling updates, but slower”. The controller is not simply counting down a timer between steps — it is waiting for evidence before it commits.

Canary vs blue-green

Argo Rollouts supports both, and it helps to remember that they are answers to different questions.

Canary is about gradual exposure. Raise the traffic a step at a time, watch what happens, then decide whether to keep going.
Blue-green is about a clean cutover. Stand the new version up beside the old one, satisfy yourself it works, then flip the active service across.

Neither is universally the right answer. Canary tends to be the better default for customer-facing services, where the whole point is for a bad version to touch as little traffic as possible before you catch it. Blue-green comes into its own when you want an obvious stable-versus-preview split and the reassurance of a fast switch back.

A concrete canary with analysis

The heart of a real Argo Rollouts setup is rarely the sheer volume of YAML — it is the promotion logic hiding inside it. The example below is deliberately small: a canary that shifts traffic in stages and refuses to promote unless a Prometheus-backed success-rate check agrees.

First, install the controller:

kubectl create namespace argo-rollouts
kubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml

Then define the analysis template:

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: checkout-success-rate
spec:
  args:
    - name: service-name
  metrics:
    - name: success-rate
      interval: 5m
      successCondition: result[0] >= 0.99
      failureLimit: 3
      provider:
        prometheus:
          address: http://prometheus.monitoring.svc.cluster.local:9090
          query: |
            sum(
              rate(http_requests_total{service="{{args.service-name}}",status!~"5.."}[5m])
            ) /
            sum(
              rate(http_requests_total{service="{{args.service-name}}"}[5m])
            )

Then wire it into the rollout:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: checkout
spec:
  replicas: 8
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app: checkout
  template:
    metadata:
      labels:
        app: checkout
    spec:
      containers:
        - name: checkout
          image: ghcr.io/example/checkout:v2
          ports:
            - containerPort: 8080
  strategy:
    canary:
      steps:
        - setWeight: 10
        - pause:
            duration: 5m
        - analysis:
            templates:
              - templateName: checkout-success-rate
            args:
              - name: service-name
                value: checkout
        - setWeight: 30
        - pause:
            duration: 10m
        - setWeight: 100

The interesting part here is not the syntax — it is the sequence.

At 10% traffic, the rollout deliberately pauses long enough to gather some real request data. The AnalysisRun then measures the service against a threshold you wrote down in advance, and if that metric fails three times it aborts, rather than quietly waving the release on to 30% and then to everyone.

There is one subtlety worth internalising early. Without a traffic-routing integration, setWeight is an approximation drawn from ReplicaSet sizes rather than exact request-level shaping. At low replica counts that is often good enough, but it is not the same thing as a true 10% split. When you genuinely need that precision, wire Rollouts up to a traffic provider and give it stableService, canaryService, and trafficRouting. That is the line between “best effort, by pod count” and “deliberate, by request path”.

Where blue-green is simpler

Blue-green is the model most teams find easiest to explain out loud, mostly because it has fewer moving parts. There is an active stack, a preview stack, and a single moment of promotion between them.

In Rollouts, that shape becomes an activeService carrying live traffic and, optionally, a previewService pointed at the new ReplicaSet before it goes live. The part that makes it genuinely interesting is what you can hang either side of the cutover with prePromotionAnalysis and postPromotionAnalysis:

prePromotionAnalysis has to be satisfied that the preview version is good enough before the switch happens at all.
postPromotionAnalysis keeps watching once traffic has moved across, and can send it back to the old version if the new one starts to degrade.

The field I would commit to memory here is scaleDownDelaySeconds. It defaults to 30 seconds, giving the service selectors and traffic infrastructure time to converge before the old ReplicaSet is scaled away. It sounds like a trivial setting, but it is exactly the sort of thing that separates a rollout which “worked” as far as the controller was concerned from one that quietly dropped traffic down in the dataplane.

How this fits with GitOps

Argo Rollouts and Argo CD get along well, but it is worth being clear that they are doing two different jobs.

Argo CD reconciles the desired state in Git into the cluster. Argo Rollouts decides how a workload that has just changed should be promoted, once that change has landed. The boundary between them is clean, and that cleanness is the point.

In practice it means you do not have to smuggle rollout logic into your CI pipelines or bend your GitOps repo layout around it. Git still holds the desired pod template, Argo CD still applies it, and Rollouts takes over only for the runtime question of whether the new version deserves to keep advancing.

It also means rollback here is operational rather than declarative — a distinction that trips people up more often than you would expect. When a rollout aborts, Argo Rollouts does not go and rewrite Git on your behalf. It scales the release back to the stable version in the cluster, but the repository still holds the bad image tag until someone, or another controller, reverts it. That gap has teeth: if Argo CD is running with auto-sync and self-heal, it can read the aborted rollout as simply out of sync and try to re-apply the very version that just failed. Teams new to the model tend to walk straight past that boundary, and only notice it the first time the two controllers start pulling against each other.

Production reality: the sharp edges

Argo Rollouts demos beautifully and is surprisingly easy to misuse once real traffic is involved. The failure modes that actually bite are rarely exotic — they are the boring, avoidable ones.

Canary percentages are only exact if your traffic layer is exact

A setWeight: 10 step reads as though it means precisely ten percent, but without traffic routing it is really just a pod-count approximation. If your service runs seven replicas, there is no way for the controller to carve traffic into a clean tenth. When exact exposure genuinely matters, reach for a supported traffic manager rather than trusting the arithmetic.

Traffic-routed canaries can overprovision

The moment you add trafficRouting, the replica arithmetic shifts under you. maxSurge and maxUnavailable are no longer the whole story, particularly once setCanaryScale is in the mix, and it is easy to find yourself running more stable and canary pods at once than you had budgeted for during a promotion. That is usually the right trade for safety — just make sure it is a trade you planned for, rather than one you discover on the cluster.

Blue-green is simple, but not cheap

Running blue and green side by side doubles parts of your footprint by design — that is the whole mechanism, not a bug. If you lean on preview environments heavily, keep them short-lived. Rollouts is built for promotion windows measured in minutes, not for quietly maintaining a second long-lived environment that drifts further from the first with every passing day.

HPA needs to target the `Rollout`

If the workload is autoscaled, point the HPA at the Rollout itself — not the Deployment you migrated from, and certainly not an individual ReplicaSet. Get this wrong and you end up refereeing two controllers that hold different opinions about who owns the replica count, usually at the least convenient moment.

When Argo Rollouts is the right call

Reach for Argo Rollouts when the cost of a bad release is high enough that “the pods became Ready” simply is not a strong enough gate to stand behind. In practice that means customer-facing APIs, services with SLOs you actually measure, and teams who already have decent metrics but are still making the promotion call by hand, one deploy at a time.

What it is not is a decoration. Adopting it to make a deployment diagram look more sophisticated is a poor reason on its own. If a service is low-risk, its traffic layer is simple, and rollback is already fast, a plain Deployment may well remain the honest choice.

The real win was never that Rollouts hands Kubernetes yet another CRD. It is that it gives your release process a proper control loop: expose a little, measure what happens, then decide. That single step — pausing to look before committing — is the one a plain Deployment has never been able to take on its own.

EKS node autoscaling with Karpenter: just-in-time capacity without node groups

Billy Walker — Thu, 09 Jul 2026 19:01:51 +0000

Most people arrive at Karpenter after a slow build-up of small frustrations rather than one dramatic outage. Managed node groups start out simple, and then one day you notice you are spending real time deciding which group should be a little bigger, which instance family to add next, and how much headroom to leave "just in case". It starts to feel less like running a cluster and more like tending a spreadsheet.

Karpenter offers a gentler mental model. Instead of pre-sizing groups and hoping one of them fits, it watches the pods that cannot be scheduled, works out the capacity they actually need, and launches EC2 nodes to match. Two things shift as a result. Scaling up follows the workload rather than a template you wrote weeks ago, and scaling down becomes a policy you set once rather than a clean-up job you keep meaning to get to. On a busy cluster, that is often the difference between paying for a permanent buffer and only asking for capacity when there is real work waiting.

How Karpenter changes EKS node autoscaling

The simplest way to describe the change is this: Cluster Autoscaler works at the level of node groups, while Karpenter works at the level of individual pending pods. On paper that reads like a minor implementation detail. In day-to-day operation it turns out to be most of the story.

With Cluster Autoscaler, you make the important decisions up front — which instance families exist, how much spare room each group keeps, and which group grows when pods start to queue. Karpenter takes those same decisions and turns them into guardrails, then chooses the actual node shape at launch time, once it can see what is really being asked for.

That tends to suit clusters where the workloads do not all look alike:

web services that spike during deployments
queue workers whose demand follows traffic rather than CPU
short-lived CI or batch pods
teams mixing Spot and on-demand capacity in the same cluster

None of this means node groups are a mistake. They are simply static by design, and that is perfectly fine right up until your workload mix changes often enough that static sizing starts to feel expensive or brittle. That is usually the moment Karpenter starts to earn its place.

NodePool and EC2NodeClass split policy from AWS plumbing

On EKS, two Karpenter resources do most of the work: NodePool and EC2NodeClass.

NodePool describes scheduling and disruption policy — what kinds of nodes are allowed, how aggressively Karpenter can consolidate, and which workloads are welcome to land there.
EC2NodeClass describes the AWS side — subnets, security groups, AMI family, the IAM role or instance profile, storage, and the other EC2-specific details.

Keeping these separate is more useful than it first appears, because it holds your capacity intent apart from the cloud plumbing underneath it. A single EC2NodeClass can back several NodePools, and each NodePool can express a different set of rules about the workloads it serves.

A minimal setup looks like this:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general
spec:
  template:
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      requirements:
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ['c', 'm', 'r']
        - key: karpenter.sh/capacity-type
          operator: In
          values: ['spot', 'on-demand']
        - key: kubernetes.io/arch
          operator: In
          values: ['amd64']
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 10m

Two things are worth remembering here, and the docs are clear about both. Karpenter will not do anything at all until at least one NodePool exists — there is no implicit default waiting in the background. And a pod's own scheduling constraints are intersected with the NodePool requirements, so if a workload asks for something the pool does not allow — an instance family, a zone, an architecture — Karpenter simply will not launch a node for it. It sounds obvious written down, but it is one of the more common ways for a pending pod to look like a capacity shortage when it is really a policy mismatch.

Why just-in-time provisioning beats a larger buffer

The traditional habit on EKS is to keep enough spare capacity around that the next deployment or burst will probably fit. The trouble with "probably" is that it quietly shows up on the bill every month, whether or not you ever use it.

Karpenter lets you run a little closer to the line, because it provisions from real pending demand rather than a guess. If a rollout suddenly needs six more replicas, it can pick capacity that fits those pods instead of inflating a general-purpose group and hoping the shape is close. When the burst passes, the same controller can quietly remove or replace whatever it over-provisioned.

This helps most when your requests are uneven. A memory-heavy worker, an ARM-only build job, and a stateless frontend rollout do not really want to share a node. Node groups force you to approximate and accept some waste, whereas Karpenter can stay much closer to what each workload actually asked for.

The catch is that your constraints have to be explicit. Instance family, architecture, capacity type, disruption tolerance — all of it needs to be written down properly. Karpenter takes away the provisioning ceremony, but it does not take away the thinking.

Consolidation, drift, and interruption handling are the real day-two features

Fast provisioning is the headline, but the part you live with day to day is what happens after a node has landed.

Karpenter's disruption controller looks after two cases that quietly save a lot of effort:

Consolidation removes empty nodes, or swaps an underutilised one for a cheaper shape when the same pods can comfortably run elsewhere.
Drift replaces nodes that have fallen out of step with the current NodePool or EC2NodeClass — a new AMI selection, say, or changed launch requirements.

In a node-group-heavy setup, those two jobs — right-sizing and AMI rollovers — usually turn into separate pieces of scheduled maintenance. Karpenter keeps comparing what you asked for with what is actually running, so the fleet does not slowly drift out of shape between maintenance windows.

There is an honest caveat here, and the Karpenter docs do not hide from it: consolidation is only ever as good as the disruption rules you allow. Preferred anti-affinity and topology spread can quietly stop nodes from consolidating even when the scheduler could place the pods elsewhere, and PodDisruptionBudgets can do the same. So when someone says Karpenter is not saving them any money, the controller is rarely the first place to look. More often it is the workload policy wrapped around the nodes it is trying to drain.

Interruption handling matters too, especially if you lean on Spot. Karpenter can react to a Spot interruption warning, bring up replacement capacity ahead of time, and drain the old node in parallel — but all of that still depends on the workload tolerating disruption. Two minutes is not long when graceful shutdown, PDBs, and placement rules are all pulling in different directions.

The cost story here is the unglamorous one: a cluster that already had plenty of capacity in aggregate, just trapped in the wrong shape. Consolidation and drift are what turn that from a tidy architecture diagram into an actual lower bill.

Spot works well when the fallback path is real

Karpenter and Spot are often mentioned in the same breath, and for good reason — it can choose from a far wider range of instance types than a hand-maintained node-group layout, which is exactly what makes Spot workable. The part worth dwelling on is that Spot only pays off when the fallback is genuinely there.

A sensible EKS pattern usually looks something like:

prefer Spot for workloads that can tolerate interruption
allow on-demand in the same NodePool for when the service cannot wait for spare Spot capacity to appear
keep topology spread and PDBs sensible, so replacing one node does not set off a second incident
test your interruption handling before you actually need it

That last point carries more weight than it might seem. "We allow on-demand fallback" is a line of configuration; proving that a constrained workload can still schedule when a Spot pool dries up is a different thing entirely, and a good deal more reassuring when it happens at an awkward hour.

Where Karpenter is better than Cluster Autoscaler, and where it is not

Karpenter tends to win out over Cluster Autoscaler when an EKS cluster carries a genuine mix of workloads and their shapes change often. It is good at turning pending work into right-sized nodes without asking you to pre-bake every possible capacity combination into managed groups ahead of time.

It is a less obvious choice when the fleet is already settled — the node groups are well understood, the workload mix barely moves, and nothing about the current setup is causing pain. If what you have today is simple and cheap, Karpenter can end up being a sophisticated answer to a question nobody was really asking.

So the useful question is not whether Karpenter is newer or cleverer, because it is comfortably both. It is whether managing node groups has quietly become a drag on your time. If it has, the added complexity is usually a fair trade. If it has not, a well-kept Cluster Autoscaler setup may still be all you need.

Takeaway

Karpenter moves EKS node autoscaling away from managing node groups and towards provisioning that follows the workload. You describe the policy once through NodePool and EC2NodeClass, and from then on capacity is launched to match the pods actually in front of the scheduler, rather than the guess someone made last quarter.

The real payoff is not only quicker scale-up. It is the combination of right-sized launches, consolidation, drift replacement, and Spot fallback that holds together under pressure. Between them, consolidation and Spot can take a meaningful bite out of node spend without making scale-up feel fragile, though the exact number will always depend on your particular workload mix.

If you are already spending your week wrestling oversized node groups, slow scale-up, or a fleet that drifts out of shape between maintenance windows, Karpenter is well worth a serious look on EKS.