Oleksandr Kuryzhev

Posted on Oct 15 Originally published at kuryzhev.cloud

Helm Chart Values for Staging vs Production Done Right

#kubernetes #devops

Originally published on kuryzhev.cloud

A deploy goes green in staging, then production comes up with one replica, a debug log level, and an ingress host that points at the wrong environment. Nobody changed the chart. The Helm chart values were assembled differently, and nobody could say exactly how. This is a common way environment drift shows up. It is usually a misunderstanding of how Helm merges values rather than a Helm bug.

What Helm chart values actually do

Values are the input to the chart's Go templates. Helm builds a single values object from several sources and renders the templates against it. The documented precedence, lowest to highest, is:

the chart's own values.yaml
values from a parent chart
files passed with -f / --values (later files win)
--set, --set-string, and --set-file flags

Maps are merged key by key, so an environment file only needs to contain what differs. Lists are not merged. A list in a later source replaces the earlier list entirely. That one rule explains many "why did my tolerations vanish" tickets.

Setting a key to null in an override file removes the default key. This is useful for dropping a default resource limit or annotation.

Helm stores the user-supplied values with each release, alongside the chart.

helm get values shows the values you supplied.
helm get values --all shows the full merged result.

These are the first commands to run when two environments disagree. Details are in the official Helm values documentation. Verify the exact behavior against the Helm version you run, since Helm 3 and Helm 4 differ in some details.

How people use Helm chart values wrong

The first pattern is copying the entire values.yaml into values-staging.yaml and values-production.yaml. Within a few releases, someone adds a key to one file and not the others. The files stop being overrides and become three divergent configurations.

The second is environment logic inside templates, such as {{ if eq .Values.env "production" }}. It looks convenient, but the chart now encodes knowledge about your environments. Staging can no longer exercise the same code path as production, and every new environment needs a template change.

The third is pipeline-side --set flags. A CI job with a dozen --set arguments is configuration that lives outside version control review.

Watch out for type coercion here:

--set enabled=true becomes a boolean.
--set image.tag=20240101 becomes an integer. Use --set-string when the value must stay a string.
Values files have the same trap through YAML parsing. An unquoted version: 1.10 is read as the float 1.1, so quote version-like strings.

Other common failure modes:

Overriding a list (hosts, env entries, tolerations) and losing the defaults.
Putting secrets in plain values files committed to Git.
Using image.tag: latest in either environment. This makes a rollback unreliable, because the same tag can point at different images.
Skipping helm template or helm lint and discovering a missing key only at install time.

The correct approach to Helm chart values

Treat values.yaml as safe, production-shaped defaults, and keep each environment file as a small diff on top. Staging should differ from production in scale and endpoints, not in structure. The templates stay identical across environments.

Here is a minimal layout. The base file holds defaults, and the environment files carry only differences:

# values.yaml (chart defaults, safe and production-shaped)
replicaCount: 3
image:
  repository: registry.example.com/shop/api
  tag: ""            # set at deploy time, never "latest"
resources:
  requests: { cpu: 250m, memory: 256Mi }
  limits: { memory: 512Mi }
ingress:
  enabled: true
  host: api.example.com
logLevel: info

# values-staging.yaml (only what differs)
replicaCount: 1
resources:
  requests: { cpu: 50m, memory: 128Mi }
ingress:
  host: api.staging.example.com
logLevel: debug

Deploy by stacking files in a fixed order. The pipeline selects the environment file and passes the image reference, and nothing else:

# render first; fails fast on template and schema errors
helm template shop ./chart -f values-staging.yaml --set-string image.tag="$IMAGE_TAG" > /dev/null

# chart values.yaml is applied automatically; the -f file overrides it,
# and any additional -f files would override in the order given
helm upgrade --install shop ./chart \
  --namespace shop-staging --create-namespace \
  -f values-staging.yaml \
  --set-string image.tag="$IMAGE_TAG" \
  --atomic --wait --timeout 5m   # roll back automatically on failure

In Helm 3, --atomic already implies --wait. In Helm 4 it is renamed to --rollback-on-failure, so adjust the flag to match your client version.

Add a values.schema.json to the chart. Helm validates merged values against it on install, upgrade, lint, and template.

A string where an integer belongs fails before anything reaches the cluster.
A typo like replicaCout is only rejected if the schema sets "additionalProperties": false for that object.

Advanced patterns

Once the basic layering works, a few patterns reduce risk further. Which one fits depends on how many environments and teams you have.

Layered files per dimension. Instead of one file per environment, split by concern: values-common.yaml, values-prod.yaml, values-eu-west.yaml. Pass them in order of increasing specificity. This scales to multiple clusters and regions without duplicating shared settings.

Convert lists to maps where you control the chart. Since lists are replaced wholesale, a map such as env: { LOG_FORMAT: json } can be overridden one key at a time. Templates can iterate the map with range $k, $v := .Values.env. Map iteration in Go templates is sorted by key, so output stays deterministic.

Secrets outside values. Reference Kubernetes Secrets by name in values and populate them with an external secrets operator. Alternatively, encrypt value files with SOPS through a plugin such as helm-secrets. Check the plugin's current maintenance status before adopting it.

GitOps tools. Argo CD supports multiple value files per application source. Flux's HelmRelease supports valuesFrom and inline values. In both, the environment is declared in Git and reviewed like code. Verify the field names against your installed version.

Review the diff, not the values. The helm diff upgrade plugin shows the rendered manifest change for a release. A one-line values change that rewrites a StatefulSet's volume template is far easier to catch there than in a values pull request.

Another gotcha: subchart values live under the subchart's name. A typo in that key is silently ignored by Helm unless a schema rejects unknown keys. Use global: only for values that genuinely apply to every subchart.

Performance notes

Helm rendering itself is rarely the bottleneck. Template rendering is local and usually finishes quickly even for large charts. Very heavy use of tpl, nested include, or huge values trees can slow it. Measure with time helm template before optimizing.

The costs that matter are at runtime and in release storage:

Release size. Helm stores each release revision in a Secret by default, and Kubernetes caps a single Secret at 1 MiB. Helm compresses the release data, but large embedded files in values or templates can still hit that limit. Use --history-max (default 10 on helm upgrade) to cap retained revisions.
Rollout time. --wait blocks until resources are ready. Your --timeout should reflect readiness probe settings and image pull times, not a guess.
Resource requests. Cutting staging requests saves cost, but extreme differences hide problems. A service that is never CPU-throttled in staging may be throttled in production. See the Kubernetes resource management docs for how requests and limits affect scheduling and throttling.
Replica parity. One staging replica is unlikely to reveal rolling-update or PodDisruptionBudget problems. If those matter, run at least two replicas in staging.

The practical rule is to keep environment differences few, explicit, and reviewable. For more infrastructure patterns, browse the rest of kuryzhev.cloud.

DEV Community