Originally published on kuryzhev.cloud
Context
We had a single Helm chart deploying roughly 15 microservices, split across staging and production, with ArgoCD 2.9 syncing everything from a monorepo. The pattern looked textbook: one values.yaml as the base, then a values-staging.yaml and a values-prod.yaml to override whatever was different per environment. On paper, this is the recommended way to manage helm values staging production configs, and honestly, we thought it was a solved problem. Helm 3.14, chart apiVersion v2, nothing exotic.
Our assumption going in was almost embarrassingly naive: "just override values.yaml per environment, how hard can it be." We'd done this a dozen times on smaller projects. What we didn't account for was scale — 15 services, two environments, and a growing team all touching the same values files without a shared mental model of what "override" actually meant in practice.
Three incidents forced us to rethink the whole approach: a silent config drift that caused a production timeout bug, a resource-sizing mistake that both wasted money and OOMKilled a payment service under load, and a security lapse where secrets ended up committed to a values file in git. None of these were exotic failures. They were the kind of thing that happens when "it's just YAML" turns into "nobody actually knows what's in prod anymore."
Mistake 1: Treating values-prod.yaml as a full override instead of a diff
Here's what we got wrong: values-staging.yaml and values-prod.yaml had both grown to over 300 lines each, and roughly 80% of those keys were identical duplicates of the base file. Nobody set out to design it this way — it just accumulated, one "quick fix" at a time, until both files were effectively full copies of the entire config tree with a handful of real differences buried inside.
The incident that exposed this: we added a new environment variable, FEATURE_FLAG_TIMEOUT, to the base values.yaml. Staging picked it up automatically because its override file didn't redefine that key. Production did not — because at some point, someone had copy-pasted the entire env block into values-prod.yaml, freezing a stale value in place. The result: production silently kept an old default for three weeks, until it caused a timeout-related outage during a traffic spike. Nobody had touched that file in weeks. Nobody thought to check it.
This is the classic gotcha with Helm's merge behavior: -f base.yaml -f override.yaml does a deep merge, but if your override file redefines a parent object, everything nested under it stops inheriting from the base. Watch out for this — a single unnecessary top-level key in an override file can silently freeze an entire subtree. We only caught it because someone happened to run helm template --debug while chasing an unrelated bug, and it looked wrong.
Mistake 2: Copy-pasting resource requests/limits without environment-aware sizing
We set resources.requests.cpu: 500m and resources.requests.memory: 512Mi once, early on, copied straight out of a tutorial, and used the identical values for both staging and production across all 15 services. Nobody revisited it. That's the mistake in one sentence: guessed numbers, never validated against real metrics, treated as permanent.
The cost side was quiet but real — about $400/month wasted on overprovisioned staging resources, because staging traffic never came close to needing 500m CPU or 512Mi memory per pod. The outage side was louder. During a real traffic spike, our payment-service pod got OOMKilled. kubectl describe pod showed exit code 137, plain and simple: the JVM-based service needed more headroom than our copy-pasted limits allowed, and we had no HPA configured because the base resource values were never tuned to actual usage in the first place — so autoscaling had nothing sane to scale from.
Lesson learned: resource values should come from actual metrics — Prometheus dashboards, VPA recommendations — not from a tutorial or a guess made under deadline pressure. And staging should be intentionally smaller than production, not identical to it. If staging and prod use the same limits, you lose the cheapest signal you have for catching cost and sizing problems before they hit production.
Mistake 3: Storing secrets and environment-specific credentials directly in values files
This one still makes me wince. A DB connection string and an API key ended up under a secrets: block in values-prod.yaml, added "temporarily" while someone was debugging a connection issue under time pressure. It was never migrated out. Reviews missed it. It sat in git, committed, for longer than anyone wants to admit.
There was no actual breach — but the exposure risk was real, and we treated it that way. We rotated the exposed API key immediately and rewrote git history with BFG Repo-Cleaner to strip the secret out of every commit that touched that file, which is its own special kind of unpleasant: force-pushing history rewrites across a shared monorepo, coordinating with everyone who had a local clone, and hoping nobody had already forked a copy somewhere. Nothing catastrophic happened. It was still a costly, embarrassing incident that ate half a day of the team's attention for a mistake that should never have been possible in the first place.
The lesson is blunt: values files should never contain secret material, full stop. Not "temporarily," not "just for this one fix." This needed to be a hard rule enforced from day one — ideally with a pre-commit hook or CI secret scanner — instead of something we retrofitted after the damage was already in git history.
What we do differently now
We rebuilt the whole values structure around layering instead of duplication. The base values.yaml holds every default that applies everywhere. The environment files contain only true deltas — replica count, resource sizing, ingress host — nothing else. We added a CI lint step that fails the pipeline if an override file duplicates unchanged base keys or grows past a line-count threshold, which forces anyone adding a "quick override" to justify it.
Here's the current shape of it:
# values.yaml — shared base, only defaults that apply everywhere
replicaCount: 2
image:
repository: registry.internal/payment-service
tag: "1.4.2" # pinned, never "latest"
pullPolicy: IfNotPresent
resources:
requests:
cpu: 250m
memory: 256Mi
limits:
cpu: 500m
memory: 512Mi
env:
LOG_LEVEL: "info"
FEATURE_FLAG_TIMEOUT: "30s"
secrets: {} # intentionally empty — never populate this in plain values files
---
# values-staging.yaml — ONLY the deltas from base
replicaCount: 1
resources:
requests:
cpu: 100m # deliberately lower than prod to surface real cost/perf issues early
memory: 128Mi
limits:
cpu: 250m
memory: 256Mi
env:
LOG_LEVEL: "debug" # different verbosity, but FEATURE_FLAG_TIMEOUT inherited from base
---
# values-prod.yaml — ONLY the deltas from base
replicaCount: 4
resources:
requests:
cpu: 500m # sized from VPA recommendation, not guessed
memory: 768Mi
limits:
cpu: 1000m
memory: 1536Mi
# secrets pulled at runtime via External Secrets Operator, NOT defined here
Resource values are no longer guessed — they come from VPA recommendations reviewed quarterly, staging is deliberately capped lower than prod on purpose, and we alert directly on OOMKilled events so we hear about sizing problems before they become outages. Secrets are gone from values files entirely, replaced with SOPS-encrypted values or the External Secrets Operator pulling from AWS Secrets Manager at runtime.
The biggest process change is that we run helm template in CI to render and diff staging vs prod before every merge to main. This is exactly the check that would have caught the FEATURE_FLAG_TIMEOUT drift on day one instead of three weeks in:
# CI diff check — fails the pipeline if an override file duplicates unchanged base keys
# and shows the rendered diff between staging and prod before merge
helm template payment-service ./chart \
-f ./chart/values.yaml \
-f ./chart/values-staging.yaml > /tmp/staging.rendered.yaml
helm template payment-service ./chart \
-f ./chart/values.yaml \
-f ./chart/values-prod.yaml > /tmp/prod.rendered.yaml
diff /tmp/staging.rendered.yaml /tmp/prod.rendered.yaml || true
# Example output that caught the original FEATURE_FLAG_TIMEOUT drift bug:
# < FEATURE_FLAG_TIMEOUT: "30s" (staging, inherited correctly)
# > FEATURE_FLAG_TIMEOUT: "10s" (prod, stale override — flagged before merge)
# mandatory pre-deploy check using helm-diff plugin
helm diff upgrade payment-service ./chart \
-f ./chart/values.yaml \
-f ./chart/values-prod.yaml \
--allow-unreleased
The helm diff upgrade plugin step became non-negotiable after the OOMKill incident — nobody deploys to prod without seeing exactly what's about to change, rendered against what's currently live. It's a five-second check that would have prevented at least two of our three incidents outright.
If you're still running helm values staging production as two nearly-identical full files, I'd stop and audit them today. Diff the rendered output between environments and see how much is actually different — I'd bet it's a lot less than what's sitting in your override files. We learned this the expensive way; you don't have to. For more on how we structure GitOps pipelines around ArgoCD, see our GitOps deployment breakdown, and check the official Helm values files documentation and ArgoCD Helm integration docs if you're setting this up from scratch.
Top comments (0)