DEV Community

Aisalkyn Aidarova
Aisalkyn Aidarova

Posted on

production-grade guide to Configuration & Secrets in Kubernetes

#architecture #devops #kubernetes #security

Mental model first (very important)

Image

Image

Image

Kubernetes does NOT magically manage configuration.
It only injects data into containers.
What the app does with it is your responsibility.

1️⃣ What DevOps must understand FIRST

Configuration problems cause:

  • Apps starting but behaving wrong
  • Random crashes after deploys
  • Downtime during config changes
  • Security leaks (very common)

Core rule

Kubernetes delivers config. Applications consume it. Kubernetes does NOT reload it automatically.

2️⃣ ConfigMap vs Secret (zero confusion rule)

Feature ConfigMap Secret
Sensitive ❌ No ✅ Yes
Base64 ❌ No ✅ Yes
Git safe ✅ Yes ❌ No
Encryption at rest ❌ No ⚠️ Optional
Examples URLs, flags passwords, tokens

DevOps rule

  • ConfigMap = behavior
  • Secret = credentials

Never mix them.

3️⃣ How configuration is injected (3 ways)

Method 1 — Environment variables (MOST COMMON)

ConfigMap 

kubectl create configmap app-config \
  --from-literal=APP_MODE=prod \
  --from-literal=LOG_LEVEL=info
Enter fullscreen mode Exit fullscreen mode

Secret 

kubectl create secret generic app-secrets \
  --from-literal=DB_USER=admin \
  --from-literal=DB_PASS=supersecret
Enter fullscreen mode Exit fullscreen mode

Inject into Pod 

env:
- name: APP_MODE
  valueFrom:
    configMapKeyRef:
      name: app-config
      key: APP_MODE
- name: DB_PASS
  valueFrom:
    secretKeyRef:
      name: app-secrets
      key: DB_PASS
Enter fullscreen mode Exit fullscreen mode

DevOps attention points

  • App restart REQUIRED to apply changes
  • Env vars are visible via kubectl describe pod
  • Never log env vars

Method 2 — EnvFrom (bulk injection) 

envFrom:
- configMapRef:
    name: app-config
- secretRef:
    name: app-secrets
Enter fullscreen mode Exit fullscreen mode

When DevOps uses this

  • Microservices
  • Standardized configs
  • Fast onboarding

Danger

  • Variable name collisions
  • Harder debugging

Method 3 — Mounted files (MOST PROFESSIONAL)

Why DevOps prefers this

  • Supports live reload (if app supports it)
  • Cleaner separation
  • Works for certificates, configs, JSON, YAML

ConfigMap as file 

volumes:
- name: config
  configMap:
    name: app-config

volumeMounts:
- name: config
  mountPath: /etc/config
Enter fullscreen mode Exit fullscreen mode

Result inside container 

/etc/config/APP_MODE
/etc/config/LOG_LEVEL
Enter fullscreen mode Exit fullscreen mode

Secret as file 

volumes:
- name: secrets
  secret:
    secretName: app-secrets
Enter fullscreen mode Exit fullscreen mode

DevOps reality

  • Kubernetes updates files automatically
  • App must watch files or reload
  • Most apps DO NOT reload by default

4️⃣ The biggest DevOps mistake (very common)

❌ “I updated ConfigMap but app didn’t change”

Why?

  • Pods don’t restart
  • Env vars are immutable
  • App doesn’t reload mounted files

Correct solutions (DevOps options)

Option 1 — Rolling restart 

kubectl rollout restart deployment app
Enter fullscreen mode Exit fullscreen mode

Option 2 — Hash-based rollout (BEST PRACTICE) 

metadata:
  annotations:
    config-hash: "{{ .Values.configHash }}"
Enter fullscreen mode Exit fullscreen mode

(Change annotation → rollout triggered)

5️⃣ Secrets: what DevOps MUST secure

Image

Image

Critical truths

  • Secrets are base64, NOT encrypted
  • Anyone with RBAC access can read them
  • GitHub leaks happen weekly

Minimum DevOps requirements

  • Enable encryption at rest (cloud KMS)
  • Restrict RBAC
  • Use namespace isolation
  • Rotate secrets

6️⃣ What NOT to do (interview traps)

❌ Put secrets in YAML
❌ Commit secrets to Git
❌ Share secrets across namespaces
❌ Use same secret in dev & prod
❌ Restart cluster for config changes

7️⃣ Production patterns DevOps uses

Pattern 1 — External Secret Managers

  • AWS Secrets Manager
  • HashiCorp Vault
  • Azure Key Vault

Flow:

Vault → ExternalSecrets → Kubernetes Secret → Pod
Enter fullscreen mode Exit fullscreen mode

Why DevOps prefers this

  • Rotation
  • Audit logs
  • Central control

Pattern 2 — Immutable config

  • New config = new deployment
  • No live mutation
  • Git controls everything

This prevents:

  • Drift
  • Mystery changes
  • “Works on my cluster”

8️⃣ Troubleshooting checklist (real-life)

App not behaving as expected 

kubectl describe pod
kubectl exec pod -- env
kubectl exec pod -- cat /etc/config/*
Enter fullscreen mode Exit fullscreen mode

Secret missing 

kubectl get secret
kubectl describe secret
Enter fullscreen mode Exit fullscreen mode

Config updated but app unchanged

  • Check restart
  • Check reload capability
  • Check correct mount path

9️⃣ How to explain this in interviews (perfect answer)

“ConfigMaps control application behavior, Secrets control credentials. Kubernetes injects them but doesn’t manage reload. DevOps must handle rollout, security, and lifecycle correctly.”

10️⃣ What senior DevOps MUST know (non-negotiable)

You must know:

  • All injection methods
  • Reload limitations
  • Security risks
  • Rollout strategies
  • External secret managers
  • RBAC implications

Final DevOps truth (important)

Most production incidents are configuration issues, not code issues.
Kubernetes only exposes them faster.

Recommended next topics (logical order)

  1. Resource requests & limits (OOMKilled lab)
  2. Rolling & Canary deployments using readiness
  3. Ingress + TLS secrets
  4. External Secrets + Vault
  5. GitOps config management

Tell me which one you want next, and I’ll give you a full production lab with failure demos.

Top comments (0)