DEV Community

Metronom
Metronom

Posted on

You can catch production failures on your laptop: probes, OOMKilled, and zero-downtime in k3d

#kubernetes #devops #tutorial #k3d

"Works on my laptop" and "survives production" are different things — but almost everything separating them is testable locally in k3d. Probe behavior, liveness restarts, rolling updates, SIGTERM shutdown: k3d reproduces all of it.

Takeaways

  • CPU throttles, memory kills. CPU is compressible — hit the limit and the container just slows down. Memory is not — exceed limits.memory and the kernel kills the process: OOMKilled, exit code 137. So a low CPU limit = slowness; a low memory limit = instant Pod death. Give memory headroom (request ~P99 + 20%); CPU can ride closer to P95.
  • Three probes, three jobs:
    • liveness fails -> container is restarted. Keep it simple (GET /healthz), never hit the DB — a dependency blip would restart a healthy container.
    • readiness fails -> Pod is pulled from the Service endpoints (no traffic), not restarted. This is the right place to check PostgreSQL.
    • startup gates the other two — until it passes, liveness/readiness are disabled. Use it for slow starters instead of a big initialDelaySeconds.
  • QoS: Guaranteed (request=limit everywhere) evicted last; BestEffort (no requests/limits) evicted first. Always set at least requests.
  • Graceful shutdown: on delete, the Pod goes Terminating, preStop runs, kubelet sends SIGTERM to PID 1, waits terminationGracePeriodSeconds (default 30s), then SIGKILL. Your app must catch SIGTERM — and uvicorn must be PID 1 (exec-form CMD, no shell wrapper) or the signal never arrives.
  • The endpoints race: removing the Pod from endpoints and sending SIGTERM happen in parallel, so traffic can still hit a shutting-down Pod (5xx during deploy). Fix: a preStop: sleep 15 so routing updates first. Keep preStop well under the grace period — it counts against the same budget.
  • Zero-downtime rollout: maxUnavailable: 0, maxSurge: 1, plus minReadySeconds. Without a readiness probe, zero-downtime is impossible — the rollout waits on readiness before retiring old Pods.

Verify locally: kubectl get endpointslices -n myapp -w while running a while true; do curl ... loop during a rollout.

Full article: https://dorokhovich.com/blog/local-k8s-making-it-production-like?utm_source=devto&utm_medium=syndication&utm_campaign=local-k8s-making-it-production-like

Top comments (0)