"I want the dev namespace off overnight without deleting anything" is one of the most reasonable requests in Kubernetes cost work. The workloads are idle sixteen hours a day, the PVCs must survive, and nobody wants to re-deploy every morning. Scaling to zero seems like two commands. It is two commands. The trouble is entirely in the third step, the one at 8am, where things refuse to come back the way they were.
Here's the full mechanics: how to take a namespace to zero, every category of thing that fights you, and the one economic truth that decides whether any of it saves money at all.
Taking a namespace to zero
The naive version:
kubectl -n dev scale deployment,statefulset --all --replicas=0
kubectl -n dev get cronjob -o name | xargs -I{} kubectl -n dev patch {} -p '{"spec":{"suspend":true}}'
Deployments and StatefulSets go to zero (pods terminate, PVCs remain), CronJobs stop scheduling new runs. Storage survives, nothing is deleted, the namespace is intact but silent. On paper, done.
What refuses to come back
The replica counts are gone. --replicas=0 overwrote the only place that number lived. At 8am, scale up to... what? Everything at 1 breaks the services that needed 4; guessing breaks differently per service. Before scaling down you must snapshot the current replica count somewhere durable, an annotation on each object works (downscaler/original-replicas: "4"), and restore from it on wake. No snapshot, no faithful morning.
HPAs fight both directions. A HorizontalPodAutoscaler with minReplicas: 2 will resurrect the deployment you just zeroed (HPA minimums win). And if you delete or suspend the HPA to stop it, that's more state to snapshot and restore. The stop procedure has to handle the autoscaler layer explicitly, not just the workloads.
GitOps undoes you on a loop. ArgoCD and Flux exist to revert drift, and a namespace at zero replicas is drift. With self-heal on, your 9pm scale-down is reverted by 9:03. The choices: annotate resources as ignored for the window, pause reconciliation on that application overnight, or make the scheduler write through Git (heavyweight but honest). Skipping this step is the most common reason "we scaled to zero but the bill didn't move".
KEDA and event-driven scalers. A KEDA ScaledObject will scale the workload right back up when its trigger fires (or hold it at minReplicaCount). Suspending KEDA cleanly means removing or pausing the ScaledObject and restoring it intact later, manifest and all.
DaemonSets don't have replicas. You can't scale a DaemonSet to zero; the honest options are a node selector trick (point it at a label no node carries overnight) or accepting they run wherever nodes still exist, which connects to the economics below.
CronJobs and the missed-window question. Suspended CronJobs skip their windows silently. A 2am backup job attached to the dev namespace stops happening the day you start suspending. Audit what's scheduled in the namespace before the first night, and after wake remember startingDeadlineSeconds: a job whose window passed during suspension will or won't fire on resume depending on it, and both behaviors surprise someone.
StatefulSets come back slowly and in order. Ordered startup (pod-0 before pod-1), volume attach time, and application-level recovery (replaying logs, rebuilding caches) mean the morning is a warm-up curve, not a light switch. Schedule the wake 15-30 minutes before humans arrive, and health-check the tier before declaring morning.
The economic fine print: zero pods is not zero dollars
Scaling a namespace to zero saves nothing by itself. You pay for nodes, not pods. The saving appears only when the emptied capacity lets the cluster autoscaler or Karpenter drain and remove nodes, and reappears as node re-provisioning time at 8am (which is most of your wake-up latency). Two implications:
- If the dev namespace shares nodes with things that stay up, the packing may leave every node alive and the saving near zero. Namespace-per-nodepool or bin-packing-aware placement decides the actual dollars.
- Measure the saving at the node/bill level, never the pod level. "We scaled 40 deployments to zero" is an activity metric; "the cluster runs 9 nodes overnight instead of 21" is money.
And keep system namespaces (kube-system, istio-system, ingress controllers, cert-manager, the monitoring stack) explicitly out of scope. Zeroing the wrong namespace converts a cost project into an incident.
This whole failure catalog is also, unsurprisingly, what productized namespace scheduling has to solve: ZopNight's namespace-level start/stop for EKS, GKE, and AKS scales Deployments and StatefulSets to zero while saving replicas, HPA, and PDB state, suspends CronJobs, evicts DaemonSets via a node selector, preserves KEDA ScaledObject manifests, and replays the whole snapshot on start, with system namespaces rejected outright. That's one implementation's answer; if you're building your own, the list above is the test plan either way.
FAQ
How do I scale a whole Kubernetes namespace to zero?
Scale all Deployments and StatefulSets to zero replicas and suspend all CronJobs in the namespace, after snapshotting current replica counts (annotations work) and neutralizing anything that will scale things back up: HPAs, KEDA ScaledObjects, and GitOps self-heal. PVCs and objects survive; only pods stop.
Does scaling to zero actually save money on EKS/GKE/AKS?
Only if node count follows. Pods are free; nodes are the bill. The saving requires the cluster autoscaler or Karpenter to remove the emptied nodes, which depends on what else shares them. Measure success as overnight node count, not scaled-down workload count.
Why do my deployments scale back up after I set replicas to zero?
Something whose job is maintaining desired state is doing its job: an HPA with a minimum above zero, a KEDA ScaledObject, or ArgoCD/Flux self-heal reverting drift. Scale-to-zero procedures must suspend or bypass that layer for the window and restore it after.
How do I restore the right replica counts in the morning?
You can't, unless you saved them: --replicas=0 destroys the previous value. Write the count into an annotation before scaling down and restore from it on wake. Tooling that does namespace scheduling properly snapshots replicas, HPA, and PDB state and replays it.
Is it safe to suspend CronJobs overnight?
Mechanically yes (suspend just stops new runs), operationally audit first: backups, retention jobs, and report generators often live in the same namespace as the app they serve. Decide per CronJob, and check startingDeadlineSeconds behavior for jobs whose window passes while suspended.
Top comments (0)