The best way to handle day-two operations for containers is to automate each layer separately: rebuild and redeploy images for patches, scale replicas on demand signals and nodes on pending capacity, and fail over through health-checked rollouts, zone spreading, and routing to healthy regions. Then test the automation on a schedule. Control Plane handles the platform side of that machinery for you, whether you run in one region of one cloud or active-active across regions and clouds.
Day-two operations are everything that happens after your containers reach production. Initial setup happens once, but patching, scaling, failing over, monitoring, rotating certificates, and testing backups repeat for the entire lifetime of every workload. This guide covers each layer of automation, what breaks when it is misconfigured, and where the boundary falls between work your team owns and work a platform can absorb.
Disclosure: Control Plane publishes this guide.
TL;DR
- Patching happens at four layers: node OS, Kubernetes version, container base image, and application dependencies.
- Autoscaling happens at three layers: replicas (HPA or KEDA), resource right-sizing (VPA), and nodes (Cluster Autoscaler or Karpenter).
- Failover happens at four layers: probes and restarts, disruption budgets and rolling updates, zone spreading, and multi-region routing.
- Observability and alerts feed every automation and catch the ones that fail.
- Backups need scheduled restore tests to prove they work.
- An operating calendar keeps recurring tasks from drifting.
- Automation goes wrong in predictable ways: restart storms, oscillating autoscalers, stuck drains, stale images, and untested failover.
- Control Plane handles Day 2 operations for you, including autoscaling, built-in observability, and security.
What’s the best way to handle day-two operations like patching, autoscaling, and failover for containers?
Four principles keep Day 2 manageable.
Immutable images. Never patch a running container. Rebuild the image, push it, and redeploy. This makes every change reproducible and auditable.
Declarative configuration. Define desired state (replica counts, resource limits, topology constraints, rollout strategy) in version-controlled manifests. Let controllers reconcile actual state to match.
Automation per layer. Each concern (node OS patches, Kubernetes upgrades, image rebuilds, replica scaling, node provisioning, health checks, certificate renewal) has its own tool and cadence. Coupling them creates fragile pipelines.
Regular testing. Automation that never runs can break unnoticed. Schedule tests of failover paths, backup restores, and rollback procedures.
Teams that treat Day 2 as an afterthought accumulate manual toil, unpatched images, untested failover, and alert fatigue. Automating each layer and verifying it on a calendar prevents that buildup.
Which Day 2 tasks should be automated, and how often?
Cadences are typical starting points. Adjust them to your provider’s support windows and your own risk tolerance.
| Task | Layer | How to automate it | How often it runs | What breaks if it is skipped |
|---|---|---|---|---|
| Node OS patching | Infrastructure | Node auto-upgrade on GKE, the node OS auto-upgrade channel on AKS, EKS Auto Mode or managed node group AMI updates on EKS, or Bottlerocket with automated updates | Weekly to monthly, or on CVE release | Known kernel and OS vulnerabilities remain exploitable |
| Kubernetes version upgrades | Infrastructure | GKE release channels, AKS auto-upgrade channels, or scheduled EKS upgrades, staged through dev then staging then production | About three times per year, following the upstream minor release cadence | Version falls out of support window; API deprecations break workloads on forced upgrade |
| Container base image patching | Application image | Renovate or Dependabot pull requests for base image digests, Trivy scanning in CI, rebuild and push on merge | When a digest pull request opens (merge security fixes within days), with a scheduled rebuild at least monthly | Container images carry known CVEs into production |
| Application dependency updates | Application image | Renovate or Dependabot PRs for language-level dependencies, SAST and SCA scanning in CI | On each release or weekly batch | Vulnerable libraries persist; compatibility drift grows |
| Replica autoscaling | Workload | HPA on CPU or memory, or on requests per second through a custom metrics adapter; KEDA for queue depth, event sources, or external metrics | Continuous (HPA evaluates every 15 seconds by default; KEDA polls event sources every 30 seconds by default) | Traffic spikes cause latency or errors; idle replicas waste spend |
| Resource right-sizing | Workload | VPA in recommendation mode, applied during next rollout | Recommendations reviewed weekly or monthly | Over-provisioned pods waste capacity; under-provisioned pods get CPU-throttled, OOMKilled, or evicted under node pressure |
| Node autoscaling | Infrastructure | Cluster Autoscaler or Karpenter, triggered by pending pods | Continuous (Cluster Autoscaler scans every 10 seconds by default; Karpenter reacts to pending pods as they appear) | Pending pods wait indefinitely; replica scaling stalls |
| Liveness, readiness, and startup probes | Workload | Defined in pod spec; tuned per application startup and runtime behavior | Continuous (probes run every 10 seconds by default, set by periodSeconds) | Unhealthy containers serve traffic (missing readiness) or hung containers that have not exited are never restarted (missing liveness) |
| Rolling updates and disruption budgets | Workload | Deployment strategy with maxUnavailable and maxSurge; PodDisruptionBudget with minAvailable or maxUnavailable | On each deployment and during node drains | Deployments cause downtime; node drains kill too many replicas at once |
| Zone spreading | Workload | Topology spread constraints with topologyKey set to zone | Applied at scheduling time | A single zone failure can take down the entire workload |
| Regional failover | Platform | DNS-based or anycast global traffic routing to healthy regions; health checks per region | Continuous health evaluation | A regional outage causes total downtime for users routed to that region |
| Certificate renewal | Security | cert-manager with ACME issuers, automated renewal before expiry | By default when one third of the certificate’s lifetime remains (30 days for a 90-day certificate) | Expired certificates cause TLS errors and outages |
| Backups and restore tests | Data | Velero for cluster state, database-native backups for data, scheduled restore tests | Backups daily or more often; restore tests monthly or quarterly | Data loss on failure; untested restores fail when needed |
| Alerting | Observability | Prometheus and Alertmanager or a managed equivalent; alerts on restart count, error rate, pod pending duration, certificate expiry | Continuous | Failures go unnoticed; mean time to detect climbs |
How do I patch containers without downtime?
To patch a container, rebuild its image. A running container is ephemeral and the image is the durable artifact. Update the dependency or base image, rebuild, scan, push, and redeploy.
The four layers of patching
Node OS. Managed Kubernetes services offer automated node image upgrades. Bottlerocket and similar minimal OSes reduce the patch surface. Nodes are drained and replaced, so PodDisruptionBudgets must be configured to keep enough replicas running during the drain.
Kubernetes version. Managed providers publish support windows for each minor version. Teams should track those windows and upgrade before a version loses support. Falling several versions behind increases the risk of API deprecation breakage, because control plane upgrades move one minor version at a time. Always test upgrades in a non-production environment first.
Container base image. Pin images by digest rather than mutable tags like latest. When a base image publisher pushes a new digest, tools like Renovate or Dependabot open a pull request to bump the digest. Trivy or a similar scanner validates the new image in CI before it merges. A mutable tag can resolve to different image content on different nodes. Moving a tag also leaves the pod spec unchanged, so Kubernetes never rolls out the patched image.
Application dependencies. Language-level packages (npm, pip, Maven, Go modules) carry their own vulnerabilities. Automated dependency update tools create PRs, and CI runs tests before merge. This layer is always the team’s responsibility because only the team knows whether a dependency update is compatible with their application logic.
Deploying patches safely
Rolling updates replace pods incrementally. A readiness probe gates traffic to new pods, so new pods receive traffic only after they report ready. Setting maxUnavailable: 1 in the deployment strategy ensures at least N-1 replicas serve traffic during the rollout. PodDisruptionBudgets protect against node drains that coincide with a rollout. If a new image fails its readiness probe, the rollout stalls and the previous version continues serving, giving the team time to investigate and roll back the bad change.
How should autoscaling be set up for containers?
Autoscaling operates at three layers. Each layer solves a different problem, and misconfiguring the interaction between them is a common source of instability.
Replica scaling on demand signals
The Horizontal Pod Autoscaler (HPA) adjusts replica count based on observed metrics. CPU and memory are the built-in resource metrics. Requests per second, concurrent connections, and response latency often track user-facing load more closely, but HPA needs a custom or external metrics adapter (such as Prometheus Adapter) or KEDA to read them.
KEDA extends HPA with scalers for queue depth, event sources, and custom metrics from systems like Kafka, RabbitMQ, or Prometheus. This is especially useful for worker workloads where CPU utilization does not correlate with backlog size.
Set a minimum replica count that matches your availability requirements. A minimum of one replica means any single failure causes downtime. A production service that must survive a zone failure needs at least two replicas in different zones. With two, losing one halves capacity, so size each replica to carry full load or run three.
Resource right-sizing
The Vertical Pod Autoscaler (VPA) analyzes actual CPU and memory consumption and recommends (or applies) adjusted resource requests. Run VPA with updateMode set to Off, which only produces recommendations, and apply changes during planned rollouts. The Recreate mode evicts pods to apply new requests, and InPlaceOrRecreate resizes pods in place where possible and falls back to eviction. Running HPA and VPA on the same metric (for example, both scaling on CPU) creates a feedback loop where one scales out while the other scales up, causing oscillation.
Node scaling on pending capacity
Cluster Autoscaler and Karpenter watch for pods that cannot be scheduled due to insufficient node capacity. Cluster Autoscaler works with predefined node groups. Karpenter provisions nodes directly from a broader set of instance types and sizes instead of resizing predefined node groups. Both remove underutilized nodes after a configurable cooldown.
Scale-down protection
HPA applies a 300-second scale-down stabilization window by default to prevent rapid scale-down after a brief traffic dip. Keep it, or lengthen it for spiky traffic, rather than lowering it. A workload that scales down aggressively and then immediately needs to scale back up wastes time on container startup and can drop requests during the gap. Kubernetes production architecture: upgrades, autoscaling, and Postgres covers how these pieces fit together in production.
How do containers fail over automatically?
Failover spans four layers, from individual container restarts to global traffic rerouting.
Probes and restarts
A liveness probe tells Kubernetes whether a container is alive. If it fails, the container is restarted. A readiness probe tells Kubernetes whether the container can accept traffic. If it fails, the pod is removed from service endpoints but not restarted. A startup probe gives slow-starting containers time to initialize before liveness and readiness checks begin.
A frequent misconfiguration is a liveness probe that is too strict or that checks downstream dependencies. If the probe’s timeout or failure threshold is too tight, normal load spikes cause the probe to fail, which triggers a restart, which increases load on remaining replicas, which causes more probe failures. This restart storm can take down an entire deployment.
Rolling updates gated on readiness
During a rolling update, the Deployment controller removes old pods only as fast as new pods become ready, within the maxUnavailable limit (25% of replicas by default). Set maxUnavailable to 0 to keep every old pod until its replacement is ready. If the new version has a bug that prevents it from becoming ready, the rollout stalls and the previous version keeps serving. Kubernetes marks the rollout as failed after progressDeadlineSeconds (600 seconds by default) but does not roll back on its own. The gate only works if readiness probes are defined and check that the application can serve requests.
Zone spreading
Topology spread constraints distribute replicas across failure domains. Setting topologyKey: topology.kubernetes.io/zone with maxSkew: 1 and whenUnsatisfiable: DoNotSchedule keeps per-zone replica counts within one of each other when pods are scheduled. Scale-down and node loss can skew the spread later. Without a constraint, the scheduler can place all replicas in a single zone, and a zone failure becomes a total outage.
Regional failover
For workloads that must survive a full region outage, replicas run in multiple regions. A global load balancer or DNS-based routing directs traffic to the nearest healthy region. Health checks at the routing layer detect when a region stops responding and shift traffic automatically. DNS-based failover is bounded by record TTLs and client caching, so keep TTLs short on failover records.
The application tier should be stateless so that any region can serve any request. Databases and other stateful systems require their own replication strategy, whether that is synchronous replication for zero data loss or asynchronous replication with a defined recovery point objective. Data replication is configured per data store and is separate from application failover.
Teams should regularly test failover without taking production down to verify that routing, data replication, and alerting all work as expected.
How does Day 2 automation go wrong?
Automation that is never tested or is misconfigured causes its own category of outages.
| Symptom | Cause | Fix |
|---|---|---|
| Restart storm under load | Liveness probe timeout or failure threshold too strict for normal load variance | Widen liveness probe thresholds; use a startup probe for slow-starting containers; use readiness, which removes a pod from endpoints without restarting it, to shed traffic from an overloaded pod |
| Autoscaler oscillation | HPA and VPA both acting on CPU, or HPA scale-down stabilization window too short | Separate the metrics each autoscaler uses; keep the HPA scale-down stabilization window at its 300-second default or longer |
| Node drain never completes or a managed node upgrade fails | PodDisruptionBudget minAvailable equals total replica count, so no pod can be evicted | Set minAvailable to N-1 or use maxUnavailable of 1; verify PDB math before applying |
| Patched base image never deployed | Mutable tag used without a redeploy trigger, or pinned digest never bumped | Pin by digest and use Renovate or Dependabot to bump it; or use an image updater (such as Flux image automation or Argo CD Image Updater) that redeploys when the tag’s digest changes |
| Rollout serves errors to users | No readiness probe defined, so new pods receive traffic immediately even if unhealthy | Define a readiness probe that validates the application can serve requests |
| Failover never tested, fails when needed | Assumption that multi-zone or multi-region setup works without validation | Schedule failover drills quarterly; simulate zone and region loss in staging |
| Certificate expired in production | cert-manager not installed, or ACME issuer misconfigured, or renewal alert not routed | Deploy cert-manager with monitored ACME issuers; alert when a certificate has not renewed by its scheduled renewal time, and on certificates expiring within 14 days |
| Scale-to-zero cold path breaks a dependency | Callers time out before the scaled-from-zero workload is ready | Set a minimum replica count of at least one for workloads with latency-sensitive callers; or ensure callers retry with appropriate timeouts |
What does a Day 2 operating calendar look like?
Spreading Day 2 work across a regular calendar prevents tasks from accumulating into risky backlogs.
| Cadence | Activities |
|---|---|
| Continuous | Replica autoscaling evaluation, node autoscaling, probe-based health checks, alerting, log and metric collection, certificate renewal checks |
| Daily | Review alerts and dashboards, triage vulnerability scan results, verify backup completion |
| Weekly | Merge dependency and base image digest PRs, review VPA recommendations, check for pending Kubernetes deprecation notices |
| Monthly | Rebuild and redeploy any image that has not picked up a base image update in the past month, review and adjust autoscaling thresholds, rotate secrets, run a restore test on one data store |
| Quarterly | Check clusters against the provider support window and upgrade the Kubernetes minor version in staging then production when due, run a full failover drill, review PodDisruptionBudget and topology spread configurations, audit IAM and RBAC policies |
Where does Control Plane fit in Day 2 operations?
Control Plane is the unbreakable platform for running production workloads. It runs serverless, standard, stateful, and cron workloads natively on AWS, GCP, Azure, and Oracle Cloud Infrastructure (OCI), VM workloads on locations joined through Control Plane’s Managed Kubernetes, and workloads on Kubernetes clusters you already run or your own machines, including on premises, active-active across regions and clouds.
Control Plane handles Day 2 operations for you, including autoscaling, built-in observability, and security.
Clusters and nodes. On the built-in locations Control Plane operates on AWS, GCP, Azure, and Oracle Cloud, your team has no Kubernetes cluster or nodes to operate. Control Plane runs the infrastructure and node capacity underneath your workloads, including patching, upgrading, and scaling it.
Certificates. By default, Control Plane issues the TLS certificates for workload endpoints and custom domains from Let’s Encrypt and rotates them every 60 days, so there is no cert-manager to run. Traffic between workloads uses mutual TLS with a unique client certificate per workload, rotated every hour.
Deployment. A Global Virtual Cloud (GVC) is Control Plane’s named set of locations drawn from cloud regions, Kubernetes clusters you register, and your own hardware. A workload deployed to a GVC runs in every location of that GVC from one definition.
Autoscaling. Control Plane scales replicas between a minimum and maximum per location on requests per second, CPU, or memory, on concurrent requests for serverless workloads, and on latency or KEDA event sources for standard and stateful workloads. Serverless workloads scale to zero natively, and standard and stateful workloads scale to zero through KEDA once it is enabled on the GVC. A new workload runs one replica per location by default; set at least two for production.
Right-sizing. Control Plane’s Capacity AI right-sizes CPU and memory between a minimum and maximum based on historical usage. It cannot run alongside CPU-based or multi-metric autoscaling, which prevents the CPU feedback loop between HPA and VPA described earlier, and it does not apply to GPU containers.
Rollouts and self-healing. During an update, each location keeps serving the previous version while new replicas start, and a new version that fails its readiness probe does not take over. By default at most one replica is unavailable at a time. A container that crashes or fails its liveness probe is restarted, and a stopped replica is replaced. A workload can be rolled back to its previous image.
Failover. Each workload’s canonical endpoint routes clients to the nearest location where the workload passes its readiness probes, and traffic shifts automatically when a location fails. Within a location, Control Plane places a workload’s replicas on separate machines, and the multi-zone option spreads them across the location’s availability zones where the location supports it. The 99.999% availability SLA applies to workloads running at least two replicas in at least two locations.
Patching. Control Plane applies security updates to the platform regularly and applies zero-day fixes as soon as they are available and verified. With Control Plane’s dynamic tags enabled, a workload redeploys automatically when its image tag’s digest changes, so an image rebuilt under the same tag rolls out without a manual deploy.
Observability. Every replica streams logs and metrics automatically, and traces are collected once tracing is enabled on the GVC or org at a sampling rate you set. Each keeps 30 days by default, adjustable independently on the org. The built-in Grafana instance ships alert rules for repeated container restarts and stuck deployments, and notifications go to any contact point you add.
Managed Kubernetes. Control Plane’s Managed Kubernetes provisions and operates Kubernetes clusters whose nodes run in your cloud accounts or on your own servers. On cloud providers such as AWS, GCP, and Hetzner, it scales each node pool between the minimum and maximum you set, can use spot instances on AWS and GCP, and upgrades a cluster to the Kubernetes version you select, one minor version per upgrade.
Backups. Control Plane’s Template Catalog includes PostgreSQL, highly available and multi-location PostgreSQL, MySQL, CockroachDB, and Cassandra, with optional scheduled backups to object storage. The multi-location PostgreSQL template keeps streaming replicas in other locations and, across three or more locations, automatically promotes one when the primary’s location is lost.
Compliance. Control Plane is PCI DSS Level 1, SOC 2 Type II, HIPAA, and GDPR compliant.
What remains yours. Your team keeps the tasks only it can do. It updates application dependencies, rebuilds images when a base image or library is patched, chooses the replication setup for each of its own databases, and runs restore tests. Control Plane can build those images remotely from a GitHub or GitLab repository with no Docker daemon.
For teams exploring how to get Kubernetes benefits without a platform team, or managing Day 2 operations for AI infrastructure, Control Plane takes on the platform layers of Day 2 described in this guide.
FAQ
Tool names and default values in these answers refer to upstream Kubernetes and its common add-ons.
What are day-two operations?
Day-two operations are the ongoing management activities performed after initial deployment, including patching, autoscaling, failover, monitoring, certificate renewal, backup testing, and incident response. Each one recurs for as long as the workload runs.
What’s the best way to handle day-two operations like patching, autoscaling, and failover for containers?
Treat images as immutable and rebuild them for every patch, declare desired state in version-controlled manifests, give each concern (patching, replica scaling, node scaling, failover) its own tool and cadence, and test failover, restores, and rollbacks on a calendar.
How do I patch containers without downtime?
Rebuild the container image with updated dependencies or base image, push it, and trigger a rolling update. The rolling update replaces pods incrementally while readiness probes gate traffic to new pods. PodDisruptionBudgets prevent node drains from removing too many replicas at once. Pin images by digest and use automated tools like Renovate or Dependabot to bump the digest when upstream images change.
Should I use HPA, VPA, or KEDA?
Use HPA for replica scaling on CPU or memory, or on request rate through a custom metrics adapter. Use KEDA to scale on queue depth, event sources, or external metrics, or to scale to zero. Use VPA in recommendation mode to right-size resource requests, and apply its recommendations during planned rollouts. Avoid running HPA and VPA on the same metric, as they will conflict and cause oscillation.
What is the difference between liveness and readiness probes?
A liveness probe determines whether a container is alive. Failure triggers a restart. A readiness probe determines whether a container can accept traffic. Failure removes the pod from service endpoints but does not restart it. Use a startup probe to hold off both checks while a slow container initializes. Keep liveness probes lenient and limited to the container’s own health, so a load spike or a failing dependency does not trigger restarts. Readiness probes can be tighter, since a failure only removes the pod from endpoints.
How do I fail over containers to another region?
Run replicas in multiple regions behind a global load balancer or DNS-based routing that health-checks each region. When a region stops passing health checks, traffic shifts to healthy regions automatically. Keep the application tier stateless, and replicate each database with its own strategy. Test failover regularly by simulating region loss in staging.
How often should I test failover?
Test failover at least quarterly. Simulate zone failures and region failures in staging or with controlled production drills. Verify that routing shifts correctly, that data is consistent after failover, and that alerts fire as expected. A failover path that has never run can fail during the incident it was built for.
How does Control Plane handle Day 2 operations?
Control Plane runs the platform side of Day 2 so your team does not have to. On its built-in locations there are no clusters or nodes for your team to manage. It scales replicas in each location on demand signals, rolls out new versions behind the readiness probes you define, routes clients to the nearest healthy location and away from a failed one, issues and renews TLS certificates, right-sizes CPU and memory with Capacity AI, and ships Grafana alert rules for container restarts and stuck deployments. Your team still owns application code and dependencies, image rebuilds, and the replication and restore testing of its own databases.
What is the biggest risk of skipping Day 2 automation?
Accumulated drift. Unpatched images carry known vulnerabilities. Untested failover fails when needed. Missing readiness probes let broken deployments serve errors. Missing autoscaling causes either wasted spend or dropped requests. Each skipped task is low-risk on any given day, but the compound risk grows until an incident forces a painful catch-up under pressure.
To hand Day 2 operations to a platform, talk to the Control Plane team or sign up free.
Originally published on the Control Plane blog.
Top comments (0)