A production-ready Kubernetes cluster starts with a managed control plane, worker nodes across three availability zones, an ingress controller or gateway behind a cloud load balancer, cluster and pod autoscaling, and stateful data either in a managed database or in an operator-managed database with replicas across zones. The hard part is Day 2: upgrades, scaling, failover, and security patches, which is why many teams run workloads on a platform that operates Kubernetes for them; Control Plane runs containers across AWS, GCP, Azure, and your own servers with autoscaling, failover, and Day 2 operations handled, with no Kubernetes cluster you have to operate.
Disclosure: Control Plane publishes this guide.
TL;DR
- Architecture. Use a managed control plane (EKS, GKE, or AKS), separate system and workload node pools, an ingress controller or Gateway API implementation behind a cloud load balancer, cert-manager for TLS, and network policies for isolation.
- Upgrades. Read release notes, check for deprecated APIs, upgrade staging first, upgrade the control plane, then roll through node pools one at a time with surge nodes and PodDisruptionBudgets.
- Autoscaling. Layer the Horizontal Pod Autoscaler (HPA) for replica count, the Vertical Pod Autoscaler (VPA) for right-sizing, and a node autoscaler (Cluster Autoscaler or Karpenter) for capacity. KEDA adds event-driven triggers.
- Stateful data. Managed databases are the simplest path. When you need Postgres in Kubernetes, use an operator such as CloudNativePG with replicas across zones, automated failover, and rehearsed backup and restore.
- When not to run it yourself. If your team spends more time operating Kubernetes than shipping product, a platform that handles Day 2 for you is the faster path.
What is a simple Kubernetes cluster architecture for production, including ingress and load balancer traffic flow?
A production cluster has three layers: the control plane, the node pools, and the networking stack that carries traffic from the internet to your pods.
Reference architecture
Managed control plane. Use a managed Kubernetes service so the cloud provider runs etcd, the API server, and control plane upgrades. The three major options are Amazon EKS, Google Kubernetes Engine (GKE), and Azure Kubernetes Service (AKS).
Node pools. Create at least two node pools. A small system node pool runs cluster add-ons (CoreDNS, metrics-server, the ingress controller). A workload node pool runs your application pods. Spread both pools across three availability zones. Use topology spread constraints or pod anti-affinity rules so replicas land in different zones. Set resource requests and limits on every pod so the scheduler can place pods accurately.
Ingress and gateway layer. An ingress controller or a Gateway API implementation sits behind a cloud load balancer and routes external HTTP and HTTPS traffic to Kubernetes Services. Use cert-manager to issue and renew TLS certificates from Let’s Encrypt or an internal CA. Use external-dns to keep DNS records in sync with your routes. Apply network policies to restrict traffic between namespaces.
Secrets management. Store secrets in an external secrets manager (AWS Secrets Manager, Google Secret Manager, Azure Key Vault, or HashiCorp Vault) and sync them into the cluster with the External Secrets Operator or the Secrets Store CSI Driver.
Request path from DNS to pod
- DNS resolution. The client resolves your domain. The record, kept current by external-dns, points to the cloud load balancer.
- Cloud load balancer. The load balancer accepts the connection, optionally terminates TLS, and forwards traffic to the ingress controller’s pods, either through node ports or directly to pod IPs (IP target mode on the AWS Load Balancer Controller, container-native load balancing on GKE). The load balancer’s health checks decide which ingress targets receive traffic.
- Ingress controller or gateway. The controller terminates TLS if the load balancer did not, matches the host and path, and forwards the request to the right Kubernetes Service. Most ingress controllers read the Service’s endpoints and send traffic directly to pod IPs.
- Service. The Service selects ready pods by label. kube-proxy or a CNI plugin such as Cilium balances connections across pod IPs.
- Pod. The container handles the request. Readiness probes decide whether a pod is listed as a Service endpoint. Liveness probes restart a container that has stopped responding.
Component table
| Component | Purpose | Common open-source choice |
|---|---|---|
| Gateway or ingress controller | HTTP routing, TLS termination | Gateway API implementations (Envoy Gateway, Cilium, Istio, Contour) |
| TLS certificates | Automated certificate issuance and renewal | cert-manager |
| DNS automation | Sync DNS records with routes | external-dns |
| CNI plugin | Pod networking, network policy | Cilium, Calico |
| Metrics | Cluster and application metrics | Prometheus |
| Tracing | Traces, metrics, and logs | OpenTelemetry |
| Secrets sync | External secrets into Kubernetes | External Secrets Operator |
| Policy engine | Admission control, compliance | OPA Gatekeeper, Kyverno |
The community ingress-nginx controller was retired in March 2026 and receives no further releases or security fixes. The Kubernetes project recommends migrating to the Gateway API.
How do I plan zero downtime Kubernetes upgrades and migrations while keeping AI infrastructure compliant and stable?
Kubernetes ships about three minor releases a year, and the project maintains release branches for the three most recent minor versions; the Kubernetes releases page lists the current versions and their end-of-life dates. Each release can remove APIs and change defaults, so a repeatable upgrade process keeps routine upgrades from turning into incidents.
Upgrade runbook
- Read the release notes. The release notes and the deprecated API migration guide list every removal and behavior change.
-
Check API deprecations. The API server exposes the
apiserver_requested_deprecated_apismetric, and managed services such as EKS and GKE flag deprecated API usage before an upgrade. Fix manifests and Helm charts before you upgrade. - Respect version skew. Upgrade one minor version at a time; the control plane cannot skip minor versions, and kubelets may lag the API server by up to three minor versions. Upgrade add-ons (CNI, CoreDNS, CSI drivers, kube-proxy) to versions that support the target.
- Upgrade a staging cluster first. Apply the new version to a non-production cluster that mirrors production. Run integration and smoke tests, and confirm every controller, operator, and custom resource still works.
- Upgrade the control plane. On a managed service this is a console action, a CLI command, or a Terraform change. Running pods keep serving while the control plane updates.
- Upgrade node pools one at a time. Add surge nodes on the new version, then cordon and drain old nodes. PodDisruptionBudgets limit how many replicas of an application can be evicted at once. Readiness probes keep traffic off replacement pods until they can serve. Give pods a preStop delay and a termination grace period so endpoint removal finishes before shutdown. Single-replica applications still drop traffic, and a PodDisruptionBudget that allows zero disruptions blocks the drain.
- Validate after each pool. Check pod health, error rates, and latency before moving to the next pool.
Blue/green cluster upgrades
For large version jumps, or when an in-place rollback is unacceptable, build a new cluster on the target version, deploy workloads to it, shift traffic at the load balancer or DNS layer, and retire the old cluster. This costs more while both clusters run and removes in-place upgrade risk. Stateful data must be replicated or restored into the new cluster, and DNS TTLs set how fast traffic moves.
GPU and AI workload considerations
The NVIDIA driver on the node, the CUDA version in the container image, and the framework version must be compatible. Validate the combination in staging before you roll GPU node pools.
Compliance and change management
Keep a change record for every upgrade: what changed, who approved it, what was tested, and how to roll it back. Store cluster configuration in version control so auditors can see the history of every change.
Which managed Kubernetes solution is best for autoscaling EC2 nodes and automatically scaling pods?
Kubernetes autoscaling works at two layers. Pod autoscaling changes how many pods run or how large they are. Node autoscaling changes the compute underneath them. On AWS, the usual answer is EKS with the HPA for pods and either Karpenter or EKS Auto Mode, which runs Karpenter for you, for EC2 nodes. GKE and AKS offer managed node provisioning of their own.
Pod autoscaling
Horizontal Pod Autoscaler (HPA). The HPA adjusts replica count from CPU, memory, or custom metrics. HPA computes CPU utilization as a percentage of the pod’s request, so accurate requests matter. Stabilization windows prevent flapping during short spikes.
Vertical Pod Autoscaler (VPA). The VPA recommends or sets resource requests from observed usage. Do not let VPA and HPA act on the same CPU or memory metric; running VPA in recommendation mode alongside HPA avoids the conflict.
KEDA. KEDA scales pods from event sources such as queue depth, stream lag, cron schedules, or Prometheus queries. It manages HPA objects for you and can scale a workload to zero.
Node autoscaling
Cluster Autoscaler. The Cluster Autoscaler adds nodes when pods cannot be scheduled and removes underused nodes. It works with node groups and managed node pools on all three major clouds.
Karpenter. Karpenter is an open-source node provisioner first built for AWS. It launches right-sized instances directly from the cloud API instead of scaling fixed node groups, and it consolidates workloads onto fewer or cheaper nodes when demand drops.
GKE Autopilot and node auto-provisioning. GKE Autopilot manages nodes for you and bills general-purpose pods by the resources they request; pods that select specific hardware, such as GPUs or a machine series, are billed by node. On GKE Standard, node auto-provisioning creates and deletes node pools as needed.
AKS node autoprovisioning. AKS node autoprovisioning, built on Karpenter, picks VM sizes from pending pod requests so you don’t predefine a node pool for each VM size.
Autoscaling comparison
| What scales | Trigger | Typical reaction time | Caveats |
|---|---|---|---|
| HPA (pod replicas) | CPU, memory, custom metrics | Seconds to low tens of seconds | Needs accurate requests; new pods still need scheduling, image pull, and readiness, which takes minutes if nodes must be added |
| VPA (pod resources) | Observed usage | Minutes; may recreate pods unless in-place resize is enabled | Conflicts with HPA on the same metric |
| KEDA (pod replicas) | Queue depth, stream lag, cron, external metrics | Seconds to low tens of seconds | Adds a controller to operate; supports scale to zero |
| Cluster Autoscaler (nodes) | Unschedulable pods | Minutes | Limited to the instance types in each node group |
| Karpenter (nodes) | Unschedulable pods, consolidation | Usually faster than node-group scaling | GPU instances launch slower; spot interruptions need fallback capacity |
| GKE Autopilot (nodes) | Pod scheduling | Minutes | Less control over node configuration |
On Control Plane, each workload autoscales per location between a minimum and maximum, targeting concurrent requests, requests per second, CPU, memory, or latency depending on workload type, and KEDA is available for event-driven scaling. Capacity AI, Control Plane’s right-sizing of CPU and memory based on historical usage, adjusts resource allocation over time. Capacity AI is not used for GPU workloads or for workloads that autoscale on CPU or on multiple metrics.
Best platforms for running stateful workloads like Postgres in containers
Running Postgres in production needs durable storage, replication with automated failover, rehearsed backups, and a plan for major version upgrades. There are three broad options.
Managed database services
The cloud provider handles replication, failover, backups, and minor-version patching, and provides tooling for major version upgrades that you schedule.
- AWS: Amazon RDS for PostgreSQL and Amazon Aurora PostgreSQL
- GCP: Cloud SQL for PostgreSQL and AlloyDB
- Azure: Azure Database for PostgreSQL
You give up some control and portability in exchange for a lighter operational load.
Kubernetes operators
Operators turn Postgres lifecycle tasks (provisioning, replication, failover, backup, restore) into Kubernetes resources. The established open-source operators are:
They run Postgres pods with persistent volumes, configure streaming replication, promote a replica when the primary fails, and archive WAL to object storage for point-in-time recovery.
Platforms with packaged Postgres templates
On Control Plane, the template catalog includes highly available Postgres (a Patroni cluster of at least three members with one leader and streaming replicas) and Postgres Multi-Location (one Patroni cluster stretched across locations with a single primary, streaming replicas in other locations, automatic promotion when a location is lost, and a leader-routing proxy per location). pgEdge and CockroachDB templates are also available. Stateful workloads on Control Plane use volume sets: persistent volumes with a performance class, snapshots, and restore from snapshot.
Key operational considerations
Synchronous vs. asynchronous replication. With synchronous replication, the primary confirms a commit only after a replica has it, so failover to that synchronous replica loses no committed transactions. The cost is added write latency, and writes stall when no synchronous replica is available unless the operator falls back to asynchronous mode, which drops the guarantee. Asynchronous replication is faster and can lose the most recent transactions during failover. Most operators let you choose.
Backups and point-in-time recovery. Take base backups to object storage and archive WAL continuously. Rehearse restores on a schedule.
Storage classes and volume performance. Use SSD-backed storage classes sized for your IOPS and throughput. Block volumes such as Amazon EBS and Persistent Disk are zonal, so a pod cannot follow its volume to another zone; replica failover is what survives a zone loss. Test what happens when a volume detaches, a node dies, or a zone goes down.
Major version upgrades. pg_upgrade needs a short outage. Copy mode scales with data size; link mode takes minutes but leaves no fallback once the new cluster starts. Run ANALYZE afterward. Logical replication to a new cluster shortens the cutover, but it does not carry DDL or sequence values, and tables need primary keys or a replica identity.
Do not autoscale database replicas with HPA. Replication topology and consistency rules make reactive scaling unsafe. Change replica counts deliberately through the operator.
Decision table
| Factor | Managed service | Kubernetes operator | Platform template |
|---|---|---|---|
| Operational burden | Lowest | Highest | Low to moderate |
| Portability | Tied to one cloud | Any conformant cluster | Any cloud or location Control Plane runs on, including your own clusters and servers |
| Customization | Limited | Full | Template values |
| Failover | Automatic when high availability is enabled | Operator-managed | Automatic Patroni promotion (Postgres templates) |
| Backup and PITR | Built in | Operator-configured | Optional logical or WAL-G backups, off by default; restore is manual |
| Major version upgrades | Managed or semi-managed | Declarative in CloudNativePG and PGO; needs planning | Manual, needs planning |
Why is Kubernetes considered complex for AI infrastructure teams and what are the biggest operational pain points?
AI and ML teams face every standard Kubernetes challenge plus problems specific to GPUs, large models, and fast-moving frameworks.
- GPU scheduling and node pools. Kubernetes treats GPUs as extended resources, and a pod gets whole GPUs unless you add sharing such as MIG, time-slicing, or MPS, or allocate devices through Dynamic Resource Allocation. Mitigation: isolate GPU node pools with labels and taints, and use the NVIDIA GPU Operator to manage drivers and the device plugin.
- Driver and CUDA version management. A mismatch between the node driver, the CUDA version in the image, and the framework causes crashes or silent failures. Mitigation: pin driver versions in node images and test the full stack in CI.
- Image sizes and pull times. AI images often run to several gigabytes, and pulling one onto a cold node can take minutes. Mitigation: pre-pull or cache images on GPU nodes and trim images with multi-stage builds.
- Autoscaling GPU nodes with slow cold starts. GPU instances take longer to launch than CPU instances and are often capacity-constrained, so a scale-up can leave pods pending. Mitigation: keep a warm GPU node or reserved capacity for latency-sensitive inference.
- Multi-tenant isolation. Teams sharing a GPU cluster can starve each other. Mitigation: use namespaces, ResourceQuotas, LimitRanges, and priority classes.
- Networking and egress costs. Moving large datasets between zones or out of a cloud adds transfer charges. Mitigation: keep data and compute in the same zone and use topology-aware scheduling.
- Storage for models and datasets. Model checkpoints and datasets need high-throughput storage that many pods can read at once. Mitigation: use shared file storage through CSI drivers (Amazon FSx for Lustre, Filestore, Azure Files) or object storage, and cache hot models on local disks.
- Skills gap. Most ML engineers are not Kubernetes operators. Mitigation: give them a platform layer that hides cluster details.
On Control Plane, workloads can attach NVIDIA GPUs, and the pricing page lists models from T4 to B200. Control Plane handles scheduling, autoscaling, and rolling updates for those workloads.
Where can I find the best in class container management and networking tools?
The CNCF Landscape is the most complete directory of cloud native projects.
Service mesh. Istio and Linkerd add mutual TLS, traffic management, and service-to-service telemetry.
CNI plugins. Cilium is eBPF-based and covers network policy, observability, and service mesh features. Calico is widely deployed and supports network policy and BGP.
Ingress and Gateway API. The Gateway API is the successor to the Ingress resource, with richer routing, traffic splitting, and role-based configuration. Implementations include Cilium, Istio, Contour, and Envoy Gateway; cloud options include the AWS Load Balancer Controller and GKE Ingress.
Observability. Prometheus for metrics, OpenTelemetry for traces, metrics, and logs, and Grafana with Alertmanager for dashboards and alerting.
Policy engines. OPA Gatekeeper and Kyverno enforce admission policies such as required labels, no privileged containers, and mandatory resource limits.
For a comparison of platforms that manage Kubernetes for you, see the best Kubernetes management platforms roundup.
When not to run Kubernetes yourself, and how Control Plane fits
If your engineers spend more hours on cluster upgrades, node patching, autoscaler tuning, and incident response than on your product, operating Kubernetes is costing you delivery time.
Control Plane is the platform for running unbreakable apps and agents. It combines any cloud, neocloud, and bare metal into what Control Plane calls a Global Virtual Cloud (GVC), a set of locations a workload runs across. It runs serverless, standard, stateful, and cron workloads on AWS, GCP, and Azure; on existing Kubernetes clusters, through Bring Your Own Kubernetes (BYOK), Control Plane’s option for connecting a cluster you already run; and on your own servers, including on premises, through Managed Kubernetes (MK8s), Control Plane’s way of turning your own servers into managed locations. Many teams run it in one cloud as well.
Control Plane handles Day 2 operations for you, including autoscaling, Capacity AI right-sizing, built-in observability, and security, with no Kubernetes cluster you have to operate. With MK8s, Control Plane operates the Kubernetes layer on the servers you join; with BYOK, you keep running the cluster you connected.
Every workload update is a rolling update: the previous version keeps serving until the new version reports ready. Readiness and liveness probes decide when a replica takes traffic and when it restarts. A workload deployed to a GVC runs active-active in every location of that GVC, and a global endpoint routes each request to the nearest healthy location. The 99.999% availability SLA applies to workloads running at least two replicas in at least two locations. Those locations can be two regions of one cloud or locations on different providers. For the cost side of that design, see active-active vs. active-passive costs.
Customers typically cut compute costs 30 to 50 percent compared with running directly on AWS, GCP, or Azure. Control Plane is PCI DSS Level 1, SOC 2 Type II, HIPAA, and GDPR compliant. Control Plane ships Terraform and Pulumi providers, so workloads fit the same infrastructure-as-code review process as the rest of your stack, plus an MCP server for AI assistants.
Teams weighing whether to build a platform team at all can read the guide to production-grade infrastructure without a platform team.
FAQ
Short answers to common production Kubernetes questions.
What is a simple Kubernetes cluster architecture for production, including ingress and load balancer traffic flow?
A managed control plane (EKS, GKE, or AKS), worker nodes across three availability zones, and an ingress controller or Gateway API implementation behind a cloud load balancer. Traffic flows from DNS to the load balancer, through the ingress or gateway where TLS usually terminates, to a Kubernetes Service, and then to pods that pass their readiness probes.
How do I plan zero downtime Kubernetes upgrades and migrations while keeping AI infrastructure compliant and stable?
Read the release notes and fix deprecated APIs, upgrade a staging cluster, upgrade the production control plane, then roll node pools one at a time with surge nodes and PodDisruptionBudgets. For AI workloads, validate GPU driver and CUDA compatibility before touching GPU node pools, and keep a change record for every upgrade.
Which managed Kubernetes solution is best for autoscaling EC2 nodes and automatically scaling pods?
On AWS, EKS with the Horizontal Pod Autoscaler for pods and Karpenter, or EKS Auto Mode, which runs Karpenter for you, for EC2 nodes. Karpenter launches right-sized instances directly instead of scaling fixed node groups. Add KEDA if workloads scale on queue depth or event streams.
What are the best platforms for running stateful workloads like Postgres in containers?
Managed database services (RDS, Aurora, Cloud SQL, AlloyDB, Azure Database for PostgreSQL) carry the lightest operational load. Inside Kubernetes, operators such as CloudNativePG, Zalando’s Postgres Operator, and Crunchy Data’s PGO handle replication, failover, and backup. Control Plane’s template catalog includes Patroni-based highly available and multi-location Postgres.
Why is Kubernetes considered complex for AI infrastructure teams?
GPU scheduling, driver and CUDA compatibility, large image pulls, slow GPU node launches, multi-tenant contention, and the gap between ML engineering and cluster operations are the main sources. Many AI teams adopt a platform layer to absorb them.
How does pod autoscaling work with node autoscaling in Kubernetes?
The HPA adds or removes pod replicas based on metrics. When new pods cannot be scheduled, the node autoscaler (Cluster Autoscaler or Karpenter) adds nodes, and it removes nodes that become underused after pods scale down. Configure both layers together.
What are the biggest operational pain points of running Kubernetes in production?
Upgrade churn across Kubernetes versions, node images, and add-ons is the most persistent. Autoscaler tuning, secrets management, network policy, observability, and incident response for stateful workloads follow. Teams without platform engineers often find Day 2 work takes more time than the initial setup.
Can I run production Kubernetes workloads without managing a cluster myself?
Yes. On Control Plane-managed compute in AWS, GCP, and Azure regions there is no cluster to operate, and autoscaling, rolling updates, observability, and security are built in. You can also connect existing clusters with BYOK, which you keep operating, or join your own servers with MK8s, where Control Plane operates the Kubernetes layer.
How do I run Postgres with high availability across multiple cloud regions?
Run a Patroni cluster with a single primary, streaming replicas in other regions, and leader-aware routing in each location, and decide whether replication to the remote regions is synchronous or asynchronous; synchronous commits across regions add a network round trip to every write. Control Plane’s Postgres Multi-Location template packages this design.
Where can I find the best container management and networking tools for Kubernetes?
Start with the CNCF Landscape. For CNI, look at Cilium and Calico; for service mesh, Istio and Linkerd; for observability, Prometheus and OpenTelemetry; for policy, OPA Gatekeeper and Kyverno.
Talk to the Control Plane team or sign up free to run your containers without operating Kubernetes.
Originally published on the Control Plane blog.
Top comments (0)