DEV Community: softverseAdmin

Amazon EKS in 2026: Managed Kubernetes on AWS, Explained for Working Engineers

softverseAdmin — Sat, 31 Oct 2026 09:15:41 +0000

TL;DR: EKS gives you a managed Kubernetes control plane and leaves everything else — nodes, networking shape, IAM wiring, add-ons — as your design decisions. The choices that matter: nodes come as managed node groups (the sensible default), self-managed groups (when you need control), or Fargate (serverless pods, with real constraints); Karpenter increasingly replaces static groups for teams that want nodes to follow demand. Networking runs on your VPC — pods get real VPC IPs from your subnets, which makes the VPC design a Kubernetes decision. IAM meets Kubernetes through Pod Identity (the current, simpler answer to "how does this pod get AWS permissions"), and the cluster is only production-shaped once the add-ons land: EBS CSI for storage, the Load Balancer Controller for ingress, external-dns and cert-manager for the public face. What EKS removes is the control plane burden. Everything in the rest of this series still applies — it just runs on AWS now.

Originally published on DevOps Enginer.

Most teams choosing EKS are not really choosing Kubernetes — they already know it. They are choosing to stop operating a control plane. What surprises them is how much remains theirs: EKS is managed Kubernetes, not managed everything, and the gap between "cluster created" and "cluster production-ready" is a series of decisions this guide walks through.

What AWS manages, and what it pointedly does not

AWS runs the control plane — API servers and etcd across multiple AZs, patched and highly available — and charges a flat per-cluster hourly fee for it. That is the entire managed surface. Nodes are your EC2 instances (or Fargate tasks), the VPC is yours, IAM is yours, and the add-ons that make a cluster useful are yours to install and maintain. Understanding this boundary prevents the most common EKS disappointment: expecting an app platform and receiving excellent infrastructure primitives.

Cluster creation itself (eksctl, Terraform, console) is the easy afternoon. The decisions below are the ones with consequences.

Nodes: three ways to run pods, and a fourth that changes the question

Managed node groups — AWS handles the EC2 lifecycle (provisioning, AMI updates, draining on termination) behind a group definition: instance type, size range, subnets, labels. This is the default recommendation: boring, supported, and sufficient for most clusters.

Self-managed node groups — you own the Auto Scaling group and the bootstrap. Justified by custom AMIs, unusual instance configuration, or requirements the managed path doesn't expose. You also own every patch and every upgrade chore — the upgrades guide applies with extra steps.

Fargate — no nodes at all: each pod runs in an isolated micro-VM with a per-pod price. The constraints are the fine print: no DaemonSets, no persistent volumes beyond EFS, pod sizes from a fixed menu, and startup latency that batch and CI workloads forgive and latency-sensitive services don't. Excellent for spiky, stateless, low-count workloads; a poor default for a whole cluster.

Karpenter — the open-source autoscaler from AWS that increasingly replaces static groups and the cluster-autoscaler pattern: it watches pending pods and launches right-sized instances (type, family, AZ chosen per workload) in under a minute, then consolidates under-used nodes away. For clusters whose demand moves — most clusters — Karpenter is where the autoscaling story lands on AWS, and its cost effect is the rare one that shows up on the bill uninvited, in the good direction.

Networking: your VPC is the cluster network

EKS uses the AWS VPC CNI by default, which means pods receive real IP addresses from your VPC subnets. The consequences are pleasant (pods are first-class network citizens; security groups can even attach per-pod) and constraining (pod density per node is limited by ENI/IP capacity, and subnet IP space is a real planning input — small subnets plus big clusters is a known way to run out of addresses). Prefix delegation raises pod density substantially where needed. This is why the subnet sizing in the VPC guide leaned generous: Kubernetes spends IP addresses like it means it.

The public face of the cluster is the AWS Load Balancer Controller: annotate an Ingress or Service and it provisions an ALB/NLB with target groups wired to your pods — the AWS-native version of the ingress patterns covered earlier. Add external-dns (Route 53 records follow your ingresses automatically) and cert-manager with ACM or Let's Encrypt, and "deploy an app, get a public HTTPS URL" becomes the paved road rather than a manual ritual.

IAM inside the cluster: Pod Identity is the current answer

Pods need AWS permissions — read this bucket, publish to that queue — and the cardinal rule is unchanged from everywhere else in AWS: per-workload roles, never node-wide credentials. A node role big enough for every pod on the node is a privilege-escalation welcome mat, per the IAM least-privilege guide.

EKS has had two mechanisms. IRSA (IAM Roles for Service Accounts) works via OIDC and annotations and remains everywhere in existing clusters. EKS Pod Identity is the newer, simpler successor: associate an IAM role with a service account through the EKS API, an agent on the node handles credentials, and the annotation archaeology of IRSA goes away. New clusters should start on Pod Identity; existing ones can run both while migrating. Either way, the design is identical: one role per workload, scoped to the resources that workload touches, with the cluster components (CSI driver, LB controller, Karpenter, external-dns) each holding their own deliberately-sized roles.

The add-ons that make it production-shaped

A fresh EKS cluster is deliberately minimal. The standard load-out, whether installed as EKS add-ons or via Helm/Argo CD:

EBS CSI driver — without it, PersistentVolumes on EBS don't exist; the storage model (gp3 classes, Retain policies, snapshots) hangs off this driver on AWS.
AWS Load Balancer Controller — ingress and services, as above.
CoreDNS and kube-proxy — shipped as EKS add-ons; keep their versions in step with cluster upgrades rather than discovering skew during one.
Metrics and logs — metrics-server or the Prometheus stack for the observability baseline, plus a log route (Fluent Bit to CloudWatch Logs or your aggregator). A cluster nobody can see into is a cluster on borrowed time.
Cluster access — modern EKS uses access entries (API-based authentication mapping IAM principals to Kubernetes permissions), replacing the hand-edited aws-auth ConfigMap that caused a generation of "I locked everyone out of the cluster" incidents. Whoever provisions the cluster should set access policy deliberately on day one; the creator-admin trap (only the creating identity can administer, ever) is an old rite of passage best skipped.

Everything after installation is the standard Kubernetes operating model — GitOps delivery, progressive rollouts, security basics. EKS changes the substrate, not the discipline.

Upgrades and cost: the two recurring appointments

Upgrades follow the standard Kubernetes rhythm with AWS holding the control-plane end: you trigger the control-plane upgrade one minor version at a time, then roll node groups (or let Karpenter drift-replace nodes onto the new AMI), then bump add-ons. Support windows matter here: standard support for a Kubernetes version on EKS runs about 14 months from its EKS release, with paid extended support beyond — the calendar in the upgrades guide applies verbatim.

Cost has three components people chronically under-model: the per-cluster control-plane fee (multiplied by every forgotten dev cluster — see the Kubernetes cost guide for why cluster sprawl is the quiet line item), the nodes themselves (where Karpenter's right-sizing and consolidation do real work, plus Spot for fault-tolerant capacity), and the AWS perimeter — NAT gateways, load balancers and data transfer that the VPC design decided before the cluster existed.

Common mistakes

Expecting a platform. EKS is infrastructure with a managed control plane. Teams that budget zero platform work discover the add-on, IAM and networking decisions arriving anyway, unscheduled.
The node role that does everything. One broad role for the node group means every pod inherits it. Per-workload identity via Pod Identity is the entire fix, and it's cheap.
Subnets sized for the demo. Pods consume VPC IPs directly; small subnets cap cluster growth in a way that surfaces mid-incident. Size for the cluster you expect, with the VPC's spare range held back.
aws-auth archaeology. Hand-editing the access ConfigMap (on clusters still using it) locks teams out with one typo. Prefer access entries; if you must edit aws-auth, do it from a session that stays open until a second login is verified.
Add-ons drifting behind the cluster version. CSI and LB controller versions have compatibility windows; treat add-on upgrades as part of the cluster upgrade runbook, not a separate someday.
A cluster per team per environment, forever. The per-cluster fee plus duplicated add-ons and perimeter makes sprawl a budget line. Consolidate with namespaces and per-team IAM where isolation requirements genuinely allow it — and keep dedicated clusters where they don't (compliance, blast radius), knowingly.

FAQ

EKS vs self-managed Kubernetes on EC2?
Self-managed gives total control and total responsibility: control-plane HA, etcd backups, and patching are yours. EKS removes exactly that layer for a flat fee. Teams choose self-managed for specific control or cost-at-scale reasons; everyone else is buying back their weekends with EKS, correctly.

EKS vs ECS Fargate?
If the team knows Kubernetes or needs its ecosystem (Helm charts, operators, the tooling in this series), EKS. If the need is "run these containers on AWS with minimal concepts," ECS is simpler and entirely respectable. The honest test is which ecosystem your hiring, tooling and existing workloads already speak.

Can I run a cluster with no managed node groups at all?
Yes — Fargate-only clusters exist and suit small, spiky, stateless estates. The DaemonSet and storage constraints decide for most teams; mixed setups (node groups for the baseline, Fargate for burst jobs) are common.

How many clusters should we have?
Fewer than instinct suggests, more than one for anything serious: separate production from non-production at minimum. Beyond that, let blast radius, compliance boundaries and team autonomy argue it out — with the per-cluster and perimeter costs on the table while they do.

Conclusion

EKS is a trade stated plainly: AWS operates the control plane; you keep the architecture. Choose node strategy deliberately (managed groups by default, Karpenter when demand moves), design the VPC like the cluster depends on it — because it literally runs on it — wire IAM per workload with Pod Identity, and install the add-ons that turn a cluster into a platform. Do those things and EKS delivers what it promises: Kubernetes without the control-plane pager. The rest of running Kubernetes well is, as ever, on you — and it's what the other fifteen parts of this series are for.

The free Kubernetes Production Checklist (via the newsletter on the homepage) includes the production-readiness items this guide builds toward — access, backups, observability and upgrade readiness.

AWS VPC in 2026: The DevOps Guide to Subnets, Routing and Not Locking Yourself Out

softverseAdmin — Sat, 31 Oct 2026 08:56:52 +0000

TL;DR: A VPC is your own slice of AWS networking, and almost everything about it is decided by five choices: the CIDR block (bigger than you think you need, because it can never grow), the subnet layout (public for load balancers, private for workloads, isolated for data — spread across at least two, ideally three AZs), the route tables (a subnet is only "public" because its route table points at the internet gateway), NAT gateways (how private subnets reach the internet — one per AZ, because a shared NAT is both a bottleneck and a bill), and the two security layers (security groups, stateful and attached to instances; network ACLs, stateless and attached to subnets). Add VPC endpoints for AWS services before the NAT bill teaches you why, enable flow logs from day one, and never use the default VPC for anything you care about.

Originally published on DevOps Enginer.

Networking is the part of AWS everyone clicks through on the way to launching something — and the part that is nearly impossible to change later. A CIDR block cannot be resized. A subnet layout chosen in an afternoon shapes every architecture decision after it. This guide is the VPC as DevOps engineers actually need to understand it: the design decisions, in the order they become permanent.

The CIDR block: the one-way door

Your VPC gets an IPv4 range like 10.20.0.0/16 — 65,536 addresses. Choose it like it matters, because:

It cannot be enlarged later. You can attach secondary CIDRs, but the clean answer is to start generous: a /16 per VPC is the common, sane default. Address space is free; running out is not.
It must not collide with anything it will ever talk to — your other VPCs (peering and Transit Gateway demand non-overlapping ranges), your office network, your VPN ranges. Collisions discovered at integration time are miserable.
Pick from private ranges everyone recognises (10.x is conventional) and keep a simple allocation register: which block belongs to which account/environment/region. A spreadsheet beats a memory, and it will be consulted at the worst possible moment if it doesn't exist.

Subnets carve the block per AZ and tier: a workable pattern is /20s per subnet (4,096 addresses each) — public, private and data tiers across three AZs consumes nine subnets and still leaves most of a /16 unallocated for the future. Remember AWS reserves five addresses in every subnet (network, router, DNS, future use, broadcast), so a subnet's usable count is its size minus five.

Subnets: three tiers, and what "public" actually means

The tier names describe routing, not intention:

Public subnets — route table has a route to an Internet Gateway (IGW). For: load balancers, NAT gateways, bastions — the small set of things that face the internet. Nothing else earns a public IP.
Private subnets — no IGW route; outbound internet goes via a NAT gateway in a public subnet. For: application servers, containers (your EKS/EC2 workloads), everything that should be reachable only through a load balancer.
Isolated (data) subnets — no internet route at all, in or out. For: databases, caches, queues — resources that should only ever talk to the application tier inside the VPC.

The sentence worth internalising: a subnet is public or private solely because of its route table. There is no checkbox. Misunderstanding this is the root of most "why can/can't my instance reach the internet" mysteries — the answer is always in the route table, never in the subnet's name.

VPC 10.20.0.0/16
├── AZ a: public 10.20.0.0/20 · private 10.20.16.0/20 · data 10.20.32.0/20
├── AZ b: public 10.20.48.0/20 · private 10.20.64.0/20 · data 10.20.80.0/20
└── AZ c: public 10.20.96.0/20 · private 10.20.112.0/20 · data 10.20.128.0/20
    (remainder of the /16 left unallocated, on purpose)

Routing and NAT: where the money and the outages live

Route tables attach per subnet (with one main table as the default). The public table points 0.0.0.0/0 at the IGW; private tables point 0.0.0.0/0 at a NAT gateway; data tables have only the local route. Two operational truths follow:

NAT gateways are per-AZ infrastructure, not a shared utility. One NAT gateway costs a fixed hourly fee plus per-GB processing — and routing three AZs through one NAT in AZ-a means (a) an AZ-a failure takes outbound internet from the whole VPC, and (b) you pay cross-AZ data transfer for the privilege. One NAT per AZ is the production pattern; the cost guide explains the bill either way. Dev environments can share one NAT and accept the risk knowingly.

VPC endpoints are the other half of the traffic bill. Without them, your private subnets reach S3, DynamoDB and every other AWS API through the NAT gateway, paying per-GB for traffic that never needed to leave AWS's network. Gateway endpoints for S3 and DynamoDB are free — there is no excuse for their absence. Interface endpoints (PrivateLink) for the likes of ECR, CloudWatch Logs and Secrets Manager cost an hourly fee per AZ but keep agent and image-pull traffic fully private — on a busy EKS cluster, pulling images through NAT is a cost and failure-domain decision, not a default.

The two security layers (they are not interchangeable)

Security groups attach to instances/ENIs, are stateful (reply traffic is automatically allowed), support allow-rules only, and can reference other security groups — "the database accepts 5432 from the app tier's group" is the single most useful pattern in AWS networking, because it tracks membership automatically as instances come and go.

Network ACLs attach to subnets, are stateless (return traffic needs its own rule — the ephemeral-port gotcha), support allow and deny rules evaluated in numbered order, and apply to everything in the subnet whether you remembered it or not.

The working model: security groups are the primary control, designed per-tier with group-to-group references; NACLs stay near-default (allow all) unless you need a subnet-level guardrail — a deny rule that stands even if someone edits a security group badly. Teams that build elaborate NACL logic usually discover the statelessness tax during their first debugging session at 2 a.m. And least privilege applies to who can edit these as much as to the rules themselves — network changes belong behind the same IAM discipline as everything else.

Flow logs: turn them on before you need them

VPC Flow Logs record accepted/rejected traffic metadata to CloudWatch Logs or S3. They answer the questions every network incident asks — "is traffic even reaching this subnet?" "what is talking to the database?" — and they cannot be back-filled. Enabling them at VPC creation costs almost nothing and is the difference between evidence and vibes during debugging; the observability habits from the Kubernetes side apply here unchanged. While you're in the console: DNS resolution and DNS hostnames on (private DNS names make internal life bearable), and no workloads in the default VPC — it exists for experiments, its layout is nobody's design, and its flat structure has surprised many auditors.

Connecting beyond one VPC

Sooner or later the VPC needs neighbours, and each option has its shape:

VPC peering — a direct link between two VPCs: cheap, fast, and non-transitive (A↔B and B↔C does not give A↔C), with CIDRs that must not overlap. Fine for a pair; a mesh of many VPCs becomes unmanageable surprisingly fast.
Transit Gateway — the hub: VPCs and VPNs attach to one router, routes propagate sensibly, and the architecture stays legible at ten VPCs and beyond. It costs more than peering and earns it the moment "many" enters the sentence.
VPN / Direct Connect — to your office or data centre. Start with site-to-site VPN (fast to stand up, internet-bound); Direct Connect is the committed, consistent-bandwidth option with lead times measured in weeks, ordered when the VPN's variability becomes a real complaint rather than a theoretical one.
For Kubernetes specifically: EKS nodes live in your private subnets and inherit all of this — the cluster's outbound internet, endpoint choices and NAT layout are VPC decisions made before the first node joins, per the networking thinking in the Kubernetes networking guide.

And define it all as code. A VPC clicked together in the console is a snowflake nobody can rebuild; the same network as a Terraform module — CIDR, tiers and AZ count as variables — is an asset every new environment inherits.

Common mistakes

The too-small CIDR. A /24 VPC feels tidy until the second AZ or the EKS cluster arrives (Kubernetes networking is address-hungry). You cannot grow it; you can only rebuild. Start at /16.
Overlapping ranges across VPCs or with the office. Peering, Transit Gateway and VPN all refuse to cooperate with collisions. Allocate from a register, not from memory.
Workloads in public subnets "temporarily". Temporary is forever. If it doesn't face the internet for a living, it's private.
One NAT gateway for three AZs — a single point of failure plus cross-AZ transfer charges. One per AZ in production; shared only where the risk is understood and accepted.
No S3/DynamoDB gateway endpoints. Free to add, and their absence sends AWS's own traffic through the metered NAT. The most expensive checkbox in AWS is the one that costs nothing.
Elaborate NACL rule sets. Stateless rules with ephemeral ports turn routine debugging into archaeology. Security groups do the real work; NACLs are guardrails.
Flow logs "later". Later is during the incident, when the missing history would have answered the question. On at creation, always.

FAQ

How many subnets do I actually need?
Per AZ: one public, one private, one data — so 6 for a solid 2-AZ setup, 9 for 3 AZs. More tiers (a separate management tier, say) are legitimate; more AZs than your region offers in your account is the usual surprise, so check AZ availability before finalising the layout.

Can I change the CIDR after creating the VPC?
You can add secondary CIDR blocks, but you cannot remove or shrink the primary one. Treat the primary block as permanent — that's why the sizing advice leans generous.

Public IP on an instance in a private subnet — possible?
No, and that's the point: no IGW route means no public reachability, whatever the instance settings say. Reach the internet via NAT (outbound) or not at all (data tier); reach the instance via load balancer, bastion or SSM Session Manager — which, not incidentally, is the modern answer to "how do I SSH in": usually, you don't; Session Manager gives audited shell access without any inbound ports.

VPC peering or Transit Gateway?
Two VPCs that need each other: peering. Anything beyond a pair, or anything involving a shared services VPC or on-prem: Transit Gateway. The cost difference is real but small next to the operational difference at scale.

Conclusion

A VPC rewards the teams who treat its first afternoon as design, not setup: a generous non-overlapping CIDR, three tiers across multiple AZs, route tables understood rather than assumed, NAT per AZ, free gateway endpoints in place, security groups referencing each other, and flow logs recording from the first packet. None of it is glamorous, and almost none of it can be fixed cheaply later — which is precisely why it's worth a guide, and worth doing once, properly.

More AWS and Kubernetes guides are published weekly on DevOps Enginer — the free Kubernetes Production Checklist is available via the newsletter on the homepage.

Terraform Modules in 2026: Reusable Infrastructure Without the Copy-Paste

softverseAdmin — Sat, 31 Oct 2026 08:43:16 +0000

TL;DR: A Terraform module is infrastructure with an interface: variables in, outputs out, opinions inside. Good modules wrap one coherent thing (a network, a cluster, an application stack), expose only the decisions callers genuinely need to make, give sensible defaults for the rest, and publish outputs that let other modules compose on top. Version them like software — semantic versions, pinned references, a changelog — because a module edit silently changes every environment that consumes it on the next plan. The failure modes are consistent: the god-module that tries to cover every case and satisfies none, the abstraction leak that forces callers to know the internals anyway, and the unpinned main branch reference that turns a routine apply into a surprise upgrade. Modules done well are how one engineer maintains ten environments; done badly, they're how a copy-paste problem becomes a shared copy-paste problem.

Originally published on DevOps Enginer.

Every growing Terraform codebase reaches the same moment: the staging VPC definition looks exactly like the production one except for six values, someone copies the folder, and from that day the two networks drift apart one forgotten edit at a time. Modules are Terraform's answer — and like most answers, they have their own failure modes. This guide covers the design decisions that separate a module library from a junk drawer.

What a module actually is

Strip the ceremony away: any directory of .tf files is a module. The root module is the one you run terraform apply in; child modules are directories (or registry entries, or git repos) that the root calls with a module block:

module "network" {
  source  = "git::https://example.com/infra-modules.git//network?ref=v1.4.0"
  # or: source = "./modules/network"   (local)
  # or: source = "terraform-aws-modules/vpc/aws"  (registry)

  cidr_block        = "10.20.0.0/16"
  az_count          = 3
  enable_flow_logs  = true
}

output "private_subnet_ids" {
  value = module.network.private_subnet_ids
}

The contract is the whole idea: variables are the questions the module asks, outputs are the promises it makes, and everything in between is its private business. A caller should never need to read a module's internals to use it — the moment they do, the abstraction has leaked and the module has failed at its one job.

Designing the interface (where modules are won or lost)

Expose decisions, not resources. Callers should pass intent — "3 AZs, flow logs on, this CIDR" — not raw resource arguments copied from the AWS provider docs. If your variable list mirrors the underlying resource's argument list one-to-one, you have not written a module; you have written a forwarding service. The module's value is its opinions: the subnet layout, the tag set, the naming scheme, the secure defaults — decided once, encoded once.

Defaults carry the opinions. A good module can be called with three or four inputs by a team that trusts its defaults:

variable "az_count" {
  type        = number
  default     = 3
  description = "How many availability zones to spread subnets across."
  validation {
    condition     = var.az_count >= 2 && var.az_count <= 4
    error_message = "az_count must be between 2 and 4 for this network design."
  }
}

variable "enable_flow_logs" {
  type    = bool
  default = true   # secure by default; turning it off is a decision, not an accident
}

Note the shape: typed variables, a description a stranger can act on, validation that fails at plan time with a human sentence, and defaults that encode the safe choice. Every default you set is a caller who never has to think about that knob — and a team convention that can never be forgotten.

Outputs are the composition layer. Publish the identifiers other modules will need (subnet IDs, the cluster endpoint, the bucket ARN) and nothing else. Outputs are a public API: teams will build on whatever you publish, so publish deliberately, and treat removing an output as a breaking change.

Size: one coherent thing

The temptation is the mega-module — module "platform" that builds the network, the cluster, the databases and the applications in one call. It feels convenient and ages terribly: every consumer inherits every decision, a change to any layer re-plans all of them, and the state file becomes a blast radius the size of your whole estate.

The working rule: a module wraps one layer of the stack that changes together and gets consumed together. Network. Cluster. Data store. Application service. Those compose:

module "network" { source = "./modules/network"  /* ... */ }
module "cluster" {
  source     = "./modules/cluster"
  subnet_ids = module.network.private_subnet_ids
}
module "app" {
  source        = "./modules/app-service"
  cluster_id    = module.cluster.id
  image         = "registry.example.com/api:${var.api_version}"
}

Composition through outputs keeps each module independently testable, independently versionable, and independently replaceable — when the cluster design changes, the network module neither knows nor cares. The same thinking, incidentally, is why state management per layer keeps blast radius small: module boundaries and state boundaries want to be the same lines.

Versioning: modules are software, treat them that way

A module consumed from a git repo at ?ref=main is an unpinned dependency on a moving target: every consumer silently upgrades whenever anyone merges anything. The first time that merge changes a default, every environment in the company plans a diff nobody asked for — during somebody's unrelated change window.

The disciplined version:

Tag releases semantically. v1.4.0 tells consumers what kind of change they're taking: patch for fixes, minor for additive inputs/outputs, major for breaking changes (renamed variables, changed defaults, removed outputs).
Pin every reference. ?ref=v1.4.0, never a branch. Upgrading a module becomes a deliberate, one-line, reviewable act per environment — staging takes it this week, production next week, and the diff in between is the changelog.
Keep a changelog. Three honest lines per release beats a generated novel: what changed, who is affected, what to check after upgrading.
Test before tagging. A module's test surface is terraform validate, a plan against a sandbox account, and — for the modules that matter — an ephemeral apply-and-destroy in CI. The registry ecosystem (terraform test, now built in) supports real test runs with mocked providers for the logic-heavy modules. Whatever the depth, a release nobody has planned against is a beta, labelled or not.

Local path modules (./modules/...) skip versioning entirely — fine for one repo's internal layers, where the "consumers" ship in the same commit. The moment a module is consumed from another repo or by another team, it has users, and users deserve versions.

Registry modules: stand on the community carefully

The public registry holds mature modules for the common shapes — the AWS VPC module alone encodes years of edge cases. Using them is good engineering, with two cautions. First, read the interface before committing: a famous module with 200 input variables demands you learn its opinions, which may not be yours. Second, a registry module is still a dependency: pin its version, and re-check its defaults on upgrade (a default flip in a popular module has surprised more than one team). When a registry module fits, it saves months; when it almost fits, a thin in-house wrapper module — your interface on the outside, the registry module inside — keeps your callers insulated from its surface.

Common mistakes

The god-module. Network + cluster + data + apps in one call: unreviewable plans, maximal blast radius, and consumers forced to accept every layer's choices. Split by layer; compose through outputs.
Variables that mirror resource arguments. A module whose inputs are the provider's docs restated provides no leverage — the opinions are the product.
Unpinned sources. ref=main turns every merge into a fleet-wide event. Pin versions; upgrade deliberately.
Everything is an output. Outputs are API. Publishing internals invites consumers to build on them, and then "internal" refactors become breaking changes.
One module for every environment difference. A variable count that grows with every new consumer is a god-module assembling itself. When a consumer's shape is genuinely different, a second module (or composition of smaller ones) beats a flag for every fork in the road.
No validation, failure at apply time. An invalid CIDR or an az_count of 7 should fail in seconds at plan, with a sentence explaining why — not fifteen minutes into an apply, midway through creating a network.

FAQ

When should code become a module?
At the second honest reuse — or the first, if the thing is a recognised layer (network, cluster) whose shape you already know. Extracting on first use for everything produces speculative abstractions; waiting until the fourth copy produces archaeology.

Local modules or a shared module repo?
One repo's layers: local paths, shipped together. Modules with consumers in other repos or teams: their own repo (or a registry), versioned and released. The boundary is about who consumes it, not about elegance.

How do I refactor a module without breaking consumers?
The same way you change any API: additively first (new optional variables with defaults, new outputs), deprecate in the changelog, and reserve removals/renames for a major version. Consumers pinned to the old major are untouched until they choose to move.

Modules vs just using Terragrunt/workspaces?
Different layers of the problem. Workspaces and wrapper tools manage environments and repetition at the root level; modules manage reusable components. Most estates of any size end up with both, and the state guidance in the state management guide applies either way. If you're still choosing the IaC tool itself, the Terraform vs Pulumi vs CloudFormation comparison is the wider view.

Conclusion

A Terraform module is a promise: give me these few decisions, and I will give you infrastructure built the way this team agreed — every time, in every environment, from one definition. Keep the interface small and opinionated, compose layers through outputs, version like software, and pin like you mean it. Do that, and the tenth environment costs you an afternoon of variables instead of a week of copy-paste — which was the entire point.

Setting up your first environments? The Kubernetes Production Checklist and the other free resources are on DevOps Enginer — new practical guides are published weekly.

Kubernetes ConfigMaps and Secrets in 2026: Configuration Without Rebuilding Images

softverseAdmin — Sat, 31 Oct 2026 08:10:22 +0000

TL;DR: ConfigMaps hold non-sensitive configuration (feature flags, endpoints, tuning values); Secrets hold sensitive values (passwords, tokens, keys) — with one warning that must be said out loud: a Kubernetes Secret is only base64-encoded, not encrypted, so anyone with API read access to it has the value. Configuration reaches pods two ways — environment variables (simple, but frozen at pod start) and volume mounts (files the kubelet keeps updated, but only for apps that re-read them). The production pattern most teams converge on: keep config in ConfigMaps, keep real secrets in an external vault (AWS Secrets Manager, Vault, and friends) and sync them in with the External Secrets Operator, encrypt etcd at rest, scope access with RBAC, and treat rotation as a designed event, not an emergency. Configuration is where "works in staging" goes to die — this guide makes it boring instead.

Originally published on DevOps Enginer.

Rebuild an image to change a log level once, and you understand why configuration exists as its own discipline. Rebuild one to change a database password during an incident, and you never design config badly again. Kubernetes gives you two objects for this job — ConfigMaps and Secrets — and a decade of teams have learned the same few lessons about them. Here they are, in one place.

ConfigMaps: the plain-text workhorse

A ConfigMap is key-value data, nothing more exotic:

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  LOG_LEVEL: "info"
  FEATURE_CHECKOUT_V2: "true"
  app.properties: |
    pool.size=20
    pool.timeout=5s

It can carry simple values and whole files (the app.properties block above lands as a file when mounted). Creation from files or literals via kubectl create configmap or your GitOps pipeline — as with everything else on this site, the manifest in Git is the version worth having; ad-hoc kubectl creates are config drift with extra steps. ConfigMaps are namespace-scoped, capped around 1 MiB, and — importantly — they are not a database: hundreds of tiny ConfigMaps are a smell pointing at a config service or a file baked sensibly into the repo, not more objects.

Secrets: base64 is not encryption (the paragraph to actually read)

A Secret looks like a ConfigMap with type: Opaque and values that appear encoded:

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
type: Opaque
stringData:          # write plaintext here; the API server encodes it
  username: app
  password: "s3cr3t-value"

That encoding is base64 — a transport format, not protection. The defences that actually protect a Secret, in the order they matter:

RBAC: who can get/list Secrets in the namespace. Read access to a Secret is the secret. This is the control that does the most work, and it is covered properly in the security basics guide.
Encryption at rest: etcd stores Secrets; enable envelope encryption (KMS-backed on managed clusters, often a toggle) so a stolen etcd backup doesn't hand over every credential in the cluster.
Not committing plaintext Secrets to Git. The most violated rule in this entire article. If manifests live in Git — they should — Secrets in plaintext manifests means the Git repo is your vault, with Git's access list, clone history and all. The fixes are the next section.

Secret types beyond Opaque exist for specific consumers (TLS secrets for ingress, docker-registry secrets for image pulls), and those integrate with their consumers directly — an ingress TLS secret, for instance, is the standard way to terminate HTTPS per the networking guide.

Getting configuration into the pod: env vars vs files

Two delivery mechanisms, with opposite trade-offs:

    env:
      - name: LOG_LEVEL
        valueFrom:
          configMapKeyRef: { name: app-config, key: LOG_LEVEL }
      - name: DB_PASSWORD
        valueFrom:
          secretKeyRef: { name: db-credentials, key: password }
    volumeMounts:
      - name: config
        mountPath: /etc/app
  volumes:
    - name: config
      configMap: { name: app-config }

Environment variables are set at container start and never change for the life of the pod. Update the ConfigMap and nothing happens until pods restart. Simple, predictable, and the right default for values that shouldn't mutate under a running process anyway.
Volume mounts present the data as files, and the kubelet updates the files when the source changes (within a minute or so, via an atomic symlink swap). But an updated file only matters if the application re-reads it — most don't. So volume-mounted config gives you eventual files, not live reload, unless the app watches the files (some frameworks do, via the ..data symlink change).

The operational consequence people trip on: changing a ConfigMap does not roll your Deployment. If the change must take effect, trigger the rollout (kubectl rollout restart) — or use the Helm trick of checksumming the config into a pod-template annotation so a config change is a template change and rolls automatically, per the Helm guide. Decide deliberately which behaviour each config deserves; "sometimes it reloads, sometimes not" is the worst of both.

Secrets in Git, solved properly: external stores and sync

Since plaintext Secrets must not live in Git, but workloads need Secrets in the cluster, the industry converged on one architecture: the vault is the source of truth; a controller syncs values into the cluster as Secrets.

The common implementation is the External Secrets Operator (ESO): you commit an ExternalSecret manifest that contains references — no values — and ESO fetches the real values from AWS Secrets Manager, GCP Secret Manager, Azure Key Vault or HashiCorp Vault using the cluster's own cloud identity:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
spec:
  refreshInterval: 1h
  secretStoreRef: { name: aws-secrets, kind: ClusterSecretStore }
  target: { name: db-credentials }
  data:
    - secretKey: password
      remoteRef: { key: prod/app/db, property: password }

What Git shows anyone: that a password exists and where it lives. What it never shows: the password. Rotation happens in the vault; ESO refreshes the Secret; pods pick it up on their next restart (or sooner, for the file-watchers). Alternatives in the same family: the cloud providers' own CSI drivers (mounting secrets as in-memory files, never persisted as API objects at all — arguably the cleanest model) and Sealed Secrets, which encrypts values with a cluster key so the ciphertext is safe to commit — simpler than a vault, with key management becoming your job instead. For a small team with no vault yet, Sealed Secrets is a legitimate step; for anything multi-cluster or compliance-flavoured, the external-store pattern wins.

Two hygiene rules whichever route you choose:

Different secrets per environment. The staging credential and the production credential being identical is how a staging compromise becomes a production incident.
Audit access. Secret reads should be logged (CloudTrail on the vault side, API audit logs in-cluster) — per the observability guide, the time to learn who read a secret is not during the postmortem.

Rotation: the event you design before it designs you

Credentials rotate — on a schedule if you're organised, during an incident if you're not. A rotation that requires editing manifests in three repos and restarting everything by hand is not a rotation procedure; it's a hope. The designed version:

Vault holds the new value alongside the old (most stores support versions; databases can hold two valid users during a transition window).
ESO syncs; a controlled rollout restarts pods against the new value (kubectl rollout restart, staged by deployment strategy so a bad credential fails a canary, not the fleet).
Old value revoked after the new one is verified serving traffic.

Notice what this procedure never involves: a human seeing the value, a value in a chat message, or a rebuild. If any of those appear in your rotation story, the design has a hole in it.

Common mistakes

Plaintext Secrets in Git. The classic. History keeps them forever, every clone keeps them everywhere. Vault + sync, or sealed ciphertext — those are the options.
Believing base64 is encryption. It is encoding, readable by anyone in milliseconds. RBAC and etcd encryption are the actual protections.
One giant shared Secret for everything. Every pod that mounts it can read all of it. Split by consumer: a pod should hold the credentials for its own job and nothing else.
Expecting ConfigMap edits to reload apps. Env vars never update; mounted files update but apps rarely re-read. Roll the pods, or wire the checksum annotation, deliberately.
The same credentials in every environment — and in the load-test cluster, and in that old namespace nobody deleted. Scope per environment, per workload.
Config in the image, secrets in the wiki. Config baked into images forces rebuilds for tuning; secrets documented in shared notes are secrets in name only. Both are the same mistake: the value lives where it isn't managed.

FAQ

ConfigMap or Secret for a database URL?
The URL itself (host, port, database name) is configuration — ConfigMap. The username and password are Secrets. Splitting them is normal and makes rotation cleaner: credentials change without touching the endpoint config.

How big can a ConfigMap be?
Around 1 MiB per object (etcd object limits). If your config approaches that, it isn't config anymore — it's data, and it belongs in object storage or a database, mounted or fetched by the app.

Should pods read config from the API directly instead?
Almost never. It couples the app to Kubernetes APIs and permissions it shouldn't hold. Env vars and files are the interface; keep the app portable and the cluster's API unexposed to workloads.

What's the difference between Sealed Secrets and External Secrets?
Sealed Secrets stores the encrypted value in Git and decrypts it in-cluster — the repo remains the distribution mechanism. External Secrets stores only a reference in Git and fetches the value from a vault at sync time. The vault model scales to rotation, audit and multi-cluster; the sealed model is simpler to adopt. Both beat plaintext.

Conclusion

Configuration is a design surface, not a chore: ConfigMaps for the values you'd print on a whiteboard, Secrets for the ones you wouldn't, RBAC and etcd encryption as the real protections, a vault with sync as the source of truth, and rotation as a rehearsed procedure. Get those five decisions right and config changes become the least interesting part of your week — which, for anything involving production credentials, is exactly the ambition.

The free Kubernetes Production Checklist (via the newsletter on the homepage) includes the configuration and secrets pre-flight items — RBAC scoping, backup tests and rotation readiness — alongside the rest of the production readiness checks.

Kubernetes Upgrades in 2026: Minor Versions Without the Downtime or the Drama

softverseAdmin — Sat, 31 Oct 2026 07:57:27 +0000

TL;DR: Kubernetes releases three minor versions a year, and each is supported for roughly a year — fall two versions behind and upgrading stops being routine maintenance and becomes a project. Safe upgrades follow a fixed order: check for removed APIs before you touch anything, upgrade the control plane first (one minor version at a time, never skip), then rotate nodes with surge capacity so workloads always have somewhere to land, respecting PodDisruptionBudgets the whole way. The two facts that shape everything: you cannot skip minor versions on the control plane, and there is no downgrade button — your rollback plan is an etcd backup and node images you can still boot. Do the preparation and an upgrade is a boring afternoon. Skip it, and the boring afternoon finds you anyway, at a worse time.

Originally published on DevOps Enginer.

Every cluster is quietly ageing. The Kubernetes release train does not stop because a team is busy, and the gap between "we should upgrade soon" and "our version just went end-of-life" is where upgrade projects turn into incidents. This guide is the upgrade as a routine: the checks, the order, and the traps — written for the person who wants upgrades to be boring.

The rules of the road

Four constraints govern every upgrade, and planning means accepting them rather than negotiating:

One minor version at a time for the control plane. Going 1.30 → 1.32 directly is not supported for kube-apiserver. Each step gets applied, verified, and only then does the next begin. (Managed control planes enforce this for you; the steps still happen.)
Version skew is a policy, not a vibe. The supported combinations are specific: kubelets may be up to three minor versions behind the API server (since 1.28), but never newer than it; kubectl works within one minor version either way. "It connected" is not the same as "it's supported."
APIs get removed. Every minor release can retire beta APIs. Workloads still referencing a removed API fail at apply time — or worse, at the moment you need to redeploy during an unrelated incident. This is what the preparation phase exists to catch.
There is no downgrade. The API server and etcd do not roll back. If an upgrade goes wrong mid-flight, recovery means restoring etcd from backup or rebuilding — which is why backups are step zero, not an afterthought.

Phase 1: Preparation (where upgrades are actually won)

Inventory the target. Read the target version's changelog and the release's "urgent upgrade notes," and check your add-ons against it: ingress controller, CNI, CSI drivers, cert-manager, cluster-autoscaler/Karpenter, metrics-server. Add-on incompatibility breaks more upgrades than Kubernetes itself does — an ingress controller that doesn't support the new version will not announce this until traffic stops.

Hunt for removed APIs. Two tools make this mechanical:

# Scans manifests / Helm releases for deprecated and removed APIs
pluto detect-helm -o wide
pluto detect-files -d ./manifests -o wide

# Scans the live cluster for APIs in use that are scheduled for removal
kubent

Fix every finding by moving to the replacement API (the tools name it) before the upgrade window. While you are in there, check the same hygiene for admission webhooks: a dead webhook with failurePolicy: Fail can block the whole cluster's writes after a control-plane change, so inventory webhooks and know which ones are load-bearing.

Back up etcd — and know how you'd restore it. On self-managed clusters, snapshot etcd and copy the snapshot off the node. On managed clusters (EKS, GKE, AKS), the provider handles the control plane, but your workload state is your problem: if everything is in Git (see my Argo CD tutorial), the cluster is rebuildable; if it isn't, a volume-level backup tool is your safety net, per the storage guide.

Check cluster health honestly. Upgrading a cluster that already has failing nodes, Pending pods, or a flapping control plane converts one problem into two. kubectl get nodes, component statuses, and a look at recent events: fix first, upgrade second.

Phase 2: Control plane first

With preparation done, the control plane goes first — always. On managed services this is one action per minor version (console, CLI or Terraform), followed by a pause: watch API server availability, controller-manager and scheduler health, and confirm nodes still report Ready before proceeding.

On self-managed clusters with kubeadm, the shape is:

# On the first control-plane node, per minor version:
apt-mark unhold kubeadm && apt-get install -y kubeadm=1.31.x-*   # target version
kubeadm upgrade plan
kubeadm upgrade apply v1.31.x
# Then upgrade kubelet + kubectl on the node and restart kubelet

kubeadm upgrade plan is the unsung hero here: it reports component versions, warns about what it can and cannot do, and refuses some genuinely bad ideas. Read its output like a pre-flight checklist, because that is what it is. Repeat for the remaining control-plane nodes with kubeadm upgrade node.

Phase 3: Nodes — rotate, don't patch in place

Worker nodes upgrade by replacement wherever possible: bring up a node at the new version, drain an old one, let workloads reschedule, remove the old node. On managed node groups this is the built-in rolling update, with one setting that decides whether it is graceful:

Max surge: allow the group to temporarily run extra nodes (e.g. +1 or +33%) during the update. Surge means a drained node's workloads already have a home; without surge, the update cordons a node and hopes capacity suffices — on a tightly packed cluster, it doesn't, and pods sit Pending mid-upgrade. This is the same capacity logic as autoscaling: replacement needs somewhere to land.
PodDisruptionBudgets are the contract that makes draining safe: drain respects voluntary-disruption budgets, so a workload with minAvailable set cannot be fully evicted by an upgrade. Workloads without PDBs can be taken to zero by a drain — if that outage would hurt, the PDB was the missing pre-work (Phase 1 again).

The manual shape, when you are rotating nodes yourself:

kubectl cordon <old-node>
kubectl drain <old-node> --ignore-daemonsets --delete-emptydir-data
# wait: confirm every workload is Running elsewhere
kubectl delete node <old-node>   # after the node itself is terminated/replaced

Watch for the two classic drain blockers: pods with local storage (--delete-emptydir-data acknowledges losing it — be sure you mean that) and pods whose PDB forbids the eviction, in which case the drain waits, correctly, and you investigate the workload rather than forcing it. --force exists; reaching for it during an upgrade is how upgrades become incidents.

On self-managed nodes where replacement isn't practical, kubelet upgrades in place: install the new kubelet (and container runtime, checking its own compatibility), restart, verify the node rejoins at the new version within skew policy.

Phase 4: Verify like you don't trust it (because you shouldn't)

"Upgrade completed" from a console is the beginning of verification, not the end:

kubectl version and kubectl get nodes — every node at the intended version, all Ready.
Core add-ons healthy: DNS resolution from a test pod, ingress serving, metrics flowing (kubectl top nodes) — the observability guide lists the signals worth watching in the hours after.
One real deployment: roll a canary workload, confirm scheduling, networking and storage attach all behave.
Add-on upgrades you deferred: now is when ingress, cert-manager and friends get their matching bumps, one at a time, while the window is still staffed.

Then leave the cluster alone for a soak period before starting the next minor version. One version, one verification, one soak — the rhythm that keeps a three-version catch-up from becoming a single three-version mystery.

Managed vs self-managed: what actually differs

The order and the rules are identical; only the hands change. Managed services move the control plane for you and offer node-group rotation with surge built in — your job concentrates on Phase 1 (API removals, add-on compatibility) and Phase 4 (verification), which is honestly where the risk always lived. Self-managed adds the kubeadm mechanics and etcd backup duty. In both worlds, the teams that get hurt are hurt by preparation gaps, not by the upgrade command itself.

Common mistakes

Skipping the removed-API scan. Everything works until the first redeploy against a removed API — during a different incident, naturally.
Trying to jump two minor versions. The control plane refuses or, worse, half-applies. One step at a time is not a suggestion.
Zero surge on node rotation. Drained workloads with nowhere to go; Pending pods; an upgrade window that ends in a rollback of the schedule rather than the version.
No PDBs on important workloads. Drain takes them to zero, voluntarily, as designed — the design just never got told they mattered.
Upgrading an unhealthy cluster. Pre-existing node or control-plane faults become upgrade casualties and misdirect the whole investigation.
Forgetting the add-ons. Kubernetes upgrades fine; the two-year-old ingress controller doesn't. Inventory includes everything with a version number, not just Kubernetes.

FAQ

How often should we upgrade?
Often enough to never be more than one minor version behind the oldest supported release — in practice, most teams settle into upgrading every 2–4 months, which keeps each event small. The cost of upgrading grows with the distance jumped; frequency is the discount.

Can we skip a version on managed Kubernetes?
No — the provider performs the same one-minor-at-a-time steps; the console just makes them look like one action. Node groups likewise step through versions.

What if something breaks mid-upgrade?
For nodes: stop the rotation, keep workloads on healthy nodes, fix forward or replace the bad node. For the control plane: there is no downgrade, which is why the pre-upgrade etcd snapshot (self-managed) or your Git-backed workload state (everywhere) is the real rollback plan.

Do workloads need downtime?
Properly built ones don't: multiple replicas, PDBs, surge capacity and rolling drains mean pods move, not stop. Single-replica workloads without PDBs will have a gap during their node's drain — the upgrade is merely revealing a property the workload always had. The deployment strategies guide covers building workloads that survive exactly this.

Conclusion

A Kubernetes upgrade is a checklist wearing a scary costume: scan for removed APIs, back up, control plane one minor at a time, nodes rotated with surge and PDBs respected, then verify like a sceptic. Run it a few times a year and it shrinks to what it should be — routine maintenance with a beginning, a middle, and an end. Let it slide for a year, and the same work returns wearing its real costume: a project, with a deadline set by an end-of-life date.

The free Kubernetes Production Checklist (via the newsletter on the homepage) includes the upgrade pre-flight items — health checks, backup verification and disruption budgets — alongside the rest of the production readiness checks.

Kubernetes Storage in 2026: PersistentVolumes, StorageClasses and StatefulSets Without the Fear

softverseAdmin — Sat, 31 Oct 2026 07:39:02 +0000

TL;DR: Kubernetes storage confuses people because it splits one idea — "my pod needs a disk" — into three objects. A PersistentVolumeClaim (PVC) is the pod's request: "I need 10Gi, writable by one node." A PersistentVolume (PV) is the actual disk that satisfies it. A StorageClass is the recipe that creates those disks on demand, with a chosen type, reclaim behaviour and expansion policy. Stateless apps should stay on Deployments with no volumes at all. Apps that genuinely own data — databases, queues, anything whose identity is its disk — belong on StatefulSets, which give each replica a stable name, a stable network identity and its own claim that survives rescheduling. Most storage incidents come from three places: the wrong reclaim policy deleting data, a volume bound to one availability zone while the pod reschedules into another, and nobody testing a restore until the day it matters.

Originally published on DevOps Enginer.

Pods are cattle; disks are pets. Everything awkward about Kubernetes storage lives in that sentence. This guide walks the moving parts in the order a request flows through them, then covers StatefulSets, expansion, snapshots, and the failure modes that actually bite.

The three objects, in the order they meet

When a pod declares it needs storage, this is the chain:

PersistentVolumeClaim — the request, written by you (or generated by a StatefulSet). It asks for a size, an access mode, and optionally a StorageClass.
StorageClass — the template. It names the provisioner (your cloud's disk service, a CSI driver, a local provisioner), the parameters (disk type, IOPS tier), the reclaim policy, and whether volumes can be expanded later.
PersistentVolume — the reality. With dynamic provisioning (the normal case on any cloud cluster), the StorageClass's provisioner creates a real disk and a PV object representing it, bound one-to-one to your claim.

You almost never create PVs by hand anymore. You write PVCs, you choose (or set a default) StorageClass, and provisioning does the rest. If a claim sits in Pending, the conversation between those three objects broke somewhere — usually the class doesn't exist, the provisioner can't create disks in that zone, or quota ran out. kubectl describe pvc <name> prints the events that tell you which.

Access modes: the line that decides your architecture

Every claim declares how the volume may be mounted, and this choice constrains everything downstream:

ReadWriteOnce (RWO) — one node at a time. The default for cloud block disks (EBS, Azure Disk, GCE PD). Fine for a database with one writer; it also means a rolling update can deadlock if the new pod lands on a different node while the old pod still holds the volume — the classic multi-attach error.
ReadWriteMany (RWX) — many nodes at once. Requires a shared filesystem (NFS, EFS, Azure Files, Filestore). Needed when several pods across nodes must see the same files; you pay for it in performance characteristics and complexity.
ReadWriteOncePod (RWOP) — one pod, not just one node (Kubernetes 1.29+ for CSI volumes). The stricter guarantee single-writer workloads actually wanted.

Read the mode before you design around a volume. Half of all "Kubernetes storage is broken" stories are a workload assuming RWX semantics on an RWO disk.

StorageClasses: the decisions hiding in the defaults

One cluster usually offers several classes — a fast SSD class, a balanced class, maybe an HDD class. The fields that matter most are the quiet ones:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-retain
provisioner: ebs.csi.aws.com        # example: AWS EBS CSI driver
parameters:
  type: gp3
reclaimPolicy: Retain               # not Delete — see below
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

reclaimPolicy decides what happens to the disk when the PVC is deleted. Delete destroys the underlying volume — convenient for throwaway environments, catastrophic for a production database whose claim gets deleted by a namespace cleanup or a careless Helm uninstall. For anything whose data matters: Retain. Retain leaves the PV in a Released state; the data survives until a human deliberately deals with it.
volumeBindingMode: WaitForFirstConsumer delays provisioning until a pod actually schedules. Without it, a disk can be created in zone A before the scheduler places the pod in zone B — and a zonal disk can never attach across zones, so the pod is stuck forever. This single field prevents one of the most common storage dead-ends on multi-zone clusters.
allowVolumeExpansion: true is only settable at class creation in effect — you cannot retroactively make a non-expandable class expandable for existing volumes in most setups. Set it now, thank yourself later.

Check what your cluster defaults to before writing a single claim:

kubectl get storageclass
kubectl describe pvc <claim> -n <namespace>
kubectl get pv

StatefulSets: when the pod's identity IS the disk

Deployments treat replicas as interchangeable — perfect for stateless services, wrong for a database cluster where db-0, db-1 and db-2 are different members with different data. StatefulSets exist for those workloads and give three guarantees Deployments don't:

Stable identity: pods are named name-0, name-1, … and keep that name across restarts, with a matching stable DNS entry per pod via a headless Service.
Per-replica storage: a volumeClaimTemplates block generates one PVC per replica (data-db-0, data-db-1, …). When a pod is deleted and recreated, it reattaches to its claim — the data follows the identity, not the node.
Ordered behaviour: replicas start and stop in order, which clustered systems that elect leaders and replicate in sequence genuinely need.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: db
spec:
  serviceName: db-headless
  replicas: 3
  selector:
    matchLabels: { app: db }
  template:
    metadata:
      labels: { app: db }
    spec:
      containers:
        - name: db
          image: postgres:16
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: fast-retain
        resources:
          requests:
            storage: 20Gi

Two operational truths about StatefulSet claims. First, scaling down does not delete the claims — scale from 3 back to 5 later and the old replicas reattach to their existing data. That is deliberate and usually what you want; it also means forgotten claims quietly keep disks (and bills) alive, which is why storage belongs in the cost optimization review. Second, deleting a StatefulSet leaves its PVCs behind too — cleanup is a separate, explicit act. Annoying on a tidy-up day; a lifesaver on a bad day.

And the honest question to ask first: does this workload need to be stateful in Kubernetes at all? A managed database service outside the cluster is often the right answer for small teams. Run state in-cluster when you have a real reason — latency, cost at scale, data locality — not by default. If you do run it, the deployment strategies guide covers how updates roll for these workloads, and it is not "like a Deployment."

Growing and protecting volumes

Expansion is the happy path, when it was prepared: with allowVolumeExpansion: true on the class, you edit the PVC's requested size upward and the provisioner grows the disk; most filesystems expand online once the pod restarts or remounts. Shrinking is not supported — a smaller disk means a new volume and a data migration, full stop.

Snapshots are the backup primitive: a VolumeSnapshot asks the CSI driver for a point-in-time copy, and a new PVC can be created from a snapshot to restore or clone. Combined with a schedule (via your backup tooling or a CronJob-driven process), this is how in-cluster state gets real backups. Two warnings from experience-shaped convention:

A snapshot is not a backup strategy until a restore has been tested. Untested restores fail in creative ways: wrong class, wrong zone, permissions, application-level inconsistency.
For databases, storage-level snapshots are crash-consistent at best unless the application quiesces writes. Where correctness matters, pair snapshots with the database's own backup tooling.

Security applies to disks too: volume contents inherit your RBAC and Pod Security posture — whoever can create a PVC from a snapshot can read the data in it, so snapshot permissions deserve the same least-privilege care as the volumes themselves.

Common mistakes

reclaimPolicy: Delete on data that matters. The claim gets deleted — by a person, a script, a namespace teardown — and the disk goes with it. Retain for production state, always.
No WaitForFirstConsumer on multi-zone clusters. Disks provisioned in the wrong zone; pods Pending forever with attach errors that look like scheduler problems.
Assuming RWX from an RWO disk. Multi-attach errors during rollouts, or worse, silent assumptions in the app design that only surface under failover.
Forgetting StatefulSet claims on cleanup. Orphaned PVs holding real disks for months — visible in the bill, invisible in kubectl get pods.
Snapshots nobody has restored from. A backup you have never tested is a hypothesis. Schedule a restore drill like you schedule the backup.
Running stateful workloads on Deployments with a shared claim. Losing stable identity and per-replica storage is how databases end up with two writers and one corrupted dataset.

FAQ

Should my web app have a PersistentVolume?
Almost never. If the app is stateless, uploads and user files belong in object storage (S3/GCS/Azure Blob) or a managed service, not a pod-local disk. PVCs are for workloads whose data is the workload — databases, brokers, search indexes.

What happens to my data if the pod is deleted?
With a PVC, nothing — the volume outlives the pod and reattaches to its replacement. With no volume (emptyDir or the container filesystem), everything is gone with the pod. That distinction is the entire subject of this article.

Can I move a volume to another cluster?
Not by copying YAML. The path is snapshot/restore (CSI snapshots, or backup tools built on them) or application-level replication/export. Plan migrations around the data tooling, not the manifests.

Why is my PVC stuck in Pending?
Read its events: kubectl describe pvc. The usual causes are a StorageClass that doesn't exist, a provisioner that can't create volumes in the needed zone, exhausted quota, or WaitForFirstConsumer waiting (correctly) for a pod that itself can't schedule — in which case the pod's events, covered in the troubleshooting guide, hold the real reason.

Conclusion

Kubernetes storage stops being scary when you keep the chain straight: claims ask, classes define, volumes provide — and StatefulSets bind identity to data for the workloads that need it. Choose the access mode honestly, set Retain and WaitForFirstConsumer before they save you, grow volumes instead of replacing them, and test restores before you need them. Disks are pets; treat them with the deliberate care pets require, and they will stop biting.

The free Kubernetes Production Checklist (via the newsletter on the homepage) includes the storage pre-flight items — reclaim policies, backup tests and disruption budgets — alongside the rest of the production readiness checks.

Kubernetes Security Basics in 2026: RBAC, Network Policies and Pod Security That Actually Get Used

softverseAdmin — Sat, 31 Oct 2026 07:13:33 +0000

TL;DR: A default Kubernetes cluster is far more open than most teams realise: any pod can talk to any other pod, many workloads run with more API permission than they need, and "the app works" often quietly means "the app runs as root with a token mounted it never uses." You do not need a security platform to fix the worst of it. Three controls close most of the real-world holes: RBAC scoped to what each identity actually does, NetworkPolicies that end the flat any-to-any network, and Pod Security Standards that stop privileged, root-running workloads at admission time. Add disciplined secrets handling and image hygiene, and a default cluster becomes a defensible one — with changes your team can roll out namespace by namespace, without a big-bang migration.

Originally published on DevOps Enginer.

Kubernetes security advice tends to arrive as a 200-item checklist, which is why so much of it never gets implemented. This guide takes the opposite approach: the small set of controls that remove the most risk per hour of effort, in the order I would apply them to a real cluster, with the manifests that matter.

Start from the uncomfortable defaults

Before the controls, be honest about what a fresh cluster permits:

The network is flat. With no NetworkPolicies, every pod can open a connection to every other pod and service in the cluster. One compromised frontend can reach your database directly.
Every pod gets a service account token mounted by default (unless you turn it off), whether or not the application ever calls the Kubernetes API.
Containers frequently run as root, with the default capabilities, on images pulled by a mutable tag like latest.
RBAC is often bypassed in practice — not by attackers first, but by teams handing out cluster-admin because a narrower role "took too long to debug."

None of this is exotic. Attackers do not need zero-days when the defaults hand them lateral movement and credentials. The good news: each default above has a specific, boring fix.

Control 1: RBAC — least privilege for humans and workloads

RBAC answers one question: who is allowed to do what to which resources? Two identities matter — the people (and CI systems) who operate the cluster, and the service accounts workloads run as.

For workloads: give each application its own ServiceAccount, and set automountServiceAccountToken: false unless the app genuinely calls the API. Most applications never do — the mounted token is pure attack surface: steal the pod, steal a credential.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: api
automountServiceAccountToken: false

When an app does need API access, bind a Role that names the exact verbs and resources — get and list on configmaps in one namespace is a role; * on * is an incident waiting for a date:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: config-reader
  namespace: production
rules:
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["get", "list", "watch"]

For humans and CI: scope roles per namespace, treat cluster-admin like root on a production database — a named, audited exception, not a team default — and audit what you have. Two commands tell you most of the story:

kubectl get clusterrolebindings -o wide
kubectl auth can-i --list --as=system:serviceaccount:production:api -n production

The second is the one to run for every workload identity: it prints exactly what that token can do, in plain text. If the list surprises you, that is the finding. This is the same least-privilege discipline as IAM on the cloud side — my AWS IAM least privilege guide works through the identical thinking for AWS policies.

Control 2: NetworkPolicies — end the flat network

A NetworkPolicy is a firewall rule for pods: it selects pods and declares which traffic may reach them (ingress) and where they may connect (egress). Policies are enforced by your CNI plugin — an important caveat: if your CNI does not enforce NetworkPolicy (the basic kubenet setup is the classic example), the objects exist and silently do nothing. Cilium, Calico and the major cloud CNIs enforce them; verify yours before trusting the YAML.

Roll out in this order, namespace by namespace:

Step 1 — default deny. One policy that selects all pods in the namespace and allows nothing changes the posture from "open unless closed" to "closed unless opened":

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: production
spec:
  podSelector: {}
  policyTypes: ["Ingress", "Egress"]

Step 2 — allow what the app actually needs. Then add narrow allowances: the frontend may reach the API on its port; the API may reach the database on 5432 and DNS on 53 (forgetting DNS egress is the classic self-inflicted outage — allow it explicitly):

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-ingress
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes: ["Ingress"]
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: frontend
      ports:
        - port: 8080

Do this in a staging namespace first, watch what breaks, and promote the same policies with the application — if you run GitOps, they belong in the same repository as the workloads, reviewed like any other change (my Argo CD tutorial covers that workflow). A compromised pod inside a default-deny namespace is a contained problem; in a flat network it is a cluster tour.

Control 3: Pod Security Standards — stop bad workloads at the door

RBAC limits identities and NetworkPolicies limit traffic, but neither stops a deployment that asks to run privileged, mount the host filesystem, or run as root. That is admission-time work, and Kubernetes has it built in: Pod Security Standards, enforced through Pod Security Admission with three profiles:

Privileged — effectively no restrictions. For system components that genuinely need host access, and nothing else.
Baseline — blocks the known privilege escalations (privileged containers, host namespaces, dangerous capabilities) while staying compatible with most typical workloads.
Restricted — the hardened target: no root, no privilege escalation, drop all capabilities, seccomp on. What user-facing workloads should aim for.

Enforcement is a namespace label — no webhook to install:

kubectl label namespace production \
  pod-security.kubernetes.io/enforce=baseline \
  pod-security.kubernetes.io/audit=restricted \
  pod-security.kubernetes.io/warn=restricted

That combination is the pragmatic rollout: enforce Baseline today (violating pods are rejected), while audit and warn on Restricted shows you — in API warnings and audit logs — exactly which workloads would fail the stricter profile, so you can fix them deliberately instead of breaking production on day one. At the workload level, the settings Restricted asks for are small and belong in every Deployment template:

securityContext:
  runAsNonRoot: true
  runAsUser: 10001
  allowPrivilegeEscalation: false
  capabilities:
    drop: ["ALL"]
  seccompProfile:
    type: RuntimeDefault

The honest blocker is usually the image: an app whose image only runs as root needs a Dockerfile fix, not an exception. Treat exceptions as named, expiring decisions — a namespace labelled back to Privileged "temporarily" has a way of becoming permanent.

Secrets: the rules that prevent the embarrassing incidents

Kubernetes Secrets are base64-encoded, not encrypted, and anyone with the right RBAC verb can read them — which loops straight back to Control 1. The working rules:

Encryption at rest on for etcd (a cluster-level setting; managed clusters differ in defaults — verify yours, don't assume).
Never bake secrets into images or commit them to Git. Use environment injection or mounted volumes from a secrets source, and scan repositories in CI so a committed credential fails the pipeline — that scanning habit is part of the wider pipeline discipline in my DevSecOps checklist.
Prefer an external secrets store (cloud secret managers, or an external-secrets operator syncing from one) as the source of truth, so rotation and audit live in one place instead of being scattered across namespaces.
Rotate like you mean it. A secret that has never been rotated should be assumed permanent — and anything permanent eventually leaks.

Image hygiene: know what you are actually running

The last basic: the artefact you deploy. Four habits cover most of it:

Pin by digest, or at least by specific version tag. latest means "whatever exists at pull time," which is neither reproducible nor auditable.
Scan in CI and watch the results. A scanner that reports into a void changes nothing; gate on fixable critical vulnerabilities for internet-facing workloads.
Use minimal bases. A small base image is fewer packages to patch and fewer surprises — the same discipline as my Docker image optimization guide, where smaller and more secure turned out to be the same work.
Pull from registries you control or trust, with image pull policies that do not silently fall back to stale local copies.

Common mistakes

NetworkPolicies on a CNI that doesn't enforce them. The YAML applies cleanly, everyone feels safer, nothing is filtered. Verify enforcement with a test connection that should fail.
cluster-admin as a convenience. Every standing cluster-admin binding is a standing invitation. Scope it, name it, review it.
Default-deny without DNS egress. The policies "work" and every lookup fails. Always allow UDP/TCP 53 to your DNS service explicitly.
Enforcing Restricted on day one. Rejected pods across production, followed by a panicked rollback to no standards at all. Baseline-enforce + Restricted-audit gets you to the same place without the outage.
Service account tokens mounted everywhere. One flag — automountServiceAccountToken: false — removes a credential from every pod that never needed it.
Secrets in Git "just for staging." Staging secrets have a habit of being production secrets. Scan, block, rotate if it already happened.

FAQ

Where do I start if the cluster has none of this?
In this order: inventory RBAC (clusterrolebindings and can-i --list for your workloads), turn off automatic token mounting where the API isn't used, then default-deny NetworkPolicies in one staging namespace, and Baseline Pod Security in audit mode cluster-wide. Each step is an afternoon, not a quarter.

Do NetworkPolicies replace a service mesh or an external firewall?
No — they cover pod-to-pod traffic inside the cluster at L3/L4. Egress to the internet, L7 rules and identity-based policy are other layers' jobs. NetworkPolicies are the floor, not the whole building.

Managed Kubernetes (EKS/GKE/AKS) — is this handled for me?
Partially. The provider secures the control plane and offers defaults, but RBAC bindings, NetworkPolicies, Pod Security labels and your images remain yours. The shared-responsibility line runs right through the middle of this article.

How do I prove it's working?
Test negatively: from a frontend pod, try to reach the database and watch it fail; deploy a privileged test pod in a Baseline namespace and watch it get rejected. A control you have never seen block anything is a hope, not a control. For the wider detection picture, see the Kubernetes observability guide.

Conclusion

Kubernetes security basics are not a product you buy; they are five defaults you deliberately change: scoped RBAC, no free tokens, a network that denies by default, admission that refuses privileged workloads, and secrets and images handled like they matter. Roll them out namespace by namespace, enforce gradually, and test that each control actually blocks something. A cluster secured that way will not make headlines — which is precisely the point.

The free Kubernetes Production Checklist (via the newsletter on the homepage) includes the security pre-flight items — RBAC review, token mounting, disruption budgets and rollback paths — alongside the rest of the production readiness checks.

Kubernetes Autoscaling in 2026: HPA, VPA, Cluster Autoscaling and KEDA — What Each One Actually Does

softverseAdmin — Sat, 31 Oct 2026 06:33:54 +0000

TL;DR: Kubernetes has four different autoscaling layers and most problems come from expecting one of them to do another one's job. The Horizontal Pod Autoscaler (HPA) changes how many pods you run. The Vertical Pod Autoscaler (VPA) changes how big each pod is. Cluster autoscaling (Cluster Autoscaler or Karpenter) changes how many nodes exist for pods to land on. KEDA scales on events — queue depth, Kafka lag, cron schedules — instead of CPU. In production you usually need at least two of these together: HPA for replicas plus node autoscaling underneath, or your new pods sit Pending forever. And none of them work without honest resource requests, because every percentage-based decision is calculated from the requests you set.

Originally published on DevOps Enginer.

Getting autoscaling right is the difference between a cluster that absorbs a traffic spike quietly and one that pages you at 3 a.m. while pods pile up unscheduled. This guide walks each layer in the order traffic actually flows through your cluster, with the configuration that matters and the mistakes that quietly break each one.

Why autoscaling fails quietly

Autoscaling rarely fails with an error. It fails silently, in three classic ways:

Pods scale, nodes don't. HPA adds replicas, but there is no room for them. They sit in Pending while the load that triggered them keeps growing.
Nodes scale, pods can't use them. The cluster adds capacity, but pod requests are set so high that each node fits only one or two pods — you pay for nodes that are mostly reserved air.
The metric lies. HPA targets 70% CPU, but the app is memory-bound, or requests were set years ago and no longer resemble reality. The scaler makes confident decisions from bad inputs.

Every layer below exists to close one of these gaps. If you haven't set requests and limits deliberately yet, start with my guide to Kubernetes resource requests and limits — autoscaling built on guessed requests just automates the guessing.

Layer 1: Horizontal Pod Autoscaler — more copies of the pod

HPA is the layer most teams already have. It watches a metric — by default CPU or memory utilisation relative to pod requests — and adjusts the replica count of a Deployment, StatefulSet or similar workload between a minimum and a maximum.

A production-shaped HPA looks like this:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 3
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 65
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Percent
          value: 100
          periodSeconds: 30
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Pods
          value: 1
          periodSeconds: 60

Three details in that manifest do most of the work:

minReplicas: 3, not 1. A single replica is a single point of failure and a slow cold start on the first spike. Three gives the scaler something to work with and survives a node loss.
The behavior block. Without it, scale-down is eager: load dips, replicas vanish, load returns, pods churn. A 300-second stabilisation window on scale-down and one pod per minute keeps the replica count calm instead of flapping. Scale-up stays aggressive — when you need capacity, you need it now.
The target is utilisation of requests. 65% of a 500m CPU request is a real number; 65% of a request nobody has reviewed since 2023 is not. This is why the requests guide matters more than the HPA manifest itself.

Prerequisites people forget: HPA needs metrics. Resource metrics come from the metrics-server add-on, which many clusters install by default but minimal or self-managed clusters may not have. If kubectl top pods returns nothing, HPA is flying blind and will do nothing — check that first before debugging the HPA object itself. Custom and external metrics (requests per second, queue length) need an adapter such as the Prometheus Adapter, which is where KEDA, below, often becomes the simpler path.

Layer 2: Vertical Pod Autoscaler — the right size for each pod

VPA attacks a different problem: the pod itself is the wrong size. It observes actual usage over time and recommends — or automatically applies — better requests.

VPA runs in three modes, and choosing the mode is the whole decision:

Off — it only shows recommendations (kubectl describe vpa becomes a right-sizing report). Safe everywhere, and honestly the mode most teams should start with for a few weeks.
Initial — requests are set when a pod is created and never changed afterwards. Low disruption, slow to adapt.
Auto — VPA evicts and recreates pods to apply new requests. Correct sizes, but your pods restart when VPA decides — never point Auto mode at something that cannot tolerate restarts.

The classic conflict: never let VPA in Auto mode and HPA fight over the same resource. If HPA scales on CPU utilisation while VPA keeps resizing the requests that utilisation is calculated from, the two controllers chase each other. The standard peace treaty: HPA on CPU (or on custom metrics), VPA in Off mode as an advisor — or VPA adjusting memory only while HPA watches CPU. My Kubernetes cost optimization guide covers why this right-sizing loop is where most of the money is: oversized requests are a silent tax on every node you run.

Layer 3: Node autoscaling — room for the pods to land on

HPA can decide it wants 20 replicas, but if the cluster has capacity for 8, the other 12 do not exist yet — they are Pending. Node autoscaling is the layer that notices unschedulable pods and adds machines, and later removes empty ones.

You have two mainstream choices:

Cluster Autoscaler is the older, conservative option. It watches for pending pods, asks the cloud provider for another node from an existing node group or instance pool, and scales groups down when nodes sit under-utilised. It is predictable and battle-tested, and its weakness is rigidity: it scales within node groups you pre-defined, so the instance types and sizes are your homework.

Karpenter (open source, originally built at AWS and now a CNCF project) takes the opposite approach: instead of scaling fixed groups, it watches pending pods, reads their actual requirements — CPU, memory, zones, labels, taints — and provisions a node shaped to fit them, often in under a minute. When nodes empty out, it consolidates: shuffles pods onto fewer, cheaper nodes and terminates the rest. The practical difference shows up exactly where Cluster Autoscaler is rigid — mixed workloads with very different shapes — and in consolidation, which steadily squeezes wasted capacity out of the cluster instead of only reacting to spikes.

Whichever you run, three settings decide whether it behaves:

PodDisruptionBudgets on important workloads. Node scale-down and consolidation drain nodes. A PDB is how a workload says "you may move me, but never take me below this many healthy replicas." Without PDBs, autoscaling will eventually evict something at a bad moment. (My deployment strategies guide covers PDBs alongside rolling updates.)
Do-not-evict annotations for the truly pinned pods — the singleton, the job mid-run. Both controllers respect signals that a pod must not be moved; use them sparingly and deliberately.
A floor under the cluster. Keep enough baseline capacity for your steady state. Autoscaling is for variance, not for running the cluster at zero slack and hoping provisioning is always fast.

Layer 4: KEDA — scaling on events, not CPU

Some workloads have no meaningful CPU signal. A worker that drains a queue is idle at 2% CPU with 10,000 messages waiting; a report generator should exist twice a day and never at night. Scaling these on CPU is scaling on a lie.

KEDA (Kubernetes Event-Driven Autoscaling, a CNCF graduated project) extends HPA rather than replacing it: it feeds external signals — SQS or Kafka backlog, RabbitMQ queue depth, database query results, cron schedules, Prometheus queries — into the scaling decision as metrics. Its two signature abilities:

Scale to zero. HPA's floor is 1 replica (well, minReplicas of 1 in practice). KEDA can take a workload to 0 when the queue is empty and bring it back when work arrives. For intermittent workers, this is the difference between paying for idle pods all month and paying for none.
Scale on the signal that predicts the work. Queue depth rises before CPU does. Scaling on the cause instead of the symptom removes most of the lag that makes CPU-based scaling feel sluggish.

A typical pattern: a ScaledObject declares the target deployment, the trigger (say, messages visible in a queue), and a threshold (one pod per 50 messages, between 0 and 15 replicas). KEDA maintains the underlying HPA for you. If you already run Argo CD, ScaledObjects are just more YAML in Git — the same review-and-rollback story as everything else.

Putting the layers together

A sane default stack for a typical web service:

Layer	Tool	Decision it makes
Pods	HPA	How many replicas, from CPU/custom metrics
Pod size	VPA (Off mode)	Whether requests match reality — advisory
Nodes	Karpenter or Cluster Autoscaler	How many machines, from pending pods
Events	KEDA (where relevant)	Scaling workers/jobs on queues and schedules

And the observability to prove it works: watch Pending pod count, HPA's current-vs-desired replicas, and node utilisation after consolidation. If you cannot see autoscaling deciding, you are trusting it blindly — the metrics to wire up are covered in my Kubernetes observability guide.

Common mistakes

Autoscaling on CPU for a memory-bound app. The metric must be the resource the workload actually runs out of — or a custom metric closer to the real demand, like requests per second.
No metrics-server, no scaling. kubectl top returning nothing means HPA sees nothing. Verify the plumbing before tuning the policy.
HPA + VPA Auto on the same resource. Two controllers, one variable, endless churn. Pick who owns CPU.
maxReplicas as an afterthought. Too low and the spike wins; absurdly high and a runaway loop or a bad deploy can scale you into a very large cloud bill. The max is a budget decision, not a formality.
Scaling pods without scaling nodes. The Pending pile-up. Every HPA rollout should be paired with the question: where do the new pods land?
No PodDisruptionBudgets. Node consolidation and scale-down will eventually evict the wrong pod at the wrong time.
Scale-down as eager as scale-up. Flapping replicas churn connections, caches and readiness. Stabilisation windows are free stability.

FAQ

Do I need all four layers?
No. Most services need HPA plus node autoscaling. Add VPA in advisory mode when you suspect requests are wrong (they usually are), and KEDA when a workload's real demand signal is a queue, a schedule or another external event rather than CPU.

Karpenter or Cluster Autoscaler?
If you are starting fresh on AWS, Karpenter's just-in-time, right-shaped provisioning and consolidation usually win. Cluster Autoscaler remains a solid, predictable choice — especially outside AWS or where node groups are deliberately fixed for compliance reasons.

Can KEDA replace HPA?
KEDA uses HPA under the hood for most triggers. What it replaces is your hand-rolled metrics plumbing, and it adds scale-to-zero, which plain HPA does not give you.

Why are my pods Pending even though autoscaling is on?
Work through the layers: HPA may have scaled (check desired replicas), but if nodes are full and the node autoscaler cannot add capacity — quota limits, no matching node group, Karpenter constraints — pods wait. kubectl describe pod on a pending pod names the failing condition; my Kubernetes troubleshooting guide walks that diagnosis step by step.

Conclusion

Autoscaling is not one feature you turn on; it is four decisions stacked on top of each other — how many pods, how big each pod is, how many nodes exist, and which signal you trust. Get the requests honest first, pair every pod scaler with node capacity underneath it, protect workloads with disruption budgets, and scale event-driven work on the events themselves. Do that, and traffic spikes become a graph you glance at — not an incident you get paged for.

Running autoscaling in production? The Kubernetes Production Checklist (free PDF via the newsletter on the homepage) covers the pre-flight checks — requests, probes, disruption budgets and rollback paths — that make scaling safe to automate.

Argo Rollouts 2026: Progressive Delivery for Kubernetes — Canary, Blue-Green and Rollbacks That Happen by Themselves

softverseAdmin — Sat, 31 Oct 2026 04:49:00 +0000

TL;DR: A plain Kubernetes Deployment can roll an update out, but it cannot send 5% of traffic to the new version, watch the error rate, and roll itself back when the numbers turn bad. Argo Rollouts can. It replaces your Deployment with a Rollout resource that adds weighted canary steps, blue-green switching, metric-driven analysis, and automatic rollback — and it slots neatly next to Argo CD if you already run GitOps. This guide installs it, ships a first canary, wires metrics into the promotion decision, and covers the mistakes that bite.

Originally published on DevOps Enginer.

My Kubernetes deployment strategies guide explains the patterns: rolling, blue-green, canary, recreate. Patterns are the easy half. The hard half is execution — who watches the new version while it serves its first real traffic, and who has the nerve (and the speed) to pull it back at 2 a.m. when the error graph twitches?

A stock Deployment answers "nobody, really." It replaces pods on a schedule of percentages of replicas, not traffic. It has no idea whether the new version is healthy in any sense that matters to users. Rollback is a human noticing, deciding, and typing.

Argo Rollouts is the controller that closes that gap. And if you already run Argo CD — I do, and my Argo CD tutorial walks through it — Rollouts is from the same Argo family and feels like the missing half: Argo CD gets the right version to the cluster; Rollouts decides how that version earns the rest of the traffic.

Why a plain Deployment is not enough

Three limits show up the first time a release goes wrong:

Replica math is not traffic math. "25% of pods are new" does not mean 25% of users see the new version — behind a Service, it is roughly proportional, but you cannot set 1%, or 5%, or hold a step deliberately.
There is no verify step. A Deployment promotes on pod readiness. A pod can be Ready and still return 500s, leak memory, or take 4 seconds per request. Readiness is not health.
Rollback is manual and slow. kubectl rollout undo works — after a human has noticed, paged, decided, and typed. The blast radius grows the whole time.

Progressive delivery is the discipline of fixing exactly these three: small traffic steps, automated verification between steps, and rollback that does not wait for a human. That is what Rollouts implements.

What Argo Rollouts actually is

Three pieces:

The Rollouts controller — a deployment in your cluster that watches Rollout resources and drives ReplicaSets, Services, and (optionally) your ingress or mesh.
The Rollout resource — a near drop-in replacement for a Deployment. Same pod template, same selectors; the difference is the strategy block, which describes how versions rotate.
The kubectl plugin — kubectl argo rollouts … for watching, promoting, pausing, and aborting rollouts from the terminal (and a web dashboard if you want one).

One honest nuance up front: precise traffic percentages need a traffic router. With a plain Service, canary steps scale ReplicaSets proportionally, so weights are approximate (with 10 replicas, your steps are effectively 10% chunks). Integrate Rollouts with NGINX Ingress, ALB, Traefik, Istio, or another supported router and the weights become real traffic weights, independent of replica count. Replica-based canary is fine to learn on; traffic-based is what you want for user-facing services.

Install Rollouts and the kubectl plugin

The controller:

kubectl create namespace argo-rollouts
kubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml

# watch it come up
kubectl -n argo-rollouts get pods

The plugin (Linux example — grab the matching binary for your OS from the same releases page):

curl -LO https://github.com/argoproj/argo-rollouts/releases/latest/download/kubectl-argo-rollouts-linux-amd64
chmod +x ./kubectl-argo-rollouts-linux-amd64
sudo mv ./kubectl-argo-rollouts-linux-amd64 /usr/local/bin/kubectl-argo-rollouts

kubectl argo rollouts version

Using latest in a learning cluster is fine. In production, pin the release version in the URL and manage the install through your GitOps repo like everything else — a controller that controls your releases should itself be released carefully.

Your first canary: a Rollout with steps

Take an ordinary Deployment, change kind: Deployment to kind: Rollout (and the apiVersion to argoproj.io/v1alpha1), and add a strategy. This example rolls a new version through 20% → pause for a human look → 50% → indefinite pause → 80% → full:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: shop-api
  namespace: production
spec:
  replicas: 10
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      app: shop-api
  template:
    metadata:
      labels:
        app: shop-api
    spec:
      containers:
        - name: shop-api
          image: ghcr.io/example/shop-api:2.4.0
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
  strategy:
    canary:
      steps:
        - setWeight: 20
        - pause: { duration: 5m }
        - setWeight: 50
        - pause: {}          # waits for a human promote
        - setWeight: 80
        - pause: { duration: 10m }

Apply it, trigger an update by changing the image tag, and watch:

kubectl argo rollouts get rollout shop-api -n production --watch

You will see the step list with the current step highlighted — this little tree view is, honestly, half the reason people adopt Rollouts. The state of a release stops being something you reconstruct from three commands and becomes something you can see.

Promote past a manual pause, or bail out:

kubectl argo rollouts promote rollout shop-api -n production
kubectl argo rollouts abort rollout shop-api -n production   # back to stable, now
kubectl argo rollouts undo rollout shop-api -n production     # roll back a finished one

Let metrics decide: analysis runs

Manual pauses are a fine start, but they still depend on a human watching a dashboard. The feature that turns Rollouts from "nicer rolling update" into progressive delivery is the AnalysisRun: during a canary step, Rollouts queries your metrics provider on an interval; if the numbers breach your conditions, it aborts and rolls back automatically.

A minimal success-rate check against Prometheus:

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
  namespace: production
spec:
  args:
    - name: service-name
  metrics:
    - name: success-rate
      interval: 30s
      count: 10
      successCondition: result[0] >= 0.95
      failureLimit: 2
      provider:
        prometheus:
          address: http://prometheus.monitoring.svc:9090
          query: |
            sum(rate(http_requests_total{service="{{ args.service-name }}",status!~"5.."}[5m]))
            /
            sum(rate(http_requests_total{service="{{ args.service-name }}"}[5m]))

Reference it from the Rollout's canary strategy, and the analysis starts automatically at the step you choose:

  strategy:
    canary:
      analysis:
        templates:
          - templateName: success-rate
        args:
          - name: service-name
            value: shop-api
        startingStep: 2
      steps:
        - setWeight: 20
        - pause: { duration: 5m }
        - setWeight: 50
        - pause: { duration: 10m }

Now the 2 a.m. scenario changes shape: the new version starts failing, the success rate dips under 95% twice, and Rollouts aborts and restores the stable ReplicaSet before anyone's phone buzzes. The human reviews a finished, contained event in the morning instead of performing an emergency.

Two disciplines make or break this:

Analyse what users feel, not what machines feel. Success rate and latency percentiles of real requests beat CPU graphs. (This is the same symptom-over-cause rule from my Kubernetes observability guide — analysis is only as good as the signals you feed it.)
Your metrics must already exist and be trustworthy. If the service emits no usable request metrics, step one is observability, not Rollouts. An automated rollback wired to a meaningless query is worse than a manual one — it fails with confidence.

Blue-green with Rollouts

Prefer an all-or-nothing switch with an instant way back? Blue-green is a strategy block away:

  strategy:
    blueGreen:
      activeService: shop-api-active
      previewService: shop-api-preview
      autoPromotionEnabled: false
      scaleDownDelaySeconds: 300

The new version comes up behind the preview Service, where you can smoke-test it against real infrastructure with zero user traffic. Promote it, and the active Service flips to the new ReplicaSet; the old one stays warm for scaleDownDelaySeconds — your undo window. With autoPromotionEnabled: false, the flip waits for kubectl argo rollouts promote. Set an analysis as a prePromotionAnalysis and even blue-green gets a metric gate before the switch.

My rule of thumb, same as in the strategies guide: canary for stateless user-facing services where gradual is possible; blue-green where a version must switch atomically (or where testing the exact production-bound version beforehand matters more than gradual exposure).

Rollouts + Argo CD: who owns what

If you run GitOps, the division of labour is clean:

Argo CD syncs the desired state from Git — including the Rollout resource and its new image tag. A release is a commit, with the audit trail Git gives you for free.
Rollouts owns the progression of that version inside the cluster: steps, pauses, analysis, promotion, abort.

Argo CD understands Rollouts natively — it reports the Rollout's health (Progressing / Healthy / Degraded) instead of the binary in-sync view you'd get with a Deployment mid-rollout, so a canary sitting at 20% doesn't show up as a permanently "progressing" mystery.

One trap: do not let two systems fight. Your CI pipeline should update the image tag in Git (or the Rollout manifest) and then stop — no kubectl promote step in CI for releases you want gated, and no second controller trying to manage the same ReplicaSets. One owner per decision: Git for what version, Rollouts for how fast it earns traffic, humans for the pauses you deliberately kept.

Mistakes that bite

Canary by replicas, expectations by traffic. With few replicas and no traffic router, a "20%" step can be one pod behind a Service — noisy and not really 20% of anything. If precise percentages matter to the release decision, wire up a traffic router first.
Analysis wired to vanity metrics. An AnalysisRun checking cluster CPU will happily promote a version that is erroring on every tenth request. Gate on per-service success rate and latency, measured on the canary itself.
No baseline. Analysis compares the canary against an absolute condition (like the 95% above). If normal for this service is 96% on a good day, a 95% gate aborts healthy releases and trains everyone to ignore aborts. Tune the gate to the service's real baseline — my requests and limits guide makes the same point about numbers copied from other people's clusters.
Promoting during an incident. If the platform is already on fire, a promote command is not a fix. Rollouts will even keep an in-flight canary paused politely while everything degrades around it. Abort first; release later.
Forgetting the HPA. Rollouts scales the stable and canary ReplicaSets to honour weights; an HPA targeting the Rollout works (point it at the Rollout, not the ReplicaSets), but an HPA someone left aimed at the old Deployment setup will confuse everyone. One more reason the migration should be a deliberate manifest change, not a weekend experiment.
revisionHistoryLimit left huge. Old ReplicaSets are your rollback targets — keep a few (2–3). Keep thirty and every get rollout becomes an archaeology dig.

When NOT to use Rollouts

Honestly: plenty of services should stay on plain Deployments.

Internal tools and low-traffic services where a bad release inconveniences three colleagues, not three thousand customers.
Batch and cron workloads — there is no traffic to graduate.
Anything whose metrics are too thin to analyse; a canary you cannot verify is just a slower rolling update.
Teams still fighting basic stability. Progressive delivery amplifies a release process; it does not repair one. Get deployments boring first (the troubleshooting guide is the place to start), then add graduation.

The measure I use, from the project-manager side of my desk: does a bad release of this service cost more than the machinery to catch it early? For the public, revenue-adjacent, or reputation-carrying services, Rollouts pays for itself the first time it aborts at step one. For everything else, it's ceremony.

FAQ

Is Argo Rollouts a replacement for Argo CD?
No — they are siblings, not competitors. Argo CD delivers desired state from Git to the cluster. Rollouts controls how a new version progresses once it's there. Many teams run both; Rollouts also works without Argo CD if your delivery is plain kubectl or another CD tool.

Can I use Rollouts without an ingress controller or service mesh?
Yes. You get replica-proportional canary steps through Services, which is approximate but functional — fine for learning and for low-stakes services. Precise, replica-independent traffic weights need a supported traffic router (NGINX, ALB, Traefik, Istio, and others).

What happens to my Deployment manifests?
You convert them: kind: Rollout replaces kind: Deployment, the pod template carries over unchanged, and you add the strategy block. Rollouts manages its own ReplicaSets from there. The conversion is small; the thinking about steps and gates is the real work.

Does an aborted canary count as downtime?
No — aborting shifts traffic back to the stable ReplicaSet, which never went away. Users saw the canary's behaviour only for the share of traffic it held, which is precisely the blast-radius reduction you adopted this for.

Can analysis roll back a blue-green release too?
Yes — pre-promotion analysis gates the switch, and post-promotion analysis can trigger a rollback to the previous ReplicaSet while it's still within its scale-down window.

Conclusion

A Deployment rolls software out. Argo Rollouts rolls software out on probation: a small share of traffic, real metrics watching, automatic retreat when the numbers say so. Install the controller, convert one service, start with replica-based canary steps and a manual pause — then wire in analysis and graduate to "the release promotes itself, and pages nobody."

Ship the versions. Let the evidence promote them.

Keep going: Deployment Strategies · Argo CD Tutorial · Kubernetes Observability · Kubernetes Production Checklist (free PDF)

Kubernetes Observability 2026: Metrics, Logs and Traces Without the Noise

softverseAdmin — Sat, 31 Oct 2026 04:22:09 +0000

TL;DR: Kubernetes observability is three signals with three different jobs — metrics tell you that something is wrong, logs tell you what happened, traces tell you where it went — plus a fourth everyone forgets: events, the cluster's own diary. The craft is not collecting more; it's discipline: golden signals per service, structured logs shipped off the pod, traces sampled honestly, strict label cardinality, and alerts that fire on symptoms users feel. Wire those in and incidents get shorter instead of dashboards getting longer.

Originally published on DevOps Enginer.

The 2 a.m. questions

Every incident, whatever the technology, comes down to two questions: what changed? and what is it doing now? I learned the audit-trail half of that lesson teaching myself AWS on my own account — my habit was to open CloudTrail and read what had actually been created and changed, instead of guessing from memory. Clusters deserve the same respect: the systems that answer those two questions in minutes are called "observable," and the ones that answer in hours are called "down."

Kubernetes makes this both easier and harder than the VM world. Easier, because everything is an API object with a state and a history. Harder, because the things you used to SSH into — containers, pods — are ephemeral by design: the pod that errored at 02:00 has been rescheduled twice by 02:15, and its logs went with it. Observability in Kubernetes is therefore not a monitoring add-on; it is the deliberate act of exporting state out of things that are designed to disappear.

This guide is the working model: what to collect, what each signal is actually for, and the disciplines that keep the signal bill — and the alert noise — from eating the team.

The three signals, and the jobs they're for

Metrics — numbers over time (request rate, error rate, latency percentiles, CPU saturation). Cheap to store, fast to query, perfect for alerting and dashboards. Useless for "why" — a metric tells you the error rate tripled, never which error.
Logs — timestamped event records. The detail layer: stack traces, request IDs, the specific "what happened." Expensive at volume, worthless unstructured — a log you can't query during an incident is a diary nobody reads.
Traces — the path of one request across services, as timed spans. The only signal that answers "which hop in this chain is slow" without an afternoon of log archaeology. Costs you instrumentation effort, repaid on the first cross-service incident.

And the unofficial fourth: events — the cluster's own record of decisions (scheduled, pulled, started, OOMKilled, failed probes). kubectl get events is the first stop in the troubleshooting workflow for a reason. Know its limit: events are retained for roughly an hour by default, so they are a live-incident tool, not an archive — anything you need historically must be exported.

Metrics: golden signals, ruthless cardinality

The Prometheus model won this layer: applications and exporters expose metrics, Prometheus scrapes and stores them, Grafana draws them. Managed equivalents exist on every cloud; the concepts transfer. Two disciplines make or break it:

Per-service golden signals. For every service users touch, four numbers on one row: latency (with percentiles, not averages — the average hides the suffering), traffic (requests per second), errors (rate and ratio), saturation (how full the service is: CPU vs. request, memory headroom, queue depth). This RED/USE-style framing is what turns 400 dashboards into one glance: a service is healthy or one of its four numbers is red. For the cluster itself, the same idea one level down: node saturation, pending pods, and requests-vs-usage — the last of which doubles as your cost signal (see the cost optimization guide).

Cardinality is the budget. Every unique label combination is a separate time series, and series are what you pay for — in memory, storage, and query speed. Labels are for dimensions with few values (service, namespace, status code class). They are never for user IDs, request IDs, full URLs, or error message text — those belong in logs and traces. The classic self-inflicted wound is a path label with unbounded values quietly multiplying a metric into a million series. Cardinality discipline is to metrics what requests discipline is to cost: invisible when done right, an incident when ignored.

Alert on the golden signals, recorded at sane resolution, and let recording rules pre-compute the expensive queries your dashboards and alerts run every minute.

Logs: structured, shipped, and correlated

Three rules convert logs from storage expense into incident fuel:

Structured or it didn't happen. JSON (or equivalent key-value) logs with level, service, and a correlation/request ID in every line. The request ID is the thread that ties a log line to a trace to a user complaint — propagate it at ingress and through every downstream call.
Off the pod, always. kubectl logs reads the current container on the current node; a rescheduled pod takes its history with it — the checklist version of this rule is blunt: pod logs die with the pod. Ship to a central store (Loki, Elasticsearch/OpenSearch, a cloud logging service) with retention tiers: hot and searchable for days, archived cheap for the compliance horizon.
Log decisions and failures, not chatter. A line per request at INFO is a billing strategy for your logging vendor. Log the boundaries (request received/completed at DEBUG or sampled), the anomalies, and the state changes — and make error logs carry the context a stranger needs at 2 a.m.: what was attempted, with what key inputs, and what the system decided.

Traces: OpenTelemetry and honest sampling

Distributed tracing's centre of gravity is now OpenTelemetry: one instrumentation standard, vendor-neutral, with auto-instrumentation for common frameworks and manual spans where the story needs them. The practical shape in a cluster:

Instrument the edges first — ingress to service, service to service, service to database. A trace that covers the hops covers the argument.
Propagate context properly (W3C traceparent headers) — a broken chain is worse than none, because it looks like data.
Sample honestly. Head-based sampling at a modest rate for normal traffic, tail-based sampling (keep the interesting ones) where your collector supports it, and one sacred rule: errors and slow requests are always kept. Tracing everything is a storage bill; tracing nothing is a hope; tracing the failures is the point.
Connect the signals: exemplars and trace IDs in your metrics and logs so an alert can jump to the metric, the metric to example traces, the trace to the exact log lines. Observability pays out at the joins.

Alerting: symptoms, budgets, and the noise floor

The fastest way to destroy an observability practice is alerting on causes ("CPU > 80%") until the team mutes the channel. The working rules:

Page on symptoms users feel: error ratio burning, latency SLO at risk, saturation heading for the cliff. Causes belong on dashboards and in tickets, where they can be investigated at human speed.
Every page carries a runbook link and answers "what do I do now?" An alert without a next action is a criticism, not a tool.
An alert budget is real. Track pages per week the way you track cost. When the budget is blown, the work is alert hygiene — deleting, demoting, or fixing the flapping few — before any new feature work. This is also the honest metric of your self-healing setup: the best alert is the one the system resolved before a human was involved, and said so quietly in the morning report.
Multi-window burn-rate alerts (fast burn pages, slow burn tickets) beat fixed thresholds for SLOs, because they measure the thing the business actually promised.

Dashboards: three that earn their tabs

Dashboard sprawl is cardinality's visual cousin. The set that survives contact with real operations is small:

Service overview — the four golden signals for one service, with deploy markers overlaid. The deploy overlay is the highest-value line on any dashboard: most incidents correlate with a change, and "what changed?" should be answerable by looking, not archaeology.
Cluster capacity — node saturation, pending pods, requests-vs-usage by namespace, and the top offenders list. Operations and cost share this view, which is as it should be.
The incident view — one screen the on-call actually opens at 2 a.m.: golden signals for the affected service, its dependencies' health, recent events, and links into logs and traces pre-filtered to the service. Build it during a calm week; you will not build it during an incident.

Everything else is a report, a debugging scratchpad, or nostalgia. Archive aggressively; a dashboard nobody opens in a quarter is a liability with a URL.

Observability as code

Dashboards and alerts defined by clicking are configuration drift with a login page. The GitOps answer applies here exactly as it does to workloads: dashboards as JSON in git, alert rules as PrometheusRule objects or files in the repo, recording rules reviewed in pull requests. The benefits compound: changes are reviewed (a threshold change is a policy change), history is blame-able in the good sense, and a rebuilt monitoring stack comes back identical — our Argo CD tutorial pattern works for the observability stack itself, which can live in the same app-of-apps layout as everything else you run.

What observability costs (and how it balloons)

Observability spend surprises teams the way cluster spend does: through accumulation, not decisions. The three leaks:

Metric cardinality — the unbounded label, priced per series (see above).
Log volume — the chatty service at INFO, priced per gigabyte, forever. Sample, tier, and give every team's its own volume number — showback works here exactly as it does for compute.
Trace retention — full-fidelity tracing retained for months, priced per span. Sample the boring, keep the broken, archive the rest cold.

Put the observability bill on the same monthly one-pager as the cluster bill. A monitoring stack that costs a visible fraction of the infrastructure it watches is normal; an unexamined one grows until finance asks the question in a tone nobody enjoys.

Mistakes that make observability useless

Collecting everything, querying nothing. Data without a dashboard, alert, or runbook attached is storage, not observability. Every signal should name the decision it feeds.
Averages for latency. The mean of a bimodal distribution describes nobody's experience. Percentiles (p50/p95/p99) or it didn't happen.
Unbounded labels. One user_id label can take down the metrics store that was supposed to tell you about the outage — during the outage.
Logs only on the pod. The first rescheduling during an incident deletes the evidence. Shipping is not optional.
Alerting on causes until nobody listens. A muted channel is worse than no channel: it looks like coverage. Symptom-based paging, budgeted and reviewed, or expect the 2 a.m. page to be the one everyone sleeps through.
Observability configured by clicking. Unreviewed thresholds and dashboards that vanish when the stack is rebuilt. If it matters, it lives in git.

FAQ

Prometheus/Grafana, or my cloud's managed monitoring?
Start with whichever your team will actually open — a managed stack beats an unmaintained Prometheus. The concepts in this guide (golden signals, cardinality, symptom alerts) are portable across both; move when scale, multi-cluster needs, or cost make the case, not before.

Do I need traces if I have good logs?
Logs tell you what each service did; traces tell you how the request's time and errors distribute across services. With two services, logs plus correlation IDs may carry you. Past a handful of hops, the cross-service incident without traces is an afternoon of archaeology — that's the moment tracing pays for its instrumentation.

What should retention be?
Tier it by use: metrics at full resolution for weeks, downsampled for a year or more (capacity questions are seasonal); logs hot and searchable for days to weeks, archived to cheap storage for your compliance horizon; traces sampled with short retention — their value is in the incident week, not the archive.

How do I observe the observability stack itself?
Meta, but necessary: scrape the scrapers, alert on Prometheus/Loki/collector health from a dead-simple external check (even an uptime ping on the alerting path), and rehearse the "monitoring is down" case once — the worst incidents are the ones you discover from users because the dashboards were the thing that broke.

Where does this fit the production checklist?
Section 5 of the Kubernetes Production Checklist — the four signals per service, logs shipped off-pod, alerts with runbooks, and the one-command answers to "what changed / what's running / what's it costing" — is this guide compressed into checkboxes. (The full checklist PDF is the free lead magnet on the DevOps Enginer homepage.)

Conclusion

Observability is not a stack you install; it's the property that your two 2 a.m. questions — what changed, and what is it doing now — have fast, trustworthy answers. Metrics for the that, logs for the what, traces for the where, events for the cluster's own account of itself, all of it disciplined by cardinality, structure, honest sampling, and alerts that respect the humans they wake. Build the three dashboards, wire the deploy markers, put it all in git, and review the alert budget like a bill — because it is one. Your future on-call self, mid-incident, is the customer. Build for them.

Platform Engineering in Practice 2026: Golden Paths, Portals, and Proof Anyone Uses Them

softverseAdmin — Sat, 31 Oct 2026 03:49:53 +0000

TL;DR: Platform engineering succeeds or fails on one question — will anyone actually use it? In practice that means: build golden paths (opinionated, end-to-end routes from idea to production) instead of handing out raw infrastructure, treat the portal as the shop window rather than the shop, run the platform as a product with internal customers who can say no, and measure adoption and lead time instead of counting what you built. Start with one thinnest viable platform for one real team, and let demand — not a roadmap deck — pull the next feature.

Originally published on DevOps Enginer.

The question that decides everything

When I moved from DevOps engineering into project management, my core question changed. As an engineer it was "is it built right?" As a PM it became "will anyone actually use it?" I didn't expect that second question to be a platform engineering question — but it is the platform engineering question, asked verbatim.

Every failed internal platform I've seen described (and every post-mortem the community shares) fails the same way: a talented infrastructure team builds something technically excellent — a beautiful self-service portal, a pristine Kubernetes estate, immaculate Terraform modules — and the product teams keep opening tickets, running their own shadow pipelines, or quietly deploying around it. The platform was built right. Nobody used it. Our earlier piece on the rise of platform engineering covered why the movement exists; this one is about the operating discipline that makes it work in practice.

What a golden path actually is

"Golden path" and "paved road" get used loosely, so let's be concrete. A golden path is an opinionated, supported, end-to-end route for one common job — the canonical example being "a new backend service, production-ready":

A service template/scaffold: repo created with the standard layout, CODEOWNERS, and a README that isn't empty.
CI already wired: build, test, image scan, deploy stages — the reusable-workflow pattern, not a blank workflow file and good wishes.
Delivery already wired: a GitOps-managed deployment (in our world, Argo CD — see the tutorial) into the standard namespaces, with requests/limits defaults from the requests and limits guide.
The un-fun parts included: TLS, ingress, dashboards, alert routes, a runbook stub, cost labels. The things every team needs and no team wants to rediscover.

Two properties make it a path rather than a template dump. First, it's supported: a named team owns it, fixes it, and answers questions about it. Second, it's optional in principle: teams may leave the path — and own everything the path would have given them. That combination (attractive enough to choose, honest about the trade) is what separates a golden path from a mandate wearing a friendlier name. Mandates produce compliance theatre; paths produce adoption metrics you can believe.

The portal is the shop window, not the shop

Backstage is the name everyone knows, and developer portals absolutely have their place: a service catalog, one searchable home for docs and ownership, a front door for scaffolding. But the portal-first project is the most common platform failure of the last few years. The pattern: six months building the portal, a launch announcement, and then the discovery that the underlying capabilities — provisioning, deployment, secrets, environments — are still ticket queues wearing a web form.

The working order is the reverse:

Capabilities first. Can a team actually create a compliant environment, deploy, and get credentials without a human in the middle? If not, no interface fixes that.
Interfaces second. CLI, GitOps repo, API, portal — in whatever order your users actually pull. Many healthy platforms run for years on templates + docs + a catalog page before a full portal earns its keep.
The portal when it pays. When discoverability becomes the bottleneck — hundreds of services, new hires drowning — the portal consolidates what already works. It amplifies a platform; it cannot substitute for one.

A useful test: if your portal went down for a day, would teams be inconvenienced or blocked? Inconvenienced means the platform is real and the portal is its window. Blocked means you built a very expensive single point of failure.

Run it as a product, because it is one

The Team Topologies framing popularised the idea of the Thinnest Viable Platform: the smallest platform that meaningfully accelerates the stream-aligned teams — and no bigger. The discipline behind the phrase is pure product management:

Internal customers, real interviews. Before building, sit with two or three product teams. Where do they actually lose time — provisioning, flaky environments, deploy approvals, observability setup? Rank by their pain, not by your curiosity. (This is the PM question again: the backlog is theirs, not yours.)
A roadmap measured in outcomes. "Ship Backstage" is an output. "New service to production in under a day, without a ticket" is an outcome. Roadmaps written in outcomes survive contact with reality; output roadmaps produce demo-ware.
Adoption is the metric that can't be gamed. Percentage of services on the golden path, lead time from first commit to production for path users vs. off-path, ticket volume for things the platform now self-serves, and the quietest, most honest signal of all: do teams recommend it to each other? Anything you count, also ask whether the number would move if the platform got worse — if not, it's a vanity metric.
Support is a feature. Office hours, a Slack channel with a response norm, docs written for the reader's deadline rather than the author's completeness. Platforms die of unanswered questions more often than of missing features.

Start thin: your first golden path in about a month

For a team starting from zero, a realistic first iteration looks like this — scoped for one pilot team with one common service shape:

Week 1 — pick the slice with the pilot team. Interview them, watch their current path to production, and write down every step that is waiting, copying, or asking. Choose the one service archetype they ship most (usually a stateless HTTP service) and define "done": scaffold → merged → deployed to production, no tickets.
Week 2 — build the scaffold. Repo template with CI wired (reusable workflows), a Helm chart or kustomize base with sane defaults, Argo CD application manifest, dashboard and alert stubs. Ugly is fine; manual steps inside the template's README get listed as known debt, not hidden.
Week 3 — the pilot team ships something real on it. Sit with them (pair, don't present). Every friction point they hit is your backlog, in their priority order. Expect the path to be wrong in at least three places — that's the pilot working as intended.
Week 4 — harden the top frictions, write the docs from the pilot's notes, and invite team two. Not an org-wide launch: a second volunteer team. Volunteers are the adoption metric that predicts the rest.

Resist two expansions during this month: a portal (see above) and a second archetype. Depth on one path beats breadth across five half-paths, every time.

Escape hatches are a feature

A golden path that fits 100% of cases is a fantasy; one that fits 80% and handles the other 20% gracefully is a platform. Design the off-ramps deliberately:

Documented deviations. "Need a non-standard database? Here's the pattern, here's what you now own (backups, upgrades, on-call), here's who to tell." The cost of leaving the path should be explicit and paid consciously, not discovered during an audit.
Extension points in the templates. Values files, overlays, and workflow inputs that let teams tune the common knobs without forking the whole scaffold. Forks are where platform improvements go to die — every fork is a team that stops receiving your fixes.
A feedback loop with teeth. Deviations are product research: if five teams all deviate the same way, the path is wrong, not the teams. Review deviation patterns quarterly and promote the common ones into the path.

This is also where the cognitive-load argument lands honestly: the goal was never to remove all infrastructure thinking from product teams — it's to remove the undifferentiated infrastructure thinking, so their attention goes to the parts that are genuinely theirs.

Where AI fits (and where it doesn't)

AI tooling slots into a platform the same way it slots into DevOps generally — as an accelerator on the path, not a replacement for it (the full picture is in our AI for DevOps guide). Practical fits: generating first drafts of runbooks and docs from the templates, summarising incidents for the platform's users, and helping scaffold consumers customise the last 10% of a template. The trap is an "AI portal" that generates arbitrary infrastructure — it produces the opposite of a golden path: a thousand unique snowflakes, each confidently generated and none of them supported. The path stays opinionated; AI helps people walk it faster.

Failure modes I've learned to recognise

Platform teams rarely fail on technology. The recurring shapes:

The ticket team with a new name. The "platform team" still spends its weeks processing provisioning tickets by hand. If the work didn't become self-service, the rebrand was the deliverable.
Ivory-tower architecture. Standards designed without the pilot-team loop, announced as mandates, measured by compliance instead of adoption. Teams route around them within a quarter — usually silently.
Building for the demo. Optimising the launch presentation (portal screenshots, feature lists) instead of the second team's onboarding. Demos end; onboarding is forever.
Platform as a cost-cutting weapon. The moment teams believe the platform exists to justify headcount reduction, trust — the actual currency of adoption — is spent. Platforms are leverage, and leverage arguments only work when the lever's users believe it too.
No migration story. A great new path that requires teams to re-platform in a flag week will be adopted in never. Provide the incremental route: new services on the path now, existing services adopt piece by piece (CI first, delivery second, observability third).

Mistakes that quietly kill platforms

Counting what you built instead of who uses it. Launch metrics belong to marketing. Adoption, lead time, and ticket deflection belong to platforms.
Treating docs as the last step. Undocumented self-service is a rumour. The docs are part of the path — write them from the pilot's notes, in week 4, not "when things settle."
Forgetting the off-path teams. The 20% with legitimate special cases are your harshest critics or your best case studies. Support their deviations visibly and they defend the platform in rooms you're not in.
Letting the platform team's own on-call rot. If the path breaks and nobody is paged, every team learns the same lesson: the path is not really supported. Reliability of the platform is the platform's first SLO.
Scaling the team before the demand. A three-person platform team serving five teams can be exactly right. Growing the platform org ahead of adoption just raises the ROI bar the platform must clear to survive its next budget review.

FAQ

How big should a platform team be?
As small as possible while owning the path end-to-end — often two to four engineers for the first golden paths. The binding constraint is scope, not headcount: one well-supported archetype per small team beats a large team spread across a portal, three archetypes, and a migration.

Backstage or build our own portal?
Start from the test in this article: capabilities first. If provisioning and deployment are already self-service and discoverability is the pain, Backstage (or a commercial IDP) consolidates them well. If the capabilities are still tickets, a portal of any origin is a prettier queue.

How do we convince leadership it's working?
Report the product metrics: adoption share, lead-time change for path users, self-service rate for the top former ticket types, and team satisfaction from a two-question quarterly survey. "We launched features" convinces nobody twice; "a new service reaches production in a day and most teams choose the path" is a renewal argument.

Platform engineering vs. DevOps vs. SRE — who owns what?
In practice: the platform team owns the paved road (templates, self-service capabilities, the catalog), SRE owns reliability outcomes and the practices around them, and "DevOps" is the culture both operate in — product teams owning what they run, with the platform making that ownership cheap. Titles vary; the seams above are the ones that work. Our DevSecOps checklist shows how the security layer weaves through the same path.

Can a small company do platform engineering?
Yes — at small scale the "platform" might be one engineer's 20% time maintaining templates and a deploy pipeline for five teams. The product discipline (thin, adopted, supported) matters more than the org chart. What small companies can't afford is the big-bang version: a dedicated team building a portal nobody asked for.

Conclusion

Platform engineering in practice is product management applied to internal infrastructure. The technology — Kubernetes, GitOps, portals, templates — is the visible tenth. The other nine-tenths are the PM questions: Who is this for? What pain does it remove? Will anyone actually use it — and how will we know? Build one thin, supported, genuinely optional golden path for one real team; measure adoption like your budget depends on it, because it does; and let the teams who walk the path pull its next features into existence. That's the whole discipline — the rest is implementation detail, and the monorepo CI guide is a good example of what one of those details looks like when it's done as a platform would do it.

CI/CD for Monorepos 2026: Build Only What Changed

softverseAdmin — Sat, 31 Oct 2026 03:30:49 +0000

TL;DR: A monorepo with naive CI rebuilds the world on every commit, so pipelines get slower exactly as the repo succeeds. The fix is affected-only delivery: detect what changed, expand through the dependency graph, build and test only that slice, lean on remote caching for the rest, and protect the main branch with required checks plus a merge queue. Start with path filters in GitHub Actions; graduate to a build tool (Nx, Turborepo, Bazel) when the graph outgrows what filters can express.

Originally published on DevOps Enginer.

One repo, everything inside

My own website lives in a single repository: the Next.js blog, the API routes, shared components — and, sitting in a folder of its own, the exported files of a 3D portfolio game I built. It's a small repo by industry standards, and it already shows the monorepo shape: unrelated things sharing one history, one issue tracker, one pipeline.

That shape is why teams choose monorepos in the first place. Shared code is actually shared — no version-pinning dance between repos. A refactor can move an API and all its callers in one atomic commit. One CI system to maintain instead of twelve. The costs arrive later, and they arrive in the pipeline: when every commit triggers every build, every test suite, and every deploy, the feedback loop that made the monorepo attractive quietly dies. A one-line change to a README queues behind a full rebuild of services it never touched.

This guide is the delivery design that keeps the benefits: CI that does work proportional to the change, not proportional to the repo.

Why naive CI breaks as the repo grows

Three failure modes, always in this order:

Time grows linearly with the repo, not the change. Ten services means ten builds per pull request — including the nine you didn't touch. At fifty services the pipeline is an hour, and engineers start batching changes to "save runs," which makes every run bigger and riskier.
Flakiness becomes everyone's blocker. One unreliable test in a service you don't own turns your PR red. Multiply by every suite in the repo and a permanently-red main branch stops being an exception.
Deploys couple things the repo was supposed to decouple. If the pipeline ships everything on every merge, the monorepo has the blast radius of a monolith with the operational overhead of microservices — the worst trade on the table.

None of these are arguments against monorepos. They're arguments against running a monorepo like it's fifty polyrepos sharing a trench coat.

The core idea: affected-only builds

Every working monorepo pipeline implements the same four steps:

Detect which files changed in this PR or push.
Map files to the projects (packages, services, apps) that own them.
Expand through the dependency graph: a change to a shared library affects every project that depends on it — transitively.
Execute builds, tests, and deploys for the affected set only. Everything else is skipped, with its previous green result standing.

Step 3 is the one teams skip, and it's the one that matters. Path filters alone answer "what changed"; correctness needs "what is affected." A change inside packages/ui that doesn't rebuild the three apps importing it hasn't saved you time — it has deferred a breakage to someone else's deploy.

Level 1: path filters in GitHub Actions

For many teams, GitHub Actions' built-in path filtering plus a change-detection action covers step 1–2 honestly, and that's enough for a long time. Path filters on the trigger:

on:
  pull_request:
    paths:
      - 'services/api/**'
      - 'packages/**'
      - '.github/workflows/api.yml'

And for finer control — one workflow that fans out per project — a changes job whose outputs gate the rest:

jobs:
  changes:
    runs-on: ubuntu-latest
    outputs:
      api: ${{ steps.filter.outputs.api }}
      web: ${{ steps.filter.outputs.web }}
    steps:
      - uses: actions/checkout@v4
      - uses: dorny/paths-filter@v3
        id: filter
        with:
          filters: |
            api:
              - 'services/api/**'
              - 'packages/shared/**'
            web:
              - 'apps/web/**'
              - 'packages/shared/**'

  build-api:
    needs: changes
    if: needs.changes.outputs.api == 'true'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: echo "build and test the API"

Notice packages/shared/** appears in both filters — a hand-maintained shadow of the dependency graph. That's the honest limit of this level: fine for a handful of projects with shallow sharing, brittle once the graph gets deep. In my own setup, personal projects live on GitHub and deploy through Vercel on push to main, and Vercel's own monorepo handling (root directory settings, ignored build steps) plays this same role for the deploy side — skip the build when the project's slice didn't change. The principle is identical at every scale: pay only for what the change touches.

Level 2: let a build tool own the graph

When hand-written filters start lying — shared packages nested inside shared packages, generated code, mixed languages — the graph needs a real owner. The well-known names, by ecosystem:

Nx — JavaScript/TypeScript-first, with nx affected computing the affected set from its project graph, plus task caching locally and remotely.
Turborepo — lighter-weight, pipeline-oriented, popular with Next.js-style app collections; remote cache is its headline feature.
Bazel — the industrial option: language-agnostic, hermetic builds, precise dependency graphs; the power is real and so is the adoption cost.
Pants / Please / moon — worth knowing exist if your repo is Python-heavy, polyglot, or JS with different tastes.

The evaluation question is never "which is best" — it's "who maintains our graph, and what does the cache hit rate look like." Whichever you choose, the affected computation and the cache are the product; the CLI is packaging. I learned the mirror of this lesson with Jenkins years ago: every capability arrives as a plugin you then own, so a tool's maintenance tail is part of its price. Monorepo tooling is no different — adopt the smallest tool whose graph you trust.

Caching: the second half of "only build what changed"

Affected-only execution shrinks the work; caching shrinks what's left. Two layers:

Dependency caching — the package manager's store (npm/pnpm, pip, Maven, Go modules) keyed by lockfile hash, via actions/cache or your tool's built-in handling. This is table stakes; a monorepo without it re-downloads the internet on every run.
Task/build caching — the monorepo tool records inputs (source files, dependency outputs, environment, command) and replays stored outputs when inputs repeat. Locally it makes the second build instant; remote caching shares those results across the team and CI, so a branch whose code matches something a teammate already built skips the work entirely.

The discipline that makes caching safe is hermetic inputs: if a build secretly reads the current time, an unpinned network fetch, or a file outside its declared inputs, the cache will eventually serve you a confidently wrong artifact. Declare inputs honestly, pin versions in lockfiles, and treat a suspicious cache result as a build bug, not as weather.

Testing strategy: fast where it's local, honest where it's shared

Running "the affected tests" has the same graph trap as building: a change to a shared library should run its dependents' tests too. Practical shape:

Unit tests run for the affected set on every PR — these must be fast and hermetic, or nothing else here works.
Integration/contract tests for the affected set plus its direct dependents. Shared libraries earn extra scrutiny: their blast radius is the argument for their existence.
End-to-end tests are the budget item. Run a smoke subset on PRs, the full suite on merge to main (or nightly), and keep them owned — an E2E test with no owner becomes a flake generator with a fan club.
Quarantine flaky tests deliberately. A flaky test in a monorepo taxes every PR in the repo. The honest pattern: automatic detection (rerun-to-pass patterns flagged), a quarantine list with an owner and an expiry date, and the test still running in a non-blocking job so the signal isn't lost. Pretending flakiness is a discipline problem instead of an engineering one is how main branches stay red.

Per-project pipeline definitions also don't have to mean copy-pasted YAML: reusable workflows and composite actions let one maintained definition serve every project in the repo, with inputs for what differs.

Releases and versioning inside one repo

A monorepo doesn't imply a single version or a single release train. Two common models:

Independent versioning — each package/service versions and releases on its own (libraries published per-package, services deployed per affected set). Fits services; needs tooling discipline (changeset-style workflows for JS repos are the well-known pattern) so humans state the bump intent with the change.
Fixed/lockstep versioning — everything versions together. Simpler mental model, fits tightly-coupled product suites and app monorepos where "the version" means the whole product.

Deploys follow the same affected logic as builds: a merge to main deploys the affected services, each through its own pipeline — and if you deliver with GitOps, each affected service's desired state updates in git independently, exactly the model in our GitOps vs traditional CI/CD comparison. The anti-pattern to refuse: a "deploy everything Friday" train created because per-service deploys were never made safe. That train is a monolith wearing a monorepo's clothes.

Protecting main: required checks and merge queues

Affected-only CI creates one subtle hazard: PR checks and main-branch checks can test different sets. A PR that skips a job because its slice didn't change can merge into a main where that job would have failed — two individually-green PRs can compose into a red main.

The mechanisms that close the hole:

Required status checks per project path, so a PR can't merge without its affected jobs passing.
A merge queue (GitHub's is built in) that tests the merged result — your PR combined with everything ahead of it in the queue — before it lands. This is the piece that makes "main is always green" a property of the system instead of a hope.
A scheduled full-repo run (nightly) as the safety net: everything builds, everything tests, and failures are triaged as platform work in the morning — not discovered by the next unrelated PR.

When a monorepo is the wrong answer

Honesty section, because the migration runs both ways. Split (or don't adopt) when:

Access control genuinely differs — regulated or client-confidential code that must not be visible to the whole engineering org doesn't belong in the shared repo. (This is a real constraint, not an aesthetic one; it decides the question on its own.)
Release cadences are hostile — a library consumed by external teams on strict versioned contracts may live better as its own repo with real published versions.
The tooling tax exceeds the sharing benefit — a two-person team with two services gains little from Bazel. Path filters and discipline scale further than conference talks admit.

Notice none of these are "the repo got big." Big is a CI design problem, and it's the one this guide solves.

Mistakes that make monorepo CI miserable

Rebuilding everything "to be safe." You've rebuilt the polyrepo's costs inside one repo and kept none of its isolation. Safety lives in the graph's correctness, not in redundant compute.
Path filters without the dependency expansion. The shared-library change that skips its dependents' builds doesn't save time; it schedules a surprise.
No owner for flaky tests. In a monorepo, one team's flake is every team's blocked Friday. Quarantine with an owner and an expiry, or fix it the week it appears.
Caches with undeclared inputs. The first poisoned artifact costs more trust than a year of cache misses. Hermetic or don't cache.
A mega-workflow file. One 2,000-line workflow that every PR triggers is un-reviewable and un-skip-able. Fan out per project; share logic through reusable workflows, not through one file's job list.
Skipping the merge queue because PR checks "already passed." They passed against yesterday's main. The queue tests the combination you're actually shipping.

FAQ

Do I need Nx/Turborepo/Bazel, or are path filters enough?
Path filters are enough while the dependency graph is shallow enough to hold in your head — a few projects, one shared layer. The moment filters start missing dependents (or including everything "just in case"), the graph has outgrown hand-maintenance and a tool should own it. Start at the cheapest level that tells the truth.

How do I handle a change to the repo root — CI config, root lockfile?
Treat root-level changes as affecting everything that consumes them: a root CI change should run the full suite (it's the pipeline changing itself), and a lockfile change affects every project resolving from it. Encode that as an explicit filter rule rather than than discovering it during an incident.

Monorepo with multiple languages — one tool or several?
Either can work. Bazel and Pants are built for the polyglot case; language-native tools (Nx for JS, plus per-language CI for the rest) work when the language subtrees share little code. The deciding factor is how much cross-language dependency actually exists — measure it before choosing.

Does a monorepo slow down git itself?
Eventually, at very large scale, clone and status operations grow — which is why the largest monorepos pair with partial clone, sparse checkout, and virtual filesystems. Most teams hit CI pain years before git pain; solve them in that order.

How does this fit platform engineering?
The affected-only pipeline is a textbook platform product: a paved road every team in the repo consumes, maintained once. It's one of the highest-leverage builds in the platform engineering story — internal customers, a real interface, and a measurable benefit (pipeline minutes) that no team has to build for itself.

Conclusion

A monorepo's promise is atomic change across everything you own; its tax is a pipeline that wants to rebuild everything you own. Affected-only delivery is how you keep the promise and refuse the tax: detect the change, expand through the graph, build and test that slice, cache aggressively but hermetically, and let a merge queue guarantee that green PRs compose into a green main. Start where you are — for most GitHub Actions users that's path filters this week and an honest look at your shared packages — and let the graph's real complexity, not fashion, tell you when it's time for a build tool. If you want the underlying Actions mechanics first, our GitHub Actions pipeline guide is the place to start.