
The AWS EKS Best Practices Guide is a strong reference, and most teams read it once, around launch. The harder part is knowing which practice to apply in which phase, and noticing when a cluster has quietly stopped following them. This post covers both, based on EKS platforms I have built and operated, including one migration where a GitOps-first approach saved about 50% of the planned time.
TL;DR
- Decide the expensive-to-change practices (networking, identity, cluster layout) at design time. Enforce the rest in the pipeline. Verify all of them continuously.
- Security goes first on anything you cannot cheaply change later, such as identity and network boundaries. It does not outrank everything else forever.
- Practices change. The
aws-authConfigMap, IRSA versus Pod Identity, and add-on defaults have all moved. A cluster that was compliant a year ago may not be today.- Drift is the default state. Treat every practice as something to check on a schedule, not something you did once.
Which phase, which practice
| SDLC phase | Practices to settle | Why then |
|---|---|---|
| Design | VPC and subnet sizing, IPv4 versus IPv6, cluster-per-environment layout, identity model, multi-tenancy boundaries | These are hard or impossible to change after workloads land. Pod IP exhaustion is a design-time failure that shows up in production. |
| Build | Infrastructure as code, GitOps contract, node provisioning (Karpenter or Auto Mode), add-on versions | Anything created by hand cannot be reviewed, repeated, or diffed later. |
| Test | Upgrade rehearsal, PodDisruptionBudgets, probes, resource requests and limits, load and scale behavior | Upgrades and scale events are where weak workload settings surface. Test them somewhere that is allowed to break. |
| Deploy | Admission policy, image provenance, progressive delivery | The pipeline is the cheapest enforcement point. A policy that blocks a bad manifest costs minutes. A finding in production costs days. |
| Operate | Observability, cost visibility, upgrade cadence, access review, drift detection | Operations is where practices decay, so this phase owns the checking. |
How I applied this on a migration
The platform was an on-premises Docker estate moved to EKS, delivered across three environments (an internal lab, a sandbox, and production) with no direct access to the later ones. That constraint forced the practices into code early.
Design decisions came first: cluster layout per environment, networking, and the identity model. Build followed with Terraform for the cluster and a GitOps Bridge for the handoff to ArgoCD, with Karpenter handling nodes. The bridge keeps infrastructure facts in one interface and application manifests free of hard-coded values. I covered that contract in The GitOps Bridge.
Because each environment came from the same code, validating in the lab meant the sandbox and production were the same shape. The saving of about 50% in delivery time came from removing per-environment hand-work, not from any one tool being fast.
On an earlier platform with several clusters, the practice that mattered most was a fixed upgrade cadence. Quarterly, rehearsed in a lower environment first, with zero downtime as the target. A predictable rhythm keeps the version gap small, and a small gap makes each upgrade routine.
Where to start, and how much it matters
Start in design, not in operations. The practices with the highest cost of change are the ones to settle first:
- IP address planning and subnet sizing.
- Cluster and account boundaries.
- Identity for both humans and workloads.
Retrofitting these means rebuilding clusters. Practices like resource limits or probe tuning can be fixed per workload at any time, so they can wait for the Test phase.
On importance, I rank by cost of being wrong. A missed probe setting causes a bad rollout. A wrong identity or network boundary causes an incident or a rebuild.
Should security come first
For identity, network boundaries, and secrets, yes. These are the decisions that are expensive to reverse and that define blast radius. Settle them in design and enforce them in the pipeline.
Security does not outrank everything indefinitely, though. A cluster that is perfectly locked down but cannot be upgraded, observed, or scaled becomes a security problem of its own, because stale versions accumulate known vulnerabilities. Reliability and operability practices protect the security posture over time.
A working order that has held up for me:
- Identity and network boundaries (design).
- Reproducibility through IaC and GitOps (build).
- Upgrade path and observability (test and operate).
- Cost controls (operate, but visible from day one).
Practices change, so recheck them
Several recommendations have moved, and an environment built a while ago may still follow the old ones.
Cluster access. The aws-auth ConfigMap is deprecated. The EKS best practices guide says to use the Cluster Access Management API (access entries) for new clusters and to migrate existing ones. Before removing ConfigMap entries that EKS created for managed node groups or Fargate profiles, confirm the matching access entries exist, as the migration guide notes.
Workload identity. EKS Pod Identity is now the recommended approach for new workloads, with IRSA still fully supported. Existing IRSA setups do not need an emergency rewrite, but new work should default to Pod Identity.
Node management. EKS Auto Mode and Karpenter change who owns node lifecycle, upgrades, and add-ons. If your cluster predates them, the manual node-group practices in older runbooks may no longer be the right ones.
Component behavior. Agents and add-ons have their own failure modes at scale. A public issue on the EKS Pod Identity agent (#171) showed credential renewals continuing for pods that no longer existed, and I opened a fix for it (pull request #192, under review at the time of writing). Keep add-on versions current and read release notes, because behavior at scale is not always in the defaults.
Catching drift
Drift happens in two ways: the cluster moves away from your code, or your code moves away from current guidance. Both need a check.
-
Cluster versus code. Run
terraform planon a schedule and alert on a non-empty diff. Let ArgoCD report out-of-sync applications and treat that as a signal, not noise. Disable manual edits where you can. - Cluster versus policy. Admission policy (Kyverno or OPA Gatekeeper) catches new violations. Run the same policies in audit mode against existing resources to find old ones.
- Cluster versus guidance. Put a recurring review on the calendar, quarterly alongside the upgrade, to compare your setup with the current best practices guide. Check access method, workload identity method, add-on versions, and deprecated API usage.
- Deprecated APIs. Scan for them before every upgrade, not after.
- Cost drift. Review node utilization and Karpenter consolidation behavior monthly. Over-provisioning accumulates quietly.
A short checklist for the quarterly review:
- Is the cluster version within the supported window?
- Is access managed through access entries, with no leftover ConfigMap entries?
- Are new workloads using Pod Identity?
- Do add-on versions match what the pipeline defines?
- Does the audit-mode policy report show zero new violations?
- Does the Terraform plan show no unexplained diff?
Takeaway
Treat best practices as a lifecycle, not a launch gate. Settle the expensive decisions in design, enforce the rest in the pipeline, and check all of them on a schedule. Security leads on identity and boundaries, and operability keeps that security from decaying.
If your cluster is more than a year old, the access method is the first thing to look at.
Top comments (1)
Drift controls work best when the feedback loop matches the stage: cheap policy checks before merge, admission controls at deploy, and periodic comparison for changes made outside the normal path. Reporting the owning team and the intended remediation is as important as detecting the difference.