EKS Upgrades Need Pod Identity Checks

I like EKS upgrades when they stay boring. The problem is that identity paths inside a cluster are often not boring at all. A control plane bump can finish cleanly, nodes can rotate on schedule, and then one quiet workload starts failing because its AWS credential flow changed in a way nobody noticed during the maintenance window.

That is why I now treat Pod Identity and IRSA validation as a required preflight, not a nice extra. In AWS and Kubernetes environments, most upgrade pain is not the version change itself. It is the hidden dependency around service accounts, trust policies, webhook injection, or SDK defaults inside the container.

If your platform team already documents privacy notes for email screening or wants better evidence for OTP-related checks, the same operational habit applies here too: freeze the expected identity story before you touch production.

Why EKS upgrades break identity in quiet ways

Most failed upgrades do not start with "cluster down." They start with one controller, one CronJob, or one internal API worker getting AccessDenied after the change. The version move just exposed a drift that had already been there.

The common traps I keep seeing are:

• a service account annotation exists, but the role trust policy no longer matches
• the workload moved from one node group to another and lost an assumption about metadata access
• the container image uses an older SDK chain than the team remembers
• nobody verified whether the app was using EKS Pod Identity or classic IRSA

AWS recommends validating workloads after cluster updates and keeping an eye on addon compatibility because skew between cluster pieces is a normal source of trouble. Source: https://docs.aws.amazon.com/eks/latest/userguide/update-cluster.html

That sounds obvious, but teams still rush. During one review I even saw a scratch note mentioning temp org mail and tempail mail in the same upgrade checklist. That was not the real issue, of course, but it told me the runbook had become a dumping ground instead of a decision tool.

The checks I run before changing the cluster version

I want a short list that an operator can finish in 15 minutes.

First, map which workloads actually need AWS credentials. Do not assume it is only controllers. Jobs that pull from S3, apps that read from SSM, and internal workers that publish to SNS are the ones that surprise people later.

Second, record the current identity path:

• service account name
• namespace
• IAM role ARN
• whether the workload uses IRSA or Pod Identity
• one successful AWS API call made by that pod today

Third, verify addon status. If the cluster relies on Pod Identity, make sure the EKS Pod Identity Agent is where you expect it and healthy before the maintenance starts. If you are still on IRSA, validate the OIDC provider and trust conditions now, not at 2 a.m.

A small smoke test for Pod Identity and IRSA

I prefer one disposable pod per critical credential path. It is not glamorous, but it catches the boring mistakes fast.

kubectl -n payments run iam-smoke \
  --rm -it --restart=Never \
  --serviceaccount=payments-api \
  --image=amazon/aws-cli:2.27.41 \
  -- sts get-caller-identity

If that works, capture the returned ARN in the change record. If it fails, do not start the upgrade yet.

For app-level validation, I also want one command that touches the real dependency:

kubectl -n payments exec deploy/payments-api -- \
  aws ssm get-parameter --name /prod/payments/stripe-key >/dev/null

This is one of those places where CI/CD can save you if you let it. A cheap preflight job that runs these commands before the upgrade window is much better than learning about drift from 500 errors after rollout.

What to inspect first when credentials drift

When a pod suddenly loses access after an upgrade, I check in this order:

1. the pod spec and service account annotation
2. the IAM role trust relationship
3. the node placement and any leftover dependency on instance profile access
4. the container's AWS SDK version and credential provider behavior

The reason for that order is simple. Most breaks are plumbing, not policy design. The app did not become evil in the last ten minutes; the wiring changed.

For teams using Pod Identity, the official EKS docs are worth rereading because the credential flow is different from older IRSA mental models. Source: https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html

A rollout checklist for CI/CD teams

My upgrade checklist is short enough that people actually use it:

• prove one successful AWS call for each critical workload before the change
• store the expected caller identity ARN in the change ticket
• run the same smoke test right after control plane and node updates
• fail the rollout if any critical workload returns a different identity path
• keep one fallback note for workloads still coupled to older IRSA assumptions

That last point matters more than people admit. Mixed clusters happen. Some teams are migrating gradually, and pretending otherwise just makes the incident review more messy then it needs to be.

Q&A

Is this only needed for major EKS upgrades?

No. I use it for addon changes, node group refreshes, and identity migrations too. The failure shape is similar enough that the same checks pay off.

Should CI own these checks or SREs?

Both, honestly. CI should run the repeatable smoke tests. SREs or platform engineers should own the final go/no-go call when the results look off.

What is the main win?

You stop treating credential failures as random post-upgrade noise. You get a clear before-and-after identity record, which makes rollback and diagnosis way less painful.