Originally published on kuryzhev.cloud
The problem we hit
It was 3:40am when the pager went off. A backup CronJob that had been running fine for months suddenly started throwing AccessDenied on s3:PutObject. Nothing had changed in the application code, no deploy had gone out, and the IAM policy attached to the role looked exactly like it always had. The only thing that had happened was a "routine" node group rotation earlier that evening — new nodes, same AMI family, same everything, or so we thought.
The on-call engineer's first instinct was to check the IAM role tied to the service account. Policy looked correct: s3:PutObject on the right bucket ARN. But the pod kept failing. Digging into CloudTrail, the actual caller wasn't the intended IRSA role at all — it was arn:aws:sts::<acct>:assumed-role/eksctl-nodegroup-role/i-0abcd.... The pod was quietly falling back to the EC2 instance profile attached to the node, not the role we thought we'd scoped down months ago.
That's when the real scope of the problem became clear. The node's IAM role was shared across twelve other unrelated pods on the same node group — logging sidecars, a metrics exporter, a couple of internal APIs. All of them had been running with broad, node-level permissions the entire time, and nobody noticed because everything "worked." The backup job breaking was almost a lucky accident — it forced us to look at something that had been a ticking time bomb for a long time. Overprivileged access doesn't announce itself; it just sits there until an incident makes you look.
Why it happens
Without a proper EKS IRSA setup, pods don't get their own AWS identity — they inherit whatever the EC2 instance profile on the node grants. That's the default behavior of the AWS SDK credential chain: if no explicit credentials are found, it walks up to instance metadata and grabs the node's role. It's broad, it's shared across every pod scheduled on that node, and it's nearly impossible to trace a specific API call back to a specific workload in CloudTrail unless you know exactly what to look for.
IRSA (IAM Roles for Service Accounts) fixes this by binding a Kubernetes ServiceAccount to a specific IAM role via OIDC federation. Here's the chain: the EKS cluster has an OIDC identity provider registered in IAM. A ServiceAccount gets annotated with eks.amazonaws.com/role-arn. When a pod using that ServiceAccount starts, a mutating webhook injects a projected token volume and two environment variables — AWS_WEB_IDENTITY_TOKEN_FILE and AWS_ROLE_ARN. The AWS SDK reads that token, calls sts:AssumeRoleWithWebIdentity, and gets short-lived credentials scoped to exactly that role.
In our incident, two things had gone wrong. First, the cluster had been recreated via Terraform months earlier and the OIDC provider registration hadn't been re-associated — a classic drift issue that nobody caught because most workloads still "worked" using node-role fallback. Second, and this is the gotcha that really got us: the trust policy's Condition block had a subtly wrong sub value from an earlier copy-paste — the namespace didn't match exactly. When the AssumeRoleWithWebIdentity call failed silently, several SDKs just fell back to the default credentials chain instead of erroring loudly. That's the dangerous part — it doesn't crash, it just quietly uses the wrong identity.
The fix (with code)
The fix has three parts: confirm the OIDC provider is actually registered, write a trust policy scoped to an exact namespace and service account (no wildcards, ever), and attach a least-privilege policy instead of reaching for AmazonS3FullAccess as a "temporary" patch. We ran this on EKS 1.27 with eksctl 1.147.0 and aws-cli 2.15.x — worth noting that anything older than aws-cli 2.9 doesn't play nicely with IRSA debugging defaults.
#!/usr/bin/env bash
# irsa-setup.sh — reproduce the fix from the incident: scoped IAM role for a single ServiceAccount
set -euo pipefail
CLUSTER_NAME="prod-eks-01"
NAMESPACE="backup-jobs"
SERVICE_ACCOUNT="s3-backup-sa"
BUCKET_ARN="arn:aws:s3:::acme-backup-bucket"
ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
# 1. Confirm OIDC provider exists (this was the actual root cause in our incident —
# the provider was missing after a cluster recreate via Terraform)
OIDC_ISSUER=$(aws eks describe-cluster --name "$CLUSTER_NAME" \
--query "cluster.identity.oidc.issuer" --output text)
OIDC_ID=$(echo "$OIDC_ISSUER" | sed 's|https://oidc.eks.*/id/||')
if ! aws iam list-open-id-connect-providers | grep -q "$OIDC_ID"; then
echo "OIDC provider missing — associating now"
eksctl utils associate-iam-oidc-provider --cluster "$CLUSTER_NAME" --approve
fi
# 2. Write a trust policy scoped to exact namespace:serviceaccount (no wildcards)
cat > trust-policy.json <<EOF
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": { "Federated": "arn:aws:iam::${ACCOUNT_ID}:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/${OIDC_ID}" },
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"oidc.eks.us-east-1.amazonaws.com/id/${OIDC_ID}:sub": "system:serviceaccount:${NAMESPACE}:${SERVICE_ACCOUNT}",
"oidc.eks.us-east-1.amazonaws.com/id/${OIDC_ID}:aud": "sts.amazonaws.com"
}
}
}]
}
EOF
# 3. Create the role — least privilege, no AmazonS3FullAccess "temporary fix"
aws iam create-role \
--role-name irsa-s3-backup \
--assume-role-policy-document file://trust-policy.json
cat > s3-policy.json <<EOF
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": ["s3:PutObject", "s3:GetObject"],
"Resource": "${BUCKET_ARN}/*"
}]
}
EOF
aws iam put-role-policy \
--role-name irsa-s3-backup \
--policy-name s3-backup-put \
--policy-document file://s3-policy.json
echo "Role ARN: arn:aws:iam::${ACCOUNT_ID}:role/irsa-s3-backup"
Once the role exists, the ServiceAccount needs the annotation applied before the pod is created — this bit us too. The mutating webhook only injects the token env vars at admission time, so annotating an existing ServiceAccount and doing a rolling update won't help until the pods are fully recreated.
# service-account.yaml — annotate BEFORE the pod is created; webhook injects env vars at admission time
apiVersion: v1
kind: ServiceAccount
metadata:
name: s3-backup-sa
namespace: backup-jobs
annotations:
eks.amazonaws.com/role-arn: "arn:aws:iam::123456789012:role/irsa-s3-backup"
---
apiVersion: batch/v1
kind: CronJob
metadata:
name: s3-backup-job
namespace: backup-jobs
spec:
schedule: "0 3 * * *"
jobTemplate:
spec:
template:
spec:
serviceAccountName: s3-backup-sa # must match trust policy "sub" exactly
containers:
- name: backup
image: amazon/aws-cli:2.15.0
command: ["aws", "s3", "cp", "/data/dump.tar.gz", "s3://acme-backup-bucket/"]
restartPolicy: OnFailure
# Verification after apply:
# kubectl exec -it <pod> -n backup-jobs -- aws sts get-caller-identity
# Expected output:
# {
# "UserId": "AROAEXAMPLE:botocore-session-1234567890",
# "Account": "123456789012",
# "Arn": "arn:aws:sts::123456789012:assumed-role/irsa-s3-backup/botocore-session-1234567890"
# }
# If Arn instead shows "assumed-role/eksctl-nodegroup-role/..." — annotation wasn't
# applied before pod creation, or SA name/namespace mismatch in trust policy.
Also worth checking manually: the token file lives at /var/run/secrets/eks.amazonaws.com/serviceaccount/token inside the pod. If that file is missing, the webhook never fired — usually because the annotation went on too late. I stopped trusting "it should be fine, we applied the annotation" the moment I saw this happen twice in the same week across two different teams.
Prevention checklist
This class of incident is entirely preventable with a few disciplined habits baked into how you manage IAM for EKS workloads.
-
One IAM role per workload, never shared. Reusing a role across unrelated pods is exactly how a backup job's misconfiguration turns into a fleet-wide privilege audit. Run
aws iam list-roles | grep irsaperiodically and map each role back to a single namespace/service account. -
Scope trust policies exactly — no wildcards. The
Conditionblock must useStringEqualswith the precisesubvalue:system:serviceaccount:<namespace>:<sa-name>. A typo or wrong namespace doesn't throw a loud error — it silently falls back to node credentials in some SDKs, which is worse than an outright failure. -
Detect OIDC provider drift automatically. After any cluster upgrade or recreation, verify the provider is still registered with
aws eks describe-cluster --name <cluster> --query "cluster.identity.oidc.issuer"against your IAM provider list. Wire this into a CI job or a Terraform plan check so it never depends on someone remembering. -
Never leave "temporary" broad policies in place. If you attach
AmazonS3FullAccessto unblock an incident at 4am, put a ticket and a revert date on it immediately. It will not get revisited otherwise. - Segregate roles by environment. Don't reuse the same trust policy pattern across dev, staging, and prod namespaces — that's how privilege leaks across environments during a "quick copy-paste."
- Use a tested module instead of hand-rolled JSON. The terraform-aws-modules IRSA module generates trust policies correctly and removes an entire class of typo bugs.
Getting the EKS IRSA setup right isn't a one-time task — it's an ongoing discipline. We now run a quarterly audit that cross-references every ServiceAccount annotation against its IAM role and trust policy, and we treat any mismatch as a P2. For more patterns on locking down cloud IAM without breaking workloads, check out the DevOps_DayS archive — there's a related piece on Terraform-managed Vault secrets that pairs well with this.
For the official reference, AWS's own IAM roles for service accounts documentation is worth bookmarking — it's the source of truth for webhook behavior across EKS versions.
Top comments (0)