DEV Community

Cover image for EKS Pod Identity vs IRSA: Where It Works, and Where Elasticsearch’s S3 Snapshots Break It
AWS Community Builders profile image Maxwell Ugochukwu
Maxwell Ugochukwu for AWS Community Builders

Posted on AI-assisted

EKS Pod Identity vs IRSA: Where It Works, and Where Elasticsearch’s S3 Snapshots Break It

#aws #kubernetes #security #devops

I used EKS Pod Identity for cert-manager, the EBS CSI driver and Vault, then ended up with a plain IAM user for Elasticsearch snapshots. Here is what failed, how to tell which identity your pod really used, and what has changed upstream since.

TL;DR: EKS Pod Identity is simpler than IRSA, and I use it wherever the software supports it. Elasticsearch’s bundled repository-s3 plugin was the exception. On Elasticsearch 8.15.0, IRSA silently fell back to the node role and Pod Identity was rejected by the bundled AWS SDK, so snapshots run on a narrowly scoped IAM user stored in the Elasticsearch keystore. Newer Elasticsearch versions behave differently, so test yours before copying my choice.

Diagram showing three credential paths for an Elasticsearch pod
Figure 1. The three credential paths an Elasticsearch pod can use for S3 snapshots, and what happened on each.

Why I Chose Pod Identity First

I built a centralized logging platform on EKS (Elasticsearch through ECK, Filebeat, Vault, cert-manager, ingress-nginx). Every component that talks to AWS needs credentials without static keys, which on EKS means IRSA or Pod Identity.

Pod Identity removes most of the ceremony from IRSA:

Capability IRSA EKS Pod Identity
Cluster prerequisite IAM OIDC provider per cluster Pod Identity Agent add-on
Role-to-ServiceAccount link ServiceAccount annotation EKS API association
Role trust policy Cluster OIDC provider + ServiceAccount conditions pods.eks.amazonaws.com (reusable across clusters)
Credential delivery Projected token exchanged with STS Link-local agent endpoint
SDK requirement Supports web identity tokens Supports container credentials from the agent (AWS lists minimum SDK versions)

The last row is where this post gets interesting. Pod Identity is only simpler if the software inside your pod can actually use it.

Where It Worked: cert-manager, EBS CSI and Vault

For cert-manager (Route 53 DNS-01 challenges), the EBS CSI driver and Vault’s AWS access, Pod Identity was a clear improvement. The trust policy trusts the EKS service principal, and the association ties the role to a namespace and ServiceAccount:

# Simplified pattern, not the full module
resource "aws_eks_pod_identity_association" "cert_manager" {
  cluster_name    = var.cluster_name
  namespace       = "cert-manager"
  service_account = "cert-manager"
  role_arn        = aws_iam_role.cert_manager.arn
}
Enter fullscreen mode Exit fullscreen mode 
{
  "Effect": "Allow",
  "Principal": { "Service": "pods.eks.amazonaws.com" },
  "Action": ["sts:AssumeRole", "sts:TagSession"]
}
Enter fullscreen mode Exit fullscreen mode

There is no OIDC provider to manage, no annotations to forget, and the same role definition works across clusters. These components use modern AWS SDKs that natively support the Pod Identity credential provider.

Where It Broke: Elasticsearch Snapshots

For Elasticsearch, I wanted the same pattern: snapshots to S3 with no static credentials. I tried both federated options in order, and both attempts ran on Elasticsearch 8.15.0.

Attempt 1: IRSA

On 8.15.0, snapshot repository registration failed with AccessDenied. The detail that mattered was the identity named in the error: it was the EC2 node’s instance role, not the IRSA role. The pod had never assumed the role at all. Three problems stacked on top of each other:

  1. No dedicated ServiceAccount by default. ECK does not create a dedicated ServiceAccount per Elasticsearch resource unless you set one in podTemplate. Without one carrying the IRSA annotation, the EKS mutating webhook has nothing to inject credentials into, and the SDK falls back to the node instance role.
  2. A sandboxed token path. Elastic documents that the plugin reads the web identity token from config/repository-s3/aws-web-identity-token-file, because a repository cannot read files outside its config directory. Having AWS_ROLE_ARN injected is not enough on its own.
  3. Token refresh failures. Reports against Elasticsearch describe the plugin failing to pick up refreshed tokens after the first one expires: snapshots work, then fail hours later until the node restarts (elastic/elasticsearch#115186).

Attempt 2: Pod Identity

Pod Identity is the newer mechanism, so I tried it next. The snapshot repository check failed with an error that points straight at the bundled SDK:

The full URI (http://169.254.170.23/v1/credentials) contained within
environment variable AWS_CONTAINER_CREDENTIALS_FULL_URI has an invalid host.
Host can only be one of [localhost, 127.0.0.1]
Enter fullscreen mode Exit fullscreen mode

When you create a Pod Identity association, EKS injects AWS_CONTAINER_CREDENTIALS_FULL_URI pointing at the link-local address 169.254.170.23. The older AWS Java SDK v1 bundled with Elasticsearch 8.15.0 only accepts localhost or 127.0.0.1 there, so it rejects the endpoint. This is tracked in elastic/cloud-on-k8s#8320, where an ECK maintainer explains that ECK is not setting the variable and the limitation sits in Elasticsearch itself.

What I Deployed: A Scoped IAM User in the Keystore

Because both federated paths were dead ends on 8.15.0, I used Elastic’s documented fallback: an IAM user with access to nothing except the snapshot bucket, loaded into the Elasticsearch keystore through spec.secureSettings:

# Simplified fragment of the Elasticsearch resource
spec:
  secureSettings:
    - secretName: es-s3-snapshot-credentials  # holds s3.client.default.access_key and secret_key
Enter fullscreen mode Exit fullscreen mode

It is a deliberate, documented exception to "no static credentials", not an oversight. The trade-off is a long-lived key that you must scope tightly and rotate yourself.

How to Tell Which Identity Your Pod Really Used

The most transferable lesson from this build is how to debug pod credential chains. When an AWS call from a pod fails with AccessDenied, read the caller ARN in the error message before touching IAM policies. If it names the node instance role, federation never kicked in, and no policy edit on your workload role will fix it.

Next, inspect what the platform actually injected into the pod:

# 1. Check for IRSA injection
kubectl exec -n <ns> <pod> -- env | grep -E 'AWS_ROLE_ARN|AWS_WEB_IDENTITY_TOKEN_FILE'

# 2. Check for Pod Identity injection
kubectl exec -n <ns> <pod> -- env | grep -E 'AWS_CONTAINER_CREDENTIALS_FULL_URI|AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE'

# 3. Verify the Pod Identity association exists in the EKS control plane
aws eks list-pod-identity-associations --cluster-name <cluster>
Enter fullscreen mode Exit fullscreen mode

Those three commands separate "the platform did not inject credentials" (missing ServiceAccount, missing association, or a pod created before the association existed) from "the application ignored or rejected them" (outdated SDK or a custom credential path).

One hardening tip: setting the IMDSv2 hop limit to 1 on your nodes stops regular pods from reaching the instance metadata service, so a failed federation produces a clear error instead of a silent fall back to the node role. Pods on the host network are the exception, and any workload that legitimately needs the metadata service will need a closer look first.

What Has Changed Upstream Since

I ran 8.15.0, and the picture has moved on. Check these before copying my workaround:

  • Token refresh: in the IRSA token refresh issue, an Elasticsearch maintainer says the underlying "connection pool shut down" failure is fixed from 8.18.0 (#115186).
  • AWS SDK v2: Elastic’s breaking-changes notes list the repository-s3 plugin moving from AWS SDK v1 to v2 in 9.1.0 (#126843, release notes). AWS SDK v2 is the line that understands Pod Identity credentials, which is the root cause of my attempt 2 failure.
  • Pod Identity on 9.1.0: a commenter in elastic/cloud-on-k8s#8320 reports that Pod Identity works on 9.1.0 if you symlink the token into the plugin’s config directory and set the two container credential variables yourself. I have not tested this workaround: 
# Reported in the issue thread, untested by me
containers:
  - name: elasticsearch
    env:
      - name: AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE
        value: /usr/share/elasticsearch/config/repository-s3/eks-pod-identity-token
      - name: AWS_CONTAINER_CREDENTIALS_FULL_URI
        value: http://169.254.170.23/v1/credentials
initContainers:
  - name: symlink-token
    command:
      - sh
      - -c
      - mkdir -p /usr/share/elasticsearch/config/repository-s3 && ln -s /var/run/secrets/pods.eks.amazonaws.com/serviceaccount/eks-pod-identity-token /usr/share/elasticsearch/config/repository-s3/eks-pod-identity-token
Enter fullscreen mode Exit fullscreen mode

If you upgrade across 9.1.0, read the SDK v2 notes first. They say you must set the signing region (s3.client.default.region) and that SDK v2 does not support the older IMDSv1 protocol.

My IAM user was the right call for 8.15.0. On a current release, testing federated credentials first is worth it, because removing a static key is a real security gain.

A Practical Decision Guide

Workload Recommended path Why
Modern operators (cert-manager, EBS CSI) Pod Identity Simplest setup, cluster-portable roles
Workloads with older AWS SDKs IRSA or a scoped static key Check SDK support for Pod Identity first
Elasticsearch repository-s3 on 8.15.x (my case) Scoped IAM user in the keystore Both federated paths failed
Elasticsearch 8.18 to 9.0 Test IRSA Token refresh fix reported in 8.18.0; the plugin is still on SDK v1
Elasticsearch 9.1 and later Test Pod Identity or IRSA SDK v2 in the plugin; Pod Identity may still need the token symlink workaround

Takeaways

  • Pod Identity is the better default, provided the container’s SDK supports it. Verify SDK compatibility before migrating stateful or Java-heavy workloads.
  • Read the failing caller ARN first. An AccessDenied naming the EC2 node role is a credential delivery problem, not an IAM policy problem.
  • Document deliberate exceptions. My repository’s troubleshooting guide explains why snapshots use an IAM user, so a future maintainer does not "fix" it back to a broken association without testing.
  • Pin advice to software versions. Workarounds expire, so write down the exact version that forced the compromise.

Project:

GitHub logo princemaxi / enterprise-k8s-observability

Enterprise Kubernetes Logging Platform

This repository deploys the EKS-based enterprise logging stack, including Elasticsearch, Kibana, Filebeat, Vault, ingress, TLS, DNS, and the Order API Terraform is intentionally split into two state boundaries:

terraform/infrastructure/environments/<env>
        -> EKS and AWS resources
terraform/platform/environments/<env>
        -> Kubernetes and Helm resources

The platform root reads EKS connection details from the infrastructure state with terraform_remote_state. It never declares a Kubernetes provider in the same state that creates EKS, so a clean deployment does not need terraform apply -target.

Deploy dev

Prerequisites are Terraform >= 1.7, AWS CLI credentials, kubectl, Docker, and an S3 bucket plus DynamoDB lock table for Terraform state. The split-root workflow expects the real backend files in the environment directories and the repo-root copies used by the helper scripts:

cp terraform/infrastructure/environments/dev/backend.hcl \
  terraform/backend-dev-infrastructure.hcl
cp terraform/platform/environments/dev/backend.hcl \
  terraform/backend-dev-platform.hcl
Enter fullscreen mode Exit fullscreen mode

The live deployment files are real configuration, not placeholders. Edit the real files in the…

View on GitHub



Troubleshooting guide: [docs/troubleshooting.md]

Have you made the leap to EKS Pod Identity yet, or are you still sticking with IRSA? Let me know what tripped you up in the comments!

Top comments (0)