Why Your EBS CSI Driver Can’t Attach Volumes (And How IRSA Fixes It)

If you're running EKS and trying to use the AWS EBS CSI Driver, you've probably hit a cryptic volume attachment error at some point. Here's a breakdown of the exact problem I faced, and how I resolved it using IAM Roles for Service Accounts (IRSA).

📊 The Problem

After installing the EBS CSI Driver via Helm, pods like Prometheus and others failed to bind their PersistentVolumeClaims (PVCs). On checking the controller logs:

kubectl logs -n kube-system -l app.kubernetes.io/name=aws-ebs-csi-driver -c ebs-plugin --tail=100

You may see an error like this:

Could not attach volume "vol-..." to node "i-...": operation error EC2: AttachVolume, https response error StatusCode: 403, api error UnauthorizedOperation

🔍 How to Check Configuration

Use these commands to validate that everything is wired correctly:

✅ Controller logs (should NOT show AttachVolume 403 errors)

kubectl logs -n kube-system -l app.kubernetes.io/name=aws-ebs-csi-driver -c ebs-plugin --tail=100

Bad output:

... AttachVolume ... UnauthorizedOperation ...

Good output:

I0330 00:04:47.133164       1 node.go:944] "CSINode Allocatable value is set" nodeName="i-035ea5232976e526c" count=26

✅ Validate CSINode registration

kubectl get csinode

Good output:

NAME                         DRIVER              ...
i-035ea5232976e526c          ebs.csi.aws.com     ...

✅ Confirm node tolerations

kubectl get daemonset ebs-csi-node -n kube-system -o jsonpath='{.spec.template.spec.tolerations}' | jq

Expected:

[
  {"operator": "Exists"},
  {"effect": "NoExecute", "key": "node.kubernetes.io/not-ready", "operator": "Exists"},
  {"effect": "NoSchedule", "key": "node.kubernetes.io/disk-pressure", "operator": "Exists"}
]

✅ Check IRSA annotations

kubectl describe sa ebs-csi-controller-sa -n kube-system

Expected:

Annotations: eks.amazonaws.com/role-arn: arn:aws:iam::<ACCOUNT_ID>:role/cltest-ebs-csi-controller-irsa

✅ Inspect IAM Role and policies

aws iam get-role --role-name cltest-ebs-csi-controller-irsa
aws iam list-attached-role-policies --role-name cltest-ebs-csi-controller-irsa

Expected:

"PolicyName": "AmazonEBSCSIDriverPolicy"

✅ Evaluate IAM policies

You can also simulate the action to confirm permissions:

aws iam simulate-principal-policy \
  --policy-source-arn arn:aws:iam::<ACCOUNT_ID>:role/cltest-ebs-csi-controller-irsa \
  --action-names ec2:AttachVolume ec2:DetachVolume \
  --resource-arns arn:aws:ec2:<REGION>:<ACCOUNT_ID>:instance/* \
  --region <REGION>

Expected:

"EvalDecision": "allowed"

🔐 IAM Inline Policy

Make sure this inline policy is also present:

resource "aws_iam_role_policy" "ebs_volume_permissions" {
  name = "allow-ebs-volume-ops"
  role = aws_iam_role.ebs_csi_controller_irsa.name

  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Sid    = "EBSVolumeActions",
        Effect = "Allow",
        Action = [
          "ec2:AttachVolume",
          "ec2:DetachVolume"
        ],
        Resource = [
          "arn:aws:ec2:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:volume/*",
          "arn:aws:ec2:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:instance/*"
        ]
      }
    ]
  })
}

💡 Use aws iam get-role-policy --role-name <name> --policy-name allow-ebs-volume-ops to verify.

🎉 Outcome

Once the proper inline policy was added, everything worked. Volumes were attached and Prometheus stopped showing PVC binding issues.

👋 Final Thoughts

The EBS CSI Driver depends on your IAM plumbing being just right. While AmazonEBSCSIDriverPolicy covers most use cases, sometimes additional permissions are needed.

Check your trust policies, policy attachments, and don't forget: inline IAM policies can help patch IAM gaps when needed.

Happy debugging!

---

🧪 Quick Summary of Checks

Check	Command	Expected Result
Controller logs	kubectl logs -n kube-system -l app.kubernetes.io/name=aws-ebs-csi-driver -c ebs-plugin --tail=100	No UnauthorizedOperation errors
CSINode registration	kubectl get csinode	Node lists ebs.csi.aws.com driver
Tolerations	`kubectl get daemonset ebs-csi-node -n kube-system -o jsonpath='{.spec.template.spec.tolerations}' \	jq`
IRSA annotation	kubectl describe sa ebs-csi-controller-sa -n kube-system	Has eks.amazonaws.com/role-arn annotation
IAM role policies	aws iam list-attached-role-policies	Contains AmazonEBSCSIDriverPolicy
IAM simulation	aws iam simulate-principal-policy	Shows "EvalDecision": "allowed"