DEV Community

Someswar Duddi
Someswar Duddi

Posted on

Hardening an AWS EKS cluster: IRSA, NetworkPolicies, restricted Pod Security and Secrets Manager via CSI

#aws #kubernetes #security #devops

A freshly created EKS cluster works, but it's permissive. Every pod can reach every other pod, workloads can run as root, and the easy way to give an application AWS access is to attach permissions to the node role, which hands them to every pod on that node.

This post works through four controls that close those gaps, using a single example workload: an orders-api deployment in the orders namespace that needs a database password from AWS Secrets Manager.

  1. IRSA — per-workload IAM roles instead of node-wide permissions.
  2. NetworkPolicies — default-deny traffic, then allow only what's needed.
  3. Pod Security Admission (restricted) — block privileged and root containers at admission time.
  4. Secrets Store CSI driver — mount Secrets Manager secrets as files, rather than copying them into Kubernetes Secrets by hand.

The examples assume eksctl, kubectl, helm and the AWS CLI, and a cluster called prod-eu in eu-west-2.

1. IAM Roles for Service Accounts (IRSA)

IRSA lets a Kubernetes service account assume an IAM role. The pod receives a projected, short-lived token, and the AWS SDK exchanges it for temporary credentials. No access keys, and no permissions on the node.

First, register the cluster's OIDC issuer with IAM (once per cluster):

eksctl utils associate-iam-oidc-provider --cluster prod-eu --region eu-west-2 --approve
Enter fullscreen mode Exit fullscreen mode

Next, a least-privilege policy. This one can read exactly one secret:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret"],
      "Resource": "arn:aws:secretsmanager:eu-west-2:123456789012:secret:prod/orders/db-??????"
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

The -?????? suffix matters: Secrets Manager appends six random characters to every secret ARN. If the secret is encrypted with a customer-managed KMS key, the role also needs kms:Decrypt on that key.

Then create the role and service account together:

aws iam create-policy --policy-name orders-api-secrets \
  --policy-document file://orders-api-secrets.json

eksctl create iamserviceaccount \
  --cluster prod-eu --region eu-west-2 \
  --namespace orders --name orders-api \
  --role-name orders-api-irsa \
  --attach-policy-arn arn:aws:iam::123456789012:policy/orders-api-secrets \
  --approve
Enter fullscreen mode Exit fullscreen mode

eksctl creates a role whose trust policy only accepts tokens for that specific service account:

"Condition": {
  "StringEquals": {
    "oidc.eks.eu-west-2.amazonaws.com/id/EXAMPLE0123456789:sub": "system:serviceaccount:orders:orders-api",
    "oidc.eks.eu-west-2.amazonaws.com/id/EXAMPLE0123456789:aud": "sts.amazonaws.com"
  }
}
Enter fullscreen mode Exit fullscreen mode

and annotates the service account with eks.amazonaws.com/role-arn. If you manage the service account in Git rather than letting eksctl own it, add that annotation yourself.

Two follow-ups make IRSA meaningful:

  • Strip the node role back to what the kubelet and CNI need. Anything extra there is available to every pod.
  • Block pods from the instance metadata service so they can't fall back to node credentials. Set the IMDSv2 hop limit to 1 on your launch template (or disablePodIMDS: true in an eksctl node group).

EKS Pod Identity is the newer alternative and avoids per-cluster OIDC setup and trust-policy editing. IRSA still works everywhere, including self-managed tooling that predates Pod Identity, which is why it's shown here; the rest of this post works the same with either.

2. NetworkPolicies

Kubernetes allows all pod-to-pod traffic by default. NetworkPolicies change that, but only if something enforces them. The Amazon VPC CNI can do this natively; enable it on the add-on:

aws eks update-addon --cluster-name prod-eu --region eu-west-2 \
  --addon-name vpc-cni \
  --configuration-values '{"enableNetworkPolicy": "true"}'
Enter fullscreen mode Exit fullscreen mode

Start with default-deny for the namespace, in both directions:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: orders
spec:
  podSelector: {}
  policyTypes: ["Ingress", "Egress"]
Enter fullscreen mode Exit fullscreen mode

Everything in orders is now isolated, including DNS, which is the usual first surprise. Allow it back:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: orders
spec:
  podSelector: {}
  policyTypes: ["Egress"]
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - { protocol: UDP, port: 53 }
        - { protocol: TCP, port: 53 }
Enter fullscreen mode Exit fullscreen mode

Then open the specific paths orders-api needs: inbound from the ingress controller, and outbound to the database subnet.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: orders-api
  namespace: orders
spec:
  podSelector:
    matchLabels:
      app: orders-api
  policyTypes: ["Ingress", "Egress"]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: ingress-nginx
      ports:
        - { protocol: TCP, port: 8080 }
  egress:
    - to:
        - ipBlock:
            cidr: 10.0.64.0/20   # private database subnets
      ports:
        - { protocol: TCP, port: 5432 }
Enter fullscreen mode Exit fullscreen mode

The trade-off is operational: every new dependency now needs a policy change, and a missing rule shows up as a timeout rather than a clear error. Roll out per namespace, and keep policies next to the application's manifests so they change together.

3. Pod Security Admission: restricted

Pod Security Admission is built into Kubernetes and is configured with namespace labels. The restricted profile rejects pods that run as root, allow privilege escalation, keep Linux capabilities, or skip a seccomp profile.

Turn on warn and audit first to see what would break, then enforce:

kubectl label namespace orders \
  pod-security.kubernetes.io/enforce=restricted \
  pod-security.kubernetes.io/enforce-version=latest \
  pod-security.kubernetes.io/warn=restricted \
  pod-security.kubernetes.io/audit=restricted \
  --overwrite
Enter fullscreen mode Exit fullscreen mode

A deployment that passes looks like this. The securityContext blocks are the part that matters:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-api
  namespace: orders
spec:
  replicas: 2
  selector:
    matchLabels: { app: orders-api }
  template:
    metadata:
      labels: { app: orders-api }
    spec:
      serviceAccountName: orders-api
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
        seccompProfile: { type: RuntimeDefault }
      containers:
        - name: app
          image: 123456789012.dkr.ecr.eu-west-2.amazonaws.com/orders-api:1.4.2
          ports: [{ containerPort: 8080 }]
          env:
            - name: SPRING_CONFIG_IMPORT
              value: "optional:configtree:/mnt/secrets/"
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities: { drop: ["ALL"] }
          volumeMounts:
            - { name: tmp, mountPath: /tmp }
            - { name: db-secrets, mountPath: /mnt/secrets, readOnly: true }
      volumes:
        - name: tmp
          emptyDir: {}
        - name: db-secrets
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: orders-db
Enter fullscreen mode Exit fullscreen mode

readOnlyRootFilesystem isn't required by restricted, but it's cheap. A Spring Boot app with embedded Tomcat needs a writable /tmp, hence the emptyDir.

Leave kube-system and other system namespaces at privileged or baseline; CNI, CSI and logging agents legitimately need host access.

4. Secrets Manager via the Secrets Store CSI driver

The CSI driver mounts secrets into the pod at start-up, fetched using the pod's own IAM identity, which is the IRSA role from step 1. Install the driver and the AWS provider:

helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
helm repo add aws-secrets-manager https://aws.github.io/secrets-store-csi-driver-provider-aws

helm install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver \
  --namespace kube-system

helm install secrets-provider-aws aws-secrets-manager/secrets-store-csi-driver-provider-aws \
  --namespace kube-system
Enter fullscreen mode Exit fullscreen mode

Then describe which secret to mount. The secret prod/orders/db holds JSON with username and password keys; jmesPath splits them into separate files:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: orders-db
  namespace: orders
spec:
  provider: aws
  parameters:
    objects: |
      - objectName: "prod/orders/db"
        objectType: "secretsmanager"
        jmesPath:
          - path: username
            objectAlias: spring.datasource.username
          - path: password
            objectAlias: spring.datasource.password
Enter fullscreen mode Exit fullscreen mode

The aliases are deliberate. With SPRING_CONFIG_IMPORT=optional:configtree:/mnt/secrets/, Spring Boot treats each file name as a property name and its contents as the value, so the datasource is configured with no code and no environment variables holding secrets.

Trade-offs to know about:

  • Rotation isn't automatic end to end. The driver can refresh mounted files (enableSecretRotation=true on the Helm chart), but Spring reads config trees at start-up. You'll need a rolling restart after rotation, or a connection pool that re-reads credentials.
  • Syncing to Kubernetes Secrets (secretObjects, with syncSecret.enabled=true) is available if you need environment variables, but it puts the value back in etcd. If you do, make sure envelope encryption with KMS is enabled on the cluster.
  • Pods won't start if the secret can't be fetched. That's usually what you want, but it makes Secrets Manager and STS part of your start-up path.

Summary

None of these controls is exotic, and together they remove the most common EKS footguns:

  • IRSA (or Pod Identity) gives each workload its own narrowly scoped IAM role; strip the node role and block IMDS so pods can't bypass it.
  • Default-deny NetworkPolicies, with explicit allows for DNS and real dependencies.
  • restricted Pod Security on application namespaces, rolled out via warn and audit first.
  • Secrets mounted from Secrets Manager using the workload's own identity, with a clear plan for rotation.

Top comments (0)