Implement multi-tenancy on Amazon EKS Auto Mode clusters

The latest CNCF Survey in 2025 highlighted the increasing adoption of containers within the enterprise, with 92% of organisations now using containers in production. Of these, 82% of them run Kubernetes, which has rapidly become the industry standard. Overall, 93% of organisations are now using, piloting, or evaluating Kubernetes, with 79% of these running managed services from the hyperscalers.

In an enterprise with many engineering teams looking to containerise their applications, a common challenge is whether to run these on a single Kubernetes cluster. This blog post focuses on the options that exist for implementing multi-tenancy on an Amazon EKS Auto Mode cluster.

Hard versus Soft Isolation Boundaries

In this blog post, we want to look at the options for two workload teams to deploy their containerised application onto Amazon EKS. Amazon EKS is a fully upstream and certified conformant and compliant version of Kubernetes. Kubernetes provides a single shared control plane and supports soft multi-tenancy through a number of isolation mechanisms. This means that a single instance of the control plane is shared among all the tenants within a cluster.

The diagram below shows the main components that are running when two pods are deployed on the same worker node on Amazon EKS.

This highlights that containers on the same worker node share much of the underlying operating system, including the Linux kernel, node networking and storage. However, even if these pods were deployed on different nodes, they will still be sharing the same control plane and therefore components like etcd which provides the backing datastore for Kubernetes state across the cluster.

The only way to guarantee hard isolation between the applications of the two workload teams is to deploy them onto separate Amazon EKS clusters in separate AWS accounts. By default, resources in one AWS account cannot access resources in another AWS account, limiting the blast radius of a misconfiguration or malicious action.

However, there is a significant operational and cost overhead of having each workload team manage its own Amazon EKS cluster. This has led to many organisations establishing a platform team, and allowing applications from different workload teams to be hosted on this central Amazon EKS platform. The rest of this blog post looks at the controls that can be put in place to achieve logical soft isolation between these applications. These controls are part of a defence in depth strategy.

This post breaks down this defence in depth strategy into the following layers:

• Layer 1 — Account boundary and Service Control Policies
• Layer 2 — Namespace, Quotas, and Pod Security isolation
• Layer 3 — Pod Identity, role chaining, and ABAC
• Layer 4 — Network isolation
• Layer 5 — Policy as code with Kyverno
• Layer 6 — GitOps deployment isolation
• Layer 7 — Observability and audit isolation

Layer 1 — Account boundary and Service Control Policies

AWS recommend using a multi-account strategy with AWS Organizations to help isolate and manage business applications and data.

Each of the workload teams have their own set of AWS resources required as part of their overall application e.g. Amazon S3 buckets, RDS databases and DynamoDB tables. These resources are hosted within the AWS account of the individual workload team. This limits the shared resources to the compute capability of Amazon EKS.

This is reinforced by the use of Service Control Policies (SCP). SCPs offer central control over the maximum available permissions for users and roles in an AWS organization. We create and apply the following two SCPs as examples:

• Prevent any non-EKS principal from setting the Pod Identity session tags (kubernetes-service-account, kubernetes-namespace, eks-cluster-arn) that our access model depends on
• Deny EKS cluster creation in any workload account, ensuring that clusters are centralised

Layer 2 — Namespace, Quotas, and Pod Security isolation

The standard practice to support soft multi-tenancy on EKS is to align with Kubernetes namespaces as a mechanism for isolating groups of resources. Namespaces allow you to divide the cluster into logical partitions. Quotas, network policies, service accounts and several other objects are all scoped to a namespace. In our example, workload team 1 and workload team 2 are assigned their own dedicated namespace.

Pod Security Admission (PSA) is a Kubernetes built-in admission controller that enforces Pod Security Standards. It has three levels:

• privileged - largely unrestricted and intended for trusted system workloads.
• baseline - blocks known privilege escalations
• restricted - the most hardened built-in profile

PSA is enabled per namespace via labels. We apply this when creating the namespace as follows:

apiVersion: v1
kind: Namespace
metadata:
  name: team-1
  labels:
    team: team-1
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: latest
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/audit: restricted

PSA governs what a pod is allowed to do when it is admitted to the cluster. It does not control which workloads can communicate with one another or what AWS resources they can access. In addition to PSA, we also extend the use of compliance-as-code with Kyverno, which we discuss later on in the blog.

We also define ResourceQuota and LimitRange to prevent the noisy neighbour problem where one team could scale their deployment and starve other workloads of nodes and memory.

ResourceQuota provides a hard ceiling on how much of the cluster's shared resources this namespace is allowed to consume. The request values (requests.cpu and requests.memory) limit the aggregate CPU and memory requests that all pods in the namespace may declare. The limits values (limits.cpu and limits.memory) define the maximum value for the sum of all limits in the namespace. In our example we allow workloads to burst above their guaranteed resources while still reserving only 8 vCPUs of scheduler capacity. We also show how we can limit the maximum number of pods that can exist and the maximum number of Kubernetes Services that can exist at any one time.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-1-quota
  namespace: team-1
spec:
  hard:
    requests.cpu: "8"
    requests.memory: 16Gi
    limits.cpu: "16"
    limits.memory: 32Gi
    pods: "50"
    services: "10"

LimitRange defines the minimum, maximum and default request/limit values per container, preventing workloads from requesting unreasonably small or large amounts of CPU and memory. The default values apply if a pod has not specified any limits.

apiVersion: v1
kind: LimitRange
metadata:
  name: team-1-limits
  namespace: team-1
spec:
  limits:
    - type: Container
      default:
        cpu: 500m
        memory: 512Mi
      defaultRequest:
        cpu: 100m
        memory: 128Mi
      max:
        cpu: "4"
        memory: 8Gi
      min:
        cpu: 50m
        memory: 64Mi

In addition, we also create a Kubernetes ServiceAccount for each workload within the namespace. A service account is scoped to a namespace and not a cluster. This means in our example below, the "team-1-sa" service account only exists in the "team-1" namespace, and cannot be used by a pod in a different namespace.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: team-1-sa
  namespace: team-1
  labels:
    team: team-1

Every namespace automatically gets a default service account created for it. If you don't specify one, pods will use default. We enforce the use of named service accounts with Kyverno which we cover later in the blog.

Layer 3 - Pod Identity, role chaining and ABAC

Each application developed by the workload team and running on the central EKS cluster needs to access AWS resources that are running in the workload team's AWS account. Historically, this was achieved using IAM Roles for Service Accounts (IRSA), which allowed you to deliver temporary AWS credentials to workloads running on EKS. This also requires enabling cross-account access and setting up an IAM OIDC provider.

At re:Invent 2023, AWS launched EKS Pod Identities as a simpler way of delivering temporary AWS credentials to your pods running on EKS. EKS Pod Identities integrate with the EKS control plane and on-cluster agent so that pods receive credentials without requiring you to create or manage an IAM OIDC identity provider. EKS Pod Identities are the AWS recommended approach for new workloads on supported node types, and is the approach adopted in this blog post.

With EKS Pod Identity, you associate a Kubernetes service account in your cluster with an IAM role in the same AWS account as the cluster. EKS uses this association to obtain temporary credentials on behalf of the pod for that IAM role and securely deliver them to pods to use the service account.

EKS Pod Identities natively support cross account access by using a target IAM role in the workload account and IAM role chaining. When you create a Pod Identity association for a Kubernetes service account, you specify both a pod IAM role in the cluster account and a target IAM role in the workload account. EKS Pod Identity uses the pod role to assume the target role and returns temporary credentials for the target role to the pod.

We set this up in terraform in the platform account as follows:

Firstly we create an IAM role called "pod_role_team_1". This has a trust policy that means only the EKS Pod Identity service is allowed to assume the role and attach session tags.

resource "aws_iam_role" "pod_role_team_1" {
  name = "pod-role-team-1"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Service = "pods.eks.amazonaws.com" }
      Action    = ["sts:AssumeRole", "sts:TagSession"]
    }]
  })
  tags = { Team = "team-1" }
}

We then create a policy that has only the permissions to assume the "Team1PodTargetRole" in the workload account and attach session tags. This permission is intentionally minimal to restrict the blast radius of it being compromised.

resource "aws_iam_policy" "pod_role_team_1_assume" {
  name = "pod-role-team-1-assume-target"
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect   = "Allow"
      Action   = ["sts:AssumeRole", "sts:TagSession"]
      Resource = "arn:${local.partition}:iam::${var.workload_team_1_account_id}:role/Team1PodTargetRole"
    }]
  })
}

The next step is to attach the policy to the "pod_role_team_1" role.

resource "aws_iam_role_policy_attachment" "pod_role_team_1_assume" {
  role       = aws_iam_role.pod_role_team_1.name
  policy_arn = aws_iam_policy.pod_role_team_1_assume.arn
}

Finally, we create a Pod Identity association. This maps the "team-1-sa" service account in the "team-1" namespace to the "pod_role_team_1" role, and crucially also specifies the target role in the workload account. Because we set target_role_arn, EKS Pod Identity performs the role chaining itself — it assumes the pod role, then assumes the target role, and delivers credentials for the target role directly to the pod.

resource "aws_eks_pod_identity_association" "team_1" {
  cluster_name    = module.eks.cluster_name
  namespace       = "team-1"
  service_account = "team-1-sa"
  role_arn        = aws_iam_role.pod_role_team_1.arn
  # Native cross-account role chaining — EKS assumes this target role for the pod
  target_role_arn = "arn:${local.partition}:iam::${var.workload_team_1_account_id}:role/Team1PodTargetRole"
}

This works by creating an association stored by the EKS control plane. On standard EKS clusters, Pod Identity requires the EKS Pod Identity Agent add-on. With EKS Auto Mode, this capability is built into the managed nodes, so there is no add-on or DaemonSet to install or manage. When a pod makes an AWS SDK request and requires credentials, the agent reads the pod's service account and namespace and queries EKS for a matching association. Finding one, EKS performs two assume-role calls in sequence on the pod's behalf:

1. It assumes pod-role-team-1 (the pod role in the cluster account).
2. Using those credentials, it assumes Team1PodTargetRole in the workload account, attaching the Kubernetes session tags as it does so.

The pod never makes either call — it simply receives the final target-role credentials. The second assume-role looks effectively like:

sts:AssumeRole(
  RoleArn = "arn:aws:iam::<workload-account>:role/Team1PodTargetRole",
  RoleSessionName = "eks-...",
  Tags = [                       # set by EKS via sts:TagSession on this call
    { Key: "kubernetes-namespace",       Value: "team-1" },
    { Key: "kubernetes-service-account", Value: "team-1-sa" },
    { Key: "eks-cluster-name",           Value: "shared-platform-cluster" },
    { Key: "eks-cluster-arn",            Value: "arn:aws:eks:..." }
  ]
)

As EKS sets these tags on the call the assumes the target role, the workload account validates them as aws:requestTag/.. (the tags on the incoming request), and locks the caller down to the pod role with aws:PrincipalArn.

In our workload account, we create the target role that EKS Pod Identity chains into. Its trust policy allows only the pod-role-team-1 role (via aws:PrincipalArn) to assume it — no other AWS principal can. The condition then validates the session tags that EKS attaches when it assumes the role: the pod must be running under the specific service account, in the specified namespace, on the exact EKS cluster. Because EKS sets these tags on the assume-role request itself, we match them with aws:RequestTag/... (not aws:PrincipalTag). This is what makes the authorization attribute-based (ABAC) rather than identity-based.

resource "aws_iam_role" "pod_target_role" {
  name = "Team1PodTargetRole"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Principal = {
        AWS = "arn:${local.partition}:iam::${var.platform_account_id}:root"
      }
      Action = [
        "sts:AssumeRole",
        "sts:TagSession"
      ]
      Condition = {
        # ABAC: validate the session tags EKS sets when assuming this role
        StringEquals = {
          "aws:RequestTag/kubernetes-service-account" = "${var.team_name}-sa"
          "aws:RequestTag/kubernetes-namespace"       = var.team_name
          "aws:RequestTag/eks-cluster-arn"            = var.cluster_arn
        }
        # Lock the caller down to exactly the pod role
        ArnEquals = {
          "aws:PrincipalArn" = var.pod_role_arn
        }
      }
    }]
  })
}

In our workload account, we can then assign a policy to this role to give it the permissions required to interact with any required AWS resources. In this case, it is just to a DynamoDB table.

The following sequence diagram highlights how this all hangs together.

With the native target-role flow, the application code needs no STS calls at all — boto3 picks up the target-role credentials from Pod Identity automatically, and the pod talks to DynamoDB directly.

Layer 4 — Network isolation

By default, all pods in a Kubernetes cluster can communicate freely — there is no restriction on pod-to-pod traffic until you add a NetworkPolicy. As soon as a policy selects a pod, the default flips: anything not explicitly allowed is denied. NetworkPolicies operate at L3/L4 (IP, port, and pod/namespace selectors)
— they cannot match an AWS resource identity such as a security group or an ALB.

One thing we discovered was that with EKS Auto Mode clusters, the Network Policy Controller is off by default. This means NetworkPolicy objects are accepted by the Kubernetes API but not enforced by the data plane, allowing cross-namespace traffic to flow freely and isolation silently fails. We enable the Network Policy Controller by applying a ConfigMap. Enforcement is then handled on each mode by an eBPF agent.

apiVersion: v1
kind: ConfigMap
metadata:
  name: amazon-vpc-cni
  namespace: kube-system
data:
  enable-network-policy-controller: "true"

Ingress — who can reach team-1 pods

We apply a Network Policy that allows pods within team-1 to talk to each other. Everything else — including other namespaces — is denied. In our example, we have a Deployment that tells Kubernetes to run 2 replicas of the team 1 app container. We expose the Pods through a Service, giving the workload a stable ClusterIP and DNS name. This is carried out as we test the applications locally using port forwarding.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-cross-namespace-ingress
  namespace: team-1
spec:
  podSelector: {}
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector: {}

If we wanted to expose the workload externally via an Application Load Balancer, we would need to allow ingress from the ALB. To enforce that only the ALB could reach these pods in this namespace, we would use Security Groups for Pods.

This is carried out by putting the pods behind their own EC2 security group, so the network interface becomes governed by security group rules just like an EC2 instance. You can achieve this with EKS Auto Mode by setting the podSecurityGroupSelectorTerms on the NodeClass, and then EKS Auto Mode will attach the selected group to the pods branch network interface. This looks as follows:

apiVersion: eks.amazonaws.com/v1
kind: NodeClass
metadata:
  name: team-workloads
spec:
  # ... subnet / role config ...
  podSecurityGroupSelectorTerms:
    - matchLabels:
        Name: team-1-pod-sg     # selects the SG that allows ingress only from the ALB SG

The selected security group would allow inbound 8080 only from the ALB's security group. This provides the identity-based control that you cannot express with just NetworkPolicy. This means you can separate the division of responsibility as:

• NetworkPolicy — pod-to-pod and namespace isolation inside the cluster (L3/L4).
• Security groups for pods (via NodeClass on Auto Mode) — AWS-resource-level control (ALB→pod, pod→RDS), using security group identity rather than IP ranges.

Egress — what can team-1 pods reach out too

Locking down egress for the team-1 pods turned out to be an interesting experience. Originally, a default-deny egress policy still allowed team-1 pods to access DynamoDB in the workload account. However, this was before the Network Policy Controller had been enabled. This broke the application in a series of timeout errors, which we were slowly able to work through. The fix was to treat egress as an explicit allow-list of the platform plumbing the pod depends on, and then layer the applications own destinations on top.

The Network Policy for egress (ignoring DynamoDB which we deal with separately) is as follows:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-and-aws-egress
  namespace: team-1
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    # DNS — the cluster DNS service IP (CoreDNS is node-local; see note)
    - to:
        - ipBlock:
            cidr: 172.20.0.10/32
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53
    # EKS Pod Identity Agent — node-local, link-local. REQUIRED for any AWS
    # access: the SDK fetches credentials from http://169.254.170.23/v1/credentials
    # (ports 80 and 2703). Without this the whole credential chain fails (see note).
    - to:
        - ipBlock:
            cidr: 169.254.170.23/32
      ports:
        - protocol: TCP
          port: 80
        - protocol: TCP
          port: 2703
    # Same-namespace pod-to-pod, incl. the team's own ClusterIP service. The
    # service CIDR is needed because the eBPF egress hook evaluates the pre-DNAT
    # service IP; cross-namespace is still blocked at the destination's ingress.
    - to:
        - podSelector: {}
        - ipBlock:
            cidr: 172.20.0.0/16       # cluster service CIDR
    # AWS Interface Endpoints (STS, ECR, CloudWatch Logs) live in the private
    # subnets. With private DNS enabled, sts.<region>.amazonaws.com resolves to a
    # private VPC IP, so this stays internal — no internet egress.
    - to:
        - ipBlock:
            cidr: 10.0.0.0/16         # platform VPC — where endpoint ENIs live
      ports:
        - protocol: TCP
          port: 443

Before EKS Auto Mode, CoreDNS would run as a regular Kubernetes pod in the kube-system namespace and labelled as k8s-app: kube-dns. This meant you could use a pod selector such as k8s-app: kube-dns as a rule to allow workloads to perform DNS lookups while still having a default-deny egress rule. With EKS Auto Mode, the Auto Mode nodes use CoreDNS running as a system service directly on each node. Each pod is configured through /etc/resolv.conf to send DNS queries to the cluster DNS service IP, which in our case was 172.20.0.10. Every Pod receives a /etc/resolv.conf file when it starts, which tells the operating system's DNS resolver which nameserver to use. We match that in our Network Policy using an IP block and allowing traffic via port 53 for DNS queries.

With EKS Auto Mode, the Pod Identity capability is provided as a built-in node-local component rather than a DaemonSet you install or manage. It exposes a link-local HTTP endpoint at 169.254.170.23 on ports 80 and 2703 that is reachable only by pods on the same node. Because the address is link-local, the request never leaves the node. Every node runs its own instance of this capability, so each pod only ever talks to the component on the node it is scheduled on. We need to allow egress to this address to allow the AWS SDK in our application code to fetch credentials.

Our team-1 app needs to access DynamoDB which is exposed via a Gateway Endpoint. The pod resolves the public DynamoDB hostname, so the initial packet is destined for one of DynamoDB's public IP addresses. The eBPF egress hook evaluates that packet before the VPC route table redirects it via the Gateway Endpoint. These IP's rotate, so there is no stable CIDR range to match. EKS Auto Mode has an ApplicationNetworkPolicy (an AWS CRD that extends NetworkPolicy with a domainNames filter). EKS Auto Mode uses the domainNames rule to allow egress to the IP addresses resolved for that FQDN.

apiVersion: networking.k8s.aws/v1alpha1
kind: ApplicationNetworkPolicy
metadata:
  name: allow-dynamodb-egress       # must NOT clash with the NetworkPolicy name
  namespace: team-1
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - domainNames:
            - "dynamodb.eu-west-1.amazonaws.com"
      ports:
        - protocol: TCP
          port: 443

Layer 5 — Policy as code with Kyverno

Pod Security Admission (PSA) provides Kubernetes built-in enforcement of the Pod Security Standards, protection against common privilege escalation risks such as privileged containers, host networking and hostPath mounts. This sets out a minimum security baseline, on top of which we add Kyverno. Kyverno is a general-purpose policy engine. We use it to enforce specific standards that PSA cannot express:

• Restricting container images to approved registries (e.g. Amazon ECR)
• Mandating CPU/memory limits
• Requiring team/app labels
• Forbidding the use of the default service account so every workload runs under an explicit least-privilege identity

We also use it as defense-in-depth, re-asserting key pod-security controls (non-root, no privileged containers, no host namespaces) so we are not relying on a single enforcement mechanism.

An example of one of our Kyverno policies is shown below:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-default-serviceaccount
spec:
  validationFailureAction: Enforce
  rules:
    - name: deny-default-sa
      match:
        any:
          - resources:
              kinds: [Pod]
              namespaces: [team-1, team-2]
      validate:
        message: "Pods must not use the 'default' service account"
        pattern:
          spec:
            =(serviceAccountName): "!default"

This policy is used to ensure all pods in the team-1 and team-2 namespace name an explicit service account. Any pod that requested the default service account (or did not specify one at all) would be rejected before it was ever scheduled. Enforcing named ServiceAccounts ensures every workload has an explicit identity, allowing us to map it to the correct IAM role using EKS Pod Identity. EKS Pod Identity maps a Kubernetes ServiceAccount (within a namespace) to a per-team IAM role.

The use of Kyverno is an example of policy as code. By expressing these requirements as policies rather than documentation, every deployment is validated consistently by the Kubernetes API server, preventing non-compliant workloads from ever entering the cluster.

Layer 6 — GitOps deployment isolation

With a multi-tenant EKS cluster running in a platform account, it was critical to limit access to the account. Workload teams are given no capability to run kubectl against the cluster, or to assume a role into the platform account. They deploy to the cluster by committing manifests to their own Git repository, and Argo CD (running as a managed EKS capability) reconciles the cluster to match. Argo CD implements a GitOps workflow where you define your application configurations in Git repositories and Argo CD automatically syncs your applications to match the desired state. For application deployments, the only identity that applies workload manifests to the cluster is the EKS Capability for Argo CD. The workload team is restricted to the Argo CD AppProject. This is where the isolation lives, and it constrains the workload team as follows:

• Source - the AppProject only permits applications sourced from the approved "team-1-gitops" repository. A team cannot point Argo CD at an arbitrary repository, and cannot sync a manifest that lives in someone else's.
• Destination - each project is pinned to a single namespace on the one registered cluster. Even if team-2 specified a namespace of team-1, it would be refused when it was synced under their project.
• Resource kinds - each project allow-lists only the resource kinds that an application requires such as Deployments, Services and ConfigMaps. Cluster-scoped resources are locked down, so a workload team cannot create their own ClusterRole or Namespace, or a CRD to escalate their privileges.

The applications sync with selfHeal: true and prune: true, which means Git acts as the source of truth. selfHeal automatically corrects drift if resources are modified outside Git, while prune removes Kubernetes resources that have been deleted from the Git repository. If the live state drifts from what has been committed to Git, Argo CD automatically reconciles it back to the desired state. This means every application change is auditable as a Git commit, with Git acting as the single source of truth for the deployed configuration.

Because we use the Amazon EKS Capability for Argo CD, AWS manages the operational aspects of Argo CD—including upgrades, high availability and scaling—allowing the platform team to focus on defining deployment policies rather than operating the GitOps platform itself

Layer 7 — Observability and audit isolation

Alongside preventing different workloads from affecting each other, it is equally important to ensure that operational data such as logs, metrics, traces and audit records are also appropriately isolated. Each workload team should be able to troubleshoot their own application without gaining visibility into another team's logs or operational data.

Application telemetry should be partitioned by tenant or namespace to ensure each team only has visibility into its own workloads. Application telemetry is commonly collected using Fluent Bit for logs and the AWS Distro for OpenTelemetry (ADOT), or another OpenTelemetry Collector, for metrics and traces. These collectors can route telemetry to a variety of backends, including Amazon CloudWatch, Amazon Managed Service for Prometheus, Amazon OpenSearch Service, Grafana, Elastic or third-party observability platforms. Regardless of the backend, logs, metrics and traces should be partitioned using Kubernetes metadata attributes such as namespace, workload, service or tenant identifiers, with access enforced by the observability platform's authorisation model.

At the platform level, Amazon EKS control plane audit logging provides a complete audit trail of every request made to the Kubernetes API server. These audit logs capture actions such as creating or deleting workloads, modifying namespaces, changing RBAC policies and accessing Kubernetes resources. Unlike application logs, audit logs are intended for the platform and security teams, providing cluster-wide visibility for operational monitoring, compliance and forensic investigations. AWS CloudTrail complements Kubernetes audit logging by recording AWS API activity, including IAM role assumptions made through EKS Pod Identity and access to AWS services. Together, Kubernetes audit logs and CloudTrail provide a complete audit trail spanning both the Kubernetes control plane and the AWS control plane.

Testing our Isolation Constraints

We are able to run a series of tests, to prove how effective a number of these layers are in providing secure isolation between tenants. Each of the following tests deliberately attempts to violate one of the isolation boundaries described earlier. The expected outcome is that the request is denied by the appropriate layer, demonstrating that no single control is relied upon in isolation.

Test 1a: Cross network traffic between namespace (expected failure)

This test checks that a pod within the team-1 namespace cannot open a TCP socket to a service inside the team-2 namespace. This results in a timeout error as the connection is dropped by the NetworkPolicy that denies all cross namespace ingress. Because a NetworkPolicy drop is silent, the connection attempt hangs until it is finally times out.

multi-account-eks-demo % kubectl exec -n team-1 deploy/team-1-app -i -- python - <<'PY'
import socket
socket.setdefaulttimeout(5)
socket.create_connection(('team-2-app.team-2.svc.cluster.local', 80))
print('CONNECTED')
PY
Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
  File "/usr/local/lib/python3.12/socket.py", line 865, in create_connection
    raise exceptions[0]
  File "/usr/local/lib/python3.12/socket.py", line 850, in create_connection
    sock.connect(sa)
TimeoutError: timed out
command terminated with exit code 1

Test 1b: Cross network traffic within namespace (expected success)

This test runs the same code as the previous test, but checking whether a pod within the team-1 namespace can open a TCP socket to a service inside its own namespace. This results in a "CONNECTED" response.

multi-account-eks-demo % kubectl exec -n team-1 deploy/team-1-app -i -- python - <<'PY'
import socket
socket.setdefaulttimeout(5)
socket.create_connection(('team-1-app.team-1.svc.cluster.local', 80))
print('CONNECTED')
PY
CONNECTED

Test 2a: Cross account IAM role chaining to own workload account (expected success)

This test confirms that the cross account IAM role chaining detailed in Layer 3 works as described. The test executed a script in a pod running in the "team-1" namespace, and returns the IAM identity being used to make the call. The returned identity shows that the pod is operating as the target role in the Team 1 workload account (not the platform account)

multi-account-eks-demo % kubectl exec -n team-1 deploy/team-1-app -i -- python - <<'PY'
import boto3, os
# Use the REGIONAL STS endpoint. A bare boto3.client('sts') targets the global
# sts.amazonaws.com (a public IP) which egress intentionally blocks, so it hangs.
# The regional host resolves to the private STS interface endpoint inside the VPC.
sts = boto3.client('sts', region_name=os.environ['AWS_REGION'])
print('IDENTITY:', sts.get_caller_identity()['Arn'])
PY
IDENTITY: arn:aws:sts::169928422290:assumed-role/Team1PodTargetRole/eks-shared-pla-team-1-app-51fa2855-9605-4a5e-95bd-3be8e8754e6a

Test 2b: Cross account IAM role chaining to another workload account (expected failure)

This test attempts to assume the Team 2 target role in the Team 2 workload account. It returns with Access Denied. This request is denied by the trust policy on the target role which will only allow an assume role if it comes from the Team 2 pod role. The trust policy also requires ABAC session tags that identify the caller as coming from the "team-2" namespace and the "team-2-sa" ServiceAccount.

multi-account-eks-demo % kubectl exec -n team-1 deploy/team-1-app -i -- python - <<PY
import boto3, botocore, os
try:
    boto3.client('sts', region_name=os.environ['AWS_REGION']).assume_role(
        RoleArn='arn:aws:iam::${ACCOUNT_3_ID}:role/Team2PodTargetRole',
        RoleSessionName='cross-team-attempt')
    print('ASSUMED - unexpected')
except botocore.exceptions.ClientError as e:
    print('DENIED:', e.response['Error']['Code'])
PY
DENIED: AccessDenied

Test 3: GitOps deployment into another team's namespace

This test attempts to deploy into the Team 2 namespace through the GitOps pipeline. It creates an Argo CD Application under the "team-1" project, sourced from Team 1's own repo, but with its destination set to the "team-2" namespace. The Application object is created successfully, but Argo CD refuses to sync it and marks it InvalidSpecError. The block comes from the Team 1 AppProject, whose destinations list only permits the "team-1" namespace on the shared cluster. A destination of "team-2" matches no allowed destination and is rejected before anything lands in Team 2's namespace.

multi-account-eks-demo % kubectl apply -f - <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: cross-team-attempt
  namespace: argocd
spec:
  project: team-1
  source:
    repoURL: "https://github.com/mlewis7127/team-1-gitops"
    targetRevision: main
    path: manifests/
  destination:
    server: arn:aws:eks:eu-west-1:424727766526:cluster/shared-platform-cluster
    namespace: team-2
EOF
application.argoproj.io/cross-team-attempt created
(.venv) multi-account-eks-demo % kubectl get application cross-team-attempt -n argocd -o jsonpath='{.status.conditions}'
[{"lastTransitionTime":"2026-06-30T10:55:42Z","message":"application destination server 'arn:aws:eks:eu-west-1:424727766526:cluster/shared-platform-cluster' and namespace 'team-2' do not match any of the allowed destinations in project 'team-1'","type":"InvalidSpecError"}]%   

Test 4a: Deploying an image from an unapproved registry

This test attempts to deploy a Pod whose image comes from Docker Hub (docker.io/library/nginx) rather than an approved AWS registry. It is rejected at admission by Kyverno. The restrict-image-registries policy requires every container image to come from an approved ECR registry (*.dkr.ecr.*.amazonaws.com/* or public.ecr.aws/*), so the Docker Hub image fails validation and the Pod is never created.

multi-account-eks-demo % kubectl apply -n team-1 -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: rogue-registry
  labels: { team: team-1, app: rogue }
spec:
  serviceAccountName: team-1-sa
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    seccompProfile: { type: RuntimeDefault }
  containers:
    - name: c
      image: docker.io/library/nginx:latest      # <-- not an approved registry
      resources:
        requests: { cpu: 50m, memory: 64Mi }
        limits:   { cpu: 100m, memory: 128Mi }
      securityContext:
        runAsNonRoot: true
        allowPrivilegeEscalation: false
        capabilities: { drop: ["ALL"] }
EOF
Error from server: error when creating "STDIN": admission webhook "validate.kyverno.svc-fail" denied the request: 

resource Pod/team-1/rogue-registry was blocked due to the following policies 

restrict-image-registries:
  validate-image-registry: 'validation error: Images must come from approved ECR registries.
    rule validate-image-registry failed at path /spec/containers/0/image/'

Test 4b: Deploying a Pod without required team labels

This test attempts to deploy a Pod that is missing the mandatory team and app labels. It is rejected at admission by Kyverno. The require-team-labels policy requires every Pod in the tenant namespaces to carry both a team and an app label (used for ownership, cost allocation, and workload selection) so a Pod with no labels fails validation and is never created.

multi-account-eks-demo % kubectl apply -n team-1 -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: rogue-labels
spec:
  serviceAccountName: team-1-sa
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    seccompProfile: { type: RuntimeDefault }
  containers:
    - name: c
      image: public.ecr.aws/docker/library/busybox:1.36
      command: ["sleep", "3600"]
      resources:
        requests: { cpu: 50m, memory: 64Mi }
        limits:   { cpu: 100m, memory: 128Mi }
      securityContext:
        runAsNonRoot: true
        allowPrivilegeEscalation: false
        capabilities: { drop: ["ALL"] }
EOF
Error from server: error when creating "STDIN": admission webhook "validate.kyverno.svc-fail" denied the request: 

resource Pod/team-1/rogue-labels was blocked due to the following policies 

require-team-labels:
  require-labels: 'validation error: Pods must have ''team'' and ''app'' labels. rule
    require-labels failed at path /metadata/labels/app/'

Test 4c: Deploying a Pod using the default service account

This test attempts to deploy a Pod that runs under the default service account. It is rejected at admission by Kyverno. The disallow-default-serviceaccount policy forbids the default service account in the tenant namespaces, because every workload must run under an explicit, named service account. The service account is the anchor for the entire Pod Identity to IAM role chain. A Pod using the default ServiceAccount would not be associated with the intended Pod Identity mapping, so it is never created.

multi-account-eks-demo % kubectl apply -n team-1 -f - <<'EOF'           
apiVersion: v1
kind: Pod
metadata:       
  name: rogue-sa                      
  labels: { team: team-1, app: rogue }
spec:                                                              
  serviceAccountName: default                     # <-- not allowed
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    seccompProfile: { type: RuntimeDefault }
  containers:
    - name: c
      image: public.ecr.aws/docker/library/busybox:1.36
      command: ["sleep", "3600"]
      resources:
        requests: { cpu: 50m, memory: 64Mi }
        limits:   { cpu: 100m, memory: 128Mi }
      securityContext:
        runAsNonRoot: true
        allowPrivilegeEscalation: false
        capabilities: { drop: ["ALL"] }
EOF
Error from server: error when creating "STDIN": admission webhook "validate.kyverno.svc-fail" denied the request: 

resource Pod/team-1/rogue-sa was blocked due to the following policies 

disallow-default-serviceaccount:
  deny-default-sa: 'validation error: Pods must not use the ''default'' service account.

Test 4d: Deploying a container that does not enforce non-root at the container level

This test attempts to deploy a Pod that sets runAsNonRoot at the Pod level (enough to satisfy Pod Security Admission) but omits it on the container's own securityContext. It is rejected at admission by Kyverno. The require-non-root policy demands runAsNonRoot on the container itself, so this Pod fails validation and is never created. This is a good example of Kyverno layering a stricter check on top of the PSA baseline. The manifest would pass PSA, but our organisational policy requires the guarantee to be explicit at the container level.

multi-account-eks-demo % kubectl apply -n team-1 -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: rogue-nonroot
  labels: { team: team-1, app: rogue }
spec:
  serviceAccountName: team-1-sa
  securityContext:
    runAsNonRoot: true                            # pod-level satisfies PSA
    runAsUser: 1000
    seccompProfile: { type: RuntimeDefault }
  containers:
    - name: c
      image: public.ecr.aws/docker/library/busybox:1.36
      command: ["sleep", "3600"]
      resources:
        requests: { cpu: 50m, memory: 64Mi }
        limits:   { cpu: 100m, memory: 128Mi }
      securityContext:                            # no runAsNonRoot here
        allowPrivilegeEscalation: false
        capabilities: { drop: ["ALL"] }
EOF
Error from server: error when creating "STDIN": admission webhook "validate.kyverno.svc-fail" denied the request: 

resource Pod/team-1/rogue-nonroot was blocked due to the following policies 

require-non-root:
  run-as-non-root: 'validation error: Containers must run as non-root. rule run-as-non-root
    failed at path /spec/containers/0/securityContext/runAsNonRoot/'

Testing Results

Attempt	Layer that blocked it	Result
Connect to another namespace	NetworkPolicy	Timed out
Assume another team's role	IAM trust policy + ABAC	AccessDenied
Deploy to another namespace	Argo CD AppProject	InvalidSpecError
Use Docker Hub	Kyverno (restrict-image-registries)	Denied at admission
Use default ServiceAccount	Kyverno (disallow-default-serviceaccount)	Denied at admission
Omit non-root	Kyverno (require-non-root)	Denied at admission

Additional Considerations

All of the layers we have provided so far give a level of isolation between tenants on a shared EKS cluster. If you need to go even further, there are two other directions. The first is to offer greater network isolation beyond what a NetworkPolicy can provide using a service mesh. The second is to offer greater compute isolation for when your threat model includes untrusted code or you don't want to rely on a shared kernel.

Option 1 - Adopting a Service Mesh (Zero Trust Networking)

By default, pod-to-pod communication in Amazon EKS Auto Mode is not protected with application-layer encryption. Pods communicate using native VPC networking, so end-to-end encryption between workloads requires application-layer TLS or a service mesh providing mutual TLS (mTLS). With VPC Encryption Controls, traffic between pods on different Nitro-based worker nodes can be encrypted in transit at the AWS networking layer. This protects packets while they traverse the AWS network but does not provide workload authentication, certificate identity or mTLS between services.

A service mesh provides each workload with a cryptographic identity. Services authenticate each other using certificates rather than relying solely on network location, IP addressing or namespace-based trust. It also allows for application-aware (Layer 7) authorisation policies based on HTTP methods, paths, or headers, rather than just a layer 4 (Transport) connection policy. Istio is the most widely adopted full-featured service mesh for Kubernetes. Adopting a service mesh brings with it additional operational complexity and infrastructure overhead. Traditionally, service meshes such as Istio injected a sidecar proxy into every pod, increasing CPU and memory consumption as well as pod startup times. More recently, Istio introduced Ambient Mesh, which replaces per-pod sidecars with shared node-level ztunnel proxies for Layer 4 functionality and optional waypoint proxies for Layer 7 policies, significantly reducing this overhead.

Option 2 - Adopting Fargate

The hardware virtualisation boundary provided by a hypervisor offers significantly stronger isolation than Linux namespaces and cgroups alone. One option to achieve this is to run pods on AWS Fargate rather than on shared EC2 worker nodes. Fargate is a separate serverless compute option (configured through Fargate profiles) rather than part of EKS Auto Mode, so adopting it means running those workloads on a different data plane from the Auto Mode managed nodes used elsewhere in this post. Fargate runs each pod within its own Firecracker microVM, providing a dedicated kernel and virtual machine isolation boundary for that pod. Unlike EC2-backed worker nodes, pods do not share a Linux kernel with other workloads, reducing the impact of a potential container escape.

There are trade-offs. Fargate does not support DaemonSets or privileged containers, making it unsuitable for some infrastructure agents and security tooling. Resources are also sized per pod, so you cannot take advantage of the workload bin-packing that EKS Auto Mode performs across managed EC2 instances.

Note that AWS now recommend EKS Auto Mode with EC2 managed instances over EKS Fargate link

Option 3 - Deploying workloads on separate nodes

EKS Auto Mode allows customers to achieve a similar workload isolation model to Fargate, using standard Kubernetes scheduling capabilities to ensure each EC2 instance runs a single application pod. To replicate Fargate’s pod isolation model where each pod runs on its own dedicated instance, you can use Kubernetes topology spread constraints. This is the recommended approach for controlling pod distribution across nodes. EKS Auto Mode will automatically provision new EC2 instances as needed to satisfy this constraint, providing a similar isolation model as Fargate while giving you access to the full range of EC2 instance types and purchasing options. In this example setting a maxSkew of 1 ensures the difference in pod count between any two nodes is at most 1, effectively resulting in one pod per node as EKS Auto Mode provisions additional instances to satisfy the scheduling constraint. The whenUnsatisfiable: DoNotSchedule attribute prevents scheduling if the constraint can't be made.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: isolated-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: isolated-app
  template:
    metadata:
      labels:
        app: isolated-app
      annotations:
        eks.amazonaws.com/compute-type: ec2
    spec:
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: isolated-app
        minDomains: 1
      containers:
      - name: app
        image: nginx
        ports:
        - containerPort: 80

You can also use use pod anti-affinity rules for stricter isolation. The podAntiAffinity rule with requiredDuringSchedulingIgnoredDuringExecution ensures that no two pods with the same label can be scheduled on the same node. This approach provides a stronger scheduling guarantee than topology spread constraints by preventing matching pods from being scheduled onto the same node.

The trade-off for this approach is efficiency and cost. Kubernetes achieves high utilisation through bin-packing multiple workloads onto shared nodes. Scheduling a single workload per EC2 instance reduces that density, resulting in more instances and higher infrastructure costs, although EKS Auto Mode continues to manage the provisioning, scaling and lifecycle of those instances automatically. This approach also preserves support for DaemonSets, privileged workloads and the broader EC2 feature set that are not available on Fargate.

Additional Security Considerations

Kubernetes Secrets

A strong security posture is to minimise the number of long-lived secrets stored within the cluster. Kubernetes Secrets are namespace-scoped, so a secret in "team-1" cannot be read by a pod in "team-2". Access is controlled through Kubernetes RBAC. EKS already encrypts etcd at rest using an AWS-owned key by default, and production clusters should add a further layer by enabling envelope encryption with a customer-managed AWS KMS key. Either way, every namespace's secrets remain stored within the same shared etcd datastore that forms part of the EKS control plane.

In our architecture, we used Pod Identity with short-lived role-based credentials. When we needed to interact with an RDS instance in a workload account, we used RDS IAM authentication, so there was no database password to hold.

The recommended approach when a secret is required is to store it in AWS Secrets Manager, with the secret scoped to the specific team that needs it. AWS Secrets Manager authorises access using IAM policies rather than Kubernetes RBAC. EKS Pod Identity allows pods to obtain AWS credentials without embedding static credentials in the cluster. This also provides additional capabilities such as automatic rotation, CloudTrail auditing, resource policies and centralised IAM-based access control.

In practice, many AWS-native workloads require few or no application secrets. Services such as Amazon S3, DynamoDB, SQS, SNS and RDS IAM authentication can all be accessed using short-lived IAM credentials delivered through EKS Pod Identity.

Image Provenance

Image provenance is another important consideration. Restricting workloads to approved registries ensures images originate from trusted locations, but it does not prove that an image has not been tampered with.

Amazon ECR supports managed container image signing using AWS Signer, automatically generating cryptographic signatures as images are pushed to the registry. Amazon EKS can then verify these signatures during deployment using its native image signature verification capability. Trusted signing profiles and verification policies define which signed images are permitted to run, helping ensure only trusted workloads are deployed. Combined with Amazon ECR image scanning, this establishes a trusted software supply chain from build through to deployment. This reduces the risk of one tenant deploying a malicious or tampered container image that could compromise the shared platform.

For organisations using a broader Kubernetes ecosystem, Sigstore Cosign remains a widely adopted alternative. Native Amazon EKS image verification currently validates Notation (Notary v2) signatures produced by AWS Signer, whereas Cosign signatures are typically verified during admission using a policy engine such as Kyverno's verifyImages.

Runtime threat detection

All of our controls described so far are preventative, and aim to stop a non-compliant workload from carrying out an activity it should not. To provide defence in depth, we are also interested in detective controls, and this is where Amazon GuardDuty EKS Protection comes in.

Amazon GuardDuty Runtime Monitoring supports Amazon EKS clusters running on Amazon EC2 instances and Amazon EKS Auto Mode. An automated agent configuration approach is available which allows GuardDuty to manage the deployment of the security agent on your behalf. With this approach, the VPC endpoint is created for you, and this is used to deliver the runtime events to the security agent. The security agent monitors process execution, file access and network activity to detect compromised containers and privilege escalation attempts, allowing them to be investigated and remediated.