Honey Kiran

Posted on Oct 27 Originally published at honeykiran.substack.com on Oct 16

50 Clusters, 50 aws-auth Files, Zero Idea Who Has Access

#aws #kubernetes #security #devops

Somebody on r/devops asked a plain question this week: how do you manage access for 50 EKS clusters with AWS IAM? No fancy wording. Just pain.

Because here is what “managing access” means at that scale today. Somebody joins the team — you edit a file. Somebody leaves — you edit the same file, fifty times, and pray you found every copy. An auditor asks who can reach production — you open fifty files and start digging. Access review becomes archaeology.

The file has a name: aws-auth. And the good news is, AWS built its replacement a while ago. It is called access entries , and it turns cluster access from a hand-edited file inside each cluster into an AWS API call you can stamp out from one place. Fifty clusters, one grant list, zero archaeology. Let us build that.

How we got here: one YAML file inside every cluster

First the basics, because this only makes sense if you know what aws-auth actually is. EKS is Amazon’s managed Kubernetes — AWS runs the control plane, you run the workloads. Kubernetes decides who can do what using its own permission system, RBAC. AWS decides who you are using IAM (Identity and Access Management — AWS’s system for identities and permissions). aws-auth is the bridge between the two: a ConfigMap — a small configuration object living inside the cluster — written in YAML, that maps an IAM user or role to Kubernetes groups.

It looks innocent enough:

apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-auth
  namespace: kube-system
data:
  mapRoles: |
    - rolearn: arn:aws:iam::123456789012:role/sre-oncall
      username: sre-oncall
      groups:
        - system:masters

One cluster, one file, no problem. Now multiply.

At fifty clusters you have fifty separate files, each edited by hand, each at a different moment by a different person. They drift. One cluster still maps a role that was renamed last quarter — silently broken. Another still grants admin to a group nobody remembers creating. A third got a “temporary” edit during an incident that became permanent the moment everyone went back to sleep. And the real insult: editing a ConfigMap is not an EKS API call, so the change does not show up in CloudTrail, AWS’s audit log of API activity. When the auditor asks “who gave this role admin on cluster 41, and when?” — the honest answer is a shrug.

There is one more trap, and it bites everyone once: to edit aws-auth you already need working admin access to that cluster. Fumble the YAML — a stray indent, a wrong group name — and you can lock out the very access you need to fix it. A guest list you can only edit from inside the party.

The new shape: access entries

Access entries move the guest list out of the cluster and into AWS, where it always belonged. An access entry is an AWS API object that says: this IAM principal — a user or, better, a role — may enter this cluster. Attached to the entry, you associate an access policy : an AWS-managed bundle of Kubernetes permissions. Four do most of the work:

AmazonEKSClusterAdminPolicy — full cluster admin (Kubernetes cluster-admin). Cluster scope only, for obvious reasons.
AmazonEKSAdminPolicy — admin inside a scope: nearly everything, short of cluster-wide control.
AmazonEKSEditPolicy — create, change and delete most resources, without the admin extras.
AmazonEKSViewPolicy — look, do not touch. List and view most resources.

Admin, Edit and View can be scoped : cluster-wide, or limited to named namespaces. That scope line is the whole game. A developer does not need “view on the cluster” — they need view on the two namespaces their service lives in, and silence everywhere else. There is also AmazonEKSAdminViewPolicy, a cluster-wide view that includes Secrets — hand that one out like you mean it, because “read-only” that can read every Secret in the cluster is read-only the way a glass house is private. Run aws eks list-access-policies to see the full current list in your account.

Here is the manual version, once, so you can feel the shape:

aws eks create-access-entry \
  --cluster-name prod-eu-1 \
  --principal-arn arn:aws:iam::123456789012:role/dev-view \
  --type STANDARD

aws eks associate-access-policy \
  --cluster-name prod-eu-1 \
  --principal-arn arn:aws:iam::123456789012:role/dev-view \
  --policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonEKSViewPolicy \
  --access-scope type=namespace,namespaces=payments

One entry, one policy, one namespace. Clean. Now: nobody is running that by hand fifty times either. The CLI is for understanding. The stamping comes later.

The guest list should live at the front desk, not be taped inside fifty different rooms.

Design the roles before you touch a single cluster

Here is where most migrations go wrong: teams automate the mess they already have — one IAM user per engineer, mapped fifty different ways — instead of designing the small set of roles the fleet actually needs. People should not be principals at this scale. Roles should be: named bundles of permission that a person temporarily wears, handed out through IAM Identity Center (AWS’s single sign-on front door), so joining and leaving is managed in exactly one place.

Four tiers cover almost every real fleet (names illustrative — rename to taste):

platform-admin
sre-oncall
dev-view
audit-view

Notice: no person’s name appears anywhere. When someone changes teams, you change their SSO group once — the roles on the clusters never change, so the fifty clusters never hear about it.

Stamp it on all fifty with Terraform

Access entries are plain AWS resources, so Terraform manages them like anything else: aws_eks_access_entry creates the entry, aws_eks_access_policy_association attaches the policy. The shape that scales is one map of who gets what, multiplied across the cluster list:

locals {
  clusters = ["prod-eu-1", "prod-us-1", "staging-eu-1"] # ...all 50

  grants = {
    platform-admin = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy"
    sre-oncall = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSAdminPolicy"
    dev-view = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSViewPolicy"
  }

  pairs = {
    for pair in setproduct(local.clusters, keys(local.grants)) :
    "${pair[0]}|${pair[1]}" => { cluster = pair[0], role = pair[1] }
  }
}

resource "aws_eks_access_entry" "this" {
  for_each = local.pairs
  cluster_name = each.value.cluster
  principal_arn = "arn:aws:iam::123456789012:role/${each.value.role}"
  type = "STANDARD"
}

resource "aws_eks_access_policy_association" "this" {
  for_each = local.pairs
  cluster_name = each.value.cluster
  principal_arn = "arn:aws:iam::123456789012:role/${each.value.role}"
  policy_arn = local.grants[each.value.role]

  access_scope {
    type = "cluster"
  }

  depends_on = [aws_eks_access_entry.this]
}

Two things worth knowing before you copy this into a real repo. First, that depends_on is not decoration: an association needs its entry to exist first, and Terraform does not always infer that on its own — at fifty clusters times several roles, applies start failing in batches until you say it out loud. Second, the scope above is cluster-wide for simplicity; for namespace-scoped tiers like dev-view, set type = "namespace" and add the namespace list per grant. The point stands either way: the grant list is one file, reviewed like code, applied everywhere identically. Drift has nowhere to be born.

Migrate without locking yourself out

Every cluster has an authentication mode that says which system EKS trusts: CONFIG_MAP (aws-auth only), API_AND_CONFIG_MAP (both), or API (access entries only — the recommended end state). Migration walks the modes in that order:

aws eks update-cluster-config \
  --name prod-eu-1 \
  --access-config authenticationMode=API_AND_CONFIG_MAP

In the middle mode, both systems work at once, and if the same principal appears in both, the access entry wins. That overlap is your safety net: create the entries, test every role through the new path — actually sign in, actually run the commands — and only then delete the matching aws-auth lines. One cluster, one role at a time, boring on purpose.

Two warnings, both earned by other people’s bad days. The mode change is a one-way door : once a cluster leaves CONFIG_MAP behind, it never goes back, so do the testing like you mean it. And inside aws-auth you will find entries EKS itself created for managed node groups and Fargate profiles — the mappings that let your nodes join the cluster at all. Do not remove those until their equivalent access entries exist, or your nodes simply stop joining and the cluster empties out politely, one NotReady at a time. Finish a cluster, flip it to API, and move to the next. Fifty small ceremonies, no big bang.

Humans use one door. Pods use another.

Everything above is about people reaching the Kubernetes API. Your workloads reaching AWS — a pod that needs to read an S3 bucket or publish to a queue — is a different door entirely, and mixing the two is how node roles end up with admin “temporarily” for two years.

The current way is EKS Pod Identity : an association binds a namespace plus a Kubernetes service account (the identity a pod runs as) to an IAM role, and an AWS-managed agent in the cluster hands the pod its credentials. No OIDC provider to build, no annotation to babysit — and the role’s temporary credentials carry session tags naming the cluster, namespace and service account, so one role can safely serve many clusters with permissions that adjust based on those tags (that trick is called ABAC — attribute-based access control).

The older way, IRSA (IAM Roles for Service Accounts), wires the cluster’s OIDC provider into IAM trust policies and annotates the service account with the role ARN. It still works, and you still need it where Pod Identity cannot go: Fargate profiles on older platform versions, and self-managed or EKS Anywhere clusters. Keep IRSA where it is load-bearing, use Pod Identity for everything new, and never let either become an excuse to hand a pod the node’s role.

Now the audit writes itself

Here is the payoff the Reddit thread was really asking for. Access entries are AWS API objects, so every change — CreateAccessEntry, AssociateAccessPolicy, deletions — lands in CloudTrail with a name and a timestamp attached. “Who changed access on any cluster this month?” stops being fifty files of archaeology and becomes one query:

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=CreateAccessEntry

Pair that with IAM Access Analyzer ‘s unused-access findings, which flag roles and permissions nobody has used inside the tracking window and recommend tighter policies to replace them. The roles people swore they needed forever and touched twice? There they are, in a findings list, with a suggested smaller policy attached. Quarterly review becomes: read the findings, shrink what is provably unused, archive what is deliberately idle with a reason written down. An afternoon, not an excavation.

You do not have an access problem on 50 clusters. You have a filing problem — the file is just YAML, and it is inside every cluster.

The 50-cluster checklist

Design 3–5 shared IAM roles first — platform-admin, sre-oncall, dev-view, audit-view — and hand them out through IAM Identity Center. No per-person principals on clusters.
Pick the smallest policy per tier — ClusterAdmin, Admin, Edit, View — and scope Admin/Edit/View to namespaces wherever a team only owns a few.
Treat AdminViewPolicy as sharp — cluster-wide view that includes Secrets is not “harmless read-only”.
Set authentication mode to API_AND_CONFIG_MAP , create entries, test every role end-to-end, then remove the matching aws-auth lines.
Protect the node mappings — never delete aws-auth entries EKS created for node groups or Fargate until equivalent access entries exist.
Finish at API mode , one cluster at a time. The mode never moves backwards, so test like it matters — it does.
Stamp everything from one Terraform module — aws_eks_access_entry + aws_eks_access_policy_association, explicit depends_on, one grant list for all 50 clusters.
Give pods their own door — Pod Identity for new workloads, IRSA only where Fargate versions or EKS Anywhere force it.
Audit from CloudTrail and Access Analyzer — access changes are API events now; unused roles get findings, not folklore.

Staging warning: run every step of this — mode changes, entry creation, ConfigMap cleanup — on a staging cluster first, with the same roles and the same module. A mode change cannot be undone, and an access mistake rehearses exactly as well as it performs.

The engineer in that thread does not need to work harder at editing files. Nobody can hand-edit fifty of anything and stay sane. Move the guest list to the front desk: a few roles, one module, an API that remembers who changed what. Fifty clusters stop being fifty special cases — one fleet, one door policy.

Honey Kiran · DevOps & Platform Engineering

DevOps, SRE, Terraform, and platform engineering — explained the practical way.

How is your fleet handling EKS access right now — aws-auth, access entries, or a mix nobody wants to open? Tell me in the comments; the honest answers are the useful ones.

Examples are illustrative — validate in your own environment before running in production.

Disclaimer

This post describes my personal experience with my own systems and accounts, in my own words. It is a first-person account of what I observed and what I do in my own work — not a statement of any company's intentions, policies, or internal decisions. Where I do not know something, the post says so.
I am not affiliated with, endorsed by, or speaking for any company, product, or vendor named in this post. All product and company names are trademarks of their respective owners and are mentioned only to describe the experience accurately.
Any error messages, documentation excerpts, or figures quoted here are reproduced from my own records or from the public sources linked in the post, believed accurate at the time of writing. Product behavior changes over time; verify against current official documentation before relying on any detail.
Nothing in this post is professional, financial, legal, or security advice. Examples are illustrative — validate in your own environment before running anything in production.
Corrections are welcome: if a company named here publishes an official statement, or a detail above is wrong, I will update this post and note the correction. The goal is an accurate public record of my experience, not a verdict on anyone.

DEV Community