Garrett Yan

Posted on Nov 2

Building a Multi-Tenant EKS Platform for 12 Teams: RBAC, Network Policies, and Karpenter with Terraform

#aws #kubernetes #devops #tutorial

November 2026 | ~18 min read

The Shared Cluster That Wasn't Sharing

Three incidents in one month. Same cluster. Different root cause every time. All preventable.

Week 1: Noisy neighbor. The data team kicked off a backfill job — 40 pods, no resource limits, no dedicated node pool. Within minutes, the job consumed 80% of cluster CPU. The payments API started returning 504s. On-call spent two hours tracking it down because kubectl top nodes showed everything was "healthy" — the nodes were just saturated. Revenue impact: $12,000 in failed transactions.

Week 2: Accidental scraper. A misconfigured frontend pod had a Prometheus scrape target pointing at * instead of its own namespace. It was hitting every metrics endpoint in the cluster every 15 seconds. Prometheus's memory ballooned from 4GB to 28GB, OOMKilled, and took every Grafana dashboard with it. The monitoring team didn't notice for 3 hours because… their monitoring was down.

Week 3: Wrong namespace delete. A junior ML engineer with cluster-admin ran kubectl delete configmap --all in what they thought was their dev namespace. It was payments-prod. The payments service lost its feature flags, rate limit configs, and API keys. Full recovery took 45 minutes from the last backup.

Every one of these incidents had the same root cause: 12 teams sharing one EKS cluster with zero governance. No RBAC beyond cluster-admin. No network isolation. No resource guardrails. No admission control. The cluster was a shared apartment where everyone had the master key and nobody locked their door.

I built the multi-tenant platform to fix this. Not a migration to multiple clusters — that's expensive and operationally brutal. A single cluster with proper tenant isolation. Namespace boundaries. Least-privilege RBAC. Deny-by-default network policies. Workload-class node pools. Policy-as-code admission control. Per-team observability with RBAC'd dashboards.

All Terraform. All production-ready. Here's everything.

The Numbers

┌─────────────────────────────────────────────────────────────────────┐
│                  MULTI-TENANT PLATFORM RESULTS                      │
├──────────────────────────────────┬──────────────┬───────────────────┤
│ Metric                           │ Before       │ After             │
├──────────────────────────────────┼──────────────┼───────────────────┤
│ Teams on cluster                 │ 12           │ 12                │
│ Namespaces                       │ 14           │ 36                │
│ RBAC model                       │ cluster-admin│ namespace-scoped  │
│ Cross-namespace access           │ 100% open    │ deny-by-default   │
│ Noisy neighbor incidents/quarter │ 7            │ 0                 │
│ Onboarding time (new team)       │ 2 weeks      │ 45 minutes        │
│ Policy violations blocked/week   │ 0            │ ~340              │
│ Karpenter NodePools              │ 1            │ 4                 │
│ P99 scheduling latency           │ 38s          │ 8s                │
│ Platform engineer time/week      │ 20 hrs       │ 5 hrs             │
│ Monthly compute cost             │ $6,800       │ $7,200 (+6%)      │
└──────────────────────────────────┴──────────────┴───────────────────┘

Yes, cost went up 6%. That's intentional. We traded $400/month for zero noisy-neighbor incidents, 75% less platform toil, and 96% faster onboarding. The three incidents in month one cost us more than $400 in engineering time alone.

Architecture Overview
Namespace Strategy and Resource Quotas
RBAC with Least Privilege
Network Policies for Tenant Isolation
Karpenter NodePools by Workload Class
Autoscaling — HPA, KEDA, and Topology Spread
Platform Governance with Kyverno
Per-Team Observability
Lessons Learned
Action Items

Architecture Overview

┌──────────────────────────────────────────────────────────────────────────┐
│                        MULTI-TENANT EKS PLATFORM                         │
│                                                                          │
│  12 Teams                                                                │
│  ┌───────┐ ┌───────┐ ┌───────┐ ┌───────┐ ┌───────┐ ┌───────┐          │
│  │Payments│ │  ML   │ │  Data │ │Frontend│ │ Auth  │ │  ...  │          │
│  └───┬───┘ └───┬───┘ └───┬───┘ └───┬───┘ └───┬───┘ └───┬───┘          │
│      │         │         │         │         │         │                 │
│      ▼         ▼         ▼         ▼         ▼         ▼                 │
│  ┌──────────────────────────────────────────────────────────┐            │
│  │              36 Namespaces ({team}-{env})                 │            │
│  │   payments-prod  payments-staging  payments-dev           │            │
│  │   ml-prod        ml-staging        ml-dev                 │            │
│  │   data-prod      data-staging      data-dev     ...       │            │
│  └────────────────────────┬─────────────────────────────────┘            │
│                           │                                              │
│  ┌────────────────────────┼─────────────────────────────────┐            │
│  │  Admission Control     │         Network Policies         │            │
│  │  ┌──────────┐    ┌─────┴──────┐    ┌──────────────┐      │            │
│  │  │ Kyverno  │    │ Cilium CNI │    │ Deny-by-      │      │            │
│  │  │ 5 policies│    │            │    │ default       │      │            │
│  │  └──────────┘    └────────────┘    └──────────────┘      │            │
│  └──────────────────────────────────────────────────────────┘            │
│                           │                                              │
│  ┌────────────────────────┼─────────────────────────────────┐            │
│  │             4 Karpenter NodePools                         │            │
│  │  ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐    │            │
│  │  │ Critical │ │ General  │ │  Batch   │ │  System  │    │            │
│  │  │ On-demand│ │ 70% spot │ │ 100% spot│ │ On-demand│    │            │
│  │  └──────────┘ └──────────┘ └──────────┘ └──────────┘    │            │
│  └──────────────────────────────────────────────────────────┘            │
│                           │                                              │
│  ┌────────────────────────┼─────────────────────────────────┐            │
│  │                 Observability                              │            │
│  │  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐    │            │
│  │  │ Grafana RBAC │  │  Prometheus  │  │  CloudWatch  │    │            │
│  │  │ per-team     │  │              │  │              │    │            │
│  │  └──────────────┘  └──────────────┘  └──────────────┘    │            │
│  └──────────────────────────────────────────────────────────┘            │
└──────────────────────────────────────────────────────────────────────────┘

The Terraform module layout:

terraform/
├── main.tf
├── variables.tf
├── outputs.tf
├── teams.tf                   # Team definitions (single source of truth)
├── namespaces.tf              # Namespaces + ResourceQuotas + LimitRanges
├── rbac.tf                    # ClusterRoles + RoleBindings + EKS access entries
├── network-policies.tf        # Cilium + deny-by-default + allow rules
├── karpenter-nodepools.tf     # 4 NodePools + EC2NodeClasses
├── autoscaling.tf             # HPA + KEDA + PDB templates
├── kyverno.tf                 # Helm release + 5 ClusterPolicies
├── observability.tf           # Grafana folders + teams + dashboards
└── modules/
    └── team-onboarding/       # Reusable module for new team setup

The foundation is a single teams local that drives everything via for_each:

# teams.tf — Single source of truth for all team configuration

locals {
  teams = {
    payments = {
      iam_roles       = ["arn:aws:iam::123456789012:role/payments-dev"]
      environments    = ["prod", "staging", "dev"]
      workload_class  = "critical"
      quota_tier      = "large"
      grafana_viewers = ["payments-lead@company.com", "payments-oncall@company.com"]
    }
    ml = {
      iam_roles       = ["arn:aws:iam::123456789012:role/ml-dev"]
      environments    = ["prod", "staging", "dev"]
      workload_class  = "batch"
      quota_tier      = "large"
      grafana_viewers = ["ml-lead@company.com"]
    }
    frontend = {
      iam_roles       = ["arn:aws:iam::123456789012:role/frontend-dev"]
      environments    = ["prod", "staging", "dev"]
      workload_class  = "general"
      quota_tier      = "medium"
      grafana_viewers = ["frontend-lead@company.com"]
    }
    auth = {
      iam_roles       = ["arn:aws:iam::123456789012:role/auth-dev"]
      environments    = ["prod", "staging", "dev"]
      workload_class  = "critical"
      quota_tier      = "medium"
      grafana_viewers = ["auth-lead@company.com"]
    }
    # ... 8 more teams follow the same pattern
  }

  # Flatten teams × environments into namespace map
  team_namespaces = merge([
    for team, config in local.teams : {
      for env in config.environments :
      "${team}-${env}" => {
        team           = team
        environment    = env
        workload_class = config.workload_class
        quota_tier     = config.quota_tier
      }
    }
  ]...)
}

Every resource downstream — namespaces, RBAC bindings, network policies, Grafana folders — iterates over this map. Adding a new team is a single block in teams.tf and a terraform apply.

Namespace Strategy and Resource Quotas

Convention: {team}-{env}. Three namespaces per team. 12 teams × 3 environments = 36 namespaces.

Why three environments per team instead of shared staging/dev namespaces? Because shared environments recreate the noisy-neighbor problem at a smaller scale. Team A's staging load test shouldn't blow up Team B's staging integration tests.

# namespaces.tf

resource "kubernetes_namespace" "team" {
  for_each = local.team_namespaces

  metadata {
    name = each.key

    labels = {
      "app.kubernetes.io/managed-by" = "terraform"
      "platform.company.io/team"     = each.value.team
      "platform.company.io/env"      = each.value.environment
      "platform.company.io/tier"     = each.value.quota_tier
      "pod-security.kubernetes.io/enforce" = each.value.environment == "prod" ? "restricted" : "baseline"
    }
  }
}

Resource Quotas by Tier

Three tiers. Prod gets the most. Dev gets enough to iterate but not enough to accidentally run a production-scale load test.

locals {
  quota_tiers = {
    large = {
      prod    = { cpu = "8",  memory = "16Gi", pods = "40", services = "10", pvcs = "10" }
      staging = { cpu = "4",  memory = "8Gi",  pods = "20", services = "5",  pvcs = "5" }
      dev     = { cpu = "2",  memory = "4Gi",  pods = "10", services = "3",  pvcs = "3" }
    }
    medium = {
      prod    = { cpu = "4",  memory = "8Gi",  pods = "20", services = "5",  pvcs = "5" }
      staging = { cpu = "2",  memory = "4Gi",  pods = "10", services = "3",  pvcs = "3" }
      dev     = { cpu = "1",  memory = "2Gi",  pods = "5",  services = "2",  pvcs = "2" }
    }
    small = {
      prod    = { cpu = "2",  memory = "4Gi",  pods = "10", services = "3",  pvcs = "3" }
      staging = { cpu = "1",  memory = "2Gi",  pods = "5",  services = "2",  pvcs = "2" }
      dev     = { cpu = "500m", memory = "1Gi", pods = "3", services = "1",  pvcs = "1" }
    }
  }
}

resource "kubernetes_resource_quota" "team" {
  for_each = local.team_namespaces

  metadata {
    name      = "${each.key}-quota"
    namespace = kubernetes_namespace.team[each.key].metadata[0].name
  }

  spec {
    hard = {
      "requests.cpu"    = local.quota_tiers[each.value.quota_tier][each.value.environment].cpu
      "requests.memory" = local.quota_tiers[each.value.quota_tier][each.value.environment].memory
      "limits.cpu"      = local.quota_tiers[each.value.quota_tier][each.value.environment].cpu
      "limits.memory"   = local.quota_tiers[each.value.quota_tier][each.value.environment].memory
      pods              = local.quota_tiers[each.value.quota_tier][each.value.environment].pods
      services          = local.quota_tiers[each.value.quota_tier][each.value.environment].services
      "persistentvolumeclaims" = local.quota_tiers[each.value.quota_tier][each.value.environment].pvcs
    }
  }
}

LimitRange for Default Container Limits

Without a LimitRange, pods with no resource requests bypass the quota entirely. This sets sane defaults so developers don't have to remember to set limits on every container.

resource "kubernetes_limit_range" "team" {
  for_each = local.team_namespaces

  metadata {
    name      = "${each.key}-limits"
    namespace = kubernetes_namespace.team[each.key].metadata[0].name
  }

  spec {
    limit {
      type = "Container"

      default = {
        cpu    = each.value.environment == "prod" ? "500m" : "250m"
        memory = each.value.environment == "prod" ? "512Mi" : "256Mi"
      }

      default_request = {
        cpu    = each.value.environment == "prod" ? "100m" : "50m"
        memory = each.value.environment == "prod" ? "128Mi" : "64Mi"
      }

      max = {
        cpu    = each.value.environment == "prod" ? "2" : "1"
        memory = each.value.environment == "prod" ? "4Gi" : "2Gi"
      }
    }
  }
}

The result: every namespace has a hard resource ceiling and every container gets default limits. The data team's backfill job? It would now hit the data-prod quota ceiling at 8 CPU instead of consuming the entire cluster.

RBAC with Least Privilege

Before: every developer had cluster-admin. After: three roles with namespace-scoped bindings.

Three Roles

team-developer: Full CRUD within their team's namespaces. Can create deployments, services, configmaps, secrets, and debug with exec/logs. Cannot access other teams' namespaces or cluster-wide resources.

team-viewer: Read-only access to their team's namespaces. For stakeholders, QA engineers, and on-call rotations that need visibility but shouldn't modify anything.

platform-admin: Cluster-wide access for the platform team. The only role that can modify namespaces, node pools, cluster policies, and CRDs.

# ClusterRole: team-developer
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: team-developer
rules:
  - apiGroups: ["", "apps", "batch"]
    resources:
      - pods
      - pods/log
      - pods/exec
      - deployments
      - replicasets
      - statefulsets
      - daemonsets
      - jobs
      - cronjobs
      - services
      - configmaps
      - secrets
      - persistentvolumeclaims
      - horizontalpodautoscalers
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  - apiGroups: ["autoscaling"]
    resources: ["horizontalpodautoscalers"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  - apiGroups: ["networking.k8s.io"]
    resources: ["ingresses"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  - apiGroups: ["policy"]
    resources: ["poddisruptionbudgets"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

# ClusterRole: team-viewer
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: team-viewer
rules:
  - apiGroups: ["", "apps", "batch", "autoscaling", "networking.k8s.io"]
    resources: ["*"]
    verbs: ["get", "list", "watch"]

Namespace-Scoped RoleBindings

The ClusterRoles define what. The RoleBindings define where. Each team gets developer bindings only in their own namespaces.

# rbac.tf

resource "kubernetes_role_binding" "team_developer" {
  for_each = local.team_namespaces

  metadata {
    name      = "${each.value.team}-developer"
    namespace = kubernetes_namespace.team[each.key].metadata[0].name
  }

  role_ref {
    api_group = "rbac.authorization.k8s.io"
    kind      = "ClusterRole"
    name      = "team-developer"
  }

  subject {
    kind      = "Group"
    name      = "${each.value.team}-developers"
    api_group = "rbac.authorization.k8s.io"
  }
}

resource "kubernetes_role_binding" "team_viewer" {
  for_each = local.team_namespaces

  metadata {
    name      = "${each.value.team}-viewer"
    namespace = kubernetes_namespace.team[each.key].metadata[0].name
  }

  role_ref {
    api_group = "rbac.authorization.k8s.io"
    kind      = "ClusterRole"
    name      = "team-viewer"
  }

  subject {
    kind      = "Group"
    name      = "${each.value.team}-viewers"
    api_group = "rbac.authorization.k8s.io"
  }
}

EKS Access Entries

EKS access entries are the modern replacement for the aws-auth ConfigMap. They're managed via the AWS API, version-controlled in Terraform, and don't have the single-point-of-failure problem where a bad ConfigMap edit locks everyone out.

resource "aws_eks_access_entry" "team" {
  for_each = local.teams

  cluster_name  = aws_eks_cluster.main.name
  principal_arn = each.value.iam_roles[0]
  type          = "STANDARD"

  kubernetes_groups = [
    "${each.key}-developers"
  ]
}

resource "aws_eks_access_policy_association" "team" {
  for_each = local.teams

  cluster_name  = aws_eks_cluster.main.name
  principal_arn = aws_eks_access_entry.team[each.key].principal_arn
  policy_arn    = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSViewPolicy"

  access_scope {
    type       = "namespace"
    namespaces = [for env in each.value.environments : "${each.key}-${env}"]
  }
}

The Migration Path

We didn't flip a switch. We ran both systems in parallel for two weeks.

Week 1: Created all ClusterRoles, RoleBindings, and access entries alongside existing cluster-admin grants. Verified every team could still do their work.
Week 2: Removed cluster-admin bindings one team at a time. Started with teams that had staging-only workloads. Payments (the most critical) went last.
Day 15: Removed the legacy aws-auth ConfigMap entries. EKS access entries became the sole authentication source.

The junior ML engineer from the Week 3 incident? They now have team-developer scoped to ml-prod, ml-staging, and ml-dev. Running kubectl delete configmap --all in payments-prod would return Forbidden.

Network Policies for Tenant Isolation

RBAC controls who can talk to the API server. Network policies control which pods can talk to each other. Without network policies, any pod can reach any other pod in the cluster — even across namespaces. This is how the misconfigured frontend scraper from Week 2 was able to hit every Prometheus endpoint.

Cilium CNI

We replaced the default AWS VPC CNI's network policy enforcement with Cilium. AWS VPC CNI supports network policies natively since EKS 1.25, but Cilium gives us two things AWS CNI doesn't: Hubble (network flow visibility) and L7 policies (HTTP method/path filtering).

# network-policies.tf

resource "helm_release" "cilium" {
  name       = "cilium"
  repository = "https://helm.cilium.io/"
  chart      = "cilium"
  version    = "1.16.4"
  namespace  = "kube-system"

  set {
    name  = "eni.enabled"
    value = "true"
  }

  set {
    name  = "ipam.mode"
    value = "eni"
  }

  set {
    name  = "egressMasqueradeInterfaces"
    value = "eth0"
  }

  set {
    name  = "routingMode"
    value = "native"
  }

  set {
    name  = "hubble.relay.enabled"
    value = "true"
  }

  set {
    name  = "hubble.ui.enabled"
    value = "true"
  }
}

Default-Deny Everything

The first policy applied to every namespace: deny all ingress and egress. Then we add explicit allows.

# Policy 1: Default deny all traffic
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress

resource "kubernetes_network_policy" "default_deny" {
  for_each = local.team_namespaces

  metadata {
    name      = "default-deny-all"
    namespace = kubernetes_namespace.team[each.key].metadata[0].name
  }

  spec {
    pod_selector {}

    policy_types = ["Ingress", "Egress"]
  }
}

Explicit Allows

With everything denied, we add back the traffic that should flow. Four allow policies per namespace:

# Policy 2: Allow same-namespace communication
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-same-namespace
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - podSelector: {}
  egress:
    - to:
        - podSelector: {}

# Policy 3: Allow DNS resolution (required for any egress)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

# Policy 4: Allow ingress from ingress controller
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-from-ingress
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/component: server
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: ingress-nginx

# Policy 5: Allow specific cross-namespace traffic (e.g., auth service)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-from-auth
spec:
  podSelector: {}
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              platform.company.io/team: auth
          podSelector:
            matchLabels:
              app: auth-service

Hubble for Traffic Mapping

Before applying deny-all, we ran Hubble for one week to map actual traffic flows. This is critical — you don't want to apply deny-by-default and discover that 6 services have undocumented cross-namespace dependencies.

# Install Hubble CLI
hubble observe --namespace payments-prod --since 7d \
  --output jsonpb | jq -r '.flow |
    select(.source.namespace != .destination.namespace) |
    "\(.source.namespace)/\(.source.labels[\"app\"]) -> \(.destination.namespace)/\(.destination.labels[\"app\"])"' \
  | sort | uniq -c | sort -rn

The output revealed 4 cross-namespace dependencies we didn't know about. We created explicit allow policies for each before flipping to deny-by-default.

resource "kubernetes_network_policy" "allow_same_namespace" {
  for_each = local.team_namespaces

  metadata {
    name      = "allow-same-namespace"
    namespace = kubernetes_namespace.team[each.key].metadata[0].name
  }

  spec {
    pod_selector {}

    ingress {
      from {
        pod_selector {}
      }
    }

    egress {
      to {
        pod_selector {}
      }
    }

    policy_types = ["Ingress", "Egress"]
  }
}

resource "kubernetes_network_policy" "allow_dns" {
  for_each = local.team_namespaces

  metadata {
    name      = "allow-dns"
    namespace = kubernetes_namespace.team[each.key].metadata[0].name
  }

  spec {
    pod_selector {}

    egress {
      to {
        namespace_selector {
          match_labels = {
            "kubernetes.io/metadata.name" = "kube-system"
          }
        }
      }

      ports {
        protocol = "UDP"
        port     = "53"
      }

      ports {
        protocol = "TCP"
        port     = "53"
      }
    }

    policy_types = ["Egress"]
  }
}

The result: the frontend scraper incident is now impossible. A pod in frontend-prod can only reach other pods in frontend-prod, the DNS service, and the ingress controller. Everything else returns connection refused.

Karpenter NodePools by Workload Class

In my previous post on Kubernetes cost optimization, I covered migrating from Cluster Autoscaler to a single Karpenter NodePool with spot instances. That works for a single-team cluster focused on cost. For a multi-tenant platform, one NodePool isn't enough — different workload classes need different scheduling guarantees.

Four NodePools, each with a distinct purpose:

┌────────────────────────────────────────────────────────────────────────┐
│                     KARPENTER NODEPOOL STRATEGY                        │
├──────────────┬───────────────┬─────────────────┬───────────────────────┤
│ Pool         │ Capacity      │ Disruption       │ Use Case             │
├──────────────┼───────────────┼─────────────────┼───────────────────────┤
│ critical     │ 100% on-demand│ Budget: 0        │ payments, auth       │
│ general      │ 70% spot / 30%│ WhenEmpty, 6h    │ frontend, backend    │
│ batch        │ 100% spot     │ WhenEmpty, 30m   │ ML training, ETL     │
│ system       │ 100% on-demand│ Budget: 0        │ monitoring, ingress  │
└──────────────┴───────────────┴─────────────────┴───────────────────────┘

Critical Pool

For workloads where a spot interruption means revenue loss. Zero disruption budget — Karpenter will never voluntarily evict nodes in this pool.

# karpenter/nodepool-critical.yaml
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: critical
spec:
  template:
    metadata:
      labels:
        workload-class: critical
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["m", "r"]
        - key: karpenter.k8s.aws/instance-generation
          operator: Gt
          values: ["5"]
        - key: karpenter.k8s.aws/instance-size
          operator: In
          values: ["large", "xlarge", "2xlarge"]
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: critical
  disruption:
    budgets:
      - nodes: "0"
  limits:
    cpu: "32"
    memory: "64Gi"
  weight: 10

General Pool

The default for most workloads. 70% spot with on-demand fallback. Consolidation runs after 6 hours of underutilization — long enough to absorb daily traffic patterns.

# karpenter/nodepool-general.yaml
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general
spec:
  template:
    metadata:
      labels:
        workload-class: general
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64", "arm64"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["m", "c", "r"]
        - key: karpenter.k8s.aws/instance-generation
          operator: Gt
          values: ["5"]
        - key: karpenter.k8s.aws/instance-size
          operator: In
          values: ["large", "xlarge", "2xlarge"]
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: general
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 6h
  limits:
    cpu: "96"
    memory: "192Gi"
  weight: 50

Batch Pool

For ML training, ETL jobs, and data backfills. 100% spot — these workloads are designed to be interrupted and resumed. Aggressive consolidation (30 minutes) because batch jobs are bursty.

# karpenter/nodepool-batch.yaml
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: batch
spec:
  template:
    metadata:
      labels:
        workload-class: batch
      annotations:
        karpenter.sh/do-not-disrupt: "false"
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64", "arm64"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["m", "c", "r", "i"]
        - key: karpenter.k8s.aws/instance-generation
          operator: Gt
          values: ["5"]
        - key: karpenter.k8s.aws/instance-size
          operator: In
          values: ["xlarge", "2xlarge", "4xlarge"]
      taints:
        - key: workload-class
          value: batch
          effect: NoSchedule
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: batch
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 30m
  limits:
    cpu: "64"
    memory: "128Gi"
  weight: 80

System Pool

For cluster infrastructure: monitoring, ingress controllers, cert-manager, Kyverno itself. Tainted so application workloads can't land here. On-demand only — you don't want your ingress controller getting spot-interrupted.

# karpenter/nodepool-system.yaml
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: system
spec:
  template:
    metadata:
      labels:
        workload-class: system
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["m"]
        - key: karpenter.k8s.aws/instance-generation
          operator: Gt
          values: ["5"]
        - key: karpenter.k8s.aws/instance-size
          operator: In
          values: ["large", "xlarge"]
      taints:
        - key: node-role
          value: system
          effect: NoSchedule
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: system
  disruption:
    budgets:
      - nodes: "0"
  limits:
    cpu: "16"
    memory: "32Gi"
  weight: 10

EC2NodeClass per Pool

Each pool gets its own EC2NodeClass with appropriate security groups and subnets:

# karpenter/ec2nodeclass-critical.yaml
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: critical
spec:
  role: KarpenterNodeRole-production
  amiSelectorTerms:
    - alias: al2023@latest
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: production-cluster
        tier: private
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: production-cluster
  blockDeviceMappings:
    - deviceName: /dev/xvda
      ebs:
        volumeSize: 100Gi
        volumeType: gp3
        encrypted: true
  metadataOptions:
    httpEndpoint: enabled
    httpProtocolIPv6: disabled
    httpPutResponseHopLimit: 1
    httpTokens: required
  tags:
    workload-class: critical
    ManagedBy: karpenter

Workload Targeting

Deployments land on the right pool via nodeSelector and tolerations:

# Example: payments service targeting the critical pool
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments-api
  namespace: payments-prod
spec:
  replicas: 3
  selector:
    matchLabels:
      app: payments-api
  template:
    metadata:
      labels:
        app: payments-api
        platform.company.io/team: payments
    spec:
      nodeSelector:
        workload-class: critical
      containers:
        - name: payments-api
          image: 123456789012.dkr.ecr.us-east-1.amazonaws.com/payments-api:v2.3.1
          resources:
            requests:
              cpu: "500m"
              memory: "512Mi"
            limits:
              cpu: "1"
              memory: "1Gi"
          ports:
            - containerPort: 8080

# Example: ML training job targeting the batch pool
apiVersion: batch/v1
kind: Job
metadata:
  name: model-training-v4
  namespace: ml-prod
spec:
  template:
    spec:
      nodeSelector:
        workload-class: batch
      tolerations:
        - key: workload-class
          value: batch
          effect: NoSchedule
      containers:
        - name: trainer
          image: 123456789012.dkr.ecr.us-east-1.amazonaws.com/ml-trainer:v4.0.0
          resources:
            requests:
              cpu: "4"
              memory: "16Gi"
            limits:
              cpu: "8"
              memory: "32Gi"
      restartPolicy: OnFailure

The scheduling latency improvement (38s → 8s) came from two things: workload-specific instance type requirements (Karpenter doesn't have to evaluate every instance type for every pod) and pre-warmed capacity in the critical pool that's always available.

Autoscaling — HPA, KEDA, and Topology Spread

NodePools handle node-level scaling. But pod-level scaling needs its own strategy. We use HPA for synchronous HTTP workloads, KEDA for event-driven workloads, and topology spread constraints to keep pods resilient across availability zones.

HPA for HTTP Workloads

Standard HPA with CPU target. The key addition: custom metrics via Prometheus adapter for RPS-based scaling, because CPU alone misses I/O-bound services that hit latency limits before CPU saturates.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: payments-api
  namespace: payments-prod
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: payments-api
  minReplicas: 3
  maxReplicas: 15
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 65
    - type: Pods
      pods:
        metric:
          name: http_requests_per_second
        target:
          type: AverageValue
          averageValue: "100"
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Percent
          value: 25
          periodSeconds: 60
    scaleUp:
      stabilizationWindowSeconds: 30
      policies:
        - type: Percent
          value: 100
          periodSeconds: 30

The behavior block is critical. Scale-up is aggressive (double pods in 30 seconds). Scale-down is conservative (remove max 25% per minute, wait 5 minutes between scale-downs). This prevents flapping during traffic spikes.

KEDA for Event-Driven Workloads

For the data team's SQS-based ETL pipeline, HPA doesn't work — there's no HTTP traffic to measure. KEDA scales based on queue depth.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: etl-processor
  namespace: data-prod
spec:
  scaleTargetRef:
    name: etl-processor
  minReplicaCount: 0
  maxReplicaCount: 20
  pollingInterval: 15
  cooldownPeriod: 300
  triggers:
    - type: aws-sqs-queue
      metadata:
        queueURL: https://sqs.us-east-1.amazonaws.com/123456789012/etl-input
        queueLength: "5"
        awsRegion: us-east-1
      authenticationRef:
        name: keda-aws-credentials
    - type: cron
      metadata:
        timezone: America/New_York
        start: "0 2 * * *"
        end: "0 6 * * *"
        desiredReplicas: "10"

This scales to zero when the queue is empty (saving compute), scales to 10 pods during the nightly batch window (2-6 AM), and dynamically scales based on queue depth during the day with 5 messages per pod.

PodDisruptionBudget

Every HPA-managed deployment gets a PDB. Without it, Karpenter's consolidation can evict all replicas simultaneously during a node drain.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: payments-api
  namespace: payments-prod
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: payments-api

Rule of thumb: maxUnavailable: 1 for prod, maxUnavailable: "50%" for staging. Never set minAvailable equal to your replica count — that makes the pod undisruptable and blocks all node drains.

Topology Spread Constraints

Pods should be spread across availability zones. A single AZ outage shouldn't take down a service.

# Production: strict spread — DoNotSchedule if AZs are imbalanced
spec:
  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: topology.kubernetes.io/zone
      whenUnsatisfiable: DoNotSchedule
      labelSelector:
        matchLabels:
          app: payments-api
    - maxSkew: 1
      topologyKey: kubernetes.io/hostname
      whenUnsatisfiable: ScheduleAnyway
      labelSelector:
        matchLabels:
          app: payments-api

# Staging: relaxed spread — ScheduleAnyway to avoid blocking deploys
spec:
  topologySpreadConstraints:
    - maxSkew: 2
      topologyKey: topology.kubernetes.io/zone
      whenUnsatisfiable: ScheduleAnyway
      labelSelector:
        matchLabels:
          app: payments-api

Production uses DoNotSchedule with maxSkew: 1 — the scheduler will refuse to place a pod if it would create an AZ imbalance. Staging uses ScheduleAnyway with maxSkew: 2 — it prefers balanced placement but won't block a deployment if AZ capacity is uneven.

Platform Governance with Kyverno

RBAC controls who. Network policies control where. Kyverno controls what. It's an admission controller that validates and mutates Kubernetes resources before they're persisted to etcd.

Why Kyverno Over OPA Gatekeeper

Both work. We chose Kyverno for one reason: policies are written in native YAML, not Rego. Our team of 12 developers can read and modify Kyverno policies without learning a new language. Gatekeeper's Rego policies are more powerful but create a bottleneck where only the platform team can write them.

Kyverno policy:    "if resource.spec.containers[*].resources.limits is empty, deny"
Gatekeeper policy: "violation[{"msg": msg}] { container := input.review.object.spec.containers[_] ...}"

Five ClusterPolicies

# Policy 1: Require resource limits on all containers
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-resource-limits
  annotations:
    policies.kyverno.io/title: Require Resource Limits
    policies.kyverno.io/severity: high
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: require-limits
      match:
        any:
          - resources:
              kinds:
                - Pod
      exclude:
        any:
          - resources:
              namespaces:
                - kube-system
                - karpenter
                - monitoring
      validate:
        message: "All containers must have CPU and memory limits defined."
        pattern:
          spec:
            containers:
              - resources:
                  limits:
                    cpu: "?*"
                    memory: "?*"

# Policy 2: Require standard labels
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-standard-labels
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: require-team-label
      match:
        any:
          - resources:
              kinds:
                - Deployment
                - StatefulSet
                - Job
      exclude:
        any:
          - resources:
              namespaces:
                - kube-system
                - karpenter
                - monitoring
      validate:
        message: "Deployments must have 'platform.company.io/team' and 'app' labels."
        pattern:
          metadata:
            labels:
              platform.company.io/team: "?*"
              app: "?*"

# Policy 3: Deny privileged containers
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: deny-privileged
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: deny-privileged-containers
      match:
        any:
          - resources:
              kinds:
                - Pod
      exclude:
        any:
          - resources:
              namespaces:
                - kube-system
      validate:
        message: "Privileged containers are not allowed."
        pattern:
          spec:
            containers:
              - securityContext:
                  privileged: "false"

# Policy 4: Restrict image registries
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-image-registries
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: allowed-registries
      match:
        any:
          - resources:
              kinds:
                - Pod
      exclude:
        any:
          - resources:
              namespaces:
                - kube-system
                - karpenter
      validate:
        message: "Images must come from approved registries: ECR or approved public repos."
        pattern:
          spec:
            containers:
              - image: "123456789012.dkr.ecr.*.amazonaws.com/* | public.ecr.aws/*"

# Policy 5: Require topology spread for production deployments
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-topology-spread
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: require-zone-spread
      match:
        any:
          - resources:
              kinds:
                - Deployment
              namespaces: "*-prod"
      preconditions:
        all:
          - key: "{{ request.object.spec.replicas }}"
            operator: GreaterThan
            value: 1
      validate:
        message: "Production deployments with >1 replica must have topology spread constraints."
        pattern:
          spec:
            template:
              spec:
                topologySpreadConstraints:
                  - topologyKey: "topology.kubernetes.io/zone"
                    maxSkew: "1"

Rollout Strategy

We deployed every policy in Audit mode first. Kyverno generates PolicyReport resources that show what would be blocked without actually blocking it.

# kyverno.tf

resource "helm_release" "kyverno" {
  name       = "kyverno"
  repository = "https://kyverno.github.io/kyverno/"
  chart      = "kyverno"
  version    = "3.3.4"
  namespace  = "kyverno"

  create_namespace = true

  set {
    name  = "replicaCount"
    value = "3"
  }

  set {
    name  = "resources.limits.memory"
    value = "512Mi"
  }
}

Week 1-2: All policies in Audit mode. Ran kubectl get policyreport -A daily to see violations. Found 340+ violations per week — mostly missing labels and resource limits.

Week 3: Sent violation reports to each team's Slack channel. Gave them two weeks to fix.

Week 5: Flipped to Enforce. The remaining violations were blocked at admission. Teams that hadn't fixed their manifests got immediate kubectl errors with clear messages explaining what was wrong.

# Check current violations
kubectl get policyreport -A -o json | jq '.items[] |
  select(.results != null) |
  .results[] |
  select(.result == "fail") |
  {namespace: .resources[0].namespace, policy: .policy, message: .message}'

The ~340 violations per week stabilized at ~15 after two months — mostly new deployments from developers who hadn't read the onboarding docs. The error messages from Kyverno are clear enough that they self-correct without opening a support ticket.

Per-Team Observability

Each team gets their own Grafana folder with pre-built dashboards scoped to their namespaces. The platform team gets a global view. For the full monitoring stack setup (Prometheus, Grafana, Kubecost, CloudWatch), see my previous post on EKS cost monitoring — this section covers the multi-tenant RBAC layer on top of that stack.

# observability.tf

resource "grafana_folder" "team" {
  for_each = local.teams

  title = "Team: ${title(each.key)}"
}

resource "grafana_team" "team" {
  for_each = local.teams

  name    = each.key
  email   = "${each.key}-team@company.com"
  members = each.value.grafana_viewers
}

resource "grafana_folder_permission" "team" {
  for_each = local.teams

  folder_uid = grafana_folder.team[each.key].uid

  permissions {
    team_id    = grafana_team.team[each.key].id
    permission = "View"
  }

  permissions {
    team_id    = grafana_team.platform["platform"].id
    permission = "Admin"
  }
}

Pre-Provisioned Dashboards

Every team folder gets three dashboards automatically:

Namespace Overview: CPU/memory utilization, pod count, restart count, OOMKill events
HPA Status: Current vs desired replicas, scaling events, CPU/memory targets vs actual
Network Policy Drops: Cilium drop counters per namespace — shows blocked traffic that might indicate a misconfiguration or an actual isolation policy doing its job

resource "grafana_dashboard" "team_overview" {
  for_each = local.team_namespaces

  folder    = grafana_folder.team[each.value.team].uid
  overwrite = true

  config_json = templatefile("${path.module}/dashboards/namespace-overview.json.tpl", {
    namespace   = each.key
    team        = each.value.team
    environment = each.value.environment
  })
}

The platform team's global dashboard shows:

Cluster-wide resource utilization by NodePool
Per-team cost allocation (via Kubecost labels)
Kyverno policy violation trends
Network policy drop rates across all namespaces
Karpenter scheduling latency and consolidation events

This gives the platform team the signal they need to identify teams that are consistently hitting quotas (time to increase the tier) or teams that are underutilizing their allocation (time to right-size).

Lessons Learned

1. RBAC is the foundation — everything else is theater without it.

Network policies, Kyverno, quotas — none of it matters if every developer has cluster-admin. A developer with cluster-admin can delete network policies, modify quotas, and bypass admission control. RBAC is the first thing you implement. We made the mistake of deploying network policies before RBAC. A developer deleted a deny-all policy that was blocking their cross-namespace call. We re-applied it. They deleted it again. The fix was RBAC, not the policy.

2. Network policies should be deny-by-default from day one.

Retrofitting deny-by-default onto an existing cluster is painful. You have to map every traffic flow, create allow rules for each one, and then flip the switch — while praying you didn't miss a dependency. If we'd started with deny-by-default, teams would have declared their dependencies as part of the deployment process from the beginning. Do it on day one.

3. Four Karpenter NodePools is the sweet spot.

We started with two (critical + general) and quickly realized batch workloads needed their own pool with aggressive consolidation, and system components needed isolation from application workloads. Four covers every workload class we've seen. We considered a fifth for GPU workloads but haven't needed it yet — when we do, the pattern is established.

4. Kyverno policies need an audit phase before enforce.

We tried enforcing the require-resource-limits policy immediately. It broke 23 deployments across 8 teams in the first hour. The two-week audit phase with Slack notifications gave teams time to fix their manifests without a production outage. Every new policy now follows the same audit → notify → enforce cycle.

5. Platform engineering is a product, not a project.

The initial build took 8 weeks. But the ongoing work — new team onboarding, quota adjustments, cross-namespace policy exceptions, Karpenter tuning, Kyverno policy updates — is continuous. We treat the platform like a product with a backlog, sprint planning, and a dedicated on-call rotation. The 20 hrs/week → 5 hrs/week reduction happened because we automated the repetitive parts (onboarding, quota changes), not because the work went away.

Action Items

Phase 1: Foundation (Weeks 1-3)

[ ] Map existing cluster: namespaces, RBAC bindings, network traffic flows (Hubble)
[ ] Define team structure in teams.tf locals block
[ ] Create namespaces with {team}-{env} convention and for_each
[ ] Apply ResourceQuotas (3 tiers) and LimitRanges to all namespaces
[ ] Deploy ClusterRoles: team-developer, team-viewer, platform-admin
[ ] Create RoleBindings per team per namespace
[ ] Migrate from aws-auth ConfigMap to EKS access entries
[ ] Revoke all cluster-admin grants (platform-admin team last)
[ ] Install Cilium CNI and enable Hubble
[ ] Apply default-deny network policies + allow rules per namespace
[ ] Validate: team members can only access their own namespaces

Phase 2: Workload Isolation (Weeks 4-5)

[ ] Deploy 4 Karpenter NodePools: critical, general, batch, system
[ ] Create EC2NodeClass per pool with appropriate SGs and subnets
[ ] Migrate critical workloads (payments, auth) to critical pool
[ ] Migrate batch workloads (ML, ETL) to batch pool with tolerations
[ ] Deploy HPA for all HTTP services with scale-down stabilization
[ ] Deploy KEDA for event-driven workloads (SQS, cron)
[ ] Add PodDisruptionBudgets to all HPA-managed deployments
[ ] Add topology spread constraints to all prod deployments
[ ] Validate: spot interruption in batch pool doesn't affect critical workloads

Phase 3: Governance and Observability (Weeks 6-8)

[ ] Install Kyverno (3 replicas, resource limits)
[ ] Deploy 5 ClusterPolicies in Audit mode
[ ] Run audit for 2 weeks — send violation reports to team Slack channels
[ ] Switch policies to Enforce mode
[ ] Create Grafana folders and teams per team
[ ] Deploy pre-built dashboards (namespace overview, HPA status, network drops)
[ ] Build platform team global dashboard
[ ] Create onboarding runbook: new team → teams.tf entry → terraform apply
[ ] Validate: new team onboarding completes in <1 hour

The full Terraform source for this platform is ~1,200 lines across 10 files. The most complex part isn't any individual resource — it's the for_each composition that makes everything driven from a single team definition. Get the data model right in teams.tf, and the rest follows mechanically.

If you're running a shared EKS cluster with more than 3 teams and you don't have RBAC, network policies, and admission control, the incidents I described aren't hypothetical — they're a matter of time. Build the platform before the incidents force you to.