DEV Community: Paweł Swiridow

Decoupling Ingress with TargetGroupBinding in EKS

Paweł Swiridow — Wed, 28 Jan 2026 11:00:00 +0000

As we scale our EKS clusters, relying solely on Kubernetes Ingress objects to provision AWS Application Load Balancers (ALBs) can become restrictive. Sometimes we need to attach an EKS Service to a pre-existing ALB managed by Terraform, or we need complex routing rules that are easier to manage in HCL (Terraform) than in K8s annotations.

The AWS Load Balancer Controller supports a custom resource called TargetGroupBinding (TGB). This allows us to provision the Load Balancer and Target Group in Terraform (Infrastructure layer) and simply "bind" our Kubernetes Service to it at runtime (Application layer).

This guide walks through how to set up an AWS Target Group in Terraform and register a Prometheus instance to it using Helm values.

Prerequisites

Before proceeding, ensure your environment meets the following requirements. This architecture relies on specific AWS components to route traffic directly to Pods:

Amazon EKS Cluster: A running EKS cluster is required.
AWS VPC CNI Plugin: We will be using target_type = "ip". This mode requires the AWS VPC CNI (the default networking plugin for EKS), which assigns native AWS VPC IP addresses to Pods. This allows the ALB to route traffic directly to the Pod IP, bypassing the worker node's kube-proxy.
AWS Load Balancer Controller: You must have the AWS Load Balancer Controller (v2.0+) installed and running in your cluster. This controller is responsible for installing the TargetGroupBinding CRD and actively managing the registration of targets.
- Note: Ensure the controller has the necessary IAM permissions to elasticloadbalancing:RegisterTargets and elasticloadbalancing:DeregisterTargets.

Part 1: Infrastructure (Terraform)

First, we need to create the Target Group. The critical setting here is target_type = "ip".

When using the AWS Load Balancer Controller with the AWS VPC CNI, we want the ALB to send traffic directly to the Pod IP addresses, bypassing NodePorts.

`main.tf`

resource "aws_lb_target_group" "prometheus_tg" {
  name        = "eks-prometheus-tg"
  port        = 9090
  protocol    = "HTTP"
  vpc_id      = module.vpc.vpc_id

  # CRITICAL: Must be 'ip' for direct Pod routing via AWS LB Controller
  target_type = "ip"

  health_check {
    path                = "/-/healthy"
    protocol            = "HTTP"
    matcher             = "200"
    interval            = 30
    timeout             = 5
    healthy_threshold   = 3
    unhealthy_threshold = 3
  }

  tags = {
    Environment = "production"
    ManagedBy   = "terraform"
  }
}

# We need to export this ARN to pass it to our Helm chart later
output "prometheus_tg_arn" {
  value = aws_lb_target_group.prometheus_tg.arn
}

Note: Do not use aws_lb_target_group_attachment in Terraform. The AWS Load Balancer Controller running inside the cluster will manage the targets dynamically as Pods come and go.

Part 2: Security (IAM Least Privilege)

By default, the generic AWS Load Balancer Controller policy is permissive (often using Resource: *). In a production environment - especially one with multiple teams sharing an AWS account - we should adhere to Least Privilege.

We must restrict the Controller's ability so it can only register/deregister targets for this specific Target Group, preventing it from accidentally modifying other Load Balancers.

Add this policy to the IAM Role used by your Load Balancer Controller ServiceAccount:

`iam.tf`

data "aws_iam_policy_document" "lb_controller_tgb_policy" {
  statement {
    sid       = "AllowRegisterTargets"
    effect    = "Allow"
    actions   = [
      "elasticloadbalancing:RegisterTargets",
      "elasticloadbalancing:DeregisterTargets"
    ]
    # Scope permissions strictly to the specific Target Group ARN created above
    resources = [aws_lb_target_group.prometheus_tg.arn]
  }

  statement {
    sid       = "AllowDescribeHealth"
    effect    = "Allow"
    actions   = [
      "elasticloadbalancing:DescribeTargetHealth"
    ]
    resources = [aws_lb_target_group.prometheus_tg.arn]
  }
}

resource "aws_iam_policy" "tgb_strict_policy" {
  name        = "eks-alb-controller-tgb-restricted"
  description = "Restricted access for TargetGroupBinding to specific TGs only"
  policy      = data.aws_iam_policy_document.lb_controller_tgb_policy.json
}

Part 3: The Glue (TargetGroupBinding CRD)

The TargetGroupBinding CRD tells the controller: "Watch this Kubernetes Service, and whenever its endpoints change, update this specific AWS Target Group."

A raw TGB manifest looks like this:

apiVersion: elbv2.k8s.aws/v1beta1
kind: TargetGroupBinding
metadata:
  name: prometheus-tgb
  namespace: monitoring
spec:
  serviceRef:
    name: prometheus-k8s # The name of your K8s Service
    port: 9090           # The port defined in the Service
  targetGroupARN: <YOUR_TF_OUTPUT_ARN>

Part 4: Application Deployment (Prometheus Helm Values)

We don't want to apply that YAML manually. We want it version-controlled with our Prometheus deployment.

Most Prometheus Helm charts (like kube-prometheus-stack) support an extraManifests or additionalManifests property in their values.yaml. This allows us to inject arbitrary K8s objects-like our TGB-directly during the Helm install.

Here is how you configure your values.yaml to register Prometheus to the Terraform-managed Target Group.

`values.yaml`

# Configuration for kube-prometheus-stack
prometheus:
  service:
    port: 9090
    # Ensure the service selector matches what the TGB expects
    # usually standard, but good to verify.

# Injecting the Custom Resource Definition
extraManifests:
  - apiVersion: elbv2.k8s.aws/v1beta1
    kind: TargetGroupBinding
    metadata:
      name: prometheus-binding
      namespace: monitoring # Must match the release namespace
    spec:
      serviceRef:
        name: prometheus-kube-prometheus-prometheus # Default name in the stack
        port: 9090
      targetGroupARN: "arn:aws:elasticloadbalancing:us-east-1:1234567890:targetgroup/eks-prometheus-tg/6d0ecf831eec9f09"

Summary of Flow

Terraform creates the empty Target Group (Mode: IP).
Helm deploys Prometheus + the TargetGroupBinding CR.
AWS Load Balancer Controller sees the new Binding CR.
Controller looks up the Pod IPs backing the Prometheus Service.
Controller registers those Pod IPs into the AWS Target Group automatically.

This approach gives us the stability of Terraform-managed Infrastructure (the ALB and Listeners) with the flexibility of Kubernetes-managed endpoints.

Secure AWS Access in Kubernetes: Transitioning from Secrets to IRSA or Pod Identity

Paweł Swiridow — Mon, 22 Sep 2025 09:07:05 +0000

Traditional approaches to AWS authorization from Kubernetes involved creating IAM users with permanent access keys and storing these credentials within the cluster. Modern solutions eliminate stored credentials entirely, using temporary tokens that expire automatically and follow the principle of least privilege.
Think of it like upgrading from physical keys to smart cards for your Kubernetes workloads - instead of storing a master key in every pod, your containers get temporary access cards that only work for specific resources and expire automatically.

The old way: stored credentials (what to avoid)

Before diving into modern solutions, let's understand what we're moving away from. Traditional approaches to AWS access from Kubernetes typically involved:

Create IAM users with access keys (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY)
Store these credentials in Kubernetes secrets, environment variables, or config files
Hardcode them in container images or application code

Here's what this looked like in a typical Kubernetes deployment:

apiVersion: v1
kind: Secret
metadata:
  name: aws-credentials
  namespace: default
type: Opaque
data:
  AWS_ACCESS_KEY_ID: ABCDEF
  AWS_SECRET_ACCESS_KEY: GHIJKL
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: legacy-app
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: legacy-app
  template:
    metadata:
      labels:
        app: legacy-app
    spec:
      containers:
      - name: legacy-app
        image: my-app:latest
        env:
        - name: AWS_ACCESS_KEY_ID
          valueFrom:
            secretKeyRef:
              name: aws-credentials
              key: AWS_ACCESS_KEY_ID
        - name: AWS_SECRET_ACCESS_KEY
          valueFrom:
            secretKeyRef:
              name: aws-credentials
              key: AWS_SECRET_ACCESS_KEY
        - name: AWS_REGION
          value: "us-west-2"

This approach creates several security and operational challenges in Kubernetes environments:

Security risks:

Long-lived credentials don't expire automatically
Keys can be extracted from running containers or Kubernetes secrets
Credential leaks provide persistent access until manually revoked
Difficult to audit credential usage across pods and namespaces

Operational challenges:

Manual key rotation across all pods and clusters
Credential sprawl as keys get copied across namespaces and environments
Complex secret management across multiple Kubernetes clusters

Modern authorization methods for Kubernetes

Modern AWS authorization from Kubernetes eliminates stored credentials by using temporary tokens that are automatically refreshed. Two primary solutions exist for Kubernetes-to-AWS authentication:

IRSA (IAM roles for service accounts)

IRSA uses OpenID Connect (OIDC) to bridge Kubernetes service accounts with AWS IAM roles. When a pod needs AWS access, it presents a JWT token that AWS validates through the cluster's OIDC provider, receiving temporary credentials in return.

How it works:

Pod receives a JWT token from the Kubernetes API server automatically
AWS SDK includes this token in API requests
AWS STS validates the token through the EKS cluster's OIDC provider
STS returns temporary AWS credentials to the pod
Pod uses these credentials to access AWS services

EKS pod identity

EKS Pod Identity is AWS's newer approach, using a cluster add-on that handles credential exchange automatically. A Pod Identity agent runs as a DaemonSet on each Kubernetes node, intercepting AWS API calls and fetching credentials from AWS's Pod Identity service.

How it works:

Pod makes AWS API calls through the AWS SDK
Pod Identity agent (running as DaemonSet) intercepts these calls
Agent requests credentials from the EKS Pod Identity service
Service returns temporary credentials to the agent
Original API call proceeds with valid credentials

Implementation with Terraform

Let's implement both approaches using consistent IAM policies. We'll use a common S3 access policy for both examples that pods can use to access AWS services:


resource "aws_iam_policy" "s3_access" {
  name        = "kubernetes-app-s3-access"
  description = "S3 access policy for Kubernetes applications"

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "s3:GetObject",
          "s3:PutObject",
          "s3:DeleteObject"
        ]
        Resource = "arn:aws:s3:::my-k8s-app-bucket/*"
      },
      {
        Effect = "Allow"
        Action = [
          "s3:ListBucket"
        ]
        Resource = "arn:aws:s3:::my-k8s-app-bucket"
      }
    ]
  })
}

Setting up IRSA for EKS

Configure your EKS cluster with OIDC enabled to support IRSA:


module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 19.0.0"

  cluster_name    = "my-k8s-cluster"
  cluster_version = "1.28"

# create an OpenID Connect Provider for EKS to enable IRSA (IAM Roles for Service Accounts)
  enable_irsa = true

  vpc_id     = var.vpc_id
  subnet_ids = var.private_subnets

  eks_managed_node_groups = {
    main = {
      desired_size = 2
      max_size     = 4
      min_size     = 1
      instance_types = ["t3.medium"]
    }
  }
}

Create the IAM role that your Kubernetes service account will assume:


module "irsa_role" {
  source = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"

  role_name = "k8s-app-irsa-role"

  role_policy_arns = {
    s3_access = aws_iam_policy.s3_access.arn
  }

  oidc_providers = {
    main = {
      provider_arn               = module.eks.oidc_provider_arn
      namespace_service_accounts = ["default:k8s-app-service-account"]
    }
  }
}

Setting up EKS pod identity

Enable the Pod Identity add-on in your EKS cluster:


module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 19.0.0"

  cluster_name    = "my-k8s-cluster"
  cluster_version = "1.28"

  cluster_addons = {
    eks-pod-identity-agent = {
      addon_version = "v1.3.4-eksbuild.1"
    }
  }

  vpc_id     = var.vpc_id
  subnet_ids = var.private_subnets

  eks_managed_node_groups = {
    main = {
      desired_size = 2
      max_size     = 4
      min_size     = 1
      instance_types = ["t3.medium"]
    }
  }
}

Create the Pod Identity association to link your Kubernetes service account with AWS permissions:


module "pod_identity" {
  source = "terraform-aws-modules/eks-pod-identity/aws"

  name = "k8s-app-pod-identity"

  additional_policy_arns = {
    S3Access = aws_iam_policy.s3_access.arn
  }

  associations = {
    default = {
      cluster_name    = module.eks.cluster_name
      namespace       = "default"
      service_account = "k8s-app-service-account"
    }
  }
}

Kubernetes manifests

IRSA application deployment


apiVersion: v1
kind: ServiceAccount
metadata:
  name: k8s-app-service-account
  namespace: default
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/k8s-app-irsa-role
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: k8s-app-irsa
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: k8s-app-irsa
  template:
    metadata:
      labels:
        app: k8s-app-irsa
    spec:
      serviceAccountName: k8s-app-service-account
      containers:
      - name: app
        image: my-app:latest
        env:
        - name: AWS_REGION
          value: "us-west-2"
        ports:
        - containerPort: 8080

Pod identity application deployment


apiVersion: v1
kind: ServiceAccount
metadata:
  name: k8s-app-service-account
  namespace: default
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: k8s-app-pod-identity
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: k8s-app-pod-identity
  template:
    metadata:
      labels:
        app: k8s-app-pod-identity
    spec:
      serviceAccountName: k8s-app-service-account
      containers:
      - name: app
        image: my-app:latest
        env:
        - name: AWS_REGION
          value: "us-west-2"
        ports:
        - containerPort: 8080

Notice how both modern approaches eliminate the need for stored credentials entirely in your Kubernetes cluster. The Pod Identity deployment is even simpler, requiring no special annotations on the service account.

Comparing modern approaches for Kubernetes

IRSA strengths and limitations

Strengths:

Works with any Kubernetes cluster, not just EKS
Mature technology with extensive community support
Fine-grained control over JWT token configuration
Compatible with other OIDC-based systems and service meshes

Limitations:

More complex setup and troubleshooting in Kubernetes
Requires OIDC provider configuration at the cluster level
JWT tokens can become large and cause issues with some applications
Manual annotation management required for each service account

Pod identity strengths and limitations

Strengths:

Simplified setup and ongoing management in EKS
Native AWS integration with better performance
Automatic credential refresh and rotation handled by the agent
Built-in audit logging capabilities through CloudTrail
No JWT token size limitations or complexity

Limitations:

EKS-only solution, not portable to other Kubernetes distributions
Newer technology with less community knowledge and troubleshooting guides
Requires EKS cluster version 1.24 or higher
Less customization flexibility compared to OIDC-based approaches

Security best practices for Kubernetes workloads

Regardless of which modern approach you choose for your Kubernetes cluster, follow these security principles:

Principle of least privilege:


# Good: Specific permissions for specific resources used by pods
resource "aws_iam_policy" "restricted_s3" {
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = ["s3:GetObject"]
        Resource = "arn:aws:s3:::k8s-specific-bucket/app-folder/*"
      }
    ]
  })
}

# Avoid: Overly broad permissions that pods don't need
resource "aws_iam_policy" "too_broad" {
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = ["s3:*"]
        Resource = "*"
      }
    ]
  })
}

Additional security measures for Kubernetes:

Use Kubernetes namespace isolation to limit service account scope
Implement regular auditing of role usage and permissions across clusters
Monitor credential usage through AWS CloudTrail and Kubernetes audit logs
Set up alerts for unusual access patterns from pods
Use Kubernetes Network Policies to restrict pod-to-pod communication
Regularly review and rotate any remaining long-lived credentials in your cluster

Choosing the right approach

Choose IRSA when:

You need multi-cloud or hybrid Kubernetes support
You have complex OIDC integration requirements
Your team has existing IRSA expertise
You need fine-grained token control

Choose Pod Identity when:

You're using EKS exclusively
You want simplified management overhead
You're starting new projects from scratch
You prefer AWS-native solutions

Migrate from stored credentials when:

You're currently using long-lived access keys
You want to improve security posture
You need better credential rotation
You want to reduce operational overhead

Conclusion

Modern AWS authorization methods eliminate the security risks and operational complexity of stored credentials in Kubernetes environments. Both IRSA and EKS Pod Identity provide secure, temporary credential access that follows cloud security best practices while integrating seamlessly with Kubernetes workflows.

For new EKS projects, Pod Identity offers the simplest path forward with native AWS integration and reduced management overhead. For existing Kubernetes environments or multi-platform requirements, IRSA provides proven reliability and flexibility across different cluster types.

The most important step is moving away from stored credentials entirely in your Kubernetes clusters. Choose the modern approach that best fits your team's expertise and infrastructure requirements - both represent significant improvements over traditional credential management and will enhance your cluster's security posture while simplifying operations.

Secure AWS Access in Kubernetes: Transitioning from Secrets to IRSA or Pod Identity

Paweł Swiridow — Mon, 22 Sep 2025 07:00:00 +0000

The old way: stored credentials (what to avoid)

Before diving into modern solutions, let's understand what we're moving away from. Traditional approaches to AWS access from Kubernetes typically involved:

Create IAM users with access keys (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY)
Store these credentials in Kubernetes secrets, environment variables, or config files
Hardcode them in container images or application code

Here's what this looked like in a typical Kubernetes deployment:

apiVersion: v1
kind: Secret
metadata:
  name: aws-credentials
  namespace: default
type: Opaque
data:
  AWS_ACCESS_KEY_ID: ABCDEF
  AWS_SECRET_ACCESS_KEY: GHIJKL
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: legacy-app
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: legacy-app
  template:
    metadata:
      labels:
        app: legacy-app
    spec:
      containers:
      - name: legacy-app
        image: my-app:latest
        env:
        - name: AWS_ACCESS_KEY_ID
          valueFrom:
            secretKeyRef:
              name: aws-credentials
              key: AWS_ACCESS_KEY_ID
        - name: AWS_SECRET_ACCESS_KEY
          valueFrom:
            secretKeyRef:
              name: aws-credentials
              key: AWS_SECRET_ACCESS_KEY
        - name: AWS_REGION
          value: "us-west-2"

This approach creates several security and operational challenges in Kubernetes environments:

Security risks:

Long-lived credentials don't expire automatically
Keys can be extracted from running containers or Kubernetes secrets
Credential leaks provide persistent access until manually revoked
Difficult to audit credential usage across pods and namespaces

Operational challenges:

Manual key rotation across all pods and clusters
Credential sprawl as keys get copied across namespaces and environments
Complex secret management across multiple Kubernetes clusters

Modern authorization methods for Kubernetes

Modern AWS authorization from Kubernetes eliminates stored credentials by using temporary tokens that are automatically refreshed. Two primary solutions exist for Kubernetes-to-AWS authentication:

IRSA (IAM roles for service accounts)

How it works:

Pod receives a JWT token from the Kubernetes API server automatically
AWS SDK includes this token in API requests
AWS STS validates the token through the EKS cluster's OIDC provider
STS returns temporary AWS credentials to the pod
Pod uses these credentials to access AWS services

EKS pod identity

How it works:

Pod makes AWS API calls through the AWS SDK
Pod Identity agent (running as DaemonSet) intercepts these calls
Agent requests credentials from the EKS Pod Identity service
Service returns temporary credentials to the agent
Original API call proceeds with valid credentials

Implementation with Terraform

Let's implement both approaches using consistent IAM policies. We'll use a common S3 access policy for both examples that pods can use to access AWS services:


resource "aws_iam_policy" "s3_access" {
  name        = "kubernetes-app-s3-access"
  description = "S3 access policy for Kubernetes applications"

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "s3:GetObject",
          "s3:PutObject",
          "s3:DeleteObject"
        ]
        Resource = "arn:aws:s3:::my-k8s-app-bucket/*"
      },
      {
        Effect = "Allow"
        Action = [
          "s3:ListBucket"
        ]
        Resource = "arn:aws:s3:::my-k8s-app-bucket"
      }
    ]
  })
}

Setting up IRSA for EKS

Configure your EKS cluster with OIDC enabled to support IRSA:


module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 19.0.0"

  cluster_name    = "my-k8s-cluster"
  cluster_version = "1.28"

# create an OpenID Connect Provider for EKS to enable IRSA (IAM Roles for Service Accounts)
  enable_irsa = true

  vpc_id     = var.vpc_id
  subnet_ids = var.private_subnets

  eks_managed_node_groups = {
    main = {
      desired_size = 2
      max_size     = 4
      min_size     = 1
      instance_types = ["t3.medium"]
    }
  }
}

Create the IAM role that your Kubernetes service account will assume:


module "irsa_role" {
  source = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"

  role_name = "k8s-app-irsa-role"

  role_policy_arns = {
    s3_access = aws_iam_policy.s3_access.arn
  }

  oidc_providers = {
    main = {
      provider_arn               = module.eks.oidc_provider_arn
      namespace_service_accounts = ["default:k8s-app-service-account"]
    }
  }
}

Setting up EKS pod identity

Enable the Pod Identity add-on in your EKS cluster:


module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 19.0.0"

  cluster_name    = "my-k8s-cluster"
  cluster_version = "1.28"

  cluster_addons = {
    eks-pod-identity-agent = {
      addon_version = "v1.3.4-eksbuild.1"
    }
  }

  vpc_id     = var.vpc_id
  subnet_ids = var.private_subnets

  eks_managed_node_groups = {
    main = {
      desired_size = 2
      max_size     = 4
      min_size     = 1
      instance_types = ["t3.medium"]
    }
  }
}

Kubernetes manifests

IRSA application deployment


apiVersion: v1
kind: ServiceAccount
metadata:
  name: k8s-app-service-account
  namespace: default
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/k8s-app-irsa-role
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: k8s-app-irsa
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: k8s-app-irsa
  template:
    metadata:
      labels:
        app: k8s-app-irsa
    spec:
      serviceAccountName: k8s-app-service-account
      containers:
      - name: app
        image: my-app:latest
        env:
        - name: AWS_REGION
          value: "us-west-2"
        ports:
        - containerPort: 8080

Pod identity application deployment


apiVersion: v1
kind: ServiceAccount
metadata:
  name: k8s-app-service-account
  namespace: default
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: k8s-app-pod-identity
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: k8s-app-pod-identity
  template:
    metadata:
      labels:
        app: k8s-app-pod-identity
    spec:
      serviceAccountName: k8s-app-service-account
      containers:
      - name: app
        image: my-app:latest
        env:
        - name: AWS_REGION
          value: "us-west-2"
        ports:
        - containerPort: 8080

Comparing modern approaches for Kubernetes

IRSA strengths and limitations

Strengths:

Works with any Kubernetes cluster, not just EKS
Mature technology with extensive community support
Fine-grained control over JWT token configuration
Compatible with other OIDC-based systems and service meshes

Limitations:

More complex setup and troubleshooting in Kubernetes
Requires OIDC provider configuration at the cluster level
JWT tokens can become large and cause issues with some applications
Manual annotation management required for each service account

Pod identity strengths and limitations

Strengths:

Simplified setup and ongoing management in EKS
Native AWS integration with better performance
Automatic credential refresh and rotation handled by the agent
Built-in audit logging capabilities through CloudTrail
No JWT token size limitations or complexity

Limitations:

EKS-only solution, not portable to other Kubernetes distributions
Newer technology with less community knowledge and troubleshooting guides
Requires EKS cluster version 1.24 or higher
Less customization flexibility compared to OIDC-based approaches

Security best practices for Kubernetes workloads

Regardless of which modern approach you choose for your Kubernetes cluster, follow these security principles:

Principle of least privilege:


# Good: Specific permissions for specific resources used by pods
resource "aws_iam_policy" "restricted_s3" {
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = ["s3:GetObject"]
        Resource = "arn:aws:s3:::k8s-specific-bucket/app-folder/*"
      }
    ]
  })
}

# Avoid: Overly broad permissions that pods don't need
resource "aws_iam_policy" "too_broad" {
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = ["s3:*"]
        Resource = "*"
      }
    ]
  })
}

Additional security measures for Kubernetes:

Use Kubernetes namespace isolation to limit service account scope
Implement regular auditing of role usage and permissions across clusters
Monitor credential usage through AWS CloudTrail and Kubernetes audit logs
Set up alerts for unusual access patterns from pods
Use Kubernetes Network Policies to restrict pod-to-pod communication
Regularly review and rotate any remaining long-lived credentials in your cluster

Choosing the right approach

Choose IRSA when:

You need multi-cloud or hybrid Kubernetes support
You have complex OIDC integration requirements
Your team has existing IRSA expertise
You need fine-grained token control

Choose Pod Identity when:

You're using EKS exclusively
You want simplified management overhead
You're starting new projects from scratch
You prefer AWS-native solutions

Migrate from stored credentials when:

You're currently using long-lived access keys
You want to improve security posture
You need better credential rotation
You want to reduce operational overhead

Conclusion

Configuring EKS Managed Node Groups to Use a Proxy with Terraform

Paweł Swiridow — Thu, 18 Sep 2025 08:33:05 +0000

In enterprise environments, security and network policies are paramount. It's common practice for servers, including Kubernetes worker nodes, to reside in private subnets with no direct outbound internet access. Instead, all egress traffic must be routed through a centrally managed and monitored HTTP/HTTPS proxy. While this enhances security, it introduces a configuration challenge for services like Amazon EKS, which need to pull container images, communicate with AWS APIs, and perform other bootstrap tasks.

When using the popular terraform-aws-modules/eks/aws module to provision your cluster, you need a reliable, automated way to inject this proxy configuration into your managed node groups. The goal is to ensure that the operating system, the container runtime (containerd), and the Kubernetes components themselves are all proxy-aware from the moment they boot.

This article details a robust solution using the cloudinit_pre_nodeadm hook provided by the module to configure your EKS nodes for a proxy environment.

The core challenge: comprehensive proxy awareness

Simply setting HTTP_PROXY and HTTPS_PROXY environment variables is not enough. A modern EKS node, built on the Amazon Linux 2 EKS Optimized AMI, has several key components that must be configured independently:

The Base Operating System: For shell sessions and general system utilities.
The Container Runtime (containerd): This is critical. Without proxy settings, containerd will fail to pull any images from public registries like Docker Hub or even private registries outside the VPC, such as Amazon ECR.
The EKS Bootstrap Process (nodeadm): The new bootstrapping agent for EKS AMIs needs to communicate with the EKS control plane. This communication should bypass the proxy.
Package Managers (yum): If you need to install any additional packages as part of your user data, yum must also be configured to use the proxy.

The solution is to use a cloud-init script that runs before the main EKS bootstrapping logic, ensuring the entire environment is correctly configured before any Kubernetes processes are started.

Understanding the bootstrap hook: `cloudinit_pre_nodeadm`

Before diving into the code, it's essential to understand the mechanism we're using. The name cloudinit_pre_nodeadm breaks down into two key concepts:

cloud-init: This is the de facto industry standard for early-stage initialization of cloud instances. When a new EC2 instance boots for the first time, it runs cloud-init, which executes a series of user-provided instructions (user data) to configure the operating system, install packages, create files, and set up users before the machine is considered fully "ready".
nodeadm: Starting with the EKS Optimized Amazon Linux 2023 AMI, nodeadm is the official bootstrap agent responsible for configuring a node and joining it to an EKS cluster. It handles tasks like retrieving cluster certificates, configuring the kubelet, and applying node labels and taints. It has replaced the legacy bootstrap.sh script.

The cloudinit_pre_nodeadm parameter, provided by the terraform-aws-modules/eks/aws module, is a powerful hook that lets you run a custom cloud-init script at a precise moment: after basic OS initialization but before the nodeadm service is started.

This timing is critical. By executing our script before nodeadm, we ensure that the foundational network environment—including our proxy settings—is already in place. When nodeadm, containerd, and the kubelet eventually start, they inherit the correct configuration from the environment, allowing them to function properly within the restricted network.

The solution: using `cloudinit_pre_nodeadm` in Terraform

Our solution will pass the Kubernetes service CIDR from our Terraform configuration directly into the user data script. This removes hardcoded values and makes the solution reusable across different clusters.

Here is the Terraform code block that implements the complete proxy configuration.

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 21.0"

  # Define the service CIDR for the cluster
  cluster_service_ipv4_cidr = "10.100.0.0/16"

  # ... other EKS cluster configuration ...

  eks_managed_node_groups = {
    main_nodes = {
      # ... other node group configuration like instance_types, min_size, etc. ...

      cloudinit_pre_nodeadm = [
        {
          content_type = "text/x-shellscript"
          content      = <<-EOT
            #!/bin/bash
            set -ex

            # Define your proxy endpoint
            PROXY="http://your-proxy-url:3128" # standard proxy port

            # Use IMDSv2 to securely fetch instance metadata
            TOKEN=`curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"`
            MAC=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/mac)
            VPC_CIDR=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/network/interfaces/macs/$MAC/vpc-ipv4-cidr-blocks | xargs | tr ' ' ',')
                        K8S_SERVICE_CIDR="${module.eks.cluster_service_ipv4_cidr}"

            # Define your NO_PROXY list. This is critical for internal and AWS service communication.
            # You might need to add your Pod CIDR as well.
            NO_PROXY="$VPC_CIDR,$K8S_SERVICE_CIDR,localhost,127.0.0.1,169.254.169.254,.amazonaws.com,.svc.cluster.local,.svc,.cluster.local"

            # 1. Configure system-wide proxy settings
            echo "Setting up /etc/environment"
            cat << EOF > /etc/environment
            HTTP_PROXY=$PROXY
            HTTPS_PROXY=$PROXY
            NO_PROXY=$NO_PROXY
            http_proxy=$PROXY
            https_proxy=$PROXY
            no_proxy=$NO_PROXY
            EOF

            # 2. Configure containerd proxy via systemd override
            echo "Configuring containerd service"
            mkdir -p /etc/systemd/system/containerd.service.d
            cat << EOF > /etc/systemd/system/containerd.service.d/http-proxy.conf
            [Service]
            EnvironmentFile=/etc/environment
            EOF

            # 3. Configure nodeadm proxy via systemd override
            echo "Configuring nodeadm service"
            mkdir -p /etc/systemd/system/nodeadm.service.d
            cat << EOF > /etc/systemd/system/nodeadm.service.d/http-proxy.conf
            [Service]
            EnvironmentFile=/etc/environment
            EOF

            # 4. Configure yum proxy
            echo "Configuring yum"
            echo "proxy=$PROXY" >> /etc/yum.conf

            # 5. Reload systemd daemon and restart containerd to apply changes
            echo "Reloading systemd and restarting containerd"
            systemctl daemon-reload
            systemctl restart containerd
          EOT
        }
      ]
    }
  }

  # ... other module configuration ...
}

Dissecting the script

Let's break down the content of the cloud-init script to understand each step.

set -ex: This is a standard best practice for shell scripting. set -e ensures the script will exit immediately if a command fails, and set -x prints each command before it is executed, providing clear debug output in the system logs.
Fetching metadata with IMDSv2: The script dynamically fetches the VPC CIDR block directly from the EC2 metadata service. This is crucial for the NO_PROXY variable, ensuring that any traffic destined for other resources within the VPC bypasses the proxy. It uses the more secure IMDSv2 method, which requires a session token.
Defining NO_PROXY: The NO_PROXY variable is just as important as HTTP_PROXY. It specifies a comma-separated list of domains and IP ranges that should not use the proxy. Our list includes:
- The dynamically fetched $VPC_CIDR for all internal VPC traffic.
- The Kubernetes service CIDR passed from module via $K8S_SERVICE_CIDR (e.g., 10.100.0.0/16).
- localhost and the metadata service address (169.254.169.254).
- Key AWS and Kubernetes service endpoints to ensure direct communication with the control plane and other AWS APIs. Note the addition of .amazonaws.com to ensure services like ECR, S3, and EC2 are accessed directly via VPC Endpoints if they exist.
/etc/environment: This file provides the system-wide environment variables for all users and processes. It's the foundation of our configuration.
Systemd overrides: The modern, correct way to modify a systemd service like containerd or nodeadm is to create an override file. We create http-proxy.conf for both services. The EnvironmentFile=/etc/environment directive instructs systemd to load all variables from our global configuration file before starting the service. This is cleaner and more maintainable than modifying the main service files directly.
yum.conf: A simple line added to /etc/yum.conf makes the package manager proxy-aware.
Reload and restart: Finally, systemctl daemon-reload forces systemd to re-read its configuration files, and systemctl restart containerd applies the new environment variables to the container runtime immediately. nodeadm will pick up its configuration when it runs shortly after this script completes.

Successfully deploying EKS worker nodes in a network firewalled behind an HTTP proxy is a common enterprise challenge, but it doesn't have to be a complex one. By leveraging the cloudinit_pre_nodeadm hook within the terraform-aws-modules/eks/aws module, you gain precise control over the node's bootstrap sequence.

This method provides a declarative, automated, and robust solution. By injecting a dynamic script that configures the operating system, containerd, and nodeadm before these critical services start, you ensure nodes join the cluster and pull images reliably. This Infrastructure as Code approach not only solves the immediate technical problem but also creates a repeatable, version-controlled pattern for building secure and compliant Kubernetes platforms on AWS.

Centralized EKS Logging Across AWS Accounts Using Fluent Bit, Pod Identity and OpenSearch

Paweł Swiridow — Wed, 13 Aug 2025 08:47:52 +0000

Centralized log collection is essential in distributed Kubernetes environments. Amazon EKS combined with Fluent Bit offers a lightweight and flexible log forwarding solution. In this guide, we configure Fluent Bit to run on an EKS cluster in one AWS account (Account A) and forward logs to an OpenSearch domain hosted in another account (Account B).

We’ll secure the communication using EKS Pod Identity and a cross-account IAM role protected with an external_id. This ensures that no long-lived credentials are exposed and the trust relationship cannot be abused.

What is Fluent Bit?

Fluent Bit is a lightweight, high-performance log processor and forwarder. Designed with efficiency in mind, it’s ideal for environments like Kubernetes, where resource constraints matter. It collects logs from various sources (e.g., files, containers, syslog), applies filtering and transformation, and then routes them to different destinations such as OpenSearch, Elasticsearch, or cloud services.

What is OpenSearch?

OpenSearch is an open-source search and analytics engine derived from Elasticsearch 7.10. It’s developed and maintained by AWS and the community. OpenSearch supports full-text search, distributed indexing, and real-time analytics — making it an excellent backend for log aggregation, monitoring, and observability platforms.

Architecture overview

account A: hosts the EKS cluster and Fluent Bit
account B: hosts the OpenSearch domain
Fluent Bit pods use a pod identity to assume a cross-account role that grants access to write logs into OpenSearch

To prevent the confused deputy problem, the trust policy in account B includes an external_id that must be known and passed by the caller (Fluent Bit).

Prerequisites

EKS cluster deployed in account A with Pod Identity Agent installed
OpenSearch domain deployed in account B
Terraform v1.6+
kubectl access to the cluster
helm

Creating the pod identity role (account A)

resource "aws_iam_policy" "log_writer_assuming_policy" {
  name = "LogWriterAssumingPolicy"

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "sts:AssumeRole",
          "sts:TagSession"
        ]
        Resource = "arn:aws:iam::ACCOUNT_B_ID:role/fluentbit-opensearch-writer"
      }
    ]
  })
}

module "fluentbit_pod_identity" {
  source  = "terraform-aws-modules/eks-pod-identity/aws"
  version = "1.11.0"

  name = "fluentbit-pod-identity"

  additional_policy_arns = {
    LogWriterAssumingPolicy = aws_iam_policy.log_writer_assuming_policy.arn
  }

  associations = {
    main = {
      namespace       = "middlewares"
      service_account = "fluentbit"
      cluster_name    = var.cluster_name
    }
  }
}

The log_writer_assuming_policy is a policy that allows sts:AssumeRole on the cross-account writer role in account B.

EKS Pod Identity uses session tags (therefore sts:TagSession action) to pass metadata during the AssumeRole operation — specifically:

Kubernetes namespace
Service account name
Cluster name

These are passed as session tags when the EKS control plane assumes the IAM role on behalf of the pod. AWS uses those tags internally to enforce the correct role-to-service-account association, ensuring that only the intended pods can assume that role.

Creating the OpenSearch writer role (account B)

This role trusts only account A to assume it and requires an external ID for additional protection:

data "aws_iam_policy_document" "trust_policy" {
  statement {
    effect = "Allow"

    principals {
      type        = "AWS"
      identifiers = ["arn:aws:iam::ACCOUNT_A_ID:role/fluentbit-pod-identity"]
    }

    actions = ["sts:AssumeRole"]

    condition {
      test     = "StringEquals"
      variable = "sts:ExternalId"
      values   = ["shared-external-id-between-a-and-b-accounts"]
    }
  }
}

resource "aws_iam_role" "fluentbit_writer" {
  name               = "fluentbit-opensearch-writer"
  assume_role_policy = data.aws_iam_policy_document.trust_policy.json
}

Attaching OpenSearch permissions

This policy grants Fluent Bit permission to write to the OpenSearch domain (cluster):

resource "aws_iam_policy" "write_to_opensearch" {
  name = "opensearch-write-access"

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "es:ESHttpPost",
          "es:ESHttpPut"
        ]
        Resource = "arn:aws:es:AWS_REGION:ACCOUNT_B_ID:domain/OPENSEARCH_DOMAIN_NAME/*"
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "attach_writer_policy" {
  role       = aws_iam_role.fluentbit_writer.name
  policy_arn = aws_iam_policy.write_to_opensearch.arn
}

Deploying Fluent Bit on EKS with helm

We use the official Helm chart from the eks-charts repo. It supports pod identity and AWS-signed requests to OpenSearch.

Make sure the IAM role from account B is correctly assumed by Fluent Bit using the pod identity role from account A. Also, pass the external ID as a Helm value. Remember that AWS_Role_ARN has to be an ARN of role in account B. Service account name should be the same as we set up in fluentbit_pod_identity module.

helm repo add eks-charts https://aws.github.io/eks-charts
helm repo update

helm upgrade --install fluentbit eks-charts/aws-for-fluent-bit \
  --namespace middlewares \
  --create-namespace \
  --set serviceAccount.name="fluentbit-service-account-name" \
  --set output.opensearch.enabled=true \
  --set output.opensearch.host="vpc-my-domain.region.es.amazonaws.com" \
  --set output.opensearch.port=443 \
  --set output.opensearch.awsAuth=true \
  --set output.opensearch.awsExternalId="shared-external-id-between-a-and-b-accounts" \
  --set output.opensearch.AWS_Role_ARN="arn:aws:iam::ACCOUNT_B_ID:role/fluentbit-opensearch-writer" \
  --set output.opensearch.index="index-name" \
  --set output.opensearch.type="_doc"

Verifying the setup

Check the logs of the Fluent Bit pods:

kubectl logs -n middlewares -l app.kubernetes.io/name=fluentbit

You should see lines indicating successful writes:

[ info] [output:es:es.0] HTTP status=200

You can also search in the OpenSearch dashboard to confirm logs are being indexed in eks-logs.

Conclusion

This setup allows Fluent Bit to securely ship logs from EKS in account A to OpenSearch in account B. By using EKS Pod Identity and a cross-account IAM role with an external ID, we avoid exposing secrets while still maintaining a strong trust boundary.

This architecture is well-suited for multi-account environments and aligns with AWS best practices for identity and access control.

DEV Community: Paweł Swiridow

Decoupling Ingress with TargetGroupBinding in EKS

Prerequisites

Part 1: Infrastructure (Terraform)

main.tf

Part 2: Security (IAM Least Privilege)

iam.tf

Part 3: The Glue (TargetGroupBinding CRD)

Part 4: Application Deployment (Prometheus Helm Values)

values.yaml

Summary of Flow

Secure AWS Access in Kubernetes: Transitioning from Secrets to IRSA or Pod Identity

The old way: stored credentials (what to avoid)

Modern authorization methods for Kubernetes

IRSA (IAM roles for service accounts)

EKS pod identity

Implementation with Terraform

Setting up IRSA for EKS

Setting up EKS pod identity

Kubernetes manifests

IRSA application deployment

Pod identity application deployment

Comparing modern approaches for Kubernetes

IRSA strengths and limitations

Pod identity strengths and limitations

Security best practices for Kubernetes workloads

Choosing the right approach

Conclusion

Secure AWS Access in Kubernetes: Transitioning from Secrets to IRSA or Pod Identity

The old way: stored credentials (what to avoid)

Modern authorization methods for Kubernetes

IRSA (IAM roles for service accounts)

EKS pod identity

Implementation with Terraform

Setting up IRSA for EKS

Setting up EKS pod identity

Kubernetes manifests

IRSA application deployment

Comparing modern approaches for Kubernetes

IRSA strengths and limitations

Pod identity strengths and limitations

Security best practices for Kubernetes workloads

Choosing the right approach

Conclusion

Configuring EKS Managed Node Groups to Use a Proxy with Terraform

The core challenge: comprehensive proxy awareness

Understanding the bootstrap hook: cloudinit_pre_nodeadm

The solution: using cloudinit_pre_nodeadm in Terraform

Dissecting the script

Centralized EKS Logging Across AWS Accounts Using Fluent Bit, Pod Identity and OpenSearch

What is Fluent Bit?

What is OpenSearch?

Architecture overview

Prerequisites

Creating the pod identity role (account A)

Creating the OpenSearch writer role (account B)

Attaching OpenSearch permissions

Deploying Fluent Bit on EKS with helm

Verifying the setup

Conclusion

`main.tf`

`iam.tf`

`values.yaml`

Understanding the bootstrap hook: `cloudinit_pre_nodeadm`

The solution: using `cloudinit_pre_nodeadm` in Terraform