Navigating AWS EKS with Terraform: Understanding EKS Cluster Configuration

In our journey through the AWS EKS ecosystem, we have laid a solid foundation with VPC networking and security groups. Now we will discuss the Terraform resources required to implement a working cluster.

The entire project will look like this by the end:

.
├── eks.tf
├── modules
│   └── aws
│       ├── eks
│       │    ├── main.tf
│       │    ├── outputs.tf
│       │    ├── versions.tf
│       │    └── variables.tf
│       └── vpc
│            ├── main.tf
│            ├── outputs.tf
│            ├── versions.tf
│            └── variables.tf
├── versions.tf
├── variables.tf
└── vpc.tf

Cluster Resource

The Terraform configuration creates an Amazon EKS cluster with enhanced security and logging features:

• Cluster Configuration:

  • name: Sets the cluster's name to a variable, allowing for customizable deployments.
  • role_arn: Specifies the IAM role that EKS will assume to create AWS resources for the cluster.
  • enabled_cluster_log_types: Enables logging for audit, API, and authenticator logs, enhancing security and compliance.

• VPC Configuration:

  • Integrates the cluster with specified private and public subnets, allowing workloads to be placed accordingly for both security and accessibility.
  • Associates the cluster with a specific security group to control inbound and outbound traffic.

• Encryption Configuration:

  • Utilizes a KMS key for encrypting Kubernetes secrets, safeguarding sensitive information.
  • Specifies that the encryption applies to Kubernetes secrets, ensuring that secret data stored in the cluster is encrypted at rest.

• Dependencies:

  • Ensures the cluster is created after the necessary IAM role and policy attachments are in place, maintaining the deployment order and security posture.

# Fetch current AWS account details
data "aws_caller_identity" "current" {}

############################################################################################################
### EKS CLUSTER
############################################################################################################
resource "aws_eks_cluster" "main" {
  name                      = var.cluster_name
  role_arn                  = aws_iam_role.eks_cluster_role.arn
  enabled_cluster_log_types = var.enabled_cluster_log_types

  vpc_config {
    subnet_ids             = concat(var.private_subnets, var.public_subnets)
    security_group_ids     = [aws_security_group.eks_cluster_sg.id]
    endpoint_public_access = true

  }

  encryption_config {
    provider {
      key_arn = aws_kms_key.eks_encryption.arn
    }
    resources = ["secrets"]
  }

  depends_on = [
    aws_iam_role_policy_attachment.eks_cluster_policy,
  ]
}

The Kubernetes provider configuration is necessary for Terraform to interact with your Amazon EKS cluster's Kubernetes API

data "aws_eks_cluster_auth" "main" {
  name = var.cluster_name
}

provider "kubernetes" {
  host                   = aws_eks_cluster.main.endpoint
  cluster_ca_certificate = base64decode(aws_eks_cluster.main.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.main.token
}

OIDC for EKS: Secure Access Management

Integrating OIDC with your EKS cluster enhances security by facilitating identity federation between AWS and Kubernetes. This configuration allows for secure role-based access control, which is crucial for managing access to your cluster.

############################################################################################################
### OIDC CONFIGURATION
############################################################################################################

data "tls_certificate" "eks" {
  url = aws_eks_cluster.main.identity[0].oidc[0].issuer
}

resource "aws_iam_openid_connect_provider" "eks" {
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = [data.tls_certificate.eks.certificates[0].sha1_fingerprint]
  url             = aws_eks_cluster.main.identity[0].oidc[0].issuer
}

Secrets Encryption

In the EKS cluster resource above, this KMS key is referenced. It is needed to encrypt Kubernetes secrets in the cluster.

############################################################################################################
### KMS KEY
############################################################################################################
resource "aws_kms_key" "eks_encryption" {
  description         = "KMS key for EKS cluster encryption"
  policy              = data.aws_iam_policy_document.kms_key_policy.json
  enable_key_rotation = true
}

# alias
resource "aws_kms_alias" "eks_encryption" {
  name          = "alias/eks/${var.cluster_name}"
  target_key_id = aws_kms_key.eks_encryption.id
}

data "aws_iam_policy_document" "kms_key_policy" {
  statement {
    sid = "Key Administrators"
    actions = [
      "kms:Create*",
      "kms:Describe*",
      "kms:Enable*",
      "kms:List*",
      "kms:Put*",
      "kms:Update*",
      "kms:Revoke*",
      "kms:Disable*",
      "kms:Get*",
      "kms:Delete*",
      "kms:ScheduleKeyDeletion",
      "kms:CancelKeyDeletion",
      "kms:Decrypt",
      "kms:DescribeKey",
      "kms:Encrypt",
      "kms:ReEncrypt*",
      "kms:GenerateDataKey*",
      "kms:TagResource"
    ]
    principals {
      type = "AWS"
      identifiers = [
        "arn:aws:iam::${data.aws_caller_identity.current.account_id}:root",
        data.aws_caller_identity.current.arn
      ]
    }
    resources = ["*"]
  }

  statement {
    actions = [
      "kms:Encrypt",
      "kms:Decrypt",
      "kms:ReEncrypt*",
      "kms:GenerateDataKey*",
      "kms:DescribeKey"
    ]
    principals {
      type        = "Service"
      identifiers = ["eks.amazonaws.com"]
    }
    resources = ["*"]
  }
}


resource "aws_iam_policy" "cluster_encryption" {
  name        = "${var.cluster_name}-encryption-policy"
  description = "IAM policy for EKS cluster encryption"
  policy      = data.aws_iam_policy_document.cluster_encryption.json
}

data "aws_iam_policy_document" "cluster_encryption" {
  statement {
    actions = [
      "kms:Encrypt",
      "kms:Decrypt",
      "kms:ListGrants",
      "kms:DescribeKey"
    ]
    resources = [aws_kms_key.eks_encryption.arn]
  }
}

# Granting the EKS Cluster role the ability to use the KMS key
resource "aws_iam_role_policy_attachment" "cluster_encryption" {
  policy_arn = aws_iam_policy.cluster_encryption.arn
  role       = aws_iam_role.eks_cluster_role.name
}

Node Groups: Scaling and Optimization with Amazon EKS

Node Groups in Amazon EKS allow the management of EC2 instances as Kubernetes nodes. These are critical for running your applications and can be customized to suit specific workload needs through AWS Managed Node Groups, Self-Managed Node Groups, and AWS Fargate.

AWS Managed Node Groups

Managed Node Groups simplify node management by automating provisioning, lifecycle management, and scaling. They offer:

• Automated Updates: Automatic application of security patches and OS updates.
• Customization: Via Launch Templates for AMIs, instance types, etc.
• Scalability: Directly adjustable scaling configurations, Kubernetes labels, and AWS tags.

Self-Managed Node Groups

Offering full control over configuration, Self-Managed Node Groups require manual scaling and updates but allow for:

• Custom AMIs: For specific compliance or policy requirements.
• Manual Management: Users handle software and OS updates, leveraging AWS CloudFormation for operations.

Serverless Option: AWS Fargate with Amazon EKS

AWS Fargate abstracts server management, allowing specifications for CPU and RAM without considering instance types. This offers a simplified, serverless option for running containers in EKS.

Comparison Table

Criteria	Managed Node Groups	Self-Managed Nodes	AWS Fargate
Deployment to AWS Outposts	No	Yes	No
Windows Container Support	Yes	Yes (but with at least one compulsory Linux node)	No
Linux Container Support	Yes	Yes	Yes
GPU Workloads	Yes (Amazon Linux)	Yes (Amazon Linux)	No
Custom AMI Deployment	Yes (via Launch Template)	Yes	No
Operating System Maintenance	Automated by AWS	Manual	N/A
SSH Access	Yes	Yes	No (No node OS)
Kubernetes DaemonSets	Yes	Yes	No
Pricing	EC2 instance cost	EC2 instance cost	Fargate pricing per Pod
Update Node AMI	Automated notification & one-click update in EKS console for EKS AMI; manual for custom AMI	Can not be done from EKS console. Manual process using external tools	N/A
Update Node Kubernetes Version	Automated notification & one-click update in EKS console for EKS AMI; manual for custom AMI	Can not be done from EKS console. Manual process using external tools	N/A
Use Amazon EBS with Pods	Yes	Yes	No
Use Amazon EFS with Pods	Yes	Yes	Yes
Use Amazon FSx for Lustre with Pods	Yes	Yes	No

AWS Fargate for EKS provides a higher level of abstraction, managing more server aspects for you. It eliminates the need to select instance types, focusing solely on the required CPU and RAM for your workloads.

Our Managed Node Groups Configuration

For our EKS module, we will use the managed node group Terraform resource.
Our terraform resources below creates a series of node groups for an EKS cluster, where each group's configuration is based on user-defined variables. These node groups are essential for running your Kubernetes workloads on AWS, providing the compute capacity as EC2 instances.

• Specifies the EKS cluster to which these node groups belong.
• Defines the size (min, max, desired) of each node group, allowing for scalability.
• Associates a launch template to dictate the configuration of EC2 instances within the node group.
• Allows selection of instance types, AMI types, and capacity types (e.g., On-Demand or Spot Instances) for flexibility and cost optimization.
• Includes a mechanism to force updates to node groups when the launch template changes.

############################################################################################################
### MANAGED NODE GROUPS
############################################################################################################
resource "aws_eks_node_group" "main" {
  for_each = var.managed_node_groups

  cluster_name    = aws_eks_cluster.main.name
  node_group_name = each.value.name
  node_role_arn   = aws_iam_role.node_role.arn
  subnet_ids      = var.private_subnets

  scaling_config {
    desired_size = each.value.desired_size
    max_size     = each.value.max_size
    min_size     = each.value.min_size
  }

  launch_template {
    id      = aws_launch_template.eks_node_group.id
    version = "$Default"
  }

  instance_types       = each.value.instance_types
  ami_type             = var.default_ami_type
  capacity_type        = var.default_capacity_type
  force_update_version = true
}

Launch Template

This defines a blueprint for the EC2 instances that will be launched as part of the node groups. This template encapsulates various settings and configurations for the instances.

• Configures instance networking, including security groups for controlling access to/from the instances.
• Sets instance metadata options to enhance security, like enforcing IMDSv2 for metadata access.
• Details the block device mappings for instance storage, including the root EBS volume size, type, and deletion policy.
• Automatically tags instances for easier management, cost tracking, and integration with Kubernetes.
• Ensures that the launch template is created with a lifecycle policy that avoids conflicts and dangling resources.

This combined setup allows for automated management of the underlying infrastructure for Kubernetes workloads, leveraging EKS-managed node groups and EC2 launch templates for fine-grained control over instance properties and scaling behaviours.

############################################################################################################
### LAUNCH TEMPLATE
############################################################################################################  
resource "aws_launch_template" "eks_node_group" {
  name_prefix = "${var.cluster_name}-eks-node-group-lt"
  description = "Launch template for ${var.cluster_name} EKS node group"

  vpc_security_group_ids = [aws_security_group.eks_nodes_sg.id]

  metadata_options {
    http_endpoint               = "enabled"
    http_tokens                 = "required"
    http_put_response_hop_limit = 2
    instance_metadata_tags      = "enabled"
  }

  block_device_mappings {
    device_name = "/dev/xvda" # Adjusted to the common root device name for Linux AMIs

    ebs {
      volume_size           = 20    # Disk size specified here
      volume_type           = "gp3" # Example volume type, adjust as necessary
      delete_on_termination = true
    }
  }

  tags = {
    "Name"                                      = "${var.cluster_name}-eks-node-group"
    "kubernetes.io/cluster/${var.cluster_name}" = "owned"
  }

  lifecycle {
    create_before_destroy = true
  }
}

EKS Add-ons: Enhancing Cluster Capabilities

With the aws_eks_addon resource we can pass a list of addons to install. The defaults for this project will be vpc-cni, kube-proxy, coredns and aws-ebs-csi-driver

EKS Add-ons Explanation

• vpc-cni: The Amazon VPC CNI (Container Network Interface) plugin allows Kubernetes pods to have the same IP address inside the pod as they do on the VPC network. This plugin is responsible for providing high-performance networking for Amazon EKS clusters, enabling native VPC networking for Kubernetes pods.

• kube-proxy: Manages network rules on each node. This allows network communication to your pods from network sessions inside or outside of your cluster. It makes sure that each node has the latest network rules for communicating with other nodes in the cluster.

• coredns: A flexible, extensible DNS server that can serve as the Kubernetes cluster DNS. It translates human-readable hostnames like www.example.com into IP addresses. CoreDNS is used for service discovery within the cluster, allowing pods to find each other and services to scale up and down.

• aws-ebs-csi-driver: The Amazon EBS CSI (Container Storage Interface) Driver provides a way to configure and manage Amazon EBS volumes. It allows you to use Amazon EBS volume as persistent storage for stateful applications in EKS. This driver supports dynamic provisioning, snapshots, and resizing of volumes.

############################################################################################################
# PLUGINS
############################################################################################################
data "aws_eks_addon_version" "main" {
  for_each = toset(var.cluster_addons)

  addon_name         = each.key
  kubernetes_version = aws_eks_cluster.main.version
}

resource "aws_eks_addon" "main" {
  for_each = toset(var.cluster_addons)

  cluster_name                = aws_eks_cluster.main.name
  addon_name                  = each.key
  addon_version               = data.aws_eks_addon_version.main[each.key].version
  resolve_conflicts_on_create = "OVERWRITE"
  resolve_conflicts_on_update = "OVERWRITE"
  depends_on = [
    aws_eks_node_group.main
  ]
}

IAM Roles and Policies: Securing EKS

IAM Roles for EKS

Securing your EKS clusters involves creating and associating specific IAM roles and policies, ensuring least privilege access to AWS resources. This step is fundamental in protecting your cluster's interactions with other AWS services.

• EKS Cluster Role and Policies:

  • A role (eks_cluster_role) is created for the EKS cluster to interact with other AWS services.
  • The role trusts the eks.amazonaws.com service to assume the role.
  • Attached policies:
    • CloudWatchFullAccess: Grants full access to CloudWatch, allowing the cluster to log and monitor.
    • AmazonEKSClusterPolicy: Provides permissions that Amazon EKS requires to manage clusters.
    • AmazonEKSVPCResourceController: Allows the cluster to manage VPC resources for the cluster.

• Node Group Role and Policies:

  • An instance profile (eks_node) is created for EKS node groups, associating them with a role (node_role) that nodes assume for AWS service interaction.
  • The role trusts the ec2.amazonaws.com service to assume the role.
  • Attached policies:
    • AmazonEKSWorkerNodePolicy: Grants nodes in the node group the permissions required to operate within an EKS cluster.
    • AmazonEKS_CNI_Policy: Allows nodes to manage network resources, which is necessary for the Amazon VPC CNI plugin to operate.
    • AmazonEBSCSIDriverPolicy: Allows nodes to manage EBS volumes, enabling dynamic volume provisioning.
    • AmazonEC2ContainerRegistryReadOnly: Provides read-only access to AWS Container Registry, allowing nodes to pull container images.

• VPC CNI Plugin Role:

  • A specific role (vpc_cni_role) is created for the VPC CNI plugin to operate within the EKS cluster.
  • The role trusts federated access via the cluster's OIDC provider, specifically for the aws-node Kubernetes service account within the kube-system namespace.
  • Attached policy:
    • AmazonEKS_CNI_Policy: Grants the necessary permissions for the VPC CNI plugin to manage AWS network resources.

############################################################################################################
### IAM ROLES
############################################################################################################
# EKS Cluster role
resource "aws_iam_role" "eks_cluster_role" {
  name               = "${var.cluster_name}-eks-cluster-role"
  assume_role_policy = data.aws_iam_policy_document.eks_assume_role_policy.json
}

data "aws_iam_policy_document" "eks_assume_role_policy" {
  statement {
    actions = ["sts:AssumeRole"]

    principals {
      type        = "Service"
      identifiers = ["eks.amazonaws.com"]
    }
  }
}

# EKS Cluster Policies
resource "aws_iam_role_policy_attachment" "eks_cloudwatch_policy" {
  policy_arn = "arn:aws:iam::aws:policy/CloudWatchFullAccess"
  role       = aws_iam_role.eks_cluster_role.name
}

resource "aws_iam_role_policy_attachment" "eks_cluster_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
  role       = aws_iam_role.eks_cluster_role.name
}

resource "aws_iam_role_policy_attachment" "eks_vpc_resource_controller_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSVPCResourceController"
  role       = aws_iam_role.eks_cluster_role.name
}

# Managed Node Group role
resource "aws_iam_instance_profile" "eks_node" {
  name = "${var.cluster_name}-node-role"
  role = aws_iam_role.node_role.name
}

resource "aws_iam_role" "node_role" {
  name               = "${var.cluster_name}-node-role"
  assume_role_policy = data.aws_iam_policy_document.assume_role_policy.json
}

data "aws_iam_policy_document" "assume_role_policy" {
  statement {
    actions = [
      "sts:AssumeRole"
    ]
    principals {
      type        = "Service"
      identifiers = ["ec2.amazonaws.com"]
    }
  }
}

# Node Group Policies
resource "aws_iam_role_policy_attachment" "eks_worker_node_policy" {
  role       = aws_iam_role.node_role.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
}

resource "aws_iam_role_policy_attachment" "eks_cni_policy" {
  role       = aws_iam_role.node_role.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
}

resource "aws_iam_role_policy_attachment" "eks_ebs_csi_policy" {
  role       = aws_iam_role.node_role.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy"
}

resource "aws_iam_role_policy_attachment" "eks_registry_policy" {
  role       = aws_iam_role.node_role.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
}

# VPC CNI Plugin Role
data "aws_iam_policy_document" "vpc_cni_assume_role_policy" {
  statement {
    actions = ["sts:AssumeRoleWithWebIdentity"]
    effect  = "Allow"

    condition {
      test     = "StringEquals"
      variable = "${replace(aws_iam_openid_connect_provider.eks.url, "https://", "")}:sub"
      values   = ["system:serviceaccount:kube-system:aws-node"]
    }

    principals {
      identifiers = [aws_iam_openid_connect_provider.eks.arn]
      type        = "Federated"
    }
  }
}

resource "aws_iam_role" "vpc_cni_role" {
  assume_role_policy = data.aws_iam_policy_document.vpc_cni_assume_role_policy.json
  name               = "${var.cluster_name}-vpc-cni-role"
}

resource "aws_iam_role_policy_attachment" "vpc_cni_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
  role       = aws_iam_role.vpc_cni_role.name
}

Cluster Access and Role Based Access Control

When an Amazon EKS cluster is created, the AWS identity (user or role) that creates the cluster is automatically granted system:masters permissions, providing full administrative access to the cluster. This ensures immediate operational control without additional configuration steps.

However, in collaborative environments or larger organizations, more than one individual or service may need administrative access to manage the cluster effectively. To extend administrative privileges securely, additional IAM roles can be created and configured with permissions similar to the cluster creator.

This Terraform configuration defines an IAM role (aws_iam_role.eks_admins_role) designed for EKS Administrators, granting them similar administrative capabilities over the EKS cluster. It uses an IAM policy document (data.aws_iam_policy_document.eks_admins_assume_role_policy_doc) to specify that the role can be assumed by specified AWS principals. This approach adheres to the principle of least privilege, allowing for controlled access to the EKS cluster based on organizational requirements.

It's best to also add the node role with the username system:node:{{EC2PrivateDNSName}} in groups "system:bootstrappers", "system:nodes", though it is actually added to the ConfigMap automatically, it will get removed by your next terraform apply command if not defined explicitly.

Lastly, the kubernetes_config_map.aws_auth resource updates the aws-auth ConfigMap in the EKS cluster. This action registers the role with the cluster, mapping it to system:masters, thus granting it administrative access similar to the cluster creator. This setup ensures that designated administrators can manage the cluster without using the credentials of the original creator, enhancing security and operational flexibility.

############################################################################################################
### CLUSTER ROLE BASE ACCESS CONTROL
############################################################################################################
# Define IAM Role for EKS Administrators
resource "aws_iam_role" "eks_admins_role" {
  name = "${var.cluster_name}-eks-admins-role"

  assume_role_policy = data.aws_iam_policy_document.eks_admins_assume_role_policy_doc.json
}

# IAM Policy Document for assuming the eks-admins role
data "aws_iam_policy_document" "eks_admins_assume_role_policy_doc" {
  statement {
    actions = ["sts:AssumeRole"]
    principals {
      type        = "AWS"
      identifiers = ["arn:aws:iam::${data.aws_caller_identity.current.account_id}:root"]
    }
    effect = "Allow"
  }
}

# Define IAM Policy for administrative actions on EKS
data "aws_iam_policy_document" "eks_admin_policy_doc" {
  statement {
    actions   = ["eks:*", "ec2:Describe*", "iam:ListRoles", "iam:ListRolePolicies", "iam:GetRole"]
    resources = ["*"]
  }
}

# Create IAM Policy based on the above document
resource "aws_iam_policy" "eks_admin_policy" {
  name   = "${var.cluster_name}-eks-admin-policy"
  policy = data.aws_iam_policy_document.eks_admin_policy_doc.json
}

# Attach IAM Policy to the EKS Administrators Role
resource "aws_iam_role_policy_attachment" "eks_admin_role_policy_attach" {
  role       = aws_iam_role.eks_admins_role.name
  policy_arn = aws_iam_policy.eks_admin_policy.arn
}

# Update the aws-auth ConfigMap to include the IAM group
resource "kubernetes_config_map" "aws_auth" {
  metadata {
    name      = "aws-auth"
    namespace = "kube-system"
  }

  data = {
    mapRoles = yamlencode([
      {
        rolearn  = aws_iam_role.eks_admins_role.arn
        username = aws_iam_role.eks_admins_role.name
        groups   = ["system:masters"]
      },
      {
        rolearn  = aws_iam_role.node_role.arn
        username = "system:node:{{EC2PrivateDNSName}}"
        groups   = ["system:bootstrappers", "system:nodes"]
      }
    ])
    mapUsers = yamlencode([
      {
        userarn  = data.aws_caller_identity.current.arn
        username = split("/", data.aws_caller_identity.current.arn)[1]
        groups   = ["system:masters"]
      }
    ])
  }

}

Conclusion: Mastering EKS Management

By delving into advanced configuration options, securing your clusters with IAM, and employing effective logging, monitoring, and scaling strategies, you can create Kubernetes clusters that are not only robust and scalable but also optimized for cost, performance, and security. The journey to mastering AWS EKS with Terraform is ongoing, and each step forward enhances your ability to manage complex cloud-native ecosystems effectively.

The complete project can be found here:
Sample Implementation GitHub Repo

Comments

[shengzhen_dev]

what a nice tutorial on provision EKS via Terraform, absolutely great.