DEV Community: Ravindu Fernando

A Production Ready EKS Deployment with IaC & GitOps - Part 5 - Deploying EKS Cluster

Ravindu Fernando — Sun, 02 Feb 2025 14:41:32 +0000

Refer here for corresponding Git commit: Provision EKS Cluster

In this article, we'll explore how to deploy the Amazon EKS cluster using Terraform. We'll examine each component and explain how to customize them for your specific requirements.

For this EKS deployment, we'll add / make changes for these files:

eks.tf: Contains the EKS cluster configuration
variables.tf: Contains variable definitions

Adding the required the Variables

Let's start with the new variables in variables.tf:


variable "cluster_name" {
  default     = "opsninja"
}

variable "worker_group_name" {
  default     = "opsninja"
}

variable "cluster_creator_admin_permission" {
  description = "Enable admin permissions to access Kubernetes API - "
  type        = bool
  default     = true
}

variable "cluster_endpoint_public_access" {
  description = "Enable or disable public access to the EKS cluster endpoint"
  type        = bool
  default     = true
}

variable "public_access_cidrs" {
  description = "List of CIDRs allowed for public access to the EKS cluster"
  type        = list(string)
  default     = ["0.0.0.0/0"]
}

variable "coredns_addon_version" {
  description = "Version of the CoreDNS addon"
  type        = string
  default     = "v1.11.3-eksbuild.2"
}

variable "kube_proxy_addon_version" {
  description = "Version of the kube-proxy addon"
  type        = string
  default     = "v1.31.2-eksbuild.2"
}

variable "vpc_cni_addon_version" {
  description = "Version of the VPC CNI addon"
  type        = string
  default     = "v1.19.0-eksbuild.1"
}

variable "pod_identity_agent_addon_version" {
  description = "Version of the POD IDENTITY addon"
  type        = string
  default     = "v1.3.2-eksbuild.2"
}

variable "app_namespace" {
  default = "opsninja"
}

EKS Configuration Breakdown

Here's the complete configuration for the EKS cluster.

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "20.29.0"

  cluster_name    = local.cluster_name
  cluster_version = "1.31"

  vpc_id                         = module.vpc.vpc_id
  subnet_ids                     = module.vpc.private_subnets
  control_plane_subnet_ids       = module.vpc.intra_subnets

  cluster_endpoint_private_access       = true
  cluster_endpoint_public_access        = var.cluster_endpoint_public_access
  cluster_endpoint_public_access_cidrs  = var.public_access_cidrs

  enable_cluster_creator_admin_permissions = var.cluster_creator_admin_permission

  eks_managed_node_group_defaults = {
    ami_type = "AL2_x86_64"
    disk_size = 100
  }

  cluster_addons = {
    coredns = { 
      addon_version = var.coredns_addon_version
      resolve_conflicts = "OVERWRITE"
    }
    kube-proxy = {
      addon_version = var.kube_proxy_addon_version
      resolve_conflicts = "OVERWRITE"
    }
    eks-pod-identity-agent = {
      addon_version = var.pod_identity_agent_addon_version
      resolve_conflicts = "OVERWRITE"
    }
    vpc-cni = {
      addon_version = var.vpc_cni_addon_version
      resolve_conflicts = "OVERWRITE"
      before_compute           = true
      configuration_values = jsonencode({
        env = {
          # Reference docs https://docs.aws.amazon.com/eks/latest/userguide/cni-increase-ip-addresses.html
          ENABLE_PREFIX_DELEGATION = "true"
          WARM_PREFIX_TARGET       = "1"
        }
      })
    }
  }

  enable_irsa = true

  cluster_security_group_additional_rules = {
    egress_nodes_ephemeral_ports_tcp = {
      description                = "To node 1025-65535"
      protocol                   = "tcp"
      from_port                  = 1025
      to_port                    = 65535
      type                       = "egress"
      source_node_security_group = true
    }
  }

  node_security_group_additional_rules = {
    ingress_self_all = {
      description = "Node to node all ports/protocols"
      protocol    = "-1"
      from_port   = 0
      to_port     = 0
      type        = "ingress"
      self        = true
    }
    egress_all = {
      description      = "Node all egress"
      protocol         = "-1"
      from_port        = 0
      to_port          = 0
      type             = "egress"
      cidr_blocks      = ["0.0.0.0/0"]
      ipv6_cidr_blocks = ["::/0"]
    }
  }

  eks_managed_node_groups = {
    opsninja-wg = {
      name = "${var.worker_group_name}-${var.env}-wg"

      instance_types = ["m6a.xlarge"]

      create_security_group                 = false
      attach_cluster_primary_security_group = false

      min_size     = 1
      max_size     = 6
      desired_size = 1

      iam_role_additional_policies = {
        policy1 = "arn:${local.partition}:iam::aws:policy/AmazonSSMManagedInstanceCore",
        policy_cw = "arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy"
      }

      pre_bootstrap_user_data = <<-EOT
        #!/bin/bash
        LINE_NUMBER=$(grep -n "KUBELET_EXTRA_ARGS=\$2" /etc/eks/bootstrap.sh | cut -f1 -d:)
        REPLACEMENT="\ \ \ \ \ \ KUBELET_EXTRA_ARGS=\$(echo \$2 | sed -s -E 's/--max-pods=[0-9]+/--max-pods=90/g')"
        sed -i '/KUBELET_EXTRA_ARGS=\$2/d' /etc/eks/bootstrap.sh
        sed -i "$${LINE_NUMBER}i $${REPLACEMENT}" /etc/eks/bootstrap.sh
      EOT
    }
  }
  node_security_group_tags = {
    "karpenter.sh/discovery" = local.cluster_name
  }

}

resource "kubernetes_namespace" "app_namespace" {
  metadata {
    name = var.app_namespace
  }
}

Let's examine each part in detail:

Network Integration

vpc_id                         = module.vpc.vpc_id
subnet_ids                     = module.vpc.private_subnets
control_plane_subnet_ids       = module.vpc.intra_subnets

cluster_endpoint_private_access      = true
cluster_endpoint_public_access       = var.cluster_endpoint_public_access
cluster_endpoint_public_access_cidrs = var.public_access_cidrs

Customization Points:
We are referring to the VPC module we provisioned earlier to obtain most of the information.

cluster_endpoint_public_access: Set to false for private-only clusters. However, in our case, we can set this to 'true' while we provision infrastructure.
public_access_cidrs: Restrict access to specific IP ranges. In thr variables.tf we have set this to 0.0.0.0/0 which allows all IPs to access the Kubernetes API Server. This is insecure. If you have a bastion host or a server which you would like to access the EKS cluster with, you can enter that IP here.
Control plane placed in intra subnets for enhanced security
Worker nodes placed in private subnets

Node Group Configuration

eks_managed_node_group_defaults = {
  ami_type = "AL2_x86_64"
  disk_size = 100
}

eks_managed_node_groups = {
  opsninja-wg = {
    name = "${var.worker_group_name}-${var.env}-wg"

    instance_types = ["m6a.xlarge"]

    create_security_group                 = false
    attach_cluster_primary_security_group = false

    min_size     = 1
    max_size     = 6
    desired_size = 1

    iam_role_additional_policies = {
      policy1 = "arn:${local.partition}:iam::aws:policy/AmazonSSMManagedInstanceCore",
      policy_cw = "arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy"
    }
  }
}

In this configuration we are using the following specs. But you can customize them:

Using Amazon Linux 2 AMI
100GB root volume for container images and system files
m6a.xlarge instances for good price/performance ratio
Autoscaling configuration: min=1, max=6
SSM and CloudWatch policies attached for monitoring and management

Add-ons Configuration

cluster_addons = {
  coredns = { 
    addon_version = var.coredns_addon_version
    resolve_conflicts = "OVERWRITE"
  }
  kube-proxy = {
    addon_version = var.kube_proxy_addon_version
    resolve_conflicts = "OVERWRITE"
  }
  eks-pod-identity-agent = {
    addon_version = var.pod_identity_agent_addon_version
    resolve_conflicts = "OVERWRITE"
  }
  vpc-cni = {
    addon_version = var.vpc_cni_addon_version
    resolve_conflicts = "OVERWRITE"
    before_compute           = true
    configuration_values = jsonencode({
      env = {
        ENABLE_PREFIX_DELEGATION = "true"
        WARM_PREFIX_TARGET      = "1"
      }
    })
  }
}

Important notes:

Update addon versions as new releases become available. In our case we have pinned the addon versions since terraform would otherwise show a drift everytime we apply.
VPC CNI configured with prefix delegation for better IP address management. You can read more about it: here. Without this configuration, there is a limit on number of IPs that are made available for a Node. To bypass this limitation, we have done a hack on pre_bootstrap_user_data aswell under eks_managed_node_group. More on that later in this article..

Security Group Rules

cluster_security_group_additional_rules = {
  egress_nodes_ephemeral_ports_tcp = {
    description                = "To node 1025-65535"
    protocol                   = "tcp"
    from_port                  = 1025
    to_port                    = 65535
    type                       = "egress"
    source_node_security_group = true
  }
}

node_security_group_additional_rules = {
  ingress_self_all = {
    description = "Node to node all ports/protocols"
    protocol    = "-1"
    from_port   = 0
    to_port     = 0
    type        = "ingress"
    self        = true
  }
  egress_all = {
    description      = "Node all egress"
    protocol         = "-1"
    from_port        = 0
    to_port          = 0
    type            = "egress"
    cidr_blocks      = ["0.0.0.0/0"]
    ipv6_cidr_blocks = ["::/0"]
  }
}

What this configuration does::

Allows necessary communication between control plane and nodes
Enables node-to-node communication for pod networking
Permits outbound internet access for nodes

Bootstrap Customization

pre_bootstrap_user_data = <<-EOT
  #!/bin/bash
  LINE_NUMBER=$(grep -n "KUBELET_EXTRA_ARGS=\$2" /etc/eks/bootstrap.sh | cut -f1 -d:)
  REPLACEMENT="\ \ \ \ \ \ KUBELET_EXTRA_ARGS=\$(echo \$2 | sed -s -E 's/--max-pods=[0-9]+/--max-pods=90/g')"
  sed -i '/KUBELET_EXTRA_ARGS=\$2/d' /etc/eks/bootstrap.sh
  sed -i "$${LINE_NUMBER}i $${REPLACEMENT}" /etc/eks/bootstrap.sh
EOT

Configuration Notes:

Modifies max pods per node to 90
However note that if you want to use a different AMI, the pre_bootstrap_user_data here might not work (Although it should work for atleast most of the Linux AMIs).

Namespace Creation

resource "kubernetes_namespace" "app_namespace" {
  metadata {
    name = var.app_namespace
  }
}

We are creating the namespace where our application will ultimately run aswell.

Accessing the cluster

Once you have applied the changes above through terraform, you will finally have your cluster created.

To access your cluster, now you can configure kubectl:

aws eks update-kubeconfig --name <cluster-name> --region <region>

A Production Ready EKS Deployment with IaC & GitOps - Part 4 - Deploying VPC

Ravindu Fernando — Sun, 02 Feb 2025 12:03:43 +0000

Refer here for corresponding Git commit: Deploying VPC

In this article, we'll explore how to set up a production-ready VPC for our Amazon EKS cluster using Terraform. We'll walk through each component and explain how you can customize them for your specific needs.

File Structure

For this VPC deployment, we'll work with two main files:

data.tf: Contains data sources for AWS resources
vpc.tf: Contains the VPC module configuration

Understanding the Data Sources

Let's start with data.tf:

data "aws_availability_zones" "available" {
  filter {
    name   = "opt-in-status"
    values = ["opt-in-not-required"]
  }
}

locals {
  cluster_name = "${var.cluster_name}-${var.env}"
}

This data source fetches available AWS Availability Zones (AZs) in our region that don't require opt-in. This ensures our VPC spans across reliable and readily available AZs.

We have a local defined to append the environment to the cluster name we define in variables.tf.

variables.tf file would be updated with the following code block;

variable "vpc_name" {
  default     = "opsninja"
}

variable "cluster_name" {
  default     = "opsninja"
}

VPC Configuration Breakdown

Here's the complete configuration for the VPC.
vpc.tf

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "5.15.0"

  name = "${var.vpc_name}-${var.env}-vpc"

  cidr = "10.1.0.0/16"
  azs  = slice(data.aws_availability_zones.available.names, 0, 3)

  private_subnets = ["10.1.1.0/24", "10.1.2.0/24", "10.1.3.0/24"]
  public_subnets  = ["10.1.4.0/24", "10.1.5.0/24", "10.1.6.0/24"]
  intra_subnets   = ["10.1.7.0/24", "10.1.8.0/24", "10.1.9.0/24"]

  enable_nat_gateway   = true
  single_nat_gateway   = true
  enable_dns_hostnames = true

  public_subnet_tags = {
    "kubernetes.io/cluster/${local.cluster_name}" = "shared"
    "kubernetes.io/role/elb"                      = 1
  }

  private_subnet_tags = {
    "kubernetes.io/cluster/${local.cluster_name}" = "shared"
    "kubernetes.io/role/internal-elb"             = 1
    "karpenter.sh/discovery"                      = local.cluster_name
  }
}

Now, let's examine each part of the VPC configuration in vpc.tf:

Module Source and Version

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "5.15.0"

We're using the official AWS VPC Terraform module. You can update the version number as newer releases become available.

VPC Naming

name = "${var.vpc_name}-${var.env}-vpc"

The VPC name follows the format: [vpc_name]-[environment]-vpc. For example: opsninja-prod-vpc

Customization Point:

Modify var.vpc_name in variables.tf to match your project name
The env variable determines the environment (prod, dev, etc.)

Network Configuration

cidr = "10.1.0.0/16"
azs  = slice(data.aws_availability_zones.available.names, 0, 3)

private_subnets = ["10.1.1.0/24", "10.1.2.0/24", "10.1.3.0/24"]
public_subnets  = ["10.1.4.0/24", "10.1.5.0/24", "10.1.6.0/24"]
intra_subnets   = ["10.1.7.0/24", "10.1.8.0/24", "10.1.9.0/24"]

Customization Points:

CIDR Range: Modify 10.1.0.0/16 based on your network requirements. Note that Each pod requires its own IP address. Consider future growth in pod count. For 1000 pods: A /24 subnet (256 IPs) might be too small. For 5000+ pods: Consider larger subnets like /22 (1024 IPs) or bigger. Current configuration (/24) provides ~250 usable IPs per subnet.

Subnet CIDRs: Adjust subnet ranges ensuring they fall within your VPC CIDR. Private subnets host your worker nodes and pods. Size private subnets larger than public/intra subnets.
Example sizing for high pod density:

 private_subnets = ["10.1.0.0/22", "10.1.4.0/22", "10.1.8.0/22"]    # ~1000 IPs each
 public_subnets  = ["10.1.12.0/24", "10.1.13.0/24", "10.1.14.0/24"] # ~250 IPs each
 intra_subnets   = ["10.1.15.0/24", "10.1.16.0/24", "10.1.17.0/24"] # ~250 IPs each

Number of AZs: The slice function selects the first 3 AZs, modify if needed

Network Features

enable_nat_gateway   = true
single_nat_gateway   = true
enable_dns_hostnames = true

Configuration Notes:

single_nat_gateway: Set to true for cost optimization, set to false for high availability
enable_dns_hostnames: Required for EKS functionality
NAT Gateway: Required for private subnet internet access

Subnet Tagging

public_subnet_tags = {
  "kubernetes.io/cluster/${local.cluster_name}" = "shared"
  "kubernetes.io/role/elb"                      = 1
}

private_subnet_tags = {
  "kubernetes.io/cluster/${local.cluster_name}" = "shared"
  "kubernetes.io/role/internal-elb"             = 1
  "karpenter.sh/discovery"                      = local.cluster_name
}

These tags are crucial for:

EKS cluster networking
Load balancer provisioning
Karpenter node auto-discovery. We haven't deployed Karpenter yet, but since we are configuring the VPC, might as well add those tags.

A Production Ready EKS Deployment with IaC & GitOps - Part 3 - Backend & Provider Configuration

Ravindu Fernando — Sun, 02 Feb 2025 10:20:30 +0000

Refer here for corresponding Git commit: Backend & Provider Configuration

In this part of the series, we will provision an AWS EKS Cluster using Terraform. To maintain infrastructure as code (IaC), we are using Terraform to define and manage the EKS cluster and related resources in an automated and repeatable manner.

Before diving into the actual cluster provisioning, let’s go through the file structure and the Terraform backend & provider configurations that will be used throughout this guide.

Terraform File Structure

Our initial Terraform setup is structured as follows:

├── terraform.tf
├── main.tf
├── variables.tf

We will add more terraform files here for other components such as vpc, eks etc..

terraform.tf: The Foundation

The terraform.tf file serves as the foundational configuration file for our Terraform project. It contains two critical sections:

Backend Configuration

backend "s3" {
    bucket         = "opsninja-tf-state-prod"
    region         = "us-east-1"
    key            = "opsninja/v1/terraform-prod.tfstate"
    dynamodb_table = "opsninja-tf-state-prod"
}

In Part 2 of this article series we provisioned the S3 bucket and the DynamoDB table. You should use the same table name and bucket name here.

This configuration specifies where Terraform should store its state file. Using an S3 bucket with DynamoDB for state locking is a common practice in production environments, ensuring safe concurrent access to the infrastructure state.

Here's what each property does:

bucket - The name of the S3 bucket where Terraform will store the state file
region - The AWS region where the S3 bucket is located
key - The path within the S3 bucket where the state file will be stored. In your case:
- opsninja/v1/terraform-prod.tfstate represents the full path to the state file
dynamodb_table - The name of the DynamoDB table used for state locking. This prevents multiple users from modifying the infrastructure at the same time

This configuration ensures your Terraform state is stored remotely and can be accessed securely by team members while preventing concurrent modifications.

Provider Requirements

required_providers {
    aws = {
        source  = "hashicorp/aws"
        version = "5.76.0"
    }
    # Additional providers...
}

This section defines all the providers needed for our infrastructure and their specific versions, ensuring consistency across team members and environments.

main.tf: Provider Configuration

The main.tf file focuses on configuring the providers declared in terraform.tf. It contains the specific settings and authentication methods for each provider. Key aspects include:

AWS Provider Configuration

provider "aws" {
    region = var.region
    profile = var.profile
    default_tags {
        tags = {
            application_name = var.app
            application_env = var.env
            terraform = "true"
            managed_by = "Terraform"
        }
    }
}

This configuration sets up AWS-specific settings, including default tags that will be applied to all resources.

Kubernetes-Related Providers The file also includes configurations for kubectl, kubernetes, and helm providers, all necessary for managing Kubernetes resources.

variables.tf: Variable Definitions

The variables.tf file defines all the variables used across our Terraform configuration:

variable "app" {
    description = "Application name"
    type        = string
    default     = "opsninja"
}

This file makes our configuration more flexible and reusable across different environments.

Now that we have a clear idea about what our project structure is, you can add the following files to your terraform project.

terraform.tf

terraform {

  backend "s3"{
    bucket                 = "opsninja-tf-state-prod"
    region                 = "us-east-1"
    key                    = "opsninja/v1/terraform-prod.tfstate"
    dynamodb_table         = "opsninja-tf-state-prod"
  }

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "5.76.0"
    }

    random = {
      source  = "hashicorp/random"
      version = "~> 3.5.1"
    }

    tls = {
      source  = "hashicorp/tls"
      version = "~> 4.0.4"
    }

    cloudinit = {
      source  = "hashicorp/cloudinit"
      version = "~> 2.3.2"
    }

    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.11.0"
    }

    kubectl = {
      source  = "alekc/kubectl"
      version = ">= 2.0.0"
    }
  }

  required_version = "~> 1.3"

}

main.tf

provider "aws" {
  region = var.region
  profile = var.profile
  default_tags {
    tags = {
      application_name = var.app
      application_env = var.env
      terraform = "true"
      managed_by = "Terraform"
    }
  }
}

provider "kubectl" {
  apply_retry_count      = 15
  host                   = module.eks.cluster_endpoint
  cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
  load_config_file       = false

  exec {
    api_version = "client.authentication.k8s.io/v1beta1"
    command     = "aws"
    args        = ["eks", "get-token", "--cluster-name", module.eks.cluster_name]
  }
}

provider "kubernetes" {
  host                   = module.eks.cluster_endpoint
  cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
  exec {
    api_version = "client.authentication.k8s.io/v1beta1"
    command     = "aws"
    args = ["eks", "get-token", "--cluster-name", module.eks.cluster_name]
  }
}

provider "helm" {
  kubernetes {
    host                   = module.eks.cluster_endpoint
    cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
    exec {
      api_version = "client.authentication.k8s.io/v1beta1"
      args        = ["eks", "get-token", "--cluster-name", module.eks.cluster_name]
      command     = "aws"
    }
  }
}

variables.tf

##### Globals #####

variable "app" {
  description = "Application name. Used as a prefix to most resources created (Ex: Policies)"
  type        = string
  default     = "opsninja"
}

variable "env" {
  description = "Environment. Used to prefix most resources to identify resources environment-wise"
  type        = string
  default     = "prod"
}

variable "region" {
  description = "AWS region"
  type        = string
  default     = "us-east-1"
}

variable "profile" {
  description = "AWS Profile. Used to provision resources"
  type        = string
  default     = "terraform"
}

A Production Ready EKS Deployment with IaC & GitOps - Part 2 - Configuring Terraform Backend & Validate Prerequisites

Ravindu Fernando — Sun, 02 Feb 2025 07:58:45 +0000

Refer here for corresponding Git commit: Provision Terraform Backend

Pre-requisites

Throughout this guide, all infrastructure and applications will either be named or prefixed with "opsninja" for easy identification. Our deployments will take place in the us-east-1 region.

Before we start deploying our AWS EKS cluster using Terraform, we need to ensure that the required tools are installed and properly configured.

1. Install AWS CLI v2

The AWS CLI (Command Line Interface) v2 is required to interact with AWS services. To install it, follow these steps:

For macOS

brew install awscli

For Linux

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

For Windows

Download the installer from the official AWS website: AWS CLI v2
Run the installer and follow the setup instructions.
Verify the installation:

   aws --version

You should see an output similar to:

   aws-cli/2.x.x Python/x.x.x Linux/x86_64

2. Install Helm (Kubernetes Package Manager)

Helm is a package manager for Kubernetes that simplifies the deployment of applications and services.

For macOS

brew install helm

For Linux

curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

For Windows (using Chocolatey)

choco install kubernetes-helm

Verify the installation:

helm version

You should see an output similar to:

version.BuildInfo{Version:"v3.x.x", GitCommit:"xxxx", GoVersion:"go1.x.x"}

3. Install AWS IAM Authenticator

The AWS IAM Authenticator is required for authentication with the Amazon EKS cluster.

For macOS

brew install aws-iam-authenticator

For Linux

curl -o aws-iam-authenticator https://amazon-eks.s3.us-west-2.amazonaws.com/1.24.3/2022-06-24/bin/linux/amd64/aws-iam-authenticator
chmod +x ./aws-iam-authenticator
mv ./aws-iam-authenticator /usr/local/bin/

For Windows

Download the binary from the official AWS EKS release and add it to your system's PATH.

Verify the installation:

aws-iam-authenticator version

Expected output:

{"Version":"0.x.x","Commit":"xxxx"}

4. Configure AWS CLI with a Profile Named `terraform`

We will be using a dedicated AWS CLI profile called terraform to provision our infrastructure with Terraform.

Run the following command to configure AWS CLI with the `terraform` profile:

aws configure --profile terraform

You'll be prompted to enter your AWS credentials:

AWS Access Key ID [None]: <YOUR_ACCESS_KEY>
AWS Secret Access Key [None]: <YOUR_SECRET_KEY>
Default region name [None]: us-east-1
Default output format [None]: json

Verify the Profile

To check if the profile is configured correctly, run:

aws configure list --profile terraform

or list all available profiles:

aws configure list-profiles

This profile will be used throughout this guide to authenticate and interact with AWS when deploying infrastructure using Terraform.

5. Validate AWS Configuration

To confirm that the AWS CLI and profile are correctly set up, run:

aws sts get-caller-identity --profile terraform

This should return details of the authenticated IAM user or role.

With AWS CLI, Helm, and AWS IAM Authenticator installed and the terraform profile configured, we are now ready to proceed with configuring the Terraform backend

Configuring the Terraform Backend

Throughout this guide, all infrastructure and applications will either be named or prefixed with "opsninja" for easy identification. Our deployments will take place in the us-east-1 region.

For better state management, we will configure a Terraform backend using AWS S3 and DynamoDB. This will allow us to store our Terraform state remotely and enable state locking to prevent conflicts in collaborative environments.

Terraform Backend Configuration (`main.tf`)

Below is the Terraform backend configuration, which uses an S3 bucket for state storage and a DynamoDB table for state locking.

provider "aws" {
  region = var.region
  profile = "terraform"
}

resource "aws_s3_bucket" "terraform_state_bucket" {
  bucket = var.s3_bucket_name

  versioning {
    enabled = true
  }

  server_side_encryption_configuration {
    rule {
      apply_server_side_encryption_by_default {
        sse_algorithm = "AES256"
      }
    }
  }

  lifecycle {
    prevent_destroy = true
  }

  tags = {
    Name        = var.s3_bucket_name
    Environment = var.environment
    App         = var.app
    ManagedBy   = "Terraform"
  }
}

resource "aws_dynamodb_table" "terraform_lock_table" {
  name         = var.dynamodb_table_name
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "LockID"

  attribute {
    name = "LockID"
    type = "S"
  }

  tags = {
    Name        = var.dynamodb_table_name
    Environment = var.environment
    App         = var.app
    ManagedBy   = "Terraform"
  }
}

Terraform Variables (`variables.tf`)

variable "app" {
  description = "App Name"
  type        = string
  default     = "opsninja"
}

variable "region" {
  description = "AWS region"
  type        = string
  default     = "us-east-1"
}

variable "s3_bucket_name" {
  description = "The name of the S3 bucket for storing Terraform state"
  type        = string
  default     = "opsninja-tf-state-prod"
}

variable "dynamodb_table_name" {
  description = "The name of the DynamoDB table for state locking"
  type        = string
  default     = "opsninja-tf-state-prod"
}

variable "environment" {
  description = "Environment for tagging purposes (e.g., dev, prod)"
  type        = string
  default     = "prod"
}

Terraform Output (`output.tf`)

output "s3_bucket_name" {
  value = aws_s3_bucket.terraform_state_bucket.bucket
}

output "dynamodb_table_name" {
  value = aws_dynamodb_table.terraform_lock_table.name
}

Deploying the Terraform Backend

Once the backend is defined, apply Terraform using the following command:

terraform init
terraform apply

Deployment Output and State Management

Once the deployment is complete, the output will display the S3 bucket and DynamoDB table that were created. Make sure to note these values, as they will be required for future steps in this guide.

Additionally, the state file for backend provisioning is saved locally. This file must be preserved, as it is necessary to manage the lifecycle of the S3 bucket and DynamoDB table in the future.

A Production Ready EKS Deployment with IaC & GitOps - Part 1 - Introduction

Ravindu Fernando — Sun, 25 Aug 2024 15:11:12 +0000

Introduction to AWS EKS Deployment

This series is designed to take you through the process of deploying a production-ready application on AWS EKS (Elastic Kubernetes Service) from scratch. Whether you're a beginner developer or someone looking to refine your skills, this series will help you confidently set up and deploy your application in the least amount of time, without the need to dig through extensive documentation.

What We Will Cover

This article series will guide you through deploying and managing an AWS EKS cluster, integrating essential services, and automating your workflows. Below is an overview of the steps we will take.

1. Configuring Terraform Backend and Validating Pre-requisites

Before deploying our infrastructure, we'll configure the Terraform backend to store our state securely and validate pre-requisites to ensure we have the required permissions and dependencies in place.

2. Provisioning an AWS EKS Cluster with Terraform

Using Terraform, we'll provision an AWS EKS cluster along with essential resources such as Elastic File System (EFS), Storage Classes (SC), and other networking components.

3. Setting Up Cluster Autoscaler

We'll configure Cluster Autoscaler, which dynamically adjusts the number of nodes in your cluster based on workload demands, optimizing resource usage and cost.

4. Deploying & Configuring Karpenter for Node Provisioning

We will deploy Karpenter, an advanced node provisioning tool for Kubernetes, and set up IAM roles, instance profiles, and policies to enable automated scaling.

5. Enabling CloudWatch Container Insights for Monitoring

Once Karpenter is deployed, we'll enable AWS CloudWatch Container Insights to gain visibility into cluster metrics, logs, and resource utilization.

6. Setting Up an Ingress Controller on EKS

We'll configure an Ingress Controller, a crucial component for managing external access to services within the cluster, ensuring your applications are securely exposed.

7. Installing ArgoCD for GitOps-based Deployment

We'll install and configure ArgoCD, a declarative GitOps tool that automates deployments and ensures version-controlled application rollouts.

8. Using EFS Mounts in Containers

You'll learn how to create Persistent Volumes (PVs) and Persistent Volume Claims (PVCs) to mount AWS EFS inside your containers, enabling stateful applications.

9. Deploying a Sample Application with ArgoCD

With our infrastructure ready, we'll deploy a sample application using ArgoCD to demonstrate continuous deployment and GitOps practices.

10. Configuring Application Load Balancer as an Ingress Controller

We'll set up an AWS Application Load Balancer (ALB) as an ingress solution, allowing efficient routing of external traffic to your services.

11. DNS Propagation to Application Load Balancer from a Custom Domain

You'll learn how to configure DNS propagation to map your custom domain to the ALB, making your application accessible via a user-friendly URL.

12. Adding SSL Certificates for Secure Communication

To secure traffic to your application, we'll integrate SSL/TLS certificates, ensuring encrypted connections and HTTPS accessibility.

This structured approach ensures you gain a comprehensive understanding of AWS EKS, from cluster setup to automated deployment and monitoring. Stay tuned as we explore each topic in depth and build a fully functional EKS environment from the ground up! 🚀

DEV Community: Ravindu Fernando

A Production Ready EKS Deployment with IaC & GitOps - Part 5 - Deploying EKS Cluster

Adding the required the Variables

EKS Configuration Breakdown

Network Integration

Node Group Configuration

Add-ons Configuration

Security Group Rules

Bootstrap Customization

Namespace Creation

Accessing the cluster

A Production Ready EKS Deployment with IaC & GitOps - Part 4 - Deploying VPC

File Structure

Understanding the Data Sources

VPC Configuration Breakdown

Module Source and Version

VPC Naming

Network Configuration

Network Features

Subnet Tagging

A Production Ready EKS Deployment with IaC & GitOps - Part 3 - Backend & Provider Configuration

Terraform File Structure

terraform.tf: The Foundation

main.tf: Provider Configuration

variables.tf: Variable Definitions

A Production Ready EKS Deployment with IaC & GitOps - Part 2 - Configuring Terraform Backend & Validate Prerequisites

Pre-requisites

1. Install AWS CLI v2

For macOS

For Linux

For Windows

2. Install Helm (Kubernetes Package Manager)

For macOS

For Linux

For Windows (using Chocolatey)

3. Install AWS IAM Authenticator

For macOS

For Linux

For Windows

4. Configure AWS CLI with a Profile Named terraform

Run the following command to configure AWS CLI with the terraform profile:

Verify the Profile

5. Validate AWS Configuration

Configuring the Terraform Backend

Terraform Backend Configuration (main.tf)

Terraform Variables (variables.tf)

Terraform Output (output.tf)

Deploying the Terraform Backend

Deployment Output and State Management

A Production Ready EKS Deployment with IaC & GitOps - Part 1 - Introduction

Introduction to AWS EKS Deployment

What We Will Cover

1. Configuring Terraform Backend and Validating Pre-requisites

2. Provisioning an AWS EKS Cluster with Terraform

3. Setting Up Cluster Autoscaler

4. Deploying & Configuring Karpenter for Node Provisioning

5. Enabling CloudWatch Container Insights for Monitoring

6. Setting Up an Ingress Controller on EKS

7. Installing ArgoCD for GitOps-based Deployment

8. Using EFS Mounts in Containers

9. Deploying a Sample Application with ArgoCD

10. Configuring Application Load Balancer as an Ingress Controller

11. DNS Propagation to Application Load Balancer from a Custom Domain

12. Adding SSL Certificates for Secure Communication

4. Configure AWS CLI with a Profile Named `terraform`

Run the following command to configure AWS CLI with the `terraform` profile:

Terraform Backend Configuration (`main.tf`)

Terraform Variables (`variables.tf`)

Terraform Output (`output.tf`)