DEV Community: Pin Xiong

Easy to use ECS

Pin Xiong — Fri, 15 Jul 2022 14:00:09 +0000

Elastic Container Service(ECS) is a highly scalable and fast container management service that makes it easy to run, stop, and manage containers on a cluster.

Today, I'll show you how to use it!

Architecture

There are several steps as below, and all of source codes have been put in my github repo(infrastructure and golang-web).

Prerequisites

Create an aws account with the IAM AdministratorAccess permission
Install and configure aws CLI
Install and configure Terraform

Setup environment

Terraform saves all changes in *.tfstate file, so we'd better store *.tfstate in aws s3 bucket instead of local machine. This step will build a aws s3 bucket to store *.tfstate file.

Init Terraform

Download infrastructure first

$ cd setup
$ terraform init

Create s3 bucket

$ terraform apply

You will be prompted to enter an aws region code, such as us-east-1. After that, you need to make sure the listed resources that will be crated and then enter yes

You can see the output s3_bucket_terraform_state below

Outputs:

s3_bucket_terraform_state = "**********-us-east-1"

Build resources

Now, we begin to build the resources including VPC, subnets, ECS and Pipeline.

Setup remote backup

$ cd ../region/virginia

and then, update the block of s3 in providers.tf file

backend "s3" {
    bucket  = "**********-us-east-1"
    key     = "terraform/backend.tfstate"
    region  = "us-east-1"
    encrypt = "true"
  }

The following things maybe need to be modified:

Set bucket as the output of s3_bucket_terraform_state
Set key as the path to store *.tfstate file in s3 bucket
Update region as the region code that you entered when creating s3 bucket above
Set encrypt as true

Create resources

You can modify the configuration in the main.tf file according to your needs, and then run the following commands

$ terraform init
$ terraform apply

You will be prompted to enter yes after confirmed the listed resources.

After 10 minutes, all resources are created.

Deploy your code

Download golang-web first and then set your git remote as below

$ git remote add aws ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/golang-web

$ git remote -v

$ git push aws master:master
Enumerating objects: 62, done.
Counting objects: 100% (62/62), done.
Delta compression using up to 12 threads
Compressing objects: 100% (55/55), done.
Writing objects: 100% (62/62), 25.36 KiB | 6.34 MiB/s, done.
Total 62 (delta 26), reused 3 (delta 0), pack-reused 0
To ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/golang-web
 * [new branch]      master -> master

Now, everything is done, let's take a look at the outputs in aws.

Outputs

First, we can see VPC and subnet have been created.

VPC
Subnets

and then the codepipe has been built

CodeCommit
CodeBuild
CodePipeline
ECR

we also can see that a public endpoint is created.

ELB
Target group

Now, It's time to see CES

ECS Dashboard
ECS Cluster
ECS Service
Task Definition

Verification

Access public endpoint
Scale out
Access start api will exhaust CPU resource serval times

and then see the monitor in Cloud watch

the ecs has been created 2 new tasks

Scale in Access stop api will release CPU resource serval times

and then see the monitor in Cloud watch

the ecs has been draining tasks

How to build EKS with Terraform

Pin Xiong — Sat, 23 Apr 2022 12:32:15 +0000

ESK(Amazon Elastic Kubernetes Service) is a fully managed Kubernetes cluster by aws. Now, we want to build EKS with Terraform. All the codes have been put in github.

Preparation

We need to build VPC first and then build EKS, you can refer to How to build VPC with Terraform

Code structure

infrastructure
├── module
│   ├── networking
│   ├── eks
│   │   ├── main.tf
│   │   ├── outputs.tf
│   │   └── variables.tf
│   └── jumpserver
│       ├── main.tf
│       ├── outputs.tf
│       └── variables.tf
├── region
│   └── virginia
│       ├── main.tf
│       └── providers.tf
└── setup
    ├── main.tf
    ├── outputs.tf
    ├── providers.tf
    └── variables.tf

EKS

Now, we begin to build EKS, the source codes in eks folder are listed as below:

main.tf

locals {
  name               = var.name
  vpc_id             = var.vpc_id
  public_subnets_id  = var.public_subnets_id
  private_subnets_id = distinct(flatten(var.private_subnets_id))
  desired_size       = var.desired_size
  max_size           = var.max_size
  min_size           = var.min_size
  security_group_ids = var.security_group_ids
  tags               = var.eks_tags
}

# Define the role to be attached EKS
resource "aws_iam_role" "eks_role" {
  name               = "eks-role"
  assume_role_policy = jsonencode({
    "Version" : "2012-10-17",
    "Statement" : [
      {
        "Effect" : "Allow",
        "Action" : [
          "sts:AssumeRole"
        ],
        "Principal" : {
          "Service" : [
            "eks.amazonaws.com"
          ]
        }
      }
    ]
  })
  tags = merge({
    Name : "EKS Role"
  }, local.tags)
}

# Attach the CloudWatchFullAccess policy to EKS role
resource "aws_iam_role_policy_attachment" "eks_CloudWatchFullAccess" {
  policy_arn = "arn:aws:iam::aws:policy/CloudWatchFullAccess"
  role       = aws_iam_role.eks_role.name
}

resource "aws_iam_role_policy_attachment" "eks_AmazonEKSClusterPolicy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
  role       = aws_iam_role.eks_role.name
}

# Optionally, enable Security Groups for Pods
resource "aws_iam_role_policy_attachment" "eks_AmazonEKSVPCResourceController" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSVPCResourceController"
  role       = aws_iam_role.eks_role.name
}

# Default Security Group of EKS
resource "aws_security_group" "security_group" {
  name        = "${local.name} Security Group"
  description = "Default SG to allow traffic from the EKS"
  vpc_id      = local.vpc_id

  ingress {
    from_port       = "0"
    to_port         = "0"
    protocol        = "TCP"
    security_groups = local.security_group_ids
  }

  tags = merge({
    Name = "${local.name} Security Group"
  }, local.tags)

}

# EKS Cluster
resource "aws_eks_cluster" "eks" {
  name    = local.name
  version = "1.22"

  enabled_cluster_log_types = [
    "api",
    "audit",
    "authenticator",
    "controllerManager",
    "scheduler"
  ]
  role_arn = aws_iam_role.eks_role.arn

  timeouts {}

  vpc_config {
    endpoint_private_access = true
    endpoint_public_access  = true
    public_access_cidrs     = [
      "0.0.0.0/0",
    ]
    security_group_ids = [
      aws_security_group.security_group.id
    ]
    subnet_ids = flatten([var.public_subnets_id, var.private_subnets_id])
  }

  tags = merge({
    Name = local.name
  }, local.tags)
}

resource "aws_iam_role" "node_group_role" {
  name                  = format("%s-node-group-role", lower(aws_eks_cluster.eks.name))
  path                  = "/"
  force_detach_policies = false
  max_session_duration  = 3600
  assume_role_policy    = jsonencode(
    {
      Statement = [
        {
          Action    = "sts:AssumeRole"
          Effect    = "Allow"
          Principal = {
            Service = "ec2.amazonaws.com"
          }
        },
      ]
      Version = "2012-10-17"
    }
  )
}

resource "aws_iam_role_policy_attachment" "node_group_AmazonEC2ContainerRegistryReadOnly" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
  role       = aws_iam_role.node_group_role.id
}

resource "aws_iam_role_policy_attachment" "node_group_AmazonEKSWorkerNodePolicy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
  role       = aws_iam_role.node_group_role.id
}

resource "aws_iam_role_policy_attachment" "node_group_AmazonEC2RoleforSSM" {
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonEC2RoleforSSM"
  role       = aws_iam_role.node_group_role.id
}

resource "aws_iam_role_policy_attachment" "node_group_AmazonEKS_CNI_Policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
  role       = aws_iam_role.node_group_role.id
}

resource "aws_iam_role_policy_attachment" "node_group_CloudWatchAgentServerPolicy" {
  policy_arn = "arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy"
  role       = aws_iam_role.node_group_role.id
}

resource "aws_eks_node_group" "node_group" {
  cluster_name  = aws_eks_cluster.eks.name
  disk_size     = 0
  capacity_type = "SPOT"
  labels        = {
    "eks/cluster-name"   = aws_eks_cluster.eks.name
    "eks/nodegroup-name" = format("nodegroup_%s", lower(aws_eks_cluster.eks.name))
  }
  node_group_name = format("nodegroup_%s", lower(aws_eks_cluster.eks.name))
  node_role_arn   = aws_iam_role.node_group_role.arn

  subnet_ids = local.private_subnets_id

  scaling_config {
    desired_size = local.desired_size
    max_size     = local.max_size
    min_size     = local.min_size
  }

  timeouts {}

  lifecycle {
    create_before_destroy = true
  }

  tags = merge({
    Name                 = local.name
    "eks/cluster-name"   = local.name
    "eks/nodegroup-name" = format("%s Node Group", aws_eks_cluster.eks.name)
    "eks/nodegroup-type" = "managed"
  }, local.tags)
}

variables.tf

variable "name" {
  description = "The EKS name"
}

variable "vpc_id" {
  type        = string
  description = "The VPC id"
}

variable "public_subnets_id" {
  description = "The public subnets id"
  type        = list
}

variable "private_subnets_id" {
  description = "The private subnets id"
  type        = list
}

variable "desired_size" {
  description = "The desired size of node"
  type        = number
  default     = 1
}

variable "max_size" {
  description = "The maximum size of node"
  type        = number
}

variable "min_size" {
  description = "The minimum size of node"
  type        = number
  default     = 0
}

variable "security_group_ids" {
  description = "The security groups to access EKS"
  type        = list
  default     = []
}

variable "eks_tags" {
  description = "A map of tags to add to EKS"
  type        = map(string)
  default     = {}
}

outputs.tf

output "endpoint" {
  value = aws_eks_cluster.eks.endpoint
}

Jump Server

We cannot access EKS directly, so we can use a EC2 as jump server to do it. The source codes in jumpserver folder are listed as below:

main.tf

locals {
  vpc_id             = var.vpc_id
  instance_type      = var.instance_type
  instance_ami       = var.instance_ami
  subnet_id          = var.public_subnets_id[0][0]
  security_group_ids = var.security_group_ids
  tags               = var.shared_tags
}

# Define the security group for jump server.
resource "aws_security_group" "jump-server" {
  name   = "Security group for jump server"
  vpc_id = local.vpc_id

  ingress {
    from_port   = 22
    protocol    = "TCP"
    to_port     = 22
    cidr_blocks = ["0.0.0.0/0"]
    description = "Allow SSH inbound traffic"
  }

  # Allow all outbound traffic
  egress {
    from_port        = 0
    protocol         = "-1"
    to_port          = 0
    cidr_blocks      = ["0.0.0.0/0"]
    ipv6_cidr_blocks = ["::/0"]
    description      = "Allow all outbound traffic"
  }

  tags = merge({
    Name : "Security group for jump server"
  }, local.tags)
}

# Define the role to be attached ec2 instance of the jump server
resource "aws_iam_role" "jump-server" {
  name               = "jump-server-role"
  assume_role_policy = jsonencode({
    "Version" : "2012-10-17",
    "Statement" : [
      {
        "Effect" : "Allow",
        "Action" : [
          "sts:AssumeRole"
        ],
        "Principal" : {
          "Service" : [
            "ec2.amazonaws.com"
          ]
        }
      }
    ]
  })
  tags = merge({
    Name : "Jump Server Role"
  }, local.tags)
}

# Attach the AdministratorAccess policy to jump server role
resource "aws_iam_role_policy_attachment" "jump-server-AdministratorAccess" {
  policy_arn = "arn:aws:iam::aws:policy/AdministratorAccess"
  role       = aws_iam_role.jump-server.name
}

# Define iam instance profile for ec2 instance of the message transmission service
resource "aws_iam_instance_profile" "jump-server-profile" {
  name = "jump-server-profile"
  role = aws_iam_role.jump-server.name
  tags = merge({
    Name : "Jump Server Profile"
  }, local.tags)
}

# Define the ec2 instance
resource "aws_instance" "jump-server" {
  count                       = 1
  instance_type               = local.instance_type
  ami                         = local.instance_ami
  associate_public_ip_address = true
  hibernation                 = false
  subnet_id                   = local.subnet_id
  vpc_security_group_ids      = setunion([aws_security_group.jump-server.id], local.security_group_ids)
  iam_instance_profile        = aws_iam_instance_profile.jump-server-profile.name
  tags                        = merge({
    Name = "Jumper Server"
  }, local.tags)
}

variables.tf

variable "vpc_id" {
  type = string
}

variable "instance_type" {
  type    = string
  default = "t3.micro"
}

variable "instance_ami" {
  type    = string
  default = "ami-0f9fc25dd2506cf6d"
}

variable "public_subnets_id" {
  description = "The public subnet id"
  type        = list
}

variable "security_group_ids" {
  description = "The security group for jump server"
  type        = list
  default = []
}

variable "shared_tags" {
  description = "A map of tags to add to Jump Server"
  type        = map(string)
  default     = {}
}

outputs.tf

output "jump-server-security-group-id" {
  value = aws_security_group.jump-server.id
}

Virginia

The source code region/virginia folder are listed as below:

providers.tf

terraform {
  required_version = ">= 0.13.7"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 4.8.0"
    }
    random = {
      source  = "hashicorp/random"
      version = "3.1.0"
    }
    null = {
      source  = "hashicorp/null"
      version = "3.1.0"
    }
    local = {
      source = "hashicorp/local"
      version = "2.1.0"
    }
    template = {
      source = "hashicorp/template"
      version = "2.2.0"
    }
    # Kubernetes Provider
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "2.1.0"
    }
  }

  backend "s3" {
    bucket  = "pin-terraform-state-us-east-1"
    key     = "terraform/backend.tfstate"
    region  = "us-east-1"
    encrypt = "true"
  }
}

main.tf

locals {
  region             = "us-east-1"
  availability_zones = ["${local.region}a", "${local.region}b", "${local.region}c"]
  tags               = {
    "Environment" : "PROD"
    "Project" : "Infrastructure"
  }
}

provider "aws" {
  region = local.region
}

module "Networking" {
  source                     = "../../module/networking"
  name                       = "VPC"
  availability_zones         = local.availability_zones
  vpc_cidr_block             = "10.0.0.0/16"
  public_subnets_cidr_block  = ["10.0.32.0/24", "10.0.96.0/24", "10.0.224.0/24"]
  private_subnets_cidr_block = ["10.0.0.0/19", "10.0.64.0/19", "10.0.128.0/19"]
  vpc_tags                   = local.tags
}

module "JumpServer" {
  source            = "../../module/jumpserver"
  vpc_id            = module.Networking.vpc_id
  public_subnets_id = module.Networking.public_subnets_id
  shared_tags       = local.tags
}

module "EKS" {
  source             = "../../module/eks"
  name               = "EKS"
  vpc_id             = module.Networking.vpc_id
  public_subnets_id  = module.Networking.public_subnets_id
  private_subnets_id = module.Networking.private_subnets_id
  desired_size       = 4
  max_size           = 16
  eks_tags           = local.tags
}

Now, we can build it

$ terraform init

$ terraform apply

Result

How to build VPC with Terraform

Pin Xiong — Wed, 20 Apr 2022 14:39:31 +0000

It's one of the most important things that everyone need to build VPC after applied a new aws account. The purpose of this blog is to help you build it using Terraform. All the codes have been put in github.

Design for networking

Public subnet: Deploying websites can be accessed directly by the Internet
Private subnet: Deploying applications only can be accessed by the websites in Public subnet. Also, the application can access to the Internet through NAT Gateway

How to build

Before creating a VPC, we need to install TerraForm first.

Code structure

infrastructure
├── module
│   └── networking
│       ├── main.tf
│       ├── outputs.tf
│       └── variables.tf
├── region
│   └── virginia
│       ├── main.tf
│       └── providers.tf
└── setup
    ├── main.tf
    ├── outputs.tf
    ├── providers.tf
    └── variables.tf

Setup environment

The source codes in setup folder are listed as below:

main.tf

locals {
  region = var.region
}

# Define a s3 bucket to store terraform state file.
resource "aws_s3_bucket" "terraform_state" {
  bucket        = format("terraform-state-%s", local.region)
  force_destroy = false
  lifecycle {
    ignore_changes = [bucket]
  }
}

resource "aws_s3_bucket_versioning" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id
  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.bucket
  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "AES256"
    }
  }
}

outputs.tf

output "s3_bucket_terraform_state" {
  value = aws_s3_bucket.terraform_state.bucket
}

providers.tf

terraform {
  required_version = ">= 0.13.7"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 4.8.0"
    }
  }
}

variables.tf

variable "region" {
  description = "The name of the aws Region"
  type        = string
}

And then, we need to create s3 bucket for remote backend, the s3 bucket name is the output of s3_bucket_terraform_state.

$ terraform init

$ terraform apply

Networking

The source codes in networking folder are listed as below:

main.tf

locals {
  name                       = var.name
  cidr_block                 = var.vpc_cidr_block
  public_subnets_cidr_block  = var.public_subnets_cidr_block
  private_subnets_cidr_block = var.private_subnets_cidr_block
  availability_zones         = var.availability_zones
  tags                       = var.vpc_tags
}

# VPC
resource "aws_vpc" "vpc" {
  cidr_block           = local.cidr_block
  enable_dns_hostnames = true
  enable_dns_support   = true

  tags = merge( {
    Name = local.name
  }, local.tags)
}

# Subnets
# Internet Gateway for Public Subnet
resource "aws_internet_gateway" "internet_gateway" {
  vpc_id = aws_vpc.vpc.id
  tags   = merge({
    Name = "${local.name} Internet Gateway"
  }, local.tags)
}

# EIP for NAT
resource "aws_eip" "nat_eip" {
  vpc        = true
  depends_on = [aws_internet_gateway.internet_gateway]
}

# NAT
resource "aws_nat_gateway" "nat_gateway" {
  allocation_id = aws_eip.nat_eip.id
  subnet_id     = element(aws_subnet.public_subnet.*.id, 0)

  tags = merge({
    Name = "${local.name} Nat Gateway"
  }, local.tags)
}

# Public subnet
resource "aws_subnet" "public_subnet" {
  vpc_id                  = aws_vpc.vpc.id
  count                   = length(local.public_subnets_cidr_block)
  cidr_block              = element(local.public_subnets_cidr_block, count.index)
  availability_zone       = element(local.availability_zones, count.index)
  map_public_ip_on_launch = true
  tags                    = merge( {
    Name = "${local.name} Public Subnet ${element(var.availability_zones, count.index)}"
  }, local.tags)
}

# Private Subnet
resource "aws_subnet" "private_subnet" {
  vpc_id                  = aws_vpc.vpc.id
  count                   = length(local.private_subnets_cidr_block)
  cidr_block              = element(local.private_subnets_cidr_block, count.index)
  availability_zone       = element(local.availability_zones, count.index)
  map_public_ip_on_launch = false
  tags                    = merge({
    Name = "${local.name} Private Subnet ${element(var.availability_zones, count.index)}"
  }, local.tags)
}


# Routing tables to route traffic for Private Subnet
resource "aws_route_table" "private" {
  vpc_id = aws_vpc.vpc.id

  tags = merge({
    Name = "${local.name} Private Route Table"
  }, local.tags)
}

# Routing tables to route traffic for Public Subnet
resource "aws_route_table" "public" {
  vpc_id = aws_vpc.vpc.id

  tags = merge({
    Name = "${local.name} Public Route Table"
  }, local.tags)
}

# Route for Internet Gateway
resource "aws_route" "public_internet_gateway" {
  route_table_id         = aws_route_table.public.id
  destination_cidr_block = "0.0.0.0/0"
  gateway_id             = aws_internet_gateway.internet_gateway.id
}

# Route for NAT
resource "aws_route" "private_nat_gateway" {
  route_table_id         = aws_route_table.private.id
  destination_cidr_block = "0.0.0.0/0"
  nat_gateway_id         = aws_nat_gateway.nat_gateway.id
}

# Route table associations for Public Subnets
resource "aws_route_table_association" "public" {
  count          = length(local.public_subnets_cidr_block)
  subnet_id      = element(aws_subnet.public_subnet.*.id, count.index)
  route_table_id = aws_route_table.public.id
}

# Route table associations for Private Subnets
resource "aws_route_table_association" "private" {
  count          = length(local.private_subnets_cidr_block)
  subnet_id      = element(aws_subnet.private_subnet.*.id, count.index)
  route_table_id = aws_route_table.private.id
}

# Default Security Group of VPC
resource "aws_security_group" "security_group" {
  name        = "${local.name} Security Group"
  description = "Default SG to allow traffic from the VPC"
  vpc_id      = aws_vpc.vpc.id
  depends_on  = [
    aws_vpc.vpc
  ]

  ingress {
    from_port = "0"
    to_port   = "0"
    protocol  = "-1"
    self      = true
  }

  egress {
    from_port = "0"
    to_port   = "0"
    protocol  = "-1"
    self      = "true"
  }

  tags = merge({}, local.tags)
}

outputs.tf

output "vpc_id" {
  value = aws_vpc.vpc.id
}

output "public_subnets_id" {
  value = [aws_subnet.public_subnet.*.id]
}

output "private_subnets_id" {
  value = [aws_subnet.private_subnet.*.id]
}

output "security_groups_ids" {
  value = [aws_security_group.security_group.id]
}

output "public_route_table" {
  value = aws_route_table.public.id
}

variables.tf

variable "name" {
  description = "The VPC name"
}

variable "vpc_cidr_block" {
  description = "CIDR block of the vpc"
}

variable "public_subnets_cidr_block" {
  type        = list
  description = "CIDR block for Public Subnet"
}

variable "private_subnets_cidr_block" {
  type        = list
  description = "CIDR block for Private Subnet"
}

variable "availability_zones" {
  type        = list
  description = "AZ in which all the resources will be deployed"
}

variable "vpc_tags" {
  description = "A map of tags to add to VPC"
  type        = map(string)
  default     = {}
}

Build VPC

The source codes in virginia folder are listed as below:

main.tf

locals {
  // the region code of virginia
  region             = "us-east-1"
  availability_zones = ["${local.region}a", "${local.region}b", "${local.region}c"]
  tags               = {
    "Environment" : "PROD"
    "Project" : "Infrastructure"
  }
}

provider "aws" {
  region = local.region
}

module "Networking" {
  source                     = "../../module/networking"
  name                       = "VPC"
  availability_zones         = local.availability_zones
  vpc_cidr_block             = "10.0.0.0/16"
  public_subnets_cidr_block  = ["10.0.32.0/24", "10.0.96.0/24", "10.0.224.0/24"]
  private_subnets_cidr_block = ["10.0.0.0/19", "10.0.64.0/19", "10.0.128.0/19"]
  vpc_tags                   = local.tags
}

providers.tf

terraform {
  required_version = ">= 0.13.7"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 4.8.0"
    }
  }

  backend "s3" {
    // the s3 bucket name
    bucket  = "*********"
    key     = "terraform/backend.tfstate"
    region  = "us-east-1"
    encrypt = "true"
  }
}

Now, we can build the VPC

$ terraform init

$ terraform apply