DEV Community: Oloruntobi Olurombi

Step-by-Step Guide: Deploying a Static Web Application in OpenShift Using a Custom S2I Builder Image

Oloruntobi Olurombi — Mon, 09 Dec 2024 14:06:27 +0000

This guide takes you through the process of creating a custom Source-to-Image (S2I) builder image, using it to deploy a static web application in OpenShift, and making the application accessible externally by updating the route with your cluster's public IP.

By the end of this guide, you'll have a fully functional static web application running on your OpenShift cluster.

Prerequisites

Before diving into the process, ensure the following are set up:

OpenShift CLI (oc): Installed and configured to connect to your OpenShift cluster.
OpenShift Cluster: Access to a running OpenShift cluster with a project where you can create resources.
Git: Installed on your system for cloning repositories.
Environment Variable: The OpenShift cluster's public IP should be stored in an environment variable called OPENSHIFT_CLUSTER_PUBLICIP.
Basic Terminal Skills: Knowledge of executing commands and navigating files in the terminal.

Part 1: Verify OpenShift Setup

Step 1: Open the Terminal

Open a terminal on your local system or through the OpenShift web console:

In the web console, navigate to Terminal ==> New Terminal.

Check the version of OpenShift to confirm your setup is active:

oc version

Part 2: Create a Custom S2I Builder Image

S2I (Source-to-Image) builder images are designed to take application source code and package it into a runnable container image. In this part, you'll create a custom builder image tailored for static web applications.

Step 2: Define the S2I Repository

Set the GitHub repository URL that contains the S2I source code for a simple static web server:

CLOUDACADEMY_GITHUB_S2I_URL=https://github.com/cloudacademy/openshift-s2i-simplewebsvr.git

Step 3: Create the Builder Image

Run the following command to create a new build named static-webapp-builder using the S2I repository:

oc new-build --strategy=docker --name static-webapp-builder --code $CLOUDACADEMY_GITHUB_S2I_URL

This command initialises a build process using the provided source code and Docker strategy.

Step 4: Verify the Build

Check if the build is completed:

oc get builds

Inspect all resources created in the current project:

oc get all

View the build logs to confirm its success:

oc logs build/static-webapp-builder-1

Part 3: Deploy the Static Web Application

Now that your S2I builder image is ready, you’ll use it to build and deploy a static web application.

Step 5: Build and Launch the Application
Run the following command to create and deploy an application named dashboard using the static-webapp-builder image:

oc new-app static-webapp-builder~https://github.com/ColorlibHQ/gentelella.git --name dashboard

This command automatically creates the build and deployment configuration, service, and image stream.

Step 6: Monitor the Build Process

Check the build logs to ensure everything works correctly:

oc logs -f bc/dashboard

Verify all created resources:

oc get all

Confirm that the dashboard pod is running:

oc get pods --field-selector status.phase=Running

Ensure the dashboard service has been registered:

oc get svc

Part 4: Expose the Application to External Traffic

By default, OpenShift services are not exposed outside the cluster. In this section, you’ll expose the application and update the route.

Step 7: Expose the Service

Expose the dashboard service to create a route:

oc expose svc/dashboard

verify the route:

oc get routes

Step 8: Update the Route

The route created in the previous step uses a default Fully Qualified Domain Name (FQDN). You’ll now update it to match the public IP address of your OpenShift cluster.

Install yq for YAML Processing

Download and install the yq utility:

curl -O --location https://github.com/mikefarah/yq/releases/download/2.4.1/yq_linux_amd64
sudo mv yq_linux_amd64 /usr/bin/yq
sudo chmod +x /usr/bin/yq

Export and Edit the Route Configuration

Export the current route configuration:

oc get route dashboard -o yaml > route.yaml

Update the FQDN using yq:

yq w --inplace route.yaml spec.host dashboard.$OPENSHIFT_CLUSTER_PUBLICIP.nip.io

Apply the Updated Route

Delete the existing route:

oc delete route dashboard

Recrate the route with the update configuration:

oc apply -f route.yaml

Confirm the updated route:

oc get route dashboard

Part 5: Test the Application

Step 9: Verify Connectivity

Test the application’s connectivity using curl:

curl -I -s http://dashboard.$OPENSHIFT_CLUSTER_PUBLICIP.nip.io/production/

Step 10: Access the Application

Generate and display the application URL:

echo http://dashboard.$OPENSHIFT_CLUSTER_PUBLICIP.nip.io/production/

Copy the URL and open it in your browser to view the deployed static web application.

Summary

You have successfully:

Created a custom S2I builder image.
Built and deployed a static web application using OpenShift.
Exposed the application to external traffic by configuring a route.

This hands-on exercise demonstrates the flexibility of OpenShift in building, deploying, and managing applications seamlessly. Explore further by customising your static application or integrating advanced features! 🚀

Deploying Jenkins on Kubernetes using Helm

Oloruntobi Olurombi — Wed, 16 Oct 2024 21:49:19 +0000

This article walks through creating and deploying a custom Helm chart to deploy Jenkins on Kubernetes. Helm is a package manager for Kubernetes, which helps automate the deployment of applications by packaging them into charts. Helm simplifies application deployment by managing Kubernetes resources as a single unit.

The deployment involves creating a Helm chart, defining the necessary configurations, deploying Jenkins, and accessing the Jenkins console through the web browser.

Why Helm?

Helm streamlines Kubernetes application deployments by:

Packaging Kubernetes resources: Helm charts contain all the manifests required to deploy an application, allowing easier management and updates.
Version control: Helm maintains release versions, making it easy to upgrade or rollback applications.
Customisation: Helm charts allow parameterisation, so you can use the same chart to deploy different instances of the same application by tweaking configuration values.
Repeatability: Helm makes it easy to deploy applications consistently across multiple environments.

Step 1: Creating a Helm Chart

To begin, we will create a Helm chart called myhelmchart that will deploy Jenkins.

helm create myhelmchart

This command generates a directory structure with the basic files and templates needed for a Kubernetes deployment. Let’s break down the structure of the generated chart:

Helm Chart Directory Structure

Chart.yaml: Defines the metadata of the Helm chart, such as the chart name, version, and appVersion (version of the deployed application).
values.yaml: Default values that the chart uses, such as image name, ports, and service type. These values can be overridden during installation.
templates/: Contains the Kubernetes manifests (YAML files) for deployments, services, and other Kubernetes resources.
NOTES.txt: Provides instructions after a chart is installed.

Step 2: Update Chart.yaml

Next, update Chart.yaml to specify the Jenkins version we want to deploy:

cd myhelmchart
vim Chart.yaml

Modify the appVersion field to reflect the Jenkins version:

appVersion: "2.249.2"

Step 3: Clean Up Unnecessary Files

We will delete unnecessary files and directories to simplify our Helm chart:

rm -rf templates/*.yaml templates/tests

Step 4: Create Deployment and Service Manifests

In the templates/ directory, create two files: one for the Jenkins deployment and one for the service.

1. Create Jenkins Deployment Manifest

cd templates
touch jenkins-deployment.yaml
vim jenkins-deployment.yaml

Add the following code to jenkins-deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ template "myhelmchart.fullname" . }}
  labels:
    app: jenkins
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jenkins
  template:
    metadata:
      labels:
        app: jenkins
    spec:
      containers:
        - name: jenkins
          image: {{ printf "%s:%s" .Values.image.repository .Values.image.tag | quote }}
          env:
            - name: JENKINS_USERNAME
              value: {{ default "test" .Values.jenkinsUsername | quote }}
            - name: JENKINS_PASSWORD
              {{- if .Values.jenkinsPassword }}
              value: {{.Values.jenkinsPassword }}
              {{- else }}
              value: testPassword
              {{- end }}
          ports:
            {{- range .Values.containerPorts }}
            - name: {{ .name }}
              containerPort: {{ .port }}
            {{- end }}

This manifest defines a Deployment with the following key elements:

metadata: Specifies the name and labels for the deployment.

spec.replicas: Ensures there is only one replica (Jenkins instance) running.

spec.template: Describes the pod template with metadata and containers. It sets environment variables JENKINS_USERNAME and JENKINS_PASSWORD from the chart’s values.yaml file, with default values if not provided.

containers: Defines the Jenkins container, using the image specified in values.yaml, and exposes ports from the same file.

2. Create Jenkins Service Manifest

touch jenkins-service.yaml
vim jenkins-service.yaml

Add the following content to jenkins-service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: {{ template "myhelmchart.fullname" . }}
spec:
  type: {{ .Values.service.type }}
  ports:
    - name: http
      port: 8080
      targetPort: http
      nodePort: {{ .Values.service.nodePort }}
  selector:
    app: jenkins

This manifest defines a Service to expose the Jenkins pod:

spec.type: Sets the service type to NodePort, which exposes the application on a specific port of the node (EC2 instance).
ports: Maps port 8080 of the Jenkins pod to a NodePort specified in values.yaml.
selector: Ensures the service targets pods labeled with app: jenkins.

Step 5: Update NOTES.txt

The NOTES.txt file provides information after a successful Helm installation. Let’s update it:

echo “This is my first chart” > NOTES.txt

Step 6: Update `values.yaml`

The values.yaml file stores default values used by the templates. Update it with the following configuration:

image:
  repository: bitnami/jenkins
  tag: latest

jenkinsUsername: ""
jenkinsPassword: ""

containerPorts:
  - name: http
    port: 8080

service:
  type: NodePort
  nodePort: 30080

This configuration:

image.repository: Uses the official Bitnami Jenkins image.
jenkinsUsername and jenkinsPassword: Default values can be passed during installation.
service: Sets the service type to NodePort, exposing Jenkins on port 30080.

Step 7: Run the Helm Template Command

Before installing the chart, verify the templates using the helm template command, which renders the Kubernetes manifests without applying them:

helm template myhelmchart . -s templates/jenkins-deployment.yaml

Step 8: Install the Helm Chart

Now install the Helm chart using the helm install command:

helm install myhelmchart ./myhelmchart

This deploys Jenkins to your Kubernetes cluster.

Step 9: Get Public IP of the Node and Access Jenkins

To access Jenkins via the web browser, we need the public IP of the node (EC2 instance) hosting the Jenkins pod.

Get the public IP of the node:

kubectl get nodes -o wide

Look for the external/public IP address of the node that the Jenkins pod is running on.

Access Jenkins in the browser:

Open your web browser and navigate to:

http://<Public_IP>:30080

Step 10: Access Jenkins Console

When you access Jenkins in the browser, it will prompt you for a username and password. Use the credentials set in your Helm chart:

Username: The value of jenkinsUsername (or test if not set).
Password: The value of jenkinsPassword (or testPassword if not set).

Once you log in, you’ll have access to the Jenkins dashboard.

Conclusion

In this article, we created a Helm chart to deploy Jenkins on Kubernetes, customised the chart with a deployment and service manifest, and accessed Jenkins via a browser. Helm makes managing Kubernetes applications more efficient by providing easy-to-use templates and a simple installation process.

☕️ If this article helped you avoid a tech meltdown or gave you a lightbulb moment, feel free to buy me a coffee! It keeps my code clean, my deployments smooth, and my spirit caffeinated. Help fuel the magic here!.

Happy deploying!

End-to-End Deployment and Monitoring of EKS and Flask Apps with Terraform, GitHub Actions, Helm, Prometheus and Grafana

Oloruntobi Olurombi — Sat, 14 Sep 2024 20:50:50 +0000

In the realm of cloud-native applications, automation plays a vital role in simplifying deployments and optimising infrastructure management. In this article, we will delve into how to automate the provisioning of an Amazon EKS cluster and deploy a Flask application using Terraform and GitHub Actions. Along the way, we'll cover essential security best practices to safeguard your environment, explore monitoring techniques to ensure system health, and discuss building a resilient CI/CD pipeline for continuous integration and deployment. This comprehensive approach will help you achieve efficient, secure, and scalable cloud-native application management.

Overview
Prerequisites
Infrastructure Automation: EKS Cluster Setup
Application Deployment: Flask App on EKS
Monitoring with Prometheus and Grafana
Security Best Practices
Conclusion

Overview

The goal of this project is to automate the deployment of a containerised Flask application on an EKS (Elastic Kubernetes Service) cluster. Using Terraform to provision AWS resources and GitHub Actions to automate the CI/CD pipeline, this setup allows for seamless infrastructure management and application deployment.

Why Terraform?

Terraform enables you to write declarative code for infrastructure. Instead of manually creating resources like VPCs, subnets, or an EKS cluster, we automate everything via Infrastructure as Code (IaC).

Why GitHub Actions?

GitHub Actions provides a powerful way to integrate CI/CD, testing, static analysis, and security checks into the code deployment process.

Prerequisites

Before diving into the automation, here are the prerequisites you’ll need to get started:

AWS Account: Create an AWS account if you don’t have one.
IAM Access Keys: Set up access keys with permissions for managing EKS, EC2, and S3.
S3 Bucket: Create an S3 bucket to store your Terraform state files securely.
AWS CLI: Install and configure the AWS CLI.
Terraform: Make sure Terraform is installed on your local machine or use GitHub Actions for automation.
GitHub Secrets: Add AWS credentials (access keys, secret keys) and other sensitive data as GitHub secrets to avoid hardcoding them.
Synk: Create a Synk account and get your Token.
SonarCloud: Create a SonarCloud account and get your Token, Organisation key and Project key.

Infrastructure Automation: EKS Cluster Setup

Automating infrastructure deployment is key to maintaining scalable, consistent, and reliable environments. In this project, Terraform is utilised to automate the provisioning of an EKS cluster, its node groups, and the supporting AWS infrastructure. This includes VPC creation, IAM roles, S3 bucket setup, and cloud resources like CloudWatch and CloudTrail for logging and monitoring.

Terraform Setup

Let’s start by provisioning the necessary infrastructure. Below is the detailed explanation of the key resources defined in the Terraform files.

EKS Cluster and Node Group (main.tf):

This provision an EKS cluster and node group with IAM roles attached.
The cluster supports encryption using a KMS key, and the worker nodes are set up to scale between a minimum of 2 nodes. Outputs include the cluster name and endpoint for easy reference.

touch main.tf

terraform {
  backend "s3" {
    bucket = "regtech-iac"
    key = "terraform.tfstate"
    region = "us-east-1"
    encrypt = true 
  }
}

# Provides an EKS Cluster
resource "aws_eks_cluster" "eks_cluster" {
  name     = var.cluster_name
  role_arn = aws_iam_role.eks_cluster_role.arn

  version = "1.28"

  vpc_config {
    subnet_ids = [aws_subnet.public_subnet_1.id, aws_subnet.public_subnet_2.id, aws_subnet.public_subnet_3.id]
  }

  encryption_config {
    provider {
      key_arn = aws_kms_key.eks_encryption_key.arn
    }
    resources = ["secrets"]
  }

  # Ensure that IAM Role permissions are created before and deleted after EKS Cluster handling.
  # Otherwise, EKS will not be able to properly delete EKS managed EC2 infrastructure such as Security Groups.
  depends_on = [
    aws_iam_role_policy_attachment.eks_cluster_policy_attachment,
    aws_iam_role_policy_attachment.eks_service_policy_attachment,
  ]
}

# Provides an EKS Node Group 

resource "aws_eks_node_group" "eks_node_group" {
  cluster_name    = aws_eks_cluster.eks_cluster.name
  node_group_name = var.node_group_name
  node_role_arn   = aws_iam_role.eks_node_group_role.arn
  subnet_ids      = [aws_subnet.public_subnet_1.id, aws_subnet.public_subnet_2.id, aws_subnet.public_subnet_3.id]

  scaling_config {
    desired_size = 2
    max_size     = 2
    min_size     = 2
  }
  update_config {
    max_unavailable = 1
  }

  # Ensure that IAM Role permissions are created before and deleted after EKS Node Group handling.
  # Otherwise, EKS will not be able to properly delete EC2 Instances and Elastic Network Interfaces.
  depends_on = [
    aws_iam_role_policy_attachment.eks_worker_node_policy_attachment,
    aws_iam_role_policy_attachment.eks_cni_policy_attachment,
    aws_iam_role_policy_attachment.ec2_container_registry_readonly,
  ]
}

# Extra resources 
resource "aws_ebs_volume" "volume_regtech"{
    availability_zone = var.az_a
    size = 40
    encrypted = true
    type = "gp2"
    kms_key_id        = aws_kms_key.ebs_encryption_key.arn
}

resource "aws_s3_bucket" "regtech_iac" {
  bucket = var.bucket_name
}

resource "aws_s3_bucket_server_side_encryption_configuration" "regtech_iac_encrypt_config" {
    bucket = aws_s3_bucket.regtech_iac.bucket
    rule {
        apply_server_side_encryption_by_default {
        kms_master_key_id = aws_kms_key.s3_encryption_key.arn  
        sse_algorithm = "aws:kms"
        }
    }
}


# OutPut Resources
output "endpoint" {
  value = aws_eks_cluster.eks_cluster.endpoint
}

output "eks_cluster_name" {
    value = aws_eks_cluster.eks_cluster.name
}

Networking (vpc.tf):

Defines a VPC, public subnets for the EKS cluster, and private subnets for other resources, ensuring flexibility in network architecture.

vpc.tf

# Provides a VPC resource
resource "aws_vpc" "main" {
  cidr_block       = var.vpc_cidr_block
  instance_tenancy = "default"

  tags = {
    Name = var.tags_vpc
  }
}


# Provides an VPC Public subnet resource
resource "aws_subnet" "public_subnet_1" {
  vpc_id     = aws_vpc.main.id
  cidr_block = var.p_s_1_cidr_block
  availability_zone = var.az_a
  map_public_ip_on_launch = true

  tags = {
    Name = var.tags_public_subnet_1
  }
}

resource "aws_subnet" "public_subnet_2" {
  vpc_id     = aws_vpc.main.id
  cidr_block = var.p_s_2_cidr_block
  availability_zone = var.az_b
  map_public_ip_on_launch = true

  tags = {
    Name = var.tags_public_subnet_2
  }
}

resource "aws_subnet" "public_subnet_3" {
  vpc_id     = aws_vpc.main.id
  cidr_block = var.p_s_3_cidr_block
  availability_zone = var.az_c
  map_public_ip_on_launch = true

  tags = {
    Name = var.tags_public_subnet_3
  }
}

# Provides an VPC Private subnet resource
resource "aws_subnet" "private_subnet_1" {
  vpc_id     = aws_vpc.main.id
  cidr_block = var.private_s_1_cidr_block
  availability_zone = var.az_private_a
  map_public_ip_on_launch = false 

  tags = {
    Name = var.tags_private_subnet_1
  }
}

resource "aws_subnet" "private_subnet_2" {
  vpc_id     = aws_vpc.main.id
  cidr_block = var.private_s_2_cidr_block
  availability_zone = var.az_private_b
  map_public_ip_on_launch = false 

  tags = {
    Name = var.tags_private_subnet_2
  }
}

resource "aws_subnet" "private_subnet_3" {
  vpc_id     = aws_vpc.main.id
  cidr_block = var.private_s_3_cidr_block
  availability_zone = var.az_private_c
  map_public_ip_on_launch = false 

  tags = {
    Name = var.tags_private_subnet_3
  }
}

IAM Roles (iam.tf):

IAM roles and policies for the EKS cluster, node groups, and autoscaler. Includes roles for security services like CloudWatch and CloudTrail, ensuring robust monitoring.

iam.tf

# Declare the aws_caller_identity data source
data "aws_caller_identity" "current" {}


# IAM Role for EKS Cluster Plane 

resource "aws_iam_role" "eks_cluster_role" {
    name = var.eks_cluster_role_name

    assume_role_policy = jsonencode({
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Principal": {
                    "Service": "eks.amazonaws.com"
                },
                "Action": "sts:AssumeRole"
            }
        ]
    })
}

resource "aws_iam_role_policy_attachment" "eks_cluster_policy_attachment" {
    policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
    role = aws_iam_role.eks_cluster_role.name 
}

resource "aws_iam_role_policy_attachment" "eks_service_policy_attachment" {
  role       = aws_iam_role.eks_cluster_role.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSServicePolicy"
}

# IAM Role for Worker node

resource "aws_iam_role" "eks_node_group_role" {
    name = var.eks_node_group_role_name

    assume_role_policy = jsonencode({
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Principal": {
                    "Service": "ec2.amazonaws.com"
                },
                "Action": "sts:AssumeRole"
            }
        ]
    })
}

resource "aws_iam_role_policy_attachment" "eks_worker_node_policy_attachment" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
  role       = aws_iam_role.eks_node_group_role.name
}

resource "aws_iam_role_policy_attachment" "eks_cni_policy_attachment" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
  role       = aws_iam_role.eks_node_group_role.name
}

resource "aws_iam_role_policy_attachment" "ec2_container_registry_readonly" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
  role       = aws_iam_role.eks_node_group_role.name
}

resource "aws_iam_instance_profile" "eks_node_instance_profile" {
    name = var.eks_node_group_profile
    role = aws_iam_role.eks_node_group_role.name
}


# Policy For volume creation and attachment

resource "aws_iam_role_policy" "eks_node_group_volume_policy" {
  name   = var.eks_node_group_volume_policy_name
  role   = aws_iam_role.eks_node_group_role.name
  policy = jsonencode({
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": [
          "ec2:CreateTags",
          "ec2:DescribeTags",
          "ec2:DescribeVolumes",
          "ec2:DescribeVolumeStatus",
          "ec2:CreateVolume",
          "ec2:AttachVolume"
        ],
        "Resource": "arn:aws:ec2:${var.region}:${data.aws_caller_identity.current.account_id}:volume/*"
      }
    ]
  })
}


# IAM Role for CloudWatch 

resource "aws_iam_role" "cloudwatch_role" {
  name = "cloudwatch_role_log"

  assume_role_policy = jsonencode({
    "Version": "2012-10-17",
    "Statement": [
      {
        "Action": "sts:AssumeRole",
        "Principal": {
          "Service": "cloudwatch.amazonaws.com"
        },
        "Effect": "Allow",
        "Sid": ""
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "cloudwatch_policy_attachment" {
  role       = aws_iam_role.cloudwatch_role.name
  policy_arn = "arn:aws:iam::aws:policy/CloudWatchLogsFullAccess"
}

# IAM Role for CloudTrail

resource "aws_iam_role" "cloudtrail_role" {
  name = "cloudtrail_role_log"

  assume_role_policy = jsonencode({
    "Version": "2012-10-17",
    "Statement": [
      {
        "Action": "sts:AssumeRole",
        "Principal": {
          "Service": "cloudtrail.amazonaws.com"
        },
        "Effect": "Allow",
        "Sid": ""
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "cloudtrail_policy_attachment" {
  role       = aws_iam_role.cloudtrail_role.name
  policy_arn = "arn:aws:iam::aws:policy/AWSCloudTrail_FullAccess"
}


# KMS Key Policy for Encryption

resource "aws_kms_key" "ebs_encryption_key" {
  description = "KMS key for EBS volume encryption"
}

resource "aws_kms_key" "s3_encryption_key" {
  description = "KMS key for S3 bucket encryption"
}

resource "aws_kms_key" "eks_encryption_key" {
  description = "KMS key for EKS secret encryption"
}


resource "aws_s3_bucket_policy" "regtech_iac_policy" {
  bucket = aws_s3_bucket.regtech_iac.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          Service = "cloudtrail.amazonaws.com"
        }
        Action = "s3:GetBucketAcl"
        Resource = "arn:aws:s3:::${aws_s3_bucket.regtech_iac.bucket}"
      },
      {
        Effect = "Allow"
        Principal = {
          Service = "cloudtrail.amazonaws.com"
        }
        Action = "s3:PutObject"
        Resource = "arn:aws:s3:::${aws_s3_bucket.regtech_iac.bucket}/AWSLogs/${data.aws_caller_identity.current.account_id}/*"
        Condition = {
          StringEquals = {
            "s3:x-amz-acl" = "bucket-owner-full-control"
          }
        }
      }
    ]
  })
}

CloudWatch and Monitoring (cloudwatch.tf):

This provisions CloudWatch log groups, an SNS topic for alerts, and a CloudWatch alarm to monitor CPU utilisation. CloudTrail logs are configured to monitor S3 and management events.

touch cloudwatch.tf

resource "aws_cloudwatch_log_group" "eks_log_group" {
  name              = "/aws/eks/cluster-logs-regtech"
  retention_in_days = 30
}

resource "aws_cloudtrail" "security_trail" {
  name                          = "security-trail-log"
  s3_bucket_name                = aws_s3_bucket.regtech_iac.bucket
  include_global_service_events = true
  is_multi_region_trail         = true
  enable_log_file_validation    = true

  event_selector {
    read_write_type           = "All"
    include_management_events = true

    data_resource {
      type = "AWS::S3::Object"
      values = ["arn:aws:s3:::${aws_s3_bucket.regtech_iac.bucket}/"]
    }
  }
}


resource "aws_sns_topic" "alarm_topic" {
    name = "high-cpu-alarm-topic"
}

resource "aws_sns_topic_subscription" "alarm_subscription" {
    topic_arn = aws_sns_topic.alarm_topic.arn
    protocol = "email"
    endpoint = "oloruntobiolurombi@gmail.com"
}

resource "aws_cloudwatch_metric_alarm" "cpu_alarm" {
  alarm_name          = "high_cpu_usage"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = "2"
  metric_name         = "CPUUtilization"
  namespace           = "AWS/EC2"
  period              = "120"
  statistic           = "Average"
  threshold           = "70"

  alarm_actions = [
    aws_sns_topic.alarm_topic.arn
  ]
}

AutoScaler IAM (iam-autoscaler.tf):

This will provision roles and policies for enabling the EKS Cluster Autoscaler are included, which will help in adjusting the number of worker nodes based on resource demands.

data "aws_iam_policy_document" "eks_cluster_autoscaler_assume_role_policy" {
  statement {
    actions = ["sts:AssumeRoleWithWebIdentity"]
    effect  = "Allow"

    condition {
      test     = "StringEquals"
      variable = "${replace(aws_iam_openid_connect_provider.eks.url, "https://", "")}:sub"
      values   = ["system:serviceaccount:kube-system:cluster-autoscaler"]
    }

    principals {
      identifiers = [aws_iam_openid_connect_provider.eks.arn]
      type        = "Federated"
    }
  }
}

resource "aws_iam_role" "eks_cluster_autoscaler" {
  assume_role_policy = data.aws_iam_policy_document.eks_cluster_autoscaler_assume_role_policy.json
  name               = "eks-cluster-autoscaler"
}

resource "aws_iam_policy" "eks_cluster_autoscaler" {
  name = "eks-cluster-autoscaler"

  policy = jsonencode({
    Statement = [{
      Action = [
                "autoscaling:DescribeAutoScalingGroups",
                "autoscaling:DescribeAutoScalingInstances",
                "autoscaling:DescribeLaunchConfigurations",
                "autoscaling:DescribeTags",
                "autoscaling:SetDesiredCapacity",
                "autoscaling:TerminateInstanceInAutoScalingGroup",
                "ec2:DescribeLaunchTemplateVersions"
            ]
      Effect   = "Allow"
      Resource = "*"
    }]
    Version = "2012-10-17"
  })
}

resource "aws_iam_role_policy_attachment" "eks_cluster_autoscaler_attach" {
  role       = aws_iam_role.eks_cluster_autoscaler.name
  policy_arn = aws_iam_policy.eks_cluster_autoscaler.arn
}

output "eks_cluster_autoscaler_arn" {
  value = aws_iam_role.eks_cluster_autoscaler.arn
}

Routing (`security_groups.tf`)

This defines the security groups required for your infrastructure. Security groups act as virtual firewalls that control the inbound and outbound traffic to your resources.

touch security_groups.tf

# Provides a security group 
resource "aws_security_group" "main_sg" {
    name = "main_sg"
    description = var.main_sg_description
    vpc_id = aws_vpc.main.id 

    ingress  {
        description = "ssh access"
        from_port = 22
        to_port = 22
        protocol = "tcp"
        cidr_blocks = ["0.0.0.0/0"]
    }

    ingress  {
        description = "Kubernetes API access"
        from_port = 443
        to_port = 443 
        protocol = "tcp"
        cidr_blocks = ["0.0.0.0/0"]
    }

    egress {
        from_port = 0
        to_port = 0
        protocol = -1 
        cidr_blocks = ["0.0.0.0/0"]
    }

    tags = {
        Name = var.tags_main_sg_eks
    }
}

Variables (`variables.tf`)

This file holds the variable definitions that are utilized throughout your Terraform configurations. These variables serve as configurable parameters, offering default values that can be modified or overridden based on specific requirements. By centralizing variable management, this approach ensures flexibility and reusability, allowing you to tailor infrastructure settings without altering the core configuration files. This is particularly useful when deploying infrastructure across different environments or making adjustments to scale resources.

touch variables.tf

variable "region" {
    type = string 
    default = "us-east-1"
}

variable "bucket_name" {
    type = string 
    default = "regtech-logs"
}

variable "aws_access_key_id" {
    type = string
    default = ""
}

variable "aws_secret_access_key" {
    type = string
    default = ""
}

variable "tags_vpc" {
    type = string 
    default = "main-vpc-eks"
}

variable "tags_public_rt" {
    type = string 
    default = "public-route-table"
}

variable "tags_igw" {
    type = string 
    default = "internet-gateway"
}

variable "tags_public_subnet_1" {
    type = string 
    default = "public-subnet-1"
}

variable "tags_public_subnet_2" {
    type = string 
    default = "public-subnet-2"
}

variable "tags_public_subnet_3" {
    type = string 
    default = "public-subnet-3"
}

variable "tags_private_subnet_1" {
    type = string 
    default = "private-subnet-1"
}

variable "tags_private_subnet_2" {
    type = string 
    default = "private-subnet-2"
}

variable "tags_private_subnet_3" {
    type = string 
    default = "private-subnet-3"
}

variable "tags_main_sg_eks" {
    type = string
    default = "main-sg-eks"
}

variable "instance_type" {
    type = string 
    default = "t2.micro"
}

variable "cluster_name" {
    type = string 
    default = "EKSCluster"
}

variable "node_group_name" {
    type = string 
    default = "SlaveNode"
}

variable "vpc_cidr_block" {
    type = string 
    default = "10.0.0.0/16"
}

variable "p_s_1_cidr_block" {
    type = string 
    default = "10.0.1.0/24"
}

variable "az_a" {
    type = string 
    default = "us-east-1a"
}

variable "p_s_2_cidr_block" {
    type = string 
    default = "10.0.2.0/24"
}

variable "az_b" {
    type = string 
    default = "us-east-1b"
}

variable "p_s_3_cidr_block" {
    type = string 
    default = "10.0.3.0/24"
}

variable "az_c" {
    type = string 
    default = "us-east-1c"
}

variable "private_s_1_cidr_block" {
    type = string 
    default = "10.0.4.0/24"
}

variable "az_private_a" {
    type = string 
    default = "us-east-1c"
}

variable "private_s_2_cidr_block" {
    type = string 
    default = "10.0.5.0/24"
}

variable "az_private_b" {
    type = string 
    default = "us-east-1c"
}

variable "private_s_3_cidr_block" {
    type = string 
    default = "10.0.6.0/24"
}

variable "az_private_c" {
    type = string 
    default = "us-east-1c"
}

variable "main_sg_description" {
    type = string 
    default = "Allow TLS inbound traffic and all outbound traffic"
}


variable "eks_node_group_profile" {
    type = string 
    default = "eks-node-group-instance-profile_log"
}

variable "eks_cluster_role_name" {
    type = string 
    default = "eksclusterrole_log"
}

variable "eks_node_group_role_name" {
    type = string 
    default = "eks-node-group-role_log"
}

variable "eks_node_group_volume_policy_name" {
    type = string 
    default = "eks-node-group-volume-policy"
}

variable "eks_describe_cluster_policy_name" {
    type = string 
    default = "eks-describe-cluster-policy_log"
}

variable "tags_nat" {
    type = string 
    default = "nat-gateway_eip"
}

variable "tags_k8s-nat" {
    type = string 
    default = "k8s-nat"
}

Provider (`provider.tf`)

This step is essential in any Terraform project, as it defines the provider configuration, specifying which cloud platform or service you'll be interacting with. In this case, the provider is AWS, and this configuration establishes the connection between Terraform and the AWS environment. It ensures that all infrastructure resources are provisioned and managed within the specified cloud platform. Properly setting up the provider is foundational to the entire Terraform workflow, enabling seamless communication with AWS services such as EC2, S3, and EKS. Without it, Terraform wouldn't know where to deploy or manage the infrastructure.

touch

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

# Configure the AWS Provider
provider "aws" {
  region = var.region
  access_key = var.aws_access_key_id
  secret_key = var.aws_secret_access_key 
}

IAM OpenID (`oidc.tf`)

This component sets up an IAM OpenID Connect (OIDC) provider, which plays a critical role in enabling secure authentication and identity management for your AWS resources. By establishing this OIDC provider, your Kubernetes clusters can seamlessly integrate with AWS IAM, allowing you to manage permissions and roles for applications running within the cluster. This is particularly important for securely granting temporary, limited access to AWS services like S3 or DynamoDB, without the need for hardcoding credentials. The OIDC provider facilitates trust between AWS IAM and external identity providers, enabling scalable, secure access control across your infrastructure.

touch oidc.tf

data "tls_certificate" "eks" {
  url = aws_eks_cluster.eks_cluster.identity[0].oidc[0].issuer
}

resource "aws_iam_openid_connect_provider" "eks" {
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = [data.tls_certificate.eks.certificates[0].sha1_fingerprint]
  url             = aws_eks_cluster.eks_cluster.identity[0].oidc[0].issuer
}

GitHub Actions Workflow setup: `eks-setup.yaml`

The eks-setup.yaml file is designed to automate the process of deploying an EKS cluster on AWS. This workflow streamlines the entire infrastructure setup, eliminating manual intervention and ensuring consistency across deployments.

Purpose:

This workflow automates the provisioning of AWS infrastructure, focusing on setting up an Amazon EKS cluster. By leveraging Terraform within GitHub Actions, it ensures that your EKS cluster is deployed efficiently and consistently, aligning with infrastructure-as-code best practices.

Steps:

AWS Login: Configures the AWS credentials required for secure authentication, ensuring Terraform has the proper access to interact with AWS services.

Terraform Initialisation: Initialises Terraform by downloading and configuring the necessary provider plugins (such as AWS), setting up the working environment to handle the infrastructure resources.

Terraform Plan: Generates a detailed execution plan, outlining the changes Terraform will make to the infrastructure, without actually applying those changes yet. This step helps verify the proposed updates.

Terraform Apply: Executes the Terraform configuration, applying the planned changes and provisioning the EKS cluster along with any related resources. This fully automates the creation of the Kubernetes control plane, networking, and worker nodes in AWS.

This workflow is essential for ensuring a repeatable, scalable deployment process while maintaining the flexibility to adjust infrastructure configurations based on changing requirements.

name: Set up EKS with Terraform

on: push

env:
  AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
  AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
  AWS_REGION: ${{ secrets.AWS_REGION }}
  EKS_CLUSTER_NAME: ${{ secrets.EKS_CLUSTER_NAME }}

jobs:
  LogInToAWS:
    runs-on: ubuntu-latest
    steps:
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-access-key-id: ${{ env.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ env.AWS_SECRET_ACCESS_KEY }}
          aws-region: ${{ env.AWS_REGION }}

  TerraformInit:
    runs-on: ubuntu-latest
    needs: LogInToAWS
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Initialize Terraform
        run: terraform init

  TerraformPlan:
    runs-on: ubuntu-latest
    needs: TerraformInit
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Terraform Plan
        run: terraform plan

  TerraformApply:
    runs-on: ubuntu-latest
    needs: TerraformPlan
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Apply Terraform configuration
        run: terraform apply -auto-approve

Essentially, the workflow carries out several key tasks to automate the deployment process:

First, it checks out the latest version of your code from the repository, ensuring that the most up-to-date changes are available.
Next, it configures the necessary AWS credentials to securely authenticate with your cloud environment, allowing the workflow to interact with AWS services.
Finally, it initialises Terraform, setting up the working environment, and applies the EKS cluster configuration, provisioning the infrastructure based on the defined Terraform scripts. This ensures the EKS cluster is deployed correctly, ready to manage Kubernetes resources efficiently.

Lastly, we need to configure our control machine, which has kubectl installed, to communicate with the EKS cluster. This involves updating the machine with the appropriate cluster name and AWS region. By doing so, kubectl can interact with the cluster, allowing us to manage resources such as pods, services, and deployments. Without this step, the control machine won’t be able to send commands to the cluster, which is crucial for managing and monitoring our Kubernetes environment effectively.

aws eks update-kubeconfig --region region-code --name my-cluster

With our infrastructure now fully provisioned and operational, the next step is to set up our Flask application and deploy it onto the newly created environment. This involves configuring the necessary application dependencies, setting up environment variables, and ensuring the app is containerised for deployment. Once everything is configured, we can seamlessly deploy the Flask app to our infrastructure, leveraging the scalability and reliability of the EKS cluster to manage the application. This deployment marks the transition from infrastructure setup to delivering a functional, production-ready application.

Flask App and Docker Setup:

This guide will walk you through setting up a basic Flask application, configuring the necessary files, and preparing it for testing. We’ll also cover how to set up a Python virtual environment to manage dependencies and keep your project isolated.

Step 1: Set Up the Flask Application

Create a new directory for your application

Start by creating a dedicated directory for your Flask project. This keeps everything organised:

mkdir regtech-docker-app
cd regtech-docker-app

(Optional) Set Up a Python Virtual Environment

It’s highly recommended to use a Python virtual environment to isolate your project dependencies. This ensures that packages installed for this project won’t affect other Python projects:

python3 -m venv venv
source venv/bin/activate

Once the virtual environment is active, you’ll see (venv) before your terminal prompt.

Install Flask

Next, you need to install Flask. Before doing so, create your main application file:

touch app.py

Then install Flask using pip:

pip install Flask

Create the Flask Application

Now, let's populate the app.py file with a basic Flask app. This app includes CSRF protection and loads configurations from a separate config.py file:

from flask import Flask
from flask_wtf.csrf import CSRFProtect
from config import Config

app = Flask(__name__)
app.config.from_object(Config)

csrf = CSRFProtect(app)

@app.route('/')
def hello():
    return "Hello, Welcome to Zip Reg Tech!"

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=5000)

This sets up a simple route (/) that returns a greeting message when accessed.

Create a requirements.txt File

To ensure all dependencies are properly documented, generate a requirements.txt file. This file will list all installed packages, including Flask:

pip freeze > requirements.txt

Now add the following dependencies:

blinker==1.8.2
click==8.1.7
Flask==3.0.3
itsdangerous==2.2.0
Jinja2==3.1.4
MarkupSafe==2.1.5
Werkzeug==3.0.4
pytest
requests
Flask-WTF

This step ensures that anyone who clones your repository can easily install all the required dependencies using pip install -r requirements.txt.

Create the Configuration File

You’ll need a configuration file to manage environment-specific settings like the app’s secret key. Let’s create a config.py file:

touch config.py

Populate it with the following code:

# config.py
import os
import secrets

class Config:
    SECRET_KEY = os.getenv('SECRET_KEY', secrets.token_urlsafe(32))

This file uses the environment variable SECRET_KEY if available; otherwise, it generates a secure token on the fly. You can set the environment variable in your Docker container or deployment environment.

Write Unit Tests for the Flask App

To ensure the application works as expected, let’s add some unit tests. Create a test_app.py file:

touch test_app.py

Inside the file, add the following code to test the root endpoint (/):

from app import app 

def test_hello():
    with app.test_client() as client:
        response = client.get('/')
        assert response.data == b"Hello, Welcome to Zip Reg Tech!"
        assert response.status_code == 200

This test simulates an HTTP request to the Flask app and verifies that the correct response is returned.

Create Integration Tests

To simulate real-world conditions, we’ll write an integration test that starts the Flask app, makes a request to it, and verifies the response. Create a test_integration.py file:

touch test_integration.py

Now add the following code for integration testing:

import requests
from app import app
import multiprocessing
import time

# Run Flask app in a separate process for integration testing
def run_app():
    app.run(host="0.0.0.0", port=5000)

def test_integration():
    # Start the Flask app in a background process
    p = multiprocessing.Process(target=run_app)
    p.start()

    # Give the app a moment to start up
    time.sleep(2)

    # Make an HTTP request to the running Flask app
    response = requests.get('http://localhost:5000/')

    # Check that the response is as expected
    assert response.status_code == 200
    assert response.text == "Hello, Welcome to Zip Reg Tech!"

    # Terminate the Flask app process
    p.terminate()

This integration test launches the Flask app in a separate process and makes an HTTP request to the / route. It then verifies the status code and response body before terminating the app.

By following these steps, you have a fully functional Flask app with unit and integration tests. The next step will be setting up Docker for containerisation, ensuring your app is ready for deployment in any environment.

Step 2: Set Up Docker and Push the Image to Amazon Elastic Container Registry (ECR)

In this section, we'll create a Docker image for our Flask app, configure the necessary Docker files, and then push the image to Amazon Elastic Container Registry (ECR) for deployment.

Create the Dockerfile

A Dockerfile is a script that defines how your Docker image is built. In the same directory as your Flask application, create a new file named Dockerfile and add the following content:

# Use an official Python runtime as a base image
FROM python:3.9-slim

# Set the working directory in the container
WORKDIR /app

# Copy the current directory contents into the container at /app
COPY . /app

# Install the dependencies
RUN pip install --no-cache-dir -r requirements.txt

# Make port 5000 available to the world outside this container
EXPOSE 5000

# Define environment variable to prevent Python from buffering stdout/stderr
ENV PYTHONUNBUFFERED=1

# Run the application
CMD ["python", "app.py"]

Base Image: We are using python:3.9-slim as our base image. This is a lightweight version of Python that reduces the size of the final Docker image.

Working Directory: The WORKDIR /app command sets the working directory for subsequent instructions.

Copying Files: The COPY . /app command copies the current directory’s contents (including your Flask app and config files) into the container.

Install Dependencies: The RUN pip install --no-cache-dir -r requirements.txt installs all the necessary dependencies as specified in the requirements.txt file.

Expose Port: The EXPOSE 5000 command allows the Flask app to communicate over port 5000 from inside the container.

Set Environment Variable: ENV PYTHONUNBUFFERED=1 ensures that Flask logs are outputted in real time rather than being buffered.

Run the App: The CMD directive specifies that app.py will be executed when the container starts.

Create the .dockerignore File

To keep your Docker image clean and avoid copying unnecessary files into the container, create a .dockerignore file in your project’s root directory. This file works similarly to .gitignore, telling Docker which files to exclude from the build context.

Create the .dockerignore file with the following content:

venv/
__pycache__/
*.pyc
*.pyo
*.pyd
.Python

venv/: Prevents your local Python virtual environment from being included in the Docker image.

__pycache__/: Excludes Python cache files generated during development.

File extensions: Ignoring Python compiled files like .pyc, .pyo, and .pyd.

.Python: Excludes the Python build files.

Build and Test the Docker Image Locally

To make sure everything works, you can build and run your Docker image locally. In your project directory, run the following commands:

docker build -t regtech-docker-app .
docker run -p 5100:5000 regtech-docker-app

docker build: This command creates the Docker image, tagging it as regtech-docker-app.

docker run: Runs the Docker container on port 5100, forwarding traffic from your machine to the container.

At this point, your Flask app should be running at http://localhost:5100.

Create an Elastic Container Registry (ECR) Repository

Log in to the AWS Console:

Open the AWS Management Console and search for ECR (Elastic Container Registry).

Create a New Repository:

On the ECR landing page, click the Create repository button.
Give your repository a name (e.g., regtech-docker-app), then click Create.

Confirm Repository Creation: Once the repository is created, you’ll be redirected to a confirmation screen showing the repository details.

You’ll see the repository listed, confirming it was successfully created:

Setup Kubernetes Deployment and Service:

In this section, we will configure a Kubernetes Deployment and a NodePort Service for our Dockerized Flask application. This setup will ensure that your Flask app runs smoothly and is accessible from outside the Kubernetes cluster.

Create a Kubernetes Deployment and Service

A Deployment will handle managing the Flask application, making sure the desired number of Pods are always running. The NodePort Service will expose the Flask application on a specific port that can be accessed externally from outside the cluster. It maps a port on the Kubernetes nodes to the port where the Flask app is running inside the Pods.

Create a file named deploy.yml with the following content:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: regtech-app-deployment
  labels:
    app: regtech-app
spec:
  replicas: 1
  selector:  
    matchLabels:
      app: regtech-app
  template:
    metadata:
      labels:
        app: regtech-app
    spec:
      containers:
      - name: regtech-app 
        image: REPOSITORY_TAG
        ports: 
        - containerPort: 5000 

---
apiVersion: v1 
kind: Service
metadata: 
  name: regtech-app-service
spec:
  type: NodePort
  selector:
    app: regtech-app
  ports:
    - protocol: TCP
      port: 5000
      targetPort: 5000
      nodePort: 30030

Explanation of the Configuration

Deployment:

Deploys a single replica of the Flask application.

Each Pod runs a container from your Docker image specified in REPOSITORY_TAG.

The container exposes port 5000, which is the port the Flask application listens on.

Service:

Creates a NodePort Service that makes the Flask application accessible externally.

Maps port 5000 in the cluster to port 5000 on the Flask application container.

Exposes the application on NodePort 30030 (or a random port in the range 30000-32767 if not specified).

Deploying the Flask Application

The deployment of the Flask application is managed through a CI/CD pipeline, which automates the process of building the Docker image and deploying it to the Kubernetes cluster.

Required GitHub Secrets

You'll need to configure the following secrets in your GitHub repository:

AWS_ACCESS_KEY_ID: Your AWS access key.

AWS_SECRET_ACCESS_KEY: Your AWS secret access key.

AWS_REGION: The AWS Region where your cluster resides.

EKS_CLUSTER_NAME: Your EKS Cluster name.

NEW_GITHUB_TOKEN: Your github access token

ORGANIZATION_KEY: Your Sonarcloud organisation key

PROJECT_KEY: Your Sonarcloud project key

SONAR_TOKEN: Your SonarQube token.

SONAR_HOST_URL: The URL of your SonarQube server (for SonarCloud, this can be https://sonarcloud.io).

SNYK_TOKEN: The API token from Snyk.

Adding Secrets to GitHub:

Navigate to your GitHub repository.

Go to Settings > Secrets > Actions.

Add the required secrets (SONAR_TOKEN, SONAR_HOST_URL, and SNYK_TOKEN).

CI/CD Pipeline Process (regtech-app.yaml)

Here’s a step-by-step breakdown of the CI/CD workflow:

Linting and Static Analysis (SonarCloud):

Ensures code quality and identifies potential issues.

Unit and Integration Tests:

Validates the functionality of your code before deployment.

Security Scan (Snyk):

Detects vulnerabilities in your code dependencies.

Build Docker Image:

Packages the Flask application into a Docker image.

Push to Amazon ECR:

Publishes the Docker image to Amazon Elastic Container Registry (ECR).

Deploy to EKS:

Deploys the Docker image to your EKS cluster using kubectl.

Rollback:

Automatically rolls back the deployment if it fails.

Here’s the workflow configuration for regtech-app.yaml:

name: Deploy Flask App to EKS  

on:
  push:
    branches:
      - main
  pull_request:
    branches:
      - main

env:
  AWS_REGION: ${{ secrets.AWS_REGION }}
  EKS_CLUSTER_NAME: ${{ secrets.EKS_CLUSTER_NAME }}

jobs:
  # Step 1: Source Code Testing (Linting, Static Analysis, Unit Tests, Snyk Scan)
  Lint-and-Static-Analysis:
    name: Linting and Static Analysis (SonarQube) 
    runs-on: ubuntu-latest 
    steps:
      - name: Checkout repository 
        uses: actions/checkout@v3
        with:
          fetch-depth: 0

      - name: SonarCloud Scan
        uses: sonarsource/sonarcloud-github-action@master  
        env:
          GITHUB_TOKEN: ${{ secrets.NEW_GITHUB_TOKEN }} 
          #ORGANIZATION_KEY: ${{ secrets.ORGANIZATION_KEY }} 
          SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }} 
        with:
          args: >
            -Dsonar.organization=${{ secrets.ORGANIZATION_KEY }}
            -Dsonar.projectKey=${{ secrets.PROJECT_KEY }} 
            -Dsonar.exclusions=venv/**
            -Dsonar.c.file.suffixes=-
            -Dsonar.cpp.file.suffixes=-
            -Dsonar.objc.file.suffixes=-

      - name: Check SonarCloud Quality Gate 
        run: |
          curl -u ${{ secrets.SONAR_TOKEN }} "https://sonarcloud.io/api/qualitygates/project_status?projectKey=${{ secrets.PROJECT_KEY }}" | grep '"status":"OK"' || exit 1

  UnitAndIntegrationTests:
    name: Unit and Integration Tests on Source Code
    runs-on: ubuntu-latest
    needs: Lint-and-Static-Analysis
    steps:
      - name: Checkout repository 
        uses: actions/checkout@v3

      - name: Set up Python 
        uses: actions/setup-python@v4
        with:
          python-version: '3'

      - name: Check Python version
        run: python --version

      - name: Verify venv creation
        run: ls -la venv/bin/

      - name: Clean up and recreate virtual environment
        run: |
          rm -rf venv
          python3 -m venv venv

      - name: Create virtual environment
        run: |
          python3 -m venv venv

      - name: Check Python executable path
        run: |
          which python3

      - name: List directory contents
        run: |
          cd /home/runner/work/regtech_accessment_cicd
          ls -la

      - name: Install dependencies 
        run: |
          cd /home/runner/work/regtech_accessment_cicd/regtech_accessment_cicd
          source venv/bin/activate
          ls -la venv venv
          pip3 install -r requirements.txt

      - name: Run Unit Tests 
        run: |
          cd /home/runner/work/regtech_accessment_cicd/regtech_accessment_cicd
          source venv/bin/activate 
          pytest test_app.py  

      - name: Run Integration tests
        run: |
          cd /home/runner/work/regtech_accessment_cicd/regtech_accessment_cicd
          source venv/bin/activate
          pytest test_integration.py   

  SNYK-SCAN:
    name: Dependency Scanning (Snyk)
    runs-on: ubuntu-latest 
    needs: UnitAndIntegrationTests
    steps:
      - name: Checkout repository 
        uses: actions/checkout@master

      - name: Set up Python 
        uses: actions/setup-python@v4
        with:
          python-version: '3'

      - name: Check Python version
        run: python --version

      - name: Clean up and recreate virtual environment
        run: |
          rm -rf venv
          python3 -m venv venv

      - name: Create virtual environment
        run: |
          python3 -m venv venv

      - name: Check Python executable path
        run: |
          which python3

      - name: List directory contents
        run: |
          cd /home/runner/work/regtech_accessment_cicd
          ls -la

      - name: Install dependencies 
        run: |
          cd /home/runner/work/regtech_accessment_cicd/regtech_accessment_cicd
          source venv/bin/activate
          ls -la venv venv
          pip3 install -r requirements.txt

      - name: Set up Snyk
        uses: snyk/actions/python-3.10@master
        env:
          SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}
        with:
          args: --severity-threshold=high 

# Step 2: Build Docker Image 
  BuildImage-and-Publish-To-ECR:
    name: Build and Push Docker Image
    runs-on: ubuntu-latest 
    needs: SNYK-SCAN 
    steps:
    - name: Checkout 
      uses: actions/checkout@v4 

    - name: Login to ECR 
      uses: docker/login-action@v3 
      with:
        registry: 611512058022.dkr.ecr.us-east-1.amazonaws.com
        username: ${{ secrets.AWS_ACCESS_KEY_ID }}  
        password: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        region: ${{ secrets.AWS_REGION }}

    - name: Build Image 
      run: | 
        docker build -t regtech-app .
        docker tag regtech-app:latest 611512058022.dkr.ecr.us-east-1.amazonaws.com/regtech-app:${GITHUB_RUN_NUMBER}
        docker push 611512058022.dkr.ecr.us-east-1.amazonaws.com/regtech-app:${GITHUB_RUN_NUMBER}

# Step 3: Docker Image Testing (Integration Tests Inside Container)
  Integration-Tests:
    name: Integration Tests on Docker Image
    runs-on: ubuntu-latest 
    needs: BuildImage-and-Publish-To-ECR
    steps:

      - name: Login to ECR 
        uses: docker/login-action@v3 
        with:
          registry: 611512058022.dkr.ecr.us-east-1.amazonaws.com
          username: ${{ secrets.AWS_ACCESS_KEY_ID }}  
          password: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          region: ${{ secrets.AWS_REGION }}

      - name: Pull Docker Image from ECR 
        run: |
          docker pull 611512058022.dkr.ecr.us-east-1.amazonaws.com/regtech-app:${GITHUB_RUN_NUMBER}

      - name: Run Integration Tests inside Docker Container 
        run: |
          docker run --rm -v $(pwd):/results 611512058022.dkr.ecr.us-east-1.amazonaws.com/regtech-app:${GITHUB_RUN_NUMBER} pytest --junitxml=/results/integration-test-results.xml

      - name: List Files in Current Directory
        run: |
          ls -l 

      - name: Upload Integration Test Results 
        uses: actions/upload-artifact@v3 
        with: 
          name: integration-test-results
          path: integration-test-results.xml
          if-no-files-found: warn

# Step 4: Install Kubectl
  Install-kubectl:
    name: Install Kubectl on The Github Actions Runner
    runs-on: ubuntu-latest
    needs: Integration-Tests
    steps:
    - name: Checkout 
      run: |
        curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/OS_DISTRIBUTION/amd64/kubectl
        chmod +x ./kubectl
        sudo mv ./kubectl /usr/local/bin/kubectl

# Step 5: Deploy To EKS
  Deploy-To-Cluster:
    runs-on: ubuntu-latest
    needs: Install-kubectl 
    steps:
    - name: Checkout
      uses: actions/checkout@v4

    - name: Configure credentials
      uses: aws-actions/configure-aws-credentials@v4
      with:
        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        aws-region: ${{ secrets.AWS_REGION }}

    - name: Download KubeConfig File 
      env:
        KUBECONFIG: ${{ runner.temp }}/kubeconfig

      run: | 
        aws eks update-kubeconfig --region ${{ secrets.AWS_REGION }} --name ${{ secrets.EKS_CLUSTER_NAME }} --kubeconfig $KUBECONFIG
        echo "KUBECONFIG=$KUBECONFIG" >> $GITHUB_ENV 
        echo $KUBECONFIG 

    - name: Deploy to EKS 
      run: |
        sed -i "s|image: REPOSITORY_TAG|image: 611512058022.dkr.ecr.us-east-1.amazonaws.com/regtech-app:${GITHUB_RUN_NUMBER}|g" ./deploy.yml
        kubectl apply -f ./deploy.yml

    - name: Check Deployment Status 
      id: check-status 
      run: |
        kubectl rollout status deployment.apps/regtech-app-deployment || exit 

    - name: Rollback Deployment 
      if: failure()
      run: | 
        echo "Deployment failed. Rolling back..."
        kubectl rollout undo deployment.apps/regtech-app-deployment

To determine which node your deployment is running on, follow these steps:

First, get the details of your running pods, including the node they are scheduled on, by running the following command:

kubectl get po -o wide

This command will provide detailed information about the pods, including their IP addresses, node assignments, and the container status.

Next, identify the public IP address of the node where your pod is running. To do this, list all the nodes in your cluster:

kubectl get nodes -o wide

You'll see a list of the nodes in your Kubernetes cluster, along with their corresponding external IPs.

Once you've identified the public IP of the node, copy it. You'll combine this public IP with the NodePort assigned to your Flask application to access it externally.

Finally, construct the URL by combining the public IP of the node with the NodePort (e.g., 30030 from the example). Enter this into your browser like so:

http://<PUBLIC_IP>:<NODE_PORT>

Hit enter, and if everything is set up correctly, your Flask app will be live and accessible from the browser.

Monitoring with Prometheus and Grafana

After deploying your application, it's crucial to set up monitoring to track its performance, health, and any potential issues in real-time. In this guide, we’ll walk through setting up Prometheus for monitoring, leveraging Grafana for visualization later.

Prometheus

Prometheus is a powerful monitoring and alerting toolkit that collects and stores metrics from both your application and the Kubernetes cluster itself.

We will use Helm, the Kubernetes package manager, to deploy Prometheus. Follow the steps below:

Create a namespace for Prometheus:

kubectl create namespace prometheus

Add the Prometheus Helm chart repository:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

Deploy Prometheus using Helm:

helm upgrade -i prometheus prometheus-community/prometheus --namespace prometheus --set alertmanager.persistence.storageClass="gp2" --set server.persistentVolume.storageClass="gp2"

This command installs or upgrades the Prometheus instance in the prometheus namespace, setting the storage class for both Alertmanager and Prometheus Server's persistent volumes to gp2 (Amazon EBS General Purpose volumes).

Verify the deployment:

kubectl get pods -n prometheus

At this point, some of your pods might be in a Pending state due to missing Amazon Elastic Block Store (EBS) volumes. This is because the Amazon EBS CSI (Container Storage Interface) driver is required to manage EBS volumes for Kubernetes. Let’s set up the driver.

Amazon EBS CSI Driver Setup

Create an IAM OIDC identity provider for your cluster:

eksctl utils associate-iam-oidc-provider --cluster $cluster_name --approve

Ensure that eksctl is installed on your control machine for this step.

Create an IAM role for the Amazon EBS CSI plugin:

eksctl create iamserviceaccount \
    --name ebs-csi-controller-sa \
    --namespace kube-system \
    --cluster my-cluster \
    --role-name AmazonEKS_EBS_CSI_DriverRole \
    --role-only \
    --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \
    --approve

Add the Amazon EBS CSI driver as an EKS add-on:

To check the required platform version:

aws eks describe-addon-versions --addon-name aws-ebs-csi-driver

To install the EBS CSI driver add-on using eksctl:

eksctl create addon --name aws-ebs-csi-driver --cluster my-cluster --service-account-role-arn arn:aws:iam::111122223333:role/AmazonEKS_EBS_CSI_DriverRole --force

Replace my-cluster with your actual cluster name and with your AWS account number.

Verify the installation of the EBS CSI driver:

eksctl get addon --name aws-ebs-csi-driver --cluster my-cluster

Update the EBS CSI driver if needed.

eksctl update addon --name aws-ebs-csi-driver --version v1.11.4-eksbuild.1 --cluster my-cluster \
  --service-account-role-arn arn:aws:iam::111122223333:role/AmazonEKS_EBS_CSI_DriverRole --force

Re-check the pod status:

kubectl get pods -n prometheus

Your Prometheus pods should now be in the Running state.

Label the Prometheus server pod:

To connect the Prometheus server pod with a service:

kubectl label pod <pod-name> app=prometheus

Expose Prometheus via NodePort

Prometheus has a built-in web interface for accessing metrics. To access Prometheus externally, we will expose it via a NodePort service.

Create a Prometheus service YAML file:

touch prometheus-service.yml

Define the NodePort service configuration:

apiVersion: v1
kind: Service
metadata:
  name: prometheus-nodeport
  namespace: prometheus
spec:
  selector:
    app: prometheus
  ports:
    - name: web
      port: 9090
      targetPort: 9090
      protocol: TCP
      nodePort: 30000  # You can choose any available port on your nodes
  type: NodePort

Apply the service configuration:

kubectl apply -f prometheus-service.yml

Access Prometheus:

Now you can access the Prometheus web UI by navigating to:

http://<NODE_PUBLIC_IP>:30000

Replace with the public IP address of any node in your cluster.

By following these steps, we now have a fully functioning Prometheus setup monitoring both our Kubernetes cluster and application metrics. Next, we will integrate Grafana to create rich, visual dashboards for real-time analysis of the collected metrics.

Grafana: Visualizing and Monitoring with Dashboard

A visualisation tool that integrates with Prometheus to create dashboards. It also allows you to set up alerts for events like high CPU usage, memory spikes, or pod failures in your EKS cluster.

Here’s how to deploy Grafana on your EKS cluster using Helm:

Add the Grafana Helm repository:

First, you'll need to add the Grafana Helm chart repository to Helm:

helm repo add grafana https://grafana.github.io/helm-charts

Create a Grafana namespace:

Set up a dedicated namespace for Grafana:

kubectl create namespace grafana

Create a Grafana YAML configuration file:

Grafana requires a configuration file to connect it to Prometheus. Create a file called grafana.yml with the following content:

  datasources:
  datasources.yaml:
    apiVersion: 1
    datasources:
    - name: Prometheus
      type: prometheus
      url: http://prometheus-server.prometheus.svc.cluster.local
      access: proxy
      isDefault: true

Ensure the Prometheus URL is correctly specified to point to the Prometheus service in your cluster.

Deploy Grafana using Helm:

Use Helm to deploy Grafana into your Kubernetes cluster. You can set up persistent storage for Grafana dashboards, configure the admin password, and provide the path to your grafana.yml file:

helm install grafana grafana/grafana \
    --namespace grafana \
    --set persistence.storageClassName="gp2" \
    --set persistence.enabled=true \
    --set adminPassword='EKS!sAWSome' \
    --values /home/ec2-user/grafana.yaml \
    --set service.type=NodePort

Replace the adminPassword with a strong password of your choice, and ensure the path to your grafana.yml file is correct.

Verify the deployment

Check if the Grafana pods are running successfully in the Grafana namespace:

kubectl get pods -n grafana

Access Grafana:

Grafana will be exposed using a NodePort. You can access the Grafana UI using the public IP of any node in your cluster and the specified NodePort. For example:

nodeport:30281

Replace with the actual IP address of one of your nodes. The default NodePort is set to 30281, but it can vary based on your configuration.

Once you’ve accessed the Grafana web UI, log in using the credentials:

Username: admin

Password: The admin password you set during the Helm installation.

Create a new dashboard:

After logging in, you can create your first dashboard. To make things easier, you can import a pre-built dashboard tailored for Kubernetes monitoring.

Click "Create" → "Import" on the Grafana console.

Import a pre-built Kubernetes dashboard:

On the "Find and Import Dashboards for Common Applications" section, input the dashboard ID 17119 and click "Load".

Configure the data source:

Select Prometheus as the data source for the dashboard.

Click "Import" to load the dashboard.

Configure the data source:

Select Prometheus as the data source for the dashboard.

Click "Import" to load the dashboard.

View your dashboard:

After importing the dashboard, you can now visualize the performance of your Kubernetes cluster. The dashboard will display real-time metrics such as CPU and memory usage, pod status, and more.

By following these steps, we have a Grafana instance running on your EKS cluster, integrated with Prometheus to collect and visualise metrics.

Conclusion

By integrating Terraform and GitHub Actions, we fully automate the setup and management of our AWS infrastructure and Kubernetes-based application deployments. This setup ensures:

Scalability: You can easily scale your infrastructure to meet demand.

Efficiency: Automating deployments speeds up the process, reduces errors, and makes your development workflow smoother.

Security: Following security best practices protects your application and data.

Happy deploying!

gggg

Oloruntobi Olurombi — Sat, 14 Sep 2024 10:05:42 +0000

ggggg

fff

Oloruntobi Olurombi — Sat, 14 Sep 2024 09:56:13 +0000

In the world of cloud-native applications, automation is crucial to streamline deployments and manage infrastructure efficiently. In this article, we’ll explore how to automate the setup of an EKS cluster and deploy a Flask application using Terraform and GitHub Actions. We’ll also touch on security best practices, monitoring, and how to ensure a robust CI/CD pipeline.

Overview
Prerequisites
Infrastructure Automation: EKS Cluster Setup
Application Deployment: Flask App on EKS
Monitoring with Prometheus and Grafana
Security Best Practices
Conclusion

Overview

Why Terraform?

Why GitHub Actions?

GitHub Actions provides a powerful way to integrate CI/CD, testing, static analysis, and security checks into the code deployment process.

Prerequisites

Before diving into the automation, here are the prerequisites you’ll need to get started:

AWS Account: Create an AWS account if you don’t have one.
IAM Access Keys: Set up access keys with permissions for managing EKS, EC2, and S3.
S3 Bucket: Create an S3 bucket to store your Terraform state files securely.
AWS CLI: Install and configure the AWS CLI.
Terraform: Make sure Terraform is installed on your local machine or use GitHub Actions for automation.
GitHub Secrets: Add AWS credentials (access keys, secret keys) and other sensitive data as GitHub secrets to avoid hardcoding them.
Synk: Create a Synk account and get your Token.
SonarCloud: Create a SonarCloud account and get your Token, Organisation key and Project key.

Infrastructure Automation: EKS Cluster Setup

Terraform Setup

Let’s start by provisioning the necessary infrastructure. Below is the detailed explanation of the key resources defined in the Terraform files.

EKS Cluster and Node Group (main.tf):

touch main.tf

terraform {
  backend "s3" {
    bucket = "regtech-iac"
    key = "terraform.tfstate"
    region = "us-east-1"
    encrypt = true 
  }
}

# Provides an EKS Cluster
resource "aws_eks_cluster" "eks_cluster" {
  name     = var.cluster_name
  role_arn = aws_iam_role.eks_cluster_role.arn

  version = "1.28"

  vpc_config {
    subnet_ids = [aws_subnet.public_subnet_1.id, aws_subnet.public_subnet_2.id, aws_subnet.public_subnet_3.id]
  }

  encryption_config {
    provider {
      key_arn = aws_kms_key.eks_encryption_key.arn
    }
    resources = ["secrets"]
  }

  # Ensure that IAM Role permissions are created before and deleted after EKS Cluster handling.
  # Otherwise, EKS will not be able to properly delete EKS managed EC2 infrastructure such as Security Groups.
  depends_on = [
    aws_iam_role_policy_attachment.eks_cluster_policy_attachment,
    aws_iam_role_policy_attachment.eks_service_policy_attachment,
  ]
}

# Provides an EKS Node Group 

resource "aws_eks_node_group" "eks_node_group" {
  cluster_name    = aws_eks_cluster.eks_cluster.name
  node_group_name = var.node_group_name
  node_role_arn   = aws_iam_role.eks_node_group_role.arn
  subnet_ids      = [aws_subnet.public_subnet_1.id, aws_subnet.public_subnet_2.id, aws_subnet.public_subnet_3.id]

  scaling_config {
    desired_size = 2
    max_size     = 2
    min_size     = 2
  }
  update_config {
    max_unavailable = 1
  }

  # Ensure that IAM Role permissions are created before and deleted after EKS Node Group handling.
  # Otherwise, EKS will not be able to properly delete EC2 Instances and Elastic Network Interfaces.
  depends_on = [
    aws_iam_role_policy_attachment.eks_worker_node_policy_attachment,
    aws_iam_role_policy_attachment.eks_cni_policy_attachment,
    aws_iam_role_policy_attachment.ec2_container_registry_readonly,
  ]
}

# Extra resources 
resource "aws_ebs_volume" "volume_regtech"{
    availability_zone = var.az_a
    size = 40
    encrypted = true
    type = "gp2"
    kms_key_id        = aws_kms_key.ebs_encryption_key.arn
}

resource "aws_s3_bucket" "regtech_iac" {
  bucket = var.bucket_name
}

resource "aws_s3_bucket_server_side_encryption_configuration" "regtech_iac_encrypt_config" {
    bucket = aws_s3_bucket.regtech_iac.bucket
    rule {
        apply_server_side_encryption_by_default {
        kms_master_key_id = aws_kms_key.s3_encryption_key.arn  
        sse_algorithm = "aws:kms"
        }
    }
}


# OutPut Resources
output "endpoint" {
  value = aws_eks_cluster.eks_cluster.endpoint
}

output "eks_cluster_name" {
    value = aws_eks_cluster.eks_cluster.name
}

Networking (vpc.tf):

Defines a VPC, public subnets for the EKS cluster, and private subnets for other resources, ensuring flexibility in network architecture.

vpc.tf

# Provides a VPC resource
resource "aws_vpc" "main" {
  cidr_block       = var.vpc_cidr_block
  instance_tenancy = "default"

  tags = {
    Name = var.tags_vpc
  }
}


# Provides an VPC Public subnet resource
resource "aws_subnet" "public_subnet_1" {
  vpc_id     = aws_vpc.main.id
  cidr_block = var.p_s_1_cidr_block
  availability_zone = var.az_a
  map_public_ip_on_launch = true

  tags = {
    Name = var.tags_public_subnet_1
  }
}

resource "aws_subnet" "public_subnet_2" {
  vpc_id     = aws_vpc.main.id
  cidr_block = var.p_s_2_cidr_block
  availability_zone = var.az_b
  map_public_ip_on_launch = true

  tags = {
    Name = var.tags_public_subnet_2
  }
}

resource "aws_subnet" "public_subnet_3" {
  vpc_id     = aws_vpc.main.id
  cidr_block = var.p_s_3_cidr_block
  availability_zone = var.az_c
  map_public_ip_on_launch = true

  tags = {
    Name = var.tags_public_subnet_3
  }
}

# Provides an VPC Private subnet resource
resource "aws_subnet" "private_subnet_1" {
  vpc_id     = aws_vpc.main.id
  cidr_block = var.private_s_1_cidr_block
  availability_zone = var.az_private_a
  map_public_ip_on_launch = false 

  tags = {
    Name = var.tags_private_subnet_1
  }
}

resource "aws_subnet" "private_subnet_2" {
  vpc_id     = aws_vpc.main.id
  cidr_block = var.private_s_2_cidr_block
  availability_zone = var.az_private_b
  map_public_ip_on_launch = false 

  tags = {
    Name = var.tags_private_subnet_2
  }
}

resource "aws_subnet" "private_subnet_3" {
  vpc_id     = aws_vpc.main.id
  cidr_block = var.private_s_3_cidr_block
  availability_zone = var.az_private_c
  map_public_ip_on_launch = false 

  tags = {
    Name = var.tags_private_subnet_3
  }
}

IAM Roles (iam.tf):

IAM roles and policies for the EKS cluster, node groups, and autoscaler. Includes roles for security services like CloudWatch and CloudTrail, ensuring robust monitoring.

iam.tf

# Declare the aws_caller_identity data source
data "aws_caller_identity" "current" {}


# IAM Role for EKS Cluster Plane 

resource "aws_iam_role" "eks_cluster_role" {
    name = var.eks_cluster_role_name

    assume_role_policy = jsonencode({
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Principal": {
                    "Service": "eks.amazonaws.com"
                },
                "Action": "sts:AssumeRole"
            }
        ]
    })
}

resource "aws_iam_role_policy_attachment" "eks_cluster_policy_attachment" {
    policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
    role = aws_iam_role.eks_cluster_role.name 
}

resource "aws_iam_role_policy_attachment" "eks_service_policy_attachment" {
  role       = aws_iam_role.eks_cluster_role.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSServicePolicy"
}

# IAM Role for Worker node

resource "aws_iam_role" "eks_node_group_role" {
    name = var.eks_node_group_role_name

    assume_role_policy = jsonencode({
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Principal": {
                    "Service": "ec2.amazonaws.com"
                },
                "Action": "sts:AssumeRole"
            }
        ]
    })
}

resource "aws_iam_role_policy_attachment" "eks_worker_node_policy_attachment" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
  role       = aws_iam_role.eks_node_group_role.name
}

resource "aws_iam_role_policy_attachment" "eks_cni_policy_attachment" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
  role       = aws_iam_role.eks_node_group_role.name
}

resource "aws_iam_role_policy_attachment" "ec2_container_registry_readonly" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
  role       = aws_iam_role.eks_node_group_role.name
}

resource "aws_iam_instance_profile" "eks_node_instance_profile" {
    name = var.eks_node_group_profile
    role = aws_iam_role.eks_node_group_role.name
}


# Policy For volume creation and attachment

resource "aws_iam_role_policy" "eks_node_group_volume_policy" {
  name   = var.eks_node_group_volume_policy_name
  role   = aws_iam_role.eks_node_group_role.name
  policy = jsonencode({
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": [
          "ec2:CreateTags",
          "ec2:DescribeTags",
          "ec2:DescribeVolumes",
          "ec2:DescribeVolumeStatus",
          "ec2:CreateVolume",
          "ec2:AttachVolume"
        ],
        "Resource": "arn:aws:ec2:${var.region}:${data.aws_caller_identity.current.account_id}:volume/*"
      }
    ]
  })
}


# IAM Role for CloudWatch 

resource "aws_iam_role" "cloudwatch_role" {
  name = "cloudwatch_role_log"

  assume_role_policy = jsonencode({
    "Version": "2012-10-17",
    "Statement": [
      {
        "Action": "sts:AssumeRole",
        "Principal": {
          "Service": "cloudwatch.amazonaws.com"
        },
        "Effect": "Allow",
        "Sid": ""
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "cloudwatch_policy_attachment" {
  role       = aws_iam_role.cloudwatch_role.name
  policy_arn = "arn:aws:iam::aws:policy/CloudWatchLogsFullAccess"
}

# IAM Role for CloudTrail

resource "aws_iam_role" "cloudtrail_role" {
  name = "cloudtrail_role_log"

  assume_role_policy = jsonencode({
    "Version": "2012-10-17",
    "Statement": [
      {
        "Action": "sts:AssumeRole",
        "Principal": {
          "Service": "cloudtrail.amazonaws.com"
        },
        "Effect": "Allow",
        "Sid": ""
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "cloudtrail_policy_attachment" {
  role       = aws_iam_role.cloudtrail_role.name
  policy_arn = "arn:aws:iam::aws:policy/AWSCloudTrail_FullAccess"
}


# KMS Key Policy for Encryption

resource "aws_kms_key" "ebs_encryption_key" {
  description = "KMS key for EBS volume encryption"
}

resource "aws_kms_key" "s3_encryption_key" {
  description = "KMS key for S3 bucket encryption"
}

resource "aws_kms_key" "eks_encryption_key" {
  description = "KMS key for EKS secret encryption"
}


resource "aws_s3_bucket_policy" "regtech_iac_policy" {
  bucket = aws_s3_bucket.regtech_iac.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          Service = "cloudtrail.amazonaws.com"
        }
        Action = "s3:GetBucketAcl"
        Resource = "arn:aws:s3:::${aws_s3_bucket.regtech_iac.bucket}"
      },
      {
        Effect = "Allow"
        Principal = {
          Service = "cloudtrail.amazonaws.com"
        }
        Action = "s3:PutObject"
        Resource = "arn:aws:s3:::${aws_s3_bucket.regtech_iac.bucket}/AWSLogs/${data.aws_caller_identity.current.account_id}/*"
        Condition = {
          StringEquals = {
            "s3:x-amz-acl" = "bucket-owner-full-control"
          }
        }
      }
    ]
  })
}

CloudWatch and Monitoring (cloudwatch.tf):

This provisions CloudWatch log groups, an SNS topic for alerts, and a CloudWatch alarm to monitor CPU utilisation. CloudTrail logs are configured to monitor S3 and management events.

touch cloudwatch.tf

resource "aws_cloudwatch_log_group" "eks_log_group" {
  name              = "/aws/eks/cluster-logs-regtech"
  retention_in_days = 30
}

resource "aws_cloudtrail" "security_trail" {
  name                          = "security-trail-log"
  s3_bucket_name                = aws_s3_bucket.regtech_iac.bucket
  include_global_service_events = true
  is_multi_region_trail         = true
  enable_log_file_validation    = true

  event_selector {
    read_write_type           = "All"
    include_management_events = true

    data_resource {
      type = "AWS::S3::Object"
      values = ["arn:aws:s3:::${aws_s3_bucket.regtech_iac.bucket}/"]
    }
  }
}


resource "aws_sns_topic" "alarm_topic" {
    name = "high-cpu-alarm-topic"
}

resource "aws_sns_topic_subscription" "alarm_subscription" {
    topic_arn = aws_sns_topic.alarm_topic.arn
    protocol = "email"
    endpoint = "oloruntobiolurombi@gmail.com"
}

resource "aws_cloudwatch_metric_alarm" "cpu_alarm" {
  alarm_name          = "high_cpu_usage"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = "2"
  metric_name         = "CPUUtilization"
  namespace           = "AWS/EC2"
  period              = "120"
  statistic           = "Average"
  threshold           = "70"

  alarm_actions = [
    aws_sns_topic.alarm_topic.arn
  ]
}

AutoScaler IAM (iam-autoscaler.tf):

This will provision roles and policies for enabling the EKS Cluster Autoscaler are included, which will help in adjusting the number of worker nodes based on resource demands.

data "aws_iam_policy_document" "eks_cluster_autoscaler_assume_role_policy" {
  statement {
    actions = ["sts:AssumeRoleWithWebIdentity"]
    effect  = "Allow"

    condition {
      test     = "StringEquals"
      variable = "${replace(aws_iam_openid_connect_provider.eks.url, "https://", "")}:sub"
      values   = ["system:serviceaccount:kube-system:cluster-autoscaler"]
    }

    principals {
      identifiers = [aws_iam_openid_connect_provider.eks.arn]
      type        = "Federated"
    }
  }
}

resource "aws_iam_role" "eks_cluster_autoscaler" {
  assume_role_policy = data.aws_iam_policy_document.eks_cluster_autoscaler_assume_role_policy.json
  name               = "eks-cluster-autoscaler"
}

resource "aws_iam_policy" "eks_cluster_autoscaler" {
  name = "eks-cluster-autoscaler"

  policy = jsonencode({
    Statement = [{
      Action = [
                "autoscaling:DescribeAutoScalingGroups",
                "autoscaling:DescribeAutoScalingInstances",
                "autoscaling:DescribeLaunchConfigurations",
                "autoscaling:DescribeTags",
                "autoscaling:SetDesiredCapacity",
                "autoscaling:TerminateInstanceInAutoScalingGroup",
                "ec2:DescribeLaunchTemplateVersions"
            ]
      Effect   = "Allow"
      Resource = "*"
    }]
    Version = "2012-10-17"
  })
}

resource "aws_iam_role_policy_attachment" "eks_cluster_autoscaler_attach" {
  role       = aws_iam_role.eks_cluster_autoscaler.name
  policy_arn = aws_iam_policy.eks_cluster_autoscaler.arn
}

output "eks_cluster_autoscaler_arn" {
  value = aws_iam_role.eks_cluster_autoscaler.arn
}

Routing (`security_groups.tf`)

This defines the security groups required for your infrastructure. Security groups act as virtual firewalls that control the inbound and outbound traffic to your resources.

touch security_groups.tf

# Provides a security group 
resource "aws_security_group" "main_sg" {
    name = "main_sg"
    description = var.main_sg_description
    vpc_id = aws_vpc.main.id 

    ingress  {
        description = "ssh access"
        from_port = 22
        to_port = 22
        protocol = "tcp"
        cidr_blocks = ["0.0.0.0/0"]
    }

    ingress  {
        description = "Kubernetes API access"
        from_port = 443
        to_port = 443 
        protocol = "tcp"
        cidr_blocks = ["0.0.0.0/0"]
    }

    egress {
        from_port = 0
        to_port = 0
        protocol = -1 
        cidr_blocks = ["0.0.0.0/0"]
    }

    tags = {
        Name = var.tags_main_sg_eks
    }
}

Variables (`variables.tf`)

This contains the variable definitions used across your Terraform configurations. These variables provide default values and can be overridden as needed.

touch variables.tf

variable "region" {
    type = string 
    default = "us-east-1"
}

#variable "profile" {
#  type = string
#  default = "tobi"
#}

variable "bucket_name" {
    type = string 
    default = "regtech-logs"
}

variable "aws_access_key_id" {
    type = string
    default = ""
}

variable "aws_secret_access_key" {
    type = string
    default = ""
}

variable "tags_vpc" {
    type = string 
    default = "main-vpc-eks"
}

variable "tags_public_rt" {
    type = string 
    default = "public-route-table"
}

variable "tags_igw" {
    type = string 
    default = "internet-gateway"
}

variable "tags_public_subnet_1" {
    type = string 
    default = "public-subnet-1"
}

variable "tags_public_subnet_2" {
    type = string 
    default = "public-subnet-2"
}

variable "tags_public_subnet_3" {
    type = string 
    default = "public-subnet-3"
}

variable "tags_private_subnet_1" {
    type = string 
    default = "private-subnet-1"
}

variable "tags_private_subnet_2" {
    type = string 
    default = "private-subnet-2"
}

variable "tags_private_subnet_3" {
    type = string 
    default = "private-subnet-3"
}

variable "tags_main_sg_eks" {
    type = string
    default = "main-sg-eks"
}

variable "instance_type" {
    type = string 
    default = "t2.micro"
}

variable "cluster_name" {
    type = string 
    default = "EKSCluster"
}

variable "node_group_name" {
    type = string 
    default = "SlaveNode"
}

variable "vpc_cidr_block" {
    type = string 
    default = "10.0.0.0/16"
}

variable "p_s_1_cidr_block" {
    type = string 
    default = "10.0.1.0/24"
}

variable "az_a" {
    type = string 
    default = "us-east-1a"
}

variable "p_s_2_cidr_block" {
    type = string 
    default = "10.0.2.0/24"
}

variable "az_b" {
    type = string 
    default = "us-east-1b"
}

variable "p_s_3_cidr_block" {
    type = string 
    default = "10.0.3.0/24"
}

variable "az_c" {
    type = string 
    default = "us-east-1c"
}

variable "private_s_1_cidr_block" {
    type = string 
    default = "10.0.4.0/24"
}

variable "az_private_a" {
    type = string 
    default = "us-east-1c"
}

variable "private_s_2_cidr_block" {
    type = string 
    default = "10.0.5.0/24"
}

variable "az_private_b" {
    type = string 
    default = "us-east-1c"
}

variable "private_s_3_cidr_block" {
    type = string 
    default = "10.0.6.0/24"
}

variable "az_private_c" {
    type = string 
    default = "us-east-1c"
}

variable "main_sg_description" {
    type = string 
    default = "Allow TLS inbound traffic and all outbound traffic"
}


variable "eks_node_group_profile" {
    type = string 
    default = "eks-node-group-instance-profile_log"
}

variable "eks_cluster_role_name" {
    type = string 
    default = "eksclusterrole_log"
}

variable "eks_node_group_role_name" {
    type = string 
    default = "eks-node-group-role_log"
}

variable "eks_node_group_volume_policy_name" {
    type = string 
    default = "eks-node-group-volume-policy"
}

variable "eks_describe_cluster_policy_name" {
    type = string 
    default = "eks-describe-cluster-policy_log"
}

variable "tags_nat" {
    type = string 
    default = "nat-gateway_eip"
}

variable "tags_k8s-nat" {
    type = string 
    default = "k8s-nat"
}

Provider (`provider.tf`)

This is crucial in any Terraform project as it defines the provider configuration, which in this case is AWS.

touch

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

# Configure the AWS Provider
provider "aws" {
  region = var.region
  access_key = var.aws_access_key_id
  secret_key = var.aws_secret_access_key 
}

IAM OpenID (`oidc.tf`)

This provides an IAM OpenID Connect provider.

touch oidc.tf

data "tls_certificate" "eks" {
  url = aws_eks_cluster.eks_cluster.identity[0].oidc[0].issuer
}

resource "aws_iam_openid_connect_provider" "eks" {
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = [data.tls_certificate.eks.certificates[0].sha1_fingerprint]
  url             = aws_eks_cluster.eks_cluster.identity[0].oidc[0].issuer
}

Automated EC2 Control Plane and EKS Cluster Deployment Using Terraform and GitHub Actions

Oloruntobi Olurombi — Tue, 03 Sep 2024 09:33:57 +0000

In the fast-paced world of DevOps, automation is key to managing and scaling cloud infrastructure efficiently. Setting up an EC2 control plane and EKS cluster manually can be time-consuming, error-prone, and difficult to maintain, especially as infrastructure grows. That’s where tools like Terraform and GitHub Actions come into play. By leveraging Infrastructure as Code (IaC) with Terraform and automating deployments through Continuous Integration and Continuous Deployment (CI/CD) pipelines with GitHub Actions, we can streamline the entire process, reduce human error, and ensure consistent, repeatable infrastructure setups.

Terraform allows us to define infrastructure in code, making it easy to version, share, and collaborate on infrastructure changes. It supports a wide range of cloud providers, making it a powerful tool for managing cloud resources like EC2 instances and EKS clusters. On the other hand, GitHub Actions provides a flexible CI/CD platform directly integrated with our codebase, enabling us to automate everything from code testing to infrastructure provisioning.

In this article, I’ll walk you through the steps to automate the deployment of an EC2 control plane and EKS cluster using Terraform, integrated with GitHub Actions. You’ll learn how to define your infrastructure in Terraform, set up GitHub Actions workflows to automate the deployment process, and implement best practices to ensure your infrastructure is scalable, secure, and resilient.

Whether you're a seasoned DevOps engineer looking to enhance your toolkit or a cloud enthusiast eager to dive into automation, this guide will provide you with practical insights and hands-on techniques to get started. By the end of this article, you’ll be equipped with the knowledge to automate your cloud infrastructure deployments, allowing you to focus more on innovation and less on manual setup.

Prerequisites

Before diving into the automation process, ensure you have the following set up:

Create an AWS Account: If you haven’t already, sign up for an AWS account. This will be your gateway to provisioning and managing cloud resources.
Install AWS CLI: The AWS Command Line Interface (CLI) is a powerful tool that allows you to interact with AWS services from your terminal. Make sure you have it installed and configured with your credentials.
Create an S3 Bucket: Terraform uses a state file to keep track of your infrastructure. Create an S3 bucket to store this state file securely. This is crucial for maintaining consistency and enabling collaboration when managing your infrastructure.
Create IAM Keys - Access Key and Secret Key: Generate IAM keys to enable Terraform and GitHub Actions to authenticate and manage your AWS resources. Make sure you have both the access key and the secret key ready.
Create a Key Pair: If you plan to SSH into your EC2 instances, create a key pair in AWS. This will allow you to securely connect to your instances once they’re up and running.

These prerequisites lay the foundation for the automated deployment process. With these in place, you’ll be ready to follow along with the Terraform configurations and GitHub Actions workflows that will bring your EC2 control plane and EKS cluster to life.

Project Structure

To keep everything organised, we’ll structure our project as follows:

Eks-setup-github-action/
├── iam_roles.tf
├── main.tf
├── provider.tf
├── routing.tf
├── security_groups.tf
├── variables.tf
└── vpc.tf

iam_roles.tf: This file contains the IAM roles and policies needed for your EC2 instances and EKS cluster to function securely.
main.tf: The main configuration file where we’ll define the resources that Terraform will provision, such as the EC2 instances and EKS cluster.
provider.tf: This file specifies the AWS provider and its configuration, including region and authentication details.
routing.tf: Manages the routing tables and routes within the VPC, ensuring that traffic is correctly routed between subnets and other AWS resources.
security_groups.tf: Defines the security groups, controlling inbound and outbound traffic to our EC2 instances and other components.
variables.tf: A centralized place for defining variables used across your Terraform configuration, making it easier to manage and reuse values.
vpc.tf: Contains the configuration for Virtual Private Cloud (VPC), including subnets, internet gateways, and other networking components.

By organising our Terraform files this way, we’ll ensure that our infrastructure code is modular, maintainable, and easy to understand. Each file has a specific purpose, making it easier to manage changes and scale the setup as needed.

Now, let’s start by looking at the provider.tf file, which is the foundation of our Terraform project.

The provider.tf File

The provider.tf file is crucial in any Terraform project as it defines the provider configuration, which in this case is AWS. This file tells Terraform which cloud provider to use and how to authenticate with it.

Below is the content of our provider.tf file:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

# Configure the AWS Provider
provider "aws" {
  region     = var.region
  access_key = var.aws_access_key_id
  secret_key = var.aws_secret_access_key
}

Deep Dive:

terraform { required_providers { ... } }: This block specifies that we're using the AWS provider from HashiCorp's registry. The version constraint (~> 5.0) ensures that we use a version compatible with our configuration, avoiding breaking changes from newer versions.

provider "aws" { ... }: This block configures the AWS provider with the necessary credentials and region. The region, access_key, and secret_key are pulled from variables (var.region, var.aws_access_key_id, and var.aws_secret_access_key), making the configuration flexible and secure.

region: The AWS region where our resources will be deployed, such as us-west-2.

access_key and secret_key: These are the AWS credentials that Terraform uses to authenticate and manage infrastructure. By using variables, you can avoid hardcoding sensitive information directly in your configuration files.

This provider.tf setup ensures that Terraform can communicate with AWS using the credentials and region you've specified. It also makes it easy to switch regions or credentials by simply changing the variable values, keeping your infrastructure code adaptable and secure.

With our provider configured, we can now move on to the core of our Terraform setup.

The vpc.tf File

The vpc.tf file contains the configuration for our Virtual Private Cloud (VPC), including the VPC itself and its associated subnets. Proper network setup is crucial for the secure and efficient operation of our EC2 instances and EKS cluster.

Here’s a look at the content of vpc.tf:

# Provides a VPC resource
resource "aws_vpc" "main" {
  cidr_block       = var.vpc_cidr_block
  instance_tenancy = "default"

  tags = {
    Name = var.tags_vpc
  }
}

# Provides VPC Public subnet resources
resource "aws_subnet" "public_subnet_1" {
  vpc_id                  = aws_vpc.main.id
  cidr_block              = var.p_s_1_cidr_block
  availability_zone       = var.az_a
  map_public_ip_on_launch = true

  tags = {
    Name = var.tags_public_subnet_1
  }
}

resource "aws_subnet" "public_subnet_2" {
  vpc_id                  = aws_vpc.main.id
  cidr_block              = var.p_s_2_cidr_block
  availability_zone       = var.az_b
  map_public_ip_on_launch = true

  tags = {
    Name = var.tags_public_subnet_2
  }
}

resource "aws_subnet" "public_subnet_3" {
  vpc_id                  = aws_vpc.main.id
  cidr_block              = var.p_s_3_cidr_block
  availability_zone       = var.az_c
  map_public_ip_on_launch = true

  tags = {
    Name = var.tags_public_subnet_3
  }

Deep Dive:

aws_vpc "main_vpc": Creates the VPC with a specified CIDR block. DNS support and DNS hostnames are enabled to allow instances within the VPC to resolve internal and external domain names.
aws_internet_gateway "igw": Attaches an Internet Gateway to the VPC, enabling resources within the VPC to communicate with the internet.
aws_subnet "public_subnet_1" and aws_subnet "public_subnet_2": Define public subnets within the VPC. These subnets have the map_public_ip_on_launch attribute set to true, which assigns public IP addresses to instances launched in these subnets.
aws_subnet "private_subnet_1": Defines a private subnet within the VPC. Instances in this subnet do not have direct access to the internet.
aws_route_table "public_rt": Creates a route table for the public subnets. The route table includes a route that directs all outbound traffic (0.0.0.0/0) to the Internet Gateway.
aws_route_table_association "public_subnet_1_association" and aws_route_table_association "public_subnet_2_association": Associates the public route table with the public subnets, ensuring that traffic from these subnets is routed through the Internet Gateway.
aws_security_group "main_sg": Defines a security group for EC2 instances. It allows inbound SSH traffic (port 22) from any IP address and permits all outbound traffic.
aws_vpc_endpoint "s3_endpoint": Creates a VPC endpoint for S3, allowing instances within the VPC to access S3 buckets without traversing the internet. This enhances security and reduces latency.

The vpc.tf file sets up the networking environment for your infrastructure. By defining the VPC, subnets, route tables, and security groups, you ensure that your resources are correctly networked and secured. This modular approach allows you to maintain and scale your infrastructure effectively while keeping your network configuration organized and manageable.

Now it is time to start building our resources.

The main.tf File

The main.tf file is where we define the actual resources that Terraform will provision. This file includes configurations for the EC2 instance, EKS cluster, and associated components.

Here’s a look at the content of main.tf:

terraform {
  backend "s3" {
    bucket = "terraform-state-bucket-tobi"
    key    = "terraform.tfstate"
    region = "us-east-1"
    encrypt = true 
    #profile = "tobi"
  }
}

# Provides an EC2 instance resource
data "aws_ami" "amazon-linux-2" {
  most_recent = true

  filter {
    name   = "owner-alias"
    values = ["amazon"]
  }

  filter {
    name   = "name"
    values = ["amzn2-ami-hvm*"]
  }
}

# Provides an EC2 Instance for Control Plane
resource "aws_instance" "control_plane" {
  depends_on = ["aws_internet_gateway.igw"]

  ami                         = data.aws_ami.amazon-linux-2.id
  instance_type               = var.instance_type
  associate_public_ip_address = true
  iam_instance_profile        = aws_iam_instance_profile.ec2_instance_profile.name
  key_name                    = "bastion"
  vpc_security_group_ids      = [aws_security_group.main_sg.id]
  subnet_id                   = aws_subnet.public_subnet_1.id
}

# Provides an EKS Cluster
resource "aws_eks_cluster" "eks_cluster" {
  name     = var.cluster_name
  role_arn = aws_iam_role.eks_cluster_role.arn

  version = "1.28"

  vpc_config {
    subnet_ids = [
      aws_subnet.public_subnet_1.id,
      aws_subnet.public_subnet_2.id,
      aws_subnet.public_subnet_3.id
    ]
  }

  depends_on = [
    aws_iam_role_policy_attachment.eks_cluster_policy_attachment,
    aws_iam_role_policy_attachment.eks_service_policy_attachment,
  ]
}

# Provides an EKS Node Group 
resource "aws_eks_node_group" "eks_node_group" {
  cluster_name    = aws_eks_cluster.eks_cluster.name
  node_group_name = var.node_group_name
  node_role_arn   = aws_iam_role.eks_node_group_role.arn
  subnet_ids      = [
    aws_subnet.public_subnet_1.id,
    aws_subnet.public_subnet_2.id,
    aws_subnet.public_subnet_3.id
  ]

  scaling_config {
    desired_size = 2
    max_size     = 2
    min_size     = 2
  }

  update_config {
    max_unavailable = 1
  }

  depends_on = [
    aws_iam_role_policy_attachment.eks_worker_node_policy_attachment,
    aws_iam_role_policy_attachment.eks_cni_policy_attachment,
    aws_iam_role_policy_attachment.ec2_container_registry_readonly,
  ]
}

# Output Resources
output "endpoint" {
  value = aws_eks_cluster.eks_cluster.endpoint
}

output "ec2_public_ip" {
  value = aws_instance.control_plane.public_ip 
}

output "ec2_instance_id" {
  value = aws_instance.control_plane.id
}

output "eks_cluster_name" {
  value = aws_eks_cluster.eks_cluster.name
}

Deep Dive:

terraform { backend "s3" { ... } }: Configures the backend to use an S3 bucket for storing the Terraform state file. This ensures that your state is centralized and protected with encryption.

data "aws_ami" "amazon-linux-2": Fetches the latest Amazon Linux 2 AMI ID, which is used for the EC2 instance.

resource "aws_instance" "control_plane": Defines an EC2 instance that will act as the control plane. It specifies the AMI, instance type, and other settings.

resource "aws_eks_cluster" "eks_cluster": Configures the EKS cluster, including its version and VPC configuration. It also ensures that IAM roles and policies are in place before and after cluster creation.

resource "aws_eks_node_group" "eks_node_group": Sets up an EKS node group with scaling and update configurations. This node group will run your containerized workloads.

output "endpoint": Outputs the EKS cluster endpoint URL, which is necessary for accessing the cluster.

output "ec2_public_ip": Outputs the public IP address of the EC2 instance.

output "ec2_instance_id": Outputs the ID of the EC2 instance.

output "eks_cluster_name": Outputs the name of the EKS cluster.

The main.tf file provides a comprehensive setup for your EC2 control plane and EKS cluster, ensuring that all necessary resources are created and configured. With this file, you can easily manage and scale your infrastructure while maintaining a clean and organised configuration.

With our primary resources defined, we now turn our attention to the IAM roles required for our setup. IAM roles are crucial for managing permissions and access controls for both our EC2 instances and EKS cluster. The iam_roles.tf file handles the creation of these roles and their associated policies.

The iam_roles.tf File

The iam_roles.tf file defines the IAM roles and policies necessary for the EC2 instance, EKS cluster, and worker nodes. This setup ensures that each component has the appropriate permissions to interact with other AWS services securely.

Here’s the content of iam_roles.tf:

# Declare the aws_caller_identity data source
data "aws_caller_identity" "current" {}

# IAM Role For EC2
resource "aws_iam_role" "ec2_instance_role" {
    name = var.ec2_instance_role_name

    assume_role_policy = jsonencode({
        Version = "2012-10-17",
        Statement = [
            {
                Action = "sts:AssumeRole",
                Effect = "Allow",
                Principal = {
                    Service = "ec2.amazonaws.com"
                }
            }
        ]
    })
}

# Policies For EC2 IAM Role

# Attach Policies 
resource "aws_iam_role_policy_attachment" "ec2_full_access" {
    role = aws_iam_role.ec2_instance_role.name
    policy_arn = "arn:aws:iam::aws:policy/AmazonEC2FullAccess"
}

resource "aws_iam_role_policy_attachment" "ec2_read_only_access" {
    role = aws_iam_role.ec2_instance_role.name 
    policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ReadOnlyAccess"
}

# Create an Instance Profile (for attaching the role to an EC2 instance)
resource "aws_iam_instance_profile" "ec2_instance_profile" {
    name = var.ec2_instance_profile
    role = aws_iam_role.ec2_instance_role.name
}

# IAM Role for EKS Cluster Plane 
resource "aws_iam_role" "eks_cluster_role" {
    name = var.eks_cluster_role_name

    assume_role_policy = jsonencode({
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Principal": {
                    "Service": "eks.amazonaws.com"
                },
                "Action": "sts:AssumeRole"
            }
        ]
    })
}

resource "aws_iam_role_policy_attachment" "eks_cluster_policy_attachment" {
    policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
    role = aws_iam_role.eks_cluster_role.name 
}

resource "aws_iam_role_policy_attachment" "eks_service_policy_attachment" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSVPCResourceController"
  role = aws_iam_role.eks_cluster_role.name
}

# IAM Role for Worker node
resource "aws_iam_role" "eks_node_group_role" {
    name = var.eks_node_group_role_name

    assume_role_policy = jsonencode({
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Principal": {
                    "Service": "ec2.amazonaws.com"
                },
                "Action": "sts:AssumeRole"
            }
        ]
    })
}

resource "aws_iam_role_policy_attachment" "eks_worker_node_policy_attachment" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
  role       = aws_iam_role.eks_node_group_role.name
}

resource "aws_iam_role_policy_attachment" "eks_cni_policy_attachment" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
  role       = aws_iam_role.eks_node_group_role.name
}

resource "aws_iam_role_policy_attachment" "ec2_container_registry_readonly" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
  role       = aws_iam_role.eks_node_group_role.name
}

# Create a Policy That Allows The eks:DescribeCluster Action
resource "aws_iam_policy" "eks_describe_cluster_policy" {
  name        = var.eks_describe_cluster_policy_name
  description = "Policy to allow describing EKS clusters"
  policy      = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action   = "eks:DescribeCluster"
        Effect   = "Allow"
        Resource = "arn:aws:eks:${var.region}:${data.aws_caller_identity.current.account_id}:cluster/${var.cluster_name}"
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "eks_describe_cluster_policy_attachment" {
  role       = aws_iam_role.ec2_instance_role.name
  policy_arn = aws_iam_policy.eks_describe_cluster_policy.arn
}

data "aws_caller_identity" "current": Retrieves information about the current AWS account. This is used to dynamically reference the account ID in policies.

IAM Roles for EC2:

aws_iam_role "ec2_instance_role": Defines a role for the EC2 instance with an assume role policy allowing EC2 to assume this role.

aws_iam_role_policy_attachment "ec2_full_access" and ec2_read_only_access: Attach policies to the EC2 role to provide full and read-only access to EC2 resources.

aws_iam_instance_profile "ec2_instance_profile": Creates an instance profile to attach the EC2 role to the EC2 instance.

IAM Roles for EKS Cluster:

aws_iam_role "eks_cluster_role": Defines a role for the EKS cluster with an assume role policy allowing EKS to assume this role.

aws_iam_role_policy_attachment "eks_cluster_policy_attachment" and eks_service_policy_attachment: Attach policies necessary for the EKS cluster to operate.

IAM Roles for EKS Worker Nodes:

aws_iam_role "eks_node_group_role": Defines a role for EKS worker nodes with an assume role policy allowing EC2 to assume this role.

aws_iam_role_policy_attachment "eks_worker_node_policy_attachment", eks_cni_policy_attachment, and ec2_container_registry_readonly: Attach policies to the worker node role to allow it to interact with EKS, manage networking, and access container images.

aws_iam_policy "eks_describe_cluster_policy" and aws_iam_role_policy_attachment "eks_describe_cluster_policy_attachment": Create a policy to allow describing EKS clusters and attach it to the EC2 role.

The iam_roles.tf file ensures that each component of your infrastructure has the appropriate permissions to operate effectively and securely. Proper role and policy management is essential for maintaining a secure and functional environment.

Now that we’ve set up the IAM roles, let’s move on to configuring the networking components of our infrastructure. Proper routing and internet access are crucial for ensuring that our resources can communicate as needed.

The routing.tf File

The routing.tf file handles the creation of routing tables and internet gateways for your VPC. This setup ensures that your public and private subnets can route traffic appropriately.

Here’s the content of the routing.tf file:

# Provides a resource to create a VPC routing table
resource "aws_route_table" "public_rt" {
  vpc_id = aws_vpc.main.id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.igw.id
  }

  tags = {
    Name = var.tags_public_rt
  }
}

# Provides a resource to create an association between a route table and Public subnets
resource "aws_route_table_association" "public_subnet_1_association" {
    subnet_id = aws_subnet.public_subnet_1.id
    route_table_id = aws_route_table.public_rt.id 
}

resource "aws_route_table_association" "public_subnet_2_association" {
    subnet_id = aws_subnet.public_subnet_2.id
    route_table_id = aws_route_table.public_rt.id 
}

resource "aws_route_table_association" "public_subnet_3_association" {
    subnet_id = aws_subnet.public_subnet_3.id
    route_table_id = aws_route_table.public_rt.id 
}

# Provides a resource to create a VPC Internet Gateway
resource "aws_internet_gateway" "igw" {
  vpc_id = aws_vpc.main.id

  tags = {
    Name = var.tags_igw
  }
}

# Provides a resource to create a private route table 
resource "aws_route_table" "private_rt" {
    vpc_id = aws_vpc.main.id
}

# Provides a resource to create an association between a route table and Private subnets
resource "aws_route_table_association" "private_subnet_1_association" {
    subnet_id = aws_subnet.private_subnet_1.id
    route_table_id = aws_route_table.private_rt.id 
}

resource "aws_route_table_association" "private_subnet_2_association" {
    subnet_id = aws_subnet.private_subnet_2.id
    route_table_id = aws_route_table.private_rt.id 
}

resource "aws_route_table_association" "private_subnet_3_association" {
    subnet_id = aws_subnet.private_subnet_3.id
    route_table_id = aws_route_table.private_rt.id 
}

Deep Dive:

aws_route_table "public_rt": Creates a routing table for public subnets. This routing table has a default route (0.0.0.0/0) that directs traffic to the internet gateway, allowing instances in public subnets to access the internet.

aws_route_table_association: Associates the public routing table with the public subnets. This ensures that traffic from these subnets is routed correctly according to the public_rt table.

aws_internet_gateway "igw": Creates an internet gateway and attaches it to the VPC. This gateway allows instances in the public subnets to access the internet.

aws_route_table "private_rt": Creates a routing table for private subnets. This table does not have a route to the internet gateway, ensuring that private subnets do not have direct internet access.

aws_route_table_association: Associates the private routing table with the private subnets, ensuring that traffic is routed according to the private_rt table.

The routing.tf file ensures that your VPC is properly configured to handle traffic both internally and to/from the internet. With the routing and internet access in place, your infrastructure will be able to interact with external resources and services as needed.

The security.tf File

The security.tf file defines the security groups required for your infrastructure. Security groups act as virtual firewalls that control the inbound and outbound traffic to your resources, such as EC2 instances and EKS clusters.

Here’s the content of the security.tf file:

# Provides a security group 
resource "aws_security_group" "main_sg" {
    name = "main_sg"
    description = var.main_sg_description
    vpc_id = aws_vpc.main.id 

    ingress  {
        description = "ssh access"
        from_port = 22
        to_port = 22
        protocol = "tcp"
        cidr_blocks = ["0.0.0.0/0"]
    }

    ingress  {
        description = "Kubernetes API access"
        from_port = 443
        to_port = 443 
        protocol = "tcp"
        cidr_blocks = ["0.0.0.0/0"]
    }

    egress {
        from_port = 0
        to_port = 0
        protocol = -1 
        cidr_blocks = ["0.0.0.0/0"]
    }

    tags = {
        Name = var.tags_main_sg_eks
    }
}

Deep Dive:

aws_security_group "main_sg": Defines a security group named main_sg associated with your VPC. This security group is crucial for controlling access to your EC2 instances and EKS cluster.

Ingress Rules:

SSH Access: Allows incoming traffic on port 22 (SSH) from any IP address (0.0.0.0/0). This is essential for remote access to your EC2 instances.

Kubernetes API Access: Allows incoming traffic on port 443 (HTTPS) from any IP address (0.0.0.0/0). This rule is necessary for accessing the Kubernetes API server if you’re using EKS.
Egress Rules:

Outbound Traffic: Allows all outbound traffic (protocol -1) to any IP address (0.0.0.0/0). This ensures that instances can initiate outbound connections to the internet or other resources.

The security.tf file is vital for defining the security posture of your infrastructure. By setting up appropriate security groups, you ensure that your resources are protected from unwanted access while allowing necessary communication for their operation.

With the security groups set up to manage traffic to and from your resources, it's important to ensure that all the necessary variables are defined. These variables help to parameterize your Terraform configurations, making them flexible and easier to manage.

The variables.tf File

The variables.tf file contains the variable definitions used across your Terraform configurations. These variables provide default values and can be overridden as needed.

Here’s the content of the variables.tf file:

variable "region" {
    type = string 
    default = "us-east-1"
}

variable "bucket_name" {
    type = string 
    default = "terraform-state-bucket-tobi"
}

variable "aws_access_key_id" {
    type = string
    default = ""
}

variable "aws_secret_access_key" {
    type = string
    default = ""
}

variable "tags_vpc" {
    type = string 
    default = "main-vpc-eks"
}

variable "tags_public_rt" {
    type = string 
    default = "public-route-table"
}

variable "tags_igw" {
    type = string 
    default = "internet-gateway"
}

variable "tags_public_subnet_1" {
    type = string 
    default = "public-subnet-1"
}

variable "tags_public_subnet_2" {
    type = string 
    default = "public-subnet-2"
}

variable "tags_public_subnet_3" {
    type = string 
    default = "public-subnet-3"
}

variable "tags_private_subnet_1" {
    type = string 
    default = "private-subnet-1"
}

variable "tags_private_subnet_2" {
    type = string 
    default = "private-subnet-2"
}

variable "tags_private_subnet_3" {
    type = string 
    default = "private-subnet-3"
}

variable "tags_main_sg_eks" {
    type = string
    default = "main-sg-eks"
}

variable "instance_type" {
    type = string 
    default = "t2.micro"
}

variable "cluster_name" {
    type = string 
    default = "EKSCluster"
}

variable "node_group_name" {
    type = string 
    default = "SlaveNode"
}

variable "vpc_cidr_block" {
    type = string 
    default = "10.0.0.0/16"
}

variable "p_s_1_cidr_block" {
    type = string 
    default = "10.0.1.0/24"
}

variable "az_a" {
    type = string 
    default = "us-east-1a"
}

variable "p_s_2_cidr_block" {
    type = string 
    default = "10.0.2.0/24"
}

variable "az_b" {
    type = string 
    default = "us-east-1b"
}

variable "p_s_3_cidr_block" {
    type = string 
    default = "10.0.3.0/24"
}

variable "az_c" {
    type = string 
    default = "us-east-1c"
}

variable "private_s_1_cidr_block" {
    type = string 
    default = "10.0.4.0/24"
}

variable "az_private_a" {
    type = string 
    default = "us-east-1c"
}

variable "private_s_2_cidr_block" {
    type = string 
    default = "10.0.5.0/24"
}

variable "az_private_b" {
    type = string 
    default = "us-east-1c"
}

variable "private_s_3_cidr_block" {
    type = string 
    default = "10.0.6.0/24"
}

variable "az_private_c" {
    type = string 
    default = "us-east-1c"
}

variable "main_sg_description" {
    type = string 
    default = "Allow TLS inbound traffic and all outbound traffic"
}

variable "ec2_instance_role_name" {
    type = string 
    default = "ec2-instance-role"
}

variable "ec2_instance_profile" {
    type = string 
    default = "ec2-instance-profile"
}

variable "eks_cluster_role_name" {
    type = string 
    default = "eksclusterrole-2"
}

variable "eks_node_group_role_name" {
    type = string 
    default = "eks-node-group-role"
}

variable "eks_describe_cluster_policy_name" {
    type = string 
    default = "eks-describe-cluster-policy"
}

Deep Dive:

Region and Access Keys: Define your AWS region and credentials for Terraform to interact with AWS.
Tags: Tagging helps to identify and manage resources, making it easier to organize and maintain them.
Instance and Cluster Details: Variables for instance types, cluster names, and other configurations help to customize your deployments.
CIDR Blocks and Availability Zones: Define your network ranges and availability zones for proper network segmentation.

The variables.tf file is essential for defining all the customizable parameters that Terraform uses to create and manage your infrastructure. It helps in keeping your Terraform configurations flexible and adaptable to different environments or requirements.

With your infrastructure set up using Terraform, it’s time to automate the deployment and management of your resources through Continuous Integration (CI) and Continuous Deployment (CD). Leveraging GitHub Actions for CI/CD allows you to automate the processes of building, testing, and deploying your infrastructure, making your workflow more efficient and less error-prone.

CI/CD Integration: Automating Deployment and Management

In this section, we will introduce how to set up GitHub Actions workflows to manage your Terraform configurations and automate the deployment of your Elastic Kubernetes Service (EKS) cluster.

Adding GitHub Secrets

Before you can use GitHub Actions to deploy your infrastructure, you need to add sensitive information like AWS credentials and SSH keys as secrets in your GitHub repository. These secrets will be used in the workflow to interact with AWS and other services securely.

To add secrets to your GitHub repository:

Navigate to Your Repository: Go to the repository where you want to add secrets.
Access Repository Settings: Click on the “Settings” tab in the repository.
Select Secrets and Variables: In the left sidebar, click on "Secrets and variables” and then "Actions."
Add New Secrets: Click on the "New repository secret" button. Add the following secrets:

AWS_ACCESS_KEY_ID: Your AWS Access Key ID.

AWS_SECRET_ACCESS_KEY: Your AWS Secret Access Key.

AWS_REGION: The AWS region where your resources are located.

EKS_CLUSTER_NAME: The name of your EKS cluster.

EC2_SSH_PRIVATE_KEY: The private SSH key for accessing your EC2 instances.

GitHub Actions Workflow for EKS Setup

To automate the setup of your EKS cluster, we will create a GitHub Actions workflow that performs the following steps:

Log in to AWS: Configure AWS credentials.
Initialize Terraform: Prepare Terraform for execution.
Plan Terraform Deployment: Preview the changes Terraform will make.
Apply Terraform: Provision the infrastructure defined in your Terraform configuration.
Install Tools on EC2: Ensure necessary tools like AWS CLI and kubectl are installed on the EC2 instance.

Here is the content of the GitHub Actions workflow file eks-setup.yaml:

name: Set up EKS With Terraform

on: push

env: 
  AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
  AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
  AWS_REGION: ${{ secrets.AWS_REGION }}
  EKS_CLUSTER_NAME: ${{ secrets.EKS_CLUSTER_NAME}}

jobs:
  LogInToAWS:
    runs-on: ubuntu-latest
    steps:
    - name: Configure credentials
      uses: aws-actions/configure-aws-credentials@v4
      with:
        aws-access-key-id: ${{ env.AWS_ACCESS_KEY_ID }}
        aws-secret-access-key: ${{ env.AWS_SECRET_ACCESS_KEY }}
        aws-region: ${{ env.AWS_REGION }}

  TerraformInit:
    runs-on: ubuntu-latest
    needs: LogInToAWS
    steps:
    - name: Checkout
      uses: actions/checkout@v4

    - name: Initialize Terraform
      run: terraform init

  TerraformPlan:
    runs-on: ubuntu-latest
    needs: TerraformInit
    steps:
    - name: Checkout
      uses: actions/checkout@v4

    - name: Initialize Terraform
      run: terraform init

    - name: Plan Terraform
      run: terraform plan

  TerraformApply-InstallTools:
    runs-on: ubuntu-latest
    needs: TerraformPlan
    steps:
    - name: Checkout
      uses: actions/checkout@v4

    - name: Initialize Terraform (Again, if needed)
      run: terraform init

    - name: Apply Terraform
      run: terraform apply -auto-approve

    - name: Get EC2 Public IP
      id: get_public_ip
      run: |
        terraform output -json > tf_output.json
        EC2_PUBLIC_IP=$(jq -r '.ec2_public_ip.value' tf_output.json)
        if [ -z "$EC2_PUBLIC_IP" ]; then
          echo "Error: EC2 Public IP is empty."
          exit 1
        fi
        echo "EC2_PUBLIC_IP=$EC2_PUBLIC_IP" >> $GITHUB_ENV
        echo "Captured EC2 Public IP: $EC2_PUBLIC_IP"

    - name: Ensure EC2 Public IP is Not Empty
      if: ${{ env.EC2_PUBLIC_IP == '' }}
      run: |
        echo "Error: EC2 Public IP is empty. Exiting."
        exit 1    

    - name: Install SSH Client
      run: sudo apt-get update && sudo apt-get install -y sshpass

    - name: Setup SSH Key
      run: |
        echo "${{ secrets.EC2_SSH_PRIVATE_KEY }}" > /tmp/private_key
        chmod 600 /tmp/private_key

    - name: Wait for EC2 Instance to be Ready
      run: sleep 100 

    - name: Print Environment Variables
      run: |
        echo "EC2_PUBLIC_IP=${{ env.EC2_PUBLIC_IP }}"

    - name: Debug Public IP
      run: |
        echo "Public IP: ${{ env.EC2_PUBLIC_IP }}"

    - name: SSH and Install AWS CLI and kubectl
      run: |
        set -x
        ssh -o StrictHostKeyChecking=no -i /tmp/private_key ec2-user@${{ env.EC2_PUBLIC_IP }} << 'EOF'
          sudo yum install -y unzip

          # Check if AWS CLI exists
          if ! command -v aws &> /dev/null
          then
            echo "AWS CLI not found, installing..."
            curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
            unzip awscliv2.zip
            sudo ./aws/install 
          else
            echo "AWS CLI already installed"
          fi

          # Check if kubectl exists
          if ! command -v kubectl &> /dev/null
          then
            echo "kubectl not found, installing..."
            curl -O "https://s3.us-west-2.amazonaws.com/amazon-eks/1.28.11/2024-07-12/bin/linux/amd64/kubectl"
            chmod +x ./kubectl
            mkdir -p $HOME/bin && cp ./kubectl $HOME/bin/kubectl && export PATH=$HOME/bin:$PATH
          else
            echo "kubectl already installed"
          fi

          kubectl version --client
          #aws eks update-kubeconfig --region ${{ env.AWS_REGION }} --name ${{ env.EKS_CLUSTER_NAME }}
        EOF

    - name: Verify EC2 Instance State 
      run: |
        INSTANCE_STATE=$(aws ec2 describe-instances --instance-ids $(terraform output -raw ec2_instance_id) --query 'Reservations[*].Instances[*].State.Name' --output text)  
        echo "EC2 Instance State: $INSTANCE_STATE"
        if [[ "$INSTANCE_STATE" != "running" ]]; then
          echo "Error: EC2 instance is not running."
          exit 1
        fi

Deep dive of the Workflow

LogInToAWS: Configures AWS credentials to allow the workflow to interact with your AWS account.

TerraformInit: Initialises Terraform to prepare it for running.

TerraformPlan: Plans the changes Terraform will make, allowing you to review them before applying.

TerraformApply-InstallTools: Applies the Terraform configurations, retrieves the EC2 public IP, and ensures necessary tools are installed on the EC2 instance.

By integrating this GitHub Actions workflow into your project and configuring the required secrets, you ensure that your infrastructure is provisioned consistently and managed efficiently with automated processes. This setup minimizes manual intervention, reduces the risk of errors, and accelerates the deployment of your applications.

Pushing Code to Trigger the Pipeline

Once the setup is complete, the final step is to push your code to the repository to trigger the pipeline.

Here’s how you can do it:

Add Your Files:

Ensure all your Terraform configuration files and the GitHub Actions workflow file (eks-setup.yaml) are added to your local repository.

Commit Your Changes:

Use Git commands to commit your changes. For example:

git add .
git commit -m "Add Terraform configurations and GitHub Actions workflow"

Push to Repository:

Push your changes to the remote repository on GitHub:

git push origin main

Verify Pipeline Execution:

After pushing your code, navigate to the "Actions" tab in your GitHub repository to monitor the progress of your workflow. The pipeline will automatically trigger, executing the defined steps, from initialising and planning Terraform to applying the configurations and setting up necessary tools.

Summary

By following the steps outlined in this article, you have set up a robust and automated deployment pipeline. We started by defining and managing your infrastructure with Terraform, then moved on to integrate GitHub Actions for a seamless CI/CD process. Finally, by pushing your code to the repository, you triggered the pipeline, ensuring your infrastructure deployment is automated and consistent.

Conclusion

Automating your infrastructure deployment with Terraform and GitHub Actions brings significant benefits, including reduced manual intervention, improved efficiency, and enhanced consistency. By leveraging these tools, you maintain infrastructure as code, which simplifies version control and collaboration.

I hope this guide has equipped you with the knowledge and skills needed to implement a fully automated deployment pipeline for your cloud infrastructure. If you have any questions or need further assistance, please feel free to reach out or leave a comment.

Happy deploying!

AWS Elastic Beanstalk vs. AWS Amplify: Choosing the Right Tool for Your App’s Success

Oloruntobi Olurombi — Wed, 21 Aug 2024 22:18:20 +0000

Both AWS Elastic Beanstalk and AWS Amplify are popular managed services for deploying and managing applications, but they cater to different use cases and developer needs. Let’s explore these two services, highlighting their differences, trade-offs, and considerations to help you choose the right tool for your project.

Understanding AWS Elastics Beanstalk

AWS Elastic Beanstalk is a Platform-as-a-Service (PaaS) that helps developers deploy and manage applications in several languages, including Java, .NET, Node.js, PHP, Python, Ruby, Go, and Docker. It abstracts much of the infrastructure management by automatically handling scaling, load balancing, health monitoring, and environment management.

Key Features:

Multi-language support.
Automatic scaling and load balancing.
Full control over the underlying infrastructure, including EC2 instances, security groups, and more.
Integration with other AWS services like RDS, S3, and CloudWatch.

Understanding AWS Amplify

AWS Amplify is a development platform tailored for building and deploying full-stack web and mobile applications, particularly those with front-end frameworks like React, Angular, and Vue.js. Amplify provides an end-to-end environment, offering tools for backend development, hosting, and continuous integration/continuous deployment (CI/CD).

Key Features:

Seamless integration with frontend frameworks.
Built-in support for authentication, APIs (GraphQL/REST), storage, and data synchronisation.
Continuous deployment with Git-based workflows (GitHub, GitLab, Bitbucket).
Simple hosting with globally distributed CDN.

Use Cases

When to Use AWS Elastic Beanstalk:

Traditional Web Applications: Elastic Beanstalk is well-suited for applications that require more complex server-side logic, multiple backend services, or custom compute environments.
Multi-Language/Stack Applications: If your project involves multiple languages or technology stacks, Beanstalk provides flexibility by supporting various programming environments.
Legacy Applications: For legacy systems or applications that need to be lifted and shifted into the cloud with minimal refactoring, Beanstalk’s managed environment is ideal.
Full Control Over Infrastructure: If you need detailed control over the underlying infrastructure, such as choosing specific EC2 instance types or networking configurations, Beanstalk is the better option.

When to Use AWS Amplify:

Modern Web and Mobile Applications: Amplify is optimised for serverless, full-stack applications where the frontend is built with popular JavaScript frameworks like React, Angular, or Vue.js.
Single-Page Applications (SPAs): Amplify excels in building SPAs with features like real-time data synchronisation, authentication, and serverless functions.
Rapid Prototyping and Development: For teams building and iterating quickly, Amplify’s out-of-the-box backend, hosting, and CI/CD features significantly reduce development overhead.
Mobile App Development: Amplify offers robust tools for mobile development, with built-in support for authentication, push notifications, and analytics.

Key Differences

Aspect	AWS Elastic Beanstalk	AWS Amplify
Target Audience	Backend developers, full-stack developers	Frontend developers, mobile and full-stack teams
Supported Languages/Frameworks	Java, .NET, Node.js, PHP, Python, Ruby, Go, Docker	React, Angular, Vue.js, Flutter, and other frontends
Infrastructure Control	High - You can customise EC2, security groups, VPC	Low - Mostly abstracted and serverless
Scaling	Automatic, with customizable EC2 instances	Automatic, fully managed (serverless)
CI/CD	Git-based (Jenkins, GitLab CI, etc.)	Integrated with Amplify CI/CD pipeline
Deployment Models	Single and multi-environment deployments	Single and multi-environment deployments
Ease of Use	Medium - Requires some infrastructure knowledge	High - Focuses on simplicity and rapid iteration

Trade-Offs and Considerations

Control vs. Abstraction:

Elastic Beanstalk: Provides more control over the infrastructure, which can be beneficial for fine-tuning performance, cost, and security settings. However, this comes with increased complexity and management overhead.

Amplify: Highly abstracted and serverless, making it easier to manage but with less control over the underlying infrastructure.

Scalability:

Elastic Beanstalk: Offers more flexibility in defining how applications scale, with custom EC2 instance types and scaling rules. Ideal for applications with more complex performance requirements.

Amplify: Automatically handles scaling but in a fully managed, serverless environment, better suited for applications with bursty or unpredictable traffic.

Flexibility in Application Architecture:

Elastic Beanstalk: Better suited for monolithic or microservices-based applications that require complex routing, service mesh setups, or hybrid architectures.

Amplify: Tailored for simpler, serverless applications where the backend is largely API-driven, typically with a frontend focus.

Integration and Ecosystem:

Elastic Beanstalk: Has deeper integration with traditional enterprise systems and supports a wide array of AWS services, making it more versatile for diverse use cases.

Amplify: Offers streamlined integration with Amplify-specific services, like Cognito for authentication, AppSync for GraphQL, and S3 for storage, optimising modern development stacks.

Cost Considerations:

Elastic Beanstalk: Costs vary based on the chosen EC2 instances, load balancers, and databases, with potential for higher costs for more complex setups.

Amplify: Primarily serverless, which often translates to cost savings, especially for low-traffic applications. However, usage-based pricing can lead to unexpected spikes.

Conclusion

Choosing between AWS Elastic Beanstalk and AWS Amplify depends on your application’s specific needs, development workflow, and infrastructure requirements. If your project demands more control over the backend, custom scaling, and multi-language support, Elastic Beanstalk is likely the better choice. On the other hand, if you’re focused on rapidly developing and deploying modern web or mobile applications with minimal infrastructure management, AWS Amplify offers the simplicity and speed you need.

In summary:

Choose Elastic Beanstalk for traditional web applications, multi-language stacks, and applications requiring fine-grained infrastructure control.

Choose Amplify for serverless, frontend-heavy applications, rapid prototyping, and mobile app development.

Understanding these differences and trade-offs will help you make an informed decision, ensuring your application scales effectively while meeting your technical and business goals.

Mastering Infrastructure as Code with Terraform Modules: A Practical Guide

Oloruntobi Olurombi — Mon, 19 Feb 2024 09:46:06 +0000

In the realm of modern infrastructure management, Infrastructure as Code (IaC) has emerged as a cornerstone practice for automating and managing cloud resources efficiently. Among the myriad of tools available, Terraform stands out as a powerful and versatile choice for provisioning, configuring, and managing infrastructure resources. Within Terraform, modules offer a potent mechanism for organizing and reusing infrastructure code, enabling teams to maintain scalable, modular, and maintainable infrastructure configurations. In this article, we delve into the world of Terraform modules, exploring their fundamentals, benefits, and providing practical examples to illustrate their usage.

Understanding Terraform Modules

At its core, a Terraform module is a collection of Terraform configuration files encapsulated in a directory, representing a set of resources with well-defined inputs and outputs. Modules abstract infrastructure components into reusable building blocks, promoting code reuse, consistency, and collaboration among teams. Modules can range from simple configurations, such as a single AWS EC2 instance, to complex architectures comprising multiple interconnected resources like VPCs, subnets, load balancers, and more.

Benefits of Terraform Modules

Reusability: Modules enable the encapsulation of infrastructure components, allowing them to be reused across projects and environments. This promotes consistency and reduces duplication of code.

Abstraction: Modules abstract the complexity of underlying infrastructure configurations, providing a clean interface with inputs and outputs. This simplifies the consumption of infrastructure resources, making it easier to manage and understand.

Scalability: With modules, infrastructure configurations can scale effortlessly as projects grow in complexity. Teams can compose intricate architectures by combining and nesting modules, maintaining clarity and organisation.

Collaboration: Modules facilitate collaboration among teams by promoting standardisation and sharing of best practices. Teams can develop and share modules internally or leverage community-contributed modules from the Terraform Registry.

Practical Examples

In this article, I propose the creation of an AWS EC2 Instance Module. By doing so, we can develop a straightforward Terraform module designed to facilitate the provisioning of an AWS EC2 instance. This module will streamline the process of deploying and managing EC2 instances within our infrastructure, enhancing efficiency and scalability in our cloud environment.

Step 1

To begin, we'll generate a main.tf file and populate it with the following configurations. The main.tf file serves as the primary configuration file in Terraform, where we define the desired state of our infrastructure using declarative code. In this file, we specify the resources and configurations, such as AWS EC2 instances, networking components, and any other infrastructure elements required for our project. By organising our infrastructure specifications within main.tf, we establish a clear and concise blueprint for deploying and managing resources effectively using Terraform's infrastructure as code paradigm.

touch main.tf

// main.tf

variable "instance_type" {
  description = "EC2 instance type"
  default     = "t2.micro"
}

variable "ami" {
  description = "AMI for the EC2 instance"
}

resource "aws_instance" "ec2_instance" {
  ami           = var.ami
  instance_type = var.instance_type
}

Next, we'll proceed to create our variables.tf file and insert the provided content. The variables.tf file in Terraform serves a crucial role in defining variables that can be used across our Terraform configuration files. By centralising variable definitions in this file, we promote reusability, maintainability, and consistency in our infrastructure code. These variables can represent various parameters, such as AWS region, instance types, key pairs, or any other configurable values required for our Terraform deployment. Utilising variables enhances the flexibility and scalability of our infrastructure, allowing for easier customisation and adaptation to changing requirements.

touch variables.tf

// variables.tf

variable "instance_type" {}
variable "ami" {}

Now, let's proceed by creating an outputs.tf file and incorporating the provided directives. In Terraform, the outputs.tf file plays a crucial role in defining the outputs or results of our infrastructure deployment. These outputs can include essential information or attributes of deployed resources that we may need to reference or utilise externally. By declaring outputs in this file, we enable easier access to important data about our infrastructure, such as public IP addresses, DNS names, or any other relevant details. This facilitates integration with other systems or processes and enhances the overall observability and usability of our Terraform-managed infrastructure.

touch outputs.ft

// outputs.tf

output "public_ip" {
  value = aws_instance.ec2_instance.public_ip
}

Step 2

Let's explore how to integrate and utilise the EC2 Instance Module we've developed within a separate Terraform configuration. This process of consuming the module involves referencing and incorporating it into another Terraform configuration file to provision EC2 instances efficiently and consistently across our infrastructure. By leveraging modularisation, we promote code reuse, maintainability, and scalability in our Terraform projects. This approach allows us to streamline the deployment of EC2 instances while maintaining flexibility and standardisation throughout our infrastructure provisioning workflows.

To integrate our EC2 Instance Module into a separate Terraform configuration, we must initiate the process by generating a new main.tf file. This file will serve as the primary configuration blueprint for orchestrating the consumption of our module and orchestrating the deployment of EC2 instances. By encapsulating our module consumption logic within this new main.tf file, we establish a clear separation of concerns and ensure modularity and maintainability within our Terraform project. This approach facilitates the seamless integration of our module into existing or future infrastructure deployments, enabling efficient management and provisioning of EC2 instances across our environment.

touch main.tf

Let's proceed by incorporating the following script into the freshly generated main.tf file. This file serves as the central configuration hub for orchestrating various Terraform resources and operations. By appending these script, we define the specific actions and resources required to achieve our infrastructure provisioning goals. This step contributes to the comprehensive definition of our Terraform configuration, ensuring that all necessary components are accurately specified and configured. Additionally, it establishes a clear roadmap for the Terraform engine to follow when executing our infrastructure deployment plan, enhancing the predictability and reliability of the provisioning process.

// main.tf

module "ec2_instance" {
  source  = "./ec2_instance_module"
  ami     = "ami-0c55b159cbfafe1f0"
}

Advanced Usage and Best Practices

Parameterisation and Input Validation:

Terraform modules support parameterisation, allowing users to customise module behaviour through input variables. It's essential to define clear input variable descriptions and provide sensible defaults for improved usability. Additionally, input validation can be implemented using conditional expressions or validation rules to ensure the correctness of input values.

Composition and Nesting:

Modules can be composed and nested to create complex infrastructure configurations. By combining smaller, reusable modules, teams can construct sophisticated architectures while maintaining modularity and encapsulation. However, it's crucial to strike a balance between granularity and complexity to avoid overly nested structures.

Versioning and Dependency Management:

Versioning is critical for managing module dependencies and ensuring reproducibility across environments. Terraform modules can be versioned using version control systems like Git or by publishing modules to the Terraform Registry. Semantic versioning practices should be followed to communicate changes effectively and prevent unexpected modifications.

Testing and Continuous Integration:

Testing Terraform modules is essential to validate functionality, detect errors, and ensure reliability. Automated testing frameworks such as Terratest or integration with continuous integration (CI) pipelines can help automate testing processes and provide early feedback on module changes. Additionally, infrastructure changes should undergo rigorous testing in staging environments before deployment to production.

Conclusion

Terraform modules serve as indispensable tools for managing infrastructure configurations effectively. By encapsulating infrastructure components into reusable modules, teams can achieve greater consistency, scalability, and collaboration in their infrastructure management practices. With a solid understanding of Terraform modules and their benefits, teams can streamline their infrastructure provisioning workflows and unlock the full potential of Infrastructure as Code. Embracing advanced usage patterns and best practices empowers teams to build resilient, scalable, and maintainable infrastructure architectures with Terraform.

Terraform in Plain English

Oloruntobi Olurombi — Tue, 30 Jan 2024 08:33:17 +0000

In the fast-paced world of cloud computing, managing infrastructure efficiently is a must. Enter Terraform – a superhero in the realm of Infrastructure as Code (IaC). In this practical guide, I'll take a high-level look at how Terraform can simplify your life by allowing you to define, deploy, and manage your cloud infrastructure using straightforward code.

What is Terraform?

Terraform is like a magic wand for cloud infrastructure. It lets you write down what you want your infrastructure to look like, and then it makes it happen. No more clicking around in various dashboards – just code your wishes, and Terraform does the heavy lifting.

Declarative Magic:

With Terraform, you don't need to worry about the nitty-gritty details of how to get things done. Instead of scripting out every step, you declare what you want, and Terraform figures out the best way to make it happen. It's like having a personal assistant for your infrastructure tasks.

Multi-Cloud Flexibility:

Say goodbye to being tied down to a single cloud provider. Terraform lets you play the field by supporting multiple cloud providers. Whether it's AWS, Azure, Google Cloud, or others – Terraform has your back. You can manage all your clouds using the same familiar Terraform language.

Building Blocks with Modules:

Ever wish you could reuse bits of your infrastructure like Lego blocks? Well, with Terraform modules, you can! Modules are like pre-built Lego sets – they encapsulate a bunch of resources and configurations. Need a web server? There's a module for that. Database? Yep, there's one for that too. It's code you don't have to write.

Code Version Control:

Imagine having an undo button for your infrastructure changes. With Terraform, you can store your infrastructure code in a version control system like Git. This means you can roll back changes, see who did what, and collaborate with your team without breaking a sweat.

Multi-Environment Simplicity:

Going from development to testing to production has never been smoother. With Terraform, you can use the same code for different environments. It's like having a blueprint for your infrastructure that works seamlessly, no matter where it's deployed.

Say Goodbye to Snowflake Servers:

Terraform encourages the idea of 'immutable infrastructure.' Instead of tinkering with existing servers, you just replace them with new ones when changes are needed. This not only keeps your infrastructure consistent but also makes it easier to recover from any mishaps.

Scalability Made Simple:

As your business grows, so does your infrastructure. Terraform makes scaling a breeze. Need more servers? Just tweak your code, and Terraform handles the rest. It's the secret sauce for keeping up with the demands of a rapidly expanding digital landscape.

Collaborate with Confidence:

Terraform brings teams together. With a shared language and consistent configurations, collaboration becomes seamless. No more confusion about who did what – Terraform makes it clear, ensuring that everyone is on the same page when it comes to managing your infrastructure.

Conclusion:

Terraform is not just for the tech wizards; it's for anyone who wants to simplify their cloud infrastructure management. With its user-friendly approach, multi-cloud support, reusable modules, time-saving features, scalability, and collaborative benefits, Terraform is the key to turning your infrastructure dreams into reality – without the headache. Ready to make your cloud journey a breeze? It's time to embrace the magic of Terraform. Your infrastructure will thank you.

Streamlining DevOps Workflows: Integrating Atlassian BitBucket and Slack for Seamless Pipelines and Docker Management

Oloruntobi Olurombi — Mon, 11 Sep 2023 17:17:31 +0000

In today's rapidly evolving software development landscape, the successful execution of DevOps practices has become instrumental in delivering high-quality software at speed. DevOps, which is a fusion of development and operations, emphasises collaboration, automation, and continuous integration/continuous deployment (CI/CD) pipelines to streamline the software development lifecycle.

To achieve this seamless workflow, organisations are increasingly turning to robust tools like Atlassian BitBucket for version control, collaboration, and code deployment. Additionally, Slack has emerged as a leading communication and collaboration platform, providing teams with a central hub for real-time communication and integrations with various development tools.

In this comprehensive guide, we will explore how to harness the power of Atlassian BitBucket and Slack to create a harmonious environment for managing CI/CD pipelines and Docker containers. This integration will empower your development and operations teams to collaborate effectively, automate workflows, and deploy code with confidence.

Accessing Your BitBucket Dashboard

Our journey begins on the BitBucket dashboard, the central control panel for managing repositories, code, and collaborations. If you don't already have a BitBucket account, you can sign up for one on their website. Once you're logged in, follow these steps:

Creating a New Repository:

Click on Create a new repository

On the Create a new repository do the following:

A. Project name: Give your project a meaningful name to categorize your work effectively.
B. Repository name: Provide a unique name for the repository, reflecting the project or application.
C. Click on the "Create Repository" button to finalize the setup.

D. Confirm that the repository was created successfully.

Setting Up Slack

Effective communication and collaboration are paramount in DevOps, and Slack provides a perfect platform for real-time interactions. Here's how to set up a Slack workspace:

Go to the Slack website and click on the "Try For Free" button.

Provide your email address and click "Continue".
On the "Create a New Slack Workspace" page, click the "Create a Workspace" button.
Provide a name for your company or team.

Next, add a workspace name and click "Next".

While you can invite team members, for this article's sake, let's skip this step and click "Next".

Provide information about what your team is currently working on.

Confirm that your workspace is now active.

Create a channel named "devops-engineering".

Connecting BitBucket to Slack

With your BitBucket repository and Slack workspace in place, it's time to connect the two for seamless collaboration and automated notifications:

Return to your BitBucket repository page and click on "Repository settings."

Under the "Slack" section, click on the "Connect" button.

Allow BitBucket Cloud to access your Slack workspace by clicking "Allow".

Add a chat subscription by selecting your workspace and channel. Click the "Add" button.

Confirm the subscription.

Cloning Your BitBucket Repository

To start working with your code locally, you'll need to clone the BitBucket repository to your local machine:

On the BitBucket repository page, click "Clone" and copy the repository URL link.

Use the repository URL link to run git clone in your command line.

git clone repo_url

Navigate into the cube360 directory.

cd cube360

Create a new file called index.html and add
Hello World!!!

touch index.html

<p>Hello World!!!</p>

Add index.html to the staging area with git add index.html.

git add index.html

Run git status to check the status of your repository.

git status

Commit the changes with git commit -m "Initial commit of project Hello World!!".

git commit -m "Initial commit of project Hello World!!"

Push your code to BitBucket with git push.

git push

Check the "devops-engineering" channel in Slack, and you should see an alert about the push action.

To further test the integration, add
This is a great DevOps Collaboration feature
to index.html.

<p>This is a great DevOps Collaboration feature</p>

Run git add * and git commit -m "Added more code".

git add * and git commit -m "Added more code"

Push your changes to BitBucket with git push.

git push

BitBucket Pipelines for Building and Deployment

Now, let's delve into the heart of DevOps - automation. BitBucket Pipelines is a robust CI/CD solution that enables automated building, testing, and deployment of your code. We'll start by configuring the necessary files and directories:

Return to your BitBucket repository dashboard and select the "Pipelines" option.

On your local machine, create the following files and directories:

A. Create a file called "Dockerfile" with touch Dockerfile.

touch Dockerfile

B. Create another file named "bitbucket-pipelines.yml" with touch bitbucket-pipelines.yaml.

touch bitbucket-pipelines.yaml

C. Create a new file named "nginx.conf" with touch nginx.conf.

touch nginx.conf

D. Create a directory called "www-data" with mkdir www-data.

mkdir www-data

E. Move index.html into the "www-data/" directory with mv index.html www-data/.

mv index.html www-data/

Add, commit, and push your changes to BitBucket with the following commands:

git add .

git status

git commit -m "Updated with Docker code"

git push

Now confirm the recent changes on Bitbucket repository dashboard.

Now, update the files on your local machine and commit them to your remote repository:

A. Update your Dockerfile.

# docker pull xxxxxxxx/yyyyyyy:zzzzzzzzz
# docker run --name nginx-webserver -d -p 8080:80 xxxxxxxx/yyyyyyy:zzzzzzzzz

FROM ubuntu:14.04
RUN apt-get update
RUN apt-get install -y nginx
ADD nginx.conf /etc/nginx/nginx.conf

ADD ./www-data /www-data

EXPOSE 80
CMD ["nginx"]

This is a Dockerfile, a text-based script used to define a Docker image. The Dockerfile describes how to build an image for running an NGINX web server using an Ubuntu 14.04 base image. Here's a breakdown of each part of the Dockerfile:

Base Image Selection:

   FROM ubuntu:14.04

This line specifies the base image for the new image being built. In this case, it uses the "ubuntu:14.04" image as the starting point. The resulting image will have Ubuntu 14.04 as its base operating system.

Package Installation:

   RUN apt-get update
   RUN apt-get install -y nginx

These lines use the RUN instruction to execute commands within the image during the build process.

The first command, apt-get update, updates the package repositories to ensure the package manager (APT) has the latest package information.

The second command, apt-get install -y nginx, installs the NGINX web server. The -y flag automatically confirms any prompts during the installation.

Configuration File Addition:

   ADD nginx.conf /etc/nginx/nginx.conf

This line copies a file named "nginx.conf" from the context (the directory where the Dockerfile is located) into the image.

Specifically, it adds the configuration file to the "/etc/nginx/nginx.conf" location within the image. This is where NGINX looks for its configuration by default.

Directory Addition:

   ADD ./www-data /www-data

This line copies the contents of a directory named "www-data" from the context into the image.

It places the contents into the "/www-data" directory within the image.

Port Exposure:

   EXPOSE 80

This line informs Docker that the container will listen on port 80 at runtime. It doesn't actually publish the port, but it documents that the container expects external traffic on port 80.

Command Execution:

   CMD ["nginx"]

This line specifies the default command to run when a container based on this image is started.

It runs the "nginx" command, which starts the NGINX web server within the container.

B. Update your bitbucket-pipelines.yml file.

# This is a sample build configuration for Docker.
# Check our guides at https://confluence.atlassian.com/x/O1toN for more examples.
# Only use spaces to indent your .yml configuration.
# -----
# You can specify a custom docker image from Docker Hub as your build environment.
image: atlassian/default-image:2

pipelines:
  default:
    - step:
        name: Build NGINX Docker Image
        services:
          - docker
        script: # Modify the commands below to build your repository.
          # Set $DOCKER_HUB_USERNAME and $DOCKER_HUB_PASSWORD as environment variables in repository settings
          - export IMAGE_NAME=oloruntobi/my-nginx-image:$BITBUCKET_COMMIT

          # build the Docker image (this will use the Dockerfile in the root of the repo)
          - docker build -t $IMAGE_NAME .
          # authenticate with the Docker Hub registry
          - docker login --username $DOCKER_HUB_USERNAME --password $DOCKER_HUB_PASSWORD
          # push the new Docker image to the Docker registry
          - docker push $IMAGE_NAME

          - echo "finished building!"

This code is a configuration file written in YAML format for defining a build process using Docker within an Atlassian Bitbucket Pipelines. It describes how to build a Docker image for an NGINX web server and push it to a Docker registry, like Docker Hub. Here's a breakdown of the code:

Docker Image Configuration:

image: atlassian/default-image:2: This specifies the Docker image to be used as the build environment. In this case, it's using the "atlassian/default-image" version 2 image as the base image for running the build steps.

Pipelines Configuration:

pipelines: This section defines the pipelines for different stages of the build process. In this case, there's only one pipeline named "default."

Pipeline Step:

- step:: This indicates the start of a build step within the "default" pipeline.

Step Name:

name: Build NGINX Docker Image: This is the name of the build step and is used for identification and display purposes.

Service Configuration:

services:: This section specifies the services that should be available during the build step. In this case, it includes the "docker" service, indicating that Docker should be available to execute Docker-related commands.

Build Script:

script:: This section defines the sequence of commands to be executed during the build step. These commands will be run within the Docker image specified earlier.

Environment Variable Setup:

export IMAGE_NAME=oloruntobi/my-nginx-image:$BITBUCKET_COMMIT: This command sets an environment variable called IMAGE_NAME. It uses the repository name "oloruntobi/my-nginx-image" and appends the Bitbucket commit hash to create a unique name for the Docker image.

Docker Build:

docker build -t $IMAGE_NAME .: This command builds a Docker image using the Dockerfile located in the root of the repository. The -t flag assigns a tag (in this case, the value of IMAGE_NAME) to the image.

Docker Login:

docker login --username $DOCKER_HUB_USERNAME --password $DOCKER_HUB_PASSWORD: This command authenticates with the Docker Hub registry using the provided Docker Hub username and password. This authentication is necessary to push the Docker image to Docker Hub.

Docker Push:
- docker push $IMAGE_NAME: This command pushes the newly built Docker image to the Docker registry (in this case, Docker Hub). It uses the IMAGE_NAME variable as the image name and tag.

Echo Finished:
- echo "finished building!": This is a simple informational message that will be printed to the build log, indicating that the build process has finished.

C. Update the nginx.conf file.

daemon off;
pid /var/lib/nginx/nginx.pid;

events {
  worker_connections 1024;
}

http {
  include /etc/nginx/mime.types;
  index index.html;

  server {
    listen *:80;

    location / {
      root /www-data;
    }
  }
}

Let's break down the different sections and directives in this configuration:

Global Directives:

daemon off;: This directive controls whether NGINX runs in the background as a daemon. Setting it to "off" means NGINX will run in the foreground, which is useful for Docker containers or when running NGINX directly from the command line.

pid /var/lib/nginx/nginx.pid;: This directive specifies the path where NGINX will write its process ID (PID) file. This file contains the process ID of the running NGINX instance.

events Block:

events { ... }: This block contains event-related configuration directives.

worker_connections 1024;: This directive sets the maximum number of connections that a worker process can handle simultaneously. In this case, it's set to 1024 connections, which is a reasonable default value.

http Block:

http { ... }: This block defines HTTP-related configuration directives for NGINX.

include Directive:

include /etc/nginx/mime.types;: This directive includes the specified file, "/etc/nginx/mime.types," which contains a mapping of file extensions to MIME types. It's used to determine the MIME type of files served by NGINX.

index Directive:

index index.html;: This directive specifies the default file to serve when a directory is requested. In this case, if a directory is requested, NGINX will look for an "index.html" file and serve it if found.

server Block:

server { ... }: This block defines a server block, which represents an individual HTTP server in NGINX.
listen :80;: This directive specifies that the server should listen on port 80 for incoming HTTP requests. The asterisk () means it will listen on all available network interfaces.

location Block:

location / { ... }: This block defines a location block within the server block. It specifies how NGINX should handle requests that match the location pattern (in this case, the root path "/").

root /www-data;: This directive sets the root directory for serving content for this location. Requests to the root path ("/") will be served from the "/www-data" directory. This is where the web content is expected to be located.

In summary, this NGINX configuration sets up a basic HTTP server listening on port 80, serving content from the "/www-data" directory, and using "index.html" as the default file.

D. Use the following commands to add, commit, and push these changes:

git add .

git status

git commit -m "Adding more code to our repo"

git push

E. Next, configure environment variables in the repository settings:

Click on "Repository Settings.

Under "Repository Variables," create two variables named DOCKER_HUB_USERNAME and DOCKER_HUB_PASSWORD.

Update the bitbucket-pipelines.yml file by adding your Docker Hub username and image name.

Push the changes to your BitBucket repository.

git add .

git status

git commit -m "Adding more code to our repo"

git push

Go to the Pipelines page on your BitBucket repository and confirm that the pipeline passes successfully. Additionally, check Docker Hub to verify that your artifact is available.

F. Now, let's move forward:

Start the Docker daemon via Docker Desktop.

Run your container with the following steps: a. Check for containers on your local machine with docker ps -a

docker ps -a

b. Pull your image from Docker Hub with docker pull username/my-nginx-image:tag

docker pull username/my-nginx-image:tag

c. Start your container with docker run --name nginx-webserver -d -p 8080:80 username/my-nginx-image:tag

docker run --name nginx-webserver -d -p 8080:80 username/my-nginx-image:tag

d. Confirm the status of the container with docker ps

docker ps

e. Test your application by running curl localhost:8080

curl localhost:8080

Slack Notifications for BitBucket Pipelines

To receive notifications in Slack when your pipeline succeeds or fails, follow these steps:

Return to the BitBucket repository page and click on "Repository Settings".

Under the "SLACK" section, click on the "Settings" button.

Add "pipeline failed" and "pipeline succeeded" options under the master section.

Add a new line to your index.html page:
This DevOps is DevOpsing by automation!
.

<p>This DevOps is DevOpsing by automation!</p>

Push the change to BitBucket.

git add .

git status

git commit -m "adding more code to index.html file"

git push

Confirm by checking the Slack channel and BitBucket Pipelines page.

Conclusion

By integrating Atlassian BitBucket and Slack into your DevOps workflow, you've taken a significant step toward streamlining your development process. This comprehensive integration enables you to automate code deployments, manage Docker containers efficiently, and receive real-time notifications on pipeline statuses.

In the fast-paced world of software development, where collaboration and automation are key, this integration empowers your team to deliver code more swiftly and reliably. It fosters a culture of collaboration and transparency, allowing development and operations teams to work together seamlessly.

As you embark on your DevOps journey with BitBucket and Slack, remember that this is just the beginning. Continuous improvement and adaptation to your team's unique needs will further enhance your DevOps practices, helping you deliver exceptional software with speed and confidence. Happy DevOps-ing!

Unlocking the Power of Amazon RDS Database Backups: Ensuring Data Resilience and Business Continuity

Oloruntobi Olurombi — Mon, 04 Sep 2023 18:54:15 +0000

Data is the lifeblood of modern businesses, and safeguarding it against potential disasters is paramount. Amazon RDS (Relational Database Service) offers a robust backup solution that ensures your data remains secure and accessible even in the face of unexpected events. In this article, we'll explore how to perform manual backups using Amazon RDS and dive into the steps for both the AWS console and the command line interface (CLI).

Manual Backups: A Step-by-Step Guide

Amazon RDS provides a straightforward way to create manual backups of your databases, whether you prefer to use the AWS Management Console or the command-line interface (CLI). Here's a detailed walkthrough of how to create manual backups using both methods:

Using the AWS Management Console:

Navigate to Amazon RDS

To initiate a manual backup via the AWS console, start by navigating to the Amazon RDS page on the AWS Management Console.

Select the "Databases" option to proceed.

Choose the Database

From the list of available databases, select the specific database that you wish to back up.

Take Snapshot

Click on the "Actions" button and choose the "Take Snapshot" option. This action triggers the backup process.

Snapshot Configuration

DB Instance: Select the appropriate DB instance for which you want to create the backup.

Snapshot Name: Give your snapshot a meaningful name. This helps in identifying it later.

After configuring these details, click on the "Take Snapshot" button to initiate the backup process.

Confirmation

Once the snapshot creation process begins, you will receive a confirmation message indicating that the snapshot is being created. You can track the progress of the backup from the RDS console.

Manual Backups via AWS CLI

Amazon RDS also provides the flexibility of creating manual backups through the command line interface (CLI). Here's how you can achieve this:

AWS CLI Installation: Ensure that you have the AWS CLI installed and configured with the necessary credentials.

Execute Command: Use the following command to create a manual backup via the CLI:

aws rds create-db-snapshot --db-snapshot-identifier CLISnapshot --db-instance-identifier my-db-instance

Replace CLISnapshot with your desired snapshot name.

Replace my-db-instance with the name of the DB instance for which you want to create the backup.

Executing this command will initiate the backup process programmatically.

Why Manual Backups Matter:

Manual backups play a crucial role in data resilience and business continuity. They allow you to take snapshots of your database at specific points in time, preserving your data in a known state. These backups are invaluable for:

Disaster Recovery: In case of data corruption, accidental deletion, or system failures, manual backups provide a reliable restore point.

Testing and Development: Manual backups are useful for creating a copy of your database for testing, development, or analytics without affecting the production environment.

Data Retention Policies: They enable you to meet compliance requirements by retaining historical data.

In conclusion, Amazon RDS database backups, whether performed through the AWS console or CLI, are a vital component of a comprehensive data protection strategy. By creating manual backups, you can safeguard your data, ensuring that your business remains resilient and prepared for any unexpected challenges.

Mastering the Art of Establishing an Amazon RDS Database: A Comprehensive Guide to Setting Up and Optimising

Oloruntobi Olurombi — Mon, 28 Aug 2023 13:07:00 +0000

Amazon Web Services (AWS) provides a vast array of services that empower businesses to deploy, manage, and scale their applications with ease. Among these, Amazon RDS (Relational Database Service) offers a convenient solution for creating, operating, and scaling relational databases in the cloud. In this guide, we will walk you through the step-by-step process of creating an Amazon RDS database, utilising MySQL as the engine, and configuring it for optimal performance.

Accessing the AWS Console and Searching for RDS

Log in to your AWS account and navigate to the AWS Management Console. In the search bar, type "RDS" to access the Amazon RDS dashboard.

Creating the Database

a. Once on the Amazon RDS dashboard, locate the option to "Create Database" and click on it.

b. In the "Choose a database creation method" section, select "Standard create."

c. Under "Engine options," choose "MySQL" as the database engine.

d. For "Templates," opt for the "Production" template, which comes with configurations suitable for production workloads.

e. In the "Availability and durability" section, select "Single DB instance" for deployment options.

f. Configure the following settings:

DB cluster identifier: Set it to a descriptive name like "database-1".
Master username: Choose a username, often "admin" for administrative access.
Master password: Enter a strong and secure password for the master account.

g. Under "Instance configuration," choose the appropriate DB instance class from the Standard class options (including m classes), based on your performance requirements.

h. Configure storage option following these settings:

Storage type: Choose "General Purpose SSD (gp3)" for a balance of performance and cost.
Storage autoscaling: Enable this feature by clicking "Enable storage autoscaling".
Maximum storage threshold: Set the maximum storage threshold to 100 GB.

i. For "Connectivity," leave the default settings as they are. These settings include networking details like VPC, subnet group, and publicly accessible options.

j. Similarly, leave the "Database authentication" settings at their default configuration.

k. In the "Additional configuration" section, provide an initial database name like "MyDB".

l. Once you've configured all the necessary settings, click on the "Create database" button to initiate the creation process.

Verification

After clicking "Create database," AWS will begin provisioning your RDS instance. Once the process is complete, navigate to the RDS dashboard and verify that your newly created database, named "database-1," exists and is available.

Congratulations! You've successfully created an Amazon RDS database using MySQL as the engine. This comprehensive guide has taken you through each step, from selecting engine options to configuring storage and verifying the database's existence. With your newly established RDS database, you're well on your way to managing your data efficiently and securely within the AWS ecosystem.

Feel free to connect with me on Linkedin

DEV Community: Oloruntobi Olurombi

Step-by-Step Guide: Deploying a Static Web Application in OpenShift Using a Custom S2I Builder Image

Prerequisites

Part 1: Verify OpenShift Setup

Step 1: Open the Terminal

Part 2: Create a Custom S2I Builder Image

Step 2: Define the S2I Repository

Step 3: Create the Builder Image

Step 4: Verify the Build

Part 3: Deploy the Static Web Application

Step 6: Monitor the Build Process

Part 4: Expose the Application to External Traffic

Step 7: Expose the Service

Step 8: Update the Route

Export and Edit the Route Configuration

Part 5: Test the Application

Step 9: Verify Connectivity

Step 10: Access the Application

Summary

Deploying Jenkins on Kubernetes using Helm

Why Helm?

Step 1: Creating a Helm Chart

Helm Chart Directory Structure

Step 2: Update Chart.yaml

Step 3: Clean Up Unnecessary Files

Step 4: Create Deployment and Service Manifests

1. Create Jenkins Deployment Manifest

2. Create Jenkins Service Manifest

Step 5: Update NOTES.txt

Step 6: Update values.yaml

Step 7: Run the Helm Template Command

Step 8: Install the Helm Chart

Step 9: Get Public IP of the Node and Access Jenkins

Step 10: Access Jenkins Console

Conclusion

End-to-End Deployment and Monitoring of EKS and Flask Apps with Terraform, GitHub Actions, Helm, Prometheus and Grafana

Table of Contents

Overview

Why Terraform?

Why GitHub Actions?

Prerequisites

Infrastructure Automation: EKS Cluster Setup

Terraform Setup

EKS Cluster and Node Group (main.tf):

Networking (vpc.tf):

IAM Roles (iam.tf):

CloudWatch and Monitoring (cloudwatch.tf):

AutoScaler IAM (iam-autoscaler.tf):

Routing (security_groups.tf)

Variables (variables.tf)

Provider (provider.tf)

IAM OpenID (oidc.tf)

GitHub Actions Workflow setup: eks-setup.yaml

Purpose:

Steps:

Flask App and Docker Setup:

Step 1: Set Up the Flask Application

Step 2: Set Up Docker and Push the Image to Amazon Elastic Container Registry (ECR)

Setup Kubernetes Deployment and Service:

Explanation of the Configuration

Deploying the Flask Application

Required GitHub Secrets

Adding Secrets to GitHub:

CI/CD Pipeline Process (regtech-app.yaml)

Monitoring with Prometheus and Grafana

Prometheus

Grafana: Visualizing and Monitoring with Dashboard

Conclusion

gggg

fff

Table of Contents

Overview

Why Terraform?

Why GitHub Actions?

Prerequisites

Infrastructure Automation: EKS Cluster Setup

Terraform Setup

EKS Cluster and Node Group (main.tf):

Networking (vpc.tf):

IAM Roles (iam.tf):

Step 6: Update `values.yaml`

Routing (`security_groups.tf`)

Variables (`variables.tf`)

Provider (`provider.tf`)

IAM OpenID (`oidc.tf`)

GitHub Actions Workflow setup: `eks-setup.yaml`

Routing (`security_groups.tf`)

Variables (`variables.tf`)

Provider (`provider.tf`)

IAM OpenID (`oidc.tf`)