Challa Parthasaradi

Posted on Jun 5

Implementation of EC2-based Application with Auto Scaling, CI/CD, and Enhanced Monitoring

#aws #ec2 #codepipeline #rds

1. Executive Summary

This document outlines the steps to implement an EC2-based application using Auto Scaling, CI/CD, and enhanced monitoring. The goal is to create a scalable, high-availability infrastructure for a web application, leveraging AWS services such as EC2, RDS, CloudWatch, Code Deploy, and Auto Scaling.

2. Infrastructure Overview

Architecture Diagram

2.1 Components List

EC2 Instances: For hosting the web application.
Amazon RDS: Provides managed database services.
AWS Auto Scaling: Ensures application scalability and availability.
CloudWatch and CloudTrail: Used for monitoring and logging.
AWS Backup: Handles backups and disaster recovery.
CI/CD Pipeline: Automates application deployment.

2.2 Key Design Decisions

Scalability: Achieved through Auto Scaling.
High Availability: Implemented with multi-AZ deployments.
Security: Enhanced with IAM roles, security groups.

1. Networking

Create a VPC
Go to the AWS Management Console.
Navigate to VPC > Your VPCs.
Click Create VPC:
Name tag: Mumbai-VPC
IPv4 CIDR block: 10.10.0.0/16
Leave other options as default and click Create.

2. Create Subnets

Have 4 subnets: 2 public and 2 privates. Ensure subnets are in different availability zones for high availability.

Public Subnet 1

Go to Subnets and click Create Subnet.
Name tag: Public-Subnet-1
VPC ID: Select Mumbai-VPC.
Availability Zone: ap-south-1a (or any zone in Mumbai).
IPv4 CIDR block: 10.10.0.0/24
Click Create.

Public Subnet 2

Click Create Subnet again.
Name tag: Public-Subnet-2
VPC ID: Select Mumbai-VPC.
Availability Zone: ap-south-1b.
IPv4 CIDR block: 10.10.1.0/24
Click Create.

Private Subnet 1

Click Create Subnet again.
Name tag: Private-Subnet-1
VPC ID: Select Mumbai-VPC.
Availability Zone: ap-south-1a.
IPv4 CIDR block: 10.10.2.0/24
Click Create.

Private Subnet 2

Click Create Subnet again.
Name tag: Private-Subnet-2
VPC ID: Select Mumbai-VPC.
Availability Zone: ap-south-1b.
IPv4 CIDR block: 10.10.3.0/24
Click Create.

3. Create an Internet Gateway

Go to Internet Gateways and click Create Internet Gateway.
Name tag: Mumbai-IGW.
Attach it to the VPC:
Select Mumbai-VPC.
Click Attach Internet Gateway

4. Create a NAT Gateway

Go to NAT Gateways and click Create NAT Gateway.
Name tag: Mumbai-NAT.
Subnet: Select one of the public subnets (EX.Public-Subnet-1).
Elastic IP: Allocate a new Elastic IP.
Click Create NAT Gateway.

5. Create Route Tables

You need two route tables: one public and one private.

Public Route Table

Go to Route Tables and click Create Route Table.
Name tag: Public-RT.
VPC: Select Mumbai-VPC.
Click Create.
Add a route:
Go to Routes and click Edit routes.
Destination: 0.0.0.0/0
Target: Select the Internet Gateway (Mumbai-IGW).
Click Save changes.
Associate public subnets:
Go to Subnet Associations, click Edit Subnet Associations.
Select Public-Subnet-1 and Public-Subnet-2.
Click Save associations.

Private Route Table

Go to Route Tables and click Create Route Table.
Name tag: Private-RT.
VPC: Select Mumbai-VPC.
Click Create.
Add a route:
Go to Routes and click Edit routes.
Destination: 0.0.0.0/0
Target: Select the NAT Gateway (Mumbai-NAT).
Click Save changes.
Associate private subnets:
Go to Subnet Associations, click Edit Subnet Associations.
Select Private-Subnet-1 and Private-Subnet-2.
Click Save associations.

Finaly VPC Resource Map.

2.IAM role We need have a Two role one is EC2 and another one is deployment

1.Create the Role for EC2

1.Go to IAM:

Open the IAM Management Console.
Click Roles in the navigation pane.

2.Create a New Role:

Click Create role.
Select AWS service as the trusted entity type.
Choose EC2 as the service that will use the role.
Click Next.

3.Attach Policy:

Search for AmazonEC2RoleforAWSCodeDeploy,AmazonSSMFullAccess, CloudWatchActionsEC2Access in the list of policies.
Select it by checking the box.
Click Next.

4.Name the Role:

Enter a name for the role (EX. EC2CodeDeployRole).
Review the settings and click Create role.

2.Create the Role for Code Deploy

1.Go to IAM:

Open the IAM Management Console.
Click Roles in the navigation pane.

2.Create a New Role:

Click Create role.
Select AWS service as the trusted entity type.
Choose Codedeploy as the service that will use the role.
Click Next.

3.Attach Policy:

Search for AWSCodeDeployRole and AmazonS3FullAccess in the list of policies.
Select both policies by checking their boxes.
Click Next.

4.Name the Role:

Enter a name for the role (EX. CodeDeployRoleWithS3).
Review the settings and click Create role.

3.Launch Template Creation

Navigate to the EC2 Dashboard in the AWS Management Console.
Select Launch Templates from the left-hand menu.
Click Create Launch Template.
Configure the Launch Template
Launch Template Name: Enter a unique name (EX. AutoScalingTemplate).
AMI: Select Amazon Linux 2 AMI (HVM).
Instance Type: Choose t3.micro.
Key Pair: Select an existing key pair or create one if required.
IAM Instance Profile: Select EC2CodeDeployRole (or your specific IAM role).

2 Add the User Data Script

#!/bin/bash
# Script to install aws cli and codedeploy agent
# Install aws cli
$ yum install pip -y
$ pip install awscli
# Install codedeploy agent
$ yum update
$ yum install ruby wget -y
$ cd /home/ec2-user
$ wget https://aws-codedeploy-ap-south-1.s3.ap-south-1.amazonaws.com/latest/install
$ chmod +x ./install
$ ./install auto
$ sudo service codedeploy-agent start
$ sudo service codedeploy-agent status
# Install httpd
$ yum install httpd -y
$ echo "<h1>Hello From EC2: v1.0</h1>" > /var/www/html/index.html
$ service httpd start; chkconfig httpd on

Explanation:

The script installs the AWS CLI, CodeDeploy agent, and Apache HTTP server (httpd).
It creates an HTML page to verify the deployment ("Hello From EC2: v1.0").
It starts the httpd service to serve the page and configures it to start on boot.

3 Finalize Launch Template.

Monitoring and Other Settings: Adjust settings as necessary, but the default options should be fine for most use cases.
Click Create Launch Template.

4.My target group configuration

Create Target Group

Navigate to EC2 Dashboard
Go to the EC2 Dashboard in the AWS Management Console.
Go to Target Groups
In the left-hand navigation panel, scroll down to Load Balancing and click on Target Groups. - Create Target Group
Click the Create target group button.
Select the following options:
Target type: Choose Instances (for EC2 instances).
Protocol: Choose HTTP or HTTPS
Port: Set to 80 for HTTP or 443 for HTTPS.
VPC: Select the VPC.
Health check protocol: Choose HTTP.
Health check path: Set to / or the relevant path to check health (EX. /health).
Create the Target Group
Click on Create to finalize the creation of the Target Group.

5. Application load balancer configuration.

Create Application Load Balancer (ALB)

1.Navigate to EC2 Dashboard

Go to the EC2 Dashboard in the AWS Management Console.

2.Create Load Balancer

In the left-hand navigation pane, under Load Balancing, click on Load balancers.
Click Create Load Balancer.
Choose Application Load Balancer.

3.Configure Load Balancer

Name: Give your ALB a unique name (EX. MyALB).
Scheme: Choose internet-facing to make the ALB publicly accessible.
IP address type: Choose ipv4.
Listeners: HTTP on port 80 is created. You can also add an HTTPS listener if needed (you would need an SSL certificate) you can add HTTPS ports 443.

4.Configure Availability Zones

Select Availability Zones and choose public subnets in each desired availability zone. These public subnets will route external traffic to your ALB.

5.Security Group

Assign a security group that allows inbound traffic on the listener ports (EX.. HTTP port 80 or HTTPS port 443) from the public internet.
Ensure your security group allows inbound access for HTTP/HTTPS.

6.Create Load Balancer

Click Create Load Balancer to finalize the ALB creation.

6.Create Auto Scaling Group

Navigate to Auto Scaling Groups in the EC2 Dashboard.
Click Create Auto Scaling Group.

1.Configure the Auto Scaling Group

Auto Scaling Group Name: Enter a name (EX. Partha-auto).
Launch Template: Select the Launch Template (EX, AutoScalingTemplate) created earlier.
VPC and Subnets: Choose the private subnets where the instances will run.

2.Configure Group Size

Minimum Instances: Set to 1.
Maximum Instances: Set to 5.
Desired Instances: Set to 1.

3.Configure Load Balancer Integration

Choose Attach to a Load Balancer.
Select Application Load Balancer (ALB).
Listener Protocol: Select HTTP or HTTPS.
Target Group: Select the Target Group created with private subnets.
Health Check Path: Set to / or the relevant health check path.

4.Configure Scaling Policies

Scaling Policy Type: Select Target Tracking Scaling Policy.
Metric: Choose Average CPU Utilization or Average Memory Utilization.
Target Value: Set to 75% for CPU or a suitable value for memory.
Instance Warm-up: Set to 300 seconds (5 minutes).

5.Health Checks

EC2 Health Checks: Enable to ensure instances are healthy before scaling.
Enable ELB Health Checks (optional): If using the Load Balancer, enable this to ensure the instances are healthy before routing traffic.

6.Finalize Auto Scaling Group

Review your settings and click Create Auto Scaling Group.

Add the Lifecycle hooks.

Ec2 instance server with Private

7.Public OpenVPN Server Setup

1 Launch an OpenVPN Server

Create a new EC2 instance in a public subnet:
Use an Amazon Linux 2 AMI.
Instance Type: t3.micro.
Security Group: it is selected default.

2.Configure OpenVPN by following the OpenVPN setup guide document.

3.Assign an Elastic IP to the instance for public accessibility.

OpenVPN connection

When I have launched the private server after that i can took Ip address then while i can paste it the browser it could showing like this small web site is open.

8.Create an Amazon RDS MySQL Instance

1.Navigate to RDS Dashboard

In the AWS Management Console, navigate to RDS under Services.

2.Create a New Database

Click on Create database.
Database Creation Method: Select Standard Create.

3.Choose Database Engine

Choose MySQL as the database engine.

4.Set Database Version

Select the MySQL version you want to use. It’s recommended to choose the latest stable version.

5.Instance Configuration

DB Instance Class: Select db.t3.micro (for the smaller instance type).
Storage: Configure the allocated storage (default of 20 GiB should suffice for testing purposes).
Multi-AZ Deployment: Enable Multi-AZ Deployment to ensure high availability.
DB Instance Identifier: Give your database instance a name (Ex. MyRDSInstance).
Master Username: Set a Master Username (EX. admin).
Master Password: Set a Master Password and confirm it.

6.VPC and Subnet Configuration

VPC: Choose the VPC where your EC2 instance resides
Subnet Group: Choose a subnet group that includes private subnets to ensure that your RDS instance is deployed in a private subnet.
Public Accessibility: Set No since the RDS instance should not be publicly accessible.
VPC Security Group: Create a new security group or select an existing one. Make sure the security group allows traffic on port 3306 (MySQL).

7.Additional Configuration

Database Name: Optionally set a database name (EX. mydatabase).
Backup: Enable automated backups for your RDS instance (this is enabled by default).

8.Launch Database

Click Create database to launch the RDS instance.

10.Configure Security Group for RDS Access

1.Create a Security Group for EC2 Access

Go to the EC2 Dashboard and navigate to Security Groups under Network & Security.
Click Create Security Group.
Name: Name it something like EC2-to-RDS-SG.
Description: Add a description (EX.. "Security group allowing EC2 to access RDS").
VPC: Select the VPC where both EC2 and RDS are deployed.
Inbound Rules:
Type: Select MySQL/Aurora.
Protocol: TCP.
Port Range: 3306 (MySQL default port).
Source: Choose the Security Group ID.

Ec2 Instance Security Group

11.Create an AWS Backup

1.Go to AWS Backup in the AWS Management Console.
2.Click Create backup plan.
3.Choose Custom Backup Plan.
4.Name your backup plan (EX. DailyBackupPlan).
5.Set Backup frequency to Daily and choose a Backup window.
6.Under Backup Vault, select the default or create a new vault.

Assign Resources (EC2 & RDS & EBS)

1.In your backup plan, click Assign resources.
2.Choose EC2 and RDS as the resource types.
3.Select the specific EC2 instances and RDS databases you want to back up.
4.Set an IAM role (the default role will be automatically created).
5.Click Assign to confirm the resources.

Configure Retention Policy

1.In the backup plan, under Backup rule, set Retain backups for to 1 day.
2.This will automatically delete backups after 1 day.

Review & Save

1.Review your backup plan.
2.Click Create Plan to save and apply the plan.

Above screenshot EC2 Instance server and EBS volume Backup.

This is RDS database backup

Whenever database is deleted or terminate at the time you can restore the same database for RDS AWS backup.

11.Source Control Integration (GitHub with AWS Code Pipeline)

1.Go to AWS CodePipeline:

Navigate to AWS CodePipeline in the AWS Management Console.

2.Create a Pipeline:

Click on Create Pipeline to start the setup.

3.Configure Pipeline Settings:

Provide a Pipeline Name.
Choose or create a Service Role to allow CodePipeline to access AWS resources.

4.Source Stage Configuration:

Under the Source stage, select GitHub as the source provider.
Authenticate GitHub: Use OAuth or a Personal Access Token (PAT) to connect your GitHub account to AWS.
Repository: Select the repository that contains your application code.
Branch: Choose the branch (Ex.main) to monitor for changes.
Set the Output Artifact to store the source code in an S3 bucket.

5.Save: Once everything is configured, click Next.

12. Deployment Stage Setup (Using Code Deploy)

1.Add Deployment Stage:

After the Source stage, click on Add Stage and name it Deploy.

2.Select CodeDeploy as the Deployment Provider:

Choose AWS CodeDeploy as the deployment provider.

3.Create or Select CodeDeploy Application:

If you haven't already, create a CodeDeploy Application in the AWS Management Console.
Choose EC2/On-Premises as the compute platform.

4.Create or Select Deployment Group:

Create a Deployment Group that specifies Auto Scaling Group to deploy the code to.
Select the Target Deployment Group created earlier (or create a new one).

5.Define Deployment Configuration:

Choose the deployment configuration that suits your use case (EX. In-place)

6.Define Deployment Actions:

Make sure you have an appspec.yml file in your repository that contains deployment.

7.Save: Once configured, click Next and Deploy the application.

Revision S3 bucket

Whenever code deploy successful it will automatically store s3 bucket in zip file.

Code deploys stages.

Next go to the Load balancer DNS url then copy the url while open the browser and pasted we can see the web application.

Steps to Generate SSL Certificates with load balancer DNS

Step-1: create certificate in ACM.
Step-2: Create record set in route53 and assign DNS url address in record set and create domain to the website.

With Domain using for Load balancer DNS.

13.CloudTrail Setup

1.Navigate to CloudTrail:

Go to AWS Management Console.
In the Search bar, type CloudTrail and click on it.

2.Create a Trail:

In the CloudTrail Dashboard, click on Create trail.

3.Enable Trail in All Regions:

Choose Apply trail to all regions to enable CloudTrail in all regions.
This ensures that logs from every region are captured.

4.Configure S3 Bucket for Log Storage:

In the Storage Location section, choose Create new S3 bucket or select an existing S3 bucket to store the logs.
If creating a new bucket, provide a unique bucket name.
Enable log file encryption (optional, for added security).
Enable log file validation to ensure integrity.
Click Next.
5.Confirm and Create:
Review your CloudTrail settings and click Create to activate the trail.
Cloud trail logs screenshot

whenever you need to check cloud trails logs user activities for example if you want check server start and stop who has server start and stop you can check

14.CloudWatch Configuration

1.Custom Metrics

1.Navigate to CloudWatch:

Go to AWS Management Console.
In the Search bar, type CloudWatch and click on it.

2.Create Custom Metrics:

In the CloudWatch dashboard, click Metrics from the left-hand menu.
Click Create Metric.

3.Monitor CPU Utilization:

Select EC2 from the available metric namespaces.
Choose Per-Instance Metrics and select CPUUtilization.
Choose the EC2 instances you want to monitor and click Create.

4.Monitor Disk Space and Memory.

For monitoring disk space, you'll need to enable CloudWatch Agent on your EC2 instances.
Go to the EC2 instance and install the CloudWatch agent by running the following below commands.

$ sudo yum install amazon-cloudwatch-agent -y
$ cd /opt/aws/amazon-cloudwatch-agent/etc
$ sudo wget https://ibexcatalogapplication.s3.ap-southeast-2.amazonaws.com/instancemetrics.json
$ sudo chmod 777 instancemetrics.json
$ sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a append-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/etc/instancemetrics.json
$ systemctl restart amazon-cloudwatch-agent.service

2.Alarm Creation

1.Create CloudWatch Alarms:

Go to CloudWatch Dashboard in the AWS Management Console.
Under the Alarms section in the left-hand menu, click Create Alarm.

2.Choose Metric:

Select the metric you want to monitor (EX. CPUUtilization, DiskSpaceUtilization, MemoryUsage).
If it's a custom metric, go to Custom Metrics and select the one you've created.

3.Set Alarm Threshold:

Select Threshold type: Static or Anomaly detection.
For a static threshold:
Choose Greater than a specific value.
Set the threshold value (EX. CPU > 80%).
Set the period (EX. 5 minutes).
Choose Statistic (EX. Average, Maximum).

4.Set Actions:

Define what happens when the alarm is triggered. You can:
Send a notification to an SNS topic (EX. email).
Auto-scaling actions (EX. Add instance).
Click Create Topic to create a new SNS topic for notifications.
If you have already created SNS then you can select.

5.Configure Alarm:

Name your alarm (EX. "High CPU Utilization").
Set the actions (EX. Send a notification to an SNS topic).
Click Create Alarm.

Whenever we have loaded the server like CPU and disk, memory. then it will automatically get a notification for automatically create an ec2 instance through autoscaling, we can use stress commands

sudo yum install stress -y stress --cpu 4 --timeout 120, stress --hdd 2 --hdd-bytes 1G --timeout 120, stress --hdd 2 --hdd-bytes 1G --timeout 120.

Autoscaling to launch instance for load the server at to receive the notification.

GitHub repositories like in blow.

(https://github.com/pathasaradi/code_deploy_autoscaling)

Conclusion

This document outlined the step-by-step implementation of an EC2-based application with Auto Scaling, CI/CD integration, and enhanced monitoring. By following this guide, users can build a robust, scalable, and secure architecture leveraging key AWS services such as EC2, Auto Scaling, RDS, Code Deploy, CloudWatch, and Application Load Balancer.

Key Achievements

Scalability & High Availability: Configured Auto Scaling and multi-AZ deployments to handle traffic fluctuations and maintain uptime.
Security: Implemented best practices with IAM roles, VPC design, security groups, and encryption mechanisms.
Automation: Streamlined application deployment using CI/CD pipelines, reducing manual intervention.
Monitoring: Enhanced observability with CloudWatch, providing actionable insights and logging for troubleshooting and performance optimization.

Troubleshoots