DEV Community: SANDESH D MANOCHARYA

Deploying a Sample HTML Application to AWS EC2 with CodePipeline, CodeDeploy, and GitHub

SANDESH D MANOCHARYA — Wed, 08 Jan 2025 06:41:37 +0000

Step 1: Create IAM Roles

Create two IAM Roles. One for the service AWS EC2 and another for the service AWS CodeDeploy.

Go to the service "IAM", select "Roles" in the left panel and click on "Create role" on the right top.

First let us create a role "Role_EC2CodeDeploy" for the service EC2.

To do so ensure that "AWS service" is selected.

From the dropdown of "Use case", under "Commonly used services", select the service or use case "EC2".

Click "Next" and "Next"

Give a name to the role as "Role_EC2CodeDeploy" and click on "Create role".

Either during the role creation or after the role creation we need to add permissions by searching CodeDeploy in the search bar and selecting the policy "AmazonEC2RoleforAWSCodeDeploy".

A new role is created. I am going to use this role on an EC2 machine. This role allows the service AWS EC2 to access another service AWS CodeDeploy.

Similarly let us create another role "Role_CodeDeploy" for the service CodeDeploy.

But this time instead of EC2, select the service or use case "CodeDeploy".

It will take the permission policy "AWSCodeDeployRole" automatically. I am going to use this role on CodeDeploy.

Give a name "Role_CodeDeploy" to the role and click on "Create role".

Now both the roles are created.

Step 2: Create EC2 Instance and Attach the Role "Role_EC2CodeDeploy"

We can launch any number of instances based on our requirements. But I will launch only one instance "Demo_AWSCodeDeploy" in this example.

I launched an Amazon Linux 2023 AMI t2.micro EC2 instance.

Make sure that you have added the ports 22 and 80 as inbound rules.

Under "Advanced details", from the IAM instance profile dropdown, select the role "Role_EC2CodeDeploy" which was created recently for the service EC2.

Under "Advanced details" itself scroll down and in the "User data" section paste the following script to automatically install all the packages or dependencies immediately after launching the EC2 instance.

_#!/bin/bash
sudo yum -y update
sudo yum -y install ruby
sudo yum -y install wget
cd /home/ec2-user
wget https://aws-codedeploy-ap-south-1.s3.ap-south-1.amazonaws.com/latest/install
sudo chmod +x ./install
sudo ./install auto
sudo yum install -y python-pip
sudo pip install awscli
_

Click on "Launch instance".

The EC2 instance "Demo_AWSCodeDeploy" is launched and running successfully.

If we click on the instance ID and see the details of the instance then we can see the IAM role "Role_EC2CodeDeploy" attached.

That is all fine. But I want to try each command individually. So I do not use the user data script.

Connect to the instance through Git Bash or any other CLI of your choice.

First let us update the package lists.

sudo yum -y update

Now let us install necessary packages.

sudo yum -y install ruby

sudo yum -y install wget

Let us create a project directory and navigate to it.

_mkdir -p /home/ec2-user/Projects/GitHub_CodeDeploy

cd /home/ec2-user/Projects/GitHub_CodeDeploy

ls -la_

Now let us download CodeDeploy agent installer.

_wget

ls -la

Let us make the installer executable.

sudo chmod +x ./install

Let us install the CodeDeploy agent in auto mode.

sudo ./install auto

Let us install pip

sudo yum install -y python-pip

Let us install awscli

sudo pip install awscli

Let us confirm the installation of awscli

aws - version

Step 3: Create an Application and Deployment Group under CodeDeploy

Go to the service "CodeDeploy", select "Applications" in the left side bar and click on "Create application".

Give a name "WebsiteForDevOpsStuffs" for the application.

Select "EC2" from Compute platform dropdown.

Click on "Create application".

An application "WebsiteForDevOpsStuffs" is created.

Click on "Create deployment group".

Give the name "DevOpsStuffsDeploymentGroup" to the deployment group.

Attach the role "Role_CodeDeploy" to the service "AmazonCodeDeploy" by selecting "Role_CodeDeploy" from the "Service role" dropdown.

Select "In place" as "Deployment type".

Select "Amazon EC2 instances" as "Environment configuration".

In the Tag group, select "Name" and name of the newly created EC2 instance as value from dropdowns.

Leave all other settings as their default values.

Disable "Load balancer".

Click on "Create deployment group".

Deployment group "DevOpsStuffsDeploymentGroup" was created successfully.

Step 4: Create a Pipeline and Integrate GitHub with CodePipeline

Go to the left panel, expand "CodePipeline" and and click on Pipelines"

Click on "Create pipeline", give a name "DevOpsStuffsPipeline" to it, and leave the rest of the things default. By default it stores the artifacts in S3.

Then click "Next".

Select GitHub (Version 2) from dropdown, paste the connection link if you already have one or click on "Connect to GitHub" to create one.

Select repository name and branch name from dropdowns.

Select "No filter" for specifying how you want to trigger the pipeline and leave rest as default.

Click on "Skip build stage" since we are not building the code in this project.

Select "AWS CodeDeploy" from the dropdown for "Deploy Provider".

Click on "Next".

It will automatically take Region, select the application and the deployment group from dropdowns.

Click on "Next".

Review all the things and click on "Create pipeline".

Now deployment will be started using AWS CodeDeploy. If we would have specified the number of EC2 instances as 4 during the time of launch in Step 2 then the application would have been deployed on all 4 EC2 instances now.

First the code will be checked out from GitHub repo.

After some time code or application will be deployed on EC2 instance or server.

You can click on "View details" to see the summary. As the code will be stored in S3 bucket by default, you can go to the service "AWS S3" and see the bucket.

Step 5: Access the Application

Let us access the application on port 80 as already we have added the inbound rule 80 in the EC2 instance.

Copy the Public DNS of the EC2 instance and paste it in the browser.

There we go!!

Let us go to GitHub repo, make some minor changes in the file index.html by clicking on Edit/Pencil icon and commit the changes as follows.

Now go back to Amazon CodePipeline and just refresh the page.

Then go back to the browser and just refresh the page.

There we go!!

The AmazonCodePipeline has automatically identified the code changes in the GitHub repo and triggered the build.

Here is the GitHub repo of the project:

Ansible Kickstart for Absolute Beginners

SANDESH D MANOCHARYA — Wed, 08 Jan 2025 06:19:39 +0000

In this blog let us understand how to install Ansible, how to create an inventory file and how to run a playbook.

Requirement: An AWS account and basics of EC2.

Step 1: Create three EC2 Instances.

Create three EC2 instances with the same key-pair and security group.

Rename them as Ansible-ControlNode, Ansible-ManagedNode1 and Ansible-ManagedNode2.

Step 2: Download and install Git.

On your local machine download and install Git using below link so that we can use Git Bash:

You can use any CLI such as Command Prompt, WSL (Windows Subsystem for Linux), VS Code Terminal, etc as alternatives for Git Bash.

Step 3: Connect to each instance using Git Bash.

Navigate to the folder where you stored the .pem file locally.

Right click and select "Open Git Bash here".

You will get the below screen in a new Git Bash window.

Go to AWS Management Console, search for the service EC2 and click on Instances.

Select the 1st instance Ansible-ControlNode and click on Connect to connect to it.

Make sure that the SSH client is selected and copy the ssh command displayed at the bottom.

Go to Git Bash, right click and paste the copied ssh command. Then hit Enter.

Then type yes and hit Enter if it prompts the confirmation about the connection.

There we go!!…..
We are successfully connected to the 1st instance. Its private IP is 172.31.45.125.

Open two more separate Git Bash windows and repeat these connection steps for another two instances (Managed Nodes) as well. Their private IPs are 172.31.33.145 and 172.31.41.176.

Open the .pem file locally in the notepad and copy the content.

Create a .pem file in all three instances with the same file name and paste the same content.

Now you will see the same content in the file .ssh/authorized_keys in all three instances.

Step 4: Install Ansible on 1st instance Ansible-ControlNode.

_sudo apt update

sudo apt install ansible

ansible –-version_

Step 5: Create an inventory file.

In the 1st instance Ansible-ControlNode, create a directory ansible_quickstart and navigate to it.

Also in the same directory where you have created a .pem file in the 1st instance, create a .ini file for inventory by adding the Public IPs of 2nd and 3rd instances to a group myhosts.

NOTE: Whenever we restart the EC2 instances we get new Public IPs. So in the below screenshots the Public IPs may vary from one screenshot to another.

Step 6: Verify your inventory.

Make sure that the output of the following command has listed the Public IPs that we have added in the previous step.

ansible-inventory -i inventory.ini - list

Step 7: Ping the group myhosts in your inventory.

Ensure the permission of the .pem file is readable only by the owner in all three instances. We can change it to 644 by using the chmod command. But since I am the owner of the .pem file I set the permission to 600 itself.

For 1st instance:
chmod 600 /home/ubuntu/ansible_quickstart/sandy-devops-stuffs-mumbai.pem

For 2nd and 3rd instances:

chmod 600 /home/ubuntu/sandy-devops-stuffs-mumbai.pem

Also I modified the inventory file as follows to ensure my inventory.ini and Ansible configuration are correctly set up to use the SSH key.

_[myhosts]

13.233.123.32 ansible_user=ubuntu ansible_ssh_private_key_file=/home/ubuntu/ansible_quickstart/sandy-devops-stuffs-mumbai.pem

43.205.239.162 ansible_user=ubuntu ansible_ssh_private_key_file=/home/ubuntu/ansible_quickstart/sandy-devops-stuffs-mumbai.pem_

Now run the ansible ping command in the 1st instance.

ansible myhosts -m ping -i inventory.ini

ansible -m ping myhosts -i inventory.ini

OR
ansible -m ping -i inventory.ini myhosts

There we go!!…..

The control node is successfully connected to the managed nodes and now the control node can manage the managed nodes.

NOTE: Pass the -u option with the ansible ping command if the username is different on the control node and the managed node(s).

NOTE: It is easy and straightforward to write an inventory file in .ini format. But if the number of managed nodes increases then it is a best practice to write an inventory file in .yaml format as shown below which is equivalent to the file inventory.ini which we have already created.

Step 8: Create and run the playbook to ping the hosts.
Now create a file playbook.yaml with following content in the directory ansible_quickstart on 1st instance:

NOTE: ansible.builtin.ping: and ansible.builtin.debug: in the above playbook are collections and modules of Ansible.

Ansible.builtin is one of the Ansible collections.

ping and debug are modules in the collection ansible.builtin.

ping module - Try to connect to the host, verify a usable python and return pong on success.

debug module - Print statements during execution.

To know more about all other collections & modules and to understand what they do please refer to the following links.

Run the following command on 1st instance:

ansible-playbook playbook.yaml -i inventory.ini

There we go!!…..

We have successfully run a simple Ansible playbook to ping to the hosts listed in the inventory file.

In this way we can write other playbooks and run them on the control node to deploy and configure the applications on all the managed nodes at a time instead of doing it on each node manually.

Credit: Ansible official document Introduction to Ansible

Storing the Terraform State File in Remote Backend (S3 bucket)

SANDESH D MANOCHARYA — Mon, 26 Aug 2024 09:40:05 +0000

In this article let us build a simple terraform script to create an EC2 instance (you can create any resource of your choice) and then let us store the state file in S3 bucket.
Storing Terraform state files in an S3 bucket is a recommended best practice because it provides a central location for storing and managing your infrastructure's state files. Here's a step-by-step guide on how to store a Terraform state file in an S3 bucket:
Prerequisites:
Install Terraform on your local machine.
AWS account with the necessary IAM permissions to create S3 buckets and manage EC2 instances.

Step 1:
Create an IAM user (for example 'SDM-TerraformStateInS3').

Note:
While creating the IAM user do not attach any policy and do not add the user to any group. Just create the IAM user with a password of your choice.

Step 2:
Create a S3 bucket for example ('sdm-terraform-state-bucket-1') manually in the region 'ap-south-1'. You can choose any region of your choice.

Note:
In your backend configuration file 'backend_config.tf' (which will be created in Step 8), it's a good practice to specify the same region where you have created the S3 bucket. This helps ensure that your Terraform backend configuration aligns with the region where the bucket is located, which can help avoid potential issues related to region mismatch.
By specifying the same region as your S3 bucket in the backend block in the file 'backend_config.tf', you ensure that Terraform communicates with the correct S3 bucket in the designated region when storing and retrieving the state file. This alignment between your Terraform backend configuration and the region of your S3 bucket helps maintain consistency and ensures that Terraform functions as expected.

Step 3:
Create a DynamoDB table (for example 'SDM-terraform-lock') for state locking with the Partition key 'LockID' and its type 'String'.

Note:
When creating a DynamoDB table for use as a Terraform state lock, it's important to ensure that the table is created in the same region that you specify in your Terraform backend configuration (backend "s3" block in the file 'backend_config.tf' which will be created in Step 8) to maintain consistency. Terraform will interact with the DynamoDB table in the specified region to manage state locks.

Step 4:
With the help of JSON code create your own 'Customer managed' IAM policy (for example 'SDM-Terraform-S3') for S3. The JSON code for the policy is as follows:

Then the created policy looks like this in the list of policies in IAM service of AWS:

Note:
Instead of creating our own 'Customer managed' policy we could attach 'AWS managed' policy 'AmazonS3FullAccess' to the IAM user. But the AWS managed policy 'AmazonS3FullAccess' provides full access to Amazon S3 resources, allowing users to perform a wide range of actions on S3 buckets and objects. If your primary goal was to store Terraform state files in an S3 bucket and manage infrastructure with Terraform, attaching this policy would be useful and sufficient for storing Terraform state files in S3.
Here's why it's useful:
S3 Bucket Operations: The 'AmazonS3FullAccess' policy grants permissions for various S3 bucket operations, including creating, listing, deleting, and updating buckets. These permissions are necessary for creating and managing an S3 bucket for Terraform state storage.
Object Operations: The policy allows users to perform actions on S3 objects (files) within a bucket, which includes the ability to upload, download, and delete objects. This is important for managing the Terraform state file in the bucket.

However, it's essential to consider the principle of least privilege when granting permissions. While 'AmazonS3FullAccess' provides broad access to S3, it may grant more permissions than strictly necessary for your use case. For security best practices:
Use a More Specific Policy: If possible, create a custom IAM policy tailored to the specific actions and resources needed for your use case. This allows you to grant only the permissions required, reducing the potential attack surface.
Consider State Locking: If you plan to use Terraform in a collaborative environment with multiple users, consider using Terraform's state locking feature, which uses DynamoDB to manage locks. Ensure that your IAM policies grant appropriate permissions for DynamoDB if you implement state locking.
Regularly Review and Audit Policies: Periodically review and audit your IAM policies to ensure they align with your current infrastructure and security requirements. Remove unnecessary permissions and ensure that permissions are granted on a need-to-know basis.

Hence, 'AmazonS3FullAccess' can be useful for managing Terraform state files in an S3 bucket, but it's essential to review and fine-tune your IAM policies to meet your specific security and infrastructure needs while adhering to security best practices.

Step 5:
Attach this policy 'SDM-Terraform-S3' to IAM user 'SDM-TerraformStateInS3'.

Step 6:
Manually create an AWS EC2 Ubuntu instance (for example 'SDM-Terraform') with instance type 't2.micro' in the region 'ap-south-1'.

Then ssh into it.
Create a directory 'S3'.
mkdir S3
Navigate into the directory 'S3'.
cd S3

Step 7:
Install Terraform in it using the following commands.
Update the system:
sudo apt-get install unzip

Confirm the latest version number on the terraform website given below:
https://www.terraform.io/downloads.html

Download latest version of the terraform (substituting newer version number if needed):
wget https://releases.hashicorp.com/terraform/1.5.7/terraform_1.5.7_linux_amd64.zip

Install unzip:
unzip terraform_1.5.7_linux_amd64.zip

Extract the downloaded file archive:
unzip terraform_1.5.7_linux_amd64.zip

Move the executable into a directory searched for executables:
sudo mv terraform /usr/local/bin/

Confirm the installation:
terraform --version

Step 8:
Create a backend configuration file 'backend_config.tf' with the following content inside the directory 'S3'.

Step 9:
Create an infra (EC2 instance) configuration file or resource definition file 'ec2.tf' with the following content inside the directory 'S3'.

Note:
Keep both the files 'backend_config.tf' and 'ec2.tf' in the same directory 'S3'.
In this setup, 'backend_conf.tf' contains the Terraform backend configuration for state storage, while 'ec2.tf' contains your EC2 instance resource definitions.

Note:
We can combine the contents of both 'backend_conf.tf' and 'ec2.tf' into a single Terraform configuration file. Here's how you can structure it:

Note:
In this combined file:
The provider block configures the AWS provider.
The resource block defines an AWS EC2 instance.
The terraform block includes the backend configuration for S3 and DynamoDB.
This single file contains both the resource definition and the backend configuration, and you can use it to create the EC2 instance and manage the state in S3.
The earlier provided guidance with two separate files is based on the typical recommended project structure and best practices for Terraform configuration management. Using separate files for backend configuration and resource definitions is often recommended for the following reasons:

Modularity and Organization: Separating backend configuration from resource definitions allows you to keep your infrastructure code organized and modular. It's easier to manage different aspects of your configuration in distinct files, making it more maintainable, especially in larger projects.
Collaboration: In collaborative environments, different team members may be responsible for different parts of the configuration. By having separate files, team members can work on the backend configuration independently of resource definitions, reducing conflicts when merging changes in version control.
Flexibility: Separating backend configuration enables you to reuse resource definitions across different environments or projects while changing only the backend configuration as needed.

However, using a single file can be a valid approach for smaller or less complex projects, as it simplifies the file structure. The choice of whether to use separate files or a single file depends on the specific requirements and complexity of your project, as well as your personal preference for organization.
The above provided guidance on both approaches gives you flexibility and helps you choose the one that best suits your needs.
If you go with two separate files for adopting best practice then that sounds like a good choice! Separating your Terraform configuration into two separate files for backend configuration and resource definitions aligns with best practices, especially as your projects grow in complexity or involve collaboration with multiple team members.

Step 10:
Run the following commands:
terraform init
terraform plan
terraform apply
Now go to the service 'EC2' in AWS and go to 'Instances'. Then in list of instances you will see an instance with the name 'SDM-TestTfStateinS3' in Running state.

Also if you go to the service 'S3' in AWS, if you go to 'Buckets' and if you go to the bucket 'sdm-terraform-state-bucket-1' then you will be able to see the terraform state file in that bucket.

Note:
We could store the state file in GutHub as well. But it has some drawbacks.
Yes, you can store your Terraform state file in a version control system (VCS) like GitHub, but it's generally not recommended for several reasons:

Concurrency and Locking: VCS systems like GitHub do not provide built-in mechanisms for handling concurrent access and locking of the state file. In a collaborative environment, multiple team members could attempt to modify the state file simultaneously, leading to conflicts and potential data corruption.
Performance: Terraform state files can become large and contain sensitive information. Storing them in a VCS can impact the performance of the repository and could expose sensitive data if not properly protected.
Versioning: VCS systems are designed for source code versioning, not infrastructure state. Managing state in a VCS can become unwieldy as your infrastructure grows and changes.
Security: Storing sensitive information, such as secrets or access keys, in a VCS is generally discouraged due to security concerns. State files may contain sensitive information, and their exposure should be minimized.
Ease of Collaboration: Remote backends like Amazon S3 and others (e.g., Azure Blob Storage, Google Cloud Storage) are specifically designed for storing Terraform state files. They provide features like state locking and access control, making it easier for teams to collaborate safely.

Using a remote backend like S3 or an equivalent is recommended because it addresses these concerns and is purpose-built for managing Terraform state. It provides a secure, centralized, and scalable solution for storing and managing state files, especially in team environments.
While you could store Terraform configurations in a VCS like GitHub, it's generally better to use a remote backend for managing the state files. This separation of concerns allows you to benefit from the strengths of each tool: VCS for code collaboration and versioning, and a remote backend for state management and collaboration on infrastructure changes.
Hope this article helps you to understand the purpose of storing Terraform state files into remote backend such as S3 buckets. Please comment if you have any suggestions or improvements.