DEV Community: Palak Bhawsar

AI-Powered Sentiment Analysis for Product Reviews & Visualization

Palak Bhawsar — Sat, 18 Jan 2025 12:43:25 +0000

In this project, we will create an automated pipeline for analyzing the sentiment of product reviews. Instead of building a full application, we will upload product reviews in JSON format to an Amazon S3 bucket. As soon as a new file is uploaded to S3, an S3 event notification will trigger an AWS Lambda function. This Lambda function will use the Amazon Comprehend API to perform sentiment analysis on each review in the uploaded file. Once the sentiment data is processed, the Lambda function will upload the analyzed results to a new S3 bucke t in JSON format. Amazon Athena will then be used to query the sentiment data stored in S3. Finally, the data will be seamlessly integrated with Amazon QuickSight for interactive visualization, providing insightful analysis of the sentiment trends.

Prerequisite:

AWS Account
QuickSight Account Setup
Experience working with AWS Services
Programming skills: Python, JSON

Step 1: Create S3 Buckets

Lets get our hands dirty and start by creating two S3 buckets one for storing product reviews and the other to store the data analyzed by Amazon Comprehends sentiment analysis.

Search for Amazon S3 in the services in AWS account.
Click on create bucket.
Select bucket type as General Purpose.
Provide unique name for the bucket, for example product-review-bucket-789
Object Ownership select as ACL Disabled.
Click Create Bucket.
Upload Product Review JSON file to S3 Bucket
JSON file link
Create one more bucket to store sentiment analysis data sentiment-analysis-bucket-1234

Step 2: Create Lambda Function

Search for Lambda in the services.
Click on Create a function.
Select Author from scratch.
Provide a name for the Lambda function, for example, LambdaForSentimentAnalysis.
Select the runtime as Python 3.13.
Choose Create a custom role and go to the IAM console.
Choose the service as Lambda.
Select AmazonS3FullAccess and ComprehendFullAccess. [Note: It is a good practice to provide granular access for a particular bucket and Comprehend job.]
Click Create role.
Go back to the Lambda screen and select the role just created in Existing roles.
Finally, click Create function.

Step 3: Write Python code

Open editor of your choice and create a file.
Write a code to read product review JSON file from the S3 bucket.
Use the Amazon Comprehend API to analyze sentiment.
Store the analyzed sentiment data in the output S3 bucket in JSON.
GitHub Code Link

Step 4: Create S3 Event Notification to Trigger Lambda

Select Product review S3 bucket product-review-bucket-789
Under Properties, click create event notification.
Provide a name to the event, for example, NewReviewUploadTrigger.
Select All object create events.
Choose Lambda function created in above step.
Click save changes.

Step 5: Set up Athena for Querying S3 Data

Select Athena in services.
Select Query your data with Trino SQL.
Click Launch query editor.
In Athena console, click on the "Settings" icon
Under Query result location, specify an S3 bucket to store Athena query
Use Query editor create a new database.

Step 6: Setup Amazon QuickSight for Visualization

Go to QuickSight in the AWS Console and setup Amazon QuickSight Account.
Attach an IAM policy to allow QuickSight to access Athena and S3.
Click on Datasets New Dataset.
Give a name to Data Source Name.
Select Athena Workgroup as Primary.
Select Database and Tables created in previous step.
Select Directly Query your Data and click Visualize then Create.
Choose visuals like Pie chart, Bar etc select Group/x-axis as sentiment.

Thank you for taking time to read my article. If I've overlooked any steps or missed any details, please don't hesitate to get in touch.

Feel free to reach out to me anytime Contact me

~ Palak Bhawsar

]]>

CI/CD pipeline for Terraform Project

Palak Bhawsar — Wed, 27 Mar 2024 05:11:32 +0000

In this article, we will be creating an automated CI/CD pipeline for a Terraform project, with a focus on adhering to security and coding best practices. The pipeline will be designed to trigger automatically upon code push to GitHub, and will encompass code analysis, security analysis, testing, and the typical Terraform workflow stages such as initialization, planning, and applying changes.

Prerequisite:

GitHub and AWS account
Basic knowledge of Terraform and AWS
Knowledge of Jenkins and CI/CD

Let's understand the purpose of all these tools within our CI/CD pipeline.

TFLint is a popular open-source static analysis tool designed for Terraform. It performs automated checks on Terraform configurations to identify potential issues, errors, and violations of best practices. TFLint helps maintain code quality, consistency, and reliability in Terraform projects.

Tfsec is a static analysis tool used to scan Terraform code to identify security gaps in IaC. It analyzes Terraform codebases to identify potential security issues such as misconfigurations, insecure settings, and other issues that might expose infrastructure to risks.

Terratest is an open source testing framework for infrastructure defined using Terraform. It performs unit tests, integration tests, and end-to-end tests for the cloud-based infrastructure and helps identify security vulnerabilities early on.

1. Setup Jenkins Server

Launch an EC2 instance and install Jenkins on it to set up the job for running the pipeline. Follow the blogs below to set up a Jenkins server on an EC2 instance.

https://palak-bhawsar.hashnode.dev/install-jenkins-in-ec2-instance-using-user-data-script

2. Install tools in Jenkins

Connect to the Jenkins EC2 instance and install all these tools. The Terraform tool is needed to run Terraform commands. TFLint is required to perform linting on Terraform configuration files. TFsec is needed to scan Terraform configuration files to identify any security vulnerabilities. Go is needed to run Terratest to execute unit and integration test cases.

Install Terraform
Install TFLint
Install TFSec
Install go

3. Attach IAM role to Jenkins server

Create an IAM role with the necessary permissions to create resources in AWS, and then attach this role to the EC2 instance where Jenkins is installed. Attaching this role to Jenkins is crucial as it grants the necessary permissions for provisioning resources within the AWS infrastructure.

4. Create Webhook

Webhook in Jenkins triggers the pipeline automatically when any changes are done in the GitHub repository like commit and push. Go to Jenkins dashboard and copy the Jenkins URL. Go to GitHub repository settings. In the left pane select Webhooks. Click Add webhook and paste the Jenkins URL in the Payload URL by appending the URL with /github-webhook/ in the end of URL. Select the events when you want to trigger the pipeline, I have selected Just the push event and click Add webhook.

5. Project struture and Code

In this project, I am creating an AWS EC2 instance named "test_instance" with a specified AMI and instance type. It enables HTTP access to instance metadata and follows best practices by encrypting both the root block device and an additional EBS volume.

Under test folder, I have created main_test.go file to verify the changes in Terraform configuration, an EC2 instance of type t2.micro is created with the creator name as Palak. If any of these conditions fail, the test will report a failure.

The tflint.hcl file is used to configure TFLint. We can specify which plugins TFLint should use and their versions. If your Terraform code interacts with AWS resources, you might enable the AWS plugin and specify its version.

https://github.com/palakbhawsar98/Terraform-CI-CD-Pipeline

6. Create Jenkins pipeline

Go to Jenkins Dashboard click New Item -> Give a name to the pipeline -> Select Pipeline -> Click Ok. Add Description of your pipeline -> Build Triggers -> GitHub hook trigger for GITScm polling.

Scroll to the last in the Pipeline section and from the dropdown select Pipeline script from SCM. Under SCM, select Git and enter your GitHub project repository URL. If your GitHub repository is private then add credentials. Also, enter the branch name in Branches to build and the Jenkinsfile name in Script Path and click Save. Finally, Click Build Now to run the pipeline.

7. Troubleshooting

The pipeline failed at the TFlint stage due to my Terraform configuration explicitly utilizing undeclared variables.

The Tfsec stage failed because encryption was not enabled for the EBS volume attached to the EC2 instance.

Finally, all pipeline stages ran successfully once it met best practices and security requirements.

Terraform workflow: init, plan, apply

Thank you for taking time to read my article. If I've overlooked any steps or missed any details, please don't hesitate to get in touch.

Feel free to reach out to me anytime Contact m e

~ Palak Bhawsar

Monitoring AWS Services Using CloudWatch

Palak Bhawsar — Tue, 30 Jan 2024 15:11:20 +0000

In this article we will explore monitoring using AWS CloudWatch and creation of CloudWatch alarms for various AWS services leveraging AWS SNS (Simple Notification Service) and SES (Simple Email Service) to send emails to the team.

Prerequisite:

AWS account
Basic understanding of different AWS services

Monitoring is a method of keeping an eye on your infrastructure. If something fails or breaches occur, you should receive a notification. Amazon CloudWatch collects various metrics, and based on the thresholds set, it triggers an alarm. These alarms can then initiate automated actions if already set up, or else send notifications.

For instance, suppose you have deployed an application on an EC2 instance, but suddenly, the traffic to your application surges beyond what your instance can handle. In such situations, creating alarms for EC2 allows you to monitor the instance's performance. When the traffic exceeds a certain limit, these alarms can alert you or automatically increase the size of the instance. This scaling up enhances your application's performance and availability to meet the increased demand.

OK State : The OK state indicates that the monitored metric is within the acceptable range and has not breached the defined threshold.

ALARM State : The ALARM state indicates that the monitored metric has breached the defined threshold. When the metric enters the ALARM state, CloudWatch triggers the associated alarm, and any configured alarm actions are executed.

INSUFFICIENT_DATA State : The INSUFFICIENT_DATA state indicates that there is insufficient data to determine the state of the alarm. This state typically occurs when CloudWatch does not have enough data points to evaluate the metric against the defined threshold.

Important terminologies in CloudWatch:

📈Metrics: Metrics in CloudWatch are variables that represent specific data points or measurements about performance of AWS resources. These metrics are continuously collected and monitored by CloudWatch. For example CPUUtilization, MemoryUtilization, and DatabaseConnections are some of the metrics.

Alarms: Alarms allow you to set thresholds on metrics. When a metric breaches the threshold, CloudWatch triggers an alarm, enabling you to take proactive actions or receive notifications. For example when CPU utilization is more than 80% trigger an alarm.

📉Thresholds: Thresholds are predefined values that you set on metrics to define acceptable performance ranges for your AWS resources. For example, you might set a threshold for CPU utilization at 80% or 90%.

Period: The period refers to the length of time over which CloudWatch aggregates data points for a specific metric.

Evaluation Period : The evaluation period determines the number of consecutive data points that must breach the alarm threshold before CloudWatch triggers the alarm. For example if we set the evaluation period to 3 for a 5-minute period, CloudWatch requires three consecutive 5-minute data points to breach the threshold before triggering the alarm.

📄Namespaces : Namespaces are containers for metrics, categorizing them based on their origin or source. For example, AWS services have their own namespaces like "AWS/EC2" or "AWS/S3".

💹Datapoint: Data points represent specific values or measurements of a metric at a particular point in time. For example we have an EC2 instance running a web server and CloudWatch collects data at every minute so data point indicates that at this hour of this day, the CPU utilization of the EC2 instance was X%

Step 1:Create Amazon SNS topic:

In the AWS account services, search for SNS. Go to Topics in side bar and click Create topic, select type as Standard, enter a topic name and click create topic.

Step 2: Subscribe to SNS topic

Go to Subscriptions in the left panel, click Create subscription, choose your SNS topic ARN, select protocol as Email and enter the email addresses of your team members. Team will receive a confirmation email to confirm their subscription.

Step 3: Setup SES for sending emails

Go to AWS services and search for SES and verify the email addresses or domains that you want to send emails from. Once done go to your email and confirm the email address verification.

Step 4:Create CloudWatch alarm for EC2

Go to AWS services and search for CloudWatch, In the left navigation pane, click on All alarms , then click Create Alarm, Select the metric you want to monitor (e.g., CPU utilization for an EC2 instance).

Select statistics, Period, and threshold. Select existing SNS topic and click Next, enter Alarm name and click Create Alarm.

When CPUUtilization threshold is crossed, alarm will be triggered and you will receive an email notification.

Below is the email notification received on verified email:

Important metrics for Monitoring AWS services

Metrics to monitor for EC2:

Elastic compute cloud (EC2) is a web services that enable users to rent virtual servers, known as instances, along with other computing resources.

CPU Utilization : CPU utilization indicates how much of the CPU resources are being utilized by your EC2 instances.
Memory Utilization : Monitoring utilization metric ensure that your instances have enough memory available to handle the workload efficiently.
Network Traffic : Monitoring inbound and outbound network traffic helps identify trends and anomalies in data transfer.
Disk I/O : Monitoring Disk I/O metrics provide insights into how much data is being read from and written to the instance's storage volumes.
Disk Utilization : Monitoring disk space usage helps prevent instances from running out of storage capacity, which can lead to application failures or downtime.

Metrics to monitor for CloudFront:

Amazon CloudFront is a content delivery network (CDN) service provided by Amazon Web Services (AWS) that helps distribute content to users globally with low latency and high transfer speeds.

Requests : Monitor the total number of requests served by CloudFront.
Data Transfer : Track the volume of data transferred by CloudFront.
HTTP Status Codes : Monitor HTTP status codes returned by CloudFront. This includes 2xx, 3xx, 4xx, and 5xx status codes. An increase in 4xx or 5xx errors might indicate issues that need attention.
Origin Response Time : Track the time taken by the origin server to respond to CloudFront requests. High response times can impact overall latency and user experience.
Error Rate : Monitor the rate of errors encountered by CloudFront, increase in error rates may indicate underlying issues with content delivery or origin servers.

Metrics to monitor Application Load Balancer:

Application Load Balancer (ALB) distributes incoming application traffic across multiple targets, such as Amazon EC2 instances, containers, and IP addresses, within one or more Availability Zones.

Request Count : Total number of requests handled by the ALB over time. This metric helps you understand the traffic volume your application is receiving.
Healthy Host Count : Track the number of healthy instances registered with the ALB's target groups.
Unhealthy Host Count : Track the number of unhealthy instances registered with the ALB's target groups.
Latency : Monitor the latency of requests processed by the ALB. Latency metrics provide insights into the responsiveness of your application.
HTTP Error Rates : Track the rate of HTTP 4xx and 5xx error responses returned by the ALB.
Active Connection Count : Monitor the number of active connections to the ALB over time. This metric helps you understand the level of concurrent connections your application is handling.
Target Response Time : Track the response time of backend instances when responding to requests forwarded by the ALB.

Metrics to monitor Amazon S3:

Amazon Simple Storage Service (S3) is a storage service that provides object storage through a web service interface.

Bucket Size : Monitor the total size of objects stored in each S3 bucket. This can help understand your storage utilization.
Number of Objects : Track total number of objects stored in each S3 bucket. Monitoring object count helps you manage your data and identify any unexpected increases or decreases in the number of objects.
Request Metrics : Monitor various S3 request metrics, including the number of GET, PUT, POST, DELETE, and LIST requests made to each bucket.
Data Transfer Metrics : Track the amount of data transferred in and out of each S3 bucket.
Bucket Access Metrics : Monitor access metrics such as the number of requests from different AWS services and different regions made to bucket.

Metrics to monitor Amazon RDS:

Amazon Relational Database Service (RDS) is a web service that makes it easy to set up, operate, and scale a relational database in the cloud.

CPU Utilization : Monitor the CPU utilization of your RDS instances to ensure that they have sufficient processing power to handle the workload.
Database Connections : Track the number of active database connections to your RDS instances.
Read and Write IOPS : Monitor the read and write I/O operations per second for your RDS instances.
Storage Usage : Keep track of the amount of storage used by your RDS instances.
Free Storage Space : Monitor the amount of free storage space available in your RDS instances.
Database Throughput : Monitor the throughput of data flowing in and out of your RDS instances.
Database Latency : Monitor the latency of database queries and transactions.

Thank you for taking time to read my article. If I've overlooked any steps or missed any details, please don't hesitate to get in touch.

Feel free to reach out to me anytime Contact m e

~ Palak Bhawsar

Architect two tier Secure and Scalable AWS Infrastructure with Terraform

Palak Bhawsar — Wed, 16 Aug 2023 04:55:36 +0000

In this article, we will design an architecture that meets security, availability and scalability requirements and also streamlines deployment and management processes. We will create AWS Route53 for DNS management, Application Load Balancer (ALB) for load distribution, Auto Scaling Groups to dynamically adjust capacity, VPC setup for network isolation, AWS EC2 instances and MySQL for deploying a web application, and many other AWS services to explore the power of Infrastructure as Code (IaC) tool Terraform. Additionally, we will take a step further by not just deploying Python-MySQL application, but also containerizing it using Docker. This modernization approach enhances portability and consistency while simplifying management. Moreover, we'll ensure that our application has an online presence with our custom domain.

Prerequisite:

Installed and configured AWS CLI and Terraform
GitHub and AWS account
Basic knowledge about Terraform
Understanding of different AWS services

Notes:

I suggest building the infrastructure and resources manually first to fully understand how they work together. If you encounter any problems with the Terraform user data script, try manually deploying the code by SSHing into the instance. This hands-on approach will give you a solid grasp of the setup and troubleshooting skills you need for successful automation later.

Code link:

Step 1: State management using S3 and DynamoDB

The first step is to create an AWS S3 bucket and a DynamoDB table to store and lock state files. Terraform state is a critical component that keeps track of the resources we create and manage. Instead of storing this state locally, we'll store it remotely using AWS S3 and leverage DynamoDB for state locking. Let's first add the below code in the Terraform project folder and execute Terraform init, plan and apply commands.

############################## S3.tf #################################
resource "aws_s3_bucket" "dev-remote-state-bucket" {
    bucket = "dev-remote-state-bucket"
      versioning {
        enabled = true
    }
    tags = {
        Name = "S3 Remote Terraform State Store"
    }
}


############################## dynamo_db.tf ##########################
resource "aws_dynamodb_table" "terraform-state-lock" {
    name = "terraform-state-lock"
    read_capacity = 5
    write_capacity = 5
    hash_key = "LockID"
    attribute {
        name = "LockID"
        type = "S"
    }
    tags = {
        "Name" = "DynamoDB Terraform State Lock Table"
    }
}

The above AWS resources will get created after executing terraform commands. Now, let's add a provider block and execute all commands again make sure you delete terraform.tfstate file from your local and update the Terraform backend with the AWS S3 bucket and DynamoDB table name as shown below.

############################## provider.tf #############################
terraform {
     backend "s3" {
        bucket = "dev-remote-state-bucket"
        key = "terraform.tfstate"
        region = "us-east-1"
        dynamodb_table = "terraform-state-lock"
    }
    required_providers {
      aws = {
        version = "~>5.0"
        source = "hashicorp/aws"
      }
    }
}

Step 2: Purchase a domain name

The first step in establishing your application's online presence is to purchase a domain name. You can choose a domain provider of your preference. I have chosen GoDaddy for domain registration services. You can also use AWS Route 53 for domain management which allows seamless integration with your AWS resources.

Since I have purchased the domain from GoDaddy to integrate it with other AWS services. We need to delegate domain rights to AWS Route53. This will allow you to easily manage your DNS records within AWS, ensuring better integration with your resources and services.

Step 3: Hosted zone in AWS Route 53

An AWS Route 53 hosted zone is a container that holds information about how you want to route traffic for a specific domain, such as palakbhawsar.in. It will provide us with the necessary nameservers that we can use to delegate the domain management to Route 53. We will later associate the hosted zone with an Application load balancer to enable dynamic routing and load balancing of traffic across multiple resources.

We can then take the name servers from the output and update the GoDaddy account to delegate domain management to Route 53.

Go to your GoDaddy account->Domain portfolio-> DNS-> Nameservers->Change nameservers->Add nameservers->Save.

Step 3: Add SSL Certificate for Domain

An SSL certificate ensures encrypted communication between users' browsers and your web application. We will utilize AWS Certificate Manager (ACM) to easily provision a free SSL/TLS certificate for our custom domain. The below code sets up an ACM certificate, creates a Route53 DNS record for certificate validation, and then validates the certificate. The validation process involves confirming ownership of the domain through DNS records, and it's a critical step in securing your website with an SSL/TLS certificate.

Step 4: VPC setup for network isolation

In this step, we'll establish a Virtual Private Cloud (VPC) in the us-east-1 region, linking it to an Internet Gateway for external connectivity. The setup includes the creation of both public and private subnets across the us-east-1a and us-east-1b availability zones. Route tables for these subnets will be configured and associated accordingly. Route tables act as guides for network traffic, determining its destination based on predefined rules. This orchestrated process encompasses the development of a comprehensive networking environment, laying the foundation for the subsequent building blocks of your infrastructure.

Step 5: Create an IAM role for EC2 Instances

In this step, we will set up a special role for EC2 instances. The application running inside these instances needs to talk to a MySQL database to get credentials. Plus, it also requires access to database credentials stored in an AWS tool called Systems Manager. So, we're creating this role and permitting it to communicate with MYSQL and the System manager.

Step 6: Create instances and security groups

In this step, we will create EC2 instances within each availability zone, leveraging a user data script that will orchestrate the deployment of our Python-MySQL application. The application will operate on port 80. We will create a security group that allows inbound traffic on port 80 for HTTP traffic, port 443 for HTTPS communication, and port 22 for SSH access.

Step 7: Create RDS and security group

In this step, we will create a MySQL DB instance in a private subnet and attach the security group to the MySQL DB instance. The security group will allow traffic from the EC2 instance security group on port 3306. This architecture is designed to seamlessly handle contingencies. If the MySQL database instance in the us-east-1a availability zone encounters a failure, a failover plan comes into play and the standby MySQL instance in the us-east-1b availability zone takes over the role of the primary database.

Step 8: Create an Application Load Balancer

In this step, we will create an AWS Application Load Balancer (ALB) with associated components. Starting with the security group that permits incoming HTTP (port 80) and HTTPS (port 443) traffic while allowing outbound traffic. Following this will create a target group and ALB listener to set up for HTTPS on port 443, along with a default fixed-response action. Additionally, specific listener rules are defined for paths like "/signup," "/signin," and "/dashboard," each forwarding traffic to the designated target group is EC2 instances. An SSL certificate is associated with the listener via an ACM certificate ARN. Lastly, a Route 53 A record is generated to associate the ALB's DNS name with the root domain ("palakbhawsar.in"), ensuring connectivity to the ALB for incoming requests.

Step 9: Create an Autoscaling group

In this step, we will create an Auto Scaling group, which is a way to manage a collection of Amazon EC2 instances. First, create a launch configuration that defines how instances will be launched. It specifies an Amazon Machine Image (AMI) to use, sets the instance type, and assigns a security group for the instances. This launch configuration is used to create instances. Next, create the Auto Scaling group, this group ensures a specific number of instances are running at all times, adapting to demand. It uses the previously created launch configuration. The group's settings include minimum and maximum instance counts (2 to 4 instances), and it will always try to maintain 2 instances by default. Finally, associate the target group to the Application Load Balancer (ALB) and distribute incoming traffic among the instances in the Auto Scaling group.

Step 10: Create a parameter in AWS System Manager

In this step, we will create a secret locker for a password by creating a parameter in the AWS system manager securely. It will create a special place called "mysql_password" where you can store your password secretly. You give it a name, in this case, "mysql_psw," and you tell it that the secret is a secure string, like a hidden message. Then, you put your password, let's say it's "12345678," into this secret locker. Now, whenever you need that password, you can open this locker and get it. It's like having a secret vault just for your password.

After successfully deploying the infrastructure, it's time to verify its functionality. Open your browser and access the web application using your domain: palakbhawsar.in/signup.

As we added SSL/TLS certificate for our domain the connection is secure and can be accessed via HTTPS.

Go ahead and sign up, then sign in to validate the connection with the MySQL database.

Voila, it's a success! Your application is up and running smoothly.

I am going to destroy this infrastructure due to the costs associated with AWS services. As a result, accessing the application using my domain might not be possible anymore.

🥳🥳 Congratulations on completing this project!

Thank you for taking time to read my article. If I've overlooked any steps or missed any details, please don't hesitate to get in touch.

👉 Feel free to reach out to me anytime Contact m e

Palak Bhawsar

S3 Event-Driven Email Notifications using AWS Lambda and SQS

Palak Bhawsar — Wed, 17 May 2023 09:34:49 +0000

In this project, I will be creating an S3 bucket, Queue, and Lambda function. Whenever an object is uploaded in the S3 bucket, an S3 event will trigger the queue and the lambda function will run which will check if the object name contains the word "sensitive" If so it will send an email notification to the team saying some sensitive file is added in the bucket.

Prerequisite:

AWS account and familiarity with AWS services

Let's first understand these AWS services:

Simple Storage Service(S3): A highly scalable object storage service that allows you to store and retrieve large amounts of data. It provides durability, availability, and security for your data and can be used for various purposes, such as backup, static website hosting, and content distribution.

Simple Queue Service(SQS): A fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications. It allows you to send, store, and receive messages between software components.

Function as a Service(Lambda): A computing service that lets you run code without provisioning or managing servers. It allows you to execute your code in response to events, such as changes in data, HTTP requests, or scheduled intervals.

Simple Email Service(SES): A fully managed service provided by AWS that enables you to send and receive email messages. It handles the underlying infrastructure and scaling aspects, allowing you to focus on sending and managing emails without the need to provision or manage servers.

Step 1: Create an S3 bucket

Go to the AWS console and search for " S3" in services. Click Create bucket and give a unique name to the bucket -> choose the AWS region where you want your bucket to reside. Keep other settings default and click Create bucket.

Step 2: Create SQS

Go to the AWS console and search for " SQS" in services. Click Create queue -> Choose the queue type as Standard -> Give a name to the queue -> Keep other settings default and click Create queue.

Now attach the access policy to the queue so that it can communicate with the S3 bucket when the S3 event is triggered.

Step 3: Attach policy to SQS

Go to the Queue that we have created in the previous step and select access policy. Click edit to update the permissions.

Take the Resource arn from the existing policy shown in your console. Go to the bucket that we have created and in the properties section you will get bucket arn, copy it and paste it into aws:SourceArn in the below policy and save.

{
  "Version": "2012-10-17",
  "Id": "Policy1684249599450",
  "Statement": [
    {
      "Sid": "Stmt1684249581034",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "sqs:*",
      "Resource": "arn:aws:sqs:us-east-1:148418490226:project-orion-queue",
      "Condition": {
        "ArnEquals": {
          "aws:SourceArn": "arn:aws:s3:::project-orion-files"
       }
      }
    }
  ]
}

Step 4: Create S3 Event

Go to your bucket and select Properties, scroll down to Event notifications and click Create event notification. Give a name to your event then checkmark All object create events.

Select destination as SQS queue -> Select the queue that we have created in previous steps -> click save changes.

Step 5: Create Lambda function

Go to AWS services and search for Lambda. Select Lambda and click Create function. Select Author from Scratch-> Give a name to function -> Select runtime as Python3-> Click Create function.

Now go to Configuration -> Permissions -> click Role name

Add permissions to this lambda role so that it can communicate with SQS and SES services. Click Add permissions-> Attach policies-> Search for AmazonSESFullAccess and AmazonSQSFullAccess and add this policy to the Lambda.

Step 6: Add Lambda Trigger

We want this function to run when SQS has something to process. Click Add trigger in the Lambda function and select SQS as the trigger source. Select the SQS name that we have created. Keep the Batch window as 1 then click Add.

Step 7: Create SES and verify the Email

Go to AWS services and search for SES. Select SES and click Verified Identities -> Create identity-> Email address-> Enter email address-> Click Create Identity.

Check your mailbox and verify the email address. Create an identity for both the sender's email address and the receiver's email address. Once the identity is verified move to the next step.

Step 8: Deploy the Lambda function

Go to Lambda and add the Python code which will filter out the object uploaded in the S3 bucket and send the email to the receiver if the object name contains the word "sensitive"

Clone this GitHub repository: https://github.com/palakbhawsar98/AWS-Lambda-SQS-Project

After adding Python code in the Lambda function, click Deploy. Now to test this, upload some objects in the S3 bucket. Go to monitor and click view CloudWatchLogs-> Select the latest Log Stream. Check if the email is sent to the receiver and the queue is empty.

As soon as you upload a file in the S3 bucket and the name of the file contains the word "sensitive" an email will be sent.

Congratulations🥳🥳, we have completed the serverless project successfully.

If you face any issues contact me on my socials 👉 Contact m e

Thank You

Palak Bhawsar

Automated CI/CD pipeline for Java Project

Palak Bhawsar — Wed, 22 Mar 2023 04:14:58 +0000

In this article, we will be creating an automated CI/CD pipeline for your Java project using Jenkins, Docker, and AWS. With this pipeline, your project will be automatically built, tested, and deployed to your AWS EC2 instance every time you push code to your GitHub repository.

Prerequisite:

GitHub and DockerHub account
AWS account and knowledge of EC2 Instances
Knowledge of Jenkins, Docker, and Maven

Step 1: Setup Jenkins Server

First set up the Jenkins server to create a pipeline. Launch an EC2 instance and install Java and Jenkins into it. Follow this article to set up a Jenkins server . You need to install Maven and Docker as well. Maven to build the Java project and Docker to build the docker image in Jenkins. SSH into your instance and install maven and Docker using the below commands. For Jenkins, I am using an Ubuntu instance.

# Install Maven in Jenkins
sudo apt-get install maven -y
## Update packages
sudo apt-get update
## Install Docker
sudo apt-get install docker.io -y
## Add Jenkins user to Docker group
sudo usermod -a -G docker jenkins

Set JAVA_HOME and MAVEN_HOME using the below commands. To set the variable permanently, you should add it to the .bashrc file in your home directory.

echo "export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64" >> ~/.bashrc
echo "export MAVEN_HOME=/usr/share/maven" >> ~/.bashrc

1.1. Install Plugins

Install plugins to integrate Jenkins with GitHub, Maven, and EC2. Go to Manage Jenkins , and select Manage plugins. Under available plugins search for the below plugins and Install without restart

Git
Pipeline maven integration
Pipeline stage view
SSH Agent

1.2. Configure Java and Maven in Jenkins

Go to Manage Jenkins, select Global tool configuration , and scroll down to add JDK and Maven path that we exported in the above steps as shown below. Uncheck the Install automatically checkbox, Give Name and Path and click Save.

1.3. Create Webhook

Webhook in Jenkins triggers the pipeline automatically when any changes are done in the GitHub repository like commit and push. Go to Jenkins dashboard and copy the URL in the browser. Now go to GitHub repository settings. In the left pane select Webhooks.

Click Add webhook and paste the Jenkins URL in the Payload URL by appending the URL with /github-webhook/ as shown below. Select the events when you want to trigger the pipeline, I have selected Just the push event and click Add webhook.

Step 2: Add DockerHub credential in Jenkins

To build and push the Docker image to DockerHub, we need to add docker credentials in Jenkins. Go to your DockerHub account and select the dropdown near username and click Account settings.

Click Security in the left panel and click New Access Token. Give a description and click Generate. Copy the token and close. Keep this token as we will be adding this to the Jenkins credential in the next steps.

Go to the Jenkins dashboard -> Manage Jenkins -> Manage credentials. Click System -> Global credentials and then click Add credentials. Select the kind as Username and Password and enter your DockerHub username and enter the token in Password that we have generated in dockerHub. Give the ID and description of your choice but remember this ID as we will be using it in Jenkinsfile and click Create.

2.1. Create a Repository in DockerHub

Go to the DockerHub registry, select Repositories and click Create repository. Give a name to the repository and select visibility as Public and click Create.

Step 3: Setup Docker in EC2 Instance

Launch an EC2 instance and install docker into it. Open port 22 to SSH into the instance and open port 8081 as I will be exposing my Java application to port 8081. After successful deployment, you will be able to access the application in the browser using http://PUBLIC_IP:8081/hello

I have launched the Amazon Linux instance and setup Docker using the below commands:

# Update packages
sudo yum update
# Install Docker
sudo yum install docker
# Add ec2-user to Docer group
sudo usermod -a -G docker ec2-user
# Enable docker service at AMI boot time
sudo systemctl enable docker.service
# Start the Docker service
sudo systemctl start docker.service

3.1. Add EC2 credentials in Jenkins

Go to Jenkins Dashboard -> Manage Jenkins -> Manage Credentials. Click on Global and Add Credentials and select SSH Username with the private key under Kind. Enter the ID and Description of your choice. Make note of this ID as we will be using it in Jenkinsfile. Enter the username of the EC2 instance that you have launched. For the Amazon Linux instance, the username is ec2-user. Select Enter directly and click Add and copy-paste the private key that you have created while launching an instance and click Create.

Step 4: Write Jenkinsfile and Dockerfile

Write Jenkinsfile with all the below steps to fetch, build, test and deploy Java application. Remember to add the below Jenkinsfile in the root directory of your project in GitHub.

GitHub Project

pipeline {
  agent any

  environment {
    DOCKERHUB_CREDENTIALS = credentials('docker-hub-cred')
    REMOTE_SERVER = 'your-remote-server-ip'
    REMOTE_USER = 'your-remote-server-user'            
  }

  // Fetch code from GitHub

  stages {
    stage('checkout') {
      steps {
        git branch: 'main', url: 'https://github.com/palakbhawsar98/JavaWebApp'

      }
    }

   // Build Java application

    stage('Maven Build') {
      steps {
        sh 'mvn clean install'
      }

     // Post building archive Java application

      post {
        success {
          archiveArtifacts artifacts: '**/target/*.jar'
        }
      }
    }

  // Test Java application

    stage('Maven Test') {
      steps {
        sh 'mvn test'
      }
    }

   // Build docker image in Jenkins

    stage('Build Docker Image') {

      steps {
        sh 'docker build -t javawebapp:latest .'
        sh 'docker tag javawebapp palakbhawsar/javawebapp:latest'
      }
    }

   // Login to DockerHub before pushing docker Image

    stage('Login to DockerHub') {
      steps {
        sh 'echo $DOCKERHUB_CREDENTIALS_PSW | docker login -u $DOCKERHUB_CREDENTIALS_USR --password-stdin'
      }
    }

   // Push image to DockerHub registry

    stage('Push Image to dockerHUb') {
      steps {
        sh 'docker push palakbhawsar/javawebapp:latest'
      }
      post {
        always {
          sh 'docker logout'
        }
      }

    }

   // Pull docker image from DockerHub and run in EC2 instance 

    stage('Deploy Docker image to AWS instance') {
      steps {
        script {
          sshagent(credentials: ['awscred']) {
          sh "ssh -o StrictHostKeyChecking=no ${REMOTE_USER}@${REMOTE_SERVER} 'docker stop javaApp || true && docker rm javaApp || true'"
      sh "ssh -o StrictHostKeyChecking=no ${REMOTE_USER}@${REMOTE_SERVER} 'docker pull palakbhawsar/javawebapp'"
          sh "ssh -o StrictHostKeyChecking=no ${REMOTE_USER}@${REMOTE_SERVER} 'docker run --name javaApp -d -p 8081:8081 palakbhawsar/javawebapp'"
          }
        }
      }
    }
  }
}

Also, write Dockerfile with the instructions to build the Java project and keep this file in the root directory of the project.

#Define your base image 
FROM eclipse-temurin:17-jdk-focal 

#Maintainer of this image
LABEL maintainer="Palak Bhawsar" 

#Copying Jar file from target folder                                                                                       
COPY target/web-services.jar web-services.jar  

#Expose app to outer world on this port                                                                                                                                                                                                                                                                          
EXPOSE 8081   

#Run executable with this command  
ENTRYPOINT ["java", "-jar", "web-services.jar"]

Step 5: Create a Jenkins pipeline

Go to Jenkins Dashboard click New Item -> Give a name to the pipeline -> Select Pipeline -> Click Ok

Add Description of your pipeline -> Build Triggers -> GitHub hook trigger for GITScm polling. Scroll to the last in the Pipeline section and from the dropdown select Pipeline script from SCM. Under SCM, select Git and enter your GitHub project repository URL. If your GitHub repository is private then add credentials.

Also, enter the branch name in Branches to build and the Jenkinsfile name in Script Path and click Save. Finally, Click Build Now to run the pipeline.

The Jenkins pipeline run successfully and the docker image got deployed to the EC2 instance.

Access your Java Application in the browser using the public IP of the AWS instance at port 8081

http://publicIP:8081/hello

Congratulations🥳🥳, we have successfully created an Automated CI/CD pipeline for Java application.

If you face any issues contact me on my socials 👉 Contact me

Thank You

Palak Bhawsar

Create free tier AWS account, create IAM user and set CloudWatch alarm for billing

Palak Bhawsar — Mon, 27 Feb 2023 17:02:31 +0000

In this article, we will see how to create a free tier AWS account step by step. We will create the IAM user and enable the Multi-factor Authentication (MFA) for the root user and IAM user to secure your account. Also to safeguard our AWS account from over-expenditure we will set billing alarms and create an SNS topic that will send you an email notification if you go beyond a certain limit.

Let's first understand some of these terminologies:

Amazon Web Services (AWS): AWS is a cloud provider platform that provides computing, networking and storage services on demand that scales easily.

Root User: The root user is the account owner and is created when the AWS account is created. This is the default user and should not be used or shared.

IAM User: An IAM user is a resource to give access to your AWS account to specific users and provides them specific permissions to access resources in your AWS account.

IAM Policy: IAM policies define permissions for actions that users or groups can perform in an AWS account. Users and groups are assigned JSON documents called policies.

AWS CloudWatch alarm: CloudWatch alarm helps you to watch CloudWatch metrics and to receive notifications when the metrics fall outside of the levels (high or low thresholds) that you configure. In our case, we are setting a CloudWatch alarm to monitor the billing threshold.

Simple Notification Service (SNS): SNS is a managed service that provides message delivery from publishers to subscribers. Publishers communicate asynchronously with subscribers by sending messages to a topic. SNS topic is a logical access point that acts as a communication channel.

Step 1: Create an AWS account

Go to the below AWS site to create a free tier account and click Create a Free Account button.

Free tier account

Enter your email address and give any name to your AWS account. You can also change this name later. Click Verify email address. You will get the verification code in your email. Enter the code and click Verify.

Create a root user password. Make sure that you create a strong password containing more than 8 digits then click Continue.

In the next step, select Personal - for your own projects. Enter your Full Name, Phone Number, Country or Region, and your address details. Click the checkbox after reading AWS Customer Agreement and click continue to the next steps.

In the next step, enter your Credit or Debit card details. It will deduct a minimal amount just for verification purposes and click continue. On the next page, it will ask for the OTP and Rs. 2.00 will get deducted from your account. Finally, enter the OTP and click on Submit button.

Confirm your Identity on the next page by verifying your Phone number. This phone number will be used to send you verification codes in the future. Select text message and enter your phone number, do the security check and click Send SMS. Enter the OTP sent to your phone number.

Choose a Basic support-Free plan which comes under the free tier and finally click on Complete Sign up.

Go to the AWS management console and select sign in to the console and sign in using the email and password that we set up previously for the root user.

The root user has access to every AWS service and resource in an account. If the credentials for the root account are stolen, it may lead to unnecessary costs in your account therefore it's recommended to not use a root account and instead create an IAM user.

2. Enable Multi-factor authentication

Search for IAM in the services section and select IAM. The IAM dashboard will appear, in the security recommendation click Add MFA and then Assign MFA on the next page.

Give a name to the Device name box and select an Authenticator app and click next. Download the Google Authenticator app on your phone. It will generate a 6 digits verification code that you will have to enter whenever you log in as a root user.

Click on the Show QR code on the next page. Open the Google Authenticator app on your phone and click on the '+' button and scan the QR code visible on your AWS account screen. Enter the 6 digits code from the app in the box MFA code 1. Wait for 30 seconds and enter the next code in the box MFA code 2 and click Add MFA.

Multi-factor authentication is enabled for your root account.

3. Create IAM User

Go to Users in the left panel and click Add users. Give the name to the user and click next. This user will not have any permission by default. Attach the policy to this user to access AWS resources. Select Attach policies directly and AdministratorAccess for this user and click next and Create user. The user is created successfully.

4. Create credentials for IAM user

Click on your username and go to Security Credentials and click Enable console access under Console sign-in. Select enable and a custom password on the next window.

Create a strong password for this user and checkmark User must create new password at next sign-in and click on Apply and download the CSV file

Go to the dashboard and in right, click on the Create button . It will create the Account alias for your IAM user so that whenever you log in you don't have to enter the Account ID instead you can use the alias that you have created.

Enable MFA for IAM user : Go to User, click on your username and go to security credentials and click Assign MFA device. Set the MFA for this user in the same way that we have created for the root user.

5. Create a Billing Alert

To safeguard your account from using services that are not under the free tier and to get email notifications whenever you cross the minimum bill amount we have to create a billing alarm. Go to the dropdown under your root user and click Billing dashboard.

Click on Bills in the right panel and then Billing preferences. Select the checkbox for Receive PDF Invoice By Email and Receive Free Tier Usage Alerts and Receive Billing Alerts. Give your email address to receive the notifications for billing alerts and click save preferences.

6. Create CloudWatch Alarm

You can monitor your estimated AWS charges by using Amazon CloudWatch. Search for CloudWatch in AWS services and select it. Billing metric data is stored in the US East (N. Virginia), make sure you are in N. Virginia region before creating alarms.

Click on All alarms in the left panel and select Create alarm. Select metric and scroll down, you will see billing as we have created an alert for billing in the previous step.

Click on Billing and Total estimated charge. Click USD and Select metric. On the next page define the threshold value. If you want to get an alert if your billing amount gets more than 2 USD then enter that value and click Next.

7. Create an SNS Topic

Select create new topic and give a name to this topic and enter your email address then click Create topic and click Next. On the next page give a name to the alarm and click Next and Create alarm.

Go to your email and confirm the subscription. The alarm will trigger when your account billing exceeds the threshold you have specified. The status is Ok as my billing is less than 2 USD.

Now, Sign out from the root user. Sign in as an IAM user and use an alias (instead of AccountID) that we created in the previous step. In the username enter the name of the IAM user and the password that we set up. Get the verification code from google authenticator for IAM users and click sign in. The window will prompt you to set a password. Create a new password, and make sure it's strong and has more than 8 digits.

Congratulations🥳🥳, we have created AWS free tier account and learned about IAM users, IAM policy, and CloudWatch alarm.

If you face any issues contact me on my socials 👉 Contact m e

Create pipeline using Jenkins GitHub and AWS for Java Project

Palak Bhawsar — Tue, 31 Jan 2023 06:49:02 +0000

In this article, we are going to create a continuous integration pipeline for a Maven project. This pipeline will fetch the code from GitHub, build the code, run the test cases, and will store the generated artifacts in the AWS S3 bucket.

Prerequisite:

AWS account with user permission to upload files in S3
AWS access key and secret access key for a user
GitHub account
Understanding of Jenkins
Familiarity with Maven commands to build and run Java project

Let's first understand what Jenkins, GitHub, Webhook, S3, and Maven are:

Jenkins is an open-source automation tool written in Java with plugins built for Continuous Integration and Continuous deployment/delivery purposes.

GitHub is a code hosting platform for collaboration and version control. It lets you and others work together on projects.

Webhook is a mechanism to automatically trigger the build of a Jenkins project in response to a commit pushed to a Git repository.

Maven is a powerful project management tool that is based on POM (project object model). It is used for project build, dependency, and documentation.

JUnit is an open-source Unit Testing Framework for JAVA. It is useful for Java Developers to write and run repeatable tests.

Amazon S3 (Simple Storage Service) is an object storage service that offers industry-leading scalability, data availability, security, and performance.

1. Launch an EC2 instance and Install Jenkins

First set up the Jenkins server to create a pipeline. Launch an EC2 instance and install Java and Jenkins into it.

Follow this article to set up Jenkins server

You need to install Maven as well to build the project. SSH into the instance and install maven using the below command.

sudo apt-get install maven -y

Set JAVA_HOME and MAVEN_HOME using the below commands. To set the variable permanently, you should add it to the bashrc file in your home directory.

echo "export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64" >> ~/.bashrc

echo "export MAVEN_HOME=/usr/share/maven" >> ~/.bashrc

Check if the variables are exported correctly.

tail -3 ~/.bashrc

2. Install Plugins

Install plugins to integrate Jenkins with GitHub, Maven, and AWS S3. Go to Manage Jenkins , and select Manage plugins. Under available plugins search for the below plugins and Install without restart

GitHub

Maven Integration

S3 publisher

3. Configure Java and Maven in Jenkins

Go to Manage Jenkins, select Global tool configuration , and scroll down to add JDK and Maven path that we exported in the above steps as shown below. Uncheck the Install automatically checkbox and click Save.

4. Create S3 bucket to store artifacts

Search for S3 in AWS services and click Create bucket. Give a unique name to the bucket and choose the region in which you want your bucket to reside. Leave all the setting default and click Create bucket at last.

Go to Jenkins dashboard, select Manage Jenkins then configure System. Scroll to the end to add Amazon S3 profiles. Give S3 bucket name in the profile name and access key and secret access key as shown below and click Save.

5. Create Webhook

6. Create Jenkins pipeline

Go to Jenkins dashboard and click New item. Enter an item name and select Freestyle project and click Ok. Give a small description of your project and select GitHub project, and enter the URL of your GitHub project. Scroll down and select Git as source code management and enter the repository URL of your project. If your repository is private then add credentials. Enter your branch name under Branches to build section.

Select the GitHub hook trigger for GITScm polling under build triggers. In Build Steps select Invoke top-level Maven targets. Select MAVEN_HOME which we configured in the maven version and enter the clean install test in Goals.

In Post-build Actions, select Publish artifacts to S3 Bucket. Select your AWS profile name and in the source enter the artifact path that you want to store in the S3 bucket. Give the name of your S3 bucket in the Destination bucket. Select the bucket region in which you have created your bucket. Keep all other settings default and click apply save.

Finally, click Build now in the left panel, and you will see the pipeline run successfully.

Check the artifact stored in the S3 bucket after the successful build.

Congratulations 🥳, the pipeline ran successfully and artifacts got stored in S3

If you face any issues contact me on my socials 👉 Contact m e

Easily manage your Terraform Remote State file with Terraform Cloud

Palak Bhawsar — Tue, 03 Jan 2023 10:50:26 +0000

In this article, we will see how to store terraform remote state file in the terraform cloud. Also how to create Terraform cloud account and workspaces to manage remote state file. We are storing terraform state file in Terraform cloud to keep it safe as it contains sensitive information about the infrastructure.

What is Terraform?

Terraform is an open-source infrastructure as a code tool created by HashiCorp, that lets you provision, build, change, and version cloud and on-prem resources. It lets you define both Cloud and on-prem resources in human-readable configuration files that you can version reuse, and share.

What is Infrastructure as Code?

Infrastructure as code (IaC) is the process that allows you to manage infrastructure with configuration files rather than through a graphical user interface. IAC allows you to build, change, and manage your infrastructure in a safe, consistent, and repeatable way by defining resource configurations that you can version, reuse, and share.

What is Terraform state file?

When we create infrastructure after executing terraform apply command. Terraform creates a state file called terraform.tfstate. This state file contains all the information about the resources created using Terraform and keeps track of resources created by your configuration and maps them to real-world resources. Terraform state file is a sensitive file as it contains information about the infrastructure that we have created.

You should never push terraform state file to any version control system like GitHub. Store terraform.tfstate file in the backend to keep it safe.

What is Terraform cloud?

Terraform Cloud is HashiCorps managed service offering. It eliminates the need for unnecessary tooling and documentation for practitioners, teams, and organizations to use Terraform in production. Terraform Cloud enables infrastructure automation for provisioning, compliance, and management of any cloud, data center, and service.

The backend supported by Terraform:

Amazon S3
Azure Storage
Google Cloud Storage
HashiCorps Terraform Cloud and Terraform Enterprise.

In this article, we are using Terraform Cloud as backend to store Terraform state file.

Prerequisite

AWS account and AWS Access key and Secret key created
Terraform installed on your IDE
AWS CLI installed and configured on your IDE
Basic understanding of AWS services and Terraform

Step-1: Create Terraform cloud account

Go to https://app.terraform.io/ and sign-up for a free account. After login creates an organization and a workspace. Choose Start from the scratch workflow.

Step-2: Create an organization

Create a new organization, enter your organization name and email address as shown below, and click Create organization.

Step-3: Create a workspace

Choose the workflow of your choice, here I am selecting a CLI-driven workflow.

Give a name and description to your workflow and click Create workspace.

Step-4: Clone repository

Clone this code, add the backend block in the configuration file and enter your organization-name and workspace name that you have created in previous steps. This block ensures that terraform.tfstate file doesn't get created in your workspace locally and it gets created in the remote backend i.e. Terraform cloud.

terraform {
  backend "remote" {
    organization = "organization-name"
    workspaces {
      name = "terraform-backend-handson"
    }
  }

Step-5: Add environment variables

Go to the variables section in the left bar and click Add variable. Here we can add variables that will be used by configuration files.

Select the Environment variable and Add AWS_ACCESS_KEY and SECRET_ACCESS_KEY and don't forget to checkbox Sensitive and click Save variable.

After adding environment variables the console will look like this. You can also add terraform variables in this step.

Step-6: Terraform Login

Go to your project directory in the console and execute terraform login command and enter the value yes. Terraform will open the browser. Click Create API token.

Copy the API token and paste it into the terminal where we have executed terraform login command.

Now run the terraform init, terraform plan, and terraform apply commands to provision the infrastructure, You will see the console where the build starts.

After the triggered build is successful you will able to see the resources created in the AWS console, as well as the state file in Terraform Cloud.

Congratulations, triggered CLI workflow run successfully and the state file got created in terraform cloud.

If you face any issues contact me on my socials 👉 Contact m e

Terraform Cloud: Manage Terraform Remote State file

Palak Bhawsar — Tue, 03 Jan 2023 10:50:26 +0000

What is Terraform?

Terraform is an open-source infrastructure as a code tool created by HashiCorp, that lets you provision, build, change, and version cloud and on-prem resources. It lets you define both Cloud and on-prem resources in human-readable configuration files that you can version reuse, and share.

What is Infrastructure as Code?

Infrastructure as code (IaC) is the process that allows you to manage infrastructure with configuration files rather than through a graphical user interface. IAC allows you to build, change, and manage your infrastructure in a safe, consistent, and repeatable way by defining resource configurations that you can version, reuse, and share.

What is Terraform state file?

The backend supported by Terraform:

Amazon S3
Azure Storage
Google Cloud Storage
HashiCorps Terraform Cloud and Terraform Enterprise.

In this article, we are using Terraform Cloud to store Terraform state file.

Prerequisite

AWS account and AWS Access key and Secret key created
Terraform installed on your IDE
AWS CLI installed and configured on your IDE
Basic understanding of AWS services and Terraform

Step-1: Create Terraform cloud account

Go to https://app.terraform.io/ and sign-up for a free account. After login creates an organization and a workspace. Choose Start from the scratch workflow.

Step-2: Create an organization

Create a new organization, enter your organization name and email address as shown below, and click Create organization.

Step-3: Create a workspace

Choose the workflow of your choice, here I am selecting a CLI-driven workflow.

Give a name and description to your workflow and click Create workspace.

Step-4: Clone repository

Clone this repository https://github.com/palakbhawsar98/Terraform/tree/main/terraform-remote-state-hands-on and Add the backend block in the configuration file and enter your organization-name and workspace name that you have created in previous steps. This block ensures that terraform.tfstate file doesn't get created in your local but in the remote backend i.e Terraform cloud.

terraform {
  backend "remote" {
    organization = "organization-name"
    workspaces {
      name = "terraform-backend-handson"
    }
  }

Step-5: Add environment variables

Go to the variables section in the left bar and click Add variable. Here we can add variables that are used by configuration files.

Select the Environment variable and Add AWS_ACCESS_KEY and SECRET_ACCESS_KEY and don't forget to checkbox Sensitive and click Save variable.

After adding environment variables the console will look like this. You can terraform variables also in this step.

Step-6: Terraform Login

Go to your project directory in the console and execute terraform login command and enter the value yes. Terraform will open the browser. Click Create API token.

Copy the API token and paste it into the terminal.

Now run the terraform init, terraform plan, and terraform apply commands to provision the infrastructure, You can see the console where the build will start.

After the triggered build is successful you will able to see the resources created in the AWS console, as well as the state file in Terraform Cloud.

Terraform: Create VPC, Subnets, and EC2 instances in multiple availability zones

Palak Bhawsar — Sat, 17 Dec 2022 10:37:03 +0000

In this article, I will demonstrate how to create VPC, Subnets, EC2 instances, Internet gateway, NAT gateway, and Security groups using Terraform in two availability zones.

Architecture

Prerequisite

AWS account and AWS Access key and Secret key created
Terraform installed on your IDE
AWS CLI installed and configured on your IDE
Basic understanding of AWS services and Terraform

Objective

Choose a region in which you want your VPC to reside and availability zones where you want to create public and private subnets for high availability.
Decide the CIDR blocks range for your VPC and Subnets.
Create public and private subnets in each availability zone.
Create an internet gateway to allow communication between your VPC and the internet.
Create an EC2 instance in each public subnet in both the availability zones and create AWS key pair to SSH into your instances.
Create a route table for the public and private subnets and associate the route table with subnets to control where network traffic is directed.
Create a NAT gateway to enable private subnets to connect to services outside your VPC. A NAT gateway must be in a public subnet.
Finally, create a VPC security group and open port 80 to allow HTTP traffic from anywhere and open port 22 to SSH into the instances.

Code Repository

Use GitHub to find providers.tf, variables.tf, and outputs.tf files.

Let's get started with the configuration of the project

Create VPC

We are creating VPC in the us-east-1 region and attaching it to the internet gateway.

resource "aws_vpc" "vpc" {
  cidr_block = "10.0.0.0/16"
  tags = {
    Name = "my-vpc"
  }
}

resource "aws_internet_gateway" "internet_gateway" {
  vpc_id = aws_vpc.vpc.id
  tags = {
    Name = "inernetGW"
  }
}

Create public and private subnets

Creating one public and one private subnet in both us-east-1a and us-east-1b zones.

resource "aws_subnet" "vpc_public_subnet" {
  vpc_id = aws_vpc.vpc.id
  count = length(var.subnets_count)
  availability_zone = element(var.availability_zone, count.index)
  cidr_block = "10.0.${count.index}.0/24"
  map_public_ip_on_launch = true

  tags = {
    Name = "pub-sub-${element(var.availability_zone, count.index)}"
  }
}

resource "aws_subnet" "vpc_private_subnet" {
  count = length(var.subnets_count)
  availability_zone = element(var.availability_zone, count.index)
  cidr_block = "10.0.${count.index + 2}.0/24"
  vpc_id = aws_vpc.vpc.id

  tags = {
    Name = "pri-sub-${element(var.availability_zone, count.index)}"
  }
}

Create a route table and associate it with the public subnet

A route table contains a set of rules that are used to determine where network traffic is directed. Associate a public subnet with the default route (0.0.0.0/0) pointing to an internet gateway.

resource "aws_route_table" "public_route_table" {
  vpc_id = aws_vpc.vpc.id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.internet_gateway.id
  }
  tags = {
    Name = "public-route-tbl"
  }
}

resource "aws_route_table_association" "public_route_table_association" {
  count = length(var.subnets_count)
  subnet_id = element(aws_subnet.vpc_public_subnet.*.id, count.index)
  route_table_id = aws_route_table.public_route_table.id
}

Create a NAT gateway and associate it with Elastic IP

Create a public NAT gateway in a public subnet and associate it with an elastic IP address to route traffic from the NAT gateway to the Internet gateway for the VPC.

resource "aws_eip" "elasticIP" {
  count = length(var.subnets_count)
  vpc = true
}

resource "aws_nat_gateway" "nat_gateway" {
  count = length(var.subnets_count)
  allocation_id = element(aws_eip.elasticIP.*.id, count.index)
  subnet_id = element(aws_subnet.vpc_public_subnet.*.id, count.index)

  tags = {
    Name = "nat-GTW-${count.index}"
  }
}

Create a route table and associate it with the private subnet

resource "aws_route_table" "private_route_table" {
  count = length(var.subnets_count)
  vpc_id = aws_vpc.vpc.id
  route {
    cidr_block = "0.0.0.0/0"
    nat_gateway_id = aws_nat_gateway.nat_gateway[count.index].id
  }

  tags = {
    Name = "private-route-tbl"
  }
}

resource "aws_route_table_association" "private_route_table_association" {
  count = length(var.subnets_count)
  subnet_id = element(aws_subnet.vpc_private_subnet.*.id, count.index)
  route_table_id = element(aws_route_table.private_route_table.*.id,
  count.index)
}

Create security group

For the inbound connections open port 80 to allow HTTP traffic from anywhere and open port 22 to SSH into the instance and open all the ports for the outbound connections.

resource "aws_security_group" "vpc_sg" {
  name = "vpc_sg"
  description = "Security group for vpc"
  vpc_id = aws_vpc.vpc.id

  ingress {
    from_port = 22
    to_port = 22
    protocol = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    from_port = 80
    to_port = 80
    protocol = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port = 0
    to_port = 0
    protocol = -1
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name = "vpc-sg"
  }

}

Create EC2 instances in public subnets

Create EC2 instance with user-data scripts to install Apache server and access static webpage. Also, create AWS key pair to SSH into instances

resource "tls_private_key" "key" {
  algorithm = "RSA"
  rsa_bits = 4096
}

resource "local_file" "private_rsa_key" {
  content = tls_private_key.key.private_key_pem
  filename = "private_rsa_key"
}

resource "aws_key_pair" "public_rsa_key" {
  key_name = "public_rsa_key"
  public_key = tls_private_key.key.public_key_openssh
}

resource "aws_instance" "my_app_server" {
  ami = var.instance_ami
  instance_type = var.instance_size
  key_name = aws_key_pair.public_rsa_key.key_name
  count = length(var.subnets_count)
  subnet_id = element(aws_subnet.vpc_public_subnet.*.id, count.index)
  security_groups = [aws_security_group.vpc_sg.id]
  associate_public_ip_address = true

  user_data = <<-EOF
  #!/bin/bash
  sudo apt update -y
  sudo apt install apache2 -y
  sudo systemctl start apache2
  sudo systemctl enable apache2
  sudo apt install git -y
  git clone https://github.com/palakbhawsar98/FirstWebsite.git
  cd /FirstWebsite
  sudo cp index.html /var/www/html/
  EOF

  tags = {
    Name = "my_app_server-${count.index}"
  }
}

We are ready to deploy all our changes to AWS. Perform the below commands:

terraform init to initialize the working directory and download all the plugins for providers.
terraform fmt to rewrite Terraform configuration files to a canonical format and style.
terraform validate to check that our code is error-free.
terraform plan to create the execution plan for the resources we are going to create in AWS.
terraform apply to execute the actions proposed in a terraform plan and to deploy your infrastructure.

You can see the resources created in AWS Console. Take the public IPV4 and search it in the browser on port 80 that we opened for HTTP connections.

You can access the HTML static webpage using public IPV4

If you want you can destroy the infrastructure we just create using terraform destroy command.

Launch an EC2 Instance using Terraform

Palak Bhawsar — Mon, 31 Oct 2022 10:23:50 +0000

Terraform is an open-source infrastructure as code software tool created by HashiCorp. It enables users to define and provision infrastructure using a high-level configuration language known as HCL (Hashicorp Configuration Language). Terraform is an IAC tool, used primarily by DevOps teams to automate various infrastructure tasks. The provisioning of cloud resources, for instance, is one of the main use cases of Terraform. It's a cloud-agnostic, open-source provisioning tool written in the Go language and created by HashiCorp.

IAC or Infrastructure as Code enables us to manage infrastructure in the form of code. Using Terraform we can automate the process of provisioning and destroying infrastructure.

Let's get started. . .

1. Download Terraform on a Linux machine

Open your Linux terminal and run the following commands to Install Terraform.

sudo apt-get update && sudo apt-get install -y gnupg software-properties-common curl
curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
sudo apt-get update
sudo apt-get install terraform

Check the downloaded version of Terraform using the below command:

terraform version

2. Create AWS User with permission

Create a User and give it permission to interact with AWS from the local machine. Login to the AWS account and in services search for IAM. In side pane select Users and click Add users , enter name of the user and check the box Access key - Programmatic access and click next:Permissions

Select Attach existing policies directly from the top menu and check AmazonEC2FullAccess and click Next: Tags , Review , and Add user.

Copy and keep the Access key ID and Secret access key safe for later use or Download the CSV file.

3. Configure AWS keys

Install AWS CLI using the below commands:

sudo apt install awscli


aws configure

Enter the region and access key details when prompted:

AWS Access Key ID :
AWS Secret Access key:
Default region: us-east-1
Default Output: json

Configure profile to store access key and secret access key. Cat config file located at path ~/.aws/

Edit the file and enter the access key and secret access key and give a name to profile

[default]
aws_access_key_id = <ACCESS_KEY_ID>
aws_secret_access_key = <SECRET_ACCESS_KEY>
region = us-east-1

4. Create terraform configuration file

A Terraform configuration is a complete document in the Terraform language that tells Terraform how to manage a given collection of infrastructure. A configuration can consist of multiple files and directories. Create a configuration file with a .tf extension in any editor or IDE and save the file.

provider "aws" {
   region = "us-east-1"
   }

resource "aws_instance" "my_ec2_instance" {
  ami = "ami-08c40ec9ead489470"
  instance_type = "t2.micro"

  tags = {
    Name = "FirstEC2Instnace"
  }
}

To format the configuration file run the below command in the terminal in the same directory where the .tf file is present.

terraform fmt

Initialize terraform using the below command. It's the same as the git init command in GIT. This command will initialize the working directory containing Terraform configuration files and install any required plugins

terraform init

To check that the configuration file is error-free, run the following command after terraform init:

terraform validate

Run the below which lets you preview the actions Terraform would take to modify your infrastructure or save a speculative plan which you can apply later.

terraform plan

Finally, run the below command. This is the same as git push. This command executes the actions proposed in a terraform plan. It is used to deploy your infrastructure. Typically apply should be run after terraform init and terraform plan. Enter yes when prompted "Enter a value:".

terraform apply

You can see the instance is running in your AWS Console. You will see terraform.tfstate file is created after running the terraform apply command. This state is used by Terraform to map real-world resources to your configuration, keep track of metadata, and improve performance for large infrastructures. This state is stored by default in a local file named "terraform. tfstate", but it can also be stored remotely, which works better in a team environment.

5. Destroy infrastructure

To destroy the resources you have just created run the below command. The terraform destroy command terminates resources managed by your Terraform project. This command is the inverse of terraform apply in that it terminates all the resources specified in your Terraform state. It does not destroy resources running elsewhere that are not managed by the current Terraform project.

terraform destroy