DEV Community: Arjun Menon

Building a Mass Emailing System Using AWS Lambda and SES

Arjun Menon — Wed, 01 May 2024 07:00:48 +0000

A mass emailing system allows businesses to reach out to customers efficiently, and building one using Amazon Web Services (AWS) can be both scalable and cost-effective. In this blog post, I'll guide you through setting up a mass emailing system using AWS Lambda, Simple Email Service (SES), and S3.

Step 1: Create an S3 Bucket

Amazon S3 will store the CSV files containing the email addresses of the recipients.

Navigate to S3: In the AWS Management Console, click on S3 under the Storage category.
Create a Bucket: Click on Create bucket. Name your bucket (it must be unique across AWS), select a region close to you for better performance, and then create the bucket.

Step 2: Set Up AWS Simple Email Service (SES)

AWS SES will handle the email sending operations.

Navigate to SES
Verify Your Email Address: Before sending emails, SES requires verification of your ownership of the email address. On the left pane, go to Identities, and then verify the email address you want to send from.

Important Note: As we are using SES in sandbox mode, you must manually verify each email address that you wish to send emails to before they can receive emails from your application. This is a restriction in the sandbox environment intended to prevent spam and abuse.

Add and verify all Recipient Email Addresses: You can use Temp Mail for testing.

Step 3: Create an IAM Role for Lambda

This role will allow your Lambda function to access AWS resources securely.

Navigate to IAM: Go to IAM under Security, Identity, & Compliance.
Create a New Role: Click on Roles, then Create role. Choose Lambda for the service that will use this role, then proceed to permissions.
Attach Policies: Add policies like AmazonSESFullAccess, AmazonS3FullAccess, and AWSLambdaExecute.

Step 4: Create a Lambda Function

This function will process the CSV file and send emails.

Navigate to Lambda: Go to Lambda under Compute.
Create Function: Choose Author from scratch, name your function, select Python 3.x for the runtime.
Set Permissions: Choose the IAM role you created earlier.
Create Function: Finalize the creation of your function.

Step 5: Write Lambda Function Code

The function will read from S3, parse CSV, and send emails via SES.

import boto3import csvdef lambda_handler(event, context): s3 = boto3.client('s3') ses = boto3.client('ses') bucket_name = event['Records'][0]['s3']['bucket']['name'] key = event['Records'][0]['s3']['object']['key'] response = s3.get_object(Bucket=bucket_name, Key=key) lines = response['Body'].read().decode('utf-8').split() for row in csv.DictReader(lines): ses.send_email( Source='your-verified-email@example.com', Destination={'ToAddresses': [row['email']]}, Message={ 'Subject': {'Data': 'Your Email Subject'}, 'Body': {'Text': {'Data': "Your email content"}} } )

Step 6: Set Up S3 Event Trigger

Trigger your Lambda function when a CSV file is uploaded to S3.

Select Your Function: In Lambda, open your functions configuration.
Add Trigger: Choose S3 from the list. Configure the trigger for PUT events, and set the file prefix or suffix (like .csv).
Save: Save your configuration.

Step 7: Test Your Setup

Upload a CSV file with an email column to your S3 bucket and check the Lambda execution logs. Make sure the emails are being sent out successfully.

Email Recieved:

Reminder: As we are using SES in sandbox mode, you must manually verify each email address that you wish to send emails to before they can receive emails from your application. This is a restriction in the sandbox environment intended to prevent spam and abuse.

Conclusion

You've now successfully set up a basic mass emailing system using AWS. This system is not only scalable but also cost-effective, perfect for businesses looking to manage their email marketing efforts. Always ensure to comply with legal requirements and manage opt-ins and opt-outs to respect user privacy and avoid spam.

Monitoring Docker Logs Using Grafana and Prometheus

Arjun Menon — Sun, 21 Jan 2024 14:24:01 +0000

Introduction

In this comprehensive tutorial, we'll guide you through the process of sending Docker logs to Grafana for effective monitoring and visualization. This setup involves launching two EC2 instances, installing Grafana on Instance 1, and Prometheus with Docker on Instance 2. By the end of this guide, you'll have a fully functional Grafana dashboard displaying Docker container metrics.

Prerequisites

AWS account with EC2 instances
Basic knowledge of Linux commands
SSH access to both EC2 instances

Step 1: Launch EC2 Instances

Launch two EC2 instances on AWS, naming them Instance 1 and Instance 2. Ensure you have the necessary security groups, key pairs, and IAM roles configured for smooth communication.

Step 2: Install Grafana on Instance 1

SSH into Instance 1 and install Grafana using the following commands:

sudo apt-get updatesudo apt-get install -y software-properties-commonsudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -sudo apt-get updatesudo apt-get install -y grafanasudo systemctl start grafana-serversudo systemctl enable grafana-server

Access Grafana's web interface by navigating to http://<instance_1_ip>:3000 in your browser. Default login credentials are usually admin/admin.

Step 3: Install Prometheus and Docker on Instance 2

SSH into Instance 2 and install Prometheus and Docker:

sudo apt-get updatesudo apt-get install -y prometheus docker.io

Step 4: Configure Docker Metrics Collection

Create a file named daemon.json in /etc/docker/ with the following content:

{ "metrics-addr": "0.0.0.0:9323", "experimental": true}

Restart Docker:

sudo systemctl restart docker

You should now see your docker metrics by going to <instance_2_ip>:9323/metric.

Step 5: Configure Prometheus for Docker Metrics

Edit the prometheus.yml file in /etc/prometheus/ and add a new job:

- job_name: 'docker_metrics' static_configs: - targets: ['<instance_2_ip>:9323']

Restart Prometheus:

sudo systemctl restart prometheus

Verify the new job is visible in Prometheus targets.

Step 6: Configure Grafana Data Source

Log in to Grafana.
Navigate to "Settings" > "Data Sources."
Click "Add your first data source" and select Prometheus.
Set the URL to <instance_2_ip>:9090 and click "Save & Test."

Step 7: Create Grafana Dashboard

Go to the "+" icon on the left sidebar and select "Dashboard."
Click "Add new panel" and choose Prometheus as the data source.
Use the sample query:
Customize the panel and dashboard as needed.

Conclusion

Congratulations! You've successfully set up Grafana to visualize Docker container metrics collected by Prometheus. This comprehensive guide ensures you have a smooth experience from launching EC2 instances to creating a functional Grafana dashboard. Feel free to explore more queries and customize your dashboard for a tailored monitoring solution.

Follow me on LinkedIn.

Checkout my GitHub profile.

Deploying a Three-Tier Application with CI/CD using Jenkins, ReactJS, NodeJS, and MongoDB on Kubernetes

Arjun Menon — Sat, 13 Jan 2024 17:17:29 +0000

Introduction

In this tutorial, we will guide you through the process of deploying a three-tier application on Kubernetes using ReactJS for the frontend, NodeJS for the backend, and MongoDB as the database. We'll use the provided GitHub repository and Docker containers for each tier. Additionally, we'll create Kubernetes manifest files to orchestrate the deployment.

Prerequisites

Before starting, ensure you have the following prerequisites:

AWS account with IAM administrator access.
An EC2 instance with Ubuntu in your preferred region.
All the required code is available in this GitHub repository.

Step 1: IAM Configuration

Create a user eks-admin with AdministratorAccess and generate security credentials (Access Key and Secret Access Key).

# AWS CLI commandaws configure

Step 2: EC2 Setup

Launch an Ubuntu instance and SSH into it.

Step 3: Install AWS CLI v2

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"sudo apt install unzipunzip awscliv2.zipsudo ./aws/install -i /usr/local/aws-cli -b /usr/local/bin --update

Step 4: Install Docker

sudo apt-get updatesudo apt install docker.iodocker pssudo chown $USER /var/run/docker.sock

Step 5: Install kubectl

curl -o kubectl https://amazon-eks.s3.us-west-2.amazonaws.com/1.19.6/2021-01-05/bin/linux/amd64/kubectlchmod +x ./kubectlsudo mv ./kubectl /usr/local/binkubectl version --short --client

Step 6: Install eksctl

curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmpsudo mv /tmp/eksctl /usr/local/bineksctl version

Step 7: Setup EKS Cluster

eksctl create cluster --name three-tier-cluster --region us-west-2 --node-type t2.medium --nodes-min 2 --nodes-max 2aws eks update-kubeconfig --region us-west-2 --name three-tier-clusterkubectl get nodes

Step 8: Clone GitHub Repository

git clone https://github.com/ArjunMnn/TWSThreeTierAppChallengecd TWSThreeTierAppChallenge

Step 9: Build Docker Images

Build Docker images for all three tiers (frontend, backend, and MongoDB). Refer to the provided Dockerfiles in the repository.

Step 10: Create Kubernetes Manifests

Use the provided manifest files to deploy the application on Kubernetes.

kubectl create namespace workshopkubectl apply -f backend-deployment.yamlkubectl apply -f backend-service.yamlkubectl apply -f frontend-deployment.yamlkubectl apply -f frontend-service.yamlkubectl apply -f deploy.yamlkubectl apply -f secrets.yamlkubectl apply -f service.yamlkubectl apply -f full_stack_lb.yaml

Step 11: Install AWS Load Balancer

curl -O https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.5.4/docs/install/iam_policy.jsonaws iam create-policy --policy-name AWSLoadBalancerControllerIAMPolicy --policy-document file://iam_policy.jsoneksctl utils associate-iam-oidc-provider --region=us-west-2 --cluster=three-tier-cluster --approveeksctl create iamserviceaccount --cluster=three-tier-cluster --namespace=kube-system --name=aws-load-balancer-controller --role-name AmazonEKSLoadBalancerControllerRole --attach-policy-arn=arn:aws:iam::626072240565:policy/AWSLoadBalancerControllerIAMPolicy --approve --region=us-west-2sudo snap install helm --classichelm repo add eks https://aws.github.io/eks-chartshelm repo update ekshelm install aws-load-balancer-controller eks/aws-load-balancer-controller -n kube-system --set clusterName=my-cluster --set serviceAccount.create=false --set serviceAccount.name=aws-load-balancer-controllerkubectl get deployment -n kube-system aws-load-balancer-controllerkubectl apply -f full_stack_lb.yaml

Step 12: Test the Application

Access the Application

Open your web browser and navigate to the DNS of the AWS Load Balancer. This will take you to the frontend of your three-tier application.

Create Tasks

On the frontend, you should see the application interface.
Click around and interact with the application, creating tasks to test the functionality.

Verify MongoDB Entries

To ensure that entries are being added to the MongoDB database, follow these steps:

Access MongoDB Shell

# Example SSH commandkubectl exec -it -n workshop <mongodb-pod-name> -- mongo

Check Database and Collection

use tododb.tasks.find()

This will show you the entries in the tasks collection of the todo database. You should see the tasks you created through the application.

Step 13: Configure Jenkins for CI/CD

Configure AWS Credentials

In Jenkins, go to "Manage Jenkins" > "Manage Credentials."
Under the "Stores scoped to Jenkins" section, click on "(global)".
Add new credentials:

Configure Kubeconfig Credential

In Jenkins, go to "Manage Jenkins" > "Manage Credentials."
Under the "Stores scoped to Jenkins" section, click on "(global)".
Add new credentials:

Note: Kubeconfig file is in the path /home/ubuntu/.kube/config. Download it to your local using scp.

Create a Jenkins Pipeline Job

In Jenkins, click on "New Item."
Choose "Pipeline" and give your job a name.
Under the "Pipeline" section, select "Pipeline script from SCM."
Set the SCM to Git and provide your repository URL.
In the "Script Path," specify the path to your Jenkinsfile (e.g., Jenkinsfile).
Save the job.

Trigger the Pipeline

Make a change in your GitHub repository, commit, and push.
Jenkins should automatically detect the change and trigger the pipeline.
Monitor the Jenkins dashboard for the progress of each stage in the pipeline.

Step 14: Verify CI/CD Deployment

Check AWS ECR

Open the AWS ECR console.
Verify that the images for the frontend and backend have been pushed successfully.

Check Kubernetes Deployment

Open your Kubernetes dashboard or use the following command:

kubectl --kubeconfig=/path/to/kubeconfig get deployments -n workshopkubectl --kubeconfig=/path/to/kubeconfig get services -n workshopkubectl --kubeconfig=/path/to/kubeconfig get pods -n workshop

Verify that the frontend and backend deployments are running.

Make a Change in GitHub

Make another change in your GitHub repository, commit, and push.
Jenkins should automatically trigger the pipeline again.

Verify Updated Deployment

Check the AWS ECR console for new image versions.
Monitor the Jenkins dashboard for the progress of each stage in the updated pipeline.
Verify that the changes reflect in your Kubernetes deployment.

Conclusion

If you can access the application through the DNS and see the tasks you created in the MongoDB database, congratulations! You have successfully deployed and tested your three-tier application on Kubernetes.

Remember to clean up your resources after testing by deleting the EKS cluster and any associated resources to avoid incurring unnecessary costs. Use the following command:

eksctl delete cluster --name three-tier-cluster --region us-west-2

This will delete the EKS cluster and associated resources.

CI/CD Pipeline using AWS CodeCommit, Codebuild, CodeDeploy and CodePipeline

Arjun Menon — Thu, 28 Dec 2023 12:47:51 +0000

Introduction

Welcome to this comprehensive guide where we'll walk through the process of creating a CI/CD pipeline for deploying a Todo List web app using AWS DevOps services. This tutorial is designed to be beginner-friendly, providing detailed steps for setting up CodeCommit, CodeBuild, CodeDeploy, and CodePipeline. We'll be installing Nginx and deploying a Todo List web app on an EC2 instance.

Code

All the required code is available at this repository.

Step 1: Setting Up CodeCommit

Create a CodeCommit Repository:
Create IAM User and Generate Credentials:
Clone Repository Locally:
Create Project Files:

git add .git commit -m "Initial commit"git push origin main

Step 2: Configuring CodeBuild

Create a Build Project:

# buildspec.yamlversion: 0.2phases: install: commands: - echo Installing Nginx... - sudo apt-get update - sudo apt-get install nginx -y build: commands: - echo Build started on `date` - cp index.html /var/www/html/ - cp styles.css /var/www/html/ - cp app.js /var/www/html/ - cp appspec.yml /var/www/html/ post_build: commands: - echo Restarting Nginx...artifacts: files: - index.html - appspec.yml - scripts/** - app.js - styles.css

Configure Artifacts:
Run the Build:

Step 3: Implementing CodeDeploy

Create CodeDeploy Application:
Create Deployment Group:
Install CodeDeploy Agent on EC2 Instance:

#!/bin/bash # This installs the CodeDeploy agent and its prerequisites on Ubuntu 22.04. sudo apt-get update sudo apt-get install ruby-full ruby-webrick wget -y cd /tmp wget https://aws-codedeploy-us-east-1.s3.us-east-1.amazonaws.com/releases/codedeploy-agent_1.3.2-1902_all.deb mkdir codedeploy-agent_1.3.2-1902_ubuntu22 dpkg-deb -R codedeploy-agent_1.3.2-1902_all.deb codedeploy-agent_1.3.2-1902_ubuntu22 sed 's/Depends:.*/Depends:ruby3.0/' -i ./codedeploy-agent_1.3.2-1902_ubuntu22/DEBIAN/control dpkg-deb -b codedeploy-agent_1.3.2-1902_ubuntu22/ sudo dpkg -i codedeploy-agent_1.3.2-1902_ubuntu22.deb systemctl list-units --type=service | grep codedeploy sudo service codedeploy-agent status

Create appspec.yaml and Scripts:

appspec.yaml:

# appspec.yamlversion: 0.0os: linuxfiles: - source: / destination: /var/www/html/hooks: AfterInstall: - location: scripts/install_nginx.sh timeout: 300 runas: root ApplicationStart: - location: scripts/start_nginx.sh timeout: 300 runas: root

install_nginx.sh

# scripts/install_nginx.sh#!/bin/bashapt-get updateapt-get install nginx -y

start_nginx.sh

# scripts/start_nginx.sh#!/bin/bashservice nginx start

Build and Deploy:
Configure EC2 Role:
Restart CodeDeploy Agent:

Step 4: Setting Up CodePipeline

Configure CodePipeline:
Create Pipeline:
Test the Web App:

Conclusion

Congratulations! You've successfully set up a CI/CD pipeline for deploying a Todo List web app using AWS DevOps tools. This tutorial has covered each step in detail, providing you with a solid foundation for implementing similar projects in the future.

Follow me on LinkedIn.

Checkout my GitHub profile.

Deploying a Todo App on AWS ECS with ECR

Arjun Menon — Mon, 25 Dec 2023 13:44:55 +0000

Introduction

In the world of cloud computing, deploying applications has become more efficient and scalable. Amazon Web Services (AWS) provides a range of services that enable developers to easily deploy and manage their applications. In this step-by-step guide, we'll walk through the process of deploying a Todo app on AWS ECS (Elastic Container Service) using ECR (Elastic Container Registry). This tutorial assumes you have a basic understanding of AWS services and have an AWS account.

Step 1: Set Up an EC2 Instance

Launch an EC2 instance with Amazon Linux or any other suitable Amazon Machine Image (AMI).
SSH into the instance.

ssh -i your-key.pem ec2-user@your-instance-ip

Step 2: Clone the Todo App Repository

Clone the Todo app repository from your version control system (e.g., GitHub).

git clone https://github.com/your-username/todo-app.gitcd todo-app

Step 3: Set Up ECR Repository

Navigate to the AWS Management Console and open the Elastic Container Registry (ECR).
Create a new repository named node-app.

Step 4: Install Docker and AWS CLI on EC2 Instance

Install Docker and the AWS CLI on your EC2 instance.

sudo apt update -ysudo apt install docker.iosudo usermod -a -G docker ubuntucurl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"unzip awscliv2.zipsudo ./aws/install

Step 5: Authenticate Docker with ECR

Run the following command to retrieve an authentication token and authenticate your Docker client with your ECR registry.

aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws/u1o8l9k6

Step 6: Build and Push Docker Image to ECR

Build your Docker image and push it to the ECR repository.

docker build -t node-app .docker tag node-app:latest public.ecr.aws/u1o8l9k6/node-app:latestdocker push public.ecr.aws/u1o8l9k6/node-app:latest

Step 7: Create ECS Cluster

Navigate to ECS in the AWS Management Console.
Create a new cluster named node-todo-cluster.
Choose Fargate as the launch type.

Step 8: Create Task Definition and Deploy

Create a new task definition named node-todo-app-task-definition.
Select the task definition and click on "Deploy," then "Run Task."

Step 9: Check the App

Once the task is running, note the public IP of your Fargate instance.
Open a web browser and navigate to <public-ip>:8000 to check if the Todo app is running.

Conclusion

Congratulations! You have successfully deployed a Todo app on AWS ECS using ECR. This step-by-step guide covered setting up an EC2 instance, cloning the Todo app repository, configuring ECR, installing Docker and AWS CLI, building and pushing a Docker image, creating an ECS cluster, defining tasks, and finally deploying and checking the app. This workflow provides a scalable and efficient way to deploy containerized applications on AWS.

Follow me on LinkedIn.

Checkout my GitHub profile.

Deploying a Game with AWS Elastic Beanstalk

Arjun Menon — Sat, 23 Dec 2023 06:53:04 +0000

Introduction

Welcome to this step-by-step tutorial on deploying a game using AWS Elastic Beanstalk. In this guide, we will walk through the process of creating an application, setting up a web server environment, and deploying a Dockerized game using Elastic Beanstalk.

Prerequisites

Before you begin, make sure you have:

An AWS account with the necessary permissions.
Docker installed on your local machine.
A Dockerfile for your game setup.

Step 1: Login to the AWS Console

Open your web browser and navigate to the AWS Management Console.
Log in with your AWS account credentials.

Step 2: Search for the Elastic Beanstalk Service

In the AWS Management Console, use the search bar at the top and type "Elastic Beanstalk."
Click on the Elastic Beanstalk service in the search results.

Step 3: Create a New Application

In the Elastic Beanstalk dashboard, click "Create Application."

Step 4: Select Web Server Environment

Choose the type of environment that will host your application. Here, we select the web server environment.

Step 5: Name the Application

Provide a name for your application.

Step 6: Name Environment and Add Description

Assign a name to the environment and include a brief description.
You can leave the domain name blank for an auto-generated value.

Step 7: Choose Docker Platform

Select the Managed Platform and specify Docker as the platform type.

Step 8: Upload Docker File

Opt for "Upload your code" and upload the Dockerfile you've created. Here's an example Dockerfile:

   FROM ubuntu:22.04 RUN apt-get update RUN apt-get install -y nginx zip curl RUN echo "daemon off;" >>/etc/nginx/nginx.conf RUN curl -o /var/www/html/master.zip -L https://github.com/ArjunMnn/2048-game/archive/refs/heads/master.zip RUN cd /var/www/html/ && unzip master.zip && mv 2048-game-master/* . && rm -rf 2048-game-master master.zip EXPOSE 80 CMD ["/usr/sbin/nginx", "-c", "/etc/nginx/nginx.conf"]

Step 9: Select Free Tier Eligible Option

Choose the option that falls under the AWS Free Tier to ensure cost-effective deployment.

Step 10: Configure Service Access

Step 11: Review and Submit

Review the settings and click "Create" to confirm the deployment.

Step 12: Initiate Environment Creation

The process of creating the environment will begin. Wait for the deployment to complete.

Step 13: Load the Domain

After successful environment creation, access your game using the provided domain.

Conclusion

Congratulations! You have successfully deployed a game using AWS Elastic Beanstalk. Your game should now be accessible via the provided domain, providing a scalable and cost-effective hosting solution.

Follow me on LinkedIn.

Checkout my GitHub profile.

Building a Serverless Node.js API with AWS and Serverless Framework

Arjun Menon — Sun, 10 Dec 2023 09:25:40 +0000

Welcome to this comprehensive tutorial on creating a serverless Node.js API using AWS services and the Serverless Framework. In this tutorial, we'll walk through the process of setting up a serverless project, deploying it to AWS using the Serverless Framework, and testing the APIs using Postman. The project allows you to manage tasks in a DynamoDB table, demonstrating basic CRUD operations.

Prerequisites

Before we start, make sure you have the following:

Sign up on serverless.com.
Node.js and npm installed. You can download them from nodejs.org.
Serverless Framework installed globally. Run the following command:

npm install -g serverless

AWS IAM User credentials for deployment.

Project Setup

Step 1: Clone the Project Repository

Clone the project repository from GitHub:

git clone https://github.com/ArjunMnn/aws-node-http-api-project-dev.git

Step 2: Install Dependencies

Navigate to the project directory and install the required npm packages:

cd aws-node-http-api-project-dev
npm install

Step 3: Configure AWS Credentials

Run the following command to configure your AWS credentials:

serverless config credentials --provider aws --key YOUR_ACCESS_KEY --secret YOUR_SECRET_KEY

Replace YOUR_ACCESS_KEY and YOUR_SECRET_KEY with your AWS IAM User credentials.

Serverless Configuration

The serverless.yaml file defines the AWS infrastructure for your serverless project. Open the serverless.yaml file and replace the placeholder values in the provider section with your specific AWS configuration.

provider:
  name: aws
  runtime: nodejs14.x
  region: YOUR_AWS_REGION
  iamRoleStatements:
    - Effect: Allow
      Action:
        - dynamodb:*
      Resource:
        - arn:aws:dynamodb:YOUR_AWS_REGION:YOUR_ACCOUNT_ID:table/KaamKaro

Replace YOUR_AWS_REGION and YOUR_ACCOUNT_ID with your AWS region and account ID.

Serverless Functions

The project consists of several functions, each handling specific API endpoints. The function code is in separate files in the src directory.

hello.js: Handles the root endpoint and returns a welcome message.
kaamBharo.js: Adds a new task to the DynamoDB table.
kaamDikhao.js: Retrieves all tasks from the DynamoDB table.
kaamHatao.js: Deletes a task from the DynamoDB table.
kaamKhatamKaro.js: Updates the completion status of a task in the DynamoDB table.

DynamoDB Table

The serverless.yaml file includes a DynamoDB table definition. Ensure that you have the necessary permissions to create a DynamoDB table in your AWS account.

resources:
  Resources:
    KaamKaro:
      Type: AWS::DynamoDB::Table
      Properties:
        TableName: KaamKaro
        BillingMode: PAY_PER_REQUEST
        AttributeDefinitions:
          - AttributeName: id
            AttributeType: S
        KeySchema:
          - AttributeName: id
            KeyType: HASH

Deploy the Serverless Project

Run the following command to deploy the project to AWS:

serverless deploy

This command will package and deploy your project using AWS CloudFormation.

Testing with Postman

Use Postman to test your APIs. Here are some sample requests:

GET /: Retrieve a welcome message.
POST /kaam: Add a new task to the DynamoDB table.
GET /kaam: Retrieve all tasks from the DynamoDB table.
PUT /kaam/{id}: Update the completion status of a task in the DynamoDB table.
POST /kaam/{id}: Delete a task from the DynamoDB table.

Conclusion

Congratulations! You've successfully created a serverless Node.js API using AWS and the Serverless Framework. This project serves as a foundation for building scalable and cost-effective serverless applications. Feel free to explore and extend the functionality based on your requirements.

Follow me on LinkedIn.

Check out my GitHub profile.

A Beginner's Guide to VPC Peering in AWS: Connecting Instances Across Virtual Networks

Arjun Menon — Thu, 07 Dec 2023 07:39:31 +0000

Introduction

Embarking on your cloud computing journey with AWS opens up a world of possibilities. One crucial aspect of managing your infrastructure efficiently is establishing connections between Virtual Private Clouds (VPCs). In this tutorial, we'll guide you through the process of setting up VPC peering, allowing seamless communication between instances in distinct VPCs.

Creating Test VPC

Title: Setting Up the Test Environment

Create a VPC: Begin by creating a VPC named "test-vpc" with the CIDR block 10.0.0.0/16. This step lays the foundation for your testing environment.
Subnet Creation: Establish a subnet named "test-public-subnet" within "test-vpc" using the CIDR block 10.0.0.0/16. Subnets help organize resources within your VPC.
Route Table Configuration: Create a route table named "test-rt" associated with "test-vpc." This table will define the traffic flow within the VPC.
Internet Gateway Setup: Introduce an internet gateway named "test-igw" and attach it to "test-vpc." This step enables instances within the VPC to communicate with the internet.
Routing to Internet: Edit the routes in the "test-rt" route table, allowing all access to the internet gateway "test-igw." This is a crucial step for outbound internet communication.
Subnet Association: Add the created subnet "test-public-subnet" to the route table associations. This associates the subnet with the specified route table.
EC2 Instance Launch: Launch an EC2 instance named "test-ec2-instance" within "test-vpc." Ensure to add an inbound rule for HTTP traffic and select CIDR 10.0.0.0/16.

Creating Prod VPC

Title: Setting Up the Production Environment

Create a VPC: Mirror the steps for the production environment. Create a VPC named "prod-vpc" with the CIDR block 172.16.0.0/16.
Subnet Creation: Establish a subnet named "prod-public-subnet" within "prod-vpc" using the CIDR block 172.16.0.0/16.
Route Table Configuration: Create a route table named "prod-rt" associated with "prod-vpc." This table will define the traffic flow within the production VPC.
Internet Gateway Setup: Introduce an internet gateway named "prod-igw" and attach it to "prod-vpc."
Routing to Internet: Edit the routes in the "prod-rt" route table, allowing all access to the internet gateway "prod-igw."
Subnet Association: Add the created subnet "prod-public-subnet" to the route table associations.
EC2 Instance Launch: Launch an EC2 instance named "prod-ec2-instance" within "prod-vpc." Ensure to add an inbound rule for HTTP traffic and select CIDR 172.16.0.0/16.

Testing Isolation:

Title: Verifying VPC Isolation

Now that both environments are set up, try to ping one EC2 instance from another. As expected, the attempt will fail, highlighting the isolated nature of these VPCs.

Setting Up VPC Peering

Title: Bridging the Gap with VPC Peering

Create Peering Connection: Navigate to the VPC dashboard and select "Peering Connections." Create a peering connection, providing names for identification, and select both VPCs.
Accept Peering Connection: Once the peering connection is created, accept the connection request. This step establishes a direct link between the two VPCs.

Security Group Configuration

Title: Ensuring Secure Communication

Test EC2 Security Group: Go to the security group of "test-ec2-instance" and edit inbound rules. Add rules for All ICMP IPv4, specifying CIDR blocks 10.0.0.0/16 and 172.16.0.0/16.
Prod EC2 Security Group: Similarly, edit the inbound rules for the security group of "prod-ec2-instance." Add rules for All ICMP IPv4 with CIDR blocks 10.0.0.0/16 and 172.16.0.0/16.

Route Table Adjustments

Title: Directing Traffic Across VPCs

Test Route Table: In the "test-rt" route table, edit routes and add a route for the peering connection to "prod-vpc" with CIDR 172.16.0.0/16. This enables traffic from "test-vpc" to reach "prod-vpc."
Prod Route Table: Similarly, in the "prod-rt" route table, add a route for the peering connection to "test-vpc" with CIDR 10.0.0.0/16. This allows traffic from "prod-vpc" to reach "test-vpc."

Testing Communication

Title: Ensuring VPC Peering Success

With VPC peering established and configurations in place, attempt to ping the EC2 instances across VPCs. This time, the ping should succeed, demonstrating the successful communication between instances in different VPCs.

Conclusion

In conclusion, this tutorial has walked you through the process of setting up VPC peering in AWS, connecting instances across different virtual networks. By following these steps, you've bridged the gap between isolated environments, enabling seamless communication. This fundamental skill is essential for building complex and interconnected architectures on the AWS cloud.

Follow me on LinkedIn.

Checkout my GitHub profile.

Deploying a Two-Tier Web App with MySQL on Amazon RDS

Arjun Menon — Tue, 05 Dec 2023 14:04:21 +0000

Welcome to Day 4 of our AWS mastery journey! Today, we're embarking on an exciting project - deploying a two-tier application. Tier 1 features a Dockerized web app, while Tier 2 hosts a MySQL server on Amazon RDS. Let's dive into the world of containers, AWS, and seamless cloud integration.

Setting the Stage: Unleashing the Power of Docker and AWS

Defining Your Project: Start by conceptualizing your project. Our goal is to create a two-tier application where the first tier is a Dockerized web app, and the second tier is a MySQL server hosted on Amazon RDS.

Tier 1: Dockerized Web App

Installation: Begin by installing Docker on your EC2 instance. This will be the backbone of your containerized web app.
Cloning the Repository: Clone this repository which contains the Dockerfile.
Image Creation: Execute docker build to create an image from your Dockerfile. This encapsulates your app and its dependencies.

Tier 2: MySQL on Amazon RDS

Configuring Amazon RDS:

Database Instance Creation: Navigate to the AWS Management Console, select Amazon RDS, and create a new database instance. Choose MySQL as the engine.
Instance Settings: Define your instance details, including DB credentials, storage, and network configurations.

Connecting EC2 to RDS:

Go to your RDS database. Scroll to the "Connected compute resources" section and add your ec2 instance.
In your EC2 instance, run the following commands:

 # Install mysql client
 sudo apt update
 sudo apt install mysql-client

 # Connecting your ec2 instance to the rds database.
 mysql -u admin -h <endpoint> -P 3306 -p

MySQL console will open. In the MySQL console, run the following:

 create database aws;
 CREATE TABLE messages (
     id INT AUTO_INCREMENT PRIMARY KEY,
     message TEXT
 );

Deployment: Bringing Your Application to Life

Running the docker container: Run the docker container we created before with MySQL environment variables as below

 sudo docker run -d -p 5000:5000 -e MYSQL_HOST=mysql -e MYSQL_USER=admin -e MYSQL_PASSWORD=admin123 -e MYSQL_DB=aws two-tier

Testing the Full Setup: Validate your two-tier application in the browser, by going on the URL <http://ec2-public-ip>:5000.

Conclusion

Congratulations on successfully deploying a two-tier application on AWS! Today's journey has taken you through the intricacies of Docker, Amazon RDS, and the orchestration of cloud resources.

Stay tuned for Day 5, where we'll explore advanced AWS services, propelling your cloud expertise even further!

Follow me on LinkedIn.

Checkout my GitHub profile.

Building a Secure Foundation with WAF

Arjun Menon — Sun, 03 Dec 2023 15:43:56 +0000

Introduction:

Welcome back to the 7 Days of AWS challenge! Today, we'll focus on fortifying our infrastructure with AWS Web Application Firewall (WAF). Buckle up as we delve into the essentials of WAF and embark on a hands-on project to bolster our security measures.

Understanding AWS Web Application Firewall (WAF):

AWS Web Application Firewall (WAF) is your shield against malicious attacks on your web applications. It allows you to create custom rules that filter and monitor HTTP and HTTPS requests. WAF works seamlessly with Amazon CloudFront and Application Load Balancers, giving you granular control over the traffic entering your applications.

To get started, navigate to the AWS WAF console. Here, you can create WebACLs (Access Control Lists) to define the rules that govern which requests are allowed or denied. The rule sets include conditions such as IP addresses, geographic locations, and even custom rules based on the content of the requests.

Hands-On Project: Building a Secure Infrastructure

Step 1 - Creating a Launch Template:

Our first step is to create a Launch Template, a powerful resource that defines the parameters for launching instances within our Auto Scaling Groups.

In the EC2 Dashboard, find the "Launch Templates" section.
Click "Create Launch Template" and configure the necessary settings, including the Amazon Machine Image (AMI), instance type, and storage.
In the "Advanced Details" section, under "User data," paste the following script to deploy a todo list web app from a Docker image:

 #!/bin/bash

 # Update the package list
 sudo apt-get update -y

 # Install Docker
 sudo apt-get install -y docker.io
 sudo systemctl start docker
 sudo systemctl enable docker

 # Pull the Docker image from DockerHub
 sudo docker pull arjunmmnn/todo-app:latest

 # Run the Docker container, exposing port 8001
 sudo docker run -d -p 80:8001 arjunmmnn/todo-app:latest

Save the template.

Step 2 - Auto Scaling Group (ASG):

Now, let's use the Launch Template to set up an Auto Scaling Group, ensuring our applications can handle varying workloads seamlessly.

In the EC2 Dashboard, under "Auto Scaling Groups," click "Create Auto Scaling Group."
Select the Launch Template you created earlier.
Configure scaling policies, desired capacity, and other parameters.
Complete the setup.

Step 3 - Application Load Balancer (ALB):

With our Auto Scaling Group in place, it's time to distribute traffic efficiently using an Application Load Balancer.

In the EC2 Dashboard, navigate to "Load Balancers" and click "Create Load Balancer."
Choose "Application Load Balancer" and configure the settings, including listener configurations.
Add the instances from your Auto Scaling Group to the target group.

Securing Your Infrastructure with WAF:

Subtitle: Step 4 - Configuring WAF WebACL:

Now that our infrastructure is set up lets enhance its security by configuring a WebACL in WAF.

In the AWS WAF console, go to WebACLs and click "Create WebACL."
Define the conditions for your WebACL, specifying the rules to allow or block requests.
Associate the WebACL with your Application Load Balancer.

By configuring a WAF WebACL, you add an additional layer of protection to your applications, safeguarding them against various cyber threats.

Subtitle: Step 5 - Testing the Todo App:

It's time to put our todo list web app to the test. Open your web browser and navigate to:

http://<Your_Load_Balancer_DNS_Name>:80

Replace <Your_Load_Balancer_DNS_Name> with the DNS name of your Application Load Balancer. If everything is configured correctly, you should see your todo list web app in action. Congratulations on successfully deploying and testing your application!

Conclusion:

Congratulations! You've successfully strengthened your AWS infrastructure on Day 2 of the challenge. Understanding the nuances of AWS Web Application Firewall and implementing a secure project with launch templates, auto-scaling groups, and application load balancers are crucial steps in your AWS journey.

Stay tuned for Day 3, where we'll explore more AWS services to optimize and enhance your cloud experience. Remember, each day brings us closer to mastering the art of AWS.

Let's connect on LinkedIn.

Checkout my GitHub profile.

Declarative CICD Pipeline with GitHub Integration, Jenkins, Docker, and Agent-Based Automation

Arjun Menon — Wed, 22 Nov 2023 08:41:11 +0000

Efficient deployment is a cornerstone of successful software development. In this tutorial, we'll guide you through the process of setting up a Jenkins pipeline to automate the deployment of your Django web application using Docker. What makes this tutorial special is the utilization of two EC2 instances a Jenkins master server and an agent server to facilitate seamless deployment.

Section 1: Prerequisites and Project Setup

1.1 Prerequisites:

Before initiating the deployment process, confirm that the following prerequisites are met:

Two ec2 instances for master and agent servers.
Jenkins installed and configured on the master instance.
Jdk, Docker and docker-compose are installed on both instances.
A DockerHub account for storing Docker images.

Section 2: Configuring Jenkins Master and Agent

2.1 Setting Up Jenkins Master:

Access your Jenkins master server and navigate to "Manage Jenkins" -> "Manage Nodes and Clouds."
Click on "New Node" to add the agent EC2 instance.
Name the node, select "Permanent Agent," and click "OK."
Specify the necessary configurations, ensuring Docker is installed on the agent. Refer to this article for more details

2.2 Creating a New Pipeline in Jenkins:

Access Jenkins and click on "New Item."
Choose "Pipeline" and provide a name for your project.
In the pipeline configuration, select "Pipeline script from SCM."
Choose "Git" as the SCM, and enter your GitHub repository URL.
Save the configuration.

Section 3: Building the Declarative CICD Pipeline

3.1 Modifying Jenkinsfile:

Update your Jenkinsfile with the following steps to include agent configuration and deployment:

pipeline {
    agent { label 'dev-server' }
    stages {
        stage('code clone') {
            steps{
                echo 'cloning'
                git url: "https://github.com/ArjunMnn/django-notes-app", branch: "main"
            }
        }
        stage('build') {
            steps{
                 echo 'building'
                 sh 'docker build --no-cache -t my-note-app .'
            }
        }
        stage('push to dockerhub') {
            steps{
                echo 'pushing'
                withCredentials([usernamePassword(credentialsId:"dockerHub",passwordVariable:"dockerHubPass",usernameVariable:"dockerHubUser")]){
                    sh "docker tag my-note-app ${env.dockerHubUser}/my-note-app:latest"
                    sh "docker login -u ${env.dockerHubUser} -p ${dockerHubPass}"
                    sh "docker push ${env.dockerHubUser}/my-note-app:latest"
                }
            }
        }
        stage('deploy') {
           steps{
                echo 'deploying'
                sh "docker-compose down && docker-compose build --no-cache && docker-compose up -d --build"
           }
        }
    }
}

Section 4: Configuring DockerHub Credentials

4.1 Jenkins Master:

Navigate to "Manage Jenkins" -> "Manage Credentials."
Add your DockerHub credentials.

Section 5: Creating Docker Compose File

Include a comprehensive Docker Compose file in your GitHub repository.
Configure services, networks, and volumes based on your Django app and any related services.

Section 6: Configuring GitHub Webhook

In your GitHub repository, navigate to "Settings" -> "Webhooks" -> "Add webhook."
Set the Payload URL to your Jenkins master server's webhook URL.
Configure the webhook to trigger on push events.

Section 7: Testing the Pipeline

Click on build now. It should deploy the app.
Observe Jenkins triggering the pipeline on the master server, with deployment occurring on the agent server.

Conclusion

You've successfully established a robust CICD pipeline for your Django web application, incorporating GitHub integration, Jenkins, Docker, and an agent-based approach. This comprehensive guide allows for easy customization to suit your specific development needs. Explore additional optimizations and enhancements to further enhance your deployment workflow. Happy coding!

Let's connect on LinkedIn.

Checkout my GitHub profile.

Complete Jenkins CI/CD Project with GitHub Integration

Arjun Menon — Sun, 19 Nov 2023 14:24:00 +0000

Welcome to this comprehensive guide on setting up a Jenkins CI/CD pipeline for deploying a Node.js To-Do app using Docker Compose, seamlessly integrated with GitHub. This tutorial will walk you through each step, providing detailed explanations and additional information to ensure a smooth setup.

Introduction

In the ever-evolving landscape of software development, Continuous Integration and Continuous Deployment (CI/CD) have become integral components for delivering reliable and efficient applications. Jenkins, coupled with Docker and GitHub, offers a powerful solution to automate your application deployment process.

This tutorial focuses on the seamless integration of Jenkins with GitHub using webhooks. With GitHub webhooks, any changes made to your repository trigger automatic deployments through Jenkins, ensuring your application is always up-to-date.

Prerequisites

Before we begin, ensure the following prerequisites are met:

Your ec2 instance is up and running.
Jenkins is installed and running.
Docker and Docker Compose are installed on your server.

Setting Up Jenkins Job

Step 1: Create a New Freestyle Project

Open Jenkins and Navigate to Dashboard: Log in to your Jenkins instance and click on "New Item" to create a new project.
Enter Project Details: Provide a name for your project, such as "todo-node-app," and select the "Freestyle project" option. Click "OK" to create the project.

Step 2: Configure GitHub Project

Enable GitHub Project: Check the "GitHub project" checkbox.
Enter Project URL: In the "Project URL" field, input the URL of your GitHub repository (e.g., https://github.com/ArjunMnn/node-todo-cicd/).

Step 3: Set Up Source Code Management

Choose Git: In the "Source Code Management" section, select "Git."
Enter Repository URL: Input the URL of your GitHub repository (e.g., https://github.com/ArjunMnn/node-todo-cicd.git).

Step 4: Configure Credentials

Generate SSH Keys: On your EC2 instance, run ssh-keygen to generate SSH keys. Use the generated public key for GitHub
Add SSH Key to GitHub: In GitHub settings, add a new SSH key with the generated public key.
Jenkins Credentials: In Jenkins credentials section, select "SSH username with private key" in the "Kind" field. Use "github-jenkins" as the ID and "ubuntu" as the username. Enter the private key directly.

Step 5: Set Up Build Triggers

Enable GitHub Hook Trigger: In the "Build Triggers" section, check "GitHub hook trigger for GITScm polling." This ensures that Jenkins is triggered upon code changes in your GitHub repository.

Step 6: Define Build Steps

Add Execute Shell Build Step: In the "Build" section, add an "Execute shell" build step. Enter the command docker-compose down && docker-compose build --no-cache && docker-compose up -d --build to handle Docker Compose actions.

Step 7: Save the Jenkins Job

Click "Save" to store your Jenkins job configuration.

GitHub Webhook Configuration

Step 8: GitHub Webhook Setup

Navigate to GitHub Settings: In your GitHub repository, click on "Settings" and select "Webhooks."
Add Webhook: Click "Add webhook" to set up a new webhook.

Step 9: Configure Webhook

Enter Payload URL: In the "Payload URL" field, enter <ipaddress:8080/github-webhook/>. Ensure to replace <ipaddress> with the actual IP of your Jenkins server.
Set Content Type: Choose "application/json" as the content type.
Save Webhook: Click "Add webhook" to save your webhook configuration.

Install GitHub Integration Plugin

Step 10: Install GitHub Integration Plugin

Navigate to Jenkins Plugin Manager: In Jenkins, go to "Manage Jenkins" > "Manage Plugins."
Install Plugin: In the "Available" tab, search for "GitHub Integration" and install the plugin. Restart Jenkins when prompted.

Step 11: Test the Jenkins Job

Build Now: Go to your Jenkins job and click "Build Now" to test if the setup is correct.
Check Console Output: Monitor the build console output for any errors or issues.

Step 12: Test the Deployment

Open Browser: In your web browser, navigate to <ipaddress:8000> to verify that your app is successfully deployed.

Step 13: Verify Webhook

Make Changes in GitHub: Make changes to your GitHub repository to test the webhook.
Check Jenkins: Observe Jenkins for triggered builds and successful deployments.
Verify Deployment: Confirm the changes by checking the deployed app in the browser at <ipaddress:8000>.

Conclusion

Congratulations! You've successfully configured a Jenkins CI/CD pipeline to automate the deployment of your Node.js To-Do app, integrated seamlessly with GitHub. Embrace the power of CI/CD for efficient and reliable software delivery. Happy coding! 🎉

Let's connect on LinkedIn