DEV Community: Sri Vishnuvardhan A

Face Recognition System for provisioning AWS instance using Terraform scripts and generating Mail & WhatsApp alerts

Sri Vishnuvardhan A — Thu, 24 Jun 2021 09:52:47 +0000

In this article, We are going to know how to create a face recognition system by CV2 and use it for performing some automation tasks in Real-time such as Provisioning AWS instance and its dependencies such as VPC, Subnet, Security Groups, Route tables, Internet Gateway, and 5GB EBS Volume which is done by triggered Terraform scripts.

We are also going to know how to generate Whatsapp and Email alerts when our face is recognized using Python.

Technologies used: Python CV2 and Terraform

Contents Overview:

Model1
- Data Collection
- Model Training
- Face Recognition
- Running Terraform Scripts to provision AWS Instance
Model2
- Data Collection
- Model Training
- Face Recognition
- Generating WhatsApp and Email Alerts

Data Collection

First step of in Machine Learning world is always a Data Collection. Without data, nothing can be achieved. In this step, we are going to collect 100 samples of my face with the size of 200 x 200 pixels and stored in the folder named as Vishnu.

This folder name is crucial because this folder name will be our Object name/face name for our model in future. So, give Human name to your folder.

Here, We are going to use haarcascade_frontalface_default.xml file as a face detector.

Model Training

In this phase, model is created using LBPHFaceRecognizer algorithm and the model is trained to recognize my face whenever my face appears in real time in video or appear in image.

Finally, the model is created and now the model has the capability to identify or recognize my face.

Face Recognition

Final step is to do Face Recognition. Our desired output would be the model should recognize my face and display the confident score and say "Hi Vishnu" and ask to hit Enter to perform further operations.

The above figure shows that our model predicted successfully and if we press enter, in the mean time, our Terraform scripts will automatically run and launch VPC, Subnet, Security Groups, Route tables, Internet Gateway, 5GB EBS Volume and finally provisioned AWS instance.

EC2 Dashboard

VPC Dashboard

Model2

Now, We are going to create another model for demonstrating automatic Email and WhatsApp alert messages when the face is detected and recognized. Here, we are going to use the face of Actor Vijay.

Data Collection

Same Procedure. Samples are collecting and storing in another folder named "Vijay" for training the model.

Model Training

Model is trained using LBPH(Local Binary Pattern Histogram) algorithm and used to trained the samples of Actor Vijay's face.

Above figure shows model is trained successfully and attain its capability of Face Recognition.

Face Recognition

When the Vijay's face is recognized, it will give the confident score and will say "Hi Vijay" and asked to hit enter to perform further operations. When we hit enter, it will send Email and Whatsapp messages alert to the mentioned mail ID and WhatsApp number.

WhatsApp Alert

Email Alert

Output will be..

Above figures shows that desired output was got and it shows our model was trained successfully and predicted correctly.

If you want to play, feel free to download the code in the GitHub by click here

That's it. Thank you for your reads. Stay tuned for my upcoming more interesting articles!!

JavaScript: The Company use cases

Sri Vishnuvardhan A — Tue, 22 Jun 2021 16:25:58 +0000

JavaScript is high-level, often just-in-time compiled, and multi-paradigm. It has curly-bracket syntax, dynamic typing, prototype-based object-orientation, and first-class functions.

Out of over 1.6 billion websites on the Internet, JavaScript is used in 95% of them.

Common examples of JavaScript uses and applications include:

Presentations
Web Development
Server Applications
Web Applications
Games
Mobile Applications

Microsoft

Microsoft needs to work closely with JavaScript to built its Edge web browser.All browsers want to process and execute JavaScript efficiently, so Microsoft has advanced and keeps its personal JavaScript engine for Edge

PayPal

PayPal has manifestly been the usage of JavaScript at the frontend in their internet site for a protracted time, that’s only the beginning.

The online payment giant was one of the earliest adopters of NodeJS. During an overhaul in their account evaluate web page, they determined to attempt constructing the web page in Node on the equal time as their standard Java development.

Uber

Uber wants to handle loads of data in real time. They have a millions of requests coming continuously, and that’s not just hits on a page. Uber wants to track driver locations, rider locations, and incoming ride requests.

It has to seamlessly sort that data and match riders as rapid as possible.

LinkedIn is predicated on NodeJS for its mobile web website online. A few years back, LinkedIn used Rails for its mobile web website online. As with different different huge Rails applications, it become slow, monolithic, and it scaled poorly.

LinkedIn converted to NodeJS to clear up its scaling problems. Node’s asynchronous abilities allowed the LinkedIn mobile web website online to carry out extra fast than earlier than at the same time as the usage of fewer resources.

These are some of the companies which uses Javascript for their various purposes especially User Interface part. But it is not limited to this much companies, more than a million.

Hope this article insightful to you to know about Javascript and its real life use cases. Have a good day!!

Setting up Fast & Secure Globally scaled Content Delivery Network with high availability using AWS Cloudfront & S3

Sri Vishnuvardhan A — Wed, 16 Jun 2021 02:16:25 +0000

In this article, we are going to see about what is the Content Delivery Network, why we need them, what are its use cases and then finally we are going to set up our own Content Delivery Network with Fast, Secure and high availability using one of the most powerful services provided by AWS namely Cloudfront.

Content Delivery Network

A Content Delivery Network (CDN) is a globally distributed network of web servers whose purpose is to provide faster content delivery. The content is replicated and stored throughout the CDN so the user can access the data that is stored at a location that is geographically closest to the user.

This is different and more efficient than the traditional method of storing content on just one, central server. A client accesses a copy of the data near to the client, as opposed to all clients accessing the same central server, in order to avoid bottlenecks near that server.

High content loading speed ==positive User Experience

CDN Architecture model

The above figure clearly illustrates the typical CDN model. When a user requests the content, for the first time it will send to Content Provider, then Content Provider will send their copy of the document known as Source to CDN and that copy is stored as digital information which is created, licensed and ready for distribution to an End User.

If the User requests the content again, he will receive the content from CDN only which is located nearer to the geographical location of the user, not from Content Provider. We can reduce latency and ensure high availability.

Benefits of CDN over Traditional method

CDN enables global reach
100% percent availability
Your reliability and response times get a huge boost
Decrease server load
Analytics

Use-Cases of CDN

Optimized file delivery for emerging startups
Fast and secure E-Commerce
Netflix-grade video streaming
Software Distribution, Game Delivery and IoT

CloudFront

credits: whizlabs

Amazon CloudFront is a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency, high transfer speeds, all within a developer-friendly environment. CloudFront uses Origins from S3 for setting up its Content Delivery Network.

Cloudfront uses Edge locations to store the data cache. Currently, AWS now spans 77 Availability Zones within 24 geographic regions around the world and has announced plans for 18 more Availability Zones and 6 more AWS Regions in Australia, India, Indonesia, Japan, Spain, and Switzerland.
AWS uses DNS to find the nearest data centers for storing the caches. Data comes from Origin to Edge location and Edge location to our PC.

Practicals

Now I am going to show you how to setup your Own custom Content Delivery Network for your content which includes images, videos, etc.

Pre-requisites

AWS account
AWS instance
EBS volume

For saving time, I already launched one Amazon instance and one EBS volume sized 10GB and attached to its instance.

Steps

Installing HTTPD server
Making its document root persistent
Storing the content in S3
Deployed it to CloudFront

Installing HTTPD server

Since the package manager “YUM” is already installed in Amazon Linux 2, so run the following commands for configuration of HTTPD server in that instance.

yum install httpd -y

Then we have to start the service.

yum start httpd

We have to enable the service. So that we need not start the service again and again after every reboot.

yum enable httpd

Making its document root persistent

Since the OS in Amazon 2 Linux is RHEL 8, so the document root of HTTPD server is /var/www/httpd. The document root is the location where the HTTPD server reads and deploys the webpage in web server.

We have to make that document root persistent in order to secure the data from being lost due to OS crash etc.
For that, you have to ready with the previously created one EBS volume and done with the partition. Then run the following command.

mount /var/www/html /partition

Simple Storage Service(S3)

Since the Origin for CloudFront is S3, we have to setup S3 so that we can get the Origin Domain name for CloudFront. In S3, the folders are said to be buckets and files are said to be Objects.

First step is to create a bucket by using the following command syntax.

aws s3 mb s3://bucketname

The second step is to move/copy the objects to the buckets by using following command syntax.

aws s3 mv object-location s3://bucketname

By default, Public access for S3 buckets is blocked. We have to release the access by running the following command syntax in your Windows Command Prompt.

aws s3api put-public-access-block --bucket bucketname --public-access-block-configuration “BlockPublicAcls=false, IgnorePublicAcls=false, BlockPublicPolicy=false, RestrictPublicBuckets=false”

aws s3api put-object-acl --acl "public-read" --bucket "bucketname" --key objectname

We had completed the S3 setup, the HTML code of the webpage is shown below.

CloudFront

Since Origin S3 setup is over, now we have to setup the CloudFront by creating one distribution using Origin Domain Name from S3 and Object name as Default Root Object.

We have to run the following command to create a distribution

aws cloudfront create-distribution --origin-domain-name vishnu1234.s3.amazonaws.com --default-root-object Scam-1992.jpg

After creating the distribution, CloudFront will give one domain address, we have to copy that domain address to that HTML code and it will replace the S3 domain address.

Finally our Output will be*…*

Thank you all for your patience to read this article.Stay tuned for my upcoming interesting articles.Have a good day.

Network Topology Setup in such a way that System A can ping to two Systems B & C but both systems not able to ping each other

Sri Vishnuvardhan A — Fri, 11 Jun 2021 04:58:36 +0000

Note: We are not going to use any firewall, since it is simple and everyone aware of this, so we are going to use unique way to achieve this topology setup.

Concepts used: Routing tables and Netmask

In this practical, We are going to use three Redhat Enterprise Linux Virtual Machines which are hosting by Oracle VirtualBox.

I already explained these Routing table and Netmask concepts in my previous blog, if you want to know these basic concepts then please refer this

Note: Before starting these VM's, make sure that it is connected in Host-only adapter mode and the adapter name should be the same in all three VMs.

Environment Setup

System A

First step is to change the IP and set the Netmask by running the following command.

ifconfig enp0s3 162.168.1.1/29

Second step is to create the route rule by running the following command.

route add -net 162.168.1.0 netmask 255.255.255.248 enp0s3

System B

First step is to change the IP and set the Netmask by running the following command.

ifconfig enp0s3 162.168.1.2/31

Second step is to create the route rule by running the following command.

route add -net 162.168.1.0 netmask 255.255.255.254 enp0s3

System C

First step is to change the IP and set the Netmask by running the following command.

ifconfig enp0s3 162.168.1.4/31

Second step is to create the route rule by running the following command.

route add -net 162.168.1.0 netmask 255.255.255.254 enp0s3

Ping Checking

Now, We completed the environment setup. Now we have to run the ping command in order to check the status of Network connectivity.

Pinging System B from System A

Pinging System A from System B

Pinging System C from System A

Pinging System A from System C

Pinging System B from System C

Pinging System C from System B

How is this Possible?

In simple words, it happened because of routing tables that use the Netmask for the IP range.

Let's take a look at the routing rule in System A

route add -net 162.168.1.0 netmask 255.255.255.248 enp0s3

In this, Netmask 255.255.255.248 denotes the IP range which have the connection between them.
If you convert to binary form means, it looks like this.

11111111 11111111 11111111 11111000

For the last three places(where the zeros located), we can accommodate 8 combinations. So it decided the number of IP's (IP range) that can connect each other.

The range of IPs are that can connect each other is 162.168.1.0–162.168.1.7.

So in this way, System A can ping to System B (162.168.1.2) and System C (162.168.1.4).

Then if you look at the routing rule in System B.

route add -net 162.168.1.0 netmask 255.255.255.254 enp0s3

The netmask is specified as 255.255.255.254
If you convert to binary form means, it looks like this.

11111111 11111111 11111111 11111110

For the last one place(where the zero located), We can accommodate 2 combinations. So it decided the number of IPs (IP range) that can connect each other.

So the range of IP's that are connected to each other are two in numbers namely 168.162.1.0 and 162.168.1.1.

In this way System B can ping to System A. It is noted that IP of System C (162.168.1.4) is not in range, that's the reason for not ping.

Finally, if you take a look on routing rule in System C.

route add -net 162.168.1.0 netmask 255.255.255.254 enp0s3

Here, the Netmask specified is 255.255.255.254. So the last one place(where the zero located), we can accommodate 2 combinations. So it decided the number of IPs (IP range) that can connect each other.

So the range of IPs that are connected to each other are two in numbers namely 168.162.1.0 and 162.168.1.1.

In this way, System C can ping to System A. It is noted that IP of B(162.168.1.2) is not in range, that’s the reason for not ping.

Thats it. Hope this practical help you to undertand the concepts of Netmask and Routing tables and its use cases in Networking. Stay tuned for my next article!!

Setting up your own High Availability managed WordPress hosting using Amazon RDS

Sri Vishnuvardhan A — Fri, 11 Jun 2021 04:27:10 +0000

Hosting your own WordPress website is interesting right!! Ok, come on let's do it!!

We are going to do this practical from Scratch. From the Creation of our Own VPC, Subnets, Internet Gateway, Route tables to Deployment of WordPress.

Here, We are going to use Amazon Web Service’s RDS service for hosting our own WordPress site. Before that, let's take a look at a basic introduction to RDS service.

Amazon Relational Database Service is a distributed relational database service by Amazon Web Services (AWS). It is a web service running in the cloud designed to simplify the setup, operation, and scaling of a relational database for use in applications.

Administration processes like patching the database software, backing up databases and enabling point-in-time recovery are managed automatically.

Features of AWS RDS

Lower administrative burden. Easy to use
Performance. General Purpose (SSD) Storage
Scalability. Push-button compute scaling
Availability and durability. Automated backups
Security. Encryption at rest and in transit
Manageability. Monitoring and metrics
Cost-effectiveness. Pay only for what you use

Ok, let's jump onto the practical part!!.We will do this practical from scratch. Since it will be long process, so we divided this into 5 small parts namely,

Creating a MySQL database with RDS
Creating an EC2 instance
Configuring your RDS database
Configuring WordPress on EC2
Deployment of WordPress website

Creating a MySQL database with RDS

Before that, we have to do two pre-works namely the Creation of Virtual Private Cloud(VPC), Subnets and Security groups. These are more important because in order to have a reliable connection between WordPress and MySQL database, they should be located in the same VPC and should have the same Security Group.

Since Instances are launched on Subnets only, Moreover RDS will launch your MySQL database in EC2 instance only, that we cannot able to see since it is fully managed by AWS.

We are going to create our own VPC. For that, we have to specify IP range and CIDR. We specified IP and CIDR as 192.168.0.0/16.
What is CIDR?. I explained this in my previous blog in very detail. You can refer here.

Let's come to the point. After specifying the IP range and CIDR,then enter your VPC name.

Now, VPC is successfully created with our specified details.
Next step is to launch the subnet in the above VPC.

For Creating Subnets, you have to specify which VPC the lab should launch. We already have our own VPC named “myvpc123”.

And then we have to specify the range of Subnet IP and CIDR. Please note that the Subnet range should come under VPC range, it should not exceed VPC range.

For achieving the property of High Availability, We have to launch minimum two subnets, so that Amazon RDS will launch its database in two subnets, if one subnet collapsed means, it won't cause any trouble.

Now, two Subnets with their specified range of IPs and CIDR are launched successfully inside our own VPC and they are available.

Next step is to create a security group in order to secure the WordPress and MySQL databases. Note that both should have the same Security Group or else it won't connect.

For creating a Security Group, we have to specify which VPC it should be launched and adding a Description is mandatory.

Then we have to specify inbound rules, for making this practical simple, we are allowing all traffic to access our instance.

Now, the Security Group is successfully created with our specified details.
Now let's jump into part 1 which is about Creating a MySQL database with RDS.

Select Create database, then select Standard create and specify the database type.

Then you have to specify the Version. Version plays a major role in MySQL when integrating with WordPress, so select the compactible version or else it will cause serious trouble at the end. Then select the template, here we are using Free-tier since it won't be chargeable.

Then you have to specify the credentials such as Database Instance name, Master username and Master password.

Most important part is a selection of VPC, you should select the same VPC where you will launch your EC2 instance for your WordPress and we can’t modify the VPC once the database is created. Then select the Public access as No for providing more security to our database. Now, the people outside of your VPC can’t able to connect to your database.

Then you have to specify the Security group for your database. Note that the Security Group for your database and WordPress should be the same or else it will cause serious trouble.

Note that Security Groups is created per VPC. After selecting Security Group, then click Ok to create the RDS database.

Creating an EC2 instance

Before creating an instance, there should be two things you have to be configured namely Internet Gateway and Route tables. It is used for providing outside internet connectivity to an instance launched in the subnet.

Internet Gateway is created per VPC. First, we have to create one new Internet Gateway with the specified details.

Then you have to attach Internet Gateway to the VPC.

Next step is to create Routing tables. Note that Route table is created per Subnet.

We have to specify which VPC in which your subnet is available to attach routing table with it, specify Name and click create to create the route table.

Then click Edit route to edit the route details namely destination and target. Enter destination as 0.0.0.0/0 for accessing any IP anywhere on the Internet and target is your Internet Gateway.

After entering the details, click Save routes.

We created a Route table, then we have to attach that table to your Subnet. For that click Edit route table association and select your subnet where you want to attach the route table with it.

Now, lets jump into the task of creating an EC2 instance. First, you have to choose the AMI image in which you used for creating an EC2 instance, here I selected Amazon Linux 2 AMI for that.

Then you have to select Instance type, here I selected t2.micro since it comes under free tier.

Then you have to specify the VPC, Subnet for your instance and you have to enable Auto-assign Public IP in order to get your Public IP to your instance.

Then you have to add storage for your instance. It is optional only.

Then you have to specify the tags which will be more useful especially for automation.

Then you have to select the Security Group for your instance. It should be the same as your database have.

And click Review and Launch. Then you have to add Keypair to launch your EC2 instance. If you didn't have Keypair means, you can create at that time.

Configuring your RDS database

At this point, you have created an RDS database and an EC2 instance. Now, we will configure the RDS database to allow access to specific entities.

You have to run the below command in your EC2 instance in order to establish the connection with your database.

export MYSQL_HOST=<your-endpoint>

You can find your endpoint by clicking database in the RDS
dashboard. Then you have to run the following command.

mysql --user=<user> --password=<password> dbname

This output shows the database is successfully connected to an EC2 instance.
In the MySQL command terminal, you have to run the following commands in order to get all privileges to your account.

CREATE USER 'vishnu' IDENTIFIED BY 'vishnupassword'; GRANT ALL PRIVILEGES ON dbname.* TO vishnu; FLUSH PRIVILEGES; Exit

Configuring WordPress on EC2

For Configuring WordPress on EC2 instance, the first step is to configure the webserver, here I am using Apache webserver. For that, you have to run the following commands.

sudo yum install -y httpd sudo service httpd start

Next step would be download the WordPress application from the internet by using wget command. Run the following code to download the WordPress application.

wget https://wordpress.org/latest.tar.gz tar -xzf latest.tar.gz

Then we have to do some configuration, for this follow the below steps.

cd wordpress cp wp-config-sample.php wp-config.php cd wp-config.php

Go inside the wp-config.php file and enter your credentials (including your password too)

Then, Goto this link and copy all and paste it to replace the existing lines of code.

Next step is to deploy the WordPress application. For that, you have to run the following commands in order to solve the dependencies and deploy WordPress in the webserver.

sudo amazon-linux-extras install -y lamp-mariadb10.2-php7.2 php7.2 sudo cp -r wordpress/* /var/www/html/ sudo service httpd restart

That’s it. You have a live, publicly-accessible WordPress installation using a fully-managed MySQL database on Amazon RDS.
Then if you enter your WordPress instance IP in your browser, you will land your WordPress home page.

After you filled in your credentials, you will get your own homepage.

That's it. You launched your own application in your own instance and your database is managed by AWS RDS service.

Thank you all for your reads. Stay tuned for my upcoming articles!!

How to limit the size of contribution as a data node in HDFS cluster?

Sri Vishnuvardhan A — Wed, 09 Jun 2021 02:42:41 +0000

Hope this title makes some sense to you that what we are going to discuss.

In this article, we will going to see this from scratch. Here, we use Linux partition concepts for limiting the size of the contribution of the data node in HDFS Cluster. You may think Why we need to limit the size, because we can't shut down the data node when it is exhausted or run out of memory and also it will limits the dynamic storage.

Note:

In this task, the OS we used is Redhat Linux(RHEL8) and you can use any Linux OS and this RHEL8 is installed on top of Oracle Virtualbox.

Pre-requisites:

Hadoop 1.2.1
Java jdk-8u171 should be installed in your system

Hadoop

Apache Hadoop is an open-source framework which is used to store massive amount of data ranging from Kilobyte to Petabytes. It functions based on clustering of multiple computers by distributed storage instead of one single Large computer thus results in reduction of cost and time.

Linux partitions

In RHEL8 Linux, there are three types of partition namely Primary, Extended and Logical partition. Normally only the four partitions can be done per hard disk. Because the metadata of each partitions stored in 64 bytes only and size of metadata for one partition is 16 bytes.

So we have to do some trick, we have to divide harddisk into two partitions namely three Primary partition and one Extended partition.

In Extended partition, partition is considered as one new Hard disk thus we can perform this trick again to get more partitions.

Totally, we can create three Primary partitions, one extended partitions and Sixty Logical partitions. Totally 64 partitions can create and we can use 63 partitions. since one Extended partition, we can't store any data in it.

Before creating partition, we required a raw hard disk. You can buy a new hard disk or create a Virtual hard disk in Oracle Virtualbox.

Please follow the below steps to create a Virtual hard disk in RHEL8 using Oracle virtual box.

Open Settings in Oracle VirtualBox
Select Add hard disk in Controller: SATA menu
Select Create Disk Image icon
Select Virtual Disk Image option
Then select Dynamically allocated option in order to make storage Dynamically filled.
Give your required storage and then click Create button
Then attach the storage and click choose button
You can find your created storage details in your dashboard

Now we have a raw hard disk, in raw hard disk we can’t store any data. We have to do partition to utilize this hard disk.

Steps involved to create a partition:

Physical partition
Format
Mount

Physical partition:

We have to decide how much space we required for our partition. After that run

fdisk -l

to find out details of hard disk details.

You can find the hard disk named /dev/sdb which we previously created. Then execute the following command

fdisk /dev/sdb

and enter n for new partitions and enter p for primary partitions and select how many sectors or GB according to your required size and enter “w” to save. You can found these in below image.

Enter below command to check whether the partition was created or not

fdisk -l /dev/sdb

But we can create only 4 partitions, so we have to do some tricks. We can create 3 Primary partitions and create one Extended partition for remaining size.

Extended partition is treated like a raw Hard disk, We can create Logical partitions inside Extended partition and Logical partition sector’s range will be same as range of Extended partition. You can seen this in below image.

Format:

First step is done. Then we have to format the disk. Format process is like creating an index in hard disk which is used by OS for searching a file and displayed to us when we click to open a file. Enter the following command for formatting the hard disk in "ext4" format. You can use any format according to your requirements.

mkfs.ext4 /dev/sdb1

Mount:

Next step is to mount the hard disk in any of the folder/directory in OS. Because, we cant go inside or access the folder in hard disk, for this we have to link/mount this to OS folder.

For this, we have to enter the following commands in Linux terminal. We created drive1 folder in root directory and its size is 1GB.

mount /dev/sdb1 /drive1

Configuring HDFS cluster

Hadoop works based on multiple clusters which comprises of three types of nodes namely Master node, Client node and Data node.

Master node

It stores the metadata of files stored in data node. It is used to provide the metadata of datanodes to client and thereby it act as a connecting bridge between them.

Client node

This node is an End-user node that decides the number of replication blocks and block size. Risk factor plays a major role in deciding replication blocks. It is directly proportional to number of replication blocks. By default,the number of replication blocks is three which can be increased or decreased based on our applications.

Data node

It is also known as slave node which is used to store the data which is provided by client node.

These three nodes combined to form a Hadoop Distributed FileSystem (HDFS) cluster. But before configuring this cluster, we are going to see how to do partition in RHEL-8 Linux since it will help us to achieve our goal of limiting the size of contribution of data node to their master node.

Master node configuration:

This configuration involves two files namely hdfs-site.xml file which involves storage folder for metadata and core-site.xml for networking part.

In core-site.xml file, we have to enter the IP as 0.0.0.0 in order to giving permissions to connect with any IP as a master node. we have to enter port number as 9001 since it is a default port number for Hadoop.The configuration of core-site.xml file is shown below.

Then enter "w" have to start namenode by using following command.

hadoop-daemon.sh start namenode

You can check whether the node is running or not by using Java Process(JPS)

Data node configuration:

It also involves two files namely hdfs-site.xml file which involves storage folder for metadata and core-site.xml for networking part. The configuration of hdfs-site.xml is shown below.

In core-site.xml file, we have to enter master IP in order to contribute your storage with it.

Then run the following commands to start a datanode

hadoop-daemon.sh start datanode

Finally, your data node is connected with master node and formed a HDFS-cluster and you can check this with a webGUI. For this you have to perform following commands in your browser.

http://masterip:50070

Finally we limited our storage to 1 GiB for datanode to namenode. You can see this in following picture.

Thank you for reading this article. Hope you got a clarity about setting HDFS cluster and Linux partitions. Please kindly share your views as a comments so that I can improvise myself to give you all a quality content.

An Impact of Confusion Matrix on Cyber Security

Sri Vishnuvardhan A — Sat, 05 Jun 2021 15:21:33 +0000

In this article, we are going to see the use cases of Confusion Matrix and its impact on Cyber Security.

Confusion Matrix

A confusion matrix is a table that is often used to describe the performance of a classification model on a set of test data for which the true values are known. The confusion matrix itself is relatively simple to understand, but the related terminology can be confusing.

There are two possible predicted classes: "yes" and "no".

If we were predicting the presence of a disease, for example, "yes" would mean they have the disease, and "no" would mean they don't have the disease.

The classifier made a total of 165 predictions in which
Out of those 165 cases, the classifier predicted "yes" 110 times, and "no" 55 times.

In reality, 105 patients in the sample have the disease, and 60 patients do not.

True positives (TP): These are cases in which we predicted yes (they have the disease), and they do have the disease.

True negatives (TN): We predicted no, and they don't have the disease.

False positives (FP): We predicted yes, but they don't actually have the disease. (Also known as a "Type I error")

False negatives (FN): We predicted no, but they actually do have the disease. (Also known as a "Type II error")

Real life use cases in Cyber Security

Here we are going to investigate the performance of our classifier in identifying ransomware as a whole. We then move on to discuss how well our classifier can identify a specific type of crypto ransomware.

Data Collection

We collect over 100MB of ransomware traffic traces from malware traffic-analysis.net, resulting in 265 unique bidirectional ransomware related flows. We collect another 100MB of network traffic that is malware free (clean) to use as a baseline. The clean data consists of flows corresponding to web browsing, file streaming, and file downloading.

The false positive rate describes how often clean traffic is misclassified as ransomware. The false positive rate also needs to be as low as possible to prevent the unwarranted blocking of clean traffic. Furthermore, a high false positive rate results in a base rate fallacy issue, which quickly results in a massive number of falsely identified ransomware traffic.

To measure the classifier’s success, we also look at the F1 score.

The F1 score is a weighted average of the recall and precision scores and provides an idea of the balance between the false negative and false positive rates.

Initial Classification Model

We first tune our stream processor to extract 28 unique features
from our collected network traffic. These features are fed into the classifier, which first ensures the data contains the same number of malicious flows as clean flows in order to prevent classification bias.

The data is then split into two, unequal sets. One set consists of
70% of the data and is used for training and the other set holds the remaining 30% of traffic and is used for testing the learned model.

A 810-fold cross validation (CV) is performed on our data splitting to ensure our splitting model is unbiased. The confusion matrix shows the results of our classifier using 28 different
features. Even with a smaller set of traffic data, 200MB, we are
able to achieve a respectable recall of 0.89, a precision of 0.83, and an F1 score of 0.87.

Feature Reduction

Feature reduction is a key method used in machine learning to
increase classification accuracy while simultaneously reducing the
computational cost of the model. In order to reduce the number
of feature in our model, we identify the top eight most influential features in classifying ransomware traffic

fig 1:The confusion matrix of our 28-feature random
forest classifier shows a recall of 0.89, a precision of 0.83, and
an F1 score of 0.86.

fig 2: The confusion matrix of our 8-feature classifier shows similar results to that of our 28-feature classifier with a recall of 0.87, precision of 0.86, and F1 score of 0.87.

Conclusion

The 8-feature model has a slightly lower recall score at 0.87 but produces a higher precision and F1 scores of 0.86 and 0.87, respectively. However, Figure 1b shows a slightly smaller AUC for the 8-feature ROC indicating that the 8-feature classifier performs about 1.4% worse than the 28-feature model. This slight performance loss will be worth the computational savings when running classification at line rate.

Create, Crop, Swap and Collate a BGR image using Python

Sri Vishnuvardhan A — Thu, 03 Jun 2021 15:22:30 +0000

In this article, We are going to see how to create a coloured image using Python. Not only that, we are also going to see how to make changes in that image which includes Crop, Swap and collate the images.

In Simple words, an Image is just a 2D/3D array. If it is grey image, then it is a 2D array and if it is a color image, then it is a 3D array.

Concepts used: Arrays

Aray Slicing

In Python array, there are multiple ways to print the whole array with all the elements, but to print a specific range of elements from the array, we use Slice operation.

Slice operation is performed on array with the use of colon(:).

To print elements from beginning to a range use [:Index], to print elements from end use [:-Index], to print elements from specific Index till the end use [Index:], to print elements within a range, use [Start Index:End Index] and to print whole List with the use of slicing operation, use [:]. Further, to print whole array in reverse order, use [::-1].

credit: GeeksForGeeks

Array

A Python Array is a collection of common type of data structures

Arrays are same as lists. Only difference between them is list is only meant for row-wise operation but Arrays can do both row-wise and column-wise operations too.

The arrays are especially useful when you have to process the data dynamically.

Python arrays are much faster than list as it uses less memory.

Creating your own image

We already know that image is just an 2D/3D array. First step is to create a set of arrays with our desired dimension/pixels using the following commands.

import cv2 import numpy as np size= 600,600,3 zero=np.zeros(size, dtype=np.uint8)

Here, I will create my image with the dimension of 600x600 pixels and the name of the array is zero.
Its output will look like

Next step is to give/assign the color to that image since it is a plain black image.

It is done by using the concept of arrays and color codes which are available in internet. You can refer this color code by using below link.
https://www.rapidtables.com/web/color/RGB_Color.html

One thing you should note, the color code followed in arrays is BGR not RGB.

The code and its outputs are given below.

zero[0:200,0:200]=[255,153,51] zero[0:200,200:400]=[0,0,0] zero[0:200,400:600]=[0,255,255] zero[200:400,0:200]=[204,0,204] zero[200:400,200:400]=[0,255,128] zero[200:400,400:600]=[0,0,255] zero[400:600,0:200]=[0,102,204] zero[400:600,200:400]=[51,153,255] zero[400:600,400:600]=[0,153,0]

The Output is

Left side image is our generated image and right side image is the master-piece developed in Paint.

Crop and Swap the image

Again we use the concept of Array slicing to crop and swap the image. You can refer to this in below code.

#cropping and replace(swap) the squares zero[100:200,100:200]=[0,153,0] zero[100:200,400:500]=[255,153,51] zero[200:400,100:200]=[0,255,255] zero[200:400,200:400]=[204,0,204] zero[200:400,400:500]=[0,255,128] zero[400:500,100:200]=[51,153,255] zero[400:500,200:400]=[0,102,204] zero[400:500,400:500]=[0,0,0] zero[100:200,200:400]=[0,255,255]

The output is

Yes, Finally We successfully swapped the boxes.

Collate the image

You may notice this option in your Image editor in your phone. Yes, Now we are going to create our own Image Collager using concatenate function in Python.

There are two types of Concatenation namely Horizontal concatenate and Vertical Concatenate.

We are going to create an another 3D array and concatenate them horizontally with that existing array/image.

The code is listed below.

#Creating another same array(image) for horizontal concatenating size= 600,600,3 Hconcate=np.zeros(size, dtype=np.uint8) Hconcate[0:200,0:200]=[255,153,51] Hconcate[0:200,200:400]=[0,0,0] Hconcate[0:200,400:600]=[0,255,255] Hconcate[200:400,0:200]=[204,0,204] Hconcate[200:400,200:400]=[0,255,128] Hconcate[200:400,400:600]=[0,0,255] Hconcate[400:600,0:200]=[0,102,204] Hconcate[400:600,200:400]=[51,153,255] Hconcate[400:600,400:600]=[0,153,0]

```#Horizontal concatenating axis=1

Vertical concatenating axis=0

horizontal_concate=np.concatenate((zero,Hconcate),axis=1)```

After concatenation done, We can see our output using cv2 library in Python by using below code.

cap=cv2.imshow("photo",horizontal_concate)
cv2.waitKey()
cv2.destroyAllWindows()

waitKey is used to hold the operation and when we press enter, destroyAllWindows will destroy the process and process will shut down.

The output of Concatenation is shown in below figure.

Finally we done Concatenation too.

Thank you all for your reads. Stay tuned for my upcoming more interesting articles.

Launch a GUI Application inside the Docker Container

Sri Vishnuvardhan A — Mon, 31 May 2021 14:35:14 +0000

In this article, we are going to see how to run/launch the Jupyter notebook inside the docker container.

By default, we cant run any GUI applications on Docker container. By performing some trick, we are going to run those applications inside the docker container.

We have to know our GUI Display variable of our localhost by running the following command.

echo $DISPLAY

Here, DISPLAY variable is :0

Then we have to create the docker container by using following command.

docker run -it --name mygui --net=host --env="DISPLAY" -v /root:/root centos:latest

Here, we are giving docker network as host network in order to reach the host and access the host environment, since we are going to use host's Graphical Interface to run the GUI applications on Docker containers.

We are importing a environmental variable DISPLAY in order to get that display to run GUI applications.

Now, you are inside the Docker container. But, you may notice that your hostname on bash is not changed.

You can also verify using your IP or running any non-default applications.

Next step is to install firefox and Python3, because Jupyter is a python library so we required Python and Jupyter runs on the top of firefox.

You can install those packages using the following command.

yum install firefox python3 -y

After installing those packages, final step is to install Jupyter using pip3 by using following command.

pip3 install jupyter

Now, your Jupyter library is installed. You can use it by running the following command.

jupyter notebook --allow-root

Since, due to security concern, Jupyter cant run in root mode.So, we add --allow-root in order to launch Jupyter in root mode.

Finally, we launch Jupyter notebook inside the container. Now, you can run any ML models on Jupyter.

Thats it. Thank you for your reads. Stay tuned for my upcoming article!!

Providing Idempotence property to HTTPD Service restart using Ansible handlers

Sri Vishnuvardhan A — Wed, 31 Mar 2021 08:01:24 +0000

Before moving to the topic, let's familiar with one unique unfamiliar concept/vocabulary ie Idempotence. Actually, this concept is borrowed from Mathematics.

Hope all are familiar with Zero. If any number multiply with zero in "N" number of times, its output will be throughout the same. This concept is called Idempotence.

Any operations are said to be Idempotent if the result of performing once is exactly equal to the result of performing multiple times.

In other words, An Idempotent operation is one that can be applied multiple times without changing the result beyond the initial application.

Ok, let's take one real industry use-case so it will make more sense to you.

It is assumed that you are familiar with the HTTPD web server which is used for the deployment of your code to your web-client.

We all know that HTTPD server's default port number is 80 and its default document root is /var/www/html. HTTPD server can only deploy the files present in Document root.

But we can change all these settings by entering inside the HTTPD configuration file, but after make changes we have to restart the HTTPD so that only the changes made by you will update or else it wont.

We already know how to launch the webserver in Docker container using Ansible. If not, please refer to my previous blog by clicking here

Here we are going to launch the web server in Managed node's EC2 instance.

Challenges

If you run the following scripts to start the webserver,

if there will be any change in configuration file like changing port number or changing document root, the change won't get updated while running playbook for next time.

It can be solved by replacing started with restarted, but it will cause other problems, the problem is every time if you run the playbook, it will always restart because it has no intelligence to check whether there is a change or not and it also consumes more resources.

For us, it should not run everytime, it should run only once when the property of its configuration file changes.

For solving this challenge, We are going to use Ansible File handlers. It can run only once when it get notified.

Code with Explanation

Here, the Installation of the HTTPD web server is done using yum module and Copying the files to its document root by using copy module.

So far, it is simple only.

Then We are going to change the default port number of httpd server in Managed node's configuration file using Regular Expression (ReGex).

And we are also using Notify keyword, it will notify/trigger restart operation.
Note: Notify will only notify to the Restart handler when there is any change of Replacement operation, or else it won't notify and sends the trigger to restart HTTPD.

If there will be no change, then "Start HTTPD" will run and it simply start the service.

In this way, We can save the CPU & RAM resources whenever there will be a need, then only HTTPD will restart, it won't run every time when playbooks are executed.

You can find these YAML scripts through my Github.

Github link: https://github.com/vishnuswmech/Providing-Idempotence-property-to-HTTPD-Service-restart-using-Ansible-handlers.git

If you have queries, You can also connect with me though

Mail: vishnuanand97udt@gmail.com

Linkedin: https://www.linkedin.com/in/sri-vishnuvardhan/

Launching the Web server and deploying the code on top of the Docker by using Ansible Playbook

Sri Vishnuvardhan A — Mon, 29 Mar 2021 11:36:40 +0000

In this article, We are going to configuring the Apache Web server and deploying the code on it on the docker container in the managed node by using ansible-playbook.

Before jumping to the practical, let's get familiar with Ansible.

Ansible

It is an open-source tool mainly used for configuration management, application deployment and in some cases also for software provisioning. Ansible is written in Python language and it uses ssh protocol.

It uses YAML file for configuration management of the system.

You can configure multiple systems by sitting in one system. The systems that need the configuration are called as Managed nodes and the system that configured the multiple managed nodes can be called as Controller Node. Typically,there will be only one controller node that will configure the "N" number of managed nodes.

Ansible has an agentless architecture, means the software that helps the configuration need to be installed only on a single Controller node only. This is the core competency and unique feature for Ansible that possess, this helps Ansible to have a market share of 74.67% in the configuration-management market.

The above figure reflects the interest of using Ansible for configuration management of their managed nodes.

Let's get into the practical.

Note: I used Amazon 2 Linux EC2 instances for this practical. I used container image as httpd

Controller node setup

First step is to configure the Ansible software or package in the Controller node. Since Ansible is written in Python, so we can install ansible by using pip3 command.

yum install python3 -y

pip3 install ansible

Then you create an inventory file which contains all the meta data of the managed nodes which you want to configure. You can find the inventory file which I configured in below figure.

After creating the inventory file, you have to update it in configuration file. The ansible's configuration file is located in /etc/ansible/ansible.cfg . If not there means, you can create ansible folder using mkdir command and ansible.cfg using touch command.

The above figure shows the configuration file and how to update the inventory in it.

After completion, you can use the below command to verify there is a connection between managed node and controller node.
ansible all -m ping

The above figure shows that the controller node has a reliable connection with the managed nodes.

Practical

Steps involved

Configuring Docker repo
Creating directory for mount
Copy the file to that directory
Python3 installation
Installation of Docker module
Installation of IP tables
Docker Installation
Starting/Executing Docker service
Launch of Docker Container

1.Configuring Docker repo

For installation of Docker, first step is to creating one yum repo called Docker.repo. Using this repo, we can install docker in our OS.

2. Creating directory for mount

Usually code is stored in file only and files are stored in directory. The deployment of code in docker container requires storage. We are providing storage by mounting docker storage to this directory. So that, in future if you want to deploy any webapp in container, you can store that program file in this baseOS directory.

3. Copy

After the creation of directory, next step is to copy the file you want to deploy in webserver in the above mentioned directory.

4. Python3 installation

Next step is to install the Python3 using yum module, since ansible is written in Python and so that we can install the python-docker module in future.

5. Installation of Docker module

Then we have to install the docker module in python by using pip

6. Installing IP tables

In some cases, this step is important, since IP tables are required to start the docker service in some systems. Because in some versions of docker, docker has no pre-installed IP tables. To avoid those conflicts, we are installing IPtables using yum module.

7. Docker Installation

Next step is to install the docker by using yum module.

8. Start and make Docker service permanent

Then we have to start the service of Docker and enable the service for avoiding the service stop due to system reboot.

9. Pull image from DockerHub

We are pulling httpd image from DockerHub by using docker pull command

10. Launching Container using docker image

Finally we are going to launch one container named httpd140 with exposed port of 8040 and also we attached docker container httpd's document root to the baseOS or host's directory. Since, the file in document root can only deployed it to the webserver.

Output

You can check this scripts from my github
GitHub URL: https://github.com/vishnuswmech/Launching-the-Web-server-and-deploying-the-code-on-top-of-the-Docker-by-using-Ansible-Playbook.git

We are done with it!!. Thank you all for your reads. Stay tuned for upcoming articles!! Suggestions are always welcome!!

Setting up your own Network that can ping Google but not able to ping Facebook in the same system without using a firewall

Sri Vishnuvardhan A — Sat, 27 Mar 2021 12:34:17 +0000

The first question that arises on your mind after seeing this title is “Why I want to block Facebook? What is the need for this?”. The answer to your question is maybe you have kids in your home or they may be lying to you by saying they are attending online classes but actually they are wasting their precious time on Social Networks. It is more common nowadays since this Pandemic happened.

Ok. Let's come to the point.

Here we are going to see how to block Facebook but ensure access to Google in the same system. Maybe you had seen this set up in your college systems where we are not allowed to use some kind of the websites.

Note: Here I am using Redhat Enterprise Linux (RHEL8) which is hosting by Oracle VirtualBox.

But before starting this practical, you should know some basic Linux Networking concepts and terminologies.

IP address

In simple words, it is like your mobile number which is used for identified you uniquely. Every computer has their unique IP address. IP stands for “Internet Protocol”, which is the set of rules governing the format of data sent via the internet or local network.

IP addresses are not random. They are mathematically produced and allocated by the Internet Assigned Numbers Authority (IANA), a division of the Internet Corporation for Assigned Names and Numbers (ICANN). ICANN is a non-profit organization that was established in the United States in 1998 to help maintain the security of the internet and allow it to be usable by all.

Each time anyone registers a domain on the internet, they go through a domain name registrar, who pays a small fee to ICANN to register the domain.

An IP address is of two types based on the number of octets namely IPv4 and IPv6.

IPv4

The above figure clearly explains IPv4. Its size is 32 bits or 4 bytes. Each number in the set can range from 0 to 255. So, the full IP addressing range goes from 0.0.0.0 to 255.255.255.255.

That means it can provide support for 2³² IP addresses in total around 4.29 billion. That may seem like a lot, but all 4.29 billion IP addresses have now been assigned, leading to the address shortage issues we face today.

IPv6

IPv6 utilizes 128-bit Internet addresses. Therefore, it can support 2¹²⁸ Internet addresses — 340,282,366,920,938,463,463,374,607,431,768,211,456 of them to be exact. The number of IPv6 addresses is 1028 times larger than the number of IPv4 addresses. So there are more than enough IPv6 addresses to allow for Internet devices to expand for a very long time.

We can find your system IP address in RHEL8 by using the following command.

ifconfig enp0s3

Here our IPv4 address is 192.168.43.97 and IPv6 address is fe80::ad91:551e:e05a:5ab8.

Netmask

Netmask plays a major role in finding the range of IPs in which they can ping each other. It has two parts namely Network ID and Host ID.

For example, with an IP address of 192.168.100.1 and a subnet mask of 255.255.255.0, the network ID is 192.168.100.0 and the host ID is 1. With an IP of 192.168.100.1 and a subnet mask of 255.0.0.0, the network ID is 192 and the host ID is 168.100.1

In the above example, NetMask is 255.255.255.0 and if we convert host ID into binary, it has 8 zeros. 2⁸ =256 IPs are available to connect.

CIDR

Classless Inter-Domain Routing, or CIDR, was developed as an alternative to traditional subnetting. The idea is that you can add a specification in the IP address itself as to the number of significant bits that make up the routing or networking portion.

For example, we could express the idea that the IP address 192.168.0.15 is associated with the netmask 255.255.255.0 by using the CIDR notation of 192.168.0.15/24. This means that the first 24 bits of the IP address given are considered significant for the network routing.

In simple words, CIDR is the count of the total number of Ones in Netmask.

Gateway

A gateway is a router that provides access for IP packets into and/or out of the local network. The term “default gateway” is used to mean the router on your LAN which has the responsibility of being the first point of contact for traffic to computers outside the LAN.

The default gateway IP of your system can be found by using the following command on RHEL8.

route -n

Here, in the routing table, the Gateway IP is 192.168.43.146. The Destination IP mentioned 0.0.0.0 indicates that we can go anywhere on the Internet and accessing any websites without any restriction.

The above concepts so far I explained are enough to understand this practical. Here comes the practical part.

The first step is to delete one of the route rule in the routing table. We have to delete the rule which permits the user to access any kind of website. It is done by running the following command.

route del -n 0.0.0.0 netmask 0.0.0.0 gw 192.168.43.146 enp0s3

After this, if you want to ping google or Facebook, it won't possible

For now, even if you are having an internet connection, you feel like you are offline because your system doesn't know the gateway address, so impossible to go out.

For this, you have to add one rule to your IP table for granting access to Google only. It is done by the below command.

route add -n googleip netmask 255.255.255.0 gw 192.168.43.146

You can find Google IP for your PC by running the below command.

nslookup google.com

After running these commands, you can notice that your Facebook IP is not pinging and at the same time your Google IP is pinging and you have good connectivity with Google.

That's it.

Thank you all for your reads. Stay tuned!! for my more upcoming interesting articles!!

DEV Community: Sri Vishnuvardhan A

Face Recognition System for provisioning AWS instance using Terraform scripts and generating Mail & WhatsApp alerts

Contents Overview:

Data Collection

Model Training

Face Recognition

EC2 Dashboard

VPC Dashboard

Model2

Data Collection

Model Training

Face Recognition

WhatsApp Alert

Email Alert

JavaScript: The Company use cases

Microsoft

PayPal

Uber

LinkedIn

Setting up Fast & Secure Globally scaled Content Delivery Network with high availability using AWS Cloudfront & S3

Content Delivery Network

CDN Architecture model

Benefits of CDN over Traditional method

Use-Cases of CDN

CloudFront

Practicals

Pre-requisites

Steps

Installing HTTPD server

Making its document root persistent

Simple Storage Service(S3)

CloudFront

Network Topology Setup in such a way that System A can ping to two Systems B & C but both systems not able to ping each other

Environment Setup

System A

System B

System C

Ping Checking

Pinging System B from System A

Pinging System A from System B

Pinging System C from System A

Pinging System A from System C

Pinging System B from System C

Pinging System C from System B

How is this Possible?

Setting up your own High Availability managed WordPress hosting using Amazon RDS

Features of AWS RDS

Creating a MySQL database with RDS

Creating an EC2 instance

Configuring your RDS database

Configuring WordPress on EC2

How to limit the size of contribution as a data node in HDFS cluster?

Note:

Pre-requisites:

Contents:

Hadoop

Linux partitions

Steps involved to create a partition:

Physical partition:

Format:

Mount:

Configuring HDFS cluster

Master node

Client node

Data node

Master node configuration:

Data node configuration:

An Impact of Confusion Matrix on Cyber Security

Confusion Matrix

Real life use cases in Cyber Security

Data Collection

Initial Classification Model

Feature Reduction

Conclusion

Create, Crop, Swap and Collate a BGR image using Python

Aray Slicing

Array

Creating your own image

Crop and Swap the image

Collate the image