DEV Community: Jaya chandrika reddy

Hello Hadoop | Learn Hadoop in just a few minutes easily!

Jaya chandrika reddy — Fri, 05 Jun 2020 17:59:21 +0000

Originally written at: https://jayachandrika.com

Lets explore Hadoop(Click to skip to that topic):

Hadoop: Whats the buzz around it?
Hadoop EcoSystem
HDFS: Hadoop Distributed File System
YARN: Yet Another Resource Negotiator
MapReduce: Mappers and Reducers
Pig : A fast and easy alternative to MapReduce
TEZ
Hive
Ambari: UI
Mesos: An Alternative to YARN
Spark: Queen of Hadoop Chessboard
HBase
Apache STORM
OOZIE
ZooKeeper
Data Ingestion: Sqoop + Flume + Kafka
External Data Storages: MySQL, Cassandra, MongoDB
Query Engines: Drill + Hue + Phoenix + Presto + Zeplin

Platform: Hortonworks Sandbox

Hadoop: Whats the buzz around it?

Hadoop is an open source software platform with utilities or tools, for distributed storage and distributed processing on very large files and datasets, performed on several computer

I agree if it doesnt make complete sense yet. Lets break it down.

Example: Lets say we got employee information of millions of people working in Google, Amazon , Facebook and we have to combine all their information and remake the list in decreasing order of salary.

To load such big data into one normal computer and perform operations on it, making the computer to process each every employee, will either make computer to burst out like a SpaceX rocket or take a lo...ng time to process that our next generation have to see the output!

If we alloted this data to several computers, each computer processing a small piece of it, it gives fast output without the overload of processing huge data.

This is what Hadoop does,

distributed storage-The huge data is distributed to several computers and is stored there.

distributed processing-Instead of one costly computer with high processing power, several normal computers can process the big data.

In short, Hadoop has tools to store, process this big data giving us the expected results.

Lets see how exactly Big Data is handled in Hadoop!

Hadoop EcoSystem

Hadoop has many components, each has its own purpose and functions.(Kind of like each hero in Endgame has their own movie.)

HDFS, YARN and MapReduce belong to core Hadoop Ecosystem while others were added later on to solve specific problems. Lets explore each one of them, one by one.

Lets say we have a huge chunks of potato(Big data) with us and we wish to make French fries, Chips and Nuggets.

In order to make them, we divide potato into 3 blocks of potato for french fries, potato for chips and nuggets,
which is what HDFS does, dividing the big chunk of data and distribute to several computers.
All resources or ingrediants required for each dish are alloted respectively, similiar to YARN which manages the resources needed for each chunk or cluster.
We transform this potato to the desired dish which is same as MapReducers which transform data in efficient manner and aggregate the data giving us the desired result.

Lets explore in more detail about each component.

HDFS: Hadoop Distributed File System

As it says, it distributes the big file across many computers as follows:

The big file is converted to blocks(each block size max upto 128 Mb)
Blocks are stored across many computers.
Redundancy:To handle failure of one computer, more than one copy of each block is stored in each computer. So that if one fails,other computers will have, failed computer's data.

YARN: Yet Another Resource Negotiator

Its the heartbeat of the hadoop system, all the data processing happens here.
It provides the computational resources like CPUs, memory, etc for application executions.
It does job scheduling and manages the resources like what nodes are available for, which ones should be allowed to run tasks etc.
Usually, its advised not to code against YARN directly. Instead, use one of the higher-level tools, for example Hive, to make your life easier.

In a way, like how Operating System allots resources to a computer, YARN allocates resources in Hadoop.

MapReduce: Mappers and Reducers

So we got data in HDFS, resources allocated from YARN. Now how to make get our desired results from this? We code using MapReduce.

MapReduce has two functions: Mappers and Reducers.
Here python or java programming is used to code those functions.Its overall, a programming model to process and compute across the data.

Mappers split the data and map them according to the desired output.
Reducers take the output of the mappers and aggregate or combine the data to give result.

As input gets in, mapper transforms data one by one and organizes according to the output we desire.

The reducer, reduces the mapper output to the aggregated form which we require as output.

Pig : A fast and easy alternative to MapReduce

Pig Latin, a scripting language, uses SQL like syntax, forming relationship, selecting a data, filtering it, transforming it etc. Contains User defined functions to help us create our own functions.
Not all business problems can be solved by converting to mappers and reducers form. So Pig Latin sitting on top of MapReduce, converts our Pig Latin script into map reduce form.
Pig script can be run by either of the 3 below:
- Grunt(Command Line Interpreter)
- Script(place the script in file and running it)
- Ambari (Directly from Pig View)

TEZ

We might wonder if using pig on top of mapreduce might increase latency. Tez is present in between them, to optimize and make it 10x faster.
It makes Hive, Pig and Map Reduce jobs faster, optimising physical dataflow and resource usage.
Using DAG(Directed Acyclic Graphs) it sees the workflow of the jobs and finds the optimal way among it making our job easier.

Suppose:we have three ways, 1-2-3 , 4-9-8, 2-7-9 and we need to find the smallest path, we choose 1-2-3. DAG works in somewhat similar way.

The output is processed again and again Through DAG giving us the optimal solution.

Hive

It takes SQL queries as input from command prompt and transforms them to mapreduce jobs and automatically gets the job done.
We can store the result of query into a view, perform a query onto entire cluster and define our own functions.
Though Pig and Hive might be similiar,Hive is used for completely structured Data whereas Pig Hadoop Component is used for semi structured data.
Pig is relatively faster than Hive.
Hive is slow and not useful for realtime transactions unlike Pig and Spark.
It might give the feel of a relational database, but it cant delete, insert etc as there are no tables, joins, primary keys and the data is just a flat text, de-normalized.

As you can observe,the queries used are same as SQL queries.

Ambari: UI

Its pretty much a web interface to graphically interact with the cluster and its usage.
It gives us a high-level view of your cluster, what’s running, what system you are using, what resources are available/in use.
It shows statistics about the cluster like CPU Usage,Memory Usage,Network Usage, Cluster load etc.
All services like HDFS, YARN etc can be used on a cluster using Ambari.
Even Hadoop can be installed using Ambari.

Mesos: An Alternative to YARN

Its simliar to YARN but then difference is Mesos is more general.
In simple words, YARN works only on Hadoop components like mapreduce etc where as Mesos can allcate resources to outside of hadoop like web servers.
YARN is for analytical and long process jobs where as Mesos is for both long and short process jobs.

Spark: Queen of Hadoop Chessboard

Spark engine is pretty fast, infact 100x faster than mapreduce and is used for large scale data processing.
Like Hive -> it can handle SQL queries, like Apache STORM ->it handles real time streaming hence we call it the queen.
Processing in Spark happens in this way:
- Driver Program, containing script to control what happens in the job.
- Cluster Manager,which can be YARN, Mesos or its own Spark Cluster Manager.
- Executor, it contains cache prioritizing memory based solution unlike other disk based solutions.
Spark has many components:
- Spark Streaming: Handles Realtime data.
- Spark SQL: SQL interface, takes SQL queries.
- MLLib: Machine Learning operations can be performed without about mappers and reducers to solve a problem.
- GraphX: All graph related problems and analytics are done using this.
- Spark Core: It provides distributed task dispatching, scheduling, and basic I/O functionalities.

Spark uses a specialized fundamental data structure known as RDD (Resilient Distributed Datasets).Its an object representing dataset and we can perform operations on that object just like in java.

HBase

Its basically a scalable NoSQL Database.
A columnar data storage, pretty fast and finds it useful for high data transaction rates.
CRUD(Create,Read,Update and Delete) operations are employed here.

Apache STORM

Processes continuous realtime streaming data like from sensors or weblogs etc.
Even spark streaming does this but more in batch intervals but storm handles data event wise.
Spout is the term used for sources of the streaming data i.e., they produce. Bolts is the term used for those which transform or aggregate this streaming data.
Usually kafka(which sends data into cluster) and storm are used as a pair.

OOZIE

Runs and Schedules various jobs/tasks on Hadoop.
Instructions are given via XML file.

ZooKeeper

Co-ordinates everything on cluster. Like which is the master node, task assigned for each worker node, which node is up and which is down and which worker nodes are available etc.
If a master node dies, it chooses who the next master should be.

Example:This phenomenon can be observed in whatsapp groups or in videocalls where if the host leaves the group, someone else is assigned as the next host.

Outside Hadoop EcoSystem, we can have databases to store the data and Query Engines to get input data to the cluster. Lets have a look at them!

Data Ingestion: Sqoop + Flume + Kafka

So how do you get data into the cluster? Sqoop,Flume and Kafka help with this via data ingestion.

Sqoop: Sqoop is a command-line interface application tool for transferring data between relational database servers and Hadoop.

Its just a way of tying hadoop database to relational database.

Flume: Another way of sending data into cluster and it acts as a buffer between the data coming in and storing, so that cluster wont go down with overwhelming data coming in realtime.

Kafka: This is more a general publisher/subscriber model where the publishers send/produce data as topics which are subscribed by consumers.

Collects any sort of data from PCs, webservers etc and broadcast it into Hadoop Cluster.

External Data Storages: MySQL, Cassandra, MongoDB

MySQL: Typical SQL based relational database.

Cassandra : Its a NoSQL columnar database with no single point of failure. It favours availability and partial- tolerance in CAP theorem.

MongoDB : A NoSQL database which uses document model. Its used when we deal with huge data and its pretty flexible.

Query Engines: Drill + Hue + Phoenix + Presto + Zeplin

All these are interactive query engines to fetch required data from databases under them like Hbase, Cassandra etc.

Drill lets us use SQL queries, even when the databases under it (Hbase, Cassandra, MongoDB etc) are non relational Databases.
Phoenix is similar to drill with SQL queries but with ACID properties(Atomicity,Consistency,Isolation and Durability).Its fast but works only HBase.
Presto is similar to Drill and unlike Drill, it connects to Cassandra and executes query across the cluster.
Hue (Hadoop User Experience), an SQL Cloud Editor, is used for query and as UI in Cloudera just like Ambari to HortonWorks.
Zeppelin has Notebook UI to visualize the data similar to jupyter notebooks for TensorFlow and share, interact with the code.

It might be overwhelming to see many components doing the same work.Ultimately based on the task and the type of database(whether its relational or non structered data etc) and other factors, we need to decide the right component and utilize it and it comes with knowledge of the components,practice and experience.

Hope you had fun learning Hadoop, i for one certainly had fun sharing about it!!

Originally written at: https://jayachandrika.com

Now you can follow us to explore more interesting topics, in Instagram at: @code_voyager

Great to have you here, lets get back for the next post and explore more!

Have an amazing day!

Learn Docker basics quickly!

Jaya chandrika reddy — Tue, 26 May 2020 15:04:54 +0000

Article originally written at https://jayachandrika.com

In this Article lets explore :

Docker:What the hell is it?
But why Docker??
What are Containers
Images
Magic Commands

Have you ever built an application but it doesn't run on all platforms? Or been sick of installing different dependencies and libraries for each service/component in the application? Dodged the whole nine yards just to deploy, scale and manage a simple application??

Then DOCKER is waiting for you!!!

Docker:What the hell is it?

Docker is a set of platform as a service products that uses OS-level virtualization to deliver software in packages called containers.

(Well. . .thats the GOOGLE definition for you! )

To wrap it up simply,it is just a BOX!

Imagine,

A printer box, we insert a paper and get printed text or colors on it.

So how is it working ? Perhaps all the colors and required material is already present in the printer.
Consider a car wash, car is sent in, many process like washing, cleaning etc happen and at the end we get a shiny clean car (or a less rusty one).

So, docker is like a printer box or car wash setup in this instance, taking an input and giving an output basically.

It is basically a box with an application and its related dependencies, libraries etc inside it

But why docker?

Imagine we are building a zoo, we toss all animals like monkeys, Lion and Crocodiles in one big park with only one kind of food and only one kind of environment, lets say, land.

The bananas eaten by monkeys can’t be used by Lions and crocodiles, the meat for Lions can’t be eaten by monkeys. Monkeys want trees and crocodiles need a water space in the environment.

In similar way, if we are building an application that has services like a database and framework etc, The libraries and dependencies(food) used by databases (cassandra, postgresql, mongodb etc) may not be compatible/suitable for frameworks (strapi, nodejs express etc).

Eg: MongoDb version 4.26 might not be compatible/suitable for NodeJs version 4.16.4

To make it easy to develop, run and maintain an application and to operate on all the platforms we came up docker. The problem in zoo example can be solved by providing each animal, a separate boundary with the required food and environment.

Containers

In docker we have those separate boundaries called containers. We place each service in one container with all the libraries and dependencies required for it.

Container is a completely isolated environment with their own processes, network interfaces and their own mounts.

Wait, doesnt it look familiar? Yes,you are right if you guessed it, they remind us of Virtual machines. If its similar to VMs, Why on Earth do we need to use Containers??

Well, Each VM has a separate OS along with libraries, dependencies and application, which increases overload and utilization, making it difficult to operate it in small PCs(though we might not use it there).

Whereas Containers share the same OS, considerably less overload and utilization than VMs. Hence they are just the darlings preferred for application development, deployment and management.

Images

These are readymade templates of services like MongoDb, NodeJs etc, to run in containers. We can either use existing images from docker.io or create our own.

Creating an Image of the service helps in shipping it for deployments. Then it is just running a simple command to get the server up and running. We dont have to worry about creating environment, installing dependencies and what not.

Magic commands

Its time to blow some fairy pixel dust and do some magic!

Lets start with running an image in a container,using run command.

Run redis image in container

docker run redis

List the version details

docker version

List the Running containers

docker ps

List the Running & Not Running containers

docker ps -a

Remove a container

docker rm [container ID]

Note: To remove a container make sure it is stopped first using stop command.

List the images

docker images

Remove an image

docker rmi [image_name]

Just download the image but not run in the container

docker pull [image_name]

Thank you so much for reading my content. More of such content coming on this blog, so stay tuned.

Article originally written at https://jayachandrika.com

Draw Shapes,Text,Polygons,Blend,Blur images | Using OpenCV With Python | Easy code

Jaya chandrika reddy — Tue, 26 May 2020 09:24:14 +0000

In this Article lets learn:

[Blend: How to Combine 2 Images?]
[Draw Shapes & Text on Image]
[Draw a Polygon]
[Blur an image]

Pre Requesites: Jupyter or any python editor

Blend: How to Combine 2 Images?

Blending in OpenCV is joining two images of same size into each other.

We can make cool posters, blend your loved ones picture with their favourite character on background and gift them, bring out your creativity and what not!

Lets begin the magic!

    import cv2  #opencv library
    import matplotlib.pyplot as plt # Matplotlib used for visualization
    %matplotlib inline

    # Reading the jpeg formmatted image from its path to img1
    img1=cv2.imread(r'./img/hp.JPG')

    # Reading the jpeg formatted image from its path to img2
    img2=cv2.imread(r'./img/doc.JPG')

    # Converting from BGR(blue,green,red) color space to RGB using cvtColor
    img1=cv2.cvtColor(img1,cv2.COLOR_BGR2RGB)
    img2=cv2.cvtColor(img2,cv2.COLOR_BGR2RGB)

    # Displaying the image 1
    plt.imshow(img1)

    # Displaying the image 2
    plt.imshow(img2)

Output:

    #'1' Flips image to left side, '0' to upside down and '-1' to combination of both 0 and 1
    flip_img=cv2.flip(img1,1)

    # Showing flipped image
    plt.imshow(flip_img)

Output:

    # Gives the shape (height,width, no.of color channels(RGB)) of image1
    flip_img.shape

Output:

(2093, 2652, 3)

    # Gives the shape (height,width, no.of color channels(RGB)) of image2
    img2.shape

Output:

(2127, 3840, 3)

    # Both images should be of same size to blend.
    flip_img=cv2.resize(flip_img,(2652,2093))
    img2=cv2.resize(img2,(2652,2093))

    # Increase in alpha increases img1 weightage,beta for img2
    blended=cv2.addWeighted(src1=flip_img,alpha=0.9,src2=img2,beta=0.9,gamma=0)

Output:

    # Saves the blended img under the name blended.JPG
    cv2.imwrite('blended_img.png',blended)

Output:

True

Draw Shapes on Image

Different shapes can be drawn on an image, to mark an object or create a figure etc.

Rectangles
Hollow circles
Solid Circles
Lines

and many more can be drawn using OpenCV.

    import cv2 #  opencv library
    import numpy as np
    # Matplotlib used for visualization purpose
    import matplotlib.pyplot as plt
    %matplotlib inline

    # Array of size 512x512 filled with zero as values,
    # to create an image with black color.
    img=np.zeros(shape=(512,512,3),dtype=np.int16)

    # Displays the shape(Height,width,no.of color channels(RGB))
    img.shape

Output:

(512, 512, 3)

    # To display the image
    plt.imshow(img)

    # cv2.rectangle(image, start_point, end_point, color, thickness)
    cv2.rectangle(img,pt1=(0,0),pt2=(100,100),color=(0,0,255),thickness=10)

Output:

array([[[ 0, 0, 255],
[ 0, 0, 255],
[ 0, 0, 255],
...,
[ 0, 0, 0],
[ 0, 0, 0],
[ 0, 0, 0]],..

    plt.imshow(img)

    # cv2.circle(image, center_coordinates, radius, color, thickness)
    cv2.circle(img,center=(150,150),radius=100,color=(255,0,0),thickness=10)

Output:

array([[[ 0, 0, 255],
[ 0, 0, 255],
[ 0, 0, 255],
...,
[ 0, 0, 0],
[ 0, 0, 0],
[ 0, 0, 0]],
[[ 0, 0, 255],
[ 0, 0, 255],
[ 0, 0, 255],
...,

    plt.imshow(img)

    # Negative values for thickness results in filled/solid circles
    cv2.circle(img,center=(200,200),radius=100,color=(255,255,0),thickness=-10)

Output:

array([[[ 0, 0, 255],
[ 0, 0, 255],
[ 0, 0, 255],
...,
[ 0, 0, 0],
[ 0, 0, 0],
[ 0, 0, 0]],

plt.imshow(img)

    # cv2.line(image, start_point, end_point, color, thickness)
    cv2.line(img,pt1=(275,275),pt2=(512,512),color=(0,255,0),thickness=10)

Output:

array([[[ 0, 0, 255],
[ 0, 0, 255],
[ 0, 0, 255],
...,
[ 0, 0, 0],
[ 0, 0, 0],
[ 0, 0, 0]],

   plt.imshow(img)

    # Font of the text
    text_style=cv2.FONT_ITALIC

    # Syntax
    #cv2.putText(image, text, org, font, fontScale, color[, thickness[, lineType[, bottomLeftOrigin]]])

    cv2.putText(img,text='HAKUNA MATATA',org=(10,200),fontFace=text_style,fontScale=2,color=(0,0,255),thickness=3,lineType=cv2.LINE_AA)

Output

array([[[0, 0, 0],
[0, 0, 0],
[0, 0, 0],
...,
[0, 0, 0],
[0, 0, 0],
[0, 0, 0]],..

   plt.imshow(img)

Draw a Polygon

    import cv2
    import numpy as np

    # Matplotlib used for visualization purpose
    import matplotlib.pyplot as plt
    %matplotlib inline

    # Array of size 512x512 filled with zero as values,
    # to create an image with black color.
    img=np.zeros(shape=(512,512,3),dtype=np.int16)

    # Assigning vertices to polygon
    poly=np.array([[100,200],[200,100],[400,300],[200,400]],dtype=np.int32)
    poly.shape

Output:

    (4, 2)

    # Reshaping according to opencv format
    poly_new=poly.reshape((-1,1,2))
    poly_new.shape

Output:

(4, 1, 2)

    # cv2.polylines(img, pts, isClosed, color[, thickness[, lineType[, shift]]])
    cv2.polylines(img,[poly_new],isClosed=True,color=(0,255,0),thickness=10)

Output:

array([[[0, 0, 0],
[0, 0, 0],
[0, 0, 0],
...,

    plt.imshow(img)

Blur an image

There are plenty of blurring techniques available.Some of the methods are:

Default Blurring Technique
Gaussian Blurring Technique
Smoothing Technique
Median Blurring Technique
Bilateral Filter Blurring Technique

Lets explore each one of them:

     import cv2  # opencv library

    # Matplotlib used for visualization
    import matplotlib.pyplot as plt
    %matplotlib inline
    import numpy as np

    img=cv2.imread(r'./img/image.JPG')
    img=cv2.cvtColor(img,cv2.COLOR_BGR2RGB)
    plt.imshow(img)

    kernel=np.ones((5,5),np.float)/25

    # Default Blurring Technique
    plt.imshow(cv2.blur(img,ksize=(5,5)))

    # Gaussian Blurring Technique
    plt.imshow(cv2.GaussianBlur(img,(5,5),0))

    # Smoothing Technique
    plt.imshow(cv2.filter2D(img,-1,kernel))

    # Median Blurring Technique
    # Reduces unwanted noise
    plt.imshow(cv2.medianBlur(img,5))

    # Bilateral Filter Blurring Technique
    # Reduces unwanted noise with sharp edges but slow
    plt.imshow(cv2.bilateralFilter(img,9,75,75))

Try out all these and feel free to ask me doubts via gmail at: jayachandrika56@gmail.com

Now you can follow us to explore more interesting topics, in Instagram at: @code_voyager

Hope you had fun experimenting with the code.
Have an amazing day!

OpenCV Basics

Jaya chandrika reddy — Thu, 07 May 2020 20:43:15 +0000

Want to combine two images using just a line of code? Or wanna blur an image? Lets find out how!

In this Article (Click to skip to that topic):

[Blend images: What is it?]
[Code to Blend]
[Methods to Blur an Image ]
[Code to Blur Image]

Blend images: What is it?

Blending images in opencv, is combining to images of same size. So we have to remember to resize those two images to same size(which will be shown in the code later).

Using this, we can make pretty cool posters, wallpapers, beautiful creations in your imaginations and many more...

So lets learn how to do it in a simple way

bigarr=np.arange(0, 10).reshape((2,5))
bigarr

#'arange' gives an array withn values from 0 to 9, 'reshapes' converts into a matrix of 2x5

Output:

array([[0, 1, 2, 3, 4],
[5, 6, 7, 8, 9]])

bigarr[:, 2]

#Here only column 2 is considered but not row as colon(:) is mentioned

Output:

array([2, 7])

For more practice on Numpy visit

 https://numpy.org/devdocs/user/quickstart.html

## Display Image in Opencv using Numpy

Now,how to open or show an image using our numpy knowledge:

We use libraries

PIL(Python Imaging Library)
Matplotlib( For visualization) and import Image from PIL

Image.open(image file path)       

# Opens and identifies the given image file

Each pixel has a shape.

For Eg:(1080,1080,3)which is (pixel height,pixel width,color channels)
The 3 color channels are red,green and blue(RGB).

plt.imshow                         

# Displays the image

Example

import numpy as np
import matplotlib.pyplot as plt 
%matplotlib inline
from PIL import Image

img=Image.open(r'C:\Desktop\image.jpg') 

# Opens Image

img

#To Preview Image

Each pixel has a shape.

For Eg:(1080,1080,3) which is (pixel height,pixel width,color channels).

The 3 color channels are red,green and blue(RGB).

plt.imshow             

# Displays the image

Example:

img_array=np.asarray(img)

# Converting image into array of values

img_array.shape

# Displaying the shape of the array

Output:

(1080,1920,3)

plt.imshow(img_array)

# Displaying array into Image

What if only one channel is used in displaying image instead all 3 in RGB?

plt.imshow(array[:,:,0])           # Displays image with only Red channel.

plt.imshow(array[:,:,1])           # Displays image with only Green channel.

plt.imshow(array[:,:,2])           # Displays image with only Blue channel.

## Gray Image using Numpy

What if we want a gray image like in old movies?

The parameter cmap=’gray’ is used to convert color images into a gray image.

plt.imshow(array[:,:,0],cmap=’gray’) 

# Displays image with grayscale values.

Example

plt.imshow(img_array[:,:,0])

#Displaying only RED channel in RGB. Replace 0 with 1 for Green and 2 for Blue Channel.

plt.imshow(img_array[:,:,0],cmap='gray')

#Converting the color image into grayscale using cmap

For more code on fliping, resizing etc visit the blog, follow the Instagram page mentioned below 😄

If you are building a project or wanna upgrade your skills for the job or just excited to build cool stuff, learn opencv easily and quickly from Instagram page code_voyager:

https://www.instagram.com/p/B_5ey22pLu_/?igshid=1t6k2adh9a9zk

Or more detailed version feel free to visit 😄:
https://jayachandrika.com/computer-vision-images

Looking forward for you to join us!😊