DEV Community: hellocodeclub

Install Hadoop on Mac – Ultimate Step by Step Guide

hellocodeclub — Mon, 28 Mar 2022 21:20:21 +0000

In this tutorial you will learn how to install the hadoop system in your mac machine, including a brief reminder of what hadoop is and its architecture.

When talking about hadoop, first idea that probably crosses your mind is big data. Hadoop emerged with big data as there was a need to store massive amount of data. Not only store it, but analyse it and access it in a reliable, scalable and affordable manner.

The hadoop system solves two key big data problems. First problem, what if one of the computer fails? Traditionally if a machine fails, all information stored is lost, unless there is a backup. The hadoop system have mechanisms to avoid this problem.

The second challenging problem was combining the information from different hard drives. When you are analysing large amount of data saved in many hard drives, accessing and combining this information can be a challenging. Fortunately, Hadoop also tackles this issue.

What is Hadoop?

Hadoop is an open source software optimised for reliable and scalable distributed computing. What does that means? Distributed computing means that instead of a single computer carrying out a processing task, the task is performed by several machines. Multiple computers, all connected together to attempt one goal.

The Hadoop software includes mechanisms that avoid data loss. And It is a scalable system, meaning more computers can be added to the system as the data grows.

Hadoop is designed to handle large files. Once you store a file in Hadoop, the file will be split in smaller pieces and each piece stored in different a machine within the cluster. Plus each file block is replicated in several machines to avoid data loss.

The whole system can be scaled up from one server to thousand of servers. As a result, the computation and storage power of each server is combined resulting in a really powerful system.

Learn more about Hadoop's architecture

Learn about different installation modes

Standalone installation

As we have seen, you can install Hadoop on mac in three different mode. One of them is standalone. Standalone means that there are no separated daemons, all Hadoop processes are running on the same JVM. See below the steps to install Hadoop in standalone mode:

Step 1) Check Java is installed

Hadoop is a software written in Java, and behind the scenes is using Java. Therefore first thing to do is indicating to hadoop where Java is installed. To do so, you need to set up the JAVA_HOME enviroment variable on your machine.

export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk-14.0.2.jdk/Contents/Home/

Then check that java is installed, by running the following command:

java --version

Step 2) Download Hadoop

You can download hadoop from the following website: Download Hadoop

Select any of the mirrors, then pick a version and download the file called hadoop-X.Y.Z.tar.gz.

Step 3) Set up Hadoop environment variables

Next step to install hadoop on mac is creating the hadoop environment variables. You will need to create the HADOOP_HOME environment variable, which will point to the directory where you uncompressed the previous file. You should also add this path the your PATH variable. See below the command you should execute:

export HADOOP_HOME=~/sw/hadoop-x.y.z
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

Run Hadoop on Pseudo Distributed Mode

Graph Implementation Example in Java

hellocodeclub — Mon, 28 Mar 2022 20:34:59 +0000

In this article you will see a graph implementation example in Java from scratch. You will learn how to create and use a graph data structure in Java, practicing with a real exercise that I have seen in many interview processes.

The example is an exciting and challenging problem to solve. Firstly I will introduce the problem itself and a possible way to solve it in Java.

The problem

A delivery service company creates a system that will plan a route for a delivery truck to deliver customer orders. The planner will create a delivery area for each order, which is abstracted as a grid. The challenge is to write an algorithm to find a path for the truck to deliver the order.

The delivery area is represented as a two-dimensional grid of integers where:

1 represents an accessible area
0 represents a non-accessible area
2 represents the order destination

This exercise is an excellent example in Java of how using the correct data structure can turn a seemingly complex problem into something much more straightforward. The problem becomes simpler when you use a graph implementation.

Some stuff you should consider before starting. The truck can’t leave the grid and should get to the order destination through the accessible areas. And the truck can move one cell up, down, left, or right at a time.

Here are some examples of different cases with the inputs and the corresponding outputs:

  Example 1
  grid = [[1,1,0],[0,1,2][0,1,1]]
  Output: [0,0][0,1][1,1][2,1][2,2][1,2]

  Example 2
  grid = [[1,1,1,1],[0,1,0,1],[0,1,0,1],[0,1,1,0],[0,1,2,1]]
  Output: [0,0][0,1][1,1][2,1][3,1][3,2]

Solution

The best solution to this problem is using a java graph implementation. In general, a graph is the right solution for any situation where you need to navigate a grid.

This solution will have two classes: Cell class representing each matrix cell, and DeliveryArea representing the area the truck should navigate. See the skeleton below, just the class and the method signatures. We will complete the implementation in the next section.

public class Cell {
    public int row;
    public int col;

    public Cell(int row, int col){
        this.row = row;
        this.col = col;
    }

    public String hashKey(){
        return String.valueOf(this.row)+String.valueOf(this.col);
    }
}

import java.util.List;
import java.util.Map;

public class DeliveryArea {

    private final static int ACCESSIBLE_CELL = 1;
    private final static int NON_ACCESSIBLE_CELL = 0;
    private final static int DESTINATION = 2;

    public int[][] matrix;

    public DeliveryArea(int[][] matrix){
        this.matrix = matrix;
    }
    private Map<String, List<Cell>> buildGraph(){
        return null;
    }
    public List<Cell> findRoute(){
        return null;
    }
}

Check out the full solution

How To write a Rock Paper Scissors Game in Java - Step by Step

hellocodeclub — Thu, 07 Oct 2021 07:58:16 +0000

In this article, I want to help you to design a Rock Paper Scissors Game in Java step by step. And first step is overcoming your self-doubt and concentrate.

You will put together everything you know about programming to build a rock paper scissors game that makes you feel proud of yourself. Let's see how you will achieve this step by step.

We will start by analysing the game and defining the key entities in our problem. We will use the object oriented programming approach, meaning we will turn the entities into classes.

Next we will define a skeleton implementation, in other words creating the classes but no code. And last we will fill up the classes with code and have a working rock paper scissors game.

Define How your Rock Paper Scissors Game works

Let’s start by defining how the game should work. You will be creating a rock paper scissors text-based java game.

The first step is writing down the steps of what the game should. By doing so, it can help us identify the classes and actions that we will need our game. The steps are:

The Game welcomes the user.
The game requests the user to enter his choice(rock, paper or scissors)
The user types the selected option.
The game picks randomly among rock, paper or scissors.
Calculates the results.
The game updates the total score( how many games you play in total and won games)
Display the score.
Ask the user if he wants to continue.

Writing down how the game will work is a great start. It helps you identify some of the main entities of the problem, so later you can turn them into class. In this case, we will create the following classes:

RockPaperScissorsGame: the entry point of the application
Game Controller: this class will represent the game itself, containing the scores, the player choices, and operations related to the game mechanism like calculate who won, etc.
GameOption: This class represents different options the player can select: rock, paper, or scissors.
Player: It helps indicate you won, the user, the machine, or the tie.
Score: It contains the total number of games player, and the number of games played by the user.

Create a Skeleton Implementation

Now that we have an overall view of how the rock paper scissors java game will work and the main classes in our program, we can start creating them and give shape to our game.
At this option, I will advise you not to add implementation. The skeleton should contain the classes and the empty operations of these classes. These operations define how the classes interact with each other. For now the implementation is not necessary.

RockPaperScissorsGame class

This class needs a main method since it is the entry point of our java program. Since we are going to interact with the game, we will create an instance of the class to access its operations.

public class RockPaperScissorsGame {

public static void main(String[] args){

        GameController gameController = new RockPaperScissorsGameController();

    }
}

SEE HOW TO CREATE ALL OTHER CLASSES HERE

Complete the Implementation

Now we have created our application’s skeleton, and we have a better overall understanding of how the game will work. So it is an excellent time to start filling the gaps and get the game working.

Let’s go back to our game’s entry point. The entry point which is the main method. Every statement executed by the game will be inside the main method.

Since you need to receive inputs from the user, we will need a Scanner class. Additionally, the game will run until the user indicates he wants to exit, so we will need a do-while loop to keep repeating.

Inside the loop, the game will:

Request the user selection and save it.
Pick a random option
Calculate the result
And finally, display the game results and ask the user if he wants to continue.

SEE ALL JAVA CODE TO COMPLETE THE GAME HERE

I hope you enjoy this article, and thank you so much for reading and supporting this blog! Happy Coding! 🙂

Apache Spark Java Tutorial: Simplest Guide to Get Started

hellocodeclub — Mon, 09 Nov 2020 07:58:41 +0000

This article is an Apache Spark Java Complete Tutorial, where you will learn how to write a simple Spark application. No previous knowledge of Apache Spark is required to follow this guide. Our Spark application will find out the most popular words in US Youtube Video Titles.

Firstly, I have introduced Apache Spark, its history, what it is, and how it works. And to continue, you will see how to write a simple Spark application.

History of Apache Spark

The history of Apache Spark emerged at UC Berkeley, where a group of researchers acknowledges the lack of interactivity of the MapReduce jobs. Depending on the dataset’s size, a large MapReduce job could take hours or even days to complete. Additionally, the whole Hadoop and MapReduce ecosystem was complicated and challenging to learn.

Apache Hadoop framework was an excellent solution for distributed systems introducing parallelism programming paradigm on distributed datasets. The worker nodes of a cluster will execute computations and aggregates results, producing an outcome. However, Hadoop has a few shortcomings: the system involves an elaborate set-up and not quite interactive. Fortunately, Apache Spark brought simplicity and speed to the picture.

What is Apache Spark?

Apache Spark is a computational engine that can schedule and distribute an application computation consisting of many tasks. Meaning your computation tasks or application won’t execute sequentially on a single machine. Instead, Apache Spark will split the computation into separate smaller tasks and run them in different servers within the cluster. Therefore, maximizing the power of parallelism.

Another critical improvement over Hadoop is speed. Using in-memory storage for intermediate computation results makes Apache Spark much faster than Hadoop MapReduce.

Architecture with examples

Apache Spark uses a master-slave architecture, meaning one node coordinates the computations that will execute in the other nodes.

The master node is the central coordinator which executor will run the driver program. The driver program will split a Spark job is smaller tasks and execute them across many distributed workers. The driver program will communicate with the distributed worker nodes through a SparkSession.

Set Up Spark Java Program

Write an Apache Spark Java Program

And finally, we arrive at the last step of the Apache Spark Java Tutorial, writing the code of the Apache Spark Java program. So far, we create the project and download a dataset, so you are ready to write a spark program that analyses this data. Specifically, we will find out the most frequently used words in trending youtube titles.

More Detailed Explanations of the below code

public class YoutubeTitleWordCount {
    private static final String COMMA_DELIMITER = ",";
    public static void main(String[] args) throws IOException {

        Logger.getLogger("org").setLevel(Level.ERROR);
        // CREATE SPARK CONTEXT
        SparkConf conf = new SparkConf().setAppName("wordCounts").setMaster("local[3]");
        JavaSparkContext sparkContext = new JavaSparkContext(conf);

        // LOAD DATASETS
        JavaRDD<String> videos = sparkContext.textFile("data/youtube/USvideos.csv");

        // TRANSFORMATIONS
        JavaRDD<String> titles =videos
                .map(YoutubeTitleWordCount::extractTitle)
                .filter(StringUtils::isNotBlank);

        JavaRDD<String> words = titles.flatMap( title -> Arrays.asList(title
                .toLowerCase()
                .trim()
                .replaceAll("\\p{Punct}","")
                .split(" ")).iterator());

        // COUNTING
        Map<String, Long> wordCounts = words.countByValue();
        List<Map.Entry> sorted = wordCounts.entrySet().stream()
                .sorted(Map.Entry.comparingByValue()).collect(Collectors.toList());

        // DISPLAY
        for (Map.Entry<String, Long> entry : sorted) {
            System.out.println(entry.getKey() + " : " + entry.getValue());
        }

    }


    public static String extractTitle(String videoLine){
        try {
            return videoLine.split(COMMA_DELIMITER)[2];
        }catch (ArrayIndexOutOfBoundsException e){
            return "";
        }
    }
}

I hope you enjoy this article, and thank you so much for reading and supporting this blog!

Creating a Speech Recognition Program with Python & Google API

hellocodeclub — Tue, 03 Nov 2020 09:59:56 +0000

Speech Recognition means that the program will capture the words produced by a person and converts them into written words. It can be handy to generate subtitles, transcript a meeting discussion, and many other use cases.

Converting speech to text is quite a complex machine learning problem where an algorithm needs to receive every sound produced by a person and identify the corresponding written letters. Plus, depending on the language used, different sounds might correspond to other characters. As a result, speech recognition is too complex to be solved using a traditional programming approach.

Fortunately, big companies like Google, Amazon, IBM, and others have already solved this problem. They collected many audios, fed this data to algorithms using machine learning techniques, and produced trained algorithms to convert speech to text with really high accuracy. Plus, these algorithms are available through API's to easily integrate them into your programs.

This article will show you how using Python, and the Google API can transcribe audio with a few code lines. Let's get started!

Python Speech Recognition using Google Api

Google offers a Speech-To-Text service through an API, meaning that you can send a request with an audio file, and you will receive the transcription of the audio file. This service makes simple, including python speech recognition functionality in your programs.
See how to set-up a google account and configure it to access Google Speech-To-Text API

Write the python program

Once you have done all configuration needed to use the google speech-to-text API, you can move on to the last step, write the python program.

Our program will need the third-party library google-cloud-speech, which will send requests to Google. You can install this library running the following command from your terminal:

>> pip install --upgrade google-cloud-speech

Lastly, you can copy the code below and save it as a python script. Please note the audio file should be in the same folder as the script. Also, you will need to replace the file name test.wav with your file name.

from google.cloud import speech
import os
import io

# Creates google client
client = speech.SpeechClient()

# Full path of the audio file, Replace with your file name
file_name = os.path.join(os.path.dirname(__file__),"test.wav")

#Loads the audio file into memory
with io.open(file_name, "rb") as audio_file:
    content = audio_file.read()
    audio = speech.RecognitionAudio(content=content)

config = speech.RecognitionConfig(
    encoding=speech.RecognitionConfig.AudioEncoding.LINEAR16,
    audio_channel_count=2,
    language_code="en-US",
)

# Sends the request to google to transcribe the audio
response = client.recognize(request={"config": config, "audio": audio})

# Reads the response
for result in response.results:
    print("Transcript: {}".format(result.alternatives[0].transcript))

In case your file has a different extension, you can convert it using an online file converter. Go to m4a to wav converter.

If your program is working correctly, this is the output you will see after executing your script:

>> python speech_to_text.py # Replace with your program file name

Output:

Transcript: hey there in this area you will learn how you can set your django version there are a few ways
Transcript:  there are a few ways to check your django version and in this video I will show you a few of them I will also show you how you can upgrade and downgrade your django version

Any errors?See some possible errors and how to fix them.

Hope you enjoy his tutorial and thank you so much for reading! Happy coding!

How to Create Tkinter Progress Bar and DropDown

hellocodeclub — Sun, 18 Oct 2020 08:38:28 +0000

In this article I will show you step by step how you can add a progress bar and a dropdown widget to you user interface developed using Tkinter. This article has two parts. First I will explain you how to create a window and add the dropdown. And how to add a callback, so the interface reacts if the user changes the dropdown. In the second part, I will demonstrate how to add a progress bar using Tkinter. Let’s get started.

Check out the youtube video tutorial for this article

Install Tkinter

The Tkinter module is a built-in module, therefore you don’t need to install it. You can just import the module in your python file and use it.

Create a Tkinter Window

First I will create an empty window. Here is the code to create a window of size 200 by 200:

from tkinter import Tk

class App(Tk):
    def __init__(self):
        super().__init__()  # Create the window
        self.geometry('200x200') #Resize

app = App()
app.mainloop()

What is this code exactly doing? I have created a class that inherit from Tk. The Tk class contains all the graphical functionality to create a window. In the constructor method, __init__ I will call the constructor of the super class. That will create the window.

Then I have resized the window to 200 by 200 using the .geometry() method.

You can paste this code into IDLE or PyCharm, run it, and you will get your window created. Let’s move on to the next step, adding a dropdown.

Add Tkinter DropDown

Now I m gonna show you how to add a dropdown like the one below. The dropdown allows you to select the gender, Male or Female. You can change these options to anything, and add as many options as you like.

Here is the code to add the dropdown widget. I have placed the new piece of code between ##, just to mark the new piece of code added.

from tkinter import Tk, OptionMenu,StringVar

class App(Tk):
    def __init__(self):
        super().__init__()
        self.geometry('200x200')

        #NEW: Adding the dropdown ##
        self.gender_selected = StringVar()
        self.gender_selected.set("Select Gender")
        self.dropdown = OptionMenu(self, self.gender_selected, *["Male", "Female"])
        self.dropdown.grid(row=0,column=1)
        ###########################

app = App()
app.mainloop()

Code Explanation

What is this code doing? First I have created a variable that will hold the value selected in the dropdown, called gender_selected, see lines 9 and 10. If you select a different dropdown value, it will be saved in this variable.

Next, I have created the dropdown widget using the OptionMenu class. Lines 11 and 12. Please note that you need to import this class to use it, otherwise you will get a syntax error when trying to run the program. The OptionMenu class takes three parameters: the window that will contain the widget, the variable that holds the widget value, and a list with the dropdown option.

Next I have specified where in the window I want to place the dropdown, using the .grid() method. This is an important step, if I forget to add this, the widget won’t be displayed.

Find out how to trigger an action when the dropdown selected value changes

Add tkinter progess bar to your app

I hope you enjoy the tutorial and thank you so much for reading! Happy Coding! :)

Python Errors: Nameerror name is not defined and more

hellocodeclub — Sun, 18 Oct 2020 08:07:23 +0000

Errors are inevitable when you are programming. As you write code, errors will start raising. The better you understand these errors, the easier it will be to avoid them. In this article you will learn the main python errors, how to interpret them and how they arise. For example, python nameerror name is not defined, what does this mean? You will find out by the end of this tutorial.

he goal of an error, or exception, is flagging something unexpected happened while running the code. Some of these situation arise frequently. Therefore python contains some built-in exceptions that capture the more frequent unexpected situation. Below we will go through each of those exception types and see what’s the meaning behind.

SyntaxError: invalid syntax

This error occurs when the code you write doesn’t follow the python syntax rule. For example, not closing a parenthesis will lead to a syntax error. The python parser won’t be able to parse the code if it doesn’t follow the syntax rule, therefore it can’t process it any further. Let’s see some examples:

Example #1

list = [1, 23, 45, 0, 9]
for item in list
    print(item)

Output:

  File line 2
    for item in list
                   ^
SyntaxError: invalid syntax

This code raised an unexpected situation, because line 2 is missing the colon at the end, which breaks the python syntax rules.

Example #2

list = [1, 23, 45, 0, 9]
for item in list:
    print(item

Output:

  File line 4

                 ^
SyntaxError: unexpected EOF while parsing

The code above raised an error because line 3 is missing the closing parenthesis.

Python Nameerror name is not defined

You will encounter a nameerror ( name is not defined) when a variable is not defined in the local or global scope. Or you used a function that wasn’t defined anywhere in your program. For example, you will see this error if you try to print a variable that wasn’t defined. You might also see this error when you use a built-in library, but forget to import the library first. Let’s see a few code examples:

Example #1

number = 1
print(num)

Output:

Traceback (most recent call last):
  File line 4, in <module>
    print(num)
NameError: name 'num' is not defined

Usually this error is highlighting that there is a typo in one of the variable names.

Example #2

def print_age(age):
    print('My age is: '+str(age))

print__age(14)

Output:

Traceback (most recent call last):
  File line 4, in <module>
    print__age(14)
NameError: name 'print__age' is not defined

This issue is similar to the previous example, but applied to function. Although there is a “print age” function, the function name is print, underscore and age, however when I called the function I used double underscore __. That’s why the code can’t find the function

Find out How to Fix – no module named tkinter error

How to Fix Valueerror: too many values to unpack

Fix Typeerror a bytes-like object is required not ‘str’

I hope you enjoyed the article and thanks for reading! Happy Coding! 😃

Runnable vs Callable

hellocodeclub — Sat, 03 Oct 2020 08:37:48 +0000

In this article you will learn what is a runnable , what is a callable and the difference between the two in java, runnable vs callable. These concepts are important when you are dealing with concurrency.

Concurrency basically means there is not just one execution thread in your program, but several executions or threads, potentially happening simultaneously. Therefore different pieces of the code might be executed at the same time.
Let's get started!

Runnable

The Runnable interface in java enables you to define a block of code that can be executed within its own thread. Therefore if you create several runnable classes and execute them, and you will have several threads running at the same time within the same process or program. This is known as concurrency.

See examples of how to use a runnable interface

Callable

Next is callable. Callable is also a java interface and as Runnable, you can use it to run tasks in parallel. However there is a key difference. The callable can return the result of the task or throw an exception. The Callable interface is included in Java to address some of runnable limitations.

See examples of how to use a runnable interface

Runnable vs Callable - The difference

The main difference between Runnable and Callable is that Callable will return the result of executing the task to the caller. If you use Runnable you can't return anything, any result will need to be saved in separated shared structure or database. To understand this difference runnable vs callable, you can compare the two interfaces.

public interface Runnable {
    public abstract void run();
}

public interface Callable<V> {
    V call() throws Exception;
}

How to execute a runnable
How to execute a callable

I hope you enjoy the article and thank you so much for reading and supporting this blog! 🙂

How to Check Django Version – Ultimate Guide 2020

hellocodeclub — Fri, 21 Aug 2020 08:27:26 +0000

There are different ways you can check your django version. In this article I will share a couple of them. I will also share how you can upgrade and downgrade the django version. How you can uninstall django, something that you might do if you are running into problems and decide to uninstall it and install Django again.

Finding the django version can be useful if you want to make sure your python and django versions are compatible. I have also included a section to show how to find the python version that you are using.

Find your django version can also be useful to make sure a feature you are using, is supported that the django version installed

How can I check Django version?

The way you check your Django version depends on if you installed Django on you base python installation or you installed the module in a virtual environment. Let’s see first how to check your version on the base python installation.

Checking Django version with Idle

To achieve this, open the idle terminal and type the following:

import django
django.get_version()

Output(you should get something similar):

'3.0.8'

You could also use the following code, that will return the same results, but slightly different format:

import django
django.VERSION

Output:

(3, 0, 8, 'final', 0

Checking Django version with pip

You can also check the version through pip using. Run the following command on your terminal:

pip3 freeze

Output:

Django==3.0.8  # Django version
docutils==0.14
dominate==2.5.1
Flask==1.1.1
Flask-Bootstrap==3.3.7.1
importlib-metadata==1.6.0

If django is in your python installation, it will be listed along with any other module installed.

Check out your django version in a virtualenv or using the django scripts

Upgrade Django version

You can upgrade django and any module installed using the pip command. You can run the following command from your terminal, or the Pycharm terminal. As before, if you are working on a virtual environment, make sure you activate the environment before running the command:

pip install --upgrade django

This will upgrade to the latest available version

Downgrade Django version

In case you would like to downgrade, you can do this via pip as well, using the following command:

pip install -v Django==3.0.5 #Replace with your version

You can find a list with all the available django version here

As before in case you are working on a virtualenv, make sure it is activated. Or if you have several python version installed in your computer, make sure your pip command you are using belongs to the python installation you are using to run your python program. You can check this by running the linux which command from your terminal.

which pip

The output should be same folder or subfolder as where your python version is installed.

Check out the following article to see how to install and uninstall python, and see your local python version