DEV Community: anujd64

Access Logs Count

anujd64 — Wed, 13 Mar 2024 04:50:52 +0000

This practical is very similar to WordCount

Only changes are in input filetype and the code.
This time we will copy access_log_short.csv into hdfs and rest everything is identical.

Code and input file for this Practical can be found here:
Code & Input

Word Count Hadoop

anujd64 — Wed, 13 Mar 2024 04:11:19 +0000

Video version of this article: https://www.youtube.com/watch?v=wTkffAYsCBw
Credits: @UnboxingBigData

Code
Open Eclipse and create a new Java project.

Right click on project and click on Build Path > select Configure Build Path...

Click on libraries tab and click on Add External JARs...

Navigate to hadoop common folder in my case it is home/hadoop-3.3.6/share/hadoop/common and select all the JAR files.

Again click on Add External JARs...

Navigate to hadoop mapreduce folder in my case it is home/hadoop-3.3.6/share/hadoop/mapreduce and select all the JAR files.

Now the libraries tab will look like this:

Click apply and close.

Now create the three classes, code for the classes is given at the end of the article. I am just copy pasting the classes in the src folder of our Eclipse project.

Now we will be exporting JAR for out project, right click on the project and select export.

Select the export destination:

Select the main class for the project:

You will find the JAR at the selected location:

Create an input.txt file like so:

Now just execute the following commands one by one

cd Desktop
start-all.sh
hadoop fs -mkdir /wc_input
hadoop fs -put input.txt /wc_input
hadoop jar WordCount.jar /wc_input/input.txt /wc_output/
hadoop fs -cat /wc_output/part-00000
stop-all.sh

What do these commands do ?

-mkdir command creates the folder wc_input in the hdfs(hadoop's distributed file system)
-put will copy the input.txt file to the folder we just created.
jar command uses the code in the JAR file exported to perform map reduce on the input.txt file and saves the output in wc_output directory.
-cat command just prints out the contents of the output produced by the mapreduce operation.
to delete a directory in hdfs use

hadoop fs -rm -r <DIRECTORY_PATH>

Your terminal should look like this:

Code for the classes:

WC_Runner.java

import java.io.IOException;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.FileInputFormat;
import org.apache.hadoop.mapred.FileOutputFormat;
import org.apache.hadoop.mapred.JobClient;
import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.mapred.TextInputFormat;
import org.apache.hadoop.mapred.TextOutputFormat;
public class WC_Runner {
    public static void main(String[] args) throws IOException{
        JobConf conf = new JobConf(WC_Runner.class);
        conf.setJobName("WordCount");
        conf.setOutputKeyClass(Text.class);
        conf.setOutputValueClass(IntWritable.class);
        conf.setMapperClass(WC_Mapper.class);
        conf.setCombinerClass(WC_Reducer.class);
        conf.setReducerClass(WC_Reducer.class);
        conf.setInputFormat(TextInputFormat.class);
        conf.setOutputFormat(TextOutputFormat.class);
        FileInputFormat.setInputPaths(conf,new Path(args[0]));
        FileOutputFormat.setOutputPath(conf,new Path(args[1]));
        JobClient.runJob(conf);
    }
}

WC_Reducer.java

import java.io.IOException;
import java.util.Iterator;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.MapReduceBase;
import org.apache.hadoop.mapred.OutputCollector;
import org.apache.hadoop.mapred.Reducer;
import org.apache.hadoop.mapred.Reporter;

public class WC_Reducer  extends MapReduceBase implements Reducer<Text,IntWritable,Text,IntWritable> {
    public void reduce(Text key, Iterator<IntWritable> values,OutputCollector<Text,IntWritable> output,
                       Reporter reporter) throws IOException {
        int sum=0;
        while (values.hasNext()) {
            sum+=values.next().get();
        }
        output.collect(key,new IntWritable(sum));
    }
}

WC_Mapper.java

import java.io.IOException;
import java.util.StringTokenizer;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.MapReduceBase;
import org.apache.hadoop.mapred.Mapper;
import org.apache.hadoop.mapred.OutputCollector;
import org.apache.hadoop.mapred.Reporter;
public class WC_Mapper extends MapReduceBase implements Mapper<LongWritable,Text,Text,IntWritable>{
    private final static IntWritable one = new IntWritable(1);
    private Text word = new Text();
    public void map(LongWritable key, Text value,OutputCollector<Text,IntWritable> output,
                    Reporter reporter) throws IOException{
        String line = value.toString();
        StringTokenizer  tokenizer = new StringTokenizer(line);
        while (tokenizer.hasMoreTokens()){
            word.set(tokenizer.nextToken());
            output.collect(word, one);
        }
    }

}

Hadoop configuration

anujd64 — Sun, 03 Mar 2024 08:42:45 +0000

Video version of this article: https://www.youtube.com/watch?v=Slbi-uzPtnw
Credits:@codewitharjun

(Video uses cli text editor to edit config files, this tutorial will use normal text editor.)

First install java-jdk-8

sudo apt install openjdk-8-jdk

(Optional) To check it’s there

cd /usr/lib/jvm

Now ensure that you are at the root of the terminal if not run
cd ~

Open .bashrc file

sudo gedit .bashrc

And paste in the following block

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 
export PATH=$PATH:/usr/lib/jvm/java-8-openjdk-amd64/bin 
export HADOOP_HOME=~/hadoop-3.3.6/ 
export PATH=$PATH:$HADOOP_HOME/bin 
export PATH=$PATH:$HADOOP_HOME/sbin 
export HADOOP_MAPRED_HOME=$HADOOP_HOME 
export YARN_HOME=$HADOOP_HOME 
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop 
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native 
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native" 
export HADOOP_STREAMING=$HADOOP_HOME/share/hadoop/tools/lib/hadoop-streaming-3.3.6.jar
export HADOOP_LOG_DIR=$HADOOP_HOME/logs 
export PDSH_RCMD_TYPE=ssh

sudo apt-get install ssh

For following commands check your hadoop version number at the time of writing this it is 3.3.6

Now go to hadoop.apache.org website download the tar file.
Direct Link
Website Link

Once downloaded execute:
(To extract the tar file)

tar -zxvf ~/Downloads/hadoop-3.3.6.tar.gz

For all the configuration below ensure you are in hadoop-3.3.6/etc/hadoop directory.

cd hadoop-3.3.6/etc/hadoop

Many of the files might have <configuration> tag already so watch before you paste in new configurations.

Now open hadoop-env.h:

sudo gedit hadoop-env.h

Paste the following in hadoop-env.h:

JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

(set the path for JAVA_HOME)
You might not need to use sudo in the following commands but to avoid permission issues I have added it to everything.
Let's configure other files similarly:

sudo gedit core-site.xml

<configuration> 
 <property> 
 <name>fs.defaultFS</name> 
 <value>hdfs://localhost:9000</value>  </property> 
 <property> 
<name>hadoop.proxyuser.dataflair.groups</name> <value>*</value> 
 </property> 
 <property> 
<name>hadoop.proxyuser.dataflair.hosts</name> <value>*</value> 
 </property> 
 <property> 
<name>hadoop.proxyuser.server.hosts</name> <value>*</value> 
 </property> 
 <property> 
<name>hadoop.proxyuser.server.groups</name> <value>*</value> 
 </property> 
</configuration>

sudo gedit hdfs-site.xml

Change <User> with your ubuntu username !

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href=''?>
<configuration>
<property>
  <name>dfs.name.dir</name>
  <value>file:///home/<USER>/pseudo/dfs/name</value>
</property>
<property>
  <name>dfs.data.dir</name>
  <value>file:///home/<USER>/pseudo/dfs/data</value>
</property>
<property>
    <name>dfs.replication</name>
    <value>1</value>
</property>
</configuration>

sudo gedit mapred-site.xml

<configuration> 
 <property> 
 <name>mapreduce.framework.name</name>  <value>yarn</value> 
 </property> 
 <property>
 <name>mapreduce.application.classpath</name> 
<value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value> 
 </property> 
</configuration>

sudo gedit yarn-site.xml

<configuration> 
 <property> 
 <name>yarn.nodemanager.aux-services</name> 
 <value>mapreduce_shuffle</value> 
 </property> 
 <property> 
 <name>yarn.nodemanager.env-whitelist</name> 
<value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREP END_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value> 
 </property> 
</configuration>

Hadoop is now configured.

Next execute following one by one:

ssh
ssh localhost 
ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa 
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys 
chmod 0600 ~/.ssh/authorized_keys 
hadoop-3.3.6/bin/hdfs namenode -format
export PDSH_RCMD_TYPE=ssh

To start hadoop
start-all.sh

To check if hadoop is running go to http://localhost:9870/

To stop hadoop
stop-all.sh

This is an update version to this article