DEV Community

Aditya Pidurkar
Aditya Pidurkar

Posted on

Installing Apache Hadoop 3.4.2 on Linux 2026 (Single Node Cluster) Complete Step-by-Step Guide

#beginners #java #linux #tutorial

S1. Update the System

Before installing Hadoop, update your system packages.

sudo apt update
sudo apt upgrade -y
Enter fullscreen mode Exit fullscreen mode

S2. Install and Configure Java 21

Apache Hadoop is written in Java and requires a compatible Java Runtime Environment (JRE) and Java Development Kit (JDK).

This guide uses OpenJDK 21 (LTS).

Java Compatibility

Installed Java Version Action
Java 21 Continue with the installation
Java 17 May work with some Hadoop versions, but this guide uses Java 21
Java 22–25 Install Java 21 and switch to it
Java Not Installed Install OpenJDK 21

S2.1 Check if Java is Installed 

java -version
javac -version
Enter fullscreen mode Exit fullscreen mode

If Java is NOT Installed

Install OpenJDK 21.

sudo apt update
sudo apt install openjdk-21-jdk -y
Enter fullscreen mode Exit fullscreen mode

Verify the installation.

java -version
javac -version
Enter fullscreen mode Exit fullscreen mode

If Java is Already Installed

Check the installed version.

java -version
javac -version
Enter fullscreen mode Exit fullscreen mode

Compare the output with the compatibility table above.

If Java 21 is already active, continue to the next step.

Otherwise, switch to Java 21.

S2.2 List All Installed Java Versions 

update-java-alternatives -l
Enter fullscreen mode Exit fullscreen mode

Example:

java-1.17.0-openjdk-amd64
java-1.21.0-openjdk-amd64
Enter fullscreen mode Exit fullscreen mode

S2.3 Switch to Java 21 

sudo update-alternatives --config java
sudo update-alternatives --config javac
Enter fullscreen mode Exit fullscreen mode

Select the option corresponding to Java 21.

S2.4 Verify the Active Java Version 

java -version
javac -version
Enter fullscreen mode Exit fullscreen mode

Both commands should display Java 21.

S2.5 Find the Correct JAVA_HOME

Different Linux distributions may install Java in different directories.

Instead of copying someone else's path, determine it automatically.

readlink -f $(which java)
Enter fullscreen mode Exit fullscreen mode

Example:

/usr/lib/jvm/java-21-openjdk-amd64/bin/java
Enter fullscreen mode Exit fullscreen mode

Therefore,

JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64
Enter fullscreen mode Exit fullscreen mode

Note: Never blindly copy the JAVA_HOME path from a tutorial. Always determine it using the command above.

S3. Download Apache Hadoop 3.4.2

Move to the directory where you want to download Hadoop.

cd ~/Downloads
Enter fullscreen mode Exit fullscreen mode

Download Hadoop from the official Apache Archive.

wget https://archive.apache.org/dist/hadoop/common/hadoop-3.4.2/hadoop-3.4.2.tar.gz
Enter fullscreen mode Exit fullscreen mode

Verify the download.

ls -lh hadoop-3.4.2.tar.gz
Enter fullscreen mode Exit fullscreen mode

Expected output:

-rw-r--r-- 1 user user 734M hadoop-3.4.2.tar.gz
Enter fullscreen mode Exit fullscreen mode

S4. Extract and Install Hadoop

After downloading the Hadoop archive, the next step is to extract it and install Hadoop on your system.

In this guide, Hadoop will be installed under /opt/hadoop, which is the standard location for optional third-party software on Linux.

S4.1 Extract the Archive

Move to the directory where the archive was downloaded.

cd ~/Downloads
Enter fullscreen mode Exit fullscreen mode

Extract the archive.

tar -xvzf hadoop-3.4.2.tar.gz
Enter fullscreen mode Exit fullscreen mode

This will create a directory named:

hadoop-3.4.2
Enter fullscreen mode Exit fullscreen mode

Verify the extraction.

ls
Enter fullscreen mode Exit fullscreen mode

Expected output:

hadoop-3.4.2
hadoop-3.4.2.tar.gz
Enter fullscreen mode Exit fullscreen mode

S4.2 Move Hadoop to /opt

Move the extracted directory to /opt.

sudo mv hadoop-3.4.2 /opt/hadoop
Enter fullscreen mode Exit fullscreen mode

S4.3 Verify the Installation Directory

Check that Hadoop has been moved successfully.

ls /opt
Enter fullscreen mode Exit fullscreen mode

Expected output:

hadoop
Enter fullscreen mode Exit fullscreen mode

View the contents of the Hadoop directory.

ls /opt/hadoop
Enter fullscreen mode Exit fullscreen mode

Expected output:

bin
etc
include
lib
libexec
licenses
sbin
share
NOTICE.txt
README.txt
...
Enter fullscreen mode Exit fullscreen mode

S4.4 Change Ownership

By default, the Hadoop directory is owned by the root user because it was moved using sudo.

Change the ownership so your user can modify the configuration files without requiring root privileges every time.

sudo chown -R $USER:$USER /opt/hadoop
Enter fullscreen mode Exit fullscreen mode

Verify the ownership.

ls -ld /opt/hadoop
Enter fullscreen mode Exit fullscreen mode

Example output:

drwxr-xr-x <username> <username> ...
Enter fullscreen mode Exit fullscreen mode

S4.5 Verify the Hadoop Version

Run the Hadoop executable directly.

/opt/hadoop/bin/hadoop version
Enter fullscreen mode Exit fullscreen mode

Expected output:

Hadoop 3.4.2
Source code repository ...
Compiled by ...
Enter fullscreen mode Exit fullscreen mode

At this point, Hadoop has been successfully installed on the system. However, it cannot yet be executed from any directory because its binaries have not been added to the system's PATH.

Note: We are intentionally using the full path (/opt/hadoop/bin/hadoop) for now. In the next section, we will configure environment variables so that the hadoop, hdfs, and yarn commands can be run from anywhere in the terminal.

S5. Configure Environment Variables

After installing Hadoop, we need to configure environment variables. These variables tell the operating system where Hadoop and Java are installed, allowing Hadoop commands to be executed from any directory.

The main environment variables are:

Variable Purpose
JAVA_HOME Points to the Java installation directory
HADOOP_HOME Points to the Hadoop installation directory
HADOOP_CONF_DIR Specifies the Hadoop configuration directory
PATH Allows Hadoop commands to be executed from anywhere

S5.1 Check Your Default Shell

Before editing your shell configuration, determine which shell you are using.

echo $SHELL
Enter fullscreen mode Exit fullscreen mode

Possible outputs:

/bin/bash
Enter fullscreen mode Exit fullscreen mode

or

/bin/zsh
Enter fullscreen mode Exit fullscreen mode

S5.2 Open the Shell Configuration File

If you are using Bash 

nano ~/.bashrc
Enter fullscreen mode Exit fullscreen mode

If you are using Zsh 

nano ~/.zshrc
Enter fullscreen mode Exit fullscreen mode

S5.3 Add the Hadoop Environment Variables

Append the following lines to the end of the file.

# Java
export JAVA_HOME=$(dirname $(dirname $(readlink -f $(which java))))

# Hadoop
export HADOOP_HOME=/opt/hadoop
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

# Hadoop Components
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME

# Native Libraries
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native

# PATH
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
Enter fullscreen mode Exit fullscreen mode

Why use $(dirname $(dirname $(readlink -f $(which java))))?

Instead of hardcoding the Java installation path, this command automatically determines the correct JAVA_HOME directory. This makes the guide more portable across different Linux distributions and Java installation methods.

S5.4 Save the File

For Nano:

  • Press Ctrl + O
  • Press Enter
  • Press Ctrl + X

S5.5 Reload the Configuration

Bash 

source ~/.bashrc
Enter fullscreen mode Exit fullscreen mode

Zsh 

source ~/.zshrc
Enter fullscreen mode Exit fullscreen mode

S5.6 Verify the Environment Variables

Check that Hadoop is configured correctly.

echo $JAVA_HOME
Enter fullscreen mode Exit fullscreen mode

Example:

/usr/lib/jvm/java-21-openjdk-amd64
Enter fullscreen mode Exit fullscreen mode

Check the Hadoop installation directory.

echo $HADOOP_HOME
Enter fullscreen mode Exit fullscreen mode

Expected output:

/opt/hadoop
Enter fullscreen mode Exit fullscreen mode

Check the configuration directory.

echo $HADOOP_CONF_DIR
Enter fullscreen mode Exit fullscreen mode

Expected output:

/opt/hadoop/etc/hadoop
Enter fullscreen mode Exit fullscreen mode

S5.7 Verify Hadoop Commands

You should now be able to run Hadoop commands from any directory.

Check the Hadoop version.

hadoop version
Enter fullscreen mode Exit fullscreen mode

Check the HDFS version.

hdfs version
Enter fullscreen mode Exit fullscreen mode

Check the YARN version.

yarn version
Enter fullscreen mode Exit fullscreen mode

Each command should report Hadoop 3.4.2.

Note: If any of these commands return command not found, ensure that the shell configuration file has been reloaded using the source command or open a new terminal session.

S6. Configure Passwordless SSH

Apache Hadoop uses SSH to start and stop its daemons. Even in a single-node cluster, passwordless SSH is required for Hadoop scripts such as start-dfs.sh and start-yarn.sh.

S6.1 Check if OpenSSH is Installed

Run the following command:

ssh -V
Enter fullscreen mode Exit fullscreen mode

If OpenSSH is installed, you will see output similar to:

OpenSSH_10.x
Enter fullscreen mode Exit fullscreen mode

If the command is not found, install the OpenSSH server.

sudo apt update
sudo apt install openssh-server -y
Enter fullscreen mode Exit fullscreen mode

S6.2 Start and Enable the SSH Service

Start the SSH service.

sudo systemctl start ssh
Enter fullscreen mode Exit fullscreen mode

Enable it to start automatically on boot.

sudo systemctl enable ssh
Enter fullscreen mode Exit fullscreen mode

Check the service status.

sudo systemctl status ssh
Enter fullscreen mode Exit fullscreen mode

The status should show active (running).

Exit the status screen by pressing Q.

S6.3 Generate an SSH Key Pair

If you do not already have an SSH key, generate one.

ssh-keygen -t rsa -b 4096
Enter fullscreen mode Exit fullscreen mode

When prompted for the file location, simply press Enter to accept the default location.

When prompted for a passphrase, press Enter twice to leave it empty.

Example:

Enter file in which to save the key:
/home/<username>/.ssh/id_rsa
Enter fullscreen mode Exit fullscreen mode

S6.4 Enable Passwordless SSH

Append the public key to the authorized keys file.

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
Enter fullscreen mode Exit fullscreen mode

Set the correct permissions.

chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
Enter fullscreen mode Exit fullscreen mode

S6.5 Test the SSH Connection

Run:

ssh localhost
Enter fullscreen mode Exit fullscreen mode

The first time you connect, you may see:

Are you sure you want to continue connecting (yes/no)?
Enter fullscreen mode Exit fullscreen mode

Type:

yes
Enter fullscreen mode Exit fullscreen mode

If everything is configured correctly, you will be logged into your own machine without entering a password.

Exit the SSH session.

exit
Enter fullscreen mode Exit fullscreen mode

Why is Passwordless SSH Required?

Hadoop's management scripts use SSH to launch and stop services such as the NameNode, DataNode, ResourceManager, and NodeManager. Without passwordless SSH, these scripts will repeatedly prompt for a password, making automated startup impossible.

S7. Create Hadoop Data Directories

Hadoop requires a few directories to store its metadata and temporary files. These directories are referenced by the Hadoop configuration files.

Create the required directories by running the following commands:

mkdir -p ~/hadoopdata/namenode
mkdir -p ~/hadoopdata/datanode
mkdir -p ~/hadooptmp
Enter fullscreen mode Exit fullscreen mode

You can verify that the directories were created successfully using:

tree -L 2 ~
Enter fullscreen mode Exit fullscreen mode

If the tree command is not installed, install it using:

sudo apt install tree -y
Enter fullscreen mode Exit fullscreen mode

Expected output:

/home/<username>
├── hadoopdata
│   ├── datanode
│   └── namenode
└── hadooptmp
Enter fullscreen mode Exit fullscreen mode

Note: The directory names must match the paths specified in the Hadoop configuration files (core-site.xml and hdfs-site.xml). If you changed those paths, create the directories accordingly.

S8. Install the Hadoop Configuration Files

Instead of manually editing multiple Hadoop configuration files, use the installation script provided in the GitHub repository. The script downloads and installs all the required configuration files automatically.

Download the installation script.

wget https://raw.githubusercontent.com/itsadityapidurkar/hadoop-installation-guide/main/linux/install-configs.sh
Enter fullscreen mode Exit fullscreen mode

Make the script executable.

chmod +x install-configs.sh
Enter fullscreen mode Exit fullscreen mode

Run the script.

./install-configs.sh
Enter fullscreen mode Exit fullscreen mode

The script downloads and installs the following configuration files:

  • core-site.xml
  • hdfs-site.xml
  • mapred-site.xml
  • yarn-site.xml
  • workers

These files are automatically copied to:

/opt/hadoop/etc/hadoop
Enter fullscreen mode Exit fullscreen mode

If the script completes successfully, you should see:

========================================
 Apache Hadoop Configuration Installer
========================================

[+] Downloading configuration files...
[✓] Download complete.

[+] Installing configuration files...

[✓] Configuration files installed successfully!

Installed to:
/opt/hadoop/etc/hadoop
Enter fullscreen mode Exit fullscreen mode

Note: The provided configuration files are intended for a Single Node Hadoop 3.4.2 installation. If you modified the directory paths while following this guide, update the configuration files accordingly before installing them.

S9. Configure hadoop-env.sh

The hadoop-env.sh file is used to configure environment variables required by Hadoop. The only change required for this guide is to set the JAVA_HOME variable.

Open the file using a text editor.

nano /opt/hadoop/etc/hadoop/hadoop-env.sh
Enter fullscreen mode Exit fullscreen mode

Search for the line containing:

export JAVA_HOME=
Enter fullscreen mode Exit fullscreen mode

or

# export JAVA_HOME=
Enter fullscreen mode Exit fullscreen mode

Replace it with:

export JAVA_HOME=$(dirname $(dirname $(readlink -f $(which java))))
Enter fullscreen mode Exit fullscreen mode

Save the file and exit.

For Nano:

  • Press Ctrl + O
  • Press Enter to save the file.
  • Press Ctrl + X to exit.

You can verify that the JAVA_HOME path is detected correctly by running:

echo $(dirname $(dirname $(readlink -f $(which java))))
Enter fullscreen mode Exit fullscreen mode

Example output:

/usr/lib/jvm/java-21-openjdk-amd64
Enter fullscreen mode Exit fullscreen mode

Note: This command automatically detects the Java installation directory, making the configuration portable across different Linux distributions and Java installation paths.

S10. Format the NameNode

Before starting Hadoop for the first time, the NameNode must be formatted. This initializes the Hadoop Distributed File System (HDFS) and creates the required metadata.

Run the following command:

hdfs namenode -format
Enter fullscreen mode Exit fullscreen mode

During the formatting process, you may be prompted with the following message if the NameNode directory already contains data:

Re-format filesystem in Storage Directory root=... ? (Y or N)
Enter fullscreen mode Exit fullscreen mode

Type:

Y
Enter fullscreen mode Exit fullscreen mode

and press Enter.

If the formatting is successful, the last few lines of the output should look similar to:

...
INFO common.Storage: Storage directory ... has been successfully formatted.
INFO namenode.FSImageFormatProtobuf: Saving image file ...
INFO namenode.NameNode: SHUTDOWN_MSG:
Enter fullscreen mode Exit fullscreen mode

Important: Formatting the NameNode permanently removes all existing HDFS metadata and data stored in the configured NameNode and DataNode directories. This command should only be executed during the initial setup or when you intentionally want to reset the Hadoop cluster. Do not run this command every time you start Hadoop.

S11. Start HDFS

Now that the NameNode has been formatted, you can start the Hadoop Distributed File System (HDFS).

Run the following command:

start-dfs.sh
Enter fullscreen mode Exit fullscreen mode

If HDFS starts successfully, you should see output similar to:

Starting namenodes on [localhost]
Starting datanodes
Starting secondary namenodes [<hostname>]
Enter fullscreen mode Exit fullscreen mode

To verify that all HDFS daemons are running, execute:

jps
Enter fullscreen mode Exit fullscreen mode

Expected output:

NameNode
DataNode
SecondaryNameNode
Jps
Enter fullscreen mode Exit fullscreen mode

Note: The order of the processes may vary depending on your system.

You can also verify that the NameNode is running by opening the following URL in your web browser:

http://localhost:9870
Enter fullscreen mode Exit fullscreen mode

If HDFS has started successfully, the NameNode web interface will be displayed, showing information about the cluster, storage usage, live DataNodes, and the HDFS filesystem.

S12. Start YARN

After starting HDFS, the next step is to start YARN (Yet Another Resource Negotiator), which is responsible for resource management and job scheduling in the Hadoop cluster.

Run the following command:

start-yarn.sh
Enter fullscreen mode Exit fullscreen mode

If YARN starts successfully, you should see output similar to:

Starting resourcemanager
Starting nodemanagers
Enter fullscreen mode Exit fullscreen mode

To verify that the YARN daemons are running, execute:

jps
Enter fullscreen mode Exit fullscreen mode

Expected output:

NameNode
DataNode
SecondaryNameNode
ResourceManager
NodeManager
Jps
Enter fullscreen mode Exit fullscreen mode

Note: The order of the processes may vary depending on your system.

You can also verify that YARN is running by opening the following URL in your web browser:

http://localhost:8088
Enter fullscreen mode Exit fullscreen mode

If YARN has started successfully, the ResourceManager web interface will open, displaying information about the cluster, available resources, running applications, and node status.

At this point, your Hadoop Single Node Cluster is up and running.

S13. Verify the Hadoop Installation

After starting both HDFS and YARN, verify that all Hadoop services are running correctly.

Check the running Hadoop daemons.

jps
Enter fullscreen mode Exit fullscreen mode

Expected output:

NameNode
DataNode
SecondaryNameNode
ResourceManager
NodeManager
Jps
Enter fullscreen mode Exit fullscreen mode

View the HDFS cluster report.

hdfs dfsadmin -report
Enter fullscreen mode Exit fullscreen mode

If the cluster is running correctly, the report will display information about the configured capacity, remaining storage, live DataNodes, and other cluster details.

Next, open the NameNode Web UI in your browser.

http://localhost:9870
Enter fullscreen mode Exit fullscreen mode

The NameNode dashboard should display information such as:

  • Cluster Summary
  • Live DataNodes
  • Storage Information
  • HDFS Overview

Then open the ResourceManager Web UI.

http://localhost:8088
Enter fullscreen mode Exit fullscreen mode

The ResourceManager dashboard should display:

  • Cluster Metrics
  • Node Status
  • Running Applications
  • Available Resources

If all the above commands and web interfaces are accessible, your Hadoop Single Node Cluster has been installed and configured successfully.

Hadoop Web UI Port Quick-Reference

Service Web UI URL Default Port Description
HDFS NameNode http://localhost:9870 9870 File system status, capacity, browser directories
YARN ResourceManager http://localhost:8088 8088 Job executions, node resource allocation
HDFS DataNode http://localhost:9864 9864 Individual DataNode status & blocks

S14. Shut Down the Hadoop Cluster

To stop the Hadoop daemons and release system resources safely, stop the YARN resource manager and HDFS services in order.

Run the following commands:

stop-yarn.sh
stop-dfs.sh
Enter fullscreen mode Exit fullscreen mode

To verify that all Hadoop services have been stopped successfully, run:

jps
Enter fullscreen mode Exit fullscreen mode

Expected output:

Jps
Enter fullscreen mode Exit fullscreen mode

Only the Jps process itself should be running, confirming that HDFS and YARN have successfully shut down.

Top comments (0)