DEV Community

Big Data Today
Big Data Today

Posted on

Install Hadoop on WSL2 Ubuntu (2026): Complete Step-by-Step Guide

#hadoop #bigdata #wsl #ubuntu

If you've searched "install Hadoop on WSL," you've probably found guides from 2020–2023, most written for WSL1 or old Hadoop 3.2/3.3.0 builds. This guide is different: it's WSL2-specific, uses the current Hadoop 3.5.x line, and includes the exact errors WSL throws that generic Linux guides don't warn you about — SSH not starting, slow HDFS I/O on the Windows filesystem, and JAVA_HOME issues.

What you need before starting

  • Windows 10 (build 19041+) or Windows 11
  • WSL2 enabled with an Ubuntu distro installed
  • About 20-30 minutes and ~3GB free disk space

Check your WSL version first:

wsl -l -v
Enter fullscreen mode Exit fullscreen mode

If Ubuntu doesn't show VERSION 2, upgrade it:

wsl --set-version Ubuntu 2
Enter fullscreen mode Exit fullscreen mode

Step 1: Update Ubuntu and install Java

Open your WSL Ubuntu terminal and run:

sudo apt update && sudo apt upgrade -y
sudo apt install openjdk-17-jdk -y
Enter fullscreen mode Exit fullscreen mode

Note: Hadoop 3.5.0 requires Java 17 on the server side (this is new — earlier 3.4.x releases used Java 11). If you have an older Java installed from a previous setup, you'll need to update it before continuing.

Verify it installed correctly:

java -version
Enter fullscreen mode Exit fullscreen mode

Find your Java install path — you'll need this later:

readlink -f $(which java)
Enter fullscreen mode Exit fullscreen mode

This usually points to something like /usr/lib/jvm/java-17-openjdk-amd64/bin/java — the JAVA_HOME path is everything before /bin/java.

Step 2: Set up passwordless SSH

This is the step most guides gloss over, and it's the #1 place WSL setups fail. Hadoop's start scripts use SSH to talk to "localhost" even for a single-node setup, but WSL doesn't run an SSH server by default.

sudo apt install openssh-server -y
Enter fullscreen mode Exit fullscreen mode

Generate a key and authorize it for your own account:

ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 0600 ~/.ssh/authorized_keys
Enter fullscreen mode Exit fullscreen mode

Start the SSH service manually (WSL doesn't auto-start services like a normal Linux boot would):

sudo service ssh start
Enter fullscreen mode Exit fullscreen mode

Test it:

ssh localhost
Enter fullscreen mode Exit fullscreen mode

If it connects without asking for a password (after the first "yes" to trust the host), you're good. Important WSL quirk: this SSH service does not persist across WSL restarts — every time you reopen your WSL terminal after a reboot, you'll need to run sudo service ssh start again before starting Hadoop. (Later in this guide I'll show how to add this to your .bashrc so it starts automatically.)

Step 3: Download and extract Hadoop 

cd ~
wget https://downloads.apache.org/hadoop/common/hadoop-3.5.0/hadoop-3.5.0.tar.gz
tar -xzf hadoop-3.5.0.tar.gz
mv hadoop-3.5.0 hadoop
Enter fullscreen mode Exit fullscreen mode

Performance tip specific to WSL: keep this in your Linux home directory (~/hadoop), not under /mnt/c/.... Reading/writing across the Windows-Linux filesystem boundary is significantly slower — this alone causes a lot of "Hadoop feels really slow" complaints that are actually just a filesystem choice.

Step 4: Set environment variables

Open ~/.bashrc:

nano ~/.bashrc
Enter fullscreen mode Exit fullscreen mode

Add these lines at the bottom (adjust JAVA_HOME to match what you found in Step 1):

export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64
export HADOOP_HOME=$HOME/hadoop
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
# Auto-start SSH for Hadoop on WSL
sudo service ssh start > /dev/null 2>&1
Enter fullscreen mode Exit fullscreen mode

Reload it:

source ~/.bashrc
Enter fullscreen mode Exit fullscreen mode

Step 5: Configure Hadoop's XML files

All files below are in $HADOOP_HOME/etc/hadoop/.

hadoop-env.sh — find the line starting with export JAVA_HOME= and set it explicitly:

export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64
Enter fullscreen mode Exit fullscreen mode

core-site.xml:

<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://localhost:9000</value>
  </property>
</configuration>
Enter fullscreen mode Exit fullscreen mode

hdfs-site.xml — first create data directories:

mkdir -p ~/hadoop/hdfs/namenode
mkdir -p ~/hadoop/hdfs/datanode
Enter fullscreen mode Exit fullscreen mode

Then:

<configuration>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
  <property>
    <name>dfs.namenode.name.dir</name>
    <value>file:///home/YOUR_USERNAME/hadoop/hdfs/namenode</value>
  </property>
  <property>
    <name>dfs.datanode.data.dir</name>
    <value>file:///home/YOUR_USERNAME/hadoop/hdfs/datanode</value>
  </property>
</configuration>
Enter fullscreen mode Exit fullscreen mode

(Replace YOUR_USERNAME with your actual WSL username — check with whoami.)

mapred-site.xml:

<configuration>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
</configuration>
Enter fullscreen mode Exit fullscreen mode

yarn-site.xml:

<configuration>
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
</configuration>
Enter fullscreen mode Exit fullscreen mode

Step 6: Format the NameNode 

hdfs namenode -format
Enter fullscreen mode Exit fullscreen mode

Do this only once, on first setup. Running it again on an existing cluster wipes your HDFS data.

Step 7: Start Hadoop 

start-dfs.sh
start-yarn.sh
Enter fullscreen mode Exit fullscreen mode

Verify everything is running:

jps
Enter fullscreen mode Exit fullscreen mode

You should see: NameNode, DataNode, SecondaryNameNode, ResourceManager, NodeManager. If you see all five, your setup worked.

Step 8: Check the web UI

Because WSL2 handles localhost forwarding automatically, you can open these directly in your Windows browser:

  • NameNode UI: http://localhost:9870
  • ResourceManager UI: http://localhost:8088

Common errors and fixes

ssh: connect to host localhost port 22: Connection refused
SSH isn't running. Run sudo service ssh start. Remember this doesn't persist across WSL restarts unless you added it to .bashrc in Step 4.

JAVA_HOME is not set and could not be found
Your hadoop-env.sh JAVA_HOME path is wrong or empty. Re-check with readlink -f $(which java) and update both hadoop-env.sh and .bashrc.

NameNode won't start after a reboot / stuck in safe mode
This usually means a previous shutdown wasn't clean. Try hdfs dfsadmin -safemode leave. If that doesn't help, check $HADOOP_HOME/logs/*namenode*.log for the actual error — don't reformat unless you're okay losing your test data.

Permission denied errors on HDFS directories
Make sure the directories in hdfs-site.xml are owned by your user, not root: sudo chown -R $USER:$USER ~/hadoop/hdfs.

Everything feels slow
Check whether your Hadoop install or data directories are under /mnt/c/. Move them into your Linux home directory as described in Step 3.

Uninstalling Hadoop from WSL (if you need to start fresh)

If you already have an older Hadoop version installed and want to remove it cleanly before installing 3.5.0, here's the full teardown:

1. Stop all running daemons first:

stop-dfs.sh
stop-yarn.sh
Enter fullscreen mode Exit fullscreen mode

Confirm nothing's left running:

jps
Enter fullscreen mode Exit fullscreen mode

You should only see Jps itself in the list. If anything else is still listed, kill it manually:

kill -9 <PID>
Enter fullscreen mode Exit fullscreen mode

2. Delete the Hadoop installation and any leftover data:

rm -rf ~/hadoop
rm -f ~/hadoop-*.tar.gz
Enter fullscreen mode Exit fullscreen mode

This removes the install directory along with your namenode/datanode data folders, since they live inside it.

3. Clean up ~/.bashrc:

nano ~/.bashrc
Enter fullscreen mode Exit fullscreen mode

Delete the JAVA_HOME, HADOOP_HOME, HADOOP_MAPRED_HOME, HADOOP_COMMON_HOME, HADOOP_HDFS_HOME, YARN_HOME, and PATH lines you added earlier (leave the sudo service ssh start line if you want SSH to keep auto-starting for future use — otherwise remove that too). Save, then reload:

source ~/.bashrc
Enter fullscreen mode Exit fullscreen mode

4. (Optional) Remove SSH setup, if you don't need it for anything else:

rm -f ~/.ssh/id_rsa ~/.ssh/id_rsa.pub
nano ~/.ssh/authorized_keys   # remove the corresponding key entry
sudo apt remove --purge openssh-server -y
Enter fullscreen mode Exit fullscreen mode

5. (Optional) Remove Java, if nothing else on your system depends on it:

sudo apt remove --purge openjdk-11-jdk -y
sudo apt autoremove -y
Enter fullscreen mode Exit fullscreen mode

(Skip this if you use Java for anything else — check with java -version first to see what's actually installed before removing it.)

6. Verify it's fully gone:

which hadoop
echo $HADOOP_HOME
Enter fullscreen mode Exit fullscreen mode

Both should return nothing. Open a fresh terminal to be sure your .bashrc changes have taken effect, then you're clear to follow the install steps above from scratch with 3.5.0.

Top comments (0)