DEV Community: Seamoon Pandey

Apache Hadoop on Ubuntu 22.04 (Beginner-Friendly, Java 8 Without Breaking Java 21)

Seamoon Pandey — Sat, 31 Jan 2026 14:43:23 +0000

Hadoop is still a core concept in data engineering — especially for understanding HDFS, YARN, and distributed systems.

The biggest blocker for beginners today is Java version conflicts.
Most modern systems run Java 17 or 21, while Hadoop prefers Java 8.

This guide shows you how to:

Install Hadoop from scratch
Use Java 8 only for Hadoop
Keep your system Java untouched
Avoid auto-start, battery drain, and common traps

What you’ll build

Hadoop 3.3.6
Single-node (pseudo-distributed)
Ubuntu 22.04
Java 8 for Hadoop only
Manual start/stop (laptop-friendly)

Prerequisites

Ubuntu 22.04
Internet connection
sudo access
No prior Hadoop knowledge required

Step 1: System preparation

sudo apt update
sudo apt install -y ssh rsync curl

Hadoop relies on SSH — even on a single machine.

Step 2: Create a dedicated Hadoop user (recommended)

sudo adduser hadoop
sudo usermod -aG sudo hadoop
su - hadoop

From here on, everything runs as the hadoop user.

Step 3: Enable passwordless SSH

Hadoop services communicate over SSH.

ssh-keygen -t rsa -P ""
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
ssh localhost

If it logs in without asking for a password → you’re good.

Step 4: Java setup (important)

Hadoop works best with Java 8.
You do NOT need to uninstall Java 17/21.

Verify Java 8 exists:

ls /usr/lib/jvm/java-8-openjdk-amd64

If it exists, continue.
(You can install it with sudo apt install openjdk-8-jdk if missing.)

Step 5: Download Hadoop

cd ~
wget https://downloads.apache.org/hadoop/common/hadoop-3.3.6/hadoop-3.3.6.tar.gz
tar -xvzf hadoop-3.3.6.tar.gz
mv hadoop-3.3.6 hadoop

Step 6: Hadoop environment variables

Edit .bashrc:

nano ~/.bashrc

Add at the bottom:

export HADOOP_HOME=$HOME/hadoop
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

Apply:

source ~/.bashrc

Step 7: Bind Hadoop to Java 8 (critical step)

This avoids breaking your other Java projects.

nano ~/hadoop/etc/hadoop/hadoop-env.sh

Set:

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

Verify:

hadoop version

You should see:

Java version: 1.8.0

Step 8: Hadoop configuration

Move into config directory:

cd ~/hadoop/etc/hadoop

core-site.xml

<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://localhost:9000</value>
  </property>
</configuration>

hdfs-site.xml

<configuration>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
  <property>
    <name>dfs.name.dir</name>
    <value>file:/home/hadoop/hadoopdata/namenode</value>
  </property>
  <property>
    <name>dfs.data.dir</name>
    <value>file:/home/hadoop/hadoopdata/datanode</value>
  </property>
</configuration>

Create directories:

mkdir -p ~/hadoopdata/namenode ~/hadoopdata/datanode

mapred-site.xml

cp mapred-site.xml.template mapred-site.xml

<configuration>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
</configuration>

yarn-site.xml

<configuration>
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
</configuration>

Step 9: Format the NameNode (run ONCE)

hdfs namenode -format

⚠️ Running this again wipes HDFS metadata.

Step 10: Start Hadoop

start-dfs.sh
start-yarn.sh

Verify:

jps

Expected processes:

NameNode
DataNode
SecondaryNameNode
ResourceManager
NodeManager

Step 11: Access Web UIs

HDFS: http://localhost:9870
YARN: http://localhost:8088

If these load → Hadoop is running correctly.

Step 12: Test HDFS

hdfs dfs -mkdir /test
hdfs dfs -put ~/.bashrc /test
hdfs dfs -ls /test

Expected output:

Found 1 items
-rw-r--r-- 1 hadoop supergroup ...

About the “native-hadoop” warning

You may see:

WARN NativeCodeLoader: Unable to load native-hadoop library

This is normal.

Hadoop falls back to pure Java
No functionality is lost
Safe to ignore for learning & development

Does Hadoop auto-start on boot?

No.

Hadoop:

Does NOT register as a system service
Does NOT start after reboot
Uses zero resources unless manually started

To stop:

stop-dfs.sh
stop-yarn.sh

Viola you're done.

Transferring SSH Keys (`id_rsa` and `id_rsa.pub`) to a New System

Seamoon Pandey — Mon, 28 Oct 2024 08:51:54 +0000

SSH keys are essential for secure communication between computers. If you're migrating to a new system but want to retain your existing SSH keys (id_rsa and id_rsa.pub), follow these steps to seamlessly transfer them.

Step 1: Locate Your SSH Keys

First, identify the location of your SSH keys on your old system. Typically, they reside in the ~/.ssh/ directory.

cd ~/.ssh/
ls -al

Step 2: Backup Your Keys

Ensure the safety of your SSH keys by creating a backup. Copy them to a secure location, such as an encrypted USB drive or a trusted cloud storage service.

cp id_rsa id_rsa.pub /path/to/backup/location

Step 3: Transfer the Keys to the New System

Use the scp (secure copy) command to transfer the SSH keys to your new system.

scp /path/to/backup/location/id_rsa /path/to/backup/location/id_rsa.pub user@new_system_ip:~/.ssh/

Replace user with your username on the new system and new_system_ip with the IP address or hostname of your new system.

Step 4: Set Permissions

Ensure that the permissions of your id_rsa file are secure. Only the owner should have read and write permissions.

chmod 600 ~/.ssh/id_rsa

Step 5: Add Public Key to Authorized Keys

Append the contents of your id_rsa.pub file to the authorized_keys file on your new system. This allows you to authenticate using your SSH private key.

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

Step 6: Test the Connection

Finally, test the SSH connection to your new system to ensure everything is set up correctly.

ssh user@new_system_ip

If successful, you should log in to your new system without being prompted for a password.

Congratulations! You've successfully transferred your SSH keys to your new system, ensuring secure access to your remote servers.

Feel free to customize the instructions further based on your specific environment or requirements!

Reset password mailer implementation in rails 7 api, devise_token_auth, and sendgrid-ruby.

Seamoon Pandey — Mon, 28 Oct 2024 08:45:37 +0000

In this guide, we’ll walk through setting up password reset functionality in a Rails app using devise_token_auth for authentication and SendGrid for email delivery. With this, users can request password reset links sent directly to their emails.

Dependencies

Add devise_token_auth and sendgrid-ruby to your Gemfile:

# Sendgrid for sending emails
gem 'sendgrid-ruby'

# Devise for authentication
gem 'devise_token_auth'

Then install the dependencies:

bundle install

Or add each gem individually:

bundle add 'sendgrid-ruby'
bundle add 'devise_token_auth'

Setup

To generate Devise token auth files, run:

rails g devise_token_auth:install User auth

Now, open the generated migration file ...devise_token_auth_create_users.rb, and uncomment these lines:

t.string   :confirmation_token
t.datetime :confirmed_at
t.datetime :confirmation_sent_at
t.string   :unconfirmed_email

Run the migration:

bin/rails db:migrate

In your user.rb model file, add:

extend Devise::Models
include DeviseTokenAuth::Concerns::User

# Add :recoverable for password recovery
devise :database_authenticatable, :registerable, :recoverable

In config/initializers/devise_token_auth.rb, configure password reset URL:

config.change_headers_on_each_request = false
config.default_password_reset_url = "#{ENV['FRONTEND_URL']}/reset-password"

Routes Configuration

In config/routes.rb, add:

mount_devise_token_auth_for "User", at: "auth", controllers: {
  passwords: "user/passwords"
}

Controller Setup

Create controllers/user/passwords_controller.rb for handling password reset actions:

# frozen_string_literal: true
class User::PasswordsController < DeviseTokenAuth::PasswordsController
  # POST /auth/password
  def create
    email = resource_params[:email]
    if email.blank?
      return render json: { success: false, errors: ["Email cannot be blank"] }, status: :unprocessable_entity
    end

    resource_class.send_reset_password_instructions(email: email)
    render json: { success: true, message: "If the email exists, password reset instructions have been sent." }, status: :ok
  end

  # PUT /auth/password
  def update
    if password_update_params[:reset_password_token].blank?
      authenticate_user!

      unless current_user.valid_password?(password_update_params[:old_password])
        return render json: { success: false, errors: ["Old password is incorrect"] }, status: :unprocessable_entity
      end

      if current_user.update(password: password_update_params[:password])
        render json: { success: true, message: "Password updated successfully for authenticated user" }
      else
        render json: { success: false, errors: current_user.errors.full_messages }, status: :unprocessable_entity
      end

    else
      resource = resource_class.reset_password_by_token(password_update_params)
      if resource.nil?
        return render json: { success: true, message: "If the email exists, password reset instructions have been sent." }
      end

      if resource.errors.present?
        return render json: { success: false, errors: resource.errors.full_messages }, status: :unprocessable_entity
      end

      resource.update(password: password_update_params[:password])
      render json: { success: true, message: "Password updated successfully" }
    end
  rescue StandardError => e
    render json: { success: false, errors: ["An unexpected error occurred: #{e.message}"] }, status: :internal_server_error
  end

  private

  def password_update_params
    params.permit(:reset_password_token, :password, :password_confirmation, :old_password)
  end

  def resource_params
    params.permit(:email)
  end
end

Now, users can initiate a password reset by sending a POST request to /auth/password.

Customizing the Email Template

To modify the reset email, update views/devise/mailer/reset_password_instructions.html.erb:

<p>Hello <%= @resource.email %>!</p>
<p>Someone requested a password reset. You can reset it via the link below.</p>
<% reset_url = DeviseTokenAuth.default_password_reset_url + "?reset_password_token=#{@token}" %>
<p><%= link_to 'Change my password', reset_url %></p>
<p>If you didn’t request this, please ignore this email.</p>
<p>Your password won’t change until you create a new one.</p>

Environment Setup

Sign in to SendGrid, set up your API key, and verify your sender email.
Use dotenv or Rails credentials to securely store the following in your .env file:

SENDGRID_API_KEY=SG.******************
SENDER_EMAIL=your_verified_email@domain.com

Configure SendGrid in your environment files (config/environments/development.rb and config/environments/production.rb):

config.action_mailer.perform_deliveries = true
config.action_mailer.smtp_settings = {
  address: "smtp.sendgrid.net",
  port: 587,
  authentication: :plain,
  user_name: "apikey", # Keep as "apikey"
  password: ENV["SENDGRID_API_KEY"],
  domain: ENV["BASE_URL"],
  enable_starttls_auto: true
}

Wrapping Up

And that’s it! Now, users can reset their passwords by requesting a link through your Rails API with Devise Token Auth and SendGrid.

References

Swapping Numbers Without a Temporary Variable in C using XOR like a pro.

Seamoon Pandey — Thu, 22 Feb 2024 02:48:34 +0000

Swapping the values of two variables without using a temporary variable is a classic programming problem. One elegant solution to this problem in C involves using bitwise XOR operations.

Algorithm

Consider two variables a and b. The goal is to swap their values.

Initialize a and b with the values to be swapped.
Perform the following steps:
```
a ^= b;
b ^= a;
a ^= b;
```

Explanation

Let's break down the algorithm step by step:

a ^= b;: XOR (^=) a with b and store the result back in a. After this operation, a contains the result of a XOR b.
b ^= a;: XOR b with the new value of a (which was the original value of b). After this operation, b contains the result of b XOR (a XOR b), which simplifies to a.
a ^= b;: XOR a with the new value of b (which was the original value of a). After this operation, a contains the result of (a XOR b) XOR a, which simplifies to b.

Now, a holds the original value of b and b holds the original value of a, effectively swapping their values without using a temporary variable.

Example

Consider the following example:

int a = 5, b = 7;
a ^= b;
b ^= a;
a ^= b;

Ruby on Rails Api authentication with rails cookie session.

Seamoon Pandey — Sun, 10 Dec 2023 03:55:45 +0000

Backend (Rails API)

Step 1: Create a New Rails API Project

To start building your Rails API, open your terminal and run:

rails new your_api_project_name --api -d postgresql
cd your_api_project_name

Explanation: This command initializes a new Rails API project, enabling API-only functionality and using PostgreSQL as the database.

Step 2: Integrate Devise and Devise Token Auth

Add Devise and Devise Token Auth to your Gemfile:

gem 'devise_token_auth'

Then, execute the following commands:

bundle install
rails generate devise:install
rails generate devise_token_auth:install
rails db:migrate

Explanation: These commands install and configure Devise along with Devise Token Auth. They set up authentication for your User model and create the necessary files and database tables.

Step 3: Configure ApplicationController

In app/controllers/application_controller.rb, add the following lines:

class ApplicationController < ActionController::API
  include ActionController::Cookies
  include DeviseTokenAuth::Concerns::SetUserByToken
end

Explanation: This configures your ApplicationController to include modules for handling cookies and setting the user based on the authentication token.

Step 4: Configure Application

In config/application.rb, add the following lines:

config.api_only = true
config.session_store :cookie_store, key: '_your_api_session', httponly: true, same_site: :strict, secure: true
config.middleware.use ActionDispatch::Cookies
config.middleware.use config.session_store, config.session_options

Explanation: These configurations ensure that your API works in an API-only mode, sets up secure cookie storage, and adds necessary middleware.

Step 5: Configure Devise Token Auth

In config/initializers/devise_token_auth.rb, add the following:

config.cookie_enabled = true
config.cookie_attributes = {
  httponly: true,
  encrypt: true,
  same_site: :strict,
  secure: true
}

In config/routes.rb, mount Devise Token Auth for the 'User' model:

mount_devise_token_auth_for 'User', at: 'auth'

Explanation: These configurations enable token authentication and set up cookie attributes for added security.

Frontend (Using Axios)

Step 6: Set Up Axios for Authenticated Requests

Install Axios using:

yarn add axios

Now, when making authenticated requests, use Axios with withCredentials: true:

axios.get('your/api/endpoint', { withCredentials: true })
  .then(response => {
    // Handle the response
  })
  .catch(error => {
    // Handle the error
  });

Explanation: Axios is a popular HTTP client for the browser and Node.js. The withCredentials: true option ensures that cookies are sent with requests, crucial for authentication.

Step 7: Middleware for Authenticated Requests

Consider creating a middleware for handling authentication in your frontend. Middleware can centralize logic for authenticated requests.

Step 8: Manage Authenticated State with a Cookie

Use a logged_in cookie to manage the authenticated state on the client side. Set it to true on login and false on logout or session expiration.

Explanation: This helps keep track of the user's authentication status on the frontend, providing a seamless user experience.

This guide covers the setup of a Rails API with token-based authentication using Devise and Devise Token Auth, along with frontend integration using Axios. Feel free to customize these steps based on your project's specific needs. If you have any questions or need further clarification, Mate, do some more research!