DEV Community: Seamoon Pandey

Bloom Filters, Explained Simply

Seamoon Pandey — Tue, 11 Aug 2026 08:36:09 +0000

TL;DR

A row of bits, all zero. To store something, hash it a few times and flip the slots those hashes point at. To look something up, hash it again and check the same slots.

Any slot still 0 means it's definitely not there. All slots 1 means probably there, so go check the real database.

It can only be wrong in one direction: false alarms, never misses. Costs about 10 bits per item for a 1% error rate, so 10 million records fit in roughly 12 MB. It can't list what's in it, can't count, and can't delete.

Its whole job is killing 99% of expensive lookups before they happen.

If that already makes sense, you're done. The rest is the same thing slowly, with a worked example.

Start with the annoying problem

You're building a signup page. Someone types a username, and you have to tell them whether it's taken.

Easy. Ask the database.

The trouble is that you're asking it constantly. Every keystroke in that box turns into a network call and a disk lookup, and you've got a few million users and a lot of visitors. So you're burning real resources to discover, over and over, that "xkq_dragon_9981" is available.

And it usually is available. That's the part worth noticing. Almost every name someone tries is free, which means almost every one of those expensive lookups was pointless before you even made it.

What you want is something small and fast sitting in front of the database, catching all the obvious misses so they never reach it.

That's a Bloom filter.

The idea, in one sentence

It tells you "definitely not in the set" or "probably in the set." Very fast. Almost no memory. And it never stores your actual data.

Everything below is just how that works and what it costs you.

Setting one up

You need a row of slots, each holding a 0 or a 1. All zeros to start.

A real one might have millions of slots. We'll use 16, because 16 fits on a screen.

index:  0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15
value:  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0

You also need hash functions. We'll use three.

If "hash function" sounds intimidating, it shouldn't. For our purposes it's just a rule that takes any input and spits out a number. It needs two properties and no others:

Same input, same output, always. Feed it "alex" today and next year and you get the same number both times.
Different inputs land in different places, spread reasonably evenly across 0 to 15.

That's it. That's the whole job description.

Adding a name

To add "alex", run it through all three hash functions.

h1("alex") = 2
h2("alex") = 7
h3("alex") = 13

Flip those three slots to 1.

index:  0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15
value:  0  0  1  0  0  0  0  1  0  0  0  0  0  1  0  0

Now add "priya". Her hashes give 4, 7, 11.

Slot 7 is already 1. You just leave it. Nothing anywhere records that two different names are now leaning on that same slot, and that detail is going to matter quite a lot later.

index:  0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15
value:  0  0  1  0  1  0  0  1  0  0  0  1  0  1  0  0

One more. "sam" gives 0, 9, 13.

index:  0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15
value:  1  0  1  0  1  0  0  1  0  1  0  1  0  1  0  0

Three usernames stored.

Look at what isn't in that row, though. The strings "alex", "priya" and "sam" appear nowhere. The filter didn't keep them. It kept the marks they left on the way through, and that's a genuinely different thing.

Looking something up

Same procedure. Hash the input three times, check those three slots.

When some slot is 0

Someone types "maria". Hashes give 1, 6, 12.

Slot 1 is 0.

"maria" is not taken. Skip the database entirely.

And that's a certainty, not a guess. Think about why. If "maria" had ever been added, slot 1 would have been flipped to 1 right then. It's still 0. So she was never added. There's no room in that argument for the filter to be wrong.

When every slot is 1, and the name really is taken

Someone types "alex". Slots 2, 7, 13, all 1.

Probably taken. You go query the database, it confirms, you show the "sorry, try another" message. The filter did its job and routed the check where it belonged.

When every slot is 1, and the name is actually free

Someone types "jordan". Hashes give 0, 4, 13.

All three are 1.

Except nobody ever added "jordan". Slot 0 came from "sam". Slot 4 from "priya". Slot 13 from "alex". Three unrelated names happened to cover exactly the three slots that "jordan" checks, and the filter has no way to tell the difference.

So it says "probably taken," you query the database, the database says no such user, and you tell the person the name is free.

You made one unnecessary lookup. That's it. That's the whole price of being wrong.

The rule everything hangs on

"No" is always right. "Yes" only means "maybe."

A Bloom filter can produce a false positive. It cannot produce a false negative. It will never tell you something is absent when it's actually sitting right there.

That lopsidedness isn't a flaw nobody got around to fixing. It's the trade. You accept a handful of wasted lookups, and you get back a structure small enough to live in memory and fast enough to check in nanoseconds.

One thing follows from this that people miss, so it's worth saying flatly: a Bloom filter is only useful when something trustworthy sits behind it. It screens. It doesn't decide. Every "maybe" has to go somewhere that knows the real answer. Used on its own, with nothing to verify against, it's just a machine that's confidently wrong every so often.

Why anyone accepts occasional wrong answers

The numbers.

Take 10 million usernames. Store them properly at roughly 20 bytes apiece and you're at 200 MB, and that's before you build any kind of index on top to search them quickly.

Put them in a Bloom filter at 10 bits per entry and you get about a 1% false positive rate using 12.5 MB.

So 12.5 MB of memory kills 99% of your database traffic. That's not a tuning improvement. It's a different kind of answer to the question.

The two dials

Slots. More slots, fewer accidental overlaps, fewer false positives, more memory. Nothing surprising.

Hash functions. This one is not "more is better," and it catches people out.

Use only one and a single collision produces a false positive. Too fragile.

Use fifty and every insert flips fifty slots. The array fills with 1s almost instantly, and once nearly everything is a 1, everything looks like a maybe. A saturated filter says "probably yes" to every question you ask it, which makes it precisely as useful as having no filter at all.

The sweet spot is roughly "flip enough bits that the array ends up about half full." In numbers:

Bits per item	Hash functions	False positive rate
5	3	~9%
8	6	~2%
10	7	~0.8%
16	11	~0.05%

In practice you won't work this out by hand. You tell the library how many items you expect and what error rate you can live with, and it picks both dials for you. But it's worth knowing the shape of that table, if only so you can tell whether what you're asking for is reasonable before you ask.

What it can't do

Worth being blunt here, because people reach for Bloom filters and then get irritated that they aren't hash tables.

It can't tell you what's in it. You can't ask which usernames it knows about. It holds bits. There is nothing to read back out.

It can't count. Slot 7: flipped once, or ten thousand times? No idea. The filter has no opinion.

It can't delete. Say you want to remove "alex" and you go reset slots 2, 7 and 13. You've just broken "priya", who needs slot 7, and "sam", who needs slot 13. Now the filter starts saying "definitely not present" about names that are present, and the moment "no" stops being trustworthy the entire structure is worthless. If you need removal, look at counting Bloom filters (a small counter in each slot instead of a single bit) or cuckoo filters (a newer alternative that handles deletion and is often smaller anyway). Both cost you something.

It goes bad quietly. Size a filter for 10 million entries, feed it 100 million, and it saturates. Every answer becomes "maybe." It won't throw an error. It won't log a warning. It just stops being useful while continuing to look like it's working fine, which is honestly the failure mode to actually watch for.

Where these show up

Databases skipping disk reads. By a wide margin the most common real use. Cassandra, HBase, RocksDB and friends spread data across many files on disk, and checking every file for a key is slow. So each file carries a Bloom filter of its own keys. "Definitely not in here" means skip the file completely, no disk read at all. Structurally it's the exact same problem as the signup form: cheap check first, expensive check only if you have to.

Browsers checking for dangerous URLs. Shipping the full list of known malicious sites to every browser isn't practical. Ship a compact filter instead. Anything that comes back "definitely safe" resolves locally in an instant, and only the rare maybe goes to a server. Nice side effect: most of your browsing history never leaves your machine.

Caches ignoring one hit wonders. A big chunk of CDN requests are for things nobody will ever ask for twice, and caching those is wasted space. So keep a filter of what you've seen once, and only bother caching something on the second request.

Blockchain clients. Every Ethereum block header carries a 2048 bit Bloom filter summarising the log addresses and topics inside that block. A client hunting for events from one particular contract checks the header filter and skips the entire block on a "no," instead of parsing every transaction since genesis.

So, when should you reach for one?

Three conditions, and you want all three.

You're doing a lot of lookups against a big set. Most of those lookups come back negative. And there's something slow behind the check, a disk, a network call, another service, that you'd like to stop bothering.

Hit all three and a Bloom filter is close to free money. Miss one and it's probably not worth the complexity. If your set is small enough to sit in a hash table, use the hash table. If most lookups are hits anyway, the filter is just extra work in front of a query you were going to make regardless.

The mental model that sticks: it's a bouncer, not a judge. Turns away the obvious no's instantly, waves everything else through to someone who actually knows.

Burton Bloom published this in 1970, in a paper about hyphenation in typesetting software. It now holds up databases, browsers and blockchains. Simple ideas tend to outlast clever ones.

My laptop told me to load the kernel first, so I did (a missing-kernel horror story on EndeavourOS)

Seamoon Pandey — Mon, 22 Jun 2026 09:24:26 +0000

So I rebooted my machine like a normal person and got slapped with a blue GRUB screen:

error: file '/boot/vmlinuz-linux-lts' not found.
you need to load the kernel first

Cool. Very helpful. Load the kernel first. With what, GRUB? My good looks?

I dived deep into the issue just to find this was a pretty common issue caused due to some update cancellations in between update sessions breaking the packages(most probably a patience issue) as well as spent almost an extra of about half an hour session with my claude for more specific information on the issue or rather solution.

If you're reading this because you got the same message, good news: your system is almost certainly fine and you do not need to reinstall anything. I went through the full panic arc so you don't have to, and here's roughly how it went.

First, the dumb mistake everyone makes

I knew the rough shape of the fix from somewhere in the back of my brain: boot a live USB, chroot in, fix it. So naturally, sitting at the grub> prompt, I confidently typed:

grub> mount /dev/sdXn /mnt
error: can't find command `mount'.
grub> arch-chroot /mnt
error: can't find command `arch-chroot'.

GRUB just stared at me. Turns out the GRUB shell is not Linux. It has its own tiny set of commands, and mount and arch-chroot are not among them. Those are things you run later, from an actual live environment. Lesson one: the blue prompt is not a terminal, stop treating it like one.

What the error actually means

GRUB's whole job is to grab a kernel and its initramfs, load them into memory, and hand off. The error shows up when GRUB has a menu entry pointing at a kernel file, in my case /boot/vmlinuz-linux-lts, but that file simply isn't there. Nothing to load means nothing to boot. Hence the passive-aggressive "you need to load the kernel first."

So the real mystery is: where did my kernel go?

Poking around from the GRUB shell

Before doing anything drastic you can actually snoop on your disks from grub>. List partitions with:

ls

I got back something like (hd0,gpt5) (hd0,gpt4) (hd0,gpt3) (hd0,gpt2) (hd0,gpt1). Then you just go fishing through them:

ls (hd0,gpt5)/

The one that spits out boot/ etc/ home/ usr/ var/ ... is your Linux root. Mine was gpt5. Then the moment of truth:

ls (hd0,gpt5)/boot/

And /boot contained... amd-ucode.img and grub/. That's it. No vmlinuz-linux. No initramfs-linux.img. My kernel had ghosted me. (Also, if you see an empty boot/efi folder here, relax, that's just an unmounted mountpoint, not the problem.)

At this point I genuinely considered just wiping the whole thing and starting fresh with some calmer distro. Do not do this. I'll get to why.

Making the rescue USB without a second computer

I only have the one laptop, which felt like a dealbreaker until I remembered it dual-boots Windows, and the Windows install was sitting there completely untouched on its own partition. So I booted into Windows straight from the BIOS boot menu (mash F12 / F2 / Esc at power-on depending on your machine), downloaded the EndeavourOS ISO, and flashed it to a USB stick with Rufus.

If you run EndeavourOS, grab the EndeavourOS ISO. Vanilla Arch, grab the Arch ISO. Matching your actual distro saves you keyring headaches later.

The live USB also tried to ruin my day

Booted the USB, picked the friendly default entry ("open source drivers, all GPUs"), and watched the screen blink at me forever. Black, blink, black, blink, nothing. This is apparently a beloved tradition on laptops with NVIDIA graphics: the normal graphical boot just refuses to come up.

The fix is the entry nobody reads: fallback / nomodeset. Reboot the USB, pick that, and it boots ugly but functional. That got me to a desktop. If you're stuck blinking, that's your move.

Finally, the actual repair

Opened a terminal and ran the one command that changes nothing and tells you everything:

lsblk -f

This shows your partitions and their filesystems. I was looking for two things: my root partition (ext4, helpfully labeled endeavouros) and my EFI partition (small, FAT32). Mine were nvme0n1p5 and nvme0n1p1. Yours might be sda5 / sda1 on an older SATA drive. Use your real names, not mine.

Then mount everything and jump in:

sudo mount /dev/nvme0n1p5 /mnt
sudo mkdir -p /mnt/boot/efi
sudo mount /dev/nvme0n1p1 /mnt/boot/efi
sudo arch-chroot /mnt

When the prompt flips to something like [root@EndeavourOS /]#, you're now living inside your installed system instead of the USB. This is where the real commands work.

The plot twist

I figured I'd just reinstall the kernel and go to bed. But first I checked what was actually installed:

pacman -Q linux linux-lts mkinitcpio

And got this beautiful nonsense:

linux 7.0.12.arch1-1
linux-lts 6.18.36-1
error: package 'mkinitcpio' was not found

There it was. According to pacman, both kernels were perfectly installed. But mkinitcpio was gone. And mkinitcpio is the thing that actually builds your initramfs and gets the kernel images written into /boot during install. No mkinitcpio means the kernel package "installs" but never produces the files GRUB needs. The database lies to your face while /boot sits there empty.

How does mkinitcpio go missing? Usually an upgrade that got interrupted, a /boot that filled up mid-write, or a cleanup / package removal that took more than you meant it to. I genuinely don't remember which crime I committed, but the result was the same.

The fix that finally worked

Install the missing piece, then reinstall the kernels so the hooks actually fire this time:

pacman -S mkinitcpio
pacman -S linux linux-lts amd-ucode

Say yes to the reinstall. (I'm on AMD, hence amd-ucode. Intel folks want intel-ucode. Don't want the LTS kernel? Drop linux-lts and the dead menu entry disappears on its own.)

If pacman starts whining about keyrings or signatures, refresh them first with pacman -Sy archlinux-keyring (add endeavouros-keyring on EndeavourOS) and try again.

Then check that the files actually showed up this time:

ls /boot

And finally, the good stuff:

vmlinuz-linux                     vmlinuz-linux-lts
initramfs-linux.img               initramfs-linux-lts.img
initramfs-linux-fallback.img      initramfs-linux-lts-fallback.img
amd-ucode.img                     grub/

A populated /boot. Chef's kiss. The kernels exist again.

Wrapping it up

Tell GRUB about its newly recovered kernels, then get out:

grub-mkconfig -o /boot/grub/grub.cfg
exit
sudo umount -R /mnt
reboot

Pull the USB on the way down. It booted straight into my normal GRUB menu and into a working desktop like nothing had happened.

While you're in there, two quick checks

Run df -h /boot and make sure it isn't basically full, because a stuffed /boot is a great way to cause this exact mess again. And if you're paranoid, mkinitcpio -P rebuilds every initramfs by hand, though the kernel reinstall already does it for you.

Stuff I'd tell past me

The grub> prompt is not bash. Stop typing Linux commands into it.
Don't yank the power during a pacman -Syu. Let kernel upgrades finish, especially on a laptop.
Keep an eye on /boot free space.
Read what pacman is about to remove before hitting yes. If it wants to take linux or mkinitcpio with it, that's your sign to stop.
Keep a flashed live USB in a drawer. It turns "my laptop is bricked" into "ugh, fifteen minutes."

And please don't just reinstall

I almost did. It feels clean when you're frustrated. But reinstalling throws away all your config, takes way longer than the repair, and worst of all, fiddling with partitions next to a Windows install is a fantastic way to accidentally delete Windows too. The repair touches exactly two things: the kernel files and the GRUB config. The reinstall touches your entire disk. You need the same USB stick either way, so just try the repair first.

My system, my dotfiles, my Windows install, all completely fine the whole time. I was just missing a couple of files and one very important package. That's it.

I Forgot My Linux Password — And Reset It in 5 Minutes

Seamoon Pandey — Tue, 09 Jun 2026 21:55:31 +0000

Most Linux users assume their login password is the primary barrier protecting their system. Then one day they forget it.

The surprising part? On many Linux distributions, resetting a forgotten password is remarkably easy if you have physical access to the machine.

This isn't a vulnerability. It's a fundamental aspect of how local system administration works on Linux.

Let's explore why password recovery works, how it's done, and what it teaches us about Linux security.

The Situation

Imagine you've been away from your laptop for a few weeks.

You boot into your favorite Linux distribution, confidently type what you think is your password, and get:

Authentication failure

You try again.

Still wrong.

A few more attempts later, you realize you've forgotten your password.

Many people panic at this point, assuming they need to reinstall the operating system. Fortunately, that's rarely necessary.

Why Password Recovery Is Possible

Linux separates several security concepts:

User account passwords
Root (administrator) privileges
Physical access to the machine
Disk encryption

If someone can boot the operating system and obtain a root shell, they can change any user's password.

The key insight is that Linux trusts administrators to manage user accounts.

When you boot into recovery mode, you're effectively entering a maintenance environment where administrative actions are allowed.

Recovery Mode on Ubuntu-Based Distributions

On distributions such as Ubuntu, Lubuntu, Kubuntu, Xubuntu, and Linux Mint, the process is straightforward.

Step 1: Access GRUB

During boot:

Hold Shift on BIOS systems
Press Esc repeatedly on UEFI systems

This should display the GRUB menu.

Step 2: Enter Recovery Mode

Navigate to:

Advanced options for Ubuntu

Then select:

(recovery mode)

Step 3: Open a Root Shell

Choose:

root - Drop to root shell prompt

You now have administrative access.

Step 4: Remount the Filesystem

The root filesystem is usually mounted read-only.

Remount it as writable:

mount -o remount,rw /

Step 5: Reset the Password

First, identify the username if necessary:

ls /home

Then change the password:

passwd username

Example:

passwd john

Enter a new password twice.

Step 6: Reboot

reboot

You're done.

What About Other Linux Distributions?

The same principle applies across most Linux systems.

Distribution	Typical Recovery Method
Ubuntu Family	Recovery Mode
Debian	Single User Mode
Fedora	Emergency Mode
Arch Linux	Recovery Shell or Live USB
EndeavourOS	Same as Arch
openSUSE	Rescue Mode

The interface differs, but the idea remains identical:

Obtain root access
Run passwd
Set a new password

The Security Lesson

Many new Linux users assume:

"If someone doesn't know my password, they can't access my computer."

That's only partially true.

If an attacker has:

Physical access
The ability to boot the system
An unencrypted drive

Then they can often reset passwords without knowing the original one.

This is expected behavior, not a bug.

Full-Disk Encryption Changes Everything

Now consider a system protected by LUKS full-disk encryption.

Before Linux even starts booting, you'll see something similar to:

Please unlock disk...

Without the encryption passphrase:

Recovery mode is inaccessible
User data remains unreadable
Password reset procedures become ineffective

This is why security professionals often say:

Your real protection isn't your login password—it's disk encryption.

The login password protects your account.

Disk encryption protects your data.

Why Linux Allows This

At first glance, being able to reset passwords seems insecure.

In reality, it solves a practical problem.

System administrators occasionally forget passwords. Machines become inaccessible. Recovery tools are necessary.

Linux follows a long-standing philosophy:

Physical access usually implies administrative control.

If you want stronger protection against physical attacks, encryption—not account passwords—is the answer.

Final Thoughts

Resetting a forgotten Linux password is surprisingly simple across most distributions.

The process demonstrates an important security principle:

Passwords protect accounts.
Root access controls the system.
Physical access is powerful.
Encryption protects data.

The next time someone asks, "What happens if I forget my Linux password?" the answer is usually:

"Boot into recovery mode, reset it, and carry on."

The more interesting question is:

"What protects the machine if someone else does the same thing?"

The answer is full-disk encryption.

Apache Hadoop on Ubuntu 22.04 (Beginner-Friendly, Java 8 Without Breaking Java 21)

Seamoon Pandey — Sat, 31 Jan 2026 14:43:23 +0000

Hadoop is still a core concept in data engineering — especially for understanding HDFS, YARN, and distributed systems.

The biggest blocker for beginners today is Java version conflicts.
Most modern systems run Java 17 or 21, while Hadoop prefers Java 8.

This guide shows you how to:

Install Hadoop from scratch
Use Java 8 only for Hadoop
Keep your system Java untouched
Avoid auto-start, battery drain, and common traps

What you’ll build

Hadoop 3.3.6
Single-node (pseudo-distributed)
Ubuntu 22.04
Java 8 for Hadoop only
Manual start/stop (laptop-friendly)

Prerequisites

Ubuntu 22.04
Internet connection
sudo access
No prior Hadoop knowledge required

Step 1: System preparation

sudo apt update
sudo apt install -y ssh rsync curl

Hadoop relies on SSH — even on a single machine.

Step 2: Create a dedicated Hadoop user (recommended)

sudo adduser hadoop
sudo usermod -aG sudo hadoop
su - hadoop

From here on, everything runs as the hadoop user.

Step 3: Enable passwordless SSH

Hadoop services communicate over SSH.

ssh-keygen -t rsa -P ""
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
ssh localhost

If it logs in without asking for a password → you’re good.

Step 4: Java setup (important)

Hadoop works best with Java 8.
You do NOT need to uninstall Java 17/21.

Verify Java 8 exists:

ls /usr/lib/jvm/java-8-openjdk-amd64

If it exists, continue.
(You can install it with sudo apt install openjdk-8-jdk if missing.)

Step 5: Download Hadoop

cd ~
wget https://downloads.apache.org/hadoop/common/hadoop-3.3.6/hadoop-3.3.6.tar.gz
tar -xvzf hadoop-3.3.6.tar.gz
mv hadoop-3.3.6 hadoop

Step 6: Hadoop environment variables

Edit .bashrc:

nano ~/.bashrc

Add at the bottom:

export HADOOP_HOME=$HOME/hadoop
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

Apply:

source ~/.bashrc

Step 7: Bind Hadoop to Java 8 (critical step)

This avoids breaking your other Java projects.

nano ~/hadoop/etc/hadoop/hadoop-env.sh

Set:

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

Verify:

hadoop version

You should see:

Java version: 1.8.0

Step 8: Hadoop configuration

Move into config directory:

cd ~/hadoop/etc/hadoop

core-site.xml

<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://localhost:9000</value>
  </property>
</configuration>

hdfs-site.xml

<configuration>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
  <property>
    <name>dfs.name.dir</name>
    <value>file:/home/hadoop/hadoopdata/namenode</value>
  </property>
  <property>
    <name>dfs.data.dir</name>
    <value>file:/home/hadoop/hadoopdata/datanode</value>
  </property>
</configuration>

Create directories:

mkdir -p ~/hadoopdata/namenode ~/hadoopdata/datanode

mapred-site.xml

cp mapred-site.xml.template mapred-site.xml

<configuration>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
</configuration>

yarn-site.xml

<configuration>
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
</configuration>

Step 9: Format the NameNode (run ONCE)

hdfs namenode -format

⚠️ Running this again wipes HDFS metadata.

Step 10: Start Hadoop

start-dfs.sh
start-yarn.sh

Verify:

jps

Expected processes:

NameNode
DataNode
SecondaryNameNode
ResourceManager
NodeManager

Step 11: Access Web UIs

HDFS: http://localhost:9870
YARN: http://localhost:8088

If these load → Hadoop is running correctly.

Step 12: Test HDFS

hdfs dfs -mkdir /test
hdfs dfs -put ~/.bashrc /test
hdfs dfs -ls /test

Expected output:

Found 1 items
-rw-r--r-- 1 hadoop supergroup ...

About the “native-hadoop” warning

You may see:

WARN NativeCodeLoader: Unable to load native-hadoop library

This is normal.

Hadoop falls back to pure Java
No functionality is lost
Safe to ignore for learning & development

Does Hadoop auto-start on boot?

No.

Hadoop:

Does NOT register as a system service
Does NOT start after reboot
Uses zero resources unless manually started

To stop:

stop-dfs.sh
stop-yarn.sh

Viola you're done.

Transferring SSH Keys (`id_rsa` and `id_rsa.pub`) to a New System

Seamoon Pandey — Mon, 28 Oct 2024 08:51:54 +0000

SSH keys are essential for secure communication between computers. If you're migrating to a new system but want to retain your existing SSH keys (id_rsa and id_rsa.pub), follow these steps to seamlessly transfer them.

Step 1: Locate Your SSH Keys

First, identify the location of your SSH keys on your old system. Typically, they reside in the ~/.ssh/ directory.

cd ~/.ssh/
ls -al

Step 2: Backup Your Keys

Ensure the safety of your SSH keys by creating a backup. Copy them to a secure location, such as an encrypted USB drive or a trusted cloud storage service.

cp id_rsa id_rsa.pub /path/to/backup/location

Step 3: Transfer the Keys to the New System

Use the scp (secure copy) command to transfer the SSH keys to your new system.

scp /path/to/backup/location/id_rsa /path/to/backup/location/id_rsa.pub user@new_system_ip:~/.ssh/

Replace user with your username on the new system and new_system_ip with the IP address or hostname of your new system.

Step 4: Set Permissions

Ensure that the permissions of your id_rsa file are secure. Only the owner should have read and write permissions.

chmod 600 ~/.ssh/id_rsa

Step 5: Add Public Key to Authorized Keys

Append the contents of your id_rsa.pub file to the authorized_keys file on your new system. This allows you to authenticate using your SSH private key.

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

Step 6: Test the Connection

Finally, test the SSH connection to your new system to ensure everything is set up correctly.

ssh user@new_system_ip

If successful, you should log in to your new system without being prompted for a password.

Congratulations! You've successfully transferred your SSH keys to your new system, ensuring secure access to your remote servers.

Feel free to customize the instructions further based on your specific environment or requirements!

Reset password mailer implementation in rails 7 api, devise_token_auth, and sendgrid-ruby.

Seamoon Pandey — Mon, 28 Oct 2024 08:45:37 +0000

In this guide, we’ll walk through setting up password reset functionality in a Rails app using devise_token_auth for authentication and SendGrid for email delivery. With this, users can request password reset links sent directly to their emails.

Dependencies

Add devise_token_auth and sendgrid-ruby to your Gemfile:

# Sendgrid for sending emails
gem 'sendgrid-ruby'

# Devise for authentication
gem 'devise_token_auth'

Then install the dependencies:

bundle install

Or add each gem individually:

bundle add 'sendgrid-ruby'
bundle add 'devise_token_auth'

Setup

To generate Devise token auth files, run:

rails g devise_token_auth:install User auth

Now, open the generated migration file ...devise_token_auth_create_users.rb, and uncomment these lines:

t.string   :confirmation_token
t.datetime :confirmed_at
t.datetime :confirmation_sent_at
t.string   :unconfirmed_email

Run the migration:

bin/rails db:migrate

In your user.rb model file, add:

extend Devise::Models
include DeviseTokenAuth::Concerns::User

# Add :recoverable for password recovery
devise :database_authenticatable, :registerable, :recoverable

In config/initializers/devise_token_auth.rb, configure password reset URL:

config.change_headers_on_each_request = false
config.default_password_reset_url = "#{ENV['FRONTEND_URL']}/reset-password"

Routes Configuration

In config/routes.rb, add:

mount_devise_token_auth_for "User", at: "auth", controllers: {
  passwords: "user/passwords"
}

Controller Setup

Create controllers/user/passwords_controller.rb for handling password reset actions:

# frozen_string_literal: true
class User::PasswordsController < DeviseTokenAuth::PasswordsController
  # POST /auth/password
  def create
    email = resource_params[:email]
    if email.blank?
      return render json: { success: false, errors: ["Email cannot be blank"] }, status: :unprocessable_entity
    end

    resource_class.send_reset_password_instructions(email: email)
    render json: { success: true, message: "If the email exists, password reset instructions have been sent." }, status: :ok
  end

  # PUT /auth/password
  def update
    if password_update_params[:reset_password_token].blank?
      authenticate_user!

      unless current_user.valid_password?(password_update_params[:old_password])
        return render json: { success: false, errors: ["Old password is incorrect"] }, status: :unprocessable_entity
      end

      if current_user.update(password: password_update_params[:password])
        render json: { success: true, message: "Password updated successfully for authenticated user" }
      else
        render json: { success: false, errors: current_user.errors.full_messages }, status: :unprocessable_entity
      end

    else
      resource = resource_class.reset_password_by_token(password_update_params)
      if resource.nil?
        return render json: { success: true, message: "If the email exists, password reset instructions have been sent." }
      end

      if resource.errors.present?
        return render json: { success: false, errors: resource.errors.full_messages }, status: :unprocessable_entity
      end

      resource.update(password: password_update_params[:password])
      render json: { success: true, message: "Password updated successfully" }
    end
  rescue StandardError => e
    render json: { success: false, errors: ["An unexpected error occurred: #{e.message}"] }, status: :internal_server_error
  end

  private

  def password_update_params
    params.permit(:reset_password_token, :password, :password_confirmation, :old_password)
  end

  def resource_params
    params.permit(:email)
  end
end

Now, users can initiate a password reset by sending a POST request to /auth/password.

Customizing the Email Template

To modify the reset email, update views/devise/mailer/reset_password_instructions.html.erb:

<p>Hello <%= @resource.email %>!</p>
<p>Someone requested a password reset. You can reset it via the link below.</p>
<% reset_url = DeviseTokenAuth.default_password_reset_url + "?reset_password_token=#{@token}" %>
<p><%= link_to 'Change my password', reset_url %></p>
<p>If you didn’t request this, please ignore this email.</p>
<p>Your password won’t change until you create a new one.</p>

Environment Setup

Sign in to SendGrid, set up your API key, and verify your sender email.
Use dotenv or Rails credentials to securely store the following in your .env file:

SENDGRID_API_KEY=SG.******************
SENDER_EMAIL=your_verified_email@domain.com

Configure SendGrid in your environment files (config/environments/development.rb and config/environments/production.rb):

config.action_mailer.perform_deliveries = true
config.action_mailer.smtp_settings = {
  address: "smtp.sendgrid.net",
  port: 587,
  authentication: :plain,
  user_name: "apikey", # Keep as "apikey"
  password: ENV["SENDGRID_API_KEY"],
  domain: ENV["BASE_URL"],
  enable_starttls_auto: true
}

Wrapping Up

And that’s it! Now, users can reset their passwords by requesting a link through your Rails API with Devise Token Auth and SendGrid.

References

Swapping Numbers Without a Temporary Variable in C using XOR like a pro.

Seamoon Pandey — Thu, 22 Feb 2024 02:48:34 +0000

Swapping the values of two variables without using a temporary variable is a classic programming problem. One elegant solution to this problem in C involves using bitwise XOR operations.

Algorithm

Consider two variables a and b. The goal is to swap their values.

Initialize a and b with the values to be swapped.
Perform the following steps:
```
a ^= b;
b ^= a;
a ^= b;
```

Explanation

Let's break down the algorithm step by step:

a ^= b;: XOR (^=) a with b and store the result back in a. After this operation, a contains the result of a XOR b.
b ^= a;: XOR b with the new value of a (which was the original value of b). After this operation, b contains the result of b XOR (a XOR b), which simplifies to a.
a ^= b;: XOR a with the new value of b (which was the original value of a). After this operation, a contains the result of (a XOR b) XOR a, which simplifies to b.

Now, a holds the original value of b and b holds the original value of a, effectively swapping their values without using a temporary variable.

Example

Consider the following example:

int a = 5, b = 7;
a ^= b;
b ^= a;
a ^= b;

Ruby on Rails Api authentication with rails cookie session.

Seamoon Pandey — Sun, 10 Dec 2023 03:55:45 +0000

Backend (Rails API)

Step 1: Create a New Rails API Project

To start building your Rails API, open your terminal and run:

rails new your_api_project_name --api -d postgresql
cd your_api_project_name

Explanation: This command initializes a new Rails API project, enabling API-only functionality and using PostgreSQL as the database.

Step 2: Integrate Devise and Devise Token Auth

Add Devise and Devise Token Auth to your Gemfile:

gem 'devise_token_auth'

Then, execute the following commands:

bundle install
rails generate devise:install
rails generate devise_token_auth:install
rails db:migrate

Explanation: These commands install and configure Devise along with Devise Token Auth. They set up authentication for your User model and create the necessary files and database tables.

Step 3: Configure ApplicationController

In app/controllers/application_controller.rb, add the following lines:

class ApplicationController < ActionController::API
  include ActionController::Cookies
  include DeviseTokenAuth::Concerns::SetUserByToken
end

Explanation: This configures your ApplicationController to include modules for handling cookies and setting the user based on the authentication token.

Step 4: Configure Application

In config/application.rb, add the following lines:

config.api_only = true
config.session_store :cookie_store, key: '_your_api_session', httponly: true, same_site: :strict, secure: true
config.middleware.use ActionDispatch::Cookies
config.middleware.use config.session_store, config.session_options

Explanation: These configurations ensure that your API works in an API-only mode, sets up secure cookie storage, and adds necessary middleware.

Step 5: Configure Devise Token Auth

In config/initializers/devise_token_auth.rb, add the following:

config.cookie_enabled = true
config.cookie_attributes = {
  httponly: true,
  encrypt: true,
  same_site: :strict,
  secure: true
}

In config/routes.rb, mount Devise Token Auth for the 'User' model:

mount_devise_token_auth_for 'User', at: 'auth'

Explanation: These configurations enable token authentication and set up cookie attributes for added security.

Frontend (Using Axios)

Step 6: Set Up Axios for Authenticated Requests

Install Axios using:

yarn add axios

Now, when making authenticated requests, use Axios with withCredentials: true:

axios.get('your/api/endpoint', { withCredentials: true })
  .then(response => {
    // Handle the response
  })
  .catch(error => {
    // Handle the error
  });

Explanation: Axios is a popular HTTP client for the browser and Node.js. The withCredentials: true option ensures that cookies are sent with requests, crucial for authentication.

Step 7: Middleware for Authenticated Requests

Consider creating a middleware for handling authentication in your frontend. Middleware can centralize logic for authenticated requests.

Step 8: Manage Authenticated State with a Cookie

Use a logged_in cookie to manage the authenticated state on the client side. Set it to true on login and false on logout or session expiration.

Explanation: This helps keep track of the user's authentication status on the frontend, providing a seamless user experience.

This guide covers the setup of a Rails API with token-based authentication using Devise and Devise Token Auth, along with frontend integration using Axios. Feel free to customize these steps based on your project's specific needs. If you have any questions or need further clarification, Mate, do some more research!