How I Built HashDup: A Fast, Memory-Safe Duplicate File Finder CLI in Node.js

#javascript #node #performance #opensource

Over time, every workstation accumulates duplicate files: duplicated photo backups, repeated downloads of large zip archives, cloned repositories, and redundant ISOs. They quietly eat gigabytes of expensive SSD storage.

Most people write a simple script that reads all files and compares their hashes. But on large drives, that naive approach has two fatal problems:

Performance bottleneck: Hashing thousands of files wastes CPU and disk I/O when 90% of files have unique sizes and can never be duplicates.
Memory crashes (OOM): Reading multi-gigabyte files into memory (fs.readFileSync) crashes the Node.js event loop with out-of-memory errors.

To solve this efficiently, I built HashDup — a fast, memory-safe CLI duplicate finder built with native Node.js streams and a smart 2-phase scanning engine.

The 2-Phase Scanning Architecture

HashDup uses an algorithmic filtering pipeline that avoids unnecessary disk reads:

[All Target Files]
       │
       ▼
Phase 1: Size-First Grouping (O(1) filter)
  └── Discard all files with unique byte sizes without reading contents!
       │
       ▼ (Only files with identical sizes remain)
Phase 2: Chunked Streaming SHA-256 Hashing
  └── Streams files in 64KB buffers (Flat O(1) memory consumption)
       │
       ▼
[Identified Duplicate Sets & Reclaimable Space]

1. Phase 1: Size-First Filtering

Before reading any file content, HashDup runs a lightweight metadata scan using fs.promises.stat().

If a file has a unique size in bytes, it is physically impossible for it to be a duplicate of any other file. We discard these immediately. On a drive with 10,000 files, this eliminates 90%–95% of candidates in milliseconds!

2. Phase 2: Memory-Safe Streaming SHA-256

For candidate files that share the exact same byte size, we compute cryptographic SHA-256 digests.

Instead of buffering whole files into RAM, HashDup uses Node.js streams:

import fs from 'node:fs';
import crypto from 'node:crypto';

function hashFile(filePath) {
  return new Promise((resolve, reject) => {
    const hash = crypto.createHash('sha256');
    const stream = fs.createReadStream(filePath);

    stream.on('data', chunk => hash.update(chunk));
    stream.on('end', () => resolve(hash.digest('hex')));
    stream.on('error', reject);
  });
}

This ensures that whether a file is 2 KB or 10 GB, Node.js memory usage stays constant at around 30MB of RAM.

Safety & Reporting Features

Waste Analytics: Displays exact duplicate groupings, file paths, and calculates total reclaimable space (MB/GB).
Dry-Run by Default: Inspect duplicates in a clean CLI table before touching or deleting anything.
Zero Heavy Dependencies: Built entirely with Node.js built-ins (crypto, stream, fs/promises).

Quick Start

You can test it right now on any directory using npx:

# Scan current directory and preview duplicates
npx hashdup-cli scan .

# Scan with custom output format or dry-run
npx hashdup-cli scan ~/Downloads --min-size 1MB

Source Code & Community Feedback

HashDup is open source under the MIT license.

🔗 GitHub Repository: github.com/mahdyarmonfared/hashdup-cli

I'd love to hear from other CLI developers:

How do you handle file comparisons on massive datasets (e.g., partial hashing vs full SHA-256)?
Would you find an interactive terminal cleanup wizard (TUI) useful for safe one-click deletion?

Feedback, PRs, and ⭐ stars on GitHub are always appreciated!

DEV Community