DEV Community

poolion
poolion

Posted on

Duplicate File Finder: Find Exact Duplicate Files Using MD5 Hashing in Python

#opensource #python #tutorial #cli

Duplicate File Finder: A Minimal CLI Tool to Find Exact Duplicate Files in Python

I built duplicate-file-finder to help clean up disk space. This script identifies exact duplicate files by content hash and optionally removes extras — perfect for those messy Downloads folders or old project backups!

What It Does

The tool scans a directory and finds files with identical contents, regardless of where they're located:

  • Find mode: Lists all duplicate groups you can review
  • Remove mode: Deletes all but one copy per group
  • Report mode: Preview duplicates without deleting

It uses MD5 (default) for speed or SHA1 for larger files. Both algorithms come from Python's built-in hashlib — no pip install needed!

No dependencies, no virtualenv setup — just drop it anywhere and run!

Installation

Use any Python 3 environment:

python find_duplicates.py --help
Enter fullscreen mode Exit fullscreen mode

That's it! The argparse module handles CLI parsing automatically.

Usage Examples

1. Find Duplicates in Downloads Folder

Check your download folder for exact copies:

python find_duplicates.py find -s ~/Downloads
Enter fullscreen mode Exit fullscreen mode

Output shows each group with all file paths:

Found 3 duplicate groups:
2        files:       /home/user/Downloads/installer-1.msi     >    /home/user/Downloads/backup_installer.msi
5         files:   /media/photos/IMG_101.jpg                   >    /media/camera-backup/IMG_101.jpg
Enter fullscreen mode Exit fullscreen mode

2. Report (Preview) Before Deleting

See what would be wasted, but don't delete yet:

python find_duplicates.py report -s ~/Pictures
Enter fullscreen mode Exit fullscreen mode

Shows duplicate groups with wasted space calculated!

3. Remove Duplicates (Interactive Confirmation)

Review first, then confirm removal:

# Interactive - asks "Remove all extras?" before deleting  
python find_duplicates.py remove -s /var/tmp/downloads

# Automatic deletion without confirmation prompt
python find_duplicates.py remove -s ~/Downloads -y
Enter fullscreen mode Exit fullscreen mode

How I Built It

The implementation is straightforward Python with three core functions:

1. Hash Computation

I use hashlib.md5() or hashlib.sha1():

def compute_hash(file_path):
    hasher = hashlib.md5()
    with open(file_path, 'rb') as f:
        for chunk in iter(lambda: f.read(8192), b''):
            hasher.update(chunk)
    return hasher.hexdigest()
Enter fullscreen mode Exit fullscreen mode

Why hash? This compares file contents byte-by-byte, not filenames. File A at /docs/file.pdf matches File B at /backups/file.pdf only if their bytes are identical.

2. Directory Scanning

I use os.walk() to traverse recursively:

for root, dirs, files in os.walk(source_dir):
    for f in files:
        fp = os.path.join(root, f)
        size = os.path.getsize(fp)
        if size < 256 or size > 10_000_000:
            continue  # Skip tiny/very large files
        hash_val = compute_hash(fp)
        file_hashes[hash_val].append(fp)
Enter fullscreen mode Exit fullscreen mode

Smart filtering: I skip files <256 bytes (unlikely to be intentional duplicates) and >10MB (takes seconds per file). For media servers, adjust these thresholds!

3. Grouping Duplicates

After hashing all files, I filter for groups with 2+ entries:

duplicates = [files for hash_val, files in file_hashes.items() if len(files) > 1]
Enter fullscreen mode Exit fullscreen mode

Output Format Example

Find command:

python find_duplicates.py find -s /var/tmp/downloads
Enter fullscreen mode Exit fullscreen mode

Output:

Found 4 duplicate groups:
2   files:   /var/tmp/cleanup/sample-doc.pdf        >   /media/backup/sample-doc.pdf
5   files:   /home/user/Photos/screenshot.png       >   /mnt/NAS/backups/screenshot.png
                                                    >   /home/user/Archived/IMG_0492.screenshot
Enter fullscreen mode Exit fullscreen mode

Complete Options Summary

Find Duplicates 

python find_duplicates.py find \
    -s <directory-path> \
    [-a md5|sha1] \
    [-q quiet-mode-no-output-for-groups]
Enter fullscreen mode Exit fullscreen mode
  • -s: Source directory to scan (required)
  • -a: Hash algorithm: md5 fast, sha1 more secure for large files
  • -q: Silent mode — only print total count

Remove Duplicates 

python find_duplicates.py remove \
    -s <directory-path> \
    [-y skip-confirmation-prompt] \
    [-a md5|sha1]
Enter fullscreen mode Exit fullscreen mode
  • Without -y: Interactively confirms each group before deleting extras
  • With -y: Automatic deletion without confirmation (use carefully!)

Preview Duplicates (Report) 

python find_duplicates.py report \
    -s <directory-path> \
    [-a md5|sha1]
Enter fullscreen mode Exit fullscreen mode

Shows all groups with file counts and wasted space before any action.

Safety First!

Before running remove — especially with-y`:

  1. Always run report or find first to preview duplicates
  2. Check paths carefully — the tool uses actual file locations, not names
  3. Test on a small folder before scanning your entire /media

The tool never modifies files until you explicitly choose remove. Even then, -y is just an automation flag — double-check output!

Why Build This?

Most "duplicate finder" tools are heavy GUI apps or require Python packages. This one uses only the standard library:

  • Pure Python with argparse, pathlib/os, hashlib
  • Works on any Linux/macOS/Windows system with Python 3 installed
  • Perfect for server cleanup scripts

I wrote this to help people reclaim space from:

  • Multiple downloads of the same installer
  • Screenshots copied to different locations
  • Project backups that duplicated media files

Full Code Repository

Want to see the implementation? Published at:

https://github.com/Poolion/find_duplicates

The code is minimal and heavily commented if you want to study or reuse it!

If you find this useful, you can support development: https://www.buymeacoffee.com/poolion

Top comments (0)