Your Drive Is Full of Files You Already Have. Here's a Safe Way to Find Them

Every hard drive has a hoarding problem. Camera bursts save 40 near-identical photos. Backup scripts copy the same folder into three locations. Downloads accumulate the same installer under different names. Before you know it, a 2 TB drive is "full" — and you can't actually point at what's eating the space.

The usual advice is to run a duplicate finder. But here's what nobody tells you: most duplicate finders are dangerous or useless at scale. They either hash every byte of every file (slow on big drives), or they group by size only and delete things that merely look alike.

The two real problems

1. False positives. Two files with the same size are not the same file. A size-only "duplicate" scan will happily flag two different 8 MB PDFs as identical. If you act on that, you delete data.

2. No undo. Many tools delete duplicates on the spot. One misclick and your original is gone — the "duplicate" you kept was actually the one with the newer comments.

What I built: a cautious duplicate finder

A single-file Python CLI (pure standard library, zero dependencies) that treats your files like evidence, not junk:

• Two-stage hashing — group candidates by size first, then confirm with SHA-1, and optionally re-verify the survivors with SHA-256 before reporting. No byte-for-byte hash, no false "duplicate" verdict.
• Report first, act never — default mode writes a Markdown/JSON report and touches nothing.
• A quarantine directory instead of delete — --delete-to moves duplicates to an isolated folder, keeping the first copy in place. You get your space back and a full undo path if you change your mind.
• --dry-run before anything destructive, plus --exclude and --min-size filters for the noise (skip system dirs, only scan files above 1 MB, etc.).
• Exit code contract — 1 when duplicates are found, so you can hook it into cron or CI and get alerted that your backup dir is breeding copies again.

# Scan and report only (the safe default)
python duplicate_finder.py --dirs ~/Photos ~/Backups

# Move confirmed duplicates to a quarantine dir, keeping one copy of each
python duplicate_finder.py --dirs ~/Photos --delete-to ~/dupes_quarantine --dry-run

# Reclaim the space for real
python duplicate_finder.py --dirs ~/Photos --delete-to ~/dupes_quarantine

Fully local — your file list never leaves your machine, which matters when the drive in question contains client documents or family photos.

Why "cautious" wins

The best dedup tool isn't the one that deletes the most. It's the one you trust enough to run on a drive you actually care about. Size-group + double-hash confirmation + quarantine-with-undo is that trust. The full kit (scanner + quarantine + JSON/Markdown reports + docs) is at AgentChip.

Your storage is finite. Your duplicate copies aren't.

---

Originally published on the AgentChip blog.