If your photo library has grown out of control, there's a good chance a chunk of it is exact duplicates: the same file copied during a backup, a sync, or an import you ran twice. Before reaching for any tool, it's worth knowing you can find these with something already on your machine.
The quick way: sha256sum
On Linux (GNU coreutils), you can hash every file in a folder and group the ones with identical hashes. Identical hash means identical content, byte for byte.
find . -type f \( -iname "*.jpg" -o -iname "*.jpeg" -o -iname "*.png" \) -exec sha256sum {} \; > hashes.txt
sort hashes.txt | uniq -w64 -D
The first line walks the current folder recursively and writes one hash per image into hashes.txt. The second sorts by hash and prints every line that belongs to a group of identical files. uniq -w64 -D compares only the first 64 characters (the hash itself).
On macOS, use shasum -a 256 instead of sha256sum, and install GNU coreutils (brew install coreutils) to get a uniq that supports -w. It is then available as guniq.
What this gets you, and what it doesn't
This approach is great for exact duplicates: the same file, same bytes, maybe just renamed or moved. It won't catch:
- The same photo re-exported at a different resolution
- A slightly recompressed JPEG
- A screenshot vs. the original image it was taken from
For those "similar but not identical" cases you need perceptual comparison, not a cryptographic hash. That is a different problem and outside the scope of a one-liner.
Also worth noting: the commands above only report duplicates. They don't move or delete anything, which is a good thing. You don't want a script silently deleting files based on a hash match without you checking first.
When you outgrow the one-liner
The commands above work fine for one folder and exact matches. It starts to get annoying once you have:
- Thousands of photos across many nested folders
- Near-duplicates (same photo, different edit, export or resolution)
- A need for a report you can review before anything is touched
- A desire to not delete files outright, just quarantine them
That's the situation I built Duplicate Photo Finder Pro for. It is mine and it is paid. It scans folders for exact duplicates (SHA-256, the same method as above) and also for similar images via perceptual hashing with an adjustable threshold. It runs in dry-run mode first, writes a CSV/JSON report, and moves duplicates to a quarantine folder instead of deleting them, keeping whichever copy you choose (oldest, largest or highest resolution) as the original. It runs fully offline, with no cloud upload.
Duplicate Photo Finder Pro, 12 EUR: https://skeilster.gumroad.com/l/nejwxp
Top comments (0)