How I added exact-duplicate detection to a free WordPress plugin, what broke on a 3,099-image multilingual site, and the SQL that fixed it.
Last week I shipped duplicate image detection in Filikod, the free alt text audit plugin I maintain. On paper the feature is small: fingerprint every image file, group identical fingerprints, show the groups. Then I ran it on a real client site. 3,099 images, three languages under WPML, a media library migrated twice. Three of my assumptions did not survive the afternoon. This is the story, without the plugin's source code, because the lessons matter more than my implementation.
Assumption 1: a duplicate has a recognisable name
I expected duplicates to look like photo.jpg and photo-1.jpg. Some do. Most do not. On my own site the same dashboard screenshot existed under three names, uploaded in July, August and September by three different people, with three different titles and three different alt texts. Nothing in the WordPress admin connects them. You can scroll the media grid for an hour and never notice.
So the identity of an image has to be the content of the file, not its name. A fingerprint of the bytes is enough to catch exact copies, and exact copies are the only ones you can act on with confidence. A resized or re-encoded variant is another file, another decision, another project. I left near-duplicates out on purpose.
One subtlety that cost me an hour: since WordPress 5.3, a large upload is scaled down and the attachment points at the scaled file, while the original stays on disk. Two uploads of the same photo produce two scaled files that are usually identical, but "usually" is not a guarantee, a site that changed image library between the two uploads gets different bytes. Fingerprint the original when WordPress kept one. And keep the fingerprint in sync: the file behind an attachment changes after upload far more often than you think, through scaling, image optimisers, the built-in editor, replacement plugins, thumbnail regeneration. All of them end by saving the attachment metadata, which is the moment to recompute.
Assumption 2: same file, same problem
The first scan of the client site listed several hundred groups. Almost all of them were the same thing: one image, three attachments, one per language. WPML Media Translation, Polylang's media translation and Bogo all create a separate attachment per language, pointing at the same physical file. Same bytes, same fingerprint, and absolutely not a duplicate to clean up. Delete one of those "copies" and the Spanish version of the page loses its image.
Worse, the alt text audit had been quietly wrong for months on that site. The French, English and Spanish versions of one product photo carried the same alt text, which is normal for a brand name, and the audit counted them as three "duplicated ALT" errors, dragging the score down for something nobody should fix.
The fix is a rule, not a hack: a translation set is one image. Every multilingual plugin keeps its links somewhere queryable, WPML in its own table, Polylang in a hidden taxonomy, Bogo in a post meta, so the whole map of "which attachment belongs to which set" can be loaded in one query per request instead of one call per image. Once you have the map, a group of identical files collapses to one entry per translation set, and a group that shrinks to a single entry is not a group anymore. The same rule applies to the alt audit: an alt is only duplicated when it sits on two genuinely distinct images.
Weglot, GTranslate and TranslatePress translate on the fly and never create a second attachment. WPGlobus keeps every language in one post. MultilingualPress runs one site per language. Nothing to do for those, but you have to know it to say so in the documentation.
After the fix the same library gave 29 groups and 30 redundant files. Every one of them real: the same substrate photo used on two product pages under two names, a pictogram uploaded twice by two editors, a partner logo that existed four times. That is a list a human can actually work through in twenty minutes.
Assumption 3: the media library contains files
The scan finished with 232 images "still waiting". They were not waiting. They were attachments whose file no longer existed on disk: leftovers of two migrations, still listed in the media library with a title and an alt text, still referenced in a few pages, rendering as broken images to visitors. Nothing in the WordPress admin surfaces this. The library shows a generic icon and moves on.
I turned that into its own tab, with the expected path and the page each attachment is used on. On a site that has been migrated, it may be the most useful list of the three.
A side lesson for anyone auditing media on WPML or Polylang
Both plugins filter admin queries to the currently selected admin language. My audit was silently skipping two thirds of that library, because the admin bar was set to French, until I told the query to ignore third-party filters:
'suppress_filters' => true,
If you have ever written a media audit, an export, or a cleanup script that runs in the admin of a multilingual site, check your counts against a raw SQL count. Mine were off by 1,838 images.
What it deliberately does not do
The plugin never deletes or merges anything. Rewriting every reference to a file across post content, page builder JSON, custom fields and widgets is where duplicate cleaners earn their one-star reviews, and it is not something I want a plugin to do behind someone's back. It shows every copy, the page it is attached to, marks the oldest one, and leaves the decision to the person in front of the media library. It also warns at upload time when the file you just added already exists, which is the moment the duplicate is cheapest to avoid.
Numbers from that first afternoon
| Images in the library | 3,099 |
| Fingerprinted | 2,867 |
| Attachments without a file on disk | 232 |
| Groups before handling translations | several hundred |
| Real duplicate groups after | 29 |
| Redundant files, disk space | 30, 11 MB |
| Images the alt audit had been skipping | 1,838 |
The version with all of this is Filikod 1.2.0, free and GPL on WordPress.org. If you run it on a multilingual site and the duplicates page shows you a translation set, tell me: it means there is a fourth assumption I have not met yet.
Top comments (0)