I work at SEOSage, which builds desktop SEO audit software. The steps below work with any crawler.
Your pages should not compete with each other.
Duplicate content rarely triggers a penalty — that is a myth for most sites. The real damage is quieter: when three near-identical pages exist, search engines split the ranking signals between them, pick a canonical you did not intend, and none of the versions ranks as well as one consolidated page would. This audit finds the internal competition and ends it.
The three kinds of duplication
- Exact duplicates — the same page served at multiple URLs: trailing-slash variants, www vs non-www, HTTP vs HTTPS, uppercase vs lowercase, print versions. Usually a configuration problem, not a content one.
- Near-duplicates — pages that differ only slightly: sorted and filtered views, paginated series, location pages with the city name swapped, tag pages echoing category pages. The most common and the hardest to see without a crawl.
- Thin duplicates — pages with almost nothing unique: auto-generated tag archives, empty category shells, boilerplate service pages cloned per suburb. These dilute the site even when each URL is technically unique.
The audit: compare, don't guess
- Compare the metadata. Export every title, meta description, and H1 from a full crawl. Sort by title: identical titles across different URLs are your first suspect list. Duplicate meta descriptions are the second. This one export catches most exact and template-level duplicates in minutes.
- Check the content. For the suspect URLs, compare actual page content — many crawlers flag near-duplicate content with a similarity threshold. Focus on the template: if every location page is 95% identical, the fix is one content rewrite strategy, not fifty edits.
- Map the canonicals. For each duplicate cluster, check where the canonical tag points. The most common finding: canonicals pointing at themselves on pages that should defer to a master version, or canonicals on pages Google has chosen to ignore because the content barely matches.
- Cross-check Search Console. The Page indexing report's "Duplicate, Google chose different canonical than user" and "Duplicate without user-selected canonical" entries confirm which clusters Google actually sees as duplicates — audit evidence, straight from the source.
Fix by cluster, not by URL
Each cluster gets one decision:
- Consolidate — one page should exist where three near-duplicates do. Pick the winner, move any unique value into it, and redirect the rest to it.
- Canonicalise — when the duplicate URLs must stay live (sorted views, print pages), point every variant's canonical at the clean master URL.
- Differentiate — when the pages deserve to exist separately (real location pages), give each genuinely unique content: local specifics, distinct services, original copy. Swapped city names are not unique content.
- Remove — thin auto-generated pages that serve no user get noindexed or deleted, with their internal links updated.
Prioritise by traffic
Consolidation touches rankings, so start where the stakes are: clusters containing pages that already bring visitors. I overlay GA traffic on the crawl in SEOSage so the duplicate clusters sort by traffic — the same traffic-first habit, applied to duplication. Merge the earning clusters first; the thin archives can wait their turn.
The maintenance rule
Duplication regenerates. New plugins create new parameter views, new templates clone the old boilerplate, new locations get another swapped-city page. Re-run the title/meta comparison after every site change — it is the fastest tripwire in the whole audit set, and it catches new clusters before they split your signals again.
Disclosure: I work at SEOSage, a desktop SEO audit tool. These steps work with any crawler you already use. How many URLs on your site share the same title tag right now?
Top comments (0)