Preventing Search Cannibalization: Dynamic URL Graph Deduplication

#webdev #seo #nextjs #javascript

As your web application, blog, and documentation scale, an insidious SEO penalty begins to silently erode your organic traffic: Keyword Cannibalization.

Cannibalization happens when multiple pages on the same domain compete for identical or heavily overlapping search queries. Search engine ranking algorithms get confused about which URL is the authoritative source, dividing backlink equity and click-through signals between them.

The result? Instead of one page ranking at Position #2, both pages bounce between Position #15 and #45.

In ⚡ PLYXO (CRO • SEO • AIO • AEO • GEO), our Claude-SEO crawler builds an internal semantic graph of every published URL to identify cannibalization before it destroys your traffic.

1. Common Developer Anti-Patterns that Cause Cannibalization

Pattern A: Fragmented Product & Feature Pages

Creating both /features/crm and /products/crm-platform where 80% of the copy describes the same core features.

Pattern B: The Blog Title Overlap

Publishing multiple articles over six months without a clear canonical pillar:

Post 1: "How to Optimize Next.js Web Performance"
Post 2: "Next.js Performance Tips for 2026"
Post 3: "Speed Up Next.js Applications"

Pattern C: Query Parameter Sprawl

Failing to canonicalize faceted navigation parameters (/products?sort=price&category=tools vs /products?category=tools).

2. Algorithmic Detection: Computing Jaccard Title & Heading Similarity

To catch cannibalization automatically, Plyxo compares the token sets of titles and <h2> headings across all internal pages:

export function computeJaccardSimilarity(tokensA: Set<string>, tokensB: Set<string>): number {
  const intersection = new Set([...tokensA].filter(t => tokensB.has(t)));
  const union = new Set([...tokensA, ...tokensB]);

  if (union.size === 0) return 0;
  return intersection.size / union.size;
}

export function detectCannibalization(pages: Array<{ url: string; headings: string[] }>) {
  const conflicts: Array<{ urlA: string; urlB: string; similarity: number }> = [];

  for (let i = 0; i < pages.length; i++) {
    for (let j = i + 1; j < pages.length; j++) {
      const tokensA = new Set(pages[i].headings.flatMap(h => h.toLowerCase().split(/\W+/)));
      const tokensB = new Set(pages[j].headings.flatMap(h => h.toLowerCase().split(/\W+/)));

      const similarity = computeJaccardSimilarity(tokensA, tokensB);
      // Similarity > 0.6 indicates heavy semantic overlap
      if (similarity > 0.6) {
        conflicts.push({ urlA: pages[i].url, urlB: pages[j].url, similarity });
      }
    }
  }

  return conflicts;
}

3. The 3 Technical Strategies to Fix Cannibalization

The 301 Redirect Consolidation: Merge two weak, overlapping articles into one comprehensive master guide, and permanently 301 redirect the deleted URL to transfer its historical backlink equity.
Canonical Pointing: If both pages must exist for user experience reasons, point an unambiguous rel="canonical" tag from the secondary page to the primary pillar page.
Internal Anchor Text Alignment: Ensure internal links pointing to Page A always use Page A's primary keyword, while links pointing to Page B use distinctly differentiated anchor phrases.

4. Explore the Open-Source Cannibalization Scanner

Audit your own site's URL graph with Plyxo:

👉 pixelfogg/Plyxo-CRO-SEO-AIO-AEO-GEO on GitHub

DEV Community