DEV Community

Hanin Mp
Hanin Mp

Posted on

Do your canonical signals agree? Audit the HTML tag, the HTTP header and the final URL in Python

#seo #webdev #python #tutorial

A canonical tag is a hint, not a command. Google weighs it against redirects, internal links, sitemaps and hreflang. When those signals disagree, Google picks its own canonical, and that is where "Duplicate, Google chose different canonical than user" in Search Console comes from.

Most audits only look at the <link rel="canonical"> tag. In practice the conflicts hide between three places:

  1. The final URL after redirects
  2. The <link rel="canonical"> in the HTML
  3. The Link: <...>; rel="canonical" HTTP header (common on PDFs and CDN-managed pages)

This script compares all three for one URL, using only the Python standard library.

What it flags

  • No canonical anywhere
  • Multiple conflicting canonical tags (plugins fighting each other is the usual cause)
  • A canonical placed in <body>, which Google ignores
  • HTML canonical and HTTP header canonical that disagree
  • A canonical pointing to a different URL than the one served
  • A canonical on http instead of https
  • A canonical containing a query string

The script 

#!/usr/bin/env python3
"""Check whether a URL's canonical signals agree with each other.

Usage: python3 canonical_check.py https://example.com/page
"""
import sys
import urllib.request
from html.parser import HTMLParser
from urllib.parse import urljoin, urlsplit, urlunsplit


class CanonicalParser(HTMLParser):
    def __init__(self):
        super().__init__()
        self.canonicals = []
        self.in_head = True

    def handle_starttag(self, tag, attrs):
        if tag == "body":
            self.in_head = False
        if tag != "link":
            return
        a = dict(attrs)
        rels = (a.get("rel") or "").lower().split()
        if "canonical" in rels and a.get("href"):
            self.canonicals.append((a["href"].strip(), self.in_head))


def header_canonical(link_header):
    """Pull the canonical URL out of a Link header, if present."""
    if not link_header:
        return None
    for part in link_header.split(","):
        if 'rel="canonical"' in part or "rel=canonical" in part:
            start, end = part.find("<"), part.find(">")
            if start != -1 and end > start:
                return part[start + 1:end].strip()
    return None


def normalise(url):
    """Lowercase host, drop fragment and a lone trailing slash on the path."""
    p = urlsplit(url)
    path = p.path.rstrip("/") or "/"
    return urlunsplit((p.scheme, p.netloc.lower(), path, p.query, ""))


def analyse(final_url, html, link_header):
    parser = CanonicalParser()
    parser.feed(html)
    issues = []

    html_tags = [urljoin(final_url, href) for href, _ in parser.canonicals]
    outside_head = [h for h, in_head in parser.canonicals if not in_head]
    header_val = header_canonical(link_header)
    header_val = urljoin(final_url, header_val) if header_val else None

    if not html_tags and not header_val:
        issues.append("No canonical declared in HTML or HTTP header")
    if len(set(map(normalise, html_tags))) > 1:
        issues.append("Multiple conflicting canonical tags in HTML")
    if outside_head:
        issues.append("Canonical found in <body>, which Google ignores")
    if html_tags and header_val and normalise(html_tags[0]) != normalise(header_val):
        issues.append("HTML canonical and HTTP header canonical disagree")

    chosen = html_tags[0] if html_tags else header_val
    if chosen and normalise(chosen) != normalise(final_url):
        issues.append("Canonical points to a different URL than the one served")
    if chosen and urlsplit(chosen).scheme == "http":
        issues.append("Canonical uses http, not https")
    if chosen and urlsplit(chosen).query:
        issues.append("Canonical contains a query string")

    return {
        "final_url": final_url,
        "html_canonicals": html_tags,
        "header_canonical": header_val,
        "issues": issues,
    }


def fetch(url):
    req = urllib.request.Request(url, headers={"User-Agent": "canonical-check/1.0"})
    with urllib.request.urlopen(req, timeout=15) as resp:
        html = resp.read(500_000).decode("utf-8", errors="replace")
        return resp.geturl(), html, resp.headers.get("Link")


if __name__ == "__main__":
    if len(sys.argv) != 2:
        sys.exit("Usage: python3 canonical_check.py <url>")
    final_url, html, link = fetch(sys.argv[1])
    result = analyse(final_url, html, link)
    print("Final URL:       ", result["final_url"])
    print("HTML canonical:  ", result["html_canonicals"] or "none")
    print("Header canonical:", result["header_canonical"] or "none")
    print("Issues:" if result["issues"] else "No issues found.")
    for issue in result["issues"]:
        print(" -", issue)
Enter fullscreen mode Exit fullscreen mode

How to read the output

Run it against a parameter URL such as https://example.com/shoes?utm_source=mail. If the final URL still carries the parameter and the canonical points to /shoes, that is healthy: the canonical is doing its job.

The two results that need action:

  • Canonical differs from the served URL on a page that should rank. A product page that canonicalises to its category is the classic case. Google will usually follow it, and the product page drops out of the index.
  • HTML and header disagree. One layer (often a CDN or a plugin) is rewriting the other. Pick one source of truth and remove the second.

Limits worth knowing

  • It reads the raw HTML, so a canonical injected by JavaScript will not show up. Compare with the rendered DOM before deciding it is missing.
  • It checks one URL at a time. Loop it over a sitemap export for a site-wide view, and keep the request rate polite.
  • A matching canonical does not guarantee Google agrees. Confirm the result with the URL Inspection tool in Search Console.

More checks in this series

This is part 8 of my Python checks for technical SEO. Related posts:

What is the most surprising canonical conflict you have found on a live site?

Top comments (0)