Find Sitemap URLs with Conflicting Canonicals

A sitemap lists /guides/, but the page declares /resources/guides/ as its canonical URL. Which address should your publishing pipeline keep? Audit the sitemap entry, the final URL after redirects, and the HTML canonical separately. Report the disagreement; then a human decides which URL represents the page.

Google recommends putting the URLs you want shown as canonical in a sitemap and avoiding contradictory canonical signals. A sitemap entry and an HTML rel="canonical" are signals, though: the checker below finds inconsistent declarations. It cannot declare which URL Google will choose. See Google's sitemap guidance and canonical methods.

AI disclosure: This article was drafted with AI. The author must verify its technical claims and code before publication and select the truthful DEV editor disclosure tier.

What exactly are we checking?

For every <loc> in one XML URL sitemap, compare three values:

Value	What it tells us
Sitemap URL	The URL the publisher submitted as a canonical candidate.
Final response URL	The URL reached after ordinary HTTP redirects.
HTML canonical	The URL named by a <link rel="canonical" href="…"> in the returned HTML.

For example, if a sitemap includes https://example.test/a and the fetched HTML points to https://example.test/b, the script prints CONFLICT. If /a redirects to /b, it also prints REDIRECTED. Neither output is proof of a search penalty; both indicate a place to examine your routing and page-generation rules.

Run a bounded local audit

Save the following as audit_canonicals.py. Pass it a trusted local sitemap XML file, not a remote URL. It checks at most 200 entries, uses an eight-second request timeout, and reads only the first 512 KB of each HTML response. Run it only against a site you are authorized to inspect. Python 3.12 needs no third-party package.

from html.parser import HTMLParser
from pathlib import Path
from urllib.error import HTTPError, URLError
from urllib.parse import urljoin, urlsplit, urlunsplit
from urllib.request import Request, urlopen
import sys
import xml.etree.ElementTree as ET




def local_name(tag):
    return tag.rsplit("}", 1)[-1]




def sitemap_urls(path):
    root = ET.parse(path).getroot()
    if local_name(root.tag) != "urlset":
        raise ValueError("Provide a URL sitemap, not a sitemap index")
    for entry in root:
        if local_name(entry.tag) == "url":
            for child in entry:
                if local_name(child.tag) == "loc" and child.text:
                    yield child.text.strip()
                    break




class CanonicalLinks(HTMLParser):
    def __init__(self):
        super().__init__()
        self.hrefs = []


    def handle_starttag(self, tag, attrs):
        if tag.lower() != "link":
            return
        values = dict(attrs)
        if "canonical" in (values.get("rel") or "").lower().split():
            if values.get("href"):
                self.hrefs.append(values["href"])




def comparable(url):
    parts = urlsplit(url)
    return urlunsplit((parts.scheme.lower(), parts.netloc.lower(),
                       parts.path or "/", parts.query, ""))




def audit(path):
    urls = list(sitemap_urls(path))
    if len(urls) > 200:
        raise ValueError("Split the sitemap or adjust the audited limit")


    for url in urls:
        if urlsplit(url).scheme not in ("http", "https"):
            print("INVALID_URL", url, sep="\t")
            continue
        try:
            request = Request(url, headers={"User-Agent": "CanonicalAudit/1.0"})
            with urlopen(request, timeout=8) as response:
                final_url = response.geturl()
                if "text/html" not in response.headers.get("Content-Type", ""):
                    print("REVIEW_NON_HTML", url, sep="\t")
                    continue
                body = response.read(512_000)
        except (HTTPError, URLError, TimeoutError) as exc:
            print("FETCH_ERROR", url, type(exc).__name__, sep="\t")
            continue


        parser = CanonicalLinks()
        parser.feed(body.decode("utf-8", errors="replace"))
        canonicals = [comparable(urljoin(final_url, href))
                      for href in parser.hrefs]
        notes = []
        if comparable(url) != comparable(final_url):
            notes.append("REDIRECTED")
        if not canonicals:
            notes.append("REVIEW_NO_HTML_CANONICAL")
        elif len(set(canonicals)) > 1:
            notes.append("REVIEW_MULTIPLE_CANONICALS")
        elif canonicals[0] != comparable(url):
            notes.append("CONFLICT")
        print("|".join(notes) or "OK", url, sep="\t")




if __name__ == "__main__":
    audit(Path(sys.argv[1]))

Run python3 audit_canonicals.py sitemap.xml. A sitemap entry with an explicit matching HTML canonical prints OK. A missing HTML canonical prints REVIEW_NO_HTML_CANONICAL, which is a review item, not an automatic error. A redirect and mismatch can appear together.

For a small local check, I ran the script against a temporary HTTP server with these fixtures:

Fixture path	Returned page behavior	Observed classification
/ok	Canonical points to /ok.	OK
/wrong	Canonical points to /other.	CONFLICT
/missing	No HTML canonical tag.	REVIEW_NO_HTML_CANONICAL
/redirect	Redirects to /ok; sitemap still lists /redirect.	`REDIRECTED
{% raw %}/multiple	Two different HTML canonicals.	REVIEW_MULTIPLE_CANONICALS

The five outputs matched the expected classifications under Python 3.12. This is a test of the script's branch behavior, not proof that a real crawler interprets every live site in the same way. Recreate equivalent fixtures for your site's redirect and template rules before running a larger audit.

The comparison removes URL fragments and lowercases the scheme and host. It deliberately preserves paths and query strings: changing either can identify a different resource. Review harmless URL normalization differences rather than rewriting them blindly.

Verify a reported conflict before fixing it

Take one CONFLICT row and answer four questions: Did the submitted URL redirect? Is the HTML canonical absolute or relative? Is the destination page actually the preferred version? Does the sitemap generator use the same URL rule as the page renderer? Fix the responsible generator or redirect rule, then rerun the audit.

This small script reads one uncompressed URL sitemap, follows normal redirects, and scans the fetched HTML only. It does not parse sitemap indexes, render JavaScript, inspect Link HTTP headers, enforce robots.txt, or tell you what Google selected. It also uses a simplified UTF-8 decode; adapt it for a site's declared charset. Large sites should add rate limiting, host allowlisting, index traversal, and respectful scheduling before running a broader audit.

The useful output is a review queue with reasons, not a green SEO score. A person can inspect the URL pair, correct conflicting publication signals, and verify the intended result with the appropriate search tools.