How to find broken links and redirects hiding in your XML sitemap (with Python)

#seo #python #webdev #tutorial

Google expects your sitemap to list only live, canonical URLs. After a migration or a product cleanup, sitemaps quietly fill up with 404s and 301s, and Search Console starts complaining. Here's how to audit one in ~40 lines of Python.

1. Find the sitemaps

Start with robots.txt - most sites list their sitemaps there:

import re, requests
def sitemaps_from_robots(origin):
    txt = requests.get(f"{origin}/robots.txt", timeout=20).text
    found = re.findall(r"(?im)^\s*sitemap:\s*(\S+)", txt)
    return found or [f"{origin}/sitemap.xml"]

2. Walk sitemap indexes and collect URLs

import gzip, xml.etree.ElementTree as ET
NS = {"s": "http://www.sitemaps.org/schemas/sitemap/0.9"}
def urls_from_sitemap(url, depth=0):
    body = requests.get(url, timeout=30).content
    if url.endswith(".gz"): body = gzip.decompress(body)
    root = ET.fromstring(body)
    if root.tag.endswith("sitemapindex") and depth < 5:
        for loc in root.findall("s:sitemap/s:loc", NS):
            yield from urls_from_sitemap(loc.text.strip(), depth + 1)
    else:
        for u in root.findall("s:url", NS):
            yield u.findtext("s:loc", namespaces=NS).strip(), u.findtext("s:lastmod", namespaces=NS)

3. Check every URL politely

def check(url):
    r = requests.head(url, allow_redirects=False, timeout=20)
    if r.status_code in (405, 501):  # some servers don't support HEAD
        r = requests.get(url, allow_redirects=False, timeout=20, stream=True)
    return r.status_code, r.headers.get("location")

Keep concurrency low (2-3 requests at a time per site), respect robots.txt disallow rules, and write rows with status >= 400 or a location header to a CSV. Those are the entries to fix or remove.

Edge cases you'll hit

Sitemap indexes nested several levels deep, .xml.gz files, and plain-text sitemaps.
robots.txt that returns an HTML error page with status 200.
Servers that answer HEAD with 404/405 but GET with 200.
Huge sitemaps (50k URLs/50 MB per file) - stream or cap them.

If you don't want to maintain it

Disclosure: I built a hosted version of exactly this, Sitemap Scraper on Apify, which handles the edge cases above, runs many sites at once and exports to CSV/Sheets ($0.20 per 1,000 URLs, $0.50 per 1,000 status checks). The script above works fine for one site too.

DEV Community