Find Broken Links and Redirect Chains in Your Sitemap with Python (Standard Library Only)

Your XML sitemap is supposed to be a list of pages you want indexed. In practice it drifts. A page gets deleted and the CMS keeps listing it. A slug changes, a redirect goes in, and the sitemap still points at the old URL. Someone adds a second redirect on top of the first, and now a crawler has to take two hops to reach the page. None of this shows up as an XML error, so a sitemap validator passes the file and the problems sit there.

This post builds a small sitemap URL status checker in Python using only the standard library (urllib, xml.etree, concurrent.futures, html.parser). It reads a sitemap or a sitemap index, requests every URL without letting urllib follow redirects on its own, walks each redirect chain hop by hop, and writes a CSV you can sort in a spreadsheet.

It flags:

• 4xx and 5xx responses on listed URLs
• listed URLs that redirect at all (a sitemap should list final URLs)
• redirect chains longer than one hop, and redirect loops
• redirects that end on a 4xx or 5xx page
• temporary redirects (302, 303, 307) inside a chain
• redirects whose target is not itself in the sitemap
• final pages whose rel="canonical" points at a different URL
• connection errors (refused connections, timeouts, DNS failures)

Tool descriptions for appie.run are as of 5 Nov 2026, checked on their pages that day. The script was tested on Python 3.13 against a local test site, shown below.

Disclosure: Appie.run publishes this article. Two free tools mentioned near the end are ours. The script does not depend on them.

Where this fits

Earlier posts in this series covered validating a sitemap's XML and comparing two sitemaps before and after a migration. The validator post ended with a short status-code loop that let urlopen follow redirects and compared the final URL. That tells you a redirect happened, but not how many hops it took, which codes were involved, or whether it looped. The comparison post looked at the first hop only. This script records the whole chain.

Why stop urllib from following redirects

urllib.request.urlopen follows redirects for you, up to 10 hops, and gives up with an error when it detects a loop. That is convenient for downloading a page and useless for auditing one, because the intermediate hops disappear. The fix is a redirect handler whose redirect_request returns None. urllib then raises an HTTPError for every 3xx response, and the error object still carries the status code and the Location header. The script reads those, resolves relative locations with urljoin, and requests the next hop itself.

The script

Save this as sitemap_status.py. It should run on Python 3.8 or later; we tested it on 3.13.

#!/usr/bin/env python3
"""Check every URL in an XML sitemap for broken links and redirect chains.

Standard library only. Usage:
    python3 sitemap_status.py https://example.com/sitemap.xml --out report.csv
"""
import argparse
import csv
import gzip
import sys
import threading
import time
import urllib.error
import urllib.request
import xml.etree.ElementTree as ET
from concurrent.futures import ThreadPoolExecutor
from html.parser import HTMLParser
from urllib.parse import urljoin, urldefrag

NS = "{http://www.sitemaps.org/schemas/sitemap/0.9}"
UA = "sitemap-status/1.0 (+https://example.com/contact)"


class NoRedirect(urllib.request.HTTPRedirectHandler):
    """Make urllib hand 3xx responses back to us instead of following them."""

    def redirect_request(self, req, fp, code, msg, headers, newurl):
        return None


OPENER = urllib.request.build_opener(NoRedirect)


def fetch_bytes(src, timeout):
    """Read a sitemap from a URL or a local file, unzipping .gz content."""
    if src.startswith(("http://", "https://")):
        req = urllib.request.Request(src, headers={"User-Agent": UA})
        with urllib.request.urlopen(req, timeout=timeout) as resp:
            data = resp.read()
    else:
        with open(src, "rb") as fh:
            data = fh.read()
    return gzip.decompress(data) if data[:2] == b"\x1f\x8b" else data


def sitemap_urls(src, timeout, seen=None, depth=0):
    """Return page URLs in sitemap order, following sitemap indexes."""
    seen = seen if seen is not None else set()
    if src in seen or depth > 3:
        return []
    seen.add(src)
    root = ET.fromstring(fetch_bytes(src, timeout))
    locs = [el.text.strip() for el in root.iter(NS + "loc") if el.text]
    if root.tag == NS + "sitemapindex":
        urls = []
        for child in locs:
            urls += sitemap_urls(child, timeout, seen, depth + 1)
        return urls
    if root.tag == NS + "urlset":
        return locs
    raise ValueError(f"{src}: root is {root.tag}, expected urlset or sitemapindex")


class CanonicalFinder(HTMLParser):
    """Grab the href of the first <link rel="canonical">."""

    def __init__(self):
        super().__init__()
        self.canonical = None

    def handle_starttag(self, tag, attrs):
        a = dict(attrs)
        if tag == "link" and self.canonical is None and \
                "canonical" in (a.get("rel") or "").lower().split():
            self.canonical = a.get("href")


class Throttle:
    """Space requests out across all worker threads."""

    def __init__(self, delay):
        self.delay, self.lock, self.next_at = delay, threading.Lock(), 0.0

    def wait(self):
        with self.lock:
            now = time.monotonic()
            sleep_for = max(0.0, self.next_at - now)
            self.next_at = max(now, self.next_at) + self.delay
        time.sleep(sleep_for)


def request(url, timeout, throttle):
    """GET one URL without following redirects. Returns (status, location, html)."""
    throttle.wait()
    req = urllib.request.Request(url, headers={"User-Agent": UA})
    try:
        with OPENER.open(req, timeout=timeout) as resp:
            html = ""
            if "html" in resp.headers.get("Content-Type", ""):
                html = resp.read(200_000).decode("utf-8", "replace")
            return resp.status, "", html
    except urllib.error.HTTPError as err:  # 3xx, 4xx and 5xx all land here
        return err.code, err.headers.get("Location", ""), ""
    except (urllib.error.URLError, TimeoutError, ConnectionError) as err:
        return 0, "", f"error: {getattr(err, 'reason', err)}"


def check(url, opts, throttle, listed):
    """Follow one URL's redirect chain and return a report row."""
    chain, codes, current, issues = [url], [], url, []
    for _ in range(opts.max_hops + 1):
        status, location, html = request(current, opts.timeout, throttle)
        codes.append(status)
        if 300 <= status < 400 and location:
            nxt = urldefrag(urljoin(current, location))[0]
            if nxt in chain:
                issues.append("redirect loop")
                chain.append(nxt)
                break
            chain.append(nxt)
            current = nxt
            continue
        break
    else:
        issues.append(f"redirect limit reached (--max-hops {opts.max_hops})")

    final = codes[-1]
    hops = len(chain) - 1
    if final == 0:
        issues.append(html)  # the connection error message
    elif 400 <= final < 500:
        issues.append(f"client error {final}")
    elif final >= 500:
        issues.append(f"server error {final}")
    elif 300 <= final < 400 and not issues:
        issues.append(f"{final} with no Location header")

    if hops:
        issues.append("listed URL redirects" if hops == 1 else f"redirect chain of {hops} hops")
        if any(c in (302, 303, 307) for c in codes[:hops]):
            issues.append("temporary redirect in chain")
        if final == 200 and chain[-1] not in listed:
            issues.append("redirect target not in sitemap")

    canonical = ""
    if final == 200 and html:
        finder = CanonicalFinder()
        finder.feed(html)
        if finder.canonical:
            canonical = urljoin(chain[-1], finder.canonical)
            if canonical != chain[-1]:
                issues.append("final URL canonicalises elsewhere")

    return {
        "url": url,
        "status": codes[0],
        "final_url": chain[-1],
        "final_status": final,
        "hops": hops,
        "chain": " -> ".join(f"{u} [{c}]" if i < len(codes) else u
                             for i, (u, c) in enumerate(zip(chain, codes + [0] * 2))),
        "canonical": canonical,
        "issues": "; ".join(issues) or "ok",
    }


def main():
    ap = argparse.ArgumentParser(description=__doc__)
    ap.add_argument("sitemap", help="sitemap or sitemap index: URL or local file")
    ap.add_argument("--out", default="sitemap-status.csv")
    ap.add_argument("--workers", type=int, default=4, help="parallel requests (default 4)")
    ap.add_argument("--delay", type=float, default=0.5,
                    help="seconds between request starts, across all workers (default 0.5)")
    ap.add_argument("--timeout", type=float, default=15)
    ap.add_argument("--max-hops", type=int, default=5)
    ap.add_argument("--limit", type=int, default=0, help="check only the first N URLs")
    opts = ap.parse_args()

    urls = list(dict.fromkeys(sitemap_urls(opts.sitemap, opts.timeout)))  # dedupe, keep order
    if opts.limit:
        urls = urls[:opts.limit]
    listed = set(urls)
    throttle = Throttle(opts.delay)
    print(f"checking {len(urls)} URLs with {opts.workers} workers", file=sys.stderr)

    with ThreadPoolExecutor(max_workers=opts.workers) as pool:
        rows = list(pool.map(lambda u: check(u, opts, throttle, listed), urls))

    fields = ["url", "status", "final_url", "final_status", "hops", "chain", "canonical", "issues"]
    with open(opts.out, "w", newline="", encoding="utf-8") as fh:
        writer = csv.DictWriter(fh, fieldnames=fields)
        writer.writeheader()
        writer.writerows(rows)

    problems = [r for r in rows if r["issues"] != "ok"]
    for r in problems:
        print(f"{r['status']:>3} {r['url']}  {r['issues']}")
    print(f"{len(rows) - len(problems)} ok, {len(problems)} with issues; report: {opts.out}")
    return 1 if problems else 0


if __name__ == "__main__":
    sys.exit(main())

What each part does

• fetch_bytes and sitemap_urls read a sitemap from a URL or a local file, unzip gzip content by checking the magic bytes, and follow <sitemapindex> files into their child sitemaps. A seen set and a depth limit stop an index that lists itself from looping. Duplicate URLs across child sitemaps are checked once.
• NoRedirect is the handler described above. Every 3xx comes back to us as an HTTPError with its Location header.
• request sends one GET with a descriptive User-Agent and a timeout. It reads up to 200 KB of the body only when the response is HTML, which is enough to find the canonical tag in the <head> without downloading whole pages. GET is used instead of HEAD because some servers answer HEAD differently from GET, and we need the HTML for the canonical check anyway.
• check walks the chain. Each new hop is compared against the URLs already in the chain, so A -> B -> A stops as a loop instead of running until the hop limit. If the chain is still redirecting after --max-hops follow-ups, it stops and says so.
• CanonicalFinder is a tiny HTMLParser subclass that keeps the first <link rel="canonical"> it sees. The href is resolved against the final URL, so relative canonicals work.
• Throttle spaces out request starts across all worker threads. With the defaults (4 workers, 0.5 seconds), the script starts at most two requests per second no matter how many threads are waiting. Workers mainly help when a slow response would otherwise hold up the queue.
• Exit code is 1 when any URL has an issue, so a CI job can fail on it.

Testing it against a local site

You should not test a crawler by pointing it at someone else's server. We built a tiny local site with http.server that has one of each problem, then ran the script against it. Save this as test_server.py:

"""Tiny local site for testing sitemap_status.py."""
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

BASE = "http://127.0.0.1:8765"
PAGE = '<html><head><link rel="canonical" href="{}"></head><body>hi</body></html>'

REDIRECTS = {
    "/old-pricing": (301, "/pricing"),           # one hop, fine target
    "/blog/a": (301, "/blog/b"),                 # chain: a -> b -> c
    "/blog/b": (302, "/blog/c"),
    "/loop-1": (301, "/loop-2"),                 # loop
    "/loop-2": (301, "/loop-1"),
    "/moved": (301, "/gone"),                    # redirect to a 404
    "/plans": (301, BASE + "/pricing/"),     # absolute Location, trailing slash variant
}
PAGES = {
    "/": "/",
    "/pricing": "/pricing",
    "/blog/c": "/blog/c",
    "/pricing/": "/pricing",                     # canonical points elsewhere
    "/print/about": "/about",                    # canonicalises to another URL
}

class H(BaseHTTPRequestHandler):
    def do_GET(self):
        p = self.path
        if p in ("/sitemap_index.xml", "/sitemap-pages.xml", "/sitemap-blog.xml"):
            body = open("." + p, "rb").read()
            self.send_response(200); self.send_header("Content-Type", "application/xml")
            self.end_headers(); self.wfile.write(body); return
        if p in REDIRECTS:
            code, loc = REDIRECTS[p]
            self.send_response(code); self.send_header("Location", loc); self.end_headers(); return
        if p in PAGES:
            body = PAGE.format(BASE + PAGES[p]).encode()
            self.send_response(200); self.send_header("Content-Type", "text/html; charset=utf-8")
            self.end_headers(); self.wfile.write(body); return
        if p == "/broken":
            self.send_response(500); self.end_headers(); return
        self.send_response(404); self.end_headers()

ThreadingHTTPServer(("127.0.0.1", 8765), H).serve_forever()

Then two child sitemaps and an index. sitemap-pages.xml lists /, /pricing, /old-pricing, /plans, /print/about, /moved, /broken and /missing. sitemap-blog.xml lists /blog/a, /blog/c and /loop-1. The index points at both:

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap><loc>http://127.0.0.1:8765/sitemap-pages.xml</loc></sitemap>
  <sitemap><loc>http://127.0.0.1:8765/sitemap-blog.xml</loc></sitemap>
</sitemapindex>

Start the server in one terminal and run the checker in another:

python3 test_server.py
python3 sitemap_status.py http://127.0.0.1:8765/sitemap_index.xml --out report.csv

Output:

checking 11 URLs with 4 workers
301 http://127.0.0.1:8765/old-pricing  listed URL redirects
301 http://127.0.0.1:8765/plans  listed URL redirects; redirect target not in sitemap; final URL canonicalises elsewhere
200 http://127.0.0.1:8765/print/about  final URL canonicalises elsewhere
301 http://127.0.0.1:8765/moved  client error 404; listed URL redirects
500 http://127.0.0.1:8765/broken  server error 500
404 http://127.0.0.1:8765/missing  client error 404
301 http://127.0.0.1:8765/blog/a  redirect chain of 2 hops; temporary redirect in chain
301 http://127.0.0.1:8765/loop-1  redirect loop; redirect chain of 2 hops
3 ok, 8 with issues; report: report.csv

The run took about 8 seconds for 11 sitemap URLs (17 page requests including redirect hops) at the default delay, and it exited with code 1.

Here is how each flagged row maps to the setup:

Listed URL	What the server does	What the script reports
/old-pricing	301 to /pricing, which is in the sitemap	listed URL redirects
/plans	301 to /pricing/, whose canonical is /pricing	redirects, target not in sitemap, canonicalises elsewhere
/print/about	200, canonical points at /about	final URL canonicalises elsewhere
/moved	301 to /gone, which returns 404	client error 404 after a redirect
/broken	500	server error 500
/missing	404	client error 404
/blog/a	301 to /blog/b, then 302 to /blog/c	chain of 2 hops, temporary redirect in chain
/loop-1	301 to /loop-2, which 301s back	redirect loop

The /plans row is the one people miss by eye. The redirect works and the page loads, but it lands on the trailing-slash variant of a page whose canonical says the real URL is /pricing. The fix is to list /pricing in the sitemap and point the redirect straight at it.

The CSV has one row per sitemap URL, with the full chain in one column:

url,status,final_url,final_status,hops,chain,canonical,issues
http://127.0.0.1:8765/blog/a,301,http://127.0.0.1:8765/blog/c,200,2,http://127.0.0.1:8765/blog/a [301] -> http://127.0.0.1:8765/blog/b [302] -> http://127.0.0.1:8765/blog/c [200],http://127.0.0.1:8765/blog/c,redirect chain of 2 hops; temporary redirect in chain
http://127.0.0.1:8765/loop-1,301,http://127.0.0.1:8765/loop-1,301,2,http://127.0.0.1:8765/loop-1 [301] -> http://127.0.0.1:8765/loop-2 [301] -> http://127.0.0.1:8765/loop-1,,redirect loop; redirect chain of 2 hops

We also checked three edge cases:

• Hop limit. With --max-hops 1, /blog/a stops after the second redirect and reports redirect limit reached (--max-hops 1); redirect chain of 2 hops; temporary redirect in chain.
• Local gzip file. A gzipped sitemap saved on disk (local.xml.gz) is read without any extra flag.
• Connection errors. A URL on a port with nothing listening is reported as status 0 with error: [Errno 111] Connection refused, and the run continues.

Reading the report

A few rules of thumb for the CSV:

• Fix 4xx and 5xx first. Either restore the page, redirect it, or remove it from the sitemap. A 5xx on a single run can be a blip, so rerun those rows before acting.
• Listed URL redirects. Replace the sitemap entry with the final_url. Your sitemap generator is usually the real fix, not the file.
• Chains. Point the first URL straight at the final one. Chains tend to build up across several migrations, so check old redirect rules too, not only the newest.
• Loops. These are always a configuration bug, often two rules (one adding a trailing slash, one removing it) fighting each other.
• Temporary redirects. If the move is permanent, use 301 or 308.
• Canonicalises elsewhere. List the canonical URL in the sitemap, or fix the canonical tag if it is the tag that is wrong.

Polite defaults and running it on a real site

The defaults are deliberately slow: 4 workers, a 0.5 second gap between request starts, a 15 second timeout and at most 5 follow-up hops. On your own site you can raise --workers and lower --delay if the server copes. On a client's site, ask first and keep the defaults. Change the contact URL in the UA string to something real, so the site owner can see who is crawling and reach you.

Use --limit for a first pass:

python3 sitemap_status.py https://www.example.com/sitemap_index.xml --limit 50 --out sample.csv

For large sites, check a sample from each child sitemap rather than every URL on every run.

Known limits

• It reads server HTML only. A canonical tag added by JavaScript after load will not be seen.
• Canonical comparison is exact. http versus https, a trailing slash or a different host all count as "elsewhere". That is usually what you want, but expect a few rows to need a human look.
• Canonical from HTTP headers is not checked. Some sites send Link: <...>; rel="canonical" as a header, mostly for PDFs. Add a check on resp.headers if you rely on that.
• No robots.txt handling. The script requests what the sitemap lists. If you point it at a site you don't run, read its robots.txt yourself.
• Status 0 covers several failures. The issues column has the actual error text.
• Rate limiting. If a site answers 429, slow down and rerun those rows; the script reports 429 as a client error and does not retry.

Free tools for the steps around the script

If you want to see what a sitemap contains before running anything:

• Sitemap URL Extractor (ours, free). Paste sitemap XML or enter a sitemap URL and it returns one URL per line from the urlset <loc> tags. Given a sitemap index, it lists the child sitemap locations; run it again on a child to get that file's page URLs.
• XML Sitemap Validator (ours, free). It checks the root element, absolute URLs, lastmod, changefreq, priority, duplicates and the 50,000 URL limit, and gives a score that starts at 100 and drops for errors and warnings.

Both ask for a captcha before each run. Neither requests the listed pages or follows redirects, which is the gap this script fills.

FAQ

Why not use requests or Scrapy?
They are good libraries. This script avoids dependencies so it runs in a minimal CI image or on a client's server with nothing to install.

Why not let urllib follow redirects and compare the final URL?
You lose the chain. You can see that a URL moved, but not whether it took one hop or four, which codes were used, or whether it looped.

Is a single redirect in a sitemap a problem?
It is not an error, but the sitemap should list the final URL. Search engines have to make an extra request for every listed URL that redirects.

Does it check links inside pages?
No. It checks the URLs listed in the sitemap. For links inside page content you need a crawler.

Can I run it in CI?
Yes. It exits with 1 when any URL has an issue. Run it against staging after deploys, with a --limit if the sitemap is large.

Why does my redirect target show "not in sitemap" when the page is fine?
The target URL is not listed exactly as written in the sitemap. Often it is a trailing slash or http versus https difference. Either list the target or point the redirect at the URL you do list.

Wrap-up

Broken links and redirect chains in a sitemap are cheap to find and easy to fix once you have the list. Run the script after migrations, CMS changes and redirect rule edits. If you want the URL list or a protocol check in a browser first, the Appie free tools cover both, and Appie's apps start on a Free plan with 500 credits and no credit card.

RG, Founder, Appie.run