How to check a sitemap for broken links (with a short Python script)

#seo #python #webdev #tutorial

Your XML sitemap is a list of pages you want search engines to crawl. If it contains pages that no longer exist, pages that redirect, or pages you have blocked, it stops being a clean list. Here is a quick way to check yours.

What should be in a sitemap

Only canonical pages that return a normal 200 status and that you want in search results. Leave out 404 and 410 pages, redirected URLs, pages with noindex, pages blocked in robots.txt and duplicate versions of the same page. One sitemap file can hold up to 50,000 URLs or 50 MB uncompressed.

Step 1: list every URL

Open the sitemap, or use a tool that extracts the URLs. If you use a sitemap index, remember that it points to other sitemap files, so you need to read those too. The script below does that.

Step 2: check the status of each URL

This script reads a sitemap (or a sitemap index), then asks each URL for its status without following redirects. Only run it on your own site, or with permission, and keep the delay so you do not overload the server.

import time
import requests
import xml.etree.ElementTree as ET

SITEMAP = "https://example.com/sitemap.xml"
NS = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}

def urls_from(sitemap_url):
    root = ET.fromstring(requests.get(sitemap_url, timeout=15).content)
    if root.tag.endswith("sitemapindex"):
        for loc in root.findall("sm:sitemap/sm:loc", NS):
            yield from urls_from(loc.text.strip())
    else:
        for loc in root.findall("sm:url/sm:loc", NS):
            yield loc.text.strip()

for url in urls_from(SITEMAP):
    r = requests.get(url, allow_redirects=False, timeout=15)
    if r.status_code != 200:
        print(r.status_code, url, r.headers.get("Location", ""))
    time.sleep(0.3)

Anything printed is a URL that does not return a plain 200.

Step 3: decide what to do

Status	Meaning	What to do
200	Page works	Keep it
301 or 308	Moved permanently	Put the final address in the sitemap
302, 303 or 307	Temporary move	Check whether it should be permanent
404 or 410	Page is gone	Remove it, or restore the page if it should exist
5xx	Server error	Fix the server problem and check again

Step 4: update and resubmit

Remove or replace the bad URLs, upload the new sitemap and submit it again in Google Search Console. Google Search Console's Pages report also lists problems such as "Submitted URL not found (404)", so check it too.

Keep it clean

Generate the sitemap automatically so deleted pages drop out of it, and run this check after every redesign or migration.

If you want a no-code version, our free Sitemap URL Extractor lists every URL in a sitemap, and the Sitemap Validator checks the file itself. The full guide, with more detail, is on seovoro.com.

DEV Community