How to audit your XML sitemap for indexing problems

Disclosure: I build and sell the paid Apify Actor mentioned near the end. Everything before that section works with free tools. This article was drafted with AI assistance; the code and examples were tested.

Your XML sitemap is supposed to be a list of pages you want indexed. In practice it often turns into a list of pages that redirect, return 404, carry a noindex tag, or tell Google that some other URL is the real one. Each of those sends a mixed signal. When Search Console shows a long "Discovered – currently not indexed" or "Crawled – currently not indexed" list, the sitemap is one of the first places to look.

This tutorial walks through a full sitemap audit:

1. Find every sitemap the site exposes.
2. Expand sitemap indexes and gzipped files into one URL list.
3. Check each URL for the problems that contradict "please index this".
4. Fix them at the source, usually the CMS or the sitemap generator.

Step 1: Find the sitemaps

Start with robots.txt. The Sitemap: directive can appear anywhere in the file, more than once, and it doesn't have to point to the same host:

curl -s https://example.com/robots.txt | grep -i '^sitemap:'

If there's no Sitemap: line, try the usual locations:

• /sitemap.xml
• /sitemap_index.xml (Yoast and many other plugins)
• /wp-sitemap.xml (WordPress core since 5.5)

Also check Search Console under Indexing → Sitemaps to see which files were actually submitted. A missing Sitemap: line in robots.txt isn't an error, but adding it helps other crawlers find the file.

Step 2: Expand everything into one URL list

A sitemap is either a <urlset> (a list of pages) or a <sitemapindex> (a list of other sitemaps). Large sites nest them and often gzip them (.xml.gz). The protocol limits each file to 50,000 URLs and 50 MB uncompressed, so big sites end up with dozens of files.

For each URL, keep:

• loc, the URL itself
• lastmod, if present
• which sitemap file it came from, so you know where to fix it

Watch for duplicates across files. The same URL listed in two sitemaps usually means two generators are running, for example a plugin plus a custom script.

Step 3: Decide what counts as a problem

A URL in a sitemap should pass all of these:

Check	Why it matters
Returns 200 without redirecting	A redirecting URL isn't the canonical one. List the final URL instead.
Not 4xx/5xx	You're asking Google to index a page that doesn't exist.
No noindex in <meta name="robots"> or the X-Robots-Tag header	It directly contradicts the sitemap. The header version is easy to miss because it's invisible in the HTML.
rel="canonical" points to itself	If it points elsewhere, you're submitting a duplicate.
Not disallowed in robots.txt	Google can't crawl it to see the content.
lastmod is a valid W3C date and not in the future	Google has said it uses lastmod only when it's consistently accurate. A bogus date trains it to ignore yours.
HTTPS and on the right host	http:// or staging-host URLs in a production sitemap are common after migrations.

Two judgment calls:

• Stale lastmod (say, older than a year) isn't an error. It's worth reviewing on content-heavy sites, because it often means the date is frozen or the page is neglected.
• Canonical comparison should resolve relative canonicals and ignore trailing-slash differences, or you'll get false positives.

Step 4: A script that does all of the above

Here's a compact, tested Python version (needs pip install httpx). It finds the sitemaps, follows nested indexes and gzip, and audits the first N URLs:

import gzip, re, sys
import xml.etree.ElementTree as ET
from datetime import datetime, timezone
from urllib.parse import urljoin, urlsplit
from urllib.robotparser import RobotFileParser
import httpx

UA = "sitemap-audit-script (contact: [e-mail])"
NS = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}
client = httpx.Client(headers={"User-Agent": UA}, timeout=20, follow_redirects=False)

def find_sitemaps(site):
    robots_url = f"https://{site}/robots.txt"
    r = client.get(robots_url)
    found = re.findall(r"(?im)^\s*sitemap:\s*(\S+)", r.text) if r.status_code == 200 else []
    if not found:
        for path in ("/sitemap.xml", "/sitemap_index.xml", "/wp-sitemap.xml"):
            if client.get(urljoin(f"https://{site}", path)).status_code == 200:
                found.append(urljoin(f"https://{site}", path))
                break
    return robots_url, found

def read_sitemap(url, seen=None, limit=200):
    seen = seen if seen is not None else set()
    if url in seen or len(seen) >= limit:
        return []
    seen.add(url)
    r = client.get(url, follow_redirects=True)
    body = r.content
    if url.endswith(".gz") or body[:2] == b"\x1f\x8b":
        body = gzip.decompress(body)
    root = ET.fromstring(body)
    if root.tag.endswith("sitemapindex"):
        out = []
        for loc in root.findall("sm:sitemap/sm:loc", NS):
            out += read_sitemap(loc.text.strip(), seen, limit)
        return out
    return [(u.findtext("sm:loc", "", NS).strip(), u.findtext("sm:lastmod", None, NS), url)
            for u in root.findall("sm:url", NS)]

def audit(url, lastmod, robots, site):
    issues = []
    if urlsplit(url).scheme != "https": issues.append("http-not-https")
    if not urlsplit(url).netloc.endswith(site): issues.append("off-domain-url")
    if not robots.can_fetch("Googlebot", url): issues.append("blocked-by-robots-txt")
    if lastmod:
        try:
            d = datetime.fromisoformat(lastmod.replace("Z", "+00:00"))
            if d.tzinfo is None: d = d.replace(tzinfo=timezone.utc)
            if d > datetime.now(timezone.utc): issues.append("future-lastmod")
        except ValueError:
            issues.append("invalid-lastmod")
    try:
        r = client.get(url)
    except httpx.HTTPError:
        return issues + ["unreachable"]
    if r.status_code in (301, 302, 307, 308):
        return issues + ["redirected"]
    if r.status_code >= 400:
        return issues + [f"http-{r.status_code}"]
    head = r.text[:200_000]
    robots_meta = " ".join(re.findall(r'<meta[^>]+name=["\']robots["\'][^>]*content=["\']([^"\']+)', head, re.I))
    if "noindex" in (robots_meta + r.headers.get("x-robots-tag", "")).lower():
        issues.append("noindex")
    m = re.search(r'<link[^>]+rel=["\']canonical["\'][^>]*href=["\']([^"\']+)', head, re.I)
    if m and urljoin(url, m.group(1)).rstrip("/") != url.rstrip("/"):
        issues.append("canonical-mismatch")
    return issues

if __name__ == "__main__":
    site = sys.argv[1]
    robots_url, sitemaps = find_sitemaps(site)
    rp = RobotFileParser(robots_url); rp.read()
    entries = []
    for sm in sitemaps:
        entries += read_sitemap(sm)
    seen = set()
    print(f"{len(sitemaps)} sitemap(s), {len(entries)} URLs")
    for url, lastmod, src in entries[:int(sys.argv[2]) if len(sys.argv) > 2 else 50]:
        dup = url in seen; seen.add(url)
        issues = audit(url, lastmod, rp, site) + (["duplicate-in-sitemaps"] if dup else [])
        if issues:
            print(url, issues)

Run it:

python sitemap_audit.py example.com 200

When I ran it against wordpress.org, it found 9 sitemap files, and the first audited URLs it flagged were /enterprise/... pages carrying noindex while still being listed in the sitemap. That's exactly the kind of contradiction this audit is meant to surface. (Whether those pages should be noindexed is a separate decision. The point is that the two signals disagree.)

Limitations of this script, so you know what to extend:

• It audits sequentially. For thousands of URLs, add async requests with low concurrency (around 5) and a clear User-Agent.
• It reads every sitemap file before auditing. On very large sites (wordpress.org lists hundreds of thousands of URLs), cap the number of files.
• Regex HTML parsing is fine for <head> tags in served HTML but isn't a real parser. It also doesn't run JavaScript, so canonicals or robots tags injected client-side won't be seen.
• It checks robots.txt rules for Googlebot only.

Step 5: Fix problems at the source

Patching the XML by hand rarely sticks, because the generator overwrites it. Instead:

• Redirected URLs: usually an http→https, trailing-slash, or www mismatch between the CMS "site URL" setting and the real canonical host. Fix the setting, and every URL in the generated sitemap changes with it.
• 404s: deleted posts or products that the generator still lists. Typical causes are caching plugins or a static sitemap file left behind from an old setup (check that /sitemap.xml isn't a stale file sitting in the web root).
• noindex pages in the sitemap: most SEO plugins exclude noindexed content automatically. If they don't, a custom template or a per-page header is overriding them. Check the X-Robots-Tag header at the server or CDN level too.
• Canonical mismatches: often paginated, filtered, or parameterized URLs that canonicalize to a parent. Remove them from the sitemap rather than changing the canonical.
• Future or invalid lastmod: usually a timezone bug or a "now()" default. Use the real content modification date.

After fixing, resubmit the sitemap in Search Console and re-run the audit. Then schedule it, because sitemaps drift every time someone installs a plugin or migrates a section.

Step 6: Automating it (optional)

If you'd rather not maintain the script, I packaged this workflow as an Apify Actor: Sitemap Extractor & SEO Index Audit. It's paid (pay-per-event), so here's exactly what it does:

• finds sitemaps via robots.txt Sitemap: lines, then falls back to /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml and other common paths
• follows nested indexes and .xml.gz, and keeps lastmod, changefreq, priority, hreflang alternates and image/video counts
• flags http-404/http-5xx, redirected, noindex (meta or header), canonical-mismatch, blocked-by-robots-txt, unreachable, stale-lastmod/future-lastmod/invalid-lastmod, duplicate-in-sitemaps, http-not-https and off-domain-url
• writes an ISSUES CSV of only the problem URLs, plus a SUMMARY with a health score (the percentage of audited URLs that are indexable)

Pricing from the README: $0.30 per 1,000 extracted URLs and $1.20 per 1,000 audited pages, so a full audit of 1,000 pages is about $1.50. You can set a max cost per run. Like the script above, it doesn't execute JavaScript.

If you just migrated a site, it pairs with my earlier post, How to find and fix lost pages after a website migration. That one covers the old URLs that dropped out of the sitemap. This one covers the URLs that are in it but shouldn't be.

Checklist

• [ ] All sitemap files found (robots.txt, common paths, Search Console)
• [ ] Indexes and gzip expanded, duplicates found
• [ ] Every URL: 200 without redirect, no noindex (meta and header), self-canonical, crawlable
• [ ] lastmod valid and honest
• [ ] Fixes made in the generator or CMS, not by hand-editing the XML
• [ ] Resubmitted and scheduled to re-run