DEV Community

Hanin Mp
Hanin Mp

Posted on Edited on

Is your sitemap lastmod telling the truth? Audit it against server headers in Python

#seo #webdev #python #tutorial

Most sitemaps I look at have a lastmod on every URL. Fewer have a lastmod that means anything.

Google has said it uses lastmod only when it is consistently and verifiably accurate. If every URL carries the date of the last sitemap build, the signal is noise, and crawlers are free to ignore it. So before arguing about crawl budget, it is worth checking whether your dates are honest.

This post builds a small audit with the Python standard library. No pip install, no API keys.

What the audit checks

  1. Distribution. If one date covers most of the URLs, the sitemap is probably being bulk-stamped by a plugin or a deploy job.
  2. Today and the future. Dates equal to the fetch date on many URLs, or dates in the future, are both red flags.
  3. Corroboration. Where the server sends a Last-Modified header, the sitemap should not claim a change that is much newer than the header says.

The third check has a limit worth stating plainly: plenty of CMSs and CDNs send no Last-Modified header at all. The script reports that as "no header to compare" instead of guessing.

The script 

#!/usr/bin/env python3
"""Audit sitemap <lastmod> values against what the server says. Standard library only.

Usage: python lastmod_audit.py https://example.com/sitemap.xml [--limit 40] [--slack-days 2]
"""
import argparse
import sys
import urllib.request
import xml.etree.ElementTree as ET
from collections import Counter
from datetime import datetime, timedelta, timezone
from email.utils import parsedate_to_datetime

NS = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}
UA = {"User-Agent": "lastmod-audit/1.0"}


def fetch(url, method="GET", timeout=15):
    req = urllib.request.Request(url, headers=UA, method=method)
    return urllib.request.urlopen(req, timeout=timeout)


def parse_dt(value):
    """Parse W3C datetime from a sitemap. Date-only values count as midnight UTC."""
    value = value.strip()
    try:
        dt = datetime.fromisoformat(value.replace("Z", "+00:00"))
    except ValueError:
        return None
    return dt if dt.tzinfo else dt.replace(tzinfo=timezone.utc)


def read_sitemap(url, depth=0):
    root = ET.fromstring(fetch(url).read())
    rows = []
    if root.tag.endswith("sitemapindex") and depth == 0:
        for loc in root.findall("sm:sitemap/sm:loc", NS)[:5]:
            rows += read_sitemap(loc.text.strip(), depth + 1)
        return rows
    for u in root.findall("sm:url", NS):
        loc = u.find("sm:loc", NS)
        lm = u.find("sm:lastmod", NS)
        if loc is not None:
            rows.append((loc.text.strip(), parse_dt(lm.text) if lm is not None and lm.text else None))
    return rows


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("sitemap")
    ap.add_argument("--limit", type=int, default=40)
    ap.add_argument("--slack-days", type=int, default=2)
    a = ap.parse_args()

    rows = read_sitemap(a.sitemap)
    print(f"URLs in sitemap: {len(rows)}")
    dated = [(u, d) for u, d in rows if d]
    print(f"With lastmod: {len(dated)} ({len(dated) / max(len(rows), 1):.0%})")
    if not dated:
        sys.exit("No lastmod values to audit.")

    # 1. Distribution checks: a lastmod that never varies carries no information.
    days = Counter(d.date() for _, d in dated)
    top_day, top_n = days.most_common(1)[0]
    print(f"Most common lastmod date: {top_day} on {top_n} URLs ({top_n / len(dated):.0%})")
    today = datetime.now(timezone.utc).date()
    stamped_today = sum(1 for _, d in dated if d.date() == today)
    future = [u for u, d in dated if d > datetime.now(timezone.utc) + timedelta(days=1)]
    print(f"Stamped with today's date: {stamped_today}; in the future: {len(future)}")
    if top_n / len(dated) > 0.8 and len(dated) > 10:
        print("FLAG: one date covers most URLs. Looks bulk-stamped, so crawlers may learn to ignore it.")

    # 2. Per-URL check against Last-Modified. Only compare what the server actually sends.
    slack = timedelta(days=a.slack_days)
    checked = no_header = overclaim = 0
    for url, lm in dated[: a.limit]:
        try:
            r = fetch(url, "HEAD")
        except Exception as e:
            print(f"  skip {url}: {e}")
            continue
        checked += 1
        h = r.headers.get("Last-Modified")
        if not h:
            no_header += 1
            continue
        hdr = parsedate_to_datetime(h)
        if lm - hdr > slack:
            overclaim += 1
            print(f"  OVERCLAIM {url}: sitemap {lm.date()} vs header {hdr.date()}")
    print(f"Checked {checked}: no Last-Modified header on {no_header}, sitemap newer than header on {overclaim}")
    if no_header == checked:
        print("NOTE: no header to compare. Use a content hash or CMS revision date instead.")


if __name__ == "__main__":
    main()
Enter fullscreen mode Exit fullscreen mode

Run it with:

python lastmod_audit.py https://example.com/sitemap.xml --limit 40
Enter fullscreen mode Exit fullscreen mode

What I tested

I ran it against a local fixture server, not a live site: 14 URLs, 12 stamped with today's date and 2 with a date 60 days old. Three of those URLs sent a Last-Modified header 90 days in the past. The script printed the bulk-stamp flag (86% on one date), listed the three overclaims, and reported 11 URLs with no header. That is the behaviour I wanted, and it is the only claim I am making about the output.

How to read the results

  • Bulk-stamp flag, no overclaims: The sitemap generator is probably writing the build date. Point it at a real modified date from your CMS (post_modified in WordPress, updated_at in most headless CMSs).
  • Overclaims: Either the sitemap is wrong or the header is cached at the CDN. Check which one is stale before changing anything.
  • No headers at all: The script cannot corroborate anything. Compare against the CMS revision date, or hash the main content and store the hash with a date.
  • Date-only values: These are valid in a sitemap. I treat them as midnight UTC, so keep the slack at 1 or 2 days to avoid false overclaims from timezones.

Limits

  • It samples the first N dated URLs, not a random sample. For big sites, shuffle the list first.
  • It only follows one level of sitemap index and the first five child sitemaps.
  • A matching Last-Modified header does not prove the content changed in a meaningful way. Updating a footer year should not move lastmod.

A lastmod you can defend is worth more than a lastmod on every URL. If your dates fail this audit, fix the source of the date before you resubmit the sitemap.

More in this series

This is part of Python checks for technical SEO, small standard-library scripts for audits you can run in a minute:

What do you use as the source of truth for lastmod on your sites: CMS revision date, file mtime, or a content hash?

Top comments (0)