Your sitemap's lastmod is lying — one CSS tweak marked all 380 pages as updated

Last winter I added one line to a stylesheet on my site — letter-spacing: -0.01em on a heading — and redeployed. Nothing else changed. And yet, according to my sitemap, all 380 pages had been updated that day.

bingbot, being reasonable, went off to recrawl 380 pages. Almost none of them had changed.

This is the quiet way a static site wastes its crawl budget: not through broken links or slow responses, but through a lastmod that lies.

Why lastmod matters more than people think

Google has said for years that it mostly ignores <lastmod> unless it can trust it. Bing is blunter about it: crawl scheduling uses change history per URL. If your sitemap claims everything changed today, every day, then either:

• the crawler recrawls a lot of unchanged pages, or
• it learns your lastmod is noise and stops using it — so genuinely new pages wait behind nothing.

Either outcome is bad, and both are invisible in your analytics because nothing errors. You just notice, months later, that a batch of pages never got indexed.

The naive fix, and why it backfires

The obvious improvement over file mtimes is to hash the rendered output:

def lastmod_for(path):
    digest = hashlib.sha1(open(path, "rb").read()).hexdigest()
    return state.setdefault(path, today()) if digest != state.get(path) else state[path]

That's roughly what I shipped first. It's correct in principle and useless in practice, because the bytes of a rendered page are not stable even when the content is.

Two things move on every single deploy:

1. Cache-busting query strings. My asset references look like /assets/style.css?v=3f9a1c. The build rewrites that hash whenever the stylesheet changes. So a one-line CSS edit changes the bytes of every page that links the stylesheet — which is all of them.

2. The analytics beacon. A small script tag with a site token gets injected into <head>. Harmless, identical on every page — but if the injection order or whitespace ever shifts (a template refactor, a new minifier version, a different Python release writing attributes in a different order), the hash changes for the whole site at once.

Neither is a content change. Both are pure output noise. A hash of the raw file can't tell the difference.

Hash the meaning, not the bytes

The fix is to normalise before hashing — strip the parts that are allowed to vary without the page meaning anything different:

import re, hashlib

NOISE = [
    re.compile(r'\?v=[0-9a-zA-Z]+'),                 # cache-busting hashes
    re.compile(r'<script[^>]*data-analytics[^>]*>.*?</script>', re.S),
    re.compile(r'\s+'),                              # whitespace
]

def fingerprint(html):
    for pat in NOISE:
        html = pat.sub("", html)
    return hashlib.sha1(html.encode("utf-8")).hexdigest()

Then keep a state file mapping path → fingerprint, and only touch lastmod when the fingerprint moves:

state = json.load(open("_lastmod.json"))
changed = []
for page, html in built_pages:
    fp = fingerprint(html)
    if state.get(page, {}).get("fp") != fp:
        state[page] = {"fp": fp, "lastmod": today()}
        changed.append(page)
json.dump(state, open("_lastmod.json", "w"))
print(f"{len(changed)} pages changed")

The lastmod values in the sitemap now come from that state file, not from the clock.

The test that keeps it honest

A fingerprint system is easy to write and easy to break silently. The assertion I run in CI is deliberately dumb:

Build twice in a row. The second build must report zero changed pages.

If it reports one page, something in the pipeline is non-deterministic. If it reports 380, the normaliser stopped working — usually because someone added a new injected block and didn't add it to NOISE.

That single assertion has caught three real regressions for me:

• a minifier that started emitting a random class-name suffix
• an analytics snippet that included a build timestamp
• a Python upgrade that changed attribute order in the HTML writer

All three would have shipped an "everything changed" sitemap, and none of them would have raised anything in a normal test suite, because the pages were correct. They were just dishonest about it.

Where the state file lives

One trap worth calling out: if _lastmod.json lives only in CI and the cache is wiped, every page looks new again — the failure mode you were trying to fix.

Two options that work:

• commit it to the repo (small, boring, effective — 380 lines of JSON for my site)
• store it as a build artefact with a long retention and restore it before building

What does not work: recomputing it from git history. Git tells you when a source file changed, not when a rendered page's output changed — and one shared layout file touches all 380 outputs.

What actually changed

On my site, jianzhujisuan.com — 168 bilingual engineering calculators, 380 static pages — the effect was immediate and measurable in the logs rather than in rankings:

• deploys that change one article now mark 1 page as updated instead of 380
• the second build in any CI run reports 0 changed pages
• crawlers spend their budget on the pages that actually moved

No ranking jump. No dramatic story. Just a sitemap that stopped lying, which is what lastmod was supposed to be all along.

If you want to see the output side of this, open any tool page and view source — say the brick count calculator — the result table is in the HTML, with a fingerprint attribute that's the same on every build until the formula changes.