How to Generate an XML Sitemap for a Static Site With Python (Skip noindex Pages, Split at 50,000 URLs)

Most static site generators can produce a sitemap, but the default output often includes pages you don't want indexed: drafts marked noindex, duplicate pages that point a canonical elsewhere, and a lastmod on every URL set to the moment the build ran. This post walks through a small Python script that builds sitemap.xml from your build folder and avoids those three problems.

Disclosure: Appie.run publishes this post. Appie has a free XML Sitemap Generator, mentioned near the end. The script below has no dependency on it. Facts about the sitemap format come from sitemaps.org and Google Search Central, read on 28 Oct 2026.

If you want to check an existing sitemap rather than build one, our earlier post How to validate an XML sitemap covers the common errors.

The rules the script follows

From the sitemaps protocol and Google's Build and submit a sitemap page:

• Size limits. One sitemap holds at most 50,000 URLs or 50MB uncompressed. Past that, split it into several files and list them in a sitemap index.
• UTF-8 and escaping. The file must be UTF-8. Tag values must be entity-escaped, and URLs must be URL-escaped.
• Absolute URLs. Use https://www.example.com/page.html, not /page.html.
• Canonical URLs only. List the URLs you want in search results.
• priority and changefreq. Google says it ignores both. The script leaves them out.
• lastmod. Google uses it only if it's consistently and verifiably accurate, and it should reflect the last significant change to the page. A build timestamp on every page fails that test, so the script either uses a real date or omits the tag.
• Location. Unless you submit through Search Console, a sitemap only affects URLs under its own directory. Put it at the site root.

The script

Save this as gen_sitemap.py. It uses only the standard library (Python 3.8+).

#!/usr/bin/env python3
"""Build sitemap.xml from a static site's output folder (dist/, public/, _site/)."""
import argparse, datetime, re, subprocess
from pathlib import Path
from urllib.parse import quote
from xml.sax.saxutils import escape

MAX_URLS = 50_000  # per sitemap file, per the sitemaps protocol

NOINDEX = re.compile(r'<meta[^>]+name=["\']robots["\'][^>]+content=["\'][^"\']*noindex', re.I)
CANONICAL = re.compile(r'<link[^>]+rel=["\']canonical["\'][^>]+href=["\']([^"\']+)["\']', re.I)


def page_url(base, root, path):
    rel = path.relative_to(root).as_posix()
    if rel == "index.html":
        rel = ""
    elif rel.endswith("/index.html"):
        rel = rel[: -len("index.html")]  # keep trailing slash
    return base.rstrip("/") + "/" + quote(rel)  # percent-encode spaces, non-ASCII


def last_modified(path, root, src_root):
    """Last git commit date of the matching source file, else None.

    Build output mtimes are all "now", which makes lastmod useless, so we
    only trust git history for a file at the same relative path in --src.
    """
    if src_root:
        src_file = src_root / path.relative_to(root)
        if not src_file.exists():
            return None
        try:
            out = subprocess.run(
                ["git", "log", "-1", "--format=%cs", "--", path.relative_to(root).as_posix()],
                cwd=src_root, capture_output=True, text=True, check=True,
            ).stdout.strip()
            if out:
                return out
        except (subprocess.CalledProcessError, FileNotFoundError):
            pass
    return None


def collect(root, base, src_root):
    entries, skipped = [], []
    for path in sorted(root.rglob("*.html")):
        if path.name == "404.html":
            continue
        url = page_url(base, root, path)
        html = path.read_text(encoding="utf-8", errors="ignore")
        if NOINDEX.search(html):
            skipped.append((url, "noindex"))
            continue
        m = CANONICAL.search(html)
        if m and m.group(1).rstrip("/") != url.rstrip("/"):
            skipped.append((url, "canonical points to " + m.group(1)))
            continue
        entries.append((url, last_modified(path, root, src_root)))
    return entries, skipped


def write_urlset(entries, out):
    lines = ['<?xml version="1.0" encoding="UTF-8"?>',
             '<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">']
    for url, lastmod in entries:
        tag = f"<lastmod>{lastmod}</lastmod>" if lastmod else ""
        lines.append(f"  <url><loc>{escape(url)}</loc>{tag}</url>")
    lines.append("</urlset>")
    out.write_text("\n".join(lines) + "\n", encoding="utf-8")


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("root", type=Path, help="built site folder, e.g. dist")
    ap.add_argument("--base", required=True, help="https://www.example.com")
    ap.add_argument("--src", type=Path, help="git checkout whose files mirror the output paths")
    args = ap.parse_args()

    entries, skipped = collect(args.root, args.base, args.src)
    for url, why in skipped:
        print(f"skipped {url}: {why}")

    chunks = [entries[i:i + MAX_URLS] for i in range(0, len(entries), MAX_URLS)] or [[]]
    if len(chunks) == 1:
        write_urlset(chunks[0], args.root / "sitemap.xml")
        print(f"wrote sitemap.xml with {len(entries)} URLs")
        return
    names = []
    for n, chunk in enumerate(chunks, 1):
        name = f"sitemap-{n}.xml"
        write_urlset(chunk, args.root / name)
        names.append(name)
    today = datetime.date.today().isoformat()
    index = ['<?xml version="1.0" encoding="UTF-8"?>',
             '<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">']
    for name in names:
        loc = escape(args.base.rstrip("/") + "/" + name)
        index.append(f"  <sitemap><loc>{loc}</loc><lastmod>{today}</lastmod></sitemap>")
    index.append("</sitemapindex>")
    (args.root / "sitemap.xml").write_text("\n".join(index) + "\n", encoding="utf-8")
    print(f"wrote sitemap index with {len(names)} files, {len(entries)} URLs")


if __name__ == "__main__":
    main()

What each part does

page_url maps a file path to a URL. index.html becomes the folder URL with a trailing slash, so dist/blog/a-post/index.html becomes https://www.example.com/blog/a-post/. Paths are percent-encoded, so my page.html becomes my%20page.html.

collect walks every .html file and skips three kinds of page:

• 404.html
• pages with a robots meta tag containing noindex
• pages whose rel="canonical" points to a different URL

Each skip is printed, so you can see why a page is missing.

last_modified is the part most generators get wrong. Files in a build folder all have the build time as their modified date, which makes every lastmod identical and meaningless. The script only sets lastmod when you pass --src with a git checkout whose files sit at the same relative paths as the output (true for hand-written HTML sites, often not true for generators that render Markdown). It then uses the date of the last commit touching that file. Otherwise it leaves lastmod out, which is valid.

write_urlset escapes each URL with xml.sax.saxutils.escape, so an & in a query string becomes &.

main writes one sitemap.xml when you have 50,000 URLs or fewer. Above that it writes sitemap-1.xml, sitemap-2.xml and so on, and makes sitemap.xml a sitemap index pointing to them.

Run it

# after your build step
python3 gen_sitemap.py dist --base https://www.example.com

# with lastmod from git, for a site whose source paths match the output
python3 gen_sitemap.py dist --base https://www.example.com --src .

On a small test folder, the output looked like this:

skipped https://www.example.com/about-copy.html: canonical points to https://www.example.com/about.html
skipped https://www.example.com/drafts/x.html: noindex
wrote sitemap.xml with 5 URLs

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url><loc>https://www.example.com/about.html</loc></url>
  <url><loc>https://www.example.com/blog/a-post/</loc></url>
  <url><loc>https://www.example.com/</loc></url>
  <url><loc>https://www.example.com/my%20page.html</loc></url>
  <url><loc>https://www.example.com/tom%26jerry.html</loc></url>
</urlset>

To test the split, change MAX_URLS to 2 in a copy of the script. On the same folder it wrote three files plus an index.

Add it to your build

In a package.json:

{
  "scripts": {
    "build": "astro build && python3 gen_sitemap.py dist --base https://www.example.com"
  }
}

Or as a CI step after the build, before deploy. Then add one line to robots.txt so crawlers find it:

Sitemap: https://www.example.com/sitemap.xml

Google's docs list three ways to submit: Search Console, the Search Console API, or the Sitemap: line in robots.txt.

Known limits

• Regex HTML parsing. The noindex and canonical checks expect the attributes in the usual order (name before content, rel before href). If your templates write them the other way round, swap in html.parser from the standard library.
• No 50MB check. At 50,000 short URLs you'll be well under 50MB, but if your URLs are very long, check the file size too.
• Only HTML. PDFs and other files you want indexed need adding by hand.
• Trailing slashes. The canonical comparison ignores a trailing slash. If your site treats /page and /page/ as different pages, tighten that check.

When a script is more than you need

If you just need a one-off sitemap for a small site, a hosted tool is quicker:

• Appie's free XML Sitemap Generator crawls a public site for up to 80 pages, or takes a pasted list of up to 50,000 URLs, and returns sitemap XML. A single free run stops at 80 HTML pages or after 12 seconds, so paste a URL list for larger sites. It asks you to choose changefreq and priority; since Google ignores both, any value is fine. There's a captcha before each run.
• Screaming Frog SEO Spider is a desktop crawler that can generate XML sitemaps. Per its site on 28 Oct 2026, it crawls 500 URLs for free, and a licence to remove the limit is £199 per year.

After generating, run the file through a validator such as Appie's free XML Sitemap Validator, which checks the root element, absolute URLs, lastmod, duplicates and the 50,000 URL limit.

RG, Founder, Appie.run