DEV Community: appierun

Audit Title and Meta Description Lengths from a Sitemap with Python (Standard Library Only)

appierun — Sun, 08 Nov 2026 15:54:51 +0000

A sitemap can be valid XML and still ship titles like Home or meta descriptions that are empty, identical across fifty product pages, or long enough to get cut mid-sentence in a SERP. None of that shows up as a sitemap error. You only see it when you open each page, or when Search Console shows a title Google rewrote for you.

This post builds a small auditor in Python using only the standard library (html.parser, urllib, xml.etree, csv, concurrent.futures). It reads a sitemap or sitemap index, fetches every listed URL, extracts the <title> and meta name="description", measures their character lengths, flags missing and empty values, and marks duplicate titles or descriptions across the set. The output is a CSV you can sort in a spreadsheet.

Google Search Central docs on title links and snippets / meta descriptions were opened on 7 Nov 2026. Appie free-tool descriptions are as of 7 Nov 2026. The script was tested on Python 3.13.5 against a local fixture site, shown below.

Disclosure: Appie.run publishes this article. Two free tools mentioned near the end are ours. The script does not depend on them.

Where this fits

Earlier posts in this series covered extracting JSON-LD structured data with the standard library and finding broken links and redirect chains in a sitemap. Those check machine-readable markup and HTTP behavior. Titles and meta descriptions are the human-facing SERP text. Same pattern: parse what the server returns, print what is there, fail loudly when the page is incomplete or out of a practical length band.

What Google actually says about length

This matters, because a lot of SEO checklists still treat "60 characters" or "160 characters" as if Google published those numbers as rules. The current docs do not.

On title links (opened 7 Nov 2026), Google says every page should have a <title>, that the text should be descriptive and concise, and that you should avoid vague titles, keyword stuffing, and repeated boilerplate. It also says: there is no limit on how long a <title> element can be, but the title link is truncated in Google Search results as needed, typically to fit the device width. Google may also rewrite the title link using on-page headings, og:title, anchor text and other sources when it detects issues.

On snippets and meta descriptions (opened 7 Nov 2026), Google says snippets are primarily created from page content, and that Google sometimes uses the meta description when it describes the page better. Again: there is no limit on how long a meta description can be, but the snippet is truncated as needed to fit the device. Unique, page-specific descriptions are preferred over site-wide boilerplate. Keyword lists are less likely to be shown as a snippet. The docs even call out a "too short" example as a bad pattern.

So this script does not claim a Google-official character cutoff. It uses a practical audit band as a starting point for triage:

<title>: Practical band (defaults): 30 to 60 characters; Why it is useful anyway: Catches Home / Contact stubs and essay-length titles that almost always truncate.
meta description: Practical band (defaults): 70 to 160 characters; Why it is useful anyway: Catches empty pitches and keyword dumps; puts the important sentence first.

Override the bands with flags when your brand suffix or language needs a different window. The point of the band is to surface outliers for a human edit, not to score pages against a Google rule that does not exist.

The script

Save this as title_meta_audit.py. It should run on Python 3.8 or later; we tested it on 3.13.5.

#!/usr/bin/env python3
"""Audit <title> and meta description lengths for every URL in an XML sitemap.

Standard library only. Usage:
    python3 title_meta_audit.py https://example.com/sitemap.xml --out report.csv
    python3 title_meta_audit.py /path/to/sitemap.xml --out report.csv
    python3 title_meta_audit.py sitemap.xml --limit 20

Practical length bands (NOT Google rules; Google documents no fixed character
limit and truncates to fit the device). Defaults:
    title: 30-60 characters
    meta description: 70-160 characters
Override with --title-min/--title-max/--desc-min/--desc-max.
"""
import argparse
import csv
import gzip
import sys
import threading
import time
import urllib.error
import urllib.request
import xml.etree.ElementTree as ET
from collections import defaultdict
from concurrent.futures import ThreadPoolExecutor, as_completed
from html.parser import HTMLParser

NS = "{http://www.sitemaps.org/schemas/sitemap/0.9}"
UA = "title-meta-audit/1.0 (+https://example.com/contact)"


class TitleMetaParser(HTMLParser):
    """Collect <title> text and name=description meta content from <head>."""

    def __init__(self):
        super().__init__()
        self.title_parts = []
        self.description = None
        self._in_head = False
        self._in_title = False
        self._done = False

    def handle_starttag(self, tag, attrs):
        if self._done:
            return
        name = tag.lower()
        attr = {k.lower(): (v or "") for k, v in attrs}
        if name == "head":
            self._in_head = True
            return
        if name == "body":
            self._done = True
            return
        if not self._in_head:
            return
        if name == "title":
            self._in_title = True
            return
        if name == "meta":
            meta_name = (attr.get("name") or "").strip().lower()
            if meta_name == "description" and self.description is None:
                self.description = attr.get("content", "")

    def handle_endtag(self, tag):
        name = tag.lower()
        if name == "title":
            self._in_title = False
        if name == "head":
            self._in_head = False
            self._done = True

    def handle_data(self, data):
        if self._in_title and not self._done:
            self.title_parts.append(data)

    @property
    def title(self):
        return "".join(self.title_parts).strip()


def fetch_bytes(src, timeout):
    if src.startswith(("http://", "https://")):
        req = urllib.request.Request(src, headers={"User-Agent": UA})
        with urllib.request.urlopen(req, timeout=timeout) as resp:
            data = resp.read()
    else:
        with open(src, "rb") as fh:
            data = fh.read()
    return gzip.decompress(data) if data[:2] == b"\x1f\x8b" else data


def sitemap_urls(src, timeout, seen=None, depth=0):
    seen = seen if seen is not None else set()
    if src in seen or depth > 3:
        return []
    seen.add(src)
    root = ET.fromstring(fetch_bytes(src, timeout))
    locs = [el.text.strip() for el in root.iter(NS + "loc") if el.text]
    if root.tag == NS + "sitemapindex":
        urls = []
        for child in locs:
            urls += sitemap_urls(child, timeout, seen, depth + 1)
        return urls
    return locs


def fetch_html(url, timeout):
    req = urllib.request.Request(url, headers={"User-Agent": UA})
    with urllib.request.urlopen(req, timeout=timeout) as resp:
        charset = resp.headers.get_content_charset() or "utf-8"
        return resp.read().decode(charset, errors="replace"), resp.geturl(), resp.status


def classify(title, description, title_min, title_max, desc_min, desc_max):
    issues = []
    title_len = len(title) if title is not None else 0
    desc_len = len(description) if description is not None else 0

    if not title:
        issues.append("missing title")
    else:
        if title_len < title_min:
            issues.append(f"title too short ({title_len} < {title_min})")
        elif title_len > title_max:
            issues.append(f"title too long ({title_len} > {title_max})")

    if description is None:
        issues.append("missing meta description")
    elif description.strip() == "":
        issues.append("empty meta description")
    else:
        if desc_len < desc_min:
            issues.append(f"description too short ({desc_len} < {desc_min})")
        elif desc_len > desc_max:
            issues.append(f"description too long ({desc_len} > {desc_max})")

    return issues, title_len, desc_len


def audit_one(url, timeout, bands):
    title_min, title_max, desc_min, desc_max = bands
    try:
        html, final_url, status = fetch_html(url, timeout)
    except urllib.error.HTTPError as e:
        return {
            "url": url,
            "final_url": "",
            "status": e.code,
            "title": "",
            "title_len": 0,
            "description": "",
            "desc_len": 0,
            "issues": f"HTTP error {e.code}",
        }
    except Exception as e:
        return {
            "url": url,
            "final_url": "",
            "status": 0,
            "title": "",
            "title_len": 0,
            "description": "",
            "desc_len": 0,
            "issues": f"error: {e}",
        }
    parser = TitleMetaParser()
    parser.feed(html)
    title = parser.title
    description = parser.description
    issues, title_len, desc_len = classify(
        title, description, title_min, title_max, desc_min, desc_max
    )
    return {
        "url": url,
        "final_url": final_url,
        "status": status,
        "title": title,
        "title_len": title_len,
        "description": description if description is not None else "",
        "desc_len": desc_len,
        "issues": "; ".join(issues) if issues else "",
    }


def main(argv=None):
    p = argparse.ArgumentParser(description=__doc__)
    p.add_argument("sitemap", help="Sitemap URL or local XML path")
    p.add_argument("--out", default="title_meta_report.csv")
    p.add_argument("--timeout", type=float, default=15.0)
    p.add_argument("--workers", type=int, default=4)
    p.add_argument("--delay", type=float, default=0.25, help="Seconds between request starts")
    p.add_argument("--limit", type=int, default=0, help="Audit at most N URLs (0 = all)")
    p.add_argument("--title-min", type=int, default=30)
    p.add_argument("--title-max", type=int, default=60)
    p.add_argument("--desc-min", type=int, default=70)
    p.add_argument("--desc-max", type=int, default=160)
    args = p.parse_args(argv)

    bands = (args.title_min, args.title_max, args.desc_min, args.desc_max)
    try:
        urls = sitemap_urls(args.sitemap, args.timeout)
    except Exception as e:
        print(f"Failed to read sitemap: {e}", file=sys.stderr)
        return 2
    if args.limit and args.limit > 0:
        urls = urls[: args.limit]

    print(
        f"auditing {len(urls)} URLs with {args.workers} workers "
        f"(title {args.title_min}-{args.title_max}, "
        f"description {args.desc_min}-{args.desc_max}; practical bands)"
    )

    lock = threading.Lock()
    last_start = [0.0]
    rows = []

    def gated(url):
        with lock:
            now = time.monotonic()
            wait = args.delay - (now - last_start[0])
            if wait > 0:
                time.sleep(wait)
            last_start[0] = time.monotonic()
        return audit_one(url, args.timeout, bands)

    with ThreadPoolExecutor(max_workers=args.workers) as pool:
        futures = {pool.submit(gated, u): u for u in urls}
        for fut in as_completed(futures):
            rows.append(fut.result())

    # Preserve sitemap order
    by_url = {r["url"]: r for r in rows}
    ordered = [by_url[u] for u in urls if u in by_url]

    # Duplicate title / description across pages (after all fetches)
    title_map = defaultdict(list)
    desc_map = defaultdict(list)
    for r in ordered:
        if r["title"]:
            title_map[r["title"]].append(r["url"])
        if r["description"]:
            desc_map[r["description"]].append(r["url"])

    for r in ordered:
        extras = []
        if r["title"] and len(title_map[r["title"]]) > 1:
            extras.append("duplicate title")
        if r["description"] and len(desc_map[r["description"]]) > 1:
            extras.append("duplicate description")
        if extras:
            r["issues"] = (
                (r["issues"] + "; " if r["issues"] else "") + "; ".join(extras)
            )

    for r in ordered:
        flag = r["issues"] or "ok"
        print(
            f"{r['status']} {r['url']}  "
            f"title={r['title_len']} desc={r['desc_len']}  {flag}"
        )

    fields = [
        "url",
        "final_url",
        "status",
        "title",
        "title_len",
        "description",
        "desc_len",
        "issues",
    ]
    with open(args.out, "w", newline="", encoding="utf-8") as fh:
        w = csv.DictWriter(fh, fieldnames=fields)
        w.writeheader()
        w.writerows(ordered)

    ok = sum(1 for r in ordered if not r["issues"])
    bad = len(ordered) - ok
    print(f"{ok} ok, {bad} with issues; report: {args.out}")
    print(
        "Note: length bands are a practical audit starting point, not a Google rule. "
        "Google documents no fixed character limit for titles or meta descriptions."
    )
    return 1 if bad else 0


if __name__ == "__main__":
    sys.exit(main())

Local test

Serve a small fixture site with pages that miss titles, miss descriptions, go too short or too long, and share duplicate SERP text. Example pages (abbreviated):

<!-- good.html: title 43, description 125 -->
<title>Pricing Plans for Growing SaaS Teams | Acme</title>
<meta name="description" content="Compare Free, Starter and Standard plans. See credit limits, included tools and when to upgrade as your support volume grows.">

<!-- short-title.html: title "Home" (4 chars) -->
<title>Home</title>
<meta name="description" content="Welcome to Acme, the product platform that helps product and GTM teams ship clearer release notes and track customer feedback in one place.">

<!-- long-title.html: title 130 chars -->
<title>Acme Blog: Complete Guide to Customer Discovery Interviews, SAFE Scripts, Note Templates and Follow-up Emails for Founders in 2026</title>

<!-- no-desc.html: title present, no meta description -->
<title>About Acme: Our Story and Team</title>

<!-- short-desc.html -->
<title>Contact Acme Support and Sales</title>
<meta name="description" content="Get in touch.">

<!-- long-desc.html: description 423 chars -->
<title>Acme Features Overview for Product Teams</title>
<meta name="description" content="Explore every Acme feature in depth: ...without reinventing the wheel every sprint.">

<!-- no-title.html: description only -->
<meta name="description" content="Docs for integrating Acme with your stack, including API keys, webhooks and MCP server setup for agent workflows.">

<!-- empty-desc.html -->
<title>Careers at Acme: Open Roles</title>
<meta name="description" content="">

<!-- dup-a.html and dup-b.html share the same title and description -->
<title>Acme Product Updates and Changelog</title>
<meta name="description" content="Release notes and product updates from the Acme team. See what shipped this month and what is coming next for customers.">

A sitemap.xml lists those ten pages plus a deliberate 404 (missing.html). A tiny http.server on 127.0.0.1:8766 serves the folder.

python3 test_server.py
python3 title_meta_audit.py http://127.0.0.1:8766/sitemap.xml --out report.csv --delay 0.05

Real output from Python 3.13.5 on 7 Nov 2026 (exit code 1 because issues were found):

auditing 11 URLs with 4 workers (title 30-60, description 70-160; practical bands)
200 http://127.0.0.1:8766/good.html  title=43 desc=125  ok
200 http://127.0.0.1:8766/short-title.html  title=4 desc=139  title too short (4 < 30)
200 http://127.0.0.1:8766/long-title.html  title=130 desc=135  title too long (130 > 60)
200 http://127.0.0.1:8766/no-desc.html  title=30 desc=0  missing meta description
200 http://127.0.0.1:8766/short-desc.html  title=30 desc=13  description too short (13 < 70)
200 http://127.0.0.1:8766/long-desc.html  title=40 desc=423  description too long (423 > 160)
200 http://127.0.0.1:8766/no-title.html  title=0 desc=113  missing title
200 http://127.0.0.1:8766/empty-desc.html  title=27 desc=0  title too short (27 < 30); empty meta description
200 http://127.0.0.1:8766/dup-a.html  title=34 desc=120  duplicate title; duplicate description
200 http://127.0.0.1:8766/dup-b.html  title=34 desc=120  duplicate title; duplicate description
404 http://127.0.0.1:8766/missing.html  title=0 desc=0  HTTP error 404
1 ok, 10 with issues; report: /workspace/tmp/title-meta-audit/report.csv
Note: length bands are a practical audit starting point, not a Google rule. Google documents no fixed character limit for titles or meta descriptions.

Only good.html passed the default band with unique text. The two changelog pages passed the length checks on their own, then failed the duplicate pass once both had been fetched. That second pass is why the script waits until every URL is in before printing the final flags.

CSV columns: url, final_url, status, title, title_len, description, desc_len, issues. Sample rows:

url,status,title_len,desc_len,issues
http://127.0.0.1:8766/good.html,200,43,125,
http://127.0.0.1:8766/short-title.html,200,4,139,title too short (4 < 30)
http://127.0.0.1:8766/long-desc.html,200,40,423,description too long (423 > 160)
http://127.0.0.1:8766/dup-a.html,200,34,120,duplicate title; duplicate description
http://127.0.0.1:8766/missing.html,404,0,0,HTTP error 404

How the parser works

html.parser.HTMLParser only collects while it is inside <head>, and it stops at </head> or <body>. That keeps footer widgets and body copy from polluting the report.

The title is the concatenated text nodes inside <title>, stripped. The description is the first meta whose name is description (case-insensitive). Later duplicate description tags are ignored; fix the first one.

Character length is Python's len() on the Unicode string after decode. That is a character count, not a rendered pixel width. Google truncates by device width, not by a published character quota, which is another reason the bands are labeled as a practical starting point.

Sitemap reading matches the earlier posts in this series: xml.etree walks {http://www.sitemaps.org/schemas/sitemap/0.9}loc nodes, follows a sitemap index up to depth 3, and accepts .gz payloads. Local files and http(s) URLs both work as the sitemap argument.

Reading the report

A few rules of thumb once you have the CSV:

Missing or empty title. Google will invent a title link from headings, og:title or anchors. Write a real <title> that matches the main visible heading.
Title too short. Home, Blog, Pricing and similar stubs fail the lower band on purpose. Add the page topic and a short brand suffix.
Title too long. Put the distinctive part first. Brand suffixes are fine; keyword essays are not. Google may rewrite verbose titles anyway.
Missing or empty meta description. Not a ranking penalty by itself, but you give up a chance to pitch the page. Prefer a unique sentence over nothing.
Description too short or too long. Aim for a readable pitch with the important claim first. Truncation is width-based; front-loading still helps.
Duplicate title or description. Common after CMS templates reuse one string. Google's title and snippet docs both warn against boilerplate. Differentiate the pages people cannot tell apart in results.
HTTP errors. Fix or remove those URLs from the sitemap before you worry about meta length. Pair this script with the broken-link / redirect-chain checker if status codes are noisy.

Polite defaults and running it on a real site

Defaults: 4 workers, 0.25 seconds between request starts, 15 second timeout. Raise --workers and lower --delay on your own staging box if the server can take it. On a client site, ask first and keep the defaults. Change the contact URL in the UA string so the site owner can reach you.

python3 title_meta_audit.py https://www.example.com/sitemap.xml --limit 50 --out sample.csv
python3 title_meta_audit.py https://www.example.com/sitemap.xml --title-min 25 --title-max 70 --desc-min 50 --desc-max 180 --out custom-bands.csv

Use --limit for a first pass. For large indexes, audit one child sitemap at a time.

Known limits

Server HTML only. Titles or descriptions injected only by client-side JavaScript will not appear. View-source what the server returns.
Character count, not pixels. Wide characters and different fonts change how much fits in a SERP. The band is a triage tool.
First meta name="description" only. Unusual pages with several description tags need a human look.
No Open Graph / Twitter Card check. Those are separate preview signals; use the OG/Twitter checker from this series for link previews.
No robots.txt handling. The script requests what the sitemap lists.
Duplicate detection is exact. Trailing spaces or tiny copy edits count as different strings.

Free tools for the steps around the script

If you want a URL list in the browser before you run anything:

Sitemap URL Extractor (ours, free; checked 7 Nov 2026). Paste sitemap XML or enter a sitemap URL. You get one URL per line from urlset <loc> tags. Given a sitemap index, it lists child sitemap locations; open a child and run it again for that file's page URLs. Useful when you want to sample a subset into a shell loop before a full audit.

When a page's meta description is empty or too thin and you need draft Q&A from the page body itself:

Webpage to FAQ Generator (ours, free; checked 7 Nov 2026). Paste a single public page URL. It builds FAQs from the text on that page and is instructed to skip claims the page does not make. That will not write your meta description for you, but it is a fast way to surface page-specific sentences you can turn into a unique description.

Both ask for a captcha before each run. Neither replaces the length audit this script performs.

FAQ

Is a title over 60 characters "wrong"?
Not by Google's docs. Google documents no fixed character limit and truncates to fit the device. The default upper band is a practical flag so you review titles that will almost always truncate. Raise --title-max if your brand needs more room.

Does a missing meta description hurt rankings?
Google does not require one. It may still generate a snippet from page content. A unique description is still worth writing for pages you care about in search results.

Why flag duplicates if both pages are "in band"?
Because identical titles and descriptions make results hard to tell apart. Google's own title and snippet guidance warns against repeated or boilerplate text across pages.

Can I use Beautiful Soup or lxml?
Yes, if you already depend on them. This series stays on the standard library so the script runs anywhere Python runs, with no pip install.

Will it see meta tags added by React or Vue after load?
Only if they are present in the HTML response. Most crawlers see the first response body.

Can I run it in CI?
Yes. It exits with 1 when any URL has an issue, 2 when the sitemap cannot be read, and 0 when every URL is clean under your chosen bands.

Wrap-up

Title and meta description problems rarely show up in sitemap validators. They show up as vague SERP text, Google-rewritten titles, and pages that look the same in results. Run this auditor after template changes, migrations and bulk CMS edits. Pair it with the JSON-LD and broken-link scripts when you want a fuller technical pass. If you need a URL list from a sitemap in the browser first, the free Sitemap URL Extractor is enough to get started.

RG, Founder, Appie.run

Extract and Check Open Graph and Twitter Card Meta Tags with Python (Standard Library Only)

appierun — Sat, 07 Nov 2026 15:57:52 +0000

When someone pastes your URL into Slack, LinkedIn, iMessage or X, the preview is not reading your <title> tag and hoping for the best. It looks for Open Graph tags (og:*) and, for X, Twitter Card tags (twitter:*). A page can be fine in the browser and still look broken in a preview: missing image, wrong title, relative og:image that only works on your domain, or no twitter:card at all.

This post builds a small checker in Python using only the standard library (html.parser, urllib). It fetches a URL (or reads a local HTML file), extracts every og:* and twitter:* meta tag, and flags the ones this script treats as required for a usable link preview: og:title, og:description, og:image, og:url and twitter:card.

Open Graph rules below come from ogp.me (opened 6 Nov 2026). Twitter Card tag names and Open Graph fallback behavior come from the archived Cards Markup reference on developer.twitter.com via the Wayback Machine (snapshot retrieved 6 Nov 2026); X's current docs.x.com index no longer surfaces that page as a first-class doc. Appie free-tool descriptions are as of 6 Nov 2026. The script was tested on Python 3.13 against local fixtures shown below.

Disclosure: Appie.run publishes this article. One free tool mentioned near the end is ours. The script does not depend on it.

Where this fits

Earlier posts in this series covered checking which AI crawlers your robots.txt blocks and extracting JSON-LD structured data with the standard library. Those look at machine-readable signals search and AI systems use. Open Graph and Twitter Cards are the parallel signal for human link previews. Same idea: parse the page yourself, print what is there, fail loudly when the important tags are missing.

What "required" means here

The Open Graph protocol lists four required properties for every page: og:title, og:type, og:image and og:url. og:description is optional in the protocol but nearly always needed for a readable preview, so this script treats it as required for the checklist.

For X/Twitter Cards, the important tag is twitter:card (common values: summary, summary_large_image, app, player). The older Cards Markup reference notes that if og:type, og:title and og:description exist but twitter:card is absent, a summary card may still render via Open Graph fallback. Relying on that fallback is fragile, so the script still flags a missing twitter:card.

This is a preview hygiene checklist, not a claim that every platform uses the same rules.

The script

Save this as og_meta_check.py. It should run on Python 3.8 or later; we tested it on 3.13.

#!/usr/bin/env python3
"""Extract Open Graph and Twitter Card meta tags from a URL or local HTML file.

Standard library only. Usage:
    python3 og_meta_check.py https://example.com/page
    python3 og_meta_check.py /path/to/page.html
    python3 og_meta_check.py https://example.com/page --json
"""
import argparse
import json
import sys
import urllib.error
import urllib.request
from html.parser import HTMLParser
from urllib.parse import urljoin

UA = "og-meta-check/1.0 (+https://example.com/contact)"

# Preview checklist used by this script (not identical to every platform's rules).
# Open Graph protocol (ogp.me) requires og:title, og:type, og:image, og:url.
# This checker also treats og:description as required for useful link previews,
# and twitter:card as required for X/Twitter Cards (OG tags may fill in the rest).
REQUIRED = (
    "og:title",
    "og:description",
    "og:image",
    "og:url",
    "twitter:card",
)


class MetaParser(HTMLParser):
    """Collect og:* and twitter:* meta tags from <head>."""

    def __init__(self):
        super().__init__()
        self.tags = {}  # property/name -> list of content values (order preserved)
        self._in_head = False
        self._done = False

    def handle_starttag(self, tag, attrs):
        if self._done:
            return
        name = tag.lower()
        attr = {k.lower(): (v or "") for k, v in attrs}
        if name == "head":
            self._in_head = True
            return
        if name == "body":
            self._done = True
            return
        if not self._in_head or name != "meta":
            return
        prop = attr.get("property") or attr.get("name") or ""
        prop = prop.strip().lower()
        if not (prop.startswith("og:") or prop.startswith("twitter:")):
            return
        content = attr.get("content", "").strip()
        self.tags.setdefault(prop, []).append(content)

    def handle_endtag(self, tag):
        if tag.lower() == "head":
            self._in_head = False
            self._done = True


def fetch_html(src, timeout):
    if src.startswith(("http://", "https://")):
        req = urllib.request.Request(src, headers={"User-Agent": UA})
        with urllib.request.urlopen(req, timeout=timeout) as resp:
            charset = resp.headers.get_content_charset() or "utf-8"
            return resp.read().decode(charset, errors="replace"), resp.geturl()
    with open(src, "r", encoding="utf-8", errors="replace") as fh:
        return fh.read(), src


def first(tags, key):
    vals = tags.get(key) or []
    return vals[0] if vals else None


def analyze(tags, page_url):
    missing = [k for k in REQUIRED if not first(tags, k)]
    warnings = []
    og_image = first(tags, "og:image")
    if og_image and page_url.startswith(("http://", "https://")):
        abs_img = urljoin(page_url, og_image)
        if abs_img != og_image and not og_image.startswith(("http://", "https://")):
            warnings.append(
                f"og:image is relative ({og_image}); resolved as {abs_img}"
            )
    if first(tags, "og:title") and not first(tags, "og:type"):
        warnings.append(
            "og:type is missing; Open Graph protocol lists it as required "
            "(this script still treats the page as checkable without it)."
        )
    card = first(tags, "twitter:card")
    if card and card not in {
        "summary",
        "summary_large_image",
        "app",
        "player",
    }:
        warnings.append(
            f"twitter:card value {card!r} is unusual; common values are "
            "summary, summary_large_image, app, player"
        )
    return {
        "url": page_url,
        "tags": {k: v for k, v in sorted(tags.items())},
        "missing_required": missing,
        "warnings": warnings,
        "ok": not missing,
    }


def print_report(result):
    print(f"URL: {result['url']}")
    print()
    if result["tags"]:
        print("Found tags:")
        for key, values in result["tags"].items():
            for val in values:
                print(f"  {key}: {val}")
    else:
        print("Found tags: (none)")
    print()
    if result["missing_required"]:
        print("Missing required:")
        for key in result["missing_required"]:
            print(f"  - {key}")
    else:
        print("Missing required: (none)")
    if result["warnings"]:
        print()
        print("Warnings:")
        for w in result["warnings"]:
            print(f"  - {w}")
    print()
    print("Status:", "OK" if result["ok"] else "INCOMPLETE")


def main(argv=None):
    p = argparse.ArgumentParser(description=__doc__)
    p.add_argument("source", help="URL or path to an HTML file")
    p.add_argument("--timeout", type=float, default=15.0)
    p.add_argument("--json", action="store_true", help="Print JSON instead of text")
    args = p.parse_args(argv)
    try:
        html, final_url = fetch_html(args.source, args.timeout)
    except urllib.error.HTTPError as e:
        print(f"HTTP error: {e.code} {e.reason}", file=sys.stderr)
        return 2
    except urllib.error.URLError as e:
        print(f"URL error: {e.reason}", file=sys.stderr)
        return 2
    except OSError as e:
        print(f"File error: {e}", file=sys.stderr)
        return 2
    parser = MetaParser()
    parser.feed(html)
    result = analyze(parser.tags, final_url)
    if args.json:
        print(json.dumps(result, indent=2))
    else:
        print_report(result)
    return 0 if result["ok"] else 1


if __name__ == "__main__":
    sys.exit(main())

Local test

Create a complete fixture:

<!DOCTYPE html>
<html>
<head>
  <meta charset="utf-8">
  <title>Fallback Title</title>
  <meta property="og:title" content="Sample Product Launch" />
  <meta property="og:description" content="A short description for link previews." />
  <meta property="og:image" content="https://example.com/og.png" />
  <meta property="og:url" content="https://example.com/launch" />
  <meta property="og:type" content="article" />
  <meta name="twitter:card" content="summary_large_image" />
  <meta name="twitter:title" content="Sample Product Launch" />
</head>
<body><h1>Hello</h1></body>
</html>

Run:

python3 og_meta_check.py sample.html

Expected output (status OK, exit code 0):

URL: sample.html

Found tags:
  og:description: A short description for link previews.
  og:image: https://example.com/og.png
  og:title: Sample Product Launch
  og:type: article
  og:url: https://example.com/launch
  twitter:card: summary_large_image
  twitter:title: Sample Product Launch

Missing required: (none)

Status: OK

Now an incomplete page:

<!DOCTYPE html>
<html><head>
<title>Only a title</title>
<meta property="og:title" content="Incomplete page" />
</head><body></body></html>

python3 og_meta_check.py incomplete.html; echo "exit: $?"

Expected (status INCOMPLETE, exit code 1):

URL: incomplete.html

Found tags:
  og:title: Incomplete page

Missing required:
  - og:description
  - og:image
  - og:url
  - twitter:card

Warnings:
  - og:type is missing; Open Graph protocol lists it as required (this script still treats the page as checkable without it).

Status: INCOMPLETE
exit: 1

We ran both fixtures on Python 3.13 on 6 Nov 2026 and got the outputs above.

Against a live site:

python3 og_meta_check.py https://example.com/
python3 og_meta_check.py https://example.com/ --json

Exit code 0 means the checklist passed; 1 means required tags are missing; 2 means fetch failed.

How the parser works

html.parser.HTMLParser is intentionally picky about where it looks. The collector only records meta tags while it is inside <head>, and it stops when it sees </head> or <body>. That keeps footer widgets and body markup from polluting the report.

Open Graph tags usually use property="og:...". Twitter Card tags usually use name="twitter:...". The script accepts either attribute for both prefixes, because real pages mix them. Multiple values for the same key are kept in document order; the checklist uses the first value, which matches how most consumers prefer the first og:image.

urllib.request follows redirects by default. The report prints the final URL after redirects, which matters when og:url still points at an old path.

Common failures the script catches

No og:image. Previews fall back to a blank or a random scraped image.
Relative og:image. Some consumers resolve it; many do not. The script warns and shows the absolute form via urljoin.
Missing og:description. Title-only cards look unfinished in Slack and iMessage.
og:url missing or pointing at a tracking redirect. Prefer the canonical public URL.
No twitter:card. You may get an OG fallback on X, or nothing useful. Set the card type explicitly.
og:type missing. The protocol requires it. This script warns but does not fail the checklist solely for that, so you can prioritize preview text and image first.

What it does not do

It does not download og:image to check dimensions, file size or format.
It does not call Facebook's Sharing Debugger or X's card validator APIs.
It does not execute JavaScript. Tags injected only by client-side code will not appear. If your site is a pure CSR app, render meta tags on the server or in the prerendered HTML.
It does not crawl a whole site. Point it at one URL at a time, or wrap it in a small loop over a URL list.

If you already have an XML sitemap and want the URL list first, Appie's free Sitemap URL Extractor (checked 6 Nov 2026) turns a sitemap or sitemap index into one URL per line. Feed those URLs to this script in a shell loop. The extractor lists child sitemap locations when you give it an index; open a child and run it again for page URLs.

Minimal tags to aim for

<meta property="og:title" content="Your page title" />
<meta property="og:description" content="One or two sentences for the preview." />
<meta property="og:image" content="https://example.com/images/og.png" />
<meta property="og:url" content="https://example.com/your-page" />
<meta property="og:type" content="website" />
<meta name="twitter:card" content="summary_large_image" />

Use absolute HTTPS image URLs. Keep descriptions short. Match og:url to the canonical page people should share.

FAQ

Does this replace Facebook or X's official debuggers?
No. Use those when you need to see the cached preview a platform will show. This script is for local and CI checks before you ship.

Why flag og:description if ogp.me marks it optional?
Because link previews without a description look incomplete in the places people actually paste URLs. The protocol's required set still includes og:type, which this script warns about separately.

Can I use Beautiful Soup instead?
Yes, if you already depend on it. This series stays on the standard library so the script runs anywhere Python runs, with no pip install.

Will it see meta tags added by React or Vue after load?
Only if they are present in the HTML response. View-source what the server returns; that is what most crawlers and preview bots see on the first request.

If you are already auditing sitemaps and structured data with small Python scripts, add this checker to the same folder and run it on every template that can be shared. For pulling URLs out of a sitemap before you loop, the free Sitemap URL Extractor is enough to get started.

RG, Founder, Appie.run

Find Broken Links and Redirect Chains in Your Sitemap with Python (Standard Library Only)

appierun — Fri, 06 Nov 2026 15:58:19 +0000

Your XML sitemap is supposed to be a list of pages you want indexed. In practice it drifts. A page gets deleted and the CMS keeps listing it. A slug changes, a redirect goes in, and the sitemap still points at the old URL. Someone adds a second redirect on top of the first, and now a crawler has to take two hops to reach the page. None of this shows up as an XML error, so a sitemap validator passes the file and the problems sit there.

This post builds a small sitemap URL status checker in Python using only the standard library (urllib, xml.etree, concurrent.futures, html.parser). It reads a sitemap or a sitemap index, requests every URL without letting urllib follow redirects on its own, walks each redirect chain hop by hop, and writes a CSV you can sort in a spreadsheet.

It flags:

4xx and 5xx responses on listed URLs
listed URLs that redirect at all (a sitemap should list final URLs)
redirect chains longer than one hop, and redirect loops
redirects that end on a 4xx or 5xx page
temporary redirects (302, 303, 307) inside a chain
redirects whose target is not itself in the sitemap
final pages whose rel="canonical" points at a different URL
connection errors (refused connections, timeouts, DNS failures)

Tool descriptions for appie.run are as of 5 Nov 2026, checked on their pages that day. The script was tested on Python 3.13 against a local test site, shown below.

Disclosure: Appie.run publishes this article. Two free tools mentioned near the end are ours. The script does not depend on them.

Where this fits

Earlier posts in this series covered validating a sitemap's XML and comparing two sitemaps before and after a migration. The validator post ended with a short status-code loop that let urlopen follow redirects and compared the final URL. That tells you a redirect happened, but not how many hops it took, which codes were involved, or whether it looped. The comparison post looked at the first hop only. This script records the whole chain.

Why stop urllib from following redirects

urllib.request.urlopen follows redirects for you, up to 10 hops, and gives up with an error when it detects a loop. That is convenient for downloading a page and useless for auditing one, because the intermediate hops disappear. The fix is a redirect handler whose redirect_request returns None. urllib then raises an HTTPError for every 3xx response, and the error object still carries the status code and the Location header. The script reads those, resolves relative locations with urljoin, and requests the next hop itself.

The script

Save this as sitemap_status.py. It should run on Python 3.8 or later; we tested it on 3.13.

#!/usr/bin/env python3
"""Check every URL in an XML sitemap for broken links and redirect chains.

Standard library only. Usage:
    python3 sitemap_status.py https://example.com/sitemap.xml --out report.csv
"""
import argparse
import csv
import gzip
import sys
import threading
import time
import urllib.error
import urllib.request
import xml.etree.ElementTree as ET
from concurrent.futures import ThreadPoolExecutor
from html.parser import HTMLParser
from urllib.parse import urljoin, urldefrag

NS = "{http://www.sitemaps.org/schemas/sitemap/0.9}"
UA = "sitemap-status/1.0 (+https://example.com/contact)"


class NoRedirect(urllib.request.HTTPRedirectHandler):
    """Make urllib hand 3xx responses back to us instead of following them."""

    def redirect_request(self, req, fp, code, msg, headers, newurl):
        return None


OPENER = urllib.request.build_opener(NoRedirect)


def fetch_bytes(src, timeout):
    """Read a sitemap from a URL or a local file, unzipping .gz content."""
    if src.startswith(("http://", "https://")):
        req = urllib.request.Request(src, headers={"User-Agent": UA})
        with urllib.request.urlopen(req, timeout=timeout) as resp:
            data = resp.read()
    else:
        with open(src, "rb") as fh:
            data = fh.read()
    return gzip.decompress(data) if data[:2] == b"\x1f\x8b" else data


def sitemap_urls(src, timeout, seen=None, depth=0):
    """Return page URLs in sitemap order, following sitemap indexes."""
    seen = seen if seen is not None else set()
    if src in seen or depth > 3:
        return []
    seen.add(src)
    root = ET.fromstring(fetch_bytes(src, timeout))
    locs = [el.text.strip() for el in root.iter(NS + "loc") if el.text]
    if root.tag == NS + "sitemapindex":
        urls = []
        for child in locs:
            urls += sitemap_urls(child, timeout, seen, depth + 1)
        return urls
    if root.tag == NS + "urlset":
        return locs
    raise ValueError(f"{src}: root is {root.tag}, expected urlset or sitemapindex")


class CanonicalFinder(HTMLParser):
    """Grab the href of the first <link rel="canonical">."""

    def __init__(self):
        super().__init__()
        self.canonical = None

    def handle_starttag(self, tag, attrs):
        a = dict(attrs)
        if tag == "link" and self.canonical is None and \
                "canonical" in (a.get("rel") or "").lower().split():
            self.canonical = a.get("href")


class Throttle:
    """Space requests out across all worker threads."""

    def __init__(self, delay):
        self.delay, self.lock, self.next_at = delay, threading.Lock(), 0.0

    def wait(self):
        with self.lock:
            now = time.monotonic()
            sleep_for = max(0.0, self.next_at - now)
            self.next_at = max(now, self.next_at) + self.delay
        time.sleep(sleep_for)


def request(url, timeout, throttle):
    """GET one URL without following redirects. Returns (status, location, html)."""
    throttle.wait()
    req = urllib.request.Request(url, headers={"User-Agent": UA})
    try:
        with OPENER.open(req, timeout=timeout) as resp:
            html = ""
            if "html" in resp.headers.get("Content-Type", ""):
                html = resp.read(200_000).decode("utf-8", "replace")
            return resp.status, "", html
    except urllib.error.HTTPError as err:  # 3xx, 4xx and 5xx all land here
        return err.code, err.headers.get("Location", ""), ""
    except (urllib.error.URLError, TimeoutError, ConnectionError) as err:
        return 0, "", f"error: {getattr(err, 'reason', err)}"


def check(url, opts, throttle, listed):
    """Follow one URL's redirect chain and return a report row."""
    chain, codes, current, issues = [url], [], url, []
    for _ in range(opts.max_hops + 1):
        status, location, html = request(current, opts.timeout, throttle)
        codes.append(status)
        if 300 <= status < 400 and location:
            nxt = urldefrag(urljoin(current, location))[0]
            if nxt in chain:
                issues.append("redirect loop")
                chain.append(nxt)
                break
            chain.append(nxt)
            current = nxt
            continue
        break
    else:
        issues.append(f"redirect limit reached (--max-hops {opts.max_hops})")

    final = codes[-1]
    hops = len(chain) - 1
    if final == 0:
        issues.append(html)  # the connection error message
    elif 400 <= final < 500:
        issues.append(f"client error {final}")
    elif final >= 500:
        issues.append(f"server error {final}")
    elif 300 <= final < 400 and not issues:
        issues.append(f"{final} with no Location header")

    if hops:
        issues.append("listed URL redirects" if hops == 1 else f"redirect chain of {hops} hops")
        if any(c in (302, 303, 307) for c in codes[:hops]):
            issues.append("temporary redirect in chain")
        if final == 200 and chain[-1] not in listed:
            issues.append("redirect target not in sitemap")

    canonical = ""
    if final == 200 and html:
        finder = CanonicalFinder()
        finder.feed(html)
        if finder.canonical:
            canonical = urljoin(chain[-1], finder.canonical)
            if canonical != chain[-1]:
                issues.append("final URL canonicalises elsewhere")

    return {
        "url": url,
        "status": codes[0],
        "final_url": chain[-1],
        "final_status": final,
        "hops": hops,
        "chain": " -> ".join(f"{u} [{c}]" if i < len(codes) else u
                             for i, (u, c) in enumerate(zip(chain, codes + [0] * 2))),
        "canonical": canonical,
        "issues": "; ".join(issues) or "ok",
    }


def main():
    ap = argparse.ArgumentParser(description=__doc__)
    ap.add_argument("sitemap", help="sitemap or sitemap index: URL or local file")
    ap.add_argument("--out", default="sitemap-status.csv")
    ap.add_argument("--workers", type=int, default=4, help="parallel requests (default 4)")
    ap.add_argument("--delay", type=float, default=0.5,
                    help="seconds between request starts, across all workers (default 0.5)")
    ap.add_argument("--timeout", type=float, default=15)
    ap.add_argument("--max-hops", type=int, default=5)
    ap.add_argument("--limit", type=int, default=0, help="check only the first N URLs")
    opts = ap.parse_args()

    urls = list(dict.fromkeys(sitemap_urls(opts.sitemap, opts.timeout)))  # dedupe, keep order
    if opts.limit:
        urls = urls[:opts.limit]
    listed = set(urls)
    throttle = Throttle(opts.delay)
    print(f"checking {len(urls)} URLs with {opts.workers} workers", file=sys.stderr)

    with ThreadPoolExecutor(max_workers=opts.workers) as pool:
        rows = list(pool.map(lambda u: check(u, opts, throttle, listed), urls))

    fields = ["url", "status", "final_url", "final_status", "hops", "chain", "canonical", "issues"]
    with open(opts.out, "w", newline="", encoding="utf-8") as fh:
        writer = csv.DictWriter(fh, fieldnames=fields)
        writer.writeheader()
        writer.writerows(rows)

    problems = [r for r in rows if r["issues"] != "ok"]
    for r in problems:
        print(f"{r['status']:>3} {r['url']}  {r['issues']}")
    print(f"{len(rows) - len(problems)} ok, {len(problems)} with issues; report: {opts.out}")
    return 1 if problems else 0


if __name__ == "__main__":
    sys.exit(main())

What each part does

fetch_bytes and sitemap_urls read a sitemap from a URL or a local file, unzip gzip content by checking the magic bytes, and follow <sitemapindex> files into their child sitemaps. A seen set and a depth limit stop an index that lists itself from looping. Duplicate URLs across child sitemaps are checked once.
NoRedirect is the handler described above. Every 3xx comes back to us as an HTTPError with its Location header.
request sends one GET with a descriptive User-Agent and a timeout. It reads up to 200 KB of the body only when the response is HTML, which is enough to find the canonical tag in the <head> without downloading whole pages. GET is used instead of HEAD because some servers answer HEAD differently from GET, and we need the HTML for the canonical check anyway.
check walks the chain. Each new hop is compared against the URLs already in the chain, so A -> B -> A stops as a loop instead of running until the hop limit. If the chain is still redirecting after --max-hops follow-ups, it stops and says so.
CanonicalFinder is a tiny HTMLParser subclass that keeps the first <link rel="canonical"> it sees. The href is resolved against the final URL, so relative canonicals work.
Throttle spaces out request starts across all worker threads. With the defaults (4 workers, 0.5 seconds), the script starts at most two requests per second no matter how many threads are waiting. Workers mainly help when a slow response would otherwise hold up the queue.
Exit code is 1 when any URL has an issue, so a CI job can fail on it.

Testing it against a local site

You should not test a crawler by pointing it at someone else's server. We built a tiny local site with http.server that has one of each problem, then ran the script against it. Save this as test_server.py:

"""Tiny local site for testing sitemap_status.py."""
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

BASE = "http://127.0.0.1:8765"
PAGE = '<html><head><link rel="canonical" href="{}"></head><body>hi</body></html>'

REDIRECTS = {
    "/old-pricing": (301, "/pricing"),           # one hop, fine target
    "/blog/a": (301, "/blog/b"),                 # chain: a -> b -> c
    "/blog/b": (302, "/blog/c"),
    "/loop-1": (301, "/loop-2"),                 # loop
    "/loop-2": (301, "/loop-1"),
    "/moved": (301, "/gone"),                    # redirect to a 404
    "/plans": (301, BASE + "/pricing/"),     # absolute Location, trailing slash variant
}
PAGES = {
    "/": "/",
    "/pricing": "/pricing",
    "/blog/c": "/blog/c",
    "/pricing/": "/pricing",                     # canonical points elsewhere
    "/print/about": "/about",                    # canonicalises to another URL
}

class H(BaseHTTPRequestHandler):
    def do_GET(self):
        p = self.path
        if p in ("/sitemap_index.xml", "/sitemap-pages.xml", "/sitemap-blog.xml"):
            body = open("." + p, "rb").read()
            self.send_response(200); self.send_header("Content-Type", "application/xml")
            self.end_headers(); self.wfile.write(body); return
        if p in REDIRECTS:
            code, loc = REDIRECTS[p]
            self.send_response(code); self.send_header("Location", loc); self.end_headers(); return
        if p in PAGES:
            body = PAGE.format(BASE + PAGES[p]).encode()
            self.send_response(200); self.send_header("Content-Type", "text/html; charset=utf-8")
            self.end_headers(); self.wfile.write(body); return
        if p == "/broken":
            self.send_response(500); self.end_headers(); return
        self.send_response(404); self.end_headers()

ThreadingHTTPServer(("127.0.0.1", 8765), H).serve_forever()

Then two child sitemaps and an index. sitemap-pages.xml lists /, /pricing, /old-pricing, /plans, /print/about, /moved, /broken and /missing. sitemap-blog.xml lists /blog/a, /blog/c and /loop-1. The index points at both:

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap><loc>http://127.0.0.1:8765/sitemap-pages.xml</loc></sitemap>
  <sitemap><loc>http://127.0.0.1:8765/sitemap-blog.xml</loc></sitemap>
</sitemapindex>

Start the server in one terminal and run the checker in another:

python3 test_server.py
python3 sitemap_status.py http://127.0.0.1:8765/sitemap_index.xml --out report.csv

Output:

checking 11 URLs with 4 workers
301 http://127.0.0.1:8765/old-pricing  listed URL redirects
301 http://127.0.0.1:8765/plans  listed URL redirects; redirect target not in sitemap; final URL canonicalises elsewhere
200 http://127.0.0.1:8765/print/about  final URL canonicalises elsewhere
301 http://127.0.0.1:8765/moved  client error 404; listed URL redirects
500 http://127.0.0.1:8765/broken  server error 500
404 http://127.0.0.1:8765/missing  client error 404
301 http://127.0.0.1:8765/blog/a  redirect chain of 2 hops; temporary redirect in chain
301 http://127.0.0.1:8765/loop-1  redirect loop; redirect chain of 2 hops
3 ok, 8 with issues; report: report.csv

The run took about 8 seconds for 11 sitemap URLs (17 page requests including redirect hops) at the default delay, and it exited with code 1.

Here is how each flagged row maps to the setup:

Listed URL	What the server does	What the script reports
`/old-pricing`	301 to `/pricing`, which is in the sitemap	listed URL redirects
`/plans`	301 to `/pricing/`, whose canonical is `/pricing`	redirects, target not in sitemap, canonicalises elsewhere
`/print/about`	200, canonical points at `/about`	final URL canonicalises elsewhere
`/moved`	301 to `/gone`, which returns 404	client error 404 after a redirect
`/broken`	500	server error 500
`/missing`	404	client error 404
`/blog/a`	301 to `/blog/b`, then 302 to `/blog/c`	chain of 2 hops, temporary redirect in chain
`/loop-1`	301 to `/loop-2`, which 301s back	redirect loop

The /plans row is the one people miss by eye. The redirect works and the page loads, but it lands on the trailing-slash variant of a page whose canonical says the real URL is /pricing. The fix is to list /pricing in the sitemap and point the redirect straight at it.

The CSV has one row per sitemap URL, with the full chain in one column:

url,status,final_url,final_status,hops,chain,canonical,issues
http://127.0.0.1:8765/blog/a,301,http://127.0.0.1:8765/blog/c,200,2,http://127.0.0.1:8765/blog/a [301] -> http://127.0.0.1:8765/blog/b [302] -> http://127.0.0.1:8765/blog/c [200],http://127.0.0.1:8765/blog/c,redirect chain of 2 hops; temporary redirect in chain
http://127.0.0.1:8765/loop-1,301,http://127.0.0.1:8765/loop-1,301,2,http://127.0.0.1:8765/loop-1 [301] -> http://127.0.0.1:8765/loop-2 [301] -> http://127.0.0.1:8765/loop-1,,redirect loop; redirect chain of 2 hops

We also checked three edge cases:

Hop limit. With --max-hops 1, /blog/a stops after the second redirect and reports redirect limit reached (--max-hops 1); redirect chain of 2 hops; temporary redirect in chain.
Local gzip file. A gzipped sitemap saved on disk (local.xml.gz) is read without any extra flag.
Connection errors. A URL on a port with nothing listening is reported as status 0 with error: [Errno 111] Connection refused, and the run continues.

Reading the report

A few rules of thumb for the CSV:

Fix 4xx and 5xx first. Either restore the page, redirect it, or remove it from the sitemap. A 5xx on a single run can be a blip, so rerun those rows before acting.
Listed URL redirects. Replace the sitemap entry with the final_url. Your sitemap generator is usually the real fix, not the file.
Chains. Point the first URL straight at the final one. Chains tend to build up across several migrations, so check old redirect rules too, not only the newest.
Loops. These are always a configuration bug, often two rules (one adding a trailing slash, one removing it) fighting each other.
Temporary redirects. If the move is permanent, use 301 or 308.
Canonicalises elsewhere. List the canonical URL in the sitemap, or fix the canonical tag if it is the tag that is wrong.

Polite defaults and running it on a real site

The defaults are deliberately slow: 4 workers, a 0.5 second gap between request starts, a 15 second timeout and at most 5 follow-up hops. On your own site you can raise --workers and lower --delay if the server copes. On a client's site, ask first and keep the defaults. Change the contact URL in the UA string to something real, so the site owner can see who is crawling and reach you.

Use --limit for a first pass:

python3 sitemap_status.py https://www.example.com/sitemap_index.xml --limit 50 --out sample.csv

For large sites, check a sample from each child sitemap rather than every URL on every run.

Known limits

It reads server HTML only. A canonical tag added by JavaScript after load will not be seen.
Canonical comparison is exact. http versus https, a trailing slash or a different host all count as "elsewhere". That is usually what you want, but expect a few rows to need a human look.
Canonical from HTTP headers is not checked. Some sites send Link: <...>; rel="canonical" as a header, mostly for PDFs. Add a check on resp.headers if you rely on that.
No robots.txt handling. The script requests what the sitemap lists. If you point it at a site you don't run, read its robots.txt yourself.
Status 0 covers several failures. The issues column has the actual error text.
Rate limiting. If a site answers 429, slow down and rerun those rows; the script reports 429 as a client error and does not retry.

Free tools for the steps around the script

If you want to see what a sitemap contains before running anything:

Sitemap URL Extractor (ours, free). Paste sitemap XML or enter a sitemap URL and it returns one URL per line from the urlset <loc> tags. Given a sitemap index, it lists the child sitemap locations; run it again on a child to get that file's page URLs.
XML Sitemap Validator (ours, free). It checks the root element, absolute URLs, lastmod, changefreq, priority, duplicates and the 50,000 URL limit, and gives a score that starts at 100 and drops for errors and warnings.

Both ask for a captcha before each run. Neither requests the listed pages or follows redirects, which is the gap this script fills.

FAQ

Why not use requests or Scrapy?
They are good libraries. This script avoids dependencies so it runs in a minimal CI image or on a client's server with nothing to install.

Why not let urllib follow redirects and compare the final URL?
You lose the chain. You can see that a URL moved, but not whether it took one hop or four, which codes were used, or whether it looped.

Is a single redirect in a sitemap a problem?
It is not an error, but the sitemap should list the final URL. Search engines have to make an extra request for every listed URL that redirects.

Does it check links inside pages?
No. It checks the URLs listed in the sitemap. For links inside page content you need a crawler.

Can I run it in CI?
Yes. It exits with 1 when any URL has an issue. Run it against staging after deploys, with a --limit if the sitemap is large.

Why does my redirect target show "not in sitemap" when the page is fine?
The target URL is not listed exactly as written in the sitemap. Often it is a trailing slash or http versus https difference. Either list the target or point the redirect at the URL you do list.

Wrap-up

Broken links and redirect chains in a sitemap are cheap to find and easy to fix once you have the list. Run the script after migrations, CMS changes and redirect rule edits. If you want the URL list or a protocol check in a browser first, the Appie free tools cover both, and Appie's apps start on a Free plan with 500 credits and no credit card.

RG, Founder, Appie.run

How to Extract and Check JSON-LD Structured Data with Python (Standard Library Only)

appierun — Thu, 05 Nov 2026 15:51:52 +0000

Structured data is the small block of JSON most sites put in their HTML to tell search engines what a page is: an article, a product, an organization, a breadcrumb trail. When it breaks, nothing on the page looks wrong. The page renders, the content is there, and the only symptom is that a rich result quietly stops appearing or a search tool reports an error weeks later.

Google's own Rich Results Test is the right tool for checking one page in depth. But if you want to check 200 pages after a template change, or run a quick check in CI before a deploy, it helps to have a script you can run from the terminal. This post builds one in about 170 lines of Python using only the standard library: no pip install, no BeautifulSoup, no requests.

The script:

fetches a page (or reads a local HTML file)
pulls out every <script type="application/ld+json"> block
reports blocks that are not valid JSON, with the line and column
lists every @type it finds, including nested ones and items inside @graph
flags missing properties for a few common types, based on Google's documentation
accepts a list of URLs from a file, so you can check a whole sitemap

Google documentation details are as of 4 Nov 2026, from Google Search Central's structured data pages. Appie tool descriptions are from appie.run/tools on the same day.

Disclosure: Appie.run publishes this article. Two free tools mentioned near the end are ours.

What Google actually says about JSON-LD

A few points from Google Search Central's introduction to structured data and its general guidelines shape what the script does:

Google supports JSON-LD, Microdata and RDFa, and recommends JSON-LD.
Google can read JSON-LD that JavaScript injects into the page after load. Our script reads the HTML as the server sends it, so it will not see injected markup. That is a real limit, and the script says so when it finds nothing.
Required properties must be present for a page to be eligible for a rich result. Recommended properties are optional but can help.
Google's documentation, not schema.org, is the reference for what Google uses. schema.org defines far more types and properties than Google supports.
Valid markup does not guarantee a rich result.
Do not mark up content that is not visible on the page.

The "required" versus "recommended" split varies by type. A few examples from Google's type pages:

Article, BlogPosting, NewsArticle: Required by Google: None; Recommended by Google: headline, image, datePublished, dateModified, author
Organization: Required by Google: None; Recommended by Google: Many, including name, url, logo, sameAs
Product (product snippet): Required by Google: name, plus one of review, aggregateRating or offers; Recommended by Google: Several
BreadcrumbList: Required by Google: itemListElement; each ListItem needs position, name, and item (the last item may omit item); Recommended by Google: None listed

The script encodes exactly these and nothing more. You can extend the CHECKS dictionary for the types you use.

The script

Save this as jsonld_audit.py. It needs Python 3.8 or later; we tested it on Python 3.13.

#!/usr/bin/env python3
"""Extract JSON-LD blocks from web pages and flag common problems.

Standard library only. Usage:
    python3 jsonld_audit.py https://example.com/page [more URLs or local .html files]
    python3 jsonld_audit.py @urls.txt      (one URL or file path per line)
"""
import json
import sys
import urllib.request
from html.parser import HTMLParser

# Checks based on Google Search Central docs (read 4 Nov 2026).
# "required" means Google lists it as required for that rich result;
# "recommended" means Google lists it as recommended.
CHECKS = {
    "Article":        {"required": [], "recommended": ["headline", "image", "datePublished", "dateModified", "author"]},
    "NewsArticle":    {"required": [], "recommended": ["headline", "image", "datePublished", "dateModified", "author"]},
    "BlogPosting":    {"required": [], "recommended": ["headline", "image", "datePublished", "dateModified", "author"]},
    "Organization":   {"required": [], "recommended": ["name", "url", "logo", "sameAs"]},
    "Product":        {"required": ["name"], "one_of": ["offers", "review", "aggregateRating"]},
    "BreadcrumbList": {"required": ["itemListElement"]},
}


class JSONLDParser(HTMLParser):
    """Collects the text inside every <script type="application/ld+json">."""

    def __init__(self):
        super().__init__(convert_charrefs=True)
        self.blocks = []
        self._in_ld = False
        self._buf = []

    def handle_starttag(self, tag, attrs):
        if tag == "script":
            kind = (dict(attrs).get("type") or "").split(";")[0].strip().lower()
            if kind == "application/ld+json":
                self._in_ld = True
                self._buf = []

    def handle_data(self, data):
        if self._in_ld:
            self._buf.append(data)

    def handle_endtag(self, tag):
        if tag == "script" and self._in_ld:
            self.blocks.append("".join(self._buf))
            self._in_ld = False


def load(source):
    if source.startswith(("http://", "https://")):
        req = urllib.request.Request(source, headers={"User-Agent": "jsonld-audit/1.0"})
        with urllib.request.urlopen(req, timeout=20) as resp:
            charset = resp.headers.get_content_charset() or "utf-8"
            return resp.read().decode(charset, errors="replace")
    with open(source, encoding="utf-8", errors="replace") as fh:
        return fh.read()


def iter_items(node):
    """Yield every dict that has an @type, walking @graph, lists and nesting."""
    if isinstance(node, list):
        for item in node:
            yield from iter_items(item)
    elif isinstance(node, dict):
        if "@type" in node:
            yield node
        for key, value in node.items():
            if key != "@type":
                yield from iter_items(value)


def top_level(data):
    """The items a block declares directly: a list, an @graph, or one object."""
    if isinstance(data, list):
        return [d for d in data if isinstance(d, dict)]
    if isinstance(data, dict) and isinstance(data.get("@graph"), list):
        return [d for d in data["@graph"] if isinstance(d, dict)]
    return [data] if isinstance(data, dict) else []


def types_of(item):
    t = item.get("@type")
    return t if isinstance(t, list) else [t]


def empty(value):
    return value in (None, "", [], {})


def check_item(item):
    problems = []
    for t in types_of(item):
        rule = CHECKS.get(t)
        if not rule:
            continue
        for prop in rule.get("required", []):
            if empty(item.get(prop)):
                problems.append(f"{t}: missing required '{prop}'")
        one_of = rule.get("one_of")
        if one_of and all(empty(item.get(p)) for p in one_of):
            problems.append(f"{t}: needs one of {', '.join(one_of)}")
        for prop in rule.get("recommended", []):
            if empty(item.get(prop)):
                problems.append(f"{t}: no recommended '{prop}'")
        if t == "BreadcrumbList":
            elements = item.get("itemListElement") or []
            for i, li in enumerate(elements, 1):
                if not isinstance(li, dict):
                    continue
                if empty(li.get("position")):
                    problems.append(f"BreadcrumbList item {i}: missing 'position'")
                if empty(li.get("name")) and not isinstance(li.get("item"), dict):
                    problems.append(f"BreadcrumbList item {i}: missing 'name'")
                if empty(li.get("item")) and i < len(elements):
                    problems.append(f"BreadcrumbList item {i}: missing 'item' (only the last may omit it)")
    return problems


def audit(source):
    print(f"\n== {source}")
    try:
        html = load(source)
    except Exception as exc:  # network errors, 404s, missing files
        print(f"  could not load: {exc}")
        return 1
    parser = JSONLDParser()
    parser.feed(html)
    if not parser.blocks:
        print("  no JSON-LD found in the HTML as served (JavaScript-injected markup is not seen)")
        return 1
    issues = 0
    for n, raw in enumerate(parser.blocks, 1):
        try:
            data = json.loads(raw)
        except json.JSONDecodeError as exc:
            print(f"  block {n}: invalid JSON ({exc.msg} at line {exc.lineno}, col {exc.colno})")
            issues += 1
            continue
        items = list(iter_items(data))
        names = sorted({str(t) for item in items for t in types_of(item)})
        print(f"  block {n}: {len(items)} item(s): {', '.join(names) or 'no @type'}")
        if isinstance(data, dict) and "@context" not in data and "@graph" not in data:
            print(f"    warning: no @context")
        for item in top_level(data):
            for p in check_item(item):
                print(f"    {p}")
                issues += 1
    return 1 if issues else 0


if __name__ == "__main__":
    if len(sys.argv) < 2:
        sys.exit(__doc__)
    sources = []
    for arg in sys.argv[1:]:
        if arg.startswith("@"):
            with open(arg[1:], encoding="utf-8") as fh:
                sources += [line.strip() for line in fh if line.strip() and not line.startswith("#")]
        else:
            sources.append(arg)
    results = [audit(src) for src in sources]
    sys.exit(1 if any(results) else 0)

How it works

Finding the blocks. html.parser.HTMLParser walks the document tag by tag. When it sees a <script> tag whose type is application/ld+json, it starts collecting text until the closing </script>. The split(";") handles the occasional type="application/ld+json; charset=utf-8". This is more reliable than a regular expression, which tends to break on attributes in a different order or on extra whitespace.

Parsing. Each block goes through json.loads. When that fails, the JSONDecodeError gives a message and a position. Trailing commas, unescaped quotes inside strings, and template tags that rendered as empty values are the usual causes. A broken block is skipped entirely by parsers, so this is the most important thing the script catches.

Listing types. iter_items walks the parsed JSON recursively and yields any object with an @type. That picks up nested items such as an author that is a Person or an offers that is an Offer, and it handles @graph arrays, which many CMS plugins use to put several items in one block.

Checking. top_level returns only the items a block declares directly (the object itself, a list, or the members of @graph). The checks run on those, not on every nested object. That keeps the output quiet: a publisher Organization nested inside an Article is not expected to carry every recommended Organization property.

Exit code. The script exits with 1 if any page could not be loaded, had no JSON-LD, or had at least one issue. That makes it easy to use in CI.

Try it on a test page

Here is a deliberately flawed file, sample.html. It has a blog post without an image or modified date, a correct breadcrumb, a Product block with a trailing comma, and a Product with no offers, review or rating:

<html><head>
<script type="application/ld+json">
{"@context":"https://schema.org","@type":"BlogPosting","headline":"How we price","datePublished":"2026-11-01","author":{"@type":"Person","name":"RG"}}
</script>
<script type="application/ld+json">
{"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[
 {"@type":"ListItem","position":1,"name":"Blog","item":"https://example.com/blog"},
 {"@type":"ListItem","position":2,"name":"How we price"}]}
</script>
<script type="application/ld+json">
{"@context":"https://schema.org","@type":"Product","name":"Starter kit",}
</script>
<script type="application/ld+json">
{"@context":"https://schema.org","@type":"Product","name":"Pro kit"}
</script>
</head><body></body></html>

Running python3 jsonld_audit.py sample.html printed this on our machine:

== sample.html
  block 1: 2 item(s): BlogPosting, Person
    BlogPosting: no recommended 'image'
    BlogPosting: no recommended 'dateModified'
  block 2: 3 item(s): BreadcrumbList, ListItem
  block 3: invalid JSON (Illegal trailing comma before end of object at line 2, col 72)
  block 4: 1 item(s): Product
    Product: needs one of offers, review, aggregateRating

Each problem maps to something you would fix in the template: add image and dateModified to the blog post, remove the trailing comma, and either add offers to the second product or stop marking it up as a product.

Try it on live pages

We ran it against three public pages on 4 Nov 2026:

== https://appie.run/
  block 1: 6 item(s): Offer, Organization, SoftwareApplication, WebSite

== https://www.python.org/
  block 1: 2 item(s): SearchAction, WebSite

== https://example.com/
  no JSON-LD found in the HTML as served (JavaScript-injected markup is not seen)

The first two pages have valid JSON-LD with no flagged issues for the types the script checks. The third has no JSON-LD in its served HTML, which is expected for a placeholder domain. On your own site, "no JSON-LD found" on a page you expected to have markup means one of two things: the markup is missing, or it is injected by JavaScript. Open the page in Google's Rich Results Test to tell which.

Check a whole site from its sitemap

Put one URL per line in a text file and pass it with @:

python3 jsonld_audit.py @urls.txt > jsonld-report.txt

To build urls.txt from a sitemap, you can reuse the parser from our earlier post on validating an XML sitemap with Python, or paste the sitemap into a free extractor (more on that below). For large sites, sample one or two URLs per template (home, category, product, article) rather than every page. Structured data is almost always generated by templates, so if one product page is wrong, they usually all are.

Be polite when you run it against a site you do not own: add a short time.sleep() between requests and keep the list small.

Running it in CI

Because the script exits non-zero on problems, you can run it against a staging build before deploy:

python3 jsonld_audit.py https://staging.example.com/ https://staging.example.com/blog/sample-post/ || exit 1

If "no recommended property" warnings are too noisy for a blocking check, split them out: return 1 only for invalid JSON and missing required properties, and print the rest as warnings.

Ideas for extending it

More types. Add the types you use to CHECKS, taking required and recommended properties from Google's page for each type, not from schema.org.
Date format. Check that datePublished and dateModified parse with datetime.fromisoformat.
URL checks. Confirm that url, image and breadcrumb item values are absolute URLs on the expected host.
Visible content. Compare headline against the page's <h1> or <title> to catch markup that has drifted from what users see, which Google's guidelines warn against.
CSV output. Write one row per page and issue with the csv module for a spreadsheet-friendly report.

Tools that help around the script

The script covers the checking. A couple of free tools help on either side of it.

Google's Rich Results Test and Search Console. The Rich Results Test is the authority on whether a page is eligible for a specific rich result, and it renders JavaScript, so it sees injected markup. Search Console reports structured data issues across your site over time.

Appie Sitemap URL Extractor (ours, free). Paste sitemap XML or enter a sitemap URL and it returns one URL per line, which is the format @urls.txt expects. If you give it a sitemap index, it lists the child sitemaps; run it again on a child sitemap to get page URLs. It is at appie.run/tools/sitemap-url-extractor.

Appie Webpage to FAQ Generator (ours, free). If your audit shows pages with thin content and you want to add an FAQ section, this tool reads a single public page and writes FAQs from its text, skipping claims the page does not make. Review the result before you publish it. We covered how FAQ content fits AI answers now that Google has changed how FAQ rich results appear in this earlier post.

FAQ

Why not use BeautifulSoup or extruct?
They are good libraries. This script avoids dependencies so it runs anywhere Python does, including minimal CI images, with nothing to install.

Does valid JSON-LD guarantee a rich result?
No. Google's guidelines say structured data makes a page eligible but does not guarantee a rich result.

Why does the script say no JSON-LD on a page that has it?
The markup is probably added by JavaScript after the page loads. Google can read that, but this script reads only the HTML the server returns. Check with the Rich Results Test.

Should I follow schema.org or Google's docs?
For Google features, Google's docs. schema.org defines the vocabulary; Google documents which parts it uses and what it requires.

Does it support Microdata or RDFa?
No, only JSON-LD, which is the format Google recommends.

Can I run it on hundreds of URLs?
Yes, with the @urls.txt option. Add a delay between requests and sample by template where you can.

Wrap-up

Structured data breaks silently, usually because of a template change, and a small script catches most of it before a search tool does. If you want the sitemap URL list without writing more code, the Appie free tools include a sitemap URL extractor, and Appie's apps start on a Free plan with 500 credits and no credit card.

RG, Founder, Appie.run

Which AI Crawlers Does Your robots.txt Block? Check GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot Per URL With Python

appierun — Wed, 04 Nov 2026 15:55:08 +0000

Most teams now have an opinion about AI crawlers. Block training, allow search. Or allow everything so the product shows up in ChatGPT and Perplexity answers. Or block it all. The opinion is the easy part. Knowing what your robots.txt actually says to each crawler, for each URL, is harder than it looks, because the rules for picking a group and picking a winning rule are not the ones most people assume.

This post builds a small checker with Python's standard library. You give it a site (or a local robots.txt) and a list of paths, and it prints a table: one row per URL, one column per AI crawler, allowed or blocked. With --why it prints the exact rule that decided each cell. With --sitemap it tests every URL in your sitemap.

Dated note: crawler names and what each one is for were checked on each vendor's own documentation on 3 Nov 2026. Vendors add and rename crawlers, so check their pages before you ship a change.

Disclosure: Appie.run publishes this post. Appie has a free Sitemap to Robots.txt Generator and other sitemap tools, mentioned near the end. The script does not depend on them.

Earlier posts in this series cover validating an XML sitemap and getting AI assistants to describe your product correctly. This one sits between them: before an assistant can describe your pages, its crawler has to be allowed to read them.

Training, search and user fetches are different crawlers

Each big AI vendor now runs more than one crawler, and they do different jobs. Here is what each vendor's own page says, as of 3 Nov 2026:

GPTBot: vendor: OpenAI; what the vendor says it is for: Training. Disallowing it indicates content should not be used for training OpenAI's foundation models.
OAI-SearchBot: vendor: OpenAI; what the vendor says it is for: ChatGPT search results. OpenAI says changes can take about 24 hours to apply.
ChatGPT-User: vendor: OpenAI; what the vendor says it is for: Fetches a user asked for. OpenAI says that because these are user-initiated, "robots.txt rules may not apply".
ClaudeBot: vendor: Anthropic; what the vendor says it is for: Training.
Claude-SearchBot: vendor: Anthropic; what the vendor says it is for: Improving search result quality.
Claude-User: vendor: Anthropic; what the vendor says it is for: Fetches a user asked for. Anthropic says opt-outs are set per subdomain.
PerplexityBot: vendor: Perplexity; what the vendor says it is for: Surfacing sites in Perplexity search results. Perplexity says it is not used to train foundation models.
Perplexity-User: vendor: Perplexity; what the vendor says it is for: Fetches a user asked for. Perplexity says it "generally ignores robots.txt rules".
Google-Extended: vendor: Google; what the vendor says it is for: A control token, not a separate crawler. It covers Gemini training and grounding, and Google says it does not affect Search.

Sources: OpenAI crawlers, Anthropic's crawler article, Perplexity crawlers, Google's common crawlers.

Two things follow. First, "block AI" in one line usually blocks more than you meant, or less. Blocking GPTBot does not remove you from ChatGPT search; that is OAI-SearchBot. Second, the user-fetch agents are documented as not always following robots.txt, so the checker reports what your file says to them, not what they will do.

The four rules that decide the answer

The robots.txt standard is RFC 9309, and Google's robots.txt spec matches it on the points below.

One group per crawler. A crawler uses the group whose User-agent matches its token (case-insensitive). Only if none matches does it fall back to User-agent: *. A specific group replaces the * group. It does not add to it. If you write a GPTBot group with one rule, none of your * rules apply to GPTBot any more.
Groups for the same token merge. Two User-agent: * blocks in one file are combined into one group. This matters more than it used to, because some CDNs now prepend their own block to your file (more on that below).
Longest match wins. Inside a group, the rule with the longest matching path decides. File order does not matter. If an Allow and a Disallow are equally long, Allow wins.
* and $ are wildcards. * matches any run of characters, and a trailing $ anchors the end of the URL. Disallow: /pricing$ blocks /pricing but not /pricing?plan=pro.

Two status code rules matter too. If /robots.txt returns a 4xx, crawlers may treat the site as unrestricted (Google makes an exception for 429). If it returns a 5xx, RFC 9309 says to assume everything is disallowed. And /robots.txt itself is always allowed.

Why not urllib.robotparser?

Python ships urllib.robotparser, and it is fine for simple files. It predates RFC 9309, though, and in Python 3.13 it differs on the rules above:

It applies the first rule in file order that matches, not the longest.
It does not support * or $ inside paths.
It keeps only the first User-agent: * group and ignores later ones.
It matches user-agent names as substrings rather than exact tokens.

You can see the first three in its source (Entry.allowance, RuleLine.applies_to and _add_entry). Here is the effect on the sample file used below. This script compares the two:

"""Where urllib.robotparser and RFC 9309 matching disagree on the sample file."""
import urllib.robotparser
from ai_robots_check import AI_CRAWLERS, Robots

text = open("robots-sample.txt", encoding="utf-8").read()
ours = Robots(text)
std = urllib.robotparser.RobotFileParser()
std.parse(text.splitlines())

paths = ["/", "/pricing", "/blog/post-1", "/blog/drafts/x", "/admin/settings",
         "/admin/help/faq", "/docs/guide.pdf", "/shop?session=abc"]
for path in paths:
    url = "https://example.com" + path
    for name, tokens in AI_CRAWLERS.items():
        a = ours.check(tokens, url)[0]
        b = std.can_fetch(name, url)
        if a != b:
            print(f"{path:18} {name:16} RFC 9309: {'allow' if a else 'block':5}  robotparser: {'allow' if b else 'block'}")

Output on Python 3.13.5:

/pricing           OAI-SearchBot    RFC 9309: block  robotparser: allow
/blog/drafts/x     OAI-SearchBot    RFC 9309: block  robotparser: allow
/admin/help/faq    ChatGPT-User     RFC 9309: allow  robotparser: block
/admin/help/faq    Claude-SearchBot RFC 9309: allow  robotparser: block
/admin/help/faq    Claude-User      RFC 9309: allow  robotparser: block
/admin/help/faq    Perplexity-User  RFC 9309: allow  robotparser: block
/admin/help/faq    Googlebot        RFC 9309: allow  robotparser: block
/docs/guide.pdf    PerplexityBot    RFC 9309: block  robotparser: allow
/shop?session=abc  ChatGPT-User     RFC 9309: block  robotparser: allow
/shop?session=abc  Claude-SearchBot RFC 9309: block  robotparser: allow
/shop?session=abc  Claude-User      RFC 9309: block  robotparser: allow
/shop?session=abc  Perplexity-User  RFC 9309: block  robotparser: allow
/shop?session=abc  Googlebot        RFC 9309: block  robotparser: allow

Thirteen disagreements on a 21-line file. /blog/drafts/x is the clearest: the file says Allow: /blog/ then Disallow: /blog/drafts/. The longer rule should win and block drafts, but robotparser stops at the first match and allows them. So the checker below does its own matching.

The script

Save this as ai_robots_check.py. It needs Python 3.8 or later and nothing from pip.

#!/usr/bin/env python3
"""Check which AI crawlers your robots.txt allows, per URL, with the standard library only.

Matching follows RFC 9309 (the robots.txt standard Google also documents):
one group per crawler (most specific user-agent wins, '*' is the fallback),
longest matching rule wins, and Allow wins a tie.
"""
import argparse
import re
import sys
import urllib.error
import urllib.parse
import urllib.request
import xml.etree.ElementTree as ET

# robots.txt tokens, as published by each vendor (checked 3 Nov 2026).
# Each entry lists tokens to try in order; the first one with a group wins.
AI_CRAWLERS = {
    "GPTBot": ["gptbot"],                    # OpenAI, model training
    "OAI-SearchBot": ["oai-searchbot"],      # OpenAI, ChatGPT search results
    "ChatGPT-User": ["chatgpt-user"],        # OpenAI, user-triggered fetches
    "ClaudeBot": ["claudebot"],              # Anthropic, model training
    "Claude-SearchBot": ["claude-searchbot"],  # Anthropic, search quality
    "Claude-User": ["claude-user"],          # Anthropic, user-triggered fetches
    "PerplexityBot": ["perplexitybot"],      # Perplexity, search results
    "Perplexity-User": ["perplexity-user"],  # Perplexity, user-triggered fetches
    "Google-Extended": ["google-extended"],  # Gemini training and grounding
    "Googlebot": ["googlebot"],              # Google Search, for comparison
}
UA = "ai-robots-check/1.0 (+https://example.com/contact)"


def fetch(url, limit=2_000_000):
    req = urllib.request.Request(url, headers={"User-Agent": UA})
    with urllib.request.urlopen(req, timeout=20) as resp:
        return resp.read(limit).decode("utf-8", "replace")


def normalize(path):
    """Percent-encode consistently so '/caf%C3%A9' and '/cafÃ©' compare equal."""
    return urllib.parse.quote(urllib.parse.unquote(path), safe="/?=&*$%:@!,;+~")


def to_regex(pattern):
    """robots.txt path pattern -> regex. '*' is any run of characters, a final '$' anchors."""
    anchored = pattern.endswith("$")
    if anchored:
        pattern = pattern[:-1]
    body = ".*".join(re.escape(part) for part in normalize(pattern).split("*"))
    return re.compile(body + ("$" if anchored else ""))


class Robots:
    def __init__(self, text):
        self.groups = {}      # lowercased token -> list of (allow, pattern, regex)
        self.sitemaps = []
        self._parse(text)

    def _parse(self, text):
        agents, in_rules = [], False
        for raw in text.splitlines():
            line = raw.split("#", 1)[0].strip()
            if ":" not in line:
                continue
            field, value = (s.strip() for s in line.split(":", 1))
            field = field.lower()
            if field == "user-agent":
                if in_rules:          # a user-agent after rules starts a new group
                    agents, in_rules = [], False
                token = value.split("/")[0].strip().lower()  # 'GPTBot/1.4' -> 'gptbot'
                agents.append(token)
                self.groups.setdefault(token, [])  # groups for one token are merged
            elif field in ("allow", "disallow"):
                in_rules = True
                if not value:         # 'Disallow:' with no path means allow everything
                    continue
                for token in agents:
                    self.groups[token].append((field == "allow", value, to_regex(value)))
            elif field == "sitemap":
                self.sitemaps.append(value)
            # Crawl-delay, Content-Signal and any other fields are ignored here.

    def group_for(self, tokens):
        for token in tokens:
            if token in self.groups:
                return token
        return "*" if "*" in self.groups else None

    def check(self, tokens, url):
        """Return (allowed, group, matching rule or None)."""
        parts = urllib.parse.urlsplit(url)
        path = normalize((parts.path or "/") + ("?" + parts.query if parts.query else ""))
        if path == "/robots.txt":
            return True, None, None   # robots.txt itself is always allowed
        group = self.group_for(tokens)
        if group is None:
            return True, None, None
        best = None
        for allow, pattern, rx in self.groups[group]:
            if rx.match(path):
                length = len(pattern.rstrip("$"))
                # longest pattern wins; on a tie, Allow wins
                if best is None or length > best[0] or (length == best[0] and allow):
                    best = (length, allow, pattern)
        if best is None:
            return True, group, None
        return best[1], group, ("Allow: " if best[1] else "Disallow: ") + best[2]


def sitemap_urls(url, limit):
    """Read <loc> URLs from a sitemap or sitemap index (one level of nesting)."""
    root = ET.fromstring(fetch(url, 50_000_000))
    locs = [el.text.strip() for el in root.iter() if el.tag.endswith("loc") and el.text]
    if root.tag.endswith("sitemapindex"):
        out = []
        for child in locs:
            out += sitemap_urls(child, limit - len(out))
            if len(out) >= limit:
                break
        return out[:limit]
    return locs[:limit]


def main():
    ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    ap.add_argument("site", help="https://example.com (robots.txt is read from its root)")
    ap.add_argument("paths", nargs="*", help="paths or URLs to test (default: /)")
    ap.add_argument("--robots-file", help="test a local robots.txt instead of the live one")
    ap.add_argument("--sitemap", action="store_true", help="also test URLs from the Sitemap: lines")
    ap.add_argument("--limit", type=int, default=200, help="max sitemap URLs to test")
    ap.add_argument("--why", action="store_true", help="print the rule that decided each result")
    args = ap.parse_args()

    base = args.site.rstrip("/")
    if args.robots_file:
        text = open(args.robots_file, encoding="utf-8").read()
    else:
        try:
            text = fetch(base + "/robots.txt", limit=500 * 1024)  # Google reads 500 KiB
        except urllib.error.HTTPError as e:
            if 400 <= e.code < 500 and e.code != 429:
                text = ""   # RFC 9309: a 4xx means no restrictions
                print(f"robots.txt returned {e.code}: crawlers may treat everything as allowed")
            else:
                sys.exit(f"robots.txt returned {e.code}: treat the whole site as blocked "
                         "until it loads (RFC 9309, 'unreachable')")
        except urllib.error.URLError as e:
            sys.exit(f"could not reach {base}/robots.txt: {e.reason}")
    robots = Robots(text)

    default = [] if args.sitemap else ["/"]
    urls = [p if p.startswith("http") else base + "/" + p.lstrip("/") for p in (args.paths or default)]
    if args.sitemap:
        for sm in robots.sitemaps:
            urls += sitemap_urls(sm, args.limit)
    urls = list(dict.fromkeys(urls))  # drop duplicates, keep order
    if not urls:
        sys.exit("no URLs to test (no paths given and no Sitemap: lines found)")

    short = {name: name.replace("Perplexity", "Pplx").replace("Google-", "G-") for name in AI_CRAWLERS}
    label = {u: urllib.parse.urlsplit(u)._replace(scheme="", netloc="").geturl() or "/" for u in urls}
    width = max(len(v) for v in label.values())
    if not args.why:
        print(" " * width + "  " + " ".join(f"{short[n]:>{len(short[n])}}" for n in AI_CRAWLERS))
    blocked = {name: 0 for name in AI_CRAWLERS}
    for url in urls:
        cells = []
        for name, tokens in AI_CRAWLERS.items():
            ok, group, rule = robots.check(tokens, url)
            blocked[name] += not ok
            if args.why:
                print(f"{label[url]}  {name}: {'allowed' if ok else 'BLOCKED'} "
                      f"(group {group or 'none'}, rule {rule or 'none matched'})")
            cells.append(f"{('ok' if ok else 'NO'):>{len(short[name])}}")
        if not args.why:
            print(f"{label[url]:<{width}}  " + " ".join(cells))
    print(f"\nBlocked counts across {len(urls)} URL(s):")
    for name, n in blocked.items():
        if n:
            print(f"  {name}: {n}")
    if not any(blocked.values()):
        print("  none")


if __name__ == "__main__":
    main()

A few choices worth explaining:

AI_CRAWLERS is a plain dict. Each value is a list of tokens to try in order, so you can add a fallback (for example ["googlebot-news", "googlebot"]) or a new crawler without touching the logic.
Rule length counts the pattern as written, wildcards included, which matches Google's documented examples (allow: /page against disallow: /*.ph is a tie, so Allow wins).
Paths are percent-encoded before comparing, so /cafÃ© and /caf%C3%A9 are the same rule.
Only 500 KiB of robots.txt is read. That is Google's documented limit; anything after it is ignored.
Crawl-delay and Content-Signal lines are skipped. They do not change allow or block.

Try it

Here is a sample file with the patterns that usually cause surprises:

# Sample robots.txt with the patterns that trip people up
User-agent: *
Disallow: /admin/
Disallow: /*?session=
Allow: /admin/help/

User-agent: GPTBot
User-agent: ClaudeBot
User-Agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
Disallow: /pricing$
Allow: /blog/
Disallow: /blog/drafts/

User-agent: PerplexityBot
Crawl-delay: 5
Disallow: /*.pdf$

Sitemap: https://example.com/sitemap.xml

Run the checker against it:

python3 ai_robots_check.py https://example.com / /pricing "/pricing?plan=pro" \
  /blog/post-1 /blog/drafts/x /admin/settings /admin/help/faq /docs/guide.pdf \
  "/shop?session=abc" --robots-file robots-sample.txt

Output:

                   GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Claude-SearchBot Claude-User PplxBot Pplx-User G-Extended Googlebot
/                      NO            ok           ok        NO               ok          ok      ok        ok         NO        ok
/pricing               NO            NO           ok        NO               ok          ok      ok        ok         NO        ok
/pricing?plan=pro      NO            ok           ok        NO               ok          ok      ok        ok         NO        ok
/blog/post-1           NO            ok           ok        NO               ok          ok      ok        ok         NO        ok
/blog/drafts/x         NO            NO           ok        NO               ok          ok      ok        ok         NO        ok
/admin/settings        NO            ok           NO        NO               NO          NO      ok        NO         NO        NO
/admin/help/faq        NO            ok           ok        NO               ok          ok      ok        ok         NO        ok
/docs/guide.pdf        NO            ok           ok        NO               ok          ok      NO        ok         NO        ok
/shop?session=abc      NO            ok           NO        NO               NO          NO      ok        NO         NO        NO

Blocked counts across 9 URL(s):
  GPTBot: 9
  OAI-SearchBot: 2
  ChatGPT-User: 2
  ClaudeBot: 9
  Claude-SearchBot: 2
  Claude-User: 2
  PerplexityBot: 1
  Perplexity-User: 2
  Google-Extended: 9
  Googlebot: 2

Look at the /admin/settings row. Most bots are blocked by Disallow: /admin/ in the * group, but OAI-SearchBot and PerplexityBot are allowed. They have their own groups, so the * rules never apply to them. This is the most common surprise in real files: someone adds a group for one AI crawler and quietly drops it out of every general rule. If you want /admin/ blocked for everyone, repeat it in each named group.

Add --why to see the deciding rule:

/pricing  OAI-SearchBot: BLOCKED (group oai-searchbot, rule Disallow: /pricing$)
/pricing  ChatGPT-User: allowed (group *, rule none matched)
/blog/drafts/x  OAI-SearchBot: BLOCKED (group oai-searchbot, rule Disallow: /blog/drafts/)
/admin/help/faq  OAI-SearchBot: allowed (group oai-searchbot, rule none matched)
/admin/help/faq  ChatGPT-User: allowed (group *, rule Allow: /admin/help/)

(Trimmed to the interesting lines.) /admin/help/faq is allowed for *-group bots because Allow: /admin/help/ is longer than Disallow: /admin/. Order in the file plays no part.

When a CDN writes part of your robots.txt

Cloudflare's robots.txt setting docs (last updated 2 Nov 2026) say that when its managed robots.txt is on and your origin already has a robots.txt, Cloudflare puts its managed block in front of your file and serves both as one response. The managed block includes a User-agent: * group with Allow: /, and your own file often has a User-agent: * group too. Under RFC 9309 those two groups merge. Under urllib.robotparser, only the first one counts.

A cut-down version of that shape:

# BEGIN Cloudflare Managed content
User-agent: *
Content-Signal: search=yes,ai-train=no
Allow: /

User-agent: GPTBot
Disallow: /
# END Cloudflare Managed Content

User-agent: *
Disallow: /admin/
Disallow: /checkout/

For ClaudeBot (no group of its own in this cut-down file, so it falls back to *), the checker says /admin/users and /checkout/ are blocked, because the merged * group has Disallow: /admin/ and that is longer than Allow: /. urllib.robotparser says both are allowed, because it never sees your origin's * group. If you audit a Cloudflare-fronted site with the standard library parser, you can get the wrong answer for every URL your own rules cover.

The Content-Signal line is a separate preference signal. Cloudflare's docs say robots.txt compliance is voluntary and the file does not stop access at a technical level. The checker skips the line because it does not change allow or block.

Test every URL in your sitemap

Point the checker at a live site and add --sitemap. It reads the Sitemap: lines from robots.txt, follows a sitemap index one level, and tests up to --limit URLs:

python3 ai_robots_check.py https://yourdomain.com --sitemap --limit 500

This is the version to run after a robots.txt change, a CDN setting change or a site migration. A count like OAI-SearchBot: 37 when you expected zero tells you a group or wildcard is catching pages you wanted in ChatGPT search. Re-run with --why and the paths in question to find the rule.

To keep it from drifting, run it in CI against a staging robots.txt with --robots-file and fail the build if the blocked counts change. A short shell step that diffs the output against a saved copy is enough.

Known limits

It reads one host. Rules are per host and per subdomain. blog.example.com has its own robots.txt.
It tests what the file says, not what bots do. The user-fetch agents above are documented as not always following robots.txt. Firewalls, bot rules at your CDN and IP blocks are a separate layer the script cannot see.
No redirect handling beyond urllib's defaults. RFC 9309 allows up to five redirects for robots.txt. urllib follows redirects, but the script does not count them.
It does not cache. Crawlers may cache robots.txt for up to 24 hours, so a fix you just deployed may take a day to be seen.
The crawler list is a snapshot. New tokens appear often. Check vendor docs and add them to AI_CRAWLERS.
robots.txt is not noindex. Blocking a crawler stops it reading the page. Google's docs say a disallowed URL can still be indexed without a snippet. Use noindex (on a crawlable page) if you need it out of results.

When you do not want to run a script

Appie has free sitemap and robots tools in the browser. Each run asks for a Cloudflare Turnstile check:

Sitemap to Robots.txt Generator: you set a user agent, list allow and disallow paths one per line, add sitemap URLs, and download a robots.txt file. The page says to save it at the site root. It writes one group per run, so for a file with separate AI crawler groups, run it once per group and join the results, then check the joined file with the script above.
Sitemap URL Extractor: paste sitemap XML or a sitemap URL and get one URL per line. The page says that for a sitemap index it lists the child sitemaps, and you run each child to get its page URLs. That list is a good input for the checker's paths argument.
XML Sitemap Validator: paste sitemap XML or give its URL, and the page says it checks the root element, absolute URLs, lastmod, changefreq, priority, duplicates and the 50,000 URL limit. Worth running on the sitemap you list in robots.txt.

If what you want is for AI tools and visitors to get correct answers about your product, the crawler side is only half of it. Appie's AI Chatbots app trains a website chatbot on your URLs, sitemaps, files and Q&A pairs, so visitors on your own site get answers from your pages.

Honest limits of Appie's robots and sitemap tools

The generator writes rules; it does not test them. It will not tell you that a named group drops your * rules. That is what the checker is for.
One run at a time, with a fresh captcha each time. For hundreds of URLs or a CI check, use the script.

FAQ

If I block GPTBot, will my site disappear from ChatGPT?
Not from ChatGPT search, based on OpenAI's docs. GPTBot is for training. Search uses OAI-SearchBot, which OpenAI says is a separate, independent setting. Users can also still paste or ask for a URL, which uses ChatGPT-User.

Does Google-Extended affect my Google rankings?
Google's crawler page says Google-Extended does not affect inclusion or ranking in Google Search. It is a control token for Gemini training and grounding, with no user agent of its own, so you will not see it in your logs.

Why does my named AI crawler group ignore my Disallow: /admin/?
Because a crawler with its own group does not read the * group at all. Copy the rules you want for everyone into every named group.

What happens if my robots.txt returns a 500 error?
RFC 9309 says crawlers should assume everything is disallowed while the file is unreachable. Google's spec describes its own handling of 5xx errors in more detail. Either way, a broken robots.txt can stop crawling, so monitor it like any other endpoint.

Can I use this for non-AI crawlers?
Yes. Add any token to AI_CRAWLERS, for example "Bingbot": ["bingbot"]. The matching rules are the same for every crawler that follows RFC 9309.

Does Crawl-delay matter?
Some crawlers read it and some do not. Anthropic's article says it supports Crawl-delay, and Google's spec does not list it as a supported field. The script ignores it because it does not change allow or block.

If you want a chatbot on your site that answers from your own pages, you can try Appie's AI Chatbots on the Free plan (500 credits, no credit card) at appie.run.

RG, Founder, Appie.run

Convert Word (DOCX) to Markdown With Python (Standard Library Only) for LLM Prompts and Chatbot Knowledge

appierun — Tue, 03 Nov 2026 15:56:12 +0000

A lot of company knowledge still lives in Word files: refund policies, onboarding guides, SOPs, proposal templates. If you want that content in a prompt, a RAG index or a chatbot's knowledge base, Markdown is a better format than the .docx itself or a flat text dump. It keeps headings, lists, tables and links, and it is easy to diff, chunk and review.

This post builds a DOCX-to-Markdown converter with Python's standard library only (zipfile and xml.etree.ElementTree), so it runs in a CI job, a small container or a serverless function without a pip install. It handles headings, bold and italic, links, nested bullet and numbered lists, tables, line breaks, images and tracked changes.

Disclosure: Appie.run publishes this post. Appie has a free DOCX to Markdown converter and a few other Word tools, mentioned near the end. The script below does not depend on them.

Earlier posts in this series cover converting CSV and JSON to Markdown tables and converting PDFs to Markdown for RAG.

What is inside a .docx

A .docx file is a ZIP archive of XML files. Rename one to .zip and open it, and you will find:

word/document.xml: the body. Paragraphs are <w:p>, runs of text with the same formatting are <w:r>, the text itself is in <w:t>, tables are <w:tbl>.
word/styles.xml: style definitions. A paragraph only says <w:pStyle w:val="Heading1"/>; this file tells you what that style is.
word/numbering.xml: list definitions. A list item only says "numbering 3, level 1"; this file tells you whether that level is a bullet or a number.
word/_rels/document.xml.rels: where hyperlinks point and which file in word/media/ each image is.

So a converter has four jobs: read the body in order, look up styles to find headings, look up numbering to tell bullets from numbers, and resolve relationships for links and images.

What a useful converter has to do

Find headings by style name, not style ID. IDs such as Heading1 can differ between Word languages and between editors. The name (heading 1) and the outline level are more reliable.
Merge runs. Word splits text into runs for reasons you cannot see (spell check, edits, language tags). Without merging, you get output like **annual** **plans**.
Handle "off" switches. <w:b w:val="0"/> means not bold. A naive check for the <w:b> tag gets this wrong.
Handle both kinds of lists: numbering set directly on the paragraph, and numbering that comes from a style such as "List Bullet 2".
Keep tables as tables, with pipes escaped and multi-paragraph cells joined.
Keep tracked insertions and skip tracked deletions, so you convert the text a reader sees.
Not pass through dangerous links. A Word file can contain javascript: link targets.

The script

Save this as docx2md.py. It was tested with Python 3.13.5 on the files described below.

#!/usr/bin/env python3
"""Convert a .docx file to Markdown with the Python standard library only."""
import posixpath
import re
import sys
import zipfile
import xml.etree.ElementTree as ET
from pathlib import Path

W = "{http://schemas.openxmlformats.org/wordprocessingml/2006/main}"
R = "{http://schemas.openxmlformats.org/officeDocument/2006/relationships}"
A = "{http://schemas.openxmlformats.org/drawingml/2006/main}"
WP = "{http://schemas.openxmlformats.org/drawingml/2006/wordprocessingDrawing}"
PR = "{http://schemas.openxmlformats.org/package/2006/relationships}"


def read_xml(z, name):
    try:
        return ET.fromstring(z.read(name))
    except KeyError:
        return None


def is_on(el):
    """<w:b/> means on; <w:b w:val="0"/> or "false" means off."""
    return el is not None and el.get(W + "val", "true") not in ("0", "false", "none")


class Docx:
    def __init__(self, path, media_dir=None):
        self.z = zipfile.ZipFile(path)
        self.media_dir = Path(media_dir) if media_dir else None
        self.heading_level = self._load_styles()
        self.num_formats = self._load_numbering()
        self.rels = self._load_rels()
        self.counters = {}

    def _load_styles(self):
        """Map styleId -> heading level, using the style name or outline level."""
        levels, self.style_num, self.style_depth = {}, {}, {}
        root = read_xml(self.z, "word/styles.xml")
        if root is None:
            return levels
        for st in root.iter(W + "style"):
            sid = st.get(W + "styleId")
            name_el = st.find(W + "name")
            name = (name_el.get(W + "val") if name_el is not None else "").lower()
            m = re.fullmatch(r"heading (\d)", name)
            outline = st.find(f"{W}pPr/{W}outlineLvl")
            deep = re.fullmatch(r"list (?:bullet|number) (\d)", name)  # "List Bullet 2"
            if deep:
                self.style_depth[sid] = int(deep.group(1)) - 1
            num = st.find(f"{W}pPr/{W}numPr")  # list styles such as "List Bullet"
            if num is not None and num.find(W + "numId") is not None:
                ilvl = num.find(W + "ilvl")
                self.style_num[sid] = (num.find(W + "numId").get(W + "val"),
                                       ilvl.get(W + "val") if ilvl is not None else "0")
            if name == "title":
                levels[sid] = 1
            elif m:
                levels[sid] = int(m.group(1))
            elif outline is not None:
                levels[sid] = int(outline.get(W + "val")) + 1
        return levels

    def _load_numbering(self):
        """Map (numId, ilvl) -> numFmt such as 'bullet' or 'decimal'."""
        fmts = {}
        root = read_xml(self.z, "word/numbering.xml")
        if root is None:
            return fmts
        abstract = {}
        for an in root.findall(W + "abstractNum"):
            lv = {}
            for lvl in an.findall(W + "lvl"):
                f = lvl.find(W + "numFmt")
                lv[lvl.get(W + "ilvl")] = f.get(W + "val") if f is not None else "bullet"
            abstract[an.get(W + "abstractNumId")] = lv
        for num in root.findall(W + "num"):
            ref = num.find(W + "abstractNumId").get(W + "val")
            for ilvl, fmt in abstract.get(ref, {}).items():
                fmts[(num.get(W + "numId"), ilvl)] = fmt
        return fmts

    def _load_rels(self):
        rels = {}
        root = read_xml(self.z, "word/_rels/document.xml.rels")
        if root is not None:
            for rel in root.findall(PR + "Relationship"):
                rels[rel.get("Id")] = rel.get("Target")
        return rels

    # ---- inline content -------------------------------------------------
    def runs(self, parent, link=None):
        """Yield (text, bold, italic, link) segments for a paragraph."""
        for child in parent:
            if child.tag == W + "r":
                rpr = child.find(W + "rPr")
                bold = rpr is not None and is_on(rpr.find(W + "b"))
                ital = rpr is not None and is_on(rpr.find(W + "i"))
                for el in child:
                    if el.tag == W + "t":
                        yield el.text or "", bold, ital, link
                    elif el.tag == W + "tab":
                        yield " ", bold, ital, link
                    elif el.tag in (W + "br", W + "cr"):
                        yield "\n", False, False, None
                    elif el.tag == W + "drawing":
                        yield self.image(el), False, False, None
            elif child.tag == W + "hyperlink":
                rid = child.get(R + "id")
                target = self.rels.get(rid) if rid else None
                if target is None and child.get(W + "anchor"):
                    target = "#" + child.get(W + "anchor")
                if target and not re.match(r"(https?:|mailto:|#)", target):
                    target = None  # drop javascript:, file: and other odd targets
                yield from self.runs(child, target)
            elif child.tag in (W + "ins", W + "smartTag", W + "sdt", W + "sdtContent"):
                yield from self.runs(child, link)  # tracked insertions, content controls

    def image(self, drawing):
        blip = drawing.find(f".//{A}blip")
        doc_pr = drawing.find(f".//{WP}docPr")
        alt = (doc_pr.get("descr") or doc_pr.get("name") or "") if doc_pr is not None else ""
        target = self.rels.get(blip.get(R + "embed")) if blip is not None else None
        if not target:
            return ""
        name = posixpath.basename(target)
        if self.media_dir:
            self.media_dir.mkdir(parents=True, exist_ok=True)
            part = posixpath.normpath(posixpath.join("word", target))
            (self.media_dir / name).write_bytes(self.z.read(part))
            return f"![{alt}]({self.media_dir.name}/{name})"  # relative to the .md file
        return f"![{alt}]({name})"

    def inline(self, p):
        # merge neighbouring segments with the same formatting
        merged = []
        for text, b, i, link in self.runs(p):
            if merged and merged[-1][1:] == [b, i, link] and not text.startswith("!["):
                merged[-1][0] += text
            else:
                merged.append([text, b, i, link])
        out = []
        for text, b, i, link in merged:
            core = text.strip()
            if not core:
                out.append(text)
                continue
            lead = text[: len(text) - len(text.lstrip())]
            trail = text[len(text.rstrip()):]
            mark = "**" * b + "*" * i
            core = f"{mark}{core}{mark[::-1]}"
            if link:
                core = f"[{core}]({link})"
            out.append(lead + core + trail)
        return "".join(out)

    # ---- blocks ---------------------------------------------------------
    def paragraph(self, p):
        self.last_num = None
        text = self.inline(p).strip()
        ppr = p.find(W + "pPr")
        style = ppr.find(W + "pStyle") if ppr is not None else None
        sid = style.get(W + "val") if style is not None else None
        num = ppr.find(W + "numPr") if ppr is not None else None
        if not text:
            return None
        if sid in self.heading_level:
            plain = text.replace("**", "").replace("\n", " ")  # headings are bold already
            return "#" * min(self.heading_level[sid], 6) + " " + plain
        num_id, ilvl = self.style_num.get(sid, (None, "0"))
        if num is not None and num.find(W + "numId") is not None:
            num_id = num.find(W + "numId").get(W + "val")
            ilvl_el = num.find(W + "ilvl")
            ilvl = ilvl_el.get(W + "val") if ilvl_el is not None else ilvl
        self.last_num = num_id
        if num_id and num_id != "0":
            fmt = self.num_formats.get((num_id, ilvl), "bullet")
            depth = int(ilvl) + self.style_depth.get(sid, 0)
            # restart deeper levels when a shallower item appears
            for key in [k for k in self.counters if k[0] == num_id and int(k[1]) > depth]:
                del self.counters[key]
            indent = "    " * depth
            if fmt == "bullet":
                return f"{indent}- {text}"
            n = self.counters[(num_id, ilvl)] = self.counters.get((num_id, ilvl), 0) + 1
            return f"{indent}{n}. {text}"
        if sid and sid.lower() in ("code", "sourcecode", "htmlpreformatted"):
            return f"```\n{''.join(t.text or '' for t in p.iter(W + 't'))}\n```"
        return text.replace("\n", "  \n")

    def table(self, tbl):
        rows = []
        for tr in tbl.findall(W + "tr"):
            cells = []
            for tc in tr.findall(W + "tc"):
                parts = [self.inline(p).strip() for p in tc.iter(W + "p")]
                cell = "<br>".join(x for x in parts if x).replace("\n", "<br>")
                cells.append(cell.replace("|", "\\|"))
            rows.append(cells)
        if not rows:
            return None
        width = max(len(r) for r in rows)
        rows = [r + [""] * (width - len(r)) for r in rows]
        lines = ["| " + " | ".join(rows[0]) + " |", "|" + " --- |" * width]
        lines += ["| " + " | ".join(r) + " |" for r in rows[1:]]
        return "\n".join(lines)

    def convert(self):
        body = read_xml(self.z, "word/document.xml").find(W + "body")
        blocks, prev_list, prev_num, prev_nested = [], False, None, False
        for el in body:
            if el.tag == W + "p":
                md = self.paragraph(el)
            elif el.tag == W + "tbl":
                md = self.table(el)
            elif el.tag == W + "sdt":  # content control, e.g. a table of contents
                inner = el.find(W + "sdtContent")
                paras = inner.findall(W + "p") if inner is not None else []
                md = "\n\n".join(filter(None, (self.paragraph(p) for p in paras)))
            else:
                continue
            if not md:
                continue
            if md.startswith("```\n") and blocks and blocks[-1].endswith("\n```"):
                blocks[-1] = blocks[-1][:-3] + md[4:]  # join consecutive code lines
                continue
            is_list = bool(re.match(r"\s*(-|\d+\.) ", md))
            same_list = is_list and prev_list and (md[0] == " " or prev_nested or self.last_num == prev_num)
            sep = "\n" if same_list else "\n\n"
            blocks.append((sep if blocks else "") + md)
            prev_list, prev_num = is_list, (self.last_num if is_list else None)
            prev_nested = is_list and md[0] == " "
        return "".join(blocks).strip() + "\n"


if __name__ == "__main__":
    if len(sys.argv) < 2:
        sys.exit("usage: docx2md.py input.docx [media_dir] > output.md")
    print(Docx(sys.argv[1], sys.argv[2] if len(sys.argv) > 2 else None).convert(), end="")

Run it:

python3 docx2md.py policy.docx > policy.md          # images referenced by file name
python3 docx2md.py policy.docx media > policy.md    # also extracts images into ./media

How the pieces fit

_load_styles maps each style ID to a heading level, using the style name (title, heading 1 to heading 6) or an outline level. It also records list styles such as "List Bullet" that carry their own numbering, and treats "List Bullet 2" or "List Number 3" as one or two levels deeper.
_load_numbering resolves each numId and level to a format. Word stores list formats in two hops (num points to abstractNum), which is why there are two loops.
runs walks a paragraph and yields (text, bold, italic, link) segments. It follows hyperlinks, tracked insertions (w:ins) and content controls. Deleted text lives in w:delText, which it never reads, so tracked deletions drop out.
inline merges neighbouring segments with identical formatting, then puts the ** and * markers inside any leading or trailing spaces, because ** bold** is not valid Markdown emphasis.
paragraph decides between heading, list item, code block and plain paragraph. Numbered items keep a counter per list and level, and deeper counters reset when a shallower item appears.
convert walks the body in order, so text and tables stay in their original sequence, and puts list items on consecutive lines so they render as one list.

Try it on a test document

We built a small test file with python-docx (used only to create the fixture; the converter does not need it). It has a Title, Heading 1 and Heading 2, a paragraph with a bold run split in two, an italic run, a bold external link and a run with bold explicitly turned off, a numbered list, a bullet list with a "List Bullet 2" item, a table with a pipe in a cell and a two-paragraph cell, a soft line break, and an image with alt text.

python3 docx2md.py sample.docx media > sample.md

Output:

# Refund policy

Last updated in the Word file. This intro has an empty paragraph after it.

# Who can ask for a refund

Customers on **annual plans** can ask within *30 days*. Read the [**full terms**](https://example.com/terms) first. Not bold

## Steps

1. Open Billing
2. Click Request refund
3. Pick a reason

What you need:

- Order ID
- Account email
    - Found in the receipt footer
- Reason

## Plans

| Plan | Window | Notes |
| --- | --- | --- |
| Monthly | 7 days | Pro \| Team |
| Annual | 30 days | Prorated<br>Second line in cell |

Line one  
Line two after a soft break

![Refund flow diagram](media/image1.png)

The split bold run came out as one **annual plans**, "Not bold" stayed plain, the nested bullet is indented under its parent, the pipe in "Pro | Team" stayed inside its cell, and the image was written to media/image1.png with its alt text.

We also checked three other inputs:

A LibreOffice-made file (HTML converted to DOCX with soffice --headless), which puts numbering on each paragraph instead of on styles. Nested numbered items came out as 1., 2. under 1., and a bullet list that followed the numbered list started a new list instead of joining it.
calibre's public demo DOCX, a Word document built to exercise many features. Headings, the multi-level numbered and bulleted lists, external and internal links, inline images and simple tables came through. Its "continued list" section, where Word resumes numbering at 3 after an interrupting paragraph, also came out as 3 and 4. Its word/document.xml is about 138 KB; the Markdown is about 10 KB.
Edge cases: a tracked change file (the deleted text was dropped and the inserted text kept), a document with a javascript: link (the link was removed and its text kept, while a mailto: link survived), and consecutive paragraphs in a "Code" style (merged into one fenced block).

Convert a folder of Word files

For a shared drive export, convert everything in one pass and keep the folder structure:

from pathlib import Path
from docx2md import Docx

src, dst = Path("docs"), Path("markdown")
for path in sorted(src.rglob("*.docx")):
    if path.name.startswith("~$"):  # Word lock files
        continue
    out = dst / path.relative_to(src).with_suffix(".md")
    out.parent.mkdir(parents=True, exist_ok=True)
    media = out.parent / (out.stem + "_media")
    out.write_text(Docx(path, media).convert(), encoding="utf-8")
    print(f"{path} -> {out}")

Each Markdown file gets its own <name>_media folder, and image links are written relative to the .md file, so the output can be moved as a unit. The ~$ check skips the lock files Word leaves next to open documents.

Known limits (and how to extend it)

Footnotes, endnotes and comments are dropped. They live in word/footnotes.xml, word/endnotes.xml and word/comments.xml. To keep footnotes, read w:footnoteReference in runs, emit [^n], and append the footnote text at the end.
Merged and nested table cells are flattened. Markdown tables cannot express gridSpan, vMerge or a table inside a cell. On calibre's demo, a nested table was folded into a single cell, and a calendar built from merged cells came out with empty columns. Keep such tables as HTML or rebuild them by hand.
Formatting from character styles is ignored. Only direct bold and italic are read. Text that is bold because of a style such as "Strong" comes out plain. Add those style IDs to the bold check if your documents rely on them.
Text boxes, shapes, headers, footers and drop caps are skipped or come out oddly. A drop cap shows up as a one-letter paragraph.
Markdown characters in the text are not escaped. A paragraph that starts with # or 1. will be read as a heading or list by a Markdown renderer. This rarely matters for prompts but matters for published docs.
Table of contents entries come through as links with page numbers attached (for example [Tables 3](#_Toc...)). Drop paragraphs whose style name starts with "toc" if you do not want them.
Old .doc files are a different binary format. Convert them to .docx first, for example with soffice --headless --convert-to docx file.doc.
Untrusted uploads: the script reads files only inside the ZIP and writes images using the base name only, but it does not limit file sizes. If users upload documents, check the uncompressed sizes in zipfile.ZipFile.infolist() before converting and run it with a timeout.

Why not mammoth or pandoc?

Both are mature and handle more of the edge cases above. mammoth is built to produce clean HTML from Word files; its own documentation says its Markdown output is deprecated and recommends converting to HTML and then to Markdown (checked 2 Nov 2026). If you can install dependencies, that route (mammoth, then an HTML-to-Markdown step) or pandoc -f docx -t gfm is a sound choice. The standard-library script makes sense when you cannot add dependencies, want to read and change every rule, or only need the structure that matters for LLM input.

Is Markdown the right target for chatbot knowledge?

Usually yes. Headings give chunkers natural split points, lists and tables keep their shape, and the file is small and easy to review before it goes live. Two habits help: split on ## headings and keep the source document name with each chunk so answers can cite it, and read the converted file once before you index it. A converter cannot tell you that the refund policy in a 2023 Word file is out of date.

When you do not want to run a script

Appie has free browser tools for Word files. Each run asks for a Cloudflare Turnstile check:

Convert DOCX to Markdown: upload a .docx; the page says heading styles become Markdown headings and Word tables become Markdown tables. For older .doc files it says to save as .docx first.
AI Chat with Your Word Document: upload a .docx and ask what it says. The page says answers have to come from the text you provided, and that headings and tables are included.
DOCX to FAQ Generator: writes a Markdown FAQ from a Word file, and the page says it is instructed to skip questions the file cannot answer.

If the end goal is a website chatbot, Appie's AI Chatbots app takes DOCX, PDF, CSV and text uploads as knowledge directly, so you may not need the conversion step at all. If you work from Claude Code or Cursor, Appie's MCP server can add text content to a chatbot (its add_chatbot_content tool takes text, URLs, sitemaps or Content Writer articles, not files), which is where the Markdown from this script fits.

Honest limits of Appie's Word tools

One file per run, with a fresh captcha each time. There is no batch mode on the tool pages, so a folder of 200 SOPs is a job for the script.
No options are described for footnotes, comments or merged cells, so test a representative file before relying on the output.

FAQ

Does this work with Google Docs exports?
It should, since Google Docs can download a doc as .docx (File > Download > Microsoft Word (.docx)), but we did not test a Google Docs export for this post. Convert one file and check the headings and lists before running a whole folder.

Why are my headings coming out as plain paragraphs?
The author probably made text big and bold instead of applying a Heading style. The script only sees styles. Apply Heading styles in Word, or add a rule that treats short, fully bold paragraphs as headings.

Can I keep underline, strikethrough or colours?
Markdown has no underline or colour. Strikethrough (~~text~~) is supported by GitHub-flavored Markdown; read w:strike in runs the same way as w:b.

How do I split the output for a RAG index?
Split on lines starting with ##, keep the # title with each chunk, and store the source file path as metadata.

Is it safe to run on files from customers?
Treat them as untrusted input: limit sizes, run with a timeout, and review the Markdown before it reaches a public chatbot. The script drops javascript: and file: links, but that is not a full sanitiser.

If you want a website chatbot trained on your documents, you can try Appie's AI Chatbots on the Free plan (500 credits, no credit card) at appie.run.

RG, Founder, Appie.run

Convert a Webpage or HTML to Clean Markdown With Python (Standard Library Only) for LLM Prompts and Chatbot Knowledge

appierun — Mon, 02 Nov 2026 15:53:29 +0000

Raw HTML is a poor thing to feed a model. A typical docs page is mostly navigation, scripts, styles and wrapper divs, and the actual text is a fraction of the bytes. Markdown keeps what matters (headings, lists, links, tables, code) and drops the rest, which makes it a good format for prompts, RAG chunks and chatbot knowledge files.

This post builds a small HTML-to-Markdown converter on Python's built-in html.parser, with no pip install. It reads a local file or a URL, skips page chrome, keeps tables and code blocks, and turns relative links into absolute ones.

Disclosure: Appie.run publishes this post. Appie has free HTML and webpage to Markdown converters, mentioned near the end. The script below does not depend on them.

Earlier posts in this series cover converting PDFs to Markdown for RAG and converting Notion to Markdown.

What a useful converter has to do

Drop the chrome. <head>, <script>, <style>, <nav>, <header>, <footer>, <aside>, forms and inline SVGs add noise and no meaning.
Catch navigation that is not a <nav>. Many sites build sidebars from plain <div>s marked role="navigation".
Keep structure: headings, nested lists, bold and italic, inline code and <pre> blocks.
Make links absolute. /docs/config means nothing once the text is pasted into a prompt.
Keep tables as tables, with pipes inside cells escaped.
Collapse whitespace without breaking code blocks.

The script

Save this as html2md.py. It was tested with Python 3.13.5 on the cases below.

#!/usr/bin/env python3
"""Convert an HTML file or URL to Markdown with the standard library only."""
import sys
import urllib.request
from html.parser import HTMLParser
from urllib.parse import urljoin

SKIP = {"head", "script", "style", "nav", "footer", "header", "aside", "form", "noscript", "svg"}
SKIP_ROLES = {"navigation", "banner", "contentinfo", "search"}
BLOCK = {"p", "div", "section", "article", "main", "br", "table", "ul", "ol"}


class MarkdownParser(HTMLParser):
    def __init__(self, base_url=""):
        super().__init__(convert_charrefs=True)
        self.base = base_url
        self.out = []
        self.skip_tag = None     # tag that opened the skipped block
        self.skip_count = 0      # nesting of that tag inside the block
        self.lists = []          # stack of "ul" / "ol" with counters
        self.href = None
        self.link_text = []
        self.in_pre = False
        self.row = None
        self.rows = []

    def emit(self, text):
        if self.href is not None:
            self.link_text.append(text)
        elif self.row is not None:
            self.row[-1] += text
        else:
            self.out.append(text)

    def handle_starttag(self, tag, attrs):
        a = dict(attrs)
        if self.skip_tag:
            if tag == self.skip_tag:
                self.skip_count += 1
            return
        if tag in SKIP or a.get("role") in SKIP_ROLES:
            if tag not in {"img", "br", "input", "meta", "link"}:
                self.skip_tag, self.skip_count = tag, 1
            return
        if tag in {"h1", "h2", "h3", "h4", "h5", "h6"}:
            self.out.append("\n\n" + "#" * int(tag[1]) + " ")
        elif tag in BLOCK:
            if tag == "br":
                self.out.append("  \n")
            elif tag in {"ul", "ol"} and self.lists:
                self.out.append("")  # nested list: no blank line
            else:
                self.out.append("\n\n")
            if tag in {"ul", "ol"}:
                self.lists.append([tag, 0])
        elif tag == "li":
            kind = self.lists[-1] if self.lists else ["ul", 0]
            kind[1] += 1
            indent = "  " * (len(self.lists) - 1)
            marker = f"{kind[1]}." if kind[0] == "ol" else "-"
            self.out.append(f"\n{indent}{marker} ")
        elif tag in {"strong", "b"}:
            self.emit("**")
        elif tag in {"em", "i"}:
            self.emit("*")
        elif tag == "code" and not self.in_pre:
            self.emit("`")
        elif tag == "pre":
            self.in_pre = True
            self.out.append("\n\n" + "`" * 3 + "\n")
        elif tag == "a" and a.get("href"):
            self.href = urljoin(self.base, a["href"])
            self.link_text = []
        elif tag == "img" and a.get("src"):
            self.emit(f"![{a.get('alt', '')}]({urljoin(self.base, a['src'])})")
        elif tag == "tr":
            self.row = []
        elif tag in {"td", "th"} and self.row is not None:
            self.row.append("")

    def handle_endtag(self, tag):
        if self.skip_tag:
            if tag == self.skip_tag:
                self.skip_count -= 1
                if self.skip_count == 0:
                    self.skip_tag = None
            return
        if tag in {"ul", "ol"} and self.lists:
            self.lists.pop()
            self.out.append("\n")
        elif tag in {"strong", "b"}:
            self.emit("**")
        elif tag in {"em", "i"}:
            self.emit("*")
        elif tag == "code" and not self.in_pre:
            self.emit("`")
        elif tag == "pre":
            self.in_pre = False
            self.out.append("\n" + "`" * 3 + "\n\n")
        elif tag == "a" and self.href is not None:
            text = "".join(self.link_text).strip() or self.href
            href, self.href = self.href, None
            self.emit(f"[{text}]({href})")
        elif tag == "tr" and self.row is not None:
            self.rows.append([c.strip().replace("|", "\\|") for c in self.row])
            self.row = None
        elif tag == "table" and self.rows:
            width = max(len(r) for r in self.rows)
            rows = [r + [""] * (width - len(r)) for r in self.rows]
            lines = ["| " + " | ".join(rows[0]) + " |",
                     "|" + " --- |" * width]
            lines += ["| " + " | ".join(r) + " |" for r in rows[1:]]
            self.out.append("\n\n" + "\n".join(lines) + "\n\n")
            self.rows = []
        elif tag in {"h1", "h2", "h3", "h4", "h5", "h6", "p"}:
            self.out.append("\n\n")

    def handle_data(self, data):
        if self.skip_tag:
            return
        if self.in_pre:
            self.out.append(data)
            return
        text = " ".join(data.split())
        if text:
            # keep a space between inline pieces
            if data[:1].isspace():
                text = " " + text
            if data[-1:].isspace():
                text += " "
            self.emit(text)

    def markdown(self):
        md = "".join(self.out)
        lines = [line.rstrip() if not line.endswith("  ") else line for line in md.splitlines()]
        cleaned, blank = [], 0
        for line in lines:
            if line.strip() == "":
                blank += 1
                if blank > 1:
                    continue
            else:
                blank = 0
                line = line if line.startswith(("  ", "|")) else line.lstrip()
            cleaned.append(line)
        return "\n".join(cleaned).strip() + "\n"


def load(source):
    if source.startswith(("http://", "https://")):
        req = urllib.request.Request(source, headers={"User-Agent": "html2md/1.0"})
        with urllib.request.urlopen(req, timeout=20) as resp:
            charset = resp.headers.get_content_charset() or "utf-8"
            return resp.read().decode(charset, errors="replace"), source
    with open(source, encoding="utf-8", errors="replace") as f:
        return f.read(), ""


def main():
    if len(sys.argv) < 2:
        sys.exit("usage: html2md.py <file.html | URL> [base_url]")
    html, base = load(sys.argv[1])
    if len(sys.argv) > 2:
        base = sys.argv[2]
    parser = MarkdownParser(base)
    parser.feed(html)
    parser.close()
    sys.stdout.write(parser.markdown())


if __name__ == "__main__":
    main()

Usage:

python3 html2md.py page.html https://example.com/docs/start   # local file, with a base URL for links
python3 html2md.py https://docs.python.org/3/library/html.parser.html > html-parser.md

How it works

HTMLParser is event based. It calls handle_starttag, handle_data and handle_endtag as it walks the document, and tolerates broken markup. convert_charrefs=True turns & and friends into plain characters.
Skipping by element. When a tag in SKIP opens, or any element has a role in SKIP_ROLES, the parser records that tag name and counts nested tags of the same name. Everything is ignored until the matching close tag brings the count back to zero. Counting matters: a sidebar <div> usually contains more <div>s, and closing on the first </div> would leak the rest of the sidebar into your output.
Lists use a stack, so nested lists get two spaces of indentation per level and ordered lists keep their own counters.
Links are buffered: the text inside <a> is collected and written as [text](absolute-url) on </a>. urljoin resolves relative paths against the base URL, which is the page URL when you pass a URL.
Tables are collected row by row and written at </table>. Short rows are padded so every row has the same number of cells, and the first row becomes the header.
<pre> switches off whitespace collapsing and wraps the block in a fence.
markdown() trims trailing spaces (keeping the two-space hard line break), removes repeated blank lines and returns the result.

Test it

A small page with every awkward part:

<html><head><title>Docs</title><style>body{}</style><script>var x=1;</script></head>
<body><nav><a href="/">Home</a> <a href="/pricing">Pricing</a></nav>
<div class="sidebar" role="navigation"><div><a href="/docs/a">Page A</a></div><a href="/docs/b">Page B</a></div>
<main>
<h1>Getting started</h1>
<p>Install the <strong>CLI</strong>, then read the <a href="/docs/config">config guide</a>.</p>
<h2>Plans</h2>
<table><tr><th>Plan</th><th>Seats</th></tr><tr><td>Free</td><td>1</td></tr><tr><td>Team | Pro</td><td>10</td></tr></table>
<ul><li>Fast setup</li><li>Works with <em>any</em> site<ol><li>Step one</li><li>Step two</li></ol></li></ul>
<pre><code>pip install example
example init</code></pre>
<img src="/img/flow.png" alt="Flow diagram">
</main>
<footer>Copyright 2026</footer></body></html>

python3 html2md.py sample.html https://example.com/docs/start

Output:

# Getting started

Install the **CLI**, then read the [config guide](https://example.com/docs/config).

## Plans

| Plan | Seats |
| --- | --- |
| Free | 1 |
| Team \| Pro | 10 |

- Fast setup
- Works with *any* site
  1. Step one
  2. Step two

```
pip install example
example init
```

![Flow diagram](https://example.com/img/flow.png)

The title, styles, script, both navigation blocks and the footer are gone, the nested sidebar <div> did not leak, the pipe in "Team | Pro" stayed inside its cell and every link is absolute.

On a real page, the Python docs page for html.parser was about 55 KB of HTML when we fetched it and came out at about 14.5 KB of Markdown. Your ratio will depend on the site.

Known limits (and how to extend it)

JavaScript-rendered pages come back nearly empty, because urllib only sees the initial HTML. Render them with a headless browser first and pass the saved HTML to the script.
Header anchor links such as the Â¶ symbols on Sphinx docs survive as links. Skip <a> tags with class="headerlink" if they bother you.
Inline spacing around some tags can be off by one space (for example **Source code:**[link]). It rarely matters to a model, but check output that people will read.
Complex tables with colspan, rowspan or nested tables are flattened. Markdown tables cannot represent them, so consider keeping those as HTML.
Definition lists, footnotes and <details> are treated as plain text.
Pages without semantic markup may still include menus. If a site puts its content in <main> or <article>, an easy extension is to only emit text while inside those elements.
Respect the site. Fetch pages you are allowed to use, keep request rates low, and check the site's terms before converting content you do not own.

Is Markdown the right format?

For prompts and chatbot knowledge, usually yes: headings give chunkers natural split points, links stay readable and tables survive. Plain text is smaller but loses structure; raw HTML keeps structure but spends most of its size on markup. If you are building a RAG index, split the Markdown on ## headings and keep the page URL with each chunk so answers can cite it.

When you do not want to run a script

Appie has two free browser tools for one-off conversions:

Convert HTML to Markdown: upload an HTML file or paste markup. Headings, lists, links, images and tables are converted, and the page says input is not saved to an account.
Convert a Webpage to Markdown: enter a public URL. Links come out absolute and navigation and scripts are left out. For private pages, the page suggests pasting the HTML into the HTML converter instead.

Each run asks for a Cloudflare Turnstile check. For repeatable jobs, many pages or anything behind a login, the script is the better fit.

If the Markdown is going into a chatbot, Appie's AI Chatbots app can also take URLs and sitemaps as knowledge directly, and you can add content from Claude Code or Cursor through Appie's MCP server, as shown in this MCP walkthrough.

Honest limits of Appie's converters

One page or file per run, with a fresh captcha each time. There is no batch mode on the tool pages.
Public URLs only for the webpage tool.
No options for what to skip are described on the pages; the script lets you edit SKIP and SKIP_ROLES yourself.

FAQ

Why not BeautifulSoup plus markdownify?
They are good libraries and handle more edge cases. The standard-library version suits small containers, CI jobs and serverless functions where adding dependencies is a hassle, and it is short enough to read and change.

How do I convert a whole site?
Get the URL list from the sitemap, loop over it with a delay between requests, and write one .md file per URL. Our earlier sitemap posts on this account show how to pull URLs from a sitemap.

Does it handle non-UTF-8 pages?
For URLs it uses the charset from the Content-Type header and falls back to UTF-8, replacing undecodable bytes.

Can I keep the page title?
Read <title> separately (it is skipped with <head>) and write it as a # heading at the top if the page has no <h1>.

If you want a website chatbot trained on your pages, you can try Appie's AI Chatbots on the Free plan (500 credits, no credit card) at appie.run.

RG, Founder, Appie.run

Convert CSV and JSON to Markdown Tables for LLM Prompts and Chatbot Knowledge With Python (Escaping, Ragged Rows, Nested Fields)

appierun — Sun, 01 Nov 2026 16:01:50 +0000

Pasting a spreadsheet export into a prompt, a README or a chatbot knowledge file usually goes one of two ways: raw CSV that is hard for a person to review, or a hand-made Markdown table that breaks the moment a cell contains a pipe character. This post walks through a small, dependency-free Python script that turns CSV files and JSON arrays into Markdown tables and handles the cases that break naive converters.

Disclosure: Appie.run publishes this post. Appie has free CSV and JSON to Markdown converters, mentioned near the end. The script below has no dependency on them, and it uses only the Python standard library.

Earlier posts in this series cover converting PDFs to Markdown for RAG and converting Notion to Markdown.

What breaks naive converters

The one-liner "| " + " | ".join(row) + " |" works until your data meets the real world:

Pipes inside cells. A value like has a | pipe adds a column and shifts everything after it.
Newlines inside quoted CSV cells. A multi-line note ends the table row early.
Ragged rows. Exports often have trailing rows with fewer fields, or completely empty rows.
Different delimiters. European spreadsheet exports often use semicolons.
A byte order mark. Excel's "CSV UTF-8" export starts with a BOM, which ends up glued to your first header.
JSON records with different keys. The second object has a field the first one lacks.
Nested values. A list of tags or an address object inside a record.

The script

Save this as to_md.py. It was tested with Python 3.13 on the cases below.

import csv
import json
import sys
from pathlib import Path


def escape_cell(value):
    """Make one value safe for a Markdown table cell."""
    if value is None:
        return ""
    if isinstance(value, (dict, list)):
        value = json.dumps(value, ensure_ascii=False)
    text = str(value)
    text = text.replace("\\", "\\\\").replace("|", "\\|")
    return " ".join(text.split())  # newlines and tabs become single spaces


def rows_to_markdown(header, rows):
    lines = [
        "| " + " | ".join(escape_cell(h) for h in header) + " |",
        "| " + " | ".join("---" for _ in header) + " |",
    ]
    for row in rows:
        cells = list(row) + [""] * (len(header) - len(row))  # pad short rows
        lines.append("| " + " | ".join(escape_cell(c) for c in cells[: len(header)]) + " |")
    return "\n".join(lines)


def csv_to_markdown(path, max_rows=None):
    with open(path, newline="", encoding="utf-8-sig") as f:
        sample = f.read(4096)
        f.seek(0)
        try:
            dialect = csv.Sniffer().sniff(sample, delimiters=",;\t")
        except csv.Error:
            dialect = csv.excel
        reader = csv.reader(f, dialect)
        header = next(reader)
        rows = []
        for i, row in enumerate(reader):
            if max_rows is not None and i >= max_rows:
                break
            if any(cell.strip() for cell in row):
                rows.append(row)
    return rows_to_markdown(header, rows)


def json_to_markdown(path, max_rows=None):
    data = json.loads(Path(path).read_text(encoding="utf-8"))
    if isinstance(data, dict):
        # a dict of lists: render each list under its own heading
        parts = []
        for key, value in data.items():
            parts.append(f"## {key}\n")
            if isinstance(value, list) and value and all(isinstance(v, dict) for v in value):
                parts.append(records_to_markdown(value, max_rows))
            else:
                parts.append(escape_cell(value))
            parts.append("")
        return "\n".join(parts).strip()
    return records_to_markdown(data, max_rows)


def records_to_markdown(records, max_rows=None):
    if max_rows is not None:
        records = records[:max_rows]
    header = []
    for rec in records:  # union of keys, first-seen order
        for key in rec:
            if key not in header:
                header.append(key)
    rows = [[rec.get(k) for k in header] for rec in records]
    return rows_to_markdown(header, rows)


if __name__ == "__main__":
    src = sys.argv[1]
    limit = int(sys.argv[2]) if len(sys.argv) > 2 else None
    if src.lower().endswith(".json"):
        print(json_to_markdown(src, limit))
    else:
        print(csv_to_markdown(src, limit))

Run it with a file and an optional row limit:

python3 to_md.py plans.csv
python3 to_md.py leads.json 25

How each problem is handled

Pipes and backslashes are escaped in escape_cell, so a | b renders as one cell. Backslashes are escaped first so an existing \ does not swallow the escape.
Newlines and tabs collapse to single spaces with " ".join(text.split()). Markdown table cells cannot hold line breaks without HTML, and a prompt rarely needs them.
Ragged rows are padded with empty cells up to the header length and trimmed if they are longer. Rows where every cell is blank are skipped.
Delimiters are detected by csv.Sniffer from the first 4 KB, limited to comma, semicolon and tab, with a fallback to standard CSV if detection fails.
The BOM is removed by opening the file with encoding="utf-8-sig".
Different keys in JSON records are merged into one header in first-seen order, and missing values become empty cells.
Nested values (lists and objects) are written as compact JSON inside the cell. A top-level JSON object of lists becomes one ## heading per key, with a table under each list of records.

Test it on the awkward cases

printf 'plan,price_usd,notes\nStarter,19,"2,000 credits"\nStandard,49,"has a | pipe\nand a newline"\n,,\nFree,0\n' > plans.csv
python3 to_md.py plans.csv

Output:

| plan | price_usd | notes |
| --- | --- | --- |
| Starter | 19 | 2,000 credits |
| Standard | 49 | has a \| pipe and a newline |
| Free | 0 |  |

The quoted comma stayed in its cell, the pipe was escaped, the newline became a space, the blank row was dropped and the short Free row was padded.

A semicolon file:

printf 'name;score\nA;1\nB;2\n' > semi.csv
python3 to_md.py semi.csv

| name | score |
| --- | --- |
| A | 1 |
| B | 2 |

JSON records with different keys and a nested list:

echo '[{"name":"Ana","company":"Acme","tags":["saas","eu"]},{"name":"Ben","title":"PM"}]' > leads.json
python3 to_md.py leads.json

| name | company | tags | title |
| --- | --- | --- | --- |
| Ana | Acme | ["saas", "eu"] |  |
| Ben |  |  | PM |

A JSON object of lists:

echo '{"plans":[{"name":"Starter","credits":2000},{"name":"Standard","credits":5000}],"currency":"USD"}' > nested.json
python3 to_md.py nested.json

## plans

| name | credits |
| --- | --- |
| Starter | 2000 |
| Standard | 5000 |

## currency

USD

Is Markdown the right format for a prompt?

It depends on what you are optimising. We measured one case on our own machine: 50 records with four short fields came to 3,280 characters as JSON, 1,202 as CSV and 1,686 as a Markdown table. You can reproduce it with this snippet next to to_md.py:

import csv, io, json
from to_md import records_to_markdown

rows = [{"id": i, "name": f"user{i}", "plan": "Starter", "credits": 2000} for i in range(50)]
buf = io.StringIO()
w = csv.DictWriter(buf, fieldnames=rows[0].keys())
w.writeheader()
w.writerows(rows)
print(len(json.dumps(rows)), len(buf.getvalue()), len(records_to_markdown(rows)))

So for this data, CSV was the most compact and JSON the largest. Characters are not tokens, and your numbers will differ with your data and tokenizer, so measure your own case. Practical rules we use:

Use Markdown when the table sits inside prose (a README, a doc page, a chatbot knowledge file a person will review), or when you want the model and the reader to see the same thing.
Use CSV when the table is large and you only care about size.
Keep JSON when values are deeply nested or when the model must return the same structure.
Limit rows. The optional row limit exists because the right answer for a 20,000-row export is usually a sample plus a summary, not the whole file.

When you do not want to run a script

Appie has two free browser tools for one-off conversions:

Convert CSV to Markdown: paste a CSV export or upload the file, and the first row becomes the table header.
Convert JSON to Markdown: paste JSON or upload a .json file. Arrays of objects become tables and nested objects become headings.

Both ask for a Cloudflare Turnstile check before each run, and the pages say the file or paste is converted for that request and not saved to an Appie account. For anything repeatable, or for data you would rather not paste into a website, use the script.

If the Markdown is headed for a chatbot, Appie's AI Chatbots app can also take CSV files, text and URLs as knowledge directly, and you can add content from Claude Code or Cursor through the add_chatbot_content tool on Appie's MCP server.

Honest limits of Appie's converters

One file per run, by hand. There is no batch mode on the tool pages, and each run needs a fresh captcha check.
No control over escaping or row limits is described on the pages. If you need a specific output format, the script gives you that control.

FAQ

Why not use pandas to_markdown()?
It works well if pandas is already installed; it needs the extra tabulate package. The script above has no dependencies, which matters in small containers, CI jobs and serverless functions.

How do I keep line breaks inside a cell?
Replace the newline with <br> instead of a space in escape_cell. Many Markdown renderers, including GitHub's, show it as a line break inside the cell. LLMs read it fine, but it adds characters.

What about very wide tables?
Split them. A table with 30 columns is hard for people and models to read. Pick the columns the prompt actually needs.

Does the script handle Excel files?
No. Export the sheet as CSV first, or read it with openpyxl and pass the rows to rows_to_markdown.

If you want a website chatbot trained on files like these, you can try Appie's AI Chatbots on the Free plan (500 credits, no credit card) at appie.run.

RG, Founder, Appie.run

How to Compare Two XML Sitemaps Before and After a Site Migration With Python (Find Removed URLs and Missing Redirects)

appierun — Sat, 31 Oct 2026 15:57:04 +0000

A site migration (new domain, new CMS, new URL structure) is where search traffic quietly goes missing. Pages that existed before the move disappear from the new site, and nobody notices until rankings drop. Your old and new XML sitemaps are the quickest record of what changed. This post walks through a small Python script that diffs them, follows sitemap indexes, handles a domain change, and checks every removed URL against your redirect map.

Disclosure: Appie.run publishes this post. Appie has free sitemap tools, mentioned near the end. The script below has no dependency on them, and it uses only the Python standard library.

Earlier posts in this series cover generating a sitemap for a static site and validating a sitemap. Run the validator on both files first if you are not sure they are well formed.

What the script answers

Which URLs are in the old sitemap but not the new one (removed)?
Which are new (added)?
Which survived (kept)?
For every removed URL, is there a redirect in your map, and does the target actually exist in the new sitemap?

Question 4 is the one that saves you. A redirect pointing at a page that was never built is as bad as no redirect.

Before you run it

Save the old sitemap before launch day. Once the new site is live, the old sitemap.xml is usually gone. Download it (and every child sitemap) while you still can.
Export your redirect map as a two-column CSV: old_url,new_url. Most migration plans already have one in a spreadsheet.

The script

Save this as sitemap_diff.py:

#!/usr/bin/env python3
"""Compare two XML sitemaps (before and after a site migration)."""
import argparse
import csv
import gzip
import sys
import urllib.request
import xml.etree.ElementTree as ET
from urllib.parse import urlsplit, urlunsplit

NS = "{http://www.sitemaps.org/schemas/sitemap/0.9}"
UA = "sitemap-diff/1.0 (+https://example.com/contact)"


def read_source(src):
    """Return raw bytes from a URL or a local file path."""
    if src.startswith(("http://", "https://")):
        req = urllib.request.Request(src, headers={"User-Agent": UA})
        with urllib.request.urlopen(req, timeout=30) as resp:
            data = resp.read()
    else:
        with open(src, "rb") as fh:
            data = fh.read()
    if data[:2] == b"\x1f\x8b":  # gzip magic number
        data = gzip.decompress(data)
    return data


def collect_urls(src, seen=None, depth=0):
    """Return the set of page URLs in a sitemap, following sitemap indexes."""
    seen = seen if seen is not None else set()
    if src in seen or depth > 3:
        return set()
    seen.add(src)
    root = ET.fromstring(read_source(src))
    tag = root.tag.replace(NS, "")
    locs = [el.text.strip() for el in root.iter(NS + "loc") if el.text]
    if tag == "sitemapindex":
        urls = set()
        for child in locs:
            urls |= collect_urls(child, seen, depth + 1)
        return urls
    if tag == "urlset":
        return set(locs)
    raise ValueError(f"{src}: root element is <{tag}>, expected urlset or sitemapindex")


def normalise(url, old_host=None, new_host=None, ignore_slash=True):
    """Make URLs comparable: lowercase host, optional host swap, trailing slash."""
    parts = urlsplit(url)
    host = parts.netloc.lower()
    if old_host and host == old_host.lower():
        host = new_host.lower()
    path = parts.path or "/"
    if ignore_slash and len(path) > 1:
        path = path.rstrip("/")
    return urlunsplit((parts.scheme.lower(), host, path, parts.query, ""))


def load_redirects(path):
    """Read a two-column CSV of old_url,new_url."""
    with open(path, newline="", encoding="utf-8") as fh:
        return {row[0].strip(): row[1].strip() for row in csv.reader(fh) if len(row) >= 2}


def main():
    ap = argparse.ArgumentParser(description=__doc__)
    ap.add_argument("before", help="old sitemap: URL or file")
    ap.add_argument("after", help="new sitemap: URL or file")
    ap.add_argument("--old-host", help="host used before the move, e.g. www.old.com")
    ap.add_argument("--new-host", help="host used after the move, e.g. www.new.com")
    ap.add_argument("--redirects", help="CSV of old_url,new_url to check removed URLs against")
    ap.add_argument("--out", default="sitemap-diff.csv", help="where to write the report")
    args = ap.parse_args()

    if bool(args.old_host) != bool(args.new_host):
        ap.error("--old-host and --new-host go together")

    before_raw = collect_urls(args.before)
    after_raw = collect_urls(args.after)

    before = {normalise(u, args.old_host, args.new_host): u for u in before_raw}
    after = {normalise(u): u for u in after_raw}

    kept = before.keys() & after.keys()
    removed = before.keys() - after.keys()
    added = after.keys() - before.keys()

    redirects = load_redirects(args.redirects) if args.redirects else {}
    after_norm = set(after.keys())

    rows = []
    for key in sorted(removed):
        old = before[key]
        target = redirects.get(old, "")
        if not redirects:
            note = ""
        elif not target:
            note = "no redirect mapped"
        elif normalise(target) not in after_norm:
            note = "redirect target not in new sitemap"
        else:
            note = "ok"
        rows.append(("removed", old, target, note))
    rows += [("added", after[k], "", "") for k in sorted(added)]
    rows += [("kept", after[k], "", "") for k in sorted(kept)]

    with open(args.out, "w", newline="", encoding="utf-8") as fh:
        writer = csv.writer(fh)
        writer.writerow(["status", "url", "redirect_to", "note"])
        writer.writerows(rows)

    print(f"before: {len(before)}  after: {len(after)}")
    print(f"kept: {len(kept)}  added: {len(added)}  removed: {len(removed)}")
    if redirects:
        problems = sum(1 for r in rows if r[0] == "removed" and r[3] != "ok")
        print(f"removed URLs without a valid redirect: {problems}")
    print(f"report: {args.out}")
    return 1 if redirects and problems else 0


if __name__ == "__main__":
    sys.exit(main())

What each part does

read_source accepts a URL or a local file and decompresses .gz sitemaps by checking the gzip magic bytes rather than trusting the file name.
collect_urls reads the root element. A <urlset> returns its <loc> values. A <sitemapindex> makes the function fetch each child sitemap and merge the results. The seen set and depth limit stop a loop if an index lists itself.
normalise makes URLs comparable. It lowercases the scheme and host, drops fragments, treats /page and /page/ as the same, and, if you pass --old-host and --new-host, rewrites the old host so https://www.old-shop.com/pricing/ matches https://www.new-shop.com/pricing.
The diff is three set operations on the normalised keys. The original URLs are kept for the report, so you see exactly what each sitemap listed.
Redirect checks run only when you pass --redirects. Each removed URL is marked ok, no redirect mapped, or redirect target not in new sitemap.
Exit code is 1 when any removed URL lacks a valid redirect, so you can fail a CI job on it.

Run it

A small example with an old site at www.old-shop.com and a new one at www.new-shop.com, where the new sitemap is an index pointing to a child file:

python3 sitemap_diff.py old-sitemap.xml https://www.new-shop.com/sitemap_index.xml \
  --old-host www.old-shop.com --new-host www.new-shop.com \
  --redirects redirects.csv

Output:

before: 4  after: 5
kept: 2  added: 3  removed: 2
removed URLs without a valid redirect: 1
report: sitemap-diff.csv

And sitemap-diff.csv:

status,url,redirect_to,note
removed,https://www.old-shop.com/about-us,https://www.new-shop.com/about,redirect target not in new sitemap
removed,https://www.old-shop.com/blog/launch-post,https://www.new-shop.com/blog/launch,ok
added,https://www.new-shop.com/blog/launch,,
added,https://www.new-shop.com/company,,
added,https://www.new-shop.com/integrations,,
kept,https://www.new-shop.com/,,
kept,https://www.new-shop.com/pricing,,

The first row is the kind of bug this catches: the redirect map sends /about-us to /about, but the new site calls that page /company. Fix the map before launch, not after.

Notice that /pricing/ and /pricing count as kept because of the trailing slash rule. If your new site really treats them as different pages, run without that rule (set ignore_slash=False in normalise).

Optional: check what the live URLs actually do

The map says what should happen. After launch, check what does happen. This second script reads the report and requests each removed URL without following redirects, so you see the first hop:

#!/usr/bin/env python3
"""For each removed URL in sitemap-diff.csv, request it and report where it lands."""
import csv
import sys
import urllib.error
import urllib.request


class NoFollow(urllib.request.HTTPRedirectHandler):
    def redirect_request(self, *args, **kwargs):
        return None  # stop at the first hop so we can see the status code


opener = urllib.request.build_opener(NoFollow)


def first_hop(url, method="HEAD"):
    req = urllib.request.Request(url, method=method, headers={"User-Agent": "sitemap-diff/1.0"})
    try:
        with opener.open(req, timeout=20) as resp:
            return resp.status, ""
    except urllib.error.HTTPError as err:  # 3xx lands here because we refuse to follow
        if method == "HEAD" and err.code in (405, 500, 501):
            return first_hop(url, "GET")  # some servers reject HEAD
        return err.code, err.headers.get("Location", "")
    except urllib.error.URLError as err:
        return 0, str(err.reason)


report = sys.argv[1] if len(sys.argv) > 1 else "sitemap-diff.csv"
with open(report, newline="", encoding="utf-8") as fh:
    for row in csv.DictReader(fh):
        if row["status"] != "removed":
            continue
        code, location = first_hop(row["url"])
        flag = "" if code in (301, 308) else "  <-- check"
        print(f"{code} {row['url']} -> {location}{flag}")

Example output (illustrative URLs):

301 https://www.old-shop.com/blog/launch-post -> https://www.new-shop.com/blog/launch
404 https://www.old-shop.com/about-us ->   <-- check

Anything that is not a 301 or 308 gets flagged. A 302 is a temporary redirect, which is usually not what you want for a permanent move, and a 200 means the old page still serves content. Some servers reject HEAD requests, so the script retries those with GET. Keep the request rate polite on large sites; for tens of thousands of URLs, add a short time.sleep between requests or run it on a sample.

Wire it into a pre-launch check

# fails the job if any removed URL lacks a valid redirect
python3 sitemap_diff.py old-sitemap.xml staging-sitemap.xml \
  --old-host www.old-shop.com --new-host staging.new-shop.com \
  --redirects redirects.csv --out diff.csv

Running it against the staging sitemap on every deploy means a page dropped by a template change shows up the same day.

Known limits

A sitemap is not the whole site. Pages that were never in the old sitemap won't be checked. Combine this with a crawl or your analytics' top landing pages.
Query strings are compared as-is. If the old site used ?id=12 and the new one uses clean paths, rely on the redirect map rather than the kept count.
Relative <loc> values inside an index are read as local paths. The sitemap protocol expects absolute URLs, so this only matters for hand-made test files.
Memory. Everything is held in sets. That is fine for a few hundred thousand URLs; beyond that, stream to disk.

When a script is more than you need

For a one-off check, Appie's free tools do the core steps in the browser:

Sitemap URLs Comparison Tool takes two sitemaps, by URL or pasted XML, and reports how many URLs stayed, which were added and which were removed. The first sitemap is treated as the before state.
Sitemap URL Extractor lists one URL per line from a sitemap. Given a sitemap index, it lists the child sitemaps, so you run it again on each child.
Sitemap Finder and Checker finds sitemaps through robots.txt and the usual paths, which helps when you don't know where the old site kept them.

Each asks for a captcha before running. They don't check redirect maps or host changes, which is where the script above earns its keep.

RG, Founder, Appie.run

Create and Track Short Links From Claude Code or Cursor With an MCP Server (Appie, Bitly, Dub, Short.io)

appierun — Fri, 30 Oct 2026 16:02:40 +0000

If you ship release notes, changelogs or launch posts from your editor, the short link is usually the last manual step: open a shortener in a browser tab, paste the URL, copy the result, come back. With an MCP server, your coding agent can create the link and later tell you how many clicks it got, in the same session where you wrote the post.

This walkthrough shows the setup for four link shorteners that publish an MCP server: Appie.run, Bitly, Dub and Short.io. It covers the config for Claude Code and Cursor, the prompts that work, and the limits of each.

Plan details and prices are as of 29 Oct 2026, checked on each vendor's own docs and pricing pages that day.

Disclosure: Appie.run publishes this post, and one of the four servers is ours. We compared shorteners more broadly (not just MCP) in Bitly alternatives in 2026.

The four servers at a glance

Endpoint: Appie.run: https://appie.run/mcp; Bitly: https://api-ssl.bitly.com/v4/mcp; Dub (Links): https://mcp.dub.sh/mcp/dub-links; Short.io: https://ai-assistant.short.io/mcp (via npx mcp-remote)
Auth: Appie.run: OAuth 2.1 (ChatGPT) or esk_ project API key; Bitly: OAuth 2.1 or Bitly API token; Dub (Links): Dub API key in Mcp-Dub-Token header; Short.io: Short.io API key
Custom domain: Appie.run: No, links use s.appie.run; Bitly: Yes, on Growth and up; Dub (Links): Yes; Short.io: Yes
Edit or delete existing links via MCP: Appie.run: Not in the documented tools; Bitly: See Bitly's MCP tools reference; Dub (Links): Dub says the server lets you "programmatically manage your links"; check its tool list; Short.io: No, per Short.io's docs
Entry price: Appie.run: Free plan, 500 credits, no card; links cost 1 credit, clicks 0.001 credit each; Bitly: Bitly support says MCP is "available with some paid plans"; Core is $10/month billed annually; Dub (Links): Free plan (25 new links/month); Pro $27/month; Short.io: Free plan (1,000 branded links total); Hobby $5/month

Prerequisites

Claude Code or Cursor with MCP support.
An account with the shortener and an API key or OAuth access.
Keep keys in environment variables. Do not commit them to a shared repo.

Option 1: Appie.run

Appie's MCP server exposes two link tools, documented at appie.run/docs/mcp:

create_short_link: Arguments: long_url, optional code (custom slug); Cost: 1 credit per link; each click on the link costs 0.001 credit
link_stats: Arguments: id or code; Cost: 0 credits to read

Reads are free. Writes spend the project's credits first, then the owner's.

Claude Code:

export APPIE_API_KEY=esk_your_project_key
claude mcp add --transport http appie https://appie.run/mcp \
  --header "Authorization: Bearer $APPIE_API_KEY"

Cursor (~/.cursor/mcp.json or .cursor/mcp.json in the project):

{
  "mcpServers": {
    "appie": {
      "url": "https://appie.run/mcp",
      "headers": {
        "Authorization": "Bearer YOUR_API_KEY"
      }
    }
  }
}

Appie's docs also accept the key in an X-Api-Key header if your client prefers that.

Prompts that work:

Create a short link for https://example.com/blog/v2-release?utm_source=devto&utm_medium=post
with the code v2-devto.

Get link_stats for the code v2-devto and tell me the click count.

What the links give you: links live on the s.appie.run domain, and the URL Shortener page lists clicks and unique visitors over the last 30 days, lifetime totals, and device and location breakdowns through the stats API.

Limits: no custom domain, and the MCP tools cover create and read only. If you need branded domains or bulk edits, one of the other three is a better fit. The same MCP server also covers Appie's other apps (articles, chatbots, Reddit and LinkedIn research), which matters only if you want one server for several marketing jobs. We wrote about that in One MCP server for your marketing agent.

Option 2: Bitly

Bitly hosts its server at https://api-ssl.bitly.com/v4/mcp. Its developer docs list setup guides for ChatGPT, Claude (web, desktop and Claude Code), Cursor, VS Code and Windsurf, with OAuth 2.1 recommended and an API token as the fallback.

Claude Code or Cursor (from Bitly's docs; Claude Code reads ~/.claude/mcp.json, Cursor reads ~/.cursor/mcp.json):

{
  "mcpServers": {
    "Bitly": {
      "type": "http",
      "url": "https://api-ssl.bitly.com/v4/mcp",
      "headers": {
        "Authorization": "Bearer YOUR_BITLY_AUTH_TOKEN"
      }
    }
  }
}

Drop the headers block to use OAuth instead.

Pricing note: Bitly's support page says the MCP server is "available with some paid plans", so confirm yours before building a workflow around it. On Bitly's pricing page, Core is $10/month (billed annually, 100 links/month, 30 days of click data), Growth is $29/month (500 links/month, branded links, 4 months of data) and Premium is $199/month (3,000 links/month, city and device data). The Free plan allows 5 links a month.

Option 3: Dub

Dub runs two MCP servers: one for its partner program product and one for links. For short links you want https://mcp.dub.sh/mcp/dub-links. Dub's docs pass the API key in an Mcp-Dub-Token header.

Claude Code:

claude mcp add dub-links --transport http https://mcp.dub.sh/mcp/dub-links \
  --header "Mcp-Dub-Token: $DUB_API_KEY"

Cursor:

{
  "mcpServers": {
    "dub-links": {
      "url": "https://mcp.dub.sh/mcp/dub-links",
      "headers": {
        "Mcp-Dub-Token": "dub_xxxxxx"
      }
    }
  }
}

Pricing: Dub's Links pricing page lists a Free plan (25 new links/month, 1,000 tracked events/month, 30-day retention, API access, 3 custom domains), Pro at $27/month (12K new links/year, 1-year retention, 10 custom domains) and Business at $81/month. Dub's launch post says its MCP server is generally available on all Dub plans; confirm that the Links server is included on yours before you rely on it.

Option 4: Short.io

Short.io's docs connect through mcp-remote:

{
  "mcpServers": {
    "shortio": {
      "command": "npx",
      "args": [
        "mcp-remote",
        "https://ai-assistant.short.io/mcp",
        "--header",
        "Authorization: ${SHORTIO_API_KEY}"
      ],
      "env": {
        "SHORTIO_API_KEY": "sk_secret_key"
      }
    }
  }
}

Short.io's docs show creating links on your own domain with a custom slug, searching links, and pulling link and domain statistics (countries, browsers, operating systems, UTM campaigns). One limit they state plainly: with the current MCP implementation you cannot edit or delete an existing link; you remove it in the dashboard and create a new one.

Pricing: the Free plan includes 1,000 branded links total, 5 custom domains and 50,000 tracked clicks a month. Its comparison table lists Hobby at $5/month, Pro at $18/month and Team at $48/month.

A small release workflow

Here is a pattern that works with any of the four:

Write the release post in your repo.
Ask the agent: "Create a short link for each external URL in posts/v2-release.md, use slugs that start with v2-, and replace the long URLs in the file."
Review the diff before committing. The agent can pick the wrong URL, so check every replacement.
A week later: "Get stats for every link with a v2- slug and summarize clicks by link."

Step 3 matters. An agent with write access to your shortener can create links you did not intend, and with Bitly, Dub or Short.io those links can sit on your branded domain. Review before you publish.

Which one to pick

You already pay for Bitly, Dub or Short.io: use that vendor's server. Your existing domains and analytics come with it.
You need branded domains on a budget: Short.io's or Dub's free plans.
You want link tracking plus other marketing tools in one MCP server and do not need a custom domain: Appie.

FAQ

Do I need a paid plan to use these MCP servers?
Appie, Dub and Short.io offer free plans with API access. Bitly says its MCP server is available with some paid plans.

Can the agent post the link anywhere?
No. These servers create and read links. Publishing the post is still your step.

Are clicks from bots counted?
Each vendor filters differently, and none of the docs we read gives a full method. Treat small click counts with caution.

RG, Founder, Appie.run

Build and Train a Website Chatbot From Claude Code or Cursor (MCP Walkthrough)

appierun — Thu, 29 Oct 2026 15:57:11 +0000

You can create a support chatbot, train it on your sitemap, and test its answers without leaving your editor. The loop is five tool calls: create_chatbot, add_chatbot_content, chatbot_status, ask_chatbot, and list_chatbots when you come back later. This walkthrough uses Appie.run's MCP server, with every tool name, argument and credit cost taken from appie.run/docs/mcp on 28 Oct 2026.

Disclosure: Appie.run publishes this post, and the server used here is ours.

This is the third walkthrough in the series, after Reddit threads and reply drafts and LinkedIn profiles and leads. If you've done either, skip to Step 3.

What you need

An Appie project. The Free plan has 500 credits and needs no credit card.
A project API key that starts with esk_.
Cursor, Claude Code, or another MCP client that supports remote servers with custom headers.
A public sitemap, or a few pages of text you want the bot to answer from.

Step 1: Create the key

In Appie, open the project and go to API in the sidebar. A project admin generates one esk_ key. Two things worth knowing:

The same key is used by Event Store. Regenerating it stops both immediately.
The key has no balance of its own. Paid calls run on the project's credits, then the project owner's. Every tool result includes credits_remaining.

export APPIE_API_KEY=esk_YOUR_KEY

Keep it out of git.

Step 2: Add the server

Cursor. From Appie's docs:

{
  "mcpServers": {
    "appie": {
      "url": "https://appie.run/mcp",
      "headers": {
        "Authorization": "Bearer YOUR_API_KEY"
      }
    }
  }
}

Claude Code.

claude mcp add --transport http appie https://appie.run/mcp \
  --header "Authorization: Bearer $APPIE_API_KEY"

Run claude mcp list to check the connection. A request with a placeholder key returned HTTP 401 when we tried it on 28 Oct 2026, which is the expected response for a bad key.

Step 3: Find the sitemap you'll train on

The bot learns fastest from a sitemap, because one call pulls in every page it lists. If you don't know where your sitemap lives, Appie's free Sitemap Finder and Checker reads your robots.txt Sitemap: lines and the usual paths (including the WordPress path) and reports which files load and how many URLs they list.

A tip before you train: a sitemap often includes pages you don't want a support bot quoting, such as old blog posts, legal archives or job ads. If yours is noisy, train on a smaller sitemap file or on specific URLs instead.

Step 4: Create the bot

Ask your agent: "Create an Appie chatbot called Support Bot for example.com with a short system prompt."

The agent calls create_chatbot. name is required. Optional fields are system_prompt, openai_model (gpt-4o-mini, gpt-4o or gpt-3.5-turbo) and temperature from 0 to 1. Creation is free. Illustrative arguments:

{
  "name": "Support Bot",
  "system_prompt": "Answer questions about Example Co using only the provided content. If the answer is not in the content, say you don't know and suggest emailing support.",
  "openai_model": "gpt-4o-mini",
  "temperature": 0.2
}

A low temperature keeps answers close to your content. Note the returned chatbot_id; every later call needs it.

Step 5: Add knowledge

Call add_chatbot_content with one of these content_type values:

sitemap: Fields: url (http or https); Use it for: A whole site or section
url: Fields: url (http or https); Use it for: One page, such as pricing or a returns policy
text: Fields: title, content; Use it for: Facts that aren't on the site yet
article: Fields: article_id; Use it for: An article from Appie's Content Writer in the same project

Illustrative calls:

{ "chatbot_id": "YOUR_BOT_ID", "content_type": "sitemap", "url": "https://www.example.com/sitemap.xml" }

{
  "chatbot_id": "YOUR_BOT_ID",
  "content_type": "text",
  "title": "Support hours",
  "content": "Support replies Monday to Friday, 9am to 6pm CET. Email support@example.com."
}

The text type is handy for the questions your site never answers clearly: support hours, refund windows, which plan includes what.

Step 6: Wait for training

Processing runs in the background. Call chatbot_status with the chatbot_id to read the bot's training status and its recent sources. It's free, so poll it as often as you like. A prompt that works well:

"Check chatbot_status for YOUR_BOT_ID every 30 seconds until training finishes, then list the sources it added."

Step 7: Test it before customers do

ask_chatbot sends a message to a chatbot_id and returns a reply. It costs 1 credit per call. Pass the session_id from a previous reply to continue the same conversation, which lets you test follow-up questions.

Write a short test set and have your agent run it:

Three questions the site answers clearly (pricing, a feature, a policy).
Two questions it answers only in passing.
Two questions it doesn't answer at all. The bot should say it doesn't know, not guess.
One follow-up in the same session ("And for the annual plan?").

Ten questions cost 10 credits. If an answer is wrong, the fix is usually the content, not the prompt: add a text source with the correct fact, wait for chatbot_status, and ask again.

Step 8: Put it on your site

The MCP tools cover building, training and testing. To go live, open the chatbot in the Appie dashboard. Per appie.run/ai-chatbots, you embed it with a script tag (or the Shopify app embed), and the dashboard adds lead capture, CSV export and webhooks, and analytics.

Prompts to paste

"List my Appie chatbots and their training status."
"Create a chatbot for example.com, train it on https://www.example.com/sitemap.xml, and tell me when training is done."
"Add a text source titled Refund policy with this content: ..."
"Ask the bot these eight questions, one by one, and give me a table of question, reply, and whether the reply is supported by our site."
"Show credits_remaining after each paid call."

What it costs

list_chatbots: Credits (as of 28 Oct 2026): 0
create_chatbot: Credits (as of 28 Oct 2026): 0
add_chatbot_content: Credits (as of 28 Oct 2026): No cost stated in the docs
chatbot_status: Credits (as of 28 Oct 2026): 0
ask_chatbot: Credits (as of 28 Oct 2026): 1 per message

Building a bot is free on the pricing page too; conversations use chatbot query credits. Paid plans are Starter at $19 a month (2,000 credits) and Standard at $49 a month (5,000 credits), per appie.run/pricing.

Troubleshooting

HTTP 401. The key is missing, wrong, or was regenerated.
app_not_allowed. The person who generated the key can't use the chatbots app in that project. A project owner can use every app.
insufficient_credits. The response includes required and available.
not_found. The bot or article ID belongs to another project.
Answers ignore a page. Check chatbot_status to confirm the source was added, then ask about that page directly.

Errors come back as an MCP tool error whose text is JSON with error and message, so your agent can read and explain them.

The same actions are also available as an HTTP API and CLI; see appie.run/docs/mcp. The repo is on GitHub.

RG, Founder, Appie.run

How to Generate an XML Sitemap for a Static Site With Python (Skip noindex Pages, Split at 50,000 URLs)

appierun — Wed, 28 Oct 2026 15:51:20 +0000

Most static site generators can produce a sitemap, but the default output often includes pages you don't want indexed: drafts marked noindex, duplicate pages that point a canonical elsewhere, and a lastmod on every URL set to the moment the build ran. This post walks through a small Python script that builds sitemap.xml from your build folder and avoids those three problems.

Disclosure: Appie.run publishes this post. Appie has a free XML Sitemap Generator, mentioned near the end. The script below has no dependency on it. Facts about the sitemap format come from sitemaps.org and Google Search Central, read on 28 Oct 2026.

If you want to check an existing sitemap rather than build one, our earlier post How to validate an XML sitemap covers the common errors.

The rules the script follows

From the sitemaps protocol and Google's Build and submit a sitemap page:

Size limits. One sitemap holds at most 50,000 URLs or 50MB uncompressed. Past that, split it into several files and list them in a sitemap index.
UTF-8 and escaping. The file must be UTF-8. Tag values must be entity-escaped, and URLs must be URL-escaped.
Absolute URLs. Use https://www.example.com/page.html, not /page.html.
Canonical URLs only. List the URLs you want in search results.
priority and changefreq. Google says it ignores both. The script leaves them out.
lastmod. Google uses it only if it's consistently and verifiably accurate, and it should reflect the last significant change to the page. A build timestamp on every page fails that test, so the script either uses a real date or omits the tag.
Location. Unless you submit through Search Console, a sitemap only affects URLs under its own directory. Put it at the site root.

The script

Save this as gen_sitemap.py. It uses only the standard library (Python 3.8+).

#!/usr/bin/env python3
"""Build sitemap.xml from a static site's output folder (dist/, public/, _site/)."""
import argparse, datetime, re, subprocess
from pathlib import Path
from urllib.parse import quote
from xml.sax.saxutils import escape

MAX_URLS = 50_000  # per sitemap file, per the sitemaps protocol

NOINDEX = re.compile(r'<meta[^>]+name=["\']robots["\'][^>]+content=["\'][^"\']*noindex', re.I)
CANONICAL = re.compile(r'<link[^>]+rel=["\']canonical["\'][^>]+href=["\']([^"\']+)["\']', re.I)


def page_url(base, root, path):
    rel = path.relative_to(root).as_posix()
    if rel == "index.html":
        rel = ""
    elif rel.endswith("/index.html"):
        rel = rel[: -len("index.html")]  # keep trailing slash
    return base.rstrip("/") + "/" + quote(rel)  # percent-encode spaces, non-ASCII


def last_modified(path, root, src_root):
    """Last git commit date of the matching source file, else None.

    Build output mtimes are all "now", which makes lastmod useless, so we
    only trust git history for a file at the same relative path in --src.
    """
    if src_root:
        src_file = src_root / path.relative_to(root)
        if not src_file.exists():
            return None
        try:
            out = subprocess.run(
                ["git", "log", "-1", "--format=%cs", "--", path.relative_to(root).as_posix()],
                cwd=src_root, capture_output=True, text=True, check=True,
            ).stdout.strip()
            if out:
                return out
        except (subprocess.CalledProcessError, FileNotFoundError):
            pass
    return None


def collect(root, base, src_root):
    entries, skipped = [], []
    for path in sorted(root.rglob("*.html")):
        if path.name == "404.html":
            continue
        url = page_url(base, root, path)
        html = path.read_text(encoding="utf-8", errors="ignore")
        if NOINDEX.search(html):
            skipped.append((url, "noindex"))
            continue
        m = CANONICAL.search(html)
        if m and m.group(1).rstrip("/") != url.rstrip("/"):
            skipped.append((url, "canonical points to " + m.group(1)))
            continue
        entries.append((url, last_modified(path, root, src_root)))
    return entries, skipped


def write_urlset(entries, out):
    lines = ['<?xml version="1.0" encoding="UTF-8"?>',
             '<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">']
    for url, lastmod in entries:
        tag = f"<lastmod>{lastmod}</lastmod>" if lastmod else ""
        lines.append(f"  <url><loc>{escape(url)}</loc>{tag}</url>")
    lines.append("</urlset>")
    out.write_text("\n".join(lines) + "\n", encoding="utf-8")


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("root", type=Path, help="built site folder, e.g. dist")
    ap.add_argument("--base", required=True, help="https://www.example.com")
    ap.add_argument("--src", type=Path, help="git checkout whose files mirror the output paths")
    args = ap.parse_args()

    entries, skipped = collect(args.root, args.base, args.src)
    for url, why in skipped:
        print(f"skipped {url}: {why}")

    chunks = [entries[i:i + MAX_URLS] for i in range(0, len(entries), MAX_URLS)] or [[]]
    if len(chunks) == 1:
        write_urlset(chunks[0], args.root / "sitemap.xml")
        print(f"wrote sitemap.xml with {len(entries)} URLs")
        return
    names = []
    for n, chunk in enumerate(chunks, 1):
        name = f"sitemap-{n}.xml"
        write_urlset(chunk, args.root / name)
        names.append(name)
    today = datetime.date.today().isoformat()
    index = ['<?xml version="1.0" encoding="UTF-8"?>',
             '<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">']
    for name in names:
        loc = escape(args.base.rstrip("/") + "/" + name)
        index.append(f"  <sitemap><loc>{loc}</loc><lastmod>{today}</lastmod></sitemap>")
    index.append("</sitemapindex>")
    (args.root / "sitemap.xml").write_text("\n".join(index) + "\n", encoding="utf-8")
    print(f"wrote sitemap index with {len(names)} files, {len(entries)} URLs")


if __name__ == "__main__":
    main()

What each part does

page_url maps a file path to a URL. index.html becomes the folder URL with a trailing slash, so dist/blog/a-post/index.html becomes https://www.example.com/blog/a-post/. Paths are percent-encoded, so my page.html becomes my%20page.html.

collect walks every .html file and skips three kinds of page:

404.html
pages with a robots meta tag containing noindex
pages whose rel="canonical" points to a different URL

Each skip is printed, so you can see why a page is missing.

last_modified is the part most generators get wrong. Files in a build folder all have the build time as their modified date, which makes every lastmod identical and meaningless. The script only sets lastmod when you pass --src with a git checkout whose files sit at the same relative paths as the output (true for hand-written HTML sites, often not true for generators that render Markdown). It then uses the date of the last commit touching that file. Otherwise it leaves lastmod out, which is valid.

write_urlset escapes each URL with xml.sax.saxutils.escape, so an & in a query string becomes &.

main writes one sitemap.xml when you have 50,000 URLs or fewer. Above that it writes sitemap-1.xml, sitemap-2.xml and so on, and makes sitemap.xml a sitemap index pointing to them.

Run it

# after your build step
python3 gen_sitemap.py dist --base https://www.example.com

# with lastmod from git, for a site whose source paths match the output
python3 gen_sitemap.py dist --base https://www.example.com --src .

On a small test folder, the output looked like this:

skipped https://www.example.com/about-copy.html: canonical points to https://www.example.com/about.html
skipped https://www.example.com/drafts/x.html: noindex
wrote sitemap.xml with 5 URLs

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url><loc>https://www.example.com/about.html</loc></url>
  <url><loc>https://www.example.com/blog/a-post/</loc></url>
  <url><loc>https://www.example.com/</loc></url>
  <url><loc>https://www.example.com/my%20page.html</loc></url>
  <url><loc>https://www.example.com/tom%26jerry.html</loc></url>
</urlset>

To test the split, change MAX_URLS to 2 in a copy of the script. On the same folder it wrote three files plus an index.

Add it to your build

In a package.json:

{
  "scripts": {
    "build": "astro build && python3 gen_sitemap.py dist --base https://www.example.com"
  }
}

Or as a CI step after the build, before deploy. Then add one line to robots.txt so crawlers find it:

Sitemap: https://www.example.com/sitemap.xml

Google's docs list three ways to submit: Search Console, the Search Console API, or the Sitemap: line in robots.txt.

Known limits

Regex HTML parsing. The noindex and canonical checks expect the attributes in the usual order (name before content, rel before href). If your templates write them the other way round, swap in html.parser from the standard library.
No 50MB check. At 50,000 short URLs you'll be well under 50MB, but if your URLs are very long, check the file size too.
Only HTML. PDFs and other files you want indexed need adding by hand.
Trailing slashes. The canonical comparison ignores a trailing slash. If your site treats /page and /page/ as different pages, tighten that check.

When a script is more than you need

If you just need a one-off sitemap for a small site, a hosted tool is quicker:

Appie's free XML Sitemap Generator crawls a public site for up to 80 pages, or takes a pasted list of up to 50,000 URLs, and returns sitemap XML. A single free run stops at 80 HTML pages or after 12 seconds, so paste a URL list for larger sites. It asks you to choose changefreq and priority; since Google ignores both, any value is fine. There's a captcha before each run.
Screaming Frog SEO Spider is a desktop crawler that can generate XML sitemaps. Per its site on 28 Oct 2026, it crawls 500 URLs for free, and a licence to remove the limit is £199 per year.

After generating, run the file through a validator such as Appie's free XML Sitemap Validator, which checks the root element, absolute URLs, lastmod, duplicates and the 50,000 URL limit.

RG, Founder, Appie.run