DEV Community

Devil Scrapes
Devil Scrapes

Posted on

Your sitemap checker is probably parsing a 404 page that returned 200

#webscraping #python #api #seo

Quick answer

Your sitemap checker probably reports success on sites that have no sitemap.

GET /sitemap.xml returning HTTP 200 is not evidence a sitemap exists. A large share of CMS installs answer every unknown path with a 200 and a themed "page not found" page. Your XML parser then either throws on <!doctype html> — and you log "malformed sitemap" for a site that simply doesn't have one — or, worse, parses it as empty and reports zero URLs, which looks exactly like a site with an empty sitemap.

The fix is four lines: sniff the body, don't trust the status.

The 404-as-200 problem 🕵️

Here is the check almost everyone writes first:

r = requests.get(f"https://{domain}/sitemap.xml")
if r.status_code == 200:
    urls = parse_sitemap(r.text)      # ← this is the bug
Enter fullscreen mode Exit fullscreen mode

The status code told you the server responded. It did not tell you what it responded with. Content-Type helps and is also not sufficient — plenty of misconfigured servers send text/html for real XML, and application/xml for an error page.

The reliable signal is the first few bytes. A real sitemap starts with <?xml or a bare <urlset> / <sitemapindex> root. An HTML shell starts with <!doctype html> or <html:

def looks_like_html(content: bytes) -> bool:
    """Sniff for an HTML shell masquerading as a 200 sitemap response.

    A real sitemap starts with `<?xml` or a bare `<urlset>` /
    `<sitemapindex>` root tag; a CMS 404-as-200 or redirect landing page
    starts with `<!doctype html>` or `<html`. Checked on a small
    lower-cased prefix so we never decode/parse a large body just to
    reject it.
    """
    prefix = content[:HTML_SNIFF_WINDOW].lstrip().lower()
    return any(prefix.startswith(marker.encode()) for marker in HTML_MARKERS)
Enter fullscreen mode Exit fullscreen mode

Note the [:HTML_SNIFF_WINDOW]. Sitemaps go up to 50 MB uncompressed. Rejecting one by decoding the entire body to a string and calling .startswith() works, and it also means a hostile or misconfigured host can make you allocate 50 MB to learn nothing. Sniff a prefix.

The lstrip() matters too — a leading blank line or BOM before <?xml is common enough that a strict prefix match without it produces false rejections on perfectly good sitemaps.

Then check robots.txt, and check it second

/sitemap.xml is a convention, not a requirement. The actual declaration lives in robots.txt:

User-agent: *
Disallow: /admin

Sitemap: https://example.com/sitemap_index.xml
Sitemap: https://example.com/news-sitemap.xml
Enter fullscreen mode Exit fullscreen mode

Sites routinely publish several, under names you would never guess, on hosts that are not even the same domain (a CDN, or a sitemaps. subdomain). So the discovery order is: try /sitemap.xml, validate it by shape, then parse Sitemap: lines out of robots.txt and follow those too.

Doing robots.txt as well rather than instead is deliberate. Some sites serve a working /sitemap.xml and never mention it in robots.txt; some declare three and 404 on the conventional path. Only doing both finds every site.

Gzip: trust the magic bytes, not the header 🗜️

Sitemaps are commonly served gzipped, and the signals disagree with each other constantly:

  • some servers set Content-Encoding: gzip correctly
  • some serve a genuinely gzipped .xml.gz body and set no encoding header at all
  • some set the header on a body that is not gzipped
  • and your HTTP client may have transparently decoded it already, so by the time you look, a correctly-declared stream is plain XML

Four signals, none authoritative. The one that actually correlates with reality is the two-byte gzip magic number:

GZIP_MAGIC = b"\x1f\x8b"

def maybe_gunzip(raw, *, url, content_encoding):
    """Decompress gzip content, detected by magic bytes OR server hints.

    Some servers set `Content-Encoding: gzip` correctly, some don't set
    it at all despite serving a genuinely gzipped `.xml.gz` body, and
    curl-cffi may already have auto-decoded a properly-declared stream —
    so magic-byte sniffing is the reliable signal, not the header or the
    `.gz` extension alone.
    """
Enter fullscreen mode Exit fullscreen mode

And when the signals conflict — header says gzip, magic bytes say otherwise — it warns and proceeds with the raw body rather than raising. The header being wrong is the server's bug; refusing to read a perfectly good XML document over it would be ours.

Order matters here as well: gunzip before the HTML sniff. Otherwise a gzipped body fails the sniff on its binary prefix and every compressed sitemap on the internet gets discarded as "not XML".

Indexes of indexes, and the loop 🔁

A <sitemapindex> points at other sitemaps, which can themselves be indexes. Large sites nest two or three deep, and a recursive fetch has two ways to run forever:

  1. A self-referencing index. A sitemap that lists itself. Rarer than you would like.
  2. A cycle. Index A lists B, B lists A. This happens by accident whenever someone generates sitemaps with a template.

A depth limit alone does not fix this — it bounds the damage but you still refetch the same files repeatedly on the way down. Both guards are needed, and they do different jobs:

if entry.kind == SITEMAP_ENTRY_KIND and depth >= ctx.max_depth:
    continue
if entry.kind == SITEMAP_ENTRY_KIND and entry.loc in ctx.visited:
    continue
Enter fullscreen mode Exit fullscreen mode

The visited set makes a cycle terminate immediately instead of at the depth ceiling. The depth ceiling stops a legitimately-deep-but-finite tree from being unbounded work. Plus a per-site and a global row budget, so one enormous site cannot consume the entire run and starve the other 49 domains in your input list.

Namespaces, briefly

Sitemap XML is namespaced, so elem.tag is not urlset — it is {http://www.sitemaps.org/schemas/sitemap/0.9}urlset. And in the wild you will meet the wrong namespace URI, the http-vs-https variant, and no namespace at all.

Matching on the namespaced tag makes your parser correct against the spec and useless against the internet. Strip it:

def _local_tag(tag: str) -> str:
    """Strip the XML namespace: ``{http://.../0.9}urlset`` -> ``urlset``."""
    return tag.rsplit("}", 1)[-1] if "}" in tag else tag
Enter fullscreen mode Exit fullscreen mode

The part that generalises 🧭

Every bug above is the same bug wearing a different hat: a metadata channel was trusted over the content itself. Status code over body. Content-Encoding over magic bytes. File extension over magic bytes. Declared namespace over local tag name.

Metadata is a claim made by whoever configured the server, and it is wrong at a rate that will surprise you. The content is the thing that is actually true. When the two disagree, believe the bytes — and log the disagreement, because a mismatch is often the most interesting thing you will learn about a host.

We have hit this shape enough times to treat it as a rule. A probe that followed redirects and skipped Content-Type once logged a dead single-page-app shell as a live JSON API, and we queued a build against it. HTTP 200 is not reachability.

What the Actor gives you

One row per <url> entry, tagged with the sitemap file and source site it came from:

  • discovery via /sitemap.xml and robots.txt Sitemap: declarations
  • shape validation, so a 404-as-200 HTML page is skipped with a warning instead of parsed
  • transparent gzip handling by magic-byte detection, including undeclared .xml.gz
  • recursive <sitemapindex> expansion to a configurable depth, with a visited-URL set so cycles terminate
  • lastmod, changefreq and priority where present, namespace-agnostic parsing
  • per-site and global row budgets, so one huge site can't eat the run
  • browser-fingerprint rotation via curl-cffi and proxy rotation — a sitemap fetch gets blocked more often than you would expect

The honest limitations 🚧

  • Sitemaps are a publisher's claim about their site, not a crawl. A URL listed here may 404; an unlisted page may exist.
  • We do not fetch the listed URLs. This extracts the sitemap, it does not verify the pages.
  • 50 MB / 50,000-entry sitemap limits are the spec's. A non-compliant oversized sitemap is fetched to your configured cap and no further.
  • Sites that publish no sitemap and declare none in robots.txt yield zero rows — reported explicitly as "no sitemap found", not as an error.

Pricing

$0.20 per run plus $0.002 per URL — about $2.20 per 1,000 results. A site with no sitemap costs the start fee and nothing else.

→ XML Sitemap URL Extractor on Apify

Built by Devil Scrapes. We handle the 404s that return 200, the undeclared gzip, the self-referencing indexes and the wrong namespaces, so you get a flat table instead of a weekend.

Top comments (0)