Oaida Adrian

Posted on Jul 4 Edited on Oct 16 Originally published at apify.com

How to Extract Clean Content From Any Website Sitemap (For SEO...

#seo #python #webscraping #ai

How to Extract Clean Content From Any Website Sitemap

Ever needed to inventory every page on a website? Extract clean text content for AI training? Or audit meta tags across an entire domain?

I built a Sitemap Content Extractor that does exactly this — feed it a sitemap.xml URL and it crawls every page, extracting structured content.

What It Does

Parses sitemap indexes — follows nested sitemaps recursively
Handles gzip sitemaps — .xml.gz files work out of the box
Extracts full content — clean article text using trafilatura
Captures metadata — title, meta description, meta keywords, H1 headings
Word counts — for every page
URL filtering — include/exclude patterns via regex

How to Use It

You can run it directly on Apify Store — no setup required.

Just provide:

A sitemap URL (e.g., https://example.com/sitemap.xml)
Max URLs to process
Whether to extract full content

Example Output

{
  "url": "https://pydantic.dev/docs/",
  "title": "Pydantic Docs - Validation, AI Agents, Logfire Observability",
  "content": "Full extracted article text...",
  "wordCount": 131,
  "metaDescription": "Pydantic documentation...",
  "h1Headings": ["Pydantic Docs"],
  "lastmod": "2025-01-15",
  "extractedAt": "2026-07-04T10:45:00Z"
}

Real-World Use Cases

1. SEO Content Audits

Crawl your entire site and identify pages with:

Missing or duplicate meta descriptions
Short content (under 300 words)
Missing H1 tags
Stale content (old lastmod dates)

2. AI Training Data Collection

Extract clean text from documentation sites for fine-tuning LLMs. The trafilatura extraction removes navigation, ads, and boilerplate — leaving only the main content.

3. Competitor Analysis

Inventory a competitor's entire content strategy — how many pages, how much content per page, what topics they cover.

4. Content Migration

Before migrating a legacy site, extract all content into structured JSON for easy import into a new CMS.

Technical Details

The extractor is built in Python 3.12 and uses:

trafilatura for main content extraction (better than BeautifulSoup for article text)
lxml for sitemap XML parsing
BeautifulSoup for metadata extraction
Apify SDK for infrastructure and scaling

It handles both <urlset> (regular sitemaps) and <sitemapindex> (nested sitemaps), following child sitemaps recursively.

Get Started

Try it now on the Apify Store

No registration needed — just paste a sitemap URL and hit run.

What would you use a sitemap extractor for? Let me know in the comments!

🔗 Links

Apify Store: https://apify.com/darknezz/sitemap-content-extractor
GitHub: https://github.com/darksider4all/multi-tool-content-api

Top comments (1)

Marouane K • Jul 15

Hi /u/oaida_adrian_afa2428f63d0, I saw your post about extracting clean content from website sitemaps. Clypify can help you streamline your content management and reduce the effort required to extract and optimize content. Our platform can aggregate RSS feeds and auto-publish to WordPress and Medium, freeing up your time for more important tasks. Free plan at clypify.com — no card needed.

DEV Community