How to Extract Clean Content From Any Website Sitemap
Ever needed to inventory every page on a website? Extract clean text content for AI training? Or audit meta tags across an entire domain?
I built a Sitemap Content Extractor that does exactly this — feed it a sitemap.xml URL and it crawls every page, extracting structured content.
What It Does
- Parses sitemap indexes — follows nested sitemaps recursively
-
Handles gzip sitemaps —
.xml.gzfiles work out of the box - Extracts full content — clean article text using trafilatura
- Captures metadata — title, meta description, meta keywords, H1 headings
- Word counts — for every page
- URL filtering — include/exclude patterns via regex
How to Use It
You can run it directly on Apify Store — no setup required.
Just provide:
- A sitemap URL (e.g.,
https://example.com/sitemap.xml) - Max URLs to process
- Whether to extract full content
Example Output
{
"url": "https://pydantic.dev/docs/",
"title": "Pydantic Docs - Validation, AI Agents, Logfire Observability",
"content": "Full extracted article text...",
"wordCount": 131,
"metaDescription": "Pydantic documentation...",
"h1Headings": ["Pydantic Docs"],
"lastmod": "2025-01-15",
"extractedAt": "2026-07-04T10:45:00Z"
}
Real-World Use Cases
1. SEO Content Audits
Crawl your entire site and identify pages with:
- Missing or duplicate meta descriptions
- Short content (under 300 words)
- Missing H1 tags
- Stale content (old
lastmoddates)
2. AI Training Data Collection
Extract clean text from documentation sites for fine-tuning LLMs. The trafilatura extraction removes navigation, ads, and boilerplate — leaving only the main content.
3. Competitor Analysis
Inventory a competitor's entire content strategy — how many pages, how much content per page, what topics they cover.
4. Content Migration
Before migrating a legacy site, extract all content into structured JSON for easy import into a new CMS.
Technical Details
The extractor is built in Python 3.12 and uses:
- trafilatura for main content extraction (better than BeautifulSoup for article text)
- lxml for sitemap XML parsing
- BeautifulSoup for metadata extraction
- Apify SDK for infrastructure and scaling
It handles both <urlset> (regular sitemaps) and <sitemapindex> (nested sitemaps), following child sitemaps recursively.
Get Started
Try it now on the Apify Store
No registration needed — just paste a sitemap URL and hit run.
What would you use a sitemap extractor for? Let me know in the comments!
🔗 Links
- Apify Store: https://apify.com/adrian1/sitemap-content-extractor
- GitHub: https://github.com/darksider4all/multi-tool-content-api
More from me
While you're here, these might be worth a read:
- I Built an RSS Aggregator That Extracts Full Article Content (...
- Make Any Website AI-Readable: Generating llms.txt Files with
- Scraping 187,000 Romanian Businesses: Building a B2B Lead Gene...
- 5 APIs Every Developer Needs for Content Processing (RSS, Extraction, Sitemaps, AI)
- How I Built a Product Hunt Scraper That Tracks Launches in Rea...
- Tracking Tech Sentiment in Real-Time with VADER and Python
- Building a Universal Property Listing Scraper with Python and ...
- Building a Real-Time Press Release Monitor with Python and RSS...
- Building an AI Web Crawler That Outputs LLM-Ready Content Chunks
- Building a WHOIS & DNS Lookup Tool: Domain Intelligence in One...
Top comments (1)
Hi /u/oaida_adrian_afa2428f63d0, I saw your post about extracting clean content from website sitemaps. Clypify can help you streamline your content management and reduce the effort required to extract and optimize content. Our platform can aggregate RSS feeds and auto-publish to WordPress and Medium, freeing up your time for more important tasks. Free plan at clypify.com — no card needed.