DEV Community

Mohamed Zaghloul
Mohamed Zaghloul Subscriber

Posted on

Turn any website into clean Markdown for RAG (incremental sync, llms.txt)

#rag #llm #webscraping #tutorial

If you've ever fed a website into a RAG pipeline, you know the boring part isn't the embeddings. It's getting clean text. Nav menus, cookie banners, "related posts" widgets and footers all end up in your chunks, and then every time the site changes you either re-scrape everything or guess what changed.

I built a small Apify Actor for exactly this, called Sitemap to Markdown. This post walks through a real run on the htmx essays: what goes in, what comes out, how re-runs only pick up changes, and what it costs.

The idea: use the sitemap, not a crawler

Most sites already publish a list of every page and when it last changed: sitemap.xml. So instead of crawling links, the Actor:

  1. Finds the sitemap on its own (robots.txt, /sitemap.xml, sitemap indexes, .xml.gz)
  2. Keeps only the URLs under the path you gave it
  3. Downloads each page with plain HTTP (no headless browser)
  4. Strips the boilerplate and converts the main content to Markdown
  5. Optionally splits it into RAG-sized chunks and writes llms.txt / llms-full.txt

If a site has no sitemap, it falls back to a shallow same-site link crawl.

Input

This is the exact input I ran:

{
  "startUrls": ["https://htmx.org/essays"],
  "maxPages": 0,
  "incremental": true,
  "incrementalStateKey": "devto-demo-htmx",
  "chunkSize": 400,
  "createLlmsTxt": true
}
Enter fullscreen mode Exit fullscreen mode
  • maxPages: 0 means no limit: every essay URL in the sitemap.
  • incremental: true makes the Actor remember what it has seen, under the name in incrementalStateKey.
  • chunkSize: 400 splits each page into chunks of about 400 tokens, by heading and paragraph.

Output

You get one dataset record per page. Each page's markdown starts with YAML front matter, so it can go straight into a file-based pipeline:

---
title: "</> htmx ~ A Real World wasm to htmx Port"
url: "https://htmx.org/essays/a-real-world-wasm-to-htmx-port/"
canonical_url: "https://htmx.org/essays/a-real-world-wasm-to-htmx-port/"
lastmod: "2025-01-10"
language: "en"
word_count: 617
thin_content: false
---

# A Real World wasm to htmx Port

Joe Fioti

January 10, 2025

When I was in college, I wrote some customer service software that tied together ...
Enter fullscreen mode Exit fullscreen mode

The rest of the record (trimmed):

{
  "type": "page",
  "url": "https://htmx.org/essays/a-real-world-wasm-to-htmx-port/",
  "canonicalUrl": "https://htmx.org/essays/a-real-world-wasm-to-htmx-port/",
  "lastmod": "2025-01-10",
  "changeStatus": "new",
  "contentHash": "958fdb17b6246b5bea12b422b88cc12d12204a467949cfee7522513928b8917b",
  "wordCount": 617,
  "tokenEstimate": 897,
  "chunkCount": 3,
  "chunks": [
    { "index": 0, "heading": "A Real World wasm to htmx Port", "text": "# A Real World wasm to htmx Port\n\nJoe Fioti ...", "tokenEstimate": 324 }
  ]
}
Enter fullscreen mode Exit fullscreen mode

A few fields that matter for RAG:

  • chunks: each chunk keeps its heading, so you can store it as metadata and cite sources.
  • contentHash: a SHA-256 of the Markdown, handy for "only re-embed what changed".
  • canonicalUrl: lets you skip duplicate paths. The Actor already skips exact duplicates by default.
  • thin_content: pages with almost no main content (under 50 words by default) are flagged as thin_page instead of being written like a normal page. That includes JS-only pages, since there's no browser rendering. They're also left out of llms.txt, so they don't end up embedded as noise.

The run also saves llms.txt (a title + link + description index) and llms-full.txt (every page's Markdown in one file) to the key-value store. The second one is handy for uploading a whole site to ChatGPT, Claude Projects or NotebookLM. Here's the start of the generated llms.txt:

# htmx.org

> Pages from https://htmx.org/essays, converted to Markdown by the Sitemap to Markdown Actor on 2026-10-18.

## Pages

- [</> htmx ~ Architectural Sympathy](https://htmx.org/essays/architectural-sympathy/): In this essay, Carson Gross explores architectural sympathy as a software design principle ...
Enter fullscreen mode Exit fullscreen mode

Incremental sync: the part I actually care about

Re-scraping a whole docs site every night to catch two edited pages is wasteful. With incremental: true, the Actor keeps a small state record in a named key-value store in your own Apify account. On each run it:

  • skips pages whose sitemap lastmod hasn't changed (they aren't even downloaded)
  • downloads the rest, compares content hashes, and only outputs pages that are new or changed
  • writes a CHANGES record listing new, changed, unchanged and removed URLs

Here's what happened on the last two runs of this input. Earlier runs, with a page cap, had already synced 41 of the 56 essay pages:

Run Pages output Runtime CHANGES
Catch-up run the 15 remaining pages, all new 7.1 s new: 15, unchanged: 41
Re-run right after 0 pages 1.6 s new: 0, changed: 0, unchanged: 56, removed: 0

The CHANGES record is what you wire into your vector store: upsert new and changed, delete removed. Then a nightly schedule only does real work on days the site actually changed.

Two honest caveats:

  • removed is only reliable when the run reads the whole sitemap: maxPages: 0 and no lastmodAfter date filter.
  • maxPages limits new and changed pages per run, so with a cap, a big site fills in over several runs.

What it costs

It's pay-per-result on Apify:

  • $0.50 per 1,000 pages saved (that's $0.0005 a page), plus a $0.00005 start fee per run
  • Chunks live inside the page record, so chunking costs nothing extra; llms.txt is free too
  • A re-run where nothing changed outputs nothing, so you pay only the start fee

The full htmx essays section (56 pages) costs under 3 cents the first time and about nothing on a quiet night. You can set a max cost per run in Apify, and the Actor stops cleanly at that limit.

When not to use it

It's plain HTTP, so it won't render JavaScript-only SPAs, log in, or get past aggressive bot protection. Pages from those sites come back flagged as thin, or as a BLOCKED error instead of silently empty. For those, a browser-based crawler like Apify's Website Content Crawler is the better (more expensive) tool. This one is aimed at docs, blogs, help centers and marketing sites, which is most of what people actually want in a RAG index.

Try it

  • Store page: Sitemap to Markdown
  • It runs from the Apify console, the API, or as a tool for AI agents through the Apify MCP server.

It's new, so I'd genuinely like feedback. If the Markdown isn't clean enough for your pipeline, or a field is missing, tell me in the comments or open an issue on the Actor page.

Top comments (1)

Collapse
 
agentsearchhq profile image
AgentSearch •

Using lastmod from the sitemap as the change signal is a smart shortcut. One catch I'd watch for: some static site generators stamp every URL with the build date, so every page looks changed after each deploy. Comparing contentHash before re-embedding catches that, since the markdown itself didn't move, and it keeps a site-wide rebuild from re-embedding the whole corpus.