If you've built a RAG chatbot, you know the boring part is getting clean text out of websites. Navigation menus, cookie banners, footers and "related posts" widgets all end up in your embeddings and hurt retrieval.
The common answer is "spin up a headless browser and crawl." That works, but it's slow and memory-hungry, and on most hosted platforms it's billed by compute time, so you can't predict the cost.
Most docs sites, blogs and help centers don't need a browser at all: the content is already in the HTML. So I built a crawler that skips the browser entirely.
What it does
Website to Markdown for LLM & RAG is an Apify actor that:
- crawls from your start URLs and stays on the same domain, with optional sitemap.xml discovery
- extracts only the main content (headings, lists, code blocks and tables) as Markdown
- can chunk pages for embeddings, with overlap; you pay per page, not per chunk
- respects robots.txt and supports include/exclude URL patterns
- returns title, description, author, date, language and word count for each page
Benchmark: HTTP vs headless Chromium
Same 30 pages, 5 parallel requests, Playwright Chromium vs plain HTTP plus extraction:
| Site | HTTP (this actor) | Headless browser | Speed-up |
|---|---|---|---|
| MDN Web Docs | 1.0 s | 9.5 s | 9.8x |
| MDN Web Docs (cold) | 3.2 s | 8.1 s | 2.5x |
| Simple static blog | 1.9 s | 3.4 s | 1.8x |
| Heavy news site | finished | timed out | – |
The heavier the page (scripts, ads, trackers), the bigger the gap. Price is flat: $1 per 1,000 pages, plus $0.01 per run.
Use it from Python
from apify_client import ApifyClient
client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("datacraft-tools/web-to-markdown").call(run_input={
"startUrls": [{"url": "https://docs.python.org/3/tutorial/"}],
"maxPages": 100,
"maxCrawlDepth": 2,
"chunkSize": 2000,
"chunkOverlap": 200,
})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
print(item["url"], item["chunkIndex"], item["markdown"][:80])
Each row is ready to embed. Push markdown into Pinecone, Qdrant, Chroma or pgvector together with url and title as metadata.
When NOT to use it
If a site renders its content only with JavaScript (some single-page apps), an HTTP crawler will see an empty page and you need a browser-based crawler. For docs, blogs, news, marketing sites and knowledge bases, HTTP is the faster and cheaper choice.
I built this actor. Feedback and feature requests are welcome in the comments.
Top comments (0)