DEV Community

Ramazan Akdoğan
Ramazan Akdoğan

Posted on

Turn any website into clean Markdown for RAG without a headless browser

If you've built a RAG chatbot, you know the boring part is getting clean text out of websites. Navigation menus, cookie banners, footers and "related posts" widgets all end up in your embeddings and hurt retrieval.

The common answer is "spin up a headless browser and crawl." That works, but it's slow and memory-hungry, and on most hosted platforms it's billed by compute time, so you can't predict the cost.

Most docs sites, blogs and help centers don't need a browser at all: the content is already in the HTML. So I built a crawler that skips the browser entirely.

What it does

Website to Markdown for LLM & RAG is an Apify actor that:

  • crawls from your start URLs and stays on the same domain, with optional sitemap.xml discovery
  • extracts only the main content (headings, lists, code blocks and tables) as Markdown
  • can chunk pages for embeddings, with overlap; you pay per page, not per chunk
  • respects robots.txt and supports include/exclude URL patterns
  • returns title, description, author, date, language and word count for each page

Benchmark: HTTP vs headless Chromium

Same 30 pages, 5 parallel requests, Playwright Chromium vs plain HTTP plus extraction:

Site HTTP (this actor) Headless browser Speed-up
MDN Web Docs 1.0 s 9.5 s 9.8x
MDN Web Docs (cold) 3.2 s 8.1 s 2.5x
Simple static blog 1.9 s 3.4 s 1.8x
Heavy news site finished timed out –

The heavier the page (scripts, ads, trackers), the bigger the gap. Price is flat: $1 per 1,000 pages, plus $0.01 per run.

Use it from Python

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("datacraft-tools/web-to-markdown").call(run_input={
    "startUrls": [{"url": "https://docs.python.org/3/tutorial/"}],
    "maxPages": 100,
    "maxCrawlDepth": 2,
    "chunkSize": 2000,
    "chunkOverlap": 200,
})

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["url"], item["chunkIndex"], item["markdown"][:80])
Enter fullscreen mode Exit fullscreen mode

Each row is ready to embed. Push markdown into Pinecone, Qdrant, Chroma or pgvector together with url and title as metadata.

When NOT to use it

If a site renders its content only with JavaScript (some single-page apps), an HTTP crawler will see an empty page and you need a browser-based crawler. For docs, blogs, news, marketing sites and knowledge bases, HTTP is the faster and cheaper choice.


I built this actor. Feedback and feature requests are welcome in the comments.

Top comments (0)