Turn any website into clean Markdown for your RAG pipeline (with code)

If you've built anything with retrieval-augmented generation (RAG), you know the unglamorous truth: 80% of the work is getting clean text in. Your embeddings are only as good as the content you feed them, and raw web pages are a mess of nav bars, cookie banners, ads, share widgets, and <div> soup.

This post shows a fast, reliable way to convert any URL — or a whole site — into clean, LLM-ready Markdown, with code you can drop into a pipeline today.

Why not just requests + BeautifulSoup?

You can, and for a single static page it's fine:

import requests
from bs4 import BeautifulSoup

html = requests.get("https://example.com").text
text = BeautifulSoup(html, "html.parser").get_text()

But this falls over fast in the real world:

• You get everything — menus, footers, "related articles," cookie notices — all polluting your chunks and your embeddings.
• JavaScript-rendered sites (React, Vue, Next.js) return an empty shell; you need a real browser.
• Structure is lost — headings, lists, tables and code blocks matter to an LLM, and .get_text() flattens them.
• Crawling a whole docs site means writing link-following, dedup, robots.txt handling, and concurrency yourself.

Readability + a headless browser + a Markdown converter solves the quality problem, but now you're maintaining browser infrastructure. That's the part worth outsourcing.

The hosted approach

Website to Markdown runs the whole pipeline for you — real Chromium rendering → Mozilla Readability (main-content extraction) → Markdown with headings, links, tables and code preserved. It respects robots.txt, can crawl a site via its sitemap, and returns one clean record per page.

One page

curl -X POST "https://api.apify.com/v2/acts/fetchbase~website-to-markdown/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "urls": ["https://en.wikipedia.org/wiki/Retrieval-augmented_generation"] }'

You get back:

[{
  "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
  "title": "Retrieval-augmented generation",
  "description": "…",
  "wordCount": 2841,
  "markdown": "**Retrieval-augmented generation (RAG)** is a technique that…"
}]

A whole documentation site

Point it at the root, turn on sitemap discovery, and filter to the section you care about:

{
  "urls": ["https://docs.your-product.com"],
  "crawl": true,
  "useSitemap": true,
  "maxPages": 300,
  "includeUrlPatterns": ["*/docs/*"],
  "excludeUrlPatterns": ["*/changelog/*"]
}

Wiring it into a RAG pipeline

Here's the whole ingest step in Node — fetch clean Markdown, chunk it, embed it:

const runResp = await fetch(
  "https://api.apify.com/v2/acts/fetchbase~website-to-markdown/run-sync-get-dataset-items?token=" + process.env.APIFY_TOKEN,
  {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({
      urls: ["https://docs.your-product.com"],
      crawl: true,
      useSitemap: true,
      maxPages: 50,
    }),
  }
);
const pages = await runResp.json();

// Chunk each page's markdown and embed
for (const page of pages) {
  const chunks = chunkMarkdown(page.markdown, { maxTokens: 500 });
  for (const chunk of chunks) {
    const embedding = await embed(chunk);
    await vectorStore.upsert({
      text: chunk,
      embedding,
      metadata: { url: page.url, title: page.title },
    });
  }
}

Because the output is clean Markdown, your chunker splits on real headings instead of guessing, and your citations point at real page titles and URLs.

For big crawls (hundreds of pages), use the async endpoint (POST /v2/acts/…/runs)
and poll the run, or run it on a Schedule — the run-sync-* endpoint used
above is best for a handful of pages, since it caps at ~5 minutes.

Don't forget PDFs and Word docs

Half of most companies' knowledge lives in PDFs and .docx files, not web pages. The companion PDF & DOCX to Markdown actor handles those with the same output shape, so your ingest code stays uniform:

{ "fileUrls": ["https://example.com/whitepaper.pdf"], "outputFormat": "markdown" }

Keep the index fresh

Wrap the run in an Apify Schedule (say, nightly) and re-embed changed pages. Use the sinceDate pattern on feeds or a content hash to avoid re-embedding everything.

Cost

It's pay-per-page with no subscription and no startup fee — you're only billed for pages that extract successfully, and free-tier Apify credits cover a generous amount of testing. For a few hundred docs pages that's cents, versus the ongoing cost of running and babysitting your own browser fleet.

---

If you try it, I'd genuinely like feedback — what breaks, what's missing. Open an issue on the actor and it gets fixed fast. There's a whole suite of these (screenshots, SEO/Core-Web-Vitals audits, tech-stack detection, feeds, job APIs) at apify.com/fetchbase if they're useful to you.