Turn any website into clean Markdown for your RAG pipeline (with code)

If you've built anything with retrieval-augmented generation (RAG), you know the
unglamorous truth: 80% of the work is getting clean text in. Your embeddings
are only as good as the content you feed them, and raw web pages are a mess of
nav bars, cookie banners, ads, share widgets, and <div> soup.

This post shows a fast, reliable way to convert any URL — or a whole site —
into clean, LLM-ready Markdown, with code you can drop into a pipeline today.

Why not just requests + BeautifulSoup?

You can, and for a single static page it's fine:

import requests
from bs4 import BeautifulSoup

html = requests.get("https://example.com").text
text = BeautifulSoup(html, "html.parser").get_text()

But this falls over fast in the real world:

• You get everything — menus, footers, "related articles," cookie notices — all polluting your chunks and your embeddings.
• JavaScript-rendered sites (React, Vue, Next.js) return an empty shell; you need a real browser.
• Structure is lost — headings, lists, tables and code blocks matter to an LLM, and .get_text() flattens them.
• Crawling a whole docs site means writing link-following, dedup, robots.txt handling, and concurrency yourself.

Readability + a headless browser + a Markdown converter solves the quality
problem, but now you're maintaining browser infrastructure. That's the part
worth outsourcing.

The hosted approach

Website to Markdown runs the
whole pipeline for you — real Chromium rendering → Mozilla Readability
(main-content extraction) → Markdown with headings, links, tables and code
preserved. It respects robots.txt, can crawl a site via its sitemap, and
returns one clean record per page.

One page

curl -X POST "https://api.apify.com/v2/acts/runlayer~website-to-markdown/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "urls": ["https://en.wikipedia.org/wiki/Retrieval-augmented_generation"] }'

You get back:

[{
  "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
  "title": "Retrieval-augmented generation",
  "description": "…",
  "wordCount": 2841,
  "markdown": "**Retrieval-augmented generation (RAG)** is a technique that…"
}]

A whole documentation site

Point it at the root, turn on sitemap discovery, and filter to the section you
care about:

{
  "urls": ["https://docs.your-product.com"],
  "crawl": true,
  "useSitemap": true,
  "maxPages": 300,
  "includeUrlPatterns": ["*/docs/*"],
  "excludeUrlPatterns": ["*/changelog/*"]
}

Wiring it into a RAG pipeline

Here's the whole ingest step in Node — fetch clean Markdown, chunk it, embed it:

const runResp = await fetch(
  "https://api.apify.com/v2/acts/runlayer~website-to-markdown/run-sync-get-dataset-items?token=" + process.env.APIFY_TOKEN,
  {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({
      urls: ["https://docs.your-product.com"],
      crawl: true,
      useSitemap: true,
      maxPages: 50,
    }),
  }
);
const pages = await runResp.json();

// Chunk each page's markdown and embed
for (const page of pages) {
  const chunks = chunkMarkdown(page.markdown, { maxTokens: 500 });
  for (const chunk of chunks) {
    const embedding = await embed(chunk);
    await vectorStore.upsert({
      text: chunk,
      embedding,
      metadata: { url: page.url, title: page.title },
    });
  }
}

Because the output is clean Markdown, your chunker splits on real headings
instead of guessing, and your citations point at real page titles and URLs.

For big crawls (hundreds of pages), use the async endpoint (POST /v2/acts/…/runs)
and poll the run, or run it on a Schedule — the run-sync-* endpoint used
above is best for a handful of pages, since it caps at ~5 minutes.

Don't forget PDFs and Word docs

Half of most companies' knowledge lives in PDFs and .docx files, not web
pages. The companion PDF & DOCX to
Markdown actor handles those
with the same output shape, so your ingest code stays uniform:

{ "fileUrls": ["https://example.com/whitepaper.pdf"], "outputFormat": "markdown" }

Keep the index fresh

Wrap the run in an Apify Schedule (say, nightly) and re-embed changed pages.
Use the sinceDate pattern on feeds or a content hash to avoid re-embedding
everything.

Cost

It's pay-per-page with no subscription and no startup fee — you're only billed
for pages that extract successfully, and free-tier Apify credits cover a
generous amount of testing. For a few hundred docs pages that's cents, versus
the ongoing cost of running and babysitting your own browser fleet.

---

If you try it, I'd genuinely like feedback — what breaks, what's missing. Open
an issue on the actor and it gets fixed fast. There's a whole suite of these
(screenshots, SEO/Core-Web-Vitals audits, tech-stack detection, feeds, job APIs)
at apify.com/runlayer if they're useful to you.