DEV Community

Cover image for Turn any docs site into RAG-ready Markdown (without writing a scraper)
Mohamed Senator
Mohamed Senator

Posted on

Turn any docs site into RAG-ready Markdown (without writing a scraper)

#rag #ai #python #webscraping

Every RAG project I have seen starts the same way: someone wants a chatbot that answers questions from a documentation site, and the first week disappears into scraping. Menus end up in your embeddings, code blocks lose their language, JavaScript-rendered pages come back empty, and chunks split in the middle of a sentence.

I got tired of redoing this, so I built a small tool for it: Website to Markdown Crawler for LLMs & RAG on the Apify Store. Disclosure: it is my own Actor. This post shows the approach, the exact input I use and how to load the result into a vector store. The ideas apply to any scraper.

What "RAG-ready" actually means

Raw HTML is the wrong input for an embedding model. What you want per page is:

  1. Only the main content. No nav bars, cookie banners, footers, share buttons or sidebars.
  2. Structure kept as Markdown. Headings, lists, tables and fenced code blocks with their language, so an LLM can read it and you can chunk on headings.
  3. Chunks that respect headings. Each chunk should know which section it came from (for example First crawler > Building a crawler), stay under a size limit, and overlap a little with the previous chunk so context isn't lost at the boundary.
  4. Metadata. Title, description, canonical URL, language and dates, for filtering and for citing sources in answers.

The three problems that cost the most time

JavaScript-rendered pages. Many docs sites are React or Next.js apps, and a plain HTTP fetch returns an almost empty shell. The cheap fix is to fetch with plain HTTP first, check whether the extracted content looks missing, and only then open the page in a real browser. Rendering every page in a browser works too, but it is much slower.

Finding every page. Following links works, but a sitemap is usually faster and more complete. Some sites have no usable sitemap, so you need a fallback to link-following.

Chunking. Splitting every N characters cuts code blocks and sentences in half. Splitting on headings first and then packing up to a token limit keeps each chunk coherent.

The Actor handles all three. renderJavaScript: "auto" (the default) does the HTTP-first, browser-only-when-needed fetch; crawlMode: "sitemap" falls back to following links when a site has no sitemap; and chunkContent: true adds heading-aware chunks.

Step by step: one docs section into chunks

1. Pick the start URL and a filter

Use the docs homepage as the start URL and urlMustContain to stay inside the section you care about. Start URLs are always extracted; crawled or sitemap URLs must contain the filter text.

2. Choose a crawl mode

  • none (default): only the URLs you list.
  • sameDomain: also follow links on the same site, up to maxCrawlDepth (default 2).
  • sitemap: use the site's sitemap. Usually the fastest way to get a whole docs site.

3. Turn on chunking and set a page cap

This is the input I use:

{
  "urls": ["https://docs.apify.com"],
  "crawlMode": "sitemap",
  "urlMustContain": "docs.apify.com/actors/",
  "maxPages": 200,
  "chunkContent": true,
  "chunkSizeTokens": 800,
  "chunkOverlapTokens": 100
}
Enter fullscreen mode Exit fullscreen mode

Other useful options: outputFormat (markdown, text or both), removeBoilerplate (on by default), includeLinks, includeTables and includeMetadata (all on by default), renderJavaScript (auto, never or always) and maxConcurrency (default 5).

4. Run it from code

With the Python client (pip install apify-client):

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

run = client.actor("s3nafps/ai-ready-webpage-extractor").call(run_input={
    "urls": ["https://docs.apify.com"],
    "crawlMode": "sitemap",
    "urlMustContain": "docs.apify.com/actors/",
    "maxPages": 50,
    "chunkContent": True,
    "chunkSizeTokens": 800,
    "chunkOverlapTokens": 100,
})

pages = list(client.dataset(run["defaultDatasetId"]).iterate_items())
print(len(pages), "pages")
for page in pages[:5]:
    print(page["title"], page["estimatedTokens"], len(page["chunks"]))
Enter fullscreen mode Exit fullscreen mode

Or in JavaScript (npm install apify-client):

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('s3nafps/ai-ready-webpage-extractor').call({
  urls: ['https://docs.apify.com'],
  crawlMode: 'sitemap',
  urlMustContain: 'docs.apify.com/actors/',
  maxPages: 50,
  chunkContent: true,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items.length, 'pages');
Enter fullscreen mode Exit fullscreen mode

5. What comes back

One dataset item per page. Here is a real item from a crawl of the Crawlee docs, shortened. The chunk starts with text from the end of the previous chunk; that is the overlap.

{
  "sourceUrl": "https://crawlee.dev/js/docs/introduction/first-crawler",
  "title": "First crawler | Crawlee for JavaScript · Build reliable crawlers. Fast.",
  "description": "Your first steps into the world of scraping with Crawlee",
  "canonicalUrl": "https://crawlee.dev/js/docs/introduction/first-crawler",
  "language": "en",
  "mainContentMarkdown": "Version: 3.18\n\n# First crawler\n\nNow, you will build your first crawler...",
  "chunks": [
    {
      "index": 4,
      "headingPath": ["First crawler", "Building a crawler", "Building a CheerioCrawler"],
      "content": "Bulk add requests\n\nThe code above is for illustration of the request queue concept...",
      "estimatedTokens": 647
    }
  ],
  "headings": [{ "level": 1, "text": "First crawler" }],
  "wordCount": 1219,
  "estimatedTokens": 2236,
  "renderedWithBrowser": false,
  "crawlDepth": 1,
  "extractionStatus": "succeeded"
}
Enter fullscreen mode Exit fullscreen mode

Each item also has finalUrl, httpStatus, author, publishedDate, modifiedDate, links, tables, images, schemaOrgData and scrapedAt. The dataset has two views: Overview (one row per page) and RAG chunks (one row per chunk). Failed pages and their reasons go into the OUTPUT record in the run's key-value store.

That Crawlee crawl (sameDomain mode, urlMustContain: "/js/docs/", 25 pages) is also a good warning: 9 of the 25 pages were copies of the Quick Start page under different version folders (/docs/3.11/, /docs/3.12/ and so on). Check your first run for versioned duplicates and tighten urlMustContain before crawling more.

6. Embed the chunks

From here, embedding is a loop over chunks, using headingPath and sourceUrl as metadata:

for page in pages:
    for chunk in page["chunks"]:
        vector_store.add(
            text=chunk["content"],
            metadata={
                "url": page["canonicalUrl"] or page["sourceUrl"],
                "title": page["title"],
                "section": " > ".join(h.replace("\u200b", "") for h in chunk["headingPath"]),
            },
        )
Enter fullscreen mode Exit fullscreen mode

The replace("\u200b", "") is there because some docs themes put an invisible zero-width space in heading anchors, and it ends up in the heading text.

If you use LangChain, ApifyDatasetLoader from langchain_community.document_loaders can load the dataset directly; map mainContentMarkdown (or each chunk) to a Document.

Pricing

Pay per page, no subscription. Prices I checked today on the Store:

  • $0.003 per extracted page on the free plan (Bronze $0.0027, Silver $0.0024, Gold and above $0.002).
  • $0.00005 start fee per run (per GB of memory, default 1 GB).
  • Failed pages are not charged. Browser rendering and chunking are included.

So a 100-page docs section costs about $0.30 on the free plan, and 1,000 pages about $3. Set maxPages, and Maximum cost per run in the run options, to cap any run.

Things to know before you use it

  • Public pages only. No logins, cookies, paywalls or CAPTCHA solving.
  • Non-HTML files such as PDFs are skipped.
  • Main-content detection is heuristic. Unusual layouts can include a little extra text, so spot-check a few pages before you embed thousands.
  • Token counts are estimates (about 4 characters per token).

FAQ

Does it work on JavaScript sites?
Yes. With renderJavaScript: "auto", pages are fetched as plain HTML first and opened in a real browser only when the content comes back missing. Each item's renderedWithBrowser field tells you which path was used. For app-like sites that still come back short, set "always".

What if the site has no sitemap?
Sitemap mode falls back to following links from your start URL. You can also use sameDomain mode with maxCrawlDepth.

How do I keep costs predictable?
Set maxPages and a maximum cost per run. The run stops cleanly at the limit, and failed pages are never charged.

Which vector databases does it work with?
Any. The output is plain JSON with text, heading paths and token estimates, so it works with Pinecone, Qdrant, Weaviate, pgvector or a local store.

Can an AI agent call it directly?
Yes, through the Apify MCP server: add https://mcp.apify.com/?actors=s3nafps/ai-ready-webpage-extractor to Claude, Cursor or another MCP client.

Try it

Website to Markdown Crawler for LLMs & RAG on the Apify Store. The input form is prefilled with one URL, so a first test run costs less than a cent.

If you have a site that doesn't extract cleanly, open an issue on the Actor page with the URL. That is the fastest way for me to improve it.

Top comments (0)