DEV Community

Get Anything
Get Anything

Posted on

Turn any website into clean Markdown for your RAG pipeline (no boilerplate, with token counts)

#rag #llm #python #webscraping

If you've built a RAG pipeline, you know the boring part isn't the embeddings. It's getting clean text out of web pages. Raw HTML is full of nav bars, cookie banners, footers and "related posts", and all of it ends up in your vector database and in your prompts.

Here's the setup I use now: give a list of URLs (or one docs site) to a scraper that returns only the main content as Markdown, plus metadata and an approximate token count per page. Then chunk and embed.

What you get back

I ran it on two pages: the Wikipedia article on RAG and the Apify docs page for Actors. This is the real output (Markdown trimmed):

{
  "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
  "title": "Retrieval-augmented generation - Wikipedia",
  "date": "2023-11-05",
  "sitename": "Wikimedia Foundation, Inc.",
  "wordCount": 225,
  "tokensApprox": 375,
  "method": "http",
  "success": true,
  "markdown": "---\ntitle: Retrieval-augmented generation - Wikipedia\n...\n---\n**Retrieval-augmented generation** (**RAG**) is a technique that ..."
}
Enter fullscreen mode Exit fullscreen mode

A few things that matter for RAG:

  • The Markdown starts with a small front-matter block (title, url, date, sitename). Keep it on the first chunk so every answer can cite its source.
  • tokensApprox (about 4 characters per token) lets you budget before you embed or stuff a context window.
  • method tells you whether the page came over plain HTTP (fast, cheap) or needed the browser fallback. JS-heavy and bot-protected pages are rendered with a hardened browser automatically.
  • maxCharsPerPage truncates pages if you only want the first N characters (I used 1,500 for this demo).

Run it from Python

The scraper is an Apify Actor, so you can call it with the official client and get the dataset back in one call:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run = client.actor("get_anything/llm-web-scraper").call(run_input={
    "startUrls": [
        {"url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"},
        {"url": "https://docs.apify.com/platform/actors"},
    ],
    "outputFormat": "markdown",   # or "text" / "both"
    "includeLinks": False,        # cleaner chunks for embeddings
    "useBrowser": "auto",         # browser only when HTTP isn't enough
})

pages = list(client.dataset(run["defaultDatasetId"]).iterate_items())
for p in pages:
    print(p["title"], p["tokensApprox"], "tokens")
Enter fullscreen mode Exit fullscreen mode

Chunk it for a vector DB

Markdown headings make natural chunk boundaries. A minimal splitter:

import re

def chunk_markdown(page, max_chars=2000):
    parts = re.split(r"\n(?=#{1,3} )", page["markdown"])   # split before H1-H3
    chunks, buf = [], ""
    for part in parts:
        if len(buf) + len(part) > max_chars and buf:
            chunks.append(buf)
            buf = ""
        buf += part + "\n"
    if buf:
        chunks.append(buf)
    return [{"text": c, "url": page["url"], "title": page["title"]} for c in chunks]

docs = [c for p in pages if p["success"] for c in chunk_markdown(p)]
# -> embed `text`, store `url` + `title` as metadata (pgvector, Qdrant, Pinecone, Weaviate...)
Enter fullscreen mode Exit fullscreen mode

Ingest a whole docs site

Set crawl: true and it follows same-domain links from each start URL, with caps so a run can't run away:

{
  "startUrls": [{ "url": "https://docs.example.com/" }],
  "crawl": true,
  "maxPages": 200,
  "maxDepth": 3,
  "outputFormat": "markdown"
}
Enter fullscreen mode Exit fullscreen mode

That gives you one dataset row per page, ready to chunk the same way. It's an easy way to give an assistant a product's documentation.

Use it straight from Claude or ChatGPT (MCP)

You don't have to write code at all. Through the Apify MCP server, Claude, ChatGPT, Cursor or any MCP client can call the scraper as a tool:

{ "mcpServers": { "apify": { "url": "https://mcp.apify.com/?tools=get_anything/llm-web-scraper" } } }
Enter fullscreen mode Exit fullscreen mode

Then just ask: "Read these three URLs and summarise what each says about pricing." Step-by-step setup: Connect any Apify scraper to Claude or ChatGPT in 2 minutes.

Cost

You pay per page converted (see the Pricing tab on the Actor page). There's no subscription, and the Apify free plan's monthly credit covers plenty of testing.

Try it: LLM Web Scraper on Apify. If you need the search step too (query → top results → their pages as Markdown), I wrote that up separately: the AI Web Search Scraper.

Questions or a site it doesn't handle well? Drop a comment, I read them all.

Top comments (0)