Turn Any Website into Clean Markdown for Your LLM (RAG Pipeline)

#rag #llm #webscraping #ai

Feeding an LLM raw HTML is how you burn tokens on nav bars and cookie banners. The correct move: crawl the site → clean markdown → embed. Here's the pipeline with one Apify actor.

🤖 Try it live — chat with a Poe bot that generates the exact scraping pipeline for your use case: Lead List Generator · Job Postings Miner · Amazon Product Miner · SERP Rank Tracker · Social Lead Finder

Why markdown for RAG

~5-10x fewer tokens than HTML
Preserves structure (headings, lists, code) that embeddings need
Docs sites, help centers, blogs → ready for your vector DB

The stack

Apify — apify/website-content-crawler: full-site crawl → markdown/text. Free credits included.
Your vector DB / embedding pipeline downstream.

Step 1 — Token

Create an Apify account → Settings → Integrations → export APIFY_TOKEN=...

Step 2 — Crawl a docs site

curl -X POST "https://api.apify.com/v2/acts/apify~website-content-crawler/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "startUrls": [{"url": "https://docs.example.com"}],
    "maxCrawlDepth": 2,
    "outputFormats": ["markdown"]
  }'

Each dataset item: url, title, markdown, text — chunked-ready.

Step 3 — Straight to embeddings

from apify_client import ApifyClient
from openai import OpenAI

apify = ApifyClient("APIFY_TOKEN")
oai = OpenAI()

run = apify.actor("apify/website-content-crawler").call(run_input={
    "startUrls": [{"url": "https://docs.stripe.com"}],
    "maxCrawlDepth": 1,
    "outputFormats": ["markdown"],
})

for page in apify.dataset(run["defaultDatasetId"]).iterate_items():
    md = page.get("markdown", "")
    if len(md) < 200:
        continue
    emb = oai.embeddings.create(input=md[:8000], model="text-embedding-3-small")
    # -> upsert (page["url"], emb.data[0].embedding, md) into your vector DB
    print("embedded:", page["url"], len(md), "chars")

Decision guide

Docs/blog for LLM → website-content-crawler (this pipeline)
Structured data extraction → apify/cheerio-scraper or store actor for that site
Cloudflare-heavy → apify/puppeteer-scraper

Cost caps

maxCrawlDepth + maxPagesPerCrawl + maxConcurrency — test on 10-20 pages before the full crawl. A docs site (100 pages) costs cents.

More extraction tools: awesome-web-scraping. If you code with AI agents: npx skills add g4dr/apify-skills — the apify-web-scraper skill runs this pipeline from a prompt (repo).

What site are you feeding your LLM?

DEV Community