DEV Community

Tùng Nguyễn
Tùng Nguyễn

Posted on Edited on

Turning Any Website into Clean LLM-Ready Markdown for AI RAG Pipelines in Python

#ai #opensource #python #rag

When building Retrieval-Augmented Generation (RAG) pipelines or AI agents with LangChain, LlamaIndex, or OpenAI / Claude, feeding raw HTML directly into an embedding model wastes up to 78% of your context window on <nav>, <script>, SVG icons, cookie banners, and inline styles.

To get high-precision vector embeddings and clean chunking, you need a deterministic Website-to-Markdown Crawler that strips layout boilerplate while preserving heading hierarchies (H1–H6), code blocks, tables, and YAML metadata.

âš¡ Try the Live Cloud Actor or Open-Source Repo

🛠️ Why Standard HTML-to-Text Breaks RAG Chunking

If you simply call soup.get_text(), your headings, bullet lists, and code blocks flatten into an unstructured wall of text. Semantic chunkers (MarkdownHeaderTextSplitter in LangChain) rely on clean #, ##, and ### boundaries to keep related paragraphs together.

Python Implementation: Boilerplate Removal + Markdown Conversion 

import re
import requests
from bs4 import BeautifulSoup

NOISE_SELECTORS = [
    'script', 'style', 'noscript', 'svg', 'iframe',
    'nav', 'footer', 'header', 'aside',
    '.cookie-banner', '.newsletter', '.sidebar', '#comments'
]

def html_to_rag_markdown(url: str) -> dict:
    resp = requests.get(url, timeout=15, headers={'User-Agent': 'Mozilla/5.0'})
    soup = BeautifulSoup(resp.text, 'html.parser')

    title = soup.title.string.strip() if soup.title and soup.title.string else url
    for sel in NOISE_SELECTORS:
        for tag in soup.select(sel):
            tag.decompose()

    main = soup.find('main') or soup.find('article') or soup.body or soup
    lines = [
        "---",
        f'title: "{title}"',
        f'source_url: "{url}"',
        "---\n"
    ]

    for el in main.find_all(['h1', 'h2', 'h3', 'h4', 'p', 'pre', 'li']):
        text = el.get_text(" ", strip=True)
        if not text:
            continue
        if el.name.startswith('h'):
            level = int(el.name[1])
            lines.append(f"\n{'#' * level} {text}\n")
        elif el.name == 'pre':
            lines.append(f"\n```
{% endraw %}
\n{el.get_text()}\n
{% raw %}
```\n")
        elif el.name == 'li':
            lines.append(f"- {text}")
        else:
            lines.append(text + "\n")

    markdown = re.sub(r'\n{3,}', '\n\n', "\n".join(lines)).strip()
    word_count = len(markdown.split())
    return {
        "url": url,
        "title": title,
        "wordCount": word_count,
        "estimatedTokens": int(word_count * 1.33),
        "markdown": markdown,
    }
Enter fullscreen mode Exit fullscreen mode

☁️ Calling the Cloud API Directly in Python

With our Apify Cloud Actor, you can convert hundreds of documentation pages or knowledge bases into LLM-ready Markdown via a single REST call and pipe the JSON dataset directly into LangChain or Pinecone:

👉 Launch Website to Markdown & Clean Text Scraper for AI, RAG & LLMs on Apify Store

🌐 Live REST APIs on RapidAPI Hub & Instant Download Store

Top comments (0)