DEV Community

Oaida Adrian
Oaida Adrian

Posted on • Originally published at apify.com

Building an AI Web Crawler That Outputs LLM-Ready Content Chunks

#python #webdev #automation

Building an AI Web Crawler That Outputs LLM-Ready Content Chunks

The biggest challenge in building RAG (Retrieval-Augmented Generation) pipelines isn't the vector database or the embeddings — it's getting clean, structured content from web pages in the first place. Raw HTML is noisy, full of navigation, ads, and boilerplate that pollutes your embeddings.

I built an AI Web Crawler that extracts clean, chunked content from any website — ready to feed directly into LLMs and RAG systems.

The Problem with Web Content for AI

When you feed raw HTML to an LLM, you waste tokens on navigation menus, cookie banners, and footer links. When you feed it poorly cleaned text, retrieval quality drops because the chunks don't align with semantic boundaries.

The solution: a crawler that understands document structure and outputs content chunks designed for AI consumption.

Key Features

Smart Content Extraction

The crawler strips boilerplate (nav, footer, scripts, styles) and extracts only the main content — the actual article, documentation, or page text that matters.

Semantic Chunking

Content is split into chunks that respect heading boundaries:

def chunk_content(text, headings, max_chunk=500):
    chunks = []
    current = []
    for heading in headings:
        section = extract_section(text, heading)
        if len(section) > max_chunk:
            chunks.extend(split_long(section, max_chunk))
        else:
            chunks.append(section)
    return chunks
Enter fullscreen mode Exit fullscreen mode

Token Estimation

Every chunk includes an estimated token count, so you know exactly how much context you're working with before sending to your LLM:

tokens = len(chunk_text) // 4  # rough estimate
Enter fullscreen mode Exit fullscreen mode

Structured Data Extraction

Beyond plain text, the crawler captures:

  • Headings — document structure/hierarchy
  • Tables — structured data preserved as markdown
  • Internal/External Links — relationship mapping
  • JSON-LD — structured data from schema.org
  • OpenGraph — social metadata

Use Cases

  • RAG Pipelines: Feed chunks directly to your vector database
  • Knowledge Base Building: Crawl documentation sites systematically
  • Content Monitoring: Track changes on competitor pages
  • Dataset Creation: Build training data from web sources
  • AI Agent Tools: Give your autonomous agents a web reader

Try It

The Apify actor uses pay-per-event pricing — you only pay for pages successfully crawled.

Conclusion

The future of web scraping is AI-ready output. By chunking content semantically, estimating tokens, and preserving structure, we bridge the gap between raw web pages and LLM-ready knowledge.

Built with Python, BeautifulSoup4, httpx, and the Apify platform.

Top comments (0)