How I Built an API to Strip HTML Noise & Clean PDFs for RAG Pipelines

Oladapo Salami — Wed, 28 Oct 2026 12:51:16 +0000

When building RAG (Retrieval-Augmented Generation) applications or LLM agents, document ingestion is almost always the biggest bottleneck.

Standard web scrapers and PDF loaders constantly pull in non-essential content: header navigation, footer links, cookie banners, ads, and broken DOM trees. When fed directly into vector databases, this clutter wastes context window tokens and degrades embedding quality.

To solve this, I spent the last few weeks building Brookiro—a dedicated parsing API that cleans web pages and PDFs into structured JSON or clean Markdown before chunking.

The Core Ingestion Problem

When chunking raw HTML or poorly parsed PDFs using basic loaders, two main issues break retrieval:

Token Bloat: Blog posts or documentation pages often contain 50%+ irrelevant boilerplate (script tags, sidebar nav, promotional banners).
Table & Structural Loss: Standard PDF parsers frequently flatten multi-column layouts and tables into unstructured sentences, causing text splitters to cut right through tabular data.

How Brookiro Works

Instead of running custom BeautifulSoup scripts or heavy local PDF engines for every project, Brookiro exposes a single endpoint that handles extraction asynchronously.

URL & PDF Parsing: Handles both web URLs and uploaded PDF files.
Noise Elimination: Automatically strips non-content DOM elements, ads, and scripts.
Markdown & Structured JSON: Outputs semantically clean Markdown (preserving # headers, lists, and tables) or typed JSON extractions.

Backend Architecture

Here is the tech stack powering the backend:

Framework: Python & Django REST Framework for API endpoint handling and parsing logic.
Authentication & Rate Limiting: Integrated with Unkey for edge-based API key management.
Infrastructure: Deployed on Render.

Code Example: Integrating with Python & RAG Workflows

You can drop a request straight into your Python ingestion script:

import requests

url = "[https://brookiro.com/api/v1/extract](https://brookiro.com/api/v1/extract)"
headers = {
    "Authorization": "Bearer YOUR_API_KEY",
    "Content-Type": "application/json"
}
payload = {
    "url": "[https://example.com/target-page](https://example.com/target-page)"
}

response = requests.post(url, json=payload, headers=headers)
data = response.json()

# Access clean markdown ready for RecursiveCharacterTextSplitter
clean_markdown = data.get("markdown_content")
print(clean_markdown)

⁠

Try It Out

I set up a free tier with 100 requests every month (no credit card required) so developers can benchmark the extraction quality on their own data pipelines.