If you're feeding web content into an LLM — for RAG, for an agent's context window, for a summarization pipeline — raw HTML is the wrong input. This post covers why, and the specific extraction problems you'll hit once you try to fix it yourself.
Why raw HTML wastes your context window (and your accuracy)
A typical webpage's HTML is 70-90% markup, navigation, ads, cookie banners, and scripts that have nothing to do with the actual content. Dump that into an LLM prompt and you're paying token cost for <nav> boilerplate, burning context budget that could hold more actual retrieved content, and — worse — sometimes confusing the model with off-topic sidebar links and "related articles" widgets that get treated as part of the main content.
The fix is converting the page to clean Markdown: headings, paragraphs, lists, and links, with the chrome stripped out. This is a solved problem in principle (Readability-style content extraction has existed for years), but there are a few specific failure modes worth knowing about before you build it yourself.
Problem 1: content-extraction heuristics fail on modern SPA sites
Libraries like readability-lxml or newspaper3k work by looking at the static HTML and guessing which <div> holds the "main content" based on text density heuristics. That works fine for a WordPress blog. It fails completely on a React/Next.js/Vue app where the initial HTML response is a nearly empty shell (<div id="root"></div>) and the actual content only exists after client-side JavaScript runs.
If you're scraping server-side (no headless browser), you need to either detect this case and expand your fetch — reading more bytes to catch server-side-rendered (SSR) content that a naive byte-limited fetch would truncate — or fall back to a headless browser, which is much slower and heavier. A reasonable middle ground: detect known SPA framework signatures (React/Next.js/Vue/Angular/Svelte hydration markers) in the first chunk of HTML and adaptively increase how much you read before giving up, rather than always paying headless-browser cost.
Problem 2: "clean" shouldn't mean "lossy"
A lot of Markdown converters strip links entirely, or collapse them to plain text — which is bad for RAG, because the source URL of a claim is often exactly the citation info you want preserved for the agent to reference later. You want a converter that keeps [text](url) link structure intact, keeps heading hierarchy (so a model can tell "this was an H2 under this H1"), and drops navigation/footer/ad content without dropping legitimate body links.
Problem 3: word count and reading time aren't cosmetic — they're routing signals
If you're chunking content for a RAG pipeline, knowing the Markdown's word count before you decide how to chunk it (single chunk vs. split by heading vs. split by fixed token windows) saves a wasted round trip. Most raw-HTML-to-Markdown tools don't return this — you end up computing it yourself downstream, which is fine, but it's one more thing to build.
A minimal example of what "good" output looks like
{
"markdown_content": "# Article Title\n\nActual paragraph content with [preserved links](https://example.com) and structure...\n\n## A subheading\n\nMore content...",
"word_count": 842,
"reading_time_minutes": 4
}
Versus what you get from naive HTML stripping:
Home About Contact Subscribe Article Title Actual paragraph content with preserved
links and structure More content Copyright 2026 Privacy Policy Terms
The difference matters more than it looks like on paper — the second version has already lost the information a downstream LLM needs to understand document structure, and it's mixed navigation text directly into what looks like body content.
Combining this with metadata extraction in one call
If your pipeline needs more than just the Markdown — OpenGraph title/description for a citation card, the page's detected tech stack for competitive research, contact info for lead enrichment — doing all of that as separate scraping steps means separate fetches, separate parsing logic, and separate places for SSRF bugs to creep in (see my post on SSRF-safe URL fetching for why that matters).
This is the exact gap the Web Metadata & Contact Extractor API is built for: one GET request returns clean, link-preserving Markdown (with word count and reading time) alongside SEO/OpenGraph metadata, contacts, social links, and a 40+ signature tech-stack detector — so a RAG ingestion step doesn't need five different tools glued together. It handles the SPA-detection problem with an adaptive byte-limit fetch, and every fetch goes through the same SSRF/DNS-rebinding-safe layer regardless of what data you're asking for.
It's MIT-licensed and open source if you want to see exactly how the Markdown conversion and SPA detection work, or try the live demo with any URL, no signup required.
Top comments (0)