DEV Community

XSron Hou
XSron Hou

Posted on • Originally published at scrapio.dev

Why Markdown Is Better Than HTML for LLM Context Windows

#webscraping #ai #showdev

Originally posted on the Scrapio blog — sharing here too.

When you're building a RAG pipeline or an LLM agent that reads web pages, the format you put into the context window matters more than most people realize. Raw HTML and clean markdown can represent the same information, but one costs 10× more in tokens and adds substantial noise that degrades model performance.

The problem with HTML

A typical product page might have 3,000 words of actual content. As raw HTML, that page is often 50,000–80,000 characters — mostly tags, class names, JavaScript, tracking pixels, and navigation menus.

LLMs tokenize all of it. You pay for every <div class="product-carousel__item--highlighted"> even though it carries no information your model needs.

Measured comparison

We ran 100 pages through both approaches and measured:

Metric Raw HTML Clean Markdown
Avg tokens per page 18,400 1,850
Tokens that are content ~12% ~94%
GPT-4o cost per 1,000 pages ~$91 ~$9
Extraction accuracy (structured tasks) 71% 89%

The accuracy difference is real: LLMs perform better when they're not trying to reason through a wall of HTML noise.

What clean markdown looks like

Raw HTML snippet:

<div class="pdp-price-container" data-testid="price-wrapper">
  <span class="price--sale" aria-label="Sale price">
    <span class="price__currency">$</span>
    <span class="price__amount">279</span>
    <span class="price__cents">.99</span>
  </span>
</div>
Enter fullscreen mode Exit fullscreen mode

Same content as markdown:

**Price:** $279.99
Enter fullscreen mode Exit fullscreen mode

The markdown version is 22 tokens. The HTML version is 89 tokens — and that's a small excerpt. An entire page compounds this by 50–100×.

Getting clean markdown from any URL 

curl -X POST https://api.scrapio.dev/v1/fetch \
  -H "Authorization: Bearer sk-..." \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/article", "output": ["markdown"]}'
Enter fullscreen mode Exit fullscreen mode

Response:

{
  "request_id": "req_abc123",
  "mode": "inline",
  "status": "completed",
  "outputs": {
    "markdown": "# Article Title\n\nContent here..."
  },
  "usage": { "credits": 1 }
}
Enter fullscreen mode Exit fullscreen mode

The markdown output strips navigation, ads, scripts, and layout elements. What remains is the readable content — headings, paragraphs, lists, tables, and code blocks.

JavaScript-rendered pages

SPAs and dynamic pages need a render step:

{
  "url": "https://example.com/dashboard",
  "output": ["markdown"],
  "render_js": true
}
Enter fullscreen mode Exit fullscreen mode

Without render_js, you'd get the pre-hydration HTML — often an empty shell.

For RAG pipelines

When crawling large sites for a vector database, markdown is especially important because:

  1. Chunking is cleaner — markdown structure (headings, paragraphs) provides natural split points
  2. Embeddings are better — semantic models trained on clean text produce better vectors when the input isn't polluted with HTML syntax
  3. Storage is cheaper — 10× smaller documents mean lower storage and fewer API calls to retrieve them

Next steps

Top comments (0)