I measured how much of a web page your LLM doesn't need: 79%

If your pipeline fetches a page and hands the HTML straight to a model, you are paying to send <div> soup through a context window. I wanted to know exactly how much, so I measured it instead of guessing.

Three pages, fetched normally, then converted to markdown:

Page	Raw HTML	Markdown	Reduction
Wikipedia — Retrieval-augmented generation	244,577	64,944	3.8x
Python docs — json	111,760	25,634	4.4x
MDN — HTTP Headers	291,827	48,038	6.1x
Total	648,164	138,616	4.7x

78.6% of the characters were discarded. I checked what got dropped: markup, inline styles, tracking attributes and navigation chrome. The prose, the tables and the code examples survived intact.

A note on how I measured

I counted characters, not tokens. Tokenizers differ between models and I didn't want a number that only holds for one vendor. Characters are exact and vendor-neutral. (wc -c will give you byte counts, a few hundred off on these pages wherever UTF-8 uses multiple bytes per character — it doesn't move the ratio.)

If anything, this understates the effect. HTML tokenizes worse than prose: <div class="mw-heading mw-heading2"> becomes many tokens because tokenizers are trained on natural language, not attribute soup. So the token gap is likely wider than the character gap, not narrower. Treat 4.7x as a floor.

Why raw HTML is so expensive

I took two of those pages apart to see what the bytes actually are:

	Markup (tags)	…of which attributes	Visible text
MDN — HTTP Headers	62%	42%	38%
Wikipedia — RAG	82%	64%	18%

On the Wikipedia page, HTML attributes alone outweigh the article text three and a half to one — 156,503 characters of them against 44,030 of prose. That one page carries 985 class attributes, 794 ids, 771 hrefs, 261 data-* hooks and 70 ARIA attributes. All of it billable, none of it readable.

That's before counting the things markup ratio doesn't capture: navigation and footers repeated on every page you crawl (scrape 500 pages of one site and you pay for the same sidebar 500 times), cookie banners, and hydration payloads on framework-heavy sites.

The model has to attend over all of it. You pay on the way in, and you pay again in latency.

Doing it yourself

For a single page you don't need a service. Two well-known libraries do it. In Node:

npm install @mozilla/readability jsdom turndown

import { Readability } from "@mozilla/readability";
import { JSDOM } from "jsdom";
import TurndownService from "turndown";

const html = await fetch(url).then((r) => r.text());

const dom = new JSDOM(html, { url });
const article = new Readability(dom.window.document).parse();

const markdown = new TurndownService({ codeBlockStyle: "fenced" })
  .turndown(article.content);

Readability finds the main content, Turndown converts it. I ran exactly that snippet over the same three pages:

Page	Raw HTML	Markdown	Reduction
Wikipedia — RAG	244,577	24,294	10.1x
Python docs — json	111,760	34,405	3.2x
MDN — HTTP Headers	291,827	53,507	5.5x
Total	648,164	112,206	5.8x

5.8x — better than the 4.7x I opened with. Ten lines of code beating a service is a fine result, and if that's all you need, take it and stop reading.

But look at the Wikipedia row: 10.1x. That should make you suspicious rather than pleased.

Compression ratio is a trap

I went back and checked what got dropped, and this is the part I'd want someone to tell me before I shipped a pipeline:

On the Wikipedia page, Readability's extra compression came from silently dropping the Evaluation section, See also, and the whole reference list. Those aren't chrome. If your question happened to be about evaluation methods, your RAG now answers from a page that no longer contains the answer — and nothing in the pipeline reports a problem.

On the Python docs page it's worse, and more subtle:

new TurndownService({ codeBlockStyle: "fenced" })
  .turndown("<pre>x = 1\ny = 2</pre>")
// => "x = 1\ny = 2"    ← no fence. It's just prose now.

Turndown's fenced-code rule only fires on <pre><code>. A bare <pre> becomes plain text. And Sphinx — which generates docs.python.org and a large share of the Python ecosystem's documentation — emits bare <pre>: 15 of 15 blocks on that page, none with a <code> child.

So the output contains the code, unfenced, indented like a paragraph. The model can no longer tell the example from the sentence describing it. Nothing errors. The markdown is valid. codeBlockStyle: "fenced" was set the whole time and did nothing, because the rule it configures never matched.

Our own extractor had its own version of this, found while writing this post: it was silently replacing code blocks past the 10th one on a page with a copy of an earlier block. Valid markdown, wrong code, no error. Fixed now — but the lesson is the same one, and I'd rather show it than pretend our pipeline was born clean.

The metric to watch isn't how much you removed, it's how much of the content you kept. Count code blocks in and out. Count headings in and out. A pipeline that compresses 10x by dropping a third of the page is worse than one that compresses 4x and keeps everything.

The other edges

Client-rendered pages. On a client-rendered SPA, fetch returns an empty shell and Readability has nothing to find. You need a real browser, which means Playwright, which means you now maintain browser infrastructure. (Server-rendered frameworks are fine — the trap is that you can't tell which is which without checking.)

Everything about doing it repeatedly. Rate limits, retries with backoff, encoding detection, redirect loops, PDFs, and honoring robots.txt and the site's terms — which you should, both because it's right and because it's what keeps you unblocked.

Roughly: one page is an afternoon. A thousand pages a day, reliably, is a system.

When you'd rather not run that system

That's the part we packaged. For the record, on the Python docs page our extractor returns 15 fenced blocks out of 15 — that's the 4.4x row in the first table, and the reason it compresses less than the ten-line version compresses. Keeping things costs bytes.

One call, markdown back:

curl -X POST https://api.messora.dev/scrape \
  -H "X-API-Key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com","formats":["markdown"]}'

{
  "success": true,
  "scrape_status": "success",
  "markdown": "# Example Domain\n\nThis domain is for use in documentation examples...",
  "credits_used": 1,
  "remaining_credits": 4996
}

formats also takes json (schema-guided extraction) and raw if you want the original HTML alongside.

One design note, since it bit us in production and might be useful to steal: scrape_status is a separate field from success, with values like blocked_antibot, timeout and extraction_failed. A 200 response that returns an empty page is not the same failure as a timeout, and collapsing both into a boolean makes it impossible to decide whether retrying is worth it. Credits are only charged on success.

There's a playground with no signup, no card — paste a URL and see the markdown: messora.dev

The takeaway

Two things, and the second one matters more than the first.

Stop sending raw HTML to your model. Four to six times less input, for bytes that were never content — 82% of that Wikipedia page was markup. Ten lines of Readability and Turndown will get you there. That's about as cheap as a win gets in a RAG pipeline.

Then go check what your converter threw away. Both extractors I measured lost content silently: theirs dropped whole sections and un-fenced every code block on a Sphinx page; ours duplicated blocks past the tenth. Neither raised an error, and both produced markdown that reads fine until the answer it feeds is quietly wrong.

Count what goes in and what comes out — blocks, headings, sections. It's twenty lines of assertions, and it's the difference between a pipeline that's cheap and a pipeline that's cheap and correct.

---

Measured on 2026-08-21. Character counts on the pages as served, each one checked against the site's robots.txt first. Your numbers will differ by page; the method won't.

Comments

[Roberto Francisco junior]

Curious if others have hit this with Turndown/Readability on Sphinx-generated docs, or is this a known gotcha I just hadn't run into before?