Raw HTML vs. full-page text vs. main-content Markdown — median, range, and the cases where the saving is small.
If you build RAG pipelines or agents that read the web, every page you fetch ends up as input tokens: in the embedding step, in the context window, on every query. So what you hand the model matters. We measured it on 20 real pages instead of guessing.
TL;DR
- The median raw HTML page in our sample was ~128,000 tokens. One page would fill a 128k context window.
- LLM-oriented Markdown of the same pages needed a median 98 % fewer tokens than raw HTML and 63 % fewer than the full page converted to text.
- Most of that second saving comes from main-content extraction (dropping navigation, footers, cookie notices, teasers): median 51 % on its own. Cleanup on top added a median ~9 %.
- The range is wide: 18 % to 90 % against full-page text. Pages that are already lean save little.
Setup
- 20 public pages, German-language sites, picked to mix two kinds: 6 start/overview pages (lots of menus and teasers) and 14 articles (news, encyclopedia, documentation, statute text, recipes, guides). Sources include Wikipedia, MDN, tagesschau.de, heise online, the German Environment Agency, gesetze-im-internet.de, Chefkoch, DHL, Siemens, Verbraucherzentrale, n-tv, t-online and MSN.
- Each page was fetched fresh (no cache) and saved in four variants:
- Raw HTML — the rendered page source.
- Full-page text — the whole page converted to Markdown, menus and footer included. This is roughly what a plain "fetch this URL" tool hands a model.
- Main-content Markdown — only the main content.
- LLM Markdown — main content, further cleaned for model input, plus a short source header.
-
Tokenizer:
o200k_baseviatiktoken(the GPT-4o tokenizer). Other tokenizers give different absolute counts; the ratios move much less. - Statistic: median of the per-page savings, so a handful of huge portal pages can't drive the result.
- Measured on 1 October 2026. Pages change; a re-run will give somewhat different numbers.
Results
| Total tokens, 20 pages | Median page | |
|---|---|---|
| Raw HTML | 3,713,862 | 128,051 |
| Full-page text | 242,852 | 10,917 |
| Main-content Markdown | 109,929 | 3,241 |
| LLM Markdown | 94,987 | 2,948 |
Median saving of LLM Markdown:
| compared with | all 20 pages | start pages (6) | articles (14) |
|---|---|---|---|
| raw HTML | 98 % | 99 % | 98 % |
| full-page text | 63 % | 64 % | 59 % |
Per page, sorted by saving against full-page text
| Page | Kind | Raw HTML | Full-page text | LLM Markdown | vs. HTML | vs. full text |
|---|---|---|---|---|---|---|
| Verbraucherzentrale: guide | article | 404,544 | 5,952 | 4,902 | 98.8 % | 17.6 % |
| German copyright act § 44b | article | 1,265 | 480 | 355 | 71.9 % | 26.0 % |
| nord24: heat pump article | article | 27,875 | 7,261 | 5,365 | 80.8 % | 26.1 % |
| Verbraucherzentrale: start page | start | 562,566 | 5,573 | 3,608 | 99.4 % | 35.3 % |
| Wikipedia: Big Data | article | 94,483 | 25,259 | 15,713 | 83.4 % | 37.8 % |
| heise online: start page | start | 359,135 | 29,770 | 16,135 | 95.5 % | 45.8 % |
| Environment Agency: emissions | article | 121,354 | 20,064 | 10,354 | 91.5 % | 48.4 % |
| Chefkoch: recipe | article | 140,381 | 4,831 | 2,459 | 98.2 % | 49.1 % |
| n-tv: heat pump guide | article | 119,333 | 18,372 | 8,648 | 92.8 % | 52.9 % |
| tagesschau.de: business | start | 478,580 | 12,501 | 5,072 | 98.9 % | 59.4 % |
| heise online: news item | article | 91,736 | 5,404 | 1,858 | 98.0 % | 65.6 % |
| Siemens: start page | start | 134,748 | 6,449 | 2,059 | 98.5 % | 68.1 % |
| tagesschau.de: fuel prices article | article | 261,767 | 7,339 | 1,960 | 99.3 % | 73.3 % |
| MDN: HTML docs | article | 56,651 | 12,837 | 3,280 | 94.2 % | 74.4 % |
| t-online: heat pump article | article | 150,606 | 10,486 | 2,616 | 98.3 % | 75.1 % |
| MSN: heat pump article | article | 244,653 | 20,174 | 4,347 | 98.2 % | 78.5 % |
| DHL: private customers | start | 59,085 | 11,416 | 2,155 | 96.4 % | 81.1 % |
| Environment Agency: climate change | article | 99,713 | 11,347 | 1,337 | 98.7 % | 88.2 % |
| DHL: online postage | article | 114,545 | 10,319 | 1,119 | 99.0 % | 89.2 % |
| Chefkoch: start page | start | 190,842 | 17,018 | 1,645 | 99.1 % | 90.3 % |
What we take from it
1. Never send raw HTML to a model. Even the leanest page in the set (a statute text, 1,265 tokens of HTML) lost 72 %. Typical pages are 100k–500k tokens of HTML for a few thousand tokens of content — mostly scripts, inline styles, tracking and markup.
2. "Full page as text" is not lean either. Converting HTML to text removes the markup but keeps every menu, footer link, cookie notice and teaser. On the median page that is still more than three times the tokens of the actual content.
3. Main-content extraction is the big lever. Dropping boilerplate saved a median 51 % against full-page text; extra cleanup for LLM input added a median ~9 %. This is the honest part: much of the gain is available from any decent main-content extractor. Whichever tool you use, check that it actually does this, and spot-check its output — an extractor that cuts too much is worse than one that cuts too little.
4. The saving depends on the page. Portal and overview pages save the most; long, clean articles the least (Wikipedia: 38 %; a consumer guide: 18 %). A single median number hides that; plan with the range.
5. Fewer tokens is not only cheaper. Less noise in the context means less for the model to wade through, and less irrelevant text in your vector index competing with the passages you want to retrieve.
A rough cost sense
On the median page, full-page text is ~10,900 tokens and LLM Markdown ~2,900: about 8,000 input tokens saved per page and pass. At an illustrative $1 per million input tokens, that is about $8 per 1,000 pages per pass — and RAG pipelines pay it at embedding time, while agents pay it again every time the page lands in a context window. Plug in your own model price.
Limitations
- 20 pages is a small sample, and all are German-language sites. English sites may differ.
- Tokenizer-specific absolute numbers (
o200k_base). - Pages change; this is a snapshot from 1 October 2026.
- "Full-page text" here is our own whole-page Markdown conversion, not any specific third-party tool.
- We did not benchmark other extraction services or libraries. The comparison is between page representations, not between products.
- Token count is not answer quality. We measured retrieval separately (headings-based chunks vs. fixed-length chunks, 58 questions, one embedding model) — that is a different, smaller study.
Disclosure and method
I'm the founder of Ragfeed, a Firecrawl-compatible web-data API hosted in Germany; the four variants above were produced with it, and "LLM Markdown" is its llmMarkdown format. The script, page list and raw numbers are public at https://github.com/LeifAlbrand/ragfeed-beispiele, so you can rerun the comparison yourself — including against another Firecrawl-compatible API (--ohne-llm). If you want to try it on your own pages, there is a free tier (1,000 pages a month): https://ragfeed.de
Top comments (0)