DEV Community

agebreakco-prog
agebreakco-prog

Posted on

Turning any website into clean Markdown chunks for RAG, without a headless browser

AI disclosure: I wrote this article with AI assistance (Claude). The code comes from a small crawler we maintain, and I got every number in the output sections by running it locally on 2026-11-22.

Most RAG pipelines start with "get the docs into the vector store", and that step often turns out worse than it should. Raw HTML is full of navigation menus, cookie banners, "Edit this page" links and footers. If you embed all of that, every chunk carries the same boilerplate, retrieval gets noisy, and you pay for tokens that add nothing.

People often reach for a headless browser first. For a large share of the web, including most documentation sites, blogs and help centers, you don't need one. The HTML the server sends already contains the content. A plain fetch, an HTML parser and an HTML-to-Markdown converter will do, and the result is faster and cheaper, with fewer moving parts.

This post walks through the approach with real code: fetch, pick the main content, remove the junk, convert to Markdown, and chunk. I also show the output from three public docs pages, and the cases where this approach fails.

The stack

  • Node 20+ fetch for HTTP (with a timeout, retries and a size cap)
  • cheerio to parse and clean the HTML
  • turndown to convert the cleaned HTML to Markdown

That's it. No Chromium download, no Playwright, and no 1 GB container image.

Step 1: Fetch politely

The fetcher is ordinary, but a few details matter once you crawl more than one page: a timeout via AbortController, retries with backoff on 429 and 5xx (honoring Retry-After), and a byte cap so one huge page can't use up your memory. The crawler also checks robots.txt and keeps about 250 ms between requests to the same host.

const ctrl = new AbortController();
const timer = setTimeout(() => ctrl.abort(), timeoutMs);
const res = await fetch(url, {
  redirect: 'follow',
  signal: ctrl.signal,
  headers: { 'user-agent': USER_AGENT, accept: 'text/html,application/xhtml+xml;q=0.9,*/*;q=0.5' },
});
if ((res.status === 429 || res.status >= 500) && attempt < retries) {
  const ra = Number(res.headers.get('retry-after'));
  await sleep(Number.isFinite(ra) && ra > 0 ? Math.min(ra, 30) * 1000 : 1000 * 2 ** attempt);
  continue;
}
Enter fullscreen mode Exit fullscreen mode

Then cheerio.load(body) gives you a jQuery-like $.

Step 2: Find the main content

Most sites wrap the actual content in something semantic. Try the usual candidates in order. If a selector matches several elements, keep the one with the most text, and only accept it if it has a meaningful amount of text:

function pickMain($) {
  const candidates = ['article', 'main', '[role="main"]', '#content', '#main',
    '.post-content', '.entry-content', '.article-body', '.content'];
  for (const sel of candidates) {
    const els = $(sel);
    if (!els.length) continue;
    let best = null; let bestLen = 0;
    els.each((_, e) => { const l = $(e).text().trim().length; if (l > bestLen) { best = e; bestLen = l; } });
    if (best && bestLen > 200) return $(best);
  }
  return $('body');
}
Enter fullscreen mode Exit fullscreen mode

The 200-character threshold matters. Some pages have an empty <article> used as a card wrapper, and without the check you would get an empty result.

Step 3: Remove the junk

Before picking the root, remove elements that are never content. After picking it, remove the layout parts (nav, header, footer, aside) that live inside the main column:

const JUNK = 'script,style,noscript,template,iframe,svg,canvas,form,button,input,select,textarea,dialog,'
  + '[role="navigation"],[role="banner"],[role="contentinfo"],[aria-hidden="true"],[hidden],'
  + '.cookie,.cookies,#cookie-banner,.cookie-banner,.consent,.newsletter,.advert,.ads,.ad,.share,.social,.breadcrumbs,.breadcrumb';

$(JUNK).remove();
const root = pickMain($);
root.find('nav,header,footer,aside').remove();
root.find('.mw-editsection,.hash-link,a.anchor,.headerlink,.toc,#toc,.sr-only,.visually-hidden').remove();
Enter fullscreen mode Exit fullscreen mode

One more heuristic catches what the selectors miss, like in-content menus, language lists and tag clouds. If a block is mostly link text, has at least 8 links, and is less than half of the page, drop it:

const rootLen = root.text().replace(/\s+/g, ' ').trim().length || 1;
root.find('ul,ol,div,section,table').each((_, e) => {
  const el = $(e);
  const t = el.text().replace(/\s+/g, ' ').trim();
  if (t.length < 80) return;
  const lt = el.find('a').text().replace(/\s+/g, ' ').trim().length;
  if (lt / t.length > 0.75 && el.find('a').length >= 8 && t.length < rootLen * 0.5) el.remove();
});
Enter fullscreen mode Exit fullscreen mode

The "less than half of the page" condition keeps the heuristic from deleting the content of an index page that really is a list of links.

Finally, make every href and src absolute with new URL(href, finalUrl). A chunk that says [see here](../guide) is useless once it's sitting in a vector store.

Step 4: Convert to Markdown

Turndown does most of the work. Two custom rules made a real difference for docs:

const td = new TurndownService({ headingStyle: 'atx', codeBlockStyle: 'fenced', bulletListMarker: '-' });

// Docs sites wrap <pre> in syntax-highlighting spans; use the raw text instead.
td.addRule('pre', {
  filter: (n) => n.nodeName === 'PRE',
  replacement: (_c, n) => `\n\n\`\`\`\n${n.textContent.replace(/\n+$/, '')}\n\`\`\`\n\n`,
});

// Turndown core drops table structure; emit simple GFM tables.
td.addRule('tableCell', { filter: ['th', 'td'], replacement: (c) => ` ${c.replace(/\n+/g, ' ').trim()} |` });
Enter fullscreen mode Exit fullscreen mode

(The full version also has rules for table rows that add a | --- | separator after the header row.)

If the result has no # heading near the top, prepend the page title. That way the first chunk always says what document it came from.

Step 5: Chunk on paragraph boundaries

There are many chunking strategies. This one is deliberately simple: split on blank lines, pack paragraphs into a chunk until the next one would exceed the budget, and carry a small overlap into the next chunk.

const estimateTokens = (s) => Math.ceil(s.length / 4);

function chunk(md, size, overlap) {
  const paras = md.split(/\n{2,}/);
  const chunks = [];
  let cur = '';
  for (const p of paras) {
    if (cur && estimateTokens(cur + '\n\n' + p) > size) {
      chunks.push(cur);
      const tail = overlap ? cur.slice(-overlap * 4) : '';
      cur = tail ? tail + '\n\n' + p : p;
    } else cur = cur ? cur + '\n\n' + p : p;
  }
  if (cur.trim()) chunks.push(cur);
  return chunks.map((text, i) => ({ index: i, text, tokenEstimate: estimateTokens(text) }));
}
Enter fullscreen mode Exit fullscreen mode

The token estimate is "4 characters per token". It's close enough for English prose when you're sizing chunks, but it's not your embedding model's tokenizer. If you're close to a hard model limit, count with the real tokenizer.

Real output

I ran it on three docs pages and one single-page app, with chunkSizeTokens: 800 and chunkOverlapTokens: 50:

{
  "startUrls": [
    { "url": "https://docs.python.org/3/tutorial/classes.html" },
    { "url": "https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Caching" },
    { "url": "https://nodejs.org/api/fs.html" },
    { "url": "https://app.netlify.com/" }
  ],
  "chunkSizeTokens": 800,
  "chunkOverlapTokens": 50
}
Enter fullscreen mode Exit fullscreen mode

The whole run, including Node startup, took 2.3 seconds on a laptop.

Page Words Token estimate Chunks Largest chunk
Python tutorial: Classes 5,476 9,775 14 800
MDN: HTTP caching 5,394 10,061 14 800
Node.js: File system API 39,815 108,659 165 1,663
app.netlify.com 2 (skipped) 0 n/a

The start of the MDN page's first chunk, with no nav, sidebar or footer:

# HTTP caching

The HTTP cache stores a response associated with a request and reuses the stored response for subsequent requests.

There are several advantages to reusability. First, since there is no need to deliver the request to the origin server, then the closer the client and cache are, the faster the response will be. ...
Enter fullscreen mode Exit fullscreen mode

And the item for the Netlify app, which renders everything client-side:

{
  "url": "https://app.netlify.com/",
  "statusCode": 200,
  "error": "Only 2 words extracted. The page may need JavaScript rendering.",
  "markdown": "# Netlify"
}
Enter fullscreen mode Exit fullscreen mode

That check (wordCount < minWords, default 20) is the most useful line in the whole pipeline. Without it, a JavaScript-rendered page returns HTTP 200 and an almost empty document, and that empty document quietly goes into your index.

What's not great (yet)

Looking closely at the output, there are some real flaws you should know about if you copy this:

  • Code blocks can be split. Splitting on blank lines doesn't know about code fences. A code block that contains blank lines can end up across two chunks. In the Node.js fs page, 16 of 165 chunks had a code fence opened in one chunk and closed in the next. The fix is to tokenize fenced blocks as one unit before splitting.
  • The overlap is character-based. The overlap is the last overlap * 4 characters of the previous chunk, so it often starts mid-word (proxy cache" implementations that...) and sometimes inside a code block. Overlapping by whole sentences or paragraphs would be cleaner.
  • One oversized paragraph becomes one oversized chunk. The largest Node.js chunk was about 1,663 tokens against an 800 budget, because a single block was bigger than the budget. Add a fallback that splits long blocks by sentence or by line.
  • Site-specific leftovers. Node's docs keep [#](...) permalink anchors next to each heading, and MDN code blocks keep a stray language label (http) on the line above the fence. Neither breaks retrieval, but both are noise. Per-site cleanup rules fix them.
  • No chunk metadata. Each chunk would retrieve better if it carried the nearest heading path (HTTP caching > Freshness > max-age). That's the next thing I'd add.

The limits of "no browser"

  • JavaScript-rendered pages are not supported. If the content only appears after client-side JS runs (most dashboards, many marketing sites built as SPAs, some modern docs platforms), plain HTTP gets an empty shell. The pipeline detects this and flags the page instead of returning junk, but it can't render the page. For those, you need Playwright or Puppeteer.
  • No logins, no paywalls. It reads public pages only.
  • Bot protection. Sites behind aggressive anti-bot challenges will return a challenge page or a 403. This approach doesn't try to get around that, and it shouldn't.
  • The heuristics are heuristics. pickMain and the link-density filter work well on docs, blogs and help centers. On unusual layouts they can keep a sidebar or, more rarely, cut a list you wanted. Spot-check a few pages per site before you index thousands.

For server-rendered docs, though, this covers most of what a RAG ingestion step needs, in a few hundred lines and a couple of seconds per batch.

If you'd rather not run it yourself

We packaged this exact code (with the crawl mode, include/exclude URL patterns, depth limits and robots.txt handling) as an Apify actor: Website to Markdown for LLM & RAG. Full disclosure: it's ours. It's free to use until December 6, 2026, and after that it costs about $1 per 1,000 pages converted. Skipped and failed pages aren't charged. It has the same limits described above, including no JavaScript rendering.

If you only need a few pages, the snippets above are most of it. Copy them and adjust the selectors for your site.

Top comments (0)