Turn any web page into LLM-ready Markdown for RAG

Retrieval-augmented generation (RAG) works better when documents are clean prose instead of raw HTML. Navigation bars, scripts, and layout noise waste tokens and confuse chunkers.

This tutorial shows a practical pattern: fetch a public URL, convert the main content to Markdown, chunk it, and (optionally) embed the chunks. We’ll call the public URL to Markdown (LLM-ready) API on RapidAPI:

https://rapidapi.com/evan-DewNPsuYp/api/url-to-markdown-llm-ready

You’ll need a RapidAPI key. Start on the free plan while you prototype.

Why Markdown?

• Headings become natural chunk boundaries (#, ##, …).
• Lists and tables survive in a form LLMs already understand.
• Token counts are closer to what you send the model than raw HTML is.

This API is static HTML only — it does not run JavaScript. Sites that render all content client-side may return little text. It also rejects oversized HTML (about 200 KB body cap) with HTTP 413. Those limits are fine for many blogs and docs pages; know them before you design a crawler.

1. Subscribe and set the key

export RAPIDAPI_KEY='your-key-here'

Host header (from the public listing): url-to-markdown-llm-ready.p.rapidapi.com

2. Fetch Markdown

curl --request GET \
  --url 'https://url-to-markdown-llm-ready.p.rapidapi.com/v1/markdown?url=https%3A%2F%2Fexample.com&include_links=true&include_images=false' \
  --header "X-RapidAPI-Key: $RAPIDAPI_KEY" \
  --header 'X-RapidAPI-Host: url-to-markdown-llm-ready.p.rapidapi.com'

Successful responses look like:

{
  "ok": true,
  "title": "Example Domain",
  "markdown": "This domain is for use in documentation examples...",
  "chars": 156,
  "tokensEstimate": 39,
  "truncated": false,
  "contentSource": "body",
  "finalUrl": "https://example.com/"
}

tokensEstimate is a rough chars / 4 heuristic — useful for budgeting, not a substitute for your model’s tokenizer.

Useful query flags:

Param	Default	Meaning
include_links	true	Keep [text](url) links
include_images	false	Emit  when true
include_tables	true	Keep GitHub-style tables
max_chars	200000	Soft-truncate long Markdown

3. Node loader for a RAG pipeline

// markdownLoader.mjs
const HOST = "url-to-markdown-llm-ready.p.rapidapi.com";

export async function urlToMarkdown(targetUrl, opts = {}) {
  const endpoint = new URL(`https://${HOST}/v1/markdown`);
  endpoint.searchParams.set("url", targetUrl);
  endpoint.searchParams.set("include_links", String(opts.includeLinks ?? true));
  endpoint.searchParams.set("include_images", String(opts.includeImages ?? false));
  endpoint.searchParams.set("include_tables", String(opts.includeTables ?? true));
  if (opts.maxChars) endpoint.searchParams.set("max_chars", String(opts.maxChars));

  const res = await fetch(endpoint, {
    headers: {
      "X-RapidAPI-Key": process.env.RAPIDAPI_KEY,
      "X-RapidAPI-Host": HOST,
    },
  });
  const data = await res.json();
  if (!res.ok || !data.ok) {
    const err = new Error(data.error || `HTTP ${res.status}`);
    err.code = data.code;
    throw err;
  }
  return data;
}

/** Naive heading-aware splitter — replace with your preferred chunker. */
export function chunkMarkdown(md, { maxChars = 1200 } = {}) {
  const parts = md.split(/\n(?=#{1,6} )/g);
  const chunks = [];
  let buf = "";
  for (const part of parts) {
    if ((buf + "\n" + part).length > maxChars && buf) {
      chunks.push(buf.trim());
      buf = part;
    } else {
      buf = buf ? buf + "\n" + part : part;
    }
  }
  if (buf.trim()) chunks.push(buf.trim());
  return chunks;
}

const page = await urlToMarkdown("https://example.com", { maxChars: 20000 });
const chunks = chunkMarkdown(page.markdown);
console.log(page.title, chunks.length, page.tokensEstimate);

4. Plug into embeddings (sketch)

Pseudocode only — swap in your embedding provider:

for (const [i, text] of chunks.entries()) {
  // const vector = await embed(text);
  // await vectorDb.upsert({ id: `${page.finalUrl}#${i}`, text, vector, meta: { title: page.title } });
  console.log(`chunk ${i}:`, text.slice(0, 80));
}

At query time, embed the user question, retrieve top-k chunks, and pass them as context to your LLM.

Failure modes worth handling

Symptom	Likely cause
HTTP 413 / too_large	HTML over ~200 KB — try a cleaner URL or a different extractor
Sparse Markdown	JS-rendered SPA — need a browser-based tool instead
ssrf_blocked	Private/localhost URL — expected safety behavior
upstream_error	Target site 4xx/5xx

Related

Need a link-preview card (title + image) instead of full Markdown? Use URL Metadata & Open Graph Extractor.

Keep keys on the server, cache by URL, and stay inside your RapidAPI quota while you validate retrieval quality. Once chunk quality looks good on a handful of real pages, wire the same loader into LangChain / LlamaIndex custom document loaders the same way.