How to convert web pages and PDFs to clean Markdown for LLMs and RAG (2026)

If you feed web pages to an LLM, the first problem isn't the model. It's the input. A typical page is mostly navigation, scripts, cookie banners, footers and class names, and the text you actually care about is a small part of it. You pay for every one of those tokens, and the noise makes retrieval worse too.

This guide shows how to turn web pages and PDFs into clean Markdown that is ready for a prompt or a vector database: first in Apify Console without code, then from Python, then as a tool for AI agents over MCP.

The problem: raw HTML is mostly noise

To put a number on it, I took the Wikipedia article Retrieval-augmented generation and compared three versions of the same page:

Version	Characters	Approx. tokens
Raw HTML (downloaded with curl)	252,873	~63,000
Main content as Markdown, default settings	29,041	7,260
Markdown without references, edit links and hyperlinks	10,752	2,688

The token column is characters divided by 4, which is a rough rule of thumb, not your model's tokenizer. The exact count will differ, but the order of magnitude won't: the raw HTML is roughly ten times bigger than the article text, and most of what you'd send is markup.

Markdown is a good target format because it keeps the structure that matters to a model (headings, lists, tables, links) and drops everything else. Headings also give you natural places to split a document into chunks.

Option 1: write it yourself

For a handful of sites you control, a small script is often enough. trafilatura extracts the main content and can output Markdown directly, readability-lxml is a Python port of the classic Readability algorithm, and markdownify turns any HTML into Markdown. Something like trafilatura.extract(html, output_format="markdown") gets you surprisingly far.

The work starts when you go beyond a few pages: JavaScript-rendered sites that return an empty shell over plain HTTP, PDFs, robots.txt, retries on 429s, relative links, duplicate pages in a crawl, chunking, and somewhere to run it all on a schedule. There are also hosted services for this, such as Apify's own RAG Web Browser, Jina Reader and Firecrawl, and they're worth comparing on price and output for your pages.

Option 2: a hosted URL to Markdown Actor

For the rest of this walkthrough I'll use URL to Markdown Converter, an Actor on the Apify platform (disclosure: built by CleanData Labs). It uses trafilatura and markdownify under the hood, so the extraction is similar to what you'd write yourself, and adds the parts around it:

• main-content extraction (article mode), full-page Markdown (full), or auto, which falls back to the full page when too little text is found,
• output as markdown, text or cleaned html,
• text-based PDFs converted the same way as web pages,
• a headless browser only when the plain HTTP result is too thin (renderJs: "auto"),
• small crawls by following links or reading the sitemap, with include/exclude URL patterns,
• metadata per page (title, author, publish date, language, canonical URL) and approxTokens,
• optional chunks for RAG, with overlap.

It respects robots.txt by default and sends an identifiable User-Agent.

Pricing: $0.002 per converted page, and PDFs cost the same as web pages. Failed, blocked, robots.txt-disallowed and duplicate URLs are free. Apify adds its standard $0.00005 Actor start fee per run. The free Apify plan includes $5 of monthly credit, which covers a lot of test pages.

Step-by-step in Apify Console

1. Open the Actor page and click Try for free. Sign in or create a free Apify account.
2. Paste your pages or PDF links into URLs. Replace the two prefilled demo URLs with your own.
3. Leave Output formats on markdown unless you need plain text or HTML as well. Every extra format makes the output bigger.
4. Leave Extraction mode on Auto, or pick Main content only (article) if you never want menus and footers.
5. To trim more, use Remove elements (CSS selector), for example .newsletter, .related-posts, #comments. Turn off Keep links if you don't need URLs in the text; it saves tokens.
6. For RAG, set Chunk size (approx. tokens) to something like 800 and Chunk overlap (approx. tokens) to 80.
7. Click Start, then open the Output tab. The overview table shows title, language, words and ~tokens per page, and you can export everything as JSON, CSV or Excel.

For a JavaScript-heavy single-page app, set JavaScript rendering to Always and, if content loads late, add a Wait for selector (browser) such as main article.

What the output looks like

This is a shortened row from a real run on the Wikipedia page above, with references and links removed and chunking on:

{
  "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
  "statusCode": 200,
  "title": "Retrieval-augmented generation - Wikipedia",
  "publishedAt": "2023-11-05",
  "siteName": "Wikimedia Foundation, Inc.",
  "language": "en",
  "canonicalUrl": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
  "fetchedWith": "http",
  "markdown": "# Retrieval-augmented generation\n\n**Retrieval-augmented generation** (**RAG**) is a technique that enables large language models (LLMs) to retrieve and incorporate new information from external data sources. ...",
  "characters": 10752,
  "words": 1590,
  "approxTokens": 2688,
  "truncated": false,
  "chunks": [
    {"index": 0, "text": "# Retrieval-augmented generation\n\n...", "approxTokens": 722}
  ]
}

fetchedWith tells you how the page was read: http, browser or pdf. Failed URLs come back as rows with an error field (and blocked: true for 401/403/429 and bot walls), so you can see what went wrong without paying for it. That run converted one page and cost $0.00205 ($0.002 for the page plus the start fee). It took about 20 seconds end to end, most of it container startup.

Python example with apify-client

Install the official client with pip install apify-client and copy your API token from Settings → API & Integrations in Apify Console. This is the input from the run above, tested with apify-client 3.4:

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

run_input = {
    "urls": ["https://en.wikipedia.org/wiki/Retrieval-augmented_generation"],
    "outputFormats": ["markdown"],
    "includeLinks": False,
    "removeElementsCssSelector": ".reflist, .mw-references-wrap, sup.reference, .mw-editsection, .navbox",
    "chunkSizeTokens": 800,
    "chunkOverlapTokens": 80,
}

run = client.actor("cleandata-labs/url-to-markdown").call(
    run_input=run_input,
    max_total_charge_usd=0.10,  # hard cost cap for this run
)

for page in client.dataset(run.default_dataset_id).iterate_items():
    if page.get("error"):
        print("failed:", page["url"], page["error"])
        continue
    print(page["title"], page["approxTokens"], "tokens,", len(page["chunks"]), "chunks")

Output:

Retrieval-augmented generation - Wikipedia 2688 tokens, 4 chunks

Two notes from running it. max_total_charge_usd is a useful safety net for crawls, because the Actor stops cleanly when the cap is reached, but Apify won't accept a value below $0.10. And the CSS selectors in removeElementsCssSelector are site-specific: the ones above are Wikipedia's reference and edit-link classes. For other sites, open the page in your browser's dev tools and look for the wrapper around comments, related posts or newsletter boxes.

Splitting into chunks for RAG

With chunkSizeTokens set, every page gets a chunks list. Chunks are split on paragraph boundaries, and very long paragraphs are cut at the nearest word boundary. In the run above the four chunks came out at 722, 786, 792 and 625 approximate tokens. Each chunk after the first begins with an overlap: up to the last chunkOverlapTokens (here ~80) approximate tokens of the previous chunk, starting at a word boundary. So a chunk can start mid-sentence; that's normal for overlap and usually fine for embeddings.

Here is a minimal way to turn the rows into records for a vector database. embed() stands in for whatever embedding model you use:

records = []
for page in client.dataset(run.default_dataset_id).iterate_items():
    if page.get("error"):
        continue
    for ch in page.get("chunks", []):
        records.append({
            "id": f'{page["finalUrl"]}#{ch["index"]}',
            "text": ch["text"],
            "metadata": {
                "url": page["finalUrl"],
                "title": page["title"],
                "language": page.get("language"),
                "publishedAt": page.get("publishedAt"),
            },
        })

# vectors = embed([r["text"] for r in records])
# then upsert ids, vectors and metadata into Pinecone, Qdrant, pgvector, ...

If you prefer to chunk yourself (by heading, with a real tokenizer, or with LangChain or LlamaIndex splitters), leave chunkSizeTokens at 0 and split the markdown field instead. The Actor's chunker is deliberately simple.

To index a whole docs section rather than single pages, start from the sitemap and limit it:

{
  "urls": ["https://docs.apify.com/"],
  "useSitemap": true,
  "includeUrlGlobs": ["https://docs.apify.com/academy/*"],
  "maxPagesPerStartUrl": 50,
  "chunkSizeTokens": 800
}

You can also follow links with Crawl depth (maxCrawlDepth). Crawls stay on the same host, and duplicate pages (same final URL or same content) are skipped and not charged.

Using it from AI agents via MCP

If you work with Claude, Cursor, VS Code or another MCP client, you can give the agent this Actor as a tool through the official Apify MCP server. The URL below loads only this one Actor:

https://mcp.apify.com?tools=cleandata-labs/url-to-markdown

It's also listed in the official MCP registry as io.github.CleanData-Labs/url-to-markdown.

In Claude Code:

claude mcp add --transport http url-to-markdown "https://mcp.apify.com?tools=cleandata-labs/url-to-markdown"

In Cursor (~/.cursor/mcp.json or .cursor/mcp.json in a project):

{
  "mcpServers": {
    "url-to-markdown": {
      "url": "https://mcp.apify.com?tools=cleandata-labs/url-to-markdown"
    }
  }
}

On the first connection the client opens the Apify sign-in (OAuth). If you'd rather use a token, send it as an Authorization: Bearer <YOUR_APIFY_TOKEN> header. Runs are billed to your Apify account at the same $0.002 per page.

The agent then sees a tool called cleandata-labs--url-to-markdown and can be asked things like:

Read https://docs.apify.com/platform/actors as Markdown and summarize it in 5 bullet points.

A tip for agent use: set maxCharacters (for example 20000, around 5k tokens) so a long page doesn't flood the agent's context, and use maxPagesPerStartUrl to keep crawls small. Every row includes approxTokens, so the agent can decide how much of a page to read.

Limitations

• Pages behind logins, paywalls or bot protection are not bypassed. They come back as error rows with blocked: true.
• PDFs need a text layer. Scanned PDFs aren't OCR'd, and PDF tables come out as plain text.
• Crawling is meant for small sections of a site. For crawling an entire large website, use a dedicated crawler.
• Token counts are approximate (characters ÷ 4).

Wrapping up

If you only need a few known pages, trafilatura or readability plus markdownify in a short script is a perfectly good answer. If you want PDFs, JavaScript fallback, sitemaps, chunks and an MCP tool without running anything yourself, URL to Markdown Converter is quick to try on the free plan. You can schedule it in Apify Console or trigger it from Make, Zapier or n8n.

Whatever you use, the content still belongs to the sites you fetch it from. Check their terms and copyright before you store, train on or republish it.