I built the two Apify actors I use in the examples. I will say that now, so you can read the rest with the right amount of skepticism.
The job is simple to describe and easy to overpay for. You have a documentation site, or a list of URLs. You want clean Markdown, one record per page, with a title and a source URL, so you can chunk it and embed it. You do not want nav bars, cookie banners, or a bill you cannot explain to yourself before you press Run.
If that is your job, you can do it for about $0.10 per 1,000 static pages.
Why raw HTML makes RAG worse
Most "load this website" tutorials fetch the HTML and throw it at a splitter. The splitter then dutifully indexes the footer, the cookie dialog, and three copies of the sidebar. Your bot will someday answer a pricing question with "Accept all cookies," and it will sound very confident.
Markdown helps because headings, lists, tables, and code blocks are still there, and the chrome is mostly gone. You still have to chunk carefully (I will get to that), but you are no longer embedding CSS class names.
Two other traps:
- Some tools also search the web. That is useful when you do not have URLs. It is wasted money when you already do.
- Some tools start a headless browser on every page. That is required for a React shell that ships an empty
#root. It is not required for a lot of docs sites, which are still just HTML.
What I actually use
I use two actors on Apify Store:
- A list of known URLs: Web Page to Markdown, $0.50 per 1,000 HTTP pages, $1.00 per 1,000 browser pages.
- A docs site, blog, or help center: Website to Markdown Crawler, $0.10 per 1,000 HTTP pages, $0.35 per 1,000 browser pages, $0.50 per 1,000 saved files. That last one is only if you turn on file saving.
Both try plain HTTP first. A headless Chromium starts only when a page looks like a client-rendered shell, and only that page is billed at the browser price. Platform usage is included in the event price. Failed pages, duplicates, and robots.txt blocks are not billed. There is a $0.00005 start fee per run.
They stay on public pages. robots.txt and crawl-delay are honoured. There is no login, no cookie injection, and no captcha bypass. If the docs are behind a sign-in, this is the wrong tool.
The rest of this post uses the crawler, because "here is the docs root, give me the site" is the common case.
An honest comparison
I am not going to crown a winner. These tools overlap, they do not replace each other.
Firecrawl is the polished hosted API a lot of people reach for. A standard scrape or crawl is 1 credit per page (checked 13 November 2026). The free plan is 1,000 credits a month, no card, which is a genuine way to try it. After that, Hobby is $19 a month for 5,000 credits, about $3.80 per 1,000 pages if you use the allotment. Extra Hobby credits are $5 per 1,000. Standard is $99 for 100,000 credits, about $0.99 per 1,000, or $5 per 2,000 extra credits. JSON extraction and some add-ons cost extra credits. Firecrawl also searches, maps, and interacts with pages. I do not do those things.
Apify RAG Web Browser starts at $1.33 per 1,000 fetched pages. It can search Google and then fetch the results. Use it when you need search. My actors do not search. They only fetch the URLs or the site you give them.
Apify Website Content Crawler is the official full-featured crawler. It is billed on platform usage, not a fixed per-page event. That can be cheap or not, depending on memory, proxies, and how long the run takes. I wanted a number I could write down before the run, so I went pay-per-page instead. I am not claiming I beat it on every site.
Jina Reader is still the least ceremony for a single URL: prefix the URL and you get Markdown. I use it for one-off fetches. It is not what I use to crawl a 2,000-page docs site with a cap and a sitemap.
If you want to self-host, Crawl4AI is the open-source tool I would try first. You pay in your own time and machines, not per page.
Walkthrough: crawl a docs site
You will need an Apify account and an API token from Console, under Settings, then API and Integrations. The Free plan is $0, includes $5 of prepaid usage each month, and does not require a card. Unused credit expires at the end of the cycle.
The example below starts at a public docs root, stays on that host and path, reads sitemaps and llms.txt if they exist, and stops at 200 pages so a first run cannot run away.
JavaScript, official client
import { ApifyClient } from "apify-client";
const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor("datlab/site-to-markdown-crawler").call({
startUrls: [{ url: "https://docs.apify.com/academy" }],
crawlerType: "auto",
useSitemaps: true,
useLlmsTxt: true,
maxCrawlPages: 200,
outputFormats: ["markdown"],
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
for (const item of items) {
if (!item.markdown) continue;
console.log(item.url, item.metadata?.title, item.tokenEstimate);
}
Python, official client
import os
from apify_client import ApifyClient
client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("datlab/site-to-markdown-crawler").call(
run_input={
"startUrls": [{"url": "https://docs.apify.com/academy"}],
"crawlerType": "auto",
"useSitemaps": True,
"useLlmsTxt": True,
"maxCrawlPages": 200,
"outputFormats": ["markdown"],
}
)
items = client.dataset(run["defaultDatasetId"]).list_items().items
pages = [item for item in items if item.get("markdown")]
print(f"saved {len(pages)} pages")
A successful item looks like this (trimmed):
{
"url": "https://crawlee.dev/js/docs/quick-start",
"crawl": {
"httpStatusCode": 200,
"renderedWith": "raw-http"
},
"metadata": {
"title": "Quick Start | Crawlee for JavaScript",
"canonicalUrl": "https://crawlee.dev/js/docs/quick-start",
"languageCode": "en"
},
"markdown": "# Quick Start\n\nWith this short tutorial you can start scraping…",
"wordCount": 756,
"tokenEstimate": 2373
}
renderedWith is the field I look at after a first run. If almost everything is raw-http, you got the $0.10 per 1,000 price. If almost everything is the browser, the site is probably a client-rendered app, and you should expect the $0.35 per 1,000 price. You can force playwright:chromium when you already know that.
Respect the site. Use a real maxCrawlPages. Do not point this at a host you are not allowed to crawl. Check their robots.txt first. Hacker News, for example, publishes a 30 second crawl-delay, which will make discovery slow on purpose.
Hook it to a vector store
Import paths in LangChain move around. If a name below has changed, the idea has not: each page becomes a Document, you split on Markdown headings, you prepend the title, you embed, you store.
from langchain_core.documents import Document
from langchain_text_splitters import MarkdownHeaderTextSplitter, RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import Chroma
docs = []
for item in pages:
meta = item.get("metadata") or {}
docs.append(
Document(
page_content=item["markdown"],
metadata={
"source": item.get("url"),
"title": meta.get("title"),
"canonical": meta.get("canonicalUrl"),
"rendered_with": (item.get("crawl") or {}).get("renderedWith"),
"token_estimate": item.get("tokenEstimate"),
},
)
)
header_splitter = MarkdownHeaderTextSplitter(
headers_to_split_on=[
("#", "h1"),
("##", "h2"),
("###", "h3"),
]
)
chunker = RecursiveCharacterTextSplitter(chunk_size=800, chunk_overlap=120)
chunks = []
for doc in docs:
sections = header_splitter.split_text(doc.page_content)
for section in sections:
title = doc.metadata.get("title") or ""
h1 = section.metadata.get("h1") or ""
h2 = section.metadata.get("h2") or ""
prefix = " > ".join(part for part in (title, h1, h2) if part)
section.page_content = f"{prefix}\n\n{section.page_content}" if prefix else section.page_content
section.metadata = {**doc.metadata, **section.metadata}
chunks.extend(chunker.split_documents([section]))
vectorstore = Chroma.from_documents(chunks, OpenAIEmbeddings())
Three notes, because this is the part tutorials usually skip:
- If you split only on character count, a chunk in the middle of "Authentication" loses the heading, and retrieval gets worse. Split on headings first.
- Put the page title and the parent headings into the chunk text, not only into metadata. Many embeddings never see your metadata.
- I am not going to tell you Chroma is the right store. Use whatever you already run. The mapping is the same for Pinecone, Qdrant, or Weaviate.
If you would rather stay in Python with no LangChain, the loop is the same: keep url, title, and markdown, split on headings, embed the chunks.
LlamaIndex can consume the same records. The official Apify LlamaIndex integration and the LangChain integration both work with a named actor, including these two.
If you want an agent to call the crawler as a tool, Apify hosts an MCP server at https://mcp.apify.com. You can pin the actor with a tools query like https://mcp.apify.com/?tools=fetch-actor-details,datlab/site-to-markdown-crawler. See the Apify MCP docs.
What a run actually costs
These are event prices, not a made-up crawl log. I did not invent CU figures for Website Content Crawler, and I will not.
Example A. 1,000 static docs pages, all HTTP:
1,000 * $0.0001 = $0.10, plus the $0.00005 start fee.
Example B. 200 pages, 160 HTTP and 40 browser:
160 * $0.0001 + 40 * $0.00035 = $0.016 + $0.014 = $0.03, plus the start fee.
Example C. A 5,000-page docs site, mostly static, in auto mode: about $0.50 if almost all pages stay on HTTP. That is the example I put on the Store page. If the site is a SPA, expect closer to 5,000 * $0.00035 = $1.75.
The URL-list actor is more expensive per page because it is a smaller, fetch-only job: $0.50 / $1.00 per 1,000. Use it when an agent already has the links, or when you recrawl a known set after a sitemap diff.
The $5 Free plan covers a serious test. 1,000 HTTP pages is a dime. Even a few thousand browser pages stay well under $5. Cap a run with maxCrawlPages or Apify's maximum cost per run, so a bad glob cannot surprise you.
For comparison, 1,000 Firecrawl credits on the Hobby allotment are about $3.80, or $5 if you buy extra credits, and a standard scrape is 1 credit (pricing page, 13 November 2026). That gap is real for static docs. It shrinks if you need Firecrawl's search, extract, or interact features, which I do not provide.
Recrawling later
A full recrawl of a docs site every night is usually a waste. Footers and build hashes change even when the article does not, and you will re-embed pages for no reason.
If the site has a sitemap, watch that first: new URLs, removed URLs, and lastmod changes. lastmod can lie, so I also hash the extracted content when I care. I shipped a Sitemap Change Monitor for that job. Then I send only the changed URLs back through the URL-list actor.
FAQ
Is this a Google-search browser? No. You give it URLs or a site root.
Will it log in or skip captchas? No.
What about PDFs and other files? The crawler can save same-site documents when you turn that on, at the file event price.
Can I schedule this? Yes, with Apify schedules and webhooks.
Can I use it from n8n, Make, or Zapier? Yes, through Apify's integrations. r/n8n does not want product pitches, so if you want an n8n example, I would rather publish a workflow with the JSON than drop a Store link in that subreddit.
If a page comes out wrong
Open an issue on the actor page, or comment here with the URL (public pages only). I am more interested in bad Markdown, missed tables, and wrong HTTP-versus-browser decisions than in compliments.
Actors:
Top comments (0)