Firecrawl-style scrapers advertise "web to LLM-ready Markdown," and it's worth being literal about why that's the product. HTML is hostile to every step of a RAG pipeline: boilerplate (nav, footers, cookie banners) pollutes embeddings; chunkers split on tags mid-sentence; and token counts triple for the same content — you pay to embed <div class="..."> forever.
Markdown conversion is really three services: boilerplate removal (the hard part), structure preservation (headings survive, so chunking on ## works), and token compression (typically 3-5x fewer tokens per page).
That reframes the pricing: you're not paying per page fetched, you're paying per page cleaned. Which is why rolling your own with requests+BeautifulSoup is a real alternative for a handful of known sites (you write the cleanup once) and a false economy for crawling arbitrary ones.
Free-tier credits and where they run out: https://toolfreebie.com/firecrawl-free-scraping-api/
Top comments (0)