Turn Any Website into LLM-Ready Markdown with Firecrawl and Python

#webscraping #python #ai #llm

If you're feeding web content to an LLM, raw HTML wastes most of your context window on navigation, scripts and markup. What you want is the article text as markdown, with headings and code blocks intact. Firecrawl does that conversion as an API, including JavaScript-rendered pages, so you don't maintain a scraper per site.

This post walks through three jobs: converting a single page, saving a list of URLs as markdown files, and crawling a whole docs site into a folder you can index for RAG.

Setup

Install the SDK. I pin the version so the examples keep working:

pip install firecrawl==4.55.0

You can try scrape without an API key, at a low per-IP rate limit. For anything beyond a quick test, create a free account (1,000 credits a month, one credit per page) and export the key:

export FIRECRAWL_API_KEY="fc-YOUR_API_KEY"

Convert a Single Page to Markdown

from firecrawl import Firecrawl

firecrawl = Firecrawl()  # reads FIRECRAWL_API_KEY

doc = firecrawl.scrape(
    "https://docs.firecrawl.dev/introduction",
    formats=["markdown"],
    only_main_content=True,
)
print(doc.markdown)

only_main_content=True is the setting that matters most for LLM use. It strips headers, footers, cookie banners and sidebars, and on a typical docs page that's a large share of the text. It's on by default, but I set it explicitly so nobody reading the code has to guess.

Save a List of URLs as Markdown Files

For a handful of known pages, batch_scrape sends them in one job instead of a loop of single calls:

from pathlib import Path
from urllib.parse import urlparse

from firecrawl import Firecrawl

urls = [
    "https://docs.firecrawl.dev/features/scrape",
    "https://docs.firecrawl.dev/features/crawl",
    "https://docs.firecrawl.dev/features/map",
]

firecrawl = Firecrawl()
job = firecrawl.batch_scrape(urls, formats=["markdown"], poll_interval=2, timeout=120)

out = Path("pages")
out.mkdir(exist_ok=True)
for page in job.data:
    slug = urlparse(page.metadata.source_url).path.strip("/").replace("/", "_") or "index"
    (out / f"{slug}.md").write_text(page.markdown or "", encoding="utf-8")
    print(f"saved {slug}.md")

Naming files after the URL path keeps the mapping obvious when you index them later, and makes it easy to cite the source page in an answer.

Crawl an Entire Docs Site into a Folder

Before crawling, map the site. map returns the URLs it can find, fast and cheaply, so you know what the crawl will cost before you spend the credits:

from firecrawl import Firecrawl

firecrawl = Firecrawl()
links = firecrawl.map("https://docs.firecrawl.dev", limit=500)
print(links)

If the count looks right, crawl with a limit and exclude the sections you don't need. Changelogs and blog archives are the usual ones to skip:

from pathlib import Path
from urllib.parse import urlparse

from firecrawl import Firecrawl

firecrawl = Firecrawl()
job = firecrawl.crawl(
    "https://docs.firecrawl.dev",
    limit=50,
    exclude_paths=["blog/.+", "changelog.*"],
    scrape_options={"formats": ["markdown"], "only_main_content": True},
)

out = Path("docs")
out.mkdir(exist_ok=True)
for page in job.data:
    slug = urlparse(page.metadata.source_url).path.strip("/").replace("/", "_") or "index"
    (out / f"{slug}.md").write_text(page.markdown or "", encoding="utf-8")

print(f"{job.status}: {len(job.data)} pages")

crawl blocks until the job finishes. For big sites, use start_crawl to get a job ID back immediately and check it with get_crawl_status, so a long crawl doesn't tie up your script.

Tips Before You Index Anything

Set limit on every crawl. Without it, the crawl follows every link it finds, and each page costs a credit. Excluding paths up front matters for the same reason: pages you filter out after the crawl have already been paid for.

Check for near-empty pages. Login walls and bot challenges can come back as a few lines of text, and indexing them adds noise to every search. Skipping files under a few hundred characters catches most of them.

When you chunk the files for embeddings, split on headings rather than a fixed character count. Firecrawl keeps the page's heading structure, so splitting on ## gives you chunks that each cover one topic.

Where to Go Next

Once the markdown is on disk, any embedding pipeline can index it. If you want structured fields instead of text, Firecrawl's scrape also takes a JSON schema. I cover that and the crawl, map and agent endpoints in more detail in this Firecrawl guide on alphasec.io.

This post includes a referral link: signing up through it gives us both extra credits.

DEV Community