DEV Community

Cover image for Firecrawl: Free Web Scraping API for LLMs & Agents
toolfreebie
toolfreebie

Posted on Originally published at toolfreebie.com

Firecrawl: Free Web Scraping API for LLMs & Agents

#ai #api #opensource

Firecrawl: Free Web Scraping API for LLMs & Agents

Quick answer: Firecrawl is a web-scraping API built for LLMs and AI agents: hand it a URL, get back clean Markdown or structured JSON with JavaScript rendered and boilerplate stripped. The free tier is 1,000 credits/month (~1,000 page reads), no credit card, with all five endpoints unlocked. For unlimited free use, self-host the AGPL-3.0 open-source core.

Every AI agent hits the same wall: the model reasons brilliantly, but the open web is HTML soup — nav bars, cookie banners, lazy-loaded JS, ads, anti-bot walls. Feed raw HTML into a context window and you burn tokens on markup. Firecrawl collapses that into one HTTP call: URL in, LLM-ready data out. It’s the “read” half of an agent toolkit that pairs with a search API — search finds URLs, Firecrawl reads them.

The Five Endpoints

Firecrawl isn’t one endpoint — it’s five, each solving a different shape of the web-data problem:

  • Scrape — one URL into clean Markdown / HTML / JSON
  • Crawl — recursively follow links and scrape an entire site
  • Map — instantly return every URL on a site (sitemap on demand)
  • Search — run a web query and get full page content per result in one call
  • Extract — pull structured data from one or many URLs via prompt or schema

That breadth is why Firecrawl is the default “fetch a page” tool in 2026 agent stacks, from CrewAI and LangGraph to the MCP servers wired into Cursor and Claude.

Free Tier: What You Get

  • 1,000 credits/month, reset each cycle (no rollover)
  • No credit card — sign up with email or GitHub, fc- key is live immediately
  • All five endpoints unlocked
  • 2 concurrent requests and conservative rate limits (the main ceiling)

The number that governs how far 1,000 credits stretches is the credit-to-page mapping:

Operation Credit cost What 1,000 credits buys
Scrape / Crawl / Map 1 credit per page ~1,000 pages scraped
Search 2 credits per 10 results ~5,000 results returned
Interact (browser actions) 2 credits per browser-minute ~500 minutes
Structured JSON extract 1 credit per page ~1,000 extractions

So 1,000 credits is effectively 1,000 free page reads per month. Comfortable for a personal research agent scraping 30 pages a day; you’ll blow through it in one job on a 5,000-page docs crawl. Know which side of that line you’re on.

Paid Tiers

Paid plans scale credits and concurrency together. Prices are the annual-billing rate from the official pricing page (monthly runs higher):

Plan Price (yearly) Credits/mo Concurrent
Free $0 1,000 2
Hobby $16/mo 5,000 5
Standard $83/mo 100,000 50
Growth $333/mo 500,000 100
Scale $599/mo 1,000,000 150

The jump from Hobby (5,000) to Standard (100,000) is steep — no middle tier — so the usual path is “free for prototyping, Hobby for a small live project, then a leap to Standard.” If volume is large and steady, self-host and pay nothing (below).

First Scrape in 60 Seconds

Sign up at firecrawl.dev, copy your fc- key, then export it: export FIRECRAWL_API_KEY="fc-YOUR_KEY". The v2 API base is https://api.firecrawl.dev/v2/ with a bearer token:

curl -X POST https://api.firecrawl.dev/v2/scrape \
  -H "Authorization: Bearer $FIRECRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://news.ycombinator.com", "formats": ["markdown"]}'
Enter fullscreen mode Exit fullscreen mode

The response carries data.markdown (cleaned page) and data.metadata (title, description, status). The Python SDK:

pip install firecrawl-py
Enter fullscreen mode Exit fullscreen mode 
from firecrawl import Firecrawl

firecrawl = Firecrawl(api_key="fc-YOUR_KEY")
result = firecrawl.scrape(
    "https://news.ycombinator.com",
    formats=["markdown", "html"],
)
print(result.markdown[:500])
print(result.metadata.title)
Enter fullscreen mode Exit fullscreen mode

Field names come back snake_case, and an AsyncFirecrawl class ships with identical methods for non-blocking pipelines. A Node.js SDK (npm install firecrawl) exposes the same firecrawl.scrape(url, { formats: [...] }) call. No proxy config, no headless-browser setup — one call returns model-ready text.

The Endpoints, With Code

Scrape: One URL → Clean Data

Beyond Markdown, request multiple formats in one call — markdown, html, rawHtml, links, screenshot, json. Each is computed from the same page load, so adding links costs no extra credit.

result = firecrawl.scrape(
    "https://example.com/pricing",
    formats=["markdown", "links"],
    only_main_content=True,   # drop nav, footer, sidebars
)
print(result.links)
Enter fullscreen mode Exit fullscreen mode

Crawl: Follow Links Across a Site

Crawl discovers linked pages and scrapes each — ideal for ingesting a docs site into RAG. It’s async: start a job and poll (or let the SDK wait).

job = firecrawl.crawl(
    url="https://docs.firecrawl.dev",
    limit=50,                 # cap pages so you don't drain credits
    scrape_options={"formats": ["markdown"]},
)
for page in job.data:
    print(page.metadata.source_url, len(page.markdown))
Enter fullscreen mode Exit fullscreen mode

Watch the credits: limit=50 costs up to 50 credits — 5% of your monthly budget in one call. Always set limit on the free tier.

Map: Instant Sitemap

Map returns every URL Firecrawl can find, fast, without scraping content — the cheapest way to understand a site before crawling.

res = firecrawl.map(url="https://firecrawl.dev", limit=100)
print(res.links)
# Then crawl only the subset you care about
Enter fullscreen mode Exit fullscreen mode

The efficient pattern: map a site, filter URLs to the section you want (e.g. /blog/), then scrape that list — far cheaper than a blind recursive crawl.

Search: Web Search + Full Content in One Call

Unlike a pure search API that returns snippets, Search can return the full scraped content of each result in the same response.

results = firecrawl.search(
    "best open source vector databases 2026",
    limit=5,
    scrape_options={"formats": ["markdown"]},
)
for r in results.web:
    print(r.title, r.url)
    print(r.markdown[:300])
Enter fullscreen mode Exit fullscreen mode

Ranked URLs and cleaned body text in one round-trip — no second fetch hop. Costs 2 credits per 10 results, plus 1 credit per page if you request content.

Extract: Structured JSON From Any Page

This is where Firecrawl pulls ahead of a plain scraper: run an LLM extraction pass and get JSON matching a schema — or just a prompt.

from pydantic import BaseModel

class Product(BaseModel):
    name: str
    price: float
    in_stock: bool

result = firecrawl.scrape(
    "https://example.com/product/123",
    formats=[{"type": "json", "schema": Product.model_json_schema()}],
)
print(result.json)   # {"name": "...", "price": 29.99, "in_stock": true}
Enter fullscreen mode Exit fullscreen mode

Or skip the schema and describe what you want with {"type": "json", "prompt": "Extract the top 5 story titles and points."}. For an agent, this collapses scrape + parse + validate into one typed result.

Firecrawl vs the Free Alternatives

Tool Free path JS render Structured extract Self-host Best for
Firecrawl 1,000 credits/mo, no card Yes Yes (schema/prompt) Yes (AGPL-3.0) Managed scrape + crawl for agents
Crawl4AI Fully free (OSS) Yes (Playwright) Yes (LLM strategy) Yes (Apache 2.0) Self-hosted, zero-cost, full control
Jina Reader Free, generous limit Yes Limited No (hosted) Dead-simple single-URL reads
ScrapingBee 1,000 credit trial Yes Via AI query No Traditional scraping at scale

Crawl4AI is the no-cost king if you’ll run Playwright yourself. Jina Reader (prefix any URL with r.jina.ai/) is the fastest single-page read but does no recursive crawl or schema extract. Firecrawl sits in the sweet spot: a managed API handling anti-bot and rendering, with crawl, map, search, and extraction in one place — plus the escape hatch of self-hosting when volume demands it.

Self-Hosting: Unlimited and Free

Firecrawl’s core is open source under AGPL-3.0 (SDKs are MIT), with a self-host guide on GitHub. Run the Docker Compose stack and you have your own instance with no per-page cost — just the server bill. Two honest caveats: AGPL is strong copyleft (modify it and offer it as a network service, and you must release your changes under the same license — a non-issue for internal use); and self-hosting means you own the proxy rotation, browser-pool scaling, and anti-bot upkeep the managed API handled. Self-host when volume makes the math obvious and you have ops capacity.

Firecrawl in an Agent Stack

The cleanest way to give an assistant scraping powers is the official firecrawl-mcp-server. Add it to Cursor, Claude Desktop, Cline, or any MCP client:

{
  "mcpServers": {
    "firecrawl": {
      "command": "npx",
      "args": ["-y", "firecrawl-mcp"],
      "env": { "FIRECRAWL_API_KEY": "fc-YOUR_KEY" }
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

For RAG, the common pattern is “crawl a site → chunk → embed → store in a vector DB.” Firecrawl handles the crawl; pair it with a free Cohere embedding API and a free vector database like Qdrant or Chroma for a complete free-tier ingestion stack.

Free-Tier Gotchas

  • Unbounded crawls drain credits instantly. A crawl with no limit can burn all 1,000 credits in one job. Always set limit; prefer map + filtered scrape.
  • 2 concurrent requests means no parallel fan-out. Fire 10 scrapes with asyncio.gather() and eight queue or error. Throttle to 2-at-a-time or upgrade (Hobby 5, Standard 50).
  • Credits don’t roll over. Unused credits reset each cycle; no banking a slow month.
  • Extraction quality depends on the page. The json format runs an LLM, so cluttered pages miss fields. Use a tight schema, keep only_main_content=True, and validate (Pydantic does this free).

Pair Firecrawl with a free LLM — Groq, Gemini (1M-token context for long pages), or Together AI — and the whole hobby-scale stack costs $0.

Frequently Asked Questions

Is Firecrawl really free?

Yes — the hosted free tier gives 1,000 credits (~1,000 page scrapes) per month with no credit card, and the open-source core is free to self-host without limit under AGPL-3.0. You only pay for managed scraping above 1,000 pages/month without running your own infrastructure.

What is 1 Firecrawl credit?

One credit equals one scraped page for scrape, crawl, and map. Search costs 2 credits per 10 results; browser interactions cost 2 credits per browser-minute. So the 1,000-credit free tier is effectively 1,000 page reads per month.

Does Firecrawl return structured data or just text?

Both. By default you get Markdown or HTML. Add the json format with a schema or natural-language prompt and Firecrawl runs an LLM extraction pass, returning typed JSON — names, prices, dates, whatever you define.

Firecrawl vs Crawl4AI — which should I use?

Use Firecrawl’s hosted API when you want scraping managed (anti-bot, browser pools, retries) and the free or Hobby tier covers your volume. Use Crawl4AI when you want zero per-page cost at any scale and are willing to run Playwright yourself. Firecrawl can also be self-hosted if you want its feature set without per-credit billing.

Bottom Line

Firecrawl turns messy live web pages into clean, structured data a model can use. The free tier — 1,000 page reads a month, no card, every endpoint unlocked — is enough to build a real research agent, RAG pipeline, or monitoring tool at zero cost.

  • Prototyping or personal agent? Free tier — 1,000 pages goes a long way with crawl limits set.
  • Small live project? The $16/mo Hobby plan (5,000 pages, 5 concurrent).
  • High, steady volume? Self-host the AGPL core for unlimited free scraping, or jump to Standard.

Related Reads

Originally published at toolfreebie.com.

Top comments (0)